With the recent poll on ratio re-evaluation for TU achievements, I thought I'd put something together to sum up my views.
First, in order to have it in one place, make it easier to understand and avoid contradiction; second to try and come up with helpful suggestions.
I do not have an answer to the problem as I am not certain what the problem is, but this would be how I'd go about it.
1. The problemRatio is used by people differently. Either as a rarity or a difficulty meter.
The rationale for rarity is that difficulty is subjective and can't be agreed on, so rarity is all that can be accurately measured.
The rationale for difficulty is that rarity is pointless and unfair.
Out of the two I don't personally take side, I do not care.
The problem further exacerbates because some people use comparisons between achievements in order to assess what they want to play or compare the value they give an achievement, which creates the following further issues:
-
rarity rationale: as it stands, ratios are impossible to compare from different games, so a ratio of 2 can be much rarer than a ratio of 5
-
difficulty rationale: as it is accepted being subjective but wants to be objective, it allows calculation to be based on relative rarity, which opens the door to criticism and calls to jump rationale boat; plus ‘relative’ is relative and creates debates around the subject and mixes completion into it
There are probably more issues on each sides, but no issue can discredit the rationale it is linked to. Instead it should prompt into questioning if the calculation is appropriate for its objective. In either case it seems safe to say it isn't. So how could it be fixed?
2. PredicatesSome ground rules have to be set or there is no point continuing. So let's assume and accept the following:
PR1 : The selection of site members is varied and representative of gamers
PR2 : Calculations should only use unequivocal data and never guess
PR3 : Cheaters are excluded from stats as per site rules (cf. Cheaters Policy)
I’ll say PR1 is a stretch about the representative bit, but not that bad that I’m not ready to accept it here.
3. Abbreviations & DefinitionsAs we will be using certain concepts a few times, let’s put them down to have an easy reference and type less.
TAU = TrueAchievements Users - Number of people having registered on the site
Game = Unique container ID that can include different types of items, being base, TU or DLC
GWA = Unique Gamers w/ Achievement - Number of TAU having unlocked the achievement
GWG = Unique Gamers w/ Game - Number of TAU having unlocked any achievement in the Game
GWI = Unique Gamers w/ Item - Number of TAU having unlocked any achievement in the item (where item can be either base game, TU or DLC)
Not wanting to get into what a 'Game' is and off-topic semantics, let's call here Game the overall container of a variety of content. This allows covering all cases in the new digital world where we can have free or non-free everything and where DLC can be played without having base container game.
So to be clear, it's important to understand the distinction between GWG and GWI. GWG is not the sum of GWI, only if not a single player has played more than one item is this true, otherwise
GWG <= A.GWI + B.GWI + C.GWI (where A can be base game, B TU1 & C DLC1)
I will assume TA is able to determine the number of unique players in each item or combination of items, given what it already does or plans to do.
4. Ratio Suggestions4.1 Rarity rationaleThis is probably the easiest.
Rarity is a number defining frequency of occurrence. It can be defined from a pool serving as a sample.
As such, it actually is not necessary to ponder if the fact that not every single player having an Xbox console isn't registered on the site, statistics tell us the number doesn't change (given PR1).
If I was advocating rarity as what ratio measures, I'd then probably say:
This provides a consistent ratio and allows for comparable numbers.
Rare games will have high ratios. It's not stupid. They don't 'deserve' it. It's not 'fair'. It simply is aligned to the definition of what the ratio is within the premise of the rationale of this section.
4.2 Difficulty rationaleThis is harder as I have to stick with the principle of calculating a ratio rather than suggest having it averaged from votes; after all we want to stay compliant with PR2.
So we have to accept that difficulty has to inject a bit of rarity to estimate a ratio and this will also allow me to stay away from a discussion about boosting and how this affects difficulty of obtaining an achievement.
Furthermore let’s tackle TU vs DLC here, because TU can be turned down or never prompted to a gamer depending on the case. As we are in a difficulty rationale, not rarity, this is not enough to warrant separation.
If we then accept all that for this section, let's then say we will be using relative rarity. We then have to pick one of two options: (GWI / GWA) or (GWG / GWA)
All depends on if we consider each item worth considering standalone or if we rather look at a Game as an entity that makes sense only as a whole.
Note the first is how ratios used to be calculated for everything some time ago before the DLC poll changed things.
Note also the second may be how the new TU poll proposes to change TU only calculation, although I'm not completely sure.
The reason for calculation changes has often been that certain non-base achievements don't have a 'fair' value. It was the reason for the previous DLC change, to use a geometric mean calculation and is often mentioned (with a range of examples) for the current discussion. All good and well-meant but the wrong moves as it adds complexity and steps away from consistency.
If the original calculation was felt inappropriate, the right solution should be identified and then implemented en masse, so as to maintain integrity and consistency (so my view back in the DLC poll, was that none of the available options were going to be the right answer).
Looking back at the two options we have, and given the way things have evolved, there is no way to know for sure the concept of a base game will remain prevalent. Like it or not it will probably go more the route of containers from which customers can pick and choose packages or dare I say IAPs.
Because of that, if I was advocating difficulty as what ratio measures, I'd then probably say:
This provides a consistent ratio and satisfies PR2.
Each item has the same relative rarity applied and is never stretched depending on the type of the item and a different lens being applied. Also prevents forcing all contents and ignoring accessibility limitations.
Unpopular items may have low ratios. It's not 'unfair'. It simply is aligned to the definition of what the ratio is within the premise of the rationale of this section.
The fact is, certain TU or DLC achievements stack on base game achievements and their ratio may look 'off'. I'm not disputing this but will handle this later. Either way, there is no ‘fairness’ to be talking about, if at the same time we claim and aim for objective stats.
5. The Fix ?Without taking sides for any meaning of the ratio, I’d suggest a change in any case. And I always applied a principle of consistency and not treat exceptions as a driver.
The fact is, some ratios will ‘feel’ ‘unfair’. Clearly if I need to do one thing 10 times for an achievement and the exact same thing 20 times for another, I’d want the 20 to be rewarded better than the 10 and also expect it to be rarer.
My suggestion is not to include that aspect blindly to a bag of achievements but create a new flag instead.
Some achievements are directly linked to other achievements, whilst others are completely independent. So how about considering that an achievement can be of a certain 'type', which could be managed by
a new flag designating a dependency? A new ratio calculation could then apply.
Now it could be simply adding the ratio of its predecessor but as the calculation would be completely different, instead I’d suggest the following:
If Ach1 belongs to Item1 and Ach2 belongs to Item2, and Ach1 is defined as a pre-requisite for Ach2, then:
The calculation would remain the same whether the achievements are both part of the same base game or DLC (becomes GWI/GWA) and so the idea of keeping things simple and consistent remains whilst awarding a seemingly more appropriate ratio to those that ‘deserve’ it.
Very little achievements would be impacted, keeping this an exception treatment, and no achievement that would not ‘deserve’ it would see it applied to it.
And as there already is a flag that affects ratio (unobtainables due to dev error) then it certainly is doable.
6. ConclusionAgain, I don’t claim to have the solution to something I’m not sure I understand given it doesn’t matter to me. But I reckon the above may be something to think about.
Ker
PS: Addendum regarding TA*** Spoiler - click to reveal ***May sound off topic at first but want to add a few quick comments on TA calculation.
In addition to the contentious way of how ratios are produced, there is potentially a similar reserve about the TA calculation multiplying that ratio with the standard GamerScore. Some view keeping an arbitrary value in the mix (GS), as 'dirtying' the TA as a result.
Now we need to keep in mind that a standard GamerScore being attached to the target does affect how people react and aim for it. The GS is like a carrot and the higher the prize, the more people will try to get it. The result is that for an hypothetical identical achievement in two games, if one game rewards more GS, it will likely end with a lower ratio (in particular in the difficulty rationale, a lot less impact in the rarity rationale).
So given GWA and GS are linked, if the Ratio includes GWA, and if the TA calculation includes the Ratio, it then makes sense to also use GS in the TA calculation.
This is what TA currently does.
Now how much of an impact the GS has on the ratio cannot be an exact science and you could certainly say it is not as simple as a straight multiplication, but it is at least clear the GS should feature somewhere in the TA formula.