January 10th, 2012 1:43 am

I’m just curious about the formula to calculate the final score.

No offence but in the top 50 overall there are some games that, although they are good, they don’t are even near to the top 50 overall category. When I checked the amount of votes that games received, I found out that it a very little number.

So my thought is: if I only receive 5 of 891 of votes, and for some reason they are kind to me and give me an overrated score, do I get an advantage?

Another thought: next Ludum Dare will have for sure 1000+ entries, and as the number increases, the chances of getting less scores will be higher; there will be games with 5 ratings at the end of the voting. If those 5 votes are friends of mine who also decided to participate, and give me an overrated score, do I get an advantage?

What I’m asking here is if the current formula to calculate the final position takes into consideration the amount of votes an entry got.

If not, it would be cool to make something about it, because as the number of entries arises, it will be a less manageable community.

NOTE: I’m not saying that there was something wrong with the voting, and I don’t want you to think that I’m saying this because I only care about the score. This LD was great as always: overall there were really cool games; and I managed to finish a game that I really like, and got really good feedback, so I’m very happy with it. The worries comes because I’m afraid that, as the average number of votes for a game drops (as it is dropping as the number of entries arises), in the next LDs the games will have a score that doesn’t reflect the reality, or will reflect the opinion of just 1-5 voters.

### 22 Responses to “About the voting system”

1. Randomasta says:

Actually, the least rated games get 15 or so votes. I think that’s a pretty good number. I haven’t checked every games on the top 50 tho, so I can’t really say much.

2. There might be something to it.

Generally speaking, the more votes something gets the more it gets “evened out”. Less votes, howevery, have the ability to swing more extremely, but it can go in positive and negative directions.

The question is then, how woud one even /attempt/ to insert the number of votes into the voting-formula?

• Jorjon says:

Yes, you are right about the “swinging”. If someone didn’t like a game for X reason, and vote it 1, that will hurt the game if it ends with few votes. That will be true if the current formula is an average.
As for the attempt to use the weight of votes into the formula, I don’t really know. I bet there’s something in statistics, but that’s not my field.

• Jorjon says:

That’s a nice system, I’ll keep it in mind next time I have to do something like that. But I agree that 100 votes isn’t what the average game in LD is getting, and I’m not so sure a weighted average would work with 15 votes instead of 100.

• Kelly Thomas says:

Another technique used to normalize a scoring system where the entries will receive differing vote counts is summarized neatly here in the description of BBG rating: http://boardgamegeek.com/wiki/page/BoardGameGeek_FAQ#toc4

• caranha says:

I work with mathematical computing applied to statistics… Would you guys be interested if I ran some analysis on the data, and tried to find out a system that allows for more stability in the rankings?

3. PoV says:

It’s an average with the highest and lowest score removed, to balance out any one sided “I had my friend vote for me” or “I’m angry giving out 1’s to all 20 of my games”. That’s why we’re so aggressive about telling everyone to play and rate games. The more ratings everyone gets, the better of a scoring equilibrium it reaches.

Before the end of this LD, Phil finished up a significant update to the voting system. So for next time, the number of games you play and rate prioritizes you ahead of others. If you feel you’re not getting enough votes, the solution is simple: play and rate more games. This doesn’t mean people that rate a lot get all the votes, but it means that if the average gets 15 ratings, then those that rate lots will see 25+ ratings.

Yes technically that could mean the 15 vote person might be weighted ahead of the 25 vote person, but is that a risk you want to take? Hoping that your fewer votes result in a higher number? Statistical weighting goes both ways.

• Jorjon says:

That really clarifies the system for me, thanks!
I agree that with more ratings, the score will reflect the real value of the game, but seems a little harder every time (given the amount of 0% coolness this LD had).
Something I don’t get about the removal of highest and lowest score is: If I really like a game and rate if 5-stars, does the score gets removed?

• PoV says:

It’s only 1 highest and 1 lowest. This stops a single person from significantly skewing an entry (both ways). This is related to why we don’t show specific scores anymore, as it was upsetting people seeing 1,1,1,1,1,1,1,1 ratings in their games (whether it affected them or not). If we go one way (removing a single lowest), it’s only fair that we balance things out by going the other way (single highest).

You can see a bunch of statistical mumbling about why it’s better here:

• Jorjon says:

That’s really great, thanks for the explanation! I’m delightful to know that you keep working in the system to make it better every time!

• PoV says:

With thousands of potential participants, we have to do what we can to make sure everyone gets a fair chance. :). A totally different beast today than the early days of Ludum Dare.

• Jacic says:

“We do what we must, because we can…”

Thanks for the great explanation! I’ve been kinda wondering myself.

• Raptor85 says:

This actually answers one of my questions, I’ve run into games that have NO rating (my favorite Jam game actually has no rating) which I guess must mean of the 15 ratings it recieved only 1 or 2 actually gave it a score :/. Perhaps there should be a way to say “this game won’t run for me” that doesnt increase the rating count, as obviously a bunch of people who couldnt play the game were incrementing it higher makign it look like it had been rated when it hadnt?

4. SusanTheCat says:

I must say that you guys do an awesome job.

I am really impressed by the new rating page. The number of ratings for our game went from 15 to 38. We are very happy.

Susan

5. johnfn says:

On the topic of ratings:

I think that LD should normalize votes. I think that if one guy has a voting average of 2, and another one has an average of 4, clearly a 5 from both of them doesn’t mean the same thing. If your average is 2, then a 2 from you should be normalized to a 3; likewise with 4.

On the topic of opening voting to the public:

The only way to accomplish this would be to make direct links to games impossible. The way I imagine it, any voter would be presented with just one game to vote on. When they finished with that game, they would see a link to a new game, and they would repeat the process.

The important thing here is that you would be unable to see any game that you hadn’t already rated (+ one that you were about to rate). Clearly, celebrities couldn’t get an advantage because no one would be able to find their game (:P) Better, there would be no ‘popularity contest’ or ‘friend voting’ aspects.

• SusanTheCat says:

What about games that I can’t play because I don’t have an appropriate system? (i.e. I am using a Mac) You would have to have a means to “skip” games you can’t rate. All a devious person would need to do is “skip” until they got to a celebrity game. And for the Mac user, how many games will they “skip” before they yet annoyed and stop rating altogether?

Susan

• johnfn says:

SusanTheCat, I don’t really think this is a big deal. It would be pretty easy to ban users who have a ridiculous skip to rate ratio. Further, another feature I’m interested in is a way to say “I’m running Windows/Mac/Linux” before rating games, and game creators could mark which systems their game runs on, so that you would only get games that you could play.

• Jorjon says:

Wow that’s a really good idea, like a secret vote or something like that. That would fix many things, like posting on a forum “hey everyone check my game and give it good rating”. Of course after voting all games would be accessible for everyone (especially for the press) to view. Also there would be no more “Notch is making a 48 hs game! Check it out!” articles about “Ludum Dare”.

I don’t know if it would work here, but I like this system.

As for what Susan commented, I see she has a point, the system will suck if the game doesn’t work for you. What I can only imagine is a report button with a bunch of options: “[Mac|Linux|Windows version doesn’t work”, “Inappropriate content”, etc. When a game gets certain amount (from 1 to x) of reports about a feature like “Mac version doesn’t work”, make the game unavailable to that specific sector, and tell the developer about it. If you issued the report, you can skip the game, but the Mac user that will come after you won’t even see the game. That’s assuming you can detect Mac version with the User Agent string from browsers (because public people won’t use accounts).

• SonnyBone says:

If I’m going to devote days upon days to rate over 250 games, then i want the freedom to choose what I rate. At no point should Ludum Dare remove our freedom to rate the games that we are interested in playing.

• Jorjon says:

Maybe the system can show you 5 games in a row and you pick what you think is most interesting.

• johnfn says:

Perhaps you could see every game only if you were a participant in the contest; otherwise, you could only see one game at a time?

[cache: storing page]