How Judging Works

Posted by (twitter: @philhassey)
April 24th, 2012 12:31 pm

Ahoy there, thanks so much for joining in the 10th Anniversary Ludum Dare! I gotta say, it was an amazing experience to see all 1400+ entries being submitted to the site!

The judging is pretty much the same as always. You judge entries, people judge you.

The big change this year is that now those who judge more entries will get more people judging their entry. I recommend starting judging a few entries now and keep judging them throughout the next three weeks to ensure your entry gets a good handful of ratings. Try to judge AT LEAST 25 entries. Please note that stuffing fake ratings is 100% uncool.

Also, coolness has changed: 50 ratings gets you bronze. 75 gets you silver. 100 gets you gold. Above that, you’re just awesome :) Be sure to make a post about your favorite entries to highlight the best-of-the-best!

Lastly, if you attempt to play an entry, but can’t, please just leave a comment to the user stating that you can’t judge their entry and explain why. “I don’t have a Mac” or “I can’t install Java” or whatever are all fine reasons. It will help them know how to create an entry that even more people can play next time!


69 Responses to “How Judging Works”

  1. Pierrec says:

    Woooaa Gold coolness is easyyy
    I was expecting to rate 200/300 games! Will there be some platinium medals ? ^^ Oh wait! No! Adamantium medals! No wait! Chocolate Medals!

  2. ghRibacki says:

    That’s awesome! I expect to judge as many game as possible, but some of them are too good that you don’t want to play anything else!

    Also, I think the new rule is good, less people will stuff fake ratings, I guess.

  3. Raptor85 says:

    Question: When we attempt to play an entry but cant due to platform limitations and we comment do we hit save or post comment. I’ve been just doing post comment since it doesnt “pad” the coolness rating (unless that changed this year) but I would REALLY like to clear games I can’t play off my list and the only way to do that would be to hit “save”. By the end of the first 150 I rated I had to really hunt and peck around all the games I couldnt quite remember If i already tried or not since they pretty much take up my entire front page “random” 20 now.

  4. Codexus says:

    Also please always leave a comment if you rate a game. Even if it’s just to say ‘hi, I tried your game’ it’s always appreciated to see that people are actually playing one’s game. :)

  5. mildmojo says:

    I agree w/Raptor. It’s not clear where the “Comments” below the rating go. Maybe call this “Ratings Feedback” or move it beside the ratings or somehow indicate that it’s part of the “feedback card” we’re filling out, and it won’t post to the section marked “Comments” directly below the box?

  6. Jigxor says:

    Now that I’ve got some free time, I’m trying to rate as many games as I can this time around! Doing a blog post on the best games is a great idea :)

  7. Cosmologicon says:

    Thank you for this new system. It’s excellent. I’ve already got as many ratings as I did for the whole competition last time.

    Since I’m on Linux, I want to focus on rating the Linux entries, to show them some love. I probably *could* get Windows entries working, but I know they have a much better chance of being rated than Linux entries. Should I leave a comment saying I’m focusing on Linux entries, or would that be “gaming the system”? Thanks!

  8. johnfn says:

    This new system is great. It’s fantastic to see the reviews come in when I rate a ton of other entries :) Great work!

  9. mderoy says:

    Any way for people that worked in teams for the jam to judge entries? I’d like to contribute to the judging, but the submission is under my partner’s account.

  10. AlwaysGeeky says:

    It’s so rewarding reading all the comments that others post about your game, especially since everyone seems so positive and nice. Also that encourages me to play more games too.

    I feel obliged to at least play the games of the peopel that have commented and played my game, which is a good thing 😀

    • derosa says:

      This! I’m so happy to get comments! This is my first LD entry, and I could not imagine I would get this kind of attention and comments!

      I also try to rate all the game that my commenters did. I can’t play the Windows games, but that’s a minority.

  11. Jedi says:

    Wait, I don’t have to whore out my game on IRC? :)

  12. It would be great to have an torrent with all games available for download. It’s so sad that some games from former LDs aren’t available anymore.

    My slogan: “Save the Games”!

  13. ratboy2713 says:

    Thank you so much for re-vamping the voting system. I already have more ratings than I did all last competition.

    Also, I noticed that there no longer is a community stat for voting. Is that being worked out some other way, or did you guys drop it?

    • PoV says:

      Community was a stat for people that did well in blogging/doing ancillary things with their game. Many people were scoring highly in that category, without even a single post. Not to mention, scores in community seemed to parallel other category results, so since it wasn’t working we removed it.

      • Jedi says:

        It’s sad that the community rating had to go. I really enjoyed thinking about how I could become more entrenched within the community.

        Can you please update the Rules and Guide? Community is still listed as a category.

  14. dr_soda says:

    So what happens if 1/3 of the comments on my entry are like “I don’t have an Android so I can’t play the game” and those people used the “save” button to do so?

    Does it effect rankings if they didn’t post any scores?

  15. Drabiter says:

    I personally never leave comment for rank. I comment separately with buttom comment form.

  16. nuprahtor says:

    >The big change this year is that now those who judge more entries will get more people judging their entry. I recommend starting judging a few entries now and keep judging them throughout the next three weeks to ensure your entry gets a good handful of ratings. Try to judge AT LEAST 25 entries. Please note that stuffing fake ratings is 100% uncool.

    Oh, this is not so good for people with bad internet connection, but, well, I have plenty time to fix it, so I will judge games in a week or something.

    • Raptor85 says:

      25 ratings over 20 days isn’t a big deal, and most of the games are under 2-3 megs so even 100 ratings in 20 days shouldn’t be an issue. (actually, a LOT of games are even under 1 meg, especially the html5/flash ones).

      If you’re rating games at all though you’ll be fairly visible, looks like still around 60-70% have rated between 0 and 1 games so rate 2 games and you’re more visible than 900 entries.

    • Cosmologicon says:

      In the time it takes you to play a game and post a rating, you could probably have downloaded the next game already. But yeah it would be nice if people put the download size on the entry page.

  17. Danik says:

    This is a great improvement as it will make people vote more, good job!

  18. Suese says:


    Suese here. Writer of Tiny Civilization

    After speaking with people on chat and gathering people’s opinions as well as my own, I want to suggest some ideas here, post-mortem style:

    1.Need category “Presentation”: much akin to plating in cooking competitions, this component is of big importants. Games that present themselves well, run without hassle etc, should get some favourable points for that category stand-alone. Some games can be brilliant without presenting well, and vice versa, presenting well but lacking substance.

    2.Need category “Replay value”: this is definitely the corner-stone of GREAT games, this seperates the fun and flashy from the GREATS.

    3. Need category “Completeness”: Lots of games are amazing but too short, feeling kind of like the story line was cut-off. Other games you sit there playing in amazement, wonder how so content was put into it in just a weekend.

    4. Ditch “overall”, overall score should be the sum of stars or the average number of stars per category.

    I would not be worried about changing all of the categories at once. The rest of the community will enjoy the need to adapt.

    So to re-cap the new themes would be:

    Music & Audio

    Overall = (Fun + Innovation + Presentation + Graphics + Replay-Value + Audio + Completeness ) / 7

    Next, I think the star ratings are arbitrary and should be presented in a questionaire format that defines what each star-rating means. For example:

    Music & Audio :

    0 stars – No music/audio.
    1 star – competitor put a minimal effort into this and it is bad.
    2 stars – the competitor obviously put some effort into it but it’s not very good.
    3 stars – the competitor put effort into it and it’s decent.
    4 stars – music and audio were fitting and added to the game experience.
    5 stars – extreme: anything beyond.

    I think most people thirst for this kind of upgrade to the challenge.

    Finally, this is on a personal note, I can’t stress enough that the Theme system is too wide-open and it’s also too easy to cheat and make changes after the competition. I suggest a few solutions:

    o – Along with the theme, a second randomly selected modifier, some kind of artistic or technical limitation. See Rule-o-matic voting system for my FAVORITE example of how to vote on a randomly-selected theme. This lends itself to extremely intriguing and innovative possibilities such the Theme ‘summer’ mixed with the modifier ‘Your game must use Conway’s game of life as a key mechanic’, or ‘Dungeons’ mixed with “Your game may only use 4 colours maximum on a 128×128 pixel screen”. The possibilities are endless and it also further ensures that people can’t grab a head start on their game due to the added level of uncertainty.

    o – Create an upload bin so entries cannot be tampered with after submission.

    o – Embellish the culture of the theme a bit more, take a few days to prepare a presentation before revealing the final theme to the public. I didn’t realize ‘tiny circle worlds’ were a genre of video game until 5 hours of sleep AFTER the competition. When the competition begins, we should at least have some background and perhaps some visuals about the culture behind the theme.


    • ratboy2713 says:

      I don’t agree with the added modifier, though that could just be that both of your examples give very limited games. How many different games can really be made from the game of life? I’ve seen a few and they are normally just simple puzzle games with a little variation. I think that would kill the creativity and the spirit of the contest. The same with the culture of the theme: some people took it as literal small planets, and some people took it as a small subsection of the present world. If you put out a video saying “this is what a tiny world is” you are limiting the creativity of the designers. True, there are going to be people that make games very vague so they can fit any theme, but it is a self correcting system. If you see that, you vote them down and move on.

      I agree with your proposal for a standard for voting practices, but I like the vague-ness of the theme system because I think it creates a very diverse game set. I would hate to play 1400 of the same game because the system limited the designers creativity.

      • Suese says:

        I’ve seen action games that incorporate the game of life and I’ve seen puzzlers. This is of coarse just a tiny example, if you look at how rule-o-matic works the community still votes on the modifiers and they are chosen at random from amongst the best.

        The problem with JUST a theme in my opinion is that it’s JUST too easy to get started or have something half-started before-hand and just modify it last-minute to fit the theme.

      • Suese says:

        p.s. Artistic limitations are a big part of what makes the challenge fun. 48 hours, stick to a theme. Yes limiting, but not challenging enough for the creative mind. If my idea is too far of a stretch maybe the modifier could be a bonus, meant for hard-cores who want an added challenge.

      • Suese says:

        added creative challenge*

        • Shigor says:

          I think GGJ way would be better then just force modifier on player. Set several possible modifiers and let author choose which one (if any) would be used.

  19. MadGnomeGamer says:

    I’ve earned my bronze medal! Going for gold.

  20. Tom 7 says:

    This system is great. It addresses most of my misgivings about the rating system and the sort order biases. Good work! :)

    I have some complaints left but I’m not even going to say ’em because this is so much better!

  21. digital_sorceress says:

    I think the D=R-C system has some problems.

    The formula appears to be: D = 50 + R – 5*SQRT[C’] where C’=Coolness, capped at 100

    The term 5*SQRT[C’] is what earns us *bonus* ratings above those who choose to rate no games at all.

    But the problem is that for CoolnessC bonus rating in return. Think of the economics of that… those who rate between 1 and 25 games – even though they are giving something, they will be taking more back from the community than they have contributed.

    One consequence is that there’s a high risk that people with zero coolness will receive an absurdly low number of ratings.

    I think the formula should be adjusted to D = 50 + R – C’/2.

    This way, for every C games we rate, will be earning C/2 bonus ratings in return: ie, for every two we rate, one we earn back, and one we don’t. And the one we don’t will benefit the community.

    • digital_sorceress says:

      Bad formatting. That should read:

      But the problem is that for Coolness less than 25, the number of games we rate (C) are earning us more than C bonus rating in return.

      • Shigor says:

        Yeah, I was thinking about writing something similar. While I do love the new voting system because it gets me more victims… errr, players, there are tons of games with just one rate. There’s still 16 days for judging, but unless a lot more people will start voting at least a bit, we might end with half of the games with just five or less votes.

        • digital_sorceress says:

          Yes. Having some games with very many votes and some with very few votes isn’t a good outcome.

          It’d be like baking a cake where you measure your sugar to the exact gram, but then adding a couple of crudely measured handfuls of flour.

          You have to use the same precision with all of your ingredients, otherwise the outcome will be only as precise as the crudest measurement.

          The outcome in our case being the final rankings table, and how well it has stabilised. No matter how thoroughly game “A” has been rated, it will mean nothing if game “B” has accrued only one or two ratings. How can you possibly compare those two games and decide which should be higher placed than the other?

          • Shigor says:

            Well I guess we could always return to previous system with truly random game list for rating with hope that enough people would rate….
            Or don’t put games with less then say five rates in final ladder. Five votes for the game is bare minimum, I’d rather see at least ten for game for some statistically meaningful results…

            • digital_sorceress says:

              Between the old random system and the current system, for a short time we sorted games by least ratings, which I think was the best of the three systems.

              I understand how the current system aims to encourage more voting by rewarding those who do. It’s just that the reward is so big, that it is amplifying the problems it is intending to solve.

              To address this, and building upon my previous formula, we could increase the priority of games with R less than 20:

              D = 50 + R – C’/2

              if R is less than 20, then P = (20-R) else P=0
              Then let C’ = C + P, capped at 100.

              • Raptor85 says:

                Any game with <20 votes that's not already showing up on the first page is because they're not bothering to rate any other games themselves.

              • digital_sorceress says:

                We have to accept that not everyone will bother to rate games. At the moment, those people’s games all but ignored with just 1 or 2 votes.

                You must realise that ALL games still need to be fairly rated in order to build a credible ladder. The system I’ve outlined should achieve this better than the current system, while fairly rewarding those who are rating games.

              • Milo says:

                I think that the equation:

                D = 50 + R – min(C,100)/2

                is adequate – when this is in a system, it will naturally bring up games with few ratings. Whenever someone rates a game in this system, their own D rating decreases by 1/2, but someone else’s D rating increases by 1. So, there is a net increase of 1/2 in the system. This means that, as time goes on, the total of everyone’s D ratings will increase, and thus, the minimum D rating should increase as well.

                Say the average number of games that each person rates is 10. This would mean that the average D rating would be 55, and since the disparity in D ratings should be relatively small, the minimum D rating might be close to 55 as well. Then, all games would get a minimum of around 5 ratings.

                More generally, we just need something of the form:

                D = R – B(C)

                Where R is the number of ratings, B is the “bonus function”, taking coolness and yielding the number of extra ratings a user gets. The expression C – B(C) is the number of votes that a user generated (C) minus the number of votes they consumed (B(C)) – it’s the net change in the system affected by each user. It should be positive so that people who rated nothing get to use the extra votes generated by the system.

    • Codexus says:

      I think it’s a good system.

      Those who do rate games get rewarded, and like any good leveling system you get rewards quickly at first so you get a taste for it and then have to work a bit harder for the next ones.

      I have no problems with people who don’t rate *at all* not getting much feedback. As you said even a few ratings will get you a fair number of ratings back. But not doing it at all means they are not interested in that aspect of ludum dare. It should be however made clear why they don’t get much feedback.

      I think there is a minimum number of votes to be in a category, so they would be ‘unrated’. But that might not even happen since people can also look at the list sorted by absolute number of ratings received.

      We can’t expect everyone to have the time to rate many games, but anyone should be able to test 10 games if they really want to.

  22. SusanTheCat says:

    I love the new system!

    It is a little disconcerting to see my own game on my rating list. :/


  23. Nehmulos says:

    Hi, I just wanted to note that in the beginning I was a little bit confused by having two comment boxes just next to each other (the one for the vote and the general comment).

    Maybe the “Save” button could be relabeled to “Save Vote” so that it would be obvious, that your vote will only be saved once you click that button and it’s a button to save a comment.

  24. 7Soul says:

    I miss the old list, where I could see how many votes the most popular games have…I mean, I got a lot of ratings and I’m thinking “wow, this is the most I ever got, I wonder how everybody else is doing”

  25. deepnight says:

    The coolness rule is a really really good idea :) Nice judging gameplay!

  26. madflame991 says:

    I still don’t get it: what does N/A stand for? 0 or null (in an SQL-ish sense, I mean, if it’s 0 then it will influence the mean, if it’s null it will not)?

  27. onefineline says:

    Soooo, Ludum 24 is coming to a close… when’s the next mini-LD?? I didn’t know where else to post this! =)

  28. MadGnomeGamer says:

    The single highest and lowest ratings are deleted before the average is taken…I like that! Although there goes my plan of creating a game for a platform that only me and my freind have, and us both rating it a 5 in everything. 😛 Just kidding.

  29. kibertoad says:

    Does anybody know when the results will show up?

Leave a Reply

You must be logged in to post a comment.

[cache: storing page]