On The Rating System

Posted by (twitter: @mdkess)
September 11th, 2012 9:44 pm

I think I’m having more fun rating and playing games in this contest than I did making mine – and I have to admit, it was pretty fun to make! There are a lot of very talented developers out there, I was quite surprised by the number of high quality entries into this contest. It has made me start thinking of the rating system though, and how it could be improved. As it stands, I think that the rating system is in need of some work – although I’m by no means claiming to be an authority about “the way things should be”.

First off, I think that there are too many categories. My thoughts on each one:

Overall: This rating is of course important. I think that it is the most important rating, and it should be visibly separated from the others. I also think that it should be described on the page as not being an average, rather being representative of the overall opinion of the game. When I’m rating games, I rate games on this based on my experience playing them, trying to separate the other parts.

Innovation: I also like this category – it rewards outside of the box thinkers. I’m not sure that innovation is the correct word, I think that uniqueness or something might better serve the purpose.

Fun: I think that this fits in with overall, although I do like this category, I found it hard to consistently rate games with it.

Theme: This one is really important for the spirit of the contest. I think that it should be listed beside Overall, as the two core ratings for a game.

Graphics: I don’t like this rating as it stands, since it seems to reward visual complexity rather than aesthetics, which I think is a mistake. One of my favorite games in terms of aesthetics is Collect by Amidos. Graphically it’s simple – most people in the contest have the skill to implement graphics as Amidos did – but aesthetically it’s incredibly well put together I think, subtle and tasteful. Yet this category seems that it would not reward that. Perhaps “visual style” would be a better name for it than aesthetics, although maybe we can come up with something better.

Audio: Another staple, I found this category the easiest to rate. More on this later.

Humor: I’m not sure about this category. I only gave a few games a rating within this category. I think that the problem with it is that it covers too few games – of the 70 games I’ve rated, I’ve given a humor score to maybe 5 of them. The other problem that I see with it is that it’s too exclusive to one style of game. While funny games get their own category, serious ones don’t. Scary ones don’t. I think that this category might be worth dropping altogether.

Mood: I didn’t rate any games in this category. I don’t know what its purpose is, and I don’t see how it captures something different than innovation, fun and overall. In this case, it seems too vague in its intent. Looking back at Ludum Dare 23 winners, it seems to be about some emotional response – which would be too specialized (and perhaps better labelled as “Emotion” than “Mood”).


So those are my thoughts on the rating system – nix Humor and Mood, rename Graphics to something that encompasses style instead of just graphics, and rearrange the order and emphasis on the page.


The other issue that I see is that there’s no guideline for ratings. I came up with my own system beforehand so I could be more consistent across games, but it might be nice to have a guideline beforehand to normalize things, so that people are more consistent with their ratings, but more to guide them. My system (not saying that this is the correct way, but just how I tried to stay fair in rating other peoples games). Going below three stars usually meant something wasn’t implemented:

Overall: Usually three stars. Four stars if an interesting twist, or good implementation. Five stars if particularly clever. Independent of other ratings.

Innovation: Three stars for a well implemented take on an old game. Four if there’s a twist – lots of that in this contest, with people taking interesting views on evolution. Five if it’s something I hadn’t seen before.

Fun: General feeling – I couldn’t really find a way to objectify “fun” (thankfully!), so if I enjoyed the game, I gave it a three. If I found myself hoping that it got developed post compo, 4. If I played it again, 5.

Graphics: I treated this in two ways – artistic skill – if you made something that obviously took a lot of technical skill, but also if you made something that was well put together, even if not graphically complex.

Audio: I think I had the best system for this. Baseline is 1 (N/A if no sound). +1 if you have sound effects. +1 if you have music. +1 if your sound is complete – most actions have an associated sound effect with them. +1 if the sound was really clever in some way. -1 if sound was abrasive for some

Humor/Mood: N/A if I liked your game. N/A otherwise.


Now, I’m not saying that my system is the correct one, but I think that it accomplishes my goal pretty well – namely to be as consistent as possible across rating games. Newgrounds has a system similar to this, if you hover over a rating, you get a guide:

0: Blam this piece of crap!

1:  This makes poop look good.

2: Nothing too new or interesting.

3: Not bad. In fact… I like it!

4: This Flash is crunk, fo shizzle!

5: Woot!!! All my 5 R belong to this!


Which seems pretty good, if maybe a bit immature – it encourages a score of 2-3 as the baseline, 4’s should be reasonably common while 5’s should be rare, which I think is healthy. I think that a similar guide to this could work well for the contest in keeping ratings consistent, per reviewer, across multiple games.


Alright, essay over! Agree? Disagree? What are your thoughts on the matter?

13 Responses to “On The Rating System”

  1. johnfn says:

    “While funny games get their own category, serious ones don’t. Scary ones don’t.” Yes they do. The Mood category is exactly what you want here. (Mood is to rate how scary, serious, or well-told a game was.) You might want to go back and re-rate games that you thought were serious or scary. :)

  2. hamster_mk_4 says:

    Fun, humor, and mood represent different ways a game can engage you. Besides I think Ludum Dare needs more categories, so there can be more winners, and more feedback

  3. Codexus says:

    We can’t really have guidelines as different people value different things in games, so it would be telling them the “right way” to think and I really don’t think we want to have such a closed minded approach.

    What I would personally change:

    Merge “Innovation” and “Theme” into one “Idea” category. I can tell if I like an idea but I can’t really tell if a game is genuinely innovative as I haven’t played all the games in the world. “Theme” is just too weak a category on its own.

    Merge “Humor” into “Mood”, we don’t need to have two categories for that.

    Bring back “Technical” in some way. And I’m not just saying that because its the only category that I won 😉 It was removed when engines were allowed on the basis that people wouldn’t be able to tell what was made by the contestant and what was there thanks to the engine. But with the current trend of minimalism in LD games I think we need something to encourage technical ambition again.

    When LD was created the idea of making a playable game in 48h was madness and likely to result in failure. Now this has become too common and I think we see too much conservativeness in terms of the scale of the games being made, focus on well polished minimalistic gameplay rather than insane ambition. Some maybe we should have an “Ambitious” category or something?

    • johnfn says:

      I do like the idea of an “ambition” category. It would highlight both technically impressive ideas and games that tried to do something out of the ordinary (that may not have worked). Actually the more I write about it the more I like it. This should totally be a thing. 😀

    • Maple says:

      Ambition doesn’t really sound like a good category to me, because, for example, how do you think someone would rate a failed MMORPG? Doesn’t that deserve 5 stars? I think if there would be such a category, it can’t be called “Ambition”, or, it would need to be clear what it means (perhaps questions should actually be given alongside the category as you are rating them, so that everyone is on the same page?).

      • Codexus says:

        Yeah, it has to reward some achievement not just that idea of making something big. But if somebody made a mmorpg of a sort I want to reward that even if it’s not original or fun or if it has ugly graphics and bugs. Just having done that prototype would be quite impressive.

  4. Maple says:

    > The other issue that I see is that there’s no guideline for ratings.
    There is indeed a guideline. It can be found here: http://www.ludumdare.com/compo/rules/ (scroll down to “Judging”)

    When I’m rating, I don’t think too much. Players don’t think too much when they are playing a game. They either enjoy certain aspects, or not, so I try to reflect this in the way I rate games, in the hopes that the ratings are more authentic. I don’t think too hard about the categories or try to split it up too much in my mind. What you will now read, is merely me trying to analyse the way I think about categories, but when I actually rate them, it’s pretty much by impulse.

    Here is how I think/feel about the different categories:

    I agree with Codexus on the “Theme” category being weak. The only person really able to know how well they did in Theme, is the person who made the game. In the Judging guidelines, for Theme, it says “How well an entry suits the theme. Do they perhaps do something creative or unexpected with the theme?”. The problem is, if a game is creative and unexpected in it’s use of the theme, people rate it low because it doesn’t seem to follow the theme, or it seems to have used the developer’s warped (when actually potentially creative) view of the theme as an excuse for their implementation of it. In other words, it’s impossible to know if a theme is creative or used as an excuse unless you are the developer, and as a player you can only rate the Theme category with how unoriginal and close to the literal theme as possible, so this is what people do. I personally have a lot of trouble rating Theme, because if it isn’t creative, then I should rate it low according to the Judging guidelines, and if it is creative, I don’t know weather it actually is, or if the theme is just used as an excuse for their game. Perhaps Theme should be rated according to how genuinely it feels like it follows the theme, while on the surface seeming completely unrelated to the theme. Otherwise I just can’t see someone’s Theme rating being helpful to them at all.

    So I know I’m talking a lot about the theme category… Here, have some more:
    I think themes are a big part of what makes Ludum Dare, so I don’t think the theme category should be taken away necessarily, but maybe something to just keep in mind, is that I don’t think Ludum Dare is here to get people to make sure their game follows the theme. Ludum Dare is here to get people excited about making games, and to actually make one. For some, the theme can be very off-putting because they can’t think of something to make under it. I know people who had an idea for another theme in the voting period, and were excited for it, but just didn’t participate because the winning theme discouraged their idea. While I like there being a focused theme, I think something needs to happen to compensate for people like that. I don’t know what… Perhaps it only needs to be as simple as a small message in the post that announces the theme. I dunno.

    I think this is fine, and too unlike theme to be merged. “Have you experienced this before? No? That’s what I thought.”

    -Mood and Humour-
    I don’t think these should be taken out or merged. A serious or emotion-provoking game can have funny parts or well placed humour. Furthermore, I think Mood is more like “How well did the game make you feel (negative or positive) the way that the game was trying to make you feel?” I think this is completely different to Humour. To me, Mood is a game-long thing, whereas Humour isn’t necessarily so. Humour to me is more like “The parts that were trying (or perhaps not necessarily trying) to be funny, were they funny?”

    Fun is… Well, I’ve been rating “Fun” the way that I would rate “Addictive” (but still sort of keeping “Fun” in mind, the same way that with “Audio” you think of music and sound effects separately), if it existed, because in the Judging guidelines it says something like “Did you look up at the clock and realise it was hours later?”. I think addictive, and fun, are two different things. You can be playing a game that is hard to put down (maybe you want to see the ending, so you trudge on), but it isn’t very fun to play. I’ve played many games like this. I think “Fun” is how great it feels to play. Sort of like the mood category, but not directed at how well the feeling of the game manifests within you, instead, how fun the mechanics are. With these thoughts, I believe an “Addictive” category should be added, which is how engaging the game was, and “Fun”, being how good the mechanics felt to play with. Surely Fun can affect Addictive, but a game can be addictive without being fun.

    -Graphics and Audio-
    I think these are fine as is. I don’t think these should be elaborated on or split up or anything.


    So I think I would have the categories presented something like this (perhaps questions should actually be given alongside the category as you are rating them, so that everyone is on the same page?):
    -Overall: Generally, how much did you like the experience?
    -Theme: How creative, and authentic did the creativeness of, the theme implementation feel?
    -Fun: How fun were the gameplay mechanics?
    -Addictive: How compelled were you to continue playing?
    -Mood: How close did you feel to the way the game was intending (or not so intending) to make you feel?
    -Humour: How humourous were the intended (or not so intended) parts or aspects of the game?
    -Graphics: How well did you feel the graphics enhanced your experience?
    -Audio: How well did you feel the audio enhanced your experience?

  5. sorceress says:


    Fun, Overall –> Quality
    Mood, Humor –> Emotion
    Graphics, Audio –> Aesthetics
    Theme, Innovation –> Idea

  6. I like the idea of “overall” and “theme” standing out against the others. Overall, because it determines the final rank, and theme, because that is a major part of this competition, often gets overlooked/outright ignored.

  7. csanyk says:

    I don’t like the idea of making many rating categories. The key is to have as few as needed, and no fewer. I think the existing ratings categories should be tweaked as follows:

    Eliminate Humor (merge it into mood).
    Add Controls.

    I find that I don’t have any problem at all with rating for Theme. If the theme is strongly present in the game, and especially if it’s done in a novel or unexpected way *that works*, it gets a good rating. Developers may on occasion do something too off the wall or subtle, trying to be Innovative and Themey, and fail. That’s OK. LD is for experimentation. And while I admire the attempt, I rate based on results.

    I think it pays off to be a little daring with the design, but remain conservative with respect to scope. Pull off what you know you can, or get ambitious and dare yourself to see if you can actually pull it off, both approaches are valid and there’s probably more reward in being ambitious and trying things you’ve not done before, but the cost of that approach is a higher chance of failure.

    The great thing about LD is that the cost of failure is nothing.

    The great thing about failure is that you learn from it, and build future successes out of it.

    So there’s absolutely no reason not to encourage people to do things that are likely to fail — as a developer you just have to understand that, and decide whether you’re in it to get the highest rating you can, or if you’re in it to do something you’ve never done before, and take a chance that it may be brililant or that it might not work.

  8. sfernald says:

    The ratings system is totally flawed and that’s just fine by me. That way, we don’t feel so bad when we lose 😉

  9. quill18 says:

    Humor DEFINITELY needs to go as a category. Does something like “mood” not cover humor, if the game is meant to be funny?

    Everything else is acceptable.

Leave a Reply

You must be logged in to post a comment.

[cache: storing page]