The rating system is so flawed.

Posted by
January 7th, 2014 10:20 am

Non-game with unskippable dialog #27 over-all
http://www.ludumdare.com/compo/ludum-dare-28/?action=preview&uid=30207

My -game-, judged extremely harshly for unskippable dialog: #222 over-all
One Cigarette
http://www.ludumdare.com/compo/ludum-dare-28/?action=preview&uid=8361

Does the rating system not seem completely flawed to everyone else yet?

Don’t give me that all-inclusive “everyone’s a winner if they try” rainbows and hippie crap. Inclusiveness is for the Jam, COMPETITION is for the COMPETITION. Am I right or am I right?

There’s a problem. Let’s get it fixed… seriously.


23 Responses to “The rating system is so flawed.”

  1. KumoKairo says:

    I suggest you go killing for cigarets! :)

    But you’re right, your game is totally underrated!

  2. Zephyr Raine says:

    What the hell, man?
    Ludum-Dare is not about being rated or anything. What should matter are the feedbacks and comments. It’s all about making games in a short-time, not about getting high rankings and such. It’s not about being the winner, it’s about being creative.

    If you want COMPETITION for the COMPETITION, you came to the wrong place, and you didn’t get the spirit at all.

  3. Hmm, do you have any suggestions what could be done to improve the rating system? I guess what you want to say is that you would have liked/expected a better rating for your game, how do you think this could be achieved?
    No offense, I’m just curious… :)

  4. KumoKairo says:

    If it’s not about the competition – why would anyone create this rating system? :)

    That guy is really upset, and I understand him.
    He really did put a lot of effort in that game.

    People differ, if the results don’t matter to you, it doesn’t mean everyone should share the same thought 😉

  5. pighead10 says:

    The LD rating system is an informal tool to measure what people in general think of your game compared to other games. If you get a bunch of people rating your game who are too harsh, then unlucky, but if you want a fair competition and are playing to win, then Ludum Dare is not the place for you 😛

    There aren’t any moderators to check for cheating, everything about the rating system is completely subjective and there aren’t any prizes for winning. That’s Ludum Dare!

  6. Atmospherium says:

    Sadly, I would have to agree with how the community ranked your respective games.

    Monochrome isn’t exactly the first thing that comes to mind when you think of the word “game”, but that doesn’t that it’s not a game. It told a story and did so interactively and with creativity. The slow pacing of the text (which is skippable, BTW, if you click on another item) actually served to benefit the story being told. The player only moved at a certain speed anyways, so the player ability and the dialogue worked in tandem. Since the point of the game was the story, people recognized that and judged accordingly.

    One Cigarette isn’t at all a bad game, but it left the player (or at least me) with a lot of confusion. The focus was on finding the cigarette (or the other items) and so the slowdown in the dialogue prevented me from progressing towards the goal very quickly. Unlike in Monochrome, the player moves a whole lot quicker and has a goal other than the story. As a result, the slow text felt incredibly intrusive and interfered with my enjoyment of the game.

    Monochrome tried very few things and did them incredibly well. One Cigarette tried a lot of things and did most of them well, but fell short in a few places. That’s what the community saw and that’s how the community rated.

  7. joppiesaus says:

    I think Ludum Dare is’nt really a competition, somehow. I don’t know why.
    The “vote”-sysem is just informative, it can’t be checked for cheating!
    Don’t look *just* at the ratings, but also your comments!

    Anyways, I love your game!

  8. Alex Higgins says:

    I think that the problem isn’t the system itself, but with human error.
    For example, I’ve notice that, if two games have similarly well made graphics, but the first has better gameplay and audio, then the first will get a substantially higher average score for its graphics than the second. The evaluation of each particular element of a game depends upon the quality of the rest of its elements. It’s bullshit, but it’s not uncommon when people evaluate art/music/games in general.

    Or, if two games are of similar overall quality, but one is hyped-up in people’s blog posts on the Ludum Dare front page, then people are more predisposed to rate the one higher.

    Everywhere, you see shitty art that is overrated and brilliant art that is underrated. I’ll admit that the LD rating system is kind of unfair in that each game is judged by a different panel of varying sizes and personalities, but otherwise, the issue you’re describing seem to be universal to the evaluation of art in general.

    However great your game is, its greatness is almost guaranteed to be, to some extent, irrelevant to the numeric score other people give to it. Determine for yourself how good it is, taking in other people’s feedback to some degree, but always with a grain of salt. If you think it was a good game, then I’d consider that to be a success!

  9. udo says:

    I don’t think the rating system produces anything of value beyond a rudimentary measure for yourself. With some of the high-ranking games it’s especially difficult to see WHY they score so highly. I think that’s because a lot of other factors go into the ranking system, such as how well-known you are, if and how well you advertise, if you have friends participating in LD, and what the general expectation of the community is around you.

    On the other hand, I found the ratings to be an accurate feedback to compare my games among themselves. For instance, I really slipped in the ratings this time, and I have to say it’s fair compared with my previous game. Do I feel fairly ranked? Probably not. There are lots of far worse entries than mine who got far better scores, AND there are far better entries that got ranked way too low.

    I really think the scores are beyond meaningless and unfair if you see it as a competition with the other game-makers. It only works if you see it as a competition with your earlier self.

  10. ashdnazg says:

    The rating system is just another feedback you get about your game.
    You’re right that the fact that another game got a better rating than yours doesn’t necessarily mean that game is better (although usually it does).
    However, rating discrepancies happen for a reason, it might be due to exposure, it might be because people can’t appreciate art or any other reason.

    Now you can say: “My game is better, and people are morons!”
    Or you can say: “Hey, why did people rate me differently? Could anything be done to change that in the future? If so, do I really want to make these changes?”

  11. Shigor says:

    The rating system worked when there were less entries and most of the participants had a chance to test most of the games. Now it depends quiet a lot on who gets to rate your game – you may be lucky and hit ten raters who liked your game and rate it accordingly. You may won’t get that much luck and may hit on ten people who just click, look, scratch their head being too confused by something which requires more then 10 seconds of their time and they gave you some low rating and move on… or you may have just the bad luck that there’s another game in compo similar to yours that was played by rater just before…

    If you’re well known, have friends who’ll rate your game, or you yourself rate a lot of game, you’ll rack more rates and the results are a bit more “real” – doesn’t help you with the competition which can climb over you thanks to the luck with voters.
    Also what Alex Higgins said is extremely true – the games with good ideas / one part of the presentation tend to be scored much better even in the categories where they suck.

    Another thing is that almost constantly simple “not a game” or very very simple (bordering on stupid) games with nice presentation / polish and gameplay counted in seconds rate much higher than generally better and bit more complicated games that requires player to spend at least few minutes to look around and find out what is going on – if you want to “win”, do something simple, with either great idea (which doesn’t come easily, but generally games with very good and/or innovative ideas tend to get good votes) or nice and polished, if very simple and stupid.

    There’s nothing you can do about it though, the best you can do is pretty much ignore the rates and concentrate on what people say in comments – best 5 games in categories usually deserve their place, then it gets pretty fuzzy.

    The only “fair” system that would be more objective would be if either everyone rate everyones games, OR if there was a panel of “pros” who would rate everything and that’s pretty much impossible with the number of the games now in compo – first few LD’s I’ve attended I’ve tried to play and rate everything and even then with few hundreds at most it took me too much time.

    So… either give up on voting and do it for fun (that’s after all the main reason the LD exists and most of the people participates in it), or don’t do it at all, or maybe find some smaller compo where you have a better chance to be rated objectively.

    Yes, I know it sucks, but that’s life for you…

    (BTW your game was one of the few I’ve rated out of one hundred that got good marks :)

  12. Shigor says:

    I actually spent some time thinking about how the rating system could be improved to be more objective, but… in this way LD is killed by its own success :) Rating in more rounds? (Say half the games that fail in the first round are out and then another round is taken). Versus matches?

    Just not possible with the 2000 games :/

  13. Shigor says:

    It’s also reason why there’s no organized voting and winners in the GlobalGameJam – local sites usually have their own winners, but when there’s maybe 10 teams in a location, it’s possible.

  14. sorceress says:

    Non-game with unskippable dialog #27 over-all.
    My -game-, judged extremely harshly for unskippable dialog: #222 over-all

    I don’t think the relationship between “what your game has”, and “what result you get” is ever that direct or logical. As the old saying goes: it’s not what you have, but what you do with it that counts. There’s much more depth and subtlety in judgement.

  15. Vi-King Games says:

    Well, you certainly have humor.

  16. foumart says:

    Well, maybe there is a way to organize games by genre. And rate games in the genre you are/prefer – in order to be more objective (and receive proficient rates accordingly). I took my time this LD to play some dungeon crawlers and puzzles and even spent hours on some. There are games that just need more than 4-5 min to judge like “Dungeoneer” or “One to move them all” for example. The current rating system works only for some of the games, others seems to be always underrated.

    Sadly if you read the comments of some games (check Dungeoneer’s) it’s clear that some people commented without even knowing what they’re playing. I think the main reason for the rating system discrepancy are the unproper or lazy rates.

  17. SiENcE says:

    Well, what matters me is not the rating system.

    What matters me is, that some titles can’t really be done by one man in 48h.

    There are games with ~100kb gamecode (not engine), hundrets of graphic assets, lots of sound effects and music … polished to a very high level.

    Don’t get me wrong…as long there is a timelapse for this entries…everything is ok.

    I can’t really understand why people doing such a lot of pre-ludum dare work, just to get a higher rating. They don’t understand, why gamejams are gamejams and not “I made the best game ever!”.

    so long…

    • sorceress says:

      I don’t think 100kB of code would be impossible. If your game is code heavy, with very little time spent on graphics or anything else, then it wouldn’t be so surprising if this much code is written. It’s only 4kB per hour, given 12 hours per day coding.

      Remember too that it will be assisted by tab-completion, and copy-pasting many variable/function names, and duplicating other little chunks of code to save typing out similar algorithms twice. So it might ultimately be less than half that number of key presses.

  18. SonnyBone says:

    To SiENcE: You have no evidence. You just can’t imagine it being true because your skill level isn’t at that point. Either realize that it IS possible and push yourself to get to that point, or stop making games. If you want to accuse someone of cheating, then please make a post listing all of the games you suspect were created before the compo and be prepared for a bunch of people to prove you wrong.

    To Suese: You always complain but you always come back.

    Here’s the number 1 problem with Ludum Dare: It’s simply too big and there are just too many games to rate. You can’t possibly rate every game, so the voting is always going to be different from game to game. Every game has a different panel of judges. But there is no other way to do it. This is the best way possible given the number of entries. If you want a different experience, then join a different jam. Ludum Dare will never be able to return to the days when there were only 100 or so entries. If you want ‘fair’ and ‘even’ rating system or you don’t like the rules, then you’ll just have to join another jam or make your own.

    I’ve seen games that placed in the top 100 categorical lists that violated long-standing compo rules. To me, that’s far more frustrating than anything else. It’s a combination of there being too many games (things going under the radar), people rushing through games instead of taking their time (in an effort to get more ratings on their own games), and a lack of understanding coming from entrants AND judges (not realizing the differences between compo and jam).

    I’m done with Ludum Dare for now. It’s mostly because I’m going to focus on other projects, but also because it’s just too big for me to get anything out of any more. I’ve already learned so much since LD16, and now I feel like it’s time to seriously apply those lessons to something more personal to me. There was a time when I made games to try and please the judges. When I won medals, I then decided to just make games to please myself. Now I’m in that phase where I want to do my own thing for my own reasons and not worry about what anyone else thinks about my work, so I’m going to go off and do that.

    Maybe you should go off and do your own thing since you don’t seem to ever agree with how Ludum Dare meshes with your personal game jam philosophies.

    • SiENcE says:

      But i’m not talking about code on it’s own. I talk about gamedesign, lots of graphics, codedesign aso. This takes time, a lot more time than implementation and I know a lot entires where you can’t tell me that all this was done in 48 hours! Compare some games from well know developers (high ranked with timelapse videos) to some other games (where people often write comment like: ‘very impressiv work for so little time’).

      I often participate in gamejams with lots of people and afterwards I always look at the sources for many games. I think I really know what can be done and what not.

      I don’t want to blame someone. But I think people should know that this is not the way LD is meant to be!

      I think for me, the Jam is more relaxed :).

      cheers

  19. jfroco says:

    I think I’ve noticed that there are some strong trends in LD,

    1) Gameplay: interactive stories (mostly platform games that tell a story, I should probably say) are always well received (even when the stories are super linear)… personally I don’t like that kind of game.. a game when you can’t win or lose is not a game for me… but that’s my opinion.

    2) Graphics: I don’t like the style of “Sword and Sorcery” but the community seems to love it… see the graphics category winners for several LD. If the game looks “great” it probably will receive a better rate in fun and overall.

    3) Web games: everybody prefers to rate web games rather than downloable games. Obviously.

    4) Music: chiptunes are more popular I think. If you sing and say “ludum dare” or an internal joke in your song you are going to get 5 stars.

    So, if I would like to have a better ranking I should probably follow these tendencies, but I would probably end up with a game that I wouldn’t like, and for me LD is the opportunity to escape from real life for a couple of days and do what I love: making games.

  20. ENDESGA says:

    LD is taken way too seriously. Not the competition, but the fact that everyone is trying to make profesh games.
    I compete in LD because I want to show the world my style and my games, I also compete because I learn so much.

    To me, there’s no difference between 100th place and 1st place. Yeah, it would feel pretty great to be number one but, that’s not the point of LD. The point is mostly for you to get better at making games and to get used to people judging your games.

    The way your post is presented, you sound like a sore loser, mate. How about, it doesn’t matter, and better luck next time.

Leave a Reply

You must be logged in to post a comment.

[cache: storing page]