What’s going on here?

Posted by (twitter: @_pretzelhands)
July 30th, 2012 3:00 am

I recently noticed, we have a ton of posts ranting about the rating system.

Either I missed out on the other compos, or people suddenly started disliking it?

Why now? I’ve been here for almost 6 compos now, and I very rarely heard someone complain about the rating system. Ludum Dare isn’t about winning after all! So, basically if we go for actual LD-esque rating there would be only one solution:

We disable the ratings altogether, and people just give their opinions in the comments.

If it would work like this, we have the basics of LD summed up in the rating. I quote Mike (PoV) here now:

“Our (Ludum Dare’s) role is to start something, to plant the seed of that next great indie game..”
– Mike Kasprzak, Ludum Dare 18 Keynote

And comments are perfect feedback for that:

  • You can point out what’s good and what’s bad.
  • You can make suggestions on what to change or add.
  • You can point out any bug that’s in the game.

Thus, I conclude: Rather than changing the rating system, you could just disable it. People get their feedback in the comments, and there is no prize for winning anyway.

Now, let the flaming, which points in my general direction, begin.


27 Responses to “What’s going on here?”

  1. digital_sorceress says:

    Either I missed out on the other compos, or people suddenly started disliking it? Why now? I’ve been here for almost 6 compos now, and I very rarely heard someone complain about the rating system.

    Concerns were brought up in both December and April iirc, so I wouldn’t say it is sudden.

    LD is much bigger now than it was 2 years ago, and the problems with the rating system stem from the increasing size When only 100-200 people took part, the rating system was good enough, but now that 1000+ people take part, there are serious noise and stability problems that have emerged.

    Also, the growing community means there are are growing number of people here who know about statistics, who will recognise and point out these problems.

  2. Raptor85 says:

    What’s the fun in a contest if you can’t judge how well you did against the other participants? Making a game in 48 hours based on a theme is an easy task, but rankings creates a driving force to squeeze in as much as possible due to competitive spirit, without it ludum dare would be boring and somewhat pointless.

    I mean, when playing deathmatch in Quake there’s no “prize” to be won either but if the game didn’t keep score it wouldn’t be much fun, don’t need prizes for a competition to be fun, but you do need some way to rank players…but it has to be a FAIR way!

  3. Kvisle says:

    I think the current rating system is sufficient. But I really don’t want it to be disabled – I use the ratings of my own entries to measure my own performance.

    I think it’s nice to know that I’m not getting dumber, but the other way around. :)

  4. AlwaysGeeky says:

    I agree with this sentiment.

    If it is a choice between a broken scoring system that doesn’t work and is provable to produce incorrect results, or no scoring at all… I would chose the no scoring at all.

  5. mohammad says:

    remove the ratings: and what would make ludum dare a “contest” if there’s no way to win?
    then we’d be just like any other online gaming website,where we post games and hope somebody gets their eye caught by it. ludum dare was supposed to be a contest,now without ratings it just a “post your games here when the date comes by” like newgrounds, just post and hope somebody will see it. we just need voulanteer judges who would be happy to vote for each and every game. all we need to do is get the word out.

  6. Khayet says:

    Please don’t disable the ratings. As a newb to making games, I find that a rating for my game, given by other designers/developers is a great way to motivate me. It makes a game of game making! It produces role models! It gives goals that can be achieved! It gives a measure on the ladder I am trying to climb!
    I am completely open to any reform of the rating system, but PLEASE don’t disable it.

  7. Peter Cooper says:

    I am not a good game developer but please keep the ratings. I use them to see how well I am doing in my development as a game developer. My first game came at just under the 50th percentile and my second entry at around the 66th percentile. I don’t care about “winning” as such but improving my percentile is a big motivation. It’s not about beating other people for me but having a target in mind and a way to measure my performance.

    Second, having the various types of ratings is also useful. I usually put effort into music so I tend to have a good score there, but I come very low on certain other things.. and that’s useful for me to know so I can “up my game” in future LDs. This is a big part of the fun for me.. the learning, the rounding out of my skills, and getting better each time.

    As a player, I also like going through the top 10-20 of each category at the end, because I find a lot of games I didn’t get time to judge but that I know will be mostly good.

    • digital_sorceress says:

      I agree that the competition aspect is what makes LD a good game jam.

      What about this: Instead of ranking people from 1 to 1000, what about scoring them from 1 to 16? With 1000 contestants there would be about 60 games in each tier, or 6%.

      It could either be done from the absolute scores (meaning that some tiers would have more games than others, and some might even be empty):
      4.75-5.00 gets A+
      4.50-4.75 gets A
      4.25-4.50 gets A-
      etc, all the way down to E+ E E- and then F.

      Or it could be done on the relative scores (meaning that each tier has the same number of games):
      Top 1/16 of games by overall rating gets A+
      Next 1/16 of games gets A
      Next 1/16 gets A-
      etc

      Both of these systems have their pros and cons, but both would absorb a lot of the noise, while providing enough tiers to help people see their performance relative to one another, and relative to themselves in past contests.

      • R3ason says:

        I could go for that… initially, I was a little discouraged–after being in the top 25 in terms of raw number of votes for LD23–to see my overall rating of #434 out of 1072. Okay, I was a *lot* discouraged. But, after having a few months pass, it really doesn’t bother me any more. In fact, now I think that’s pretty damn impressive for a first go at LD (and being new to game development). Now, I have a baseline with which to compare myself. LD24 will be better. Now I won’t be competing against everyone so much as competing against my previous score.

        Back on track, though, I like the idea of giving a categorical grade (A, B, C, etc). And I like having an absolute number, just so I can see if I did better than last time.

        Cheers

        • sorceress says:

          I can imagine seeing #434 would be discouraging after putting in all that work. I’m sure you weren’t alone with those feelings, since hundred of contestants would have been been presented with a bigger number than that.

          Are these the kind of feelings we want people to take away from LD?

          Today I’ve been mulling over this grading idea… In many ways it’s similar to what Blizzard did with starcraft 2 ladders. They realised from their previous games that ranking everybody 1, 2, 3, upwards can be so discouraging for players, that they introduced leagues (platinum, gold, silver, etc) as well as dividing these leagues up into smaller divisions.

          The effect being that players can determine their skill by looking at what league they are in, and aim to get better and move up to the next league, which is a realistic short step. Moving from Gold into Platinum feels much more encouraging than moving from rank 78910 up to rank 58011, or whatever.

          I feel that grades would give the same kind of benefits.

  8. For me, the prize was the rating, so I say leave it alone. (But just my opinion)

  9. mooosh says:

    I personally feel that the numerical rating system is helping me push myself as well as give a very clear and definite signal of what I need to work on next time.

    and holy smokes is seeing a top 50 rating satisfying. every game developer should want to work towards getting to that.

  10. chambers says:

    I would believe that disabling the rating was a good solution for this problem if it weren’t for a few points:

    1- The comment section isn’t as perfect as you said. Many users get less than 5 comments in their pages, and most of the people who comment just write things like “great game, keep up the good work!” or “I had fun”, which is not constructive criticism at all. I do believe the current rating system (especially because of the “coolness” factor”) propitiates this kind of comment, but I believe the situation wouldn’t be so much different if Ludum Dare stopped being a competition.

    2- One of the things that attract so many people to LD is the fact that it’s a competition. That’s one of the main differences between LD and events like Global Game Jam. I believe people want to see their games being rated by others, and as Peter Cooper said, knowing how well you “scored” lets you keep track of your progress as a game designer (or whatever you are).

    3- The fact that there’s a ranking helps the best games on getting more viewers outside the LD community. We’ve all played tens or hundreds of games during the judging period and discovered our favorite entries all by ourselves, but other people from the indie community usually get in touch with the highest-rated games only after the judging ends.

    I think the GGJ is actually quite a good comparison. At least, my experience on this year’s GGJ illustrates perfectly these three points – My team spent 48 hours making a game only to receive 3 user comments (all of them from friends or people who were in the same GGJ site as us), we never got to get constructive feedback, and I still have no idea of which were the best (or most worth playing) games of the event, since the site’s search system doesn’t include ratings as a search option. I know which was the highest-ranked game in the “overall” category in the last few LDs but I have no idea of which were the best games in the last GGJs…

  11. mohammad says:

    why not make it so that people must play and rate a game before subbmiting theirs?

  12. refreshcreations says:

    Maybe it should be mandatory to leave feedback when leaving a rating perhaps?

    Even if it’s just a sentence it feels good as the developer to see peoples opinions and feedback post event.

    I’ll be entering Augusts Ludum Dare but halfway through Sunday I’m going to be running 10k which should be interesting to say the least! It’ll only be my second but I really got a great kick from the first one.

  13. AD-Edge says:

    Id rather the voting system exactly how it is now, rather than having none at all. That would be a real shame, and I dont think its really dealing with whatever issues there are about the system. Its never been perfect, but its always been important.

    I know LD is all about the game, and I completely agree, but a lot of the enjoyment for some people (like myself) is seeing that rank/statistics at the end, seeing how your game turned out compared to everyone elses. Some people dont care so much about the votes, and thats fine, they just dont need to pay attention/care about it. But for the rest of us, it would take away from the whole LD experience.

    Just because a loud section of the community think the voting system is flawed (and makes a lot of noise over it) doesnt mean it needs to be thrown in the trash.

  14. johnfn says:

    This is just silly. Just because it’s broken (which is a matter of debate) doesn’t you should abolish it entirely. Some indication of your score is better than none at all.

    Everyone just needs to calm down. It’s all going to be OK.

  15. Codexus says:

    I feel the ratings have become meaningless. There are several problems:

    I rarely get a chance to cast a vote on the winning games. I can only vote for a small fraction of all the games so it’s most likely I won’t have voted for the winners.

    I think the results can be strongly biased because of that. A game that attracts a small but very favorable bunch of reviewers will have a strong advantage if it’s ignored by most voters. This can be because it’s a niche game or the author has some loyal followers (without being a superstar either we have seen that’s actually a disadvantage ;).

    Too many people vote without leaving a comment. Who are those silent voters who won’t leave me any feedback? Personally I always try to give a balanced feedback to the games I rate (one thing I like for one thing I didn’t like).

    This wasn’t the case when LD was not as big but now I’m disappointed by the results almost every time. Games I think are great don’t make it to the top while some of the games that have been winners recently weren’t that impressive. I don’t want to put down anyone’s game, we all try our best but I just thought there were better games that didn’t get what the recognition they deserved. Why is that? Is it really that the majority prefers those games or is it because the voting system isn’t good? I don’t know.

    • sorceress says:

      A game that attracts a small but very favorable bunch of reviewers will have a strong advantage if it’s ignored by most voters. This can be because it’s a niche game or the author has some loyal followers

      In statistical terms, we’d say that the sample of reviewers is biased, such that their opinions are not representative of the community as a whole.

      That’s an interesting point you’ve raised, and I hadn’t considered that phenomenon occurring… but I can imagine how easily that might happen.

      I’m not sure what could be done to ensure an unbiased sample of reviewers, other than telling people which games they can/can’t rate, which I don’t think would be very nice.

  16. refreshcreations says:

    Yeah, at the end of the day it is all about the fun of making a game in 2 days and that is a lot of fun!

  17. sfernald says:

    Eh, nothing will come of any of this. The guys who run this site are way to busy to make big changes, and I don’t think they’ve changed much based on user request. Any time I’ve seen one of these threads started, usually, it is ignored and and is lost in the black hole. Then someone starts another one 6 months later on the same subject and guess what, that one is ignored too. I’ve seen this for years.

    The main problem for me is I think ludum dare is getting too big. It is no longer fun for me for whatever reason. So I don’t do any more games on here. But that is just me. I hope it survives somehow, but I have my doubts. Now i just make my games and sell them. It is really best anyway. All of this is just a distraction. Make your games and put them on an app store and make money or give it away and you’ll get 1000s of people playing it and lots of reviews.

    • johnfn says:

      “The guys who run this site are way to busy to make big changes”

      Did you participate last Ludum Dare? They completely overhauled the entire system to find new games, and substantially improved it. It was a HUGE change from how it used to be. I’m astonished you can claim this when it’s definitely not the case.

      The guys who run this site are very smart. They’re busy, but they’re not so busy that they don’t stop improving the site.

    • sorceress says:

      PoV has said before that he’s too busy running his business now to do much more than push the start/stop button for each contest. I don’t know if philhassey is as busy or not. While it’s possible, I feel it is unlikely that this summer’s discussions will yield any changes.

      As for LD getting too big: It has it’s pros and cons. On one hand, it’s nice to take part in something huge, and to be able to say that you were part of it. But on the other hand, it’s become too easy to get lost in the crowd, with most contestants going unnoticed.

      Small closed knit communities can become suffocating over time, so it’s nice to see the roster steadily change and revitalise.

      • Folis says:

        afaik, Phil is pretty busy right now too.

        About LD getting bigger: I don’t mind it much. The site is now stable enough to survive Notch, and I like it, that more games are made. The “getting lost in the masses”-effect has gone pretty hard on me last compo, though.

Leave a Reply

You must be logged in to post a comment.

[cache: storing page]