Quality or Quantity: Rating with Integrity

Posted by
May 10th, 2013 7:52 am

I always find the rating part of Ludum Dare to be more trying that the coding aspect of it, but I think a lot of that is because I don’t give up on a game easily.
Maybe it has some mechanic I don’t care for or maybe it’s just difficult and I’m not very good at it. I’ll still play the game until it is as complete as possible.

The game I made this time around isn’t very good. It’s really more of a tech demo than a game and the things that I consider successes are going to be measured personally and not with fancy star widgets. The game itself could have been implemented in plain text, and that’s really where the meaningful content lies.
If I came across it while rating I would probably give it a nod for novel mediocrity and that’s about it. What I would not do is bestow my condescension without actually playing it.

Since it’s web based and the resources are loaded dynamically, I can do some simple log parsing to see how far people have progressed. There are a couple of factors that easily create a margin for error, but in general I think the results are accurate enough to assert that a lot of people apparently can’t be bothered to spend the 5 minutes it takes to complete.

After removing my log entries, 50% of unique plays move on to the second and final chamber.
This is on a game that can be completed in just a few minutes and all you have to do is read some text and click a few times.

Just to be clear, this isn’t about my entry, this is about your entry. Consider how much time and effort went into your game. It doesn’t matter how happy you are with the result or how legitimately good or bad that result is, we owe it to one another to diligently explore each other’s work so that we can provide helpful and objective feedback.

I’d like to thank everyone who completed my game. Your comments are appreciated, and you (and everyone else involved in LD) can rest assured that I will show your work the respect that it deserves instead of trying to farm it for coolness.

Apologies if this comes across as hostile. Please read in the voice of Elmo or any other endearing childhood character of your choice.


15 Responses to “Quality or Quantity: Rating with Integrity”

  1. BenW says:

    I think that low completion rates are probably going to be a given no matter what sort of game you create. My game cannot be lost and only takes about 5 minutes to finish but currently only has a completion rate of 37.8%. You aren’t going to be able to satisfy everyone but as long as some significant portion of players are seeing your game through to the end you’re probably doing something right.

    • uberneen says:

      I’m not suggesting that everyone should be satisfied, it just strikes me as common courtesy to not balk at the first inkling of unpleasantness. I know there are a lot of dedicated people out there who will give a solid playthrough too. I’m just trying to appeal to those that don’t.

      Your game is great by the way. I’m surprised that more people don’t complete, but that really just highlights the problem to me.

  2. Hey guys, just played your games, and I think I also completed them… It wasn’t that hard, really! :) The problem with people not completing games might be that people don’t have much time to spare, after all, there are over 2000 games to play, and that makes people want to rush to the next game as fast as possible. Then again, keeping people’s attention focussed on your game is one of the most difficult tasks you have to face as a game developer, and it can’t always work out in the way you whished. But that might, as explained, be also because of the sheer mass of games on display. So I guess we should go easy on people who don’t complete games… :)

  3. Spiridios says:

    I had a long ramble about how even partial reviews can provide useful feedback, but then I realized, expecting everyone to complete your game is unrealistic.

    Let’s take Portal 2: http://steamcommunity.com/stats/Portal2/achievements/
    Only 83.8% of players even complete the INTRO sequence. Further, only 51.6% get the achievement for beating the game, and that’s actually high.
    Look at Braid: http://steamcommunity.com/stats/Braid/achievements/
    84.5% get through world 2, 24.6% complete the game.
    Bastion: http://steamcommunity.com/stats/bastion/achievements/
    84.8% get the first zone achievement, 15.4% complete the story.

    Like it or not, games only get completed by a fraction of all players. And even though us fellow LDers are a special class of players, we’re still just players. We’re going to stop because the music is grating, or because dinner got done, or because the computer crashed, or because we ran out of time trying to rate just one more game. But that doesn’t make the feedback worthless. We can still comment on difficulty, controls, graphics, or the atmosphere. We can’t comment on the ending, but the point of a game isn’t the ending, the point of a game is to play it.

    • uberneen says:

      I am definitely holding this particular community to a higher standard because I think we should be able to relate to each other. In the example I’m citing completion is trivial and arguably necessary, so I don’t think it’s unrealistic at all. In fact, I’m not even counting completions, I’m counting the people who got past what is effectively the introduction.
      I don’t look at incomplete feedback as worthless either, I’m just suggesting that we not let ourselves get so overwhelmed by the volume of games that we become flippant and careless.

      • Spiridios says:

        Why? If a game leaves such a bad taste in my mouth, why should I continue? I realize there are people who had 200 coolness on the Monday after compo, they can’t have actually played any of those games. That, I can agree, is bad.

        However, you’re saying “everyone should play my game to the end, because I worked hard on it so I want people to see it.” It doesn’t work that way. If people aren’t playing your game to the end, you shouldn’t be thinking “there’s something wrong with the players.” You should be thinking “there’s something wrong with my game” and ask yourself WHY they’re not playing to the end. You created the game, it’s your job to draw the players in and keep their interest. It is NOT the players job to slog through an unpleasant experience so you can feel warm and fuzzy about everyone seeing your clever content. The fact that they’re leaving should tell you something about your game…

        • uberneen says:

          “everyone should play my game to the end, because I worked hard on it so I want people to see it.”

          I never said any such thing, and I was the first to point out that my entry is underwhelming to say the least. You’re reading something into this that was never there. I even say that I play a game until it is as complete as possible. I’m in agreement with stevejohnson’s take on that matter.

          The entire point, to use your metaphor, is that you can’t develop a taste if you don’t take a bite.

          • Spiridios says:

            My apologies for misreading your complaint then. Please reread your initial post and perhaps you may understand where my reading came from. You say “a lot of people apparently can’t be **bothered** to spend the 5 minutes it takes to **complete**” The subtext of that sentence is the people playing your game have a problem, they’re not completing it. My point is, it’s not the players that have the problem and thinking that way is poisonous.

            • uberneen says:

              I can see how the wording is conflicted and unclear. I was mostly referring to how brief and easy completion is. I also take the rating phase of LD as a part of general participation, which is why I’ll knuckle down and keep working through a game that I’m not enjoying. Here you rate because you participated and you’re rewarded for rating, so it’s not the typical landscape.
              If I were trying to get something greenlit on steam I would have a different take on the situation.

  4. nimblegorilla says:

    I played your game and got stuck going to the second chamber because I didn’t read all of the signs. After I carefully went through all of the signs again I went into the second evaluation chamber and answered all of the questions. I went to the final door and got a message that basically said “results inconclusive, please study more and retake the test”. The game then put me back in the first room.

    At this point I feel like your game is being disrespectful of my time as a player. I have already spent several minutes reading a bunch of regulations for what appears to be a slightly dystopian society and then taken some sort of personality quiz. Now your game is telling me to go back and study more?

    I think your analytics are pretty cool. You can use them to investigate which parts of your game have a high drop-off point and then make some changes to keep players engaged across those obstacles.

    • uberneen says:

      You reached an ending, which is fine. You don’t have to re-read the indoctrination material a second time. I can see how you might get the on-the-fence ending if you think that’s only ‘slightly dystopian’ 😛

      • nimblegorilla says:

        Yeah – I thought the story idea behind your game was compelling, but I think the first-person interface made it harder to get engaged. I tried to toe the line when I answered the questions because I was kind of wondering if I would get killed for answering wrong. haha

        • uberneen says:

          That is really interesting feedback. I hadn’t considered that someone might think there was an immediate consequence to answering a question, but why not?

          I was hoping the 3D environment would be more immersive, and maybe with another 12 hours it could have been. I wanted people to be able to look around, maybe play some elevator music, but it didn’t happen.
          Hopefully the pointer lock API for browsers will gain more ground. That would let it behave more like a traditional FPS.

  5. stevejohnson says:

    I haven’t played your game yet, but as a general comment: I play a game until I think I have a feel for it and think I can accurately rate it. That’s all I expect of people who play and rate my game. It’s often impractical to finish a game even when it’s short because the game gives you no indication of how much content is left.

    I don’t think “there are a lot of games” is a good excuse for skimping on rating quality. Just rate as many games as you can while maintaining your standards. Entropy will take care of the rest.

Leave a Reply

You must be logged in to post a comment.

[cache: storing page]