What Ludum Dare Rating Categories Matter Most?

Posted by
April 13th, 2016 11:23 am

When we reach the end of a Ludum Dare event and evaluate each others’ games, we grade each game with a series of categories: Overall, Fun, Coolness, Graphics, Audio, Theme, Innovation, Humor, and Mood. When the results are announced, the top 100 games in each category are displayed, and of course it is every competitor’s dream to land in at least one of those top-100 lists.

The Overall category holds pride-of-place. Arguably, the winner of the Overall category is the winner of the event. It’s satisfying to do well in the other categories, but until you’re ranked #1 Overall, there is still room for growth in your Ludum Dare performance.

Which brings me to this question: Have you ever wondered which other categories make the biggest difference in how you rank overall? For example, do people who do well Overall also tend to do especially well in Graphics, or Theme, or Innovation? Coolness is a measure of how much you played other peoples’ games; does a great Coolness ranking help you achieve a good Overall ranking? If you want to succeed Overall, does it pay to focus on Fun, or Graphics, or Audio, or Mood?

If you’re anything like me, you’re just itching to know.

And now you can.

 

For the last several Ludum Dare events I’ve analyzed the relationships between category rankings by looking at the scores of the top 100 Compo entries. In each event I’ve analyzed the correlation between how games did in each of the nine categories. What I’ve found is that there are strong correlations, and they’re not necessarily what you would expect.

Take a look at this.

Correlations per Event

Correlations per Event

What you see here is a chart of how well each category did relative to the Overall category in each of the last eleven LD events. The blue line represents Fun, for example. This shows that more than any other category, Fun correlates strongly with Overall. If you do well in Fun, you tend to do well overall; if you do poorly in Fun, you tend to do poorly overall.

The blue line changes from left to right. That’s because each Ludum Dare event has different results. Ludum Dare 24 is at the far left. In that event, Fun had a 30% correlation with Overall. That means, roughly, that for 30% of the top-100 games, the best Overall games also did well in Fun and the weakest Overall games also did more poorly in Fun.

You see a big spike at Ludum Dare 33. (See the number 33 just below the blue spike?) In that event the correlation between Fun and overall ranking was even higher: about 50%.

Our last Ludum Dare, though—34—was a strange one. This was the only event out of the last eleven in which Fun wasn’t the dominating question: Theme, Humor, and Mood—through games like Frank & Stein, Rebuild/Resist, and Molly’s Dreaming—carried the day.

So you see right off the bat that Fun is the place to be. It almost always dominates the other categories in being closely connected (statistical speak: “correlated”) with a game’s Overall ranking.

The other categories are more of a grab-bag, though still with some clear winners and losers. To see them it’s helpful to look at the same data in tabular form.

Correlation Table

Can you get a better Overall score by rating lots of other people’s games? The clear answer is noCoolness has essentially zero correlation with overall rankings. There are plenty of other good reasons to rank other people’s games, no doubt, but getting a higher overall ranking isn’t one of them.

If you had to choose a second category to focus on, after Fun, to improve your chances at Overall, it would have to be Graphics. The average correlation for Graphics is 10%. It sometimes does as well as the high teens, but went through an unpopular period during LD-28 to 30. (Just as an example, the overall winner of LD-28—One Take—ranked #89 in Graphics, while Ultimate Wish got second place in Graphics and 90th place Overall.) Truthfully, though, Graphics does not consistently correlate much better than any of the other “middle” categories: Theme, Innovation, Audio, Mood, or Humor. So I wouldn’t spend 24 of my 48 hours neglecting audio and repolishing my artwork just because Graphics does 5 points better on this table.

There are two sound conclusions here. If you want to do well Overall, make your game Fun. And by all means rate other people’s games, but don’t think that’ll earn you a higher ranking.

So far we’ve been focusing on how the other categories correlate with the Overall category. But how do the other categories correlate with each other? Do games that do well in Graphics tend to do better in Audio, for example?

Well as it happens, yes they do. A little. Here’s the data.

Category Correlations

This chart shows how strongly the different categories have correlated over all the games I examined. To use it, think of two categories; say, Graphics and Humor. Go to the column labeled Graphics, trace down to the row labeled Humor, and read the number. This tells you that Graphics and Humor have a 1% correlation; which, frankly, is insignificant.

There are really only a few significant connections. Audio correlates well with Mood. And that kind of makes sense: presumably a game with good music has a stronger overall feeling than one with no music or bad music. Graphics and Mood are also connected (8%). Audio and Fun perhaps weakly connect, though it’s doubtful 3% is significant. In general, strong performance in one category doesn’t seem to imply that you will do either well or badly in another category.

There’s the data. What does it all mean?

For my money, I want to emphasize Fun in my Ludum Dare entry. Having strong visuals (Graphics) and moody audio seem to be the next highest priorities—audio being a good emphasis since it pays off both in Audio and Mood, both of which contribute to the Overall ranking. I’m not going to worry so much about Innovating. And although personally I like to follow the Theme closely, it seems to be okay to depart from it somewhat. And Humor? Looks like we can live without it.

Of course, none of us create our games as coldly and calculatingly as this. We make games on inspiration and, frankly, the sweat and panic of a weekend coding. At that point all our noblest plans—not to mention fiddly statistics—go out the window. Still, it’s good to know that as the deadline looms, if you have to invest your last moments either adding more humor or making the game more fun, making it more fun is the better bet.

Finally, a few disclaimers. I’m no statistician, just a computer programmer, so don’t take these statistics to the bank: I could be wrong in my method or interpretation. Remember that these numbers only cover the top 100 games, so if your goal is to move from position #600 to position #300, the emphases and strategies may be entirely different. Likewise if you’re in the Jam: these numbers all came from the Compo. Lastly, please do take these statistics loosely. A great deal of the data is spotty and vague, mainly because each competitor can choose whether to be rated in a given category, and often we opt out of significant categories, so the data has lots of “holes” in it. That, and the fact that all statistics are worse than damned lies, means you should take everything you’ve just read with a big giant grain of sea salt. It’s all true as best I know, but my best knowing isn’t very good.

Tags: , , ,


4 Responses to “What Ludum Dare Rating Categories Matter Most?”

  1. Liam :D says:

    I’ve collected the data from all the previous ludum dare events. You can view this data here:

    http://liamlime.com/experiments/ldscrape/out-ludum-dare-31.html (tabular display)

    and

    http://liamlime.com/experiments/ldscrape/out-ludum-dare-31.json (json file)

    Change the 31 to whichever number to look at the data for other jams. I hope this helps you in your further analysis.

    You might also like to look at these posts:
    http://ludumdare.com/compo/2016/01/10/how-many-people-actually-got-ratings/
    http://ludumdare.com/compo/2016/01/23/ludum-dare-visualisations/

    Hope this helps!

  2. scriptorum says:

    Nice work!

    BTW, shouldn’t the average/bottom row of the overall correlation chart match up with the overall/top row of the category correlation chart?

Leave a Reply

You must be logged in to post a comment.

[cache: storing page]