Rescue Chicken Post-Mortem

Posted by (twitter: @LiamLimeGames)
May 16th, 2016 7:28 am

  Rescue Chicken Post-Mortem 

Rescue chicken was a webpage which highlighted the games for Ludum Dare 35, which had an insufficient number of votes to receive a rating by themselves. If you missed it, the page is available here: Rescue Chicken

 The idea

Rescue Chicken wasn’t a completely “new” idea. During previous jams, Sosowki maintained the tool which inspired Rescue Chicken, called Ludum Dare Rescue Rangers. This tool has however stopped appearing in Ludum Dare jams since LD31, which is why I made the replacement.

The purpose of the tool was to spotlight the games with insufficient votes. There was one major difference between Rescue Rangers and Rescue Chicken though. It may seem like a small one, but is bigger than you might think – entry ordering. Both pages ordered entries by the number of votes they received, the difference was that Rescue Rangers used ascending ordering whereas Rescue Chicken uses descending ordering. This means that in Rescue Rangers, the games with the fewest number of votes would appear at the top whereas in Rescue Chicken the games with the most votes, which have not yet reached the threshold, appeared at the top. I felt that it’s a fairly safe assumption to expect the first few games to receive the most attention from voters – data seems to prove this.

LD31 “Votes Received” distribution (Rescue Rangers)

LD35 “Votes Received” distribution (Rescue Chicken)

Notice in the LD31 data how many games have 11 votes and none have fewer than that. This is very likely because Rescue Rangers promoted those games at the top. Also notice in LD35 data how only a single game has 16 votes, only two games have 15, 14 and 13 votes each. This is likely because Rescue Chicken promoted games that were near the 17 votes threshold, meaning more of them passed. There is also a huge clump of games with exactly 17 (137) or 18 (109) votes, which is where games pass the voting threshold and no longer appear in Rescue Chicken.

When making spotlight or promotion systems, you have to make a decision about what you would like to promote – what you would like to maximize for. Rescue Rangers and Rescue Chicken fundamentally maximize for different things: Rescue Rangers maximizes for “Every game should get at least some ratings / comments.”, whereas Rescue Chicken maximizes for “As many games as possible should get numeric ratings.”.

Everyone knows that the best way to get ratings in Ludum Dare is to rate enough other entries and get your ratings through the Ludum Dare website. These are not the participants either of these pages deal with though – These pages deal with participants who invested “No effort” or “Insufficient effort” into rating other entries to get enough ratings by themselves. I want to talk in non-accusatory terms here, as there are many valid reasons why participants might not be able to rate (enough) other entries, but seeing as how the ranking system in Ludum Dare works, these three groups can be clearly identified (no effort, insufficient effort, sufficient effort). Rescue Rangers tries to ensure that all games have at least some ratings, promoting the games with the fewest ratings. These games typically come from users who have not rated any games themselves, meaning Rescue Rangers promotes the “No effort” participants. Rescue Chicken, on the other hand, orders games in the opposite order, meaning the games with the most ratings (which have not yet crossed the threshold) get the most exposure. So if someone rated a handful of games and in return received a few ratings through the Ludum Dare website, they would appear much higher on the Rescue Chicken list than someone who has done no ratings and received none in return from the LD website. Neither Rescue Rangers nor Rescue Chicken is “correct” in its approach, they simply maximize for different things.

 Exposure / promotion

The post which announced Rescue Chicken was featured on the Ludum Dare website for the last week of voting, so it got a lot of exposure. I didn’t notice much off-site exposure for the tool though – either through social media or IRC. There were occasional mentions of it, but it certainly didn’t go ‘viral’. Never the less, the tool was used.

Rescue Chicken tool usage

I think it’s obvious when the tool was featured and when it was removed from the featured list. 2722 sessions total. The part before that is my attempt at promotion on social media and IRC, as well as development and testing, the part after it is just the tiniest long tail ever. The tool is not useful anymore, folks. 😀

 Did it work? Did it help?

Looking at the graph from above (LD35 one), yes, it did. But how much did it help? That one’s trickier. If someone wants to do a more in-depth analysis on various ways of determining whether it helped or not, go ahead, the data’s at the end of this post. For the purposes of my assessment, I chose to look at the percentage of games with at least one numeric rating (other than coolness).

Games with ratings vs. total games across jams

So let’s get the overall picture out of the way first, since it seems to hover at around 80% with some slight fluctuation from jam to jam.

Percentage of games with ratings across jams.

Most of these are actually unassisted, it seems. Rescue Rangers was apparently only active during jams 26 and 31 from those listed. It’s however possible that other jams had it and that the posts that announced Rescue Rangers have since been either deleted or edited. Establishing a baseline from before jam 25 is actually very difficult, since, for example, jam 20 only had 284 participants, 98% of whom got ratings. It’s very difficult to estimate the effect that Rescue Rangers had during LD 25 and 26 due to its explosive growth before then. Still, looking at LD31, it’s obvious that the percentage of games with received rankings is much higher than those around it. You can also see a fairly obvious and sizeable decrease in the percentage of games with rankings between LD 32 and 34. I don’t think the community just became “worse” at rating, but something else did change – opting out of ratings. It has been possible to opt out of particular ratings (even all of them) since LD 32. This is the only real change I’ve seen since then, so it’s what I attribute the decline in percentage of games with rankings to. I would love it if others did more research on this though and see if this is actually true or if there are other factors as well.

Seeing the decline in the percentage of games with rankings between LD31 and LD34, I think the upswing from LD34 and LD35 is a very positive thing and can at least partially be attributed to Rescue Chicken. Even though the numbers aren’t back to the 98% glory days of LD20, returning to the standard 80% seems like a good thing.

I may very well be reading too far into this and cherry-picking its impact on percentages, but looking back at the LD35 graph and comparing it with the LD34 graph (one without Rescue Chicken or Rescue Rangers), it seems to me that the distribution of games in the 5 – 16 vote region would have been fairly constant, whereas with Rescue Chicken, it isn’t. Most of the games that would have ended up with 9 or more votes without Rescue Chicken, have in fact been pushed past the 17 votes threshold, thus receiving a vote.

LD34 “Votes Received” distribution (neither Rescue Chicken nor Rescue Rangers)

 The data and code

The data used to make all this is linked to in this post: How many people got ratings

The code for Rescue Chicken is available on GitHub: Ludum Dare Rescue Chicken (GitHub)

A live version is available on my website: Ludum Dare Rescue Chicken (Live)

For future jams, you can change a parameter in the live version and it will poll the new LD, so you can continue to use that live tool even without my presence. Simply edit the “event” argument to the appropriate Ludum Dare number: For example for LD36, do: http://liamlime.com/content/ldtools/rescuechicken/?event=36

 TL; DR

I don’t blame you. Long story short: Rescue Chicken worked out very well. What it didn’t do is give every game a rating, it has however reversed a steady decline in the percentage of games that get ratings – at least for now. Thanks to everyone who participated, you’re the best! 😀

If you want more Ludum Dare related analysis, follow me on Twitter: @LiamLimeGames

~Lime


2 Responses to “Rescue Chicken Post-Mortem”

  1. rburema says:

    ‘Chicken Post-Mortem’? Should we get the BBQ out? 😉

    Thanks for making the site & this analysis. It was fun to think those last few days that every vote you cast could potentially be the last vote someone needed to get results.

    I seem to remember that last year (LD 32), someone (because there where a few people who thought three weeks of waiting was too long) showed a graph that indicated most votes where made during the first week after LD, dropping off in weeks 2 and 3. (Regretfully not showing as far as I remember was how many games where ‘fully rated’ for each week, which is more important than total # of votes IMHO.)

    I wonder what the effects of the various rescue programs would be on a similar graph for their respective events. (I mean, I know /I/ voted a few extra games over the last weekend thanks to the rescue site, that I probably otherwise wouldn’t have…)

  2. Liam :D says:

    > Chicken Post-Mortem’? Should we get the BBQ out?

    YES! (post images edited)

Leave a Reply

You must be logged in to post a comment.

[cache: storing page]