My thoughts on Ludum Dare (and how to make it better)

Posted by (twitter: @fakepsyho)
May 10th, 2015 4:24 pm

I highly doubt that my thoughts are going to be original, considering the long history of LD and the amount of participants it has. For the record, this was my first LD, I loved every part of the LD throughout the first 48h – even though it meant that I spent my almost entire weekend alone and working :) But that excitement quickly left me as soon as the voting period has started.

Since no one likes long posts, I’ll directly dive into the problems:

1) I thought I’ll leave people valuable feedback on their games and (hopefully) this will create some interesting discussions about the games itself. Ideally, no one would feel bad about constructive criticism and we would end up in a win-win scenario. But it’s not possible right now, since: (a) there’s no threaded view of the comments (b) there’s no way of tagging people. It’s one of the (smaller) reasons why most of the comments are completely shallow, since the discussion is impossible and has to move towards the personal blogs.

2) If I understand correctly the final result is just an average of votes. If I’m wrong please correct me, but I guess at least some of this still applies. This is really bad idea and easily fixable. (a) First of all, bayesian average is a trivial improvement over it, which in general decreases the variance of the result (no more taking top spots by submissions that had votes barely over minimum). (b) The other easy thing that should be much better is to use percentile rank. You can also drop top and bottom ~10% to reduce the effect of the outliers. And yes, I really think that if someone’s giving only 1s or only 5s, then those votes should transcribe to the exactly same score. (c) Ideally, the proper way would be to toy around with the machine learning, considering the prediction of the “true” ranking should be a rather standard problem in recommendation system. It’s actually absolutely the same problem Netflix prize used if you assume that we’re predicting ratings only for the “fresh” users.

3) If I stick only to games recommended by “Play and Rate Games”, I’m going to play mostly unfinished or just very weak games. It’s because the fun and interesting games have a lot of votes already due to the self-promotion. After 5-10 games that may have no gameplay, are completely bugged or someone was clearly just starting his/her adventure with gamedev, I’m beginning to feel frustrated. This is a pretty sad situation since this forces me to hand-select the games, which in turn completely skews the rating system and it makes even worse situation for everyone else. I guess in the past there was absolutely no system at all (only hand-selection), so nowadays people look at the coolness system as something that is an improvement over what LD had before. There’s very little you can tell about the games that are completely unfinished (or sometimes even not started). In those games the feedback is either the same, or is purely motivational. This means that weak game won’t suffer too much from having fewer votes/comments. This makes since at the same time, we want for good games to have as many votes as they can, in order to reduce the variance of the top places. (a) You could use the current rankings to adjust the priority in the “play and rate” queue. If people would get more fun games in the queue, they would use it more, so in the end the weak games would still end up with decent number of votes. The main difference is that we would reduce down the huge spam of “PLEAAASSSEEE RATEEE MY GAMMMEEEE”, which honestly is not the best thing in the world. (b) Somewhat risky idea would be to stop counting the votes of hand-selected titles altogether. Or alternatively, reduce their weight (and added coolness).

Edit: Also, please read my reply to tompudding. I think I did a better job there explaining the main idea behind (3).

Anyway those were my 2 cents. If this stuff was brought up already then I hope someone can give me links to those topics.

Also, no link to my game since this would defeat the purpose of the whole post :)


16 Responses to “My thoughts on Ludum Dare (and how to make it better)”

  1. Os_Reboot says:

    I agree with this – especially #1. Its silly that we can’t even edit past comments. +1 for tagging, hope @mikekasprzak adds that in the new site.

  2. tompudding says:

    I don’t agree with number 3 at all. By far the best feature of Ludum Dare is that despite my poor skill level, the system encourages people to play my game and give me feedback. Yes maybe playing unpolished and unfinished games is not as much fun as playing the top games from professional gamedevs, but it’s gratifying and helpful for the developers of those games, especially as they’re doing the same thing for others.

    • Psyho says:

      Well I agree that (3) is a risky change that maybe won’t benefit everyone (it’s hard to tell, but IMHO it’s worth the try). My line thought is that currently “Play and Rate Games” is not working very well, so I’d like to “fix it” so that more people will use it.

      Anyway, I’ll reiterate my main points (which may have been lost since I’m not the best guy to clearly state my point):

      – First of all, you don’t fall into the category of very weak games that I mentioned (I made a very quick glance over your game). I really don’t want to give any direct examples, but the cases I’m talking about are those where for example there’s no gameplay or the game hangs up after 10 seconds. I really doubt if there’s big difference if they are going to get 20 or 40 votes. The comments are all going to be same (“nice [insert element that was present in the submission]”, “keep up the good work!”, “it looks like it could’ve been fun if it worked”). Honestly, I want to give feedback to someone who can actually use it. If we can reduce slightly the probability of getting such games, then I think it’s a win for almost everyone.

      – I hope that when “Play and Rate games” is more entertaining for everyone, more people will use it. The end result would be that those games still receive similar amount of votes, but now there’s much less self-promotion since it’s way less effective (how about limiting the amount of posts someone can make?). People spent less time on searching good games, since the system actually recommends them better games on average.

      – Ideally, there should no way of voting for games that you have hand-selected. I understand that it’s a pity to waste someone’s vote — if someone had already played a game, why not let him/her vote since we’re fighting for the overall increase of the number of votes? At the same time, it’s unfair. So every small step we can do, to reduce the number of games that are played by hand-selection, is IMHO a good one.

      • drludos says:

        Interesting thoughts!

        Thanks for sharing this, even if I’m not really convinced by your idea. I understand your point that the current rating system is perfectible, but, for my first LD (this is my third) I actually found the rating system to be a strong point of the LD event: I think I ended up by having something like 30-40 ratings and comments in total, with a lot of motivational but also game improvement related comments (and it actually motivated me to continue working on it to release an improved version, which I’m still working on).

        Personally, I choose my games using a variety of four venues, in order:
        – Games from people who rated my game.
        – Games from people who rated the game of someone who rated my game (yup, I sometimes randomly click on the names of people who commented a game I’ve just rated to have “random” game presented to me).
        – Games I find cool or intriguing on the main blog, be it from a self-promotion, a post-mortem, a “top 10” or a “these games needs charity votes” post (so, handpicking).
        – Random games from the Play & Rate page.

        I my first LD, the Play & Rate page was my most used way to choose games to rate, now I clearly favor different kinds of “handpicking”. While you may find this approach unfair (and in a way, it is), I think it favors the “personal relations” between devs.

        When I rate a game I like, I usually try to visit the blog / portfolio of the creator, to see more games from him/her. And I often follow them on Twitter too, and I’ve discovered some very interesting and talented creators this way. I also try to check the games from these creator during the next and previous LDs, to see what they have come up with.

        All in all, with a competition generating nearly 3000 games, I think it’s impossible to have a global view of what was created during the event. So, I think accepting the idea that you’ll mostly take part in only a subset of this very very big event is a key to enjoy both the “game creation” and “game rating” phase ;).

    • Anikki says:

      When I was playing and rating games a few days back, my girlfriend walked over and told me i was being way too kind with my ratings. Following that, I observed that I am way more critical with good games, sometimes so much that I’m wondering that if others do the same, if we maybe end up with too little differentiation between good and bad games, so much so that in the final ranking only the top 3 games per category get meaningful ratings and many actually good games get buried in the top 263 because they were no platformers or had no great art or just didn’t go for the LD-success formula. If we want to improve something, it is in my mind: more honest ratings and devs being more honest to themselves before submitting whether their game is playable or not.

  3. I very much agree with #1 in that I’d love to see some sort of comment threads available. It really kills my communication with other devs when I have to manually re-visit their games to see if they might have replied back or not to my feedback. On the other side of the coin though, it would be nice to be able to subscribe or something to what you get notified about. I can imagine it could get annoying for some being notified for every little reply you get.

    I think I disagree with #3. To me, the point of the “Play and Rate Games” system isn’t to provide you with all the good polished games that come out of Ludum Dare. It’s to give well rounded balance to all the participants and to give everyone the ratings they deserve based on how much they give back to the community. I think they’ve done an excellent job with this, but it’s just based on personal experience. My game’s ratings seemed to go up right along with how much I rated other games. I find it impressive how they keep the big games from smothering all the smaller ones out.

    If what you primarily want from the Play and Rate system is to show you all the good stuff, why not just wait until the final judging results are in? To me that’s what you’re looking for, a large list of all the best games that came out of the event. Of course, you just have to wait for it.

    • jackrugile says:

      Thank you, I completely agree with what you said about the rating system. It’s not a time for developers to be entertained by the best or most fun entries. It is a time to give honest feedback and ratings to those who need it and are contributing back to the community themselves. The fun part is after! Changing it to be any other way will leave hundreds of devs in the dark with no one playing their games, which will in turn most likely create even more “play my game” spam.

      I do agree that some awfully broken games get stuck lingering in the play and rate section. Perhaps there should be some sort of flag option that devs can check off if a game is completely unplayable. With enough flag checks, it will be removed from the play and rate page and a notification will be emailed to the user. That would clear up some room for hard working devs, with working games, who are giving back to the community, who just haven’t received many votes.

      • Psyho says:

        Flagging games (or binary system) has that downside that it’s often very hard decision to make. Removing a game is IMHO much worse than reducing the number of votes it gets, because it completely ruins your effort.

        “The fun part is after!” – well, the idea is to make the voting more fair, while increasing the fun that people have while rating games. If the queue is so bad that people are mostly hand-picking games, then of course they’re going to play games that are potentially fun to them. Which means (although it would be nice to have hard data on this), that games that do more self-promotion are going to be more successful in final results. Also, that time spent on doing self-promotion could be invested in rating more games. Just saying :)

    • Psyho says:

      Um, not sure if you’ve read my reply to tompudding, but the whole idea behind (3) it to actually make “Play and Rate Games” interesting enough, to make people actually use it (ideally exclusively) for rating other games. It’s just that I’d like to fix by increasing a direct reward (play more of interesting games) instead of doing some questionable gamification-like mechanic.

  4. SpringCabal says:

    1) Comments have huge issues imo. Despite often being positive and written with good intentions, I’m honestly not happy by the comments that we’ve received: I think they were too brief and not helpful. All that we learned from them was that some people liked the graphics, people didn’t like controls in general (although no information on what exactly they didn’t like or how it should be done differently), and some with ATIs/Intels had GFX driver issues (engine thing that we couldn’t do much about for this compo). This is very little information to improve on..
    This is one of the reasons I wanted streamers to cast our game as they give a far detailed review in general, but most had ATI cards so that didn’t work out well either (except in case of: https://www.youtube.com/watch?v=_-2M5YBu_2Y ).

    I’d like it if there was a system that encouraged detailed reviews (maybe by rating comments?), but I’ll certainly propose a tit-for-tat system in the next LD: give a detailed comment for our game and I’ll give you one back!

    2) I think it’s OK as it is, but maybe it would be better if there was a second step of voting/commenting after 1.5 weeks on a reduced number of games – we managed to vote 100/2.8k games meaning it’s quite likely that we haven’t voted on the winning entries!

    3) I mostly rated games by “Play and Rate Games” default view, but after a while and finding only 1-2 games I really enjoyed playing I cheated a bit by looking into new entries submitted by winners of previous LDs (example: http://ludumdare.com/compo/ludum-dare-30/). Sure enough I found a number of awesome, polished games that I trully enjoyed playing.

    • Psyho says:

      1) Yeah, encouraging longer/detailed reviews should be something worth consideration. I completely missed that one out in my post.

      2) The system that I propose (change of rating calculation) should be strictly better than what is currently available. Considering that it seems that the scores in top10 (and often in top3) are pretty much statistically insignificant, every bit of improvement would help. If the votes are available somewhere, it would be trivial to show (at least for me :) ) how much random they are.

      3) That’s an interesting way of “cheating the system” :) It’s still unfair in that sense that you’re giving more “action” to people who had some success in the past, but it’s definitely interesting to see how developers are evolving throughout different LDs.

  5. Horsed says:

    What I really like about the coolnes based rating system is that it encourages us to rate games. The more you rate, the more visible your game will be in the default view, which leads to more votes. However, there are games with a rating count that is higher than their coolnes, so their submitters receive more votes, than they “give back” to other games. I think, that’s where promotion becomes visible and it leads me to thinking, that the rating system is incomplete. While being ok with the number of ratings my game got so far (57), I don’t like that I had to rate 131 games to receive enough visibility. Don’t get me wrong, I enjoyed playing and rating most of those games and I think, this is essential to becoming a better game dev. But I don’t like that promotion is benefitial to a game’s rating count, especially not when taking part in a game jam. Although I’m not sure of how benefitial promotion is to the overall result of a game, I don’t think it would be fair for a game to have a top twenty result, while it’s submitter(s) didn’t rate any or far less games. I would love to see some statistics based on past LDs.

    In my perfect game jam, a game can only receive as much votes as the number of games, the submitter rates: If you rated 20 games and your game received 40 votes, only the first 20 votes can count towards your game’s result. I personally, would be super motivated to rate more games and I think this would put more emphasis on a critical part of Ludum Dare: Help others in becoming a better game developer by providing feedback.

    • Psyho says:

      FYI, the effect coolness is capped at 100 ratings (i.e. you don’t get anything more by rating more than 100 games).

      The thing is, that system is already skewed because people are rating hand-selected games. The whole point was: if someone has created a game that offers 15-30 seconds of entertainment, there’s very little point of giving it more ratings since (a) the final score is going to be completely insignificant anyway (b) the comments are going to be redundant after some point of time. In other words, I’d like to do some resource management and move those votes to some higher-rated games. It’s already happening now due to hand-selection, but it would be much better to push this into “Play and Rate Games” pipeline. And if you can vote only on games that are served to you, then actually we end up with system that is no longer that easy to cheat/abuse.

  6. Filth and Money says:

    I understand that people like the fact that everyone gets a fair shake at being rated, but I also agree that rating games becomes a huge chore really quickly. Hell, look at my game—it’s terrible! I don’t need anyone to tell me it’s terrible. It’s obvious; and when I play 30 terrible games by other people, I end up feeling really drained really quickly; and it takes a ton of my time.

  7. Alex Rose says:

    “(no more taking top spots by submissions that had votes barely over minimum)”

    Why?

    That’s dumb as shit, why should a game stand more chance of winning if it has more votes? What you’re essentially doing there is making “Ludum Dare celebrities” like Deepnight have more chance of winning because they automatically get more ratings without even trying.

    Likewise, it would mean sites like indiegames.com, warpdoor, indie haven etc. that post jam games would be able to sway votes over to the games they curate rather than games that are good.

    And it also encourages people to spam their game all over social media and quickly rate 500 games without playing them properly or leaving proper feedback in able to get to the top of the coolness list to get more ratings.

    On top of this, the whole point of having the front page show you games in the way it does is that:

    A. Everyone can qualify by rating other people’s games.
    B. The biggest the disjoint between the number of games you’ve rated (<=100) and the number of times you've been rated, the more chance people have to play your game. So the more ratings you give, the more ratings you get.

    Ludum Dare's rating system isn't about you just getting to play the best games, if you want to do that, wait until the competition is over.

    I think there should be a way of flagging a game as something you either can't or don't want to play though so the same games don't clog up your front screen, and the tagging is a good idea, but the rest sounds like you really missed the point.

    • Psyho says:

      “That’s dumb as shit, why should a game stand more chance of winning if it has more votes?”

      Currently, the less votes the game has, the more chances it has for winning due to variance. It’s that simple. And it’s absolutely not fair for anyone, because it makes the results just random.

      The whole point of (2) & (3) is to make rating more fair. It would be better if you would argue with what I actually wrote, instead of what you assumed I wrote there.

Leave a Reply

You must be logged in to post a comment.

[cache: storing page]