Some things to think about

Posted by (twitter: @ddrkirbyisq)
July 25th, 2016 5:31 am

Hi there!  My name is DDRKirby(ISQ) — I’ve been participating in Ludum Dare enthusiastically for the past 5 years, having submitted 12 games to both the compo and jam events.  6 of these entries have placed in the top 10 overall (not just trying to boast — this will be relevant in a later part of this post).

As many of you know already, there has been a bit of a situation regarding the upcoming August LD event.  A poll was made, with the end result being the planning of an August LD event on the current LD website without using LD’s categorical 5-star rating system.  As you might expect, this also triggered some community backlash.

I am not affiliated with Ludum Dare officially and am in no position to make demands on how the August LD event should or should not be run.  However, I wanted to provide a personal viewpoint on some of these issues.

My goal in writing this post is two-fold:

  1. To highlight some of the major disadvantages and problems with LD’s rating system.
  2. To offer some suggestions for how we can best help the no-ratings August LD event to succeed.

Please note that I am NOT attempting to convince anyone that Ludum Dare should not have a rating system.  I think that most people understand that having a scoring system can be extremely useful and valuable.  However, I think it’s important that everyone also understand the potential issues and problems involved with the current rating system and why many of us would prefer to have an August LD without it.  Let’s get right into it:

 

Winning Ludum Dare is a Big Deal

One of the things I really like about Ludum Dare is the fact that there are no monetary prizes.  As the rules page puts it, “Your prize is your product.” (and to a lesser extent, the feedback that you receive on your game as well)  This is supposed to help foster a low pressure, non-competitive environment where people can simply focus on making cool games, which is great!

However, one thing I’d like to make clear is that winning Ludum Dare is a Big Deal.  Being highlighted as one of the winners, especially in the Overall category, guarantees you and your game a TON of exposure, not only to fellow Ludum Dare entrants, but also by gaming portals, indie game sites, YouTube videos, twitch streams, and more.

When Ripple Runner won 2nd place overall in LD29, I got a huge spike in followers and network traffic.  Not only did several gaming news sites feature it in articles, but web portals started outright STEALING my swf file and hosting it themselves to profit via ad revenue.*  A Korean stream video of my game later went viral, hitting over 80,000 views.

I sell my game soundtracks on my bandcamp site.  Despite having a minimum price of $0, sales from the Ripple Runner soundtrack earned me $200 in profit after LD results were announced because of the wide attention garnered from my 2nd place ranking and would go on to earn me over $350 in total in the following months.

In other words, placing 2nd in Ludum Dare had MONETARY implications for me, almost as if I had received a $200 cash prize — and this is for a pay-what-you-want album!  I can only imagine that things would have been even bigger if I had come in 1st place, or if I had charged $5 minimum per album sale.

I would love it if Ludum Dare were not about winning and losing, but the fact is that the benefits gained from placing highly are very real and from this point on I have always felt mental pressures in the back of my mind to do my best to score highly as a result.

This unfortunately provides a large incentive for cheating the system.

 

Ways to Cheat the System

PoV has already outlined in his post that there has been evidence of cheating within LD, including:

  • A user that created multiple accounts just to give their game 5 star ratings.
  • Games that have more votes than downloads (i.e. people rated them without even playing).
  • Users that voted for many entries in an extremely short period of time (again, indicating fake “throwaway” votes that were made without playing the actual game).

These things are not unique to Ludum Dare.  This post from GameJolt outlines some additional ways in which people have tried to cheat around the rating system in their own game jam.  Remember, these are community game jams without monetary prizes.

  • An individual who hacked into/used multiple Facebook accounts to sign up and make it appear as though different people were rating their game.
  • People giving their own entries many “4-star” ratings to avoid arousing suspicion from giving themselves 5-star ratings.
  • Around 60 entries were found to have cheated by either starting early, or by using disallowed 3rd-party assets.

If I were 100% serious about winning a Ludum Dare compo and was prepared to cheat the system in order to do so, I would probably leverage some or all of the following additional strategies:

  • Enlist one or more artists to draw artwork and animation for the entire game, but still submit it to the compo event.
  • Use tilesets, background animations, and menu systems made completely ahead of time.
  • Subvert the time limit — when submission hour ends, submit a broken link to the game, but keep working for another 48 hours.  When I’m done, upload an incredibly polished game and claim to “fix the broken link”.
  • Create many user accounts using different IP addresses.  Submit random scrappy game projects under those accounts to avoid being flagged as fake accounts.  Using these accounts I could upvote myself, then be on the lookout to downvote other people’s games that seem popular.  Other randomized votes could be applied to avoid easy detection.

Some of these methods would be very difficult, if not impossible, to track using the current LD rating system.  Of course, any of them would constitute blatant cheating and disregard for the rules and as such I would never do any of them.  But the point I’m trying to make is that there is both incentive and capability for cheating here.  In addition, you wouldn’t even have to do very much of it: For Ludum Dare 28, our Jam entry missed hitting 1st place by a mere 0.02 points (4.39 vs 4.41).  As few as THREE fake 5-star ratings would have changed the outcome.

There are a multitude of ways in which the system could possibly be augmented and improved in order to prevent, mitigate, or detect the effects of cheating.  Again, the gamejolt jam post illustrates one example of what this looks like (note that it involves coordinated efforts as well as manual intervention).  However, it is IMHO unrealistic to expect any of these changes to be implemented for the upcoming August LD.**  How this should be dealt with in the future is outside the scope of this post.  I merely wanted to illustrate the potential problems that exist in the current system.

 

Ratings Can Cause Frustration For Jammers

So far the cheating issues I’ve described perhaps only matter significantly for top-ranking games, where there is a lot of potential recognition and attention on the line.  But for anyone, having your LD game judged on a quantitative — yet extremely subjective — numeric scale can be frustrating in other ways as well.  This post and this post highlight some of these frustrations, including the following:

  • Numerical ratings encourage direct competition with and comparison to other games.  Instead of supporting each other, we can become frustrated when “bad” games score higher than ours and are incentivized to rate other games more harshly.
  • Focusing on quantitative scoring results in a larger emphasis on evaluating a game as great or poor based on a set of numbers rather than providing constructive feedback on what could be improved.  (What’s more important, a few well-written reviews or a single aggregate number?)
  • Some users have complained that taking a straight average of scores means that games with less ratings tend to have better chances at scoring particularly highly due to how variance works.
  • “Celebrity” competitors sometimes*** being able to leverage their past success and popularity to earn higher scores (probably without even meaning to).
  • “Gimmick” entries being voted consistently higher in certain aspects (e.g. rhythm games or audio-centric games regularly voted #1 for Audio).
  • Wide voter preferences for certain stylistic choices (e.g. pixel art, chiptunes).
  • Ratings that just don’t make sense or indicate that users to not understand how to rate games properly (e.g. Games with NO MUSIC OR SOUND receiving audio ratings between 1.5-2 stars).
  • Ratings that are based off of the post-compo version of a game, not the 48 or 72 hour compo version (you can tell by reading their feedback comments).
  • Cheating too easy (as described above in the previous section).

I am NOT trying to say that these factors are pervasive and that our rating system is inherently terrible.  These are just some potential disadvantages that I think people should be aware of and paying attention to.

Personally, I believe that the pros of a rating system can well outweigh the cons if handled properly.  However, I’m also interested and hopeful to see what a Ludum Dare without ratings will be like, if only as a test.  If it fails horrifically, we’ll know to never do it again in the future, but I’m hoping that we can prevent that and instead make this August LD a success by focusing on qualitative feedback and community interaction instead numerical ratings.

This is where you come in.

 

How You Can Help – Leave Detailed Comment Feedback

One of the main benefits of the categorical rating system is that it can provide you with a detailed breakdown of what areas you did well in and what areas you need to improve on.  This is extremely useful, but could we do the same thing by using comments?

Here’s an example of a comment that I would give in a typical Ludum Dare, alongside 1-5 star ratings for each category:

Cool game, and nice job on your first LD!  I especially liked the animation of the main character. Next time try to add some sounds!

For our August LD, we will not be using the categorical rating system.  Therefore, I will be using the following categorical comment template to provide more detailed feedback:

Innovation: This was not too much different from a standard platformer here, but I didn’t really mind.
Fun: The game was pretty fun at the beginning, but got a bit repetitive, especially the water level which I think involved too many difficult jumps.  Maybe adding another powerup would keep the game interesting all the way through?
Theme: The main character was a goat!  This definitely fits.
Graphics: I really liked the animation of the main character. The background for the fire level was a bit too distracting, though.
Audio: The music was a bit repetitive, but catchy. You should try to make some sound effects next time, with a tool like sfxr, bfxr, or Labchirp — it only takes a few minutes and goes a long way towards making your game feel more complete!
Humor: N/A
Mood: I really liked how each level had a different theme.  I think it worked very well to establish a mood.  As I mentioned, sound effects would really help out here!
Overall: This was honestly one of the better games that I’ve played so far this LD.  It was a little buggy and could definitely still use some work, but I would love to see you work on a post-compo version.  Feel free to leave me a comment on my game’s page if you do!

Congratulations on finishing your first Ludum Dare!  I’d really appreciate it if you could take the time to leave some feedback on my game in return.  You can find it at […].

IMHO, this type of detailed feedback is infinitely more valuable than a simple numeric rating.  I would highly encourage others to provide comment feedback in a similar style.

 

How You Can Help – Comment Back

Several people have expressed concern about the ability to receive enough feedback and comments on their games due to the lack of ratings and rankings as voting incentives.****

You’ll notice that at the end of my example comment I requested the game’s author to provide me with feedback in return.  This sort of “I’ll rate yours if you rate mine” trade has been a great way to get additional feedback in past LD events, independent of the “coolness” system.

For this Ludum Dare I will be making it a point to return the favor by playing and giving feedback to anyone who leaves feedback for my game.  It would be great if others could do the same.

In addition, we should continue to leverage the compo site itself to help out jammers in need of additional comments.  Don’t have enough feedback?  Post some interesting screenshots of your game and ask for comments.  Not sure what games you should play?  Make a post inviting people to comment with links to their games.  Let’s all help each other out! :)

 

How You Can Help – Curate Outstanding Entries

Another benefit of the rating system is the ability to select the cream of the crop based on the ratings accumulated through the voting period.  The best of the best should absolutely be selected and called out so that we can all see what truly amazing games can come out of a single weekend.

For the August LD, we won’t have any sorted ranking lists from which to select these entries from.  Therefore, the onus falls on us, the jammers, to informally curate the top entries.

Some LD jammers have already been doing this in the past by posting lists of their favorite entries, or even creating video compilations or twitch streams of notable entries.  This is great, and I would encourage more jammers to start making these sorts of posts, and/or upvoting other posts if you particularly liked a game or two.  I have personally never made a “My personal favorites of LDxx” post on this site before, but I plan to for LD36.  This is your chance to shine the spotlight on whoever you think deserves it most.

 

 

 

I know that this August LD has been on shaky footing and I understand that unfortunately, not everyone will be entirely happy with its outcome.  I really hope that the community can come together and give it their all despite that.  If we put in a little extra effort, we can definitely ensure that Ludum Dare 36 will be a great success.

(It would be awesome if some of these recommendations could be highlighted in a keynote or an announcement if people agree that they are good ideas.  However, I must emphasize that they are only my personal suggestions and again, I am in no position to tell other people how they should or shouldn’t handle LD.)

Thank you so much for reading.  See you all in a month, and I’m looking forward to playing (and commenting on) your entries! 😀

=====

Addendum: Regarding Other Options for the August Ludum Dare
There are a myriad of ideas and suggestions that have been proposed to change or improve LD’s systems, or different methods in which the upcoming August LD should be run.  Some of these ideas are really interesting, but I don’t think this is the time or place to discuss most of them as compo is already happening next month and it will be difficult to enact any meaningful changes at this point due to extenuating circumstances.  I’m personally happy with any decision regarding this, but chose to focus on the reasoning behind the current state of things as well as the ways in which we can make the most of it.
That said, I would like to specifically address the idea of preserving the categorial 5-star rating and scoring system, but not displaying an actual sorted “top 100” ranking of entries.  This idea seemed great to me at first until I realized that anyone could probably just scrape the LD database and generate the top 100 list themselves.  We already have people who are well-versed in scraping LD stats for this type of information — I don’t think obscuring the sorting order is necessarily the best way of solving this issue.
If we were interested in achieving something similar, I’d be curious to see what LD might be like if your game’s rating was sent to you =privately= (as in, not displayed publicly anywhere), as that avoids some of the issues involved with cheating and competition while still providing a numeric score for your own personal use.  But again, that’s not something that I can say is possible or even advisable.

=====

*I’ve since added rudimentary copy protection to my games to avoid similar incidents.

**If I am proven wrong by a community effort, that’s great!  However, such efforts are not something I am involved in nor are they the topic of this post.

***Yes, not always, I’m aware.

****Although I personally feel the opposite, since I have often left little or no written feedback on a game at all because I felt like my 5-star rating encapsulated my vote.

Tags: ,


35 Responses to “Some things to think about”

  1. g_o says:

    This is going to be, I hope, a detailed reply to remind you some of the points I raised you (hopefully not deliberately) dodged.
    Also I post from my phone so hopefully wouldnt come out too bad.

    I just wanna assure y’all – this post isn’t fixing or solving any of the arguments we were having. In fact, it merely touched one aspect of it.
    It just claims to without saying so. But don’t be fooled. There’s a lot of discussion to be made before getting the REASONABLE conclusion and not the EASIEST one.

    Now I’m really not trying to pick a fight but this was most certainly a reply post, and not the expaination post it claimed it was. It’s pretty clear you ARE trying to convince that we needn’t the rating system. One evidence os the lack of arguments and the vast number of solutions you could’ve proposed and present.
    Now, as for the old voting system – at last I hear some pretty good arguments regarding it. I very much hear you about all the “it’s not really your prize is your product” because indeed the serious racers (like you) get hurt and it’s reasonable.
    Now – THERE are and have been proposed solutions to this.
    Firstly, keeping the old system would require making a move along the lines of : private ranking. One can share his rank if he desires.
    Now, you also forget the very important point that winning is a big deal only because *we make it a big deal*
    If we could all agree that winning is meaningless, indeed it would be meaningless.
    Now I KNOW that people don’t work like that and that’s why the private ranking might be an option or maybe just omitting the top 100 though we still need to make sure people wont make clever scripts.
    There is another solution which removes the winning aspect but still maintains the ratings and the publicity which is just overriding the ranks so that they are calculated relatively to oneself. So basically how good were you at a certain category – say innovation, relatively to the others.
    All of these solutions make cheating meaningless since one only fools himself. Not others.
    There is no need to mention that if they are private there isn’t any reason to believe anyone sharing his results cause he might as well just faked them with photoshop or somethin, given that we can’t check his results.
    So even the old system isn’t doomed.

    Still there are so many options that you just didn’t mention at all!
    Remember that itch gives us a free solution I mean – couldnt be more easy logistically. But I already elaborated in my post for tons of solutions.

    What I like the most is that you tried to defend the comments proposition while confirming my fears and doubts about it.
    I already argued that comments are going to be a great hassle to get a good feedback. And indeed you recognized that and asked nicely but seriously without popping any bubbles – when a first timer will publish his unpolished game – no one would wanna comment on his.
    People now have the responsibility (assuming the coolness system) of commenting and they wanna play something worth their analytical comment.

    In short what you’re describing is fascinatically heavily depends on the fact that you were a serious competitor.
    In fact, your point of view is good mainly for the “elite”/running for the goal portion of LD.
    And just to remind you each LD there are about 3000 entries. So much winners can come from that. Now I’m not saying you should shut up or somethin very bruta; in fact I think you got some point we need to consider when thinking of a solution, but in no way what you’re saying is complete viewpoint of the problem let alone it’s a narrow minded solution.
    Sorry if I went hard, once again – I think we need to consider what you’re saying. I don’t think you’re right as utterly and absolutely as you think, let alone giving a complete solution.
    I do appreciate your ambition. I think if you’ll search for solutions* also *for our propositions it would be wonderful.
    Thanks for sharing your opinion, and for listening to mine

    • Pitƶik4 says:

      Your proposal for private rankings could actually be really good. Rankings ought to be given with some uncertainty, though, e.g. “You rated in the top 5%” instead of “You got first place”, so that you can’t prove to someone else that you (potentially fraudulently) “won” this Real Thing by logging into your account and showing them the scores.

      It may be too late to implement private rankings, much less fuzzy rankings, in time for LD36, though—creating things like that is exactly what we need a new site for, after all. But I think it’s an idea the admins should definitely hear about.

      As for the Itch Dare, I was initially supportive of that idea myself, but PoV’s post on it made it very clear why he didn’t want to give them a bigger slice of the Ludum Dare pie than everybody else. I know you don’t have a lot of faith in PoV and think we’re following him blindly, but he’s not just the Big Authority Figure of Ludum Dare. He’s a smart guy, and the arguments he makes would be just as reasonable coming from anyone else.

      • Pitƶik4 says:

        Huh. I tried to put a link to PoV’s post there, but it didn’t work. Just append this to the domain name, I guess: /compo/2016/06/10/on-the-itch/

        • g_o says:

          Don’t be silly I read and summerized my opinions in the exact post (last comment)
          Of course I listen to what he says but it’s exactly the LACK of co crete points which are relevent to us that Im expacting out of a smart guy. What he says sounds fishy, not smart I must be honest.
          I get everyone’s admiration but remember he’s out of the pic already. Therefore his tips sounds like a kid who wasn’t invited to the party giving tips about not drinking and arguing that it’s not healthy for your liver while he drank in the last one.

          • Pitƶik4 says:

            I brought up PoV’s concerns about the Itch Dare because you keep bringing up itch.io but haven’t actually addressed his points about it. I assumed you hadn’t seen the post.

            You haven’t really addressed anything PoV has said as far as I can recall. You mostly just talk about how he’s “out of the picture” and we shouldn’t concern ourselves with him. You talk a lot about what you want, but you don’t talk about what we want to prevent and why we shouldn’t be worried about it.

            I think PoV makes good points. You disagree. So tell me why, because I won’t believe that having the rating system fully enabled for the next LD is a good idea until you can convince me that he’s blowing things out of proportion. People losing a data point on their personal growth records isn’t a big deal. Disregarding the fact that I don’t believe a lack of ratings will reduce the amount of comments—in fact, I think comments will be way up this LD because they’ll have no other medium for giving feedback—people not getting many comments isn’t exactly a horrifying prospect either, if quite unfortunate. I mean, we could go without this LD altogether and be fine. Allowing people to push others down and reap benefits by cheating the system, on the other hand? Endorsing itch.io? These things could have real consequences for Ludum Dare and possibly beyond. If you disagree, tell me why.

            • g_o says:

              The thing is I don’t recall any none personal points that justify the rejection of such a cool offer.
              Now don’t pretend like I haven’t talked about the whys because Im repeating myself.
              If theres anything to add is that once again I hear “PoV is correct” rather than “I agree with X Y Z that he said”
              I want YOUR reasons not PoV’s

              • Pitƶik4 says:

                You obviously aren’t getting what I’m saying, but I don’t know how to make myself any clearer. So I guess I’m done.

            • g_o says:

              I honestly do not undersrand.
              Like I said in the other post-what are you claiming?

  2. sorceress says:

    You’ve made some good points DDRKirby(ISQ), and I agree with most (if not all) of it.

    I think much of this has been said before in dribs and drabs, but thanks for taking the time to put it all together, so that people can weigh it up as a whole. :)

    • doomista says:

      Good points, indeed. I love the commenting system suggested. I hope I’ll manage to finish my SFML extension library in time to release it to the public to be able to use it in my first time compo (I’ve only participated in jams so far) and I’ll definitely try to use this commenting style.

  3. Will Edwards says:

    ** Whatya want? Players! Whenya want it? LD#36! **

    The to-rate-or-not-to-rate discussion is going backwards and forwards focusing on the innards of cheating etc, but I feel we ought focus instead on the ‘bigger picture’, which ought be encouraging newcomers who have just made a game and are already feeling truly fatigued to go and play everyone else’s games. Scoring and coolness was just a way to achieve that surely?

    The vast majority of LDers are going to make a game and then crash and perhaps not even play a handful of other people’s games. Its these people we want to encourage to think of LD as more of a *community* where they can express themselves artistically and programmatically rather than a *contest* where they enter and then check back in 3 weeks to see what they got.

    Personally I think we ought run with the normal scoring just like previous LDs, and without any mysterious backstage effort by PoV neither. This just to use ‘coolness’ and wanting to get a score as a ‘stick’ to get everyone playing other people’s games. I know it can be cheated, and I know how to cheat it, and I think we’re grown up enough to shrug it off. As a security professional, if I were consulted to make LD uncheatable I’d suggest we go for collaborative filtering, which would dramatically change the nature of the contest and make it much more community spirited rather than leaderboard-producing, but still,… if we view the scoring as a means to an end and not the end itself then we can just look past its gamability.

    Personally regards commenting, I’m much keener on calling out something I liked about each game, and mentioning someone else’s game that did something differently, rather than the usual ‘constructive criticism’ approach where in the end only the worst bits and limitations are mentioned. Everybody is entering for a feel-good appreciation of their work and the outcome, and its nicer to get good news rather than bad.

    • sorceress says:

      I know that ratings are important to some people. They see them as being more honest than comments, and being a quantitative way of benchmarking their own game-dev skills. For some it’s all about self-improvement: People wanting to feel they are improving over time, and seeing their scores show an upward trend, which reassures them they are infact improving. Some enjoy the competitive aspect too, and the prospect of winning drives them to do their very best work. Some people take part because they want to win.

      There is nothing wrong with either of these attitudes. They are both very natural and healthy. But Ludum Dare is apparently the only large rated event of it’s kind, so for those who value the ratings part above all else, there is sadly no alternative.

      Meanwhile, there is an old proverb which I have been reminded of recently: That in China, the word for misfortune is the same as the word for opportunity. I don’t know if this is factually true or false, but I feel the proverb still has merit and relevance.

      LD without ratings is an opportunity for each of us. It brings into question our motivations for taking part, and perhaps encourages us to explore ourselves, to find enjoyment in new ways. New motivations. New goals. New experiences. we may even discover value in something which we didn’t before.

      At least, this is how I intend to approach this game jam :)

      • g_o says:

        You know I really hate your conformist and weak standpoint.
        All of your comments so far could be sumed as : “there’s nothing we can do” with the same rotten idea of “it’s already decided”.
        It’s bullshit and you know it, you’re just reinforcing the easiest way instead of thinking what’s really good for the community.
        Now I don’t claim I am utterly right, but I do claim that lots of my arguments have yet been answered and you keep ignoring them. I’m almost kinda gutted the way that (if i figured right) you got some organization privillages and you’re doin kind of mini bureacracy on this idea.

        • sorceress says:

          ok, moderating time.
          g_o: While you have created some discussion here, your excessive posting could be considered spamming (18 of the past 50 comments). Furthermore your tone of writing is consistently disruptive and disrespectful, and could be considered trolling. Please rein this in. Consider yourself warned.

          • Wumbo says:

            Thank you Sorceress, I completely agree and was about to say something about the tone as well.

          • g_o says:

            Yeah, keep me quiet. That’s the right thing to do. Keep the opposition silent.
            Oh and yes I have feelings too, calling me a troller for making huge efforts to speak for not so little amount of people who agreed with me is disrepectful as well.
            Warn me all you want, I’m gutted either way of your disrepection.
            About my disrespection – no one complained so far. I know I’m a bit excited about getting the voting system back but that was a dirty card playing there.very uncool.

      • Wumbo says:

        Also, I feel that one of the biggest benefits might be, if we can get the larger part of the community on board with DDRKirby’s idea is that more qualitative feedback is great. Yeah, its nice for people to give you five stars, but for me personally, I remember taking to heart one comment (i. e. “the clouds move way too fast-it’s distracting”) way more than a few one or two star ratings for art.

  4. HolyBlackCat says:

    Most points are really good. I start to understand why some people want to have the next LD without rating.

    But I don’t agree that numerical rating system is bad compared to other forms of rating. (I’m replying to that section in bold.)
    Some of the developers may be dissappointed by their ratings, but I don’t see it as a real problem.
    **Numerical system has beed working nicely for years, so we must not change it to a something completely different without a really good reason.**
    Of course, small changes are fine. For example, we can compute final rating based on something better than the average value.

    • Wumbo says:

      I agree too. At first I was somewhat disappointed that I wouldn’t get ratings for my game and was worried that perhaps that would result in less people leaving actual feedback because there wouldn’t be the incentive that there usually is to rate other games. This (and slightly more so PoV’s post that this was linked to) has changed my opinion.

      While I do agree that being disappointed in your ratings isn’t a problem per ipse, there are other, much more important problems we have to address. Some issues are perhaps that practically unpreventable (i. e. starting early or working on your game after the time limit) and can be controlled to a point by the honor system. Others, however, (such as voting only for your own game, multiple times at that, or speed voting to abuse coolness) are serious issues that may both be preventable and, if ignored, could harm the integrity of the ratings and reputation of LD.

  5. Wumbo says:

    Hi guys,
    First of all, thank you very much for the post DDRKirby. Its community members like you the make LD what it is.

    I just wanted to throw these potential solutions out there to see what other people thought of them. I am in no way a web designer and have no clue how hard any of this would be to implement, so this may be completely unhelpful, idk.

    In the short term, since there really isn’t much time to add entirely new things to the site, would it be possible to make the default comment to someone’s game an empty outline of what DDRKirby suggested as a comment, instead of an empty box. I would imagine that this wouldn’t be too difficult, and adding a short message to spend extra time commenting due to the lack of ratings in conjunction with this could significantly increase the quality of comments.

    In the long term, I think one of the biggest issues that needs to be addressed is speed voting. I think it would be helpful, as PoV said, to have some sort of team voting thing. I have no clue what that would be though. Then, if this is at all easy to do, it would be really cool if we could base the coolness system on the amount of time you spend playing and commenting on games. Perhaps like a 10 minute maximum per game would solve that.

    The fake account problem may or may not be solvable by having bots search though accounts to find suspiciously high ratings on suspiciously few games, particularly games with low coolness (i. e. games that were searched for instead of found from being next in the queue). I assume you could still game the system if you really wanted to, but at the very least it would discourage people from cheating.

    As others have pointed out, this could all be solved if ratings were still given, but private. People could still see their sense of progress/get quantitative feedback but wouldn’t be as inclined to cheat. The list at the end of the competition for all the super good games could be replaced as DDRKirby suggested with homemade personal favorite lists.

    How hard are these to implement (I suspect coolness based off time is very difficult), and any flaws with these that I haven’t thought of or mentioned yet?

  6. JaJ says:

    As far as I’m concerned the coolness system is still in place. Since the first LD I entered I found comments to be more valuable than ratings. I made use of the coolness list provided by LD to rate games, and I left a comment on all games I rated, but people that commented on my own game always had priority to receive a rating and comment back. I have noticed that this is true for many other jammers as well, as I received many comments in return for my own. All of this without even asking for it.

    I’m going to miss the rating system. I still desire to beat my score of 8th overall from a few jams ago, but the lack of a rating system is not going to ruin my experience. I see good arguments for and against the old rating system, as well as alternatives, and I really don’t have a solid opinion on what is better. I do feel like the keyword of LD is community, and I do have my doubts if the old rating system really fits in there. Either way, I see this ratingless LD as an interesting experiment, we don’t know how it will turn out until we do it, so let’s all have fun this LD and evaluate after!

    (ps, I like that comment template. I think I’ll use it!)

  7. g_o says:

    Well after getting some very insulting replies I no longer can take a discussion which refuses to be intelectual.
    It’s clear that no one’s really paying attention to what I’m claiming in terms of whats reasonable and somehow think my replies are trollings. Truth been told I’m not good with people. I just made my logical claims out there.
    When it becomes this political rather than rational I’m done discussing (not to mention getting warnings from the biased organnization)
    Not sure I’m going to get in this compo.
    I left you with some food for thought. Hope at least some of you is using their brain and truely rethinking it.
    If still you find it better than have fun.
    Either way I’m pretty tired of politics at this point. Im just a programmer.
    Just do whatever you want I cared before I care no more.

    • g_o says:

      I just think that what happened here in the last months is sad.
      I’m not just disappointed but also frustrared.

      • ajayajayaj says:

        I understand what your saying, but I think this has to happen. The new website is coming soon, and one break from ratings should not be too bad. I’m sorry if I sound mean or sound like I’ve given up, but it is just the case. I would love an LD with ratings, but I’m happy we have an LD at all. PoV is working on the website so we can’t do ratings here, and hosting the ratings else where sounds like a great idea in the short-term, but could damage LD in the long-term.

        Thanks for bringing up the issue, it is something I was worried about it too, but now my opinion has changed. I think that it will be fine.

        • g_o says:

          “this has to happen” is just incorrect and obviously biased.
          Uh I don’t even want to get into another political argument, I guess no one really understood what I was saying. English isn’t my first language after all.

  8. FrederickK says:

    Sign me in!!
    I totally agree with everything Kirby said and I must say, that Ludum Dare is not a place for novice. I am sorry, but it is not.
    I have seen some devs sky rocket because of they went well on Ludum Dare, it matters A LOT.
    It is VERY UNFAIR a totally beginner compete with professionals, specially when they use assets that they made before hand.
    I have tried for so long to give people the spirit, “it is not a competition! let’s just make good games!” but it is not what happens. It should have more strict rules and be better judged, a lot is in stake, we can not have a team/dev the goes over the rules and say, “it has no prize, do not worry so much”. This kind of post appear every-time after/before jams, is not a minor issue.
    I liked the format for the feedback and to post the favorites entries, I will try to do them the best I can!

  9. Avant-Marde says:

    “Subvert the time limit — when submission hour ends, submit a broken link to the game”

    Just in order to make a point in case the new system is somehow going to try and do something about this: people might actually do this not to cheat, not to continue working on the game, but because some sort of problem has cropped up during submission hour. Export tools breaking down, computer freezing, upload speed slowing to a halt. Things that could jeopardise the submission of one’s entry that one has worked hard on for two or three days. That would suck. So entirely preventing this isn’t a good option either IMO. Worth considering when setting up the new system.

    • sorceress says:

      In a perfect world, software would be without game breaking bugs. But in reality there are. Especially in time limited jams like these, bugs will be commonplace. I agree that it would be cruel to prevent people fixing+updating their games, after spending 48/72 hours working on them.

      So I don’t think we’re saying that fixes/updates should be forbidden, but we must recognise that they do nonetheless provide a way for people to work on their games beyond the time limit, adding new content+features. And given that fixing and expanding are technically so similar (adding code and recompiling), I can see no way to prevent one but allow the other. Essentially, we’re faced with the problem that cheating and not-cheating are indistinguishable, so we should work on the assumption that they are indistinguishable.

      To fix this kind of problem, we must look beyond it, at the bigger picture. We need to address the motivations for cheating, with a rating system that’s designed to absorb the effects which cheating can have. ie, by grouping games into broad tiers (percentiles) instead of an ordered list of numerical rankings. The effects would be:

      (a) The winning games would arguably be all those in the top tier. The publicity “prize” for doing well would be spread thinner over all these games, which are all considered equally worthy of praise. The reward for winning would be much less, but more people would be winning. I believe this would be more closely aligned with the spirit of our community than the current system is.

      (b) Climbing a few places up the ordered ranking table is easy by cheating a little. But it would be much harder to climb into a new tier, so both the motivations for cheating and impact of cheating would be reduced by this proposed system.

      (c) When people receive a rank like 429/1850, that is quite a depressing figure to look at, even though it is in the top 25%. Realise that the vast majority of the community will be looking at figures like this. I believe that tiers would feel more cheerful, and less directly competitive than a three/four digit number.

      Over the past 5 years, LD has grown tenfold in size. At it’s current size, the rating system is highly unstable. Each person has the power to push a game up/down the overall rankings by about 100 places, by giving a punishing 1 star instead of deserved 5 stars for example. That’s just how bad it is. Unless we can force people to rate 10x as many games (impossible!) there is no fix for this. Rankings must go! Tiers are the future!

      • Avant-Marde says:

        Woot, you totally misunderstood. I wasn’t talking about fixing bugs at all. .-. I was talking about technical issues preventing you from exporting your executable or uploading it. Not touching a single line of code after the timeline. Just getting the game deployed. *That* final crucial part can fail sometimes.

      • Avant-Marde says:

        And I didn’t mention rating at all, so no idea why you delved into that. D:

        • sorceress says:

          Because problems don’t exist in isolation; they need to be seen in their surroundings. eg, so that we’re not compounding problem A whilst addressing problem B.

          • Avant-Marde says:

            Well, sure. Based on your post, which doesn’t really seem to have a conclusion, I can’t tell for sure where you stand when it comes to the sole concern I had, so that had me a bit confused. But maybe you were just offering me some food for thought in order for me to come to my own conclusion?

            TBH I don’t personally have any real issues with the old system (despite what you and OP have brought up), so I can’t say much. I guess I’ll trust the people who are concerned about these things to make some wise decisions. Just wanted to throw the only concern I personally had in here in case people hadn’t thought about it, to make sure that it isn’t overlooked. Making sure people don’t fail to submit their hard work is more important than making sure people can’t cheat IMO. :/

            Completely off topic now that I’ve got the attention of someone who knows their way around the LD site, BTW: are there notifications for replies to one’s comment anywhere on the LD website? I’ve checked, but haven’t found any. So I have to remember what threads I’ve posted in and then find them again to check for replies. :c

            • sorceress says:

              There are two fall backs in place already: the submission hour gives us that extra hour to get our games to compile and uploaded. And if needed we have an additional 24 hours if we’re willing to move our games into the jam instead of the compo.

              A couple of times I have had to abandon an LD game in the latter stages, because I wasn’t happy with how it was coming together. While that isn’t the same as the technical hurdle you describe of getting something compiled and uploaded in time, it is still a failure to submit. So it is emotionally similar.

              Truth is I don’t feel strongly one way or the other about that. It’s not nice to fail, even if it’s down to something outside of our own control. But sometimes that’s just how life goes, and we need to be resilient in the face of defeat.

              To some extent we can plan ahead: stopping to compile earlier, having a backup plan for hosting in case that fails us. Having more than one internet access point in case one goes down at the worst possible moment.

              In my experiences, the sadness only lasted for a couple of hours, until I came to terms with defeat, and ended up feeling respect for all those who managed to do what I did not! 😀 I remembered that I still enjoyed my weekend, and I could still enjoy seeing what others had made over their weekends. And furthermore, I knew I could still share my game with friends or post it on the blog at a later date for public consumption if I wanted to, just that it wouldn’t be rated. :)

              There are no notifications in WP, but there is this page for all your commentary needs: http://ludumdare.com/compo/wp-admin/edit-comments.php

              • @Avant-Marde: I’m in total agreement that making sure people can submit their hard work and allowing for leeway is more important than being ruthless and precise in terms of disallowing cheating of any form. To be honest I don’t think there is an easy practical solution to subverting the time limit and I think relying on “discretion” and the honor system (as we have) is still the most pragmatic option given the nature of LD. As sorceress has suggested there are various modifications that may or may not be useful towards facilitating this as well as disincentivizing abuse.

                I hope this post didn’t come off as me claiming that the rating system should clearly be abolished because of all of the issues with it. In spite of all of the problems I presented (which I think people should be aware of), I actually think previous LDs have been great overall and I wouldn’t mind having the old ratings come back (even with all of their flaws). However it was becoming clear that our upcoming August event will be run without ratings and I wanted to provide some missing context for that. More importantly, I wanted to bring some positivity and hope to what seemed to be a very negative turn of events, and also encourage people to make use of this opportunity to see exactly what can come out of an LD that is run differently.

          • Avant-Marde says:

            Also I feel clunky as ever trying to communicate in text with people I don’t really know online, so I hope I’m not coming off as rude or anything. No such intentions.

Leave a Reply

You must be logged in to post a comment.

[cache: storing page]