Compo Process Feedback

Posted by (twitter: @ludumdare)
April 22nd, 2009 12:32 pm

Hey everyone.

Since you’re all busy busy playing and voting for entries, I thought I’d open up a thread to collect some suggestions. We’ve already seen many great ideas come up in IRC, but they should really be written down. So, any thoughts on the voting process, submission process, or anything to improve how we run a compo, drop a comment to this post.

With such an big turn out for April, it looks as if we’ll have to start taking things more seriously to manage the growth. If you’re a PHP+Database+Wordpress guy (gal) that wants to help us out, stay tuned.

For now, I’d like your thoughts. Respond away.

– Mike Kasprzak (PoV)

35 Responses to “Compo Process Feedback”

  1. PoV says:

    My own suggestion I’ve been sitting on, a “submission system”. Essentially a web form that you use to submit entries to the competition. Game Name, your name (possibly associated with your wordpress account), primary screenshot, extra shots, downloads for various platforms (local uploads, and off site link support for flash entrees).

  2. Hamumu says:

    I want to harp on my issue! RULES. We had a lot of people this time who just flat out didn’t know the rules. I think we need to hammer them down into clear, concise form and make them boldface and inyoface and big and upfront for all entrants. So everybody is really clear on exactly what’s what. There will always be questions about what lib is okay and all that, but at this point, people don’t even realize they have to make all the content themselves!

    I’d like to see that kind of streamlining and straightforwardizing applied everywhere possible. Knock out all those common questions. Make a guy’s timelapse go in a special place on his entry page so it’s instantly clear if he has a timelapse and where it is (a youtube embed would be bestest even!). The entry itself too – when you are done, there’s some form you type your links into (one spot for game, one for source, a screenshot, a drop-down for what format it’s in and what it requires), and that generates a little ‘index card’ with that info that shows up at the top of the entry page, so no digging through the journal to find it. I don’t know what else, I’m just all for simplification and clarity. Formal is nice.

  3. Hamumu says:

    So ninja’d by Canuckistan.

  4. Fiona says:

    If you need any help code-wise I’ll be happy to lend a hand. I (unfortunately) spend my day job writing PHP so I know my way around it. Shoot us an e-mail on fiona at myrmidonprocess dot com. (I’ve done LD a few times so you know I’m good for it :).)

  5. jovoc says:

    I think some kind forums would be top of my list.

    It would also help with the rules/process questions too.

  6. HybridMind says:

    I think it would be fun to have a way to respond to people leaving you feedback on your game in the judging page. Also.. there is no way to really leave a PM for anyone either as an alternative.

  7. DrPetter says:

    My thought was to have a system for indicating compatibility of each entry, displayed on the main entry list page (so you won’t spend time trying an entry that will fail on your system).
    I imagined having three OS icons (win/mac/linux) that were all enabled by default, and voters could disable any of them if they found the entry didn’t work on their system (and if they are pretty sure it won’t work for others either, like entirely missing win32 build).
    The original author would be able to completely set/reset the icons and perhaps also disable the whole feature for his game if he suspects abuse. Often, the case would be a missing win32 port, and the author could switch the windows flag back on when one is uploaded.

    There are issues of course. It should be both convenient and “quick” (meaning it’ll work even if only a handful of people have tried/failed running the entry), but also not unfairly flag entries wrongfully.

    Perhaps some combination of minimum number of people removing an icon before it’s actually shown as removed (like 2 or 3), and/or requiring everyone to submit a non-empty reason for flagging (and maybe others could read+disagree and cancel that flag).
    It’s probably difficult to deal with the case of an inactive author who gets flagged. If the entry is flagged, few will bother looking at it and the flags might not be questioned and adjusted until the author logs in and does something about it.

    A simplified version that might be better in practice is if each author gets to pick the icons when posting the game. There could be some procedure forcing him to make a decision and not just (inadvertedly?) accept some default values, OR the default could just be “works on all systems” which is essentially what we have now.
    It would still have to be editable at a later stage, in light of porting etc.

    Final issue is with python/script languages that technically works on windows but requires installations/tools that not everyone has. I guess it might be relevant to have a more elaborate set of “gotchas” that could be enabled individually for the game. Like “browser game”, “windows-only”, “no windows executable”, “requires XNA”, “won’t run on windows”, etc… Can’t be too wordy though, as it would be best to just glance it directly from the entry list page. If it was implemented as a set of graphical icons, it could work.

  8. DrPetter says:

    Seeing PoV/Hamumu chatter on above, I guess my platform icon stuff could be a part of that specific+simplified submission form and entry page. I’d still like to extend the compatibility info to the entry list though, so you can easily see which entries are unavailable to you and which ones have recently added ports/fixes.

  9. noonat says:

    Submission and entry presentation streamlining above would be huge.

    Voting is also very cumbersome with this many submissions. Would be nice if you could just expand an entry line in the grid to show the “index card” for the game, comments on it, and have voting options.

    If you added these things, it would be really awesome if it were also exposed via an API so someone like me could write an Air desktop app to do it.

    On that note, my alter ego is a js+php developer, and I’d love to help if you need it.

  10. mjau says:

    To expand a bit on DrPetter’s idea, each different VM could be considered its own platform. People could then select which VMs they have installed (default to all for lazy people), so that the compo system could hilight entries they’d be able to run. Only native executables would be marked as running on windows/linux/mac.

    For example, a py2exe’d game could be marked as running on python and windows. So, people wouldn’t have to guess what platforms their particular scripting language runs on, but just select the VM and, optionally, other platforms known to work.

    Not sure how practical this would be though. A VM would have to be added to the system to be able to be selected, and there could end up being a lot of them. Also, there’d still be things like VM version differences and extra library requirements to worry about.

  11. noonat says:

    @mjau: Maybe just simple tags would be sufficient? Could have standard tags for os+language (but still allow custom tags) and allow people to filter the list by tags.

  12. mjau says:

    noonat: It might.. Problem with tags though is that they’re very informal. People have very different ideas about what should go in a tag, many tend to just write random things that pop into mind at the time, making up alternate names for tags and forgetting to tag more obvious things like what platform it runs on. I mean, we’ve got some standard tags right now, but people still get it wrong all the time, if they use them at all.

    Also, even if tags are used, they don’t really say anything about *what* in the post (or entry) they apply to, or even what it means. Someone could make a game about a python snake cleaning windows, release it for mac only, and then tag it with python and windows (obviously forgetting to tag it with mac or os x or whatever ;))

  13. Draknek says:

    I’ll second the suggestion of a forum. IRC is great, but something more persistent would be nice.

  14. PsySal says:

    I’d like to see a different format for the final competition entries page.

    Not every game is going to get played by everyone, but the system is exacerbated when it’s “Creator Name (# Reviews)”, because people will gravitate towards the more well-known community members and also those games that have already garnered more reviews… So what happens is that some games get tons of reviews but others don’t get very many at all!

    Maybe it could just list game name, in an order randomized for everybody?

  15. erik says:

    I think it would be good to have somewhere that shows off the cool things that have come out of the LD community. If someone hears about an LD48, and wants to try some of the best games that have resulted from the compos, it’s currently really hard to find those games.

  16. Gilvado says:

    Personally, I think that as an addition to PoV’s and Dr.Petter’s suggestions we should have a spot on the form where we input what platforms we *can* test, and then once every entry is in, an automated and randomized process should distribute 20 or 30 games to every person to vote on. Obviously you can still vote on any game you want, but if we required these minimum votes, then we could guarantee that every game gets enough votes, and once a person has reviewed and voted on 30 games, they know that they’ve done the minimum requirement. If the compo continues to grow, then *eventually* it will be completely impossible to vote on every entry. If by that point we already have a system in place to guarantee a random distribution of enough votes for every game, we’ll be ahead. Just think what it would be like if there were 400 entries…

  17. georgek says:

    I second an option for a randomized list — of course you could do this yourself, but it’s very convenient to have it on-site.

  18. 5parrowhawk says:

    I second DrPetter’s suggestion about platforms.

    Also, having an organized torrent for final entries would be great.

    A centralized page showing where to get each third-party lib (pymunk, ruby, XNA, etc) would also be good. Or maybe even a second torrent for third-party libs. Of course you would have to declare what you’re using before the compo, so the organizers have time to assemble it.

    And I support Gilvado’s idea. Excellent plan there.

  19. mrfun says:

    1. How about a sidebar that shows the last ten comments posted? I find these help me navigate where the action is happening on other sites.

    2. After clicking someones name, there should be a link directly from there to their entry for voting, instead of having to hit that main list every time

    3. Annons/non competitors should be able to see the voting comments also, I don’t believe they currently can…

  20. robot_guy says:

    Another vote for the the platform suggest. Not running windows it would be really nice if I could easily find all the games that I could run rather than having to hunt them down.

    However, a small plea from the bandwidth / transfer limited:

    Could we also strongly encourage people who use VM type platforms (python, Love, etc) to also provide non-packaged versions in addition to any packaged ones. As nice as it is to have a one click to play packages I’d prefer not to download the same 8 or 9 MBs of DLLs / Frameworks half a dozen times.

  21. nilsf says:

    As mentioned above, a forum would be nice for communication that doesn’t warrant a blog post but should be more persistant than IRC. (besides I suck at multitasking so I’m not a huge fan of IRC)

    Like HybridMind suggested, being able to respond to feedback would be great.

    Being able to filter platforms would be quite nice too.

    A list of all trophies would also be neat.

    On the other hand I don’t feel a whole new submission system is urgently needed, the blog thing works pretty decently IMO.

    And the 3 points mrfun raised would be nice (especially the last two).

    I also agree with the rules issue, I didn’t notice many cases of illegal content but it seems a bunch of entries didn’t include source code.

  22. greencow says:

    Make all contest entrants have badgers on their toes slowly gnawing the toes, then the feet, then the legs, and so on until only the head is left at the end of the 48 hours. This would give a better sense of time and without bodies entrants would have little excuse not to judge every game.

    The winners in each category would have their brains extracted and mushed together into a motherbrain that would go on to conquer the world. Everyone else would get robot bodies and jobs as minions of the motherbrain.

  23. jolle says:

    We need a votomagicotron. You click a button, and are presented with an entry to vote.

    It’ll magically select the entry of the entries you haven’t yet voted on by a weighted rand, where the weight is 1 + 1/(1 + nr_votes_already_got) + 0.5 * has_voted_for_you.

    You all know you want it.

  24. nilsf says:

    I like the votomagicotron idea but I’d also add the number of entries the author has rated to the weight.
    Also as someone above suggest maybe have a minimum number of entries to rate in order not to be disqualified.

  25. Banni says:

    According to the map ( there are more Europeans than North Americans doing the competitions. How about making the start time a little more Euro-friendly, by bringing it back by a few hours?
    A 4AM start/finish time in the UK made things quite difficult, compared to around 10PM as it would have been in the US

  26. erik says:

    In addition to what I posted earlier, I have a few more thoughts:

    Maybe Pellies could automatically be awarded as part of the Trophy system?

    It might help encourage more voting if people were give an score for how many games they voted on. Maybe count the number of votes cast + the number comments made. Then in the final results, score people by how their judging contribution. I know Hamumu did something like this for the LD48 he hosted. But that was before my time. How did that turn out?

    What else can we do to encourage more feedback? Maybe give each judge 20% of the list to concentrate on, with voting for the rest being optional?

    I also have thoughts regarding Hamumu’s comment. I’m all for making the rules and expectations clear. But it would be less fun if the compo became strict about enforcement. It would hurt the community if your entry didn’t count because you submitted 30 seconds late. We would be worse off if Rara Racer had be removed from the voting because it used non-original youtube graphics.

  27. greencow says:

    rara racer was a great game, but it’s not a game which is appropriate for ludum dare. it’s not worth making a fuss over at this point, as long as it serves as a reminder that ludum dare games are all about original content.

    what i’m looking forward to is the distant future, where games can be submitted through the website and through interweb magick would allow for people to play and judge all the games through a virtual machine website. This would virtually eliminate the handling time of games which could double the number of games anyone could play in the same amount of time, as well as eliminating issues of not being able to play certain games because of libraries (i’m looking at the py-people here).

    Initially it should be enough to have an optional content upload/download system which could remove the burden of finding hosting from developers and simplify downloading for judging. This could be funded through donations.

    I don’t think requiring people to judge is a good idea, but encouraging people to seek out games which have fewer ratings to judge would help to smooth the numbers. When looking for a game to judge I look at the games list for low numbers in parenthesis. Maybe make the default sorting with lowest ratings/comments at the top. Then again, this may give early voter bias to some games as the procrastinators won’t make it to the bottom of the list. As long as they’ve got a common frame of reference I suppose, it’s strange to see ratings of 1 across the board on your game, makes you wonder. The use of expert judge panels is difficult, especially for a casual event such as LD. I guess that with a higher quantity of ratings such things will be smoothed out. What if it were youtube? Thousands of ratings per game. That’s all moot with a content viewer base which is roughly equal to the content creator base, something which is not likely to change without at least the sort of web accessible content delivery system described above. Then there’s the marketing, do you know how much it costs to get ten bikini clad volunteers together to do a commercial? Nothing, they’re volunteers. Just take your camera to the beach and start filming, cause spring is here!

  28. mjau says:

    Something strange I’ve noticed: When commenting on an entry, your comment is inserted between the other comments at a seemingly random position. Submit the comment again (edit or not doesn’t seem to matter), and it moves to somewhere else. Usually the position alternates between in the middle somewhere and at the end, or sometimes the beginning, but not always. Would be nice if new (and maybe newly edited) comments always were inserted/moved to the end of the list (or beginning, as long as it’s consistent).

  29. PoV says:

    A good suggestion from IRC. A more streamlined sign-up process.

    osgeld> you really should have some type of form, like some simple, stand alone job you can link from the homepage, that way it gets better coverage, and better results
    PoV> Not sure what you mean. We have a mailing list if that's what you're suggesting.
    osgeld> no, a "help" link on
    osgeld> and link that to some generic sign up sheet, like softball
    PoV> Ah, gotcha.

  30. PoV says:

    I can’t remember if I added this yet or not, but the thought was to have some “porting support” people around. We can impose some restrictions here to help streamline things. For Python, Py2Exe’ing with PyGame and similarly common libraries. Similar arrangements for SDL and/or Allegro perhaps.

    Again, just a thought. Will have to do some actual thinking about this.

  31. sol_hsa says:

    how about..

    48h to make a game
    1 week to port (and fix deadly bugs)
    2 weeks to vote

    original 48h source and binary should be included in case of ports and fixes are posted. After the week of cooldown is done, torrent would be available.

    Actually, two torrents. First torrent available right after 48h, second after the porting period. Having a week between the contest and voting would probably also let people rest and get some distance to things (as well as just play the damn games).

  32. PoV says:

    – Remove, or adjust tags to only include tags used recently (last month or two)
    – Make user list show only people that have made a post in the last month, two, or six (when “minimized”)
    – Change the trophy name to be a popup (popdown), so you can pack them closer together in a profile page.
    – Do layout first, then fix stuff like trophy display.

  33. nilsf says:

    Have the result page also indicate rank for each entry.

  34. Jonny D says:

    I’ll summarize what I think are some good ideas:

    – IRC doesn’t cut it if you need to leave your computer for more than 10 seconds.
    *Platform Indicator*
    – With mjau’s input that it should separate interpreters (Python) from operating systems (Windows). To solve DrPetter’s problem, the icons should be voted on. A game originally has ust a big question mark for the platform. When rating the game, there is a rating (1-5) of each platform. When there are 5 platform votes, then the icons appear with a bar under each, indicating the average of the votes for that platform. If a platform is under 1 for an average, then it does not appear.
    *Application form*
    – Is this what osgeld was thinking? There should be a competition entry page where you are presented with the condensed rules (no more than a paragraph or two). Then you press ‘I agree’ to be allowed to compete.
    *Viewing ratings*
    – MrFun’s idea and mjau’s idea. Rating comments should be available to visitors and the comments should appear in a consistent location.
    – jolle’s idea.
    *List game name with the entry*
    – PsySal’s idea.
    *Minimum voting limit*
    – Gilvado’s idea. I think it should be 20 games, but what would be a fair and non-offensive indicator for not reaching the minimum? You cannot reject an entry due to the creator voting on fewer than the minimum.
    – erik’s idea. At least award trophies for 1st, 2nd, and 3rd on each of the categories.
    *Porting support*
    – Maybe add a tag to your final entry: “Needs porting”
    *Bugfix torrent*
    -Sol_HSA’s idea.

    I hope my summary helps organize this discussion, too.

  35. greencow says:

    Giving a week for porting is already part of the rules, ports are allowed after the 48h is complete, for the whole two weeks even.

    Critical bug fixing after the 48h should really not be necessary, what with all the astute programmers we are here. Of course, a showstopper that is fixed in hour 49 will be okay, but in general I think it is contrary to the 48h dev cycle to give extra time for bug fixes. That’s what hour 48 is for ;]

Leave a Reply

You must be logged in to post a comment.

[cache: storing page]