Skip to Content

random contest scoring musings

5 replies [Last post]
jwarrend
Offline
Joined: 08/03/2008

(moved here per hpox's request...)

I hate to be the fly in the ointment, and indeed, it doesn't really matter to me because I didn't even end up entering the competition, but have the judges changed? Or was it always the case that different entries would be judged by different judges?

Either way, it seems to me that while the categories are the same, the way this judge assigned points is different than the way that the previous judge did. As such, I wonder if there's really a legitimate basis for comparison between the different entries such that you could coronate a "winner".

Now, you might say "well, it doesn't really matter if you win, the point is just to have fun and enter". But if that's the case, drop the judging pretense altogether and just post all the games here on the board and we can judge for ourselves which were the "best".

I don't know; it seems to be a really tough aspect of having a competition that a lot of people are going to enter: how do you get judges to evaluate all of the games on the same scale, except to have them evaluate ALL of the games? Yet, how can you ask judges to commit to doing the latter?

Perhaps for later competitions you should use a "Hippodice-like" method where there's an initial "filter" stage, and then the finalists will actually be playtested by the judges. The fact that that club judges the 50 or so games that make it to the finals by their whole group playing them all Spring strongly suggests that trying to judge 25 entries by playtesting with only a couple of judges was overly ambitious. Hindsight is 20-20, of course, but it seems that there are changes that can be made that would make future contests better. (I would probably say "must be made")
Maybe also charging a small fee to enter, and giving the judges a small reward would also be appropriate?

As for this contest...if I had entered, I'd probably be suggesting that we just drop the scoring altogether at this point, since it's taken so long and since the scores may be relative anyway. But I couldn't presume to propose that for the other people, who presumably worked hard on their entries and would like them evaluated.

Just my thoughts...

-Jeff

phpbbadmin
Offline
Joined: 04/23/2013
Re: random contest scoring musings

Jeff,

These are all valid concerns. Let me first give a brief summary of how the competition was setup.

I approached Alien Menace about helping to judge the contest. He agreed. He accepted the duty of head judge and also recruited all of the other judges. All of the entries were to be submitted to AM, and then divided equally among all of the judges. Each judge was to play and review their entries, and then recommend the games that they thought were worth further review. After all games were given the initial review, then ALL of the judges would judge each of the recommended games. Then the judges were to decide the rankings for these recommended games.

So basically, all games got at least one review, and if it was a really good game, it would be given further review by all the judges, afterwhich they would provide a ranking for each game.

Now what REALLY happened is that head judge Alien Menace found himself too busy to fulfill the duties. XXOOCC took over the job, only to find that all but two of the other judges (Alien Menace and Wolf) were also too busy to even judge. Couple that with XXOCC's computer crashing, and it has been a long, hard road for the competition.

SOOOO, at this point, I just want to make sure that everyone at least gets their game reviewed. The contest was never really meant to be too competitive. With bragging rights being the only prize being offered, I don't feel that objective ranking is really necessary. Also, I want to maintain some sort of credibility in case we want to hold future contests.

It was also quite a learning experience for our first contest. Lord knows we won't be making the same mistakes the next time. I agree that using a large gaming group is ideal for judging game entries. Obviously Hippodice has already proven that this system works. I think the reliance on a handful of static judges is not the best way to go. It is just too time consuming for them, not to mention the obvious difficulty in finding commited judges in the first place.

As for posting the games for the entire community to review, well that's great, but that's not what these designers signed up for. Some of them might not want their game posted openly on the Internet for the whole world to read, and we have to respect that. At this point, I'd like to see each and every game submission reviewed and scored. Then we can bring the contest to a close and perhaps think about starting a new one. We can't start a new one without bringing this one to closure.

-Darke

jwarrend wrote:
(moved here per hpox's request...)

I hate to be the fly in the ointment, and indeed, it doesn't really matter to me because I didn't even end up entering the competition, but have the judges changed? Or was it always the case that different entries would be judged by different judges?

Either way, it seems to me that while the categories are the same, the way this judge assigned points is different than the way that the previous judge did. As such, I wonder if there's really a legitimate basis for comparison between the different entries such that you could coronate a "winner".

Now, you might say "well, it doesn't really matter if you win, the point is just to have fun and enter". But if that's the case, drop the judging pretense altogether and just post all the games here on the board and we can judge for ourselves which were the "best".

I don't know; it seems to be a really tough aspect of having a competition that a lot of people are going to enter: how do you get judges to evaluate all of the games on the same scale, except to have them evaluate ALL of the games? Yet, how can you ask judges to commit to doing the latter?

Perhaps for later competitions you should use a "Hippodice-like" method where there's an initial "filter" stage, and then the finalists will actually be playtested by the judges. The fact that that club judges the 50 or so games that make it to the finals by their whole group playing them all Spring strongly suggests that trying to judge 25 entries by playtesting with only a couple of judges was overly ambitious. Hindsight is 20-20, of course, but it seems that there are changes that can be made that would make future contests better. (I would probably say "must be made")
Maybe also charging a small fee to enter, and giving the judges a small reward would also be appropriate?

As for this contest...if I had entered, I'd probably be suggesting that we just drop the scoring altogether at this point, since it's taken so long and since the scores may be relative anyway. But I couldn't presume to propose that for the other people, who presumably worked hard on their entries and would like them evaluated.

Just my thoughts...

-Jeff

jwarrend
Offline
Joined: 08/03/2008
Re: random contest scoring musings

Darke, I can totally understand the woes and how they came about. It was a cool idea and I admire you for putting it together and try to work a solution even in the face of adversity!

But....

Darkehorse wrote:

SOOOO, at this point, I just want to make sure that everyone at least gets their game reviewed. The contest was never really meant to be too competitive. With bragging rights being the only prize being offered, I don't feel that objective ranking is really necessary. Also, I want to maintain some sort of credibility in case we want to hold future contests.

If you're putting emphasis on "too competitive", that's fine, but it was always my understanding that this wasn't just a review opportunity, but a contest. If it was review of a game that you wanted to provide, there shouldn't have been a theme at all. I chose not to enter because I wasn't excited enough about my game to really finish it, and that was because I was working on other projects. If I had entered and the only outcome was "your game will be reviewed", I would have said "what a waste! I have other games I care about way more!" So I think that having some sort of objective rankings had to have been an important ingredient of a contest, or else why do it? Obviously, the prize wasn't a concern to anyone, but I think objective evaluation would have been.

With no disrespect to Wolf intended, I find his reviews to be different in quality than XXOCC's, and would be upset if I had drawn him as a judge rather than XXOCC. I think of one game's originality, he simply said "Yawn." So it's not even providing feedback, just a number for which we have no basis of knowing how he arrived at it.

I completely agree that future credibility is important and that's why I posted my remarks. I would love to see this contest become a legitimate and serious competition. And I do understand that things outside of your control have happened. But when you say "we never meant the scores to mean too much", that makes me feel like there were problems in the contest from the inception, and that there were parameters that were never obvious, at least not to me. Do you feel that you adequately communicated the idea that "we don't feel objective ranking is really necessary in this competition"? I guess I just never got that sense (though I agree that it was clear that it wasn't super serious and cutthroat.

Quote:

As for posting the games for the entire community to review, well that's great, but that's not what these designers signed up for. Some of them might not want their game posted openly on the Internet for the whole world to read, and we have to respect that. At this point, I'd like to see each and every game submission reviewed and scored. Then we can bring the contest to a close and perhaps think about starting a new one. We can't start a new one without bringing this one to closure.

I agree, but zaiga, for instance, has already pulled his submission; maybe others would do likewise? If you told me, for instance, "hey, we're having trouble with getting enough judges, and all we can promise you at this point is a review, and that review might say nothing more about elements of your game than "yawn"", I know that you could get me to pull my game in a heartbeat. If, of course, I had entered in the first place...

-Jeff

phpbbadmin
Offline
Joined: 04/23/2013
Re: random contest scoring musings

Jeff,

When I said this:

Quote:

SOOOO, at this point, I just want to make sure that everyone at least gets their game reviewed. The contest was never really meant to be too competitive. With bragging rights being the only prize being offered, I don't feel that objective ranking is really necessary. Also, I want to maintain some sort of credibility in case we want to hold future contests.

I meant this regarding the current situation. As I said, initially the contest was to have an 'initial screening process' by the judges, and then the best games were to be completely reviewed by all judges. Why was it like this? So even though your game might not have made it to the final round, at least you had a judge look over, play and review your game. I think that counts for something! And as I also said, AT THIS POINT I AM TRYING TO SALVAGE THE SITUATION! That means we will have to cut corners! And that means we may get reviews that are less than
adequate.

I understand that designers who signed up may not be getting what they signed up for, and I apologize. Understand also, that I was only supposed to be a mediator between contestants and judges. At this point, I am basically running this contest blindly. I don't have access to the entries, I don't even have access to their names or e-mail addresses. The only thing I have access to is the judges via e-mail. So basically I can't even e-mail the contestants to let them know that there was a change in the contest process. The only thing I can do is post whatever information I have on this forum. That's it. Please forgive my harsh tone, but I am just as upset about how the competition turned out as anyone. But I can't change the past, I can only do my best with what I have now.

-Darke

FastLearner
Offline
Joined: 12/31/1969
random contest scoring musings

How about we save this for a "Contest Review" thread (or even better a "Suggestions for the Next Contest" thread) after the contest is over. Mid-stream when nothing can really be changed is a bad time to tear it apart, imo.

jwarrend
Offline
Joined: 08/03/2008
Re: random contest scoring musings

Darkehorse wrote:
Jeff,

When I said this:

Quote:

SOOOO, at this point, I just want to make sure that everyone at least gets their game reviewed. The contest was never really meant to be too competitive. With bragging rights being the only prize being offered, I don't feel that objective ranking is really necessary. Also, I want to maintain some sort of credibility in case we want to hold future contests.

I meant this regarding the current situation.

Ah so. That's a very important clarification, thanks!

Quote:

As I said, initially the contest was to have an 'initial screening process' by the judges, and then the best games were to be completely reviewed by all judges. Why was it like this? So even though your game might not have made it to the final round, at least you had a judge look over, play and review your game. I think that counts for something! And as I also said, AT THIS POINT I AM TRYING TO SALVAGE THE SITUATION! That means we will have to cut corners! And that means we may get reviews that are less than adequate.

That's fine, and again, I'm not trying to denigrate specific reviews so much as to say that there's no real basis that I can tell for comparing one reviewer's reviews to anothers. It was my impression from your earlier remarks that this was something you felt was an appropriate ingredient of the original competition (ie, "it's ok if things are subjective, since it's just for fun"), but if you're saying that this posture is more reflective of your current approach rather than your initial one, no objections from me! Sorry to have gotten confused!

Quote:
Please forgive my harsh tone, but I am just as upset about how the competition turned out as anyone.

Well, I'm not upset about it at all! I simply jumped in because I see a fundamental problem, and your remarks led me to believe that this was inherent more to the philosophy of the competition than the judging bailout. At any rate, I now understand more clearly that it's just an artifact of the fact that the original judging approach has been abandoned, but that it wasn't the current system, and that's what I cared most about.

Quote:

But I can't change the past, I can only do my best with what I have now.

Sure, and of course I'm not attacking you personally, just making observations that might be relevant to the next one. I think that we're pretty much on the same page, though, in terms of objectivity of evaluation, and that was what I was most concerned about.

My apologies if my remarks stoked any flames -- I really didn't mean to denigrate you, the judges, the entrants, or anyone else. As I've said, I think it was a cool idea for a contest that has been fun for people to participate in. Though I didn't happen to be one of the people, my thinking was that my lack of personal stake makes me an independent and objective observer of the situation, and that my analysis might have been in some way helpful. But at any rate, your clarification makes me realize that the situation has been well in hand from the inception. Forgive my intrusion!

Syndicate content


forum | by Dr. Radut