Skip to Content
 

GDS Scoring System 'flaw'

33 replies [Last post]
Zomulgustar
Offline
Joined: 07/31/2008

Reposted from a PM to Brykovian, per his suggestion:

----
This is my first time as a competing participant in the GDS, so I don't feel too bad over having not noticed this before...as it stands, there is a strong disincentive for entrants to vote. I understand the desire to stop entrants from inflating their own scores at others' expense, but as it stands I feel like I have to choose between feeling selfish or stupid. Would you be willing to consider an alternative?
----

I have a few specific proposals in mind, but it's probably best to open the issue to public discussion, and we can go from there assuming anyone else actually cares. ^_^

Brykovian
Brykovian's picture
Offline
Joined: 07/21/2008
GDS Scoring System 'flaw'

I'm open to (just about) anything, as long as it isn't any more difficult to administer than the current system.

For some prior discussion on this issue, see these threads:

http://www.bgdf.com/modules.php?name=Forums&file=viewtopic&t=2423

http://www.bgdf.com/modules.php?name=Forums&file=viewtopic&t=3214

I'm curious to see what folks think ...

Cheers,
-Bryk

Zomulgustar
Offline
Joined: 07/31/2008
GDS Scoring System 'flaw'

For this round, I'd want to change the system as little as possible, if at all, as people are already in the process of voting. Perhaps something as simple as an automatic 5 pts to your own entry for voting (or equivalently, a 5 point penalty for not doing so?) Since most of the voters are also entrants, this probably won't make that much difference, but I'd feel better. I certainly understand if people are reluctant to 'change the rules in the middle of the game', so to speak.

In the long run, though, I'd also like to resolve some of the concerns expressed by people not wanting to distinguish unfairly between entries they judged of similar quality. Giving each voter as many points as there are games they may vote on, and letting them distribute them to their liking, is much more flexible, but carries an increased risk of AP ^_^, while not really solving the disincentive problem in itself.

Alternatively, each 'voter' could simply rate each entry from 1-5, including their own, but without being allowed to rate yours higher than ALL other games (no net gain for yourself). By adding to their own, they're not intrinsically taking away from anyone else...but that comes with its own set of issues.

As for administration, if anyone wants to use a more complicated formula, I'd be happy to set up an OpenOffice spreadsheet to just plug the votes into, if that would simplify things. (open to inspection, of course...^_^)

Brykovian
Brykovian's picture
Offline
Joined: 07/21/2008
GDS Scoring System 'flaw'

First -- We won't change voting rules until *after* this Challenge is complete. Something about horses and streams. ;-)

Next ... I already use a spreadsheet to do the math -- so that's not the difficulty. I just mean that I don't want to have to do much other than receive PMs from voters and get their votes/ratings/whatevers.

Finally ... Be sure to keep the task of being a voter as simple as possible -- especially for those who *don't* have an entry in the mix (and we do usually have a handful of folks who don't enter but still vote).

Otherwise ... carry on with the discussion. ;-)

-Bryk

jwarrend
Offline
Joined: 08/03/2008
Re: GDS Scoring System 'flaw'

Zomulgustar wrote:

This is my first time as a competing participant in the GDS, so I don't feel too bad over having not noticed this before...as it stands, there is a strong disincentive for entrants to vote.

I think that participation in the GDS presupposes a certain level of maturity on the part of the participants such that actions such as self-voting, refraining from voting so as not to "help the competition", and voting on weaker entries simply to dilute support for "your main competitors", would be recognized to be in poor form and against the spirit of the competition. Anyone who needs to resort to these tactics to win simply for the right to put a GDS win in their mental trophy case needs to get a life, big time.

Having said that, there is one legitimate concern with voting that I've seen expressed recurrently, and it's that people are often torn between voting for what they believe to be the most promising game and the one that conforms the best to the requirements of the competition. It may be helpful to come up with an "official" ruling on which of these represents the "right" way to vote, or else perhaps have two votes, for "best design" and "best solution to the challenge".

If we had to pick one way to vote (best game or best challenge solution), I'd advocate the former, actually. The contest requirements are useful as guideposts, but it's very difficult to actually conform to all of the restrictions in each challenge, and I strongly suspect that any of the entries that see further development will shed in an eyeblink those elements that were tacked on simply to meet the challenge. To my mind, it makes sense to vote for the best game, as long as it doesn't completely flaunt the contest requirements. But I understand the opposite view and think it has merit as well. Perhaps we could take a poll as to a preferred way to vote, and then make this the law of the land. Or just leave it as the free-for-all that it is now.

-J

Hamumu
Offline
Joined: 12/31/1969
GDS Scoring System 'flaw'

I say as I suggested before, quite simply: you can't win (1st 2nd or 3rd) unless you submitted a vote. Pretty simple! Sure there are legitimate times when you can participate in the contest, but can't vote for various reasons... but be honest, do you deserve to win if that's the case? Everybody else put in the effort to critique your entry, and you didn't do the same for them. Sounds fair that they win instead of you.

I can't see any problem with this idea, it's just a tiny spot of work for the Bryk (just checking to be sure he received votes from the people he calculated to be winners).

As for who to vote for, my own system has been to vote for the game I like best as a game, but to disqualify from consideration any game that I feel doesn't meet the requirements to a reasonable extent. I think that works pretty well. Does there really need to be an official voting scheme? I think people will always just vote their own way, by whatever criteria stick most firmly in their craw.

I don't think there should be two votes for sure.

Rick-Holzgrafe
Rick-Holzgrafe's picture
Offline
Joined: 07/22/2008
GDS Scoring System 'flaw'

I don't mind leaving the GDS unchanged. There's nothing important at stake here, and I don't think we need excessive formality in the rules.

But if we decide we do need a change, then I think Mike (Hamumu) makes good sense, in his post just above this one. If you're counting opinions, I agree with Mike.

Epigone
Offline
Joined: 12/31/1969
GDS Scoring System 'flaw'

I like free-for-alls. #J

This is also my first time, and I noticed the same thing. Then I voted just to make sure my subconsious wasn't going to try to cheese the system. In any case, there definitely isn't an alternative that always works (even disregarding the self-vote issue), that's been proven.

The Schulze method is IMO the most prevalent good voting system. From the voters perspective: for as few or as many of the entries as you like, assign them numbers, say, 1-100, with higher better, duplicates allowed. You don't have to rate all entries, but those you don't will be considered strictly worse than any you do. The numbers mean nothing except to tell how you rank the entries. From the admin's perspective, to make it just as easy as it is now you would need to get someone else to write some code to translate, say, an Excel tab-delimited export into a result.

So for four entries A B C and D, for example, a voter could submit:
A 100
B 99
C 99
D 1
to denote that he really liked A and really disliked D. It says nothing about B or C except that he's ambivalent.

I see no reason to disallow voting yourself better than everyone else. It gives your entry a slight boost but doesn't change your input on, say, game X vs. game Y, where your game is neither X nor Y.

phpbbadmin
Offline
Joined: 04/23/2013
GDS Scoring System 'flaw'

Perhaps have two requirements for voting:

A) In order to win you must vote (as already stated).
B) You can't vote for your own submission.

Regarding A, it's not really fair to throw out all the votes for an entry just because the designer didn't vote. In that case, I think it would be best for the moderator to gently prod the entrant to vote.

I know a lot of people, because of their maturity, won't vote for their own entry anyway. However, some people honestly will think their entry is the best or that may just want to 'bump up' their chances of winning, whether subconsciously or on purpose. If it comes down to two close entries, and one person voted for themself and beat the person (or even tied for that matter) who didn't vote for themself, then obviously the system if flawed. I think your design will have to be obviously good if you win and and all the votes you received are from your peers.

Just my dipence,
-Darke

Zomulgustar
Offline
Joined: 07/31/2008
GDS Scoring System 'flaw'

Just to be clear, I would choose to be stupid rather than selfish. It's not that I feel the need to be competitive...far from it. In fact, I'd prefer that the energy that goes into these be redirected towards the GDW or similar strictly-cooperative efforts to help each other build better games. But if we're going to make a game out of it, I at least want it to be one that's well-designed. ^_^

Xaqery
Xaqery's picture
Offline
Joined: 07/26/2008
GDS Scoring System 'flaw'

Zomulgustar wrote:
Just to be clear, I would choose to be stupid rather than selfish. It's not that I feel the need to be competitive...far from it. In fact, I'd prefer that the energy that goes into these be redirected towards the GDW or similar strictly-cooperative efforts to help each other build better games. But if we're going to make a game out of it, I at least want it to be one that's well-designed. ^_^

1 - Zom are you in this months contest?

2 - Its not "stupid"; the voting is not a game its a job. I often fall in to the same trap. I want to make everything a game. but voting is not a game the game ends when you turn your subbmission then you just have the job of voting. As Matt said we are all mature and we know what the spirit of the job is to try and pick the best which leads me to #3

3 - For the first time I disagree with Jeff. If we are picking the best game and not the best that fits the requiments then why do we have the requirements? The "Charm" of the GDS are the requiments. The whole rest of the fourm is dedicated to everyone trying to make the best game they can. What I mean is we are already doing that out there. In my oppinion its FUN to try and come with a game that meets the requirements.

4 - We are all infatuated with rules or we wouldnt be here. Each one of us could come up with ways to vote if we had to. Its what we love to do. THat doesnt mean that what we are doing is flawed.

5 - I feel that if a entrant doesnt vote then we should asume that they had a good reason and move on. I agree with Jeff that if they didnt vote on purpose then they need to get a life but I have a hard time beleiveing that that would ever be the case here.

ok I am done with my rant.

- Dwight
THE CURENT GDS CHAMPION! :P

Zomulgustar
Offline
Joined: 07/31/2008
GDS Scoring System 'flaw'

Quote:
1 - Zom are you in this months contest?

Yup...see first post.

Quote:
2 - Its not "stupid";

Again, mirroring the language of the original post...I didn't mean to imply anyone else was _doing_ something stupid by voting, only that I _felt_ stupid voting. You have my sincere apologies for inadvertently implying otherwise. However...

Quote:
the voting is not a game its a job.

We're agreeing to follow a set of rules, making meaningful decisions, and in the process determining a winner. I'd say we're playing a game... That said, we're also accomplishing a job in the course of playing that game as well. Several, in fact.

Quote:
I often fall in to the same trap. I want to make everything a game. but voting is not a game the game ends when you turn your subbmission then you just have the job of voting. As Matt said we are all mature and we know what the spirit of the job is to try and pick the best which leads me to #3

To me, determining which game is the best is only a side effect of and/or motivator for the real 'jobs'. First, we stretch our own design skills by forcing ourselves to work under time pressure and under constraints we wouldn't impose on ourselves. Secondly, we help each other's design skills through constructive criticism, and in the process learn how others see our own work. A 5, 3, or 1 does very little to help either the voter or the entrant if there's no explanation of why. It's more important (to me) that we help each other make the best games we can than that we figure out which of us does it best. That part is the game (again, as I see it)...and it makes sense that most of us are more motivated in that context...if we didn't enjoy games, none of us would be here discussing this.

Quote:
*snip*
In my oppinion its FUN to try and come with a game that meets the requirements.

It certainly can be, but it can also be frustrating to some of us. Quite a few times over the last few months, a few of the constraints have been enough to trigger a game idea that I'd never have had on my own, then one of the remaining ones makes it impossible to enter. My problem, though. *shrug*

Quote:
4 - We are all infatuated with rules or we wouldnt be here. Each one of us could come up with ways to vote if we had to. Its what we love to do. THat doesnt mean that what we are doing is flawed.

Hence the quotes around the word. The rules have features I personally find very awkward and undesirable, and several other people have pointed out other methods which eliminate those features without introducing anything worse (again in my opinion). Despite Arrow's Theorem, some algorithms are more equal than others. I suppose the question becomes how are we going to vote on how we are going to vote?

Personally, I think it'd be best if there were neither incentive nor disincentive to vote...your vote has zero impact on your game's standing relative to other games, only affecting the relative standing among those others. Little harm comes from the separation of the jobs of producing and critiquing games, and it seems that people are self-motivated to vote, or you wouldn't have so many entrants voting despite the disincentive.

Resorting to 'uncool' tactics doesn't reflect well on a participant to be sure, but I'd think that around here we'd consider an incentive toward that behavior to be a problem with the rules rather than the player.

Xaqery
Xaqery's picture
Offline
Joined: 07/26/2008
GDS Scoring System 'flaw'

Zomulgustar wrote:
Quote:

1 - Zom are you in this months contest?

Yup...see first post.

Sorry. I thought you told me in live chat that you were not going to enter. I misunderstood.

My whole rant was an atempt to argue that The GDS is fine although we have this same disscussion about the GDS each month we find it to be good each time and that maybe we should not take it too seriously. I meant no hard feelings. I beleive you want the best for the comunity.

At first glance, during my first GDS about 4 months ago, I thoguht it was a flaw too. Then after the disscussion that month I realized it was not, that it was just on our honor to vote honestly.

- Dwight

Kreitler
Offline
Joined: 12/31/1969
GDS Scoring System 'flaw'

Zomulgustar wrote:
To me, determining which game is the best is only a side effect of and/or motivator for the real 'jobs'. First, we stretch our own design skills by forcing ourselves to work under time pressure and under constraints we wouldn't impose on ourselves. Secondly, we help each other's design skills through constructive criticism, and in the process learn how others see our own work. A 5, 3, or 1 does very little to help either the voter or the entrant if there's no explanation of why. It's more important (to me) that we help each other make the best games we can than that we figure out which of us does it best. That part is the game (again, as I see it)...and it makes sense that most of us are more motivated in that context...if we didn't enjoy games, none of us would be here discussing this.

I think this is the crux of the matter. I know I enter for two reasons:
1) It's a great discipline to create a constrained design in a short time.
2) It's wonderful to get critique back from other designers.

At first I put a lot of stock in the scores I received, but as contests continue, I find myself much more interested in the few critiques posted after the contest (granted, this could be because my designs usually score poorly. :P ).

Personally, I'd love to trade voting for a brief 1 paragraph critique on each game from each participant. In this way, we'd have a mini-GDW. My gut tells me that this would cost us entrants because not everyone has time for critiques (even brief ones). Besides, its fun to have GDS bragging rights.

One alternative system might involve selecting a single "best" game in three separate categories. For example, we might have "Best design within contest constraints", "Best overall mechanic", and "Most interesting/fun design". All entrants would select 1 game for each category and Bryk would post the results.

The only downside to this is that it's probably harder for Bryk. The upside is that he could stop running "mini-contests" like "best gadget" and "best mini-game" by making those one of the three requirements each month.

This doesn't really address the self-voting problem, but personally I think that's a non-issue because Bryk always posts the full scores for each design. You could always go in after the fact and figure out how you would have scored were you allowed to vote for your own game.

K.

Brykovian
Brykovian's picture
Offline
Joined: 07/21/2008
GDS Scoring System 'flaw'

I'll start with a quick reminder that the voting for this current GDS Challenge (December 2005) will use the current system -- each voter gives me their top 3 games through a PM. And I'd like to encourage everyone to be to please submit a vote for this current Challenge. ;-)

With that said, I've seen a few things in these posts that have inspired the following idea for a new voting system ... It would consist of 2 parts -- a "gameplay rank" and a "challenge score" ...

Gameplay Rank
There would be 3 categories for this:

  • Best of the Best - At least 1 and not more than 2 games that the voter felt were the very best games submitted based purely on the quality of the perceived gameplay.
  • Close Second - From 0 to 2 games that were close to being the best-of-the-best, but just missed the cut. (I think having 0 games in this category would be rare, but might occur if the voter thought there was a huge drop-off after the best 1 or 2.)
  • Worth a Mention - Any number of games that should at least get a mention.
Challenge Score
For the top 2 categories ("Best of the Best" and "Close Second" -- this would range from 1 to 4 games), the voter would give a score as to how well each game met the Challenge criteria using the following scale:
  • 3 - Met all Challenge critieria well
  • 2 - Met most Challenge criteria ... shakey in a few spots
  • 1 - Didn't do so well with the Challenge critieria
Scoring
I would give points to each game as follows:
  • Best of the Best = Challenge Score + 2
  • Close Second = Challenge Score
  • Worth a Mention = 1 point
Best total score would still win.

Self-Voting
Entrants are encouraged to list their game in one of the 3 categories. In whichever category they list their game, they must also list at least 1 other game with the same or better Challenge Score. This means an entrant can give their own game a "Best of the Best" Challenge Score of 3, as long as they give the same to another game too.

Sample Vote Message
This might seem like a lot of extra work for the voters on first glance, but I don't think it is. I could see the vote PM to me looking like this:

Quote:
Best of the Best -- Entry #3 (Challenge Score: 3), Entry #5 (Challenge Score: 2)
Close Second -- Entry #7 (Challenge Score: 3)
Worth a Mention -- Entries #1, 8, 13, and 14

Most people have been submitting comments and rationale for their votes ... sometimes several paragraphs. So this additional scoring granularity shouldn't be too bad of an extra burden.

So ... whadayathink?

-Bryk

[/][/][/]
jwarrend
Offline
Joined: 08/03/2008
GDS Scoring System 'flaw'

Brykovian wrote:

With that said, I've seen a few things in these posts that have inspired the following idea for a new voting system ... It would consist of 2 parts -- a "gameplay rank" and a "challenge score" ...

I think this is a clever hybrid system that addresses nicely the concerns I've seen expressed. Nice job! But...

Quote:

Self-Voting
Entrants are encouraged to list their game in one of the 3 categories.

I can't emphasize enough what a bad idea I think this is. In a sense, the contest is already "unfair" since entrants are also judges, but allowing players to vote for their own games just pushes it right over the edge. People simply cannot evaluate their own work objectively, and the outcome of the contest will be tainted if this is allowed.

It's easy to see this with an example. Let's say that 14 people voted, and factoring in the votes of the other 12 voters, Joe's game earned 22 points and Fred's game earned 19 points. Let's also say that Joe is a rather humble guy, and has too much self-respect to stoop to voting for his own game, and thus, voted for 3 other games. Fred, on the other hand, thought his game was the best, and game himself 5 points, pushing his total to 24 points, and winning the challenge. Does this really seem to be the correct outcome to the contest? To me, it doesn't. Everyone knows that you think your game is the best. Voting for your own game confirms that, but doesn't contribute any useful data to the evaluation process.

The rule must be, vote for 3 games other than your own. I can't think of a single reason why it should be otherwise that couldn't be equally well addressed with some other tweak. (For example, if self-voting is meant to increase voting by participants, simply disallow a player from winning unless he's voted, as Mike suggests).

Otherwise, good ideas!

-Jeff

phpbbadmin
Offline
Joined: 04/23/2013
GDS Scoring System 'flaw'

jwarrend wrote:
Brykovian wrote:

I think this is a clever hybrid system that addresses nicely the concerns I've seen expressed. Nice job! But...

Quote:

Self-Voting
Entrants are encouraged to list their game in one of the 3 categories.

I can't emphasize enough what a bad idea I think this is. In a sense, the contest is already "unfair" since entrants are also judges, but allowing players to vote for their own games just pushes it right over the edge. People simply cannot evaluate their own work objectively, and the outcome of the contest will be tainted if this is allowed.

-Jeff

I'm agreeing with Jeff here. Very clever hybrid system, but allowing people to vote for themselves isn't a good idea. Some people may even feel obligated to vote for themselves. Just don't allow votes for your own entry.

When folks vote, just have them make sure to list their own entry at the top, that way you don't have to cross reference your list of entries.

-Darke

seo
seo's picture
Offline
Joined: 07/21/2008
GDS Scoring System 'flaw'

I'm with Darke and Jeff on the self voting issue. I think the proposed method has merit, as it addresses the always present issue of better game vs. better requirements-matching, but I'm strongly against self voting.

So far I've been using the same criteria as Mike, simply vote for the games that feel better, as long as they meet all the requirements.

Seo

doho123
doho123's picture
Offline
Joined: 07/21/2008
GDS Scoring System 'flaw'

It is impossible to objectively voted for your own game. While every other game has a rules set that you are being introduced to for the first time with an 800-word limit to get the idea across, your game has had effectively, an inifinite amount of "words" (the concept in your head), that has been boiled down to the 800. You know exactly what your game is supposed to be, while the others have a limited description.

I see no reason why the current format needs to change; it's simple, and it's really more of an exercise than a contest anyway.

But if you need to change, I wouldn't mind something like you have 15 points to spread around any which way you want, so if there are three top games you like equally, you could give each 5/5/5, or if you have a single game you really like but are having trouble deciding between two games for a third place vote, then you could vote 7/4/2/2, etc.

Xaqery
Xaqery's picture
Offline
Joined: 07/26/2008
GDS Scoring System 'flaw'

doho123 wrote:
I see no reason why the current format needs to change; it's simple, and it's really more of an exercise than a contest anyway.

I agree.

Zomulgustar
Offline
Joined: 07/31/2008
GDS Scoring System 'flaw'

I'd sooner use something simpler like a preference-ranking method (Schulze is OK, though there are others) than add more complexity without solving the fundamental problems. OK, so I might have an odd idea of 'simple', but at least somone up there agrees. ^_^

Another alternative would be to convert it from a 'Showdown' to a 'Challenge' and get rid of the voting entirely. Just write and critique. I don't understand the "don't change anything because it isn't a serious competition" perspective...if the system isn't worth fixing, it isn't worth having, IMHO.

Edit:
Edit: never mind (hadn't thought of THAT...)

Hamumu
Offline
Joined: 12/31/1969
GDS Scoring System 'flaw'

Yeah, I see no reason to change either, I like it as is (but of any change, I still 'vote for my own"!).

Quote:
(Xaqery)
My whole rant was an atempt to argue that The GDS is fine although we have this same disscussion about the GDS each month we find it to be good each time and that maybe we should not take it too seriously.

Yeah!

And what Jeff said too. No way is voting for your own game a remotely good idea. I'm not qualified to vote on my own, and I never would. Sometimes I think I've hit something brilliant, sometimes I don't, but I do know that I'm too close to it to judge either way. So I'm gonna lose to people who are more into self-promotion than me?

I thought the current rules stated no self-voting anyway... they should!

There's never been a disputed outcome yet, nor a sore loser, so what's the problem?

Edit: Oh, and one other "pile-on-Bryk" comment. I think "Worth A Mention" is a bad idea as well, because it basically amounts to "if I didn't mention your game here, you SUCK!" That's why I like just picking the top bunch and not mentioning the rest. It leaves it wide open as to whether an entry was just not the greatest thing ever, or was garbage. So people don't have to feel insulted by not being picked. I mean, when you say "these are worth a mention", the obvious implication is "those are NOT", which is very harsh.

doho123
doho123's picture
Offline
Joined: 07/21/2008
GDS Scoring System 'flaw'

Quote:
There's never been a disputed outcome yet, nor a sore loser, so what's the problem?

Hey. Speak for yourself! I'm sore every month!!!! (stupid judges!)

Epigone
Offline
Joined: 12/31/1969
GDS Scoring System 'flaw'

One issue with any 'points-based' rating system w.r.t. the self-vote issue is the following dilemma:

you can vote for yourself - an unwarranted vote for yourself a) boosts your own score without reason, and b) lowers the scores of those you did and would have voted for.

you can't vote for yourself - if a vote for yourself WAS warranted you a) lower your own score without reason, and b) boost the scores of those you voted for.

Either of these is made moot by *lots of people voting*. : ) So vote!

The big problem in both of these methods is b). It's the subtle knife that you trip over in the dark, the metaphor you mix with other pitfalls. IMO the *easiest* way to relieve b) is the suggestion of adding a fixed number of points to a submission whose author voted (you can't vote for yourself). The other two ways are a preference based (like Schulze) system or a linear system (after ranking candidates they will get points that can be described as A+Br, r the rank). The second has the problem that every candidate must be voted for, increasing the voter's workload.

Zomulgustar
Offline
Joined: 07/31/2008
GDS Scoring System 'flaw'

I believe you mentioned the Schulze method before...it's been a while since I studied the field, but wasn't there a 'maximum likelihood' method that was fine with leaving out certain entries without automatically sticking them at the top or bottom of the totem pole? Worked on minimizing the number of people who thought the each of the pairwise preferences in the final ordering were wrong, or something like that...sound familiar? Seems like that would be really simple to vote on (just give the ordinal preferences for the games you care to rank, and leave any others out (your own mandatorily)), if potentially trickier for Bryk to administer. Nothing a macro couldn't fix, though...

Hambone
Offline
Joined: 12/31/1969
GDS Scoring System 'flaw'

This month is my first attempt at this challenge. I wanted to add my comments to some of the issues raised in this tread.

Voting for yourself: I struggled with this issue and ultimately decided not to vote for my entry. Probably because I thought others put more time into theirs than mine (good job making it look that way) and I think I did a bad job puting the rules together. Although I had a fun game in my head, I think the ultimate entry was flawed, therefore not deserving a top 3 vote. My self-criticism may be significantly different from others, so I agree with the proposal of not allowing anyone to vote for themselves, or automatically applying points if a contestant votes.

Best game vs. best solution to the challenge: I propose that submissions are filterd out prior to voting if it does not meet the challenge. After they are posted, we vote for the best game. I could not enter my best apple pie into a cake-baking contest and hope to win, even if everyone likes my pie better than the cakes.

Winning or Losing: I am enjoying the process of the Challenge. Creating something based on someone else's criteria in a short period of time, is a classic training meathod for creators/inventors/artists/writers... I am interested in training for the hobby of game design. I will win eventually, when I train enough. I am cometetive and want to win, but I don't want to win with an instant replay reveiw of the last second by the officials. I don't want the score to be close. Keep it fun, and more will participate.

The discussion: I truely enjoy the debate. The best nugget I took away from this month's challenge so far is a quote from Zomulgustar.

Quote:
if the system isn't worth fixing, it isn't worth having, IMHO.

I am going to use that in the future. I'm always trying to improve systems and everyone is annoyed by me. I now have a great comeback.

To everyone that voted for my entry, Santa is bringing a special gift for you to leave under the tree.

Kreitler
Offline
Joined: 12/31/1969
GDS Scoring System 'flaw'

Hamumu wrote:
Edit: Oh, and one other "pile-on-Bryk" comment. I think "Worth A Mention" is a bad idea as well, because it basically amounts to "if I didn't mention your game here, you SUCK!" That's why I like just picking the top bunch and not mentioning the rest. It leaves it wide open as to whether an entry was just not the greatest thing ever, or was garbage. So people don't have to feel insulted by not being picked. I mean, when you say "these are worth a mention", the obvious implication is "those are NOT", which is very harsh.

That's a good point, Mike -- but our current system also has this problem because all scores are posted for all entries. I remember a couple months ago that someone received absolutely no votes at all. Ouch! On the other hand, I like the fact that all scores are posted so I can see where I fell -- even though I'm usually in the bottom 1/3 of the heap.

On a different subject (from a different author), the whole "if it's worth keeping it's worth fixing" sentiment strikes me as a bit off-the-mark, because it implies what we're using is broken. Whether it's broken or not stems, I think, from your expectations as an entrant. As I understand it, the current system forces you to be "stupid" (not vote for yourself, thereby decreasing your chances for winning) or "selfish" (vote for yourself, thereby possibly depriving another game of equal merit from winning). But these alternatives are only relevant if you're playing to win, no?

I think most of us play for the challenge and the possibility of creating designs we wouldn't normally produce. It's also great to get some peer review. If these are the reasons we play, the current system provides a decent "relative rating" scheme -- especially since Bryk posts a summary of each entry's total votes. Anyone can look at his design and say, "Of the 14 other voters, 2 thought mine was 3rd best and 1 thought it was second best. Not bad!" (Unless you're Yogurt, of course, in which case you usually get to say, "Oh, 80% of the people thought my design was the best. Wow, I need to do better next month."). Those results don't change if you vote for yourself because you would automatically disregard your own vote from your tally (i.e., you know how you voted -- only other people's opinions matter).

I think Bryk's hybrid system has some neat features. I especially appreciate being able to judge on two axes (best overall design and best challenge entry). I think it's more complex than we need -- especially if it leads to more work for him.

Finally, I agree with Jeff that self-voting is just a bad idea, period. Even if one could be emotionally objective about one's own entry, the fact remains that we all have much more information about our own entry than the others.

K.

Which brings me back to, "If it ain't broke, don't fix it."

Xaqery
Xaqery's picture
Offline
Joined: 07/26/2008
GDS Scoring System 'flaw'

Kreitler wrote:

That's a good point, Mike -- but our current system also has this problem because all scores are posted for all entries. I remember a couple months ago that someone received absolutely no votes at all. Ouch! On the other hand, I like the fact that all scores are posted so I can see where I fell -- even though I'm usually in the bottom 1/3 of the heap.

Thats not how I see our current system. Our current system says nothing about which games people think are bad. In our current system people pick the games they think should be in the top 3. If you are not in the top 1/3 of the picks it only means people didn’t think you should be in the top 3. You really can not make many more conclusions.

This is a main reason I like the current system. It doesn’t single out the games we don’t like as much as other systems would.

Kreitler wrote:

On a different subject (from a different author), the whole "if it's worth keeping it's worth fixing" sentiment strikes me as a bit off-the-mark, because it implies what we're using is broken. Whether it's broken or not stems, I think, from your expectations as an entrant. As I understand it, the current system forces you to be "stupid" (not vote for yourself, thereby decreasing your chances for winning) or "selfish" (vote for yourself, thereby possibly depriving another game of equal merit from winning). But these alternatives are only relevant if you're playing to win, no?

I agree completely. The crux of this debate is do we want a light exercise each month (I raise my hand) or do we want something different.

I was a bit put off by the comment:

Quote:
if the system isn't worth fixing, it isn't worth having, IMHO.

If I had been Matt I think I would have been offended.

Quote:
Which brings me back to, "If it ain't broke, don't fix it."

I agree.

- Dwight

PS: Ultimatly I will be happy with anything Matt wants to do.

Epigone
Offline
Joined: 12/31/1969
GDS Scoring System 'flaw'

Zomulgustar wrote:
wasn't there a 'maximum likelihood' method that was fine with leaving out certain entries without automatically sticking them at the top or bottom of the totem pole?

Yeah, there are a lot of different ones. The reason I like assigning default comparisons is because then everyone's input carries the same weight, namely nC2 pairwise value judgments, instead of, say, "A>B" vs. "here's a total ordering".

Zomulgustar
Offline
Joined: 07/31/2008
GDS Scoring System 'flaw'

Quote:
I agree completely. The crux of this debate is do we want a light exercise each month (I raise my hand) or do we want something different.

I'm pretty sure noone here has said they don't want the exercise...the point of disagreement is the ruleset by which the exercise should be judged.

Quote:
Quote:
if the system isn't worth fixing, it isn't worth having, IMHO.
If I had been Matt I think I would have been offended.

*sigh*
I was NOT trying to say "this system is so bad that if nobody wants to fix it let's get rid of it since it isn't doing anybody any good."

I was TRYING to say "If we feel this system is performing an important function, it's worth looking into how we can make it work as well as we can. If we feel that it's not important, maybe we should focus on more important things rather than arguing about this every month."

You've pointed out several specific positive aspects this system has over many of the proposed alternatives, and those are valid reasons not to change it, especially if your sentiments are mirrored by most of the participants. "Don't take it so seriously" sounds to me more like an argument to consider the results meaningless rather than to leave the system unchanged.

Perhaps it'll sound better if I reverse the polarity of the neutron flow:
"If it's worth preserving, it's worth optimizing."

Either way, the intent was certainly not to offend. Again. As you said, sharing ideas on how to tweak rules is what we _do_ here... if we collectively gave in when people said 'it's not broke, don't fix it', we'd all still be playing chess. With four players, dice, and a 'queen' that only moves one space diagonally.

(And for those who missed the no-voting-for-yourself rule already in effect, 'selfish' is not voting at all, rather than voting for yourself.)

Kreitler
Offline
Joined: 12/31/1969
GDS Scoring System 'flaw'

Zomulgustar wrote:

*sigh*
I was NOT trying to say "this system is so bad that if nobody wants to fix it let's get rid of it since it isn't doing anybody any good."

Perhaps it'll sound better if I reverse the polarity of the neutron flow:
"If it's worth preserving, it's worth optimizing."

(And for those who missed the no-voting-for-yourself rule already in effect, 'selfish' is not voting at all, rather than voting for yourself.)

Hey Zom,

I don't think you really offended anyone. The phraseology of the original post did come on a bit strong, but as you're re-expressed it, I think your intent is much more clear. Darn text media and its lack of nuanced communication. :)

I just tried to summarize our situation, then realized that it was just a rehash of previous debates. The GDS has been down this road before. That's how we ended up with the system we have. If the main problem is "to vote or not to vote", the community has already spoken: you're on your honor to vote if you enter, and you are not allowed to vote for yourself.

My guess is that you are more concerned with giving entrants the best possible feedback on their designs. This begs the question, "What kind of feedback do people want?" and "how much time are they willing to put into judging via a new system?"

The current system represents a reasonable compromise between feedback received, time spent judging, and Bryk's time spent crunching the numbers.

You made several good points about the current system -- like "if we feel that it's not important, maybe we should focus on more important things," and "'Don't take it so seriously' sounds more like an argument to consider the results meaningless."

Fair enough. So -- outside of the "to vote or not to vote" question, which most people feel is now a non-issue -- where does the current system break down and how could we improve it?

So far, there seem to be two answers to this:
1) Not enough resolution to select between close designs.
2) Lack of ability to judge specific aspects of designs.

Can anyone add any more?

Once we agree on the direction in which to optimize, we can start talking about how to optimize.

K.

Syndicate content


forum | by Dr. Radut