Over the past few years I have put a couple of classes in Gamestar Mechanic, a video game about designing video games. It is really quite an amazing tool, if you haven’t set your kid up for a free account just stop reading this blog and go and do it. Anyways, in the game you get to be a designer and create your own sprite based video games.
As with most online games you gain experience by completing tasks. Gamestar has 4 areas that you gain experience in: Designing Games, Playing Other People’s Games, Reviewing Games and Being a Good Citizen. So, basically if you want to increase in level (your title) not only will you build games, but you will also play everyone else’s game. Win, win for everybody.
Except for some reason kids ruin it. There used to be a category on the website for the highest rated games.
I first noticed it with my awesome game: Mt. Sierpinski. Based on a beautiful fractal, it is a pretty fun game! Ok, it is a pretty challenging game, but I was really excited because it was my first game featured on the Gamestar website. This started getting me quite a few good reviews, people commenting that it was great, etc. Then, my game started showing up on the top rated games page, and this is where it got weird. The other people on the top rated page started to give me 1 star reviews, thus dropping my average score so that I would be below their game on the top rated page. Of course, the longer you are on the top of the list, the more plays you have and the more immune you are to this kind of “attack.”
On the flip side, I now log in and play highly rated games that are really, really lousy. How are they getting to be highly rated? They are trading 5 star reviews. You rate my game five stars and I will rate your game five stars. The game can be so simple and have no challenge, take 1 second to win, and they will still be rated 5 stars as long as my also, often equally simple game also gets five stars.
Since the game engine is designed to reward reviews, and reviews are inherently subjective, here is my question: When you put students into Gamestar Mechanic, what do you tell them about the rating process? Do you tell them to do what the Gamestar designers want: honest, critical and helpful reviews? Or do you say nothing, and let them interact with the review ecosystem the way that it has evolved? Which one will lead to a better understanding of the strengths and weaknesses of the user rating system (like Yelp for instance).
My favourite set of reviews, just as an aside, was a student who was mad at me (because of an incident at recess, being caught doing something naughty is never fun). As revenge against the world of authority, this student gave every one of my games the worst possible rating. I had a chuckle over that. Especially since that made some of my other students rate the game so that the effect was undone, because I am a cool teacher. Meaning by rating me poorly the student had actually increased my plays and my rating.
Which makes me wonder . . are any of the ratings valid at all?