iCombat

I decided this weekend that I would get one last round of user feedback before we begin our final phase of development and debugging in the coming weeks. This was the first close to fully baked test I had done and I even prepared a list of survey questions for everyone that tested the game. In it I had questions regarding the user’s usage patterns (average number of applications downloaded a month, average amount of time playing games a week, how much spent in the last month on apps, etc) but I focused primarily on which aspects of the game they liked and disliked (best aspect, worst aspect, what would you add/ remove, any confusion, etc). Here I found the range of user opinion to be huge, so much so that I wonder how useful the entire exercise was. Here are some of the general observations I noticed in my small game test experiment:

1. With people you know, you will never get the brutally honest answers you need so try to go outside of your immediate circle.
2. Getting a thoughtful, helpful review takes more time and thought than most people you demo it to are willing to give. it is not as easy as it looks to try something for 15 minutes and then offer intelligent suggestions or feedback so usually people will just shout out their first thoughts to give you something.
3. Sometimes you don’t actually want to hear what testers have to say – by the time you are testing your game on them the idea is pretty near finished so adding or shifting anything major (which people always suggest) is basically out of the question.
4. For many applications, it is impossible to know your end market – you may think it is high school kids, but in the end it might be picked up by an entirely different demographic. The only way to know is to put it on the market.
5. Relating to number 4, if you don’t have a clear vision of where you game should go from the beginning don’t expect your testers to answer that for you.  Odds are your sample set will not be big enough to lead you to one conclusion over another.
6. Unless you are careless and sloppy, testing it yourself will spot the glitches / bugs in the game better than your testers – you shouldn’t rely on someone else to do this who probably has a more forgiving eye and definitely less of an interest in seeing your game work flawlessly.  The flip side of this one though is that you may sweat unnecessary details more than you need to.
7. It is easier just to go to the local mall and find people to test your product on the spot than do Ad Hoc testing (in this case Starbucks gift certificates work great)

I guess the issue I have is that it is hard to separate the review from the reviewer.  And as soon as you start to look at the different demographics in your test group you find yourself needing to decide what group you want to cater to.  Herein lies my app game design dilemma – do you take the feedback of one of your groups over the other or try to cater to all of them?  For example I had my 11 year old cousin try my game and he loved it, but then several older people played and said it was just too difficult.

Well I know that the younger demographic is more active with sharing and consuming these games but it is the older people with the credit cards on file – so what do you do?  The iPhone with its diverse user base runs the full range of casual to experienced gamer so you have to decide early who you are going after.  Ultimately you just have to stick to your original idea and wait until the users in the App store validate your product one way or another.

But the question is a good one, how do you let user feedback – before or after launch for that matter – affect the vision you have behind your application?  While listening to users is incredibly important, in the era of rapid updates and constant new releases, where you draw the line is getting challenged more and more.  I wonder how long it will be before someone crowd sources the design of an iPhone game from beginning to end?

Related articles by Zemanta

• How to make an iPhone game (tuaw.com)
• School that uses game-based learning opens in NYC — Boing Boing Offworld (boingboing.net)
• Second Life web resources for February 23rd 2009 (vintfalken.com)