Quantifying Design and UX Iterations

Case Study: Wrestling Game

Conducted a study to benchmark gaming experience and iteratively tracking metrics to gauge the impact of design updates to improve onboarding and early retention.

---

OVERVIEW

Very often teams iterate on the Experience, Design and User Flow multiple times before it is pushed live, and almost all of the times I’ve been asked to test if the design performed better the following times.

RESEARCH OBJECTIVES

• Come up baseline measurement of the game’s current performance.

• To identify areas of the game that needed improvement.

• To compare designs and show how designs have changed overall overtime.

IMPACT

• Helped in prioritization in design iterations and interventions.

• Methodology, framework and study design have been replicated for all games in the organization and have been baked in the agile development roadmaps of all games.

CHALLENGES

The team was looking for feedback on design and experience to fit their agile model of game development and I wanted to convince them that research does not need to be expensive nor does it need to cause significant delays in the product roadmap or deadlines.

For the implementation of the study, I devised a plan that was:

1. Quick - so that the team does not waste efforts early on and the iterative cycle can fit into the product roadmap itself.

2. Replicable - for comparing different iterations and versions of the game.

3. Cost effective - so that this could happen iteratively without incurring huge costs.

APPROACH

The Game Experience Quantitative Framework

I created a survey to assess the gameplay experience against baseline metrics specifically catering to the player experiences in mobile games.

Formulated a mixed-methods approach that incorporated strengths of both qualitative and quantitative methodologies simultaneously for evaluating user experience both formatively and summatively.

METHOD

1. Qualitative Game play-through

Goals:

• Observe players behavior

• Understand player motivations

SAMPLE

• I recruited 30 participants from an external sample provider to match the target audience of the game to allow for statistical analysis (T-test).

• Using this method in the early phases with 30 participants, also made sure that iterative testing was also cost effective.

2. Post - Play Quantitative Survey for Assessing Baseline Metrics

Goals:

• Measure and benchmark the current version

• Compare future iterations to prior versions.

A survey administered to players, after had played through the game to assess the game against baseline player experience metrics.

I automated the calculation of metrics scores and implemented a t-test for comparison in a Jupyter notebook using Python hosted on an internal website to improve efficiency.

Example data to showcase compared to a prior version of the design

FINDINGS

I used the power of both qualitative and quantitive data to identify key pain point points and improvement opportunities.

• There were opportunities to improve the usability and game controls that are easy to manage and that have an appropriate level of sensitivity and responsiveness.

• The core mechanism (the action of play: the activity players do over and over again in the game, like jumping, collecting, flying, or shooting) and game controls took away from the thrill and action of wrestling matches in the game. Improving emotional connection with the game would decrease drop-off.

• Opportunity to further optimize the experience through communication and education and would improve game satisfaction and enjoyment.

"Make it intuitive and user friendly.  Show a Tap or Press here blinking signage.  I never so much as got a tutorial.​.​”

​“I don't like this control method at all. Especially with how the rating system seems to work I hardly had a chance to ever land a perfect or good hit with how much faster the bar moved and how much of a challenge it is to increase my rating at the time of playing.”​

IMPACT

• The consequent improvements were iteratively tested and compared with previous designs.

• This iterative method is now a standard benchmarking and iterative testing process and the method was replicated to assess player experience for other products in the organization.