Between January 2020 and July 2022, I led UX research projects at Tangible Play (Osmo) for 18+ products targeting families that have 3-14 year old children. I conducted extensive qualitative and quantative research for one of the product lines, the Math Wizards (MW) product line. The MW line provides a series of phygital math games designed for 6 to 8-year-old children, featuring physical manipulatives that children use to interact with on-screen games.

Below is a case study for the Amazing Airships in this line:

In 2020, Osmo planned to develop a new addition to the Math Wizard product line.

The game is played using a playmat and airship pieces (boxes, woods, and balloons). Players design and build airships to deliver packages to customers/characters within the Math Wizard world.

I led a Beta research project for this new math game, with two goals in mind:

1. Understand the market fit and age fit of this product. Is this product appealing to our target audience?
2. Validate the software features and hardware components. Deliver findings to the product and design team for quick iterations.
3. Evaluate how well the product fits the Math Wizards line regarding its theme, educational content and gameplay. Compare key metrics (NPS, Satisfaction, etc) with other products in the Math Wizard line.

Disclaimer: data and findings presented in the case studies are for demo purpose only; they do not reflect any actual trends or findings.

Pre-development

Market & Concept Evaluation

During the Covid-19 pandemic, it became crucial to keep track of parents’ interest in educational products as their needs for home learning resources were rising. I worked with the product team to gather research questions the team needs answer for, then collaborated with other researchers to identify key metrics (e.g., general interest in product category) that need year-over-year comparisons.

I designed a market research survey to evaluate parents’ interest in the general product category and learning subject along with product-specific questions, such as parents’ purchase interest based on a detailed product description. Then I programmed and launched the survey on SurveyMonkey with purchased target audience panel (e.g., parents who have 6-8 year old children in the United States).

Key insights:

1. Math is the learning subject that parents of 6-8 year old are most interested in investing in.
2. Potential gap in the market: Interest in purchasing an interactive math game increase significantly since March 2022, but parents who reported owning this type of educational products remained comparable.
3. Parents are interested in the product concept described in the survey. The price estimates for the concept is lower than expected, indicating the need for better design and marketing strategies to convey value.

Alpha Evaluation

Concept and Gameplay Validation

At the stage of Alpha, we need to validate the product concept with families who meet the target audience criteria for this product. Our goal is to set the baseline metrics (e.g., Net Promoter Score, Difficulty of Gameplay) for this prototype while inviting testers to share their inputs in open-ended responses.

When the product team prepared to launch the prototype on TestFlight, I started recruiting and screening 60~80 testers from our internal pool of testing families. I designed a short survey to confirm testers’ eligibility and availability, and coordinated with our scheduler to process testers’ NDA and consent forms.

To prioritize the questions in the Alpha survey, I met the cross-functional stakeholders to discuss and track changes maded to the survey draft. My goal is to ensure the key questions from product managers, designers and engineers are translated into a suitable format. When I completed the survey draft, I incorporated feedback from stakeholders and other researchers to revise and rebalance the draft.

Some of the key insights from the Alpha survey synthesis: (see the sample slide below)

1. The educational content was challenging to the 6 year olds who struggled with mental math.
2. Children had a hard time distinguishing the physical manipulatives.
3. Some pieces (e.g., balloon of “1”) were too small for children’s fingers.
4. The core gameloop of building an airship to deliver packages felt repetitive as children had limited control with the airship and constrained interactions with the game characters.

At the end of the sysnthesis, I presented the findings and recommendations to corresponding stakeholders. Based on our discussion, we decided to move forward to the Beta Evaluation in a few months to compare the results between Alpha and Beta.