Project Background

Artie's Adventures is a series of episodic chatbots on Kik that follows the titular character on his wacky adventures. Episodes can be about anything from branching narratives to mini games. The company I work for uses Artie in order to test the viability of new features that take advantage of NLP (natural language processing) and to gather data on user behavior. For this particular case study, I will focus on the Halloween episode: Ghoul Next Door.

My Role

I have been the lead designer on many of the Artie episodes, focusing on making the episodes both fun and informative for the company. I wrote the primary linear script, designed the puzzles, created paper sketches, created all media resources (icons, puzzles, etc), led usability testing, and created intents to improve the NLP of the system.

Business Goals

Discover if word puzzles are a good experience on messaging platforms
Collect data to improve NLP for puzzle games
Create a fast, low budget chatbot

User Goals

Have fun
Not notice any Business Goals

Research

We discovered three things about Kik users during our preliminary research that I kept in mind during design. Users are:

Millennials (13-25)
Really like narratives
Distracted (15 minutes is too long)

Prototyping and Design

First, I designed the narrative structure, creating a very high level outline along with a linear script. I believe it's important that design comments (to change content, tone, or otherwise) are made early before any code is inputted. More eyes on a project are always good.

To keep the experience short and sweet, I opted to give users two random puzzles (out of 6) during each play through and make the experience repeatable. This way, users with shorter attention spans can get in and out, and those who are more engaged can solve all the puzzles.

Next, I selected the puzzles from an online list that could be solved in one step (e.g. riddles, anagrams, etc) and made simple sketches that were tested for difficulty. Because the purpose of these puzzles were just to test their viability, I tended to err on the easier side.

After role playing the experience (one person pretends to be a bot), I created fast higher fidelity images in Photoshop to get the experience ready for usability testing.

Usability Testing

We ran the chatbot with a few hundred users online. One of the issues we encountered was that repeat users would sometimes receive the same line twice. I typically like to have at least three variations of every line in an experience as short as this one, but I decided to increase the element of randomization by varying his shorter phrases. Responses like “Yeah man,” “Sure thing,” or “For sure” are used by Artie in similarly ways. By randomly selecting one from a list of phrases with the same meaning, it created greater variations in his dialogue.

More significantly, I added a hint system for each of the puzzles to give struggling users a greater chance of solving it. Keeping in mind the original goal of the project, I wanted to focus on seeing how users treat puzzles in a messaging experience. It wasn’t important to see if the user could solve the puzzle. The hint system reflected that, getting progressively easier until it almost gave the user the answer.

Results

The experience was a success. It reached the top of the Kik Bot Store during its run, giving us data on over 50,000 users.

Most importantly, I learned which puzzles work best on a chatbot platform after looking through the sentiment analysis of the user’s conversation. This project was key in incorporating new game mechanics which were later used in projects with Amazon, Disney, Lionsgate, and Paramount.