How might we design a conversational agent to improve the used car buying experience?
Designing a multi-modal conversational agent.
The Scope
CarMax is America’s largest used car retailer and a fortune 500 company. Over 250 million people in the U.S. own a car and CarMax has been a key industry player. Our team worked alongside CarMax’s conversational experience team in order to identify how and where a conversational agent could be embedded into the CarMax process. The project involved rethinking the customer journey and how changing modalities could benefit the user.
The Project (ongoing)
Client: CarMax (MHCI Capstone)
Team Members: Aziz Ghadiali, Will Khulke, Carolyn Zhong, Zack Van Cleef
Role: UX Research Lead
The Solution
“Hey CarMax”: a conversational agent, situated within the test drive, that proactively engages with customers to provide information and answer questions.
Reactive Agent
The “Hey, CarMax” conversational agent is comprised of two main capabilities. The first is its ability to answer questions that users dictate to the agent. During the test drive users can ask the agent to provide them with information about the car they are driving. CarMax’s current policy is to offer solo test drives in light of COVID-19, and an agent ensures that drivers have access to accurate information. However, the agent was purposefully designed to not answer every question. Answers that might require the sales consultants’ expertise or include sensitive information are redirected to the sales consultants. This was done to efficiently utilize the skills that sales consultants have and require a balance between technology and people.
Use the link below to try out a prototype of the reactive agent!
Proactive Agent
The second and perhaps more innovative capability of our design is its ability to be proactive and engage with the user. The conversational agent not only responds to user input, but uses collected data to engage first and provide information about the car that the customer is test driving. The model to the right indicates the potential sources of this data. Designed dialogues included questions like “most customers are interested in the car history, would you like this information?” and statements like “since you’re on the highway, you might like to know that this car has assisted lane departure”.
The above models explains the thinking behind the conversational agents two capabilities; reactivity and proactivity.
Thinking Beyond the Test Drive
In addition to our solution for our immediate opportunity spaces, our team discussed how the same conversational agent could be used at different phases of the used car buying process at CarMax. While we worked deeply within the test drive phase to create a prototype, the agent was always meant to be a part of the end to end journey. Our client, CarMax, envisioned a future where customers could design their car buying experience by relying on both humans and agents. To better understand and visualize what this future might look like, I took part in designing a future customer journey map. I developed how the conversational agent might present itself and what the dialogue might look like at each major phase. The map outlines how and where our conversational agent could be situated during different phases of the journey.
use link to explore the future customer journey map https://bit.ly/futurejourney
The Process
Research, Findings, & Early Insights
Our extensive problem space required several weeks of dedicated research in order to support our design decisions and provide evidence to our clients. We conducted a round of secondary research, a low-risk prototyping session, and several rounds of customer interviews in order to equip ourselves with knowledge as we moved through the project. Each research endeavor resulted in specific findings that we used to inform our next steps. I created several of our testing protocols, both in-person and virtual, to ensure that there was consistency across each testing session.
Early & Low-Risk Prototype
Findings
01. People are open to a conversational agent helping them find and purchase a car
02. The feeling of control is important to participants when learning from a conversational agent
03. The agent’s identity influenced how participants interacted with it
04. Participants will cease interacting with an agent that does not seem “intelligent”
At the very early start of our project, the team design a quick wizard of ox testing set-up to gauge users’ general reactions to conversational agents within the context of a car. The testing involved three parts where participants interacted with a conversational agent during the research phase, test drive phase, and maintenance phase. The voice to text interface was run by members of our team as participants interacted with it to navigate the car buying process.
User Interviews
Semi-Structured Interviews: Conducted interviews with chat team agents & sales consultants to better understand how a conversational agent can bring value to not only the shoppers but to service agents as well
Contextual Inquiry: Virtually observed non-CarMax customers as they researched for cars to identify pain points in the early parts of the car buying process
Directed Storytelling: Conversed with recent CarMax customers in an unstructured environment to discover CarMax specific pain points and gain a better understanding of the overall CarMax process
Early Synthesis
Insights
01. People make a decision on the kind of car that is right for them based on their needs but make a final decision on what to buy based on personal preferences.
02. The hands-on experience validates people’s research and assures them of their decision to purchase a car.
03. When visiting CarMax, people expect to be met where they are in their car buying process.
04. People want the flexibility to choose what role the sales consultant provides in their in-store experience.
05. People want to shop for cars without social pressure but also want information that a sales consultant can provide.
We ran a multiple day synthesis session in order to make sense of all the early research data we had collected. This included data points from our expert interviews, early prototype session, and user interviews. We began by conducting an affinity diagramming session but found that the clusters didn’t point to a clear patter. While the information was valuable, our team wanted to identify more concrete insights. We tried a new method of synthesis that focused on identifying opportunity spaces and the potential they held. We used our findings from the affinity diagramming session to build out several opportunity areas in order to visually see ways in which a conversational agent could be implement. The combination of our affinity diagram and opportunity space exploration helped the team solidify five insights that we would use as we moved forward.
Validating with Storyboard Prototype
We developed general opportunity spaces where a conversational agent could bring the greatest value. But in order to narrow our scope and create more specific areas, we needed to validate our existing assumptions. We used the findings listed above to develop concepts that correlated with our identified general opportunity spaces. I used the developed concepts to create several storyboards and bring to life what the concepts might look like in an actual experience. The purpose of this research was to narrow down on our spaces as we moved into the building and prototyping stage of our project. Each storyboard depicts a specific way and point in time in which a conversational agent could be implemented to enhance the car buying process for shoppers.
Key Findings from Storyboards
We ran a virtual speed dating test with CarMax customers in order to receive their initial reactions to the different ways a conversational agent could interact with them. The facilitation of our storyboard prototype with CarMax customers led to three important findings. These findings helped us re-evaluate our general opportunity spaces in order to determine which specific spaces could actually provide value.
01. People were interested in a conversational agent that could help facilitate their own research by providing objective information.
02. The context of interaction with the conversational agent impacts people's comfortability with sharing their conversational data.
03. People tend to trust data provided by an agent when it was objective or when a sales consultant was present to provide context.
Opportunity Spaces
Selected areas in which to implement a conversational agent derived from research.
Preparing for Testing: Research Tool & WOZ
Our team relied on a text to voice research tool in order to run both our virtual and in-person tests. The research interface allowed us to preload information about the chosen test drive car and responses to potential customer questions.
The image to the right models our wizard of oz setup for our in-person testing run. A similar setup was used for the virtual run but without the need for a video feed or GPS as it was run through Zoom and shared screen. For our in-person test, we installed two phones inside the car that customers chose to test drive. One was to film the driver for safety reasons and the other was a hidden GPS phone to track the location of the drive. The entire interaction, both virtual and in-person, was run by our team. One person controlled the research interface, one researched car specific data to input, and the final person controlled the conversational agent’s visual interface.
Virtual Testing
We used our research findings to develop a protocol that would ask users to virtually test drive a car using our “Hey, CarMax” conversational agent. The testing involved both proactive and reactive interactions with the tester. A program called OBS made it possible for us to overlay our conversational agent prototype on top of a driver point of view video. Throughout the test drive users were given the opportunity to ask the agent questions about the car, as well as receive information. We used the wizard of oz setup and research tool that was described above to run this test.
Our team had the opportunity to travel to Virginia Beach and conduct in-person testing at a CarMax location that is commonly used for user testing. We adapted our virtual protocol but there were certain challenges. For example, we couldn’t control the car that was being driven as we were pulling users as they came to CarMax. As soon as we received the car information we would load the data into our research tool in preparation for the test drive. The video feed and GPS allowed our team to proactively interject with information based on the location of the test driver.
The driver’s sales consultants were also present in the control room with us. Because they are no allowed in the cars with customers due to COVID, sales consultant often miss out on questions that customers might have during the test drive. This experience made it possible for sales consultants to have access to the customers’ needs whilst removing any social pressure for the drivers.
Personas
Our research made it possible for the team to construct two personas that leveraged all the data we had collected and represented our target user group. We found throughout our research that most shoppers come into the on-lot experience with a general sense of their needs and wants. But in order to encompass two sets of shopper experiences, we created Kendra and Mark. Kendra is titled as the validation seeker. She comes into the store knowing what car she wants. For her, the in-store experience is not to research but to validate her assumptions. She needs to test drive and simply see the car in person in order to confirm what she knows. Mark, on the other hand, has a good sense of his needs but, unlike Kendra, he’s unsure what car will fulfill those needs. He is titled as the in-store explored. While he still wants to validate the needs he knows he has, it is not through a specific car. These two personas will help us ground our decisions as we begin to narrow down and prototype specific solutions. CarMax caters to a wide variety of shopper needs but we wanted to constrain our target user group in order to design a solution that could actually bring value.
Reflection
This was an invaluable project where I learned to navigate working closely with a client. It made me think about the implications that our one design might have on the end to end customer journey. Thinking about how users view conversational agents and how their expectations can impact interactions made this design project an exciting challenge. But more than anything, this project helped me see the unending opportunities that exist in the world of UX. Before beginning this project I had not considered how experience design might exist within the used car industry. Working with the conversational experience team and visiting a CarMax location showed me how widely UX can be applied. While the end of this particular project was bittersweet, I was excited to see how other experiences might be designed for users.