conversation timelines - Data x experience lab (ilab)
Imagining AR Interfaces for Live AI Conversational Support
hci research, data visualization, augmented reality, prototyping
summer 2024
conversation timelines - Data x experience lab (ilab)
Imagining AR Interfaces for Live AI Conversational Support
hci research, data visualization, augmented reality, prototyping
summer 2024
conversation timelines - Data x experience lab (ilab)
Imagining AR Interfaces for Live AI Conversational Support
hci research, data visualization, augmented reality, prototyping
summer 2024
overview
Creating tools to study the interaction between humans and AI
overview
Creating tools to study the interaction between humans and AI
overview
Creating tools to study the interaction between humans and AI
Conversation Timelines is a lightweight AR tool that allows users to view live AI-powered summaries about their conversations to support active recall and prevent sidetracking. We published our findings after conducting a user study in Graphics Interface 2025.
As the sole undergraduate HCI researcher, I led the UX/UI design of our initial prototype. With the goal of evaluating the tool's usefulness in everyday life, I conducted research, prototyped, and co-developed my design to test with users.
Timeline
1 month (+more for development)
role
Undergraduate HCI Researcher, Lead UI Designer
team
Shanna Hollingworth, Dr. Wesley Willett
tools
Figma, D3.js, XReal Interface
Skills
User flows, UX Research, HCI Research, Prototyping, Visual Design, Web Development (D3.js, Javascript, HTML, CSS)
problem
How might live data visualization be used to support conversational goals?
problem
How might live data visualization be used to support conversational goals?
problem
How might live data visualization be used to support conversational goals?
Have you ever caught yourself asking “what were we talking about” when in a conversation?
Our initial explorations were born from this everyday problem. Coupled with creative technology, we wanted to see what we could create, especially because related work on this topic revolved solely around professional contexts.
Have you ever caught yourself asking “what were we talking about” when in a conversation?
Our initial explorations were born from this everyday problem. Coupled with creative technology, we wanted to see what we could create, especially because related work on this topic revolved solely around professional contexts.
Have you ever caught yourself asking “what were we talking about” when in a conversation?
Our initial explorations were born from this everyday problem. Coupled with creative technology, we wanted to see what we could create, especially because related work on this topic revolved solely around professional contexts.
Understanding the Problem
Before beginning anything, I needed to understand the problem space. As much of the previous work in this space focused on professional scenarios, I first aimed to understand and categorize the different scenarios we could use a conversational AI support tool with, so I could design with these contexts in mind.
There are different categories of conversations
Including, but not limited to:
Including, but not limited to:
Conversations with a defined goal
Trip planning with friends and business meetings both aim to accomplish certain tasks.
Conversations with a defined goal
Trip planning with friends and business meetings both aim to accomplish certain tasks.
Conversations with a defined goal
Trip planning with friends and business meetings both aim to accomplish certain tasks.
Casual conversations
Catching up with friends and family usually don’t come with a set of goals, aside from enjoying each other’s company.
Casual conversations
Catching up with friends and family usually don’t come with a set of goals, aside from enjoying each other’s company.
Casual conversations
Catching up with friends and family usually don’t come with a set of goals, aside from enjoying each other’s company.
A combination of both
Conversations often blend into each other as they progress and topics evolve.
A combination of both
Conversations often blend into each other as they progress and topics evolve.
A combination of both
Conversations often blend into each other as they progress and topics evolve.
There are many potential pain points or goals surrounding conversation.
Here are just a few:
Here are just a few:
Re-engagement
For times when you might say “what were we talking about?”
Re-engagement
For times when you might say “what were we talking about?”
Re-engagement
For times when you might say “what were we talking about?”
Recall with a specific goal
In conversations where action items or instructions are mentioned, we may want to revisit these points.
Recall with a specific goal
In conversations where action items or instructions are mentioned, we may want to revisit these points.
Recall with a specific goal
In conversations where action items or instructions are mentioned, we may want to revisit these points.
Staying engaged in conversation
Previous related works found that looking away or using phones during conversation detracted from the interaction greatly.
Staying engaged in conversation
Previous related works found that looking away or using phones during conversation detracted from the interaction greatly.
Staying engaged in conversation
Previous related works found that looking away or using phones during conversation detracted from the interaction greatly.
design process
How did I get to the solution?
design process
How did I get to the solution?
design process
How did I get to the solution?
I worked by first researching related concepts and papers, then creating a breadth of prototypes that I narrowed down to the final prototype.
Initial Brainstorming & Research
I took inspiration from as many sources as possible to learn about effective conversation visualizations.
To begin, we took inspiration from as many sources as possible to brainstorm.
To prevent limiting my imagination, I explored all types of conversation visualization, whether it be textual or visual representations of conversational data.
We wanted to know what others did so we could take inspiration and push boundaries as much as possible.
I created an inspiration playground, where we collected conversation visualization techniques from science fiction, existing work, practical applications, textbooks, papers, and games.
To prevent limiting our imaginations, we drew from science fiction, existing work, practical applications, textbooks, papers, and games.
FIG. 1
Snippets of our inspiration playground
FIG. 1
Snippets of our inspiration playground
FIG. 1
Snippets of our inspiration playground
To ground our work in the HCI space, we also needed to look at related literature.
To begin, we took inspiration from as many sources as possible to brainstorm.
We focused on relevant papers published at HCI conferences since we wanted to submit our paper to similar venues.
We wanted to know what others did so we could take inspiration and push boundaries as much as possible.
I looked at different conversation and meeting related tools and papers to ensure we could make a useful contribution grounded in previous findings.
To prevent limiting our imaginations, we drew from science fiction, existing work, practical applications, textbooks, papers, and games.
Key insights from related papers
Key insights from related papers
AR visualization can sometimes prove more distracting than beneficial
Sometimes, less is more when creating AR tools for the user. Conversation partners also didn’t like when users looked away.
AR visualization can sometimes prove more distracting than beneficial
Sometimes, less is more when creating AR tools for the user. Conversation partners also didn’t like when users looked away.
AR visualization can sometimes prove more distracting than beneficial
Sometimes, less is more when creating AR tools for the user. Conversation partners also didn’t like when users looked away.
Many unique tools have been tested and proven useful in professional conversation
Tools that visualized speech lengths and speaker turns as well as meeting summary and captioning tools have proven useful.
Many unique tools have been tested and proven useful in professional conversation
Tools that visualized speech lengths and speaker turns as well as meeting summary and captioning tools have proven useful.
Many unique tools have been tested and proven useful in professional conversation
Tools that visualized speech lengths and speaker turns as well as meeting summary and captioning tools have proven useful.
Privacy is a unique issue
Although consent forms are always used when participating in research, we wanted to imagine an everyday version where privacy of speakers is a priority.
Privacy is a unique issue
Although consent forms are always used when participating in research, we wanted to imagine an everyday version where privacy of speakers is a priority.
Privacy is a unique issue
Although consent forms are always used when participating in research, we wanted to imagine an everyday version where privacy of speakers is a priority.
Key insights
Key insights
After meeting with my team and presenting my research findings, we itemized a set of goals to follow while designing, which are listed below:
We wanted to know what others did so we could take inspiration and push boundaries as much as possible.
A lightweight interface that adapts to live conversation.
We highly prioritized preventing cognitive overload or distracting the user by emphasizing a dynamic and minimalist tool.
A lightweight interface that adapts to live conversation.
We highly prioritized preventing cognitive overload or distracting the user by emphasizing a dynamic and minimalist tool.
A lightweight interface that adapts to live conversation.
We highly prioritized preventing cognitive overload or distracting the user by emphasizing a dynamic and minimalist tool.
Information dense, but minimally distracting
On top of creating a lightweight tool, we wanted to encode as much information as possible into minimal space, but ensure it was useful and non-distracting.
Information dense, but minimally distracting
On top of creating a lightweight tool, we wanted to encode as much information as possible into minimal space, but ensure it was useful and non-distracting.
Information dense, but minimally distracting
On top of creating a lightweight tool, we wanted to encode as much information as possible into minimal space, but ensure it was useful and non-distracting.
Something that preserves as much privacy as possible
While consent to record conversation should be required from all parties, we needed something that provided more privacy.
Something that preserves as much privacy as possible
While consent to record conversation should be required from all parties, we needed something that provided more privacy.
Something that preserves as much privacy as possible
While consent to record conversation should be required from all parties, we needed something that provided more privacy.
Iterations & Challenges
I created 25+ rapid low fidelity prototypes to help narrow down our design.
To begin, we took inspiration from as many sources as possible to brainstorm.
With these design goals in mind, I began rapid prototyping so as to not limit my creativity. From these lo-fi prototypes, I would narrow them down to a final prototype.
We wanted to know what others did so we could take inspiration and push boundaries as much as possible.
I constantly met and presented my designs to the rest of the team and our lab to receive feedback and refine my design.
To prevent limiting our imaginations, we drew from science fiction, existing work, practical applications, textbooks, papers, and games.
FIG. 2
Snapshots of early prototypes
FIG. 2
Snapshots of early prototypes
FIG. 2
Snapshots of early prototypes
How much is too much?
I experimented with large and small amounts of transcription, using skim reading techniques as I designed. Ultimately, we decided that smaller snippets on screen would be more ideal to support a more lightweight and glanceable system that didn't distract the user from conversation.
FIG. 3
Comparing high volumes of text to low volumes
How much is too much?
I experimented with large and small amounts of transcription, using skim reading techniques as I designed. Ultimately, we decided that smaller snippets on screen would be more ideal to support a more lightweight and glanceable system that didn't distract the user from conversation.
FIG. 3
Comparing high volumes of text to low volumes
How much is too much?
I experimented with large and small amounts of transcription, using skim reading techniques as I designed. Ultimately, we decided that smaller snippets on screen would be more ideal to support a more lightweight and glanceable system that didn't distract the user from conversation.
FIG. 3
Comparing high volumes of text to low volumes
Placement matters
Although it initially felt obvious to me to design around the center of vision, I needed to address the issue of maintaining gaze with conversation partners to limit distraction; thus, this unique solution was born, where we intionally choose to lay text over the viewer's center of vision.
FIG. 4
Changing placement of text visualization to the center
Placement matters
Although it initially felt obvious to me to design around the center of vision, I needed to address the issue of maintaining gaze with conversation partners to limit distraction; thus, this unique solution was born, where we intionally choose to lay text over the viewer's center of vision.
FIG. 4
Changing placement of text visualization to the center
Placement matters
Although it initially felt obvious to me to design around the center of vision, I needed to address the issue of maintaining gaze with conversation partners to limit distraction; thus, this unique solution was born, where we intionally choose to lay text over the viewer's center of vision.
FIG. 4
Changing placement of text visualization to the center
How do we encode as much information as possible without being overwhelming?
I took elements from various designs for different reasons; timelines for chronology, 2-3 word AI assisted summaries to represent conversation topics at a glance, bolding for quick parsing, and color to distinguish time granularity.
FIG. 5
Using a collection of techniques to encode more information at once.
How do we encode as much information as possible without being overwhelming?
I took elements from various designs for different reasons; timelines for chronology, 2-3 word AI assisted summaries to represent conversation topics at a glance, bolding for quick parsing, and color to distinguish time granularity.
FIG. 5
Using a collection of techniques to encode more information at once.
How do we encode as much information as possible without being overwhelming?
I took elements from various designs for different reasons; timelines for chronology, 2-3 word AI assisted summaries to represent conversation topics at a glance, bolding for quick parsing, and color to distinguish time granularity.
FIG. 5
Using a collection of techniques to encode more information at once.
solution highlights
Our Solution: An augmented reality interface that provides real time information about your conversations
solution highlights
Our Solution: An augmented reality interface that provides real time information about your conversations
solution highlights
Our Solution: An augmented reality interface that provides real time information about your conversations
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Glanceable: Instant recall with representative sentences
How do we quickly jog memory at a glance without being overly distracting?
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Glanceable: Instant recall with representative sentences
How do we quickly jog memory at a glance without being overly distracting?
Navigation: View different points in conversation easily
How can we effectively navigate summaries?
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Navigation: View different points in conversation easily
How can we effectively navigate summaries?
Conversation Partners: Keep your eyes on your partner
How can we minimize gaze change when using this tool in conversation?
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Conversation Partners: Keep your eyes on your partner
How can we minimize gaze change when using this tool in conversation?
reflections & lessons learnt
I gained a wealth of knowledge and experience at my first summer work experience!
reflections & lessons learnt
I gained a wealth of knowledge and experience at my first summer work experience!
reflections & lessons learnt
I gained a wealth of knowledge and experience at my first summer work experience!
FIG. 7
A photo from our group retreat in Canmore!
FIG. 7
A photo from our group retreat in Canmore!
FIG. 7
A photo from our group retreat in Canmore!
This summer, I was constantly growing and learning
Always be open to learning
This was my first experience in academic research and I was constantly learning new things from others and on my own. It was lots of fun!
Always be open to learning
This was my first experience in academic research and I was constantly learning new things from others and on my own. It was lots of fun!
Always be open to learning
This was my first experience in academic research and I was constantly learning new things from others and on my own. It was lots of fun!
Using and testing your designs are important
I’d done UI design for desktop and mobile applications, but AR was completely new to me and threw me off at first.
Using and testing your designs are important
I’d done UI design for desktop and mobile applications, but AR was completely new to me and threw me off at first.
Using and testing your designs are important
I’d done UI design for desktop and mobile applications, but AR was completely new to me and threw me off at first.
Passion is a powerful motivator
Learning about the projects other iLabbers were up to showed me a piece of their values and personality, demonstrating how powerful passion can be, especially in research.
Passion is a powerful motivator
Learning about the projects other iLabbers were up to showed me a piece of their values and personality, demonstrating how powerful passion can be, especially in research.
Passion is a powerful motivator
Learning about the projects other iLabbers were up to showed me a piece of their values and personality, demonstrating how powerful passion can be, especially in research.
Next steps
Conduct studies and user tests
Our team and greater lab community have helped inform our design and test our tool, but there is more to be done.
Conduct studies and user tests
Our team and greater lab community have helped inform our design and test our tool, but there is more to be done.
Conduct studies and user tests
Our team and greater lab community have helped inform our design and test our tool, but there is more to be done.
Iterate! Iterate! Iterate!
Using insights from user tests, we would re-inform our design and create something that best helps users.
Iterate! Iterate! Iterate!
Using insights from user tests, we would re-inform our design and create something that best helps users.
Iterate! Iterate! Iterate!
Using insights from user tests, we would re-inform our design and create something that best helps users.