conversation timelines - Data x experience lab (ilab)
Non-Intrusive AR for Live AI Conversation Support
hci research, data visualization, augmented reality, prototyping
summer 2024
timeline
1 month for Data x Experience Lab summer research (+ development time)
role
User Flows, UX Research, HCI Research, Prototyping, Visual Design, Web developer
team
Shanna Hollingworth, Dr. Wesley Willett
tools
Figma, XReal Interface, D3.js
conversation timelines - Data x experience lab (ilab)
Non-Intrusive AR for Live AI Conversation Support
hci research, data visualization, augmented reality, prototyping
summer 2024
timeline
1 month for Data x Experience Lab summer research (+ development time)
role
User Flows, UX Research, HCI Research, Prototyping, Visual Design, Web developer
team
Shanna Hollingworth, Dr. Wesley Willett
tools
Figma, XReal Interface, D3.js
conversation timelines - Data x experience lab (ilab)
Non-Intrusive AR for Live AI Conversation Support
hci research, data visualization, augmented reality, prototyping
summer 2024
timeline
1 month for Data x Experience Lab summer research (+ development time)
role
User Flows, UX Research, HCI Research, Prototyping, Visual Design, Web developer
team
Shanna Hollingworth, Dr. Wesley Willett
tools
Figma, XReal Interface, D3.js
overview
Creating tools to study the interaction between humans and AI
overview
Creating tools to study the interaction between humans and AI
overview
Creating tools to study the interaction between humans and AI
I was the sole undergraduate HCI researcher working on a team with a Master's student and professor, and I led the design of an AR x AI conversational support tool. With the goal of evaluating the tool's usefulness in everyday life, I conducted research, prototyped, and co-developed my design to test with users.
IMPACT
We created a novel contribution to the field
Our paper gave novel findings and considerations as a contribution to Human-AI interaction.
We created a novel contribution to the field
Our paper gave novel findings and considerations as a contribution to Human-AI interaction.
We created a novel contribution to the field
Our paper gave novel findings and considerations as a contribution to Human-AI interaction.
We published our research at Graphics Interface 2025!
I co-authored a paper at GI'25, based on my designs and lab studies/user test findings.
We published our research at Graphics Interface 2025!
I co-authored a paper at GI'25, based on my designs and lab studies/user test findings.
We published our research at Graphics Interface 2025!
I co-authored a paper at GI'25, based on my designs and lab studies/user test findings.
And more to come!
I'm still working on this project team, focusing on further paper contributions based on my designs and research to submit to relevant conferences.
And more to come!
I'm still working on this project team, focusing on further paper contributions based on my designs and research to submit to relevant conferences.
And more to come!
I'm still working on this project team, focusing on further paper contributions based on my designs and research to submit to relevant conferences.
background
What contribution can we make to the Human-AI Interaction space?
background
What contribution can we make to the Human-AI Interaction space?
background
What contribution can we make to the Human-AI Interaction space?
To begin our research, we needed an idea of what to do
Our lab is special in that students were able to explore the topics they wished to contribute to. Ultimately, our goal was to conduct research and provide novel insights as a result to submit to a relevant HCI conference.
Our research project and papers were born from simple ideas, driven by advances in AI technology, an interest in conversations, and human-AI interactions.
To begin our research, we needed an idea of what to do
Our lab is special in that students were able to explore the topics they wished to contribute to. Ultimately, our goal was to conduct research and provide novel insights as a result to submit to a relevant HCI conference.
Our research project and papers were born from simple ideas, driven by advances in AI technology, an interest in conversations, and human-AI interactions.
To begin our research, we needed an idea of what to do
Our lab is special in that students were able to explore the topics they wished to contribute to. Ultimately, our goal was to conduct research and provide novel insights as a result to submit to a relevant HCI conference.
Our research project and papers were born from simple ideas, driven by advances in AI technology, an interest in conversations, and human-AI interactions.
problem
How might live data visualization be used to support conversational goals?
problem
How might live data visualization be used to support conversational goals?
problem
How might live data visualization be used to support conversational goals?
After exploring related work by other HCI researchers in this space, our team finally settled on a problem space that we wanted to meaningfully contribute to, which was related to challenges people face daily in conversation.
Have you ever caught yourself asking “what were we talking about” when in a conversation?
Our initial explorations were born from this everyday problem. Coupled with creative technology, we wanted to see what we could create.
Have you ever caught yourself asking “what were we talking about” when in a conversation?
Our initial explorations were born from this everyday problem. Coupled with creative technology, we wanted to see what we could create.
Have you ever caught yourself asking “what were we talking about” when in a conversation?
Our initial explorations were born from this everyday problem. Coupled with creative technology, we wanted to see what we could create.
In a human computer interactions lab, we’re always aiming to push creative and technical boundaries
We wanted to imagine something we thought would be useful in all contexts, as much of the existing work has focused on professional contexts.
In a human computer interactions lab, we’re always aiming to push creative and technical boundaries
We wanted to imagine something we thought would be useful in all contexts, as much of the existing work has focused on professional contexts.
In a human computer interactions lab, we’re always aiming to push creative and technical boundaries
We wanted to imagine something we thought would be useful in all contexts, as much of the existing work has focused on professional contexts.
solution highlights
Our Solution: An augmented reality interface that provides real time information about your conversations
solution highlights
Our Solution: An augmented reality interface that provides real time information about your conversations
solution highlights
Our Solution: An augmented reality interface that provides real time information about your conversations
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Glanceable: Instant recall with representative sentences
How do we quickly jog memory at a glance without being overly distracting?
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Glanceable: Instant recall with representative sentences
How do we quickly jog memory at a glance without being overly distracting?
Navigation: View different points in conversation easily
How can we effectively navigate summaries?
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Navigation: View different points in conversation easily
How can we effectively navigate summaries?
Conversation Partners: Keep your eyes on your partner
How can we minimize gaze change when using this tool in conversation?
Available on demand: summon and dismiss visualization when needed
How do we prevent cognitive overloading, and only provide value when needed?
Conversation Partners: Keep your eyes on your partner
How can we minimize gaze change when using this tool in conversation?
my role
my role
I focused on imagining rapid prototypes designing the text visualization presentation and flows, as well as the front-end development of the tool.
I focused on imagining rapid prototypes designing the text visualization presentation and flows, as well as the front-end development of the tool.
design process
How did I get to the solution?
design process
How did I get to the solution?
design process
How did I get to the solution?
I joined the project after there was an established problem in mind.
Given an established background and problem, I was tasked with imagining a live AR tool that could support conversations on-the-fly, but was given design freedom beyond that.
I worked by first researching related concepts and papers, then creating a breadth of prototypes that I narrowed down to the final prototype.
I joined the project after there was an established problem in mind.
Given an established background and problem, I was tasked with imagining a live AR tool that could support conversations on-the-fly, but was given design freedom beyond that.
I worked by first researching related concepts and papers, then creating a breadth of prototypes that I narrowed down to the final prototype.
I joined the project after there was an established problem in mind.
Given an established background and problem, I was tasked with imagining a live AR tool that could support conversations on-the-fly, but was given design freedom beyond that.
I worked by first researching related concepts and papers, then creating a breadth of prototypes that I narrowed down to the final prototype.
understanding the problem
Navigating a unique space for growth
understanding the problem
Navigating a unique space for growth
understanding the problem
Navigating a unique space for growth
Before beginning anything, I needed to understand the problem space. As much of the previous work in this space focused on professional scenarios, I first aimed to understand and categorize the different scenarios we could use a conversational AI support tool with, so I could design with these contexts in mind.
There are different categories of conversations
Including, but not limited to:
Including, but not limited to:
Conversations with a defined goal
Trip planning with friends and business meetings both aim to accomplish certain tasks.
Conversations with a defined goal
Trip planning with friends and business meetings both aim to accomplish certain tasks.
Conversations with a defined goal
Trip planning with friends and business meetings both aim to accomplish certain tasks.
Casual conversations
Catching up with friends and family usually don’t come with a set of goals, aside from enjoying each other’s company.
Casual conversations
Catching up with friends and family usually don’t come with a set of goals, aside from enjoying each other’s company.
Casual conversations
Catching up with friends and family usually don’t come with a set of goals, aside from enjoying each other’s company.
A combination of both
Conversations often blend into each other as they progress and topics evolve.
A combination of both
Conversations often blend into each other as they progress and topics evolve.
A combination of both
Conversations often blend into each other as they progress and topics evolve.
There are many potential pain points or goals surrounding conversation.
Here are just a few:
Here are just a few:
Re-engagement
For times when you might say “what were we talking about?”
Re-engagement
For times when you might say “what were we talking about?”
Re-engagement
For times when you might say “what were we talking about?”
Recall with a specific goal
In conversations where action items or instructions are mentioned, we may want to revisit these points.
Recall with a specific goal
In conversations where action items or instructions are mentioned, we may want to revisit these points.
Recall with a specific goal
In conversations where action items or instructions are mentioned, we may want to revisit these points.
Staying engaged in conversation
Previous related works found that looking away or using phones during conversation detracted from the interaction greatly.
Staying engaged in conversation
Previous related works found that looking away or using phones during conversation detracted from the interaction greatly.
Staying engaged in conversation
Previous related works found that looking away or using phones during conversation detracted from the interaction greatly.
initial brainstorming & research
Exploring related concepts and work
initial brainstorming & research
Exploring related concepts and work
initial brainstorming & research
Exploring related concepts and work
I took inspiration from as many sources as possible to learn about effective conversation visualizations.
To prevent limiting my imagination, I explored all types of conversation visualization, whether it be textual or visual representations of conversational data.
I created an inspiration playground, where we collected conversation visualization techniques from science fiction, existing work, practical applications, textbooks, papers, and games.
I took inspiration from as many sources as possible to learn about effective conversation visualizations.
To prevent limiting my imagination, I explored all types of conversation visualization, whether it be textual or visual representations of conversational data.
I created an inspiration playground, where we collected conversation visualization techniques from science fiction, existing work, practical applications, textbooks, papers, and games.
I took inspiration from as many sources as possible to learn about effective conversation visualizations.
To prevent limiting my imagination, I explored all types of conversation visualization, whether it be textual or visual representations of conversational data.
I created an inspiration playground, where we collected conversation visualization techniques from science fiction, existing work, practical applications, textbooks, papers, and games.
FIG. 1
Snippets of our inspiration playground
FIG. 1
Snippets of our inspiration playground
FIG. 1
Snippets of our inspiration playground
To ground our work in the HCI space, we also needed to look at related literature.
We focused on relevant papers published at HCI conferences since we wanted to submit our paper to similar venues.
I looked at different conversation and meeting related tools and papers to ensure we could make a useful contribution grounded in previous findings.
To ground our work in the HCI space, we also needed to look at related literature.
We focused on relevant papers published at HCI conferences since we wanted to submit our paper to similar venues.
I looked at different conversation and meeting related tools and papers to ensure we could make a useful contribution grounded in previous findings.
To ground our work in the HCI space, we also needed to look at related literature.
We focused on relevant papers published at HCI conferences since we wanted to submit our paper to similar venues.
I looked at different conversation and meeting related tools and papers to ensure we could make a useful contribution grounded in previous findings.
Key insights from related papers
Key insights from related papers
AR visualization can someetimes prove more distracting than beneficial
Sometimes, less is more when creating AR tools for the user. Conversation partners also didn’t like when users looked away.
AR visualization can someetimes prove more distracting than beneficial
Sometimes, less is more when creating AR tools for the user. Conversation partners also didn’t like when users looked away.
AR visualization can someetimes prove more distracting than beneficial
Sometimes, less is more when creating AR tools for the user. Conversation partners also didn’t like when users looked away.
Many unique tools have been tested and proven useful in professional conversation
Tools that visualized speech lengths and speaker turns as well as meeting summary and captioning tools have proven useful.
Many unique tools have been tested and proven useful in professional conversation
Tools that visualized speech lengths and speaker turns as well as meeting summary and captioning tools have proven useful.
Many unique tools have been tested and proven useful in professional conversation
Tools that visualized speech lengths and speaker turns as well as meeting summary and captioning tools have proven useful.
Privacy is a unique issue
Although consent forms are always used when participating in research, we wanted to imagine an everyday version where privacy of speakers is a priority.
Privacy is a unique issue
Although consent forms are always used when participating in research, we wanted to imagine an everyday version where privacy of speakers is a priority.
Privacy is a unique issue
Although consent forms are always used when participating in research, we wanted to imagine an everyday version where privacy of speakers is a priority.
Thus, we created a set of design goals based on our findings.
After meeting with my team and presenting my research findings, we itemized a set of goals to follow while designing.
Listed below are a few key insights we derived in the process:
Thus, we created a set of design goals based on our findings.
After meeting with my team and presenting my research findings, we itemized a set of goals to follow while designing.
Listed below are a few key insights we derived in the process:
Thus, we created a set of design goals based on our findings.
After meeting with my team and presenting my research findings, we itemized a set of goals to follow while designing.
Listed below are a few key insights we derived in the process:
Key insights
Key insights
A lightweight interface that adapts to live conversation.
We highly prioritized preventing cognitive overload or distracting the user by emphasizing a dynamic and minimalist tool.
A lightweight interface that adapts to live conversation.
We highly prioritized preventing cognitive overload or distracting the user by emphasizing a dynamic and minimalist tool.
A lightweight interface that adapts to live conversation.
We highly prioritized preventing cognitive overload or distracting the user by emphasizing a dynamic and minimalist tool.
Information dense, but minimally distracting
On top of creating a lightweight tool, we wanted to encode as much information as possible into minimal space, but ensure it was useful and non-distracting.
Information dense, but minimally distracting
On top of creating a lightweight tool, we wanted to encode as much information as possible into minimal space, but ensure it was useful and non-distracting.
Information dense, but minimally distracting
On top of creating a lightweight tool, we wanted to encode as much information as possible into minimal space, but ensure it was useful and non-distracting.
Something that preserves as much privacy as possible
While consent to record conversation should be required from all parties, we needed something that provided more privacy.
Something that preserves as much privacy as possible
While consent to record conversation should be required from all parties, we needed something that provided more privacy.
Something that preserves as much privacy as possible
While consent to record conversation should be required from all parties, we needed something that provided more privacy.
iterations and challenges
Creating a research-backed prototype to best suit our needs
iterations and challenges
Creating a research-backed prototype to best suit our needs
iterations and challenges
Creating a research-backed prototype to best suit our needs
I created 25+ rapid low fidelity prototypes to help narrow down our design.
With these design goals in mind, I began rapid prototyping so as to not limit my creativity. From these lo-fi prototypes, I would narrow them down to a final prototype.
I constantly met and presented my designs to the rest of the team and our lab to receive feedback and refine my design.
I created 25+ rapid low fidelity prototypes to help narrow down our design.
With these design goals in mind, I began rapid prototyping so as to not limit my creativity. From these lo-fi prototypes, I would narrow them down to a final prototype.
I constantly met and presented my designs to the rest of the team and our lab to receive feedback and refine my design.
I created 25+ rapid low fidelity prototypes to help narrow down our design.
With these design goals in mind, I began rapid prototyping so as to not limit my creativity. From these lo-fi prototypes, I would narrow them down to a final prototype.
I constantly met and presented my designs to the rest of the team and our lab to receive feedback and refine my design.
FIG. 2
Snapshots of early prototypes
FIG. 2
Snapshots of early prototypes
FIG. 2
Snapshots of early prototypes
How much is too much?
I experimented with large and small amounts of transcription, using skim reading techniques as I designed. Ultimately, we decided that smaller snippets on screen would be more ideal to support a more lightweight and glanceable system that didn't distract the user from conversation.
FIG. 3
Comparing high volumes of text to low volumes
How much is too much?
I experimented with large and small amounts of transcription, using skim reading techniques as I designed. Ultimately, we decided that smaller snippets on screen would be more ideal to support a more lightweight and glanceable system that didn't distract the user from conversation.
FIG. 3
Comparing high volumes of text to low volumes
How much is too much?
I experimented with large and small amounts of transcription, using skim reading techniques as I designed. Ultimately, we decided that smaller snippets on screen would be more ideal to support a more lightweight and glanceable system that didn't distract the user from conversation.
FIG. 3
Comparing high volumes of text to low volumes
Placement matters
Although it initially felt obvious to me to design around the center of vision, I needed to address the issue of maintaining gaze with conversation partners to limit distraction; thus, this unique solution was born, where we intionally choose to lay text over the viewer's center of vision.
FIG. 4
Changing placement of text visualization to the center
Placement matters
Although it initially felt obvious to me to design around the center of vision, I needed to address the issue of maintaining gaze with conversation partners to limit distraction; thus, this unique solution was born, where we intionally choose to lay text over the viewer's center of vision.
FIG. 4
Changing placement of text visualization to the center
Placement matters
Although it initially felt obvious to me to design around the center of vision, I needed to address the issue of maintaining gaze with conversation partners to limit distraction; thus, this unique solution was born, where we intionally choose to lay text over the viewer's center of vision.
FIG. 4
Changing placement of text visualization to the center
How do we encode as much information as possible without being overwhelming?
I took elements from various designs for different reasons; timelines for chronology, 2-3 word AI assisted summaries to represent conversation topics at a glance, bolding for quick parsing, and color to distinguish time granularity.
FIG. 5
Using a collection of techniques to encode more information at once.
How do we encode as much information as possible without being overwhelming?
I took elements from various designs for different reasons; timelines for chronology, 2-3 word AI assisted summaries to represent conversation topics at a glance, bolding for quick parsing, and color to distinguish time granularity.
FIG. 5
Using a collection of techniques to encode more information at once.
How do we encode as much information as possible without being overwhelming?
I took elements from various designs for different reasons; timelines for chronology, 2-3 word AI assisted summaries to represent conversation topics at a glance, bolding for quick parsing, and color to distinguish time granularity.
FIG. 5
Using a collection of techniques to encode more information at once.
The final prototype
Creating a research-backed prototype to best suit our needs
The final prototype
Creating a research-backed prototype to best suit our needs
The final prototype
Creating a research-backed prototype to best suit our needs
After narrowing down the prototypes that best addressed our design and research goals, I iterated on the prototype and created a final version that we moved on to develop and test with users before writing and submitting our paper to GI'25!
reflections & lessons learnt
I gained a wealth of knowledge and experience at my first summer work experience!
reflections & lessons learnt
I gained a wealth of knowledge and experience at my first summer work experience!
reflections & lessons learnt
I gained a wealth of knowledge and experience at my first summer work experience!
FIG. 7
A photo from our group retreat in Canmore!
FIG. 7
A photo from our group retreat in Canmore!
FIG. 7
A photo from our group retreat in Canmore!
Although we had some hiccups along the way, we published at GI’25!
This ended up being my first research paper to be published (coming to the ACM library very soon!)
Although we had some hiccups along the way, we published at GI’25!
This ended up being my first research paper to be published (coming to the ACM library very soon!)
Although we had some hiccups along the way, we published at GI’25!
This ended up being my first research paper to be published (coming to the ACM library very soon!)
This summer, I was constantly growing and learning
Always be open to learning
This was my first experience in academic research and I was constantly learning new things from others and on my own. It was lots of fun!
Always be open to learning
This was my first experience in academic research and I was constantly learning new things from others and on my own. It was lots of fun!
Always be open to learning
This was my first experience in academic research and I was constantly learning new things from others and on my own. It was lots of fun!
Using and testing your designs are important
I’d done UI design for desktop and mobile applications, but AR was completely new to me and threw me off at first.
Using and testing your designs are important
I’d done UI design for desktop and mobile applications, but AR was completely new to me and threw me off at first.
Using and testing your designs are important
I’d done UI design for desktop and mobile applications, but AR was completely new to me and threw me off at first.
Passion is a powerful motivator
Learning about the projects other iLabbers were up to showed me a piece of their values and personality, demonstrating how powerful passion can be, especially in research.
Passion is a powerful motivator
Learning about the projects other iLabbers were up to showed me a piece of their values and personality, demonstrating how powerful passion can be, especially in research.
Passion is a powerful motivator
Learning about the projects other iLabbers were up to showed me a piece of their values and personality, demonstrating how powerful passion can be, especially in research.
Next steps
Conduct studies and user tests
Our team and greater lab community have helped inform our design and test our tool, but there is more to be done.
Conduct studies and user tests
Our team and greater lab community have helped inform our design and test our tool, but there is more to be done.
Conduct studies and user tests
Our team and greater lab community have helped inform our design and test our tool, but there is more to be done.
Iterate! Iterate! Iterate!
Using insights from user tests, we would re-inform our design and create something that best helps users.
Iterate! Iterate! Iterate!
Using insights from user tests, we would re-inform our design and create something that best helps users.
Iterate! Iterate! Iterate!
Using insights from user tests, we would re-inform our design and create something that best helps users.