AI-Assisted Handwriting-to-Digital Notes

Evaluating User Perceptions and Workflow Preferences in AI-Assisted Handwriting-to-Digital Note Transitions

Presenters

Alicia Morgan, Joe (Beiqiao) Hu

Course

EMIT [HCIN5300/ITEC5204] Emerging Interaction Techniques

Date

April 2, 2025

Background & Introduction

The Modern Note-Taking Dilemma

We use both handwriting and digital tools—handwriting aids thinking, while digital tools enhance efficiency. The main challenge is smoothly converting handwritten notes into usable digital formats without losing context.

AI as a Potential Bridge

AI can bridge analog and digital by interpreting and integrating handwritten content. This project studies user needs to guide the design of intuitive, AI-assisted note tools.

Key Concepts

1

Handwritten Notes

  • Traditional, flexible, enhances memory and creativity
  • Often non-linear / free-form
2

Digital Note-Taking

  • Efficient, searchable, organized (e.g., OneNote, Notion)
  • Typically linear
3

Hybrid Workflows

  • Combining handwritten and digital methods
  • A need for smoother transitions
4

AI Assistance

  • Task support for transcription (OCR), structuring, tagging, summarization, and linking
5

Context-Aware System

  • AI designed to understand the circumstances surrounding note creation (e.g., source material like a video, time, location, related notes)
  • Provides richer, more relevant digital outputs

Methodology

Study Approach: Qualitative, mixed-methods approach.

Study Phases

1

Recruitment & Screening

  • Posted call for participants on social networks, recruited 13 participants (1 pilot, 12 main study)
  • Focused on a mix of handwriting and digital note users
2

Consent & Pre-Study Questionnaire

  • Informed consent
  • Collected baseline data on demographics, habits, tool usage
3

Semi-Structured Interview

  • 1-hour, in-person sessions
  • Focused on note-taking habits, hybrid note-taking challenges, AI expectations
4

Prototype Interaction

  • Task: Watch a 6-minute video ("Dynamic Land Intro") and take handwritten notes
  • Implemented participant notes to prototype (Wizard-of-Oz method)
  • Participants think aloud (reactions, impressions, usability)

(Prototype process)

Prototype process diagram
5

Data Analysis

Topics Explored

  • Process and tool preferences
  • Hybrid workflow transition challenges
  • AI tool experience and desired features
  • Concerns around privacy, accuracy, and learning impact

Prototype Generation & Methodology Evolution

Initially (Participants #0-3), used a pre-made prototype example that depicted their notes "AI enhanced".

This sometimes felt inauthentic to participants ("these are not my notes").

Iterated, and developed a rapid (~5 min) technique using current AI models (like Claude 3.7 and Gemini 2.5 Pro) to generate prototypes directly from participants' (#4 onwards) notes.

This significantly improved feedback authenticity and elicited positive reactions ("like magic").

Comparison of pre-made and instantly generated prototype

(Comparison of pre-made and instantly generated prototype)

Transforming Handwritten Notes to Digital

1. Photo

Simple image scan of handwritten notes

2. OCR Text

Optical Character Recognition (OCR) – linear text, often losing structure

3. AI-Enhanced / Context-Aware

Digital notes with enhanced OCR (structure recognition), metadata (tags, date, location), contextual links (source video, web resources), summarization, and connections to related notes

Prototype based on participants' handwritten notes

(Prototype 1 - Based on participants' handwritten to generate digitalizations)

Data Analysis

Thematic analysis of interview transcripts to identify key themes, patterns, and insights.

Survey Results & Analysis

Based on survey responses from 18 participants, we analyzed demographics and note-taking habits to better understand current practices and challenges.

Participant Demographics

Age Distribution

18-24: 33%
25-34: 33%
35-44: 33%

Gender Distribution

Women: 60%
Men: 40%

Education Level

Graduate: 60%
Bachelor: 40%

Note-Taking Habits

Note-Taking Methods Comparison

METHOD
TYPES
Digital (39%)
Handwriting (28%)
Mixed (22%)
Context-based (11%)

Most users employ multiple methods based on context

Method Strengths Comparison
Creativity & Flexibility
Handwriting
Digital
Organization & Search
Handwriting
Digital
Memory Retention
Handwriting
Digital

Transition Pain Points:

Time-consuming (93%), format loss (67%), accuracy issues (67%)

Digital Tools Used

Google Docs
7
Microsoft Word
6
OneNote
5
Apple Notes
5
Notion
2
Others
3

Handwriting Frequency

Daily: 40%
Weekly: 20%
Occasionally: 40%

Transition Methods

How participants convert handwritten notes to digital

6

Manual
Retyping

33%

5

Photos/
Scanning

28%

3

No
Transition

17%

1

Other

6%

93% find the transition process time-consuming

Transition Challenges

Time Consuming
14 (93%)
Formatting Issues
10 (67%)
Recognition Errors
10 (67%)
Organizing/Filing
6 (40%)
Collaboration
5 (33%)

Note: Percentages represent how many participants (out of 18) reported each challenge.

Note-Taking Satisfaction

Satisfaction with Current Process

Neither satisfied nor dissatisfied
7 (39%)
Somewhat dissatisfied
5 (28%)
Somewhat satisfied
4 (22%)
Extremely satisfied
2 (11%)

Key Survey Insights

Survey Overview

Based on responses from 18 participants, we analyzed note-taking preferences, satisfaction levels, and challenges.

Time Constraint

93% of participants (14 of 18) reported that the time-consuming nature of transitioning notes is a significant challenge.

Format & Recognition

67% face formatting issues and recognition errors when digitalizing handwritten notes (10 of 18 participants).

Satisfaction Levels

Only 33% of participants (6 of 18) were satisfied with their current note-taking process; the majority (67%) were neutral or dissatisfied.

Findings So Far

Key Findings

AI-Assisted Note Transition

Bridging analog and digital workflows

Handwriting Value

  • Enhanced creativity & memory
  • Flexible layouts & diagrams

Digital Efficiency

  • Easy search & organization
  • Cross-device accessibility

Transition Challenges

  • Time-consuming process (93%)
  • Structure & format loss (67%)

AI Expectations

  • Structure preservation
  • Integration with existing tools

Challenges & Limitations

  • Methodology Changes: Partway through the study, we started generating prototypes directly from user notes. Early feedback may not reflect later experiences with more personalized outputs.
  • Prototype Fidelity: The tool used was a simulated prototype, not a fully developed product. Real-world performance, especially in terms of speed and handling different types of handwriting, still needs to be tested.
  • Variation in Input Notes: The quality of AI-generated output depended heavily on how detailed the original notes were. Sparse or vague notes led to weaker results, which makes it harder to ensure a consistent experience across users.
  • Challenges with Layout Generation: Accurately recreating non-linear, handwritten layouts in a clean, editable digital format (quickly and affordably) is still a technical challenge, though the technology is improving fast.
  • Sample Demographics: The participant pool (12 people) was mainly made up of Carleton University students. While this offered rich qualitative insights, the findings aren't easily generalizable to broader populations.
  • Changing Tech Landscape: The cost and accessibility of AI tools shifted during the study (e.g., Gemini 2.5 Pro becoming free), showing how fast the underlying technology—and its barriers—can change.

Discussion

Why Is This Important?

Handwriting's Relevance

It remains vital for specific cognitive tasks and situations; ignoring it leads to lost insights and friction.

Bridging the Analog-Digital Divide

Current inefficiencies create information silos; AI can unify workflows.

Enhanced Knowledge Management

AI can transform static handwritten notes into dynamic, searchable, interconnected digital assets.

Supporting Diverse Thinking Styles

AI respecting non-linear, visual handwritten notes can better support various cognitive approaches.

Future Work/Learning

Seamless integration of analog and digital methods is crucial for increasingly hybrid environments.

Finding Solutions (Design Implications)

Prioritize Accuracy & Structure

Focus development on robust recognition of text, layout, hierarchy, diagrams, and symbols.

Integrate, Don't Isolate

Build AI features as plug-ins or native functions within popular existing note-taking platforms.

Empower the User

Provide clear editing options, visually differentiate AI content, allow manual process control, and offer varying levels of AI enhancement.

Build Trust

Ensure transparency in data usage/privacy policies; offer local processing or clear data deletion options.

Focus on Augmentation

Design AI to support and highlight the user's key insights, not replace them with generic summaries. Preserve the user's "voice."

Context Needs Input

Acknowledge that true context often requires user input (meeting purpose, project tags); explore low-friction methods to capture this.

Future Directions

  • Improve recognition of complex elements (diagrams, math)
  • Test specific valuable features (nuanced summaries, advanced linking)
  • Conduct longitudinal studies on user adaptation and value perception
  • Test with diverse user groups and professions
  • Continue research on privacy-preserving AI and cognitive impacts

Discussion Questions

With the rapid advancement of large language models (LLMs) and their growing capabilities...

1

Meeting User Needs

How can researchers and designers ensure that these technologies effectively meet evolving user needs while aligning with application-specific requirements?

User
Needs

AI
Capabilities

Specific
Requirements

2

Bridging AI & Design

What strategies or methodologies can be employed to bridge the gap between cutting-edge AI development and practical, user-centered design?

AI Development

  • Technical focus
  • ML models
  • Performance

BRIDGE

User-Centered Design

  • Human focus
  • Usability
  • Experience

Thank You & Acknowledgements

The content was generated by Gemini 2.5 Pro based on our draft and then overhauling revised and simplified.

This website was generated by File Visualizer.

Questions?