ClarifAI | Mitchell Lab

The Problem

Vague feedback tells designers nothing.

Asynchronous Online Focus Groups (AOFGs) are a widely used tool to collect feedback at scale, while addressing key geographic barriers and participant time constraints; however, the feedback collected via AOFGs is often too vague or ambiguous to act on. "Q: Was this helpful? A: Kind of" gives a designer nothing to act on.

ClarifAI intercepts this vague feedback before it becomes noise. A four-module LLM pipeline filters out irrelevant responses, flags vague or ambiguous comments, and conducts a targeted follow-up dialogue to turn them into structured, actionable insights.

ClarifAI Feedback Dialogue

You said the AI summary was "kind of useful." Could you tell us a bit more about what you meant?

It was fine, just not that helpful I guess.

Thanks — was it that the summary was missing key information, that it was too long or too brief, or that the wording was unclear?

It skipped the part about dosage — that's what I actually needed.

Got it. I've logged: summary omitted dosage information — the most critical detail for this use case. ✓ Saved

How It Works

Four steps to actionable feedback.

1

Telemetry — Relevance Filter

The Telemetry module classifies whether each piece of feedback (e.g., an individual answer to a feedback question/prompt) is addressing the informational intent of the question/prompt. Off-topic or tangential responses are filtered out before they enter the pipeline.

[Telemetry LLMs demonstrate 100% alignment with expert relevance classifications.] — Placeholder until Spec Aim 1, Stage 3 results are in.

2

Flight — Vagueness & Ambiguity Detection

The Flight module further filters relevant feedback for responses that are not specific enough to act on (i.e., contain any vagueness or ambiguity). Only feedback that contains vagueness or ambiguity is escalated to the clarification dialogue, keeping the experience lightweight for users who already gave clear and relevant responses.

[Flight LLMs demonstrate 94% alignment with expert vagueness and ambiguity detection.] — Placeholder until Spec Aim 1, Stage 3 results are in.

3

CapCom — LLM Clarification Dialogue

The CapCom module engages the user in a short, targeted follow-up conversation, for each vague or ambiguous topic identified by Flight. The LLM interviewer only asks questions needed to resolve the vagueness or ambiguity, then sends the transcript of the conversation to the Payload module for final processing.

4

Payload — Summarisation & Refactoring

The Payload module extracts the relevant and granular information elicited from the user by the CapCom module, and injects the new granular information into the original feedback response — resulting in a structured, machine-readable, and specific insight that designers can act on directly.

System Architecture

From feedback to insight.

The complete ClarifAI pipeline — from AOFG collection through four LLM modules to structured, actionable output.

AOFG Platform

Stage 1

Prerequisite Task

Consent

Demographics

Stage 2

Project & Tasks

PI Usability Questions

Stage 3

Discussion Board

Per Question Discussion

LLM-assisted Pipeline

A) Telemetry

Contextual
Relevance

B) Flight

Requires
Clarification

C) CapCom

Clarification
Dialogue

D) Payload

Summarisation
& Refactor

Why It Matters

Better feedback. Better AI.

Actionability of Feedback by Condition

ClarifAI'd feedback High

Raw vague feedback (unprocessed) Low–Medium

No feedback collected None

100%

Telemetry precision

94%+

Flight accuracy

4

LLM pipeline modules

Active

Project status

To appear at UIST '26 · Detroit, MI