← Back to Playbooks

Pilot Feedback Loop

Purpose: Put the capability in front of a small user group, collect evidence, and answer: "Is it useful in real work?"

Typical duration: 1–2 weeks, depending on workflow volume and user availability.

When to use: After the first useful version proves the core workflow is plausible.

Inputs Required

  • Working first useful version
  • Known limitations and open risks
  • Identified pilot users (2–5 people who will actually use it)
  • Real examples the pilot users already understand

Principles

Focus: Is it useful?

Real users start using it in the workflow. Not a demo. Not a review session. Actual use against actual examples.

Built-in feedback mechanism. Users should be able to report issues without leaving the app. Add a feedback widget, a "report bug" button, something frictionless.

Architecture evolves with evidence. Pilot choices serve pilot needs. If something needs rebuilding for handoff, rebuild it with what you learned.

Daily feedback sessions. 15-minute syncs to review what users encountered. Quick decisions on small fixes.

Process

Step 1: Prepare For Users

  • Add feedback widget
  • Fix the most obvious limitations
  • Create minimal onboarding (enough to get started)
  • Set up usage tracking (basic analytics)

Step 2: Users Live

  • Pilot users start using it against real examples
  • Feedback triage sessions (see User Feedback Triage)
  • Rapid fixes based on feedback
  • Document patterns in what users struggle with

Step 3: Consolidate

  • Review all feedback
  • Categorize: quick fixes vs. architectural changes vs. feature requests
  • Draft readiness, hardening, or stop recommendations

Outputs

  • Working pilot with active users
  • Usage data (who's using it, how often, what features)
  • Feedback log (categorized)
  • Input for readiness, hardening, or handoff
  • Decision: harden, hand off, reframe, or stop?

Tips

  • 2–5 pilot users is enough. More users = more noise, not more signal at this stage.
  • Pick users who will actually use it, not users who will be polite about it.
  • If users stop using it after day 2, that's critical feedback. Find out why.
  • "Is it useful?" is harder than "Does it work?" Prepare for uncomfortable answers.
  • Do not fix everything. Some feedback belongs in hardening, some belongs in a later capability, and some is out of scope.

Next: Learning Transition -> Readiness & Handoff