Human-in-the-Loop (HITL) Review Process for AI Conversation Quality

Objective

Establish a structured and repeatable QA process to monitor and improve conversations, ensuring quality, compliance, and customer satisfaction.

Step 1: Review AI conversations

Daily Review

• Tool: Use the Review Page instead of querying the database to ensure a more user-friendly experience.
• Time Commitment: Spend 30 minutes daily reviewing conversations.
• Focus Areas:
  • Identify messages that require attention.
  • Flag any anomalies, inappropriate responses, or system errors.

Step 2: Identify and log issues

• Action: Create a bug in the AI Delivery board for every issue identified.
• Linkage: Include the bug ID/Link in the tracking spreadsheet for traceability.

Step 3: Documentation

• Format: Maintain an excel spreadsheet with the following columns:
  • Message ID – ID from [dbo].[ConversationLog]
  • Date/Time – Timestamp from [dbo].[ConversationLog]
  • Sender – UserId from [dbo].[Thread]
  • Issue Description – Message from [dbo].[ConversationLog]
  • Resolution / Preventive Action – Teams feedback/Resolution
  • Recommended AI Response – Teams feedback/Resolution
  • Linked Bug ID - (from AI Delivery board)

Query that can be used to get the above details:

SELECT C.Id, C.Timestamp, C.Message, T.UserId
FROM [dbo].[ConversationLog] C
Join [dbo].[Thread] T
On C.ThreadId = T.ThreadId
Where C.Message like '%calorie%'

Step 4: Weekly Review Meeting

• Schedule: 30-minute recurring meeting every week.
• Purpose: Discuss notable conversations, review flagged issues, and share insights.
• Urgency Protocol: Immediately notify the team if any alarming issues are found before the scheduled weekly meeting.

Step 5: Team Presentation

• Present findings from the excel document during the weekly meeting.
• Discuss trends, recurring issues, and improvement opportunities.
• Use feedback to evolve the Review Page and refine the QA process.

Step 6: Storage

• Upload and maintain the document in the AI Delivery Wiki.

Step 7: Updates

• Ensure the document is updated regularly and reviewed during weekly meetings.

Process Execution

Step	Task	Frequency	Tool
1	Review AI conversations	Daily	Review Page
2	Identify and log issues	Daily	Review Page + Excel
3	Create bug tickets	As needed	AI Delivery Board
4	Link bugs in spreadsheet	As needed	Excel
5	Weekly review meeting	Weekly	Teams
6	Present findings	Weekly	Excel + Wiki
7	Update documentation	Weekly	AI Delivery Wiki

Flowchart Overview

This flowchart summarizes the end-to-end lifecycle and emphasizes the single 30‑minute daily review cadence.