Wider Reach, Thinner Judgment: AI-Conducted Critical-Incident Interviews, Facilitated Workshops, and Head-Office Guidelines for Making Frontline Customer-Interaction Values Explicit

Abstract

In 42 customer-facing sites across six organizations, we randomly assigned sites within each organization to one of three ways of writing down how their staff treat customers. At "AI-interview" sites (n=14), an AI interviewer asked frontline staff to recount specific customer interactions that had gone notably well or badly, probed for what they did and why, and grouped the accounts into a draft playbook that a small staff group edited in one two-hour session. At "workshop" sites (n=14), a human facilitator ran two half-day workshops producing the same kind of playbook, and "head-office" sites (n=14) kept their existing service guidelines. More staff contributed at AI-interview sites (66%) than attended workshops (52%), and AI-interview playbooks described more distinct situations (46 versus 37, a 24% gap). But fewer of their entries said when the usual practice should be set aside (21% versus 27% for workshop playbooks). Over their first three months, the 318 new hires at these sites received customer ratings of 5.5/7 at AI-interview sites, 5.6/7 at workshop sites, and 4.8/7 at head-office sites (gaps of 15% and 17% over head-office sites). In interactions involving a problem, such as a disputed charge or a complaint, AI-interview new hires were rated 4.6/7, against 5.2/7 for workshop new hires (a 13% gap) and 4.4/7 at head-office sites.

1. Introduction

Every customer-facing organization has a few people whom colleagues describe as simply knowing how to handle customers. The usual response is to treat them as the store of that knowledge, assign them the newcomers, and hope it rubs off. It mostly does not rub off, and when those people leave, much of what they knew leaves with them. Head-office service guidelines are meant to fill the gap, but they are typically written by people who no longer stand at the counter, and frontline staff in our sample often described them as accurate but not how anyone actually works.

Making such knowledge explicit is an old problem with established methods. One of the most widely used is the critical incident technique: instead of asking people to state their principles, which they often cannot, the interviewer asks them to recount specific occasions when something went notably well or badly, and the principles are inferred from the accounts. The technique has always been limited by interviewer time. AI interviewers remove that limit, and organizations have begun using them to collect such accounts from many staff at once and to group the results automatically.

We tested whether this approach could produce a usable account of how a site's staff treat customers, including the values behind that treatment, and compared it with human-facilitated workshops and with the head-office guidelines already in place. We expected the AI approach to do well on coverage and less well on judgment, and we measured both.

2. Method (Summary)

Six organizations took part: a hotel group, a home-goods retailer, two regional banks' branch networks, a car-rental operator, and a municipal service-center network. Each contributed seven sites, which were randomly assigned within each organization: three sites to one arm and two to each of the others, rotating across organizations so that each arm totaled 14.

At AI-interview sites, all frontline staff (352) were invited to complete a paid interview with an AI interviewer by text or voice. The interviewer asked for two to four specific interactions that had gone notably well or badly and followed each with set probes: what the staff member did, what they were weighing at the time, and what they would tell a new colleague. It then grouped the accounts into situations, each with a description of the usual practice and the reasons staff gave for it. Five or six volunteer staff at each site edited this draft into a playbook in one paid two-hour session. At workshop sites, all frontline staff (341) were invited to two paid half-day workshops led by a trained facilitator, who used the same incident-recall prompts and produced a playbook in the same format. Head-office sites (353 staff) kept their organization's existing customer-service guidelines.

Playbooks were completed within ten weeks and used in onboarding and weekly team briefings from then on. Two staff with five or more years of experience from sites outside the study, blind to arm, coded each playbook entry for whether it stated when the usual practice should be set aside. All frontline staff were surveyed on whether their site's playbook or guidelines described "how we actually treat customers here" (1–7; 79% responded).

The primary outcome was customer ratings of new hires. Over the six months after playbooks were completed, sites hired 318 frontline staff (109 at AI-interview sites, 104 at workshop sites, and 105 at head-office sites). For each new hire's first three months, customers rated interactions on a post-visit survey item asking whether they had been "treated the way I hoped to be" (1–7). Interactions were flagged as involving a problem when the visit record showed a disputed charge, a complaint, or a request that could not be met. Customers rated 9,842 new-hire interactions, of which 1,377 (14%) were flagged as involving a problem.

3. Findings

At AI-interview sites, 232 of 352 staff (66%) completed an interview, recounting 1,214 incidents in total. At workshop sites, 177 of 341 staff (52%) attended at least one workshop, a 27% gap. Part-time staff took part at a higher rate at AI-interview sites (57%) than at workshop sites (45%).

AI-interview playbooks described a mean of 46 distinct situations, against 37 for workshop playbooks (a 24% gap). Head-office guidelines were organized by rule rather than situation and were not compared on this measure. Of playbook entries, 21% at AI-interview sites stated when the usual practice should be set aside, against 27% at workshop sites.

Staff rated their site's document as describing how they actually treat customers at 5.1/7 at AI-interview sites, 5.4/7 at workshop sites, and 4.2/7 at head-office sites.

Customers rated new hires' interactions at 5.5/7 at AI-interview sites, 5.6/7 at workshop sites, and 4.8/7 at head-office sites (gaps of 15% and 17% against head-office sites). In interactions involving a problem, ratings were 4.6/7, 5.2/7 (a 13% gap in favor of workshop sites), and 4.4/7.

Within the AI-interview arm, eight sites had at least two staff with five or more years of experience in their editing session. At those sites, 23% of playbook entries stated when to set the usual practice aside, against 18% at the other six, and new hires' ratings in problem interactions were 4.9/7 (n=62 new hires) against 4.2/7 (n=47).

An AI interview took a median 47 minutes of a contributor's time. Workshop attendees spent about seven hours, and facilitators logged a mean of 22 hours per site.

4. Discussion

The AI interviews did what their advocates say they do. More staff took part, including many part-timers who could not get to a half-day workshop, and the playbooks they produced covered more situations at a fraction of the staff hours. New hires trained on them did about as well as workshop-trained new hires in ordinary interactions, and both did clearly better than new hires trained on head-office guidelines, which frontline staff themselves rated as least like how they work.

Where the AI playbooks fell short was in knowing when not to follow them. Problem interactions are where a playbook's usual practice most often has to bend, and there the AI-interview new hires were rated closer to head-office new hires than to workshop new hires. The coding offers a likely reason: fewer AI-interview entries said when to set the usual practice aside. Reading the drafts suggests how this happened. When several staff described a similar situation and one of them had done the opposite of the rest, the grouping step tended to fold that account into the majority practice, and the exception, often from someone with long experience, disappeared from the draft. In workshops, that person was in the room to object.

The editing-session subgroup points the same way, though it is small and was not planned. AI-interview sites whose editors included at least two long-serving staff came noticeably closer to the workshop sites on both exception entries and problem-interaction ratings. That complicates the idea that the AI interview replaced the experienced colleague. It reached more people and recorded more of the routine, but the judgment calls still came from the experienced colleague, now in a two-hour session rather than a career of informal mentoring.

On balance, the approach was moderately effective: a real gain over head-office guidelines, at much lower cost than workshops, with a specific and measurable weakness in exactly the situations that most test a site's values. The staff who were interviewed, and especially those who sat in editing sessions, gave time that went to colleagues they had not yet met and to those colleagues' customers.

5. Limitations and Future Directions

Staff knew which arm their site was in. Workshop playbooks were drafted and edited by the same group, while AI-interview drafts passed to a separate small editing group, so the arms differ in more than the interview method, and part of the gap in exception entries may come from the editing step rather than the grouping step. Customer ratings came from those who chose to answer a post-visit survey. Problem interactions were identified from visit records, which organizations kept with differing care. The editing-session subgroup was not randomized, and sites with more long-serving staff may differ in other ways. The AI interviewer's probes were fixed in advance; different probes, especially ones asking directly when a practice should not be followed, might produce different playbooks. Six months of hiring gave about a hundred new hires per arm. A follow-up will add a probe asking for exceptions and assign long-serving staff to editing sessions at random.

Editorial Note

This report underwent CfAHC's Rapid Review Protocol, including AI-assisted literature cross-checking and a two-reviewer human sign-off.