Situation-Matched Value Notes, Not a Single Values Prompt or Default Settings: Community-Curated Guidance for AI-Drafted Replies in Member-Facing Teams

Abstract

In 36 member-facing teams across eleven community organizations, all using the same AI assistant to draft written replies, we randomly assigned teams within each organization to one of three set-ups for twelve weeks. In "value-note" teams (n=12), staff wrote short notes on what their community values in specific situations, and the assistant drew on the two or three notes matching each incoming message. "Values-prompt" teams (n=12) used one fixed paragraph describing the organization's values on every request, and "default" teams (n=12) used the assistant as supplied. Members of each community, blind to set-up, rated sent replies from value-note teams 5.3/7 for fitting "how we treat each other here," against 4.5/7 for values-prompt teams (an 18% gap) and 4.2/7 for default teams (a 26% gap). Among drafts sent without substantial staff edits, the ratings were 5.2/7, 4.3/7 (a 21% gap), and 4.1/7 (a 27% gap). Follow-up complaints or escalations within 14 days ran at 3.4 per 100 replies for value-note teams, against 4.2 (a 24% gap) and 4.4 (a 29% gap). The notes cost staff a median 34 hours per team in the first four weeks and 1.6 hours a week after that. By week twelve, 27% of incoming messages in value-note teams still matched no note, and replies to those messages were rated 4.6/7, about the same as values-prompt replies.

1. Introduction

A short instruction that one cooperative's staff wrote for themselves reads: "When a member is behind on fees, name the next step before the amount." A community clinic in our sample wrote nearly the opposite for a similar situation, asking staff to state the amount plainly and early so that no one felt anything was being hidden from them. Neither instruction is wrong, and neither would survive being averaged into a general standard of courtesy, which is roughly what an AI assistant tuned for everyone carries by default.

Organizations that want an assistant to reflect their own community's values have usually done so by adding a description of those values to every request, a single paragraph that tells the assistant who it is speaking for and how. Such a paragraph has to be general enough to apply to every message, and the instructions above are useful precisely because they are not general: each one applies to a kind of situation and says what this particular community expects in it. An alternative, borrowed from the way organizations already give assistants access to their own documents, is to keep a library of situation-specific value notes and supply only the ones that match the message at hand.

We compared these approaches with each other and with no added guidance, measuring how well replies fit each community's values as judged by its own members, and what the notes cost the staff who wrote them.

2. Method (Summary)

Eleven organizations took part: a housing cooperative network, three community health clinics' front desks, two public library systems, a regional transit operator's customer desk, two mutual insurers' member-service teams, and two member-owned sports and leisure clubs. Each had at least three teams answering written messages (email and chat) from members or patrons, and all 36 teams used the same AI assistant to produce first drafts, which staff edited and sent. Within each organization, teams were randomly assigned to the three set-ups in balanced blocks, so that each organization had every set-up and each set-up had 12 teams, for twelve weeks. In total, 284 staff took part.

In value-note teams (n=12), staff spent the first four weeks writing notes, each tied to a situation type (for example, a member disputing a charge, a complaint about another member, a request the organization cannot meet) and stating what the community values in that situation, with one example of a phrasing to use and one to avoid. Notes were drafted by frontline staff and agreed at a weekly team meeting. For each incoming message, the assistant retrieved up to three notes whose situation type matched the message and drafted with them in view. Teams could add or revise notes throughout. In values-prompt teams (n=12), staff wrote, over the same four weeks, a single paragraph of up to 200 words describing their organization's values and preferred tone, which was supplied with every request. Default teams (n=12) used the assistant as supplied. All three groups began measurement in week five.

From weeks five to twelve, 60 sent replies per team (2,160 in total) were sampled at random. Six members of each organization's community, recruited through member newsletters and noticeboards and blind to set-up, rated the replies from their own organization, with identifying details removed, on how far each reply fitted "how we treat each other here" (1–7). Staff edits were classified from the system's draft and sent versions; an edit was substantial if more than a quarter of the draft's sentences were changed or removed. Follow-up complaints and escalations within 14 days of a reply came from each organization's case records. Value-note teams logged the hours spent writing and maintaining notes.

3. Findings

Members rated replies from value-note teams 5.3/7, values-prompt teams 4.5/7 (an 18% gap), and default teams 4.2/7 (a 26% gap). Values-prompt and default replies were close to each other.

Staff substantially edited 24% of drafts in value-note teams, 29% in values-prompt teams, and 31% in default teams. Among replies sent without substantial edits, member ratings were 5.2/7, 4.3/7 (a 21% gap), and 4.1/7 (a 27% gap).

Follow-up complaints or escalations within 14 days ran at 3.4 per 100 replies in value-note teams, 4.2 in values-prompt teams (a 24% gap), and 4.4 in default teams (a 29% gap).

Value-note teams logged a median 34 hours writing notes in the first four weeks and 1.6 hours a week after that. Their libraries grew from a median 18 notes at week four to 41 at week twelve. At week twelve, 27% of incoming messages still matched no note. Across the 720 sampled replies from value-note teams, 31% answered messages that matched no note; replies to matched messages were rated 5.6/7, and replies to unmatched messages 4.6/7, close to the values-prompt figure.

Notes differed between organizations more than within them. Of the 23 situation types that appeared in at least three organizations' libraries, 15 had notes in at least two organizations that called for different handling of the same situation, as with the fees example above.

4. Discussion

The fixed values paragraph did less than expected, lifting member ratings only slightly above an assistant with no guidance at all, and the reason appears to be that a description of values general enough to apply to every message says little about what to do with any particular one, whereas a note that names a situation and the community's expectation in it gives the assistant something it can act on. Replies to messages that matched no note fell back to roughly the values-prompt level, which suggests the benefit came from the match rather than from the general attention to values that writing the notes may have produced among staff.

The finding we would least want to lose is the one about difference. The same situation type called for different handling in different communities, and members could tell: they rated their own organization's note-guided replies as more like how they treat each other, not as more polite in some general sense. An assistant adjusted once for everyone cannot hold that variation, and the notes did so without any change to the assistant itself.

That variation had to be written down by someone, and here it was frontline staff, at a median 34 hours per team before the first reply was measured and more than an hour and a half a week thereafter, time taken from answering messages. The members received the result. Just over a quarter of messages still matched no note after twelve weeks, so this is not work that finishes.

5. Limitations and Future Directions

Staff knew which set-up their team had, and writing notes may have changed how they edited drafts; the comparison among unedited drafts reduces but does not remove this concern. Member raters were volunteers who answered a call for help and may care more about their organization's conduct than members in general. The values-prompt teams spent the same four weeks writing a single paragraph, but almost certainly fewer hours, so the comparison does not equalize staff effort. Twelve weeks cannot show whether libraries are kept up once a study ends, and the 27% of unmatched messages suggests that coverage depends on continued effort. Messages that matched no note were likely less common or harder situations, so their lower ratings may reflect the messages themselves as well as the absence of a note. Matching was by situation type as staff defined it, and notes that were poorly labeled may have been missed. A follow-up will test whether a smaller, shared starter library, adapted by each community, reduces the setup hours without losing the gain.

Editorial Note

This report underwent CfAHC's Rapid Review Protocol, including AI-assisted literature cross-checking and a two-reviewer human sign-off.