Peer-Reviewed Prediction Logs, Not Private Logs or No Log: Making Slow Supplier Outcomes Visible to Procurement Buyers

Abstract

We randomly assigned 246 procurement buyers at 11 manufacturing firms to one of three practices for 10 months. "Peer-log" buyers (n=83) wrote a short prediction before each significant supplier decision, covering expected delivery timing, quality problems, and cost, and met monthly for 30 minutes with a buyer from another site to compare resolved predictions with what had happened, each acting as reviewer for the other. "Private-log" buyers (n=82) kept the same log and were prompted monthly to compare it with outcomes alone. "No-log" buyers (n=81) worked as before. Outcomes are for the 227 buyers still in post at month 10. In months 7 to 10, 17.8% of peer-log buyers' significant decisions ended in late delivery or quality rejection, against 21.9% for private-log buyers (23% higher) and 22.6% for no-log buyers (27% higher), from rates of 22.8% to 23.4% in all three groups in the six months before. On a test of 40 past supplier decisions with known outcomes, peer-log buyers scored 63.7/100, against 54.9 for private-log buyers (a 16% gap) and 52.4 for no-log buyers (a 22% gap). By month 10, 71% of peer-log buyers were still logging, against 57% of private-log buyers. Among no-log buyers, those with 10 or more years in procurement scored 53.1 on the test, against 51.8 for those with under three. In month 2, peer-log buyers rated reviewing their missed predictions as more uncomfortable (3.7/7, against 2.9/7 for private-log buyers), a gap that had closed by month 10.

1. Introduction

Most procurement teams have a veteran buyer who is said to "just know" which suppliers will deliver on time. Teams rely on that person, and newer buyers are told to ask them. The difficulty is that a supplier decision resolves months after it is made, often after the buyer has moved on to other orders, and nobody records what the buyer expected at the time. Whether the veteran's judgment is better, or only more confident, is rarely tested.

A prediction log creates the missing record: the buyer writes down what they expect before the decision, so there is something to compare with the outcome when it arrives. The Organizational Behavior Unit's report in this issue found that years in a slow-feedback role predicted little growth unless a manager traced decisions to outcomes. We asked whether buyers could do that tracing themselves, alone or with a peer, without involving a manager.

2. Method (Summary)

Eleven firms took part, and all 246 buyers in their procurement teams were randomly assigned, stratified by firm, to one of three practices for 10 months. A significant decision was any order above the firm's review threshold or any first order with a new supplier.

Peer-log buyers (n=83) entered, before each significant decision, a three-line prediction in a shared spreadsheet: whether the order would arrive on time, whether any part of it would fail quality inspection, and whether its final cost would differ from the quoted cost by more than 5%, each with a confidence rating from 1 to 5. Each was paired with a buyer at another site of the same firm. Once a month, the pair met for 30 minutes to go through each other's predictions that had resolved. A session guide asked them to begin with one prediction that had held and one that had not, and, for each miss, to ask what the buyer would have needed to know at the time. There was no scoring and nothing was reported to managers. Private-log buyers (n=82) kept the same log and received a monthly prompt to compare their resolved predictions with outcomes on their own, using the same questions. No-log buyers (n=81) worked as before.

Decision outcomes were taken from the firms' purchasing records: a decision counted as a problem if the order arrived more than five working days late or any part of it was rejected at quality inspection. Rates were calculated for the six months before the study and for months 7 to 10. At month 10, buyers took a test of 40 significant decisions from their firm's records in previous years, with supplier details and the information available at the time but not the outcome, and predicted each outcome. Decisions the buyer had personally made were excluded from their test. Answers were scored automatically against the records from 0 to 100. Years in procurement came from HR records. Logging was counted as continuing if a buyer made at least one log entry in month 10. In months 2 and 10, buyers who kept a log rated how uncomfortable reviewing their missed predictions had been (1 = not at all, 7 = very) in a survey.

3. Findings

Of the 246 buyers, 227 were still in post at month 10 (77 peer-log, 76 private-log, and 74 no-log), and the results below are for them.

For the 227 buyers analysed, in the six months before the study, 23.1% of significant decisions by buyers later assigned to peer logs ended in a problem, against 22.8% for private-log and 23.4% for no-log buyers. In months 7 to 10, the rates were 17.8%, 21.9%, and 22.6%. The private-log rate was 23% higher than the peer-log rate and the no-log rate 27% higher.

On the 40-decision test, peer-log buyers scored a mean 63.7/100, against 54.9 for private-log buyers (a 16% gap) and 52.4 for no-log buyers (a 22% gap).

At month 10, 55 of 77 peer-log buyers (71%) were still logging, against 43 of 76 private-log buyers (57%).

Experience counted for little without a log. Among no-log buyers, the 19 with 10 or more years in procurement scored 53.1 on the test, and the 24 with under three years 51.8. Among peer-log buyers, the 21 with 10 or more years scored 64.2, and the 25 with under three years 62.9.

In month 2, peer-log buyers rated reviewing their missed predictions at 3.7/7 for discomfort, against 2.9/7 for private-log buyers (peer-log ratings 28% higher). By month 10, the ratings were 2.8 and 2.7. Peer-log buyers reported spending a mean 45 minutes a month on the review sessions, including preparation, against 18 minutes for private-log buyers' monthly reviews.

4. Discussion

The month-2 discomfort ratings are the place to start. Showing a colleague a prediction that turned out wrong was harder than reading it alone. The session guide started each meeting with a prediction that had held, and the discomfort had faded by month 10, but it was a real cost in the first months. Each peer-log buyer also gave time as a reviewer of a partner's log, and the sessions took about 45 minutes a month against 18 for reviewing alone.

What the pairs got for it was a record they actually used. Private logs started with the same entries and the same prompts, but fewer private-log buyers were still logging by month 10, and their decisions improved much less. A monthly appointment with a colleague who is also showing their own misses seems to have kept the comparison going after the novelty wore off.

The result on experience complicates the picture of the veteran who "just knows." Without a log, buyers with a decade in procurement predicted outcomes only a little better than buyers with under three years. That does not make the veterans' knowledge worthless; it suggests their experience had rarely been checked against outcomes, so some of it was confidence rather than accuracy. The peer log gave both veterans and newer buyers a way to find out which of their expectations held, without needing anyone senior to do it for them.

5. Limitations and Future Directions

The 40-decision test was given only at month 10, so we rely on randomization for baseline equivalence on that measure, although problem rates before the study were similar across groups. Problem rates depend on suppliers as well as buyers, and peer-log buyers may have shifted toward suppliers they already trusted rather than judging better. Buyers may have remembered outcomes of past decisions made by colleagues, which would favour longer-serving buyers on the test. Pairs were within the same firm, so partners may have shared supplier knowledge directly, which we cannot separate from the effect of reviewing predictions. Nineteen buyers left before month 10. The discomfort measure was a single item asked only of buyers who kept a log. Because each peer-log buyer was both reviewer and reviewed, we cannot say whether giving or receiving review mattered more. A follow-up will test one-way review, in which a newer buyer reviews a veteran's log, to separate the two.

Editorial Note

This report underwent CfAHC's Rapid Review Protocol, including AI-assisted literature cross-checking and a two-reviewer human sign-off.