Golden Dataset
The operation’s answer key: examples with a known correct answer, deliberately including the hard cases.
In one sentence
The set of examples with a known correct answer against which evals measure the system. It is the official answer key for the operation.
Before
Real cases, including the hard ones
What it does
Answer key with a known correct answer
After
Evals measure against it
Analogy
It is the test’s answer key. Without it, grading becomes opinion, and opinion changes depending on who is grading and what time it is.
Example
200 letters from the last three years that the coordination team approved without reservations, each paired with the field report that originated it.
Thirty of them were deliberately chosen because they are hard: a child who moved to another city midyear, a report with contradictory information, a child who left the program, a sponsor who writes in Spanish, a scanned report with poor handwriting. These 30 cases are what make the set valuable, since 200 easy letters measure almost nothing.
The common mistake
Building the golden dataset only with clean cases. The system gets 100%, everyone celebrates, and it breaks on the first real exception, because exceptions are exactly what was not in the answer key.
The second mistake is letting the dataset stay frozen. Every case that goes wrong in production needs to go into the dataset. That is how the golden dataset makes compound engineering work: each error becomes permanent protection, and the system improves with every failure instead of merely returning to normal.
In practice
- A versioned folder with input / expected-output pairs, plus one line explaining why each case is there.
- 30 to 50 well-chosen cases beat 500 random ones.
- Operational rule: no bug is considered resolved before it becomes a case in the dataset.
- Whoever decides what the "correct answer" is needs to be named, because that decision belongs to the business, not engineering.
How to make it tangible
A versioned folder with input/expected-output pairs and one line explaining why each case is there.
Connects with
Translated from Portuguese with AI assistance.