Evals: how good is the extraction?
Every sample meeting has hand-labeled expected follow-ups. An extracted item counts as correct when the owner matches and the title contains the key phrase. Destination and due date are then checked on the matched items. The held-out meetings were written after the rules engine and were never used to tune it, so they are the honest number.
Dev set · rules engine
Used while writing the rules: expect this to look perfect.1.00precision
1.00recall
1.00F1
1.00routing accuracy
1.00due-date accuracy
| Meeting | Expected | Predicted | Matched | Misses / extras |
|---|---|---|---|---|
| northfield-pilot-scoping | 6 | 6 | 6 | — |
| summit-team-plan | 4 | 4 | 4 | — |
| leadership-sync | 7 | 7 | 7 | — |
| investor-checkin | 4 | 4 | 4 | — |
Held-out set · rules engine
Never used for tuning: this is the real generalization signal.0.57precision
0.44recall
0.50F1
0.50routing accuracy
1.00due-date accuracy
| Meeting | Expected | Predicted | Matched | Misses / extras |
|---|---|---|---|---|
| heldout-cedar-valley-site | 4 | 3 | 1 |
missed · Priya Natarajan: data retention statement missed · Javier Monestel: counsel missed · Javier Monestel: log
extra · Grace Okafor: Resubmit extra · Javier Monestel: Set that up
|
| heldout-growth-standup | 5 | 4 | 3 |
missed · Dev Patel: free-month credit missed · Dev Patel: invite job
extra · Dev Patel: Do it Monday morning
|
Reading these numbers
The rules engine is close to perfect on the meetings it was built against and drops to about half on new phrasing ("Action item for me: …", "Let me log …", third-person commitments). That gap is exactly why the production path is Claude with a typed schema, with the rules engine kept as a zero-cost fallback. To score Claude on the same sets:
ANTHROPIC_API_KEY=... python -m evals.run --engine claude