Example Center
A data management platform that curates ground-truth examples for AI model training, evaluation, and reviewer certification at Meta.
My Role
Design lead
Team
1 x Content Designer
3 x Software Engineers
Time
2025-2026
My Contribution
Product Strategy
Product Design
IMPACT
Reduced certification test curation from 8 weeks to 1–2 days, helping certify more than twice the target number of review experts.
BACKGROUND
One bet, many moving parts
Meta’s AI for Integrity bet aimed to automate 40% of human review in H1 and scale toward 80% by year-end, while matching or exceeding human performance.
Delivering that bet required multiple workstreams. Foundry (the first case study) helped teams develop and deploy AI automation. Example Center provided the foundation: trusted training and evaluation data to teach models what good decisions look like and evaluate whether they were production-ready.
Automate 80% of Human Review
40% in H1 • surpass human accuracy & speed
Other workstreams
Infrastructure & Policy
Foundry
AI Development Platform
Example Center
Training & Evaluation Data
PROBLEM
The ground-truth data gap
The quality and quantity of ground-truth labels set the ceiling for automation. To train models and evaluate their decisions, Meta needed a large, steady supply of trusted data.
But supply was falling short of demand, putting the 80% goal at risk. To hit it, Meta needed to produce far more trusted data, faster.
Demand
Supply
As automation scales
Gap putting 80% goal at risk
Ground-truth label supply vs. demand
Labels needed
PROCESS
Finding the real bottleneck
When supply ran low, Meta increased review-expert headcount. But hiring more review experts alone didn’t help if they couldn’t get certified fast enough. Certification itself was the bottleneck: each test required heavy cross-functional coordination and had to be rebuilt from scratch every cycle.
At the Skill Certification Summit, my tech lead and I traced it to two root causes:
Curating ground-truth examples for a certification test took 8 weeks of cross-team coordination.
Even with more review experts ready to be certified, each test still required three teams working in sequence for weeks:
8-week process
Operations
defined the test requirements and coordinated execution
Hand off
Data science
calculated violation type prevalence and difficulty
Hand off
Engineering
built the data pipeline and sourcing jobs
The data behind every test lived in hard-coded spreadsheets.
Test composition depended on violation-type prevalence and difficulty, which changed as policies and abuse patterns evolved. Because those values were hard-coded, every new test required manual rework to stay representative.
Hard-coded test composition
Prevalence
…
Fixed
Difficulty
…
Fixed
as policies & abuse patterns evolve
Outdated test composition
Prevalence
…
Stale
Difficulty
…
Stale
SOLUTION
Two inputs, one ready-to-use collection
I designed a self-serve curation system in Example Center. Instead of three teams coordinating for weeks, users provided two inputs — policy area and number of examples — and the system generated a ready-to-use example collection for certification tests.
The system derived decision, difficulty, and violation-type distributions from test requirements and live production data, keeping each test representative without manual rework.
1–2 days
User input
Policy area
# of examples
Behind the scenes
Auto-calculated
test composition
from test requirements + live production data
Output
Ready-to-use certification test
A self-serve test curation system
Test composition generated from test requirements and live production data
Users create a collection by choosing the collection type, violation group, and example count. Example Center generates the test composition automatically.
DESIGN
One happy path, and the two cases that break it
The design above shows the happy path: Example Center has enough matching examples, and those examples stay stable once collected. But real-world content review is messier. As policies and abuse patterns evolve, an example’s decision, difficulty, or prevalence can change. I designed for two scenarios that break the happy path:
Scenario 1: Demand can't be fully met.
When Example Center couldn’t supply every needed example, I designed a request flow that queued missing examples into the review pipeline and added them once available. This prevented teams from certifying review experts on incomplete sets.
Users request missing examples, which are queued for review and added once available.
Scenario 2: Collected examples change after the fact.
Examples can change after collection as policies and abuse patterns evolve. When a collection no longer matches the test requirements, I designed a flow that flags the change and prompts users to update it.
Changed examples are flagged so users can keep the collection aligned with the test requirements.
TRADEOFF
Recommended defaults, with room to disagree
The derived distributions — decision, difficulty, and violation type — came from the test requirements and production data, so I let the system fill them in by default.
But the certification process was still maturing. Some users had valid reasons to adjust the defaults, while fully open editing could introduce unreliable changes to the data used for certification.
I balanced this by making system-derived values recommended defaults: users could override them, but any change triggered a warning rather than a hard block. This kept the system’s values as the trusted starting point while preserving flexibility for informed edits.
Users can override recommended defaults, with warnings that explain the impact of each change.
IMPACT
From eight weeks to two days
The self-serve test curation system cut certification test creation from 8 weeks to ~1–2 days and reduced operational time and cost by 83%. It also helped certify more than 2x the target number of review experts across most policy areas, producing the ground-truth data needed for the 80% year-end automation goal.
1–2 days
Test creation time
83% reduction
Operational time & cost
2x target
Review experts certified
WHAT'S NEXT
Making the data work smarter
Auto-curating example collections for certification tests solved the supply bottleneck. Looking ahead, I see an opportunity for Example Center to evolve into a smarter data engine:
Close the loop with Foundry.
Today, data flows one way: Example Center produces ground truth, and models train on it. The next step is feeding model disagreements back into Example Center so failures become valuable training data.
Give Example Center a dedicated homepage.
With millions of examples and countless collections, users shouldn’t have to hunt for what changed. A homepage could surface relevant changes and action items proactively.
Use AI to help build the data.
An AI agent could draft candidate examples, pre-label easy examples, and flag likely mislabels, allowing experts to focus on the hardest judgments.