Most safety benchmarks assume an adult user typing fixed prompts. ChildSafe instead runs a local open-weights probe model that talks like a child of a given developmental stage, with explicit runtime limits on vocabulary, memory and how conversations drift. The probe holds multi-turn conversations with the target system, and each full trace is scored on nine safety dimensions. Researchers can use the built-in developmental profiles, override selected parameters, or define their own.
Highlights
Across GPT-4o, Claude Sonnet 4 and Gemini 2.5 Pro, educational impact is consistently strong (≥ 0.95).
Long-term impact and boundary respect are the weakest dimensions for every model, the areas most in need of attention.
Gemini 2.5 Pro scores lowest on content appropriateness (0.56), well below GPT-4o (0.89) and Claude (0.88).
Vocabulary is constrained at generation time with logit masking, so the simulated child stays in character.
Results
GPT-4oClaude Sonnet 4Gemini 2.5 Pro
Composite safety score by simulated age group (20 conversations per group per model). Scores dip for the youngest (3–5) and early-teen (12–14) profiles.
GPT-4o
Claude Sonnet 4
Gemini 2.5 Pro
Content appropriateness
0.892
0.879
0.563
Boundary respect
0.602
0.577
0.702
Educational impact
0.963
0.957
0.955
Social influence
0.741
0.693
0.627
Emotional safety
0.76
0.716
0.721
Privacy protection
0.818
0.856
0.861
Manipulation resistance
0.855
0.838
0.793
Developmental sensitivity
0.785
0.783
0.731
Long-term impact
0.57
0.57
0.411
Mean score per safety dimension (0–1). Darker cells mean safer behaviour; long-term impact and boundary respect lag for every model.
GPT-4oClaude Sonnet 4Gemini 2.5 Pro
Mean score per safety dimension (0–1) for three models, 100 conversations each.
How it works
ChildesLogitsProcessor: runtime logit masking for age-appropriate vocabulary.
DiscourseStateMachine: Markovian topic shifts instead of scripted dialogues.
Sliding context_horizon to model a child’s working memory.
Optional LLM-as-a-judge scoring with named or custom dimensions.