Show the headline metric in the program heatmap by default - #169
Merged
Conversation
The homepage heatmap rendered the bounded score (partial credit for close dollar answers) under a leaderboard whose headline is exact match, with only a paragraph naming the difference: Claude Fable 5.1 read 93% on federal income tax where its exact rate is 69%. The heatmap now defaults to exact match, offers within-1% and the bounded score as a labeled switch, states the selected metric and its definition, and says that cells are unweighted rates that do not average to the headline. Test renders the frozen data and checks the federal-tax row carries every model's exact rate, not the bounded scores. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Within 1% also accepts an answer within $1 when the reference is $0; the bounded score gives a nonzero answer to a $0 reference no credit. Test pins both against a frozen row (scenario_102 self-employment tax, Claude Haiku 4.5: $0.14 against $0 scores 100 exact and within-1%, 0 bounded). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The homepage "Program breakdown" heatmap rendered the bounded score (partial credit for close dollar answers) beneath a leaderboard whose headline is exact match, with only a paragraph naming the difference. Reading the two together, Claude Fable 5.1 showed 93% on federal income tax and 92% on state income tax where its exact-match rates are 69% and 63%, while its headline says 86.9%.
aria-pressedbuttons).No data or score change.
Tests
app/tests/heatmapMetric.test.ts: the default is exact; the metric selector returns exact and bounded values that differ where partial credit is large (federal tax 69 vs 93); the rendered federal-tax row carries every model's exact rate, not the bounded scores, and the copy names the metric.bun run lint && bun run test && bun run buildpass.🤖 Generated with Claude Code