Skip to content

perf(llm): cache stable Claude prompts - #22

Open
rabbive wants to merge 2 commits into
sugarlabs:mainfrom
rabbive:perf/anthropic-prompt-cache
Open

perf(llm): cache stable Claude prompts#22
rabbive wants to merge 2 commits into
sugarlabs:mainfrom
rabbive:perf/anthropic-prompt-cache

Conversation

@rabbive

@rabbive rabbive commented Aug 23, 2026

Copy link
Copy Markdown

Mark byte-stable planning and codegen prefixes for Anthropic prompt caching without changing prompt content.\n\nRefs #20

Mark byte-stable planning and codegen prefixes for Anthropic prompt caching without changing prompt content.\n\nRefs sugarlabs#20

@rakshityadav1868 rakshityadav1868 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one @rabbive , verified locally-prefix stays byte-stable across specs and the tests pass. One small note: the cacheable minimum is model-dependent (1024 tokens on the default sonnet-4-6, but 4096 on some models like Haiku 4.5). The planning prefix is ~2K tokens, so if someone overrides AOD_CLAUDE_MODEL to one of those, caching silently no-ops. Harmless, but maybe worth a line in the comment.

@rabbive

rabbive commented Aug 25, 2026

Copy link
Copy Markdown
Author

Addressed the review feedback in 170c0a2.

Changes made:

  • Clarified that Anthropic’s cacheable minimum is model-dependent.
  • Documented the default Sonnet model’s 1,024-token minimum.
  • Noted that the roughly 2K-token planning prefix remains uncached on models requiring 4,096 tokens.
  • Clarified that this only causes a cache miss—the request and generated output remain unaffected.

Thanks for catching this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants