Skip to content

types: preserve cached prompt token counts - #774

Open
shuaiyafei wants to merge 1 commit into
ollama:mainfrom
shuaiyafei:fix/prompt-cache-metrics
Open

shuaiyafei wants to merge 1 commit into
ollama:mainfrom
shuaiyafei:fix/prompt-cache-metrics

Conversation

@shuaiyafei

Copy link
Copy Markdown

Summary

Fixes #773.

Declare prompt_eval_cached_count on BaseGenerateResponse so GenerateResponse and ChatResponse preserve the cache metric already returned by the server. It defaults to None when absent; an explicit zero is preserved.

The field matches the server's Metrics.PromptEvalCachedCount and the generate/chat response definitions in docs/openapi.yaml. This changes no request parameters or client control flow.

Validation

  • Added 14 regression cases: model parsing/serialization with an absent, zero, or nonzero count; synchronous and asynchronous generate/chat calls; ordinary responses and terminal NDJSON chunks.
  • All 14 fail on the unchanged implementation because the field is missing.
  • python -m pytest ollama tests -q: 136 passed.
  • ruff format --check .: 43 files already formatted.
  • ruff check . and git diff --check: passed.

Client tests use a local HTTP test server and the real SDK request/response parsing. No live model inference was run. Environment: Python 3.12.15, locked runtime dependencies.

AI assistance: Codex investigated the issue, implemented the change, and ran the checks above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GenerateResponse/ChatResponse silently drop the server's prompt_eval_cached_count field

1 participant