[BUG](log): Revalidate compaction backpressure

[lyon-chroma]

A collection can keep moving through compaction while a former log host reports millions of operations as pending. The host caches the committed compaction position, so an obsolete value can inflate backlog metrics and cause writes to be rejected. This draft refreshes those positions sooner and verifies durable progress before enforcing compaction backpressure.

The incident evidence and recommended recovery checks are in the central investigation. In incident 25272, xAI collection 60f88932-e829-4c3e-8f97-7ff7355474b8 has a committed position above 70.3 million while two hosts report positions of 41.6 million and 51.5 million. Compaction continues without worker restarts. The apparent backlog clears near the 180-reinsertion refresh boundary. Customer write rejections from that incident remain unconfirmed.

The log service now refreshes cached positions every six dirty-log summarization passes by default, approximately one minute at the default interval. It also refreshes immediately when a summary reaches the write-backpressure threshold. This cadence is independent of the waiting period that allows small writes to form compaction batches. The new setting is compaction_cursor_refresh_interval_rollups; zero or one refreshes every pass.

Before rejecting a write whose object-storage summary reports a full log, the service reads the durable compaction cursor and checks the current summaries again. Requests for the same collection share the existing collection lock and can reuse a verified cursor for at most one second. Replicated-log pressure remains authoritative. A missing cursor preserves the pending range; an unreadable or corrupt cursor returns an unavailable error without admitting the write. Explicit cursor updates and cache purges invalidate the short verification cache. Fresh verification logs the collection, reported position, pending limit, durable position, and resulting decision.

Validation: six focused tests pass using local cursor files. They cover early refresh for pressured summaries, periodic refresh below the batch threshold, admission after another host commits progress, continued rejection when the pending range grows, missing cursors, replicated-log pressure, expiry of verification, and corrupt durable cursors. The existing cluster-backed reinsertion-eviction test remains for CI and is not run locally because its services are unavailable. Formatting, whitespace checks, and Clippy with warnings treated as errors pass for the library and test targets.

The tradeoff is additional cursor reads during summarization and on the first pressured request after verification expires. Summarization reads use the existing concurrency limit. The verification cache reduces repeated reads under concurrent pressure, but storage latency remains on this exceptional request path. The larger paging threshold in Terraform PR 8468 is separate mitigation and does not change the per-collection backpressure threshold.

Human release and verification TODOs:

• Review CI and merge the fix into main, then backport through the database hotfix workflow if needed for the current production release.
• Deploy the log service. The defaults enable the repair without a configuration change or staged rollout.
• Verify a collection whose committed position advances on another host clears stale summaries within the refresh cadence, and inspect revalidation decisions before claiming customer-write recovery.
• Check cursor-read volume, storage latency, backlog metrics, and write errors after deployment.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)