pip-audit-bulk: fail when any scan is incomplete

[p-linnane]

osv-scanner exits 127 with an empty JSON result when OSV is unreachable, which was recorded as a clean scan and published. Any other exit than 0 or 1 now fails the audit and lists the formula in the summary. Scans pass --no-resolve, since resolving pinned resources through deps.dev also exits 127 for some formulae.

GPT-6 Astra drafted the change. Claude Code (Opus 5.5) reviewed it, reproducing the outage behaviour and running the patched audit over all 547 formulae with no failures. I reviewed the diff.

[coderabbitai]

📝 Summary

Summary by CodeRabbit

• New Features
  • Audits now scan pinned resources without resolving additional dependencies. Reports identify failed scans and retain findings from completed scans.
  • Incomplete audits fail CI and do not publish results or generate pull requests.
• Tests
  • Added regression coverage for scanner failures, partial results, clean audits, and pinned-resource scanning.
  • CI now runs the regression tests.

The audit scans pinned formula resources without resolving additional dependencies. It records scanner failures, reports incomplete audits, and exits with a failure status. New tests exercise successful scans, scanner failures, partial findings, and the pinned-resource scan scope.

Changes: Audit execution

Layer / File(s)	Summary
Pinned-resource scan scope pip-audit-bulk, tests/test_pip_audit_bulk.py, README.md	pip-audit-bulk passes --no-resolve to osv-scanner. The README describes the pinned resources as the scan inventory. A test checks the scan result without dependency resolution.
Incomplete audit handling pip-audit-bulk, tests/test_pip_audit_bulk.py, .github/workflows/audit.yml, README.md	The script reports failed formulas, preserves findings from completed scans, and exits with status 1 when scans fail. Tests cover scanner errors, missing output, partial findings, and clean scans. The workflow runs unittest discovery, and the README documents failure reporting and the test command.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Merge Risk: 🔵 Low · up to 9f58b

The audit now fails visibly when scans are incomplete, which fixes the stated problem. A hung scanner could still stall a run, so consider adding a timeout, but this need not block merging.

Security architecture risk: 🔵 Low · up to 9f58b

The change strengthens failure handling: incomplete scans no longer allow the current audit to be published or drive automated fixes. No introduced security concern was confirmed. Remaining uncertainty concerns inventory completeness and how independent consumers recognize older published results after an audit fails.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• observed — A recognized failure in any scanned formula now stops publication and downstream automation for the whole audit batch. Completed findings remain diagnostic rather than authorizing partial automated action.

Trust Boundaries and Controls

• observed — Scanner output crosses into report interpretation only after the new exit-code and nonempty-output checks. An error exit accompanied by empty-results JSON no longer becomes a clean scan. Existing workflow gates then prevent that failed current run from reaching publication or automated PR generation.

Hardening Proposals

• proposed — If independent consumers require current-run assurance, define report freshness semantics, such as a generation timestamp and maximum acceptable age. This is optional lifecycle hardening, not a verified vulnerability introduced by this PR.

Pre-merge checks: 4

Passed checks: 4 passed

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly and concisely states the main change: pip-audit-bulk fails when a scan is incomplete.
Description check		The description directly explains the scan failure behavior, the --no-resolve change, and the validation performed.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

🟡 Minor · Add a timeout and handle FileNotFoundError for the osv-scanner call. · pip-audit-bulk:58-69

pip-audit-bulk:58-69
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Add a timeout and handle FileNotFoundError for the osv-scanner call.

subprocess.run has no timeout. A hung osv-scanner blocks the audit indefinitely. This is a likely failure mode when OSV is unreachable, which is the case this PR targets. A hang cannot reach the new failure path. Set a timeout (for example 300s). Catch subprocess.TimeoutExpired and record the formula as a failure. Pass check=False explicitly. A missing binary raises FileNotFoundError and aborts the whole run instead of recording a per-formula failure.

Proposed fix

             capture_output=True,
             text=True,
+            check=False,
+            timeout=300,
         )

Handle subprocess.TimeoutExpired where the result is consumed. Convert it to a failed result.

Source: Learnings

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 69030a02-ec43-485b-84b3-422474c74672

📥 Commits

Reviewing files that changed from the base of the PR and between fae3142 and 9f58b8c.

📒 Files selected for processing: 4

• .github/workflows/audit.yml
• README.md
• pip-audit-bulk
• tests/test_pip_audit_bulk.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

📜 Review details

⏰ Context from checks skipped due to timeout: 2

• GitHub Check: audit
• GitHub Check: workflow_syntax

🧰 Additional context used

📚 Code guidelines: 1

Generated by CodeRabbit — auto-discovered

📓 Path-based instructions: 3

For every checkout step, set `persist-credentials: false`.

📄 CodeRabbit inference engine (Generated by CodeRabbit)

Files:

• .github/workflows/audit.yml

Pin remote GitHub Actions to full commit SHAs and include a version comment; keep local reusable-workflow references as local paths.

📄 CodeRabbit inference engine (Generated by CodeRabbit)

Files:

• .github/workflows/audit.yml

Set workflow-level `permissions: {}` and grant only required permissions at the job level; reusable workflow callers should also keep their calling job's permissions restricted.

📄 CodeRabbit inference engine (Generated by CodeRabbit)

Files:

• .github/workflows/audit.yml

🪛 ast-grep (0.45.3)

tests/test_pip_audit_bulk.py

[info] 29-29: use jsonify instead of json.dumps for JSON output
Context: json.dumps(data)
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

---

[info] 41-41: use jsonify instead of json.dumps for JSON output
Context: json.dumps({formula: [f"{formula}==1.0"] for formula in results})
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

---

[error] 62-76: Command coming from incoming request
Context: subprocess.run(
[sys.executable, str(root / SCRIPT.name)],
cwd=root,
env={
**os.environ,
"PATH": str(bin_dir) + os.pathsep + os.environ["PATH"],
"CI": "1",
"GITHUB_STEP_SUMMARY": str(summary),
"QUICK_RUN": "false",
"SCANNER_RESULTS": json.dumps(results),
},
capture_output=True,
text=True,
timeout=10,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').

(subprocess-from-request)

---

[info] 71-71: use jsonify instead of json.dumps for JSON output
Context: json.dumps(results)
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

🪛 Ruff (0.16.8)

tests/test_pip_audit_bulk.py

[error] 62-62: subprocess call: check for execution of untrusted input

(S603)

🔇 Additional comments: 5

pip-audit-bulk: 2

84-91: LGTM!

---

133-144: LGTM!

tests/test_pip_audit_bulk.py: 1

1-165: LGTM!

README.md: 1

16-23: LGTM!

Also applies to: 47-48

.github/workflows/audit.yml: 1

27-29: LGTM!

[woodruffw]

The core pip-audit-bulk change itself LGTM, but I'd remove almost everything else in this PR personally 🙂