Nursing Notes LLM Audit

This project evaluates nursing documentation notes against quality criteria using an LLM, then compares predictions to labelled ground truth (Met/Unmet).

Project Structure

• experiments.ipynb: main notebook to run the full workflow.
• utils.py: helper functions for prompt creation, model calls, parsing, scoring, and reporting tables.
• .env: environment variables (for example OPENAI_API_KEY).
• requirements.txt: Python dependencies.

What the Workflow Does

1. Load labeled data from data.xlsx.
2. Select model provider (openai or hf).
3. Build a gold-standard table from the labeled columns.
4. Evaluate each target note against quality criteria.
5. Compute overall, per-note, and per-criterion accuracy.
6. Display compact and detailed prediction tables.

Data Format

Place a data.xlsx file in the project root with:

• id: unique note identifier.
• note: note text.
• One or more additional columns, each representing a quality criterion with values:
  • Met
  • Unmet

Example schema:

id	note	Criterion A	Criterion B
1	...	Met	Unmet
2	...	Unmet	Met

Setup

python -m venv .venv
source .venv/Scripts/activate  # On Windows Git Bash
pip install -r requirements.txt

Create a .env file:

OPENAI_API_KEY=your_api_key_here

Run

Start Jupyter and open the notebook:

jupyter notebook

Then run cells in experiment.ipynb from top to bottom.

Configuration

In experiment.ipynb:

• provider = 'openai' to use OpenAI API.
• provider = 'hf' to use Hugging Face transformers pipeline.
• model controls which model is used for either provider.
• example_note_ids controls which notes are used as in-context examples.

Output

The notebook produces:

• Evaluation summary (overall and breakdown metrics).
• predictions_df: compact predictions table.
• detailed_df: side-by-side predicted vs gold labels.

Notes

• Hugging Face models may require significant RAM/VRAM depending on selected model size.
• If Excel loading fails, verify openpyxl is installed and data.xlsx exists in the root directory.