M2iOR – Data Checking Pipeline

Pipeline that takes the Excel files of scientific studies (olfactory receptors, molecules, responses) as input and produces checked and enriched CSV files for the M2iOR website.

flowchart TD
    Root["M2iOR Pipeline"]

    Root --> Ingestion["Ingestion & validation<br/>Excel reading, structure check, cleanup"]
    Root --> Receptors["Receptors<br/>UniProt/NCBI identification, BLAST, mutations"]
    Root --> Molecules["Molecules<br/>PubChem enrichment, stereochemistry, images"]
    Root --> Responses["Experimental responses<br/>validation of measured values"]
    Root --> Sources["Bibliographic sources<br/>DOI validation, APA reference"]
    Root --> Export["Final export<br/>CSV + images for the website"]
    Root --> Cache["Cache<br/>avoids re-querying the same APIs"]

Loading

Full file-by-file breakdown: docs/PIPELINE.md

Getting started

Prerequisites

• Python >= 3.14
• uv — the package/dependency manager this project uses

Install uv (one-time, see the official guide for alternatives):

curl -LsSf https://astral.sh/uv/install.sh | sh

Installation

From the project root:

uv sync

This creates the .venv and installs every dependency pinned in uv.lock.

Receptor mutation analysis (scripts/find_protein_mutations.py) and the BLAST fallback (scripts/fetch_blast.py) require a local BLAST installation:

./scripts/install_blast.sh          # Linux/macOS

.\scripts\install_blast.ps1          # Windows (PowerShell)

Both fetch NCBI's official prebuilt blastp binary and drop it into the venv (.venv/bin / .venv\Scripts) — no distro/system package, no admin rights. Re-run after any uv sync that recreates .venv from scratch.

Running the pipeline

All commands are run through uv run main.py, so dependencies are resolved automatically without activating the virtual environment manually.

uv run main.py --status          # Show the processing status of every known study
uv run main.py --scan            # List Excel files, flag new/modified ones, and choose which to process
uv run main.py --pending         # Process every study still pending
uv run main.py --retry           # Retry only studies that previously failed
uv run main.py --all             # Reprocess every study, ignoring DONE/FAILED status
uv run main.py --file <path>     # Process a single Excel file
uv run main.py --force           # Reprocess every study and force BLAST re-queries
uv run main.py --force <path>    # Reprocess a single file and force BLAST re-queries
uv run main.py --check <path>    # Validate a single file without exporting anything, report every issue

Check mode and reports

--check runs every validation stage on one Excel file and reports all the issues it finds, without writing any CSV, image or registry entry (external API results are still cached). Its exit code is 0 when nothing blocks the export, 1 on validation errors, 2 on a pipeline crash or a usage error.

uv run main.py --check input/X.xlsx                                  # human-readable report
uv run main.py --check input/X.xlsx --report json                    # JSON report on stdout (logs on stderr)
uv run main.py --check input/X.xlsx --report json --report-file r.json
uv run main.py --file input/X.xlsx --blast cached --report json      # export, non-interactive, with its report

Each issue gives its severity, stage, code, sheet, Excel column/rows/cells, value and, when possible, a suggested value. Only error issues block the export; warning (needs a human look, e.g. rows the Laravel import will skip), auto_fix (a value the pipeline silently rewrites, e.g. a receptor name or a CID/CAS it corrects) and info never change what the pipeline accepts.

--blast controls pending BLAST lookups: ask prompts (default, except for --check), cached never runs one and only applies cached results (default for --check), run runs every new or retryable lookup without asking.

Molecule lookup

scripts/tools/lookup_molecule.py looks up a molecule by name, CID, CAS, SMILES or InChIKey with the pipeline's own sources, cache and rules, and diagnoses an Excel row's Name/CID/CAS consistency: how the pipeline treats the row, what each identifier actually is, and candidate corrections, each re-checked with the pipeline's rules. It never modifies the Excel file. Run it from the project root; it prints JSON.

uv run python -m scripts.tools.lookup_molecule --name "4-Methylcyclohexanol" --cas 25639-42-3
uv run python -m scripts.tools.lookup_molecule --excel input/X.xlsx --row 8 --row 17
uv run python -m scripts.tools.lookup_molecule --report r.json    # every molecule issue of a --check JSON report

Note: the input directory (Excel studies) is input/, and the output directory (M2iOR website data) is resolved as a sibling of this repository (../M2iOR_web_public-main). Both are set at the top of main.py (input_path, M2IOR_WEB_PUBLIC_PATH) — update them if your layout differs.