edge-cv-lab

A virtual edge computer-vision lab: reproduce and fix real-world video analytics failures with no cameras, no edge boxes and no GPU.

Video analytics systems (queue and dwell timing, traffic counting, footfall) tend to fail in the same few ways: objects get counted many times when they idle on a zone boundary, shadows drag boxes into the wrong lane, and events vanish when the uplink drops. These are usually debugged on site, on real hardware, slowly.

Results

Failure	Before	After	Study
Car idling on a zone boundary, ±6 px jitter (synthetic, truth: 1 visit)	naive counter: 66 enters	debounced state machine: 1	counting
Lane markings detected as cars (real intersection clip, 14 visits)	ByteTrack visit F1 0.528	30 px zone rule: F1 0.8	phantoms
Car box and truck box on one vehicle (per-class NMS)	110 ID switches	class-agnostic NMS: 11	counting
64 kbps uplink, 400 ms latency, 2 min outage (1388 events)	QoS 0: 462 lost	QoS 1: all 1388 exactly once	delivery

Every figure comes from a command in its case study, on checked-in fixtures or public CC-BY footage. Counts that people act on are only as good as the steps after the detector: tracking, zone logic and delivery. Each is measured here.

This repo shows they can be reproduced, measured and fixed on a laptop:

Piece	What it is	Needs
harness/	Offline replay harness. Saved tracks in, zone events out. Every failure mode is a deterministic test.	Python only
docker-compose.yml	The lab: looping RTSP "camera" → detector + tracker → throttled uplink → MQTT → Postgres	Docker
chaos/	One-line network impairment: 64–512 kbps uplinks, latency, outages	Docker
grafana/	Provisioned live dashboard: naive vs debounced counts, delivery lag, device health, dwell times	Docker
harness/replay/trackers/	Pluggable trackers on identical saved detections, plus a MOTChallenge bridge for any research repo. See docs/trackers.md	Python
tools/label.html	Single-file ground-truth labeller: draw a zone, tap keys while the clip plays, export JSON	A browser
harness/replay/viz.py	Comparison GIF, dwell heatmap, trajectories, per-track timeline	Python

The same zone logic module runs in both, so what passes offline is what ships to the simulated edge node.

Same detections, three trackers. The middle one hands each new car the previous car's ID, so six cars become one visit with a dwell timer that never resets. Details in docs/trackers.md.

How the hardware is simulated

Real thing	Stand-in	Fidelity
IP camera	A video file looped forever by ffmpeg into MediaMTX, served as rtsp://. The edge node cannot tell it from a real camera.	High for pipeline behaviour. Footage is whatever you supply.
Camera + model, when you have no footage	sim service: seeded synthetic vehicles with known ground truth, replayed in real time	Exact ground truth, schematic scene
Edge box	A memory-capped container running detector, tracker and zone logic	Pipeline and failure behaviour. Not NPU numerics or speed.
Cellular uplink	Toxiproxy between edge and broker: bandwidth caps, latency, outages	TCP level
Cloud	EMQX, Postgres, Grafana in containers	Same software, single node

Headline result

One synthetic car idles on a zone boundary for 10 seconds with ±6 px box jitter, then drives through. Ground truth: 1 visit.

Counter	Enter events	Error
Naive (centroid, flip on every crossing)	66	+6500%
Debounced state machine (footpoint + hysteresis + min dwell)	1	0%

Reproduce: cd harness && python -m replay.cli --tracks fixtures/boundary_jitter.jsonl --zone fixtures/zone.json --expected 1

The 24-vehicle traffic fixture (ground truth 17 visits) gives 205 naive vs 17 debounced (cd harness && python -m replay.cli --tracks fixtures/traffic.jsonl --zone fixtures/zone.json --expected 17). Lane B never enters the zone, yet shadow-stretched boxes make the naive counter log visits there:

Dwell heatmap	Trajectories

Every image in docs/img is generated from checked-in fixtures: pip install -e "harness[viz]" && make visuals && make trackers. replay.viz compare, heatmap and trajectories take --video clip.mp4 to draw over a real frame instead of the schematic road.

On real footage

A fixed CCTV camera over a signalised intersection, 106.6 s, 14 labelled visits in the zone. YOLOv8n with ByteTrack and the naive counter logs 78 visits, the debounced counter 39. On the same detections, the IoU tracker with a short buffer plus the debounced counter logs 21. No track re-enters the zone. 18 of ByteTrack's 39 debounced enters come from tracks that moved at most 6 px: white lane-marking dashes inside the zone that the detector scores as cars at low confidence, and that ByteTrack keeps because it tracks boxes down to score 0.1. Its 21 moving enters match all 14 visits with 7 false. The IoU tracker drops boxes under 0.3, which keeps most phantoms out; that, more than its tracking, is why it counts closer. Those runs, and the GIF below, use the detector's default per-class NMS, which can leave a car box and a truck box on one vehicle. With class-agnostic NMS, now the default here, ID switches against the dataset's annotated tracks fall from 110 to 11 for ByteTrack and from 368 to 68 for the short-buffer IoU tracker. Counts move less: ByteTrack's debounced count stays at 39, and the IoU tracker's goes from 21 to 26 as it stops missing 2 visits but adds 3 false ones. A zone rule that makes a track move 30 px before it can enter removes all 18 phantoms: ByteTrack's debounced count falls from 39 to 21 and visit F1 rises from 0.528 to 0.8, with the same 14 visits matched (docs/case-study-phantoms.md).

What to watch: each time a tracker loses a vehicle, the vehicle comes back under a new ID, the churn the counting study measures and each header tallies as IDs used; the bottom tracker, with the shorter buffer, loses vehicles sooner and runs through IDs faster, and closes fewer visits because its fragments are often too short for the debounced counter to keep as visits. Per-class NMS can also put two boxes with two IDs on one vehicle, so the changing tags are the failure being shown, not a rendering fault (docs/case-study-tracking.md).

The same window and crop, on class-agnostic NMS detections in both strips: on top the same short-buffer tracker, below ByteTrack at its defaults with the two fixes from the studies, the 30 px zone rule (named in the bottom header; docs/case-study-phantoms.md) and the contain090_same second-box filter (docs/case-study-straddles.md); rendered with make footage-after. What to watch below: IDs hold on most vehicles through the zone; the lane-marking dashes still get small boxes, but they are drawn grey and tagged held while the zone rule holds their enter back, so no visit opens (every track is held until it has moved 30 px, so a new vehicle shows grey for a moment and the parked car at the top left stays grey); and no vehicle carries a second tag, where the top strip puts one on the bus and on a van's cab. Where the bus passes behind the truck their two tags sit side by side, one per vehicle. The filter is included as the straddles study's winner, not because this window needs it: rendered with --filters none none the bottom header's totals are the same. IDs still change where ByteTrack loses a vehicle, as with the truck leaving at the right edge, and the header's IDs used, which counts every ID the tracker reported in the window, frames the GIF skips included, ends at 28 below against 69 above (both headers on the last frame).

Footage: MTID (Multi-View Traffic Intersection Dataset) infrastructure camera by M. B. Jensen, A. Møgelmose and T. B. Moeslund, Aalborg University, under CC BY 4.0. Frames re-encoded to video, cropped and resized, with tracking overlays added; rendered with make footage.

Full write-up with real footage: docs/case-study-tracking.md.

Quick start

Verified so far: CI runs the harness tests, checks fixtures regenerate byte-identical, and builds and import-checks the sim and ingest images. make up has been booted end to end: events reach Postgres and every Grafana panel fills. make drill has been run on a fresh lab: through a 64 kbps cap, 400 ms latency each way and a 2 minute outage, all 1388 events of the 5 scenes that finished arrived exactly once (naive-event delivery lag: median 0.86 s under latency, up to 120.05 s after the outage); with the device at QoS 0 (SIM_QOS=0) the same drill loses 462 of them. make broker-restart has been run on a fresh lab: with ingest stopped, the broker restarted and ingest kept away 60 s more, all 846 events of the 3 finished scenes arrived exactly once; without durable sessions (DURABLE_SESSIONS=false) 157 are lost. Write-up: docs/case-study-delivery.md. make up-video has been booted on a fixed CCTV intersection clip: edge-01 events reach Postgres and Grafana. The same clip, labelled, is scored offline in the case study. make visuals and make trackers are not run in CI.

Harness (no Docker):

pip install -e "harness[dev]"
make test

Full lab, no footage needed:

setopt interactive_comments 2>/dev/null || true
make up              # synthetic traffic -> throttled uplink -> MQTT -> Postgres -> Grafana
make counts          # enter/exit totals per counter
# opt-in: OCCUPANCY_GAUGE_MS=500 make up fills the "Vehicles in zone (gauge, opt-in)" panel (empty by default; docs/case-study-balance.md)

Grafana: http://localhost:3000 (no login; it queries Postgres through a role that reads the lab tables, no table writes; admin / lab to edit; GRAFANA_PORT=3001 make up if 3000 is taken). EMQX: http://localhost:18083 (admin / public, or EMQX_DASHBOARD_PASSWORD if set when the lab first starts, with no # in it; make down before changing it). Grafana, EMQX, Toxiproxy (8474) and Postgres (5432) are published on localhost only, which keeps other hosts out on Docker Engine 28.0.0 or later (older engines let hosts on the same network segment reach them; SECURITY.md); LAB_BIND=0.0.0.0 make up publishes them on every interface. A lab whose volume predates the current db/init.sql needs make down once: Postgres runs that script only on an empty volume.

Full lab with real video:

setopt interactive_comments 2>/dev/null || true
# put a fixed-camera traffic clip at media/sample.mp4 (see media/README.md)
make up-video        # RTSP loop -> YOLO + ByteTrack on CPU -> same pipeline
# opt-in: CONTAIN_SHARE=0.9 make up-video drops second boxes before tracking (docs/case-study-straddles.md)

Real camera:

CAMERA_URL='rtsp://camera.example/stream' make up-camera

MediaMTX pulls the camera over TCP and re-serves it as cam1, so the edge reads the same URL and Toxiproxy, ingest and the dashboard run as they do for a clip; the clip camera is not started, and make up-camera first removes one that make up-video left running (compose would leave it up, and it restart-loops against the pulled cam1). The switch is docker-compose.camera.yml, which only make up-camera loads; without CAMERA_URL compose stops with a message. Single-quote the URL, on the shell line and in .env, or a & or $ in it breaks the line or is expanded. Credentials go in the gitignored .env (CAMERA_URL='rtsp://user:pass@camera.example/stream'), not on the shell line (shell history keeps it), never in a compose file; docker compose config and docker inspect of the mediamtx container show them once set. make up-camera has not been run against a real camera: only its compose render and a MediaMTX-only start are checked. Draw the zone on that camera's frame in tools/label.html, from a few seconds of it saved under media/ (once up, ffmpeg -rtsp_transport tcp -i rtsp://localhost:8554/cam1 -t 10 -an -c copy media/cam.mp4; an H.265 camera's mp4 may not play there, so if it does not, re-encode with -c:v libx264 in place of -c copy), then rerun the same line with ZONE_POLYGON='<polygon from zone.json>'; until then the edge uses the default zone, which belongs to media/sample.mp4. Add RTSP_BIND=127.0.0.1 to keep MediaMTX, and the camera's picture, off other hosts. A second camera would be cam2 (MTX_PATHS_CAM2_SOURCE) and a second edge service reading it with its own DEVICE_ID; that is not wired up here.

Your own clip, end to end:

setopt interactive_comments 2>/dev/null || true
open tools/label.html          # any browser: draw the zone, tap E/X while watching at 2x, save zone.json + truth.json
pip install -e "harness[video]"
cd harness
python scripts/dump_tracks.py --video ../media/clip.mp4 --tracker bytetrack.yaml --out runs/bytetrack.jsonl
python -m replay.score --tracks runs/bytetrack.jsonl --zone zone.json --truth truth.json

score matches each predicted visit to a labelled one in time, so a miss and a double count cannot cancel out. Log every clip in media/SOURCES.md.

Break the network:

setopt interactive_comments 2>/dev/null || true
chaos/scenarios.sh 64k        # 64 kbps uplink
chaos/scenarios.sh outage 120 # 2 minute disconnect
chaos/scenarios.sh reset
make delivery                 # did every event arrive exactly once? each finished scene vs its offline replay
make drill                    # all of it on a fresh lab, ~9 min, with a delivery-lag table per phase
make broker-restart           # fresh lab, ~5 min: ingest away across a broker restart, then the same check
# before runs: SIM_QOS=0 make up (fire-and-forget events), DURABLE_SESSIONS=false make up (no durable sessions)
# Grafana: events flatline, then a catch-up spike with high delivery lag

Failure modes covered

Failure	Cause	Fix demonstrated	Fixture
Track vanishes inside the zone, visit never ends	Counter only reacts to boxes it receives	lost_ms timeout closes the visit at the last-seen time	queue.dets.jsonl
Boundary jitter inflates counts	State flips on every frame the anchor crosses the edge	N-frame and spatial hysteresis, minimum dwell, cooldown	boundary_jitter.jsonl
Shadow pulls box into adjacent lane	Centroid moves when the box stretches	Bottom-centre footpoint anchor	shadow_expansion.jsonl
Events lost on disconnect	Fire-and-forget publish	QoS 1, persistent session, bounded local queue	lab + outage
Events lost when the broker restarts	Subscriber session kept only in broker memory	EMQX durable sessions on disk	lab + make broker-restart
Double counting after reconnect	At-least-once redelivery	ULID event_id + ON CONFLICT DO NOTHING	lab + outage
Next car inherits the previous car's ID, merging visits	IoU association plus a track left parked where a vehicle vanished	Ground-contact association with a lane-shaped gate and damped coasting	queue.dets.jsonl
Zone occupancy never returns to zero, and a live "in zone" sum never shows a vehicle inside	Visits whose track is lost are never closed; the debounced counter holds each enter until its visit closes	Close a visit 3 s after its track goes silent (lost_ms, on by default); read occupancy as a gauge of visits seen in the last 500 ms, with the zone rule (opt-in: OCCUPANCY_GAUGE_MS=500, MIN_TRAVEL_PX=30)	real clip, replay.balance; docs/case-study-balance.md
Lane marking inside the zone counted as a visit	Detector scores it as a car at low confidence; ByteTrack keeps low-score boxes	Zone rule: a track must move min_travel_px before it can enter (off by default)	real clip, replay.phantoms; test_zones.py
One vehicle counted twice	A second detection on it (a duplicate, or a box on part of a bus) starts a second track	Drop a detection mostly inside a higher-scoring one of the same class, before tracking (opt-in in the edge: CONTAIN_SHARE=0.9, off by default; its effect on enters measured in the replay harness; live on the annotated clip, the on run logged fewer enters than both off runs, by more than they differed)	annotated clip, replay.secondbox; docs/case-study-straddles.md
"Online" device that is actually dead	Health inferred from network reachability	App-level heartbeat with FPS, plus MQTT last-will	lab

Roadmap: published trackers (FastTracker, UCMCTrack, TrackTrack) on real footage through the MOT bridge, scored with TrackEval; broker drain test on kind + EMQX Operator.

Design notes

• Tracks, not video, are the test input. Detection and tracking run once; zone logic then replays in milliseconds. This is what makes a regression suite practical.
• Integer millisecond timestamps, one ULID per event. Float timestamps and timestamp-derived filenames break exact matching between services.
• Fail loud. A malformed record raises with file and line number. Nothing is skipped silently.
• Fixtures are generated, seeded and checked in CI, so results are reproducible by anyone.

More in docs/architecture.md.

Limitations

• Synthetic fixtures exercise the logic and reproduce specific failure modes; they say nothing about real-world accuracy. The case study uses public footage for that.
• The edge node runs YOLO on CPU. It models pipeline behaviour, not accelerator (NPU) numerics or latency.
• Toxiproxy shapes TCP only. It does not model packet loss or radio behaviour of a real cellular link.
• The sim scene hands the counter perfect track IDs, so it tests zone logic, not tracking. Tracker comparisons run on the queue fixture's raw detections instead.
• For make up-video, tune mem_limit for your clip, and pass the clip and its zone: VIDEO=<file under media/, no spaces> ZONE_POLYGON='[[x,y],...]' make up-video (defaults: sample.mp4 and its zone).

Using this commercially

The code's Apache-2.0 licence allows commercial use on the terms under Licence, but third-party components keep their own licences, listed with where each was read in THIRD_PARTY.md. ultralytics, which the edge service and the optional ByteTrack adapter import, and its yolov8n.pt weights are AGPL-3.0, as is the grafana/grafana-oss image; the replay harness core and the sim and ingest services import no AGPL code. Distributing the edge image conveys ultralytics under AGPL-3.0; whether the edge program as a whole then falls under AGPL-3.0 is a legal reading the licence text does not settle, and the vendor offers an Enterprise licence for commercial use, so take advice before shipping it. The edge image also carries LGPL and GPL libraries, among them Qt 5 (LGPL-3.0), which its OpenCV build loads, and Debian's ffmpeg with GPL-2.0-or-later and GPL-3 libraries, whose terms apply when the image is distributed. The case studies under docs/ and the research datasets behind some of their figures are non-commercial, and the lab is not hardened: SECURITY.md lists the defaults to replace before any deployment.

Provenance

An independent personal project, written from scratch. Every figure comes from public datasets used under their own licences (docs/datasets.md) or from synthetic fixtures the harness generates. Not affiliated with or endorsed by any company.

Licence

Code: Apache License 2.0. Keep the LICENSE and NOTICE files with any copy or derivative, and the SPDX header at the top of each Python file; the notice names the author and this repository. The container images and any wheel built from harness/ do not include LICENSE or NOTICE, so add both if you distribute one. Documents under docs/ (the case studies): CC BY-NC-SA 4.0, attribution required, non-commercial, share alike; commercial use of the write-ups needs permission. Footage and datasets keep their own licences (media/SOURCES.md).

To cite or credit this work use the CITATION.cff (GitHub's "Cite this repository"), or: edge-cv-lab, Belmark Ray Nalugon, https://github.com/blmrk/edge-cv-lab.