Read logs and inspect the pipeline
Use logs for activity, health snapshots for freshness, and receiver tools for actual messages. They answer different questions.
See every stage
Console logs use UTC time, severity, service name, stage, and labeled fields. INPUT reports capture/receive activity; PROCESS reports work; OUTPUT reports publishing or delivery; LIFECYCLE reports service state.
docker compose -f config/.generated/pipeline.compose.yaml logs -f --tail 20 --no-log-prefix
docker stats --no-streamFollow inference into tracking
Compare per-camera inference output with tracker input and processing. Rates count frame batches, not the number of people. A tracker processing at the inference rate is keeping up even if the camera captures faster.
docker compose -f config/.generated/pipeline.compose.yaml logs -f --tail 30 --no-log-prefix visionengine-inference-service-people visionengine-tracking-service-entranceUnderstand queues and drops
Bounded queues protect latency and memory. With drop-oldest, a full queue discards an older waiting frame so newer work can enter. Dropped frames are not inferred and do not produce detections. Queue size, age limits, processing time, and CPU/GPU load matter together.
Health is not the data stream
Services write atomic health snapshots under /health in the generated deployment. Monitor checks their freshness and publishes device metadata; row builder turns that into nodes rows. Monitor does not collect every log line or prove detection accuracy.
Inspect actual detections
Run a diagnostic subscriber inside the inference container. It does not take messages away from the tracker. Use the generated instance name for your model and camera.
docker compose -f config/.generated/pipeline.compose.yaml exec visionengine-inference-service-people python -m services.inference.tools.receiver --connect tcp://127.0.0.1:5556 --camera-id entrance --count 5 --format json