The Robot Data Pipeline: A Unified Sensor-to-Cloud Recording and Upload Strategy for Navigation and Perception

A modern robot generates far more data than it can ever upload. A single sensor-heavy platform produces gigabytes per hour to terabytes per day, while its uplink, dock WiFi or cellular, is orders of magnitude smaller. The job of a sensor-to-cloud strategy is therefore not to move everything, but to record the right things losslessly on the edge, unify navigation and perception in one time-aligned log, curate the critical one percent, and offload it efficiently. This reference walks the whole path, from the sensor to the cloud, and names the tools and the numbers at each step.

The data rates set the entire architecture. Per-sensor figures compiled from automotive sources put a single camera at roughly 500 to 3,500 Mbit/s, a LiDAR at 20 to 100 Mbit/s, radar at 0.1 to 15 Mbit/s, and IMU, GNSS, and odometry each below 0.1 Mbit/s. Aggregated, a lower-autonomy vehicle runs around 3 Gbit/s, about 1.4 TB per hour, and a higher-autonomy one near 40 Gbit/s, about 19 TB per hour. Industry commonly repeats a figure of 20 to 40 TB per day per vehicle. These are order-of-magnitude estimates, but the conclusion is robust.

The dominant cost is uncompressed video. A raw 1080p30 camera stream is about 1,500 Mbit/s, roughly 670 GB per hour; the same stream as H.265 is roughly 1 to 2 GB per hour, a hundred to a thousand times smaller. No uplink sustains multi-gigabit rates continuously, and cellular adds caps and per-gigabyte cost, so the practical rule is to reserve the live uplink for low-rate telemetry and teleoperation, and to move bulk data another way.

Since ROS 2 Iron in May 2023, the default rosbag2 format is MCAP, an append-only, serialization-agnostic container that embeds its own message schemas so a file stays readable after the code that produced it has changed. It records ROS 1, ROS 2, Protobuf, and JSON in one file, compresses per chunk with lz4 (fast) or zstd (better ratio), and keeps an index so readers can seek by time and topic without decompressing everything. Crucially, its chunked layout is crash-resilient: a power loss mid-write loses only the in-flight chunk, not the whole file, which is why it is preferred over the older SQLite format on robots. Foxglove's guidance is to size chunks at a megabyte or more and to benchmark on the target hardware.

Recording losslessly also depends on QoS. A recorder is just another subscriber, so if a publisher offers best-effort while the recorder requests reliable, they never connect and nothing is captured. rosbag2 adapts requested QoS per topic and exposes a QoS override file so you can force reliable delivery, sufficient history depth, and durability for lossless capture. A known discovery race can still prevent recording of non-default-QoS topics, which explicit overrides mitigate.

Treat data as two classes with opposite economics. Lightweight topics, IMU, GNSS, wheel odometry, the TF transform tree, and velocity commands, are each below a megabit per second and are essential for reconstructing and replaying everything else; keep them, effectively always. Heavyweight topics, camera images and LiDAR point clouds, are the entire storage and bandwidth budget; record cameras encoded as H.264 or H.265 rather than raw, and be selective about when full-rate point clouds are kept.

Edge hardware sets the ceiling. A SATA SSD tops out near 600 MB/s and a single raw 1080p60 camera already consumes most of that, so sensor-heavy rigs need NVMe, often in RAID, plus encoding. For continuous capture, sustained-write behavior matters more than peak: write amplification, garbage collection, and thermal throttling cause latency spikes that back-pressure the recorder and drop messages, and continuous logging wears NAND far faster than office workloads, so drive replacement belongs in fleet planning.

Three strategies exist on a spectrum. Recording all topics is simplest and highest-fidelity but hits the terabyte-per-day rates above. Selective recording, by topic list or regex, is the standard lever: drop raw camera, keep detections, TF, and odometry. The most economical pattern is event-triggered recording. rosbag2 supports a snapshot mode backed by a bounded in-memory ring buffer that continuously overwrites the oldest messages and writes nothing to disk until a trigger, exposed as a snapshot service, flushes the buffer to a bag.

The design intent is to capture the seconds before and around an event, a safety intervention, an anomaly, a disengagement, a collision, without paying to store steady-state operation. In production fleets the triggers are specific: multimodal sensor disagreement, low detection confidence, and planner overrides or recovery behaviors. That is the heart of the strategy, keep only the interesting moments, at high fidelity, and let the rest scroll past.

Navigation and perception must be captured together, in the same time-aligned recording, because a navigation failure usually has a perception root cause. Navigation, in a Nav2 stack, produces odometry, the TF tree, AMCL localization, global and local costmaps, planner and controller outputs, velocity commands, behavior-tree ticks, and recovery behaviors like spin and back-up. The behavior-tree state is what tells you why the robot decided what it did, and a recovery behavior firing is itself a strong signal that something went wrong.

Perception produces LiDAR point clouds, camera images and their intrinsics, depth, object detections, and segmentation masks. When a robot stops or swerves, the planner's behavior is the symptom and a perception disagreement is often the cause; you can only diagnose that if the navigation decisions and the perception inputs sit in one recording you can replay deterministically. The same unified log is the substrate for building labeled multimodal training sets and for fleet-wide analytics.

If sensor timestamps disagree while the robot moves, fused detections land in the wrong place, TF lookups interpolate to the wrong pose, and any dataset built from the recording carries baked-in spatial error that corrupts both debugging and training. The gold standard is hardware timestamping with the Precision Time Protocol, which gives sub-microsecond alignment when every device in the path supports it. Where hardware sync is imperfect, ROS aligns messages by header stamp using message_filters, whose ApproximateTime policy matches messages within a tolerance set by a slop parameter and a queue size.

For replay, running nodes with simulated time against the recorded clock keeps everything referenced to bag time rather than wall time; mixing wall-clock sources with bag time is a classic source of mismatch. Synchronization is not a nicety here, it is the precondition for a recording being reproducible at all.

Because upload is the bottleneck, the decisive work happens before data leaves the robot. The ring buffer plus trigger pattern is the first filter. On top of it, edge tooling filters and downsamples so only needed topics and interesting segments go up: platforms like Roboto support downsampling for fast uploads, and time-series stores like ReductStore filter by label at the edge so only records matching a condition, an AI-inference flag or a status bit, replicate to the cloud. Active-learning curation goes further, selecting only the highest-value samples; one documented result trained a detector on about 10,000 selected images out of 90,000 while hitting target performance.

The autonomous-driving canonical version is shadow mode: run a model passively and upload the segments where its prediction diverges from the human or the deployed policy, feeding a data flywheel. Foxglove frames the goal plainly, finding the critical one percent that drives improvement. Everything about the pipeline serves that ratio.

The bandwidth hierarchy is wired or physical offload at the dock first, dock WiFi second, cellular last and reserved for telemetry and teleoperation. The dominant pattern is deferred bulk offload: record locally during operation and upload opportunistically when docked or on WiFi, ideally during idle periods like charging. An on-robot agent that watches for new recordings and uploads them automatically, as Foxglove's agent does, removes the manual step for a fleet.

Two mechanics matter. Split recordings into atomic files, commonly about a minute of time or roughly a gigabyte each, so uploads and deletes are parallelizable and a failure does not lose a monolith. And use resumable, chunked transfer: S3 multipart upload splits a file into 5 MB to 5 GB parts, retries only failed parts, and enables pause and resume, though incomplete uploads keep accruing storage charges until a lifecycle rule aborts them. Cost discipline means egress-tier awareness (S3 internet egress starts near 0.09 dollars per gigabyte), retention policies, and batching small records to cut request overhead.

Data lands in object storage, increasingly under a bring-your-own-storage model that keeps it in the customer's own cloud for residency while a platform provides managed indexing and query. From there the value is search and triage: Foxglove launched a unified data search and curation platform in April 2026 to query multimodal data without moving it to a separate warehouse and to turn events into repeatable datasets; Roboto runs ingestion actions and triggers that index and process logs automatically on upload. The loop closes as a flywheel, mine edge cases from recordings, relabel with model-assisted annotation and human review, retrain, validate the new model in shadow mode, and stage the rollout with rollback thresholds.

A note on the tool landscape: AWS RoboMaker reached end of support in September 2025, so current AWS pipelines use IoT Greengrass v2 at the edge into S3 and Kinesis, not RoboMaker. Many performance and cost multipliers quoted by data-platform vendors are their own benchmarks; treat them as directional.

Camera recordings carry visual PII, primarily faces and license plates, and anonymizing them is what makes retention and sharing compliant with GDPR and CCPA. The pattern is automated pre-upload redaction: a detection model locates sensitive regions per frame and blurs them, or replaces them with realistic synthetic stand-ins where the person or vehicle is the object of interest, all without a human viewing the raw footage, which is itself a privacy control. Processing in the customer's own cloud or on-premise, with audit trails, supports data sovereignty, and the ring-buffer pattern doubles as a retention control since only recent history and triggered segments are ever kept.

Put together, the path is a single pipeline with a return loop. The diagram summarizes it: record locally in an indexed, resilient MCAP file; curate at the edge with triggers, ring buffers, label filters, and active-learning selection; enforce volume-based retention so the edge disk never overflows; defer bulk upload to the dock as atomic, resumable files; land the data in object storage under bring-your-own-storage; index and search it at fleet scale; and feed the mined edge cases back into retraining and redeployment.

This pipeline is usually described for a robot recording its own operation, but the same discipline governs any capture of physical data, including the human demonstrations that train manipulation policies. The hard parts are identical: record multiple high-rate streams losslessly, hardware-synchronize them to better than a frame, curate the segments worth keeping, and move only those to the cloud. A capture stream that is not time-aligned or not losslessly recorded is not training data, it is noise with timestamps.

Blomega Lab treats capture as exactly this kind of pipeline. Fingertip force and finger bend are recorded and hardware-synced to POV video at 100 Hz, curated per verified hour, and delivered in the formats training stacks already read. The sensor-to-cloud problem and the human-skill-data problem are the same engineering problem viewed from two ends.