The mistake that defines most Raspberry Pi cloud-logging projects is treating the cloud as the primary destination. It isn't. The network between your Pi and the cloud will fail — regularly and unapologetically — and any logger that writes straight to a remote endpoint loses data every time it does.
The pattern that survives the field is offline-first: log every reading locally the moment it's taken, then let a separate loop ship the backlog upward whenever connectivity allows. Everything else in this guide — transport choice, cloud storage, the Arduino question — hangs off that one inversion.
These recommendations come from production practice, not a weekend build. The same buffering-and-ingestion discipline described here is what lets AetherMesh, my asset-tracking platform, ingest live telemetry from 1,000+ tagged assets without losing readings to flaky site networks.
What does a reliable logging architecture look like?
Five stages, strictly separated — and the separation between capture and ship is the entire trick:
SQLite is the ideal local buffer: transactional, queryable, and far more resilient to power cuts than an append-only text file. When the network is up, readings flow with a few seconds' delay. When it's down for an hour — or a week — capture continues untouched, and the ship loop drains the backlog on reconnect. No special cases, no data gaps, no 3 a.m. heroics.
import json, sqlite3, time, threading, requests
db = sqlite3.connect("readings.db", check_same_thread=False)
db.execute("""CREATE TABLE IF NOT EXISTS readings
(id INTEGER PRIMARY KEY, recorded_at REAL, payload TEXT, sent INTEGER DEFAULT 0)""")
def capture():
while True:
reading = {"deviceId": "pi-field-07", "recordedAt": time.time(),
"temperatureC": read_temperature()}
with db:
db.execute("INSERT INTO readings (recorded_at, payload) VALUES (?, ?)",
(reading["recordedAt"], json.dumps(reading)))
time.sleep(60)
def ship():
while True:
rows = db.execute(
"SELECT id, payload FROM readings WHERE sent = 0 LIMIT 500").fetchall()
if rows:
try:
r = requests.post("https://api.example.com/ingest",
json=[json.loads(p) for _, p in rows], timeout=15)
r.raise_for_status()
with db:
db.execute(f"UPDATE readings SET sent = 1 WHERE id IN "
f"({','.join(str(i) for i, _ in rows)})")
except requests.RequestException:
pass # network down — capture keeps running, retry next pass
time.sleep(30)
threading.Thread(target=capture, daemon=True).start()
ship()Note what the snippet refuses to do: it never drops a reading because the POST failed, it never blocks capture on the network, and it batches uploads instead of sending one row at a time. The recordedAt timestamp travels inside the payload, because the server must trust device time, not arrival time — a backlog flushed after an outage would otherwise land as an hour of readings that all appear to have happened at once.
Should you ship readings over HTTPS or MQTT?
Batched HTTPS, as in the snippet, is the right default for pure logging: periodic readings, minutes of acceptable latency, one direction of travel. It needs no broker, traverses any network that can reach a website, and the batch-and-acknowledge loop is trivial to reason about.
Choose MQTT instead when the requirements grow teeth — many devices, live dashboards that want readings in seconds, or commands flowing back down to the device. A broker handles fan-in and fan-out at a scale polling never will, and quality-of-service levels replace your hand-rolled retry logic with protocol guarantees.
The honest sequencing for most projects: start with HTTPS batches, and move to MQTT when — if — the live requirement arrives. The local buffer pattern transfers unchanged; only the ship loop's transport swaps. I've published both halves of that upgrade path: the MQTT broker guide covers topic design, QoS, and security, and the Raspberry Pi + Spring Boot guide shows the streaming architecture end to end.
Where should the data land in the cloud?
Three realistic options, in ascending order of ownership:
- 1.A managed IoT platform — The cloud vendors' IoT suites get you ingestion and dashboards fastest — at the price of per-device/per-message billing and your data model living inside someone else's product. Fine for evaluation; increasingly expensive and constraining as fleets and years accumulate.
- 2.A hosted time-series service — The right fit when your need is genuinely just metrics and graphs, nothing more.
- 3.Your own ingestion service and database — My default for anything that will live for years: a Spring Boot endpoint validating and persisting into PostgreSQL, time-series-indexed, on a modest VM or managed database. Boring, flat-cost, and fully owned.
The decision hinge is ownership over time. Logging data compounds in value — trend analysis, compliance evidence, model training — and its gravity makes late migrations painful. If the readings matter beyond this quarter, put them in a database you control from the first week. The ingestion endpoint is a day of work, against the decades your data may outlive any vendor's pricing model.
What about an Arduino-to-cloud data pipeline?
Same architecture, different placement of the buffer. A classic Arduino has no realistic TLS stack and little storage, so it should never talk to the cloud directly. Pair it with a Raspberry Pi acting as gateway: the Arduino reads sensors and forwards over serial or radio, and the Pi runs exactly the capture-and-ship pattern above.
This is the right shape whenever sensors are electrically awkward, need microcontroller-grade timing, or sit at low-power positions a Pi can't occupy. ESP32-class boards blur the line — they do TLS and MQTT natively, so at hobby scale they can ship straight to a broker. But their buffering is a fraction of a Pi's, so for data you can't afford to lose, the gateway pattern with its transactional local store remains the trustworthy choice. The Pi becomes the site's one honest narrator, aggregating however many microcontrollers the deployment needs.
How do you turn the logged data into dashboards and alerts?
Once readings land in your own database, visualization is the easy layer — and the discipline is resisting the urge to make it the first layer. For internal monitoring, an off-the-shelf dashboard tool pointed at your time-series tables gets you charts and threshold alerts in an afternoon. For many logging projects that's genuinely enough.
The step up is a custom view when the audience isn't engineers: an operations screen showing the five numbers someone actually watches, or a customer-facing page scoped to their own devices. At that point you're building a small web application over the same database, with authentication and roles deciding who sees which streams.
Alerting deserves more design than it usually gets. Threshold alerts on raw readings are noisy in exactly the conditions that matter — a sensor bouncing across a boundary fires dozens of notifications for one event. Production alerting evaluates rules on validated data, applies hysteresis or time-windows, and — critically — alerts on absence too: a device that has stopped reporting is usually a more urgent signal than any value it ever sent. That last rule has paid for itself on every fleet and asset deployment I've run.
What fails in the field?
The same short list, on every deployment:
- SD cards wear out — A Pi logging aggressively to its boot card will kill it. Log to an external SSD or a high-endurance card, batch the writes, and treat the card as a consumable with a replacement schedule.
- Clocks drift — A Pi without a network loses wall-clock time across power cuts, and every buffered reading inherits the error. Sync NTP on boot, fit a battery-backed RTC on offline devices, and have the backend sanity-check inbound timestamps.
- Disks fill — An unbounded buffer eventually eats the card during a long outage. Cap it, and decide deliberately whether the oldest or newest readings win when it overflows.
- Power cuts corrupt mid-write — SQLite's transactions are your defense — plain file appends corrupt.
- TLS certificates expire — Un-renewed, a year after deployment, silently. Automate the renewal on day one.
- Retention goes unscoped — A reading every thirty seconds is over a million rows per device per year. Decide early what stays raw, what gets downsampled, and what ages out.
None of these are exotic. They're the standard tax on leaving the bench — and handling them at design time is cheap.
When logging becomes a platform
One Pi logging to the cloud is a weekend of careful work. The step-change comes with fleets, tenants, alerting, and integrations — that's when the ingestion path needs real engineering, and it's the exact scope of the proof-of-concept engagement described on my IoT services page.