Cut Your Data Usage in Half with Notefile Templates

Here's a note.add request that, to me, looks completely reasonable:

{
  "req": "note.add",
  "file": "readings.qo",
  "body": {
    "level_cm": 184.62,
    "temp_c": 21.4,
    "battery_v": 3.87,
    "pump_running": true,
    "pump_cycles": 47,
    "status": "ok"
  }
}

The body of this event is 101 bytes of JSON, but only 24 of those bytes are values you care about. The other 77 are field names, quotes, colons, commas, and braces. Packed down to a fixed binary structure, those same six values only need about 11 bytes plus the status string.

In this scenario, you're shipping the literal characters "pump_running" on every single outbound Note, forever. At a reading every 15 minutes, that's roughly 35,000 copies per device per year. On a cellular Notecard with 500MB of bundled data, that may or may not be a cost you can absorb. On satellite, where Starnote for Iridium data runs $0.00075 per byte with a 10-byte minimum billable packet, it starts to present an issue.

Blues Notecard has a built-in answer to this, and it's one request: the note.template API.

Why Notefiles Are Schemaless to Begin With

Notecard's default is deliberately permissive. Individual Notes in a Notefile share no structure at all, so you can add fields, drop fields, and mix completely different shapes in one file. That's genuinely useful during development, but it has two costs:

1. The first is byte overhead. Because Notecard can't know what's coming, it stores and forwards whatever you hand it, keys and all.

2. The second is the schemaless Notefile system is memory-based and designed to hold no more than 100 unsynced Notes per Notefile. If you queue the 101st Note before a sync completes, you get an error. That's fine for a device that reports once an hour and syncs hourly. It is not fine for one that logs a burst of readings during an event, or purposefully sits offline for days.

Templates fix both problems with the same mechanism!

What note.template Actually Does

You register a template against a .qo or .qos Notefile by describing its shape. The body you send isn't data, it's a set of "hints", where each key is a field name you promise to send, and each value tells Notecard the type and width to reserve for it.

{
  "req": "note.template",
  "file": "readings.qo",
  "body": {
    "level_cm": 14.1,
    "temp_c": 12.1,
    "battery_v": 12.1,
    "pump_running": true,
    "pump_cycles": 22,
    "status": "-"
  }
}

Those magic numbers are worth bookmarking:

With the template registered, Notecard stops storing JSON objects and starts writing fixed-length binary records to flash. Notehub reverses the process on arrival, so your routes and your cloud code still receive ordinary JSON events. The effect on storage and sync capacity as an order-of-magnitude improvement, and the 100 Note limit no longer applies at all.

Better yet, Notecard will tell you exactly what you will be sending, byte-wise. The response to note.template includes a bytes field: the per-Note size that will be transmitted to Notehub, before compression. For example:

{
  "bytes": 40
}

Your own number will differ, since it depends on your field widths and, for standard (non-compact) templates, the automatic metadata covered below.

tip

Treat that bytes value as the number to optimize against. Widen a field from 12.1 to 18.1 and you can watch the cost of the change immediately, before you've deployed anything. You can also read back a live template at any time with {"req":"note.template","file":"readings.qo","verify":true}.

Adding Notes afterward with a note.add request looks exactly like it did before. The original API request from the top of this post works unchanged, and Notecard returns {"template":true} to confirm the Note was stored against the provided schema.

There's one behavior change worth knowing: templated Notefiles validate every Note on the way in. If you send a float where you promised an integer, you'll get an error:

{
  "err": "error adding note: integer expected because of template"
}

Where the Savings Actually Live

We publish measured data usage estimates for a cellular Notecard using a two-field Note body:

Some interesting findings from this table alone:

1. At one Note per sync, templates buy you almost nothing. A single sync costs about 1.2 KB whether or not there's a template, because you're paying for the session rather than the payload.

2. At 100 Notes per sync, the picture inverts. 3.5 KB becomes 1.6 KB with a standard template, and 1.2 KB with a compact one. That's a 54% and a 66% reduction, respectively.

3. The bottom row is intriguing because syncing 5,000 Notes in one session simply isn't possible without a template, because you'd have hit the 100-Note wall 4,900 Notes ago.

The takeaway isn't just "templates save data." It's templates save data in proportion to how much you batch, so templates AND a less frequent sync schedule can end up as a similar optimization (or even a compounding set of optimizations).

note

New to Notecard sync schedules (modes)? Unlike continuous mode, which holds an always-on network connection, periodic mode connects only when it needs to. Its outbound argument to hub.set is the max wait time, in minutes, before pending Notes are synced. Raising it means fewer, larger syncs, which means less battery power used and an opportunity for less data usage with templates.

Compact Templates, and Why Satellite is Different

By default, every templated Note carries some metadata along with it: a creation timestamp, several fields describing the device's location, and a timestamp for when that location was determined.

If so desired, adding "format":"compact" strips all metadata out. You can selectively get the metadata back by naming the fields you want in the template body: _time, _lat, _lon, and _ltime.

Two situations make that trade worth it. The first is that your transport requires it (more on that below). The second is that you've looked at your data usage, found the metadata is a meaningful share of it, and decided you can live without the fields you're dropping. Outside of those, the debugging value is usually worth more than the bytes.

{
  "req": "note.template",
  "file": "readings.qo",
  "format": "compact",
  "port": 21,
  "body": {
    "_time": 14,
    "level_cm": 14.1,
    "pump_running": true
  }
}

That port argument (a unique integer from 1 to 100) lets Notecard send a small numeric reference over the air instead of the full Notefile name.

Both format and port are required when using Notecard for LoRa or NTN (satellite) mode with Starnote or Notecard for Skylo. Notes queued without a template will never sync over NTN at all.

That requirement makes more sense once you see the packet economics:

On Skylo, 256 bytes is a hard ceiling. A packet that exceeds it is not transmitted, is ignored by the satellite network, and the Note is deleted. To be clear, compact templates aren't just an optimization here, they are a strict requirement that makes NTN communications possible.

tip

For devices that fail over between transports, you can actually define two templates for the same reading and let Notecard pick which one to use. For example, you can provide a verbose cellular template "delete":true and no port, but give a "lean" compact template "delete":true and a port. Before each sync Notecard discards the Notefile queue that DOES NOT match the active transport. Full instructions are available in Define NTN vs non-NTN Templates.

Tradeoffs Worth Knowing

Templates are almost, but not quite, 100% "free" in terms of their impact on your product.

Empty fields disappear. Templated Notefiles enforce omitempty, so any field holding null, false, 0, or "" is dropped from the body that reaches Notehub. A body of {"alert":true,"warning":false,"count":0,"status":"ok"} arrives as {"alert":true,"status":"ok"}. If your cloud endpoint requires all data to pass through, either add "full":true to your note.add requests or treat a missing key as the zero/null value.

Re-registering a template adds one, it doesn't replace one. Sending note.template for the same Notefile creates a new template rather than overwriting the old. Existing Notes are untouched, which is good, but a firmware bug that re-registers a slightly different schema on every boot will fill flash and sync each new template to Notehub.

Strings are constrained in compact mode. Each string value is capped at 255 bytes, and exceeding it returns a {template-incompatible} error.

Sometimes the right answer isn't a template at all. If your data is already binary, or genuinely unstructured, base64-encoding the payload argument in a note.add request might be a better option. You can attach a payload to any templated Note, and a template with an empty body ({}) gives you a payload-only Notefile.

Summary

My rule of thumb: just use templates.

They're the obvious call when you're sending the same shape of readings repeatedly, when you batch more than a handful of Notes per sync, when a device has to survive long offline stretches and queue data locally, or when you might add satellite failover later and would rather not restructure your firmware to get there.

Happy hacking! 💙

Frequently Asked Questions

What is a Notefile template on Blues Notecard?

A Notefile template is a fixed schema you register against a .qo or .qos Notefile with the note.template API. The template body names each expected field and supplies a value that hints at its data type and width. Notecard then stores conforming Notes as fixed-length binary records in flash instead of flexible JSON objects, and Notehub reassembles them into normal JSON events on the other side.

How much cellular data do Note templates actually save?

It depends almost entirely on how many Notes you sync per session. In Blues' published measurements on a cellular Notecard in minimum mode, a sync carrying 100 Notes dropped from 3.5 KB untemplated to 1.6 KB with a standard template and 1.2 KB with a compact template. A sync carrying a single Note showed essentially no difference, because per-session TCP and TLS overhead dominates at that size.

Are Note templates required for satellite or LoRa?

Yes. Templates are required for both inbound and outbound Notefiles when using Notecard for LoRa or NTN (satellite) mode with Starnote or Notecard for Skylo. Those templates must also include a format of "compact" and a unique port between 1 and 100. Notes queued without a template will never sync over NTN.

What happens to empty or zero fields in a templated Notefile?

Templated Notefiles enforce omitempty when serializing JSON, so fields holding null, false, 0, or an empty string are dropped from the body you see in Notehub. If your cloud code needs those fields present, either add "full":true to the note.add request or make your downstream logic treat a missing key as the zero value.

Can I change a Notefile template after devices are deployed?

You can send a new note.template request for the same Notefile at any time, and it applies only to new Notes without altering existing ones. Be aware that this creates an additional template rather than overwriting the old one, so churning through many templates (>25) for one Notefile will consume flash and sync bandwidth. Blues recommends one template per distinct Note schema.