Cut Your Data Usage in Half with Notefile Templates

Here's a note.add request that, to me, looks completely reasonable:

{
  "req": "note.add",
  "file": "readings.qo",
  "body": {
    "level_cm": 184.62,
    "temp_c": 21.4,
    "battery_v": 3.87,
    "pump_running": true,
    "pump_cycles": 47,
    "status": "ok"
  }
}

The body of this event is 101 bytes of JSON, but only 24 of those bytes are values you care about. The other 77 are field names, quotes, colons, commas, and braces. Packed down to a fixed binary structure, those same six values only need about 11 bytes plus the status string.

In this scenario, you're shipping the literal characters "pump_running" on every single outbound Note, forever. At a reading every 15 minutes, that's roughly 35,000 copies per device per year. On a cellular Notecard with 500MB of bundled data, that may or may not be a cost you can absorb. On satellite, where Starnote for Iridium data runs $0.00075 per byte with a 10-byte minimum billable packet, it starts to present an issue.

Blues Notecard has a built-in answer to this, and it's one request: the note.template API.

Why Notefiles Are Schemaless to Begin With

Notecard's default is deliberately permissive. Individual Notes in a Notefile share no structure at all, so you can add fields, drop fields, and mix completely different shapes in one file. That's useful during development, but it has two costs:

1. The first is byte overhead. Because Notecard can't know what's coming, it stores and forwards whatever you hand it, keys and all.

2. The second is the schemaless Notefile system is memory-based and designed to hold no more than 100 unsynced Notes per Notefile. If you queue the 101st Note before a sync completes, you get an error. That's fine for a device that reports once an hour and syncs hourly. It is not fine for one that logs a burst of readings during an event, or purposefully sits offline for days.

Templates fix both problems with the same mechanism!

What note.template Actually Does

You register a template against a .qo or .qos Notefile by describing its shape. The body you send isn't data, it's a set of "hints", where each key is a field name you promise to send, and each value tells Notecard the type and width to reserve for it.

{
  "req": "note.template",
  "file": "readings.qo",
  "body": {
    "level_cm": 14.1,
    "temp_c": 12.1,
    "battery_v": 12.1,
    "pump_running": true,
    "pump_cycles": 22,
    "status": "-"
  }
}

Those magic numbers are worth bookmarking:

With the template registered, Notecard stops storing JSON objects and starts writing fixed-length binary records to flash. Notehub reverses the process on arrival, so your routes and your cloud code still receive ordinary JSON events. The effect on storage and sync capacity has an order-of-magnitude improvement, and the 100 Note limit no longer applies at all.

Better yet, Notecard will tell you exactly what you will be sending, byte-wise. The response to note.template includes a bytes field: the per-Note size that will be transmitted to Notehub, before compression. For example:

{
  "bytes": 40
}

Your own number will differ, since it depends on your field widths and, for standard (non-compact) templates, the automatic metadata covered below.

tip

Treat that bytes value as the number to optimize against. Widen a field from 12.1 to 18.1 and you can watch the cost of the change immediately, before you've deployed anything. You can also read back a live template at any time with {"req":"note.template","file":"readings.qo","verify":true}.

Adding Notes afterward with a note.add request looks exactly like it did before. The original API request from the top of this post works unchanged, and Notecard returns {"template":true} to confirm the Note was stored against the provided schema.

There's one behavior change worth knowing: templated Notefiles validate every Note on the way in. If you send a float where you promised an integer, you'll get an error:

{
  "err": "error adding note: integer expected because of template"
}

Where the Savings Actually Live

We publish measured data usage estimates for a cellular Notecard using a two-field Note body:

Some interesting findings from this table alone:

1. At one Note per sync, templates buy you almost nothing. A single sync costs about 1.2 KB whether or not there's a template, because you're paying for the session rather than the payload.

2. At 100 Notes per sync, the picture inverts. 3.5 KB becomes 1.6 KB with a standard template, and 1.2 KB with a compact one. That's a 54% and a 66% reduction, respectively.

3. The bottom row is intriguing because syncing 5,000 Notes in one session simply isn't possible without a template, because you'd have hit the 100-Note wall 4,900 Notes ago.

The takeaway isn't just "templates save data." It's templates save data in proportion to how much you batch, so templates AND a less frequent sync schedule can end up as a similar optimization (or even a compounding set of optimizations).

note

New to Notecard sync schedules (modes)? Unlike continuous mode, which holds an always-on network connection, periodic mode connects only when it needs to. Its outbound argument to hub.set is the max wait time, in minutes, before pending Notes are synced. Raising it means fewer, larger syncs, which means less battery power used and an opportunity for less data usage with templates.

Compact Templates, and Why Satellite is Different

By default, every templated Note carries some metadata along with it: a creation timestamp, several fields describing the device's location, and a timestamp for when that location was determined. All useful data elements for your product, especially if/when you need to debug your device.

If so desired, adding "format":"compact" strips all metadata out. You can selectively get the metadata back by naming the fields you want in the template body: _time, _lat, _lon, and _ltime.

Two situations make that trade worth it. The first is that your transport requires it (more on that below). The second is that you've looked at your data usage, found the metadata is a meaningful share of it, and decided you can live without the fields you're dropping. Outside of those, the debugging value is usually worth more than the bytes.

{
  "req": "note.template",
  "file": "readings.qo",
  "format": "compact",
  "port": 21,
  "body": {
    "_time": 14,
    "level_cm": 14.1,
    "pump_running": true
  }
}

That port argument (a unique integer from 1 to 100) lets Notecard send a small numeric reference over the air instead of the full Notefile name.

Both format and port are required when using Notecard for LoRa or NTN (satellite) mode with Starnote or Notecard for Skylo. Notes queued without a template will never sync over NTN at all.

That requirement makes more sense once you see the packet economics:

On Skylo, 256 bytes is a hard ceiling. A packet that exceeds it is not transmitted, is ignored by the satellite network, and the Note is deleted. To be clear, compact templates aren't just an optimization here, they are a strict requirement that makes NTN communications possible.

tip

For devices that fail over between transports, you can actually define two templates for the same reading and let Notecard pick which one to use. For example, you can provide a verbose cellular template "delete":true and no port, but give a "lean" compact template "delete":true and a port. Before each sync Notecard discards the Notefile queue that DOES NOT match the active transport. Full instructions are available in Define NTN vs non-NTN Templates.

Use payload for Binary Data, Never the body

If your device produces mass quantities of data that isn't just a boolean, string, or number (e.g. a batch of accelerometer samples, a compressed sensor frame, a protocol buffer), Notecard gives you a dedicated payload argument on note.add. It's the most efficient way to send base64-encoded binary data with Notecard.

This is worth stating clearly, because a tempting anti-pattern is to base64-encode your binary data and add it to a string field in a Note's body. Even in a templated Notefile you'll still pay a ~33% data penalty on this.

However, when you use a payload, Notecard base64-decodes the string immediately on the device and stores the raw bytes. At sync time the payload is pulled out into its own binary buffer. Then, the JSON that goes over the air omits the payload field entirely, leading to a far more efficient transaction.

So put binary in the payload, and keep the body for the values you want to query and route on. You can do both in one Note:

{
  "req": "note.add",
  "file": "readings.qo",
  "body": { "level_cm": 184.62, "pump_running": true },
  "payload": "ASNFZ4mrze8="
}

It's important to note that templates and payloads are complements, not alternatives. Three things are worth knowing about using them together:

• You can attach a payload to any templated Note, without declaring it in the template.
• A payload is normally capped at 256 bytes, but that limit doesn't apply once the Notefile has a template. (For large objects > 8 KB, card.binary and the Sending and Receiving Large Binary Objects guide are the better path.)
• Sending note.template with an empty body ({}) gives you a payload-only Notefile:

{
  "req": "note.template",
  "file": "waveform.qo",
  "body": {}
}

Variable-Length Arrays for Bursts of Samples

One more feature that pairs well with templates, and that I suspect is under-used, are variable-length arrays. Notefile templates support arrays, which is helpful if your device reports a series of readings rather than one.

However, on modern versions of Notecard firmware, you no longer have to declare each element in the array, letting you specify an array of arbitrary length. You can declare a single integer or float and send however many elements you like:

{
  "req": "note.template",
  "file": "vibration.qo",
  "format": "compact",
  "port": 22,
  "body": {
    "_time": 14,
    "rms_g": 12.1,
    "samples": [12.1]
  }
}

To be clear, [12.1] isn't a single-element array, it's a declaration that samples holds 2-byte floats, however many of them show up:

{
  "req": "note.add",
  "file": "vibration.qo",
  "body": {
    "rms_g": 0.42,
    "samples": [0.11, 0.19, 0.31, 0.42, 0.28]
  }
}

note

Variable-length arrays are Integer or Float only. Booleans and strings still need the fixed form where every element is declared.

Tradeoffs Worth Knowing

Templates are almost, but not quite, 100% "free" in terms of their impact on your product.

Empty fields disappear. Templated Notefiles enforce omitempty, so any field holding null, false, 0, or "" is dropped from the body that reaches Notehub. A body of {"alert":true,"warning":false,"count":0,"status":"ok"} arrives as {"alert":true,"status":"ok"}. If your cloud endpoint requires all data to pass through, either add "full":true to your note.add requests or treat a missing key as the zero/null value.

Re-registering a template adds one, it doesn't replace one. Sending note.template for the same Notefile creates a new template rather than overwriting the old. Existing Notes are untouched, which is good, but a firmware bug that re-registers a slightly different schema on every boot will fill flash and sync each new template to Notehub.

Strings are constrained in compact mode. Each string value is capped at 255 bytes, and exceeding it returns a {template-incompatible} error.

Summary

My rule of thumb: just use templates.

They're the obvious call when you're sending the same shape of readings repeatedly, when you batch more than a handful of Notes per sync, when a device has to survive long offline stretches and queue data locally, or when you might add satellite failover later and would rather not restructure your firmware to get there.

Happy hacking! 💙

Frequently Asked Questions

What is a Notefile template on Blues Notecard?

A Notefile template is a fixed schema you register against a .qo or .qos Notefile with the note.template API. The template body names each expected field and supplies a value that hints at its data type and width. Notecard then stores conforming Notes as fixed-length binary records in flash instead of flexible JSON objects, and Notehub reassembles them into normal JSON events on the other side.

How much cellular data do Note templates actually save?

It depends almost entirely on how many Notes you sync per session. In Blues' published measurements on a cellular Notecard in minimum mode, a sync carrying 100 Notes dropped from 3.5 KB untemplated to 1.6 KB with a standard template and 1.2 KB with a compact template. A sync carrying a single Note showed essentially no difference, because per-session TCP and TLS overhead dominates at that size.

Are Note templates required for satellite or LoRa?

Yes. Templates are required for both inbound and outbound Notefiles when using Notecard for LoRa or NTN (satellite) mode with Starnote or Notecard for Skylo. Those templates must also include a format of "compact" and a unique port between 1 and 100. Notes queued without a template will never sync over NTN.

What happens to empty or zero fields in a templated Notefile?

Templated Notefiles enforce omitempty when serializing JSON, so fields holding null, false, 0, or an empty string are dropped from the body you see in Notehub. If your cloud code needs those fields present, either add "full":true to the note.add request or make your downstream logic treat a missing key as the zero value.

Should binary data go in a Note's body or its payload?

Always the payload. Notecard base64-decodes the payload argument you provide, and stores the raw bytes. At sync time that payload travels as binary while the JSON sent over the air omits the payload field entirely. A base64 string placed in a body field is just a string, so it stays text the whole way and carries base64's roughly 33% size inflation on every Note, plus JSON quoting around it.