Low Bandwidth Design

When building an application that is expected to operate over a long period of time, you'll want to ensure that bandwidth is preserved and monitored, wherever possible. The Notecard provides features that allow you to optimize the size of Notes at rest and in transit, as well as a set of usage monitoring APIs.

Working with Note Templates

By default, the Notecard allows for maximum developer flexibility in the structure and content of Notes. As such, individual Notes in a Notefile do not share structure or schema. You can add JSON structures and payloads of any type and format to a Notefile, adding and removing fields as required by your application.

In order to provide this simplicity to developers, the design of the Notefile system is primarily memory based and designed to support no more than 100 Notes per Notefile. As long as your data needs and sync periods ensure regular uploads of data to Notehub, this limit is adequate for most applications.

Some applications, however, will need to track and stage bursts of data that may eclipse the 100 Note limit in a short period of time, and before a sync can occur. For these types of use cases, the Notecard supports using a flash-based storage system based on Note templates.

Using the note.template request with any .qo/.qos Notefile, developers can provide the Notecard with a schema of sorts to apply to future Notes added to the Notefile. This template acts as a hint to the Notecard that allows it to internally store data as fixed-length records rather than as flexible, JSON objects, which tend to be much larger.

Creating a Template

To create a template, use the file argument to specify the Notefile to which the template should be applied. Then, use the body argument to specify a template body, similar to the way you'd make a note.add request. That body must contain the name of each field expected in each note.add request, and a value that serves as the hint indicating the data type to the Notecard. Each field can be a boolean, integer, float, or string.

{
  "req": "note.template",
  "file": "readings.qo",
  "body": {
    "new_vals": true,
    "temperature": 14.1,
    "humidity": 11,
    "pump_state": "4"
  }
}

J *req = NoteNewRequest("note.template");
JAddStringToObject(req, "file", "readings.qo");

J *body = JCreateObject();
JAddBoolToObject(body, "new_vals", true);
JAddNumberToObject(body, "temperature", 14.1);
JAddNumberToObject(body, "humidity", 11);
JAddStringToObject(body, "pump_state", "4");
JAddItemToObject(req, "body", body);

NoteRequest(req);

req = {"req": "note.template"}
req["file"] = "readings.qo"
req["body"] = {
    "new_vals": True,
    "temperature": 14.1,
    "humidity": 11,
    "pump_state": "4"
  }

rsp = card.Transaction(req)

The Notecard responds to note.template with a single bytes field, indicating the number of bytes that will be transmitted to Notehub, per note, before compression.

{
  "bytes": 40
}

warning

Please note that trying to "update" an existing template's body schema by using the same file argument used previously does not overwrite the old template, but rather creates a new one. This can become an issue if you create numerous Notefile templates (>25) to accommodate changes in data from individual Notes, as you may negate the advantage of templates by filling the flash storage on the Notecard and consuming additional cellular data by transferring each new template to Notehub.

In this scenario, we recommend defining a smaller number of consistent Notefile templates, binary-encoding the data and sending it in a note.add payload argument, or not using Notefile templates at all.

You can also specify a length argument that will set the maximum length of a payload (in bytes) that can be sent in Notes for the templated Notefile. If using Notecard firmware prior to v3.2.1, the length argument is required when using a payload with a templated Notefile.

{
  "req": "note.template",
  "file": "readings.qo",
  "body": {
    "new_vals": true,
    "temperature": 14.1,
    "humidity": 11,
    "pump_state": "4"
  },
  "length": 32
}

J *req = NoteNewRequest("note.template");
JAddStringToObject(req, "file", "readings.qo");
JAddNumberToObject(req, "length", 32);

J *body = JCreateObject();
JAddBoolToObject(body, "new_vals", true);
JAddNumberToObject(body, "temperature", 14.1);
JAddNumberToObject(body, "humidity", 11);
JAddStringToObject(body, "pump_state", "4");
JAddItemToObject(req, "body", body);

NoteRequest(req);

req = {"req": "note.template"}
req["file"] = "readings.qo"
req["body"] = {
    "new_vals": True,
    "temperature": 14.1,
    "humidity": 11,
    "pump_state": "4"
  }
req["length"] = 32

rsp = card.Transaction(req)

Using the same body as above, and a payload length of 32 results in a template of 72 bytes.

{
  "bytes": 72
}

Understanding Template Data Types

The hints in each template Note body value come with a few expectations and requirements, as well as options for advanced usage.

• Boolean values must be specified in a template as true.
• String For firmware versions prior to v3.2.1 fields must be a numeric string to specify the max length. For example, "42" for a string that can be up to 42 characters in length. As of v3.2.1 variable-length strings are supported for any field and any string can be provided when configuring the template.
• Integer fields should use a specific value to indicate their type and length based on the following:
  • 11 - for a 1 byte signed integer (e.g. -128 to 127).
  • 12 - for a 2 byte signed integer (e.g. -32,768 to 32,767).
  • 13 - for a 3 byte signed integer (e.g. -8,388,608 to 8,388,607).
  • 14 - for a 4 byte signed integer (e.g. -2,147,483,648 to 2,147,483,647).
  • 18 - for a 8 byte signed integer (e.g. -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807).
  • 21 - for a 1 byte unsigned integer (e.g. 0 to 255). Available as of v3.3.1.
  • 22 - for a 2 byte unsigned integer (e.g. 0 to 65535). Available as of v3.3.1.
  • 23 - for a 3 byte unsigned integer (e.g. 0 to 16777215). Available as of v3.3.1.
  • 24 - for a 4 byte unsigned integer (e.g. 0 to 4294967295). Available as of v3.3.1.
• Float fields should also use a specific value to indicate their type and length based on the following:
  • 12.1 - for an IEEE 754 2 byte float.
  • 14.1 - for an IEEE 754 4 byte float.
  • 18.1 - for an IEEE 754 8 byte float.

Using Arrays in Templates

If you're working with more complex data structures, it's possible to use arrays of data types when creating a template. The same definitions are used when assigning data types to the array.

{
  "req": "note.template",
  "file": "readings.qo",
  "body": {
    "new_vals": true,
    "temperature": 14.1,
    "humidity": 11,
    "pump_state": "4",
    "array_vals": [true, 14.1, 11, "4"]
  }
}

J *req = NoteNewRequest("note.template");
JAddStringToObject(req, "file", "readings.qo");

J *body = JCreateObject();
JAddBoolToObject(body, "new_vals", true);
JAddNumberToObject(body, "temperature", 14.1);
JAddNumberToObject(body, "humidity", 11);
JAddStringToObject(body, "pump_state", "4");

J *arr = JCreateArray();
JAddItemToArray(arr, JCreateBool(true));
JAddItemToArray(arr, JCreateNumber(14.1));
JAddItemToArray(arr, JCreateNumber(11));
JAddItemToArray(arr, JCreateString("4"));
JAddItemToObject(body, "array_vals", arr);

JAddItemToObject(req, "body", body);

NoteRequest(req);

req = {"req": "note.template"}
req["file"] = "readings.qo"
req["body"] = {
    "new_vals": True,
    "temperature": 14.1,
    "humidity": 11,
    "pump_state": "4",
    "array_vals": [true, 14.1, 11, "4"]
  }

rsp = card.Transaction(req)

Verifying a Template

You can use the verify:true argument to return the current template for a Notefile.

{
  "req": "note.template",
  "file": "readings.qo",
  "verify": true
}

J *req = NoteNewRequest("note.template");
JAddStringToObject(req, "file", "readings.qo");
JAddBoolToObject(req, "verify", true);

NoteRequest(req);

req = {"req": "note.template"}
req["file"] = "readings.qo"
req["verify"] = True

rsp = card.Transaction(req)

If the file provided has an active template, it will be returned in a response body.

{
 "body": {
  "new_vals": true,
  "temperature": 14.1,
  "humidity": 11,
  "pump_state": "4"
 },
 "template": true,
 "length": 32
}

Adding Notes to a Template Notefile

After a template is created, use note.add requests to create Notes that conform to the template.

{
  "req": "note.add",
  "file": "readings.qo",
  "body": {
    "new_vals": true,
    "temperature": 22.22,
    "humidity": 43,
    "pump_state": "off"
  }
}

J *req = NoteNewRequest("note.add");
JAddStringToObject(req, "file", "readings.qo");

J *body = JCreateObject();
JAddBoolToObject(body, "new_vals", true);
JAddNumberToObject(body, "temperature", 22.22);
JAddNumberToObject(body, "humidity", 43);
JAddStringToObject(body, "pump_state", "off");
JAddItemToObject(req, "body", body);

NoteRequest(req);

req = {"req": "note.add"}
req["file"] = "readings.qo"
req["body"] = {
      "new_vals": true,
      "temperature": 22.22,
      "humidity": 43,
      "pump_state": "off"
    }

rsp = card.Transaction(req)

When adding Notes to a Notefile with an active template, the following JSON object is returned by the Notecard:

{ "template": true }

Notefiles with an active template validate each Note upon a note.add request. If any value in the Note body does not adhere to the template, or if the payload is longer than specified, an error is returned. For instance, the following Note includes a float for the humidity, which was specified in the template as an integer.

{
  "req": "note.add",
  "file": "readings.qo",
  "body": {
    "new_vals": true,
    "temperature": 22.22,
    "humidity": 43.22, // mistakenly specified here as a float instead of integer
    "pump_state": "off"
  }
}

J *req = NoteNewRequest("note.add");
JAddStringToObject(req, "file", "readings.qo");

J *body = JCreateObject();
JAddBoolToObject(body, "new_vals", true);
JAddNumberToObject(body, "temperature", 22.22);
JAddNumberToObject(body, "humidity", 43.22); // mistakenly specified here as a float instead of integer
JAddStringToObject(body, "pump_state", "off");
JAddItemToObject(req, "body", body);

NoteRequest(req);

req = {"req": "note.add"}
req["file"] = "readings.qo"
req["body"] = {
      "new_vals": true,
      "temperature": 22.22,
      "humidity": 43.22, # mistakenly specified here as a float instead of integer
      "pump_state": "off"
    }

rsp = card.Transaction(req)

{
 "err": "error adding note: integer expected because of template"
}

For string values, an error is not returned on a note.add, but the provided value is truncated to the length (if specified in the template). For instance, the following Note includes a pump_state string longer than the maximum length defined in the template. The pump_state for this Note is truncated to four characters and saved as acti.

{
  "req": "note.add",
  "file": "readings.qo",
  "body": {
    "new_vals": true,
    "temperature": 22.22,
    "humidity": 43,
    "pump_state": "active" // will be saved as "acti"
  }
}

J *req = NoteNewRequest("note.add");
JAddStringToObject(req, "file", "readings.qo");

J *body = JCreateObject();
JAddBoolToObject(body, "new_vals", true);
JAddNumberToObject(body, "temperature", 22.22);
JAddNumberToObject(body, "humidity", 43);
JAddStringToObject(body, "pump_state", "active"); // will be saved as "acti"
JAddItemToObject(req, "body", body);

NoteRequest(req);

req = {"req": "note.add"}
req["file"] = "readings.qo"
req["body"] = {
      "new_vals": true,
      "temperature": 22.22,
      "humidity": 43,
      "pump_state": "active" # will be saved as "acti"
    }

rsp = card.Transaction(req)

Modifying a Template

If the needs of your application evolve, you can modify a template with another note.template request to the same Notefile. A new template can be set at any time and is non-destructive, meaning it has no impact on existing Notes in the Notefile.

For instance, you may need to modify the template field data types and/or add/remove fields:

{
  "req": "note.template",
  "file": "readings.qo",
  "body": {
    "new_vals": true,
    "temperature": 14.1, // Change to a 4 byte float
    "humidity": 11,
    "pump_state": "4",
    "pressure": 12.1 // New field
  }
}

J *req = NoteNewRequest("note.template");
JAddStringToObject(req, "file", "readings.qo");

J *body = JCreateObject();
JAddBoolToObject(body, "new_vals", true);
JAddNumberToObject(body, "temperature", 14.1); // Change to a 4 byte float
JAddNumberToObject(body, "humidity", 11);
JAddStringToObject(body, "pump_state", "4");
JAddNumberToObject(body, "pressure", 12.1); // New field
JAddItemToObject(req, "body", body);

NoteRequest(req);

req = {"req": "note.template"}
req["file"] = "readings.qo"
req["body"] = {
    "new_vals": True,
    "temperature": 14.1, # Change to a 4 byte float
    "humidity": 11,
    "pump_state": "4",
    "pressure": 12.1 # New field
  }

rsp = card.Transaction(req)

These template changes will be applied only to new Notes in the Notefile. Existing Notes remain unchanged.

Clearing a Template

To clear a template from a Notefile, simply call note.template with the Notefile name and omit the body and payload arguments. After clearing the template, all Notes written to the Notefile are stored as arbitrary JSON structures. This request, if successful, will return an empty JSON body ({}).

{
  "req": "note.template",
  "file": "readings.qo"
}

J *req = NoteNewRequest("note.template");
JAddStringToObject(req, "file", "readings.qo");

NoteRequest(req);

req = {"req": "note.template"}
req["file"] = "readings.qo"

rsp = card.Transaction(req)

Measuring Data Usage

The Notecard comes with a fixed amount of data available to send and receive over its lifetime. The amount of data transmitted and received is proportional to the amount of user data sent to the Notecard through requests like note.add. It may vary higher due to per-session TLS and TCP overhead or lower due to data compression.

Ultimately, it's up to you to determine how much data is needed in an application, and how often that data should be sent to Notehub. To support monitoring data usage in an application, the Notecard provides card.usage.get and card.usage.test requests to see current usage, and project the lifetime of the Notecard based on its current workload.

Obtaining Historical Usage

The card.usage.get request provides actual network usage statistics, and can provide this information across the Notecard's entire life since activation, or for periods of one hour, one day or a 30 day period.

note

Usage data is updated by the Notecard at the end of each cellular connection. If connected in continuous mode, usage data will not be updated until the current session ends, which can be configured with the hub.set duration argument.

A no argument request:

{
  "req": "card.usage.get"
}

J *req = NoteNewRequest("card.usage.get");

NoteRequest(req);

req = {"req": "card.usage.get"}

rsp = card.Transaction(req)

Is the same as:

{
  "req": "card.usage.get",
  "mode": "total"
}

J *req = NoteNewRequest("card.usage.get");
JAddStringToObject(req, "mode", "total");

NoteRequest(req);

req = {"req": "card.usage.get"}
req["mode"] = "total"

rsp = card.Transaction(req)

This request returns an object with the number of seconds since the Notecard was activated, the total number of bytes_sent and bytes_received, the approximate number of notes_sent and notes_received, the number of standard (sessions_standard) and TLS (sessions_secure) sessions, and the UNIX Epoch time of device activation.

{
 "seconds": 661135,
 "bytes_sent": 65445,
 "bytes_received": 136651,
 "notes_sent": 50,
 "notes_received": 18,
 "sessions_standard": 51,
 "sessions_secure": 14,
 "time": 1598479763
}

To analyze a period of time, the mode argument also accepts the values of 1hour, 1day, or 30day and an offset argument to skip backwards in time before returning stats for the mode unit specified. For instance, the following request will skip back two days, and return a single day of usage data.

{
  "req": "card.usage.get",
  "mode": "1day",
  "offset": 2
}

J *req = NoteNewRequest("card.usage.get");
JAddStringToObject(req, "mode", "1day");
JAddNumberToObject(req, "offset", 2);

NoteRequest(req);

req = {"req": "card.usage.get"}
req["mode"] = "1day"
req["offset"] = 2

rsp = card.Transaction(req)

note

To accurately determine the start of the calculated time period when using offset, use the time value of the response. Likewise, to calculate the end of the time period, add the seconds value to the time value.

SIM Usage Data on Notehub.io

Another place to view a summary of data used by a Notecard is on Notehub.io. By navigating to Devices > SIM Usage you can see the sum total of bytes transferred to-date.

This same information is available under the SIM tab when viewing device details.

note

The usage statistics provided by card.usage may differ from what appears on Notehub.io. You should consider card.usage as more accurate than the values you see in Notehub.io. This is because the card.usage statistics are derived from the Notecard "asking" the modem (every time the modem powers down - or the duration of hub.set is met) how many bytes were transferred over the cellular network. Notehub.io can only approximate this number by looking at how many bytes were transferred at a higher level (over TCP or TLS). The numbers in Notehub.io are also updated nightly by a process that queries the cellular carrier to determine how much data has been used on the SIM.

Projecting the Lifetime of Available Data

Once your Notecard is running a workload that you feel is representative of its deployed use, you can use the card.usage.test request to estimate the lifetime of available data given its current usage rate.

When called with no arguments, card.usage.test performs its projections based on all data since activation. Alternatively, use the days argument to specify the most recent number of days to analyze, hours to analyze a number of hours, and megabytes to specify the Notecard data quota from which to estimate. For example, if your project has been running production-ready firmware for the last week, and your data cap is 500 MB, you'd send the following request:

{
  "req": "card.usage.test",
  "days": 7,
  "megabytes": 500
}

J *req = NoteNewRequest("card.usage.test");
JAddNumberToObject(req, "days", 7);
JAddNumberToObject(req, "megabytes", 500);

NoteRequest(req);

req = {"req": "card.usage.test"}
req["days"] = 7
req["megabytes"] = 500

rsp = card.Transaction(req)

This request returns all of the fields that card.usage.get does so you can see actual usage over the defined period. In addition, the returned object contains the number of days used for the test, the average bytes_per_day sent during the analyzed period, and the max number of days of Notecard lifetime based on daily usage of the analyzed period. For example, if your Notecard sends around 44 kilobytes per day, it would take 11,833 days, or over 32 years before it eclipsed its data cap! Note: Usage information provided by the Notecard is representative of all network traffic, including TCP/IP and TLS overhead.

{
  "days":              2,
  "bytes_per_day":     44289,
  "max":               11833,
 
  // Fields also sent with card.usage.get
  "bytes_sent":        29327,
  "bytes_received":    59252,
  "notes_sent":        16,
  "notes_received":    13,
  "sessions_standard": 24,
  "sessions_secure":   5
}

Data Usage Estimates

Since the Cellular Notecard comes prepaid with 500MB of cellular data, a common inquiry is precisely how much of that allocation is consumed when sending/receiving data.

This section aims to lay out some commonly-used scenarios and provide metrics around what approximate data usage numbers to expect, given the following set of testing parameters.

Testing Parameters

• Tests were performed using firmware v3.4.1 on an NBGL Notecard.
• Notecard was factory reset with a card.restore request: {"req":"card.restore","delete":true}
• Measurements were taken after a cold boot of the Notecard + three (3) completed hub.sync requests to ensure any data overhead related to its first connection was cleared: {"req":"hub.sync"}
• The body of the Note used for all tests was {"temp":12.3,"alert":true}.
• When a Note Template was used, "temp" was set to float16 and "alert" was set to bool: {"req":"note.template","file":"test.qo","body":{"temp": 18.1,"alert": true}}
• The continuous mode tests were performed with inbound and outbound arguments omitted from the hub.set request: {"req":"hub.set","mode":"continuous","product":"<productuid>"}

Initial Cellular Connection

After a power cycle or card.restart, the Notecard must re-negotiate its connection with Notehub. This always involves a TLS connection (for security purposes), even if there is no user data that needs to be synced. This is why we strongly encourage users to leave the Notecard powered-on, in an idle state, and to only restart the Notecard when strictly necessary.

note

The data usage for an initial connection will vary depending on the state of the connected Notehub project. For instance, upon boot, the Notecard will sync data in any existing .db/dbs Notefiles, any pending .qo/.qos Notefiles, and any Notefile templates. This is because part of the Notecard's boot process is to examine the data present on both sides (Notecard and Notehub) to make sure everything is completely in-sync after the initial power-on.

The value range provided below is based on a factory-reset Notecard associated with a newly-created Notehub project.

Approximate Data Usage: 28KB +/- 3KB.

Periodic/Minimum Mode Usage

The Notecard, by default, is set to periodic mode to conserve power by only connecting when it needs to sync with Notehub. The following tests were performed in minimum mode, which is similar to periodic, but requires an explicit hub.sync request to send and receive data.

Continuous Mode Usage

The use of continuous mode is only recommended when the Notecard is connected to line power and requires a low latency of syncing Notes with Notehub.

When a Notecard is in continuous mode, it needs to occasionally negotiate with the carrier to maintain its session. This is why we are also documenting the usage of continuous mode in two "idle" states:

There are two options for a continuous mode connection:

• hub.set "sync":false to prevent the Notecard from checking for inbound Notefile changes.
• hub.set "sync":true to routinely check Notehub for inbound Notefile changes.

With sync:true both the Notecard and Notehub employ mechanisms to ensure that the socket connection between the Notecard and Notehub does not "silently" drop, as can happen with wireless connections. These mechanisms consume additional data usage, but allow a robust and near instantaneous notification of changes from Notehub to Notecard, which is essential in some applications.

Additional Notes and FAQs

• Why is less bandwidth required (sometimes) to send 10 Notes vs 1 Note?
  • This is due to the fact that binary compression on the Notecard is only triggered at a certain threshold. This scenario is highly dependent on the size of the Note body or payload.
• Why do syncs in continuous mode generally consume slightly less bandwidth than in periodic/minimum mode?
  • Periodic syncs require a small amount of overhead to establish a session, while continuous mode syncs are performed during an already-established session.
• Why do syncs involving .qos, .qis, or .dbs files consume more bandwidth than similar syncs with .qo, .qi, or .db Notefiles?
  • If any of the Notefiles in a sync has a .qos, .qis, or .dbs file extension the Notecard must establish a TLS connection to Notehub before performing the sync, and the establishment of a TLS connection has an overhead of ~4 KB.