Low Bandwidth Design

When building an application that is expected to operate over a long period of time, you'll want to ensure that bandwidth is preserved and monitored, wherever possible. The Notecard provides features that allow you to optimize the size of Notes at rest and in transit, as well as a set of usage monitoring APIs.

Working with Note Templates

By default, the Notecard allows for maximum developer flexibility in the structure and content of Notes. As such, individual Notes in a Notefile do not share structure or schema. You can add JSON structures and payloads of any type and format to a Notefile, adding and removing fields as required by your application.

In order to provide this simplicity to developers, the design of the Notefile system is primarily memory based and designed to support no more than 100 Notes per Notefile. As long as your data needs and sync periods ensure regular uploads of data to Notehub, this limit is adequate for most applications.

Some applications, however, will need to track and stage bursts of data that may eclipse the 100 Note limit in a short period of time, and before a sync can occur. For these types of use cases, the Notecard supports using a flash-based storage system based on Note templates.

Using the note.template request with any .qo/.qos Notefile, developers can provide the Notecard with a schema of sorts to apply to future Notes added to the Notefile. This template acts as a hint to the Notecard that allows it to internally store data as fixed-length records rather than as flexible, JSON objects, which tend to be much larger.

Creating a Template

To create a template, use the file argument to specify the Notefile to which the template should be applied. Then, use the body argument to specify a template body, similar to the way you'd make a note.add request. That body must contain the name of each field expected in each note.add request, and a value that serves as the hint indicating the data type to the Notecard. Each field can be a boolean, integer, float, or string.

{
  "req":  "note.template",
  "file": "readings.qo",
  "body": {
    "new_vals":    true,
    "temperature": 1.1,
    "humidity":    1,
    "pump_state":  "aaaa"
  }
}

The Notecard responds to note.template with a single bytes field, indicating the number of bytes that will be transmitted to Notehub, per note, before compression.

{
  "bytes": 40
}

You can also specify a maximum payload length to be accepted by the Notecard using the length argument. Note: length and body are not mutually exclusive and can be used together in a template.

{
  "req":   "note.template",
  "file":  "readings.qo",
  "body":  {
    "new_vals":    true,
    "temperature": 1.1,
    "humidity":    1,
    "pump_state":  "aaaa"
  },
  "length": 32
}

Using the same body as above, and a payload length of 32 results in a template of 72 bytes.

{
  "bytes": 72
}

Understanding Template Data Types

The hints in each template Note body value come with a few expectations and requirements, as well as options for advanced usage.

Boolean values must be specified in a template as true.
String fields must be a repeated string of a character with a length corresponding to the expected max length of the string. Actual values can be shorter than this length, but cannot be longer. Alternatively, a numeric string can be used to specify the max length. For example, "42" for a string that can be up to 42 characters in length.
Integer fields should use a specific value to indicate their type and length based on the following:
- 1 - to store the field as an 32-bit signed integer.
- 11 - for a 1 byte signed integer.
- 12 - for a 2 byte signed integer.
- 13 - for a 3 byte signed integer.
- 14 - for a 4 byte signed integer.
- 18 - for a 8 byte signed integer.
Float fields should also use a specific value to indicate their type and length based on the following:
- 1.1 - to store the field as an 8-byte double float.
- 12.1 - for an IEEE 754 2 byte float.
- 14.1 - for an IEEE 754 4 byte float.
- 18.1 - for an IEEE 754 8 byte float.

Adding Notes to a Template Notefile

After a template is created, use note.add requests to create Notes that conform to the template.

{
  "req":  "note.add",
  "file": "readings.qo",
  "body": {
    "new_vals":    true,
    "temperature": 22.22,
    "humidity":    43,
    "pump_state":  "off"
  }
}

When adding Notes to a Notefile with an active template, the following JSON object is returned by the Notecard:

{ "template": true }

Notefiles with an active template validate each Note upon a note.add request. If any value in the Note body does not adhere to the template, or if the payload is longer than specified, an error is returned. For instance, the following Note includes a float for the humidity, which was specified in the template as an integer.

{
  "req":  "note.add",
  "file": "readings.qo",
  "body": {
    "new_vals":    true,
    "temperature": 22.22,
    "humidity":    43.22, // Specified as an integer
    "pump_state":  "off"
  }
}

{
  "err": "error adding note: integer expected because of template"
}

For string values, and error is not returned on a note.add, but the provided value is truncated to the maximum length specified in the template. For instance, the following Note includes a pump_state string longer than the maximum length defined in the template. The pump_state for this Note is truncated to four characters and saved as acti.

{
  "req":  "note.add",
  "file": "readings.qo",
  "body": {
    "new_vals":    true,
    "temperature": 22.22,
    "humidity":    43,
    "pump_state":  "active" // Saved as "acti"
  }
}

Modifying a Template

If the needs of your application evolve, you can modify a template with another note.template request to the same Notefile. A new template can be set at any time and is non-destructive, meaning it has no impact on existing Notes in the Notefile.

For instance, you may need to adjust the template field data types:

{
  "req":  "note.template",
  "file": "readings.qo",
  "body": {
    "new_vals":    true,
    "temperature": 1.2, // Change to a 2 byte float
    "humidity":    1,
    "pump_state":  "7"   // Accept strings of 7 characters
  }
}

Or add and remove fields:

{
  "req":  "note.template",
  "file": "readings.qo",
  "body": {
    "new_vals":    true,
    "temperature": 1.2,
    "humidity":    1,
    "pressure":    1.1 // New field
  }
}

Both of these template changes will be applied only to new Notes in the Notefile. Existing Notes remain unchanged.

Clearing a Template

To clear a template from a Notefile, simply call note.template with the Notefile name and omit the body and payload arguments. After clearing the template, all Notes written to the Notefile are stored as arbitrary JSON structures. This request, if successful, will return an empty JSON body ({}).

{
  "req":  "note.template",
  "file": "readings.qo"
}

Measuring Data Usage

The Notecard comes with a fixed amount of data available to send and receive over its lifetime. The amount of data transmitted and received is proportional to the amount of user data sent to the Notecard through requests like note.add. It may vary higher due to per-session TLS and TCP overhead or lower due to data compression.

Ultimately, it's up to you to determine how much data is needed in an application, and how often that data should be sent to Notehub. To support monitoring data usage in an application, the Notecard provides card.usage.get and card.usage.test requests to see current usage, and project the lifetime of the Notecard based on its current workload.

Obtaining Historical Usage

The card.usage.get request provides actual network usage statistics, and can provide this information across the Notecard's entire life since activation, or for periods of one hour, one day or a 30 day period.

A no argument request:

{
  "req":  "card.usage.get"
}

Is the same as:

{
  "req":  "card.usage.get",
  "mode": "total"
}

This request returns an object with the number of seconds since the Notecard was activated, the total number of bytes_sent and bytes_received, the total number of notes_sent and notes_received, the number of standard (sessions_standard) and TLS (sessions_secure) sessions, and the UNIX Epoch time of device activation.

{
  "seconds":           661135,
  "bytes_sent":        65445,
  "bytes_received":    136651,
  "notes_sent":        50,
  "notes_received":    18,
  "sessions_standard": 51,
  "sessions_secure":   14,
  "time":              1598479763
  }

To analyze a period of time, the mode argument also accepts the values of 1hour, 1day or 30day and an offset argument to skip backwards in time before returning stats for the mode unit specified. For instance, the following request will skip back two days, and return a single day of usage data.

{
  "req":    "card.usage.get",
  "mode":   "1day",
  "offset": 2
}

Projecting the Lifetime of Available Data

Once your Notecard is running a workload that you feel is representative of its deployed use, you can use the card.usage.test request to estimate the lifetime of available data given its current usage rate.

When called with no arguments, card.usage.test performs its projections based on all data since activation. Alternatively, use the days argument to specify the most recent number of days to analyze, hours to analyze a number of hours, and megabytes to specify the Notecard data quota from which to estimate. For example, if your project has been running production-ready firmware for the last week, and your data cap is 500 MB, you'd send the following request:

{
  "req":       "card.usage.test",
  "days":      7,
  "megabytes": 500
}

This request returns all of the fields that card.usage.get does so you can see actual usage over the defined period. In addition, the returned object contains the number of days used for the test, the average bytes_per_day sent during the analyzed period, and the max number of days of Notecard lifetime based on daily usage of the analyzed period. For example, if your Notecard sends around 44 kilobytes per day, it would take 11,833 days, or over 32 years before it eclipsed its data cap! Note: Usage information provided by the Notecard is representative of all network traffic, including TCP/IP and TLS overhead.

{
  "days":              2,
  "bytes_per_day":     44289,
  "max":               11833,
 
  // Fields also sent with card.usage.get
  "bytes_sent":        29327,
  "bytes_received":    59252,
  "notes_sent":        16,
  "notes_received":    13,
  "sessions_standard": 24,
  "sessions_secure":   5
}

Low Power Design Host Firmware Update Requests