When building an application that is expected to operate over a long period of time, you'll want to ensure that bandwidth is preserved and monitored, wherever possible. The Notecard provides features that allow you to optimize the size of Notes at rest and in transit, as well as a set of usage monitoring APIs.
By default, the Notecard allows for maximum developer flexibility in the structure and content of Notes. As such, individual Notes in a Notefile do not share structure or schema. You can add JSON structures and payloads of any type and format to a Notefile, adding and removing fields as required by your application.
In order to provide this simplicity to developers, the design of the Notefile system is primarily memory based and designed to support no more than 100 Notes per Notefile. As long as your data needs and sync periods ensure regular uploads of data to Notehub, this limit is adequate for most applications.
Some applications, however, will need to track and stage bursts of data that may eclipse the 100 Note limit in a short period of time, and before a sync can occur. For these types of use cases, the Notecard supports using a flash-based storage system based on Note templates.
Using the note.template request with any .qo/.qos Notefile, developers can
provide the Notecard with a schema of sorts to apply to future Notes added
to the Notefile. This template acts as a hint to the Notecard that allows it to
internally store data as fixed-length records rather than as flexible, JSON
objects, which tend to be much larger.
To create a template, use the file argument to specify the Notefile to which
the template should be applied. Then, use the body argument to specify
a template body, similar to the way you'd make a note.add request.
That body must contain the name of each field expected in each note.add
request, and a value that serves as the hint indicating
the data type to the Notecard. Each field can be a boolean, integer, float, or
string.
{
"req": "note.template",
"file": "readings.qo",
"body": {
"new_vals": true,
"temperature": 12.1,
"humidity": 12,
"pump_state": "4"
}
}
The Notecard responds to note.template with a single bytes field, indicating
the number of bytes that will be transmitted to Notehub, per note, before
compression.
{
"bytes": 40
}
You can also specify a maximum payload length to be accepted by the Notecard
using the length argument. Note: length and body are not mutually
exclusive and can be used together in a template.
{
"req": "note.template",
"file": "readings.qo",
"body": {
"new_vals": true,
"temperature": 12.1,
"humidity": 11,
"pump_state": "4"
},
"length": 32
}
Using the same body as above, and a payload length of 32 results in a
template of 72 bytes.
{
"bytes": 72
}
The hints in each template Note body value come with a few expectations and requirements, as well as options for advanced usage.
- Boolean values must be specified in a template as
true. - String fields must be a numeric string to specify the max length. For example, "42" for a string that can be up to 42 characters in length.
- Integer fields should use a specific value to indicate their type and
length based on the following:
11- for a 1 byte signed integer.12- for a 2 byte signed integer.13- for a 3 byte signed integer.14- for a 4 byte signed integer.18- for a 8 byte signed integer.
- Float fields should also use a specific value to indicate their type and
length based on the following:
12.1- for an IEEE 754 2 byte float.14.1- for an IEEE 754 4 byte float.18.1- for an IEEE 754 8 byte float.
After a template is created, use note.add requests to create Notes that
conform to the template.
{
"req": "note.add",
"file": "readings.qo",
"body": {
"new_vals": true,
"temperature": 22.22,
"humidity": 43,
"pump_state": "off"
}
}
When adding Notes to a Notefile with an active template, the following JSON object is returned by the Notecard:
{ "template": true }Notefiles with an active template validate each Note upon a note.add request.
If any value in the Note body does not adhere to the template, or if the
payload is longer than specified, an error is returned. For instance, the
following Note includes a float for the humidity, which was specified in the
template as an integer.
{
"req": "note.add",
"file": "readings.qo",
"body": {
"new_vals": true,
"temperature": 22.22,
"humidity": 43.22, // Specified as an integer
"pump_state": "off"
}
}
{
"err": "error adding note: integer expected because of template"
}
For string values, and error is not returned on a note.add, but the provided
value is truncated to the maximum length specified in the template. For
instance, the following Note includes a pump_state string longer than
the maximum length defined in the template. The pump_state for this
Note is truncated to four characters and saved as acti.
{
"req": "note.add",
"file": "readings.qo",
"body": {
"new_vals": true,
"temperature": 22.22,
"humidity": 43,
"pump_state": "active" // Saved as "acti"
}
}
If the needs of your application evolve, you can modify a template with
another note.template request to the same Notefile. A new template can be
set at any time and is non-destructive, meaning it has no impact on existing
Notes in the Notefile.
For instance, you may need to adjust the template field data types:
{
"req": "note.template",
"file": "readings.qo",
"body": {
"new_vals": true,
"temperature": 14.1, // Change to a 4 byte float
"humidity": 11,
"pump_state": "7" // Accept strings of 7 characters
}
}
Or add and remove fields:
{
"req": "note.template",
"file": "readings.qo",
"body": {
"new_vals": true,
"temperature": 12.1,
"humidity": 11,
"pressure": 12.1 // New field
}
}
Both of these template changes will be applied only to new Notes in the Notefile. Existing Notes remain unchanged.
To clear a template from a Notefile, simply call note.template with the
Notefile name and omit the body and payload arguments. After clearing the
template, all Notes written to the Notefile are stored as arbitrary JSON
structures. This request, if successful, will return an empty JSON body ({}).
{
"req": "note.template",
"file": "readings.qo"
}
The Notecard comes with a fixed amount of data available to send and receive
over its lifetime. The amount of data transmitted and received is
proportional to the amount of user data sent to the Notecard through requests
like note.add. It may vary higher due to per-session TLS and TCP overhead or
lower due to data compression.
Ultimately, it's up to you to determine how much data is needed in
an application, and how often that data should be sent to Notehub. To support
monitoring data usage in an application, the Notecard provides card.usage.get
and card.usage.test requests to see current usage, and project the lifetime
of the Notecard based on its current workload.
The card.usage.get request provides actual network usage statistics, and
can provide this information across the Notecard's entire life since activation,
or for periods of one hour, one day or a 30 day period.
A no argument request:
{
"req": "card.usage.get"
}
Is the same as:
{
"req": "card.usage.get",
"mode": "total"
}
This request returns an object with the number of seconds since the Notecard
was activated, the total number of bytes_sent and bytes_received, the
approximate number of notes_sent and notes_received, the number of
standard (sessions_standard) and TLS (sessions_secure) sessions, and the
UNIX Epoch time of device activation.
{
"seconds": 661135,
"bytes_sent": 65445,
"bytes_received": 136651,
"notes_sent": 50,
"notes_received": 18,
"sessions_standard": 51,
"sessions_secure": 14,
"time": 1598479763
}
To analyze a period of time, the mode argument also accepts the values of
1hour, 1day, or 30day and an offset argument to skip backwards in time
before returning stats for the mode unit specified. For instance, the
following request will skip back two days, and return a single day of usage
data.
{
"req": "card.usage.get",
"mode": "1day",
"offset": 2
}
noteTo accurately determine the start of the calculated time period when using
offset, use the time value of the response. Likewise, to calculate the end
of the time period, add the seconds value to the time value.
Once your Notecard is running a workload that you feel is representative of its
deployed use, you can use the card.usage.test request to estimate the
lifetime of available data given its current usage rate.
When called with no arguments, card.usage.test performs its projections based
on all data since activation. Alternatively, use the days argument to
specify the most recent number of days to analyze, hours to analyze a number
of hours, and megabytes to specify the Notecard data quota from which to
estimate. For example, if your project has been running production-ready
firmware for the last week, and your data cap is 500 MB, you'd send the
following request:
{
"req": "card.usage.test",
"days": 7,
"megabytes": 500
}
This request returns all of the fields that card.usage.get does so you can
see actual usage over the defined period. In addition, the returned object
contains the number of days used for the test, the average bytes_per_day
sent during the analyzed period, and the max number of days of Notecard
lifetime based on daily usage of the analyzed period. For example, if your
Notecard sends around 44 kilobytes per day, it would take 11,833 days, or
over 32 years before it eclipsed its data cap! Note: Usage information
provided by the Notecard is representative of all network traffic, including
TCP/IP and TLS overhead.
{
"days": 2,
"bytes_per_day": 44289,
"max": 11833,
// Fields also sent with card.usage.get
"bytes_sent": 29327,
"bytes_received": 59252,
"notes_sent": 16,
"notes_received": 13,
"sessions_standard": 24,
"sessions_secure": 5
}
