Sending and Receiving Large Binary Objects
While Notecard is designed to be a low-bandwidth wireless device, it is also possible to sync large binary payloads with the cloud.
This is accomplished by storing raw binary data in a reserved area on the Notecard, and then having Notecard send that large block directly to Notehub. Likewise, Notecard and Notehub can work together to get a binary payload from a remote endpoint and save it to the reserved area on the Notecard.
Important Considerations When Syncing Large Binary Objects
-
In your app design, it's safe to assume the maximum space available for data in the binary storage area on the Notecard is 100KB. The exact available space (in bytes) is returned in the
maxfield in response to a card.binary request.If the total size of the binary data you are sending is > than
max, you will need to "flush" the storage with the appropriateweb.postrequest each time that limit is reached (see examples below), and then reassemble the binary data after it has been routed to your cloud. -
Notehub charges one event credit for each megabyte of data uploaded via web transactions.
Sending Large Binary Objects
Sending a large binary object from the Notecard involves two steps:
- Storing the binary payload in Notecard's reserved binary buffer.
- Syncing that buffer with Notehub so it can be routed to your cloud endpoint.
These two steps are independent, so you can mix and match any storing method
with any syncing method. Most applications pair the SDK helpers with a
web.post request.
Storing Binary Data on Notecard
You have two paths to choose from when populating Notecard's binary buffer. They
both build on the
card.binary and
card.binary.put
APIs, but the note-arduino SDK provides helper methods to ease the process.
- Storing Binary Data with the note-arduino SDK (Recommended)
- Storing Binary Data with the card.binary APIs
Storing Binary Data with the note-arduino SDK
Due to the complexities of using the card.binary APIs directly, the
recommended path is to use the helper methods provided in the
note-arduino SDK.
The following Arduino examples demonstrate storing binary data with
note-arduino:
basic binary data example
and
sending a large binary payload in chunks.
Storing a Single Binary Fragment
For small binary payloads (e.g. <= 8 KB), you can store the entire payload in
a single fragment without the need to split and reassemble it on your cloud
endpoint.
-
Define the binary data and use the
NoteBinaryStoreTransmit()function to store the data in the reserved binary space on the Notecard. The fourth argument is the offset into the Notecard's binary area where the data should be written (0 for a single-fragment upload).char buff[25] = "Hello World"; NoteBinaryStoreTransmit((uint8_t *) buff, 10, sizeof(buff), 0); -
Once the buffer is populated, continue on to Syncing Binary Data to Notehub.
Storing Multiple Binary Fragments
For larger binary payloads, you may need to split the payload into multiple smaller fragments and reassemble them on your cloud endpoint.
-
Define the size of the binary payload fragments to send to the Notecard.
#define CHUNK_SIZE 4096 uint8_t temp_buffer[CHUNK_SIZE + 128]; -
Specify the binary array and length of the binary array from your binary object and send the binary payload to the Notecard in
CHUNK_SIZEfragments.note
The binary buffer requires additional overhead, so the buffer can be encoded in place. If you wish to know the exact requirements of your binary payload, you may use
NoteBinaryCodecMaxEncodedLength(). In this example, an arbitrary overhead was specified.const uint8_t * img_map = big_img_map; const size_t img_len = big_img_len; int i = 0; size_t bytes_left = img_len; while (bytes_left) { notecard.logDebugf("\nSending chunk %d, offset: %d...\n", i, i * CHUNK_SIZE); size_t bytes_to_send = bytes_left >= CHUNK_SIZE ? CHUNK_SIZE : bytes_left; memcpy(temp_buffer, img_map + i * CHUNK_SIZE, bytes_to_send); const char *err = NoteBinaryStoreTransmit((uint8_t *)temp_buffer, bytes_to_send, sizeof(temp_buffer), i * CHUNK_SIZE); if (!err) { bytes_left -= bytes_to_send; i++; } } -
Once the buffer is populated, continue on to Syncing Binary Data to Notehub.
Storing Binary Data with the card.binary APIs
As an alternative to using the SDK helpers, you can populate the binary buffer
directly with the card.binary and card.binary.put APIs. This path requires
you to handle COBS encoding and MD5 verification yourself.
-
Issue a
card.binaryrequest to the Notecard to verify the available space (max) is larger than the size of the binary payload you want to store.>{"req":"card.binary"}{"max":130554} -
Calculate the MD5 checksum of the binary payload.
-
COBS-encode the binary payload. The note-c library (the core C library that also powers
note-arduino) includes aNoteBinaryCodecEncodefunction to simplify this process. -
Calculate the length of the new COBS-encoded payload.
-
Append a newline character to the COBS-encoded payload (
\n). -
Send a
card.binary.putrequest to the Notecard with the MD5 checksum in thestatusargument and the length of the payload in thecobsargument.Use the
offsetargument if you are supplying multiple payloads in succession, where the currentoffsetis the index location of where the previous ended.{ "req": "card.binary.put", "cobs": 5, "status": "ce6fdef565eeecf14ab38d83643b922d" } -
At this point, the Notecard is in a state where it expects the next input to be binary data, not a JSON-formatted API request. Send it the COBS-encoded payload.
000011110110100101100101011010000111100100001010 -
Next, you can optionally send a
card.binaryrequest to check for errors and verify the binary data was properly saved to the Notecard by checking the MD5 checksum:>{"req":"card.binary"}{ "connected": true, "max": 130554, "status": "ce6fdef565eeecf14ab38d83643b922d", "length": 4, "cobs": 5 }If an error occurs on the transfer it will appear in the
errfield:{"err":"md5 mismatch","max":130554} -
Once the buffer is populated, continue on to Syncing Binary Data to Notehub.
Syncing Binary Data to Notehub
After storing binary data on the Notecard, you have two options for transmitting the buffer to Notehub:
- web.post, which sends the buffer to a Proxy for Notecard Web Requests route.
- note.add, which attaches the buffer to a Note and delivers it through a standard Notehub route.
Syncing Binary Data with web.post
-
Issue a
web.postrequest with the"binary":trueandcontent(the appropriate MIME type) arguments supplied. This tells Notecard to send all the data in the binary buffer to the specified proxy route in Notehub.note
Consult the Web Transactions docs for detailed information on using the
web.postAPI and proxy routes (noting the Notecard must be connected and incontinuousmode).>{ "req": "web.post", "route": "PostBinaryDataRoute", "binary": true, "verify": true, "content": "application/octet-stream" }{"result":200}Here is the equivalent request in C using the
note-arduinoSDK:if (J *req = NoteNewRequest("web.post")) { JAddStringToObject(req, "route", "PostImageRoute"); JAddStringToObject(req, "content", "images/jpeg"); JAddBoolToObject(req, "binary", true); JAddBoolToObject(req, "verify", true); if (!NoteRequest(req)) { NoteDebug("Error sending image\n"); delay(15000); } } -
After the
web.postis complete, reset the binary buffer on Notecard by sending acard.binaryrequest with the"delete":trueargument:>{ "req": "card.binary", "delete": true }{"max":130554}Or, with the
note-arduinoSDK, callNoteBinaryStoreReset():NoteBinaryStoreReset();
Syncing Binary Data with note.add
As an alternative to web.post, you can transmit the contents of the binary
buffer by attaching it to a Note using a
note.add request with
the "binary":true and "live":true arguments. This allows the binary payload
to flow through a standard Notefile sync and Notehub route.
When using "binary":true with note.add, the "live":true argument is
required. The live argument tells Notecard to bypass saving the Note to
flash, since the binary buffer itself is not stored in the Notefile on the
Notecard.
-
Issue a
note.addrequest with"binary":trueand"live":true, specifying the Notefile (for example,binary.qo) that your Notehub route is configured to filter on.>{ "req": "note.add", "file": "binary.qo", "binary": true, "live": true }{"total":1}Here is the equivalent request in C using the
note-arduinoSDK:if (J *req = NoteNewRequest("note.add")) { JAddStringToObject(req, "file", "binary.qo"); JAddBoolToObject(req, "binary", true); JAddBoolToObject(req, "live", true); NoteRequest(req); } -
When Notecard next syncs with Notehub, the contents of the binary buffer will be delivered as the payload of the resulting event on the specified Notefile. After the sync has completed, reset the binary buffer on the Notecard before storing the next payload by sending a
card.binaryrequest with the"delete":trueargument:>{ "req": "card.binary", "delete": true }{"max":130554}Or, with the
note-arduinoSDK, callNoteBinaryStoreReset():NoteBinaryStoreReset();
Receiving Large Binary Objects
Receiving a large binary object on Notecard involves two steps:
- Syncing the binary payload from Notehub into Notecard's reserved binary buffer.
- Reading that buffer from your host microcontroller.
These two steps are independent, so you can choose how to read the buffer
regardless of how it was populated. Most applications pair a web.get request
with the SDK helpers.
Syncing Binary Data from Notehub
Before reading, you first need to get the binary payload into the Notecard's
binary buffer by issuing a web.get request with the "binary":true argument.
-
Issue a
card.binaryrequest to the Notecard to verify the available space (max) is larger than the size of the binary payload you expect to download.>{"req":"card.binary"}{"max":130554} -
Send a
web.getrequest to the specified Notehub proxy route with the"binary":trueandcontent(the appropriate MIME type) arguments supplied, which requests that the response be placed in the Notecard's binary buffer.note
Consult the Web Transactions docs for detailed information on using the
web.getAPI and proxy routes (noting the Notecard must be connected and incontinuousmode).>{ "req": "web.get", "route": "GetBinaryDataRoute", "binary": true, "content": "application/octet-stream" }{ "result": 200, "length": 78179, "cobs": 78194, "body": {} }Here is the equivalent request in C using the
note-arduinoSDK:if (J *req = NoteNewRequest("web.get")) { JAddStringToObject(req, "route", "GetImageRoute"); JAddStringToObject(req, "content", "images/jpeg"); JAddBoolToObject(req, "binary", true); if (!NoteRequest(req)) { NoteDebug("Error receiving image\n"); } } -
Next, you can send a
card.binaryrequest to verify the binary data was properly saved to the Notecard, noting the MD5 checksum returned in thestatusfield is computed before COBS-encoding, and therefore does not include the\n.>{"req":"card.binary"}{ "connected": true, "max": 130554, "status": "c381abe19c96870db6d73fb4d670ef25", "length": 78179, "cobs": 78194 }
Reading Binary Data from Notecard
You have two paths to choose from when reading the binary buffer on your host.
They both use the
card.binary and
card.binary.get
APIs, but the note-arduino SDK provides helper methods to ease the process.
- Reading Binary Data with the note-arduino SDK (Recommended)
- Reading Binary Data with the card.binary APIs
Reading Binary Data with the note-arduino SDK
Due to the complexities of using the card.binary APIs directly, the
recommended path is to use the helper methods provided in the note-arduino
SDK.
The following Arduino examples demonstrate receiving binary data with
note-arduino:
basic binary data example
and
receiving a large binary payload in chunks.
-
Get the decoded length of the downloaded binary data via a call to
NoteBinaryStoreDecodedLength():uint32_t buffer_len = 0; NoteBinaryStoreDecodedLength(&buffer_len); -
Call
NoteBinaryStoreReceive()to verify and decode the binary data. The third and fourth arguments are the decoded-byte offset and decoded length to retrieve — pass0and the fullbuffer_lento fetch the entire payload.uint8_t * my_binary_data = (uint8_t *)malloc(buffer_len); NoteBinaryStoreReceive(my_binary_data, buffer_len, 0, buffer_len); -
Clear the binary buffer on the Notecard after the host has handled the binary data.
NoteBinaryStoreReset();
Reading Binary Data with the card.binary APIs
As an alternative to using the SDK helpers, you can read from the binary buffer
directly with the card.binary.get API. This path requires you to handle COBS
decoding yourself.
-
Send a
card.binary.getrequest to Notecard to fetch the binary data:>{"req":"card.binary.get"}{"status":"39f66921b9fb84a0400a1579e3dd3210"}Binary data will immediately follow this response. It can be fetched by reading until the
\ncharacter is encountered. -
COBS-decode the binary data. If you're working in C or C++, the note-c library (the core C library that also powers
note-arduino) includes aNoteBinaryCodecDecodefunction to simplify this process. -
After successfully retrieving the binary data, clear the binary buffer on the Notecard.
>{"req":"card.binary", "delete":true}{"max":130554}Or, with the
note-arduinoSDK, callNoteBinaryStoreReset():NoteBinaryStoreReset();
Binary Uploads with Web APIs
Using Notecard's
binary storage area is the recommended path
for most binary uploads. As an alternative, the web.post API accepts
base64-encoded payload fragments that Notehub reassembles before invoking
your route, delivering a single payload to your cloud endpoint.
You may opt to utilize this alternative binary data upload path when your
payload exceeds the binary buffer on the Notecard. The card.binary path is
capped at the Notecard's reserved binary area (i.e. max in the card.binary
response, typically ~100KB). Larger payloads require multiple buffer flushes and
reassembly on your cloud endpoint, while fragment uploads let Notehub handle
reassembly before routing.
There are some tradeoffs to be aware of when using this method:
- Base64 encoding adds ~33% bandwidth overhead per fragment compared to the
raw binary sent by the
card.binarypath. - Your host must manage fragment sizing, offsets, and per-fragment MD5s manually.
- This is a synchronous path as the Notecard must be connected and in
continuousmode, the same as the otherweb.*approaches. - The maximum recommended size of each fragment depends on the type and quality of your network connection. A safe range for most scenarios is 4–8 KB.
Sending Binary Fragments
Your host will split the binary payload into fragments and send them in
successive web.post requests. Each request must set the
"content": "application/octet-stream" argument and include the following
additional arguments so Notehub can verify each fragment and place it correctly
in the reassembled payload:
total- The total size of the reassembled payload, in raw (pre-base64) bytes.offset- The byte offset of this fragment within the reassembled payload, in raw (pre-base64) bytes.status- A 32-character hex-encoded MD5 sum of the fragment's bytes, used by Notehub to verify each fragment on receipt.verify- Set totrueto request verification from Notehub once the fragment is received. Automatically set totruewhenstatusis supplied.
-
Send the first fragment of your payload with
offset: 0. The example below shows the first fragment of an 8191-byte payload:{ "req": "web.post", "route": "SensorService", "content": "application/octet-stream", "payload": "<base64-encoded first 600 raw bytes>", "status": "<hex-encoded md5 of those 600 bytes>", "offset": 0, "total": 8191 }J *req = NoteNewRequest("web.post"); JAddStringToObject(req, "route", "SensorService"); JAddStringToObject(req, "content", "application/octet-stream"); JAddStringToObject(req, "payload", "<base64-encoded first 600 raw bytes>"); JAddStringToObject(req, "status", "<hex-encoded md5 of those 600 bytes>"); JAddNumberToObject(req, "offset", 0); JAddNumberToObject(req, "total", 8191); NoteRequest(req);req = {"req": "web.post"} req["route"] = "SensorService" req["content"] = "application/octet-stream" req["payload"] = "<base64-encoded first 600 raw bytes>" req["status"] = "<hex-encoded md5 of those 600 bytes>" req["offset"] = 0 req["total"] = 8191 rsp = card.Transaction(req) -
Send each subsequent fragment, advancing
offsetby the raw byte count of the prior fragment. For example, after sending 600 bytes, the next fragment usesoffset: 600:{ "req": "web.post", "route": "SensorService", "content": "application/octet-stream", "payload": "<base64-encoded next 600 raw bytes>", "status": "<hex-encoded md5 of those 600 bytes>", "offset": 600, "total": 8191 }J *req = NoteNewRequest("web.post"); JAddStringToObject(req, "route", "SensorService"); JAddStringToObject(req, "content", "application/octet-stream"); JAddStringToObject(req, "payload", "<base64-encoded next 600 raw bytes>"); JAddStringToObject(req, "status", "<hex-encoded md5 of those 600 bytes>"); JAddNumberToObject(req, "offset", 600); JAddNumberToObject(req, "total", 8191); NoteRequest(req);req = {"req": "web.post"} req["route"] = "SensorService" req["content"] = "application/octet-stream" req["payload"] = "<base64-encoded next 600 raw bytes>" req["status"] = "<hex-encoded md5 of those 600 bytes>" req["offset"] = 600 req["total"] = 8191 rsp = card.Transaction(req) -
Continue sending fragments until the sum of fragment sizes reaches
total. When the final fragment arrives, Notehub reassembles the complete payload, invokes the proxy route, and returns the route's HTTP response to the Notecard:>{ "req": "web.post", "route": "SensorService", "content": "application/octet-stream", "payload": "<base64-encoded final fragment>", "status": "<hex-encoded md5 of final fragment>", "offset": 7800, "total": 8191 }{"result":200}If a fragment fails MD5 verification, Notehub returns an
errfield in the response so the host can retransmit that fragment.