homechevron_rightDocschevron_rightGuides & Tutorialschevron_rightNotecard Guideschevron_rightSending and Receiving Large Binary Objects

Sending and Receiving Large Binary Objects

While the Notecard is designed to be a low-bandwidth cellular device, it is also possible to sync large binary payloads with the cloud.

Starting with Notecard firmware v5.3.1, this is accomplished by storing raw binary data in a reserved area on the Notecard, and then having the Notecard send that large block directly to Notehub. Likewise, the Notecard and Notehub can work together to get a binary payload from a remote endpoint and save it to the reserved area on the Notecard.

Alternatively, if your Notecard is using a firmware version >= 3.2.1, you may continue to use the Notecard's Web APIs.

note

In your app design, it's safe to assume the maximum space available for data in the binary storage area on the Notecard is 100KB. The exact available space (in bytes) is returned in the max field in response to a card.binary request.

If the total size of the binary data you are sending is > than max, you will need to "flush" the storage with the appropriate web.post request each time that limit is reached (see examples below), and then reassemble the binary data after it has been routed to your cloud.

Sending Large Binary Objects

The card.binary and card.binary.put APIs provide the building blocks you need to store binary data in the reserved binary storage area on the Notecard. A subsequent web.post request with the appropriate arguments (see examples below) will consume the binary buffer and send the data to a specified Notehub route.

You have two paths to choose from when implementing a binary data transfer to the Notecard. They both use the card.binary APIs, but the note-arduino SDK v1.5.0+ provides helper methods to ease the process.

  1. Sending Binary Data with the note-arduino SDK (Recommended)
  2. Sending Binary Data with the card.binary APIs

Sending Binary Data with the note-arduino SDK

Due to the complexities of using the card.binary APIs directly, the recommended path is to use the helper methods provided in the note-arduino SDK.

note

The following Arduino examples demonstrate an alternative way of sending binary data with note-arduino and a note.add request: basic binary data example and sending a large binary payload in chunks .

Sending a Single Binary Fragment

  1. Define the binary data to send, use the Notecard::binaryTransmit method to store the data in the reserved binary space on the Notecard, and send a web.postrequest to transmit all of the binary data in the buffer to the specified Notehub proxy route.

    char buff[25] = "Hello World";
Notecard::binaryTransmit((uint8_t *) buff, 10, sizeof(buff), false);

J *req = NoteNewRequest("web.post");
JAddStringToObject(req, "route", "MyRoute");
JAddBoolToObject(req, "binary", true);
NoteRequest(req);

Sending Multiple Binary Fragments

For larger binary payloads, you may need to split the payload into multiple smaller fragments and reassemble them on your cloud endpoint.

  1. Define the size of the binary payload fragments to send to the Notecard.

    #define CHUNK_SIZE 4096
uint8_t temp_buffer[CHUNK_SIZE + 128];

  2. Specify the binary array and length of the binary array from your binary object and end the binary payload to the Notecard in CHUNK_SIZE fragments.

    note

    The binary buffer requires additional overhead, so the buffer can be encoded in place. If you wish to know the exact requirements of your binary payload, you may use NoteBinaryRequiredTxBuffer(). In this example, an arbitrary overhead was specified.

    const uint8_t * img_map = big_img_map;
const size_t img_len = big_img_len;

int i = 0;
size_t bytes_left = img_len;
while (bytes_left) {
  notecard.logDebugf("\nSending chunk %d, offset: %d...\n", i, i * CHUNK_SIZE);
  size_t bytes_to_send = bytes_left >= CHUNK_SIZE ? CHUNK_SIZE : bytes_left;
  memcpy(temp_buffer, img_map + i * CHUNK_SIZE, bytes_to_send);

  const char *err = Notecard::binaryTransmit((uint8_t *)temp_buffer, bytes_to_send, CHUNK_SIZE, sizeof(temp_buffer));
     
  if (!err) {
    bytes_left -= bytes_to_send;
    i++;
  }
}

  3. Issue a web.post request to the Notecard, telling it to transmit all of the binary data in the buffer to the specified Notehub proxy route.

    if (J *req = NoteNewRequest("web.post")) {
  JAddStringToObject(req, "route", "PostImageRoute");
  JAddStringToObject(req, "content", "images/jpeg");
  JAddBoolToObject(req, "binary", true);
  JAddBoolToObject(req, "verify", true);

  if (!NoteRequest(req)) {
    NoteDebug("Error sending image\n");
    delay(15000);
  }
}

  4. Clear the binary buffer on the Notecard.

    NoteBinaryReset();

Sending Binary Data with the card.binary APIs

  1. Issue a card.binary request to the Notecard to verify the available space (max) is larger than the size of the binary payload you want to store.

    > 
    {"req":"card.binary"}
    {"max":130554}

  2. Calculate the MD5 checksum of the binary payload.

  3. COBS-encode the binary payload. The note-c library includes a NoteBinaryEncode method to simplify this process.

  4. Calculate the the length of the new COBS-encoded payload.

  5. Append a newline character to the the COBS-encoded payload (\n).

  6. Send a card.binary.put request to the Notecard with the MD5 checksum in the status argument and the length of the payload in the cobs argument.

    Use the offset argument if you are supplying multiple payloads in succession, where the current offset is the index location of where the previous ended.

    {
  "req": "card.binary.put",
  "cobs": 5,
  "status": "ce6fdef565eeecf14ab38d83643b922d"
}

  7. At this point, the Notecard is in a state where it expects the next input to be binary data, not a JSON-formatted API request. Send it the COBS-encoded payload.

    000011110110100101100101011010000111100100001010

  8. Next, you can optionally send a card.binary request to check for errors and verify the binary data was properly saved to the Notecard by checking the MD5 checksum:

    > 
    {"req":"card.binary"}
    {
 "connected": true,
 "max": 130554,
 "status": "ce6fdef565eeecf14ab38d83643b922d",
 "length": 4,
 "cobs": 5
}

    If an error occurs on the transfer it will appear in the err field:

    {"err":"md5 mismatch","max":130554}

  9. Assuming the entire COBS-encoded payload was successfully saved to the Notecard, perform a web.post request with the "binary":true and content (the appropriate MIME type) arguments supplied.

    This tells the Notecard to send all the data in the binary buffer to the specified proxy route in Notehub. Consult the Web Transactions docs for detailed information on using the web.post API and proxy routes (noting the Notecard must be connected and in continuous mode).

    > 
    {
  "req": "web.post",
  "route": "PostBinaryDataRoute",
  "binary": true,
  "content": "application/octet-stream"
}
    {"result":200}

  10. After the web.post is complete, reset the binary buffer on the Notecard using a card.binary call with the "delete":true argument.

    > 
    {
  "req": "card.binary",
  "delete": true
}
    {"max":130554}

Receiving Large Binary Objects

The process to receive binary data from a cloud endpoint is the same as sending binary data, just reversed.

Issue a web.get request with the "binary":true argument to ensure the response is placed in the Notecard's binary buffer. Then, use the card.binary.get API to fetch the binary data from the Notecard.

You have two paths to choose from when implementing binary data transfer on the Notecard. They both use the card.binary APIs, but the note-arduino SDK provides helper methods to ease the process.

  1. Receiving Binary Data with the note-arduino SDK (Recommended)
  2. Receiving Binary Data with the card.binary APIs

Receiving Binary Data with the note-arduino SDK

Due to the complexities of using the card.binary APIs directly, the recommended path is to use the helper methods provided in the note-arduino SDK.

note

The following Arduino examples demonstrate receiving binary data with note-arduino: basic binary data example and receiving a large binary payload in chunks .

  1. Send a web.get request to the specified Notehub proxy route with the "binary":true and content (the appropriate MIME type) arguments supplied, which requests that the response be placed in the Notecard's binary buffer.

    if (J *req = NoteNewRequest("web.get")) {
  JAddStringToObject(req, "route", "GetImageRoute");
  JAddStringToObject(req, "content", "images/jpeg");
  JAddBoolToObject(req, "binary", true);

  if (!NoteRequest(req)) {
    NoteDebug("Error receiving image\n");
  }
}

  2. Get the length of the downloaded binary data via a call to NoteBinaryRequiredRxBuffer():

    size_t buffer_len = 0;
NoteBinaryRequiredRxBuffer(&buffer_len);

  3. Call Notecard::binaryReceive to verify and decode the binary data:

    uint8_t * my_binary_data = (uint8_t *)malloc(buffer_len);
size_t received_byte_count = 0;
Notecard::binaryReceive(my_binary_data, buffer_len, &received_byte_count);

  4. Clear the binary buffer on the Notecard after the host has handled the binary data.

    NoteBinaryReset();

Receiving Binary Data with the card.binary APIs

  1. Issue a card.binary request to the Notecard to verify the available space (max) is larger than the size of the binary payload you expect to download.

    > 
    {"req":"card.binary"}
    {"max":130554}

  2. Send a web.get request with the "binary":true and content (the appropriate MIME type) arguments supplied, which requests that the response be placed in the Notecard's binary buffer.

    Consult the Web Transactions docs for detailed information on using the web.get API and proxy routes (noting the Notecard must be connected and in continuous mode).

    > 
    {
  "req": "web.get",
  "route": "GetBinaryDataRoute",
  "binary": true,
  "content": "application/octet-stream"
}
    {
 "result": 200,
 "length": 78179,
 "cobs": 78194,
 "body": {}
}

  3. Next, you can send a card.binary request to verify the binary data was properly saved to the Notecard, noting the MD5 checksum returned in the status field is computed before COBS-encoding, and therefore does not include the \n.

    > 
    {"req":"card.binary"}
    {
 "connected": true,
 "max": 130554,
 "status": "c381abe19c96870db6d73fb4d670ef25",
 "length": 78179,
 "cobs": 78194
}

  4. Send a card.binary.get request to the Notecard to fetch the binary data:

    > 
    {"req":"card.binary.get"}
    {"status":"39f66921b9fb84a0400a1579e3dd3210"}

    Binary data will immediately follow this response. It can be fetched by reading until the \n character is encountered.

  5. COBS-decode the binary data. The note-c library includes a NoteBinaryDecode method to simplify this process.

  6. After successfully retrieving the binary data, clear the binary buffer on the Notecard.

    > 
    {"req":"card.binary", "delete":true}
    {"max":130554}

Binary Uploads with Web APIs

While using the Notecard's binary storage area is the recommended path for binary uploads, if your Notecard is on firmware >= v3.2.1, you may use the web.post and web.put APIs to send binary payloads. These APIs support sending payload fragments and Notehub will assemble these fragments before routing to your cloud application.

note

The maximum recommended size of each fragment depends on the type and quality of your cellular connection. A safe range for most scenarios is 4-8KB.

To utilize this feature, your host will need to disassemble the payload into fragments, and send these in successive web.post requests. Your request must use arguments that Notehub uses to verify each fragment and understand where the individual fragments should be placed in the reassembled payload. Those fields are:

  • total - The total size of the entire payload, in bytes, across all fragments.
  • offset - For a given fragment, the number of bytes to offset from 0 when reassembling fragments in Notehub.
  • status - A 32-character hex-encoded MD5 sum of the payload fragment. Used by Notehub to perform verification upon receipt.
  • verify - Can optionally be set to true to request verification from Notehub once the payload fragment is received. Automatically set to true when status is supplied.
{
  "req": "web.post",
  "payload": "<600 base64-encoded bytes of payload>",
  "status": "<hex-encoded md5 sum of payload fragment>",
  "route": "SensorService",
  "offset": 600,
  "total": 8191
}
J *req = NoteNewRequest("web.post");
JAddStringToObject(req, "payload", "<600 base64-encoded bytes of payload>");
JAddStringToObject(req, "status", "<hex-encoded md5 sum of payload fragment>");
JAddStringToObject(req, "route", "SensorService");
JAddNumberToObject(req, "offset", 600);
JAddNumberToObject(req, "total", 8191);

NoteRequest(req);
req = {"req": "web.post"}
req["payload"] = "<600 base64-encoded bytes of payload>"
req["status"] = "<hex-encoded md5 sum of payload fragment>"
req["route"] = "SensorService"
req["offset"] = 600
req["total"] = 8191

rsp = card.Transaction(req)

Once the payload is received, it will be verified on Notehub and the verification result will be returned in the response from the Notecard.

Additional Resources