Updating ESP32 Host Firmware

warning

While still functional, the information provided in this guide has been superseded by Notecard Outboard Firmware Update.

The Notecard and Notehub provide developers with an over-the-air device firmware update (DFU) mechanism that can be used to update Notecard firmware, or the firmware of a connected microcontroller or microprocessor host. Provided the host has the ability to run a custom bootloader or operate across partitions, and the binary to be delivered to the host is smaller than 1.5 MB in size, developers can utilize Notehub to upload firmware binaries, and the Notecard to download and deliver those binaries to the host.

note

Host DFU is available to any processor type that has the ability to update itself from a binary. And while the interactions with the Notecard and Notehub, are consistent across processors, the actual firmware update process is host-dependent. This guide specifically covers the host DFU process using the ESP32, which leverages esp-idf libraries and the ESP32's built-in partition scheme to update its own firmware while operational. For a more processor-agnostic summary of the Host DFU process and the Notecard API requests involved, see Notecard API Requests for DFU.

A complete example of the ESP32 Host DFU process can be downloaded from the note-tutorials GitHub repository. The directory contains the 1.0.0.0 and 1.1.0.0 versions of projects covered in this guide.

Required Tools

To follow along with this guide, you'll need:

• A Notecard.
• A Notecarrier AF.
• An Adafruit HUZZAH32 Feather.
• A USB mini cable for the initial firmware update and debug console monitoring.
• The Arduino IDE with ESP32 Board Support. If you don't have board support configured, you can follow the instructions in this tutorial.

Preparing the Arduino IDE

When performing an ESP32 Host DFU, you'll want to make sure you're running at least version 1.8.13 of the Arduino IDE. The ESP32 self-updates through a partition scheme, which allows the device to swap to a secondary partition in order to update the first at runtime. After selecting an ESP32 board from the Tools > Board menu, set the Partition Scheme to "Default." The other built-in options reduce the secondary partition for large primary apps and cannot be used with Host DFU.

Next, it can be helpful to change where the Arduino IDE stores firmware binaries. By default, the Arduino IDE stores binaries in temporary directories that change each time you verify or upload firmware. You'll need access to these binaries so you can upload them to Notehub. To change this location, open the Arduino Preferences menu and take note of the location of the preferences.txt file.

Close the Arduino IDE. Then, open the preferences.txt file and add a build.path line to a location of your choosing:

build.path=/tmp/Arduino/binaries

Save the file, and reopen the Arduino IDE.

Overview of the Host DFU Process

At a high-level, the host firmware update process consists of the following steps:

1. A developer creates a new firmware revision, including a version number and optional metadata that can be extracted from the binary upon upload.
2. The administrator of the Project uploads the firmware binary to Notehub. Please note that the Notecard can only deliver host binaries up to 1.5MB in size.
3. Notehub extracts the version number, metadata, and adds the binary to the Project.
4. The administrator of the Project selects one or more Notecards to update, and queues a new firmware version for update.
5. When target Notecards sync with Notehub, they will identify the new host firmware and download it progressively in the background.
6. As the Notecard receives the host firmware, it places it into a special firmware storage area of flash. Periodically throughout this process, the host firmware issues dfu.status requests to the Notecard to determine the status of a firmware binary download.
7. Once the download has completed, the host will use hub.set to halt network communication and inform the Notecard that it needs to access new firmware.
8. Next, the host will issue dfu.get requests to load chunks of the firmware binary into its own memory.
9. After the host obtains the full binary, it will re-flash and restart.
10. Finally, the host will use hub.set to place the Notecard back into periodic or continuous mode.

warning

The maximum size of a host binary file is 1.5 MB for all Notecards, with the following exceptions:

1. The Notecard WiFi v2 has a maximum of 900KB.
2. The Notecard LoRa does not support OTA host or Notecard firmware updates.

Adding DFU Capabilities to Firmware

The first step in enabling Host DFU on a device is adding update capabilities to the baseline firmware, including:

• Adding metadata with the firmware version and name in a manner that Notehub can automatically extract upon upload.
• Implementing functionality to report the current firmware version to Notehub.
• Adding logic to check DFU status, manage the binary transfer, and update once a new binary has been downloaded to the Notecard.

note

Example project code has been omitted from the snippets below for clarity.

Adding Project Metadata

In order for Notehub to properly manage updates, the host firmware needs to provide metadata that Notehub can extract from an uploaded binary. The example project includes a number of definitions in the main .ino file that are used to serve version information to Notehub. At a minimum, the PRODUCT_DISPLAY_NAME, PRODUCT_FIRMWARE_ID, PRODUCT_MAJOR, PRODUCT_MINOR, and PRODUCT_PATCH should be set.

// C Helpers to convert a number to a string
#define STRINGIFY(x) STRINGIFY_(x)
#define STRINGIFY_(x) #x

// Definitions used by firmware update
#define PRODUCT_ORG_NAME      ""
#define PRODUCT_DISPLAY_NAME  "Notecard ESP32 DFU Example"
#define PRODUCT_FIRMWARE_ID   "notecard-esp32-dfu-example-v1"
#define PRODUCT_DESC          ""
#define PRODUCT_MAJOR         1
#define PRODUCT_MINOR         0
#define PRODUCT_PATCH         0
#define PRODUCT_BUILD         0
#define PRODUCT_BUILT         __DATE__ " " __TIME__
#define PRODUCT_BUILDER       ""
#define PRODUCT_VERSION       STRINGIFY(PRODUCT_MAJOR) "." STRINGIFY(PRODUCT_MINOR) "." STRINGIFY(PRODUCT_PATCH)

Then, at the end of the file, a FIRMWARE_VERSION string and associated functions are included. The FIRMWARE_VERSION string creates a JSON object that the device sends to Notehub to report its current version, and that Notehub can read automatically when a new binary is uploaded.

// This is a product configuration JSON structure that enables the Notehub to recognize this
// firmware when it's uploaded, to help keep track of versions and so we only ever download
// firmware builds that are appropriate for this device.
#define QUOTE(x) "\"" x "\""
#define FIRMWARE_VERSION_HEADER "firmware::info:"
#define FIRMWARE_VERSION FIRMWARE_VERSION_HEADER         \
    "{" QUOTE("org") ":" QUOTE(PRODUCT_ORG_NAME)         \
    "," QUOTE("product") ":" QUOTE(PRODUCT_DISPLAY_NAME) \
    "," QUOTE("description") ":" QUOTE(PRODUCT_DESC)     \
    "," QUOTE("firmware") ":" QUOTE(PRODUCT_FIRMWARE_ID) \
    "," QUOTE("version") ":" QUOTE(PRODUCT_VERSION)      \
    "," QUOTE("built") ":" QUOTE(PRODUCT_BUILT)          \
    "," QUOTE("ver_major") ":" STRINGIFY(PRODUCT_MAJOR)  \
    "," QUOTE("ver_minor") ":" STRINGIFY(PRODUCT_MINOR)  \
    "," QUOTE("ver_patch") ":" STRINGIFY(PRODUCT_PATCH)  \
    "," QUOTE("ver_build") ":" STRINGIFY(PRODUCT_BUILD)  \
    "," QUOTE("builder") ":" QUOTE(PRODUCT_BUILDER)      \
    "}"

const char *productVersion() {
    return ("Ver " PRODUCT_VERSION " " PRODUCT_BUILT);
}

// Return the firmware's version, which is both stored within the image and which is verified by DFU
const char *firmwareVersion() {
    return &FIRMWARE_VERSION[sizeof(FIRMWARE_VERSION_HEADER)-1];
}

Reporting Firmware Status to Notehub

After specifying version metadata, the example project next includes a dfu.status request in the setup function, just after the initial hub.set to configure the ProductUID, mode and sync periods. This request calls the firmwareVersion() function above to include the full firmware version JSON to in the next Notehub sync.

J *req = notecard.newRequest("dfu.status");
if (req != NULL) {
  JAddStringToObject(req, "version", firmwareVersion());
  notecard.sendRequest(req);
}

When flashed to a device, the output on the debug console looks like this:

{"req":"dfu.status","version":"{\"org\":\"\",\"product\":\"Notecard ESP32 DFU Example\",
  \"description\":\"\",\"firmware\":\"notecard-esp32-dfu-example-v1\",
  \"version\":\"1.0.0\",\"built\":\"Nov 11 2020 08:15:13\",\"ver_major\":1,
  \"ver_minor\":0,\"ver_patch\":0,\"ver_build\":0,\"builder\":\"\"}"}

{"on":true}

Implementing DFU Functionality

In addition to adding version information is in place, reporting the current version through dfu.status, an ESP32 Host must include functionality to poll the Notecard for firmware, transfer the firmware binary from the Notecard, and use esp-idf utilities to perform the firmware update. The dfu.cpp file in the example project directory contains all of the functionality needed and can be easily modified to be used with any ESP32 project. This section covers the Notecard-specific features of that file. ESP OTA features are out of scope for this guide, but you can learn more about them in the Espressif docs.

First, the example project performs a DFU partition check in the setup function by calling dfuShowPartitions() in dfu.cpp. This function verifies that the ESP32 device is configured with two partitions and logs the result to the debug console. If everything is configured correctly in the Arduino IDE (as referenced above), a message like the following appears in the debug console:

ESP32 PARTITION SCHEME (should be two partitions to support OTA)
   partition that should be 'app0' is 'app0' at 0x00010000 (1310720 bytes)
   partition that should be 'app1' is 'app1' at 0x00150000 (1310720 bytes)

Next, the application calls the dfuPoll() function on a periodic basis. In the example firmware, polling is implemented using button-presses from the Notecarrier AF's built-in B0 button. When the button is idle (meaning it has not been pressed), dfuPoll(false) is called, which ensures that DFU status checks happen at least hourly. Alternatively, the user can force a DFU status check by double-clicking the B0 button, which calls dfuPoll(true).

switch (buttonState) {
  case BUTTON_IDLE:
    dfuPoll(false); // Perform a routine check if an hour has passed
    return;
  case BUTTON_DOUBLEPRESS:
    digitalWrite(ledPin, HIGH);
    dfuPoll(true); // Force a check now
    digitalWrite(ledPin, LOW);
    return;
}

The dfuPoll() function contains the bulk of the functionality for interacting with the Notecard and performing firmware updates to the ESP32. Provided this function is called periodically, it will work, as-is, in any ESP32 application.

At a high-level, dfuPoll() performs the following actions:

1. Issues a dfu.status request to the Notecard and checks the mode field in the response to determine whether: a) a firmware image is being downloaded by the Notecard; b) a firmware image has been downloaded and is ready for installation; or, c) no new firmware is available.
2. Once a new firmware binary has been downloaded and any pending network activity completes, it issues a hub.set request and sets the mode to dfu, signaling to the Notecard that the host is ready to retrieve firmware.
3. Switches the ESP32 to its secondary partition to start the update.
4. Performs a series of dfu.get requests to obtain the host firmware binary in 8kb chunks. Each chunk is then CRC-validated and written to a partition handle object using the esp-idf esp_ota_write function.
5. After the entire binary is retrieved, the host issues another hub.set with a mode of dfu-completed to indicate that the Notecard can resume its normal operations.
6. Performs a final CRC check, comparing the CRC of the downloaded firmware to the value provided by Notehub in the initial dfu.status response. If the values match, the boot partition is updated with the update handle.
7. Issues a dfu.status request with the stop field set to true. Upon receipt, the Notecard removes the firmware binary from its internal storage.
8. Restarts the ESP32 and begins running the new firmware.

Once dfu.cpp is added to your project and dfuPoll() is implemented in your loop,

The result of adding dfu.cpp and dfuPoll() to the loop is the following in the debug console, indicating that the host is ready for future firmware updates:

{"req":"dfu.status"}
{"on":true}

dfu: no image is ready for firmware update

Creating a Firmware Revision

After the host firmware is configured for Host DFU, a revision to the project can be made. At a minimum, any revision should increment one of the PRODUCT_ version definitions for each update, in addition to modifying the functionality of the app.

For instance, in the example project the loop contains code that obtains the temperature and voltage from the Notecard and adds those values to a Notefile through a note.add request.

double temperature = 0;
J *rsp = notecard.requestAndResponse(notecard.newRequest("card.temp"));
if (rsp != NULL) {
  temperature = JGetNumber(rsp, "value");
  notecard.deleteResponse(rsp);
}

double voltage = 0;
rsp = notecard.requestAndResponse(notecard.newRequest("card.voltage"));
if (rsp != NULL) {
  voltage = JGetNumber(rsp, "value");
  notecard.deleteResponse(rsp);
}

J *req = notecard.newRequest("note.add");
if (req != NULL) {
  J *body = JCreateObject();
  if (body != NULL) {
    JAddNumberToObject(body, "voltage", voltage);
    JAddNumberToObject(body, "temp", temperature);
    JAddItemToObject(req, "body", body);
  }
  notecard.sendRequest(req);
}

In the example project, the next revision, adds another request to read from the Notecard's built-in accelerometer using card.motion and includes the orientation in the note.add request.

char orientation[20];
rsp = notecard.requestAndResponse(notecard.newRequest("card.motion"));
if (rsp != NULL) {
  char *current_orientation = JGetString(rsp, "status");
  strcpy(orientation, current_orientation);
  notecard.deleteResponse(rsp);
}

J *req = notecard.newRequest("note.add");
if (req != NULL) {
  J *body = JCreateObject();
  if (body != NULL) {
    JAddNumberToObject(body, "voltage", voltage);
    JAddNumberToObject(body, "temp", temperature);
    JAddStringToObject(body, "orientation", orientation);
    JAddItemToObject(req, "body", body);
  }
  notecard.sendRequest(req);
}

And the version is bumped to 1.1.0.0 by changing the PRODUCT_MINOR definition.

#define PRODUCT_MAJOR     1
#define PRODUCT_MINOR     1
#define PRODUCT_PATCH     0
#define PRODUCT_BUILD     0

When making ESP32 firmware changes for DFU, you'll click the Verify button in the Arduino IDE to compile your code and create a binary in the temporary location you specified earlier. Open that directory and look for the file with a bin extension. That's the file that should be uploaded to Notehub for DFU.

Initiating a Host DFU from Notehub

The process of initiating a Host DFU in Notehub consists of two steps:

1. Uploading a new firmware binary at the Project-level.
2. Selecting a Device, Devices or Fleet to update.

Adding a Firmware Binary

After you have a firmware binary, open your project in Notehub and navigate to the Settings > Firmware screen. Make sure the "Host Firmware" tab is selected and click on the "Upload firmware" link.

On the Upload screen, select the firmware file created by the Arduino IDE, and add optional Notes about the update, or any other required metadata.

Once you click Save, Notehub will extract the version information and other metadata and return you to the Project Firmware screen, which displays summary information for each of a Project's uploaded binaries.

Applying a Firmware Update to a Device or Fleet

After uploading new firmware to your Project, click on the Devices menu item or select a Fleet in the left navigation. Select the Device or Devices you wish to update and click on the Host Firmware tab. Then, click the Update menu item.

Click Proceed on the update modal.

Click the option button next to the firmware you wish to send to the Host and click Apply.

The Device screen will update to indicate that you've requested an update and will display the requested version.

On the next sync, the Notecard is be informed that a new host binary is available and will begin downloading it without interfering with its normal operations. During the download, the Notehub Device screen will update with download progress.

Once the binary is downloaded, the DFU Status changes to "Ready to install firmware."

Tracking DFU Progress

After the Notecard has completely downloaded the firmware binary, it will inform the host that a DFU image is ready on the next dfu.status request. If you are connected to the ESP32 over a debug console, you'll see a request and response like this:

{"req":"dfu.status"}
{"body":{"crc32":1299855394,"created":1605130978,"firmware":{
  "built":"Nov 11 2020 15:31:09","firmware":"notecard-esp32-dfu-example-v1",
  "product":"Notecard ESP32 DFU Example","ver_major":1,"ver_minor":1,
  "version":"1.1.0"},"info":{},"length":247472,
  "md5":"8ee84c3969b6e8708ea411f62edef4d2","modified":1605130978,
  "name":"esp32-dfu-v1.1.0.0.ino$20201111214258.bin",
  "notes":"Add Notecard orientation to periodic readings",
  "source":"esp32-dfu-v1.1.0.0.ino.bin","type":"firmware"},
  "status":"successfully downloaded","mode":"ready","on":true}

As the Notecard provides the binary to the host, you can monitor the result of each dfu.get request.

{"req":"dfu.get","offset":0,"length":8192}

{"body":{"crc32":1299855394,"created":1605130978,"payload":"[base64 Payload]"}}

Once the transfer completes and the update is applied, you'll see some additional log messages before the host reboots and the new firmware starts running:

dfu: successfully transferred offset:245760 len:1712 (crc:4d7a3822)

{"req":"hub.set","mode":"dfu-completed"}
{}

dfu: CRC32 of image: 4d7a3822
dfu: CRC32 of download: 4d7a3822

{"req":"dfu.status","stop":true}
{"status":"successfully downloaded","mode":"ready","on":true}

dfu: restart system

Any new functionality applied to your host will be available immediately upon restart. In the example project, the Notecard orientation was added to note.add requests, which you can verify by clicking the B0 button on the Notecarrier AF and viewing the request in the debug console.

{"req":"note.add","body":{"voltage":5.12748,"temp":30.875,"orientation":"face-up"}}

Finally, in Notehub, once a host DFU is complete for a device, the DFU Status updates to "Completed."

Additional Resources

• Host DFU Firmware Examples
• Host Firmware Updates Guide
• dfu Request API