Machine Learning with Swan & Edge Impulse

The Swan from Blues Wireless is a powerful MCU. Built around a 120 MHz Arm Cortex-M4 from STMicroelectronics with 2 KB of Flash and 640 KB of RAM, it's the perfect board for edge machine learning models. In this guide, you'll learn how to build and run ML models using the Swan and Edge Impulse, a leading development platform for edge machine learning.

note

You can find the complete source for this guide, including the accelerometer data collector sketch and edge ML model and classification sketch, in this GitHub repo.

Before You Start

This tutorial and the associated sample code provide detailed instructions for building an edge ML application using the Swan and Edge Impulse. The instructions below detail the creation of a gesture classification model using an accelerometer, but can be adapted to fit any sensor or ML application.

If you wish to follow along with this guide, you'll need the following:

• A Swan MCU.
• A triple axis accelerometer. We used the SparkFun Triple Axis Accelerometer Breakout for this example.

On the software side, you'll also need to create an account and new project with Edge Impulse and install the following:

• Node.js v12 or later.
• The edge-impulse-cli, which you can install with the following command from a terminal:

COPY
npm install -g edge-impulse-cli

Finally, you'll need to install the Arduino library from the Arduino Library Manager. From the Tools > Manage Libraries... menu, search for SparkFun MMA8452Q and click "Install". Make sure to install the library for your specific sensor if you're using something different.

Forwarding Data to Edge Impulse

For this example, we will create a simple classification model with an accelerometer designed to analyze movement over a brief period of time (2 seconds) and infer how the motion correlates to one of the following four states:

1. Idle (no motion)
2. Circle
3. Slash
4. An up-and-down motion in the shape of the letter "W"

The first step in building any ML model is data acquisition and labeling, and Edge Impulse makes this simple with the data forwarder and Edge Impulse Studio. To get started collecting data from your Swan and accelerometer, you'll load a program to the device that streams X, Y, and Z values from the accelerometer to the serial port on the Swan. Edge Impulse's data forwarder can then read the output of your device and forward it to Edge Impulse studio, where you can label the data and build up a dataset for training.

Open the Arduino IDE and under the File > Examples menu, open the Example2_RawDataReading sketch for the SparkFun MMA8452Q Accelerometer.

After initializing the accelerometer, this example sketch will output raw accelerometer data to the serial console.

COPY
if (accel.available()) {      // Wait for new data from accelerometer
  // Raw of acceleration of x, y, and z directions
  Serial.print(accel.getX());
  Serial.print("\t");
  Serial.print(accel.getY());
  Serial.print("\t");
  Serial.print(accel.getZ());
  Serial.println();
}

To run this program on your Swan, click the Upload button in the Arduino IDE. Once running, you can connect to the Arduino Serial monitor to verify that the program is running and outputting a stream of accelerometer values.

note

If you've not yet configured your Swan, visit the quick start for instructions.

Once your Swan is emitting accelerometer values, the next step is to capture readings while performing the gestures you want to classify and forward these to Edge Impulse Studio. To start the forwarder, make sure you've closed the Arduino Serial Monitor and run the following from a terminal:

COPY
edge-impulse-data-forwarder

Follow the prompts to log into your Edge Impulse account, select a project, and assign names to the X, Y, and Z values from your accelerometer.

Once instructed to do so, use the URL provided by the forwarder to open Edge Impulse Studio and start capturing data.

In the "Record new data" form, make sure your Swan is selected, set a label for the gesture you plan to perform, and set the sample length to 2000 ms (2 seconds). Then, click "Start sampling".

At this point, Edge Impulse Studio will send a message to the forwarder running on your computer and instruct it to capture a two second sample from the accelerometer after a brief delay. Once the capture starts, perform a gesture you wish to classify.

After the sample is captured, it's uploaded to Edge Impulse Studio, where you can see a visual view of the accelerometer data captured by the forwarder.

Now comes the fun part! To get a solid cross-section of data for your model, you'll want to repeat this process several times for each gesture you want to classify in your application. The more data you capture, the better your model will be, so take the time to record at least a few minutes worth of data for each gesture.

Build and Train an ML Model

Once you've captured enough data from the Swan, you're ready to move on to designing and building your model. In the left-side menu of Edge Impulse Studio click "Create impulse" under the Impulse Design menu. Based on the data you collected, Edge Impulse Studio will recognize that your source data set is a time series and will make some recommendations for the window size and increase to use, which you are free to adjust as you test and iterate on the model.

Next, click on the "Add a processing block" button and add a "Spectral Analysis" block.

Then, click the "Add a learning block" button and select the "Classification (Keras)" block.

Finally, click "Save Impulse".

On the next screen, Spectral features, you can adjust and tune parameters if you wish, or keep the defaults. Click "Save parameters" when done and you'll be taken to the "Generate features" screen.

Click the "Generate features" button and, once the job completes, you'll get some estimates of on-device inferencing performance and a feature explorer that shows how your gestures are clustering on their x, y, and z axes.

You can pan, zoom and scroll around in that view to drill into your sensor data.

Once you're done playing with the feature explorer, click on the "NN Classifier" item in the left nav, and adjust the number of training cycles to 300 and learning rate to 0.00005. Then, click "Start training" and grab a cup of coffee while Edge Impulse trains your model.

After training is complete, Edge Impulse Studio will display the performance of the model, a confusion matrix, feature explorer, and on-device performance details.

Test the Model

Before running the model on the Swan, you can use the data forwarder to perform live classification against the model you've just trained. First, make sure the Swan is connected to your computer and start the Edge Impulse data forwarder.

COPY
edge-impulse-data-forwarder

Next, in Edge Impulse Studio, click the "Live classification" menu item in the left nav. Make sure your device is selected and click "Start sampling."

After the initialization delay, perform the gesture you want to classify. Once captured from the device, Edge Impulse runs the raw data against the model and displays the classification results. Using the UI, you can set the expected outcome and even move results into your training set for model refinement.

Deploy the Model to Swan

Once you're happy with the result, it's time to deploy the model to your device! Click on the Deployment item in the left nav and then click the "Arduino library" button.

Under the "Select optimizations" section, you can select either a quantized model or unoptimized, depending on your needs. The Swan has plenty of RAM and Flash, so you can choose the unoptimized model if it provides better performance without sacrificing too much speed.

Once you've selected a model, click "Build." Edge Impulse will build a model package and deliver a Zip file for you to download.

Next, before including the download into the Arduino IDE, we need to make a minor tweak to the Edge Impulse SDK to work with our STM32-based Swan.

Unzip the file you downloaded from Edge Impulse Studio and open the config.hpp file under src/edge-impulse-sdk/dsp in a text editor. Then add the following two lines around line 25:

COPY
#define EIDSP_USE_CMSIS_DSP             1
#define EIDSP_LOAD_CMSIS_DSP_SOURCES    1

Next, you'll need to include this folder as a library in the Arduino IDE so that your program can include the model and sdk. Open a new Arduino Sketch, click the Sketch menu then Include Library > Add .ZIP Library...

Navigate to the unzipped folder you just modified, select it, and click the "choose" button.

Now, head back to Sketch > Include Library, and under Contributed libraries choose the name of the folder you selected. This will add the appropriate include statement to your sketch.

With the library installed, you're ready to perform edge inferencing with the Swan. You can clone this example sketch and customize it for your sensor, but the important piece is to read from the sensor and run the Edge Impulse classifier with the captured data.

COPY
int16_t rawReadingBuf[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE] = { 0 };
float readingBuf[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE];

for (size_t ix = 0; ix < EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE; ix += 3) {
    uint64_t next_tick = micros() + (EI_CLASSIFIER_INTERVAL_MS * 1000);

    rawReadingBuf[ix] = accel.getX();
    rawReadingBuf[ix + 1] = accel.getY();
    rawReadingBuf[ix + 2] = accel.getZ();

    delayMicroseconds(next_tick - micros());
}

numpy::int16_to_float(rawReadingBuf, readingBuf, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE);
signal_t signal;
int err = numpy::signal_from_buffer(readingBuf, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, &signal);
if (err != 0) {
    ei_printf("Failed to create signal from buffer (%d)\n", err);
    return;
}

ei_impulse_result_t result = { 0 };

err = run_classifier(&signal, &result, debug_nn);
if (err != EI_IMPULSE_OK) {
    ei_printf("ERR: Failed to run classifier (%d)\n", err);
    return;
}

ei_printf("(DSP: %d ms., Classification: %d ms., Anomaly: %d ms.)\n",
    result.timing.dsp, result.timing.classification, result.timing.anomaly);
uint8_t predictionLabel = 0;
for(size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
    Serial.print("    ");
    Serial.print(result.classification[ix].label);
    Serial.print(": ");
    Serial.println(result.classification[ix].value);

    if(result.classification[ix].value > result.classification[predictionLabel].value)
        predictionLabel = ix;
}

Serial.print("\nPrediction: ");
Serial.println(result.classification[predictionLabel].label);

The final step is to upload the program to your Swan, open the Arduino Serial Monitor, and perform some gestures. The sample above performs continuous classification every few seconds, so you can expect to see a steady stream of output in the monitor.

Congratulations! You've built an edge ML model with Edge Impulse and the Swan!

Additional Resources

• Edge Impulse Docs
• Swan Quickstart Guide