Using Serial Logs Effectively

Blues University is an ongoing series of articles geared towards beginner- and intermediate-level developers who are looking to expand their knowledge of embedded development and the IoT.

As computing resources are scarce in embedded systems, effective software monitoring and debugging aren't just conveniences — they're lifelines. Often the most reliable witnesses to system events are serial logs.

Serial logs are a stream of text or binary messages sent over a serial communication interface — a real-time account of events on the system as they unfold. Unlike other verbose logging mechanisms, serial logs are resource-efficient, making them a perfect fit for the constrained environment of embedded systems.

This article explores the importance of serial logs in debugging and monitoring embedded systems.

What is a Serial Log?

A serial log is a sequence of messages an embedded system emits over a serial communication interface (UART or USB). Unlike parallel communication, serial communication transmits data sequentially, one bit at a time. In embedded systems, serial logs serve as a practical channel to capture and convey real-time data about the system's operations and interactions.

Serial logs differ primarily from other types of logs in their simplicity and low overhead. Here are a few distinctions:

• Resource efficiency: Serial logging requires minimal system resources compared to other logging mechanisms — a crucial advantage in resource-constrained embedded environments.
• Real-time data capture: Serial logs provide real-time or near real-time insights into system operations, which is critical for timely troubleshooting.
• Simplicity: The setup and configuration of serial logging are typically simpler than other logging mechanisms, making it a quick and straightforward solution for many embedded systems.

Understanding Baud Rate

Baud rate — the number of signal units per second — measures the speed of data transmission in serial communication, or how fast the data traverses a serial line. Note that each signal unit can represent more than one bit of data, so the baud rate doesn't always equal the bit rate.

The baud rate ensures that a message's sender and the receiver are in sync for smooth communication. It directly affects how quickly data is transmitted over a communication channel. A higher baud rate means quick data transmission, while a lower baud rate means slower transmission.

This speed directly affects the quality and readability of the logs. If the baud rate is mismatched, logs may appear scrambled or incomplete — or may not appear at all. However, a higher baud rate isn't always preferable. It could introduce errors, especially over longer distances or in noisy environments, as the system has less time to distinguish each bit from the background noise.

Common Baud Rates and Use Cases

There are various standardized baud rates. Two of the most common are:

1. 9600: The default for many devices, this is suitable for most applications where the quantity of data is moderate and transmission occurs over short distances.
2. 115200: This is a high-speed rate frequently used in situations requiring rapid data transfers, such as flashing firmware or sending large chunks of data over short distances.

Other baud rates include 300, 1200, 2400, 4800, 19200, 38400, and even higher. The choice of baud rate often depends on the project's specific requirements, the distance of communication, noise in the environment, and the device's capabilities.

Matching the baud rate on both ends is paramount. Data corruption occurs if the transmitting device sends data at a different baud rate than the receiving device expects. This is because the receiving end will misinterpret the incoming bits. For instance, if the sender is transmitting at 115200 to a receiver set at 9600, the receiver will read data more slowly than it's receiving it, causing a communication breakdown.

Writing to the Serial Log

Many popular platforms offer built-in libraries to facilitate serial communication. One prime example is Arduino, which comes with its own Serial library. This library abstracts the underlying hardware interactions, providing developers with user-friendly functions for handling serial communication efficiently.

Initializing Serial Communication

You must initialize the serial port before you can write or read from it. In Arduino/C, you would use the Serial.begin() function:

void setup() {
  Serial.begin(9600); // Initializes serial communication at a baud rate of 9600
}

This snippet initializes the serial communication with a baud rate of 9600. As discussed earlier, you can choose different baud rates based on your applications' needs.

Writing Messages, Warnings, and Errors to the Log

After initializing serial communication, you can use various functions from the Serial library to write messages to the log:

Serial.println("Temperature: "); // Print a message
Serial.println("WARN: Low battery!"); // Print a warning
Serial.println("ERROR: Sensor not detected!"); // Print an error

Formatting and Structuring Logs for Clarity and Ease of Parsing

Structure and clarity are essential when dealing with logs. Machines can easily parse well-structured logs, and humans can quickly understand them. Here are some tips:

Use Delimiters:

Use delimiters like commas, semicolons, or pipes to separate data fields:

Serial.print("Temperature,");
Serial.println(25);

Key-Value Pairs:

Use key-value pairs for clarity:

Serial.print("Status=OK|Battery=");
Serial.println(95);

Timestamps:

Timestamps are incredibly useful for understanding the sequence of events:

Serial.print("Timestamp=");
Serial.print(millis());
Serial.print("|Message=");
Serial.println("System Started");

Whichever format you choose, ensure consistency. This makes it easier to develop tools to parse and analyze logs later.

Viewing the Serial Log: What Are My Options?

Once you've instrumented your embedded program with serial logging, the next step is to view these logs. Available tools include standalone software applications, tools integrated into development environments, and proprietary hardware. Let's explore some of the options.

Integrated Tools in Development Environments

Arduino IDE's Serial Monitor

Arduino IDE's Serial Monitor offers a straightforward interface for viewing serial outputs. It provides options for different baud rates and allows sending data back to the host.

• Pros: It's directly integrated with the development environment, so there's no need for additional software.

• Cons: Due to its limited features compared to standalone applications, it's only suitable for basic reading and writing.

PlatformIO in VSCode

Apart from providing a platform for embedded development, PlatformIO offers a serial monitor that's more feature-rich than Arduino IDE's Serial Monitor.

• Pros: It's integrated into the powerful VSCode editor and offers more advanced features than the Arduino IDE.

• Cons: Beginners may find the extensive features overwhelming.

Standalone Software

PuTTY (Windows)

While commonly known as a Secure Shell (SSH) or Telnet client, PuTTY also supports serial communication.

• Pros: It's lightweight, versatile, and supports logging to a file directly.

• Cons: The user interface isn't intuitive. Its developers designed it primarily for network protocols, so not all features are helpful for serial communication.

CoolTerm (macOS)

The developers of CoolTerm designed it specifically for serial port communication. It supports capturing and sending data and offers options like request to send (RTS) and clear to send (CTS) handshaking.

• Pros: It has a user-friendly interface tailored for serial communication and supports multiple simultaneous connections.

• Cons: It's exclusive to macOS.

Minicom (Linux)

Minicom is a text-based serial port communications program, making it suitable for all Linux distributions, even those without a graphical user interface (GUI).

• Pros: It's lightweight, very stable, and perfect for systems without a GUI.

• Cons: It isn't intuitive for users unfamiliar with terminal-based applications and is less user-friendly than its GUI counterparts.

These options balance usability, features, and platform compatibility. Many other software tools are available, like Tera Term, RealTerm, and HyperTerminal. While each has its merits, the best tool depends on your specific needs, the platform you're using, and personal preference. Whether you prioritize a rich feature set, ease of use, or tight integration with your development environment, there's a perfect tool for viewing your serial logs.

Hardware Tools

USB-to-Serial Converters

These small hardware dongles or cables allow a computer with a USB interface to connect and communicate with devices using a serial communication protocol. They often bridge the gap between modern computers (which might not have serial ports) and older equipment or embedded devices that use serial communication.

A note on logic levels: One vital consideration when using USB-to-serial converters is logic level compatibility. Serial devices may operate at different voltage levels — standard voltages are 5V and 3.3V. Connecting a 5V logic level device to a 3.3V logic level system (or vice-versa) can damage components. Always check and confirm the voltage compatibility of devices when interfacing them.

Logic Analyzers

Logic analyzers are advanced tools for capturing, decoding, and analyzing digital signals. While you can use them for various digital signal analyses, they're especially beneficial for capturing serial communication, especially when dealing with timing problems, data integrity issues, or analyzing communication between multiple devices.

• Pros: They visually represent serial data, allowing for an in-depth analysis. They're versatile and can work with multiple protocols beyond serial communication, including Inter-Integrated Circuit (I2C), serial peripheral interface (SPI), and more.

• Cons: They're more expensive than simple USB-to-serial converters and may have a steeper learning curve for those unfamiliar with digital signal analysis.

Logging in Production

Serial logging is an invaluable tool during the development phase. However, should you keep this logging turned on for production?

Pros of Keeping Logs in Production

• Monitoring system health: Logging in production allows you to monitor the system's health continuously. By analyzing logs, you can gain insights into system temperatures, memory usage, and other critical metrics, ensuring the system operates within its intended parameters.

• Diagnosing field issues: Even after rigorous testing, some problems only manifest in real-world conditions. Having logs in production allows for quicker diagnostics and resolution when these unexpected issues crop up.

• Gathering usage statistics: Logs provide valuable data on how end users interact with a device. By analyzing these statistics, developers and product managers can make better decisions on future product features, enhancements, or optimizations.

Cons of Keeping Logs in Production

• Performance overhead: Logging, especially verbose logging, introduces computational overhead. This added processing can slightly degrade performance, affecting user experience or the system's real-time responsiveness.

• Potential security risks: Logs may sometimes unintentionally contain sensitive information. If not appropriately secured, they could introduce a vulnerability, exposing exploitable user data or system details.

• Electrically erasable programmable read-only memory (EEPROM) wear-out: EEPROM has a limited number of write cycles, typically in the range of 100,000 to 1 million cycles per cell. Constant logging can rapidly exhaust these write cycles, leading to storage failure.

• NAND or embedded Multi-Media Card (eMMC): Devices that rely on NAND flash (like eMMC modules) also have limited write cycles. While these devices deploy wear leveling — distributing write and erase cycles evenly across the memory cells — constant logging can expedite wear-out.

Best Practices for Effective Serial Logging

When done right, serial logging is an invaluable tool for developers. To maximize the benefits, follow these best practices to ensure clarity, usability, and efficiency.

Structure Logs for Clarity

Timestamping: Embedding a timestamp in every log entry provides context, allowing you to trace back events and understand the sequence in which they happened.

Serial.print(millis()); // Using Arduino's millis() for a simple timestamp
Serial.println(": System initialized.");

Categorization: It's important to distinguish among log levels using keywords like INFO, WARN, and ERROR. This categorization helps you scan filter logs quickly.

Serial.println("INFO: System started.");
Serial.println("WARN: Low battery.");

Consistent formatting: Keeping a consistent format across logs helps read and parse the logs programmatically. Decide on a format and stick to it, whether key-value pairs, JavaScript Object Notation (JSON)-style entries, or plain text.

Avoid Excessive Logging

Every log entry has a computational and storage cost, so it's vital to strike a balance between verbosity and usefulness. Log what's necessary, but avoid logging every tiny detail, especially in time-critical code sections. Consider implementing different log levels (for example, DEBUG, INFO, ERROR) and use them judiciously. This lets you filter out the noise and focus on what's important.

Periodically Review and Refine Log Messages

As your system evolves, specific log entries may become obsolete or redundant. Review the log messages regularly to ensure they remain relevant. Doing so not only keeps the logs clean but ensures you're capturing all the necessary information from newer parts or functionalities of the system.

Use the NDEBUG Switch for Development and Production

A standard pattern in software development is to use conditional compilation, such as the NDEBUG switch, to enable or disable logging based on the environment.

#define NDEBUG  // Disable assertions
#include <assert.h>

void setup() {
    Serial.begin(115200);
    
    int x = 5;
    assert(x == 10);  // This assertion will be ignored if NDEBUG is defined
    
    Serial.println("Program running...");
}

By defining or not defining NDEBUG, you can easily control the log verbosity. Typically, NDEBUG isn't defined for development, allowing for detailed logs. For production builds, defining NDEBUG can suppress these detailed logs, optimizing performance.

Conclusion

Embedded systems often operate in environments that make direct debugging challenging. In such scenarios, serial logs provide vital clues about the system's health, behavior, and potential issues. Properly structured and effectively managed logs can transform seemingly random events into discernible patterns, making problem-solving less daunting.

Blues University - Next Steps

This article if part of a broader series where you can dig deeper into each aspect of embedded development. To embark on this journey, be sure to follow Blues University, where you can explore and contribute to shaping the future of IoT.

If you're new to embedded development and the IoT, the best place to get started is with the Blues Starter Kit for Cell + WiFi and then join our community on Discourse. 💙