The Anatomy of JSON and JSONata

Blues University is an ongoing series of articles geared towards beginner- and intermediate-level developers who are looking to expand their knowledge of embedded development and the IoT.

Data representation formats organize, store, and transmit information over the internet in a standardized way. These standardized formats pass data smoothly across platforms, reducing integration challenges, ensuring compatibility, and letting other computers and humans effortlessly parse the data.

They also make data storage and transmission more efficient, reducing the space required to store data and the bandwidth needed to transmit it. This efficiency is vital for big data and content streaming.

JavaScript Object Notation (JSON) is a popular, lightweight data representation format. This guide explores JSON in depth and demonstrates how to manipulate JSON data using the JSONata query and transformation language.

Consequently, you can simplify your data structures and transform data for the IoT when using Blues Notecard and Notehub!

What Is JSON?

Douglas Crockford developed JSON in the early 2000s as a text-based, lightweight data interchange format inspired by JavaScript's object literal notation. While JSON was initially a subset of JavaScript, it has evolved into a language-independent format for data exchange among multiple platforms and programming languages. JSON represents structured data based on the JavaScript object syntax. It's also widely popular for its human readability, simplicity, and versatility.

Before JSON, Extensible Markup Language (XML) dominated the scene. XML uses a set of rules to define its syntax and encloses data in tag-defined elements that can nest hierarchically.

JSON gained traction primarily in web development mainly due to the rise of Asynchronous JavaScript and XML (AJAX) in the mid-2000s. AJAX lets web pages retrieve and display data from a server without refreshing the entire page. JSON became the preferred data format for this asynchronous transmission due to its efficient and seamless integration.

Then, the rise of representational state transfer (REST) web services and web application programming interfaces (APIs) further boosted JSON's popularity. Many APIs began offering JSON for requests and responses, and most web APIs today use JSON for data transmission (e.g. the Blues Notehub API).

JSON Structure

JSON is text-based and represents data in key-value pairs. The keys are strings, and the values can take on many forms, including string, array, number, nested JSON object, boolean, or even null. This structure resembles a hashtable or dictionary in programming languages like Python.

Let's examine the following simple JSON object representing information about a person:

{
   "name":"Jane Doe",
   "age":31,
   "employed":false,
   "likes":[
      "movies",
      "music"
   ]
}

In this object, the keys are strings (like "name" or "age") while the values are a string ("Jane Doe"), number (31), boolean (false), or array (["movies", "music"]).

Why Is JSON Popular in the IoT?

Developers use JSON in the IoT for several compelling reasons:

1. Efficient and compact: Because JSON is lightweight and compact, it's a good option for resource-constrained devices with limited processing power, bandwidth, and memory. Its minimal overhead makes JSON suitable for IoT devices, ensuring efficient data transfer between them and the cloud or other endpoints.
2. Human readable: JSON has a straightforward syntax and readability, making it easier for developers to configure, understand, and debug data structures during development and testing.
3. Platform independent: Most modern programming languages support JSON natively, enabling IoT developers to generate and parse JSON data in any technology stack. This cross-compatibility makes it more straightforward to integrate IoT devices with a wide array of platforms and services, promoting interoperability.
4. Flexible: Unlike XML, JSON is schema-less. IoT ecosystems often involve diverse devices, protocols, and sensors, where data formats evolve and vary among applications and devices. JSON's flexibility enables dynamic data representation, accommodating multiple data types, attributes, and structures without the constraints of a rigid schema. This adaptability is vital for IoT systems that handle changing requirements and evolving data sources.
5. Widely adopted: IoT often relies on web-based communication technologies and protocols. Web browsers, servers, and APIs natively support JSON, so it's a natural fit. JSON is compatible with RESTful APIs and has a rich tool and library ecosystem.

Diving Deep: The Various JSON Data Types

JSON has six data types. String, number, boolean, and null are simple data types, while arrays and JSON objects are complex data types. Let's explore them all.

String

JSON strings are sequences of zero or more Unicode characters in double quotes. The strings can also contain the following backslash escape characters:

• \n — new line
• \r — carriage return
• \t — tab
• \b — backspace
• \f — form feed
• \" — double quote
• \\ — backslash
• \/ — forward slash
• \u — trailed by four hex digits

For example, a JSON string might look like the following:

{
  "name": "Jane Doe\n"
}

Best practices include always using double quotes for string values. Place escape characters within strings using the backslash if necessary. For example, your JSON should look like the following to use quotes inside a string:

{
  "sentence": "This is how you \"quote\" a word in a string"
}

Number

JSON numbers follow JavaScript's double-precision floating-point format. Numbers can be positive or negative integers, decimals, or exponents, like the following example:

{
  "number_0": 10,
  "number_1": -19999,
  "number_2": 0.00934,
  "number_3": 1.0e+12,
  "number_4": 9.5E-9
}

Best practices include not enclosing numbers in quotes as strings. They should be numeric values.

Beware of floating-point number inconsistencies when performing precise calculations on numbers with decimals. JSON is text-based, not number-based — converting a number to a string or vice versa can incorrectly truncate or round the number.

For example, you might send or receive a number slightly greater than a 32-bit float, then serialize or deserialize the data with a JSON library using a native 32-bit decimal float. This slight loss of precision could affect calculations in your application. Expressing numbers in binary form ensures your figures are precise.

Boolean

JSON boolean values can be true or false, like the following:

{
  "bool_1": true,
  "bool_2": false
}

Best practices include not enclosing boolean values in quotes since that would treat them as strings. JSON is case-sensitive, so use lowercase for booleans and be consistent.

Null

Null is a special JSON value. Use it when there is no value to assign to a key, like the following:

{
  "data": null
}

Best practices include not surrounding null in quotes. Use null when a value is intentionally missing, undefined, or unknown.

Array

JSON arrays are ordered lists of values separated by a comma and enclosed in square brackets, like the following:

{
  "numbers": [1,2,3,4,5],
  "colors": ["blue", "red", "purple"]
}

Best practices include ensuring consistent data types within arrays. Using multiple data types could cause ambiguity.

JSON Object

JSON objects are an unordered collection of key-value pairs enclosed in curly brackets, like the following:

{
  "person": {"name": "Robbie", "age": 28},
  "address": {"street": "123 Ice St", "city": "Lazytown"}
}

JSON objects can also contain zero or more key-value pairs, with multiple key-value pairs separated by a comma.

Best practices include using curly brackets to enclose key-value pairs within objects. Ensure the keys are strings enclosed in double quotes, followed by a colon and their corresponding values. Keys must be unique within an object.

Introducing JSONata

The growing popularity and dominance of JSON as a data interchange format — especially in web and API contexts — prompted Andrew Coleman and his IBM team to develop JSONata as an open-source project. They wanted to offer a versatile and efficient way to query and transform JSON data with features similar to XPath and XQuery for XML.

JSONata is a streamlined query and transformation language designed for working with JSON data. It provides a robust and expressive way to extract, filter, and manipulate JSON data structures using sophisticated queries expressed in an intuitive and compact notation.

This language excels at working with complex JSON data structures from web APIs and services. JSONata heeds functional programming principles, treating data transformations as a series of operations or functions applied to JSON data structures.

JSONata has become increasingly important in many industries and applications for simplifying complex data structures. It enables users to work with deeply nested JSON data effortlessly via a concise and robust syntax for navigation and extraction. It also reduces the time and complexity of manipulating data by transforming JSON data without extensive custom code. Finally, it enables interoperability since it's available in multiple programming languages and integrates with many tools and services in various environments.

Read more about how Blues uses JSONata to transform JSON for efficient IoT applications.

Using JSONata to Transform JSON

Let's use the online JSONata Exerciser to understand JSONata. The JSONata Exerciser lets you experiment with JSONata expressions with a JSON editor on the left for your JSON data and a JSONata expression editor on the top right for your JSONata expressions. The results of your expressions appear on the bottom right, like in the following screenshot:

Next, let's experiment with the JSONata Exerciser's default "Invoice" JSON data below:

{
   "Account":{
      "Account Name":"Firefly",
      "Order":[
         {
            "OrderID":"order103",
            "Product":[
               {
                  "Product Name":"Bowler Hat",
                  "ProductID":858383,
                  "SKU":"0406654608",
                  "Description":{
                     "Colour":"Purple",
                     "Width":300,
                     "Height":200,
                     "Depth":210,
                     "Weight":0.75
                  },
                  "Price":34.45,
                  "Quantity":2
               },
               {
                  "Product Name":"Trilby hat",
                  "ProductID":858236,
                  "SKU":"0406634348",
                  "Description":{
                     "Colour":"Orange",
                     "Width":300,
                     "Height":200,
                     "Depth":210,
                     "Weight":0.6
                  },
                  "Price":21.67,
                  "Quantity":1
               }
            ]
         },
         {
            "OrderID":"order104",
            "Product":[
               {
                  "Product Name":"Bowler Hat",
                  "ProductID":858383,
                  "SKU":"040657863",
                  "Description":{
                     "Colour":"Purple",
                     "Width":300,
                     "Height":200,
                     "Depth":210,
                     "Weight":0.75
                  },
                  "Price":34.45,
                  "Quantity":4
               },
               {
                  "ProductID":345664,
                  "SKU":"0406654603",
                  "Product Name":"Cloak",
                  "Description":{
                     "Colour":"Black",
                     "Width":30,
                     "Height":20,
                     "Depth":210,
                     "Weight":2
                  },
                  "Price":107.99,
                  "Quantity":1
               }
            ]
         }
      ]
   }
}

This data represents a sample JSON that an e-commerce web API query might return.

Basic Syntax

JSONata's basic syntax consists of expressions enclosed within curly brackets {}. JSONata expressions query and manipulate JSON data. They're enclosed within curly brackets and can be simple or complex operations, like below:

{ expression }

Path expressions navigate JSON data to access specific elements using dot notation (.), as shown below, or array elements using square brackets ([]). They don't need the curly brackets when only accessing data.

Key1.Key2      // returns the value of the nested Key2 inside Key1

The $ refers to the JSON data's root object.

Expressions

JSONata uses expressions comprising operators and functions for extracting and transforming JSON data. JSONata defines a location path syntax to extract values from JSON.

To extract the array of orders from the sample JSON data, specify its path expression in the JSONata expression editor:

Account.Order

This expression returns the data's array of Orders, like the following screenshot:

JSONata's path expression syntax navigates JSON data by following field and key names. A nested field has a dot (.) followed by its name. In the JSON data above, the key Address has an associated JSON object value with enclosed key-value pairs. Use the dot specifier and the desired value's key to access nested values.

You used the JSONata path expression Account.Order to get the array value of the nested Order key. Now, use the following expression to get the value of the nested key Account Name:

Account."Account Name"

The quotes signify that the Account Name key is two words but should be considered one string.

JSONata also lets you subscript an array to fetch the first element in the Order array:

Account.Order[0]

Your top-level JSON data might be an array, like the following:

[
  { "ref": [ 1,2 ] },
  { "ref": [ 3,4 ] }
]

Here, no key is present to access the first object. Use $ at the start of your path expression to refer to the entire JSON input. To fetch the first object from this JSON document using JSONata, use the following path expression:

$[0]

The output is:

{ "ref": [ 1,2 ] }

Manipulating JSON Data

In addition to using expressions of operators and functions to extract and transform data, JSONata lets you manipulate data using string, numeric, comparison, and boolean expressions.

You can combine multiple strings in your JSON data using string literals. For example, to concatenate the Account Name field and the first OrderID in your JSON data, use the JSONata expression below:

Account."Account Name" & '-' & Account.Order[0].OrderID

Enter that expression for your data in the JSONata Exerciser, like in the following screenshot:

The & operator takes string operands to concatenate (for example, '-') or a path expression to a value of string type (for example, Account."Account Name").

You can also use numeric expressions on data with the following operators:

• + — addition
• - — subtraction
• * — multiplication
• / — division
• % — remainder (modulo)

Additionally, you can combine path expressions with operators to create more complex expressions. For example, you use (Price * Quantity) to get the corresponding values from Account.Order.Product and multiply them. To get the total price of each product order in the JSON data, use the expression below:

Account.Order.Product.(Price * Quantity)

The output is:

[
  68.9,
  21.67,
  137.8,
  107.99
]

Comparison expressions compare two operands and return true or false. Here are the comparison operators:

• < — less than
• <= — less than or equal to
• > — greater than
• >= — greater than or equal to
• = — equal to
• != — not equal to
• in — value is contained in an array

You can combine these expressions with other operations. For example, in the previous JSONata expression, you got the total price of each product order. To retrieve true or false values instead — depending on if each order's total price is greater than 100 — use the following expression:

Account.Order.Product.(Price * Quantity > 100)

The output is:

[
  false,
  false,
  true,
  true
]

Boolean expressions can also create more sophisticated expressions. Like conditional expressions, booleans return true or false and use the or and and operators.

Using Functions to Transform Data

JSONata enables you to create new JSON structures using expressions, like the earlier example of returning JSON arrays from path expressions. Although you can extract data values this way, you typically transform data using functions. For example, take the following JSONata expression structure:

Account.Order.{
    OrderID: $sum(Product.(Price * Quantity))
}

The output is:

[
   {
      "order103":90.57
   },
   {
      "order104":245.79
   }
]

Account.Order{} converts whatever data you extract from the nesting into a JSON object. By specifying OrderID this way, you extract all values along the path expression Account.Order.OrderID as the keys in your new structure. Set the values of these new keys using $sum(Product.(Price * Quantity)).

$sum() is a built-in JSONata function that sums numbers passed as arguments. Expanding into a path expression, Account.Order.Product.(Price * Quantity), returns the total prices of each order in the data as an array. Calling $sum() this way sums up only the prices belonging to each of the two nested Product arrays.

Your JSON data has two orders with associated product arrays containing two product descriptions each. The JSONata expression structure retrieved the sum prices of products in each order. Use the following expression if you don't want an array of JSON objects representing both order totals, just one JSON object containing all the orders and sums as key-value pairs:

$merge(Account.Order.{
  OrderID: $sum(Product.(Price * Quantity))
})

The output is:

{
  "order103": 90.57,
  "order104": 245.79
}

The $merge() function combines your array of JSON objects into a single JSON object with the array objects' key-value pairs as its entries. You have successfully transformed your JSON data into another JSON object.

Advanced Features

While JSONata excels in basic data extraction and manipulation, it truly shines when you delve into its more advanced features.

Aggregations let you perform complex calculations on data within JSON structures. JSONata provides aggregation functions that make it easy to summarize and analyze data.

You've demonstrated using the $sum() aggregation function to sum numbers in your data. Other common aggregation functions include:

• $avg() — Find the average values in an array.
• $min() — Identify the minimum value in an array.
• $max() — Discover the maximum value in an array.
• $count() — Count the number of elements in an array.

With these functions, you can quickly get insights from your JSON data, making JSONata a valuable tool for data analysis.

Chaining expressions let you string together multiple operations, creating a sequence of transformations for a refined output. Their benefits include:

1. Clarity — Chaining keeps your code concise and readable, clearly defining each transformation step.
2. Efficiency — You can perform complex transformations without extensive intermediate variables.
3. Flexibility — Chaining enables you to build custom pipelines for data processing tailored to your specific requirements.

An example of JSONata chained expressions is in the previous section, where you created a new JSON structure by combining the $sum() and $merge() functions in an expression structure. As another example, you might want to return an array of total product orders with individual product prices greater than 30, sorted according to the total order amount (Price * Quantity). To do so, enter the following JSONata expression in the JSONata Exerciser:

$sort(Account.Order.Product[Price > 30].{
    "Product Name": $."Product Name",
    "Total Cost": Price * Quantity
}, function($l, $r) {
    $l."Total Cost" > $r."Total Cost"
})

The output is:

[
   {
      "Product Name":"Bowler Hat",
      "Total Cost":68.9
   },
   {
      "Product Name":"Cloak",
      "Total Cost":107.99
   },
   {
      "Product Name":"Bowler Hat",
      "Total Cost":137.8
   }
]

This code combines the expression structure with filtering ([Price > 30]) and the $sort() function to create new JSON data from the original data.

JSONata also has comments, variables, and custom functions, such as in the $sort() function call above.

Next Steps

JSON has revolutionized storing, exchanging, and representing digital data. Its simplicity, flexibility, and widespread adoption make it an integral part of modern software development and data interchange, especially for the IoT.

JSONata makes working with JSON even more compelling. Its aggregations, chaining expressions, conditionals, variables, and custom functions enable you to perform complex data transformations, gain insights, and customize your data processing tasks to meet specific needs. Whether you're querying JSON-based APIs or analyzing large datasets, JSONata's advanced capabilities simplify your workflow and make data manipulation more efficient and expressive.

Blues University

This article if part of a broader series where you can dig deeper into each aspect of embedded development. To embark on this journey, be sure to follow Blues University, where you can explore and contribute to shaping the future of IoT.

If you're new to embedded development and the IoT, the best place to get started is with the Blues Starter Kit for Cell + WiFi and then join our community on Discourse. 💙