Succeeded! Something went wrong!
Optimisation of the payload protocol in IoT devices featured image

Optimisation of the payload protocol in IoT devices

BLOG > Entry

Internet of Things devices often run on batteries and use wireless communications to send data to client servers. Manufacturers of electronic components aim for their chips to consume as low a current as possible during operation.

Introduction

Internet of Things devices often run on batteries and use wireless communications to send data to client servers. Manufacturers of electronic components aim for their chips to consume as low a current as possible during operation. Unfortunately, IoT devices also include less energy-efficient components, such as a radio, which consumes a lot of battery power to send or receive data (in the form of electromagnetic waves). Therefore, it is important to keep the radio usage time as short as possible, this can be achieved by optimising the size of the data sent.

Uploading payload from the device to the server.
Figure 1. Uploading payload from the device to the server.

Flexible approach to types of measurement does not require changes to servers and devices

At the stage of designing a payload protocol version it is impossible to foresee all supported measurement types (i.e. temperature or pressure). Attempting to define a finite number of measurement types may result in the need to release a new version of the payload resulting in the need to maintain two versions on the servers. It is good practice to introduce generic (custom), unspecified measurement types that can be adapted at a later stage to define previously unpredicted measurement types.

While developing the payload protocol in the version 2 we first defined up to 17 basic types of measurements that we can expect to use. These are single-value types, i.e. the measurement consists of a single value, e.g. temperature 24.5°C. Additionally, we have defined several custom types.

Classification of measurement types into basic and custom payload protocol in the version 2.
Table 1. Classification of measurement types into basic and custom payload protocol in the version 2.

By defining custom types we have achieved some flexibility of measurement types and they are often defined by the type of device e.g. the YO Pulse device counts the number of pulses. To define this type of measurement we used a custom type of measurement number 24. This does not mean that the type of measurement will always define the number of pulses. In another device it may mean something completely different, e.g. in a device that measures pressure force, the measurement value may be assigned to this type.

Mechanism for detecting sensors of the same type

Telemetry devices often take measurements of the same type but from different sensors located in different sites, and their values are different, e.g. measurement of water temperature in two different tanks. In order to assign the measurement value to a specific sensor, it is necessary to assign the measurement, for example, the address of the sensor (sensorID) from which the measurement comes. Knowing the location of sensors and their addresses, we are sure that we know what we are measuring. Unfortunately, adding additional byte to each measurement extends the total length of data payload, which increases transmission time (using radio) and consequently power consumption. This approach also has its disadvantages in the form of reduced total lifetime of the device. The best solution is to find a compromise in the form of determining when the sensor address is necessary for transmission and when it can be omitted:

1. If a device is equipped with e.g. 3 sensors taking measurements of different types e.g. temperature, humidity and pressure. In this case we can avoid adding the sensor address to each measurement because we are able to clearly associate the measurement with a particular sensor and its location. However, if necessary the sensor address can be used to determine whether the sensor is internal or external.

An example is the YO H2O device taking measurements: battery voltage, temperature, humidity, flood status, position (accelerometer). We have omitted to add sensor addresses because we know which measurement relates to which sensor.

2. If the device is equipped with more sensors of the same type e.g. two temperature sensors and one humidity and pressure sensor each. In this case you should consider assigning an address for the sensors in order to easily identify which measurement applies to which sensor in the following way:

  • the sensor address is not required if the sensor is internal (inside the device enclosure) - its location is known to us;
  • the sensor address should be added to the measurement from external sensors (sensors that can be connected to the device) or the device has multiple inputs of the same type, then we have information from which sensor (input) the measurement comes from.

An example is the YO Temp device making measurements: from 3 external temperature sensors (channels 1-3) and battery voltage, temperature and humidity inside the device. The sensor addresses have been defined only for the external sensors.

One byte and several information items, division and compression of information in a byte

Let’s go back to measurement types for a moment. We can assume that the type of measurement takes one byte, so we can specify from 1 to 255. As you can easily see, we will not manage to define 255 types of measurements, especially if we use custom types. Therefore, it should be questioned whether it makes sense to reserve a whole byte only for this parameter. Well, no. Not all data takes the full space allocated for it. The reason for optimisation, as it was mentioned before, is to try to reduce the transfer of unnecessary data without losing its usefulness. It is necessary to make an analysis of what data has to be transmitted and what will be its total size. As it was mentioned before, it will not be possible to define 255 types of measurements, therefore it is possible to use a part of this byte and send other data. We will reduce the range of measurement types by half which will give us a maximum of 127 types, thanks to that we will have one free bit in the type byte for another parameter. By making further analysis we can free up additional space. By repetition of this process for all parameters we may find that we can send the same amount of information in fewer bytes at the same time without losing data resolution.

Extracting information from two bytes and merging them into a single byte using the type and precision of measurement as an example.
Figure 2. Extracting information from two bytes and merging them into a single byte using the type and precision of measurement as an example.

Using this simple data analysis and compression mechanism, we have optimised the payload length. Now we store up as many as five measurement parameters on the first three bytes:

  • measurement type - up to 63 defined measurement types;
  • measurement precision - measurement with an accuracy of up to 3 decimal places;
  • measurement delay;
  • sensor address length - if occurred up to 15 bytes;
  • measurement length - up to 15 bytes.

How the measurement parameters are allocated in common bytes.
Table 2. How the measurement parameters are allocated in common bytes.

Currently, we use only 26 types of measurements from the total number available, and the measurement can be sent with an accuracy of 3 decimal places (some sensors available on the stock do not perform measurements with such accuracy). As you can easily notice the length of a single measurement is dynamic, only the first 3 bytes have a defined order, the following bytes corresponding to the address and value can have different lengths - we do not reserve them statically but dynamically in the payload protocol.

Conclusions

Optimisation of the protocol for data transmission in IoT devices is not complicated, it consists mainly in analysing the validity of transmitting specific data and searching for a satisfactory compromise. In addition, thanks to a smaller amount of data to be transmitted, we reduce the time of radio use, which affects the extension of the battery life of the device. It also affects the performance parameters of the entire system in which the devices operate, i.e. reduced airtime, reduced probability of data collisions, etc.

The mechanisms described in the above article have been implemented in the payload protocol in the version 2 used in our devices and servers. Thanks to them, we managed to obtain a flexible payload and at the same time the size of the measurement itself is dynamic - dependent on a specific device.

Paweł Marchewka
by Paweł Marchewka
| 10 January 2022