Sensor fusion offers benefits for designers and end users

Over time, sensors have morphed from simple analogue and mechanical constructs to chip-based digital devices that connect to a machine to monitor its health as well as environmental conditions. Similarly, sensor fusion - multiple types of sensors working together to solve a problem - has combined the threads of many other technologies to create something new and very exciting.

The idea of using a computational device to sort out the data from multiple sensors and combine information to draw a conclusion has been around since at least the 1950s. But at that time it was exceptionally difficult to do. Around 1960, several mathematicians developed sets of algorithms in an effort to have a machine draw a conclusion based on inputs from multiple sensors. These filters also removed meaningless data from noise or other sources. Of course, it was not long before the military decided that this technology would be useful in their applications. Being able to process inputs from multiple sources and compare it with stored data would enable the military to better track and identify potential airborne targets and even compute the certainty of the results. With better computers and sensors, the technology was advancing, but there were still complex and expensive problems to solve.

Potential applications

When the microprocessor first became available, people described it as a solution in search of problems. The same case can be made for sensor fusion. If you have the power and intelligence to monitor multiple sensors, analyse the data in real time, and either provide a simple direction or control an action, heretofore-unthought-of applications can be almost limitless. The following examples just scratch the surface:

• Health monitoring - including healthy athletics, patient monitoring and research
• Monitoring the elderly - wellness monitoring to reduce the burden of staffing
• Automotive and transportation systems - monitoring and controlling efficiency and safety functions
• Public safety - identification of potential hazardous conditions with much greater accuracy than simple fire and security systems
• Entertainment - gaming, including controllers and virtual reality headsets
• Weather - intelligent weather forecasting stations that not only warn of changing conditions, but also control systems to prepare for a storm (for example, closing storm shutters, closing valves, etc)
• HVAC/air quality - intelligent control of room temperature, humidity, air quality, system maintenance, etc

}

While all of these types of capabilities have existed in some form for many years, the ability of a system to observe multiple sensors and come to an intelligent conclusion - and even initiate action - is revolutionary.

Converging technologies

Fortunately, as they have done in other areas of electronics, a number of IC manufacturers have taken on the task of doing the heavy lifting. With off-the-shelf sensor fusion and sensor hub chips, it is now possible to efficiently interface to a variety of digital sensors, as well as other pathways. The burden of creating your own algorithms has been eliminated.

While their terminology does vary, a number of IC manufacturers have either adapted existing lines of products or created entirely new ones to tackle sensor fusion tasks. The processing is done by a specialised controller chip, which may be identified as an MCU, a sensor hub or a sensor fusion processor. We are already seeing this technology applied in the consumer market in smart phones, activity monitors and other devices.

The latest generation of smart phones from Apple, Samsung and others contain powerful and diverse sensing capabilities, even without the need for external interfacing. These include a three-axis magnetometer, a three-axis accelerometer, and a three-axis gyroscope. This combined capability is often referred to as 9-DoF (nine degrees of freedom).

For the most part, these functions are "always on' in a smartphone. If the processing of data from these sensors was managed by the phone's central microcontroller, battery life would be significantly shortened. Instead, highly efficient dedicated MCU chips process the data as sensor hubs, using a fraction of the power. The NXP ARM M3 series of MCUs is one example. According to Chipworks, a product teardown specialist, as reported by EETimes, Apple uses a customised version of the NXP chip to monitor its sensors in the iPhone 5S: "The M7 controls functions from a variety of discrete sensors including a gyroscope, an accelerometer, and a compass." Samsung takes on the same task with a microcontroller from Atmel, the Core 8-bit AVR MCU.

With such powerful on-board sensing technology, apps are appearing that take advantage of the 9-DoF smartphone hardware to provide health and activity monitoring, or to function in concert with GPS and external data to provide even more information for the user. Now, add to those already-diverse sensor inputs data from an external device that communicates via Bluetooth and the capabilities seem limitless. The goal of the chip manufacturers is to make it practical for engineers to design systems that provide real-time sensor data which can be used to provide the desired contextual awareness with minimal power consumption and maximum battery life. Beyond smartphones, highly optimised systems can address such applications as tablets, Ultrabooks, IoT-enabled devices, gaming, healthcare, environmental monitoring and wearable computing.

Development boards are available that enable design engineers to easily 'get their feet wet' in this technology. One such example is the ATAVRSBIN2 by Atmel, a company that has embraced sensor fusion with a wide variety of products, which it refers to as the 'complete sensor ecosystem.' Atmel identified that the simultaneous analysis and fusion of data from different sensors and sensor types was not a task it could handle solo. To get past these complexities, the company partnered with a number of leading sensor manufacturers and sensor fusion specialists to provide a complete, easy-to-implement Sensor Hub Solution.

A current trend combines an MCU with three or more MEMS sensors in a single package. One example is STMicroelectronics' LIS331EB, which combines a high-precision three-axis, digital accelerometer with a microcontroller in a single 3x3x1mm-package. The microcontroller is an ultra-low-power ARM Cortex-M0, with 64-Kbyte Flash, 128-kbyte RAM, embedded timers, two I2C (master/slave) and one SPI (master/slave). The LIS331EB can also internally process data sensed by external sensors (for a total of nine), such as for gyroscope, magnetometer and pressure sensors. Functioning as a sensor hub, it fuses together all inputs with the iNEMO Engine software. STMicroelectronics' iNEMO engine sensor fusion software suite applies a set of adaptive prediction and filtering algorithms to make sense of (or fuse) the complex information coming from multiple sensors.

Freescale also offers a range of devices that combine MCUs and sensors in a single package. Their FXLC95000 Xtrinsic Motion-Sensing Platform integrates a MEMS accelerometer and a 32-bit ColdFire MCU. Similar to the STMicroelectronics device, the FXLC95000 can simultaneously manage data from internal and external sensors. Freescale was the first company to market an MCU with a sensing hub embedded that is also programmable for customer-specific applications and algorithms. Up to 16 sensor inputs can be managed by a single device, allowing calibration, compensation and sensor functions to be offloaded from the application processor. It functions with either Freescale or third-party drivers.

Other manufacturers that are well-entrenched include Bosch, Fairchild, Honeywell, MicroChip, and TI.

Sensor fusion and the Cloud

While quite a lot of functionality can be achieved at a local level, interaction with the Cloud is where the fun really begins. Remote sensor data can be processed by a sensor fusion device and sent to the Cloud for recording, further analysis or even to order an action.

For example, an unattended pump operating in a remote location is always at some risk of failing. A few years ago, a remote sensor may have been in place to identify if it were running hot or had even failed. Now, the same pump can also be monitored for vibration, exhaust chemistry, bearing noise and the external conditions around it. A predetermined program could empower the sensor fusion controller to shut down the pump or even cycle its operation until a technician can arrive. The system would also know in advance whether it is likely that the entire pump must be replaced or just a component. Here, sensor fusion could eliminate downtime as well as costly emergency service calls, and even collect data to analyse how well the pump is working over time. The same general idea applies to monitoring an aircraft engine in flight, a building elevator, or just about anything mechanical.

Another application of the Cloud is for the sensor fusion to take place there, instead of on site. With open-source sensor fusion software available, individual sensor data can be transmitted to a server, where the processing would take place.

Conclusion

Sensor fusion is a technology that has come of age, and at just the right time to take advantage of developments in sensors, wireless communication and other technologies. Once out of reach of all but the most advanced government labs, the technology is now available off-the-shelf, at prices that even fit into the BOM budget for many consumer products.

Now closely linked to mobile technology and the rapid development of lower-cost digital sensors, sensor fusion is poised for explosive growth. For the design engineer, it is a good time to apply some creative thinking and to start experimenting!

Contact Mouser Electronics to discuss particular sensor fusion opportunities.