First Responders get co-pilots in the shape of AI-powered drones

Today, I’m sitting down with Tim Jacobs - Program Manager- and Matthias Verlinde - Data Scientist - to talk about the ins and outs of this world pioneering project. In this interview we talk about: 

• What is SENSE, and why is it so revolutionary? 
• How the AI-models in this project were built 
• Why 5G connectivity is so crucial in this scenario 
• Why you can’t fix a spraying device to a drone … 
• … and why the sky is the limit 

Tim, Matthias, tell us, what is this all about?

Tim Jacobs: In a nutshell, first responders now have a new ally in the shape drones that assist them in their perilous work. These drones in the SENSE project act as ‘an AI-powered co-pilot in the sky’ while the responders are the ‘boots on the ground.’ Currently employed in firefighting operations, this project aims to establish a comprehensive network of drones throughout Belgium. Cities like Genk and Brugge have already seen these drones in action, and soon other cities like Oostende and Hasselt will follow suit. 

"The drones in the SENSE project act as ‘an AI-powered co-pilot in the sky’ while the first responders are the ‘boots on the ground.’" 

When you say "smart" drones, what exactly do you mean? 

Matthias Verlinde: These drones are outfitted with a camera that is integrated with computer vision technology. In simple terms, the drone can detect smoke plumes, identify the type of fire (chemical, forest etc.), and locate hotspots—literally, the areas with the highest risk of further fire outbreaks. The intelligence is in the cloud, which provides benefits such as always-up-to-date models and allows a much more detailed post-intervention analysis than a real-time algorithm on a device would allow. 

The drone can also recognize individuals and their positions, even if they are unconscious and hidden under foliage. It also identifies specific objects, such as open doors or windows, which may indicate someone has entered or exited. The “intelligent” camera records and analyzes all this information, which the firefighting commander can access it in real-time on a tablet. 

So, the drone can perceive much more than a human can, even if they were in the air themselves? 

Tim Jacobs: Absolutely. The drone has multiple eyes, capable of seeing everything. Each drone is equipped with two camera streams: a regular RGB stream and an infrared stream that detects heat emitted by individuals or objects. What we do is, we create a digital twin of the disaster site, a virtual replica fed with various data types, including the data captured by the drone's camera – the so-called ‘time-series data’ –  supplemented by contextual data from other sources like weather conditions, wind direction, traffic flow and the locations of gas and utility lines (in Flanders, known as KLIP data).  

Based on this comprehensive dataset, the AI model can identify patterns and make predictions. For instance, it can predict the time it will take for a chemical smoke plume to reach a residential area or a major traffic artery given the current wind direction. The firefighting commander can then make informed decisions, such as initiating evacuations or redirecting traffic. It is important to emphasize that it is a human person who makes the final decisions here. AI serves as a co-pilot, so to speak. 

"What we do is, we create a digital twin of the disaster site, a virtual replica fed with various data types."

The real-time aspect seems crucial. How do you achieve that? 

Tim Jacobs: Indeed, every second matters in such situations. The drones are piloted by a team of certified drone operators located in Oostkamp and soon Hasselt. The plan is to grow the number of ROCs (Remote Operations Centers) to five, scattered across Belgium. Real-time operations are made possible through a 5G connection, which ensures zero latency, provides sufficient bandwidth for high-definition video streaming, and remains stable and highly performant even when the network is congested, thanks to a unique feature of 5G networks called "slicing."  

In practical terms, when the drone pilot in Oostkamp steers the drone to the left, the maneuver is executed instantaneously, and the recorded footage is immediately transmitted to the firefighting commander. What the commander sees on the tablet is precisely what is happening in real-time. With 4G networks, there would be several seconds of delay, which is unacceptable in life-or-death situations. Not so with 5G. 

Here's a wild idea: why not attach a spraying device to the drone? 

Tim Jacobs: Not a wild idea at all, but unfortunately not feasible (laughs). You see, drones rely on batteries, and the heavier the drone, the more battery power it consumes. To ensure swift arrival at the scene and ample time for meaningful observations before returning, it is essential to keep the drones as lightweight as possible. Therefore, they are solely equipped with an intelligent camera and nothing else. No additional device is attached for data computation, for instance.

Hang on, an extra device? Could you elaborate on that? 

Tim Jacobs: Sure. The camera is coupled with an AI algorithm, which, in simple terms, is software that needs to run on either a physical or virtual machine. In the case of these drones, all the computing is performed in the cloud, as attaching a small device to the drone would add extra weight, a no-go. The compute happens in the cloud, not on the edge. The images captured by the drones are also stored in the cloud, where they can be accessed for post-intervention analysis. 

"Thanks to the 5G network, the connection remains undisturbed, even during network congestion."

Going back to the AI aspect, how was the model developed?

Matthias Verlinde: We base our models on architectures identical to GPT, except here we focus on images rather than text. Numerous open-source models are readily available, trained on various tasks. Starting from these pre-trained models, we fine-tuned them for our specific purposes. For example, a model capable of distinguishing between foreground and background can be trained to recognize smoke plumes.  

We then conducted validation tests to ensure the model's accuracy surpasses a certain threshold. Subsequently, the model continues to self-train through its usage and scheduled "retraining rounds" to improve its algorithm's precision over time. In the field, this process is known as MLOps, or Machine Learning Operations. The model runs in the cloud on Microsoft Azure. Citymesh, our telecom partner, takes care of the 5G connectivity and drone operations.  

Now that safety drones for emergency services are a reality, what's next? 

Matthias Verlinde: The sky is the limit, quite literally (laughs). Numerous scenarios can benefit from this. For instance, in major traffic accidents or even terrorist attacks like the one at Zaventem Airport, drones can play a crucial role. Thanks to the 5G network, the connection remains undisturbed, even during network congestion, as was the case during the Pukkelpop disaster in 2011. In fact, Brussels Airport is one of our clients, operating a 5G MPN (Mobile Private Network) and utilizing drones.  

Tim Jacobs: Additionally, drones can be employed for aerial observations, such as assessing road and water quality or inspecting telecommunication and high-voltage towers. The construction and agriculture sectors also harness the power of drones. Hyperspectral cameras attached to drones, for instance, enable farmers to detect diseases or abnormalities in crops by peering through the foliage.  

Interview with Mitch De Geest, CEO of Citymesh and firefighter  

Given your unique perspective, do you foresee any challenges or concerns with implementing AI in emergency response? 

Mitch De Geest: Absolutely, there are always challenges when introducing new technology into established workflows. One concern could be over-reliance on AI-generated data. As firefighters, we have a gut feeling that comes from years of experience, so it's crucial that AI assists our decisions without overshadowing our intuition. Also, ensuring that the AI models are accurate and adaptable to different scenarios is vital. We deal with diverse situations, and the technology should be as versatile as we are. 

With your dual role as a firefighter and someone following this project closely, what message would you give to your fellow firefighters about embracing AI-powered drones? 

Mitch De Geest: I'd say embrace it with an open mind. AI isn't here to replace us; it's here to enhance what we do. Just like how we rely on each other's expertise during operations, we can consider AI as an additional team member. It's the fusion of human judgment and technological support that will redefine how we respond to emergencies. Let's see this as an opportunity to make our jobs safer and more efficient while staying true to our mission of protecting lives.