Hear and destroy: how artificial intelligence helps shoot down cruise missiles over Ukraine

Since the outbreak of a full-scale war, the Russians have fired more than 4,700 cruise missiles into Ukraine. Enemy weapons hit not only the Ukrainian critical infrastructure but also residential buildings, schools, and kindergartens. Many innocent civilians have been killed in air and missile strikes.

The latest large-scale shelling of Ukrainian territory happened on November 23, 2022. The Ukrainian air defence shot down more than 70 out of 90 cruise missiles and all ten Shahed drones during a massive shelling on November 15. The air defence forces are reinforcing air defence means and creating complexes consisting of several modern systems to clear the Ukrainian sky of enemy missiles.

The more detection systems the defence complex contains, the more difficult it is to destroy them.

One of these systems is the Zvook AI project. It is a hardware and software complex capable of acoustically detecting cruise missiles, helicopters, drones and fighter aircraft at low-to-medium altitudes. There are currently about 40 complexes of this type up and running in Ukraine.

Dmytro Believtsov, a Zvook project member, CTO and co-founder of Respeecher, Pavlo Tsiupka, a project member and CEO of i3 Engineering, and Marian Sulym, a project member and the 125th Territorial Defence Brigade serviceman, told "Ekonomichna Pravda" (EP) about how this system protects national security of Ukraine. From EP: the information provided by the project participants has been agreed upon with the military. -- What prompted you to set up the project and what is its main goal?

-- (Pavlo) On the first day of the war, the only question in my mind was how to help the Armed Forces of Ukraine. There were lots of scary videos of cruise missile flights in Telegram chats. I knew Ukraine had the air defence system, but I was also aware of military technologies.

I understood that missiles could be hardly detected using radars at low altitudes. It is possible to detect the missile launch, track direction of the flight from the fighter, but it is impossible to intercept. Thousands of people across Ukraine could be engaged in the enemy air targets detection efforts.

They could observe the missiles' movement and transmit information, but this is a bad idea. We realized it would be better to develop an automated system that could perform the same function, i.e. notifying the air defence forces of the time and location of a missile. This is how the project idea originated.

Zvook is a hardware and software complex that acoustically detects enemy cruise missiles, helicopters, drones and jet fighters at low-to-medium altitudes. It is based on machine learning (ML) technology that recognizes the sounds of enemy air targets' engines. There are about 40 Zvook complexes operating in Ukraine at present.

-- Who is working on the project?

-- (Pavlo) We were joined by the leading Ukrainian IT and telecom specialists and the military. The Lviv Regional Military Administration and Deputy Chairman Oleksandr Kulepin convened a meeting of mobile operator representatives and organized the joint work within 30 minutes after our appeal. The Lviv RMA continues to promote project development.  

Also, the Kyivstar team responded quite promptly, so we went on our first all-around-the-country trip together. We deployed the first Zvook systems in tandem. In addition, Kyivstar experts provided us with round-the-clock support during emergencies. 

It is noteworthy that the enemy cruise missile was detected just four hours after the first Zvook facility deployment, and the military structures in charge were notified. Also, we also shared sleepless nights with i3 Engineering and GlobalLogic engineers. Ukrainian State Air Traffic Services Enterprise helped us to plan the network and make a presentation of our system in the military bodies.

The artificial intelligence model was developed by Respeecher start-up engineers Volodymyr Sydorskyi (ML team leader) and Valeriy Zablotskyi-Drohan (sound engineer, data engineer). The Label Your Data team helped the project with hundreds of hours of data manual labelling, which is one of the key factors of successful ML algorithms used in the Zvook system.    The 125th separate Territorial Defence Brigade team developed, tested and deployed detection stations on Kyivstar cell towers.

-- (Marian) Pavlo required qualified specialists who could implement the ideas for the detection stations' production, deployment and network support. He contacted the 125th separate Territorial Defence Brigade and was enthusiastically supported. The military command recruited military personnel with software development and design experience in civilian life and provided favourable conditions for their effective Zvook project participation.

At present, five servicemen from the Brigade are involved in the project, namely Artur Horbenko, a commander, Eduard Luchuk, Maksym Rudenko, Illia Pavliuk and I (Marian Sulym). Cooperation with our brigade facilitated the military communication of the project. The experience of the military and understanding of the specifics of Ukrainian cities and important critical infrastructure air defence came in handy.

-- What does this project look like now? -- (Pavlo) We have two subdivisions now, working on hardware and software development.  A hardware product (the so-called "iron") is a system deployed in a particular location.

Marian and Illia are responsible for it. Marian coordinates the team and comes up with the improvements, and a top-notch professional Illia creates automated listener kits. Our major task is to receive the sound from a special device and focus it with a mirror.

Next, we amplify and transmit this sound to be processed within a secure data transmission network. There is no single data processing centre but several nodes instead, which excludes the risk of centre destruction. Such a network can be hardly destroyed.

Working on the ML part, Dmytro's team assists in decoding the recorded signal. In addition, Dmytro helps to promote the product.  The rest of software is developed by Maksym Rudenko's team from the 125th separate Territorial Defence Brigade.

As a highly experienced programmer and an IT company co-founder, Maksym coordinates the volunteers writing the code.  -- (Dmytro) We do not have a clear hierarchy. There are manifold tasks tackled by different team members.

We understand that what makes essence is not only the development but also communication, Bizdev and marketing: the search for stakeholders, communication with the military and potential customers. Marian and Pavlo are in charge of these directions.   We are currently gathering a team to deal with operational processes and company building.

Zvook is still in its infancy stage, lacking a deep structure. -- What are the problems with enemy targets' radar detection? What has changed since the outbreak of the war?

-- (Pavlo) In the early days of the war, we had little information about cruise missiles striking Ukrainian infrastructure. However, things have changed dramatically since then. Back then, we were not aware of the enemy missile flight paths.

At present, we are capable of roughly predicting the routes to be flown. We cover the most vulnerable areas in cooperation with air defence. In addition, the radar network was built differently in Ukraine at the beginning of the war.

The main points of the missile passage are currently detected by radars. The air defence can obtain much more data. The Zvook system conducts additional reconnaissance of the enemy air targets in "blind spots".

Obviously, it is impossible to cover the entire Ukraine with radars, so we are developing new systems to cover the areas where no radars have been installed yet.

The system complements the air targets tracking in certain areas, which is extremely important for making sensible decisions on countering enemy targets. -- (Dmytro) It is essential to track as much of the path as possible. This way we make more attempts to intercept and shoot down, thus increasing the chances of destroying the missile.

-- Tell us about the technical architecture. What technology underlies the system? -- (Dmytro) Zvook is based on machine learning, which enables the system to detect enemy air targets.

ML technology revolutionized image processing in 2006, and the machines acquired almost human-like data analytics capabilities. However, machine learning can process a much larger amount of information than a human. It is crucial that neural networks not only carry out the data analysis but also learn from errors and enhance operation.

Historically, a neural network dealt with object detection in images. Sound can be identified with an image which will be recognized by the system. This is how the audio direction began to develop.

Regarding the Zvook system, we have an audio stream which contains a large number of certain sound objects existing in time, not space, such as wind, road noise, little flies, engineers' voices, cows' mooing, chainsaws cutting wood, etc. These sounds account for 99.9% of the data, which allows us to separate a very small percentage (0.1%) of what we are primarily interested in, i.e. the sounds of enemy air targets. The algorithm is intended to identify the target among many other objects and notify a station user.

We constantly train the network and provide it with the necessary data to achieve the desired result. What the ML algorithm requires is access to the right amount of accurate data. Several stations had already been deployed before we joined the Zvook project, so we had some data to work with.

-- What issues did you encounter in the development process? -- (Dmytro) First, the network was trained on Internet recordings as we were dealing with the so-called domain shift, i.e. a scenario where the initial characteristics differ from what has to be finally obtained. The team used phone-filmed videos of close missile overflights.

The sound is different from a distant missile overflight. Moreover, the phone microphone does not provide high sound quality.  The system was gradually enhanced.

The 125th brigade servicemen Marian and Maksym spent a lot of time listening to the sounds and providing constant feedback on the results required for algorithm training. -- (Pavlo) When the system proved its effectiveness in the laboratory, it was happily delivered to the brigade for on-site installation.  However, it was just the start of our journey. There were a lot of errors later that had to be fixed in the lab at night.  

Previously, the probability of accurate detection was just about 50%, not high enough to intercept the enemy air targets. The turning point came when the project earned the trust of the Lviv Regional Military Administration. They immediately responded to our cooperation request and convened a meeting of all mobile operator representatives.

On that very day, we were allowed to deploy our devices on Ukrainian territory. A few days later, we began collecting data on actual missile flights and transmitting it to the appropriate response force. The success was followed by another issue when we encountered difficulties with the accumulated data transmission to the response force.

The military bodies required higher accuracy of the data as they were simultaneously deploying their surveillance systems. Dmytro's team project involvement helped to tackle the problem. The processes have gradually improved, and now the error probability is heading towards zero.

-- (Dmytro) There are two types of errors. False positives happen when the system wrongly identifies an object as an enemy target. For example, the car engine may be identified as the sound of a missile's jet engine.

False negatives happen when the system identifies a real missile as a non-threat and reports on safety. It is crucial to deal with both types. We fix them by training Zvook to learn from its own errors.

Also, we engaged the Label Your Data start-up team, which labels data for artificial intelligence (AI). The guys helped with manual data labelling to supplement the dataset with noise sound samples corresponding to real missile sounds. Data must be accurate and reliable to prevent unnecessary waste of resources due to false detection of enemy air targets.

There is a risk of mistrust if the system fails frequently.  -- (Pavlo) Ukrainian interception facilities are developing, and it is impossible to ignore these errors. The system has to comply with extremely high requirements. 

It was acceptable some time ago to obtain at least one correct result out of five errors, but it is no longer possible.  The system is currently working in a reliable mode, so it can be trusted. -- (Marian) Initially, we entered a vicious circle.

We had to collect samples of overflights to be able to identify enemy air targets efficiently. It is problematic as they are not automatically recognized and saved.  It took us four months to solve the issue.

Having accumulated enough recordings of the overflights, we were able to launch an automatic mode. As a rule, developers try to predict the errors of their systems in the laboratories. We had no time for that.

We worked hard to meet yesterday's deadlines. A ready-made solution was put into practice without delay. We were asked to expand the network, so there was no opportunity to sit in the lab and test the product.

In some cases, the test was positive, so we deployed the facility in the field, but it failed to work. We had to make a 300 km return journey and fix it. Of course, the failure was quite demotivating, but this feeling did not last long.

-- Did you study any Zvook system analogues while developing the complex?

-- (Pavlo) We have studied many analogues, dating back to World War I, World War II and up to the present.  When radars were still ineffective and there were no automated systems, the air threats were listened to by people. They were doing their best to recognize the sounds of jet-powered missiles at the sound guard stations.

Special sound-reflecting mirrors were used for that purpose. People with excellent hearing were on duty there. Zvook is the digital version of a sound guard station.

-- (Marian, laughing and showing a photo) These are the early and mid-20th-century devices.

Interestingly, these particular systems were used in the Second World War when the Germans launched V-2 rockets at London.  While working on Zvook, I had to study specialized literature on the engineering of such systems. The system consists of a mirror that amplifies the signal and concentrates it at a specific focus point where the microphone is located.

This way the microphone can hear more. The mirror is an essential and inexpensive station element. Also, the so-called microphone arrays are used in the complex mentioned in the specialized literature.

Another team of volunteers from Lviv is developing this solution for us. It will enable the complex to determine the angular coordinates of the air targets. We did not invent anything new while developing Zvook but collected all the available cases known to humanity just at the right time.

Dmytro helped us to supplement the system with artificial intelligence because it is not enough to hear. It must be done automatically. -- (Pavlo, laughing) Yes, what you need is either twenty dudes from the philharmonic orchestra or one Dmytro. 

-- (Dmytro) Detection of cruise missiles is a unique feature of Zvook. Russia uses a lot of cruise missiles. I don't think there are similar networks of acoustic sensors anywhere in the world that would be so sensitive to this type of weapon.

Even if any other country went to war against Ukraine, Zvook could successfully detect its air targets. Cruise missiles are built on a similar principle: each has a jet engine. -- How much does it cost to develop Zvook systems?

 -- (Marian) The development includes three components, i.e. software, AI model and "the iron". We develop hardware ourselves, without any contractors' involvement, so the work on the "iron" can be estimated by the number of hours of engineers' efforts. -- (Dmytro) Zvook is not a commercial product, so there is no unit economics here.

Zvook systems are free for the military. The stations are built with our own money and donations. We are planning to scale up the production with state funds.

-- If a thousand of your systems are deployed in Ukraine, how reliable will the protection be? -- (Pavlo) No system can guarantee a hundred per cent result, but they all complement each other. It is of utmost importance to have an efficient air defence system as it successfully destroys 90 % of the enemy air targets.

The remaining 10% must be destroyed by other technologies, similar to Zvook. -- (Marian) The most appropriate way of deploying the complex is the frontier method when the detection stations are installed in lines along the state border and deep inside Ukraine. Increasing the detection range enables the deployment of the stations with a lower density, which saves manufacturing and logistics resources.

Based on the current technical characteristics of the complex and the approximate length of the deployment lines, about 600 detection stations are required. Naturally, every defence facility can be countered. The larger the complex, the more detection systems it contains, and the harder it is to destroy.

Radar stations are the top target. They can be disabled by targeting missiles at a radio emission source. The Zvook advantage is that it is a passive system without any emission.

The location of the station can be detected, but any war has an economic component: the destruction means must be at least a bit cheaper than the target. Otherwise, you will run out of resources. Therefore, fighting against the Zvook system sounds irrational.

-- (Dmytro) The so-called "parrots" are among the main methods of combating acoustic and counter-battery systems. It is the creation of additional noise that distracts the enemy from the target. For example, when a Russian sniper fires, he can ask his colleagues to fire into the air at different points.

However, this method cannot be used by the Rashists as the stations are located in the territory of Ukraine.

-- What unusual situations did you have to deal with while working on your system? -- (Marian) The funniest things happened during the false positive scenarios (the system identifies an object that is not a missile). Dmytro trained the model to detect cruise missiles.

Everything worked smoothly, but there were  lots of false positives. The system looked for similar patterns to produce an accurate result. These patterns also occur in nature.

At the initial stage, for instance, Zvook identified the cows' mooing as an enemy target. The model excluded the sounds of the wind or cars, but cows were identified as a threat. Unfortunately, we found very few recordings of cows' mooing and couldn't train the model on them.

It took us a month to get rid of those cows (laughing). -- (Pavlo) A cockchafer, or maybe a bumblebee, flew past our devices once, so everyone jumped out of their seats at the sound. We had installed new mirrors just the day before, and they provided a very high-quality sound broadcast.

The first-used mirrors can be compared with the latest ones by analogy between Full HD and 4K.  -- (Marian) Everyone was in despair back then. There were no options for countering Russian missiles.

We offered Zvook as a solution, and the system was actually brought into use on the assumption that it could make errors. There was no other alternative in those days. It would be difficult for us to develop the project without this experience.

As a result, false positive scenarios no longer happen. -- What are the project development priorities in the near future? -- (Marian) We are aware of several potential technical improvements.

Our major task is to determine the coordinates, the direction and the speed of movement of the identified target. It would be a significant step forward in target detection to assist air defence units in firing damage to enemy targets. Our team has already developed the concept of acoustic detection of the azimuth (direction) of the target and conducted the experiment that proved its accuracy.

-- (Dmytro) Our first and foremost priority is to implement the Zvook system for national security purposes. We are planning to scale up the project and expand communication with the military. We are also considering the commercial side of the project development and attraction of investments.

We want to sell our product in a global market in the bright future.

YANA YAGORY