Cloudy with a Chance of Aircraft

Luciad is a partner in the RETINA research project, which started in 2016 and will end early next year. RETINA stands for Resilient Synthetic Vision for Advanced Control Tower Air Navigation Service Provision.

The project is lead by the University of Bologna (UNIBO). The other partners are ENAV, CRIDA and Eurocontrol. The goal of the project is to investigate applications of Augmented Reality (AR) in the air traffic control tower, so that controllers will no longer be limited by what the human eye can physically see out of the tower windows. RETINA’s concept will allow the controller to have a head-up view of the airport traffic even in low visibility conditions.

No Vertigo in This Tower

Simulation Tower

To test the AR prototype, UNIBO built an immersive, human-in-the-loop simulation of the Bologna airport control tower. The simulator reproduces the view out of the actual control tower windows, using 3D models of Bologna airport and various types of aircraft. The simulator uses a Microsoft Kinect sensor to track the position of the viewer’s head and adjusts its perspective accordingly. This makes the screens feel like true windows into a virtual world and greatly enhances the sense of immersion.

Lasagna Model

Using the Microsoft HoloLens, a set of AR overlays were implemented on top of the tower simulation. The information presented on these overlays includes:

  • Runway occupancy: runway is highlighted in green when it is free, or in red when it is occupied
  • Weather: an overlay shows wind direction and speed, temperature, pressure
  • Aircraft identifications and positions: aircrafts are labeled with their callsign and other relevant information
Overlay Augmented Reality

In this picture, you can see an illustration of the outside view (left), an overlay (middle) and the combined view (right).

So You Wanna Be a Pilot?

Luciad developed a “pseudopilot application” (PPA) with our desktop solution LuciadLightspeed. The PPA shows the current traffic situation on a 2D map. The PPA operator can select an aircraft and issue a number of commands such as “land”, “take off” or “start taxiing”. These commands are sent to and executed by the tower simulator. The PPA is thus responsible for translating the verbal instructions of the air traffic controller into simulation events.

Keep Your Head Down

We also implemented a “head down display”, which reproduces some of the systems that are already commonplace in control towers today. The head down display includes Ground Radar and Tower Radar views (essentially maps that show current aircraft positions), as well as tables showing flight data such as the current status (parked, in taxiing, in flight, …), the departure and destination airport or the ETA.

These displays are generated by the pseudopilot application in the form of HTML pages. The ground and tower radar maps are generated using offscreen LuciadLightspeed views and are refreshed once per second.

Really This Is Now Augmented

The RETINA project is currently in its validation phase. The systems are being used in a series of exercises where air traffic controllers are presented with varying levels of traffic density as well as different visibility conditions (i.e. fog). Throughout these exercises, we can objectively measure whether the addition of the AR overlays reduces the time controllers spend looking at the head down interfaces. And guess what? The first batch of RETINA validation exercises was successfully completed!

This was Luciad’s Hitchhiker’s guide to Augmented Reality in air traffic control towers in a nutshell. If you want to stay informed on the progress of the project, keep an eye on the website of the RETINA project.

Below you can find a video interview with Sara Bagassi from the University of Bologna at INTERACT 2017.