Mitigation of Runway Incursions by Using a Convolutional Neural Network to Detect and Identify Airport Signs and Markings

Runway incursions have resulted in incidents, confusions, and delays in airport operation. With the aim of reducing the risk of runway incursions, in this work, we investigate the use of a machine learning (ML) approach to detect and identify airport signs and markings to enhance operational safety especially in a low-visibility scenario. An artificial intelligence (AI) sensor for detecting the pixels developed and modeled using a convolutional neural network (CNN) is developed. In this design, the neural network outputs the feature vector model after the convolution operation. A filter is used to detect the pixels of the background image of the airport environment. The weight of the feature object is then added with a maximum pool layer after a convolution layer to find the feature map. The CNN is trained to demonstrate its capability in performing object detection and identification. It is expected that the proposed approach can be used to enhance airport operational safety and mitigate the risk of runway incursion.


Introduction
On June 26, 2018, two passenger planes (a Korean Airlines B777 and an Asiana Airlines A330) at Seoul's Gimpo International Airport in South Korea collided on the ground near a boarding gate at about 8:00 a.m. The tail of the B777 and a main wing of the A330 impacted during taxiing. The pilot made a mistake in the taxiway line, resulting in a "serious runway incursion" including aircraft damage. To reduce the number of accidents, in previous research, machine learning (ML) and unsupervised learning have been used as tools. (1) In this study, artificial intelligence (AI) sensor technology is used to rapidly and accurately detect the edge of pixels using a developed convolutional neural network (CNN) model. The main purpose of this study is to identify runway numbers and directions correctly when a simulated aircraft or an unmanned aerial vehicle receives tower radio instructions and taxis into a designated airport traffic location. The objectives include simulating the recognition neuron of images, inputting the original picture pixels, training the ML model to efficiently recognize the pictured object, training the model to identify the object's curves, lines, and text shapes using a virtual airport simulation environment, and improving the control tower, radar antenna, and other basic hardware. The airport simulation environment produces all picture pixels, inputs the neural network model, outputs the feature vector model after the convolution operation, and detects the pixels of the background picture of the airport environment by using the filter. Feature map filters activate the feature map, and with the same feature object having the same weighted value, a filter points to the connection of neurons and the same weighted value connects different neuron nodes initialized synchronously in a feature map, reduces the complexity of the feature map, and adds the maximum pool layer after the convolution layer. (2) The neural network's feature vector model can find the characteristic value of the feature map more precisely, and the contribution of the experiment results is clearer. The deep learning simulation improves the AI model in its environment and is used to simulate a drone recognizing the runway number automatically, guiding itself to a suitable air traffic controller command guide designed correctly. In the future, this AI drone is expected to assist a pilot in identifying airport pathway signage. The drone will be able to learn to identify the runway number, area location, and other specific images independently. In return, this function can assist in airport tower path planning to reduce the human workload, identify the runway number and runway out-of-control areas efficiently and rapidly, and provide correct judgment to reduce the occurrence of errors, thereby effectively reducing the number of aircraft accidents caused by serious runway incursion. (3) An AI model is constructed by using ML, and a CNN is used to mark the number of the runway and the position of the aircraft. (4) The "Airport Markings and Signs (AMAS)" feature map is built to assist the pilot in visualizing the dead-angle defect and simulate it. At present, only human vision identifies runway and taxiway numbers, with pilots driving and guiding the aircraft. The drone will add an automated driving system for aircraft after enabling the drone to identify specific images such as runway numbers and area locations. Efficiently and accurately identifying runway numbers and locations will generate improved pilot judgment, preclude errors, and reduce serious runway incursion aircraft crash totals.

Materials and Methods
Now that AI technology has considerably matured, our purpose is to experiment with airport signs and markings, using object detection to identify them with ML and CNN training to improve the accuracy of the identification of AMAS. In order to reduce the number of runway incursion events, this study is conducted to generate a functional ML model by using an AI model environment instead of manual recognition. The model enables a drone to recognize airport signs and runway markings automatically and correctly direct itself to the air traffic control (ATC) path planning runway area. The proposed approach can be used to enhance airport operational safety and mitigate the risk of runway incursion.

Objective
The experimental architecture uses CNN, which is a branch of AI and ML that employs a deep feed-forward (nonrecurrent) artificial neural network model system for analyzing visual images. The success probability of ML identification can be further improved by image learning. Considering the future development trend of passenger aircraft, the drone driving system will replace pilots' operation. Currently, the tower directs the aircraft to taxi to the designated runway area, and pilots visually identify the airport runway signs along the specified pathway. The neural network learning and identification model will be used to replace the retinal neurons of pilots in guiding the aircraft to enter the runway designated by the tower. (5) The automatic identification of aircraft runway markers eliminates the errors caused by human factors, greatly reducing the burdens of aircraft pilots in avoiding accidentally entering the wrong taxiway, prevents the chance of a serious accident or aircraft collision, and coordinates the airport's tower and ground control station task dispatch data link (C4ISR). (6) It can shorten the taxiing times of operating aircraft, rapidly distribute the taxiing sequence for several drones, and automatically taxi the drones in following the tower's flight sequence scheduling system. (7) In this manner, the system can avert collision damage and support the Doppler radar scanning warning system. The expectation includes effectively shortening the taxiing queue of the passenger aircraft. The results of this study will show an increase in throughput time for passenger flights during takeoff and landing.

Runway incursion record
The objective of the experiment was to increase the accuracy of a drone's runway identification as the primary research direction. From the International Civil Aviation Organization's (ICAO's) public investigation data on aircraft crashes, incidents, and casualties, as often reported in the media, a few cases are cited here to illustrate the importance of the research direction. Reducing runway incursions is one of this paper's top priorities and purposes. The model system was working to further reduce incursions and increase airfield safety. The notable accidents and taxiway safety issues are integrated into the form shown in Table 1. The runway incursion record of last years serious accident history still included the year 2018. In order to avoid the taxiing collision accident of drones, it is important to use AI.
On June 26, 2018, two passenger planes (a Korean Airlines B777 and an Asiana Airlines A330) collided on the ground of the airport near the boarding gate at about 8:00 a.m. The tail of the B777 and a main wing of the A330 impacted during taxiing. The pilot made a mistake in the taxiway line, resulting in a serious runway incursion including aircraft damage. Figures 1(a) and 1(b) show the damage to the planes from the serious runway incursion accident due to pilot deviation (PD). The FAA website provides a total of 1832 runway accidents up to 2018, as shown in Table 2. In addition to strengthening tower control, pilot training, and accident prevention measures, it is envisioned that AI wisdom should be introduced to reduce the rate of human error and predict the risk of possible crashes. In this paper, an application of deep learning, in lieu of purely human judgment of taxiway number and precise location, is explored as a means for the automatic management of the aircraft taxiing sequence as an aid in eliminating aircraft crashes, owing to the low probability of AI error.
Abbreviations: OI = operational incident, which is a surface event attributed to ATC action or inaction, and PD = pilot deviation action of a pilot that violates any federal aviation regulation.
The airplane was taxiing after landing at Nagoya-Komaki International Airport when an F-86D Sabre collided with the aft fuselage and tail section of the C-47 while attempting to take off, killing 3 of 33 on board.

Purpose
Deep learning is used to simulate neuron activation during the human visual recognition of images. In this study, a deep learning model has been constructed and trained for this purpose. The model can independently recognize the curves, lines, characters, and numerical shapes of objects in pictures, and it operates by utilizing CNN. After processing, the model outputs feature movement vectors. Eventually, various signboards become recognizable. The main purpose of this study is to identify the runway number correctly when a simulated aircraft or an unmanned aerial vehicle receives the tower radio instructions and taxis onto a designated airport pathway. The experiment takes Taiwan Airport as the model environment. The pilot maneuvers the aircraft onto the correct taxiway intersection area and awaits the tower's instructions. Taxiing an aircraft is different from driving a vehicle. There must be no slip into the wrong runway when driving on the runway even a slight error may cause damage to the aircraft or equipment, and a major mistake can cause casualties. Several cases were previously cited in this article. If the pilot makes a mistake due to human negligence and enters the wrong airport runway, the operational capability of the aircraft is not equal to that of vehicles that can more easily dodge an obstacle and/or rapidly brake to a stop. All impacts will cause serious equipment damage, as well as possible injuries and deaths.
Global technology giants such as Google, Facebook, NVIDIA, and INTEL continue to promote AI technology, and AI technology has greatly matured. (11) The experiment described herein attempts to use ML in training a model system to improve the accuracy in identifying airport signs, thereby reducing the burden on pilots who must currently seek identification using the naked eye. The AMAS are input into the AI model system, and the feedback data of the image sensors can be identified by self-learning. After correctly identifying the feature map of the airport runway number, the CNN system is used. Continuous training improves the prediction accuracy in recognizing pictures and characters. In the paper, CNN is applied to learn the identification of airport runway numbers. In the future, drone pilots will surely replace manual pilots in operating passenger aircraft. The airport tower will automatically arrange the taxiway of the incoming aircraft and specify the location of the airport runway for takeoff or landing. In order to avoid the taxiing collision accident of drones, it is important to use AI. In the future, drone pilots will be able to reduce human error in path planning and overall reduce the airport tower and pilot workloads. The main objectives of the experimental study are to develop a new technology to link the tower with the ground airport control station center, shorten the waiting time for aircraft operating on taxiways, and apply a deep learning model during take-offs, landings, and taxiing. In the future, this system will be able to handle more drones more quickly and increase the airport's flight schedule runs per minute.

Methods
The model structure of the experimental method is to train a computer using AI to identify the airport runway marking images. The preliminary concepts can be used for image recognition. CNN or ConvNet is widely used in visual image recognition research. In the 2012 Image-Net Challenge, a deep CNN algorithm was used for the first time to challenge visual image recognition. CNN attracted many scholars to study the architectural design of the deep CNN and showed the best answer in the video "How to teach computers?" (12) Being able to recognize a picture and explain its composition and commentary details was certainly an achievement that no teams could achieve in the past. With the efforts of scientists, it has now become possible to achieve in-depth learning, improve the ability to recognize pixels in the picture, artificially identify airport-marked images, and use CNN for deep learning. The following are used: TensorFlow, Caffe, Keras, MXNet, and so forth to train computer methods, a brief description of the algorithm mathematical CNN main writing framework, reference papers to study activation functions (such as ReLU, Sigmoid, and ReLU activation functions), loss function, weight update, and a forward mathematical model construction training framework of propagation, back propagation, and optimization. (13) In the processing of airport runway marking images, a gray scale image is used as the input marking information image, and ReLU and Sigmoid activation functions are used to construct nonlinear attributes of CNN. The cross-entropy loss function is used to calculate the distance between the predicted value and the real value. The CNN architecture consists of a volume layer, a pool layer, and several full connection layers. The experimental method simulates the aircraft software in constructing a complete set for the airport environment, including the input and retrieval of pixel images for runway signs. (14) The application of deep learning image recognition and AI technologies is intended to create a system that assists pilots in taxiing along airport pathways and decreases the human error events of aircraft mistakenly intruding upon runways. (15) The success probability of aircraft taxiing into the taxiway intersection area designated by ATC is high and the investigation of aviation safety records in the past year is still common. Accidents caused by aircraft crashing into the runway by mistake are also common. According to ICAO, cases of serious runway incursion by aircraft, which contain the historical records of aircraft crashes from 2006 to 2018, have been recorded. The number of aircraft accidents worldwide was 1832 (FAA, Runway Incursion Totals for FY 2018), and there are many aircraft taxiing accidents at airports. The record shows that there are multiple serious aircraft crashes with casualties of more than five, reaching into hundreds of casualties.
In the experiment described in this paper, AI and ML technologies are used to train the identification rate by using CNN. Experimental methods are used to simulate neural networks for human learning, visual learning, and identification. (16) The resulting AI model will be able to identify an airport environment by continuous training. Runway marking patterns, including runway number, running direction, and suspension line, give the intelligent decision-making system correct taxiway plans and paths, effectively improving the identification rate of AI learning. The experimental method in this paper can encourage the ICAO to consider adopting the use of AI, via testing unmanned aerial vehicles. The entire system consists of passenger aircraft runway sign pictures, AI identification model system establishment, digital image identification operation rules, improvements to airport runway operation systems using ML, and system links, as a means to reduce unsafe aircraft movements.

Software
The software environment is shown in Fig. 2. The deep learning model found the characteristic value of the feature map more easily, and the contribution of the experimental results is apparent. The software environment used VM Service software to build an Ubuntu 16.04 system environment for frameworks in DGX-1 Hardware Basic Framework.
The hardware and software architecture shown in Fig. 3 should be installed using NVIDIA DGX-1 Server. The most important reason for the final selection of DGX-1 Server is that this device can be used to build a deep learning framework to obtain a 4X speed or higher for CUDA core operation accelerators, and according to the original manual, the universal commercial GPU core system uses this deep learning framework with optimized versions.

Hardware
The experimental hardware environment used NVIDIA DGX-1 Server to build a deep learning platform as a good choice of services and GPUs. The deep learning model simulation speeds up the process in the AI model environment and simulates the pilot for recognizing the runway number automatically and correctly. In the near future, drones may replace pilots as an accurate captain to minimize accidents.

Result
Simulated experimental environments are limited to the establishment of the Geographic Information System (GIS) logo at Taipei Song-shan Airport in Taiwan. According to the airport logo, CNN training using ML is used to increase the identification label gradient rate. (17) The label data is defined first according to the official document "Airport Signing System Standards (Document No. 150/5340-18G, AC 5340-18G-New Proposed Signs)" published by the Federal Aviation Administration of the United States Department of Transportation (FAA) in 2015, as shown in Fig. 4. The material gallery shows a picture of global airport marking specifications. The model is the basic information map needed for this study method. The library and principle operation tools adopt the CNN model of ML technology to do ML training, and further adopt the well-learned deep learning model. (18) After the nonlinear transformation of highdimensional function values, in the link propagation activation model, each link has appropriate weight and cost. The intended result is a numerical model generated by the experimentation and training. (19) Five hundred runway pictures (pixels: 1024 × 1024) of the simulated airport moving and operating area were taken as samples. Although the airport environment was complex, the AI model identified and judged the pictures in accordance with runway designation signs. As a featured vector model, the designs of filters are listed as follows: (1) runway vacated signs, (2) location signs, (3) taxiway direction signs, (4) taxiway direction signs and markings, and (5) destination signs and other indicators, as shown in Fig. 5. The deep learning model learned to identify 500 airport runway and taxiway pictures, and to output the images of the indicators it had successfully identified. The experimental results show that the deep learning methodology succeeded according to the expected experimental purposes. (20) The model can independently identify the different types of signboards and marks. The experimental results affirm that the AI model training results achieved the experimental objectives. (21,22) The deep learning model can identify 500 simulated airport environment pictures effectively and independently. Various indicators included in the simulated airport environment pictures were clearly marked, and the recognition success rate is more than 80%. The output identification for various types of indicators matched the applicable portions of the input pictures, with the successful identification of the indicators including the naming of (1) runway vacated signs, (2) location signs, (3) taxiway direction signs, (4) taxiway direction markings, and (5) destination signs or other indicator images. Output pictures with AI labels are shown in Fig. 6.

Discussion
The overall recognition rate can reach 80% or higher. In a complex environment with the rectangular framing of airport vehicles, buildings, picture angles, and so forth, the accuracy of the object is reduced to 60%. Future research work is necessary to improve the accuracy of the model identification, with the goal being output pictures as shown in Fig. 7. Table 3 shows the identification rate.

Conclusion
The purpose of the experiment was to construct an AI sensor training model to provide drones or pilots with the ability to accurately and efficiently identify various indicators contained in simulated airport environment pictures, and to employ deep learning in augmenting the AI sensor model. The experimental results described in this paper have proven that the trained deep learning model can identify and mark the runway designation sign pictures within the areas of aircraft motions at an airport. The simulated drone automatically identified runway numbers and directions, and succeeded in properly entering the tower's plane pathway to a runway. Therefore, the AI sensor model system can aide pilots by assisting them in identifying runway numbers and their relative locations. The drone learned to independently identify runway numbers, area locations, and other specific images as a possible solution for precluding runway incursion aircraft losses and accidents.