Neural Network Implementation for Lane Tracking in Self-Driving Cars

I. INTRODUCTION

Making driving safer and more comfortable has been the goal of engineers in the automotive industry over the years. Many advance features have been introduced and implemented in cars over the years to achieve this goal. One of these features is the Anti-Lock Braking System (ABS), a safety anti-skid system used to prevent wheels from locking up during breaking by maintaining tractive contact with the road surface. Electronic Stability Control (ESC) is another feature introduced by automotive engineers; ESC is a computerized technology that improves car stability by detecting and reducing loss of traction. Rear view cameras were introduced to prevent minor collisions between cars. Airbags were also used to protect occupants in collisions. Recently, some Advanced Driving Assistant Systems (ADAS) such as Lane Keeping System (LKS), Cross Traffic Alert (CTA), Cross Traffic Breaking System (CTB), Cruise Control (CC), Adaptive Cruise Control (ACC), Blind Spot Information System (BLIS), Automatic Emergency Breaking (AEB), Active High Beam Control (AHBC) and many others, have been successfully integrated into cars for safety and comfort.

In addition to these new features, millions of dollars have been spent on campaigns to inform the public of the dangers of drunk driving, failure to buckle up, and texting while driving $[22]$. Despite the new features added to cars for safety and the money spent on campaigns for safe driving, the number of people that die on roads and highways increases every year. Based on a report by the National Safety Council, a non-profit organization that works closely with federal auto-safety regulators, 42,200 people died in accidents involving motor vehicles in 2016 $[19]$, a six percent increase from 2015.

Some of the key factors that often play a major role in car accidents are:

• Distracted Driving: Distracted driving has been identified as one of the major causes of accidents. Drivers are typically distracted by using their cell phones, talking to people in the car, greeting friends in other cars, eating while driving, singing to songs and many other activities that take their attention from the road. According to research conducted by the National Safety Council, 25% of car accidents in America are related to the use of cell phones [11].

• Fatigue of the Driver: Fatigue of the driver is another major cause of accidents. A tired driver might have impaired judgment and/or reduced time to react on the road. In 2014, the NHTASA's FARS database shows 864 fatalities were caused by drowsy driving [8].

• Drunk Driving: Driving while under the influence of alcohol is the most highly reported cause of car accidents. National Highway Traffic Safety Administration (NHTSA) reported that alcohol is considered a cause for over four-hundredth of automobile accidents. 44,574 drivers in USA that were killed in accidents involved alcohol in 2013 [11].

• Over-Speeding: Centers for Disease Control and Prevention (CDC) noted in 2013 that over-speeding contributed to twenty ninth of all fatal crashes, claiming 9,613 lives. About 1 out of 3 car accident-related deaths involved over-speeding [11].

According to the statistics above, the major cause of accidents are human errors. The government, in its effort to make driving safe has put in place sanctions and measures, yet car accidents still claim more lives each year. Engineers and scientist are exploring the option of building intelligent cars, mostly called self-driving cars, to prevent human errors in accidents.

Intelligent driving has been the interest of both academic and commercial spheres over the past few years. It has been discovered that the development of efficient algorithms is the key factor in developing self-driving cars. The prospects of the application of computer vision and deep learning in self-driving applications has provided solutions to some key problems in self-driving applications. Despite the interesting results achieved, the problems of lane detection and tracking, vehicle control, mapping, scene perception, and localization in self-driving cars remain a challenge that has not been fully solved. Many of the available self driving cars are driven in a supervised environment, which is not a good representation of the real world. There is countless number of variables and conditions that must be captured in the development and testing of self driving cars before they become available to the public. Among these issues, lane tracking is one of the important features in self driving car development that requires additional attention. Many algorithms have been proposed and implemented for lane tracking $[6]$. Lane tracking requires significant computing power, making it difficult for some of the existing algorithms to meet the real-time requirements of tracking lanes on the road. In addition, some of the existing algorithms are not robust enough to handle different weather and road conditions, like faded lane markings, obstructions on lanes, or an absence of lane markings.

This research work is aimed at solving these problems. An efficient algorithm was developed on GPU using the NVIDIA Jetson TX2 to meet the real-time needs of lane tracking. The algorithm is implemented in a robust way to handle different road conditions. The algorithm is then optimized by developing a neural network model that can track lanes by learning from the output of the algorithm.

II. PAPER OUTLINE

The rest of the paper is organized as follows. Section 3 provides a general overview of the self driving car architecture. Section 4 describes the challenges in lane detection and tracking. Section 5 summarizes the novel contributions of this research work. Section 6 describes a literature review of self driving cars, explore available implementations and setbacks in these implementations, and suggest potential improvements.

Section 7 provides a brief overview of Graphical Processing Units and the NVIDIA Jetson TX2 used for the implementation process. The implementation of the proposed algorithm is described in Section 8. The evaluation of the implementation is evaluated in Section 9.

Summary and potential future works are presented in Sections 10 and 11. The entire layout of the paper is shown as a chart in Fig. 1 to help the readers understand the structure of the paper.

III. SELF DRIVING CAR ARCHITECTURE

Self driving cars mostly have various subsystems that are integrated to form a full functioning system. Some of the commonly used subsystems are: Sensor, Perception, Planning, and Control subsystems.

3.1 Sensor Subsystem

Sensors are hardware components used by self driving cars to gather data about their environment. Just like humans use their sense organs (e.g. ears, eyes) to perceive their environments, self-driving cars use different sensors to capture and perceive their environment. Some of the popular sensors used in self driving cars are Lidar, Radar, Cameras, and GPS sensors. Self driving engineers usually combine data from these sensors through a technique called sensor fusion, the fused data is then passed on to the perception sub-system where the self driving car extract useful information that helps it to navigate without accident.

3.2 Perception Subsystem

The perception subsystem processes the fused data from the sensor sub system into a well structured information that can later be used for path planning and control. This is the stage where a self driving car analyzes its environments. Perception subsystem can be further divided into two underlying subsystems; detection and localization.

1. The detection subsystem is responsible for understanding the surrounding environment. The detection subsystem includes software components like lane detection and tracking, traffic sign and traffic light classification, object detection and tracking, and free space detection.

2. The localization subsystem is responsible for using sensor and map data to estimate with minimal error the vehicles precise location.

3.3 Planning Subsystem

The data that has been processed by the perception subsystem is used by the planning subsystem for the self driving car to plan its path. Some of the components of the Planning subsystem are route planning, prediction, behavioral planning, and trajectory planning which are described below.

1. Route planning is where high level decision about the path a self-driving should take between two points on a map; for example, which roads, highways, or freeways to take.

2. Prediction component estimates what actions other cars might take in the future; for example, if a self driving car wants to merge to highway, it needs to know if the approaching car in the lane it wants to merge in will either change to another, slow down for it to merge, or speed up.

3. Behavioral planning component determines what behavior the vehicle should exhibit at any point in time. It determines the action of a self driving car at each point, for example changing lanes, accelerating or decelerating, stop-ping at a traffic light or intersection, or making a left turn onto a new street

4. The trajectory planning plots the precise path the self driving car should follow.

3.4 Control Subsystem

The last subsystem we have is the control subsystem. This system ensures the self-driving car follows the path generated from the planning sub system. This subsystem usually includes components like PID controllers, model predictive controllers, or some other controllers. The control subsystem is responsible to send acceleration, braking, and steering commands to the self-driving car.

We show how these systems interact with each other to form the architecture of a self-driving car in Fig. 2. This research paper is focused on the perception subsystem which is the most challenging in designing self-driving cars. There are many components in the perception subsystem; our interest is in the lane detection and tracking.

IV. CHALLENGES IN LANE DETECTION AND TRACKING

Lane detection and tracking is one of the components in the perception subsystem. It primarily uses frames from the video feed taken by the camera; hence its input is from the camera in the sensor subsystem. The lane lines on the road are used to guide cars to move from one point to another without going off track i.e. drive only on the drivable path. It is usually easy for human drivers to follow lane markings on the road or even use intuition if the lane markings are not visible. The task is now to make self-driving cars do the same job as human or even better. Self-driving cars must detect and track lane lines in all conditions with minimal or no errors to avoid driving off-course. Lane detection and detection might be easy for human drivers but difficult for self-driving, some of the challenges of lane detection and tracking are as follow.

4.1 Visibility of Lane Lines

Visibility of the lane lines is the major challenge in lane detection and tracking. Lane lines must be clearly visible with no obstruction for a self-driving car to detect and track it. Most roads have faded lane lines, worn-out lane lanes, obstructions, while some have no lane markings. If self-driving cars are built with assumption that lane lanes will be present and clearly visible all times, they will crash on some roads. Most self-driving car manufactures are testing their cars in a controlled environment on good roads with visible lane markings. For self-driving cars to be accessible for all on roads, this issue must be solved.

4.2 Weather Conditions

Lane detection and tracking heavily depends on the data captured by cameras in self-driving cars.

Cameras need clear sight to capture lane lines on the road. There are some weather conditions that makes visibility so hard that human drivers sometimes can't see through. Weather conditions like heavy rainfall, snowstorm, fog, and sometimes exhaust from other vehicles will make it almost impossible for a self-driving car to detect and track lane lines. The challenge is to make self-driving car handle these conditions and still drive safe.

4.3 Obstruction on Road

Unwanted materials or objects on the road can make it difficult for self-driving car to track lane lines successfully. Also, there might be leaves on the road covering lane lines during fall season. This condition is not common, but it needs to be accounted for because one occurrence of it might lead to accident that will take human life.

4.4 Latency

Latency is a major challenge in designing a self-driving. A millisecond delay in response time might cause a serious damage. A self-driving car must be able to capture, process, and make decisions based on every lane lines in a frame at a very fast rate. There are several ways to solve this problem; one way is to have used a very power core in the car for processing videos from the camera, it is also possible to use more than one core in the car, this is called multicore. Graphical processing unit is the mostly widely use device. It usually has thousands of cores that can be executed in the parallel which makes it a better choice for image processing in self-driving cars.

V. LITERATURE REVIEW/RELATED WORK

According to the report in [24, 20], it has been argued that humans are generally not good at driving. This is usually true when humans are driving under the influence of drugs, driving with partial attention on the road, or drunk driving [14]. Even if humans can't drive safe all the time, developing a self-driving car that drives better than an average human is hard to come by [4]. Self-driving cars cannot drive better than an average human until they are able to sense the environment on their own, perform computations, and make better decisions. One advantage a self-driving car has over humans is that it cannot be distracted. Adversely, it cannot also make decisions outside what it has been trained to handle. Therefore, the task is to develop self-driving cars that make the same or better decisions than humans can. A major achievement was accomplished in the 2007 DARPA Urban Challenge, when six autonomous cars successfully navigated an urban region to the finish line, out of eleven total cars. The first-place finisher could travel at an average speed of 15mph [16].

It has been estimated that by 2050, self-driving cars will be widely accepted and implemented compared to human-driven cars. This estimation is due to the increase in research and investments into self-driving cars. Fully autonomous cars are expected to be widely accepted in 2035 [15, 28].

The following are the features of a fully autonomous car as described by the Society of Automotive Engineers (SAE):

1. A fully autonomous car must be able to finish a journey from a location to another with no input from a human except for defining the destination at takeoff.

2. A fully autonomous car must be able to successfully drive on all road's conditions.

3. A fully autonomous car must always be able to adhere to all traffic rules.

The need for and expansion of self-driving cars has been controversial. Detractors contend that if we can arrive at a place where humans always obey all traffic rules, we should not invest valuable resources into the development of self-driving cars. However, supporters of autonomous cars believe that setting up a system where humans obey traffic rules always is nearly impossible. Instead, driving can become safer and more enjoyable if we develop self-driving cars mimic human driving while obeying all traffic rules and regulations. Besides saving lives and properties, the widespread adoption of self-driving cars is anticipated to have many positive benefits including: reducing traffic congestion by a factor of 10 [5], helping to cut down the number of cars on the road, increasing speed limits, increasing leisure time and productivity during commutes [23], and removing restriction for vulnerable groups [25].

Some notable research work has been carried to develop effective lane tracking algorithms for self-driving cars. In this paper, we explore past work in this area, identify its limitations, and provide solutions for the gaps. Qui et al. [21] used Gabor filter, an edge extraction method to develop algorithm that detects and tracks lanes in real-time. In their study, the features in the lane markings were enhanced using the Gabor filter to make it more visible, and Hough Transform was used to detect the lane markings under the restriction of the polar angle. The researchers used the location of the previous frame in the video to determine the region of interest for the next frame. They tested their algorithm on multi-lane and multi-scene roads with fast moving cars and were able to track the lane markings. Despite their positive contributions, one major setback with this implementation is that it cannot handle roads in bad conditions. Their algorithm would fail on a road with no lane markings, obstructed lanes, or faded lanes.

In another study, Lee et al. [13] used simple filters as well as Kalman filters to detect and track lanes. The focus of their work was to develop an algorithm for the Lane Departure Warning System. They successfully designed a commercially ready lane departure warning system on an IMX6Q board. They implemented their algorithm in real-time and also proposed a lane detection and tracking solution. Their method yielded 97% lane detection accuracy at daytime and 95% accuracy at nighttime. They were also able to run the algorithm at 15 frames per second. Although they achieved good results, their algorithm would not work on a road that is not in good condition or is without visible lane markings.

Kun et al. [28] developed a spline-based system for multilane detection and tracking. They used the extended Kalman filter tracking with their new method 'Catmull-Rom spline' to detect and track lane lines. They used the Kalman filter for the robustness of their model. They didn't make the assumption about the parallelism and the shape of the lane markings, and as such, their algorithm was able to get detailed information from the road. The algorithm was also able to detect all lane markings visible to the camera. To evaluate the performance of their implementation, they drove a test vehicle on roads with different conditions, which included worn out lane markings, construction sites, narrow corners, and exits and entries of the highways. The results of their implementation were significant; however, their implementation was not robust enough to include roads with no lane markings at all.

Chanho and Ji-Hyun [12] designed lane detection and tracking system for real-time applications. They were able to provide solutions for major problems in lane detection and tracking such as, poor visibility in bad weather, lack of clarity on lane markings, light reflections, and shadows. Researchers were also able to use an efficient region of interest to reduce the high noise and the calculation time. They used a line clustering with scan-line tests, gradient cue, and color cue to verify the characteristics of the lane markings. The accumulated statistical data was used to separate false lane and real lane markings. They designed their system to be able to handle bad weather conditions to some degree; however, their system would still fail in the absence of lane markings.

Assiq et al. [1] designed a vision-based algorithm for lane detection in self-driving cars. The images of the road were first converted to grayscale, followed by the use of a noise reduction technique to remove unnecessary information from the image. Additionally, Canny edge detection was used to detect the edges of the lane markings, while Hough Transform was used to draw lines on the lane marking edges. The lane boundaries were first detected and subsequently, the information was used to predict lane markings position. Assiq et al. [1] were able to create an algorithm robust enough to handle various lighting conditions and run in real-time. Their algorithm reportedly performed well on roads with or without lane markings and curved and straight roads in different lighting conditions. The result of their implementation provided promising results;

nevertheless, it cannot meet the real-time requirements of self-driving cars.

In a work related to Assiq et al. [1], Bounini et al. [3] developed an algorithm that uses lane boundaries and painted lines to track lanes for intelligent and autonomous vehicles. They used techniques like Canny edge detection, Hough Transform, least-square method, and Kalman filter. They used Hough Transform to initialize the algorithm at each time needed; least-square method and Kalman filter to reduce the adaptive region of interest. The downside of their work is the fact that their algorithm runs only on the CPU which cannot meet the real-time needs of self-driving cars.

A real-time Lane Detection and tracking was developed by Hajjouji et al. [9] on FPGA. They were able to detect lane lines by using Sobel operator with an adaptive threshold and Hough Transform. They used Kalman filter for tracking the detected lane lines. In order to simplify the Coordinate Rotation Digital Computer (CORDIC) algorithm, they used the gradient directions of the edge state, which made their algorithm meet the real-time requirements of tracking lane lines. The result from evaluating their algorithm shows the implementation can effectively detect and track lanes in different lighting conditions. Their result is encouraging; however, the algorithm cannot detect and track lane lines effectively in different road conditions, including those with faded lane lines, obstructed lane lines, and other factors that affect the visibility of lane lines.

In our recently published paper, we presented a GPU implementation for Lane Tracking in Self-Driving Cars $[26]$. Our Implementation was based on techniques such as Image Capturing, Gray scale conversion, Edge detection, Noise Reduction, Line Detection, Region of Interest, and Lane Boundary Extraction. Our implementation was able to detect and track lane lines on roads with different conditions. We were also able to track the lane-lines in real-time by using NVIDIA GPU for our implementation. We measured the processing time for the CPU and GPU implementations and recorded speedup of 20X. The performance of the algorithm was tested with real road data (video) and the accuracy was up to 95%. The idea of this implementation is to extract the lane boundaries and use a mathematical model to map the lane boundaries to lane line locations. In our current research, we made this process more robust by developing a neural network model that is trained by the mathematical model. The model is then used instead of the mathematical model to make the implementation more robust.

VI. NOVEL CONTRIBUTIONS

There have been a lot of research works done in the literature on lane tracking, some of which address specific problems and provide solutions. Our contribution to this topic is to address the challenges explained in section 4 of this paper. We implemented an algorithm that can detect and track lane lines when lines are visible, and accurately predict where lane lines should be when they are not visible or obstructed. We were able to handle some weather conditions like when visibility is not too bad, but situations like snowstorm and minimal to zero visibility were not covered. We solved the latency by using the NVIDIA Jetson TX with GPU for the implementation. Generally, we implemented a neural network model that can detect and track lane lines in most conditions, specifically, when lane lines are not present.

VII. GRAPHICAL PROCESSING UNITS

A graphical processing unit is a chip that is typically used for rendering images, by performing rapid mathematical calculations $[27]$. It accelerates the creation of images in a frame buffer by rapidly manipulating and altering memory using its specialized electronic circuit $[10]$. GPUs are designed for speeding-up applications that are based on image processing. Some of the differences between GPUs and CPUs are as follow:

1. CPUs are powerful for latency-oriented applications while GPUs are powerful for throughput-oriented applications.

2. The Arithmetic Logic Units of CPUs are designed to reduce operation latency. GPUs

Arithmetic Logic Units are designed to be energy efficient with long latency.

1. CPUs use large caches to convert long latency memory accesses to short latency cache. GPUs are designed with small caches to help boost memory throughput.

Computer Engineers often integrate a GPU with a CPU on the same board, either in the motherboard of a personal computer, or on a graphics card $[18]$. As the processing power of GPUs increase, it became a widely accepted choice for other computing-intensive tasks not related to graphic processing. GPUs were used in modelling and scientific calculations, providing computing power needed for machine learning and artificial intelligence software.

7.1 Nvidia Cuda

In 2007, NVIDIA developed and released Computer Unified Device Architecture (CUDA) [18]. CUDA is a comprehensive hardware and software architecture for Generaln Purpose computing on Graphics Processing Units (GPGPU). NVIDIA started exploring GPGPU and High Performance Computing (HPC) after developing CUDA. Their design involved combining large programmability, performance, and ease of use.

CUDA supports two different types of programming interfaces:

1. Device level APIs: This involves the use of GPGPU standard DirectX. It uses the high-level shader language (HLSL) to implement compute shaders.

2. OpenCL: OpenCL is a standard that was developed by Khronos group. OpenCL is used for writing OpenCL kernels.

Device level APIs and Open CL are usually implemented on GPUs from different vendors because they are independent of GPU hardware. Another device level approach which directly uses the driver is CUDA programming. CUDA programming gives the programmer enough control but is complex to use $[26]$. This approach gives the programmers the ability to make use of native support for high-level languages such as C,

C++, FORTRAN, Java, and Python, reducing code complexity and development costs.

7.2 Jetson TX2

Applications such as lane detection and tracking in automotive require a lot of computing power. This constraint coupled with the real-time requirement makes lane detection and tracking a difficult task. Lane detection and tracking in real-time in self-driving cars necessitate the use of a super-computer to effectively deploy these processes. Embedding a super-computer in a self-driving car is not feasible in terms of cost. The introduction of GPUs provides an alternative to super-computers. GPUs are of most value for image processing implementations because the type of calculations they are designed for is like those calculations used in image processing. Most information that's utilized in image processing is delineated within the kind of matrices, and GPUs are well-known for high computational performance on matrix operations.

GPUs in comparison to CPUs are much more fitting to perform matrix operations and other types of advanced math-ematical transformations $[17]$. For this reason, lane tracking runs faster on a GPU as opposed to a CPU. The implementation in this research work is therefore done on NVIDIA Jetson TX2. Jetson TX2 is an embedded system-on-module that has quad-core ARM Cortex-A57, 8GB 128-bit LPDDR4, and an integrated 256-core Pascal GPU.

As reported by NVIDIA at the time of the development of Jetson TX2, it was considered the fastest and most power efficient embedded computing system $[17]$. It runs on 7.5-watts and is best known as a supercomputer. Jetson TX2 can be integrated into a wide range of products, with minimal effort due to its variety of standard interfaces. Jetson TX2 is most useful for developing computer vision and deep learning applications; it runs on Linux. We used Jetpack for our implementation. Jet-pack comprises of the essential libraries required to program a Jetson TX2 device.

VIII. IMPLEMENTATION

The programming language used for developing the algorithm in this research work is Python. Python is the most utilized programming language in artificial intelligence applications. It encompasses a great number of inbuilt libraries for artificial intelligence and machine learning. The algorithm was implemented on CPU with an Intel i5-6330 processors running at 2.5GHz and on a Jetson TX2 device with 256 CUDA cores. The algorithm was implemented on both platforms to measure the performance advantage of using GPUs for lane detection and tracking. The implementation was tested using a large data set of videos taken from cameras attached to the front end of a car. This dataset was provided by Berkeley Deep Drive [2]. The dataset contains videos of distinctive properties. The detailed description of the implementation is described in the following sections.

8.1 Data Capturing

The primary input to the system is a sequence of colorful images taken from a camera attached to a non-stationary car. These images are frames extracted from real-time video taken by the camera. The camera is typically placed at the front-view mirror within the car for a good angle of vision. The images are extracted from the video in real-time and saved for processing in the on-board computer memory. These images are then fetched frame by frame from the memory for processing. The process of saving, fetching, and processing video frames must be done in real-time to avoid unexpected behaviors in a self-driving car.

The data used for training the neural network model used in this research was taken with a camera attached to the front view of a moving car. The camera is attached in a way to capture the drive-able area on the road, the road boundaries, lane lines when visible, obstruction on lane lines, and other features that will be useful for training the model. The video data will later be fed into the neural model frame by frame for training process.

8.2 Data Pre-Processing

The input to the neural network model is a sequence of video frames and the expected result for each frame is the round boundaries. These road boundaries will then be used to estimate the location of the lane lines. We need to train the model with a labeled data, which means we need to extract the lane boundaries from our training data. There is extra information in the data that are not required which must be removed. The data might also have noise that reduces the accuracy of the implementation. Data pre-processing is the process of removing extra details and noise in video frames.

Noise exists in most everyday applications, including in computer vision systems. Gaussian Blur is a technique in image processing that is used to blur images, a process which we undertook to remove the noise in our implementation. It is widely used in graphics software to lower image noise and cutdown details. The result of this blurring on an image creates a smooth blur that is like viewing the image through a translucent screen. In computer vision, Gaussian smoothing is mostly implemented during the pre-processing stage to enhance image structures at different scales.

8.3 Region of Interest

The frames from the video captured by the onboard camera contain a great deal of data, but we only need information about the road boundaries. For this reason, a region of interest is used to narrow down the searching range for features on the road. ROI comprises of the major information needed to be implemented to lower computation cost of unwanted data. This process is mostly used to determine the region of interest in the video frame. This was accomplished by drawing a polygon along the area that is needed in the entire video frame. This is a critical task as it determines if the selected area will contain the necessary data to detect the lane boundaries.

8.4 Data Labeling

Data labeling is a machine learning technique used for tagging data with labels used for training models. The process typically involves taking a set of unlabeled data and argument each piece of the data with meaningful tags. We have a set of unlabeled video frames from the camera attached to the car, this is the primary input to our implementation. In order to train our model with the video frames, we need to label them. For each video frame, we will label it with the locations of the boundary pixels. Doing this, we can teach our model how to locate boundary pixels in any video frame.

To estimate the road boundary data used for labeling the video frames, we used the edge detection technique. Edge detection is a common computer vision technique used for finding the boundaries of object within an image. The boundaries of a road are the sharp contrasts between the road surface and painted lines or other kinds of non-pavement surfaces. The edges in an image provide the sharp contrasts. It is therefore important to use edge detection to isolate the boundaries of the road. This process also lessens the learning data by simplifying the image considerably. There are many edge detection algorithms which have been implemented including Canny edge detection, which is considered one of the best edge detection algorithms. Our implementation is based on Canny edge detection. Canny edge detection is specifically used for detecting the edges in an image while disposing of other data. The maxima of the partial derivative of the image function M in the direction orthogonal to the edge direction, and smoothing the signal along the edge direction is derived by:

where,

Canny edge detection implements a multi-stage algorithm to detect wide ranges of edges in images. The Open CV implementation of Canny edge detection was used in this work. It requires two parameters in addition to the input image, a low and high threshold that determines if an edge should be included. The threshold at a point defines the change in intensity at that point. All the points above the high threshold will be included in the resulting image, while the points between the threshold values will only be included if the edges next to them are high threshold or included in the resulting image. Edges below the low threshold are terminated. The recommended low: high threshold ratios are 1:2 or 1:3.

8.5 Deep Neural Network Model

In our previous paper $[26]$, the left and right road boundaries were used for calculating the location of the left and right lanes using a mathematical method we formulated. This mathematical calculation is applied to each pixel in the frame to estimate the location of an equivalent pixel that corresponds to a lane marking. This process is time consuming and might reduce the accuracy of the result. To solve this problem, a deep neural network was developed to replace the mathematical model. The network was trained using the data collected from our previous implementation.

The deep learning model used in our research is Keras $[7,7]$, a Tensorflow API. Keras is a high-level API that is mostly used to build and train deep learning models. Keras is popular for its use in fast prototyping, advanced research, and production. It has the following advantages:

1. Keras interface is consistent and simple. It is optimized for common use cases.

2. Models developed with Keras are modular and composable.

3. Keras models are easily extendable. It is easy to create new layers on existing models.

8.6 Keras Model Implementation

The Keras model applied in this research project is used to implement a regression. The model we developed has two densely connected hidden layers, with one output layer as shown in Fig. 3. The two connected hidden layers are used to feed in the left and right edges of the road, while the output layer gives the right location of the lane marking. The model was trained for 1000 epochs before we achieved a minimal error in the predictions. Each input node is connected to each output node.

{"image_source":{"path":"images/6cb55a987dba2e2c709baeb4185c26f2f6b4512ccff114e7893c59e8f65f3df3.jpg"},"content":"","chart_caption":[{"type":"text","content":"Fig. 3: Structure of the neural model implemented"}],"chart_footnote":[]}

8.7 Keras Model Training Process

The Keras model is used to implement a linear regression model that is trained to predict an output based on some sets of inputs. The inputs for this model are the left and right boundaries of the road and the output, the right and left lanes, are then estimated based on the assumed lane width.

The data used to train this model is extracted from the video. The true values of the road boundaries are extracted and the right and left lanes are recorded. The data is then saved in a file of ".data" extension. To avoid the model being trained with corrupted data points, the data is first cleaned. Pandas, an open source library for data manipulation and analysis is used for data extraction, analysis, and cleaning.

The data collected is then separated into two sets-a training and a testing set. The training dataset is used to train the model after it is created; this set comprises 80% of the total data. The testing dataset is used to evaluate the performance and accuracy of the model after training; 20% of the total dataset is used for testing the model. The joint distribution of some of the data is shown in Fig. 4.

{"image_source":{"path":"images/889bb31ef274d6ace1ddfb20a29b2ce42c713cf2cb994044ecf065a83e728356.jpg"},"content":"","chart_caption":[{"type":"text","content":"Fig. 4: Joint distribution of a few pairs of columns from the training"}],"chart_footnote":[]}

The model is then created and built after data analysis and optimization. The Keras Tensorflow API is used to create a sequential mode with two densely connected, hidden layers, with one output layer. The mean absolute error of the training training process was recorded at i +/- 2.0 pixel as shown in Fig. 5. This result was achieved by changing the parameters that was used to train the model.

{"image_source":{"path":"images/8587ab608f2c94316def2ead09f7139fb19e0afdadd62dc805fe0ca6553f69ef.jpg"},"content":"","chart_caption":[{"type":"text","content":"Fig. 5: Model mean absolute error"}],"chart_footnote":[]}

IX. EVALUATION

For performance evaluation, we implemented our algorithm on a GPU (Jetson TX2) and CPU (HP Intel i5-6330 CPU 2.5GHz). We used Python programming language for the implementation. In order to measure the accuracy of the algorithm, a large database of images and videos taken in different road conditions was used [28].

The videos were taken from various locations such as high-ways, country roads, and suburban areas. The quality of lane markings in the videos was not consistent: some included full markings, dashed markings, and no markings at all. The roads are not all straight; sharp curves were also included, especially at the entries and exits of highways and country roads. Additionally, some videos have different weather and lighting conditions. Examples of lane detected in a video with our implementation is showed if Fig. 6 and Fig. 7.

The performance of the algorithm was evaluated by comparing the actual left and right lane locations to the predicted location for the left and right lanes. We recorded 92% accuracy for the left lane and 96% accuracy for the right lane.

X. SUMMARY

In our research, we developed a deep neural network model for detecting and tracking lane markings in real-time. The model was first trained by using the extracted road boundaries and the estimated lane markings. It was also trained for 50 epochs until a minimal error was recorded. The estimated lane markings used to train the model was obtained from the implementation in our previous research. In that study, we processed the videos in frames. We performed the following operations on each frame to predict the location of the lane markings.

1. Data Pre-processing: This process involves removing noise from the video frames.

2. Region of Interest: This is the step where additional information of the road not needed are discarded.

3. Data labeling: This is the stage where we identify the right and left boundaries of each video frame.

4. Training Model: This is the final process where we use the labeled data to train the neural network model.

The implementation was able to handle up to 30fps for a resolution of 1280x720 pixels. The performance of our algorithm was also evaluated by comparing the actual location of the lane markings to the predicted lane markings and we recorded 92% accuracy for the left lane and 96% accuracy for the right lane.

FUTURE WORK

Although the results obtained in this are encouraging, there is still a need to improve future implementations. The accuracy of the implementation can be improved significantly if sensors like RADAR and LIDAR are used in conjunction with the camera, a method known as sensor fusion. There are situations where the camera by itself will not be able to perform well, especially in extreme weather conditions such as fog. Using the information from RADAR/LIDAR alongside the data from the camera will make the implementation more robust.

Sensor fusion is an intelligent process that combines data from different sensors with the aim of improving an application or system performance. It enables the resulting information from different sources have less uncertainty that would be possible when sources are used individually.