Autonomous Driving using Deep Reinforcement Learning in Urban Environment

Copyright © 2019 by author(s) and International Journal of Trend in Scientific Research and Development Journal. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0) (http://creativecommons.org/licenses/ by/4.0) ABSTRACT Deep Reinforcement Learning has led us to newer possibilities in solving complex control and navigation related tasks. The paper presents Deep Reinforcement Learning autonomous navigation and obstacle avoidance of selfdriving cars, applied with Deep Q Network to a simulated car an urban environment. “The car, using a variety of sensors will be easily able to detect pedestrians, objects will allow the car to slow or stop suddenly. As a computer is far more precise and subject to fewer errors than a human, accident rates may reduce when these vehicles become available to consumers. This autonomous technology would lead to fewer traffic jams and safe road”.


INTRODUCTION
The autonomous car has been in the headlines for a decade and still continues to dominate auto headlines. The autonomous car has attracted the researchers, robotics communities and the automobile industries. Human driving is accident-prone. We are drunk, drowsy and tired. Our inefficiency to make better on-road decisions cause road accidents, property loss and deaths. The autonomous car brings us the possibilities to replace accident-prone human driver by providing comfort and safety. An autonomous car is a vehicle that can guide itself without human interaction. This kind of vehicle has become a concern reality and may pave the future system where computers take over the art of driving. An autonomous car is also known as driverless car, robot car, self-driving car or autonomous vehicle. Autonomous cars uses various kinds of technologies. They can be built with GPS sensing knowledge to help with navigation. They may use sensors and other equipment to avoid collision. They also have the ability to use a range of technology known as augmented reality, where a vehicle displays information to drivers in new and innovative ways. Autonomous car production could cause problems with existing auto insurance and traffic controls used for humancontrolled cars. It is only a matter of time before these kinds of advances allow us to outsource our daily commute to a computer. The automated car concept was originated in 1920, but it evolved truly in 1980 with CMU's Navlab and ALV projects. CMU's effort resulted in the ALVINN in 1989, the first try in car autonomy which used a neural network to drive. The United States government invested in three projects for autonomous car which were demonstrated by US Army (Demo-I, Demo-III) and DARPA (Demo-II) in the 2000s. From 2004 to 2007, DARPA organised 3 car autonomy challenges which got attention all over the world at that time. The automobile industries like Ford, Mercedes, Volkswagen, Audi, Nissan, Toyota, BMW and Volvo came to play a bigger role in the autonomous car area by announcing their plans for it. In 2014, Google introduced Google car with sensor fusion technology (mainly camera and lidar sensor). One year later, Tesla brought their autonomous car in 2015. Nvidia has recently introduced a self-driving car with end-toend convolutional neural network. In 2017, Audi stated "latest A8 would be autonomous at up to speed of 60 km/h using Audi AI. The driver doesn't require safety checks such as frequently gripping the steering wheel." The Audi A8 is claimed to be the first production car to reach level 3 autonomous driving and Audi would bethe first manufacturer to use laser scanners in addition to cameras and ultrasonic sensors for their system. On the 7 th November 2017, Waymo announced that it has begun testing driverless cars without a safety driver at the driver position. However, there is an employee in the car. And the list still

IJTSRD23442
International Journal of Trend in Scientific Research and Development (IJTSRD) @ www.ijtsrd.com eISSN: 2456-6470 Page: 1574 continues with Uber, Tata, Intel and other automobile industries. Driving is a complex problem. The evolution of autonomous car involves complex modules, which uses traditional control algorithms , SLAM based navigation systems and Advanced Driver Assistant Systems in recent days. But all the previous methods fail to replace human driver completely. But increasing computing power of GPUs, accurate sensors and deep neural networks has made complete autonomy is realisable. But recent failure of Uber and Tesla, and the listed disengagements in show that the autonomous car research needs more attention in the upcoming years to achieve 'better than human-level driving'. In the context of 'human-level control', presented a deep reinforcement learning approach which redesigned Q-Learning using a deep neural network. It introduces 4 steps to unstable Q-learning: Experience Replay, Target Network, Clipping Rewards and Skipping Frames. It achieved humanlevel performance (sometimes even better than human) tested in different complex game environment. The breakthrough achievement of DQN has motivated us to implement it in car autonomy for human-level performance.
In this paper, we present a DRL-based autonomous driving strategy for urban environment using Deep Q Network. We use sensor fusion on two types of input sensor data: vision based camera sensor and a laser sensor. The approach is tested in simulation environment, which is designed using Unity Game Engine. We also design a car prototype capable of running our approach in real-time. The prototype uses Hokuyo Lidar Sensor and camera sensors embedded with Nvidia Jetson TX2 embedded GPU. The paper structure is as follows: it introduces autonomous car and the neural network approaches in car autonomy, presents the background and mathematical representation of Deep Q Network and describes proposed methodology with the network architecture and configurations, action definitions and simulation environment. It also presents the learning curve during network training. Section IV discusses our car prototype capable of running DQN in real-time.

I. PROPOSED METHODOLOGY A. Network Architecture
We design a deep neural network based on fully-connected CNN to approximate Q-function. The network consists of 3 Convolutional layers and 4 dense layers. Two types of data Input: front camera image (RGB image of size 80 x 80) and lidar data (grid-map of size 80 x 80) are fed into the neural network, processed using the convolutional layers data type. Two data types are fused using a flatten layer, preceded by a dense layer and a pooling layer (used for data resize) for each data type. Our target network follows the same architecture, which is updated every 6000 steps. The neural network is trained to accept 5 actions with defined reward. The network uses mini-batch of size 32, trained with learning rate 0.001 Fig: 1 System Architecture

B. Action and Reward Definition
Action definition is one of the key points in Reinforcement based learning strategy. We choose 5 actions for the agent to be trained with DQN: Keep Going (reward= speed/5), Left (reward = -0.6), Right (reward = -0.2), Accelerate (reward = +1) and Brake (reward = -0.4). The basic action for our agent is to keep going (or in other terms do nothing) in the road. For a hit, a bad reward of value -6 is assigned. It needs to learn how to accelerate when the front is empty and decelerates (brake) when there are other cars or agents in front

C. Simulation Environment
We design a simulated urban environment in Unity Game Engine. The environment is based on Geneve environment and consists of 5 lanes highways. It is a city-like structure (see fig. 3) with buildings, trees, and street lights. The environment is cooperative; other cars run on the highways (see fig. 4). The basic goal in our design is to mimic the realtime urban environment as close as possible in the simulation. It allows us to train the DQN to drive the car autonomously, with almost real-time driving behavior

D. Software Architecture
The simulation test is done in a Ubuntu platform (v. 16.04LTS) installed with python 3 (v. 3.6.2). The Deep Q Network is designed using tensorflow API. OpenCV (v. 3.3.0) is used for image related tasks. It uses numpy, scipy and matplotlib. The simulator is connected to the DQN algorithm using Socket connection (python-socket-IO). Figure 5 illustrates the software used in simulation