Evaluation of Autonomous Ground
Vehicle Skills
Phillip L. Koon
CMU-RI -TR- 06-13
The Robotics Institute
Carnegie Mellon University
Pittsburgh, Pennsylvania 15213
March 2006
© 2006 Carnegie Mellon University
The views and conclusions contained in this document are tho of the authors and should not be interpreted as reprenting the official policies or endorments, either expresd or implied, of Carnegie Mellon University.
Abstract
Autonomous ground vehicles must achieve bold performance and solid reliability to mature from laboratory curiosities to fielded systems. Currently, there are no standard methods to measure, validate and compare the performance of autonomous unmanned ground vehicles, hence the impetus for this rearch. This paper documents the test methods implemented by Carnegie Mellon University’s Red Team while preparing robots for the DARPA Grand Challenge. The Red Team’s test methods were developed to enable quantitative evaluation of the effects of unit changes to the robots’ hardware and software on the robots’ over all ability to blindly track, track with perception assistance and modify bad on perception a preplanned path. This paper describes tests for evaluating and comparing navigational skills of autonomous ground vehicles. Test data collected from the Red Team’s H1ghlander and Sandstorm autonomous unmanned ground vehicles is prented. Suggestions for future test methodology rearch and test standardization are also discusd.
Abstract (2)
1 Introduction (4)
2 Acknowledgement of Support (4)
3 Red Team Racing System (4)
4 Test Objective (5)
轻伤不下火线5 Literature Review (5)
6 Test Formulation (5)
7 Blind Path Tracking Test (6)
8 Perception Assisted Path Tracking Test (7)
9 Perception Planning Test (7)
10 Test Tracks Utilized (7)
11 Test Execution (8)
12 Data Evaluation (9)
13 Conclusions (10)
14 Future Work (11)
15 References (11)
Figure 1 Major architectural components of Red Team Racing System (4)
Figure 2 ISO-3888-1 Test track for a vere lane change maneuver (6)
Figure 3 Blind and perception assisted route through modified ISO-3888-1 test track (7)
Figure 4 Perception planning route through modified ISO-3888-1 test track (7)
Figure 5 Modified ISO-3888-1 test track as implemented by Red Team (8)
Figure 6 Sandstorm (Left) and H1ghlander (Right) on the LTV and NATC test tracks (8)
Figure 7 Sandstorm Blind Tracking (9)
Figure 8 H1ghlander Blind Tracking (10)
1 Introduction
This rearch was conducted in conjunction with Carnegie Mellon University’s Red Team’s preparations
for the 2005 DARPA Grand Challenge. The 2005 DARPA Grand Challenge was a 132 mile (212
kilometer) race of autonomous ground vehicles along paved roads, dirt roads and trails through the Mojave Dert. The following describes the Red Team’s formulation and execution of a test program to measure
the quality of autonomous ground vehicle driving skills. The program pushed performance to the edge of
driving ability without taking extraordinary risks. The paper documents how Red Team regressively ud the test program to evaluate effects of unit changes to hardware and software on the overall driving skills of洗衣店加盟选赛维干洗
the autonomous ground vehicles.
2 Acknowledgement of Support
后赤壁赋原文This rearch would not have succeeded without the support of Red Team’s Systems Test group. In
particular the hard work and dedication of Michael Clark, Test Conductor, for organizing a team to carry out and execute the tests as planned. The development of the test concepts described in this document was
cloly coordinated with Red Team’s Software Leads Dr. Chris Urmson and Kevin Peterson. Members of
the Systems Test Group include: Jason Ziglar, Josh Johnston, David Ray, Tim Reid, Chris Pinkston, Evan Tahler, Bhas Nalabothula, Josh Struble, Aaron Mosher and Jarrod Snyder.
The author would also like to thank Dr. Red Whittaker, Dr. Dimi Apostolopoulos and Juan Pablo Gonzalez for review and advi during the development of this paper.
3 Red Team Racing System
The Red Team Racing System creates path plans that its autonomous ground vehicles follow at high speed. The vehicles modify the preplanned paths as required bad on local perception of the world. Figure 1 shows the major components of the Red Team Racing System. Figure 1 and the de
scription of the Red Team Racing System are paraphrad or quoted from the Red Team1 and Red Team Too2 2005 DARPA
Grand Challenge Technical Papers.
Figure 1 Major architectural components of Red Team Racing System.
The Preplanning function was performed prior to test ssions and prior to running races. Preplanning
converts the DARPA or Test Designer provided route data definition file (RDDF) into a path definition
file (PDF) that Red Team autonomous ground vehicles can follow. The PDF, a preplanned path, is loaded onto the autonomous ground vehicles. This file defines the waypoints, corridor and speeds the robots will余子俊
attempt to pass, stay within and achieve during a race or test.
The Perception function interprets range data collected from lars and radar creating a terrain cost map
哪个国家最穷and binary obstacle map. The maps are delivered to the Path Planning function.
The Path Planning function fus the preplanned path, terrain cost map and binary obstacle map into a
world model. Items in the binary obstacle map are fud to the terrain cost map by adding them as high or
infinite cost, while clear travers are added as low or no cost. Path Planning us an A-star algorithm
which considers multiple possible traversable arcs forward of vehicle position within the preplanned path’s
route corridor. Each possible arc is evaluated in terms of least cost to goal. The "best" path at any given
interval is then communicated to the Path Tracking function. Areas outside of the path definition file's
route corridor are not considered in path planning3.
The Path Tracking function evaluates a best path relative to current vehicle position and po. The
algorithm ts maximum speed and curvature and constrains the trajectory to ensure against skidding and
tip over. Path tracking pass calculated commands of desired curvature and speed to the vehicle’s drive-
by-wire system.
The Drive By Wire function receives curvature and speed commands and converts them into control signals
which position the steering, brake and throttle actuators appropriately. The Drive By Wire function
monitors vehicle steering and speed and updates actuator commands appropriately to maintain the last
commanded values.
4 Test Objective
The impetus for the Red Team’s test methodology was a desire to measure the threshold of its robots’
driving skills. The team ud the tests in a regressive manner to evaluate the effects of unit changes in
hardware and software on the robots’ over all ability to drive. Three major skills constitute driving ability.
The first skill is the ability of the robots to follow a preplanned path bad on position nsing only. The
cond skill is the ability of the robots to track a preplanned path while assisted by perception nsors. The
third skill is the ability of the robots to dynamically modify the preplanned path to avoid nd obstacles.
5 Literature Review
The literature does not yet chronicle the subject of autonomous ground vehicle test related to measuring
driving skills at speed. Review of technical reports of DARPA Grand Challenge 2005 finishers Stanford
Racing Team4, Team Gray5 and Team Terramax6 found they all placed great value on testing but did not
mention specific tests to measure driving skill. Literature review found documentation of tests to measure
土豆焖鸡翅
an unmanned vehicle’s ability to track a path at low speed7&8. An example of this is Shilcutts,
霍氏家族
Apostolopoulos’ and Whittaker’s tests to validate a meteorite arch robot’s ability to navigate a pattern
path. Although this test did validate path tracking it required post test analysis to determine pass or fail.
The test did not validate the integration of perception into the navigation process. Tests of driver assist
technologies are also well documented9,10&11. The works describes tests of lane departure and collision楚庄王绝缨
avoidance warning systems. They all involve humans in the loop and are more qualitative than quantitative
in nature. This lack of documented tests for autonomous vehicle driving skills at high speed led to a arch
of current industry standards for testing automobiles.
6 Test Formulation
Rearch of published literature on the subject of automotive dynamic testing revealed International
Organization for Standardization standard ISO-3888-1, Pasnger cars — Test track for a vere lane-change maneuver Part 1 – Double lane-change12 (Reference Figure 2). This test was designed as a means to subjectively evaluate vehicle dynamic performance. The test is subjective becau it only quantifies a
small part of a vehicle’s handling characteristics and is highly dependent on the input from the driver. This
dependence on driver skill is what made the test attractive to the author for adaptation to autonomous
ground vehicle driving skill testing.
Figure 2 ISO-3888-1 Test track for a vere lane change maneuver.
Known error in the Red Team’s autonomous ground vehicles’ po nsor of 1 meter RMS and estimated path tracker of 0.5 meter RMS drove the team to modify the original ISO-3888-1 cour adding 1.5 meter to lane width in all ctions. This additional lane width enabled quantitative measurement of performance considering total system error. The cour shown in Figure 2 can be ud with any autonomous ground vehicle by adjusting the lane width as shown. The length of the cour is fixed for all size vehicles at 125 meters. Table 1 describes the basic steps ud in the autonomous ground vehicle path tracking skills asssment. Ideally reliability would be validated by running the test over veral days without system configuration changes.
Table 1Test steps for autonomous ground vehicle path tracking skill asssment.
7 Blind Path Tracking Test
The initial skill test conducted on the modified ISO-3888-1 test cour was blind path tracking. Red Team created the route file for this test with a human driving the robot through the test cour and recording the position output of the robot’s po nsor. The path definition file was created tting the corridor to 4 meters on either side of the path center line. Figure 3 is a graphical reprentation
of the path definition file ud in the blind and perception assisted path tracking tests. When conducting the blind path tracking test the autonomous ground vehicle is configured such that only the po nsor is considered in path
planning. The abnce of all perception nsor input will limit path tracking error to that induced from the po nsor, path tracking algorithm and drive by wire actuation control errors.
1. Create a route file which travers through the ISO-3888-1 test track.
2. Create a path definition file for the route created in step 1 tting the corridor width slightly wider
than the test track’s lane width and the speed to a constant (e.g., 5 meters/c). Path definition file must include an area before the test cour begins for the robot to achieve the required constant velocity.
3. Load the path definition file into the autonomous ground vehicle
4. Command the autonomous ground vehicle to drive the route described in the path definition file.
5. Record the time the autonomous ground vehicle is on the test track entry to exit.
6. Record the number of times the autonomous ground vehicle touches or exits the test track’s
boundaries.
7. Repeat steps 2 through 6 increasing the speed by an incremental value (e.g., 2 meters/cond) until
the autonomous ground vehicle can no longer successfully traver the cour or the operation is
deemed to be unsafe. Multiple runs at each speed increment are required to demonstrate consistency. A= (1.1 x vehicle width) + .25
All dimensions in meters
C= (1.3 x vehicle width) + .25
