IVEX Safety Analytics Use-Case: Analysing Tesla Autopilot hard-braking
Introduction
Over the past few years, AD/ADAS components have become increasingly popular in commercial and retail vehicles. This has brought a litany of interesting problems that need to be solved. One of the most interesting problems is that of identifying and understanding different events triggered by AD/ADAS behavior such as hard-brakes, hard-acceleration, low time-to-collision, etc. within thousands of hours of driving, both during and after development.
At IVEX, we built Carvex, our own data collection platform, to collect driving data and fuel our product development. Carvex includes a set of state-of-the-art sensors mounted on top of a Tesla Model 3. Much of the data is collected when Tesla’s Autopilot, Tesla’s Level 2 ADAS system, is activated. As Tesla’s Autopilot is one of the most advanced ADAS systems, we are also curious about its own performance. We are particularly interested in hard-brake events related to Tesla’s Autopilot, as false positive braking is still one of the biggest challenges in ADAS systems.
In the beginning, we also experienced difficulties in inspecting such hard-brake events from our data. Carvex has been collecting 15,000 km of driving data. Although this number of kilometres is still small compared to what a normal ADAS system needs to be tested before deployment, we already found that it is not easy to identify and inspect hard-brake events without a good supporting tool.
IVEX Safety Analytics platform was developed with this as one of the major considerations. The platform supports users to identify and inspect any event of interest in AD/ADAS driving data. The Safety Analytics platform is capable of identifying and extracting interesting events about AD/ADAS behavior from driving data. It provides reports about the data analysed and facilitates the inspection of each specific event. In this blogpost, we explain how we analyze hard-brake events in our driving data using the Safety Analytics platform.
Carvex
Carvex is based on a Tesla Model 3 where we mounted a state-of-the-art sensor set for AD/ADAS and a computer to record the sensor data. The sensor set consists of RTK GNSS, Lidar, Radar, and cameras. There is also an inside camera pointing to the Tesla dashboard to monitor the state of the Tesla’s Autopilot. The system has been designed in such a way that it is resistant to adverse weather conditions since this is one of the factors that are likely to generate events of interest.
Most of the time during the data collection activities, the Autopilot was on. The Autopilot has two primary components: adaptive cruise control and lane-keeping assist. The autopilot requires active driver supervision and will sound an alarm and disengage in events that are outside its Operational Design Domain.
Carvex has been driven for over 15,000 km and in that time, has accumulated over 40 TB of data with a lot of interesting events related to the Autopilot behavior. To identify and analyse these events, we use the IVEX Safety Analytics platform. It automatically identifies various interesting events across driving logs, connects them to their potential cause and looks for changes in the autopilot behavior.
We have selected some of these scenarios, specifically the car hard-braking, either done by the autopilot or by the driver, moments after the autopilot disengages. We consider any deceleration greater than 4.0 m/s² (13.12 ft/s²) as a hard-brake. We shall observe them in detail, and analyse the context in which the hard-brakes are triggered. The hard-braking events are particularly interesting because of two reasons. First, it might be that the driver disengages the Autopilot, following a hard-braking, which would mean that there is a situation which the Autopilot cannot handle, possibly because it is outside of its ODD. Second, it might be that the hard-brake is a false-positive (hard-braking in an event where a hard-brake is not warranted), which is dangerous as an unexpected false-positive hard-brake might lead to a rear-end collision.
Analysing Tesla hard-brakes using the IVEX SA
The sensor data from Carvex is processed offline using the perception stack of Apollo, an open-source autonomous driving software. The sensor data such as images and the perception output in object-list format are then imported into the Safety Analytics platform.
The Safety Analytics platform allows us to represent all the camera streams available from an AD/ADAS vehicle. In this case, we choose to focus on the front camera and inside camera streams for this analysis, so that we can see the behavior of other vehicles alongside the autopilot.
Finding hard-brakes with the SA
The SA platform allows us to select multiple driving logs and find interesting scenarios across them, categorised by relevant parameters. We are interested in hard-brakes that reduce the speed of the car sharply. We first use the Aggregated View of the Safety Analytics platform to have an overview of all hard-brake events.
Here above, we have selected three separate driving logs from different times of the year. Each of these is between six and eight hours of driving, and they are 1,374 km long in total. As we see above, we are looking for hard-brake events that occur when the Tesla is moving faster than 50 kph. We see that there are four events in that category, and we will analyse three of them that are of interest to us.
- Hard-brake with a vehicle on the right of the ego vehicle
We can see here that the car brakes sharply although the truck does not really move into its lane. This might be dangerous for any vehicle behind the car. The Tesla is moving at 28 m/s (62.63 mph or 100.80 kph) and the trucks in the right lane are both moving approximately at 21 m/s (46.97 mph or 75.6 kph). One of the trucks drives over the lane line for approximately a second. We can see, in the figure below, the velocity vectors of the vehicles near the Carvex at considerable speed, making this a risky situation for both cars.
We can see the velocity (as estimated by our RTK GNSS) falls by around 7 m/s in approximately a second. We see that the truck in the right lane starts to brake and at the same time, it moves on top of the curvy lane line. Because of that, the autopilot responds in turn by braking sharply for one second as it might estimate that the truck is going to cut-in at a low speed. When the car brakes, the truck is 1.59m away from the front of the car, which contributes further to the risk of the situation.
We can see that the car’s autopilot tags the truck as potentially crossing into the lane of the car and thereby being a safety risk, which is the reason for the decision to brake hard.
2. Hard-brake close to a traffic signal
In this event, we can see that the car brakes extremely sharply to stop at a traffic light. This is an interesting decision as other drivers might not be able to respond as quickly, and thereby have their driveable area reduced quite significantly. As we see below, the Tesla comes to a dead stop from 20 m/s (44.74 mph or 72 kph) in 3.7 seconds, which is approx. 5.4 m/s² of deceleration. The orange light becomes visible 1.8 seconds before the hard-brake is triggered.
Considering the fact that the autopilot in this version is not capable of reacting to traffic lights (it does recognise them, as seen below), it would probably have ignored the traffic light. What we see happen below is the driver disengaging the autopilot, taking over control of the car, and choosing to brake sharply in order to avoid running the traffic light.
3. Hard-brake on encountering a roundabout
Here we see the car braking sharply before exiting into a roundabout. The roundabout is seen approximately three seconds before the hard-brake occurs while the ego vehicle is moving at 18 m/s (40.26 mph or 64.8 kph). This action would be risky if there was a vehicle behind the Tesla. Fortunately, we can see that there are none, making the situation less of a risk.
From our drivers’ experience, Tesla’s Autopilot does not handle roundabouts as it seems to be outside of its Operational Design Domain. At the same time, similar to the event above with the traffic lights, the Autopilot does not make any alarm to notify the driver. Thus, the driver takes over, with probably less-than-sufficient time to smoothly manoeuvre into the roundabout, leading to a hard-brake before entering it.
Conclusion
In this blogpost, we see how to analyse a particular scenario category, i.e. unexpected hard-brakes using the IVEX Safety Analytics platform.
- We see that it is easy to find instances of unexpected behavior and then inspect the behavior in detail
- We can get a multimodal view of a scenario and get a sense of potential causes
- We can also see how the autopilot behaves in a given circumstance and find the reasons for it
We can also come to some conclusions about the behavior of the Tesla Autopilot based on the analysis above.
- In events that are not a part of the Tesla Autopilot ODD, the autopilot disengages, which can require the driver to brake sharply
- There are events that the autopilot will perceive as being risky, such as a potential obstacle cutting in, and will hard-brake, even when the perception does not correspond to actual events
I hope we could give you some ideas on how to analyse the data from your AD/ADAS using the IVEX Safety Analytics platform. If you want to learn more about our Safety Analytics platform or IVEX in general, feel free to contact our team.
