Scenario categorization, a crucial step in AD/ADAS development and validation

How can I know that my driving system behaves correctly and can be safely deployed?

This question, at the core of every AD/ADAS manufacturer, starts with a simple and intuitive answer: the vehicle must be tested as much as possible. Therefore, most companies collect and analyze an incredible amount of data that comes from testing their driving system in real life or simulated conditions. It represents a considerable waste of time and resources, as, out of all the data generated, only a small fraction actually contains valuable information that can be used for development and testing. The fraction in question concerns situations that need to be carefully inspected in order to make sure that the system is reliable. These situations, also called scenarios, can take many forms: a vehicle cutting-in in front, an abrupt lane change, a front vehicle suddenly braking, a pedestrian suddenly crossing the street, etc.

This blog post explores why categorizing these scenarios is crucial for the development and validation of AD/ADAS systems, and how IVEX plays a key role in the field.

The cost of an ineffective scenario-based validation

In the process of validating AD/ADAS systems, vehicle manufacturers started to follow a method called Scenario-Based Validation. This workflow, which is based on testing & assessing the vehicle’s functionality in several situations, can be summarized in the following picture.

The scenario-based validation workflow.

The workflow is based on four steps:

1. Make the system drives in a pre-defined environment, which can be in simulation or real-life conditions.
2. Process the sensors’ data through a perception stack. Log all the data recorded so that it can be analyzed by the development and/or validation teams.
3. Based on the perception output, find and categorize scenarios. These are situations where it is crucial to know if the vehicle acts properly. How does the system react when there is an obstacle vehicle doing an aggressive cut-in? Does it brake correctly when a front vehicle suddenly brakes?
4. Re-configure the system so it has the desired behavior in different scenarios.

Once the system is re-configured, it can be deployed again. The workflow is repeated in a loop until the required behavior is achieved.

To assess and qualify their system on scenarios, companies in the AD/ADAS ecosystem can perform tests in a simulated environment or closed tracks with a set of pre-defined scenarios. However, to capture the full variability of the scenarios, they must also validate their development in real-life conditions (with different weather and visibility conditions, unexpected traffic, pedestrian or objects on the road, etc.).

While very effective, this process has one major drawback. There is a huge amount of data that is generated in the pipeline, out of which only a small fraction contains valuable information. This concept, called the empty miles problem, refers to the fact that most of the kilometers driven do not cover scenarios that bring insights for development and validation. Analyzing the data from the driving logs is a long and tedious process.

A good example of this can be expressed with the cut-out scenario, which is, as defined by Euro NCAP: “a situation where a car in front leaves the lane abruptly to avoid a stopped vehicle ahead, leaving the system only a short time to identify and respond to the situation”. From IVEX knowledge base, we learned that there is, on average, less than one cut-out for every 30 minutes of recording (in a regular urban or highway environment). In other words, there is a need to digest and analyze more than 30 minutes of sensors’ data to find a single potentially interesting cut-out situation. To have statistical relevance in the validation, it is required to process thousands of these scenarios, occurring on a massive number of kilometers driven. And this accounts for only a single scenario type! In practice, many more of the scenario categories (e.g. cut-in, front vehicles suddenly braking, lane changes, etc.) need to be analyzed making the associated analysis time and costs grow exponentially.

In short, while being a crucial step of the validation pipeline, analyzing the driving logs is a major bottleneck in the validation pipeline. With an intelligent way of categorizing scenarios, one can deal with this burden while drastically reducing validation and development costs.

Scenario categorization at the heart of the AD/ADAS validation

The solution to avoiding the empty miles problem is to categorize scenarios. It consists of extracting and classifying all the scenarios of interest out of a driving log. The scenarios of interest can be based on the interactions between the vehicles (such as the Euro NCAP scenarios), but also on the behavior of the driving system, or even on the environment. The concept is illustrated in the figure below.

The concept behind Scenario Categorization. Out of an initial driving log, the scenarios of interest are identified and extracted.

Having a dedicated system to automatically categorize the relevant scenarios helps reduce engineering costs. It allows the development and validation teams to focus on the situations that are relevant to them, without having the burden of manually inspecting huge data pools coming from driving logs. They quickly gain insights and can make sure that the test drives cover perfectly the scenarios they targeted for validation campaigns.

Moreover, being able to quickly extract and analyze situations of interest helps validation and development teams to refine their testing strategies. Rather than focusing on the quantity of the driving logs, they can focus on the quality. They can create personalized test cases to ensure the reliability of their systems while reducing the number of kilometers driven.

In addition to processing and analyzing data, storage has a huge cost as well. By extracting scenarios from the initial driving logs, the storage needs are considerably reduced as only subsets of data that are useful for development are kept in hot storage (e.g. Amazon AWS S3).

IVEX solution for categorizing scenarios

Based on these facts, IVEX has built a dedicated product for categorizing interesting scenarios. Powered by the Safety Analytics Platform and its advanced algorithms, this product helps engineers efficiently find and analyze situations that are the most useful to foster the development and validation of their AD/ADAS systems. It makes the entire validation process faster, and more cost-effective.

In the example below, we see multiple cut-in, cut-out, and dangerously approaching vehicles that were successfully categorized by the tool. From around 8:09 hours of driving log, we extracted 5 minutes of very valuable information. That is a reduction of 99% in terms of data storage and time required to analyze the logs.

Category coverage of the extracted scenarios on an IVEX driving log.

Time and distance saved by the scenario categorization, in comparison with the original driving log.

This overview is completed by a more detailed list of all the extracted scenarios, containing information on their relevant parameters computation. We see in the figure below several detected cut-in scenarios, ranked by their risk. Once we see the overview of our categorization, we can select each scenario to inspect it in detail.

The Categorized Scenario list. It provides an easy way to go over and select the scenarios of interest.

Perform in-depth analysis of scenarios

Now that we have seen the overview of the categorized scenarios, we can select a particular one to perform an in-depth analysis.

The following cut-in situation was extracted from one of IVEX driving logs. We see a vehicle coming abruptly from the right at a slower speed than the test vehicle, forcing the latter to brake.

A cut-in from the right, observed by IVEX, on a Belgian highway at night.

To analyze this particular scenario, we use the Scenario Inspection View of IVEX Safety Analytics, as seen in the figure below.

The Scenario Inspection View in the Scenario Categorization Tool of IVEX Safety Analytics Platform. The cut-in scenario is displayed in its full context, using a video, a 3D visualizer, and a series of relevant indicators.

By looking at the video at the top of the view, we immediately get a good understanding of the scene, the surrounding objects, the visibility, etc. The 3D visualization on the right side of the video brings more precise information on the obstacles in the scene, their type (pedestrian, truck, car, etc.), and their trajectory.

In addition to the visualizations, we gain insights into the scenarios based on several relevant indicators. In the example, we see that the cut-in has a medium risk, which is a measurement derived from a sudden decrease in the time-to-collision with the front vehicle. This can be observed in one of the small graphs at the bottom of the view. Other key indicators such as the lateral positions and velocities help understand every aspect of the scenario.

Now that we have analyzed our scenario, we might want to share it with the entire development team, or with any involved stakeholder. It is possible to do so thanks to a simple URL mechanism, which is accessible in the top left section of the view.

Scenario Search and Coverage

On top of the scenario categorization, the tool allows to search and aggregate scenarios based on various parameters. In the example below, we perform a search on the Euro NCAP cut-in from the left scenarios. For a given range of time-to-collision and lateral distance values, we observed the distribution of the categorized scenarios over the different relevant parameters.

Scenarios from the category “Euro NCAP cutin from the left” are filtered based on time-to-collision and lateral distance parameters. The results and displayed on the Category Coverage view.

Conclusion

Analyzing driving logs is a key step in the development and validation of AD/ADAS systems. However, it can represent a real bottleneck in the engineering pipeline, as two main issues occur:

• Scenarios that are useful for the engineering teams represent only a small part of the huge amount of data that needs to be analyzed. Processing and analyzing this data pool has a massive cost, in terms of time and storage.
• Without a proper analytics tool, it is impossible to have enough confidence on why and when the validation and development pipelines are over. Scenarios must be carefully analyzed to ensure that they are relevant.

IVEX has developed a product to tackle these two issues. Dedicated to categorizing scenarios in both real-world and simulated environments, it allows the user to:

• Process massive driving logs in a limited amount of time, reducing storage and engineering time.
• Access the categorized scenarios via a Web user interface or programming API
• Have an aggregated view of their driving log, the resulting scenarios, and their coverage.
• Perform search and in-depth analysis of scenarios. Assess individually every situation while keeping an eye on key primary indicators.
• Easily share the scenarios and related analysis with any involved stakeholder.

For more information about our Scenario Categorization product or IVEX in general, feel free to contact our team.