Identifying challenging scenarios for scenario-based validation of AD/ADAS systems
Scenario-based validation is an important step in the development of autonomous driving and advanced driver-assistance systems (AD/ADAS) which consists of testing the automated systems against a set of predefined scenarios. The more scenarios included, specifically challenging ones, the higher confidence one can have about the AD/ADAS system under test.
Real-world driving recording is an important source of challenging scenarios, but real-world driving recording is often a costly and tedious process. Most of the time, real-world driving records contain many “empty miles” in which nothing of interest happens, also called “empty miles” (e.g. the vehicle is just driving straight on an empty road and in perfect weather conditions). “Empty miles” only contribute marginally to the validation of the AD/ADAS system. A small fleet of test vehicles can easily create data in the order of petabytes (PB).
How much of this data is relevant? For how much time did the vehicle drove with no surrounding vehicles? Should engineers manually inspect scene by scene all non-relevant and non-risky scenarios?
Should the engineers rely completely on operations personnel annotating the relevant scenarios (during the driving) and ignore everything else? Being able to automatically identify challenging scenarios from real-world driving records brings value. It reduces the amount of data stored in “hot” storage (e.g. AWS S3), as one can decide to store only relevant and challenging scenarios in “hot” storage and the remaining data in “cold” storage (e.g. AWS Glacier). Additionally, automatically identifying challenging scenarios reduces the validation effort, as only relevant scenarios need to be inspected.
Identifying challenging scenarios is also a good practice before adapting AD/ADAS systems to a new Operational Design Domain (ODD), as it is desirable to test the system functionalities in as many scenarios as possible from that specific ODD.
Yet, identifying challenging scenarios is not a trivial task, as one must go over thousands of hours of driving data and decide whether a certain scenario is worth to be selected or not. Therefore, it is of vital importance to have an automatic and explainable way to extract these scenarios from driving records.
IVEX’s technology is capable of automatically and efficiently identifying, categorizing and extracting challenging scenarios from driving records, providing a clear and complete overview of the analyzed driving logs. IVEX software provides reports about the data analysed and facilitates the inspection of any selected scenario. This brings the potential of saving huge amounts of engineering time needed to inspect driving logs, as well as reducing storage costs and reducing testing efforts.
How to Identify Challenging Scenarios?
The first step to identifying challenging scenarios is to collect driving data using data-collection platforms equipped with sensors. Usually, these data-collection platforms should be able to record both localization and sensor information (images, point clouds, etc.). The data-collection platforms can be a fleet of vehicles manually driven by human drivers.
Given the data collected, one can use IVEX’s Safety Analytics (SA) platform to identify challenging scenarios and perform other analyses. The IVEX’s SA platform contains different metrics, ranging from behavior-based safety metrics such as time to collision (TTC, defined in ISO 15623), to perception-based metrics such as vanishing object detections. IVEX uses a combination of metrics to assess each driving segment and to extract the ones considered as challenging scenarios. Users can also add customized metrics to the platform. The identified challenging scenarios can be exported to different formats such as OpenScenario, CSV, JSON, or simply be linked to the original data input.
Examples of Challenging Scenario Detection
We analysed a driving log from Brussels to Paris and returning to Brussels completed in 8 hours in December 2020. The complete driving log size is around 2TB. We used the IVEX’s SA platform to extract several challenging scenarios (totalling 5 minutes), reducing the recording storage to 20GB which accounts for an order of magnitude in storage reduction. Below we show two of such extracted scenarios.
Truck and Car on Merge Lane in Highway
In this scenario, the ego vehicle is driving on a highway and approaching an on-ramp. There is a truck and a passenger car driving slowly on the on-ramp and aiming to merge into the lane of ego vehicle.
This situation happens on the Belgian section of the highway. According to the Belgian highway code, the truck and the car on the right must give the way to the ego vehicle. However, a reasonable driving behavior for the ego vehicle in this case would be to decelerate slightly creating more room for the truck and car to merge, as it is usually very certain that the car and truck will not come to a full stop at the lane end but they will actually try to merge.
In fact, this scenario reduces significantly the drivable space of the ego vehicle. It has two main “good practice” options, either to slow down and stay in the same lane or try to safely change to the left lane. The ego vehicle could also (and it would even have the rights) keep driving on its lane, but that could create a critical and possible dangerous situation. Since there are two vehicles cutting in (truck and car at the same time), the best option for the ego vehicle would be to change to the left lane. However, at the same time, on the left lane there is a van, so the ego vehicle cannot change lanes too soon or otherwise it will create itself another dangerous situation, as the distance between the ego vehicle and the van is not safe. Nevertheless, the ego vehicle should not change lanes too late, or otherwise it would have to brake strongly to avoid the two merging vehicles.
Road Works in Highway
This scenario poses a challenging scenario due to road works. The ego vehicle needs to drive on a temporary lane created on what is normally the road shoulder. Such scenarios are quite common in European highways, even though we are aware that not all AD/ADAS systems already need to handle them.
Depending on the level of autonomy and the ODDs (operational design domains), this scenario might be challenging and relevant to be added to a testing set. Referring to SAE Automation Levels, a Level 2 or Level 3 system entering this scenario should be able to disengage and notify the driver in time. For a Level 4 or Level 5 system, this scenario could break ODD assumptions and there could be the need of a takeover from a remote operator or to inform the ADs fleet of the unexpected road work area.
This scenario contains very complex contextual information. Firstly, since it is a road works region, the yellow lane markers should be used to indicate the lane region and driveable area. However, as can be observed in the picture, these yellow lane markers are not easily distinguishable, and the old white road lines are still quite visible, making it even more challenging. Secondly, due to the new lane marker, the ego vehicle must drive on a usually forbidden drive zone (cross white markers at the bottom of the picture) and on the road shoulder area. Last, there is a traffic sign not very well positioned indicating a temporary speed limit of 70 km/h, which the AD/ADAS system should detect and react accordingly.
Conclusions
We showed two challenging scenarios detected using IVEX’s Safety Analytics platform. As we saw, challenging scenarios detection is an activity that span through all the core functionalities of the AV stack. It needs to check the input sensors, perception component outputs and implemented behavior.
We believe that finding challenging scenarios from driving logs is one of the key-enabling factors for widespread AD/ADAS adoption. Being able to easily extract relevant scenarios from real-world driving records is a fundamental feature for effective AD/ADAS validation process.
Stay tuned for our next blogposts where we will show IVEX’s technology in more details and with even more interesting scenarios!
If you are interested in learning more about Scenario-Based Validation and Data Analytics for AD/ADAS, contact us here.
