How IVEX Safety Analytics platform supports AD/ADAS development
One of the key investment targets for major automotive companies over the past decade has been the development of software and hardware pertaining to automated driving and advanced driver assistance systems (AD/ADAS). This report from McKinsey notes that $120 billion have been invested by major automotive companies towards AD/ADAS development over the past two years. This has resulted in faster development cycles for AD/ADAS components and has, in turn, revealed major challenges that need to be addressed.
A challenge for engineers when developing AD/ADAS components is to understand the behavior of the vehicle after each modification so that improvements can be made. Since the operating environment of AD/ADAS is complicated, every change in any AD/ADAS component needs to be validated against a large number of scenarios, ranging from simulation to real-world driving. Such large-scale validation makes it difficult for engineers to interpret and inspect the resulting behavior of the vehicle.
At IVEX, we are developing a Safety Co-Pilot, a runtime AD/ADAS component that checks the safety of AD/ADAS planned trajectories and issues emergency actions when planned trajectories are unsafe. New features and safety checks are added to the Safety Co-Pilot over time. Improvements are introduced. All changes in the software must be validated to check if they were implemented correctly. Examples of our daily engineering tasks for the development of the Safety Co-Pilot include:
- Make changes and improvements to the Safety Co-Pilot.
- Validate the Safety Co-Pilot against a large number of scenarios.
- Interpret the validation results.
- Create an overview of the behavior of the vehicle under validation.
- Inspect specific driving situations.
- Share the results and interesting driving situations with our peers.
- Etc.
As the number of scenarios is large and keeps growing every day, it’s crucial for us to have a reliable and efficient supporting platform for the development of the Safety Co-Pilot. Traditional supporting tools for robotics development such as RVIZ do not catch up with our demands. For example, with RVIZ, we can only inspect the behavior of the vehicle in real-time and it is difficult for us to quickly jump to any specific event.
Instead, we rely extensively on IVEX’s Safety Analytics platform for the development of the Safety Co-Pilot. IVEX’s Safety Analytics is a cloud-based platform containing many features to support the validation and development of AD/ADAS. In this blog post, we share our experience of using the Safety Analytics platform in our Safety Co-Pilot’s development.
We interact with the Safety Analytics platform by converting the results of our scenario execution into the required data format and then uploading the converted data to a cloud-based server. The input data to the platform is defined in a Protobuf message format, including information about the environment and the behavior of the AD/ADAS during the execution, such as:
- Ego vehicle state variables such as pose, velocity, acceleration, and steering angle.
- Ego vehicle planned trajectory.
- Obstacle information such as pose, bounding box, velocity, and type.
- Contextual images.
- Labeled events, where each event consists of a label and the corresponding timestamp.
A clear and well-defined input format for the Safety Analytics helps us work seamlessly with different data formats. Our scenarios are from different sources, therefore the scenario execution results have different formats such as Autoware ROS2 Bags, Apollo Cyber Records, Carla Simulator log format, and other commercial simulator log formats. For each data format, we have an adapter, often just a Python script, to convert the data into the Safety Analytics’ Protobuf input format. Protobuf is well-known and most of our engineers are familiar with it. Usually, it takes us barely a few hours to develop an adapter to convert a new data format into Safety Analytics’s Protobuf input format.
We upload the scenario execution results to the Safety Analytics platform via a command-line interface. Both the data conversion and the data uploading are integrated into our CI/CD process. At the same time, each of our engineers can execute any specific scenario and upload the result to the Safety Analytics platform. This flexibility allows us to experiment with each change in the Safety Co-Pilot quickly.
We inspect the scenario execution results on Safety Analytics’s web interface. We can search, filter, and select relevant scenarios based on different metadata such as the date of execution, software versions, or scenario names.
As the Safety Co-Pilot is validated against a large number of scenarios, it is important for us to quickly have an overview of the results. We use the Aggregated View of the Safety Analytics platform for that. After filtering and selecting the scenarios of interest, we can access this view which quickly gives us information about all the labeled events, classified by different Operational Design Domains. The labeled events can be included in the input data to be uploaded to the Safety Analytics platform, allowing us to label any type of events relevant to our validation tasks. The Safety Analytics platform also automatically labels several standard events such as hard-brake, critical time-to-collision, and others.
For example, when we develop a new feature in the Safety Co-Pilot and want to compare the results when the ego vehicle moves at different speeds, we enable the aggregated view to show information classified by different speed ranges as in the figure below.
Besides having an overview of the events that have occurred in the scenario execution results, it is important for us to be able to quickly navigate to specific events to inspect the behavior of the vehicle. From the aggregated view, we can directly jump to any event of interest.
For example, should we want to inspect a hard-brake event when the ego vehicle is moving faster than 30 km/h, we can click on the box corresponding to the event and the speed. The distribution of the events and a list of specific timestamps are shown with the contextual image. Clicking on any of the cards leads us to the Datalog view, which provides detailed information occurring at a particular timestamp.
In the Datalog view, we can inspect the event in detail. The view has a detailed visualization that shows all information at that moment, including the state of the ego vehicle, the map, the obstacles, and their relevant information such as velocity, type, and distance to the ego vehicle. We can also jump from the scenario list directly into the Datalog view.
The Datalog view also helps us to re-play and see what happened in a scenario. We can go back and forth to each timestamp with only a few clicks. We can also see all of the labeled events and navigate to any of them. Using this view, we can quickly assess and understand any situation.
Once we find events that need further inspection, we share the exact same view with other colleagues, which can then inspect the event from their own desks. Sharing such information is crucial when working in a team as it facilitates our communication significantly and helps us to focus on the right topics.
Conclusions
The Safety Analytics platform significantly speeds up our development activities.
- We can easily integrate any driving data into the platform.
- We can quickly have an overview of the behavior of the vehicle.
- We can quickly navigate to any event of interest.
- We have access to all the useful visualization information to quickly understand a situation.
- We can quickly share information among the team.
We hope we can give you some ideas for how to improve your development as well. If you want to learn more about our Safety Analytics platform or IVEX in general, feel free to contact our team.
