The development of autonomous driving cars requires the handling of huge amounts of data produced by test vehicles and solving a number of critical challenges specific to the automotive industry.
In this talk we will describe these challenges and how we, at BMW, are overcoming them by adapting and reinventing existing big data solutions for our end-to-end data journey for autonomous driving. Our journey involves ingesting data produced by a variety of sensors into a dedicated Hadoop cluster, decoding the data, conducting quality control, processing and storing the data on the clusters, making it searchable, analyzing it and exposing it to the engineers working on the algorithms development.
In the first part of the talk we will present a general overview of the challenges we faced and the lessons we learned from them. In the second part we will deep dive into the most interesting technical issues. These include: dealing with automotive formats and standards that are not designed for distributed processing; defragmentation of sensory data; assuring the quality of the data coming from complex car hardware and software components; efficient data search across petabytes of data; and reprocessing the computing components running in the car inside the data center, which typically requires high performance computing.