This project involved transforming telemetry data from the 2017 Bridgestone World Solar Challenge using Apache Spark running in AWS Glue. This telemetry data was then used to validate a physics simulation of that solar car to compare how well power consumption is modelled.
The data transformation was done using a map-reduce technique to convert the data from the raw long format data into processed second-by-second data in a wide format. This approach has theoretical speed benefits over a traditional approach, however, the data used was not large enough to fully realise this.
The simulation runs physics calculations based on the environment and parameters of the car to estimate work done. Telemetry data was supported by hourly weather data from a series of points along the route.
The data was transformed from 2.5gb of CSV files into a 7mb parquet file. The validation results gave the simulation a MAE of 459 Watts with 75.6% of energy consumed being accounted for by the simulation model. There is future work to be completed to account for this ~25% gap in energy consumption.
This project has successfully validated how accurate the simulation environment is and provided a means to transform telemetry data.