In this post, we’ll try to explain in more detail what went into the latest openpilot release. A lot of work goes into every release, and the short list of release notes generally doesn’t fully capture the wide range of improvements in the release. So in this new experimental format of release documentation, we hope to give a better picture of what will change with this release of openpilot.
New driving model
Bigger data
This release has a new model trained on over 1 million minutes of driving, up from 400k minutes in the previous model. The increased data and diversity are generally helpful for training.
Localization bug fixes
In this model, there were several improvements made to the ground truth. Most of the ground-truth labels rely on the localization being good, when the localizer fails to estimate the car’s direction of travel, the ground truth looks completely off, as can be seen in the images below. We improved the localizer behavior, fixing some of these failures, and also wrote a filter that can detect these failures so that the cases that can’t be fixed are at least excluded from training. Fixing bugs like this significantly improves the overall accuracy of the model.
Cut-in bug fixes
The multi-hypothesis prediction method we use for lead prediction predicts multiple potential positions for leads, and a separate part of the model then picks the best hypothesis. This part of the model was failing and selecting a nonsensical hypothesis when there were large changes in the lead, e.g. during cut-ins. This was due to a bug of this selection loss not being weighted highly enough. Fixing the weighting dramatically improved cut-in prediction.
New driver monitoring model
General training improvements
The new driver monitoring (DM) model is now trained on 150k minutes of driver video from 1750 users, with roughly 10% comma three data and 90% comma twos. The inclusion of more diverse data often helps the model work well on more people.
In addition to that, PRN has been replaced with the more modern TDDFA_V2 for higher quality pose label generation. Further improvements to the ground-truthing stack include switching to soft labels for probability output bits for the model to better learn uncertainty and more aggressive filtering of buggy labels that can undermine overall model accuracy.
Wide FOV model
The comma three has a beautiful 180° camera facing the driver, and it is time for DM to take advantage of this wider field of view (FOV). The new stack was reworked to generate ground-truth for wider input frames and train models on them. The wider frames result in 42% more FOV on the comma three.
With the wider FOV, it will be easier for the model to see the driver across a variety of cars and mounting positions.
Mask augmentation
One of the recent major DM complaints is that the model doesn’t work well on drivers wearing a mask. Intuitively, one could try to solve this by training on more data with masked users, but it turns out the generated ground-truth data (even with TDDFA) is frequently inaccurate with masked users. We need to have good ground-truth data with the face covered to train the model well.
Mask augmentation comes to the rescue - we can add fake face coverings to perfectly labeled unmasked data. Templates for these “masks” are sourced from MaskTheFace and they are warped to fit the face based on its 3D reconstruction from TDDFA. A separate mask detector is used to filter out data with real masks since that data has unreliable ground-truth. Adequate noise is added when warping the mask to avoid leaking head pose information with the applied fake mask. This augmentation forces the model to learn to look for extra cues for head pose, e.g. when the nose and mouth are covered, to make for accurate predictions.
ACADOS
For this release, the MPC solvers we use to do lateral and longitudinal planning went through significant changes. We moved from ACADO to ACADOS, a newer and better maintained library for MPC problem formulation. The problem setup is now all in python which makes it a lot more readable. We changed the solver algorithm from qpOASES to HPIPM, which provides some better guarantees on how quickly the algorithm can output a usable solution. This is important as there are tight constraints on how long the planner can take to find a usable solution. The problem description and cost setup are now all done in two relatively simple python files for lateral, and longitudinal.
For lateral planning, these changes won’t affect driving, but longitudinal planning also got a few more improvements. Better cost functions ensure the solver stays numerically stable at all times. This fixes some cases where there would be abrupt changes in the solutions when a lead came into view, which could lead to jerky driving. A complex 3 MPC system was also reduced to a single MPC with all the costs combined, making it more readable and conceptually simpler. Lastly, the MPC costs related to lead behavior were changed to improve convergence to a desired follow distance behind a lead car. Below you can see the before and after plots of a simulation of starting at a variety of follow distances behind a lead car and converging to the desired follow distance. Before this retune, it would overshoot the desired follow distance, now it cleanly converges.
Smart FCW
This release also includes some big improvements to openpilot’s forward collision warning (FCW). Until recently, openpilot has used the car’s radar to try and detect imminent collisions. You might think that it wouldn’t be too hard to tell by looking at the radar when you’re about to hit something - that’s how the stock FCW systems work on most cars, after all. Unfortunately, radar has some pretty serious limitations that make these systems a lot less effective. To the radar, a stopped car and a manhole cover look exactly the same, so the FCW has to rely on a computer vision model to double-check whether there’s really a car up ahead or not. Radar-based FCW also really struggles with cut-outs - humans will often accelerate towards a lead car that’s changing into a different lane, but the radar sees that there’s still a car up ahead, so special logic is needed to prevent the FCW from sounding a false alarm. As always, whenever you’re trying to hand-tune your code to deal with all the weird edge cases of the real world, it’s much better to let machine learning do the heavy lifting. In particular, we can train our driving model to predict whether or not a human would be likely to slam on the brakes in the next few seconds. Lucky for us, it turns out that the model is really good at predicting this kind of thing! It doesn’t suffer from any of the limitations of radar, and it can even predict hard braking events for things that wouldn’t show up on a radar at all, like a deer crossing the road. So, with this release, we’ve completely removed the old radar-based FCW and instead are using the “smart” FCW even when openpilot isn’t engaged.
Laneless planning cost bug fix
Several months ago, we designed an adaptive cost for the laneless lateral planner. This cost increases the aggressiveness when the model is very certain, and was confirmed through extensive simulation testing to improve laneless driving. Unfortunately a bug in parsing the uncertainty in modeld made this cost always default to the minimum aggressiveness. This bug has now been fixed and this should improve lateral control accuracy in laneless mode!
Cars
This release features 10 new supported cars, as well as several other car related improvements. All community supported cars have been moved out of the Community Features toggle. The toggle will remain for community supported hardware, such as the comma pedal and smartDSU.
Hyundai Improvements
openpilot queries your car’s ECU firmware versions to identify the car. The queries are unique for each make, and the Hyundai query would often return different versions on the same car. In this release, we’ve improved the stability of the Hyundai firmware query.
We also added the code to support longitudinal control on Hyundais. Since it requires disabling the radar and stock safety features, Hyundais will remain with stock ACC until openpilot’s longitudinal policy performs well without a radar. However, a hidden parameter can be set to switch to openpilot longitudinal control on your Hyundai if you’d like to test it out.
Ports
- Audi S3 2015 support thanks to jyoung8607
- Honda Freed 2020 support thanks to belm0
- Hyundai Ioniq Hybrid 2020-2022 support thanks to sunnyhaibin
- Hyundai Santa Fe 2022 support thanks to sunnyhaibin
- Kia K5 2021 support thanks to sunnyhaibin
- Škoda Kamiq 2021 support thanks to jyoung8607
- Škoda Karoq 2019 support thanks to jyoung8607
- Volkswagen Arteon 2021 support thanks to jyoung8607
- Volkswagen California 2021 support thanks to jyoung8607
- Volkswagen Taos 2022 support thanks to jyoung8607
