We gave the first taste of navigate on openpilot in last year’s Taco Bell drive, and the first version of it ships in this release. openpilot with Experimental mode now features all three major components that enabled the Taco Bell drive: end-to-end lateral planning, end-to-end longitudinal planning, and navigate on openpilot. The latter two are still alpha quality, but in the upcoming releases, we’ll iterate and fix the bugs until they’re ready for chill mode.

Navigate on openpilot

Navigate on openpilot works by feeding the map view into the driving model, allowing it to see the route it’s supposed to follow. This happens in three steps: first openpilot renders the map, then the map gets compressed to make it easier for the driving model to digest, then the driving model uses this to predict where a human would drive in order to follow the map.

mapsd

<span class="h-code">mapsd</span> is a new daemon that draws the map that’s going to be fed into the navigation model. The video on the left shows what its output look like; they’re intentionally very bare-bones, just showing the roads in white and the route to follow in gray. The resulting images are only 256x256 pixels, but that’s still way too big to feed directly into the driving model, so we have to compress them first.

navmodeld

That brings us to the next step, the “map model”, which runs as the new <span class="h-code">navmodeld</span> daemon. This model takes in a picture of a map from <span class="h-code">mapsd</span> and compresses it down into a few hundred numbers that describe how the map looks. These numbers can be decompressed back into an image, as shown on the right-hand side of the video. This is, roughly speaking, what the driving model gets to see.

modeld

Finally, these numbers are passed into the driving model. During training, we can find the exact route being driven in each video clip, generate maps using these routes, and then show these maps to the driving model. Since these maps show exactly where the human driver is going to go in the future, the model learns that the best way to drive like a human is to follow the route shown in the map, taking exits and turns when necessary.

Navigation improvements

Navigation, which has remained in a relatively stable state since its initial release in 2021 (#20751), is finally able to receive some improvements to its interaction, bug fixes around its behavior, and general polish in this release.

The destination entry settings has been moved out of a settings panel and into the spotlight, by featuring it directly on the home screen. It can also be accessed in the onroad UI by tapping the new navigation settings button in the bottom corner.

Bug fixes

• Map: fix destination pin icon being centered on the destination (#29054)
• Map: fix incorrect layering of map elements (#29050)
• Map: restore adaptive map zoom, now changes based on vehicle speed (#28796)
• Map: show warning when waiting for route from connection issues or re-route backoff (#29055)
• <span class="h-code">navd</span>: ensure the arrival message doesn’t clear too quickly (#29032)
• <span class="h-code">navd</span>: fix excessive delay when clearing and picking a destination too quickly (#29067)
• <span class="h-code">navd</span>: when arriving at destination, don’t display empty banner instructions (#29034)

comma 3X support

The comma 3X is the latest generation hardware designed to run openpilot. Available for purchase now in the comma shop.

UI updates

It is important to clearly communicate openpilot engagement status. Previously, the onroad alerts and border colors could interact, and some states made it impossible to tell if openpilot was engaged. Now, the border always communicates engagement status, and the alerts’ floating appearance allows more view of the road. (#27416)

Additionally, when navigate on openpilot is active, the route path in the map will turn from blue to green, and the map is forced open.

Bug fixes

• Fix double-tapping on the experimental mode button (#27865)
• Ignore first tap to wake device (#28656)
• Prevent sidebar from showing alongside map (#28490)
• Use a stencil buffer to make onroad draw times more consistent (#28902)

Enhancements

• Adjust opacity instead of visibility to indicate engageability (#28765)
• Border always communicates engagement status (#27416)
• Map: route path color turns green when navigate on openpilot is active (#28958)
• Map: show the map when navigate on openpilot is active (#28867)
• Move navigation destination entry out of settings and to the onroad view (#28476)

multilang contributions

Multilanguage support has been brought to the offroad alerts thanks to deanlee! (#28512) Translations were also updated by AlexandreSato, crwusiz, eFiniLan, and Iamz.

AGNOS 8

This OS update includes bug fixes, package updates, and an additional two seconds shaved off the boot time. This update also improves AGNOS as a development environment. It’s now got the latest Python 3.11.4, and both <span class="h-code">pip install</span> and <span class="h-code">apt install</span> work. When you SSH in, the MOTD will greet you with a quick start guide:

  Welcome to AGNOS (GNU/Linux 4.9.103+ aarch64)

Here are some useful commands:
  Enter the tmux session where the comma service runs
    └── tmux a

  Install Ubuntu packages
    └── sudo apt update && sudo apt install 

  Install Python packages
    └── pip install 

When modifying AGNOS, keep in mind that the system partition isn't
particularly large. /data/ is the largest partition, and unlike the
rest of the partitions, it will persist across AGNOS updates.

Want a clean slate? Reflash AGNOS with https://flash.comma.a
  

Cars

In the last release, we made fingerprinting more reliable for Hyundai, Kia, and Genesis vehicles by better understanding their FW versions. In this release, we’re preparing to do the same for Toyota by gathering data from more ECUs including the hybrid control computer, transmission, and more. Stay tuned for an update in the next blog post!

Bug fixes

• Ford: fix “Park Not Available” alert when car fails to fingerprint (#28634)
• Ford: log ACC faults from the camera when using stock longitudinal (#28170)
• GM: fix ACC fault when enabling at a stop (#28699)
• Hyundai: fix blocked cancel message on select models (#28941)
• Mazda: fix blind spot monitor detection while indicating (#27919)
• Subaru: log stock AEB events (#28052)
• Volkswagen: fix ACC fault when coming to a stop on select models (#28248)

Enhancements

• Documentation: use alpha longitudinal control limits (#28709)
• GM: allow enabling down to a stop for Sierra and Silverado trucks when using alpha longitudinal control (#28708)
• GM: improve steering control for Acadia 2018 (#28923)
• Testing: introduce fuzzing test for the car interface, provides more complete coverage (#28530)

Car Ports

• Ford Focus 2018 support (#28551)
• Kia Carnival 2023 support thanks to sunnyhaibin! (#27785)

Join the team

We’re hiring great engineers to own and work on all parts of the openpilot stack. If anything here interests you, apply for a job or join us on GitHub! We’re also offering a cash hiring bounty if you refer someone.