TU Delft Uses GPT-4o-mini to Adjust Self-Driving Style

Researchers at the Delft University of Technology (TU Delft) in the Netherlands have created a system that lets passengers alter a self-driving car's behavior using natural language. By integrating OpenAI's GPT-4o-mini model, the system translates conversational requests like "I am running late, go fast" into technical adjustments for the vehicle's underlying motion planner. This setup allows passengers to customize their ride on the fly without compromising the vehicle's core safety systems.

Instead of giving the large language model direct control over steering or acceleration—which is dangerous due to slow response times and a lack of performance guarantees—the TU Delft system uses the AI to tune a model predictive-path integral controller. This controller evaluates multiple potential trajectories based on criteria such as speed, steering angle, and collision probability. The GPT-4o-mini model interprets the passenger's prompt alongside a description of the driving environment, then adjusts the weights of these criteria. Before implementing any changes, the system explains its plan in plain language and waits for the passenger to confirm.

The research team, led by postdoctoral researcher Diego Martinez-Baselga, tested their system in the nuPlan self-driving simulator during complex highway merging scenarios. Across eight different natural-language prompts, the system successfully modified the vehicle's speed and steering smoothness to match user intent. For example, prompts indicating carsickness prompted the system to prioritize smoother steering and gentler acceleration.

For autonomous vehicle practitioners, this approach solves a major hurdle in human-robot interaction by separating the flexible, subjective reasoning of language models from the deterministic safety constraints of traditional motion planners. While other researchers, such as Nicolas Baumann at ETH Zurich and Matthias Althoff at the Technical University of Munich, have explored similar LLM-based control and verification methods, the TU Delft design highlights how developers can safely introduce user-driven customization. It proves that passenger comfort can be dynamically balanced with safety-critical operations without rewriting the vehicle's core control algorithms.