DriveDreamer: Towards Real-world-driven
World Models for Autonomous Driving

World models, especially in autonomous driving, are trending and drawing extensive attention due to its capacity for comprehending driving environments. The established world model holds immense potential for the generation of high-quality driving videos, and driving policies for safe maneuvering. However, a critical limitation in relevant research lies in its predominant focus on gaming environments or simulated settings, thereby lacking the representation of real-world driving scenarios. Therefore, we introduce DriveDreamer, a pioneering world model entirely derived from real-world driving scenarios. Regarding that modeling the world in intricate driving scenes entails an overwhelming search space, we propose harnessing the powerful diffusion model to construct a comprehensive representation of the complex environment. Furthermore, we introduce a two-stage training pipeline. In the initial phase, DriveDreamer acquires a deep understanding of structured traffic constraints, while the subsequent stage equips it with the ability to anticipate future states. The proposed DriveDreamer is the first world model established from real-world driving scenarios. We instantiate DriveDreamer on the challenging nuScenes benchmark, and extensive experiments verify that DriveDreamer empowers precise, controllable video generation that faithfully captures the structural constraints of real-world traffic scenarios. Additionally, DriveDreamer enables the generation of realistic and reasonable driving policies, opening avenues for interaction and practical applications.
世界模型，尤其是自动驾驶模型，由于其理解驾驶环境的容量，正在成为趋势并吸引了广泛的注意力。已建立的世界模型在生成高质量驾驶视频和安全操纵的驾驶政策方面具有巨大的潜力。然而，相关研究的一个关键限制在于其主要关注游戏环境或模拟设置，从而缺乏真实世界驾驶场景的表示。因此，我们推出了 DriveDreamer，这是一款完全源自真实驾驶场景的开创性世界模型。考虑到在复杂的驾驶场景中建模世界需要巨大的搜索空间，我们建议利用强大的扩散模型来构建复杂环境的综合表示。此外，我们介绍了一个两级训练流水线。在初始阶段，DriveDreamer 深入了解结构化交通约束，而后续阶段使其具备预测未来状态的能力。拟议中的 DriveDreamer 是第一个根据真实驾驶场景建立的世界模型。我们在具有挑战性的 nuScenes 基准上实例化了 DriveDreamer，大量实验验证了 DriveDreamer 能够实现精确、可控的视频生成，忠实地捕捉现实世界交通场景的结构约束。此外，DriveDreamer 能够生成现实合理的驾驶策略，为交互和实际应用开辟了途径。