Description:

Autonomous navigation in dynamic and unstructured environments is a critical challenge for mobile robots. While traditional path planning algorithms can compute static routes, they often struggle with unpredictable obstacles, dynamic environments, or high-level instructions provided in natural language. Recent advances in Large Language Models (LLM) and Diffusion Models offer new possibilities for intelligent decision-making and environment prediction, enabling robots to better understand commands and anticipate changes in complex surroundings.

This research proposes a novel system that integrates LLM, Diffusion Models, and the Spot Robot to achieve real-time intelligent navigation. The LLM acts as the cognitive module, interpreting high-level natural language instructions and generating navigation strategies. The Diffusion model serves as a predictive perception system, generating dynamic environment heatmaps and simulating potential obstacles. Spot functions as the robotic executor, performing path execution, obstacle avoidance, and real-time feedback to the system. The resulting system enables a fully “natural language command → intelligent decision → autonomous execution” loop.

Objectives:

High-Level Instruction Understanding and Strategy Generation (LLM):
Develop a framework to interpret natural language commands and convert them into actionable navigation goals and strategies. Incorporate methods such as GPT or LLaMA series models, optionally combined with reinforcement learning or rule-based constraints, to ensure safe and accurate task planning.

Environment Prediction and Simulation (Diffusion):
Utilize Diffusion models to process Spot’s sensor data—including cameras, LiDAR, or depth sensors—to generate predictive environment heatmaps, anticipate moving obstacles, and simulate feasible navigation routes.

Path Planning and Execution (Spot):
Implement real-time path planning and execution by combining outputs from LLM and Diffusion models. Integrate traditional algorithms (e.g., A*, RRT*) with dynamic obstacle avoidance strategies to ensure safe navigation under unpredictable conditions.

Closed-Loop Feedback System:
Continuously update LLM and Diffusion predictions based on Spot’s real-time sensor feedback to adjust navigation strategies dynamically and improve overall robustness.

Expected Contributions:

Propose a novel integration of LLM, Diffusion Models, and robotic execution for autonomous navigation in dynamic environments.

Demonstrate the feasibility of natural language-guided navigation with predictive environment modeling.

Provide insights into how multi-modal AI systems can enhance robotic path planning, obstacle avoidance, and task execution.

Lay groundwork for real-world applications in search and rescue, indoor/outdoor inspection, and human-robot interaction scenarios.

Requirements and Technical Skills:

Proficiency in Python and experience with deep learning frameworks such as PyTorch or TensorFlow.

Familiarity with Large Language Models, Diffusion Models, and reinforcement learning or rule-based strategy integration.

Experience with robotic platforms (ROS, Spot SDK) and real-time sensor data processing.

Knowledge of path planning algorithms (A*, RRT*, D* Lite) and dynamic obstacle avoidance techniques.

Ability to design, implement, and evaluate experimental robotic systems independently.

The applicant will use publicly available datasets and receive technical support from HUN-REN SZTAKI.

External Thesis Supervisor: András László Majdik, Ph.D., 

Senior Research Fellow； Email: majdik.andras@sztaki.hun-ren.hu