UGV Rover PI ROS2 8. Web-based natural language interaction
| ||
8. Web Natural Language Interaction
This tutorial will introduce how to directly control the robot using natural language on the web page. In this tutorial, we use the Ollama command-line tool to run a large language model, enter the corresponding commands through the command line to use the Gemma model to generate JSON instructions, and send the generated JSON instructions to the robot, which supports the robot to perform tasks such as basic movement, assisted mapping, and navigation.
8.1 Ollama introduction
Ollama is an open source large language model service tool that simplifies running large language models locally and lowers the threshold for using large language models. It also provides a rich pre-built model library, including Qwen2, Llama3, Phi3, Gemma2 and other large-scale language models that are open source and can be easily integrated into various applications. Ollama's goal is to make it easy to deploy and interact with large language models, both for developers and end users.
8.2 Install and configure Ollama
1. Click Ollama official website to download the Ollama installation package. Ollama officially supports downloading for different operating systems. According to your operating system, select the corresponding installation package to download. In this tutorial, it is installed on the Windows operating system. After the download is complete, run the installation package and follow the prompts to complete the installation process.
2. After the installation is completed, you need to configure the Windows system environment variables. In the Start menu bar, search for Advanced system settings and select Environment variables in the system settings.
3. Select New in user variables, fill in the variable name OLLAMA_HOST and variable value :11434, as shown below. Click OK to save after filling in.
4. As shown in the figure below, the addition is successful. Finally, click OK to save the environment variable configuration.
8.3 Download large model to local
Ollama supports a range of models available at Ollama Model Library, below are some example models that can be downloaded, the Gemma2:9B model is used in this tutorial.
NOTE: You should have at least 8GB RAM to run a 7B model, 16GB RAM to run a 13B model, and 32GB RAM to run a 33B model. We use the Gemma2:9B model in our tutorial, so you need to ensure that your computer's RAM is 16GB or above.
1. First, make sure your computer’s RAM is 16 GB or above. Search for Task Manager in the Start menu bar, select Memory and check if the memory size is 16GB or above.
2. Then in Windows system, press Ctrl+R and enter cmd to open the terminal, and use the pull command to completely download the large model from the Ollama remote repository to the local computer.
ollama pull gemma2
3. Once the download is complete, you can run a large model for testing, and randomly test the conversation with AI.
ollama run gemma2
8.4 Change IP address
First, determine what the local IP address you are using for your Ollama deployment and remember that IP address.
- Windows system
Press Ctrl+R, enter cmd to open the terminal, and enter the command to view the IP address in the terminal:
ipconfig
Remember this IP address, and then in the Raspberry Pi operating system, open the app.py file in the /home/ws/ugv_ws/src/ugv_main/ugv_chat_ai/ugv_chat_ai/ directory, as shown below, and change the IP address before line 85 ":11434" to the local IP address you got earlier.
Press Ctrl+S to save and exit after modification.
8.5 Chassis drive
Before starting the chassis driver, by default you have completed the main program and remotely connected to the Docker container according to the content in Chapter 1 UGV Rover PI ROS2 1. Preparation.
In the container, go to the workspace of the ROS 2 project for that product:
cd /home/ws/ugv_ws
Start the robot driver node:
ros2 launch ugv_bringup bringup_lidar.launch.py use_rviz:=true
At this moment, the robot pan-tilt will be turned to the center position, and the camera will face forward, and you can view the RVIZ2 model interface of the product. When the robot rotates in place, you can see that the model will also rotate with the robot.
8.6 Launch Web AI application
Open a new Docker container terminal, click the "⭐" symbol in the left sidebar, double-click to open Docker's remote terminal, enter username: root, password: ws.
Run the web-side AI application:
ros2 run ugv_chat_ai app
After successful operation, it will appear as shown below:
Then, in the browser of the operating system where Ollama is deployed, visit the URL Raspberry Pi IP Address: 5000 to enter the interface of the AI application. As shown in the picture above, the URL visited here is http://192.168.10.131:5000.
In the chat interface with AI, we need to send some prompts to AI first, so that AI can feedback JSON instructions to control the robot's movement. Here are the tips we provide, which can be copied and sent directly to AI:
You are an assistant helping me with the simulator for robots.Here are some tips you can use to command the robot.
{"T": "1", "type": type, "data": data}
type: drive_on_heading back_up spin stop
data: num
you should return only json in code,without explanation
for examples,
"user": "move 2 units forward."
"assistant": {"T": 1, "type": "drive_on_heading", "data": 2}
"user": "move 2 units back."
"assistant": {"T": 1, "type": "back_up", "data": 2}
"user": "turn left 30 degrees."
"assistant": {"T": 1, "type": "spin", "data": 30}
"user": "stop."
"assistant": {"T": 1, "type": "stop", "data": 0}
Wait for about 1 minute, and the AI will feedback the JSON instruction to move 2m straight, as shown below:
- type: The types of robot movement. driver_on_heading: move forward; back_up: move backward; spin: rotate; stop: stop.
- data: The numerical parameters of robot movement. When moving forward or backward, the unit is m; when rotating, the unit is degrees.
Next, you need to start the relevant interface for robot movement. After the AI feeds back the JSON command, the actual robot will move accordingly. Open a new Docker container terminal and run the following command:
ros2 run ugv_tools behavior_ctrl
After the operation is completed, there will be no feedback before the robot is given motion instructions.
Example: Let the robot move forward 1m. The AI application will feedback a JSON command to move forward 1m. The ros2 run ugv_tools behavior_ctrl command will feedback parameters such as the robot's movement distance and current distance. The robot will stop after completing the 1m movement.
