Use Flask to achieve low-latency video transmission

This section introduces how to use Flask to build a web application to display the real-time picture of the robot camera, because the web application has the cross-platform feature, users can watch the real-time picture of the camera through the browser on mobile phones/PCs/tablets and other devices, and realize the wireless video transmission function.

What is Flask?

Flask is a lightweight web application framework for building web applications quickly using Python.

Lightweight: Flask is a lightweight framework with a relatively small core library, but it is flexible and extensible enough for developers to choose to add as many extensions and libraries as they need.
Simple and easy to use: Flask has a simple design and is easy to use. Its API is clear and well-documented, allowing developers to get started and build web applications quickly.
Routing system: Flask uses decorators to define URL routes, mapping requests to the appropriate handlers. This makes it intuitive and simple to create different pages and handle different requests.
Templating Engine: Flask integrates with the Jinja2 templating engine, making it easier to build dynamic content in your app. The templating engine allows you to embed dynamically generated content in HTML.
IDV Server: Flask comes with a simple IDV server for easy development and debugging. However, in a production environment, it is advisable to use a more powerful web server such as Gunicorn or uWSGI.
Plugins and extensions: Flask supports many plugins and extensions in order to add extra functionality such as database integration, authentication, form processing, and more.
RESTful support: Flask provides good support for RESTful APIs, making it easy to build and design RESTful APIs.
WSGI compatible: Flask is based on WSGI (Web Server Gateway Interface), which allows it to run on many WSGI compliant web servers.
Active community: Flask has a large and active community, which means you can easily find tons of documentation, tutorials, and third-party extensions, as well as get support.

Preparation

Since the main program will automatically run by default when the product is booted up, the main program will occupy the camera resources, in this case, this tutorial cannot be used, you need to end the main program or prohibit the main program from running automatically before restarting the robot.

It should be noted here that the regular sudo killall python method usually does not work due to the multi-threading used in the main bot program and the automatic operation is configured by crontab, so we will introduce the method of disabling the automatic running of the main program.

If you have disabled the automatic startup of the bot's main program, you do not need to perform the End Main Program section below.

End Main Program

1. Clicking on the + sign next to the tab on this page above will open a new tab called Launcher.

2. Click Terminal in Other to open the Terminal window.

3. Type bash in the terminal window and press enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python

Web Application demo

Note that the following code block cannot be run in Jupyter Lab

The following code cannot be run in Jupyter Lab because the Flask application conflicts with Jupyter Lab over the use of port numbers, and the following programs are stored in folders named 12 in tutorial_cn and tutorial_en, and there is also a file named in folder 12 template's folder is used to store web page resources, and here's how the demo runs.

1. Use the method described above to open the terminal, at this time pay attention to the folder path on the left, the default path of the newly opened terminal is the same as the file path on the left, you need to browse to the tutorial_cn or tutorial_en folder, open the terminal and enter cd 12 to browse to the 12 folder.

2. Use the following command to start the Flask web application server: python flask_camera.py

3. Then open the browser on the device in the same LAN (or this device can open a new tab in the browser), enter the IP of Jetson: 5000 (for example, if the IP address of Jetson is 192.168.10.104, open the address 192.168.10.104:5000), note that : needs to be a colon in English.

4. Use Ctrl + C in the terminal to end the run.

Introduction to Flask program

from flask import Flask, render_template, Response  # Import the Flask class from the flask library, the render_template function is used to render the HTML template, and the Response class is used to generate the response object
from picamera2 import Picamera2 # Import Picamera2 class from picamera2 library for accessing and controlling the camera 
import time # Import time module, which can be used to handle time-related tasks 
import cv2 # Import OpenCV library for image processing

app = Flask(__name__) # Create an instance of the Flask application

def gen_frames(): # Define a generator function that generates the image captured by the camera on a frame-by-frame basis 
    picam2 = Picamera2() # Create an instance of Picamera2

    # Configure camera parameters and set the format and size of the video 
    picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))

    picam2.start() # Start the camera 
    while True: 
        frame = picam2.capture_array() # Capture a frame from the camera 
        frame = cv2.cvtColor(frame, cv2. COLOR_BGR2RGB) 
        ret, buffer = cv2.imencode('.jpg', frame) # Encodes the captured image frame to JPEG format

        frame = buffer.tobytes() # Convert JPEG images to byte streams

        # Use yield to return an image byte stream, so that you can send video frames continuously to form a video stream 
        yield (b'--frame\r\n' 
            b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

@app.route('/') # Define the root route 
def index(): 
    return render_template('index.html') # Return index.html page

@app.route('/video_feed') # Define the video stream route 
def video_feed(): 
    # Returns the response object, the content is the video stream, and the content type is multipart/x-mixed-replace 
    return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')

if __name__ == '__main__': 
    app.run(host='0.0.0.0', port=5000, debug=True) # Start the Flask application, listen on port 5000 on all network interfaces, and enable debug mode

Descriptions of some key parts of the code

gen_frames(): This is a generator function that continuously captures frames from the camera, encodes them into JPEG format, and generates frame bytes as part of a multi-part response. The resulting frames are transmitted to the client in real time.

@app.route('/'): This decorator associates the index() function with the root URL(/). When a user accesses the root URL, it will render an HTML template named '12_index.html'.

@app.route('/video_feed'): This decorator associates the video_feed() function with the '/video_feed' URL. This route is used for real-time video transmission, and frames are sent as multipart responses.

app.run(host='0.0.0.0', port=5000, debug=True): This line starts the Flask development server and listens for all available network interfaces (0.0.0.0) on port 5000. debug=True option to enable the server's debug mode. The debug mode of the server.

Introduction to the Web Section

Notes: 
<!doctype html>: Declares the HTML document type. 
<html lang="en">: The root element of the HTML document, specifying that the page language is English. 
<head>: Contains meta information about the document, such as character set and page title. 
<-- Required meta tags -->: HTML annotation, reminding that these are some required meta tags. 
<meta charset="utf-8">: Specifies that the document uses the UTF-8 character set. 
<title>Live Video Based on Flask</title>: Set page title. 
<body>: Contains the visible part of the document. 
<-- The image tag below is dynamically updated with the video feed from Flask -->: HTML annotation stating that the image tag below dynamically updates to show the video stream from Flask. 
<img src="{{ url_for('video_feed') }}">: Image label that uses the video_feed route defined in Flask to get a live video stream.

Navigation menu

Jetson 12 Implement Low Latency Video Transmission with Flask

Contents