13 Displaying Real-Time Video Stream in Jupyter Lab

In the previous chapter, we used Flask to display the real-time camera feed, a method that required opening a new tab in the browser or accessing it from another device. In this chapter, we'll explore a solution for viewing the real-time video stream directly in Jupyter Lab.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the "+" icon next to the tab for this page to open a new tab called "Launcher."

2. Click on "Terminal" under "Other" to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: crontab -e.

6. If prompted to choose an editor, enter 1 and press Enter to select nano.

7. After opening the crontab configuration file, you'll see the following two lines:

@reboot ~/ugv_pt_rpi/ugv-env/bin/python ~/ugv_pt_rpi/app.py >> ~/ugv.log 2>&1
@reboot /bin/bash ~/ugv_pt_rpi/start_jupyter.sh >> ~/jupyter_log.log 2>&1

8. Add a # character at the beginning of the line with ……app.py >> …… to comment out this line.

#@reboot ~/ugv_pt_rpi/ugv-env/bin/python ~/ugv_pt_rpi/app.py >> ~/ugv.log 2>&1
@reboot /bin/bash ~/ugv_pt_rpi/start_jupyter.sh >> ~/jupyter_log.log 2>&1

9. Press Ctrl + X in the terminal window to exit. It will ask you Save modified buffer? Enter Y and press Enter to save the changes.

10. Reboot the device. Note that this process will temporarily close the current Jupyter Lab session. If you didn't comment out ……start_jupyter.sh >>…… in the previous step, you can still use Jupyter Lab normally after the robot reboots (JupyterLab and the robot's main program app.py run independently). You may need to refresh the page.

11. One thing to note is that since the lower machine continues to communicate with the upper machine through the serial port, the upper machine may not start up properly during the restart process due to the continuous change of serial port levels. Taking the case where the upper machine is a Raspberry Pi, after the Raspberry Pi is shut down and the green LED is constantly on without the green LED blinking, you can turn off the power switch of the robot, then turn it on again, and the robot will restart normally.

12. Enter the reboot command: sudo reboot.

13. After waiting for the device to restart (during the restart process, the green LED of the Raspberry Pi will blink, and when the frequency of the green LED blinking decreases or goes out, it means that the startup is successful), refresh the page and continue with the remaining part of this tutorial.

Example

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

import matplotlib.pyplot as plt  # Import the matplotlib library for plotting
import cv2  # Import the OpenCV library for image processing
from picamera2 import Picamera2  # Import the Picamera2 library for accessing the Raspberry Pi Camera
import numpy as np  # Import the NumPy library for mathematical computations
from IPython.display import display, Image  # Import IPython display functionality
import ipywidgets as widgets  # Import the ipywidgets library for creating interactive widgets
import threading  # Import the threading library for multithreading

# Create a toggle button as a stop button
stopButton = widgets.ToggleButton(
    value=False,  # The initial state of the button is unselected
    description='Stop',  # Text displayed on the button
    disabled=False,  # The button is initially enabled
    button_style='danger',  # The button style is red
    tooltip='Description',  # Tooltip displayed when hovering over the button
    icon='square'  # Icon displayed on the button
)

# Define a function for displaying the video stream
def view(button):
    # If you are using a CSI camera you need to comment out the picam2 code and the camera code.
    # Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
    # picam2 = Picamera2()  # Create an instance of Picamera2
    # Configure camera parameters, set video format and size
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
    # picam2.start()  # Start the camera
    camera = cv2.VideoCapture(-1) #Create camera example
    #Set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle=display(None, display_id=True)  # Create a display handle for updating the displayed content
    while True:
        # frame = picam2.capture_array()  # Capture a frame from the camera
        _, frame = camera.read()
        
        # You can perform frame processing here if needed (e.g., flipping, color conversion, etc.)

        _, frame = cv2.imencode('.jpeg', frame)  # Encode the frame as JPEG format
        display_handle.update(Image(data=frame.tobytes()))  # Update the displayed image
        if stopButton.value==True:  # Check if the stop button is pressed
            #picam2.close()  # If yes, close the camera
            cv2.release() # if yes, close the camera
            display_handle.update(None)  # Clear the displayed content

# Display the stop button
display(stopButton)
# Create and start a thread, with the target function as view and the stop button as the argument
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()  # Start the thread