如何从consumers.py获取一批帧？

Question

我正在开发一个计算机视觉项目，涉及使用 OpenCV 和 Django 进行对象检测和实时帧流。为了实现这一点，我正在利用 Django 通道。然而，我遇到了一个新的挑战，我需要执行视频分类。这需要向模型发送一批帧。任何人都可以提供有关如何解决此问题的任何指导吗？

我在前端添加了一个按钮。我期望的是从后端consumer.py 获取单个时间戳。我得到了开始帧的时间戳，直到单击按钮的帧的时间戳。

在consumers.py 文件中，我利用OpenCV 库迭代帧并将数据传输到WebSocket 客户端。这是代码示例（我还提供了注释以便更好地理解）。

# Import the 'cv2' module, which is the OpenCV library for computer vision tasks.
import cv2

# Import the 'json' module for working with JSON data.
import json

# Import the 'asyncio' module for asynchronous programming support.
import asyncio

# Import the 'datetime' module for working with date and time.
import datetime

# Import the 'numpy' module for numerical and array operations.
import numpy as np

# Import a custom function 'serialize_frame' from the 'processing' module in the current application.
from .processing import serialize_frame

# Import the 'AsyncWebsocketConsumer' class from Django's 'channels.generic.websocket' module.
from channels.generic.websocket import AsyncWebsocketConsumer

# Define a custom WebSocket consumer class named VideoConsumer that handles WebSocket communication.
class VideoConsumer(AsyncWebsocketConsumer):
    # Define the 'connect' method for WebSocket connection handling.

    # This method is executed when a WebSocket connection is established.
    async def connect(self):
        # Accept the WebSocket connection.
        await self.accept()

        # Retrieve the video file path from the URL route parameters.
        video_file_path = self.scope['url_route']['kwargs']['video_path']

        # Send processed frames using the video file path.
        await self.send_processed_frames(video_file_path)

    # Define the 'send_frame' method for sending video frame data and match data to the WebSocket client.

    # This method takes 'frame_data' and 'match_data' as parameters and sends them as JSON to the WebSocket client.
    async def send_frame(self, frame_data, timestamp_str):
        # Convert the frame data and match data into a JSON format and send it to the client.
        await self.send(text_data=json.dumps({
            'frame_data': frame_data,
            'timestamp': timestamp_str
        }))

    # Define the 'send_processed_frames' method for processing and sending video frames and match data.
    # This method takes 'video_file_path' as a parameter and processes video frames and related data.
    async def send_processed_frames(self, video_file_path):
        # Initialize data structures and variables.
        # A dictionary for actions

        try:
            cap = cv2.VideoCapture(video_file_path)
            
            # Start iterating over the frames
            while True:

                ret, frame = cap.read()

                #  Get the timestamp of the frame on which the prediction is being performed
                timestamp = cap.get(cv2.CAP_PROP_POS_MSEC)
                timestamp = datetime.timedelta(milliseconds=timestamp)

                timestamp_str = str(timestamp).split('.', 2)[0]

                # Encode the frames to base64 to send to the web
                frame_data = serialize_frame(frame)  # Implement the serialization function

                # Send the frame data and match data to the WebSocket client.
                await self.send_frame(frame_data, timestamp_str)
                # Adjust the frame rate as needed (e.g., 30 frames per second).
                await asyncio.sleep(0.033)


                # Release the video capture object.
            cap.release()
        except Exception as e:
            # In case of an error, send an error message to the WebSocket client.
            await self.send(text_data=f"Error: {str(e)}")

    async def receive(self, text_data):

        # Handle messages from WebSocket
        message = json.loads(text_data)

        print(message)

我已经在routing.py中指定了路由。

# Import necessary modules for URL routing in Django.
from django.urls import re_path

# Import the 'VideoConsumer' class from the 'consumers' module of the current application.
from . import consumers

# Define WebSocket URL patterns for the application.

# Create a WebSocket URL pattern that captures the 'video_path' parameter from the URL.
# It routes WebSocket requests to the 'VideoConsumer' class using the 'as_asgi()' method.
websocket_urlpatterns = [
    re_path(r'ws/video/(?P<video_path>[\w\W]+)/$', consumers.VideoConsumer.as_asgi())
]

然后我将数据从后端发送到前端。

{% extends 'main.html' %}
{% block content %}
<div class="container text-center">
    <h1>Processing Video</h1>
    <br>

    <!-- <img class="rounded" id="liveImage" width="800" height="600"> -->
    <div class="liveImage">
        <div class="text-secondary text-center" role="status">
        </div>
        <img class="rounded" id="liveImage">
    </div>  
    <hr>
    
    <h5>To classify the cover type, specify the start duration to clip the video</h5>
    <div class="d-grid gap-2">
        <button class="btn btn-primary" id="sendTimestampButton" type="button">Button</button>
    </div>
</div>

<br>

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.4/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/js/all.min.js"></script>
<script type="text/javascript">
    document.addEventListener("DOMContentLoaded", function () {
        let videoPath = "{{ video_path }}";  // Use the video path from your Django template
        let liveImage = document.getElementById("liveImage");

        // Establish a WebSocket connection to your server
        let wsProtocol = window.location.protocol === "https:" ? "wss" : "ws";
        let wsURL = `${wsProtocol}://${window.location.host}/ws/video/${videoPath}/`;
        let socket = new WebSocket(wsURL);

        // document.getElementById('sendTimestampButton').addEventListener('click', function () {
        //     // Send a message to the backend to request the timestamp
        //     socket.send(JSON.stringify({'request_timestamp': true}));
        //     console.log('TimeStamp: ');
        // });

        socket.onopen = function (event) {
            console.log('Websocket Connection established')
        }
        
        socket.onmessage = function (event) {
            let data = JSON.parse(event.data);

            document.getElementById('sendTimestampButton').addEventListener('click', function () {
            //     // Send a message to the backend to request the timestamp
                socket.send(JSON.stringify({'timestamp': data.timestamp}));
                console.log('Timestamp: ', data.timestamp);
            });

                
                liveImage.src = "data:image/jpeg;base64," + data.frame_data;
            } 

        socket.onclose = function (event) {
            console.log("WebSocket connection closed.");
        };

        socket.onerror = function (error) {
            console.error("WebSocket error:", error);
        };
    });    
</script>
{% endblock content %}

要查看完整的代码，请检查存储库：https://github.com/ShahZebYousafzai/Django-Real-Time-Inference-on-video

Answer 1

我能够解决这个问题！我在consumers.py中的

VideoConsumer

类的开头定义了一个初始变量。在 while 循环中，我将此变量描述为类变量并存储帧的当前时间戳。在 veiws.py 中，我创建了一个发送 JSON 响应的函数类。我正在获取时间戳并将其作为该函数中的 JSON 响应发送。在前端，我使用 AJAX 来获取响应。每当我单击按钮时，我都会得到该帧的时间戳。