GStreamer 和 NNStreamer 使用 VX Delegate 失败

Question

我正在使用 BoundaryDevices iMX8MP 板，并希望在 NPU 上使用 TFLite 和 VX Delegate 运行预测。这是我的 GStreamer 命令：

gst-launch-1.0 \
    videotestsrc is-live=1 ! \      
    video/x-raw,width=640,height=480 ! \
    tee name=t \    
    t. ! queue name=thread-nn max-size-buffers=2 leaky=2 ! \
    imxvideoconvert_g2d ! video/x-raw,width=300,height=300,format=RGBA ! videoconvert ! video/x-raw,format=RGB ! \      
    tensor_converter ! \
    tensor_transform mode=arithmetic option=typecast:float32,add:-127.5,div:127.5 ! \      
    tensor_filter framework=tensorflow-lite model=/home/root/nxp-nnstreamer-examples/detection/../downloads/models/detection/ssdlite_mobilene
t_v2_coco_no_postprocess.tflite custom=Delegate:External,ExtDelegateLib:libvx_delegate.so ! \      
    tensor_decoder mode=bounding_boxes option1=mobilenet-ssd option2=/home/root/nxp-nnstreamer-examples/detection/../downloads/models/detection/coco_labels_list.txt option3=/home/root/nxp-nnstreamer-examples/detection/../downloads/models/detection/box_priors.txt option4=640:480 option5=300:300 ! \
    videoconvert ! \      
    mix. \
    t. ! queue name=thread-img max-size-buffers=2 leaky=2 ! \      
    videoconvert ! \
    imxcompositor_g2d name=mix sink_0::zorder=2 sink_1::zorder=1 latency=300000000 min-upstream-latency=300000000 ! \      
    fpsdisplaysink video-sink=$VID_SINK name=test sync=false text-overlay=true -v 2>&1

我正在使用

custom=Delegate:External,ExtDelegateLib:libvx_delegate.so

在 NPU 上使用 VX Delegate。我已经确认这个动态库存在并且可以使用，因为

benchmark-model

和

modelrunner

工具都允许使用NPU。当我运行此 GStreamer 命令时，我得到以下信息：

...
ERROR: Invoke called on model that is not ready.

** (gst-launch-1.0:1613): CRITICAL **: 21:30:40.018: Failed to invoke

** (gst-launch-1.0:1613): CRITICAL **: 21:30:40.019: /usr/lib/libnnstreamer-single.so(_backtrace_to_string+0x40) [0xffff6edb73c0]/usr/lib/gstreamer-1.0/libnnstreamer.so(+0x3d9ac) [0xffff6ee1d9ac]/usr/lib/libgstbase-1.0.so.0(+0x48704) [0xffff87198704]/usr/lib/libgstbase-1.0.so.0(+0x47dd4) [0xffff87197dd4]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]/usr/lib/libgstreamer-1.0.so.0(+0x90798) [0xffff87840798]/usr/lib/libgstbase-1.0.so.0(+0x47ee8) [0xffff87197ee8]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]/usr/lib/libgstreamer-1.0.so.0(+0x90798) [0xffff87840798]/usr/lib/gstreamer-1.0/libnnstreamer.so(+0x1fba0) [0xffff6edffba0]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]/usr/lib/libgstreamer-1.0.so.0(+0x90798) [0xffff87840798]/usr/lib/libgstbase-1.0.so.0(+0x47ee8) [0xffff87197ee8]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]/usr/lib/libgstreamer-1.0.so.0(+0x90798) [0xffff87840798]/usr/lib/libgstbase-1.0.so.0(+0x47ee8) [0xffff87197ee8]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]/usr/lib/libgstreamer-1.0.so.0(+0x90798) [0xffff87840798]/usr/lib/libgstbase-1.0.so.0(+0x47ee8) [0xffff87197ee8]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]


** (gst-launch-1.0:1613): CRITICAL **: 21:30:40.019: Calling invoke function (inference instance) of the tensor-filter subplugin (tensorflow-lite for /home/root/nxp-nnstreamer-examples/detection/../downloads/models/detection/ssdlite_mobilenet_v2_coco_no_postprocess.tflite) has failed with error code (-1).

ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
Additional debug info:
../git/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
streaming stopped, reason error (-5)
Execution ended after 0:00:00.063685468
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstQueue:thread-nn: Internal data stream error.
Additional debug info:
../git/plugins/elements/gstqueue.c(992): gst_queue_handle_sink_event (): /GstPipeline:pipeline0/GstQueue:thread-nn:
streaming stopped, reason error (-5)
...

任何解决此问题的帮助将不胜感激。如果这不是发布此问题的正确堆栈交换，请告诉我可以在哪里发布以获得答案。如果您需要有关我正在运行的系统的更多信息，也请告诉我。

有关系统的更多信息：

• Linux 信息：

  Linux nitrogen8mp 5.15.71-2.2.0+yocto+gd4a57b30c1d1

• 安装了 GStreamer、NNStreamer 和 TFLite VX Delegate 的自定义 Yocto 构建。

Answer 1

https://github.com/nnstreamer/nnstreamer/issues 将是提出这个问题的正确位置。

对于管道，我建议使用较小的管道进行测试和重现。

一些小问题想请教：

• Tensorflow-lite 已返回错误（似乎解释器->Invoke() != kTfLiteOk），并且需要检查给定的输入或输出是否适合给定的模型文件。
• 检查 imxvideoconvert_g2d 是否可以执行您在设备中请求的操作（同时缩放和转换）
• 您测试过没有委托吗？