区分地标（左手和右手）

Question

我一直在尝试区分mediapipe和opencv中左右手具有相同编号的地标。问题是我们有 20 个 ID 号码，但对于双手来说，我们有 40 个地标。最终我想分别提取左侧和右侧的地标#8 的坐标。这是我迄今为止所做的最小工作，将不同颜色（圆圈）的标记放在左右手上具有相同数字的地标上，但它不起作用（它们再次获得相同的颜色）：

import cv2
import mediapipe as mp
import time

cap = cv2.VideoCapture(0)
mpHands = mp.solutions.hands
hands = mpHands.Hands(max_num_hands=2)
mpDraw = mp.solutions.drawing_utils

while True:    
    
success, img = cap.read()
img = cv2.flip(img, 1) 
time.sleep(0.1)
imgRGB = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)

results = hands.process(imgRGB)
    
if results.multi_hand_landmarks:
    
    for handLms in results.multi_hand_landmarks:        
         
        for id, lm in enumerate(handLms.landmark):
                                    
            h,w,c = img.shape    
            cx, cy, cz = int(lm.x*w), int (lm.y*h) , int (lm.z*c)
            lbl = results.multi_handedness[0].classification[0].label
                           
            if lbl == 'Right':
                
                if id == 8:
                    cv2.circle(img, (cx,cy),8, (255,0,0), cv2.FILLED)
                    
            if lbl == 'Left':
                
                if id == 8 :
                    cv2.circle(img, (cx,cy),8, (255,255,0), cv2.FILLED)

        mpDraw.draw_landmarks(img, handLms, mpHands.HAND_CONNECTIONS,\
            mpDraw.DrawingSpec(color=(121,0,255)),\
                mpDraw.DrawingSpec(color=(0,255,0)))
    
cv2.imshow("image", img)
c = cv2.waitKey(7) % 0x100
if c == 27:
    break

我希望有一个解决方案可以完全区分双手的所有 40 个地标。

Answer 1

使用此处描述的 MP API，区分手部标志的正确方法是解析 HandLandmarkerResult中的 Handedness 字段。

即

import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision

BaseOptions = mp.tasks.BaseOptions
HandLandmarker = mp.tasks.vision.HandLandmarker
HandLandmarkerOptions = mp.tasks.vision.HandLandmarkerOptions
VisionRunningMode = mp.tasks.vision.RunningMode

# Create a hand landmarker instance with the video mode:
options = HandLandmarkerOptions(
    base_options=BaseOptions(model_asset_path='/path/to/model.task'),
    running_mode=VisionRunningMode.VIDEO)  # this example is for offline video processing
my_detector = HandLandmarker.create_from_options(options)
# Use OpenCV’s VideoCapture to load the input video.
# Load the frame rate of the video using OpenCV’s CV_CAP_PROP_FPS
# You’ll need it to calculate the timestamp for each frame.
# Loop through each frame in the video using VideoCapture#read()
# Convert the frame received from OpenCV to a MediaPipe’s Image object.
mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=numpy_frame_from_opencv)
hand_landmarker_result = my_detector.detect_for_video(mp_image, frame_timestamp_ms)
right_landmarks = np.zeros((21, 2), dtype=np.float32)
left_landmarks = np.zeros((21, 2), dtype=np.float32)
for j in range(len(detection_result.hand_landmarks)):
    # Verify these belong to right hand (otherwise, left)
    handness = detection_result.handedness[j][0].display_name == "Right"
    landmarks = np.array(
        [
            [landmark.x, landmark.y]
            for landmark in detection_result.hand_landmarks[j]
        ]
    ).astype(np.float32)
    # further processing ...