TimedMetadata使用MediaStreamSource和MediaTranscoder进行编码

Question

我正在构建一个C＃.NET Core应用程序，该应用程序收集用户交互数据并将其存储在视频供稿旁边。我打算利用MPEG容器中的定时元数据来存储交互数据和h264编码的视频数据。

According to the official documentation，似乎可以在使用MediaTranscoder进行转码操作期间对元数据流进行编码。我正在使用的代码已从screen recorder example修改。通过使用自定义的MediaEncodingProfile设置来支持元数据以及同时包含MediaStreamSource和VideoStreamDescriptor的TimedMetadataStreamDescriptor来工作，然后使用它们来对MediaTranscoder进行编码。

public class Encoder
{
    private MediaEncodingProfile GetEncodingProfile()
    {
        var profile = new MediaEncodingProfile();
        var containerEncoding = new ContainerEncodingProperties
        {
            Subtype = MediaEncodingSubtypes.Mpeg4
        };

        var videoEncoding = new VideoEncodingProperties
        {
            Subtype = MediaEncodingSubtypes.H264,
            Width = configuration.Width,
            Height = configuration.Height,
            Bitrate = configuration.BitsPerSecond,
            FrameRate = { Denominator = 1, Numerator = configuration.FramesPerSecond },
            PixelAspectRatio = { Denominator = 1, Numerator = 1 }
        };

        profile.Container = containerEncoding;
        profile.Video = videoEncoding;

        return profile;
    }

    private Tuple<VideoStreamDescriptor, TimedMetadataStreamDescriptor> GetStreamDescriptors()
    {
        var videoEncoding = VideoEncodingProperties.CreateUncompressed(MediaEncodingSubtypes.Bgra8,
                                                                       configuration.InputWidth, configuration.InputHeight);

        var videoStreamDescriptor = new VideoStreamDescriptor(videoEncoding);

        var metadataEncoding = new TimedMetadataEncodingProperties
        {
            Subtype = "{36002D6F-4D0D-4FD7-8538-5680DA4ED58D}"
        };

        byte[] streamFormatData = GetMetadataStreamFormatData(); // This just sets some arbitrary bytes

        metadataEncoding.SetFormatUserData(streamFormatData);

        var metadataStreamDescriptor = new TimedMetadataStreamDescriptor(metadataEncoding);

        return new Tuple<VideoStreamDescriptor, TimedMetadataStreamDescriptor>(
                videoStreamDescriptor, metadataStreamDescriptor);
   }

   private MediaStreamSource GetMediaStreamSource(IMediaStreamDescriptor videoStreamDescriptor,
                                                  IMediaStreamDescriptor metadataStreamDescriptor)
   {
        var mediaStreamSource = new MediaStreamSource(videoStreamDescriptor, metadataStreamDescriptor)
        {
            BufferTime = TimeSpan.FromSeconds(0)
        };

        mediaStreamSource.Starting += OnStart;
        mediaStreamSource.SampleRequested += OnSampleRequested;

        return mediaStreamSource;
    }

    private void OnStart(MediaStreamSource sender, MediaStreamSourceStartingEventArgs args)
    {
        // Intentionally omitted
    }

    private void OnSampleRequested(MediaStreamSource sender, MediaStreamSourceSampleRequestedEventArgs args)
    {
        // This only gets called for the video stream, not the metadata stream
    }

    private MediaTranscoder GetTranscoder()
    {
        var transcoder = new MediaTranscoder { HardwareAccelerationEnabled = true };
        return transcoder;
    }

    public async Task TranscodeAsync()
    {
        var transcoder = GetTranscoder();

        var (videoStreamDescriptor, metadataStreamDescriptor) = GetStreamDescriptors();

        var mediaStreamSource = GetMediaStreamSource(videoStreamDescriptor, metadataStreamDescriptor);
        var encodingProfile = GetEncodingProfile();
        await using var destinationFile = File.Open(configuration.FilePath, FileMode.Create);

        var prepareTranscodeResult = await transcoder.PrepareMediaStreamSourceTranscodeAsync(
                    mediaStreamSource, destinationFile.AsRandomAccessStream(), encodingProfile);
        await prepareTranscodeResult.TranscodeAsync().AsTask();
    }
}

我面临的问题是，定时的元数据流不会引发SampleRequested事件，而只会引发视频流。在测试期间，我将定时的元数据流替换为音频流，并且视频流和音频流均正确引发了SampleRequested事件。我怀疑可能使用TimedMetadataTrack和DataCue的方式将数据添加到定时的元数据流中的方式有​​所不同，但是到目前为止，我的努力没有成功。

使用MediaTranscoder（可能还有MediaStreamSource时，在编码时向流中添加仅基于每个样本的定时元数据的正确方法是什么？

Answer 1

与工程师沟通后，现在出现了此问题。

虽然MediaTranscoder确实支持定时元数据流，但是在MF Transform Engine上设置了配置文件后，该配置文件被删除，仅支持1个音频和1个视频流。因此，使用MediaTranscoder将永远无法使用。

相关文档已更新：Transcode media files