如何正确处理来自 Elevenlabs Streaming API 的流音频？

Question

嗨，我有一个 React js 前端应用程序。我使用 Elevenlabs 将文本转换为音频。我正在使用 Elevenlabs 的流式 API，这样我就不必等待整个音频，并且可以在收到它后立即开始播放。我使用 fetch 来调用 API 并使用 AudioContext 类处理传入的音频数据。

我的代码可以运行，但随机出现 3 个问题

1. 有时它根本不起作用

2. 有时它可以工作，但音频会在抛出以下错误之间停止： “未捕获（承诺中）DOMException：无法在‘BaseAudioContext’上执行‘decodeAudioData’：无法解码音频数据”

3. 有时它工作正常，但音频不流畅/无缝，流媒体时会出现一些剪切和暂停。

以下是我的代码

import {useEffect} from "react";


function App() {

    function getStreamAudio() {

        let textResponse='Hey, have you been keeping up with the latest in the crypto world? Its been incredible to see how much its grown over the past few years. I get where you are coming from, but I truly believe this is the future of finance. The decentralized nature of cryptocurrencies means no more relying on traditional banks and intermediaries. Its all about financial empowerment for the masses. You have got a point there, but remember, every technology has its challenges in the beginning. The scams and volatility will decrease as the industry matures. Plus, the potential for blockchain technology beyond just currency is immense – supply chain management, healthcare, even voting systems could benefit.'

        const streamingURL = "https://api.elevenlabs.io/v1/text-to-speech/MY-AUDIO-ID/stream?optimize_streaming_latency=3";

        const req = new Request(streamingURL, {
            method: 'POST',
            body: JSON.stringify({text: textResponse}),
            headers: {
                'Content-Type': 'application/json',
                'xi-api-key': 'MY-API-KEY',
            },
        });

        const audioContext = new AudioContext()

        fetch(req).then((resp: any) => {

            const audioContext = new AudioContext()
            let startTime = 0
            const reader = resp.body.getReader()
            const read = async () => {

                await reader.read().then(({done, value}: { done: any, value: any }) => {
                    if (done) {
                        console.log("THE STREAM HAS ENDED")
                        return
                    }

                    audioContext.decodeAudioData(value.buffer, (audioBuffer) => {
                        const source = audioContext.createBufferSource();
                        source.buffer = audioBuffer;
                        source.connect(audioContext.destination);

                        // Wait for the current audio part to finish playing
                        source.onended = () => {
                            console.log("Ended playing: " + Date.now())
                            read()
                        };

                        if (startTime == 0) {
                            startTime = audioContext.currentTime + 0.1 //adding 50ms latency to work well across all systems
                        }
                        source.start(audioContext.currentTime)
                        startTime = startTime + source.buffer.duration

                    });

                })
            }
            read()
        })

    }

    const playAudio = async () => {

        getStreamAudio()

    };

    return (
        <div>
            <h1>Streaming Test</h1>
            <button onClick={playAudio}>Click here to Play</button>
        </div>
    );

}

export default App;

如果能帮助我改进我的代码，我将不胜感激。谢谢你。

Answer 1

也许你可以使用 elevenlabs-js npm 包。它可能对您更有用。

你可以这样使用：

const elevenLabs = require('elevenlabs-js');

// Set your API key
elevenLabs.setApiKey('YOUR_API_KEY');

elevenLabs.textToSpeech("YOUR_VOICE_ID", "Hello World!", "elevenlabs_multilingual_v2", {
    stability: 0.95,
    similarity_boost: 0.75,
    style: 0.06,
    use_speaker_boost: true
}).then(async (res) => {
    const pipe = await res.pipe;
    console.log("pipe", pipe);
    // you can use pipe for what you want
});

如果您需要帮助，我很乐意提供帮助。