Add a three.js AudioAnalyser to an HLS Stream

Question

Is it possible to add a three.js AudioAnalyser (or WebAPI AnalyserNode) to an Hls audio stream? This is what I have so far and average frequency always returns 0.

        let audioContext = new (window.AudioContext || window.webkitAudioContext)();

        let streamAudio = document.querySelector('#player');

        if (Hls.isSupported()) {

            hls.loadSource('https://stream.revma.ihrhls.com/zc2593/hls.m3u8');
            hls.attachMedia(streamAudio);           

        }

        plyr.setup(streamAudio);

        listener = new THREE.AudioListener();
        listener.hasPlaybackControl = true;
        perspectiveCamera.add(listener);
        audio = new THREE.Audio(listener);

        analyser = new THREE.AudioAnalyser(audio, fftSize);
        let frequencyData = analyser.getAverageFrequency();

        console.log(frequencyData);

        //let streamAnalyser = audioContext.createAnalyser();

        //source = audioContext.createMediaStreamSource(streamAudio);
        //source.connect(audioContext.destination);

    <audio preload="none" id="player" hidden="hidden" controls crossorigin="anonymous" />
    <script src="https://cdn.plyr.io/1.8.2/plyr.js"></script>
    <script src="https://cdn.jsdelivr.net/hls.js/latest/hls.js"></script>

The commented code represents other methods I tried with no success. The audio stream works but I am unable to read any frequency data which is what I am hoping to achieve. Is this possible?

Answer 1

If you want to connect a particular AudioNode with another one it only works if they both belong to the same AudioContext. To achieve that you could either use AudioContext.setContext to change the AudioContext used by three.js or you could just re-use the internal AudioContext created by three.js.

const source = new MediaElementAudioSourceNode(
    listener.context,
    { mediaElement: streamAudio }
);

Please note that you need to use a MediaElementAudioSourceNode instead of a MediaStreamAudioSourceNode when connecting a media element.

You can then connect it to the input of the AudioListener like this:

source.connect(listener.getInput());

In case this doesn't work it might be worth double-checking that listener.context.state actually says the AudioContext is 'running'.

And last but not least you might run into a CORS issue in case the server which serves the audio assets doesn't allow the page to inspect the response. You can check this by replacing your source with an OscillatorNode. If that works without any problems but the HLS stream doesn't you probably need to change the server's CORS configuration.

const source = new OscillatorNode(listener.context);

source.connect(listener.getInput());
source.start();