How to play raw bytes audio in pyglet?

Question

I recieve audio buffers from another program and want to play it in pyglet. The buffer is a numpy array (dtype=int16, shape=(512, 2)). I tried writing a custom source based on a look at the other source implementations.

import pyglet
from pyglet.media.codecs.base import AudioFormat, AudioData

class BufferSource(pyglet.media.Source):
    def __init__(self) -> None:
        self.audio_format = AudioFormat(channels=2, sample_size=16, sample_rate=44100)

        self._data = np.zeros((512, 2), dtype=np.int16)

    def set_audio_data(self, data: np.ndarray):
        self._data = data

    def get_audio_data(self, num_bytes, compensation_time=0):
        data = self._data.tobytes()
        duration = float(len(data)) / self.audio_format.bytes_per_second
        return AudioData(data, len(data), 0, duration, [])


window = pyglet.window.Window(100, 100)
source = BufferSource()

...

@window.event
def on_draw():
    # do draw things

    source.set_audio_data(otherprogram.get_audio_buffer())

source.play()
pyglet.app.run()

This works but the audio is somewhat laggy and not playing continuous. Does anyone have an idea how to improve this? I guess its because 'num_bytes' if greater than 2048 (512 * 2 bytes * 2 channels) but I have no control over how large the buffer is.