Memory leak when dispatching messages with Symfony Messenger component

306 views Asked by At

When dispatching a lot of messages (tens of millions) the memory limit keeps increasing and I do not know why.

Here's the code that dispatches the messages. I took out all the application logic to keep it simple:

$total = 10000000;
$iterations = \ceil($total / 10000);

for ($i = 0; $i <= $iterations; ++$i) {
    $start = \microtime(true);
    $ids = \range(0, 10000);

    foreach ($ids as $id) {
        $this->messageBus->dispatch(new Message((int) $id));
    }

    $end = \microtime(true) - $start;
    $memory = memory_get_usage() / 1024 / 1024;
    $output->writeln("Finished iteration $i in {$end}s. Memory usage: {$memory}MB");
}

This results in the following:

Finished iteration 0 in 0.88457703590393s. Memory usage: 65.843116760254MB
Finished iteration 1 in 0.8521249294281s. Memory usage: 75.336936950684MB
Finished iteration 2 in 0.8602819442749s. Memory usage: 81.830673217773MB
Finished iteration 3 in 0.87028503417969s. Memory usage: 94.406578063965MB
Finished iteration 4 in 0.86899781227112s. Memory usage: 101.90029144287MB
Finished iteration 5 in 0.85589599609375s. Memory usage: 107.39400482178MB
Finished iteration 6 in 0.86256003379822s. Memory usage: 126.88771820068MB
...etc...

Things to keep in mind:

  • The Message contains nothing but a id (int).
  • The problem is when dispatching the messages, so the handling the messages is not the issue.
  • I disabled the TraceableMessageBus, with it the same issue occurs (but with more memory usage)
  • I'm using RabbitMQ transport
  • I'm using the default message bus configuration
  • I tried disabling all the default middleware, except send_message. In that case the issue still occurs, but memory increases with 3MB instead of 10MB (with every iteration). See config used to disable middleware:
    framework:
        messenger:
            buses:
                messenger.bus.default:
                    default_middleware: false
                    middleware:
                        - messenger.middleware.send_message
    
  • There's a known 'gotcha' where the build up of logs might be the issue. I (quick and dirty) disabled $this->logger in Symfony\Component\Messenger\Middleware\SendMessageMiddleware but to no avail: memory leak remains

What is causing this? And if this is considered 'acceptable overhead' what would be best practices to deal with dispatching many messages in bulk so that the server doesn't crash?

0

There are 0 answers