Tracking down a memory leak involving Meteor and multiple DDP connections

107 views Asked by At

I have a network of multiple “nodes” running Meteor v1.5.4.2 (due to dependencies). Each of these nodes is supposed to be able to communicate with the others to fetch statistics etc. This is done using Meteors ddp-client server side on the nodes that should get information from the others.

This seemingly worked well, but when we started provoking it with a lot of changes in the network (meaning a lot of connections come and go) the memory gradually builds up until it freezes up and belly flops. I have limited experience resolving memory leaks, but by looking at heap snapshots, I found that there’s a buildup of objects called “Connection” (pasted below). Under strings I also find a lot of strings containing the certs and CRL’s used in the DDP connection, leading me to believe theres an issue with my code involving the connection handling. Ive tried to list the highlights below, removing a lot of minor logic involved.

I am at a little bit of a loss as to further approach this, so any suggestions, thoughts or ideas would be most welcome.

Thanks in advance.

Heres a compressed run down of how its connected

if(Meteor.isServer) {
    connectionHandler = new DDPConnectionHandler();
    Meteor.setInterval( () => connectionHandler.checkNodeConnections(), 5000);
}
export const DDPConnectionHandler = function() {
    this.connections = [];

    this.checkNodeConnections = () => {
        // Logic to add or remove the node connections in this.connections
        // Looping pr. node to handle
        const node = {...} // Details of the node to add/remove

        // Add new conncetion
        this.connections.push( new DDPConnection(node) );

        // Remove connection
        const index = currentConnections.indexOf(node.id);
        this.connections[index].disconnect();
        this.connections.splice(index, 1);
    };

}
export const DDPConnection = function(node) {
    let self = this;

    // setting up variables to use, pw, user, url ... etc. 

    this.connection = DDP.connect(url, { /* certs etc. for SSL */ });

    this.connection.call("login", {/* login details */}, (error, result) => {
        if( !error ) {
            // Wrap in timeout to space out the stats calls
            Meteor.setTimeout( () => { self.initNodeStats(); }, randomNumber );
        } else { /* No luck */ }
    });

    this.disconnect = () => {
        this.connection.disconnect(); // also tried with .close()
    };

    this.subscribe = (collection) => {
        // function to fetch other data
    };

    // Initialize and start recieving default basis ndoestats from current external nde
    this.initNodeStats = () => { this.getStats(); };

    this.getStats = () => {
        self.connection.call('getStats', {}, (error, result) => {
            if( error ) { /* No luck */
            } else if ( result ) { /* Pass data to handlers */ }
        });
    }

}
Connection
    _stream::ClientStream
    __proto__::Object
    _outstandingMethodBlocks::Array
    __flushBufferedWrites::()
    map::system / Map
    _methodInvokers::Object
    properties::(object properties)[]
    _bufferedWritesFlushAt::system / Oddball
    _bufferedWritesFlushHandle::system / Oddball
    _lastSessionId::system / Oddball
    _retryMigrate::system / Oddball
    _userId::system / Oddball
    _version::system / Oddball
    _versionSuggestion::system / Oddball
    onReconnect::system / Oddball
    _supportedDDPVersions::Array
    _userIdDeps::Tracker.Dependency
    _bufferedWrites::Object
    _documentsWrittenByStub::Object
    _methodHandlers::Object
    _methodsBlockingQuiescence::Object
    _serverDocuments::Object
    _stores::Object
    _subsBeingRevived::Object
    _subscriptions::Object
    _updatesForUnknownStores::Object
    _afterUpdateCallbacks::Array
    _messagesBufferedUntilQuiescence::Array
    _resetStores::system / Oddball

Update, after digging some more.

I seem to be getting a buildup of "methods" in the "_outstandingMethodBlocks" attribute on the Connection objects. Which is defined on line 129 here: https://github.com/meteor/meteor/blob/release-1.5.4.2/packages/ddp-client/livedata_connection.js#L129

Maybe theres some timeout setting I could use to stop them from being stored there?

0

There are 0 answers