Use a Python dependency in Logstash's Ruby filter plugin

Question

I'm trying to call some Python code from the Ruby filter plugin in Logstash. The problem I'm running into is caused by trying to use external dependencies in the Python code that's called by the Ruby script. As I expected, Ruby doesn't "see" those dependencies, but I've never used Ruby before, nor have I written a Logstash filter plugin, so please help. :)

Here's the logstash.conf file:

input {
    stdin {} 
filter {
    ruby {
        path => "<PATH_TO_LOGSTASH_FILTER_RUBY_SCRIPT>"
        script_params => { "python_file_path" => "<PATH_TO_PYTHON_FILE_THAT_I_WANT_TO_EXECUTE>" }
    }
    if [process_result] =~ /.+/ {
        json {
          source => "process_result"
          remove_field => [ "process_result" ]
        }
    }
}
output {
    stdout { codec => rubydebug }
}

Here's the Logstash Ruby filter script:

require "open3"

def register(params)
    @python_file_path = params["python_file_path"]
end


def filter(event)
    msg = event.get("message")
    cmd = "python3 #{@python_file_path} #{@msg}"    
    stdin, stdout, stderr = Open3.popen3(cmd)
    event.set("process_result", stdout.read)
    err = stderr.read
    if err.to_s.empty?
        filter_matched(event)
    else
        event.set("ext_script_err_msg", err)
    end
    return [event]
end

At the start of the Python file, I'm importing an external dependency. The rest of the Python file is of no relevance to this question.

When I run Logstash and pass some input, I get the following error:

Traceback (most recent call last):\n  File \"<PATH_TO_PYTHON_FILE_THAT_I_WANT_TO_EXECUTE>", line 1, 
in <module>\n    from <EXTERNAL_DEPENDENCY> import <SOMETHING>\nModuleNotFoundError: No module 
named '<EXTERNAL_DEPENDENCY>'\n",

I understand what the error is, I just don't know how to solve it. What should I do in order to make the external dependencies of my Python script usable in the Logstash Ruby plugin?

Answer 1

I ended up running the Python script as a Flask service and using Logstash's http filter plugin to send data to that service. Here's what the logstash.conf file looks like now:

input {
    stdin {} 
}
filter {
        http {
                url => "http://localhost:5000"
                body_format => "json"
                body => {"text" => "%{[message]}"}
                verb => POST
                target_body => api_result 
        }
}

output {
    stdout  { codec => rubydebug }
}

1. By default, body_format is text, so make sure to set it to json as I did if you're POST-ing.
2. For the fellow newbies out there, the message variable that is interpolated in the body => ... line refers to the message field of the current event.

In the Flask service, I accessed the data via request.json["text"].