Swift : buffer a very long argument to `Process` with pipe

Question

The problem

I'm trying to call a Python function from my Swift executable.

This worked perfectly until the arguments became too long and I started getting the error:

uncaught exception NSInternalInconsistencyException
reason: 'Couldn't posix_spawn: error 7'

which essentially means the data I'm sending the python code for processing (my argument) is too long.

A last resort alternative

The most obvious solution is writing the data to a file and sending the script a path to that file containing the data. This comes with obvious performance limitations.

A potentially better alternative

This is what I need help with: I thought it might be possible to buffer this data rather than sending it all in one chunk.

I've tried researching this, with many different combinations of keywords and have been reading many articles online for the past day but none answer my question: Is it possible to buffer the data to get around the kernel-impose ARG_MAX limitation of around 260000 bytes using Processes and Pipes?

The code

Because it can always help to see the code I'm currently using for to call the Python functions, this is what I have:

@discardableResult
public static func shell(_ args: String...) -> String {

    let task = Process()
    task.launchPath = "/usr/bin/env"
    task.arguments = args
    let pipe = Pipe()
    task.standardOutput = pipe
    task.launch()
    task.waitUntilExit()

    // return task.terminationStatus
    let data = pipe.fileHandleForReading.readDataToEndOfFile()
    let output: String = String(data: data, encoding: String.Encoding.utf8)!

    return output
}   
//  public static func shell(_ args: String...) -> String {}

And a little later:

//
//  Call classifier on data
//
let pyStcriptPath = "\(Bundle.main.bundlePath)/Classification/****Classify.py"

//
// The fonction I'm calling here, 'shell', is the one defined just above in my question
//
let pyResult = shell("python", pyStcriptPath, "\(featureVector)", "\(signalType)")

Answer 1

You could send your big payload using a pipe to the script's standard input instead. Assuming, of course, only a single argument is causing your issue (i.e., everything else you can still keep sending using regular command line arguments).

For instance:

func shell(stdin input: String, _ args: String...) -> String {
    let task = Process()
    task.launchPath = "/usr/bin/env"
    task.arguments = args

    let inPipe = Pipe()
    task.standardInput = inPipe
    inPipe.fileHandleForWriting.write(input.data(using: .utf8)!)
    inPipe.fileHandleForWriting.closeFile()

    let outPipe = Pipe()
    task.standardOutput = outPipe

    task.launch()
    task.waitUntilExit()

    let data = outPipe.fileHandleForReading.readDataToEndOfFile()
    let output = String(data: data, encoding: .utf8)!

    return output
}

A quick test:

func bigString() -> String {
    var str = ""
    for i in 0..<50_000 {
        str.append("\(i % 10)")
    }
    return str
}

let script = "/PATH/TO/SCRIPT/echo.py"
let input = bigString()
let result = shell(stdin: input, "python", script, "some arg")
print(result)

prints:

ARG: ABC
STDIN: 012345678901234567890123...

where the echo.py script is just:

import sys
print "ARG:", sys.argv[1]
print "STDIN:", sys.stdin.read()