Module not found if graalpy is packaged in Spring Boot Jar

Question

I'm building a Spring Boot (3 RC1) application with some Python code (GraalVM 22.3). Running the application in dev mode works. After building Jar with Maven I get an error:

Caused by: org.graalvm.polyglot.PolyglotException: ModuleNotFoundError: No module named 'pystac'
    at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:399) ~[org.graalvm.sdk:na]
    at ch.so.agi.sodata.stac.ConfigService.readXml(ConfigService.java:116) ~[classes!/:0.0.1-SNAPSHOT]
    at ch.so.agi.sodata.stac.SodataStacApplication.lambda$init$0(SodataStacApplication.java:60) ~[classes!/:0.0.1-SNAPSHOT]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:767) ~[spring-boot-3.0.0-RC1.jar!/:3.0.0-RC1]
    ... 13 common frames omitted

The python.Executable shows to the graalpy executable packaged in the Jar: file:/Users/stefan/sources/datenbezug/sodata-stac/target/sodata-stac-0.0.1-SNAPSHOT.jar!/BOOT-INF/classes!/venv/bin/graalpy

Grepping the Jar shows the pystac module in the Jar file, e.g. BOOT-INF/classes/venv/lib/python3.8/site-packages/pystac/item.py

Creating the context with:

var VENV_EXECUTABLE = ConfigService.class.getClassLoader()
                .getResource(Paths.get("venv", "bin", "graalpy").toString())
                .getPath();

var context = Context.newBuilder("python")
                .allowAllAccess(true)
                .option("python.Executable", VENV_EXECUTABLE)
                .option("python.ForceImportSite", "true")
                .build()

Is it possible to put the whole python stuff including third party libs into the FatJar? Or did I just miss some packaging magic?

Answer 1

The issue is that the default Truffle filesystem, AFAIK, only supports the actual filesystem of your OS, i.e., it does not "see" resources in the jar file. This is why it works in dev mode when the resources are just files on the filesystem.

There are two options how to deal with this:

• do not put the venv into resources, but deploy it separately, such that it ends up somewhere on the filesystem
• implement custom Truffle filesystem, but I am not aware of anyone who tried this with virtual environments yet, so there may be some rough edges. Most notably GraalVM LLVM Toolchain that is used to build Python native extensions for execution with GraalVM LLVM engine produces system specific binaries, so your jar will not be as portable as one would expect from a jar. If you only use pure Python packages, it may be fine.

Update:

GraalPy now provides a command line tool to generate a skeleton of Maven based Java project that puts the venv into resources and implements custom Truffle filesystem that accesses the resources. Details: https://www.graalvm.org/latest/reference-manual/python/standalone-binaries/#embedding-graalpy-in-a-java-application

It is also planned to provide a Maven archetype that will do the same, and Maven plugin that will simplify Python packages handling in the pom.xml. I'll update the answer once that is released.