What are compatible versions of pyspark and py4j packages in python

Question

I am trying to setup pyspark locally

1. I've initiated a spark session
2. created a view named people
3. tried to read the view via below command

spark.sql("Select * From people")

It throws the Py4j exception below; it seems like a version mismatch error:

Py4JError: An error occurred while calling o24.sql.
Trace:
py4j.Py4JException: Method sql([class java.lang.String, class [Ljava.lang.Object;]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
at py4j.Gateway.invoke(Gateway.java:274)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:1623)

1. Spark version: 3.3.4
2. Java version: 20.0.2
3. Python: 3.11.5
4. Pyspark: 3.5.0
5. Py4j: 0.10.9.7

I've tried different versions of Pyspark and Py4j for compatibility but they didn't work.

Answer 1

I suggest you to try the approach in this question: Error : py4j.Py4JException: Method sql([class java.lang.String, class [Ljava.lang.Object;]) does not exist

The root cause of your error seems a mismatch between spark and pyspark.

I would do the following:

1. Install the same version of pyspark and spark:

python -m pip install pyspark==3.3.4

2. Try your query again

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("example").getOrCreate()
# Your code for creating the 'people' view or other operations

# Now try your SQL query again
spark.sql("SELECT * FROM people").show()