pip install different version

Question

I'm attempting to install pypandoc version 1.4 or higher. I included this requirement in my requirements.txt file. I believe that this is the reason why PySpark installation is failing.

my pip install command:

 python -m pip install -r requirements.txt --ignore-installed --exists-action=w --disable-pip-version-check --no-warn-script-location

the output:

Answer 1

In your screenshot, you're installing pypandoc 1.11 (which is newer than 1.1 since version components are compared separately, and 11 > 1).

However, the issue here is that pypandoc hasn't been installed, it was just downloaded; if building pyspark requires it, it needs to have been separately installed first. (It should be declared a build dependency for pyspark, but pyspark 2.4.5 is 3 years old, so I doubt they're going to fix that).

In other words, first separately install pypandoc>1.4, then try again; at that point pypandoc should be importable by pyspark's setup.py.

If that still fails with the same symptom, you may need to add --no-build-isolation.

Answer 2

Look in the requirements.txt file and see if what the pypandoc version it specifies in there. To ensure that pip installs the correct version of pypandoc, make sure your requirements.txt file contains the version constraint:

pypandoc>=1.4

If you still encounter issues, you can try installing pypandoc directly using the following command:

python -m pip install "pypandoc>=1.4"

If that works, you can try to install the other dependencies in your requirements.txt file using the original pip install command you provided.

If you still get the same problem you could try upgrading pip and setuptools using:

python -m pip install --upgrade pip setuptools

... then try installing the dependencies again.