Have been trying to set it up for hours now. Nothing works.
- Latest version does not seem to have winutils support, and using it causes errors when using some important methods. (EDIT: this is likely wrong, and the winutils stuff that I have should probably be fine.)
- Older versions require to be built with Maven. However, that just gives me a
PluginExecutionException
.
I need to do this ASAP, preferably within the next 3 hours.
I have nowhere else to ask for help, it seems, especially considering that suspended an account I set up specifically for asking questions after I edited a relevant post.
Highly doubt that anybody will be able to help me.
EDIT2: the issue has, thankfully, been resolved. I was using Python 3.12, and switched to 3.11.8. That made the problem go away.
I have never used PySpark, but I do know some about python.
How are you installing pyspark? Are there any errors?
pip install pyspark
and installing the latest version of Apache Spark leads to errors when callingpyspark.sql.DataFrame.show()
methods ofDataFrame
objects.pip install pyspark
and installing an older version of Apache Spark, i.e. having a version mismatch between PySpark and Apache Spark, leads to errors even when instantiating aSparkSession
.pip install pyspark==3.3.4
previously led to an error - the system was unable to build wheels for the package. Now, it seems to install that way, but behaves the same as in the previous case../build/mvn
using Bash from the appropriate directory led toCaused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plu gin:4.4.0:compile failed.
Running this code after having installed this stuff as in case 3:
leads to this:
System environmental variables
JAVA_HOME
,HADOOP_HOME
,SPARK_HOME
are configured. The relevant binary directories are included in thePath
system environmental variable.PYTHON_SPARK
is set topython
.EDIT: Great, and now Maven can't even attempt to build the package and throws the error
Just great.
Just in case, if I install the library the first way, for the same piece of code the logs start with this:
One stackoverflow thread mentions running python 3.12. Are you running 3.12? Does it help if you use python 3.11?
Else, an interesting thing might be the "Caused by: java.io.EOFException", an End Of File Exception.
That actually worked. Thank you.
I have got to say that I have a special hatred for arcane programming errors like in this case.
I am running 3.12. I have not tried running 3.11. Highly doubt that that will change anything, but I guess I'll try it when I'm able to.
I'm not sure what I have to glean from the EOF exception.