问题
I'm running a job on a PySpark cluster for the first time. It runs perfectly in standalone mode on the name node. However, when it runs in the cluster:
spark-submit --master yarn \
--deploy-mode client \
--driver-memory 6g \
--executor-memory 6g \
--executor-cores 2 \
--num-executors 10 \
nearest_neighbor.py
It begins complaining that numpy isn't installed:
from pyspark.ml.param.shared import *
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0024/container_1582692915671_0024_01_000002/pyspark.zip/pyspark/ml/param/__init__.py", line 26, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
However, that module is confirmed to be installed on all nodes of the cluster (using pip3 install numpy). I've also confirmed that the job is running in python3:
sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0)
If comment out the call in my script to 'import numpy as np', it still complains about numpy not being installed, so I began to suspect that something besides my script was trying to import the module improperly.
By commenting out code in the script, I discovered there are some functions I'm calling that must be trying to call a different version of numpy on the back end. Specifically, this call throws the error about numpy not being installed:
isNonZeroVector = udf(lambda x: x.numNonzeros() > 0, BooleanType())
trans_corpus_df = trans_corpus_df.select('id', 'features', \
isNonZeroVector('features').alias('non_zero'))
trans_corpus_df = trans_corpus_df.filter(trans_corpus_df.non_zero == True)
It appears that Vector has methods (in this case numNonzeroes()) that is based on Numpy, and somehow it's failing to find the numpy module.
I confirmed that Python2 doesn't have numpy installed:
Python 2.7.15+ (default, Oct 7 2019, 17:39:04)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named numpy
So I installed the Python2 version of numpy on all the nodes. This didn't change the error at all, unfortunately.
I added a line to check where numpy is (print(np.file)) and it gives this:
/home/ubuntu/.local/lib/python3.6/site-packages/numpy/__init__.py
The permissions all the way down allow Ubuntu to access that directory, so I don't think it's a permissions issue.
Any tips are greatly appreciated!
Full error report:
20/02/26 14:21:19 ERROR TaskSetManager: Task 0 in stage 6.0 failed 4 times; aborting job
Traceback (most recent call last):
File "/home/ubuntu/server/nearest_neighbor.py", line 243, in <module>
main(args)
File "/home/ubuntu/server/nearest_neighbor.py", line 209, in main
hash_model = mh.fit(trans_corpus_df)
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/base.py", line 132, in fit
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 295, in _fit
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 292, in _fit_java
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o160.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: Lost task 0.3 in stage 6.0 (TID 28, ip-172-31-5-228.ec2.internal, executor 3): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/worker.py", line 377, in main
process()
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/worker.py", line 372, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 345, in dump_stream
self.serializer.dump_stream(self._batched(iterator), stream)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 141, in dump_stream
for obj in iterator:
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 334, in _batched
for item in iterator:
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 147, in load_stream
yield self._read_with_length(stream)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 172, in _read_with_length
return self.loads(obj)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 580, in loads
return pickle.loads(obj, encoding=encoding)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 869, in _parse_datatype_json_string
return _parse_datatype_json_value(json.loads(json_string))
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 886, in _parse_datatype_json_value
return _all_complex_types[tpe].fromJson(json_value)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 577, in fromJson
return StructType([StructField.fromJson(f) for f in json["fields"]])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 577, in <listcomp>
return StructType([StructField.fromJson(f) for f in json["fields"]])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 434, in fromJson
_parse_datatype_json_value(json["type"]),
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 888, in _parse_datatype_json_value
return UserDefinedType.fromJson(json_value)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 736, in fromJson
m = __import__(pyModule, globals(), locals(), [pyClass])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/__init__.py", line 22, in <module>
from pyspark.ml.base import Estimator, Model, Transformer, UnaryTransformer
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/base.py", line 24, in <module>
from pyspark.ml.param.shared import *
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/param/__init__.py", line 26, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:456)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:410)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:365)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3389)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2550)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2550)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2550)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2557)
at org.apache.spark.ml.feature.LSH.fit(LSH.scala:328)
at org.apache.spark.ml.feature.LSH.fit(LSH.scala:304)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/worker.py", line 377, in main
process()
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/worker.py", line 372, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 345, in dump_stream
self.serializer.dump_stream(self._batched(iterator), stream)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 141, in dump_stream
for obj in iterator:
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 334, in _batched
for item in iterator:
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 147, in load_stream
yield self._read_with_length(stream)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 172, in _read_with_length
return self.loads(obj)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 580, in loads
return pickle.loads(obj, encoding=encoding)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 869, in _parse_datatype_json_string
return _parse_datatype_json_value(json.loads(json_string))
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 886, in _parse_datatype_json_value
return _all_complex_types[tpe].fromJson(json_value)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 577, in fromJson
return StructType([StructField.fromJson(f) for f in json["fields"]])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 577, in <listcomp>
return StructType([StructField.fromJson(f) for f in json["fields"]])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 434, in fromJson
_parse_datatype_json_value(json["type"]),
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 888, in _parse_datatype_json_value
return UserDefinedType.fromJson(json_value)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 736, in fromJson
m = __import__(pyModule, globals(), locals(), [pyClass])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/__init__.py", line 22, in <module>
from pyspark.ml.base import Estimator, Model, Transformer, UnaryTransformer
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/base.py", line 24, in <module>
from pyspark.ml.param.shared import *
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/param/__init__.py", line 26, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:456)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:410)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
回答1:
This ended up solving the problem. I ended up adding the following lines to .bashrc in all nodes of the cluster:
export PYTHONPATH=/usr/bin/python3:/home/ubuntu/.local/lib/python3.6/site-packages:$PYTHONPATH
export PYSPARK_PYTHON=/usr/bin/python3:/home/ubuntu/.local/lib/python3.6/site-packages:$PYSPARK_PYTHON
Where the first is the path to python3 and the second is the path to the installed packages.
来源：https://stackoverflow.com/questions/60416082/pyspark-numpy-not-found-in-cluster-mode-modulenotfounderror