文档中的简单 UDF 应用函数在 Spark 3.3 中失败

Question

来自最新文档的这个简单代码不适用于 EMR Studio Spark 集群（当前版本：

3.3.1-amzn-0

）

df = spark.createDataFrame(
    [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],
    ("id", "v"))

def subtract_mean(pdf: pd.DataFrame) -> pd.DataFrame:
    # pdf is a pandas.DataFrame
    v = pdf.v
    return pdf.assign(v=v - v.mean())

df.groupby("id").applyInPandas(subtract_mean, schema="id long, v double").show()

错误看起来像这样：

An error was encountered:
An error occurred while calling o184.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 7.0 failed 4 times, most recent failure: Lost task 0.3 in stage 7.0 (TID 59) (ip-10-130-55-119.us-east-1.aws.thousandeyes.com executor 7): java.lang.RuntimeException: Failed to run command: /usr/bin/virtualenv -p python3 --no-pip --system-site-packages virtualenv_application_1693557403809_0024_0
    at org.apache.spark.api.python.VirtualEnvFactory.execCommand(VirtualEnvFactory.scala:125)
    at org.apache.spark.api.python.VirtualEnvFactory.setupVirtualEnv(VirtualEnvFactory.scala:83)
    at org.apache.spark.api.python.PythonWorkerFactory.<init>(PythonWorkerFactory.scala:95)

我确信这是 Python 包版本的问题，因为另一个用户在 Spark 的早期版本中也遇到了类似的问题（参见此处）。然而我没有成功找到合适的 pandas/pyarrow 版本来使用......

Answer 1

解决方案是向 AWS Support 开具票证，他们解决了该问题。解决方案的一部分是在第一个笔记本单元中使用它：

%%configure -f
{
    "conf": {
        "spark.pyspark.python":"python3",
        "spark.pyspark.virtualenv.enabled": "true",
        "spark.pyspark.virtualenv.type": "native", 
        "spark.pyspark.virtualenv.bin.path": "/usr/local/bin/virtualenv"
        
    }
}