我有一个pyspark数据帧,我试图使用toPandas()将其转换为pandas,但是我遇到了下面提到的错误。
我尝试了不同的选项,但得到了同样的错误: 1)将数据限制为几个记录 2)明确使用了collect()(我相信toPandas()本身使用)
在SO上探索了很多帖子,但AFAIK没有toPandas()问题。
我的数据框的快照: -
>>sc.version
2.3.0.2.6.5.0-292
>>print(type(df4),len(df4.columns),df4.count(),
(<class 'pyspark.sql.dataframe.DataFrame'>, 13, 296327)
>>df4.printSchema()
root
|-- id: string (nullable = true)
|-- gender: string (nullable = true)
|-- race: string (nullable = true)
|-- age: double (nullable = true)
|-- status: integer (nullable = true)
|-- height: decimal(6,2) (nullable = true)
|-- city: string (nullable = true)
|-- county: string (nullable = true)
|-- zipcode: string (nullable = true)
|-- health: double (nullable = true)
|-- physical_inactivity: double (nullable = true)
|-- exercise: double (nullable = true)
|-- weight: double (nullable = true)
>>df4.limit(2).show()
+------+------+------+----+-------+-------+---------+-------+-------+------+-------------------+--------+------------+
|id |gender|race |age |status |height | city |county |zipcode|health|physical_inactivity|exercise|weight |
+------+------+------+----+-------+-------+---------+-------+-------+------+-------------------+--------+------------+
| 90001| MALE| WHITE|61.0| 0| 70.51|DALEADALE|FIELD | 29671| null| 29.0| 49.0| 162.0|
| 90005| MALE| WHITE|82.0| 0| 71.00|DALEBDALE|FIELD | 36658| 16.0| null| 49.0| 195.0|
+------+------+------+----+-------+-------+---------+-------+-------+------+-------------------+--------+------------+
*had to mask few features due to data privacy concerns
错误:-
>>df4.limit(10).toPandas()
'int' object is not iterable
Traceback (most recent call last):
File "/repo/python2libs/pyspark/sql/dataframe.py", line 1968, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/repo/python2libs/pyspark/sql/dataframe.py", line 467, in collect
return list(_load_from_socket(sock_info, BatchedSerializer(PickleSerializer())))
File "/repo/python2libs/pyspark/rdd.py", line 142, in _load_from_socket
port, auth_secret = sock_info
TypeError: 'int' object is not iterable
我们的库的自定义存储库有一个用于pyspark的包,它与由spark集群提供的pyspark冲突,并且在某种程度上两者都在Spark shell上运行但在笔记本上不起作用。 因此,重命名自定义存储库中的pyspark库解决了这个问题!