我有一个DataFrame
+------+--------------------+-----------------+----
| id| titulo |tipo | formacion |
+------+--------------------+-----------------+----
|32084|A | Material | VION00001 TRADE |
|32350|B | Curso | CUS11222 LEADER|
|32362|C | Curso | ITIN9876 EVALUA|
|32347|D | Curso | CUMPLI VION1234 |
|32036|E | Curso | EVAN1111 INFORM|
我需要在formacion列中删除以VION | CUS | ITIN | VION | EVAN开头的字符,以便数据框看起来像]
+------+--------------------+-----------------+----
| id| titulo |tipo | formacion |
+------+--------------------+-----------------+----
|32084|A | Material | TRADE |
|32350|B | Curso | LEADER |
|32362|C | Curso | EVALUA |
|32347|D | Curso | CUMPLI |
|32036|E | Curso | INFORM |
+------+--------------------+-----------------+----
谢谢您的帮助
使用split
函数将列除以space
,然后获取数组的最后一个元素。
Spark2.4+
使用element_at
功能Spark < 2.4
使用reverse(split(array))[0]
#using element_at
df.withColumn("formacion",element_at(split(col("formacion"),"\\s"),-1)).show()
#or using array_index
df.withColumn("formacion",split(col("formacion"),"\\s")[1]).show()
#split reverse and get first index value
df.withColumn("formacion",reverse(split(col("formacion"),"\\s"))[0]).show()
#+-----+--------------+----------+-------------+
#| id|titulo |tipo | formacion |
#+------+--------------------+-----------------+
#|32084|A | Material | TRADE |
#|32350|B | Curso | LEADER |
#|32362|C | Curso | EVALUA |
#|32347|D | Curso | CUMPLI |
#|32036|E | Curso | INFORM |
#+-----+--------------+----------+-------------+