Spark Parallelized Collectios

Question

我对Spark非常陌生，我无法运行并行化集合，这是我的代码：

from pyspark import SparkContext as sc

words = [
    'Apache', 'Spark', 'is', 'an', 'open-source', 'cluster-computing',
    'framework', 'Apache', 'Spark', 'open-source', 'Spark'
]

# Creates a RDD from a list of words

distributed_words = sc.parallelize(words)
distributed_words.count()

我得到：

TypeError: parallelize() missing 1 required positional argument: 'c'
why?

Answer 1

您需要初始化spark Context，我们可以从Spark Session到Spark-2单词集合从parallelize中获取。

Example:

from pyspark.sql import SparkSession
spark=SparkSession.builder.appName("test").master("local").getOrCreate()
sc=spark.sparkContext
words = [
    'Apache', 'Spark', 'is', 'an', 'open-source', 'cluster-computing',
    'framework', 'Apache', 'Spark', 'open-source', 'Spark'
]
distributed_words = sc.parallelize(words)
distributed_words.count()
#11