如何从Jupyter Notebook中的PySpark远程连接到Greenplum数据库？

Question

我正在尝试通过JDBC连接将PySpark（与Jupyter Notebook）连接到Oracle VM VirtualBox上的Greenplum数据库实例，但是当我知道密码正确时，我收到以下错误：

Py4JJavaError: An error occurred while calling o424.load.
: org.postgresql.util.PSQLException: FATAL: password authentication failed 
for user "user2"

我试过了：

查看有关与PySpark连接的Greenplum DB文档

更改gp_hba.conf，sshd_conf和postgresql.conf文件中的Postgresql连接设置

使用pyspark shell并加载.jar文件

pyspark --jars 'path to .jar file'

然后运行下面提到的代码

Jupyter Notebook中的PySpark代码是：

import findspark
findspark.init('spark-2.4.1-bin-hadoop2.7')
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

option = {
    'url':"jdbc:postgresql://localhost:5432/tutorial",
    'user':"user2",
    'password':"SECRET",
    'dbschema':"faa",
    'dbtable':"otp_c",
    'partitionColumn':"airlineid"
}

gpdf = spark.read.format('greenplum').options(**option).load()

Pivotal Greenplum指示有一个连接器.jar文件，用于JDBC连接到数据库，我位于spark-2.4.1-bin-hadoop2.7 / jars / greenplum-spark_2.11-1.6.0.jar

此外，在Greenplum DB中，gp_hba.conf配置为：

# If you want to allow non-local connections, you need to add more
# "host" records.  In that case you will also need to make PostgreSQL
# listen on a non-local interface via the listen_addresses
# configuration parameter, or via the -i or -h command line switches.
# CAUTION: Configuring the system for local "trust" authentication allows
# any local user to connect as any PostgreSQL user, including the database
# superuser. If you do not trust all your local users, use another
# authentication method.
# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
# "local" is for Unix domain socket connections only
# IPv4 local connections:
# IPv6 local connections:
local    all         gpadmin         ident
host     all         gpadmin         127.0.0.1/28    trust
host     all         gpadmin         10.0.2.15/32       trust
host     all         gpadmin         ::1/128       trust
host     all         gpadmin   fe80::a00:27ff:fe84:1f3f/128      trust
local    replication gpadmin         ident
host     replication gpadmin         samenet       trust
local    gpperfmon         gpmon         md5
host     all         gpmon         127.0.0.1/28    md5
local    tutorial            +users     trust
host     tutorial            +users     trust
host all all 0.0.0.0/0 md5
#local all all md5
#local all user2 ident

sshd_config文件配置为

PasswordAuthentication yes

最后，配置了postgresql.conf文件

# - Connection Settings -

listen_addresses = '*'          # what IP address(es) to listen on;
                                    # comma-separated list of addresses;
                                    # defaults to '*', '*' = all
                                    # (change requires restart)

port=5432 ##port = 5432                         # sets the database 
listener port for
                                    # a Greenplum instance. The master and
                                    # each segment has its own port 
number.
# note: Port numbers for the Greenplum system must also be changed in the
# gp_configuration catalog. See the Greenplum Database Administrator Guide
# for instructions!
#
#

我期望连接到Greenplum DB并使用PySpark执行SQL查询，但是我收到了Py4JJavaError。

不确定存在哪些其他选项，理想情况下我想通过Jupyter笔记本连接请帮忙！

Answer 1

在pg_hba中，主机配置需要CIDR。这条线

主持教程+用户信任

不会生效。所以它落在最后一行并要求输入密码。

您可以在greenplum群集内创建一个带有密码的角色user2。