我的目标是通过Cloud Data Fusion管道将数据从Cloud SQL Postgres读取到BigQuery。
为此,我设置了一个Cloud Data Fusion实例,并为服务帐户分配了以下两个权限:(请参阅https://cloud.google.com/data-fusion/docs/how-to/create-instance#setting_up_permissions)
下一步,我将自己连接到云数据融合实例,并导航到Wrangler -> Add Connection -> Database -> Google Cloud SQL for PostgreSQL
。
作为驱动程序,我上传了我在这里下载的postgres-socket-factory-1.0.13-jar-with-dependencies.jar
:https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory
对于驱动程序配置,我设置:
对于数据库连接,我设置:
<PROJECT_NAME>:<REGION>:<INSTANCE_CONNECTION_NAME>
jdbc:postgresql://google/<DATABASE_NAME>?cloudSqlInstance=<INSTANCE_CONNECTION_NAME>&socketFactory=com.google.cloud.sql.postgres.SocketFactory
单击Test Connection后,我收到org.postgresql.Driver错误消息。
您很可能需要一个驱动程序和JDBC Socket Factory才能使其正常工作。您应该将postgres连接器与套接字工厂捆绑到一个超级jar中,看看是否能解决问题。
这是一个应该完成此任务的快速(未经测试的)pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>postgres-with-cloud-sql-socket-factory</artifactId>
<packaging>pom</packaging>
<version>0.0.1</version>
<dependencies>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.2.5</version>
</dependency>
<dependency>
<groupId>com.google.cloud.sql</groupId>
<artifactId>postgres-socket-factory</artifactId>
<version>1.0.13</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<!-- get all project dependencies -->
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<!-- bind to the packaging phase -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>