使用 Meltano SDK 对 SQL 源的歌手点击开发进行调试和故障排除

Question

有人有使用 Meltano SDK 开发 SQL 歌手水龙头的经验吗？

问题

我最大的问题是解决测试水龙头时遇到的异常。

背景

• 我正在使用 WSL2 在 Windows 计算机上运行的 Ubuntu 环境中进行开发。
• 我已经使用在 ubuntu 计算机上为 Progress Openedge（查看此处）找到的自定义 sqlalchemy 方言成功建立了与源的连接，并使用以下代码创建了 url：

pyodbc_connstr = urllib.parse.quote_plus(r"""
    DRIVER={DataDirect 7.1 OpenEdge Wire Protocol};
    HOST=xxxx;
    PORT=xxxx;
    DB=xxxxx;
    UID=sysprogress;
    PWD=;
    DEFAULTSCHEMA=PUB;
""")                                         

sa_str = "progress+pyodbc:///?odbc_connect={}".format(pyodbc_connstr)

• 我指的是这篇 Meltano 文章 作为设置项目的指南。
• 我已经根据千篇一律的提示设置了以下响应：
  • 流类型 = 3 - SQL
  • 身份验证类型 = 6 - 自定义或不适用
• 我已使用

  poetry add <local path to package>

从 Ubuntu 计算机上的本地目录将 Progress_sa SQLAlchemy 方言安装到项目中
• 我创建了一个 .env 文件来测试水龙头。
• SDK 的所有开发人员 TODO 区域均已完成。

我被困住的地方

运行时

 poetry run tap-<my tap> --config= ENV

我收到以下异常：

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 493, in invoke
    tap = cls(
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 625, in __init__
    super().__init__(*args, **kwargs)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 102, in __init__
    self.setup_mapper()
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 172, in setup_mapper
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 165, in catalog
    self._catalog = self.input_catalog or self._singer_catalog
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 318, in _singer_catalog
    for stream in self.streams.values()
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 127, in streams
    for stream in self.load_streams():
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 351, in load_streams
    for stream in self.discover_streams():
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 655, in discover_streams
    for catalog_entry in self.catalog_dict["streams"]:
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 643, in catalog_dict
    result["streams"].extend(connector.discover_catalog_entries())
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/connectors/sql.py", line 497, in discover_catalog_entries
    engine = self._engine
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/connectors/sql.py", line 305, in _engine
    self._cached_engine = self.create_engine()
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/connectors/sql.py", line 322, in create_engine
    return sqlalchemy.create_engine(
  File "<string>", line 2, in create_engine
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 680, in create_engine
    raise TypeError(
TypeError: Invalid argument(s) 'json_serializer','json_deserializer' sent to create_engine(), using configuration Progress_pyodbc/QueuePool/Engine.  Please check that the keyword arguments are appropriate for this combination of components. 

我承认我仍然是一个正在开发的Python程序员，所以我仍在学习如何有效地排除故障。

很难判断问题是否是由我在设置水龙头时所做的操作引起的。

任何帮助或建议将不胜感激。

Answer 1

Singer SDK 的 v0.32.0 版本已解决此问题。特别是https://github.com/meltano/sdk/pull/1949。