注: 这不是一个重复的 Sqoop - 语法错误 - 边界查询 - "你的SQL语法错误"
为了限制只取最近8天的数据，我使用了下面的方法。boundary-query 与 Sqoop
SELECT min(`created_at`),
max(`created_at`)
FROM `billing_db`.`billing_ledger`
WHERE `created_at` >= timestamp(date(convert_tz(now(), IF(@@global.time_zone = 'SYSTEM', @@system_time_zone, @@global.time_zone),'Asia/Kolkata')) + interval -2 DAY)"
为了便于阅读，我在这里把查询分成了多行，其实我只用单行传递给Sqoop。
边界查询的不同部分的解释为
IF(@@global.time_zone = 'SYSTEM', @@system_time_zone, @@global.time_zone)convert_tz(now(), <server-timezone>,'Asia/Kolkata')timestamp(date(<ist-timestamp> + interval -{num_days} DAY)虽然查询在MySQL上运行良好
mysql> SELECT min(`created_at`),
-> max(`created_at`)
-> FROM `billing_db`.`billing_ledger`
-> WHERE `created_at` >= timestamp(date(convert_tz(now(), IF(@@global.time_zone = 'SYSTEM', @@system_time_zone, @@global.time_zone),'Asia/Kolkata')) + interval -2 DAY);
+---------------------+---------------------+
| min(`created_at`) | max(`created_at`) |
+---------------------+---------------------+
| 2020-05-08 00:00:00 | 2020-05-10 20:12:32 |
+---------------------+---------------------+
1 row in set (0.02 sec)
在Sqoop上的以下堆栈跟踪中，它被打破了。
INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT min(), max() FROM . WHERE >= timestamp(date(convert_tz(now(), IF(@@global.time_zone = 'SYSTEM', @@system_time_zone, @@global.time_zone),'Asia/Kolkata')) + interval -2 DAY)
[2020-05-10 12:45:34,968] {ssh_utils.py:130} WARNING - 20/05/10 18:15:34 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/hadoop/.staging/job_1589114450995_0001
[2020-05-10 12:45:34,971] {ssh_utils.py:130} WARNING - 20/05/10 18:15:34 DEBUG util.ClassLoaderStack: Restoring classloader: sun.misc.Launcher$AppClassLoader@6ab7a896
[2020-05-10 12:45:34,973] {ssh_utils.py:130} WARNING - 20/05/10 18:15:34 ERROR tool.ImportTool: Import failed: java.io.IOException: java.sql.SQLSyntaxErrorException: (conn=313686) You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 12 near "), max() FROM . WHERE >= timestamp(date(convert_tz(now(), IF(@@global.time_zone = 'SYSTEM', @@system_time_zone, @@global.time_zone),'Asia/Kolkata')) + interval -2 DAY)"
at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:207)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:303)
郑重声明
使用 WHERE $CONDITIONS 需要 --query (自由形式的查询-导入)，但对于 --boundary-query 它不是强制性的。如果没有它，Sqoop只会生成以下警告信息
WARN db.DataDrivenDBInputFormat: Could not find $CONDITIONS token in query: SELECT min(), max() FROM . WHERE >= timestamp(date(convert_tz(now(), IF(@@global.time_zone = 'SYSTEM', @@system_time_zone, @@global.time_zone),'Asia/Kolkata')) + interval -2 DAY); splits may not partition data.
我一直在使用类似的复杂 boundary-query但在这个特殊情况下，它打破了我的管道中的其他地方。
我试过什么？
我试着添加 别名 在 SELECT 查询的子句是这样的
SELECT min(`created_at`) AS min_created_at,...