Pyspark - Python 设置相同时区

Question

我正在阅读一些时区为 GMT-4 的镶木地板

def get_spark():
    spark = SparkSession.builder.getOrCreate()
    spark.conf.set("spark.sql.parquet.enableVectorizedReader", "false")
    spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInRead", "LEGACY")
    spark.conf.set("spark.sql.session.timeZone", "GMT-4")
    return spark

文件展示

base_so.where(base_so.ID_NUM_CLIENTE == 2273).show()

+--------------+-----------+----------------+------------------+-------------------+-------------------+-------------------+
|ID_NUM_CLIENTE|NUM_TRAMITE|COD_TIPO_1     |COD_TIPO_2        |      FECHA_TRAMITE|      FECHA_INGRESO|  FECHA_INICIO_PAGO|
+--------------+-----------+----------------+------------------+-------------------+-------------------+-------------------+
|          2273|     238171|              X|               NN |2005-10-25 00:00:00|2005-10-25 09:26:54|1995-05-03 00:00:00|
|          2273|     238171|              X|               NMP|2005-10-25 00:00:00|2005-10-25 09:26:54|1995-05-03 00:00:00|
+--------------+-----------+----------------+------------------+-------------------+-------------------+-------------------+

当我从测试中创建数据框时，它不会留下列的日期

        spark = get_spark()
        df_busqueda = spark.createDataFrame(
            data=[
                [Decimal(2273), Decimal(238171), "SO", datetime.strptime('2005-10-25 00:00:00', '%Y-%m-%d %H:%M:%S')],
            ],
            schema=StructType(
                [
                    StructField('ID_NUM_CLIENTE', DecimalType(), True),
                    StructField('NUM_TRAMITE', DecimalType(), True),
                    StructField('COD_TIPO_1', StringType(), True),
                    StructField('FECHA_TRAMITE', TimestampType(), True),

                ]
            ),
        )

+--------------+-----------+----------------+-------------------+
|ID_NUM_CLIENTE|NUM_TRAMITE|COD_TIPO_1      |      FECHA_TRAMITE|
+--------------+-----------+----------------+-------------------+
|          2273|     238171|              SO|2005-10-24 23:00:00|
+--------------+-----------+----------------+-------------------+

如何更好地配置以使镶木地板和创建的数据帧保持相同的时区？

Answer 1

您可以在 Spark 会话中设置

timezone

。

Example:

For spark > 3:

spark.sql("SET TIME ZONE 'America/New_York'").show()
//+--------------------------+----------------+
//|key                       |value           |
//+--------------------------+----------------+
//|spark.sql.session.timeZone|America/New_York|
//+--------------------------+----------------+


spark.sql("select current_timestamp()").show()
//+--------------------------+
//|current_timestamp()       |
//+--------------------------+
//|2021-08-25 16:23:16.096459|
//+--------------------------+

For spark < 3.0:

spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.sql("select current_timestamp()").show()
//+--------------------+
//| current_timestamp()|
//+--------------------+
//|2021-08-25 20:26:...|
//+--------------------+

Answer 2

就我而言，文件是通过 NIFI 上传的，我必须将引导程序修改为相同的时区

spark-defaults.conf

spark.driver.extraJavaOptions -Duser.timezone=America/Santiago
spark.executor.extraJavaOptions -Duser.timezone=America/Santiago

Answer 3

#Import packages

import os, time
from dateutil import tz

# Format timestamp with the following snippet
os.environ['TZ'] = 'GMT+4'

time.tzset()
time.strftime('%X %x %Z')