Catch子句不使用hive_context.read.json函数

Question

我有这个PySpark脚本：

from pyspark.sql import HiveContext
from pyspark.sql import SQLContext
from pyspark import SparkContext

sc = SparkContext.getOrCreate()
hive_context = HiveContext(sc)

sc.addFile("hdfs:///user/cloudera/2904/src/LogFunction.py")
import LogFunction

try:
    df = hive_context.read.json("/user/cloudera/Projet/pareeam.json")
except IOError:
    LogFunction.WarnLog("Nope")
....
....

这是我的LogFunction.py：

import logging
from logging.handlers import RotatingFileHandler
from pyspark.sql import HiveContext
from pyspark.sql import SQLContext
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
hive_context = HiveContext(sc)

df = hive_context.read.json("/user/cloudera/2904/param.json")
Path = df[df.column.isin("LogRep")].collect()[0][1]
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s :: %(levelname)s :: %(message)s')

debug_handler = RotatingFileHandler(Path+ '/LogDebug.log', 'a', 1000000, 1)
debug_handler.setLevel(logging.DEBUG)
debug_handler.setFormatter(formatter)
logger.addHandler(debug_handler)
info_handler = RotatingFileHandler(Path+ '/LogInfo.log', 'a', 1000000, 1)
info_handler.setLevel(logging.INFO)
info_handler.setFormatter(formatter)
logger.addHandler(info_handler)
warning_handler = RotatingFileHandler(Path+ '/LogWarning.log', 'a', 1000000, 1)
warning_handler.setLevel(logging.WARNING)
warning_handler.setFormatter(formatter)
logger.addHandler(warning_handler)

error_handler = RotatingFileHandler(Path+ '/LogError.log', 'a', 1000000, 1)
error_handler.setLevel(logging.ERROR)
error_handler.setFormatter(formatter)
logger.addHandler(error_handler)

def WarnLog(a):
    logger.warning(a)

无论如何，我确信我的日志记录正在运行，因为我已经尝试了其他错误（//除以零异常）。

但似乎不是这个例子。在执行脚本时，我得到了一个

py4j.protocol.Py4JJavaError: An error occurred while calling o31.json.
: java.io.FileNotFoundException: File hdfs://quickstart.cloudera:8020/user/cloudera/Projet/pareeam.json does not exist.

在终端上但我的警告文件中没有任何内容。任何帮助为什么不捕捉错误？谢谢

Answer 1

您正在捕捉错误的异常。你的代码捕获IOError而DataFrameReader.json抛出py4j.protocol.Py4JJavaError（内部）和pyspark.sql.utils.AnalysisException

将您的代码更改为

from pyspark.sql.utils import AnalysisException

try:
    df = hive_context.read.json("/does/not/exist")
except AnalysisException:
    LogFunction.WarnLog("Nope")