Sqoop坚持5%的进步

问题描述 投票:0回答:1

我正在使用Sqoop将数据从oracle导入HDFS。当Job开始时,它会在5%的进度中停留大约1小时,这个信息是输出:

INFO mapreduce.Job: Task Id : attempt_1535519556038_0015_m_000037_0, Status : FAILED
Container launch failed for container_1535519556038_0015_01_000043 : org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
This token is expired. current time is 1536133107764 found 1536133094775
Note: System times on machines may be out of sync. Check system time and time zones.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

然后继续,直到作业成功终止并导入所有数据。所以,我的问题是,在5%的进度中挂掉工作的原因是什么?为什么要自我纠正?这是正常的吗?如果没有,是否可以与发布的信息相关?我该如何解决这个问题?

hadoop import hdfs sqoop
1个回答
1
投票

错误消息清楚地解释了“未经授权的启动容器的请求。此令牌已过期“。

其中一个选项是通过设置来增加容器的使用寿命:yarn.resourcemanager.rm.container-allocation.expiry-interval-ms默认为10分钟。

注意:如果在yarn.resourcemanager.rm.container-allocation.expiry-interval-ms配置文件中增加yarn-site.xml,则作业将起作用。

<property>
   <name>yarn.resourcemanager.rm.container-allocation.expiry-interval-ms</name>
   <value>1000000</value>
</property>
© www.soinside.com 2019 - 2024. All rights reserved.