Storm 1.0.1提交拓扑的工作者错误

Question

在向Storm 1.0.1提交拓扑时，我收到以下错误（worker.log）（尽管出现错误，拓扑确实已提交并显示在Storm UI中）：

o.a.s.d.worker [ERROR] Error on initialization of server mk-worker java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) ~[?:1.8.0_91]
    at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_91]
    at org.apache.storm.timer$mk_timer.doInvoke(timer.clj:77) ~[storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.RestFn.invoke(RestFn.java:457) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$mk_halting_timer.invoke(worker.clj:244) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.daemon.worker$worker_data$fn__8190.invoke(worker.clj:293) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.util$assoc_apply_self.invoke(util.clj:930) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.daemon.worker$worker_data.invoke(worker.clj:268) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.daemon.worker$fn__8450$exec_fn__2461__auto__$reify__8452.run(worker.clj:611) ~[storm-core-1.0.1.jar:1.0.1]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_91]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_91]
    at org.apache.storm.daemon.worker$fn__8450$exec_fn__2461__auto____8451.invoke(worker.clj:609) ~[storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?]
    at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
    at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$fn__8450$mk_worker__8545.doInvoke(worker.clj:583) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$_main.invoke(worker.clj:771) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
    at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.0.1.jar:1.0.1]
2016-07-14 11:48:29.568 o.a.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
    at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$fn__8450$mk_worker__8545.doInvoke(worker.clj:583) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$_main.invoke(worker.clj:771) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
    at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.0.1.jar:1.0.1]

还在supervisor.log中找到了这个条目：

o.a.s.util [WARN] Worker Process ea92e1b7-c870-4b4c-b0a6-00272f27f521:# There is insufficient memory for the Java Runtime Environment to continue.

但是，“free -m”命令表明有足够的可用内存。

进一步的消息似乎表明工作人员无法启动，并且supervisor.log然后多次开始记录INFO消息“{id of worker process}仍未启动”。

Answer 1

从您的问题来看，似乎您在linux platform下运行风暴，因此在Linux中，操作系统将为安全问题提供upto N number of threads to a single process。如果您的java程序跨越N个线程，那么您将获得此类异常。为此，您需要增加在以下文件中配置的N值

/proc/sys/kernel/threads-max

---

要检索当前N值：

cat /proc/sys/kernel/threads-max

---

要将N值更新为新值：

echo 123456789 > /proc/sys/kernel/threads-max

希望能帮助到你。

Answer 2

我将kafka和storm分开，现在它们在不同的VM上运行。拓扑现在运行正常，没有任何错误。