This page lists issues people have run into when using Storm along with their solutions.
Possible symptoms:
Solutions:
Possible symptoms:
Solutions:
-Djava.net.preferIPv4Stack=true
to the supervisor child options and restarting the supervisor. Worker processes are crashing on startup with no stack trace
Symptoms:
Solutions:
Symptoms:
Solutions:
Symptoms:
Solution:
Symptoms:
Solution:
Symptoms:
java.lang.RuntimeException: java.util.ConcurrentModificationException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55)
at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56)
at org.apache.storm.disruptor$consume_loop_STAR_$fn__1597.invoke(disruptor.clj:67)
at org.apache.storm.util$async_loop$fn__465.invoke(util.clj:377)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:390)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:409)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:408)
at java.util.HashMap.writeObject(HashMap.java:1016)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:959)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1480)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
at org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
Solution:
Symptoms:
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55)
at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56)
at org.apache.storm.disruptor$consume_loop_STAR_$fn__1596.invoke(disruptor.clj:67)
at org.apache.storm.util$async_loop$fn__465.invoke(util.clj:377)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:24)
at org.apache.storm.daemon.worker$mk_transfer_fn$fn__4126$fn__4130.invoke(worker.clj:99)
at org.apache.storm.util$fast_list_map.invoke(util.clj:771)
at org.apache.storm.daemon.worker$mk_transfer_fn$fn__4126.invoke(worker.clj:99)
at org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3904.invoke(executor.clj:205)
at org.apache.storm.disruptor$clojure_handler$reify__1584.onEvent(disruptor.clj:43)
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:81)
... 6 more
or
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
~[storm-core-0.9.3.jar:0.9.3]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
~[storm-core-0.9.3.jar:0.9.3]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
~[storm-core-0.9.3.jar:0.9.3]
at
org.apache.storm.disruptor$consume_loop_STAR_$fn__759.invoke(disruptor.clj:94)
~[storm-core-0.9.3.jar:0.9.3]
at org.apache.storm.util$async_loop$fn__458.invoke(util.clj:463)
~[storm-core-0.9.3.jar:0.9.3]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
Caused by: java.lang.NullPointerException: null
at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na]
at
org.apache.storm.daemon.worker$mk_transfer_fn$fn__3548.invoke(worker.clj:129)
~[storm-core-0.9.3.jar:0.9.3]
at
org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3282.invoke(executor.clj:258)
~[storm-core-0.9.3.jar:0.9.3]
at
org.apache.storm.disruptor$clojure_handler$reify__746.onEvent(disruptor.clj:58)
~[storm-core-0.9.3.jar:0.9.3]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.9.3.jar:0.9.3]
... 6 common frames omitted
Solution:
OutputCollector
. All emits, acks, and fails must happen on the same thread. One subtle way this can happen is if you make a IBasicBolt
that emits on a separate thread. IBasicBolt
's automatically ack after execute is called, so this would cause multiple threads to use the OutputCollector
leading to this exception. When using a basic bolt, all emits must happen in the same thread that runs execute
.