KafkaSpoutの作業例
私は最近、Apache Kafkaに精通し、プロデューサー-コンシューマーの実用的な例を持っています。
私の次のステップは、KafkaをSpoutとBoltと統合することです。そして、利用可能な例（ほとんどが古い）をローカルで機能させるのに苦労しています。
ローカルテキストファイルからデータを読み取る次の例の作業storm-book/examples-ch02-getting_startedを取得しました。
同じリポジトリにstorm-book/examples-ch04-spouts kafka-spout の例がありますが、動作させることができません。
次の例も試しました cep.kafka しかし、次のエラーが発生しました-
5034 [Thread-11-words] INFO org.Apache.curator.framework.imps.CuratorFrameworkImpl - Starting
5047 [Thread-11-words] ERROR backtype.storm.util - Async loop died!
Java.lang.NoSuchMethodError: org.Apache.zookeeper.ZooKeeper.<init>(Ljava/lang/String;ILorg/Apache/zookeeper/Watcher;Z)V
at org.Apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.Java:29) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.Java:169) ~[curator-framework-2.4.0.jar:na]
at org.Apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.Java:94) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.HandleHolder.getZooKeeper(HandleHolder.Java:55) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.ConnectionState.reset(ConnectionState.Java:219) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.ConnectionState.start(ConnectionState.Java:103) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.Java:188) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.Java:234) ~[curator-framework-2.4.0.jar:na]
at storm.kafka.ZkState.<init>(ZkState.Java:62) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at storm.kafka.KafkaSpout.open(KafkaSpout.Java:85) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$fn__3371$fn__3386.invoke(executor.clj:522) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) ~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.AFn.run(AFn.Java:24) [clojure-1.5.1.jar:na]
at Java.lang.Thread.run(Thread.Java:745) [na:1.8.0_05]
5049 [Thread-11-words] ERROR backtype.storm.daemon.executor -
Java.lang.NoSuchMethodError: org.Apache.zookeeper.ZooKeeper.<init>(Ljava/lang/String;ILorg/Apache/zookeeper/Watcher;Z)V
at org.Apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.Java:29) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.Java:169) ~[curator-framework-2.4.0.jar:na]
at org.Apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.Java:94) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.HandleHolder.getZooKeeper(HandleHolder.Java:55) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.ConnectionState.reset(ConnectionState.Java:219) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.ConnectionState.start(ConnectionState.Java:103) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.Java:188) ~[curator-client-2.4.0.jar:na]
at org.Apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.Java:234) ~[curator-framework-2.4.0.jar:na]
at storm.kafka.ZkState.<init>(ZkState.Java:62) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at storm.kafka.KafkaSpout.open(KafkaSpout.Java:85) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$fn__3371$fn__3386.invoke(executor.clj:522) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) ~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.AFn.run(AFn.Java:24) [clojure-1.5.1.jar:na]
at Java.lang.Thread.run(Thread.Java:745) [na:1.8.0_05]
5088 [Thread-11-words] ERROR backtype.storm.util - Halting process: ("Worker died")
Java.lang.RuntimeException: ("Worker died")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.4.jar:0.9.4]
at clojure.lang.RestFn.invoke(RestFn.Java:423) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__4693$fn__4694.invoke(worker.clj:491) [storm-core-0.9.4.jar:0.9.4]
at backtype.storm.daemon.executor$mk_executor_data$fn__3272$fn__3273.invoke(executor.clj:240) [storm-core-0.9.4.jar:0.9.4]
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:473) [storm-core-0.9.4.jar:0.9.4]
at clojure.lang.AFn.run(AFn.Java:24) [clojure-1.5.1.jar:na]
at Java.lang.Thread.run(Thread.Java:745) [na:1.8.0_05]
作成を実行してKafka Spout）を実行する方法を学習するのと同じ問題があったときに、これ Githubリポジトリ を見つけました非常に便利で、KafkaSpoutから残りのボルトにタプルを放出させることができました。
これは、このためのトポロジを作成する方法の概要です。
public class TestTopology {
public static void main(String[] args) {
String zkIp = "192.168.59.103";
String nimbusHost = "192.168.59.103";
String zookeeperHost = zkIp +":2181";
ZkHosts zkHosts = new ZkHosts(zookeeperHost);
SpoutConfig kafkaConfig = new SpoutConfig(zkHosts, "myKafkaTopic", "", "storm");
kafkaConfig.scheme = new SchemeAsMultiScheme(new JsonScheme() {
@Override
public Fields getOutputFields() {
return new Fields("events");
}
});
KafkaSpout kafkaSpout = new KafkaSpout(kafkaConfig);
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("eventsEmitter", kafkaSpout, 8);
builder.setBolt("eventsProcessor", new RollingCountBolt(2, 1), 8)
.fieldsGrouping("requestsEmitter", new Fields("request"));
//More bolts stuffzz
Config config = new Config();
config.setMaxTaskParallelism(5);
config.put(Config.TOPOLOGY_TICK_Tuple_FREQ_SECS, 2);
config.put(Config.NIMBUS_Host, nimbusHost);
config.put(Config.NIMBUS_THRIFT_PORT, 6627);
config.put(Config.STORM_ZOOKEEPER_PORT, 2181);
config.put(Config.STORM_ZOOKEEPER_SERVERS, Arrays.asList(zkIp));
try {
StormSubmitter.submitTopology("my-topology", config, builder.createTopology());
} catch (Exception e) {
throw new IllegalStateException("Couldn't initialize the topology", e);
}
}
}
お役に立てれば、
ホセ・ルイス