Storm怎么写一个爬虫

Storm怎么写一个爬虫

这篇文章主要讲解了“Storm怎么写一个爬虫”，文中的讲解内容简单清晰，易于学习与理解，下面请大家跟着小编的思路慢慢深入，一起来研究和学习“Storm怎么写一个爬虫”吧！

packagecom.digitalpebble.storm.crawler.bolt.indexing;importjava.util.Map;importorg.slf4j.LoggerFactory;importbacktype.storm.task.OutputCollector;importbacktype.storm.task.TopologyContext;importbacktype.storm.topology.OutputFieldsDeclarer;importbacktype.storm.topology.base.BaseRichBolt;importbacktype.storm.tuple.Tuple;importcom.digitalpebble.storm.crawler.StormConfiguration;importcom.digitalpebble.storm.crawler.util.Configuration;/***Agenericboltforindexingdocumentswhichdetermineswhichendpointtouse*basedontheconfigurationanddelegatestheindexingtoit.***/@SuppressWarnings("serial")publicclassIndexerBoltextendsBaseRichBolt{privateConfigurationconfig;privateBaseRichBoltendpoint;privatestaticfinalorg.slf4j.LoggerLOG=LoggerFactory.getLogger(IndexerBolt.class);publicvoidprepare(Mapconf,TopologyContextcontext,OutputCollectorcollector){config=StormConfiguration.create();//gettheimplementationtouse//andinstanciateitStringclassName=config.get("stormcrawler.indexer.class");if(className==null){thrownewRuntimeException("Noconfigurationfoundforindexing");}try{finalClass<BaseRichBolt>implClass=(Class<BaseRichBolt>)Class.forName(className);endpoint=implClass.newInstance();}catch(finalExceptione){thrownewRuntimeException("Couldn'tcreate"+className,e);}if(endpoint!=null)endpoint.prepare(conf,context,collector);}publicvoidexecute(Tupletuple){if(endpoint!=null)endpoint.execute(tuple);}publicvoiddeclareOutputFields(OutputFieldsDeclarerdeclarer){if(endpoint!=null)endpoint.declareOutputFields(declarer);}}

感谢各位的阅读，以上就是“Storm怎么写一个爬虫”的内容了，经过本文的学习后，相信大家对Storm怎么写一个爬虫这一问题有了更深刻的体会，具体使用情况还需要大家实践验证。这里是恰卡编程网，小编将为大家推送更多相关知识点的文章，欢迎关注！