flume

Please share your thoughts. The requirement is to migrate the data in MySQL db to Hadoop/HBase for analytic purposes. The data should be migrated real time or near real time. Can flume support this. What can be a better approach. AvkashChauhan The direct answer to your question is yes. Flume is designed as a distributed data transport and aggregation system for event/log structured data. If set up "correctly" flume can push data for continuous ingestion in Hadoop. This is when Flume is set up correctly to collect data from various sources (in this case MySql) and I am sure if data is available

概要 日志分析，有两个主要模块日志收集以及分析统计。日志收集主要实现日志数据源的获取。分析统计是对数据源的聚合和统计分析。 日志收集又分为离线收集和热数据收集：离线收集的日志收集服务器与日志分析系统完全隔离，服务器日志以文本的格式输出到指定文件，然后通过logstash或者flume等文本收集系统进行传输，从而形成对应的日志数据源。 分析统计主要根据日志的数据源新进聚合和处理，聚合的过程主要是分布式日志的数据聚拢，因为聚拢的策略一般是 遵循fifo准则 和fcfs算法，保证时间优先进行日志聚合。因为目前日志分析在一定程度上要求实时性，对日志数据源的处理，又多了些预处理、流计算等模式进行数据的实时处理运算，以实现日志数据最终存储或者内存的格式即查询或者图形化展示需要的内容。 热门应用的介绍 1、 logstash elk 是logstash,elasticsearch,kibana 的缩写，这个架构组合非常适合做日志分析，其就是离线式日志实时分析的代表，而且有目前已知的最强大的社区支持。logstash shipper利用file input读取日志文件，filter组件进行正则日志筛选，然后通过 broker 进行聚合，日志通过es out已有的组件输出到 elasticsearch。kibana根据elasticsearch的索引进行实时查询。成功案例参照 新浪 、芒果TV等

I have been using hadoop 2.4.1 and flume 1.5.0.1 for experimenting and pretty new to these. All hadoop libraries are included in the classpath from $HADOOP_HOME/share/*.jar. I have flume-conf.properties as below: agentMe.channels = memory-channel agentMe.sources = my-source agentMe.sinks = log-sink hdfs-sink agentMe.channels.memory-channel.type = memory agentMe.channels.memory-channel.capacity = 1000 agentMe.channels.memory-channel.transactionCapacity = 100 agentMe.sources.my-source.type = syslogtcp #agentMe.sources.my-source.bind = 192.168.X.X agentMe.sources.my-source.port = 8100 agentMe

There is tiny problem when I try Cloudera 5.4.2. Base on this article Apache Flume - Fetching Twitter Data http://www.tutorialspoint.com/apache_flume/fetching_twitter_data.htm It tries to fetching tweets using Flume and twitter streaming for data analysis. All things are happy, create Twitter app, create directory on HDFS, configure Flume then start to fetch data, create schema on top of tweets. Then, here is the problem. Twitter streaming converts tweets to Avro format and send Avro events to downsteam HDFS sinks, when Hive table backed by Avro load the data, I got the error message said

1.从前到后从你教育背景(学过哪些课)到各个项目你负责的模块,问的很细(本以为他是物理学博士,但是所有的技术都懂) 2.hadoop 的 namenode 宕机,怎么解决 先分析宕机后的损失，宕机后直接导致client无法访问，内存中的元数据丢失，但是硬盘中的元数据应该还存在，如果只是节点挂了， 重启即可，如果是机器挂了，重启机器后看节点是否能重启，不能重启就要找到原因修复了。但是最终的解决方案应该是在设计集群的初期 就考虑到这个问题，做namenode的HA。 3.一个datanode 宕机,怎么一个流程恢复 Datanode宕机了后，如果是短暂的宕机，可以实现写好脚本监控，将它启动起来。如果是长时间宕机了，那么datanode上的数据应该已经 被备份到其他机器了，那这台datanode就是一台新的datanode了，删除他的所有数据文件和状态文件，重新启动。 4.Hbase 的特性,以及你怎么去设计 rowkey 和 columnFamily ,怎么去建一个table 因为hbase是列式数据库，列非表schema的一部分，所以在设计初期只需要考虑rowkey 和 columnFamily即可，rowkey有位置相关性，所以 如果数据是练习查询的，最好对同类数据加一个前缀，而每个columnFamily实际上在底层是一个文件，那么文件越小，查询越快，所以讲经

flume 三大组件 source 收集 channel 聚集 sink 输出 使用Flume关键就是写配置文件 A 配置source B 配置channel C 配置sink D 把以上3个组件串起来 1.通过IP端口 接收数据 a1 agent名称r1 数据源名称k1 sinks名称c1 channel名称 # example.conf: A single-node Flume configuration # Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind =hadoop000a1.sources.r1.port = 44444 # Describe the sink a1.sinks.k1.type = logger # Use a channel which buffers events in memory a1.channels.c1.type = memory # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1

I have a simple flume setup with a HTTP souce and a sink that writes the POST request payload to a file. (This complete setup is on a Linux machine). After that my task is to do a performance test on ths setup. So I decided to use JMeter (this is the first time, I am using it). So I created a test plan on my windows machine (using GUI) and then copied it to the jmeter/bin folder in the linux enviornment. When I tried ruuning it - java -jar ApacheJMeter.jar -n -t flume_http_test.jmx I am getting this error ERROR - jmeter.JMeter: Error in NonGUIDriver java.lang.RuntimeException: Could not find