scribe

以下是我在公司内部分享的关于分布式日志收集系统的PPT内容，现在与大家分享，希望对于需要使用的人能够起到基本的入门作用或是了解！ 1.分布式日志收集系统:背景介绍 许多公司的平台每天会产生大量的日志（一般为流式数据，如，搜索引擎的pv，查询等），处理这些日志需要特定的日志系统，一般而言，这些系统需要具有以下特征： （1） 构建应用系统和分析系统的桥梁，并将它们之间的关联解耦； （2） 支持近实时的在线分析系统和类似于Hadoop之类的离线分析系统； （3） 具有高可扩展性。即：当数据量增加时，可以通过增加节点进行水平扩展。 2.分布式日志收集系统：Facebook Scribe主要内容 （1）Scribe简介及系统架构 （2）Scribe技术架构 （3）Scribe部署结构 （4）Scribe主要功能和使用方案 （5）Scribe的具体应用实例 （6）Scribe的扩展 （7）Scribe研究体会 3.Scribe简介 Scribe是facebook开源的日志收集系统，在facebook内部已经得到大量的应用。 Scribe是基于一个使用非阻断C++服务器的thrift服务的实现。它能够从各种日志源上收集日志，存储到一个中央存储系统 （可以是NFS，分布式文件系统等）上，以便于进行集中统计分析处理。它为日志的“分布式收集，统一处理”提供了一个可扩展的，高容错的方案。 4

最近准备整合多个系统的日志，因而想到构建一套分布式日志存储系统，首先考虑的是Scribe，不过编译安装耗费了很多时间（Scribe相关文档确实少了点，相比Flume） 环境：Ubuntu13.04 32bit 组件: scribe 2.x(最新版) thrift 0.9.0 boost 1.54 fb303 thrift自带 至于其他所依赖的相关包（如libevent、automake、flex、bison等）可根据configure的结果进行安装或更新。 1、安装boost 官网下载boost，版本要求高于1.36（作者用的是1.54.0），按照文档中给出的方法编译： $tar --bzip2 -xf /home/vincent/Download/boost_1_54_0.tar.bz2 $cd /home/vincent/Download/boost_1_54_0 $./bootstrap.sh --help $./bootstrap.sh --prefix=opt/boost_1.54 $./b2 install 至此，boost库编译完成，编译时无需指定其他选项，在--prefix参数指定的目录下生成include和lib文件夹存放头文件和库文件。 2、安装thrift 下载thrift包，安装是经典的三大步： $tar zxvf thrift-0.9.0.tar.gz

I try to use scribe for authorization with facebook and twitter. There is no problems with facebook. But when I try requestToken = service.getRequestToken() for twitter I get Cannot get String from a null object > ru.myx.ae3.exec.ExecErrorDefault > Cannot get String from a null object > java.lang.IllegalArgumentException : org.scribe.utils.Preconditions.check(Preconditions.java:84) : org.scribe.utils.Preconditions.checkNotNull(Preconditions.java:31) : org.scribe.utils.StreamUtils.getStreamContents(StreamUtils.java:22) : org.scribe.model.Response.parseBodyContents(Response.java:43) : org.scribe

This does NOT happen when I use it with a GET. i.e. this DOES NOT WORK: ProxyAuthenticator pa = new ProxyAuthenticator(); Authenticator.setDefault( pa ); OAuthService service = new ServiceBuilder().provider( TwitterApi.class ).apiKey( strAPIKey ).apiSecret( strAPISecret ).build(); Token requestToken = service.getRequestToken(); Scanner in = new Scanner( System.in ); Verifier verifier = new Verifier(in.nextLine()); String url = "http://api.twitter.com/1/statuses/update.json?status=helllllooooo"; OAuthRequest request = new OAuthRequest( Verb.POST, url ); service.signRequest( accessToken ,

Yes, it's a long question with a lot of detail... So, my question is: How can I stream an upload to Vimeo in segments? For anyone wanting to copy and debug on their own machine: Here are the things you need: My code here . Include the Scribe library found here Have a valid video file (mp4) which is at least greater than 10 MB and put it in the directory C:\test.mp4 or change that code to point wherever yours is. That's it! Thanks for helping me out! Big update: I've left a working API Key and Secret for Vimeo in the code here . So as long as you have a Vimeo account, all the code should work

Hi I'm using the scribe library for OAuth communication with LInkedIn. I have a login class and a query class. The login class creates a service to get my request token and secret and creates my access token. The access token is then saved to the preferences file. This seems to work fine and I can make a successful api call after all the tokens have been created. In my OAuth query class i retrieve the access token, create another service and try to make an API call, but when ever I load an activity which makes calls this class it causes my app to crash. I have tested to make sure that the

I need to make a connection with Viagogo website using oAuth. Referring to their documentation I need to create a request similar to the following one Using the example in step 1A, this means you may generate a signature base string that looks like the following: GET&http%3A%2F%2Fapi.viagogo.net%2FPublic%2FSimpleOAuthAccessRequest&oauth_consumer_key%3Dtestkey%26oauth_nonce%3Dmyn0nc3%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1292404912%26oauth_version%3D1.0%26scope%3DAPI.Public I am using the following code but when I comment lines 1,2 it return unauthorized error , and when I

I'm currently implementing Reddit OAuth2 login into my web app. The handshake and token exchange work fine when testing locally but when running on the server (hosted on 'OpenShift' DIY cartridge) I get the following error: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 512 to 1024 (inclusive) Which is results in java.lang.RuntimeException: Could not generate DH keypair I've been searching most of the day and have found different solutions ranging from changing Java version to using BouncyCastle. However, I'm using the Scribe