avro

问题 I have a concrete class which I am Serializing in Byte array to be sent to a Kafka topic. For serializing I am using ReflectDatumWriter . Before sending the bytes[] I am putting schema ID in first 4 bytes with schema ID after checking some online tutorial. I am able to send the message but while consuming it in Avro console consumer I am getting response as : ./bin/kafka-avro-console-consumer --bootstrap-server 0:9092 --property schema.stry.url=http://0:8081 --property print.key=true --topic

问题 There are plugins that can generate java file using Avro schema. Is there a reverse of that? I would like to generate Avro schema based on java file. 回答1: Kite SDK has a lot of useful things to work with Avro Schema. Take a look to this http://kitesdk.org/docs/1.1.0/Inferring-a-Schema-from-a-Java-Class.html 来源： https://stackoverflow.com/questions/45700725/generate-avro-file-based-on-java-file

问题 I'm studying Avro RPC and I'm trying to create a simple example to better understand it. But I'm facing a difficulty: I can't run a server with more than one Responder , as the NettyServer constructor only allows me to use one: public NettyServer(Responder responder, InetSocketAddress addr) So, if I have more than one IDL, like this: @namespace("foo.bar") protocol FooProtocol { void foo(); } @namespace("foo.bar") protocol BarProtocol { void bar(); } I'm not able to add both to my NettyServer

问题 Can somebody share example of reading avro using java in spark? Found scala examples but no luck with java. Here is the code snippet which is part of code but running into compilation issues with the method ctx.newAPIHadoopFile . JavaSparkContext ctx = new JavaSparkContext(sparkConf); Configuration hadoopConf = new Configuration(); JavaRDD<SampleAvro> lines = ctx.newAPIHadoopFile(path, AvroInputFormat.class, AvroKey.class, NullWritable.class, new Configuration()); Regards 回答1: You can use the

问题 I am trying to setup a kafka producer with KafkaAvroSerialzer for value. And I am facing this error wheneve rit is trying to created the Producer. I am using all the jars provided in confluent 5.2.1 java.lang.NoClassDefFoundError: Could not initialize class io.confluent.kafka.schemaregistry.client.rest.RestService at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.<init>(CachedSchemaRegistryClient.java:104) at io.confluent.kafka.schemaregistry.client

问题 I'm new to Maven and installed 3.5.0 in Windows 64 bit machine. I have added Avro dependency in pom.xml as specified in https://avro.apache.org/docs/1.8.2/gettingstartedjava.html. I am facing Build failure and getting ArrayIndexOutOfBoundsException when doing mvn clean compile from the command line: [DEBUG] ======================================================================= [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] --------

问题 I'm desparately trying to configure serializer instances to use in my storm topology. The storm documentation states, there are 2 ways to register serializers : 1. The name of a class to register. In this case, Storm will use Kryo’s FieldsSerializer to serialize the class. This may or may not be optimal for the class – see the Kryo docs for more details. 2. A map from the name of a class to register to an implementation of com.esotericsoftware.kryo.Serializer. I want to use 2. -> Map<String,

问题 I would like to serialize data from a JSON object and send it throught the network with kafka as an end. Now I have an avro schema in a file, that determinate the fields necessary to send to kafka for the logging system: {"namespace": "com.company.wr.messages", "type": "record", "name": "Log", "fields": [ {"name": "timestamp", "type": "long"}, {"name": "source", "type": "string"}, {"name": "version", "type": "string"}, {"name": "ipAddress", "type": "string"}, {"name": "name", "type": "string"

问题 I can do, { "type": "record", "name": "Foo", "fields": [ {"name": "bar", "type": { "type": "record", "name": "Bar", "fields": [ ] }} ] } and that works fine, but supposing I want to split the schema up into two files such as: { "type": "record", "name": "Foo", "fields": [ {"name": "bar", "type": "Bar"} ] } { "type": "record", "name": "Bar", "fields": [ ] } Does Avro have the capability to do this? 回答1: Yes, it's possible. I've done that in my java project by defining common schema files in

问题 I have an avro schema, and I am writing data with that schema to an AvroSequenceFileOutputFormat . I looked in the file and can confirm that the schema is there to read. I call the function avro = load 'part-r-00000.avro' using AvroStorage(); and it gives me the error message ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2245: Cannot get schema from loadFunc org.apache.pig.builtin.AvroStorage Details at logfile: /Users/ajosephs/Code/serialization-protocol/output/pig_1391635368675.log Does