avro

问题 This question concerns Avro version 1.8.1. We have the following field in our AVRO schema: "name" : "sale_price", "type" : ["bytes", "null"], "logicalType": "decimal", "precision": 18, "scale": 17, As you can see, the logicalType of the field has been defined as decimal . But when we use the avro-maven-plugin , it does not generate the correct data type within the generated Java source files. Instead it generates, java.nio.ByteBuffer . How would you have the correct data type generated within

问题 I am using https://github.com/julianpeeters/avrohugger sbt plugin to generate Scala case classes for Avro .avsc files. How can I use the same plugin in a Gradle project? 回答1: I created gradle plugin for generating Scala case classes from Avro schema and it internally uses avrohugger library. So now it's enough to add this plugin to your project: plugins { id 'com.zlad.gradle.avrohugger' version '0.2.1' } When you want to compile Scala sources you need Scala library dependency too: plugins {

问题 I am new to Scala and Avro and I am simply trying to serialise a case class case class MyRecord(my_number: Int) It should not be easier... def serializeSubmapRecord(record: MyRecord): Array[Byte] = { val out = new ByteArrayOutputStream() val encoder = EncoderFactory.get.binaryEncoder(out, null) val writer = new GenericDatumWriter[MyRecord](avro_schema) writer.write(record, encoder) encoder.flush out.close out.toByteArray } where I obtain avro_schema as val avro_my_schema = scala.io.Source

问题 I am trying to create Union field in Avro schema and send corresponding JSON message with it but to have one of the fields - null . https://avro.apache.org/docs/1.8.2/spec.html#Unions What is example of simplest UNION type (avro schema) with corresponding JSON data? (trying to make example without null/empty data and one with null/empty data). 回答1: Here you have an example. Null enum {"name": "Stephanie", "age": 30, "sex": "female", "myenum": null} Not null enum {"name": "Stephanie", "age":

问题 Caused by: org.apache.kafka.common.errors.SerializationException: Error retrieving Avro schema for id 1 Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Subject not found.; error code: 40401 Confluent Version 4.1.0 I am consuming data from a couple of topics(topic_1, topic_2) using KTable, joining the data and then pushing the data onto another topic(topic_out) using KStream. (Ktable.toStream()) The data is in avro format When I check the schema by

问题 I am trying to deserialize an avro byte stream into a scala case class object. Basically, i had a kafka stream with avro encoded data flowing and now there is an addition to the schema and i am trying to update the scala case class to include the new field. The case class looks like this /** Case class to hold the Device data. */ case class DeviceData(deviceId: String, sw_version: String, timestamp: String, reading: Double, new_field: Option[String] = None ) { this() = this("na", "na", "na",

问题 We have observed that when we store the data in Avro format, it converts byte stream to binary, due to which all the comments gets converted to “from deserializer”. We found a jira bug for this issue as well, few confirms, this issue has been addressed with 0.13 version. We are using hive 1.1 (Cloudera). But we are still facing the issue. Jira :- https://issues.apache.org/jira/browse/HIVE-6681 https://www.bountysource.com/issues/1320154-describe-on-a-table-returns-from-deserializer-for-column

问题 I have some Avro classes that i generated, and am now trying to use them in Spark. So I imported my avro generated java class, “twitter_schema”, and refer to it when I deserialize. Seems to work but getting a Cast exception at the end. My Schema: $ more twitter.avsc { "type" : "record", "name" : "twitter_schema", "namespace" : "com.miguno.avro", "fields" : [ { "name" : "username", "type" : "string", "doc" : "Name of the user account on Twitter.com" }, { "name" : "tweet", "type" : "string",

问题 I know that BigQuery supports Avro file upload and I'm successful in loading Avro file into BigQuery. Using below command, java -jar avro-tools-1.7.7.jar fromjson --codec snappy --schema-file SourceSchema.avsc Source.json > Output.snappy.avro I have generated an Avro file with Snappy compression and trying to load into BigQuery but Load job fails with below errors, Errors: file-00000000: The Apache Avro library failed to parse file file-00000000. (error code: invalid) Is it possible to load

问题 I get the following error when I want to send an AVRO message which contains a field that has the type long: Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 61 Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to org.joda.time.DateTime I use Confluent 3.2.0 and Apache Spark 2.2.0. This error is thrown in a Spark Job which processes AVRO messages and prints them in a console. In the AVRO schema, the corresponding