gobblin

来源： https://stackoverflow.com/questions/63455670/trying-to-use-apache-gobblin-to-read-salesforce-data-using-soap-apis-instead-o

问题 I'm new to Gobblin and trying to read JSON Kafka message and convert it to AVRO then store it in HDFS. My current job file is like a blow: job.name=GobblinKafkaQuickStart job.group=GobblinKafka job.description=Gobblin quick start job for Kafka job.lock.enabled=false kafka.brokers=localhost:9092 source.class=org.apache.gobblin.source.extractor.extract.kafka.KafkaSimpleSource extract.namespace=org.apache.gobblin.extract.kafka converter.classes=org.apache.gobblin.converter.json

问题 I'm new to Gobblin and trying to read JSON Kafka message and convert it to AVRO then store it in HDFS. My current job file is like a blow: job.name=GobblinKafkaQuickStart job.group=GobblinKafka job.description=Gobblin quick start job for Kafka job.lock.enabled=false kafka.brokers=localhost:9092 source.class=org.apache.gobblin.source.extractor.extract.kafka.KafkaSimpleSource extract.namespace=org.apache.gobblin.extract.kafka converter.classes=org.apache.gobblin.converter.json

问题 I'm new to the Gobblin and trying to ingest data from the Kafka to HDFS. I was able to flow the Kafka-HDFS Ingestion example successfully. But now I need to add a time-based writer partition option to my job. I did go through the TimeBasedWriterPartitioner google forum and come up with the bellow solution as suggested by Zongjun. I create a separate Java project for my time-based writer partition class: import gobblin.writer.partitioner.TimeBasedWriterPartitioner; public class

问题 I'm new to the Gobblin and trying to ingest data from the Kafka to HDFS. I was able to flow the Kafka-HDFS Ingestion example successfully. But now I need to add a time-based writer partition option to my job. I did go through the TimeBasedWriterPartitioner google forum and come up with the bellow solution as suggested by Zongjun. I create a separate Java project for my time-based writer partition class: import gobblin.writer.partitioner.TimeBasedWriterPartitioner; public class

问题 I am getting this error while I try to import the gobblin distribution into my IDE , I have tried both inteliJ and eclipse , not able to find any luck. Below are the errors which I get when I try to import. In Eclipse the error is: *org.gradle.tooling.BuildException: Could not run build action using Gradle distribution 'https://services.gradle.org/distributions/gradle-3.3-bin.zip'. * For InteliJ *Cause: startup failed: build file 'C:\Users\sayyad.ghazi\Desktop\gob\gobblin-master\gobblin