Failed to construct instance from factory method DataflowRunner#fromOptions in beamSql, apache beam

问题

I'm specifying dataflow runner in my beamSql program below :

DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
    options.setStagingLocation("gs://gcpbucket/staging");
    options.setTempLocation("gs://gcpbucket/tmp");
    options.setProject("beta-19xxxx");
    options.setRunner(DataflowRunner.class);
    Pipeline p = Pipeline.create(options);

But I'm getting below exception :

Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
at my.proj.StarterPipeline.main(StarterPipeline.java:34)Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 4 more Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@782830e, only URLClassLoaders are supported.
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage(PipelineResources.java:43)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:262)

can anyone help me out understand what exactly is the issue ?

回答1:

Downgrading java 9 to java 8 did the trick.

回答2:

This problem can happen when using Beam SDK with JVM > 8.

Support for JVM 11 (or 9) is only experimental, starting from Beam SDK 2.12 and at least up to Beam SDK 2.16. This answer provides a bit more context: https://stackoverflow.com/a/57710742/1046584

Regarding this specific stack trace, it seems related to this issue: https://issues.apache.org/jira/browse/BEAM-3718

来源：https://stackoverflow.com/questions/49227327/failed-to-construct-instance-from-factory-method-dataflowrunnerfromoptions-in-b

标签

google-cloud-dataflow

apache-beam