apache-beam

My question is, behind the scene, for element-wise Beam DoFn (ParDo), how does the Cloud Dataflow parallel workload? For example, in my ParDO, I send out one http request to an external server for one element. And I use 30 workers, each has 4vCPU. Does that mean on each worker, there will be 4 threads at maximum? Does that mean from each worker, only 4 http connections are necessary or can be established if I keep them alive to get the best performance? How can I adjust the level of parallelism other than using more cores or more workers? with my current setting (30*4vCPU worker), I can

Are there any guidelines or limitations for using stateful processing and timers with the Beam Dataflow runner (as of v2.1.0)? Things such as limitations on the size of state or frequency of updates etc.? The candidate streaming pipeline would use state and timers extensively for user session state, with Bigtable as durable storage. Here is some general advice for your use case Please aggregate multiple elements then set a timer. Please don't create a timer per element, which would be excessive. Try and aggregate state, instead of accumulating large amount of state. I.e. aggregate as a sum and

Join two Pcollection with CoGroupsByKey approach taking hours to execute the 8+ millions records. Noted from another stackoverflow post CoGbkResult has more than 10000 elements,reiteration (which may be slow) is required that "CoGbkResult has more than 10000 elements, reiteration (which may be slow) is required." Any suggestion to improve this performance using this approach. Here is the code snippet, PCollection<TableRow> pc1 = ...; PCollection<TableRow> pc2 = ...; WithKeys<String, TableRow> withKeyValue = WithKeys.of((TableRow row) -> String.format("%s",row.get("KEYNAME"))) .withKeyType

Is it possible to use the equivalent of --autodetect in DataFlow? i.e. can we load data into a BQ table without specifying a schema, equivalent to how we can load data from a CSV with --autodetect ? ( potentially related question ) If you are using protocol buffers as objects in your PCollections (which should be performing very well on the Dataflow back-end) you might be able to use a util I wrote in the past. It will parse the schema of the protobuffer into a BigQuery schema at runtime, based on inspection of the protobuffer descriptor. I quickly uploaded it to GitHub , it's WIP, but you