google-cloud-pubsub

I am trying to setup a Dataflow job using the google provided template PubSub to BigQuery . I see an option to specify the Cloud Pub/Sub input topic but I don't see any option to specify Pub/Sub input subscription in GCP console UI. If I provide the topic, job would automatically create a subscription to read the messages from the provided topic. Problem with this is, the job will see only messages published to the topic after the Dataflow job has started. Anything published before to the same topic would be ignored. I don't have any complex transformations to do in my job. So the google

I'm using pubsub to push messages into an App Engine app written in node on the flexible environment. Is there a way I can limit my endpoints to only traffic from pubsub? In the standard environment, App Engine has handlers that can define admin only requests and secure endpoints. However, this functionality is not available in the flexible environment. Is it possible to set up Firewall rules for only Google requests (Firewall appears to be application wide, not endpoint?), is there a standard method to secure endpoints or do I need to custom roll a solution? Turns out Google has posted a

The docs for PubSub state that the max payload after decoding is 10MB. My question is whether or not it is advantageous to compress the payload at the publisher before publishing to increase data throughput? This especially can be helpful if the payload has a high compression ratio like a json formatted payload. If you are looking for efficiency on PubSub I would first concentrate on using the best API, and that's the gRPC one. If are using the client libraries then the chance is high that it's using gRPC anyway. Why gRPC? gRPC is binary and your payload doesn't need to go through hoops to be

I need to perform a very simple transformation on some data (extract a string from JSON), then write it to PubSub - I'm attempting to use a custom python Dataflow job to do so. I've written a job which successfully writes back to Cloud Storage, but my attempts at even the simplest possible write to PubSub (no transformation) result in an error: JOB_MESSAGE_ERROR: Workflow failed. Causes: Expected custom source to have non-zero number of splits. Has anyone successfully written to PubSub from GCS via Dataflow? Can anyone shed some light on what is going wrong here? def run(argv=None): parser =

I'm trying to setup my development environment. Instead of using google cloud pubsub in production, I've been using the pubsub emulator for development and testing. To achieve this I set the following environment variable: export PUBSUB_EMULATOR_HOST=localhost:8586 This worked for the python google pubsub library but when I switched to using java apache beam for google dataflow, the pipeline still points to production google pubsub. Is there a setting, environment variable or method on the pipeline that I need to set so that the pipeline reads for the local pubsub emulator? I found the

I am using the java library to subscribe to a subscription from my code. Using sbt: "com.google.cloud" % "google-cloud-pubsub" % "0.24.0-beta" I followed this guide to write a subscriber: https://cloud.google.com/pubsub/docs/pull val projectId = "test-topic" val subscriptionId = "test-sub" def main(args: Array[String]): Unit = { val subscriptionName = SubscriptionName.create(projectId, subscriptionId) val subscriber = Subscriber.defaultBuilder(subscriptionName, new PastEventMessageReceiver()).build() subscriber.startAsync() System.in.read() } class PastEventMessageReceiver extends