google-cloud-spanner | 易学教程

SpannerIO Dataflow 2.3.0 stuck in CreateDataflowView

阅读更多关于 SpannerIO Dataflow 2.3.0 stuck in CreateDataflowView

问题 In the pipeline, I'm reading from Pub/Sub and I'm attempting to write to spanner. Writing to BigTable works, but spanner is a better option for my needs. In the image below I've expanded the relevant steps. In the top right corner is the "Debug spanner" step, which shows the proper messages via LOG. I'm quite confident that SpannerIO.Write sees those messages as well. Even without this debugging step, the messages still don't get written to spanner. There is no exception whatsoever, but as

GCP: What is the best option to setup a periodic Data pipeline from Spanner to Big Query

阅读更多关于 GCP: What is the best option to setup a periodic Data pipeline from Spanner to Big Query

Task : We have to setup a periodic sync of records from Spanner to Big Query. Our Spanner database has a relational table hierarchy. Option Considered I was thinking of using Dataflow templates to setup this data pipeline. Option1 : Setup a job with Dataflow template 'Cloud Spanner to Cloud Storage Text' and then another with Dataflow template 'Cloud Storage Text to BigQuery'. Con : The first template works only on a single table and we have many tables to export. Option2 : Use 'Cloud Spanner to Cloud Storage Avro' template which exports the entire database. Con : I only need to export

External Backups/Snapshots for Google Cloud Spanner

阅读更多关于 External Backups/Snapshots for Google Cloud Spanner

问题 Is it possible to snapshot a Google Cloud Spanner Database/table(s)? For compliance reasons we have to have daily snapshots of the current database that can be rolled back to in the event of a disaster: is this possible in Spanner? Is there intention to support it if not? For those who might ask why we would need it as Spanner is replicated/redundant etc - it doesn't guard against human error (dropping a table by accident) or sabotage/espionage hence the question and requirement. Thanks, M

Google-Cloud: Jetty ALPN/NPN has not been properly configured

阅读更多关于 Google-Cloud: Jetty ALPN/NPN has not been properly configured

Getting exception whilst using Google Pubsub to list topics, my web application is running on tomcat. public static List<String> listTopics(GcpCredentials gcCredentials, String project) throws GCPException, IOException { List<String> topics = new ArrayList<>(); TopicAdminClient client = getTopicClient(gcCredentials); ProjectName projectName = ProjectName.create(project); ListTopicsPagedResponse response = client.listTopics(projectName); for (Topic topic :response.iterateAll()) { topics.add(topic.getNameAsTopicName().getTopic()); } return topics; }` Exception: java.lang.IllegalArgumentException

Local development with cloud-spanner

阅读更多关于 Local development with cloud-spanner

Is there any way to do local development with cloud spanner? I've taken a look through the docs and the CLI tool and there doesn't seem to be anything there. Alternatively, can someone suggest a SQL database that behaves similarly for reads (not sure what to do about writes)? EDIT: To clarify, I'm looking for a database which speaks the same flavour of SQL as Cloud Spanner so I can do development locally. The exact performance characteristics are not as important as the API and consistency behaviour. I don't think Cockroach meets these requirements? There is currently no local development

What's the difference between BigQuery and BigTable? [closed]

阅读更多关于 What's the difference between BigQuery and BigTable? [closed]

问题 Closed . This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Update the question so it focuses on one problem only by editing this post. Closed 5 months ago . Is there any reason why someone would use BigTable instead of BigQuery? Both seem to support Read and Write operations with the latter offering also advanced 'Query' operations. I need to develop an affiliate network (thus I need to track clicks and 'sales') so I'm quite confused

Easiest primary key for main table?

阅读更多关于 Easiest primary key for main table?

问题 My main table, Users , stores information about users. I plan to have a UserId field as the primary key of the table. I have full control of creation and assignment of these keys, and I want to ensure that I assign keys in a way that provides good performance. What should I do? 回答1: You have a few options: 1) The most generic solution is to use UUIDs, as specified in RFC 4122. For example, you could have a STRING(36) that stores UUIDs. Or you could store the UUID as a pair of INT64 s or as a

Error using SpannerIO in apache beam

阅读更多关于 Error using SpannerIO in apache beam

This question is a follow-up to this one . I am trying to use apache beam to read data from a google spanner table (and then do some data processing). I wrote the following minimum example using the java SDK: package com.google.cloud.dataflow.examples; import java.io.IOException; import org.apache.beam.sdk.Pipeline; import org.apache.beam.sdk.PipelineResult; import org.apache.beam.sdk.io.gcp.spanner.SpannerIO; import org.apache.beam.sdk.options.PipelineOptions; import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.values.PCollection; import com.google.cloud

Why is Google's TrueTime API hard to duplicate?

阅读更多关于 Why is Google's TrueTime API hard to duplicate?

问题 I'm not sure why the press in general says that Google's TrueTime API is hard to replicate (Wired, Slashdot, etc). I can understand how it would be a tough thing to get the low error intervals that Google is achieving, but I don't see how the API itself would be very difficult. For example, I whipped up a hacked together version. Here's the interval. typedef struct TT_interval { struct timeval earliest; struct timeval latest; } TT_interval; Here's the now function. int TT_now(TT_interval*

Why is Google's TrueTime API hard to duplicate?

阅读更多关于 Why is Google's TrueTime API hard to duplicate?

I'm not sure why the press in general says that Google's TrueTime API is hard to replicate (Wired, Slashdot, etc). I can understand how it would be a tough thing to get the low error intervals that Google is achieving, but I don't see how the API itself would be very difficult. For example, I whipped up a hacked together version. Here's the interval. typedef struct TT_interval { struct timeval earliest; struct timeval latest; } TT_interval; Here's the now function. int TT_now(TT_interval* interval) { struct ntptimeval tv; struct timeval delta; struct timeval* earliest_p = &(interval->earliest)