google-bigquery

问题 I'm having trouble wrapping my head around using a string function such as STARTS_WITH or an operator such as LIKE in a LEFT JOIN ON where the parameters of either come from both tables in the join. Rather than try to explain the abstracts, I've distilled a small example ... Let's consider a table called fuzzylog that has a key field fullname that I want to canonicalize by joining to a names table that has the same column. The key field in fuzzylog may be a bit messy or free-form so a direct

问题 I'm using cloud Dataflow to import data from Pub/Sub messages to BigQuery tables. I'm using DynamicDestinations since these messages can be put into different tables. I've recently noticed that the process started consuming all resources and messages stating that the process is stuck started showing: Processing stuck in step Write Avros to BigQuery Table/StreamingInserts/StreamingWriteTables/StreamingWrite for at least 26h45m00s without outputting or completing in state finish at sun.misc

问题 I'm using cloud Dataflow to import data from Pub/Sub messages to BigQuery tables. I'm using DynamicDestinations since these messages can be put into different tables. I've recently noticed that the process started consuming all resources and messages stating that the process is stuck started showing: Processing stuck in step Write Avros to BigQuery Table/StreamingInserts/StreamingWriteTables/StreamingWrite for at least 26h45m00s without outputting or completing in state finish at sun.misc

问题 https://cloud.google.com/bigquery/docs/creating-partitioned-tables shows how to create partitioned table in Python. I've been there, I've done that. Now the question is, how to do the same thing with Java API? What is the corresponding Java code doing the same thing as the Python one below: { "tableReference": { "projectId": "myProject", "tableId": "table1", "datasetId": "mydataset" }, "timePartitioning": { "type": "DAY" } } Java with missing partitioning: Job createTableJob = new Job();

问题 I have a federated source in BigQuery which is pointing to some CSV files in GCS. When I try to read to the federated BigQuery table as a source for a Dataflow pipeline, it throws the following error: 1226 [main] ERROR com.google.cloud.dataflow.sdk.util.BigQueryTableRowIterator - Error reading from BigQuery table Federated_test_dataflow of dataset CPT_7414_PLAYGROUND : 400 Bad Request { "code" : 400, "errors" : [ { "domain" : "global", "message" : "Cannot list a table of type EXTERNAL.",

问题 I have a federated source in BigQuery which is pointing to some CSV files in GCS. When I try to read to the federated BigQuery table as a source for a Dataflow pipeline, it throws the following error: 1226 [main] ERROR com.google.cloud.dataflow.sdk.util.BigQueryTableRowIterator - Error reading from BigQuery table Federated_test_dataflow of dataset CPT_7414_PLAYGROUND : 400 Bad Request { "code" : 400, "errors" : [ { "domain" : "global", "message" : "Cannot list a table of type EXTERNAL.",

问题 I've created a view to join two tables in Big Query. When I use that view as a data source in Data Studio I get a configuration error. The query used to create the view looks like this: SELECT emp.name, emp.gender, emp.age, pay.salary FROM [project:doug.employees] as emp JOIN [project:doug.payrates] as pay on emp.name = pay.name Within Big Query the view works as expected. I can query it without any problems. If I try to use the view directly as a data source in Data Studio, I get the

问题 I've created a view to join two tables in Big Query. When I use that view as a data source in Data Studio I get a configuration error. The query used to create the view looks like this: SELECT emp.name, emp.gender, emp.age, pay.salary FROM [project:doug.employees] as emp JOIN [project:doug.payrates] as pay on emp.name = pay.name Within Big Query the view works as expected. I can query it without any problems. If I try to use the view directly as a data source in Data Studio, I get the

问题 How many bytes do the following types take up in BigQuer: Timestamp Datetime Date My guess was a date could be stored in 2 bytes, and a timestamp perhaps 8, but wasn't sure and it doesn't say on the https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types page. 回答1: The size of BigQuery's data types is as follows: Data type Size INT64/INTEGER 8 bytes FLOAT64/FLOAT 8 bytes NUMERIC 16 bytes BOOL/BOOLEAN 1 byte STRING 2 bytes + the UTF-8 encoded string size BYTES 2 bytes + the

问题 How many bytes do the following types take up in BigQuer: Timestamp Datetime Date My guess was a date could be stored in 2 bytes, and a timestamp perhaps 8, but wasn't sure and it doesn't say on the https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types page. 回答1: The size of BigQuery's data types is as follows: Data type Size INT64/INTEGER 8 bytes FLOAT64/FLOAT 8 bytes NUMERIC 16 bytes BOOL/BOOLEAN 1 byte STRING 2 bytes + the UTF-8 encoded string size BYTES 2 bytes + the