google-bigquery

问题 I have a set of text files that are uploaded every 5 minutes into the google cloud storage. I want to put them into BigQuery in every 5 minutes (because text files uploaded into Cloud Storage in every 5 min). I know text files cant to be uploaded into BigQuery. What is the best approach for this? Sample of a text file Thanks in advance. 回答1: He is an alternative approach, which will use an event-based Cloud Function to load data into BigQuery. Create a cloud function with "Trigger Type" as

问题 recently there was a change in Bigquery UI and it seems that is no longer possible to schedule a stored procedure to execute automatically. Using the UI, just keeps asking to insert a destination table. If I put a dummy table, the schedule is created but then when tries to execute just throws an error saying that we can't have a destination table when executing a stored procedure. Is anyone having this issue and has any kind of workaround ? Thanks in advance. 回答1: You can opt out of the

问题 I understand that BigQuery supports merging two tables. Currently, the INSERT operation allows inserting explicit values into a table, such as INSERT dataset.Inventory (product, quantity) VALUES('top load washer', 10), ('front load washer', 20), ('dryer', 30), ('refrigerator', 10), ('microwave', 20), ('dishwasher', 30), ('oven', 5) Is there a way to do something similar with the MERGE operation? For example, something like: MERGE dataset.DetailedInventory T USING('top load washer', 10), (

问题 Say I have a BQ table containing the following information | id | test.name | test.score | |---- |----------- |------------ | | 1 | a | 5 | | | b | 7 | | 2 | a | 8 | | | c | 3 | Where test is nested. How would I pivot test into the following table? | id | a | b | c | |---- |--- |--- |--- | | 1 | 5 | 7 | | | 2 | 8 | | 3 | I cannot pivot test directly, as I get the following error message at pivot(test) : Table-valued function not found . Previous questions (1, 2) don't deal with nested columns

问题 I've got a bunch of tables in BigQuery that I'd like to rename. The primary reason for this is because I'd like to avail of the wildcard table feature in my queries. The original format of my table names are not designed to work well with this feature. Is it possible to programmatically/easily rename a lot of tables in BigQuery? 回答1: Currently, there is no way to rename a table in BigQuery. The way to achieve this is by running a BigQuery copy job that will copy the table with the new name,

问题 I have a dataflow job where I will read from bigquery query first (in standard sql). It works perfectly in direct runner mode. However I tried to run this dataflow in dataflow runner mode and encountered this error : response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Thu, 24 Dec 2020 09:28:21 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff',

问题 When I try to search for historical data for events on BigQuery I can only find events from the last week. I couldn't find anywhere the information if this is correct, or if I am doing something wrong. Basically it seems that I can only query events_YYYYMMDD from the past week. Is this correct? 回答1: Did you by any chance enable the BigQuery export from Firebase a week ago? Keep in mind: Firebase will only write to BigQuery that it receives from the moment you enable the integration. It won't

问题 I need any log or data of Audiences by Big Query, how to consult this in a non-manual way through events? I need firebase analytics groups and audience data 回答1: When you connect the Analytics from your Firebase project to BigQuery, only the raw events are exported to BigQuery. Neither audience definitions, not audience membership are exported to BigQuery. So if you want to see audiences in your BigQuery based reports, you'll have to (re)create both the audience definitions and memberships

问题 I'm trying to get the value of multiple columns in an array and set them as a variable that can be used in the loop to do something else. Thanks. DECLARE the_array ARRAY<STRUCT<value1 STRING,value2 STRING>>; SET the_array = ( SELECT ARRAY_AGG(STRUCT(value1,value2)) FROM `project.dataset.table` WHERE somthing = 'somthing' ); LOOP SET i = i + 1; SET var1 = the_array[ORDINAL(????)]; // what do I do here? SET var2 = the_array[ORDINAL(???)]; // what do I do here? IF i > ARRAY_LENGTH(the_array)

问题 I have a dataset containing tables with similar table names ending in yyyymmdd. For example: myproject:mydataset.Sales20140815 myproject:mydataset.Sales20140816 myproject:mydataset.Sales20140817 myproject:mydataset.Sales20140818 ... myproject:mydataset.Sales20140903 myproject:mydataset.Sales20140904 Is there any way to write the BigQuery to query the latest table in the dataset (for the above example it is myproject:mydataset.Sales20140904 )? 回答1: N.N. answer is good, but relying on the