google-bigquery

问题 I'm new to using bq, Big Query command line utility. I have a more complex SQL clause that unfortunately has both apostrophe and quote characters within the SQL statement. Since both characters are in the SQL statement I'm looking for some replacement for them, otherwise one or the other will be interpretted as an "end of query" delimitter. Below is the query that I am trying to run, which works great on the Big Query html interface, but not so great using bq command line utility. Suggestions

问题 I have column in one of the BigQuery table which looks like this. {"name": "name1", "last_delivered": {"push_id": "push_id1", "time": "time1"}, "session_id": "session_id1", "source": "SDK", "properties": {"UserId": "u1"}} Is there any was to get the output like this in GBQ ?? (basically flatten the entire column into different columns) name last_delivered.push_id last_delivered.time session_id source properties.UserId name1 push_id1 time1 session_id1 SDK uid1 Let's say a = {"name": "name1",

问题 I have column in one of the BigQuery table which looks like this. {"name": "name1", "last_delivered": {"push_id": "push_id1", "time": "time1"}, "session_id": "session_id1", "source": "SDK", "properties": {"UserId": "u1"}} Is there any was to get the output like this in GBQ ?? (basically flatten the entire column into different columns) name last_delivered.push_id last_delivered.time session_id source properties.UserId name1 push_id1 time1 session_id1 SDK uid1 Let's say a = {"name": "name1",

问题 I want to update an existing table I have in BigQuery with a single new row. I found an example how this could be done if I have that json in a file in Google Cloud Storage: $confLoad->setSourceUris(array( "gs://json_file_bucket/some_name.json" )); $conf = new \Google_JobConfiguration; $conf->setLoad($confLoad); $job->setConfiguration($conf); $service = new \Google_BigQueryService($this->client); $running = $service->jobs->insert($this->projectId, $job); But I don't understand how this can be

问题 I am trying to use the BigQuery package to interact with Pandas DataFrames. In my scenario, I query a base table in BigQuery, use .to_dataframe(), then pass that to load_table_from_dataframe() to load it into a new table in BigQuery. My original problem was that str(uuid.uuid4()) (for random ID's) was automatically being converted to bytes instead of string, so I am forcing a schema instead of allowing it to auto-detect what to make. Now, though, I passed a job_config with a job_config dict

问题 Can not find any solution to this: Need to convert date(int) in the format - 20190101 to DATE in the format "2019-01-01" I have tried: Cast(cast(20190101 AS STRING) AS date) FORMAT_DATE("%F",CAST(20190101 AS STRING)) And so on...no success 回答1: You are looking for PARSE_DATE() : select PARSE_DATE('%Y%m%d', CAST(20190101 AS STRING)) 回答2: Below is for BigQuery Standard SQL #standardSQL SELECT PARSE_DATE('%Y%m%d', CAST(20190101 AS STRING)) 来源： https://stackoverflow.com/questions/54791436/big

问题 Can not find any solution to this: Need to convert date(int) in the format - 20190101 to DATE in the format "2019-01-01" I have tried: Cast(cast(20190101 AS STRING) AS date) FORMAT_DATE("%F",CAST(20190101 AS STRING)) And so on...no success 回答1: You are looking for PARSE_DATE() : select PARSE_DATE('%Y%m%d', CAST(20190101 AS STRING)) 回答2: Below is for BigQuery Standard SQL #standardSQL SELECT PARSE_DATE('%Y%m%d', CAST(20190101 AS STRING)) 来源： https://stackoverflow.com/questions/54791436/big

问题 I have an XCom associated with the Task ID database_schema stored in Airflow that is the JSON schema for a dataset sales_table that I want to load into BigQuery. The data for the BigQuery dataset sales_table comes from a CSV file retailcustomer_data.csv stored in Google Cloud Storage. The operator for loading the data from GCS to BigQuery is as follows: gcs_to_bigquery = GoogleCloudStorageToBigQueryOperator(task_id = 'gcs_to_bigquery', bucket = bucket, source_objects = ['retailcustomer_data

问题 I am trying to load a PIPE delimited file and running bq load command from windows platform. It is not accepting pipe delimiter in the command. E.g. I am trying to use -F operator to specify the delimiter and could specify space delimiter but it stops working when I specify pipe delimiter. C:\Windows>bq load -F" " "cmdwh_bq_prod.testtabPIPE" "c:\temp\testPIPE.txt" PlatformVersion:int64,AnalyticsSessionID:int64,OutletGroup:string Upload complete. ..... rest of the processing...... ..... rest

问题 I have a BigQuery dataset located in the new "asia-northeast1" region. I'm trying to run a Dataflow templated pipeline (running in Australia region) to read a table from it. It chucks the following error, even though the dataset/table does indeed exist: Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found { "code" : 404, "errors" : [ { "domain" : "global", "message" : "Not found: Dataset grey-sort-challenge:Konnichiwa_Tokyo", "reason" : "notFound" } ],