google-cloud-dataprep

问题 I am using Dataprep within Google Cloud Console. I am trying to export my recipe as a CSV. Export settings are the following: The issue I am facing is that the final result is showing a replicate header, like the following, without any clear reason since the header should be present just once: Any idea why this is happening? Any help would be much appreciated :-) Marco 回答1: It's actually a bug. Trifacta is already on it: Slack conversation: 来源： https://stackoverflow.com/questions/63305077

问题 Google Cloud Dataprep seems great and we've used it to manually import static datasets, however I would like to execute it more than once so that it can consume new files uploaded to a GCS path. I can see that you can setup a schedule for Dataprep, but I cannot see anywhere in the import setup how it would process new files. Is this possible? Seems like an obvious need - hopefully I've missed something obvious. 回答1: You can add a GCS path as a dataset by clicking on the + icon left of the

问题 Google Cloud Dataprep seems great and we've used it to manually import static datasets, however I would like to execute it more than once so that it can consume new files uploaded to a GCS path. I can see that you can setup a schedule for Dataprep, but I cannot see anywhere in the import setup how it would process new files. Is this possible? Seems like an obvious need - hopefully I've missed something obvious. 回答1: You can add a GCS path as a dataset by clicking on the + icon left of the

问题 Google Cloud Dataprep seems great and we've used it to manually import static datasets, however I would like to execute it more than once so that it can consume new files uploaded to a GCS path. I can see that you can setup a schedule for Dataprep, but I cannot see anywhere in the import setup how it would process new files. Is this possible? Seems like an obvious need - hopefully I've missed something obvious. 回答1: You can add a GCS path as a dataset by clicking on the + icon left of the

问题 Does the Import/Export Flow option only work for the same project the original flow comes from? Having exported a flow from the flows page, I can't seem to import it into another account Thanks 回答1: To import a for you need to click on the 3 dot next to the "create a flow" button and chose the zip file that was created when you exported a flow. When you say you can't seem to import into and other account, can you be more specific? do you get a error message? If so you may encounter a bug

问题 I have a CSV file of the following format that I am trying to wrangle with GCP dataprep. Timestamp Tag Value 2018-05-01 09:00:00 Temperature 40.1 2018-05-01 09:00:00 Humidity 80 2018-05-01 09:05:00 Temperature 40.2 2018-05-01 09:05:00 Humidity 80 2018-05-01 09:10:00 Temperature 40.0 2018-05-01 09:10:00 Humidity 82 The data extends in 5 minutes interval for 2 weeks. I would like to transform it such that at each 10 minute interval, I am displaying the average(or min/max/median) of the previous

问题 I've designed a data transformation in Dataprep and am now attempting to run it by using the template in Dataflow. My flow has several inputs and outputs - the dataflow template provides them as a json object with key/value pairs for each input & location. They look like this (line breaks added for easy reading): { "location1": "project:bq_dataset.bq_table1", #... "location10": "project:bq_dataset.bq_table10", "location17": "project:bq_dataset.bq_table17" } I have 17 inputs (mostly lookups)

问题 I've created and run a DataPrep job, and am trying to use the template from python on app engine. I can successfully start a job using gcloud dataflow jobs run --parameters "inputLocations={\"location1\":\"gs://bucket/folder/*\"}, outputLocations={\"location1\":\"project:dataset.table\"}, customGcsTempLocation=gs://bucket/DataPrep-beta/temp" --gcs-location gs://bucket/DataPrep-beta/temp/cloud-dataprep-templatename_template however trying to use python on app engine; service = build('dataflow'

问题 Both Dataprep and Dataflow can be used for ETL tasks. In fact Dataprep seems to use Dataflow jobs. Is it that the only difference that Dataprep provides tools to write dataflow jobs with a user interface ? 回答1: Both dataflow and dataprep can transform data for sure. The main difference is who is using the technology. Does your project need self-service data transformation by data users such as data engineers or business users such as analysts and data scientists? Then dataprep is the choice.

问题 I have quite a long set of transforms, which I'd like to break into modules (each in it's own flow). I can't see a way of chaining these, other than scheduling consecutive timeslots. Has anyone managed this, or do I need to build one massive flow? 回答1: In your flow page, click on the three dots ... near your recipe. In the menu, you have Create Reference Dataset . Once created, you will see a new logo under your recipe, you can then click on the menu and choose Add to flow 来源： https:/