amazon-s3

问题 This question already has answers here : Loading JSON data to AWS Redshift results in NULL values (3 answers) Closed 2 years ago . I am loading data from S3 into Redshift using the COPY command, the gzip flag and the 'auto' format, as per this documentation on loading from S3, this documentation for using the 'auto' format in AWS, and this documentation for addressing compressed files. My data is a highly nested JSON format, and I have created the redshift table such that the column names

问题 This question already has answers here : Loading JSON data to AWS Redshift results in NULL values (3 answers) Closed 2 years ago . I am loading data from S3 into Redshift using the COPY command, the gzip flag and the 'auto' format, as per this documentation on loading from S3, this documentation for using the 'auto' format in AWS, and this documentation for addressing compressed files. My data is a highly nested JSON format, and I have created the redshift table such that the column names

问题 I'm using this code to create an authenticated Amazon Url http://developer.longtailvideo.com/trac/browser/testing/files/s3-generator.php Is it possible to somehow let Amazon know that the file needs to be force-downloaded by adding a Content-Disposition header? 回答1: After gathering the individual pieces of this puzzle, I was able to create this Ruby method that properly signs a query string url using the aws secret key. My resources for this: RESTObjectGET documentation Signing and

问题 I'm using Amazon EMR. I have some log data in s3, all in the same bucket, but under different subdirectories like: "s3://bucketname/2014/08/01/abc/file1.bz" "s3://bucketname/2014/08/01/abc/file2.bz" "s3://bucketname/2014/08/01/xyz/file1.bz" "s3://bucketname/2014/08/01/xyz/file3.bz" I'm using : Set hive.mapred.supports.subdirectories=true; Set mapred.input.dir.recursive=true; When trying to load all data from "s3://bucketname/2014/08/": CREATE EXTERNAL TABLE table1(id string, at string, custom

问题 I'm using Amazon EMR. I have some log data in s3, all in the same bucket, but under different subdirectories like: "s3://bucketname/2014/08/01/abc/file1.bz" "s3://bucketname/2014/08/01/abc/file2.bz" "s3://bucketname/2014/08/01/xyz/file1.bz" "s3://bucketname/2014/08/01/xyz/file3.bz" I'm using : Set hive.mapred.supports.subdirectories=true; Set mapred.input.dir.recursive=true; When trying to load all data from "s3://bucketname/2014/08/": CREATE EXTERNAL TABLE table1(id string, at string, custom

问题 I have a large number of files that have incorrect mimetypes in a bucket, as well as no expires set. How can I change them all? I'm using Knox: https://github.com/LearnBoost/knox I'm trying to iterate over it. How do I get a list of all files in a folder? When I do this client.get('/folder').on('response', function(res){ console.log(res) res.on('data', function(chunk){ console.log(chunk); }); }).end(); I see osmething about an xml file, how do I access it? 回答1: It looks like the library you

问题 I implemented spark application. I've created spark context: private JavaSparkContext createJavaSparkContext() { SparkConf conf = new SparkConf(); conf.setAppName("test"); if (conf.get("spark.master", null) == null) { conf.setMaster("local[4]"); } conf.set("fs.s3a.awsAccessKeyId", getCredentialConfig().getS3Key()); conf.set("fs.s3a.awsSecretAccessKey", getCredentialConfig().getS3Secret()); conf.set("fs.s3a.endpoint", getCredentialConfig().getS3Endpoint()); return new JavaSparkContext(conf); }

问题 I implemented spark application. I've created spark context: private JavaSparkContext createJavaSparkContext() { SparkConf conf = new SparkConf(); conf.setAppName("test"); if (conf.get("spark.master", null) == null) { conf.setMaster("local[4]"); } conf.set("fs.s3a.awsAccessKeyId", getCredentialConfig().getS3Key()); conf.set("fs.s3a.awsSecretAccessKey", getCredentialConfig().getS3Secret()); conf.set("fs.s3a.endpoint", getCredentialConfig().getS3Endpoint()); return new JavaSparkContext(conf); }

问题 In version 1 of their SDK, Amazon provided some really useful methods that could be used to explore the contents of buckets using Tree, ChildCollection, LeafNode, BranchNode, etc. Unfortunately, I've had a difficulty time replicating their functionality with version 2 of the SDK, which doesn't seem to include such methods. Ideally, I'd like to do something similar to the example below, which is taken from the v1 SDK. tree = bucket.as_tree directories = tree.children.select(&:branch?).collect(

问题 I would like to use Amazon's .NET SDK for accessing a publicly readable bucket, but the AWSClientFactory.CreateAmazonS3Client method for creating an AmazonS3 forces you to input your AwsAccessKey and AwsSecretAccessKey. I do not want to include these in my client-side application, but I still want to use the SDK provided by Amazon to access these publicly readable objects on S3. Is this possible? 回答1: I also asked this question on the AWS forum and an official Amazon employee responded: As of