S3 Select CSV Headers

Question

I am using S3 Select to read csv file from S3 Bucket and outputting as CSV. In the output I only see rows, but not headers. How do I get output with headers included.

<

Answer 1

Red Boy's solution doesn't allow you to use the column names in the query and instead, you have to use the column indexes. This wasn't good for me so my solution was to do another query to only get the headers and concatenate them with the actual query result. This is on JavaScript but the same should apply to Python:

      const params = {
        Bucket: bucket,
        Key: "file.csv",
        ExpressionType: 'SQL',
        Expression: `select * from s3object s where s."date" >= '${fromDate}'`,
        InputSerialization: {'CSV': {"FileHeaderInfo": "USE"}},
        OutputSerialization: {'CSV': {}},
      };

      //s3 select doesn't return the headers, so need to run another query to only get the headers (see '{"FileHeaderInfo": "NONE"}')
      const headerParams = {
        Bucket: bucket,
        Key: "file.csv",
        ExpressionType: 'SQL',
        Expression: "select * from s3object s limit 1", //this will only get the first record of the csv, and since we are not parsing headers, they will be included
        InputSerialization: {'CSV': {"FileHeaderInfo": "NONE"}},
        OutputSerialization: {'CSV': {}},
      };

      //concatenate header + data -- getObject is a method that handles the request
      return await this.getObject(s3, headerParams) + await this.getObject(s3, params);