Avoid creation of _$folder$ keys in S3 with hadoop (EMR)

Question

I am using an EMR Activity in AWS data pipeline. This EMR Activity is running a hive script in EMR Cluster. It takes dynamo DB as input and stores data in S3.

This is the EMR step used in EMR Activity

s3://elasticmapreduce/libs/script-runner/script-runner.jar,s3://elasticmapreduce/libs/hive/hive-script,--run-hive-script,--hive-versions,latest,--args,-f,s3://my-s3-bucket/hive/my_hive_script.q,-d,DYNAMODB_INPUT_TABLE1=MyTable,-d,S3_OUTPUT_BUCKET=#{output.directoryPath}

where

out.direcoryPath is :

s3://my-s3-bucket/output/#{format(@scheduledStartTime,"YYYY-MM-dd")}

So this creates one folder and one file in S3. (technically speaking it creates two keys 2017-03-18/<some_random_number> and 2017-03-18_$folder$)

2017-03-18
2017-03-18_$folder$

How to avoid creation of these extra empty _$folder$ files.

EDIT: I found a solution listed at https://issues.apache.org/jira/browse/HADOOP-10400 but I don't know how to implement it in AWS data pipeline.

Answer 1

EMR doesn't seem to provide a way to avoid this.

Because S3 uses a key-value pair storage system, the Hadoop file system implements directory support in S3 by creating empty files with the "_$folder$" suffix.

You can safely delete any empty files with the <directoryname>_$folder$ suffix that appear in your S3 buckets. These empty files are created by the Hadoop framework at runtime, but Hadoop is designed to process data even if these empty files are removed.

https://aws.amazon.com/premiumsupport/knowledge-center/emr-s3-empty-files/

It's in the Hadoop source code, so it could be fixed, but apparently it's not fixed in EMR.

If you are feeling clever, you could create an S3 event notification that matches the _$folder$ suffix, and have it fire off a Lambda function to delete the objects after they're created.

Answer 2

There's no way in S3 to actually create an empty folder. S3 is an object store so everything is an object in there. When Hadoop uses it as a filesystem, it requires to organize those objects so that it appears as a file system tree, so it creates some special objects to mark an object as a directory. You just store data files, but you can choose to organize those data files into paths, which creates a concept similar to folders for traversing.

If you just don't create a folder, but place files in the path you want - that should work for you. You don't have to create a folder before writing files to it in S3.

Also this may help: https://qubole.zendesk.com/hc/en-us/articles/213496246-How-To-Remove-Dir-marker-folders-in-S3-NativeFS-

Answer 3

Use below script in EMR bootstrap action to solve this issue. Patch provided by AWS

#!/bin/bash

# NOTE: This script replaces the s3-dist-cp RPM on EMR versions 4.6.0+ with s3-dist-cp-2.2.0.
# This is intended to remove the _$folder$ markers when creating the destination prefixes in S3.

set -ex

RPM=bootstrap-actions/s3-dist-cp-2.2.0/s3-dist-cp-2.2.0-1.amzn1.noarch.rpm

LOCAL_DIR=/var/aws/emr/packages/bigtop/s3-dist-cp/noarch

# Get the region from metadata
REGION=$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone/ 2>/dev/null | head -c -1)

# Choose correct bucket for region
if [ $REGION = "us-east-1" ]
then
    BUCKET=awssupportdatasvcs.com
else
    BUCKET=$REGION.awssupportdatasvcs.com
fi

# Download new RPM
sudo rm $LOCAL_DIR/s3-dist-cp*.rpm
aws s3 cp s3://$BUCKET/$RPM /tmp/
sudo cp /tmp/s3-dist-cp-2.2.0-1.amzn1.noarch.rpm $LOCAL_DIR/

echo Rebuilding Repo
sudo yum install -y createrepo
sudo createrepo --update -o /var/aws/emr/packages/bigtop /var/aws/emr/packages/bigtop
sudo yum clean all