Query by “$path” field

问题

I want to query by a file / group of files under a partition inside a table. I found out that when I'm using the "$path" field Athena scans the entire partition, and not the files I want

Is there a way to make this kind of query more efficient and scan only the given files? Something like partition pruning for files...

Here is a sample query:

SELECT *
FROM my_table
WHERE day = '2019-01-01'
      AND "$path" = 's3://my-bucket/my-table/day=2019-01-01/my_file'

回答1:

No. It's not possible to get Athena to scan only the file you want by using $path, or any other method that I know of, without partitioning your table differently.

If this is a common operation I suggest making your partitions smaller and match the files better, but if it's just something you do once in a while I wouldn't worry too much about it.

If you have multiple access patterns, and this isn't the primary, but still not uncommon pattern, you can create a separate table using the org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat input format, and create a 1:1 structure of partitions with symlink.txt files pointing to the files of the original table. You can read more about this way of creating tables in this StackOverflow answer (the second half) – but I think it will be a very complicated way to solve it.

来源：https://stackoverflow.com/questions/55795031/query-by-path-field