Aggregating over _field_names in elasticsearch 5

Question

I'm trying to aggregate over field names in ES 5 as described in Elasticsearch aggregation on distinct keys But the solution described there is not working anymore.

My goal is to get the keys across all the documents. Mapping is the default one.

Data:

PUT products/product/1
{
    "param": {
        "field1": "data",
        "field2": "data2"
    }   
}

Query:

GET _search
{
    "aggs": {
        "params": {
            "terms": {
                "field": "_field_names",
                "include" : "param.*",   
                "size": 0
            }

        }
    }
}

I get following error: Fielddata is not supported on field [_field_names] of type [_field_names]

Answer 1

After looking around it seems the only way in ES > 5.X to get the unique field names is through the mappings endpoint, and since cannot aggregate on the _field_names you may need to slightly change your data format since the mapping endpoint will return every field regardless of nesting.

My personal problem was getting unique keys for various child/parent documents.

I found if you are prefixing your field names in the format prefix.field when hitting the mapping endpoint it will automatically nest the information for you.

PUT products/product/1
{
    "param.field1": "data",
    "param.field2": "data2",
    "other.field3": "data3"
}   

GET products/product/_mapping
{
    "products": {
        "mappings": {
            "product": {
                "properties": {
                    "other": {
                        "properties": {
                            "field3": {
                                "type": "text",
                                "fields": {
                                    "keyword": {
                                        "type": "keyword",
                                        "ignore_above": 256
                                    }
                                }
                            }
                        }
                    },
                    "param": {
                        "properties": {
                            "field1": {
                                "type": "text",
                                "fields": {
                                    "keyword": {
                                        "type": "keyword",
                                        "ignore_above": 256
                                    }
                                }
                            },
                            "field2": {
                                "type": "text",
                                "fields": {
                                    "keyword": {
                                        "type": "keyword",
                                        "ignore_above": 256
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

Then you can grab the unique fields based on the prefix.

Answer 2

This is probably because setting size: 0 is not allowed anymore in ES 5. You have to set a specific size now.

POST _search
{
    "aggs": {
        "params": {
            "terms": {
                "field": "_field_names",
                "include" : "param.*",   
                "size": 100                <--- change this
            }

        }
    }
}