ElasticSearch river JDBC MySQL not deleting records

Question

I\'m using the JDBC plugin for ElasticSearch to update my MySQL database. It picks up new and changed records, but does not delete records that have been removed from MySQL.

Answer 1

i am still relatively new to elastic and had been using jdbc river for my project. If i understood correctly, which not necessarily could be the case, this is how it works:

1. Fetches all rows (specified by the SQL statement in the river) from the database.
2. calculates a digest from (id,type and index) of all the fetched rows (if new rows were added or rows were deleted this should change).
3. for all rows re-index the documents. This will automatically increment the version of each document.
4. increment version of the river stored in the _river index (custom)
5. if the calculated digest in #3 is different than the one that is stored in the _river index then:
   • store it
   • run housekeeping function (deletes all docs with lower version numbers).

so considering that you would want to have a housekeeping running you need to have versioning to be set to true and subsequently this implies that digesting should be set to true as well.

So having said that your river should look like this:

curl -XPUT 'localhost:9200/_river/account_river/_meta' -d '{
    "type" : "jdbc",
    "jdbc" : {
        "driver" : "com.mysql.jdbc.Driver",
        "url" : "jdbc:mysql://localhost:3306/test",
        "user" : "test_user",
        "password" : "test_pass",
        "sql" : "SELECT `account`.`id` as `_id`, `account`.`id`, `account`.`reference`, `account`.`company_name`, `account`.`also_known_as` from `account` WHERE NOT `account`.`deleted`",
        "strategy" : "simple",
        "poll" : "5s",
        "autocommit" : true,
        "index": {
          "index" : "headphones",
          "type" : "Account",
          "versioning" : true,
          "digesting" : true
        }
    }
}'

note that versioning and digesting should be part of index definition and not jdbc definition