How should I use sql_last_value in logstash?

Question

I\'m quite unclear of what sql_last_value does when I give my statement as such:

statement => \"SELECT * from mytable where id > :sql_last         


        

Answer 1

There are a few things to take care of:

1. If you have run Logstash earlier without the schedule, then before running Logstash with schedule, delete the file:

   $HOME/.logstash_jdbc_last_run
   

   In Windows, this file is found at:

   C:\Users\\.logstash_jdbc_last_run
   

2. The "statement =>" in Logstash config should have "order by" the tracking_column.

3. tracking_column should be given correctly.

Here is an example of the Logstash config file:

    input {
jdbc {
    # MySQL DB jdbc connection string to our database, softwaredevelopercentral
    jdbc_connection_string => "jdbc:mysql://localhost:3306/softwaredevelopercentral?autoReconnect=true&useSSL=false"
    # The user we wish to execute our statement as
    jdbc_user => "root"
    # The user password
    jdbc_password => ""
    # The path to our downloaded jdbc driver
    jdbc_driver_library => "D:\Programs\MySQLJava\mysql-connector-java-6.0.6.jar"
    # The name of the driver class for MySQL DB
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    # our query
    schedule => "* * * * *"
    statement => "SELECT * FROM student WHERE studentid > :sql_last_value order by studentid"
    use_column_value => true
    tracking_column => "studentid"
}
}
output {
stdout { codec => json_lines }
elasticsearch { 
   hosts => ["localhost:9200"]
   index => "students"
   document_type => "student"
   document_id => "%{studentid}"
   }

}

To see a working example of the same you can check my blog post: http://softwaredevelopercentral.blogspot.com/2017/10/elasticsearch-logstash-kibana-tutorial.html