Logstash SQL Server Data Import

Answer 1

input {
  jdbc {
    jdbc_driver_library => "C:\Program Files\Microsoft JDBC Driver 6.0 for SQL Server\sqljdbc_6.0\enu\jre8\sqljdbc42.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://[SERVER NAME];databaseName=[DATABASE NAME];"
    jdbc_user => "[USERNAME]"
    jdbc_password => "[PASSWORD]"
    statement => "SELECT eventId, sessionId FROM Events;"
  }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "events3"
  }
  stdout { codec => rubydebug }
}

You need to download sqljdbc drivers from https://www.microsoft.com/en-au/download/details.aspx?id=11774 and wherever you will unzip those drivers just give that path in jdbc_driver_library. Try to unzip those drivers into same path as shown in code.

Answer 2

Do it like this:-

input {
  jdbc {
    jdbc_driver_library => "sqljdbc4.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://192.168.2.126:1433;databaseName=test
    jdbc_password => "sa@sa2015"
    schedule => "0 0-59 0-23 * * *"
    statement => "SELECT ID , Name, City, State,ShopName FROM dbo.Shops"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "50000"
  }
}
filter {
}
output {
  stdout { codec => rubydebug }
    elasticsearch { 
        protocol => "http"
        index => "shops"
        document_id => "%{id}"
        hosts => "your_host_here"

    }
}

Answer 3

I think that path to the "sqljdbc4.jar" file is not correct. Here is the config I am using to query data from a sql db into elasticsearch (logstash.conf):

input {
  jdbc {
    jdbc_driver_library => "D:\temp\sqljdbc\sqljdbc_4.2\enu\sqljdbc42.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://DBSVR_NAME;user=****;password=****;"
    jdbc_user => "****"
    jdbc_password => "****"
    statement => "SELECT *
FROM [DB].[SCHEMA].[TABLE]"
  }
}
filter {
}
output {
  elasticsearch {
    hosts => "localhost"
    index => "INDEX_NAME"
    document_type => "DOCUMENT_TYPE"
    document_id => "%{id}"
    protocol => "http"
  }
  stdout { codec => rubydebug }
}

I downloaded the Microsoft JDBC Driver for SQL Server from here: "https://msdn.microsoft.com/en-us/sqlserver/aa937724.aspx"

Extracted the files to the path specified in "jdbc_driver_library"

Then I ran the plugin command: "plugin install logstash-input-jdbc" to install the logstash input jdbc plugin.

And finally running logstash: "logstash -f logstash.conf".

As an aside: I am also using Elasticsearch.Net in a .Net service app to refresh the data "http://nest.azurewebsites.net/"

And this vid: "Adding Elasticsearch To An Existing .NET / SQL Server Application" "https://www.youtube.com/watch?v=sv-MflnT9qI" discuses using a Service Broker queue to get the data out of sql. We are currently exploring this as an option.

Edit - Updated host to hosts as in documentation here https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts