How to Log to Elastic Search by NLog or SeriLog with authentications

Question

We are currently use azure scale set (many VMs on one source group with load balance and one availability set), we used to use NLog to log our web app action and errors, but now we asked/needs to use Elastic Search and also use centralized log for all azure vm instances instead of file per each instance.

I am new to ES and LogStash concepts, Do I need to replace NLog with something else? and How I move to use ES and unify all logs in one (I think to make nlog store in azure storage table as unify results or do I needs to use LogStash or you prefer something else)?

What is the most logging that give support for .net core app on azure multi VMs as described above?

Any help please?

Answer 1

Since you want to log into ElasticSearch, it's best if you use a Logging framework which supports Structured Logging which for NLog this feature is in beta RTM.

You can Serilog as your Logging Framework which supports Structured Logging.

And there's also Serilog sink for Elasticsearch https://github.com/serilog/serilog-sinks-elasticsearch

You should add these nuget packages to your project :

Serilog
Serilog.Sinks.ElasticSearch  

This is a sample code for config of serilog to sink to Elasticsearch

var logger = new LoggerConfiguration()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
    {
        ModifyConnectionSettings = x => x.SetBasicAuthentication(username, password);
    })
    .CreateLogger();

Answer 2

For NLog there is a target "NLog.Targets.ElasticSearch" (nuget) which uses the Elasticsearch.Net package.

Usage:

<nlog>
  <extensions>
    <add assembly="NLog.Targets.ElasticSearch"/>
  </extensions>
  <targets>
    <target name="elastic" xsi:type="BufferingWrapper" flushTimeout="5000">
      <target xsi:type="ElasticSearch"  
         requireAuth="true"
         username="myUserName"
         password="coolpassword"/>
    </target>
  </targets>
  <rules>
    <logger name="*" minlevel="Info" writeTo="elastic" />
  </rules>
</nlog>

Docs for the parameters: https://github.com/ReactiveMarkets/NLog.Targets.ElasticSearch/wiki

Please note:

• If you need to use Elasticsearch.Net 6 (equivalent to Elastic Search version 6 or later), you need NLog.Targets.ElasticSearch version 5.
• For Elasticsearch.Net 5 you need to use NLog.Targets.ElasticSearch 4

Answer 3

Many recommends that the application should not write directly to ElasticSearch, but should just write to local files.

Then have a service (Ex. FileBeat) to upload the contents of the log-files into ElasticSearch.

This will optimize network traffic to the ElasticSearch instance (bulk), and will ensure logging is not lost if problems with the network or ElasticSearch instance is restarted because of maintenance.