Getting CloudTrail Logs into Logstash

问题

I am trying to get logs from ClouldTrail into ElasticSearch so that we can see what is going on in our AWS account better.

I have set up both Logstash and ElasticSearch on my machine (Ubuntu 14.04), and can push text from stdin to ElasticSearch. However when I try to use the S3 input nothing is added to ElasticSearch.

Here is the conf file Im using, I have removed my amazon keys

input {
  s3 {
    bucket => 'ko-cloudtrail-log-bucket'
    secret_access_key => ''
    access_key_id => ''
    delete => false
    interval => '60'
    region => 'eu-west-1'
    type => 'CloudTrail'
    codec => cloudtrail {}
    }
}

output {
    stdout {}
    elasticsearch {
            host => '127.0.0.1'
     }
}

I have install the logstash-codec-cloudtrail codec but the documentation is pretty sparse.

I get no errors in my terminal even when running Logstash with -v and nothing is printed to stdout. Is there something I am missing?

回答1:

Here's my cloudtrail input. It works great with one minor issue- it duplicates records. As the prefix indicates, I put the cloudtrail logs at s3://bucketname/cloudtrail, not the root.

The mutations are optional. The eventSource mutation is to make the logs a little more readable, and the ruby ingest_time gives me a date the record showed up in ELK- otherwise, it only has the time of the event. Finally, I drop a very common record that just adds noise to my system.

input {
  s3 {
    bucket => "bucketname"
    delete => false
    interval => 60 # seconds
    prefix => "cloudtrail/"
    type => "cloudtrail"
    codec => "cloudtrail"
    credentials => "/etc/logstash/s3_credentials.ini"
    sincedb_path => "/opt/logstash_cloudtrail/sincedb"
  }
}

filter {
  if [type] == "cloudtrail" {
    mutate {
      gsub => [ "eventSource", "\.amazonaws\.com$", "" ]
      add_field => {
        "document_id" => "%{eventID}"
      }
    }
    if ! [ingest_time] {
      ruby {
        code => "event['ingest_time'] = Time.now.utc.strftime '%FT%TZ'"
      }
    }


    if [eventSource] == "elasticloadbalancing" and [eventName] == "describeInstanceHealth" and [userIdentity.userName] == "secret_username" {
      drop {}
    }
  }
}

The credentials.ini format is explained on the s3 input page; it's just this:

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

来源：https://stackoverflow.com/questions/29819970/getting-cloudtrail-logs-into-logstash