logstash

问题 I am struggling with data manipulation in logstash version 5.1, where some of the data come from open text fields with HTML markups. Most of the time it comes with only one marker, like this: <.p> XYZ <./p> and I am dealing with it using Grok. but when it comes like this: <.p><.b><.strong> XYZ <./strong><./b><./p> simple grok cant filter it out. My question is if there is a built-in filter for HTML markup or do i have to develop my own using regular expressions? Or do you know if in versions

问题 I have a logstash pipeline that fetches data from mysql using jdbc input connecter aggregates data for users based on user id pushes aggregated data to elasticsearch cluster It fetches large amount of data (e.g 2 million rows) from mysql server and uses cursor fetch with "jdbc_fetch_size" of 100000 so that it does not load all the rows at once (not page size, limit + offset but fetch_size) to avoid out of memory exception. Below is my configuration: input { jdbc { jdbc_driver_class => "com

问题 I'm trying to request database with logstash jdbc plugins and returns a csv output file with headers with logstash csv plugin. I spent a lot of time on logstash documentation but I'm still missing a point. With the following logstash configuration, the results give me a file with headers for each row. I couldn't find a way to add the headers for only the first row in the logstash configuration. Helps very much appreciated. Output file _object$id;_object$name;_object$type;nb_surveys;csat_score

问题 I am exploring Logstash to receive inputs on HTTP. I have installed http plugin using: plugin install logstash-input-http The installation was successfull. Then I tried to run logstash using following command: logstash -e 'input {http {port => 8900}} output {stdout{codec => rubydebug}}' But logstash terminates without giving any error as such. Don't know how to verify whether plugin is installed correctly or not. And how to utilize the http plugin to test a sample request. Thanks in Advance!

问题 I am using below Logstash configuration for doing incremental indexing in this whenever a new row inserted or updated I am able to get those particular rows from MSSQL server and insert it as a document in elasticsearch but the challenge is with delete operation. Logstash configuration file input { jdbc { jdbc_driver_library => "" jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc_connection_string => "jdbc:sqlserver://xxxxx;databaseName=xxxx;" jdbc_user => "xxxx" jdbc

问题 I am using logstash with input jdbc, and would like to embed one object inside another with aggregate. How can I use add recursive? Ie add an object inside another object? This would be an example: { "_index": "my-index", "_type": "test", "_id": "1", "_version": 1, "_score": 1, "_source": { "id": "1", "properties": { "nested_1": [ { "A": 0, "B": "true", "C": "PEREZ, MATIAS ROGELIO Y/O", "Nested_2": [ { "Z1": "true", "Z2": "99999" } }, { "A": 0, "B": "true", "C": "SALVADOR MATIAS ROMERO",

问题 I am looking for migrating the data from elastic search version 5.5 to 7.7, I found the following two ways, 1) Re_Index API using Java API : https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-high-document-reindex.html#java-rest-high-document-reindex which looks easy (as it involves java code) and able to see all the possible cases for moving all the documents from version 5.5 to 7.7 2) Re_Index API using Curl : https://www.elastic.co/guide/en/elasticsearch

问题 I have 3m records. Headers are value, type, other_fields .. Here I need to load the data as in this I need to specify type as context for that value in the record. Is there any way to do this with log stash? or any other options? val,val_type,id Sunnyvale it labs, seller, 10223667 回答1: For this, I'd use the new CSV ingest processor First create the ingest pipeline to parse your CSV data PUT _ingest/pipeline/csv-parser { "processors": [ { "csv": { "field": "message", "target_fields": [ "val",

问题 This may be a simple question, but in my logs the spaces between different fields are uncertain, that mean in some logs I can see two spaces and in some three between the same fields. How do we accommodate this in GROK? 回答1: Grok is at it's heart an overlay on Regex's. So in your grok pattern, you can directly use Regex syntax: %{WORD} +%{WORD} So "space+" means one or more spaces. "space*" means 0 or more spaces. Grok also has a pattern %{SPACE} that is equivilent to " *" 回答2: You can use %

问题 This may be a simple question, but in my logs the spaces between different fields are uncertain, that mean in some logs I can see two spaces and in some three between the same fields. How do we accommodate this in GROK? 回答1: Grok is at it's heart an overlay on Regex's. So in your grok pattern, you can directly use Regex syntax: %{WORD} +%{WORD} So "space+" means one or more spaces. "space*" means 0 or more spaces. Grok also has a pattern %{SPACE} that is equivilent to " *" 回答2: You can use %