Logstash
A Logstash pipeline has two required elements, input and output, and one optional element, filter. The input plugins consume data from a source, the filter plugins modify the data as you specify, and the output plugins write the data to a destination.
Logstash -> Elasticsearch -> Kibana
- In logstash, specify output as elasticsearch and specify the index
- In Kibana UI, specify index
- Kibana will retrieve the data with the correct index.
grok filter
Reading:
- grok pattern syntax
- grok pattern examples
- optional parts in grok pattern
- Look out for your spaces! If there is a space before the optional field and a space after, the filter will only match two spaces.
- If a field does not exist, it will break Kibana. See Available fields not showing in kibana (Issue resolved)