In this blog post I’m going to detail all the steps needed to load data from a CSV file into the Elastic Stack, so we can explore it, make visualizations, augment the original data and use more advanced techniques such as machine learning in order to find some possible patterns in the data.
How do we match thousands of documents against a dynamic whitelist/blacklist in Elasticsearch?
It is useful sometimes to have day of week and day of month in fields that are separate from the
@timestamp so we can make aggregations or even machine learning jobs to find a potential correlation between your events and weekdays.
A common error people face when putting an Elasticsearch cluster to production has to do with memory locking. Tipically users would see errors like “Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out. Increase RLIMIT_MEMLOCK (ulimit)” or “memory locking requested for elasticsearch process but memory is not locked”.