In this light, here is a comparison of Open Source NOSQL databases Cassandra, Mongodb, CouchDB, Redis, Riak, RethinkDB, Couchbase (ex-Membase), Hypertable, ElasticSearch, Accumulo, VoltDB, Kyoto Tycoon, Scalaris, OrientDB, Aerospike, Neo4j and HBase:
https://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis/
esrally is a benchmark tool for elasticsearch, https://github.com/elastic/rally
Testing Script:
esrally --pipeline=benchmark-only --target-hosts=server1:9200,server2:9200,server3:9200,server4:9200,server5:9200
Server status in Kibana dashboard:
P.S. Please benchmark your ES cluster based on your real business scenario & cluster topology
We konw X-Pack is is an extension that bundles security, monitoring, reporting, and graph capabilities into one package.
From ELK stack 6.3, X-Pack is integrated into Elasticsearch, you can try it by 30-day-trial license. After the trail you could choose buy the license or downgrade to the normal license.
1.Down & Unzip & Enter
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz
tar -zvxf elasticsearch-6.3.2.tar.gz && cd elasticsearch-6.3.2
2.Enable monitor & Start
echo "xpack.monitoring.enabled: true" >>config/elasticsearch.yml
echo "xpack.security.enabled: true" >>config/elasticsearch.yml
echo "xpack.watcher.enabled: true" >>config/elasticsearch.yml
echo "xpack.ml.enabled: true" >>config/elasticsearch.yml
echo "xpack.graph.enabled: true" >>config/elasticsearch.yml
echo "xpack.monitoring.collection.enabled: true" >>config/elasticsearch.yml
bin/elasticsearch
1.Download & Unzip & Entry
wget wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-darwin-x86_64.tar.gz
tar -zvxf kibana-6.3.2-darwin-x86_64.tar.gz && cd kibana-6.3.2-darwin-x86_64
2.Start
bin/kibana
3.Visit kibana, http://localhost:5601, you will see the dashboard, next we will enable 30-day-trail license.
3.1.Click Management on left menu
3.2.Click License Management
3.3.Click Start trial button
3.4.Click Start my trial button
3.5.Start trail license done
1.Enter elasticsearch directory, execute command as below to generate password for users, please choose one
bin/elasticsearch-setup-passwords interactive(you need to enter password for every user)
bin/elasticsearch-setup-passwords auto(will generate password for users automatically)
2.Enter kibana directory, stop kibana and set username/password in kibana.yml, then start kibana
echo "elasticsearch.username: kibana" >>config/kibana.yml
echo "elasticsearch.password: kibana123" >>config/kibana.yml
bin/kibana
3.After finishing all the settings, you will see the login page
4.Enter the username/password which you set in kibana.yml, then you could login success
Recently I need do some statistic, so I choose ELK to build it. I will introduce about how to clean logs and send the logs to elasticsearch by logstash.
Logstash Version: 5.6.6
Firstly add a new configuration file named xxx.conf under config directory, the content are as below, please replace “xxx” with your business.
input {
file {
path => "/**/xxx.log"
codec => plain {
charset => "UTF-8"
}
tags => ["xxx"]
}
file {
path => "/**/xxx.log"
codec => plain {
charset => "UTF-8"
}
tags => ["xxx"]
}
}
filter {
if "xxx" in [tags] {
dissect {
mapping => {
"message" => "%{timestamp} - [%{thread}] - [%{level}] - [%{class}] - xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}"
}
}
}
if "xxx" in [tags] {
dissect {
mapping => {
"message" => "%{timestamp} - [%{thread}] - [%{level}] - [%{class}] - xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}"
}
}
}
}
#replace @timestamp
#filter {
#date {
#match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"]
#target => ["@timestamp"]
#}
#}
output {
if "xxx" in [tags] {
elasticsearch {
index => "xxx"
hosts => ["http://xxx:9200"]
}
}
if "xxx" in [tags] {
elasticsearch {
index => "xxx"
hosts => ["http://xxx:9200"]
}
}
}
Then start logstash with this configuration file
bin/logstash -f config/xxx.conf
After this, please configure kibana dashborad, and you will get some cool charts.
P.S.
There are many kinds of input/filter/output, like jdbc/redis/kafka/mongodb, please refer to the official document.
If you are familiar with grok filter, you can also filter logs as below:
filter {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}-%{SPACE}[.*]%{SPACE}-%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx}"
}
}
}
Wang's Tech Blog 
Reply