Tagged: ELK Toggle Comment Threads | Keyboard Shortcuts

  • Wang 23:33 on 2021-02-05 Permalink | Reply
    Tags: , ELK, ,   

    In this light here is a comparison of… 

    In this light, here is a comparison of Open Source NOSQL databases CassandraMongodbCouchDBRedisRiakRethinkDBCouchbase (ex-Membase)HypertableElasticSearchAccumuloVoltDBKyoto TycoonScalarisOrientDBAerospikeNeo4j and HBase:

    https://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis/

     
  • Wang 17:03 on 2020-11-11 Permalink | Reply
    Tags: ELK, , ,   

    Open Distro for Elasticsearch

     
  • Wang 20:23 on 2020-03-17 Permalink | Reply
    Tags: , ELK   

    RCA for ES OOM

     
  • Wang 00:14 on 2019-09-04 Permalink | Reply
    Tags: , ELK,   

    Elasticsearch Benchmark By Esrally 

    esrally is a benchmark tool for elasticsearch, https://github.com/elastic/rally

    Testing Script:

    esrally --pipeline=benchmark-only --target-hosts=server1:9200,server2:9200,server3:9200,server4:9200,server5:9200
    

    Server status in Kibana dashboard:

    P.S. Please benchmark your ES cluster based on your real business scenario & cluster topology

     
  • Wang 23:12 on 2018-06-28 Permalink | Reply
    Tags: ELK,   

    [ELK] Enable X-Pack in ELK stack 6.3 

    We konw X-Pack is is an extension that bundles security, monitoring, reporting, and graph capabilities into one package.

    From ELK stack 6.3, X-Pack is integrated into Elasticsearch, you can try it by 30-day-trial license. After the trail you could choose buy the license or downgrade to the normal license.

    ES

    1.Down & Unzip & Enter

    wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz
    tar -zvxf elasticsearch-6.3.2.tar.gz && cd elasticsearch-6.3.2
    

    2.Enable monitor & Start

    echo "xpack.monitoring.enabled: true" >>config/elasticsearch.yml
    echo "xpack.security.enabled: true" >>config/elasticsearch.yml
    echo "xpack.watcher.enabled: true" >>config/elasticsearch.yml
    echo "xpack.ml.enabled: true" >>config/elasticsearch.yml
    echo "xpack.graph.enabled: true" >>config/elasticsearch.yml
    echo "xpack.monitoring.collection.enabled: true" >>config/elasticsearch.yml
    
    bin/elasticsearch
    

    Kibana

    1.Download & Unzip & Entry

    wget wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-darwin-x86_64.tar.gz
    tar -zvxf kibana-6.3.2-darwin-x86_64.tar.gz && cd kibana-6.3.2-darwin-x86_64
    

    2.Start

    bin/kibana
    

    3.Visit kibana, http://localhost:5601, you will see the dashboard, next we will enable 30-day-trail license.

    3.1.Click Management on left menu

    3.2.Click License Management

    3.3.Click Start trial button

    3.4.Click Start my trial button

    3.5.Start trail license done


    X-Pack

    1.Enter elasticsearch directory, execute command as below to generate password for users, please choose one

    bin/elasticsearch-setup-passwords interactive(you need to enter password for every user)
    bin/elasticsearch-setup-passwords auto(will generate password for users automatically)
    

    2.Enter kibana directory, stop kibana and set username/password in kibana.yml, then start kibana

    echo "elasticsearch.username: kibana" >>config/kibana.yml
    echo "elasticsearch.password: kibana123" >>config/kibana.yml
    
    bin/kibana
    

    3.After finishing all the settings, you will see the login page

    4.Enter the username/password which you set in kibana.yml, then you could login success

     
  • Wang 22:39 on 2018-03-18 Permalink | Reply
    Tags: ELK, logstash   

    [ELK] Configure logstash 

    Recently I need do some statistic, so I choose ELK to build it. I will introduce about how to clean logs and send the logs to elasticsearch by logstash.

    Logstash Version: 5.6.6

    Firstly add a new configuration file named xxx.conf under config directory, the content are as below, please replace “xxx” with your business.

    input {
        file {
            path => "/**/xxx.log"
            codec => plain {
                charset => "UTF-8"
            }
            tags => ["xxx"]
        }
        file {
            path => "/**/xxx.log"
            codec => plain {
                charset => "UTF-8"
            }
            tags => ["xxx"]
        }
    }
    
    filter {
        if "xxx" in [tags] {
            dissect {
                mapping => {
                    "message" => "%{timestamp} - [%{thread}] - [%{level}] - [%{class}] - xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}"
                }
            }
        }
        if "xxx" in [tags] {
            dissect {
                mapping => {
                    "message" => "%{timestamp} - [%{thread}] - [%{level}] - [%{class}] - xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}, xxx:%{xxx}"
                }
            }
        }
    }
    
    #replace @timestamp
    #filter {
    #date {
    #match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"]
    #target => ["@timestamp"]
    #}
    #}
    
    output {
        if "xxx" in [tags] {
            elasticsearch {
                index => "xxx"
                hosts => ["http://xxx:9200"]
            }
        }
        if "xxx" in [tags] {
            elasticsearch {
                index => "xxx"
                hosts => ["http://xxx:9200"]
            }
        }
    }
    

    Then start logstash with this configuration file

    bin/logstash -f config/xxx.conf
    

    After this, please configure kibana dashborad, and you will get some cool charts.

    P.S.

    There are many kinds of input/filter/output, like jdbc/redis/kafka/mongodb, please refer to the official document.


    If you are familiar with grok filter, you can also filter logs as below:

    filter {
        grok {
            match => {
                "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}-%{SPACE}[.*]%{SPACE}-%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx},%{SPACE}xxx:%{GREEDYDATA:xxx}"
            }
        }
    }
    
     
c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel
%d bloggers like this: