Tagged: Docker Toggle Comment Threads | Keyboard Shortcuts

  • Wang 19:25 on 2018-08-11 Permalink | Reply
    Tags: , , , Docker, ,   

    Auto scaling in kubernetes 

    When we deploy a API in kubernets we must define replication number for the pod, but as we know there will be high traffic during peak time and we usually can’t estimate service capacity exactly at first time, in this case we must scale our service like creating more pods to share online traffic to avoid service crash down.

    We usually scale service manually before using kubernetes, append more nodes during peak time and destroy nodes when the traffic became smooth.

    In kubernetes there’s a kind of feature called HPA(Horizontal Pod Autoscaler) which could help your scale service automatically. You could specify minimum and maximum replica number in yaml file, HPA will monitor pod’s CPU and Memory by collecting pod’s metric, if HPA found your pod’s metric is over the threshold number which you defined in yaml file, it will create more pods automatically and join the service cluster to load the traffic.

    Here is a simple HPA samle:

    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
      name: hpa-demo
      namespace: test-ns
        app: hpa-demo
        component: api
        apiVersion: apps/v1
        kind: Deployment
        name: hpa-demo
      minReplicas: 3
      maxReplicas: 10
      - type: Resource
          name: memory
          targetAverageUtilization: 75
      - type: Resource
          name: cpu
          targetAverageUtilization: 75

    I defined there’s will be at least 3 replicas for the pod, if the CPU or Memory usage is over 75%, HPA will create at most 10 pods.

    HPA monitor pod’s metric by using metrics-server.

  • Wang 22:03 on 2018-07-30 Permalink | Reply
    Tags: Docker, ,   

    Deploy service by Helm in Kubernets 

    As we know, if you want to deploy a service, you need at first write several yaml files like deployment/service/ingress file and so on.

    Then execute several times kubectl create -f <Yaml File> when you create service, also you need delete several times when you destroy service, It’s a little boring…

    Although you could write all the configurations in just one yaml file, but it’s hard to maintain. For example you can’t define variable which used in many pods, you can’t upgrade or rollback deployment easily..

    By using Helm you will find it’s very easy to solve these problems, just execute one command like helm install <Chart>, then helm will deploy all the pods at meanwhile, you could check deployment’s status by helm list, upgrade service by helm upgrade and so on.

    There’s lots of stable charts in Helm repository, you could also define chart yourself if it doesn’t meet your requirement.

    Here is chart’s structure from Helm official website:

      Chart.yaml          # A YAML file containing information about the chart
      LICENSE             # OPTIONAL: A plain text file containing the license for the chart
      README.md           # OPTIONAL: A human-readable README file
      requirements.yaml   # OPTIONAL: A YAML file listing dependencies for the chart
      values.yaml         # The default configuration values for this chart
      charts/             # A directory containing any charts upon which this chart depends.
      templates/          # A directory of templates that, when combined with values,
                          # will generate valid Kubernetes manifest files.
      templates/NOTES.txt # OPTIONAL: A plain text file containing short usage notes
  • Wang 23:42 on 2018-05-11 Permalink | Reply
    Tags: , , Docker, ,   

    Website down 

    Today I received alert email suddenly which said my blog site went down…😂😂😂

    So I logged in server and checked containers’s status, everything looked fine

    [root@blog xiaowang]# docker stack ps blog
    ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
    qwsjjol3jk2f        blog_mysql.1        mysql:5.7           blog                Running             Running 15 days ago                       
    n9gbil4zcavy        blog_nginx.1        nginx:1.13.8        blog                Running             Running 15 days ago                       
    hg778gcc35vz        blog_wordpress.1    wordpress:4.9.1     blog                Running             Running 15 days ago

    When I checked the port, everything also looked fine

    [root@blog xiaowang]# netstat -tuapn | egrep '80|443'
    tcp6       4      0 :::80                   :::*                    LISTEN      12146/dockerd       
    tcp6       2      0 :::443                  :::*                    LISTEN      12146/dockerd       
    tcp6      74      0 ::1:80                  ::1:47352               CLOSE_WAIT  -                   
    tcp6       3      0 ::1:80                  ::1:47348               CLOSE_WAIT  -                   
    tcp6      74      0 ::1:80                  ::1:47402               CLOSE_WAIT  -                   
    tcp6      78      0 ::1:443                 ::1:56994               CLOSE_WAIT  -                   
    tcp6      78      0 ::1:443                 ::1:56944               CLOSE_WAIT  -                   
    tcp6      74      0 ::1:80                  ::1:47350               CLOSE_WAIT  -

    But when I executed “curl http://localhost, it was blocked, so I guess something wrong with local network.

    After checking I executed “sysctl -w net.ipv4.ip_forward=1” to enable ip forward, and I finally could access the port. So I executed “echo “net.ipv4.ip_forward=1″ >> /etc/sysctl.conf” to make it permanent.

    I’m using google cloud, I guess maybe they have reset the network which I didn’t make it permanent before.

  • Wang 22:13 on 2018-02-21 Permalink | Reply
    Tags: , , , Docker,   

    Manage BDP by ambari 

    It’s boring and complicated to manage bigdata platforms, there are so many softwares need to be installed and coordinated to make them work well together, so I tried ambari to manage them.

    1.run centos7 container

    docker run -dit --name centos7 --privileged --publish 8080:8080 centos:7 /usr/sbin/init

    2.operate container

    2.1.enter container

    docker exec -it centos7 bash

    2.2.update yum and install tools

    yum update -y && yum install -y wget

    2.3.download the ambari repository

    wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/ -O /etc/yum.repos.d/ambari.repo

    2.4.install the ambari

    yum install -y ambari-server
    yum install -y ambari-agent

    2.5.install mysql as metastore, ,create mysql repo under /etc/yum.repos.d

    cat << 'EOF' >/etc/yum.repos.d/mysql.5.7.repo
    name=MySQL 5.7 Community Server

    2.6.install mysql server

    yum install -y mysql-community-server

    2.7.start mysql

    systemctl start mysqld

    2.8.create mysql user && init database

    mysql_default_password=`grep 'temporary password' /var/log/mysqld.log | awk -F ': ' '{print $2}'`
    mysql -u root -p${mysql_default_password} -e "set global validate_password_policy=0; set global validate_password_length=4;" --connect-expired-password
    mysqladmin -u root -p${mysql_default_password} password ${mysql_password}
    mysql -u root -p${mysql_password} -e "create database ambari default charset 'utf8'; flush privileges;"
    mysql -u root -p${mysql_password} -e "grant all privileges on ambari.* to ambari@'' identified by 'ambari'; flush privileges;"
    mysql -u root -p${mysql_password} -e "use ambari; source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;"

    2.9.download mysql driver

    mkdir ${driver_path}
    wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.45/mysql-connector-java-5.1.45.jar -O ${driver_path}/mysql-connector.jar

    2.10.setup ambari server, pay attention to database configuration, need select mysql manually

    ambari-server setup

    2.11.modify ambari database configuration

    echo "server.jdbc.driver.path=${driver_path}/mysql-connector.jar" >> /etc/ambari-server/conf/ambari.properties

    2.12.start ambari

    ambari-server start
    ambari-agent start
    ambari-server setup --jdbc-db=mysql --jdbc-driver=${driver_path}/mysql-connector.jar

    3.login, default accuont: admin/admin


    The above steps are configured on single server,  if you wanna build cluster with several servers, you also need configure ssh key(please google for specific steps, it’s simple) and start ambari-agent on slave servers.

    Below are screenshots of a mini cluster which was built by 4 servers:

  • Wang 20:05 on 2018-01-22 Permalink | Reply
    Tags: Docker,   

    Deploy apps with docker swarm 

    I received alert email that my website crushed down, after checking, I found mysql container is stoped..

    I checked system log, found infos as below:

    Jan 22 18:42:39 ip-172-31-28-84 kernel: Out of memory: Kill process 597 (mysqld) score 226 or sacrifice child
    Jan 22 18:42:39 ip-172-31-28-84 kernel: Killed process 597 (mysqld) total-vm:1128616kB, anon-rss:228980kB, file-rss:0kB, shmem-rss:0kB

    I think the process is killed by kernel for lack of memory, because the server only has 1GB memory ..

    [root@ip-172-31-28-84 log]# free -h
                  total        used        free      shared  buff/cache   available
    Mem:           990M        559M         83M        113M        348M        133M
    Swap:            0B          0B          0B

    I restarted mysql container, and check containers’s status:

    [root@ip-172-31-28-84 log]# docker stats --no-stream
    CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
    9e5a47485105        0.00%               28.66MiB / 990.8MiB   2.89%               90.9MB / 43.2MB     24.9MB / 0B         2
    c9187825cc0c        0.00%               273.8MiB / 990.8MiB   27.63%              3.95GB / 1.02GB     11GB / 2.58MB       11
    628e301d00a1        0.04%               217.9MiB / 990.8MiB   21.99%              10.4MB / 136MB      101MB / 363MB       31

    there is no limitation on resources, so mysql will occupy more memory which caused being killed.

    After thinking about this, I decided deploy by docker swarm which will start container if stoped, and also could restrict resources for every container.

    1.init docker swarm on single server

    docker swarm init

    2.modify blog-compose.yml to support swarm, please follow gist


    3.deploy service

    docker stack deploy -c blog-compose.yml blog

    4.check container status

    [root@ip-172-31-28-84 docker]# docker stack services blog
    ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
    0l68syg6q1bi        blog_nginx          replicated          1/1                 nginx:1.13.8        *:80->80/tcp,*:443->443/tcp
    cx82xalbzdzu        blog_wordpress      replicated          1/1                 wordpress:4.9.1     
    xulj5sbkbapb        blog_mysql          replicated          1/1                 mysql:5.7           

    5.check container stats

    [root@ip-172-31-28-84 docker]# docker stats --no-stream
    CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
    08bc88c00f0c        0.04%               189.7MiB / 250MiB   75.86%              70.5kB / 1.02MB     14MB / 13.9MB       30
    64d37b150392        0.00%               29.02MiB / 50MiB    58.05%              12.6kB / 14.7kB     1.24MB / 0B         2
    f33ecf2c045e        0.00%               92.32MiB / 300MiB   30.77%              1.03MB / 76.8kB     27.8MB / 0B         9

    The memory of each container is restricted, it will not occupy more memory than limitation, I will keep on watching to see if works well.

  • Wang 23:27 on 2018-01-06 Permalink | Reply
    Tags: , , Docker, ,   

    Build blog with Docker/WordPress with https 

    1.install docker

    1.1.update yum

    sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
    name=Docker Repository baseurl=https://yum.dockerproject.org/repo/main/centos/7/ enabled=1 gpgcheck=1 gpgkey=https://yum.dockerproject.org/gpg EOF 

    1.2.install docker

    sudo yum update -y
    sudo yum install -y docker-engine
    sudo curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
    sudo chmod +x /usr/local/bin/docker-compose
    sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

    1.3.start docker

    sudo systemctl enable docker
    sudo systemctl start docker

    2.https/nginx configuration

    2.1.replace certificate

    replace domain.key/chained.pem with your certificate, you could apply free certificate on Let’s Encrypt

    2.2.nginx configuration

    replace wanghongmeng.com with your domain in nginx.conf


    3.1.wordpress initialize

    login http://xxx.com, setup wordpress

    3.2.install https plugin

    install Really Simple SSL plugin, setup whole site covered by https



  • Wang 21:46 on 2018-01-06 Permalink | Reply
    Tags: , , Docker,   

    Build blog with Docker/WordPress 

    1.install docker

    1.1.update yum

    sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
    name=Docker Repository baseurl=https://yum.dockerproject.org/repo/main/centos/7/ enabled=1 gpgcheck=1 gpgkey=https://yum.dockerproject.org/gpg EOF 

    1.2.install docker

    sudo yum update -y
    sudo yum install -y docker-engine
    sudo curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
    sudo chmod +x /usr/local/bin/docker-compose
    sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

    1.3.start docker

    sudo systemctl enable docker
    sudo systemctl start docker

    2.start wordpress by docker-compose

    sudo docker-compose -f blog-compose.yml up -d

    3.test wordpress


    P.S. start container by docker instead of docker-compose

    docker run --name blog-mysql -v /var/lib/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=mysql -e MYSQL_DATABASE=blog -e MYSQL_USER=blog -e=MYSQL_PASSWORD=blog -d mysql:5.7 --character-set-server=utf8 --collation-server=utf8_general_ci
    docker run --name blog-wordpress --link blog-mysql:mysql -e WORDPRESS_DB_USER=blog -e WORDPRESS_DB_PASSWORD=blog -e WORDPRESS_DB_NAME=blog -p 8080:80 -d wordpress:4.9.1
Compose new post
Next post/Next comment
Previous post/Previous comment
Show/Hide comments
Go to top
Go to login
Show/Hide help
shift + esc
%d bloggers like this: