Tagged: Nginx Toggle Comment Threads | Keyboard Shortcuts

  • Wang 22:12 on 2019-02-11 Permalink | Reply
    Tags: , , , , , Nginx, , , ,   

    Guarantee service availability in kubernetes 

    A good service not only provide good functionalities, but also ensure the availability and uptime.

    We reinforce our service from QoS, QPS, Throttling, Scaling, Throughput, Monitoring.

    Qos

    There’re 3 kinds of QoS in kubernetes: Guaranteed, Burstable, BestEffort. We usually use Guaranteed, Burstable for different services.

    #Guaranteed
    resources:
      requests:
        cpu: 1000m
        memory: 4Gi
      limits:
        cpu: 1000m
        memory: 4Gi
    
    #Burstable
    resources:
      requests:
        cpu: 1000m
        memory: 4Gi
      limits:
        cpu: 6000m
        memory: 8Gi
    
    QPS

    We did lots of stress test on APIs by Gatling before we release them, we mainly care about mean response time, std deviation, mean requests/sec, error rate (API Testing Report), during testing we monitor server metrics by Datadog to find out bottlenecks.

    We usually test APIs in two scenarios: internal, external. External testing result is much lower than internal testing because of network latency, network bandwidth and son on.

    Internal testing result

    ================================================================================
    ---- Global Information --------------------------------------------------------
    > request count                                     246000 (OK=246000 KO=0     )
    > min response time                                     16 (OK=16     KO=-     )
    > max response time                                   5891 (OK=5891   KO=-     )
    > mean response time                                    86 (OK=86     KO=-     )
    > std deviation                                        345 (OK=345    KO=-     )
    > response time 50th percentile                         30 (OK=30     KO=-     )
    > response time 75th percentile                         40 (OK=40     KO=-     )
    > response time 95th percentile                         88 (OK=88     KO=-     )
    > response time 99th percentile                       1940 (OK=1940   KO=-     )
    > mean requests/sec                                817.276 (OK=817.276 KO=-     )
    ---- Response Time Distraaibution ------------------------------------------------
    > t < 800 ms                                        240565 ( 98%)
    > 800 ms < t < 1200 ms                                1110 (  0%)
    > t > 1200 ms                                         4325 (  2%)
    > failed                                                 0 (  0%)
    ================================================================================
    

    External testing result

    ================================================================================
    ---- Global Information --------------------------------------------------------
    > request count                                      33000 (OK=32999  KO=1     )
    > min response time                                    477 (OK=477    KO=60001 )
    > max response time                                  60001 (OK=41751  KO=60001 )
    > mean response time                                   600 (OK=599    KO=60001 )
    > std deviation                                        584 (OK=484    KO=0     )
    > response time 50th percentile                        497 (OK=497    KO=60001 )
    > response time 75th percentile                        506 (OK=506    KO=60001 )
    > response time 95th percentile                       1366 (OK=1366   KO=60001 )
    > response time 99th percentile                       2125 (OK=2122   KO=60001 )
    > mean requests/sec                                109.635 (OK=109.631 KO=0.003 )
    ---- Response Time Distribution ------------------------------------------------
    > t < 800 ms                                         29826 ( 90%)
    > 800 ms < t < 1200 ms                                1166 (  4%)
    > t > 1200 ms                                         2007 (  6%)
    > failed                                                 1 (  0%)
    ---- Errors --------------------------------------------------------------------
    > i.g.h.c.i.RequestTimeoutException: Request timeout after 60000      1 (100.0%)
     ms
    ================================================================================
    
    Throttling

    We throttle API by Nginx limit, we configured ingress like this:

    annotations:
      nginx.ingress.kubernetes.io/limit-connections: '30'
      nginx.ingress.kubernetes.io/limit-rps: '60'
    

    And it will generate Nginx configuration dynamically like this:

    limit_conn_zone $limit_ZGVsaXZlcnktY2RuYV9kc2QtYXBpLWNkbmEtZ2F0ZXdheQ zone=xxx_conn:5m;
    limit_req_zone $limit_ZGVsaXZlcnktY2RuYV9kc2QtYXBpLWNkbmEtZ2F0ZXdheQ zone=xxx_rps:5m rate=60r/s;
    
    server {
        server_name xxx.xxx ;
        listen 80;
        
        location ~* "^/xxx/?(?<baseuri>.*)" {
            ...
            ...        
            limit_conn xxx_conn 30;
            limit_req zone=xxx_rps burst=300 nodelay;
            ...
            ...        
    }
    
    Scaling

    We use HPA in kubernetes to ensure auto (Auto scaling in kubernetes), you could check HPA status in server:

    [xxx@xxx ~]$ kubectl get hpa -n test-ns
    NAME       REFERENCE             TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
    api-demo   Deployment/api-demo   39%/30%, 0%/30%   3         10        3          126d
    
    [xxx@xxx ~]$ kubectl get pod -n test-ns
    NAME                           READY     STATUS    RESTARTS   AGE
    api-demo-76b9954f57-6hvzx      1/1       Running   0          126d
    api-demo-76b9954f57-mllsx      1/1       Running   0          126d
    api-demo-76b9954f57-s22k8      1/1       Running   0          126d
    
    
    Throughput & Monitoring

    We integrated Datadog for monitoring(Monitoring by Datadog), we could check detail API metrics from various dashboards.

    Also we could calculate throughout from user, request, request time.

     
  • Wang 21:44 on 2018-11-20 Permalink | Reply
    Tags: , , , , , Nginx,   

    Sticky session in Kubernetes 

    As we know RESTful API is stateless, every request will be forward to backend server by round robin mechanism.

    But in some scenario we need sticky session which means request from one client should be forward to one backend server.

    After checking kubernetes documentation we added some annotations under ingress configuration, and it works well.

    annotations:
      nginx.ingress.kubernetes.io/affinity: "cookie"
      nginx.ingress.kubernetes.io/session-cookie-name: "router"
      nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
    

    If you open Developer Tools in Chrome, you will find the cookie.

     
  • Wang 19:05 on 2018-01-20 Permalink | Reply
    Tags: aliyun, , , Nginx,   

    Proxy Aliyun’s domain to AWS’s server 

    I registed my domain “wanghongmeng.com” on Aliyun, and applied free EC2 server for one year on AWS.

    After building my blog on AWS, I set A parse to the server’s IP of AWS.

    But yesterday I received email from Aliyun which said that my server was not in Aliyun after they checking, it was not allowed, I have to miggrate my blog server to Aliyun, otherwise they will undo my authority number.

    After thinking about this, for saving money(Aliyun is not free for one year), I solved it by the way below:

    1.Set A parse to my friend’s server ip which was bought in Aliyun

    2.Add a piece of configuration in his nginx.conf:

    server {
        listen  80;
        server_name  wanghongmeng.com www.wanghongmeng.com;
    
        location / {
            rewrite ^/(.*)$ https://$server_name/$1 permanent;
        }
    }
    
    server {
        listen 443;
        server_name wanghongmeng.com www.wanghongmeng.com;
        ssl on;
        ssl_certificate "Location of Pem File";
        ssl_certificate_key "Location of Key File";
        ssl_session_timeout 5m;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers "Your Algorithm";
        ssl_session_cache shared:SSL:50m;
        ssl_prefer_server_ciphers on;
    
        location / {
            proxy_pass  http://AWS's IP:443/;
        }
    }
    

    3.Expose 443 port on my AWS, and only accept requests from my friend’s server IP:

    server {
        listen  443;
        
        set $flag 0;
        if ($host = 'www.wanghongmeng.com') {
            set $flag 1;
        }
        if ($host = 'wanghongmeng.com') {
            set $flag 1;
        }
        if ($flag = 0){
            return 403;
        }    
        
        location / {
            allow "My Friend's Server IP";
            proxy_pass  http://blog-ip;
        }
    }
    

    Things done! 😀😀

     
  • Wang 18:29 on 2018-01-13 Permalink | Reply
    Tags: , Nginx,   

    Prevent web site being mirrored 

    I thought something before, when I check nginx’s log, I found a wired hostname.

    After checking, I think our website was mirrored.

    I think they parsed their domain by CNAME to our domain, and we don’t do any host check at that time.

    To prevent being mirrored again, I add host check configuration in nginx.conf

    set $flag 0;
    if ($host = 'www.wanghongmeng.com') {
        set $flag 1;
    }
    if ($host = 'wanghongmeng.com') {
        set $flag 1;
    }
    if ($flag = 0){
        return 403;
    }
    

    By adding this, nginx will check every request to see if it’s from our domain, if not, return 403 response code.

    After this, our website was no longer mirrored again.

    Nginx Version: 1.9.12

     
c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel
%d bloggers like this: