Tagged: stress test Toggle Comment Threads | Keyboard Shortcuts

  • Wang 23:43 on 2020-10-26 Permalink | Reply
    Tags: , , , stress test   

    Kubernetes Failure Stories

     
  • Wang 00:14 on 2019-09-04 Permalink | Reply
    Tags: , , stress test   

    Elasticsearch Benchmark By Esrally 

    esrally is a benchmark tool for elasticsearch, https://github.com/elastic/rally

    Testing Script:

    esrally --pipeline=benchmark-only --target-hosts=server1:9200,server2:9200,server3:9200,server4:9200,server5:9200
    

    Server status in Kibana dashboard:

    P.S. Please benchmark your ES cluster based on your real business scenario & cluster topology

     
  • Wang 22:12 on 2019-02-11 Permalink | Reply
    Tags: , , , , , , , , , stress test   

    Guarantee service availability in kubernetes 

    A good service not only provide good functionalities, but also ensure the availability and uptime.

    We reinforce our service from QoS, QPS, Throttling, Scaling, Throughput, Monitoring.

    Qos

    There’re 3 kinds of QoS in kubernetes: Guaranteed, Burstable, BestEffort. We usually use Guaranteed, Burstable for different services.

    #Guaranteed
    resources:
      requests:
        cpu: 1000m
        memory: 4Gi
      limits:
        cpu: 1000m
        memory: 4Gi
    
    #Burstable
    resources:
      requests:
        cpu: 1000m
        memory: 4Gi
      limits:
        cpu: 6000m
        memory: 8Gi
    
    QPS

    We did lots of stress test on APIs by Gatling before we release them, we mainly care about mean response time, std deviation, mean requests/sec, error rate (API Testing Report), during testing we monitor server metrics by Datadog to find out bottlenecks.

    We usually test APIs in two scenarios: internal, external. External testing result is much lower than internal testing because of network latency, network bandwidth and son on.

    Internal testing result

    ================================================================================
    ---- Global Information --------------------------------------------------------
    > request count                                     246000 (OK=246000 KO=0     )
    > min response time                                     16 (OK=16     KO=-     )
    > max response time                                   5891 (OK=5891   KO=-     )
    > mean response time                                    86 (OK=86     KO=-     )
    > std deviation                                        345 (OK=345    KO=-     )
    > response time 50th percentile                         30 (OK=30     KO=-     )
    > response time 75th percentile                         40 (OK=40     KO=-     )
    > response time 95th percentile                         88 (OK=88     KO=-     )
    > response time 99th percentile                       1940 (OK=1940   KO=-     )
    > mean requests/sec                                817.276 (OK=817.276 KO=-     )
    ---- Response Time Distraaibution ------------------------------------------------
    > t < 800 ms                                        240565 ( 98%)
    > 800 ms < t < 1200 ms                                1110 (  0%)
    > t > 1200 ms                                         4325 (  2%)
    > failed                                                 0 (  0%)
    ================================================================================
    

    External testing result

    ================================================================================
    ---- Global Information --------------------------------------------------------
    > request count                                      33000 (OK=32999  KO=1     )
    > min response time                                    477 (OK=477    KO=60001 )
    > max response time                                  60001 (OK=41751  KO=60001 )
    > mean response time                                   600 (OK=599    KO=60001 )
    > std deviation                                        584 (OK=484    KO=0     )
    > response time 50th percentile                        497 (OK=497    KO=60001 )
    > response time 75th percentile                        506 (OK=506    KO=60001 )
    > response time 95th percentile                       1366 (OK=1366   KO=60001 )
    > response time 99th percentile                       2125 (OK=2122   KO=60001 )
    > mean requests/sec                                109.635 (OK=109.631 KO=0.003 )
    ---- Response Time Distribution ------------------------------------------------
    > t < 800 ms                                         29826 ( 90%)
    > 800 ms < t < 1200 ms                                1166 (  4%)
    > t > 1200 ms                                         2007 (  6%)
    > failed                                                 1 (  0%)
    ---- Errors --------------------------------------------------------------------
    > i.g.h.c.i.RequestTimeoutException: Request timeout after 60000      1 (100.0%)
     ms
    ================================================================================
    
    Throttling

    We throttle API by Nginx limit, we configured ingress like this:

    annotations:
      nginx.ingress.kubernetes.io/limit-connections: '30'
      nginx.ingress.kubernetes.io/limit-rps: '60'
    

    And it will generate Nginx configuration dynamically like this:

    limit_conn_zone $limit_ZGVsaXZlcnktY2RuYV9kc2QtYXBpLWNkbmEtZ2F0ZXdheQ zone=xxx_conn:5m;
    limit_req_zone $limit_ZGVsaXZlcnktY2RuYV9kc2QtYXBpLWNkbmEtZ2F0ZXdheQ zone=xxx_rps:5m rate=60r/s;
    
    server {
        server_name xxx.xxx ;
        listen 80;
        
        location ~* "^/xxx/?(?<baseuri>.*)" {
            ...
            ...        
            limit_conn xxx_conn 30;
            limit_req zone=xxx_rps burst=300 nodelay;
            ...
            ...        
    }
    
    Scaling

    We use HPA in kubernetes to ensure auto (Auto scaling in kubernetes), you could check HPA status in server:

    [xxx@xxx ~]$ kubectl get hpa -n test-ns
    NAME       REFERENCE             TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
    api-demo   Deployment/api-demo   39%/30%, 0%/30%   3         10        3          126d
    
    [xxx@xxx ~]$ kubectl get pod -n test-ns
    NAME                           READY     STATUS    RESTARTS   AGE
    api-demo-76b9954f57-6hvzx      1/1       Running   0          126d
    api-demo-76b9954f57-mllsx      1/1       Running   0          126d
    api-demo-76b9954f57-s22k8      1/1       Running   0          126d
    
    
    Throughput & Monitoring

    We integrated Datadog for monitoring(Monitoring by Datadog), we could check detail API metrics from various dashboards.

    Also we could calculate throughout from user, request, request time.

     
  • Wang 20:16 on 2018-07-04 Permalink | Reply
    Tags: , stress test   

    [Gatling] API test report 

    Recent I did performance test on user API, I simulated sending 246K requests during 300 seconds.

    1.Test Report

    ================================================================================
    ---- Global Information --------------------------------------------------------
    > request count                                     246000 (OK=246000 KO=0     )
    > min response time                                     16 (OK=16     KO=-     )
    > max response time                                   5891 (OK=5891   KO=-     )
    > mean response time                                    86 (OK=86     KO=-     )
    > std deviation                                        345 (OK=345    KO=-     )
    > response time 50th percentile                         30 (OK=30     KO=-     )
    > response time 75th percentile                         40 (OK=40     KO=-     )
    > response time 95th percentile                         88 (OK=88     KO=-     )
    > response time 99th percentile                       1940 (OK=1940   KO=-     )
    > mean requests/sec                                817.276 (OK=817.276 KO=-     )
    ---- Response Time Distribution ------------------------------------------------
    > t < 800 ms                                        240565 ( 98%)
    > 800 ms < t < 1200 ms                                1110 (  0%)
    > t > 1200 ms                                         4325 (  2%)
    > failed                                                 0 (  0%)
    ================================================================================
    
    

    2.Server Status

    I did the test on server which the network latency is quite low, if I did the test from external environment, I think the QPS will decrease several times.

     
  • Wang 19:45 on 2018-06-22 Permalink | Reply
    Tags: , , , scala, stress test   

    [Gatling] Customize your scripts based on scenario 

    Sometimes record function may not fulfill your requirement, in this case you need write scala script yourself according to gatling document.

    Below is a simple sample to test two APIs: employee & health check

    package me.hongmeng.stress.test
    
    import io.gatling.core.Predef._
    import io.gatling.http.Predef._
    
    class SimpleSimulation extends Simulation {
    
      val hostname = "http://localhost:8080"
    
      val httpProtocol = http
        .baseURL(hostname)
        .inferHtmlResources()
        .acceptHeader("*/*")
        .contentTypeHeader("application/json")
        .userAgentHeader("Gatling/2.3.1")
    
      val headers = Map("accept-encoding" -> "gzip, deflate", "cache-control" -> "no-cache")
    
      val employeeScenario = scenario("create_experiment_simulation")
        .exec(
          http("employee")
            .post("/v1/employees")
            .headers(headers)
            .body(
              StringBody("{"name": "xiaowang","major": "software"}")
            )
        )
    
      val healthScenario = scenario("health_simulation")
        .exec(
          http("health_check")
            .get("/actuator/health")
            .headers(headers)
        )
    
      setUp(
          employeeScenario.inject(atOnceUsers(10)),
          healthScenario.inject(atOnceUsers(10))
        ).protocols(httpProtocol)
    }
    
     
  • Wang 21:21 on 2018-06-13 Permalink | Reply
    Tags: , , , stress test   

    [Gatling] Know about stress test tool 

    Gatling is kind of stress test tool to test your app’s performance, for detail infos, please refer to official document.

    Next I will summurize how to install gatling on your Macbook.

    1.Download gatling

    wget https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/2.3.1/gatling-charts-highcharts-bundle-2.3.1-bundle.zip
    

    2.Unzip & Enter

    unzip gatling-charts-highcharts-bundle-2.3.1-bundle.zip & cd gatling-charts-highcharts-bundle-2.3.1-bundle
    

    3.Start record function

    bin/record.sh
    

    After starting record script, you could see the UI as below:

    Please configure the proxy port and click Start button. Then you could set your browser proxy and surf the target URL, gatling will capture the content as below:

    If you finished and wanna save the result, please click Stop & Save button, gatling will generate a new scala script which save all your behaviors just now, and the script will be under user-files/simulations/

    xxx@xxx 2.3.1 $ ll user-files/simulations/
    total 328
    drwxr-xr-x@ 4 1154257814  80     136 Aug 13 11:30 .
    drwxr-xr-x@ 5 1154257814  80     170 Mar  6 22:59 ..
    -rw-r--r--  1 1154257814  80  166657 Aug 13 11:54 RecordedSimulation.scala
    drwxr-xr-x@ 4 1154257814  80     136 Mar  6 22:59 computerdatabase
    

    If you wanna change the generated file location, please modify outputFolder in conf/recorder.conf

    hongmeng.wang@X0621 2.3.1 $ cat conf/recorder.conf 
    recorder {
        core {
            className=RecordedSimulation
            encoding=utf-8
            harFilePath=""
            headless=false
            mode=Proxy
            outputFolder="/usr/local/Cellar/gatling/2.3.1/user-files/simulations"
            package=""
            ...
            ...
    
     
c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel
%d bloggers like this: