Tagged: Performance Toggle Comment Threads | Keyboard Shortcuts

  • Unknown's avatar

    Wang 19:25 on 2018-08-11 Permalink | Reply
    Tags: , , , , , Performance   

    Auto scaling in kubernetes 

    When we deploy a API in kubernets we must define replication number for the pod, but as we know there will be high traffic during peak time and we usually can’t estimate service capacity exactly at first time, in this case we must scale our service like creating more pods to share online traffic to avoid service crash down.

    We usually scale service manually before using kubernetes, append more nodes during peak time and destroy nodes when the traffic became smooth.

    In kubernetes there’s a kind of feature called HPA(Horizontal Pod Autoscaler) which could help your scale service automatically. You could specify minimum and maximum replica number in yaml file, HPA will monitor pod’s CPU and Memory by collecting pod’s metric, if HPA found your pod’s metric is over the threshold number which you defined in yaml file, it will create more pods automatically and join the service cluster to load the traffic.

    Here is a simple HPA samle:

    apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-demo
  namespace: test-ns
  labels:
    app: hpa-demo
    component: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hpa-demo
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 75
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 75

    I defined there’s will be at least 3 replicas for the pod, if the CPU or Memory usage is over 75%, HPA will create at most 10 pods.

    HPA monitor pod’s metric by using metrics-server.

     
  • Unknown's avatar

    Wang 19:45 on 2018-06-22 Permalink | Reply
    Tags: , Performance, , Scala,   

    [Gatling] Customize your scripts based on scenario 

    Sometimes record function may not fulfill your requirement, in this case you need write scala script yourself according to gatling document.

    Below is a simple sample to test two APIs: employee & health check

    package me.hongmeng.stress.test

import io.gatling.core.Predef._
import io.gatling.http.Predef._

class SimpleSimulation extends Simulation {

  val hostname = "http://localhost:8080"

  val httpProtocol = http
    .baseURL(hostname)
    .inferHtmlResources()
    .acceptHeader("*/*")
    .contentTypeHeader("application/json")
    .userAgentHeader("Gatling/2.3.1")

  val headers = Map("accept-encoding" -> "gzip, deflate", "cache-control" -> "no-cache")

  val employeeScenario = scenario("create_experiment_simulation")
    .exec(
      http("employee")
        .post("/v1/employees")
        .headers(headers)
        .body(
          StringBody("{"name": "xiaowang","major": "software"}")
        )
    )

  val healthScenario = scenario("health_simulation")
    .exec(
      http("health_check")
        .get("/actuator/health")
        .headers(headers)
    )

  setUp(
      employeeScenario.inject(atOnceUsers(10)),
      healthScenario.inject(atOnceUsers(10))
    ).protocols(httpProtocol)
}
     
  • Unknown's avatar

    Wang 21:21 on 2018-06-13 Permalink | Reply
    Tags: , Performance, ,   

    [Gatling] Know about stress test tool 

    Gatling is kind of stress test tool to test your app’s performance, for detail infos, please refer to official document.

    Next I will summurize how to install gatling on your Macbook.

    1.Download gatling

    wget https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/2.3.1/gatling-charts-highcharts-bundle-2.3.1-bundle.zip

    2.Unzip & Enter

    unzip gatling-charts-highcharts-bundle-2.3.1-bundle.zip & cd gatling-charts-highcharts-bundle-2.3.1-bundle

    3.Start record function

    bin/record.sh

    After starting record script, you could see the UI as below:

    Please configure the proxy port and click Start button. Then you could set your browser proxy and surf the target URL, gatling will capture the content as below:

    If you finished and wanna save the result, please click Stop & Save button, gatling will generate a new scala script which save all your behaviors just now, and the script will be under user-files/simulations/

    xxx@xxx 2.3.1 $ ll user-files/simulations/
total 328
drwxr-xr-x@ 4 1154257814  80     136 Aug 13 11:30 .
drwxr-xr-x@ 5 1154257814  80     170 Mar  6 22:59 ..
-rw-r--r--  1 1154257814  80  166657 Aug 13 11:54 RecordedSimulation.scala
drwxr-xr-x@ 4 1154257814  80     136 Mar  6 22:59 computerdatabase

    If you wanna change the generated file location, please modify outputFolder in conf/recorder.conf

    hongmeng.wang@X0621 2.3.1 $ cat conf/recorder.conf 
recorder {
    core {
        className=RecordedSimulation
        encoding=utf-8
        harFilePath=""
        headless=false
        mode=Proxy
        outputFolder="/usr/local/Cellar/gatling/2.3.1/user-files/simulations"
        package=""
        ...
        ...
     
  • Unknown's avatar

    Wang 19:44 on 2018-02-25 Permalink | Reply
    Tags: , , Performance,   

    [Performance Test] MR vs Tez 

    I tested performance about MR and Tez on my laptop, it’s single server, so it’s not very accurate.

    I create two tables to do the test which contains the datasets I downloaded from GBIF.

    gbif_0004998: 327,316 rows

    gbif_0004991: 6,914,665 rows

    1.test gbif_0004998
    create by MR
    hive> set hive.execution.engine=mr;
Hive-on-MR is deprecated in Hive 2 and may no be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> CREATE TABLE gbif.gbif_0004998
> STORED AS ORC
> TBLPROPERTIES("orc.compress"="snappy")
> AS SELECT * FROM gbif.gbif_0004998_ori;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = wanghongmeng_20180224190744_f3fb257a-829e-40c2-974b-5abeb3d88693
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1519462946874_0007, Tracking URL = http://localhost:8088/proxy/application_1519462946874_0007/
Kill Command = /usr/local/Cellar/hadoop/2.8.2/bin/hadoop job -kill job_1519462946874_0007
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2018-02-24 19:07:53,043 Stage-1 map = 0%, reduce = 0%
2018-02-24 19:08:10,204 Stage-1 map = 100%, reduce = 0%
Ended Job = job_1519462946874_0007
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://localhost:9000/user/hive/warehouse/gbif.db/.hive-staging_hive_2018-02-24_19-07-44_762_5371659277950436672-1/-ext-10002
Moving data to directory hdfs://localhost:9000/user/hive/warehouse/gbif.db/gbif_0004998
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1 HDFS Read: 130582415 HDFS Write: 13510429 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 28.28 seconds
    create by Tez
    hive> set hive.execution.engine=tez;
hive> CREATE TABLE gbif.gbif_0004998
> STORED AS ORC
> TBLPROPERTIES("orc.compress"="snappy")
> AS SELECT * FROM gbif.gbif_0004998_ori;
Query ID = wanghongmeng_20180224193755_bd7fda12-bfd7-4abf-9c3e-0f90b9b58607
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1519462946874_0013)

----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED 
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0 
----------------------------------------------------------------------------------------------
VERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 15.65 s 
----------------------------------------------------------------------------------------------
Moving data to directory hdfs://localhost:9000/user/hive/warehouse/gbif.db/gbif_0004998
OK
gbif_0004998_ori.gbifid gbif_0004998_ori.datasetkey gbif_0004998_ori.occurrenceid gbif_0004998_ori.kingdom gbif_0004998_ori.phylum gbif_0004998_ori.class gbif_0004998_ori.orders gbif_0004998_ori.family gbif_0004998_ori.genus gbif_0004998_ori.species gbif_0004998_ori.infraspecificepithet gbif_0004998_ori.taxonrank gbif_0004998_ori.scientificname gbif_0004998_ori.countrycode gbif_0004998_ori.locality gbif_0004998_ori.publishingorgkey gbif_0004998_ori.decimallatitude gbif_0004998_ori.decimallongitude gbif_0004998_ori.coordinateuncertaintyinmeters gbif_0004998_ori.coordinateprecision gbif_0004998_ori.elevation gbif_0004998_ori.elevationaccuracy gbif_0004998_ori.depth gbif_0004998_ori.depthaccuracy gbif_0004998_ori.eventdate gbif_0004998_ori.day gbif_0004998_ori.month gbif_0004998_ori.year gbif_0004998_ori.taxonkey gbif_0004998_ori.specieskey gbif_0004998_ori.basisofrecord gbif_0004998_ori.institutioncode gbif_0004998_ori.collectioncode gbif_0004998_ori.catalognumber gbif_0004998_ori.recordnumber gbif_0004998_ori.identifiedby gbif_0004998_ori.license gbif_0004998_ori.rightsholder gbif_0004998_ori.recordedby gbif_0004998_ori.typestatus gbif_0004998_ori.establishmentmeans gbif_0004998_ori.lastinterpreted gbif_0004998_ori.mediatype gbif_0004998_ori.issue
Time taken: 16.631 seconds
    query by MR
    hive> set hive.execution.engine=mr;
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> select count(*) as total from gbif_0004998 where mediatype = 'STILLIMAGE';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = wanghongmeng_20180224194412_0c9a74e1-b01e-4b92-8db4-f31522d44bd9
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1519462946874_0016, Tracking URL = http://localhost:8088/proxy/application_1519462946874_0016/
Kill Command = /usr/local/Cellar/hadoop/2.8.2/bin/hadoop job -kill job_1519462946874_0016
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2018-02-24 19:44:24,034 Stage-1 map = 0%, reduce = 0%
2018-02-24 19:44:33,661 Stage-1 map = 100%, reduce = 0%
2018-02-24 19:44:40,063 Stage-1 map = 100%, reduce = 100%
Ended Job = job_1519462946874_0016
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1 Reduce: 1 HDFS Read: 30539 HDFS Write: 105 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
total
28918
Time taken: 28.529 seconds, Fetched: 1 row(s)
    query by Tez
    hive> set hive.execution.engine=tez;
hive> select count(*) from gbif_0004998 where mediatype = 'STILLIMAGE';
Query ID = wanghongmeng_20180224193902_f03b627e-e091-4632-87e5-0d8af6484032
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1519462946874_0013)

----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED 
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0 
Reducer 2 ...... container SUCCEEDED 1 1 0 0 0 0 
----------------------------------------------------------------------------------------------
VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 5.97 s 
----------------------------------------------------------------------------------------------
OK
total
28918
Time taken: 6.438 seconds, Fetched: 1 row(s)
    2.test gbif_0004991
    create by MR
    hive> set hive.execution.engine=mr;
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> CREATE TABLE gbif.gbif_0004991
> STORED AS ORC
> TBLPROPERTIES("orc.compress"="snappy")
> AS SELECT * FROM gbif.gbif_0004991_ori;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = wanghongmeng_20180224191238_19301476-a77f-45fa-a405-05a8732a45e9
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1519462946874_0010, Tracking URL = http://localhost:8088/proxy/application_1519462946874_0010/
Kill Command = /usr/local/Cellar/hadoop/2.8.2/bin/hadoop job -kill job_1519462946874_0010
Hadoop job information for Stage-1: number of mappers: 14; number of reducers: 0
2018-02-24 19:16:32,473 Stage-1 map = 0%, reduce = 0%
2018-02-24 19:17:32,678 Stage-1 map = 0%, reduce = 0%
2018-02-24 19:17:40,248 Stage-1 map = 7%, reduce = 0%
2018-02-24 19:17:51,119 Stage-1 map = 11%, reduce = 0%
2018-02-24 19:17:52,207 Stage-1 map = 18%, reduce = 0%
2018-02-24 19:17:58,625 Stage-1 map = 21%, reduce = 0%
2018-02-24 19:18:13,859 Stage-1 map = 25%, reduce = 0%
2018-02-24 19:18:15,999 Stage-1 map = 32%, reduce = 0%
2018-02-24 19:18:30,537 Stage-1 map = 36%, reduce = 0%
2018-02-24 19:18:31,625 Stage-1 map = 39%, reduce = 0%
2018-02-24 19:18:32,759 Stage-1 map = 43%, reduce = 0%
2018-02-24 19:19:17,117 Stage-1 map = 46%, reduce = 0%
2018-02-24 19:19:19,250 Stage-1 map = 50%, reduce = 0%
2018-02-24 19:19:25,639 Stage-1 map = 54%, reduce = 0%
2018-02-24 19:19:28,825 Stage-1 map = 57%, reduce = 0%
2018-02-24 19:19:32,031 Stage-1 map = 61%, reduce = 0%
2018-02-24 19:19:33,101 Stage-1 map = 64%, reduce = 0%
2018-02-24 19:19:39,470 Stage-1 map = 68%, reduce = 0%
2018-02-24 19:19:42,677 Stage-1 map = 71%, reduce = 0%
2018-02-24 19:19:54,459 Stage-1 map = 75%, reduce = 0%
2018-02-24 19:19:58,723 Stage-1 map = 79%, reduce = 0%
2018-02-24 19:20:04,147 Stage-1 map = 82%, reduce = 0%
2018-02-24 19:20:06,277 Stage-1 map = 86%, reduce = 0%
2018-02-24 19:20:15,977 Stage-1 map = 93%, reduce = 0%
2018-02-24 19:20:20,269 Stage-1 map = 96%, reduce = 0%
2018-02-24 19:20:36,398 Stage-1 map = 100%, reduce = 0%
Ended Job = job_1519462946874_0010
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://localhost:9000/user/hive/warehouse/gbif.db/.hive-staging_hive_2018-02-24_19-12-38_616_5758586722663198282-1/-ext-10002
Moving data to directory hdfs://localhost:9000/user/hive/warehouse/gbif.db/gbif_0004991
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 14 HDFS Read: 3539512736 HDFS Write: 342789525 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 481.311 seconds
    create via Tez
    hive> set hive.execution.engine=tez;
hive> CREATE TABLE gbif.gbif_0004991
> STORED AS ORC
> TBLPROPERTIES("orc.compress"="snappy")
> AS SELECT * FROM gbif.gbif_0004991_ori;
Query ID = wanghongmeng_20180224192800_111872d9-059b-4a8a-9fd7-e3ea02af8898
Total jobs = 1
Launching Job 1 out of 1
Tez session was closed. Reopening...
Session re-established.
Status: Running (Executing on YARN cluster with App id application_1519462946874_0013)

----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED 
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0 
----------------------------------------------------------------------------------------------
VERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 241.12 s 
----------------------------------------------------------------------------------------------
Moving data to directory hdfs://localhost:9000/user/hive/warehouse/gbif.db/gbif_0004991
OK
gbif_0004991_ori.gbifid gbif_0004991_ori.datasetkey gbif_0004991_ori.occurrenceid gbif_0004991_ori.kingdom gbif_0004991_ori.phylum gbif_0004991_ori.class gbif_0004991_ori.orders gbif_0004991_ori.family gbif_0004991_ori.genus gbif_0004991_ori.species gbif_0004991_ori.infraspecificepithet gbif_0004991_ori.taxonrank gbif_0004991_ori.scientificname gbif_0004991_ori.countrycode gbif_0004991_ori.locality gbif_0004991_ori.publishingorgkey gbif_0004991_ori.decimallatitude gbif_0004991_ori.decimallongitude gbif_0004991_ori.coordinateuncertaintyinmeters gbif_0004991_ori.coordinateprecision gbif_0004991_ori.elevation gbif_0004991_ori.elevationaccuracy gbif_0004991_ori.depth gbif_0004991_ori.depthaccuracy gbif_0004991_ori.eventdate gbif_0004991_ori.day gbif_0004991_ori.month gbif_0004991_ori.year gbif_0004991_ori.taxonkey gbif_0004991_ori.specieskey gbif_0004991_ori.basisofrecord gbif_0004991_ori.institutioncode gbif_0004991_ori.collectioncode gbif_0004991_ori.catalognumber gbif_0004991_ori.recordnumber gbif_0004991_ori.identifiedby gbif_0004991_ori.license gbif_0004991_ori.rightsholder gbif_0004991_ori.recordedby gbif_0004991_ori.typestatus gbif_0004991_ori.establishmentmeans gbif_0004991_ori.lastinterpreted gbif_0004991_ori.mediatype gbif_0004991_ori.issue
Time taken: 252.548 seconds
    query via MR
    hive> set hive.execution.engine=mr;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> select count(*) from gbif_0004991 where mediatype = 'STILLIMAGE';
Query ID = wanghongmeng_20180224192630_b2934027-2423-4945-864b-6ce663e676fa
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1519462946874_0012, Tracking URL = http://localhost:8088/proxy/application_1519462946874_0012/
Kill Command = /usr/local/Cellar/hadoop/2.8.2/bin/hadoop job -kill job_1519462946874_0012
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1
2018-02-24 19:26:43,988 Stage-1 map = 0%, reduce = 0%
2018-02-24 19:27:00,086 Stage-1 map = 50%, reduce = 0%
2018-02-24 19:27:03,287 Stage-1 map = 74%, reduce = 0%
2018-02-24 19:27:05,422 Stage-1 map = 100%, reduce = 0%
2018-02-24 19:27:08,595 Stage-1 map = 100%, reduce = 100%
Ended Job = job_1519462946874_0012
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 2 Reduce: 1 HDFS Read: 602777 HDFS Write: 106 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
total
374998
Time taken: 38.903 seconds, Fetched: 1 row(s)
    query via Tez
    hive> set hive.execution.engine=tez;
hive> select count(*) from gbif_0004991 where mediatype = 'STILLIMAGE';
Query ID = wanghongmeng_20180224193241_f4edd363-fdb8-4461-b687-4b775e8719c0
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1519462946874_0013)

----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED 
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 2 2 0 0 0 0 
Reducer 2 ...... container SUCCEEDED 1 1 0 0 0 0 
----------------------------------------------------------------------------------------------
VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 16.54 s 
----------------------------------------------------------------------------------------------
OK
total
374998
Time taken: 17.258 seconds, Fetched: 1 row(s)
    3.summary

    Table

    Total Count

    Create Table

    Query

    gbif_0004998

    327,316

    		MR28.28sMR28.529s
    Tez16.631sTez6.438s

    gbif_0004991

    6,914,665

    		MR481.311sMR38.903s
    Tez252.548sTez17.258s
     
  • Unknown's avatar

    Wang 18:23 on 2018-02-10 Permalink | Reply
    Tags: , Chrome, Performance, , Web   

    not good..

     

Newer posts →

c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel