Tagged: Restful Toggle Comment Threads | Keyboard Shortcuts

  • Wang 22:16 on 2020-12-08 Permalink | Reply
    Tags: , , , , Restful   

    Circuit Breaker Pattern

  • Wang 19:38 on 2019-11-06 Permalink | Reply
    Tags: , , , Restful   

    Chaos Engineering 

    Chaos Engineering is a great idea — build an automated solution/tool to randomly attempt to break a system in some way; ultimately to learn how the system behaves in such situations. Then you can use your newfound knowledge to find ways to make the system more fault tolerant during these failure conditions in the future.

    From :https://medium.com/better-programming/chaos-engineering-chaos-testing-your-http-micro-services-acc99d145515

  • Wang 12:43 on 2019-06-17 Permalink | Reply
    Tags: , , Restful   


  • Wang 22:12 on 2019-02-11 Permalink | Reply
    Tags: , , , , , , , Restful, ,   

    Guarantee service availability in kubernetes 

    A good service not only provide good functionalities, but also ensure the availability and uptime.

    We reinforce our service from QoS, QPS, Throttling, Scaling, Throughput, Monitoring.


    There’re 3 kinds of QoS in kubernetes: Guaranteed, Burstable, BestEffort. We usually use Guaranteed, Burstable for different services.

        cpu: 1000m
        memory: 4Gi
        cpu: 1000m
        memory: 4Gi
        cpu: 1000m
        memory: 4Gi
        cpu: 6000m
        memory: 8Gi

    We did lots of stress test on APIs by Gatling before we release them, we mainly care about mean response time, std deviation, mean requests/sec, error rate (API Testing Report), during testing we monitor server metrics by Datadog to find out bottlenecks.

    We usually test APIs in two scenarios: internal, external. External testing result is much lower than internal testing because of network latency, network bandwidth and son on.

    Internal testing result

    ---- Global Information --------------------------------------------------------
    > request count                                     246000 (OK=246000 KO=0     )
    > min response time                                     16 (OK=16     KO=-     )
    > max response time                                   5891 (OK=5891   KO=-     )
    > mean response time                                    86 (OK=86     KO=-     )
    > std deviation                                        345 (OK=345    KO=-     )
    > response time 50th percentile                         30 (OK=30     KO=-     )
    > response time 75th percentile                         40 (OK=40     KO=-     )
    > response time 95th percentile                         88 (OK=88     KO=-     )
    > response time 99th percentile                       1940 (OK=1940   KO=-     )
    > mean requests/sec                                817.276 (OK=817.276 KO=-     )
    ---- Response Time Distraaibution ------------------------------------------------
    > t < 800 ms                                        240565 ( 98%)
    > 800 ms < t < 1200 ms                                1110 (  0%)
    > t > 1200 ms                                         4325 (  2%)
    > failed                                                 0 (  0%)

    External testing result

    ---- Global Information --------------------------------------------------------
    > request count                                      33000 (OK=32999  KO=1     )
    > min response time                                    477 (OK=477    KO=60001 )
    > max response time                                  60001 (OK=41751  KO=60001 )
    > mean response time                                   600 (OK=599    KO=60001 )
    > std deviation                                        584 (OK=484    KO=0     )
    > response time 50th percentile                        497 (OK=497    KO=60001 )
    > response time 75th percentile                        506 (OK=506    KO=60001 )
    > response time 95th percentile                       1366 (OK=1366   KO=60001 )
    > response time 99th percentile                       2125 (OK=2122   KO=60001 )
    > mean requests/sec                                109.635 (OK=109.631 KO=0.003 )
    ---- Response Time Distribution ------------------------------------------------
    > t < 800 ms                                         29826 ( 90%)
    > 800 ms < t < 1200 ms                                1166 (  4%)
    > t > 1200 ms                                         2007 (  6%)
    > failed                                                 1 (  0%)
    ---- Errors --------------------------------------------------------------------
    > i.g.h.c.i.RequestTimeoutException: Request timeout after 60000      1 (100.0%)

    We throttle API by Nginx limit, we configured ingress like this:

      nginx.ingress.kubernetes.io/limit-connections: '30'
      nginx.ingress.kubernetes.io/limit-rps: '60'

    And it will generate Nginx configuration dynamically like this:

    limit_conn_zone $limit_ZGVsaXZlcnktY2RuYV9kc2QtYXBpLWNkbmEtZ2F0ZXdheQ zone=xxx_conn:5m;
    limit_req_zone $limit_ZGVsaXZlcnktY2RuYV9kc2QtYXBpLWNkbmEtZ2F0ZXdheQ zone=xxx_rps:5m rate=60r/s;
    server {
        server_name xxx.xxx ;
        listen 80;
        location ~* "^/xxx/?(?<baseuri>.*)" {
            limit_conn xxx_conn 30;
            limit_req zone=xxx_rps burst=300 nodelay;

    We use HPA in kubernetes to ensure auto (Auto scaling in kubernetes), you could check HPA status in server:

    [xxx@xxx ~]$ kubectl get hpa -n test-ns
    api-demo   Deployment/api-demo   39%/30%, 0%/30%   3         10        3          126d
    [xxx@xxx ~]$ kubectl get pod -n test-ns
    NAME                           READY     STATUS    RESTARTS   AGE
    api-demo-76b9954f57-6hvzx      1/1       Running   0          126d
    api-demo-76b9954f57-mllsx      1/1       Running   0          126d
    api-demo-76b9954f57-s22k8      1/1       Running   0          126d
    Throughput & Monitoring

    We integrated Datadog for monitoring(Monitoring by Datadog), we could check detail API metrics from various dashboards.

    Also we could calculate throughout from user, request, request time.

  • Wang 22:43 on 2018-10-08 Permalink | Reply
    Tags: , , , , , , Restful,   

    Nginx ingress in kubernetes 

    There are 3 ways to expose your service: NodePort, LoadBalancer, Ingress, next I will introduce about how to use ingress.

    1.Deploy ingress controller

    You need deploy ingress controller at first which will start nginx pods, then nginx will bind domains and listen to the requests.

    I built a common ingress chart for different service, I only need change values-<service>.yaml and deploy script if any changes.

    Another key point is that you must be clear about ingress-class, different service use different ingress-class, it will be quite messy if you mistake them.

      - /nginx-ingress-controller
      - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
      - --configmap=$(POD_NAMESPACE)/nginx-configuration
      - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      - --ingress-class={{ .Values.server.namespace }}
      - --sort-backends=true

    2.Configure service ingress

    Next we need configure service ingress which will append nginx server configuration dynamically.

    I also built a service chart which include environment configurations, Jenkins & Helm will use different values-<env>.yaml when execute pipeline deployment.

    Ingress example:

    apiVersion: extensions/v1beta1
    kind: Ingress
      name: {{ .Values.app.name }}{{ .Values.deploy.subfix }}
      namespace: {{ .Values.app.namespace }}
        kubernetes.io/ingress.class: "{{ .Values.ingress.class }}"
        kubernetes.io/tls-acme: "true"
        nginx.ingress.kubernetes.io/enable-cors: "false"
        nginx.ingress.kubernetes.io/rewrite-target: /
        nginx.ingress.kubernetes.io/proxy-body-size: 10m
      - host: {{ .Values.ingress.hostname }}
          - path: {{ .Values.ingress.path }}
              serviceName: {{ .Values.app.name }}{{ .Values.deploy.subfix }}
              servicePort: {{ .Values.container.port }}
  • Wang 19:45 on 2018-06-22 Permalink | Reply
    Tags: , , Restful, Scala,   

    [Gatling] Customize your scripts based on scenario 

    Sometimes record function may not fulfill your requirement, in this case you need write scala script yourself according to gatling document.

    Below is a simple sample to test two APIs: employee & health check

    package me.hongmeng.stress.test
    import io.gatling.core.Predef._
    import io.gatling.http.Predef._
    class SimpleSimulation extends Simulation {
      val hostname = "http://localhost:8080"
      val httpProtocol = http
      val headers = Map("accept-encoding" -> "gzip, deflate", "cache-control" -> "no-cache")
      val employeeScenario = scenario("create_experiment_simulation")
              StringBody("{"name": "xiaowang","major": "software"}")
      val healthScenario = scenario("health_simulation")
  • Wang 21:21 on 2018-06-13 Permalink | Reply
    Tags: , , Restful,   

    [Gatling] Know about stress test tool 

    Gatling is kind of stress test tool to test your app’s performance, for detail infos, please refer to official document.

    Next I will summurize how to install gatling on your Macbook.

    1.Download gatling

    wget https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/2.3.1/gatling-charts-highcharts-bundle-2.3.1-bundle.zip

    2.Unzip & Enter

    unzip gatling-charts-highcharts-bundle-2.3.1-bundle.zip & cd gatling-charts-highcharts-bundle-2.3.1-bundle

    3.Start record function


    After starting record script, you could see the UI as below:

    Please configure the proxy port and click Start button. Then you could set your browser proxy and surf the target URL, gatling will capture the content as below:

    If you finished and wanna save the result, please click Stop & Save button, gatling will generate a new scala script which save all your behaviors just now, and the script will be under user-files/simulations/

    xxx@xxx 2.3.1 $ ll user-files/simulations/
    total 328
    drwxr-xr-x@ 4 1154257814  80     136 Aug 13 11:30 .
    drwxr-xr-x@ 5 1154257814  80     170 Mar  6 22:59 ..
    -rw-r--r--  1 1154257814  80  166657 Aug 13 11:54 RecordedSimulation.scala
    drwxr-xr-x@ 4 1154257814  80     136 Mar  6 22:59 computerdatabase

    If you wanna change the generated file location, please modify outputFolder in conf/recorder.conf

    hongmeng.wang@X0621 2.3.1 $ cat conf/recorder.conf 
    recorder {
        core {
  • Wang 22:16 on 2018-04-22 Permalink | Reply
    Tags: , , Restful,   

    [Spring Boot2] Access log trace 

    If I wanna trace access logs, I usually configured in nginx and tomcat, only print business logs in java api.

    In this case you need check both nginx logs and api logs in monitor tools like kinaba, it’s a little bit inconvenience, so I wanna also add request access logs in api side.

    Generally we can create a new interceptor which extends OncePerRequestFilter, it will intercept all the requests and you can define your own logic inside. But it maybe a little complex in POST request, if you intercept and handle POST request body, when the request arrived controller, the request body will be null, because you have read the stream in your interceptor, so you have to create another request wrapper which extends HttpServletRequestWrapper, then pass the request wrapper to the filter chain.

    Actually we can make it in more easily way according to spring boot’s component:  spring-boot-starter-actuator. actuator component contains lots of useful endpoints like health check/shutdown hook/metrics and so on. There are also a class named HttpTraceRepository, which will record recent access logs. Specific documentation please go to Spring Boot Actuator.

    I extended HttpTraceRepository and combined with Aspectj, which will print both request and parameters to log, and extract it into framework module, any project could include it very easily.

    Printed logs as below:

    2018-05-03 06:56:10,651 - [http-nio-8080-exec-6] - [INFO ] - [xxx.xxx.xxx.api.framework.aop.LogHttpTraceRepository.around(line:47)] - RequestParam:[1,null,null], ResponseEntity:{"timestamp":1525330570651,"status":200,"error":null,"exception":null,"message":"success","path":null,"method":null,"data":{"experimentId":1,"results":[]}}
    2018-05-03 06:56:10,654 - [http-nio-8080-exec-6] - [INFO ] - [xxx.xxx.xxx.api.framework.aop.LogHttpTraceRepository.add(line:32)] - GET http://localhost:8080/v1/employees?deparment_id=1 - 200 - 15ms
    2018-05-03 06:56:10,655 - [http-nio-8080-exec-6] - [INFO ] - [xxx.xxx.xxx.api.framework.aop.LogHttpTraceRepository.add(line:34)] - RequestHeader:{host=[localhost:8080], connection=[keep-alive], cache-control=[no-cache], user-agent=[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36], postman-token=[e9a16988-64a6-1363-bdbd-6b63fe40da5c], accept=[*/*], dnt=[1], accept-encoding=[gzip, deflate, br], accept-language=[zh-CN,zh;q=0.9,en;q=0.8,ja;q=0.7,zh-TW;q=0.6]}, ResponseHeader:{Content-Type=[application/json;charset=UTF-8], Transfer-Encoding=[chunked], Content-Encoding=[gzip], Vary=[Accept-Encoding], Date=[Thu, 03 May 2018 06:56:10 GMT]}


    public class LogHttpTraceRepository implements HttpTraceRepository {
        public List<HttpTrace> findAll() {
            return Collections.emptyList();
        public void add(HttpTrace trace) {
            HttpTrace.Request request = trace.getRequest();
            HttpTrace.Response response = trace.getResponse();
            long timeToken = trace.getTimeTaken();
            log.info("{} {} - {} - {}ms", request.getMethod(), request.getUri(),
                    response.getStatus(), timeToken);
            log.info("RequestHeader:{}, ResponseHeader:{}", request.getHeaders(), response.getHeaders());
        @Around("execution(* com.xxx..controller.*.*(..))")
        Object around(ProceedingJoinPoint proceedingJoinPoint) throws Throwable {
            StringBuilder logBuilder = new StringBuilder();
            Optional<String> requestOptional = JsonUtils.toJson(proceedingJoinPoint.getArgs());
            requestOptional.ifPresent(request -> logBuilder.append("RequestParam:").append(request).append(", "));
            Object response = proceedingJoinPoint.proceed();
            Optional<String> responseOptional = JsonUtils.toJson(response);
            responseOptional.ifPresent(res -> logBuilder.append("ResponseEntity:").append(res));
            return response;
  • Wang 21:34 on 2018-04-20 Permalink | Reply
    Tags: , , Restful,   

    [Spring Boot2] Common exception handler 

    When we develop API, how to handle exception is very important for us to locate error, find out exact error code.

    Recently I extracted exceptionHandler to a common framework, and deploy it into our internal maven repository. If anyone wanna use it, just add maven dependency, and it will works automatically.

    1.Add internal repository and dependency in your pom.xml


    2.Add package scan annotation, the package prefix must be the same as framework’s package

    @ComponentScan(basePackages = "com.xxx.xxx.api")

    Then it will intercept all the exception and print error logs and respond to the client.

    Error Logs:

    2018-04-19 07:02:24,785 - [http-nio-8080-exec-2] - [ERROR] - [xxx.xxx.xxx.api.framework.aop.ApiExceptionHandler.internalErrorHandler(line:26)] - org.springframework.jdbc.BadSqlGrammarException: 
    ### SQL: SELECT id, experiment_id, start_at, model, created_at, created_by, updated_at, updated_by FROM XXX WHERE experiment_id = ?
    ### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'xxx_xxx.XXX' doesn't exist
    ; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'xxx_xxx.XXX' doesn't exist
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:235)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
        at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:73)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
    Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'xxx_xxx.XXX' doesn't exist
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
        ... 95 more

    Client Response:

        "timestamp": "2018-05-03T07:02:24Z",
        "status": 500,
        "error": "Internal Server Error",
        "message": "n### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'xxx_xxx.xxx' doesn't existn### The error may exist in file [/Users/xxx/java/xxx-api-xxx/target/classes/com/xxx/xxx/api/xxx/mapper/xxxMapper.xml]n### The error may involve defaultParameterMapn### The error occurred while setting parametersn### SQL: SELECT id, department_id, start_at, created_at, created_by, updated_at, updated_by FROM xxx WHERE department_id = ?n### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'xxx_xxx.xxx' doesn't existn; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'xxx_xxx.xxx' doesn't exist",
        "path": "/v1/employees",
        "method": "GET"


    public class ApiExceptionHandler extends ResponseEntityExceptionHandler {
        @ResponseStatus(code = HttpStatus.INTERNAL_SERVER_ERROR)
        public ApiResponseEntity internalErrorHandler(Exception exception, ServletWebRequest request) {
            log.error("{}", ExceptionUtils.getStackTrace(exception));
            return new ApiResponseEntity(HttpStatus.INTERNAL_SERVER_ERROR.value(), HttpStatus.INTERNAL_SERVER_ERROR.getReasonPhrase(),
                    exception.getMessage(), request);
        @ResponseStatus(code = HttpStatus.NOT_FOUND)
        public ApiResponseEntity notFoundHandler(Exception exception, ServletWebRequest request) {
            log.warn("{}", exception.getMessage());
            return new ApiResponseEntity(HttpStatus.NOT_FOUND.value(), HttpStatus.NOT_FOUND.getReasonPhrase(),
                    exception.getMessage(), request);
        @ResponseStatus(code = HttpStatus.BAD_REQUEST)
        public ApiResponseEntity illegalArgumentException(Exception exception, ServletWebRequest request) {
            log.warn("{}", exception.getMessage());
            return new ApiResponseEntity(HttpStatus.BAD_REQUEST.value(), HttpStatus.BAD_REQUEST.getReasonPhrase(),
                    exception.getMessage(), request);
        protected ResponseEntity<Object> handleExceptionInternal(
                Exception exception, @Nullable Object body, HttpHeaders headers, HttpStatus status, WebRequest request) {
            log.warn("{}", exception.getMessage());
            if (HttpStatus.INTERNAL_SERVER_ERROR.equals(status)) {
                request.setAttribute(WebUtils.ERROR_EXCEPTION_ATTRIBUTE, exception, WebRequest.SCOPE_REQUEST);
            ServletWebRequest servletWebRequest = (ServletWebRequest) request;
            ApiResponseEntity apiResponseEntity = new ApiResponseEntity(status.value(), status.getReasonPhrase(), exception.getMessage(), servletWebRequest);
            return new ResponseEntity<>(apiResponseEntity, headers, status);
  • Wang 22:32 on 2018-03-28 Permalink | Reply
    Tags: , , , Restful,   

    [Spring Boot2] Demo 

    Recently Spring Boot has released version 2.0.0.RELEASE, so I did a small demo which included the basic CRUD, I have uploaded the code to github.

    There are two branches, master is the normal branch, and docker branch will create docker image when you build.

Compose new post
Next post/Next comment
Previous post/Previous comment
Show/Hide comments
Go to top
Go to login
Show/Hide help
shift + esc
%d bloggers like this: