Tagged: cluster Toggle Comment Threads | Keyboard Shortcuts
-
Wang
-
Wang
Slurm
Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Slurm requires no kernel modifications for its operation and is relatively self-contained. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.
https://slurm.schedmd.com/ -
Wang
-
Wang
-
Wang
PoC of Apache Druid
As we have some business requirements about data aggregation and online processing, so we did a quick PoC on Apache Druid. Next I will show how to build druid quickly and start your ingestion task.
1.Select release version which is compatible to your existing system and download the package.
2.Choose what kind of druid service you want to start with
- For single node, just execute the script under bin directory which is start with start-single-server-, or you can execute start-micro-quickstart
- For multiple node cluster, please update the configuration files under start-micro-quickstart in one node and sync to other nodes. If you want to connect to your hadoop cluster, please copy corresponding hadoop xml files and kerberos keytab under druid.
Then you start druid service in every node by execute start-cluster script.
3.Visit druid through browser, http://IP:8888
Next I load the data from local file and can ingest the data file as a datasource, and finally query data by SQL.
Task configuration
{ "type": "index_parallel", "id": "index_parallel_wikiticker-2015-09-12-sampled_2020-02-18T11:17:29.236Z", "resource": { "availabilityGroup": "index_parallel_wikiticker-2015-09-12-sampled_2020-02-18T11:17:29.236Z", "requiredCapacity": 1 }, "spec": { "dataSchema": { "dataSource": "wikiticker-2015-09-12-sampled", "parser": { "type": "string", "parseSpec": { "format": "json", "timestampSpec": { "column": "time", "format": "iso" }, "dimensionsSpec": { "dimensions": [ "channel", "cityName", "comment", "countryIsoCode", "countryName", "isAnonymous", "isMinor", "isNew", "isRobot", "isUnpatrolled", "namespace", "page", "regionIsoCode", "regionName", "user" ] } } }, "metricsSpec": [ { "type": "count", "name": "count" }, { "type": "longSum", "name": "sum_added", "fieldName": "added", "expression": null }, { "type": "longSum", "name": "sum_deleted", "fieldName": "deleted", "expression": null }, { "type": "longSum", "name": "sum_delta", "fieldName": "delta", "expression": null }, { "type": "longSum", "name": "sum_metroCode", "fieldName": "metroCode", "expression": null } ], "granularitySpec": { "type": "uniform", "segmentGranularity": "DAY", "queryGranularity": "HOUR", "rollup": true, "intervals": null }, "transformSpec": { "filter": null, "transforms": [] } }, "ioConfig": { "type": "index_parallel", "firehose": { "type": "local", "baseDir": "/opt/druid-0.16.0/quickstart/tutorial", "filter": "wikiticker-2015-09-12-sampled.json.gz", "parser": null }, "appendToExisting": false }, "tuningConfig": { "type": "index_parallel", "maxRowsPerSegment": null, "maxRowsInMemory": 1000000, "maxBytesInMemory": 0, "maxTotalRows": null, "numShards": null, "partitionsSpec": null, "indexSpec": { "bitmap": { "type": "concise" }, "dimensionCompression": "lz4", "metricCompression": "lz4", "longEncoding": "longs" }, "indexSpecForIntermediatePersists": { "bitmap": { "type": "concise" }, "dimensionCompression": "lz4", "metricCompression": "lz4", "longEncoding": "longs" }, "maxPendingPersists": 0, "forceGuaranteedRollup": false, "reportParseExceptions": false, "pushTimeout": 0, "segmentWriteOutMediumFactory": null, "maxNumConcurrentSubTasks": 1, "maxRetry": 3, "taskStatusCheckPeriodMs": 1000, "chatHandlerTimeout": "PT10S", "chatHandlerNumRetries": 5, "maxNumSegmentsToMerge": 100, "totalNumMergeTasks": 10, "logParseExceptions": false, "maxParseExceptions": 2147483647, "maxSavedParseExceptions": 0, "partitionDimensions": [], "buildV9Directly": true } }, "context": { "forceTimeChunkLock": true }, "groupId": "index_parallel_wikiticker-2015-09-12-sampled_2020-02-18T11:17:29.236Z", "dataSource": "wikiticker-2015-09-12-sampled" }Task running status
Task finished, you can see the item in datasource/segment/query
-
Wang
SpringOne Platform 2019 in Austin, https://springoneplatform.io/
-
Wang
Istio playbook
Cloud platforms provide a wealth of benefits for the organizations that use them. However, there’s no denying that adopting the cloud can put strains on DevOps teams. Developers must use microservices to architect for portability, meanwhile operators are managing extremely large hybrid and multi-cloud deployments. Istio lets you connect, secure, control, and observe services.
First, download Istio release version, unzip the package and enter the directory.
Second, verify installation environment
bin/istioctl verify-install
Next, deploy Istio and select the demo profile which enable many features like tracing/kiali/grafana
bin/istioctl manifest apply --set profile=demo
Then, check Istio pods’ status, make sure all the related pods are running
Istio Commands
- authn: Interact with Istio authentication policies
- authz: (authz is experimental. Use
istioctl experimental authz) - convert-ingress: Convert Ingress configuration into Istio VirtualService configuration
- dashboard: Access to Istio web UIs like kiali, grafana, prometheus, jaeger
- deregister: De-registers a service instance
- experimental: Experimental commands that may be modified or deprecated
- help: Help about any command
- kube-inject: Inject Envoy sidecar into Kubernetes pod resources
- manifest: Commands related to Istio manifests
- profile: Commands related to Istio configuration profiles
- proxy-config: Retrieve information about proxy configuration from Envoy [kube only]
- proxy-status: Retrieves the synchronization status of each Envoy in the mesh [kube only]
- register: Registers a service instance (e.g. VM) joining the mesh
- validate: Validate Istio policy and rules
- verify-install: Verifies Istio Installation Status or performs pre-check for the cluster before Istio installation
- version: Prints out build version information
-
Wang
Elasticsearch Benchmark By Esrally
esrally is a benchmark tool for elasticsearch, https://github.com/elastic/rally
Testing Script:
esrally --pipeline=benchmark-only --target-hosts=server1:9200,server2:9200,server3:9200,server4:9200,server5:9200
Server status in Kibana dashboard:
P.S. Please benchmark your ES cluster based on your real business scenario & cluster topology
-
Wang
Kubernetes node in “NotReady” status
Rencetly I found some k8s nodes became “NotReady”, I checked disk and memory, they both seems fine.
[xxx@xxx-xxx ~]# kubectl describe node xxx-xxx ... ... Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- ... PIDPressure False Fri, 10 May 2019 09:24:43 +0900 Fri, 10 May 2018 00:10:12 +0900 KubeletHasSufficientPID kubelet has sufficient PID available ...
Then I restarted kubelet on server and checked logs, I found:
[xxx@xxx-xxx ~]# systemctl status kubelet ● kubelet.service - Kubernetes Kubelet Server Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled) ... May 10 12:30:30 xxx-xxx kubelet[16776]: F0322 12:30:30.810434 16776 server.go:233] failed to run Kubelet: Running with swap on is not supported, plea... ...
So I checked server’s status and turn off swap, then I restarted kubelet and the nodes went well.
[xxx@xxx-xxx ~]# swapoff -a [xxx@xxx-xxx ~]# systemctl restart kubelet
-
Wang
Guarantee service availability in kubernetes
A good service not only provide good functionalities, but also ensure the availability and uptime.
We reinforce our service from QoS, QPS, Throttling, Scaling, Throughput, Monitoring.
Qos
There’re 3 kinds of QoS in kubernetes: Guaranteed, Burstable, BestEffort. We usually use Guaranteed, Burstable for different services.
#Guaranteed resources: requests: cpu: 1000m memory: 4Gi limits: cpu: 1000m memory: 4Gi #Burstable resources: requests: cpu: 1000m memory: 4Gi limits: cpu: 6000m memory: 8GiQPS
We did lots of stress test on APIs by Gatling before we release them, we mainly care about mean response time, std deviation, mean requests/sec, error rate (API Testing Report), during testing we monitor server metrics by Datadog to find out bottlenecks.
We usually test APIs in two scenarios: internal, external. External testing result is much lower than internal testing because of network latency, network bandwidth and son on.
Internal testing result
================================================================================ ---- Global Information -------------------------------------------------------- > request count 246000 (OK=246000 KO=0 ) > min response time 16 (OK=16 KO=- ) > max response time 5891 (OK=5891 KO=- ) > mean response time 86 (OK=86 KO=- ) > std deviation 345 (OK=345 KO=- ) > response time 50th percentile 30 (OK=30 KO=- ) > response time 75th percentile 40 (OK=40 KO=- ) > response time 95th percentile 88 (OK=88 KO=- ) > response time 99th percentile 1940 (OK=1940 KO=- ) > mean requests/sec 817.276 (OK=817.276 KO=- ) ---- Response Time Distraaibution ------------------------------------------------ > t < 800 ms 240565 ( 98%) > 800 ms < t < 1200 ms 1110 ( 0%) > t > 1200 ms 4325 ( 2%) > failed 0 ( 0%) ================================================================================
External testing result
================================================================================ ---- Global Information -------------------------------------------------------- > request count 33000 (OK=32999 KO=1 ) > min response time 477 (OK=477 KO=60001 ) > max response time 60001 (OK=41751 KO=60001 ) > mean response time 600 (OK=599 KO=60001 ) > std deviation 584 (OK=484 KO=0 ) > response time 50th percentile 497 (OK=497 KO=60001 ) > response time 75th percentile 506 (OK=506 KO=60001 ) > response time 95th percentile 1366 (OK=1366 KO=60001 ) > response time 99th percentile 2125 (OK=2122 KO=60001 ) > mean requests/sec 109.635 (OK=109.631 KO=0.003 ) ---- Response Time Distribution ------------------------------------------------ > t < 800 ms 29826 ( 90%) > 800 ms < t < 1200 ms 1166 ( 4%) > t > 1200 ms 2007 ( 6%) > failed 1 ( 0%) ---- Errors -------------------------------------------------------------------- > i.g.h.c.i.RequestTimeoutException: Request timeout after 60000 1 (100.0%) ms ================================================================================
Throttling
We throttle API by Nginx limit, we configured ingress like this:
annotations: nginx.ingress.kubernetes.io/limit-connections: '30' nginx.ingress.kubernetes.io/limit-rps: '60'
And it will generate Nginx configuration dynamically like this:
limit_conn_zone $limit_ZGVsaXZlcnktY2RuYV9kc2QtYXBpLWNkbmEtZ2F0ZXdheQ zone=xxx_conn:5m; limit_req_zone $limit_ZGVsaXZlcnktY2RuYV9kc2QtYXBpLWNkbmEtZ2F0ZXdheQ zone=xxx_rps:5m rate=60r/s; server { server_name xxx.xxx ; listen 80; location ~* "^/xxx/?(?<baseuri>.*)" { ... ... limit_conn xxx_conn 30; limit_req zone=xxx_rps burst=300 nodelay; ... ... }Scaling
We use HPA in kubernetes to ensure auto (Auto scaling in kubernetes), you could check HPA status in server:
[xxx@xxx ~]$ kubectl get hpa -n test-ns NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE api-demo Deployment/api-demo 39%/30%, 0%/30% 3 10 3 126d [xxx@xxx ~]$ kubectl get pod -n test-ns NAME READY STATUS RESTARTS AGE api-demo-76b9954f57-6hvzx 1/1 Running 0 126d api-demo-76b9954f57-mllsx 1/1 Running 0 126d api-demo-76b9954f57-s22k8 1/1 Running 0 126d
Throughput & Monitoring
We integrated Datadog for monitoring(Monitoring by Datadog), we could check detail API metrics from various dashboards.
Also we could calculate throughout from user, request, request time.
Mr Wang and 
Chunyi are discussing. 
Reply