Tagged: Cluster Toggle Comment Threads | Keyboard Shortcuts

  • Wang 18:51 on 2022-08-28 Permalink | Reply
    Tags: , Cluster, ,   

    What is High Performance Computing? 

    High Performance Computing most generally refers to the practice of aggregating computing power in a way that delivers much higher performance than one could get out of a typical desktop computer or workstation in order to solve large problems in science, engineering, or business.

    https://www.usgs.gov/core-science-systems/sas/arc/about/what-high-performance-computing

     
  • Wang 23:08 on 2022-05-04 Permalink | Reply
    Tags: Cluster, , , ,   

    K8S Tools Sharing 

    Kubecost Core Architecture Overview


    Kustomize

    kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.


     
  • Wang 20:30 on 2022-04-22 Permalink | Reply
    Tags: , , , Cluster,   

    Pulumi vs. Terraform

     
  • Wang 23:23 on 2022-01-16 Permalink | Reply
    Tags: Alluxio, , Cluster, ,   

    Alluxio 

    Data Locality: Bring your data close to compute.
    Make your data local to compute workloads for Spark caching, Presto caching, Hive caching and more.
    Data Accessibility: Make your data accessible.
    No matter if it sits on-prem or in the cloud, HDFS or S3, make your files and objects accessible in many different ways.
    Data On-Demand: Make your data as elastic as compute.
    Effortlessly orchestrate your data for compute in any cloud, even if data is spread across multiple clouds.

    https://www.alluxio.io/
     
  • Wang 21:06 on 2021-07-14 Permalink | Reply
    Tags: Cluster, , , ,   

    Very well explanation on K8S network 

     
  • Wang 19:59 on 2021-06-09 Permalink | Reply
    Tags: , Cluster, ,   

    Slurm 

    Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Slurm requires no kernel modifications for its operation and is relatively self-contained. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.

    https://slurm.schedmd.com/

     
  • Wang 11:31 on 2020-08-13 Permalink | Reply
    Tags: , Cluster, , , , ,   

    Jupyter Gateway + JupyterHub 

    https://jupyter.org/enterprise_gateway/

     
  • Wang 21:38 on 2020-06-13 Permalink | Reply
    Tags: , , Cluster,   

    Data Pipelines with Apache Airflow

     
  • Wang 20:44 on 2019-12-24 Permalink | Reply
    Tags: , Cluster   

    PoC of Apache Druid 

    As we have some business requirements about data aggregation and online processing, so we did a quick PoC on Apache Druid. Next I will show how to build druid quickly and start your ingestion task.

    1.Select release version which is compatible to your existing system and download the package.

    2.Choose what kind of druid service you want to start with

    • For single node, just execute the script under bin directory which is start with start-single-server-, or you can execute start-micro-quickstart
    • For multiple node cluster, please update the configuration files under start-micro-quickstart in one node and sync to other nodes. If you want to connect to your hadoop cluster, please copy corresponding hadoop xml files and kerberos keytab under druid.

    Then you start druid service in every node by execute start-cluster script.

    3.Visit druid through browser, http://IP:8888


    Next I load the data from local file and can ingest the data file as a datasource, and finally query data by SQL.

    Task configuration

    {
      "type": "index_parallel",
      "id": "index_parallel_wikiticker-2015-09-12-sampled_2020-02-18T11:17:29.236Z",
      "resource": {
        "availabilityGroup": "index_parallel_wikiticker-2015-09-12-sampled_2020-02-18T11:17:29.236Z",
        "requiredCapacity": 1
      },
      "spec": {
        "dataSchema": {
          "dataSource": "wikiticker-2015-09-12-sampled",
          "parser": {
            "type": "string",
            "parseSpec": {
              "format": "json",
              "timestampSpec": {
                "column": "time",
                "format": "iso"
              },
              "dimensionsSpec": {
                "dimensions": [
                  "channel",
                  "cityName",
                  "comment",
                  "countryIsoCode",
                  "countryName",
                  "isAnonymous",
                  "isMinor",
                  "isNew",
                  "isRobot",
                  "isUnpatrolled",
                  "namespace",
                  "page",
                  "regionIsoCode",
                  "regionName",
                  "user"
                ]
              }
            }
          },
          "metricsSpec": [
            {
              "type": "count",
              "name": "count"
            },
            {
              "type": "longSum",
              "name": "sum_added",
              "fieldName": "added",
              "expression": null
            },
            {
              "type": "longSum",
              "name": "sum_deleted",
              "fieldName": "deleted",
              "expression": null
            },
            {
              "type": "longSum",
              "name": "sum_delta",
              "fieldName": "delta",
              "expression": null
            },
            {
              "type": "longSum",
              "name": "sum_metroCode",
              "fieldName": "metroCode",
              "expression": null
            }
          ],
          "granularitySpec": {
            "type": "uniform",
            "segmentGranularity": "DAY",
            "queryGranularity": "HOUR",
            "rollup": true,
            "intervals": null
          },
          "transformSpec": {
            "filter": null,
            "transforms": []
          }
        },
        "ioConfig": {
          "type": "index_parallel",
          "firehose": {
            "type": "local",
            "baseDir": "/opt/druid-0.16.0/quickstart/tutorial",
            "filter": "wikiticker-2015-09-12-sampled.json.gz",
            "parser": null
          },
          "appendToExisting": false
        },
        "tuningConfig": {
          "type": "index_parallel",
          "maxRowsPerSegment": null,
          "maxRowsInMemory": 1000000,
          "maxBytesInMemory": 0,
          "maxTotalRows": null,
          "numShards": null,
          "partitionsSpec": null,
          "indexSpec": {
            "bitmap": {
              "type": "concise"
            },
            "dimensionCompression": "lz4",
            "metricCompression": "lz4",
            "longEncoding": "longs"
          },
          "indexSpecForIntermediatePersists": {
            "bitmap": {
              "type": "concise"
            },
            "dimensionCompression": "lz4",
            "metricCompression": "lz4",
            "longEncoding": "longs"
          },
          "maxPendingPersists": 0,
          "forceGuaranteedRollup": false,
          "reportParseExceptions": false,
          "pushTimeout": 0,
          "segmentWriteOutMediumFactory": null,
          "maxNumConcurrentSubTasks": 1,
          "maxRetry": 3,
          "taskStatusCheckPeriodMs": 1000,
          "chatHandlerTimeout": "PT10S",
          "chatHandlerNumRetries": 5,
          "maxNumSegmentsToMerge": 100,
          "totalNumMergeTasks": 10,
          "logParseExceptions": false,
          "maxParseExceptions": 2147483647,
          "maxSavedParseExceptions": 0,
          "partitionDimensions": [],
          "buildV9Directly": true
        }
      },
      "context": {
        "forceTimeChunkLock": true
      },
      "groupId": "index_parallel_wikiticker-2015-09-12-sampled_2020-02-18T11:17:29.236Z",
      "dataSource": "wikiticker-2015-09-12-sampled"
    }
    

    Task running status

    Task finished, you can see the item in datasource/segment/query

     
  • Wang 23:22 on 2019-10-25 Permalink | Reply
    Tags: , Cluster, , , , ,   

    SpringOne Platform 2019 in Austin, https://springoneplatform.io/

     
c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel
%d bloggers like this: