Updates from May, 2018 Toggle Comment Threads | Keyboard Shortcuts

  • Wang 17:52 on 2018-05-19 Permalink | Reply

    New domain!! hongmeng.me

  • Wang 23:42 on 2018-05-11 Permalink | Reply
    Tags: , , , ,   

    Website down 

    Today I received alert email suddenly which said my blog site went down…😂😂😂

    So I logged in server and checked containers’s status, everything looked fine

    [root@blog xiaowang]# docker stack ps blog
    ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
    qwsjjol3jk2f        blog_mysql.1        mysql:5.7           blog                Running             Running 15 days ago                       
    n9gbil4zcavy        blog_nginx.1        nginx:1.13.8        blog                Running             Running 15 days ago                       
    hg778gcc35vz        blog_wordpress.1    wordpress:4.9.1     blog                Running             Running 15 days ago

    When I checked the port, everything also looked fine

    [root@blog xiaowang]# netstat -tuapn | egrep '80|443'
    tcp6       4      0 :::80                   :::*                    LISTEN      12146/dockerd       
    tcp6       2      0 :::443                  :::*                    LISTEN      12146/dockerd       
    tcp6      74      0 ::1:80                  ::1:47352               CLOSE_WAIT  -                   
    tcp6       3      0 ::1:80                  ::1:47348               CLOSE_WAIT  -                   
    tcp6      74      0 ::1:80                  ::1:47402               CLOSE_WAIT  -                   
    tcp6      78      0 ::1:443                 ::1:56994               CLOSE_WAIT  -                   
    tcp6      78      0 ::1:443                 ::1:56944               CLOSE_WAIT  -                   
    tcp6      74      0 ::1:80                  ::1:47350               CLOSE_WAIT  -

    But when I executed “curl http://localhost, it was blocked, so I guess something wrong with local network.

    After checking I executed “sysctl -w net.ipv4.ip_forward=1” to enable ip forward, and I finally could access the port. So I executed “echo “net.ipv4.ip_forward=1″ >> /etc/sysctl.conf” to make it permanent.

    I’m using google cloud, I guess maybe they have reset the network which I didn’t make it permanent before.

  • Wang 21:46 on 2018-05-08 Permalink | Reply
    Tags: ,   

    wuxia beauties..

  • Wang 22:37 on 2018-05-07 Permalink | Reply
    Tags: ,   

    [Presto] Kerberos trouble shooting 

    When I configured presto cluster to connect hive by kerberos, I met some problems which cost me too much time to solve them, so I summarized the problems, hope could help others.

    1.Append -Djava.security.krb5.conf=”krb5.conf location” to etc/jvm.properties

    8) Error in custom provider, java.lang.NoClassDefFoundError: Could not initialize class com.facebook.presto.hive.authentication.KerberosHadoopAuthentication
      at com.facebook.presto.hive.authentication.AuthenticationModules$1.createHadoopAuthentication(AuthenticationModules.java:59) (via modules: com.facebook.presto.hive.authentication.HiveAuthenticationModule -> io.airlift.configuration.ConditionalModule -> com.facebook.presto.hive.authentication.AuthenticationModules$1)
      while locating com.facebook.presto.hive.authentication.HadoopAuthentication annotated with @com.facebook.presto.hive.ForHiveMetastore()
        for the 2nd parameter of com.facebook.presto.hive.authentication.KerberosHiveMetastoreAuthentication.<init>(KerberosHiveMetastoreAuthentication.java:44)

    2.Specify hdfs-site.xml/core-site.xml in hive.properties like hive.config.resources=xxx/core-site.xml,xxx/hdfs-site.xml

    Query 20180504_150148_00018_v6ndf failed: java.net.UnknownHostException: xxx

    3.Download hadoop-lzo jar into plugin/hive-hadoop2

    Query 20180504_150959_00002_3f2qe failed: Unable to create input format org.apache.hadoop.mapred.TextInputFormat
    Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
            at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139)
            at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:180)
            at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
            ... 19 more
    Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
            at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
            at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132)
            ... 21 more

    4.Export KRB5_CONFIG & get kerberos tgt, use kinit command

    Query 20180504_153940_00000_nrsgy failed: Failed to list directory: hdfs://xxx/user/hive/warehouse/xxx.db/xxx

    5.More than one coordinator in the cluster

    2018-05-04T18:10:56.410Z   WARN    http-worker-4560    com.facebook.presto.execution.SqlTaskManager    Switching coordinator affinity from hhbts to qhnep
    2018-05-04T18:10:56.500Z    WARN    http-worker-4560    com.facebook.presto.execution.SqlTaskManager    Switching coordinator affinity from qhnep to c83wr
    2018-05-04T18:10:56.578Z    WARN    http-worker-4395    com.facebook.presto.execution.SqlTaskManager    Switching coordinator affinity from c83wr to ujj9n
    2018-05-04T18:10:56.749Z    WARN    http-worker-4432    com.facebook.presto.execution.SqlTaskManager    Switching coordinator affinity from ujj9n to wdsxf
    2018-05-04T18:10:57.009Z    WARN    http-worker-4584    com.facebook.presto.execution.SqlTaskManager    Switching coordinator affinity from wdsxf to hhbts
  • Wang 16:56 on 2018-05-02 Permalink | Reply
    Tags: , , , ,   

    [Presto] Connect hive by kerberos 

    For data security, hadoop cluster usually implement different security mechanisms, most commonly used mechanism is kerberos. Recently I tested how to connect hive by kerberos in presto.

    1.Add krb5.conf/keytab/hdfs-site.xml/core-site.xml in every node.

    2.Modify etc/jvm.properties, append -Djava.security.krb5.conf=”krb5.conf location”

    3.Create hive.properties under etc/catalog

    cat << 'EOF' > etc/catalog/hive.properties
    hive.metastore.client.keytab="keytab location"
    hive.config.resources="core-site.xml and hdfs-site.xml" location

    4.Download hadoop-lzo jar into plugin/hive-hadoop2

    wget http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.jar -O plugin/hive-hadoop2

    5.Get principal tgt

    export KRB5_CONFIG="krb5.conf location"
    kinit -kt "keytab location" xxx@xxx.com

    6.Restart presto

    bin/launcher restart
Compose new post
Next post/Next comment
Previous post/Previous comment
Show/Hide comments
Go to top
Go to login
Show/Hide help
shift + esc
%d bloggers like this: