New domain!! hongmeng.me
Updates from May, 2018 Toggle Comment Threads | Keyboard Shortcuts
-
Wang
-
Wang
Website down
Today I received alert email suddenly which said my blog site went down…😂😂😂
So I logged in server and checked containers’s status, everything looked fine
[root@blog xiaowang]# docker stack ps blog ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS qwsjjol3jk2f blog_mysql.1 mysql:5.7 blog Running Running 15 days ago n9gbil4zcavy blog_nginx.1 nginx:1.13.8 blog Running Running 15 days ago hg778gcc35vz blog_wordpress.1 wordpress:4.9.1 blog Running Running 15 days ago
When I checked the port, everything also looked fine
[root@blog xiaowang]# netstat -tuapn | egrep '80|443' tcp6 4 0 :::80 :::* LISTEN 12146/dockerd tcp6 2 0 :::443 :::* LISTEN 12146/dockerd tcp6 74 0 ::1:80 ::1:47352 CLOSE_WAIT - tcp6 3 0 ::1:80 ::1:47348 CLOSE_WAIT - tcp6 74 0 ::1:80 ::1:47402 CLOSE_WAIT - tcp6 78 0 ::1:443 ::1:56994 CLOSE_WAIT - tcp6 78 0 ::1:443 ::1:56944 CLOSE_WAIT - tcp6 74 0 ::1:80 ::1:47350 CLOSE_WAIT -
But when I executed “curl http://localhost“, it was blocked, so I guess something wrong with local network.
After checking I executed “sysctl -w net.ipv4.ip_forward=1” to enable ip forward, and I finally could access the port. So I executed “echo “net.ipv4.ip_forward=1″ >> /etc/sysctl.conf” to make it permanent.
I’m using google cloud, I guess maybe they have reset the network which I didn’t make it permanent before.
-
Wang
wuxia beauties..
-
Wang
[Presto] Kerberos trouble shooting
When I configured presto cluster to connect hive by kerberos, I met some problems which cost me too much time to solve them, so I summarized the problems, hope could help others.
1.Append -Djava.security.krb5.conf=”krb5.conf location” to etc/jvm.properties
8) Error in custom provider, java.lang.NoClassDefFoundError: Could not initialize class com.facebook.presto.hive.authentication.KerberosHadoopAuthentication at com.facebook.presto.hive.authentication.AuthenticationModules$1.createHadoopAuthentication(AuthenticationModules.java:59) (via modules: com.facebook.presto.hive.authentication.HiveAuthenticationModule -> io.airlift.configuration.ConditionalModule -> com.facebook.presto.hive.authentication.AuthenticationModules$1) while locating com.facebook.presto.hive.authentication.HadoopAuthentication annotated with @com.facebook.presto.hive.ForHiveMetastore() for the 2nd parameter of com.facebook.presto.hive.authentication.KerberosHiveMetastoreAuthentication.<init>(KerberosHiveMetastoreAuthentication.java:44) ... ...2.Specify hdfs-site.xml/core-site.xml in hive.properties like hive.config.resources=xxx/core-site.xml,xxx/hdfs-site.xml
Query 20180504_150148_00018_v6ndf failed: java.net.UnknownHostException: xxx
3.Download hadoop-lzo jar into plugin/hive-hadoop2
Query 20180504_150959_00002_3f2qe failed: Unable to create input format org.apache.hadoop.mapred.TextInputFormat Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found. at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139) at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:180) at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45) ... 19 more Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132) ... 21 more4.Export KRB5_CONFIG & get kerberos tgt, use kinit command
Query 20180504_153940_00000_nrsgy failed: Failed to list directory: hdfs://xxx/user/hive/warehouse/xxx.db/xxx
5.More than one coordinator in the cluster
2018-05-04T18:10:56.410Z WARN http-worker-4560 com.facebook.presto.execution.SqlTaskManager Switching coordinator affinity from hhbts to qhnep 2018-05-04T18:10:56.500Z WARN http-worker-4560 com.facebook.presto.execution.SqlTaskManager Switching coordinator affinity from qhnep to c83wr 2018-05-04T18:10:56.578Z WARN http-worker-4395 com.facebook.presto.execution.SqlTaskManager Switching coordinator affinity from c83wr to ujj9n 2018-05-04T18:10:56.749Z WARN http-worker-4432 com.facebook.presto.execution.SqlTaskManager Switching coordinator affinity from ujj9n to wdsxf 2018-05-04T18:10:57.009Z WARN http-worker-4584 com.facebook.presto.execution.SqlTaskManager Switching coordinator affinity from wdsxf to hhbts
-
Wang
[Presto] Connect hive by kerberos
For data security, hadoop cluster usually implement different security mechanisms, most commonly used mechanism is kerberos. Recently I tested how to connect hive by kerberos in presto.
1.Add krb5.conf/keytab/hdfs-site.xml/core-site.xml in every node.
2.Modify etc/jvm.properties, append -Djava.security.krb5.conf=”krb5.conf location”
3.Create hive.properties under etc/catalog
cat << 'EOF' > etc/catalog/hive.properties connector.name=hive-hadoop2 hive.metastore.uri=thrift://xxx:9083 hive.metastore.authentication.type=KERBEROS hive.metastore.service.principal=xxx@xxx.com hive.metastore.client.principal=xxx@xxx.com hive.metastore.client.keytab="keytab location" hive.config.resources="core-site.xml and hdfs-site.xml" location EOF
4.Download hadoop-lzo jar into plugin/hive-hadoop2
wget http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.jar -O plugin/hive-hadoop2
5.Get principal tgt
export KRB5_CONFIG="krb5.conf location" kinit -kt "keytab location" xxx@xxx.com
6.Restart presto
bin/launcher restart
Wang's Tech Blog 
Reply