24.Azkaban调度脚本的编写
启动azkaban
在启动了rver和excutor之后，在浏览器打开azkaban，会发现不能打开，⽇志报这个错误
at sun.curity.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.curity.ad(InputRecord.java:527)
at sun.curity.adRecord(SSLSocketImpl.java:975)
at sun.curity.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.curity.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.curity.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
bay.jetty.curity.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
bay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2019/09/0115:18:24.499 +0800 WARN [log] [Azkaban] EXCEPTION
javax.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.curity.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.curity.ad(InputRecord.java:527)
at sun.curity.adRecord(SSLSocketImpl.java:975)
at sun.curity.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.curity.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.curity.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
bay.jetty.curity.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
bay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2019/09/0115:18:24.680 +0800 WARN [log] [Azkaban] EXCEPTION
javax.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.curity.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.curity.ad(InputRecord.java:527)
at sun.curity.adRecord(SSLSocketImpl.java:975)
at sun.curity.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.curity.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.curity.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
bay.jetty.curity.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
bay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2019/09/0115:18:24.809 +0800 WARN [log] [Azkaban] EXCEPTION
javax.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.curity.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
jon huntsman
at sun.curity.ad(InputRecord.java:527)
at sun.curity.adRecord(SSLSocketImpl.java:975)
at sun.curity.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.curity.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.curity.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
bay.jetty.curity.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
bay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2019/09/0115:26:14.746 +0800 WARN [log] [Azkaban] EXCEPTION
javax.ssl.SSLHandshakeException: Remote host clod connection during handshake
at sun.curity.adRecord(SSLSocketImpl.java:994)
at sun.curity.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.curity.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.curity.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
bay.jetty.curity.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
bay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caud by: java.io.EOFException: SSL peer shut down incorrectly
at sun.curity.ad(InputRecord.java:505)
at sun.curity.adRecord(SSLSocketImpl.java:975)
... 5 more
注意了，之前⼀直不能打开是因为使⽤了http，这⾥强调⼀下，⼀定不能⽤http，必须⽤https，然后选择⾼级选项，运⾏访问不安全的地址，因为这个地址被默认为不安全的
编写upload.job脚本
# upload.job
type=command
command=bash uploadFile2Hdfs.sh
编写uploadFile2Hdfs.sh脚本
#!/bin/bash
#t java env
export JAVA_HOME=/opt/modules/jdk1.8.0_65
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
#t hadoop env
export HADOOP_HOME=/opt/modules/hadoop-2.6.0
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH
#版本1的问题：
#虽然上传到Hadoop集群上了，但是原始⽂件还在。如何处理？
#⽇志⽂件的名称都是xxxx.log1，再次上传⽂件时，因为hdfs上已经存在了，会报错。如何处理？
#如何解决版本1的问题
# 1、先将需要上传的⽂件移动到待上传⽬录
# 2、在讲⽂件移动到待上传⽬录时，将⽂件按照⼀定的格式重名名
# /export/software/hadoop.log1 /export/data/click_log/xxxxx_click_log_{date}
the next
#⽇志⽂件存放的⽬录
log_src_dir=/home/hadoop/logs/log/
#待上传⽂件存放的⽬录
log_toupload_dir=/home/hadoop/logs/toupload/
day_01=`date -d'-1 day' +%Y-%m-%d`
syear=`date --date=$day_01 +%Y`
smonth=`date --date=$day_01 +%m`
sday=`date --date=$day_01 +%d`
英语中的感叹句
#echo $day_01
#echo $syear
#echo $smonth
#echo $sday
#⽇志⽂件上传到hdfs的根路径
soutechdfs_root_dir=/data/clickLog/$syear/$smonth/$sday
hadoop fs -mkdir -p $hdfs_root_dir
#打印环境变量信息
echo "envs: hadoop_home: $HADOOP_HOME"
#读取⽇志⽂件的⽬录，判断是否有需要上传的⽂件
echo "log_src_dir:"$log_src_dir
ls $log_src_dir | while read fileName
do
if [[ "$fileName" == access.log ]]; then
# if [ "access.log" = "$fileName" ];then
date=`date +%Y_%m_%d_%H_%M_%S`
it是什么意思#将⽂件移动到待上传⽬录并重命名
#打印信息
echo "moving $log_src_dir$fileName to $log_toupload_dir"xxxxx_click_log_$fileName"$date"
mv $log_src_dir$fileName $log_toupload_dir"xxxxx_click_log_$fileName"$date
#将待上传的⽂件path写⼊⼀个列表⽂件willDoing
echo $log_toupload_dir"xxxxx_click_log_$fileName"$date >> $log_toupload_dir"willDoing."$date
fi
done
#找到列表⽂件willDoing
ls $log_toupload_dir | grep will |grep -v "_COPY_" | grep -v "_DONE_" | while read line
do
#打印信息
echo "toupload is in file:"$line
#将待上传⽂件列表willDoing改名为willDoing_COPY_
mv $log_toupload_dir$line $log_toupload_dir$line"_COPY_"
#读列表⽂件willDoing_COPY_的内容（⼀个⼀个的待上传⽂件名），此处的line 就是列表中的⼀个待上传⽂件的path cat $log_toupload_dir$line"_COPY_" |while read linedde是什么意思
do
#打印信息
echo "$line to $hdfs_root_dir"
hadoop fs -put $line $hdfs_root_dir
done
mv $log_toupload_dir$line"_COPY_" $log_toupload_dir$line"_DONE_"
done
然后把这两个脚本打包
进⼊azkaban的登录页⾯
gugu5
independenceday选择刚刚打包的upflow.zip
嘉华培训怎么样
这个时候把⽇志⽂件上传到集群上
查看HDFS⽬录，证明⽂件上传成功
北京新东方英语培训
