HsunTzu version Beat 1.0 Very Fast Hdfs Origin File To Compress Decompress Untar Tarball LISENCE. MIT 工欲善其事必先利其器 Good tools are prerequisite to the successful execution of a job First You need install jdk 8 scala 2.12.1 + sbt 1.0.4 + hadoop 2.8.1 + , also you can edit the version on build.sbt and ./project/build.properties Get git clone git@github.com:mullerhai/HsunTzu.git cd ./HsunTzu Compile sbt clean compile Package sbt update sbt assembly Run hadoop jar HsunTzuPro-beat-2.0.jar inputPath outPath CompressType PropertiesFilePath inputCodec OutputCodec you will see the logger info on console output something. before run. you need to know. CompressType. use. number instead of. compress method invoke case "1" => exec.originFileToCompressFile case "2" => exec.tarFileToOriginFile case "3" => exec.compressFileToOriginFile case "4" => exec.oneCompressConvertOtherCompress compress codec use num instead of. codec class ,example. use. 1 instead of snappycodec case "0" => deflateCode case "1" => snappyCodec case "2" => gzipCodec case "3" => lz4Codec case "4" => bZip2Codec case "5" => defaultCodec case _ => deflateCodec PropertiesFilePath you need create one property file , example. /usr/local/info.properties you need decleard the file prefix in. Key.[files]. of your property file. for. select compress or decompress or untar if. you want to. decleard. the. hdfs address and port and operetion user ,you need. to fill the file. in. property file like this. intention. the key. must. like this [ hdfsAddr hdfsPort FsKey. FsUserKey. hadoopUser HDFSPORTDOTSUFFIX. files] !! hdfsAddr=hdfs://192.168.255.161:9000 hdfsPort=9000 FsKey=fs.defaultFS FsUserKey=HADOOP_USER_NAME hadoopUser=linkedme_hadoop HDFSPORTDOTSUFFIX= :9000/ files=biz,ad_status,ad_behavior argument you need to. declard six ,maybe. the last argument is not can use. rum example. hadoop jar HsunTzuPro-beat-2.0.jar /facishare-data/taru/20170820 /facishare-data/gao 1 /usr/local/info.properties 1 0 this. is just for originFile.to CompressFile, and. compress file use. snappyCodec. hadoop jar HsunTzuPro-beat-2.0.jar /facishare-data/taruns/taruns/tarun/20170820/ /facishare-data/xin 4 /usr/local/info.properties 0 1 this is just. for deflateCodec compress files. convert to snappyCodec Compress files. oneCompressConvertOtherCompress hadoop jar HsunTzuPro-beat-2.0.jar /facishare-data/taruns/taruns/tarun/20170820/ /facishare-data/xin 3 /usr/local/info.properties 3 0 this is just lz4Codec Compress files. Decompress. to. origin files hadoop jar HsunTzuPro-beat-2.0.jar /facishare-data/taruns/taruns/tarun/20170820/ /facishare-data/xin 2 /usr/local/info.properties 0 0 this is just tarball file. to origin file
