Title: [2014.10.10] Realize the Batch Compression with Building Par2 Repair Files of Each Items @ Linux (Update) [Print this page] author: 夢 time: 2014-9-20 12:07 Title: [2014.10.10] Realize the Batch Compression with Building Par2 Repair Files of Each Items @ Linux (Update)
Limitation & Key point:
01. The size of splited volumes of every items forcing limite to 1GB and the size of every reapir files is as close as possible the volume size unless the compressed source is less then 1GB.
02. If the size of the compressed source is more than 1TB, the script need to improve.
03.Par2 repaire files are building with some rules, and the rules is related with the par2 file name.
04.The script will automatically create some other files are convenient to let me troubleshoot issue and I don't want take off these files, gays can edit the shell script to delect these files yourself (but it is not easy).
05.Before you run the script, please install the rar and par2 in your linux.
06.Before you run the script, please edit some variable content and the content of target file to match your using environment and requirement, I will describe the examples at next part.
07.Before you compiler the source code of test7 ~ test9, please double confirm that you have already installed g++.
08.If the size of compressed source is less then 5GB, the script will force using method 11 (refer the next point) to build the single par2 reapire (core) file.
09.All the items run rar archiver with rr3 and about 15% par2 repair files, if the source size is less then 1GB, the level of redundancy of par2 automatically increase to 100%.
10.Please don't fill out any special characters in every script's variables, but the files name or folders name (blong with $dirWhere location) is allowed.
---releae notes---
2010/10/10 version 1.0
Source code:
A : script
Guys create the script first into at any path you like of your computer, but you need to edit the variable ($programWhere) to match your environment ( see the example ).
#!/bin/sh
# improte the variables
dirWhere='/home/userid/a/'
saveWhere="/home/userid/save/"
programWhere="/home/userid/software/"
findList="/home/userid/software/source"
newNameList="/home/userid/software/target"
tempList="/home/userid/software/templist"
password="P.A.S.S.W.O.R.D"
#10.old ,11-new
par2_method="10"
rarVersion="r5d11fin"
#mode01=unpacket,mode02=packet
information_mode="02"
# check for bad situation 1
if [ ! -f "$newNameList" ] ; then
echo 'error, list does not exist'
exit
fi
# assign the path to te working directory
cd "$dirWhere"
# search and get the the name of working directory
find . -maxdepth 1 > $tempList
cat $tempList | sort > $findList
sed -i "s/\.\///g" $findList
sed -i '/^\.$/d' $findList
cd "$programWhere"
#rm -f $tempList
# check for the bad situation 2
lineTotal_01=` cat "$findList" | wc -l `
lineTotal_02=` cat "$newNameList" | wc -l`
if [ $lineTotal_01 -ne $lineTotal_02 ] ; then
echo 'error, the list numbers does not match with the folders and files nambers of working directory。'
exit
fi
#start to enter the working loop
# 初始化用來當迴圈「計數器」的變數的值
loopTimes=1
echo -e ""
echo "* The core of this scrip start working *"
echo -e ""
# 迴圈開始
while read fileName
do
echo "loopTimes=$loopTimes"
#mapping新檔名
# 新檔名 = 取出「更名列表清單」特定行數的資料
# 行數 = 迴圈計數器的值(由1開始,依序增加讀取)
lineNum=` echo $loopTimes"p" `
newFileName=` sed -n $lineNum $newNameList `
mkdir $saveWhere$newFileName
sleep 2
# 壓縮->修名->重新命名
#壓縮
if [ $loopTimes -ne 0 ] ; then
echo "$fileName" > "$dirWhere"comment
echo -e "" >> "$dirWhere"comment
#echo "* This shell script is created by bdmv.info * " >> "$dirWhere"comment
#echo "* People are only allowed to use the script for nonprofit reserch *" >> "$dirWhere"comment
rar a -m0 -hp"$password" -v1073741824b -rr3 -ep1 -z"$dirWhere"comment $saveWhere"$newFileName"/"$newFileName".$rarVersion.rr3 "$dirWhere""$fileName"
echo -e "\n\n"
echo "echo 1.rar壓縮"
echo -e "\n\n"
sleep 2
fi
#取得壓縮檔完整數量
rarFileNumber=` ls -laR $saveWhere"$newFileName" |grep "^-" |awk 'END{print NR}' `
rarFileSize=` ls -l $saveWhere"$newFileName" | awk '{ sum += $5} END { printf("%.3f\n",sum/1024/1024/1024)}'`
tree "$dirWhere" >> "$saveWhere"ALL.TREE\(FULL.MODE\).txt
tree -d "$dirWhere" >> "$saveWhere"ALL.FOLDER.TREE\(FULL.MODE\).txt
sed -i '2d' "$saveWhere""ALL.TREE(FULL.MODE).txt"
sed -i '2d' "$saveWhere""ALL.FOLDER.TREE(FULL.MODE).txt"
project_start=` sed -n 1p $newNameList `
test "$project_start" != "$project_end" && rar a -m0 -ep1 -hp$password "$saveWhere"\["$project_start"\].to.\["$project_end"\].ALL.INFORMATION.rar "$saveWhere"ALL.FILENAME.txt "$saveWhere"ALL.TREE\(PART.MODE\).txt "$saveWhere"ALL.FILE.MAP.txt "$saveWhere"ALL.TREE\(FULL.MODE\).txt "$saveWhere"ALL.FOLDER.TREE\(FULL.MODE\).txt "$saveWhere"ALL.FOLDER.TREE\(PART.MODE\).txt "$saveWhere"ALL.FILENAME.sfv && test "$information_mode" = "02" && rar a -m0 -ep1 -hp$password "$saveWhere"\("$project_start"\).to.\("$project_end"\).information.v2.packet.rar "$saveWhere"*.information.v2.rar && rm "$saveWhere"*.information.v2.rar
test "$project_start" = "$project_end" && rar a -m0 -ep1 -hp$password "$saveWhere"\["$project_end"\].ALL.INFORMATION.rar "$saveWhere"ALL.FILENAME.txt "$saveWhere"ALL.TREE\(PART.MODE\).txt "$saveWhere"ALL.FILE.MAP.txt "$saveWhere"ALL.TREE\(FULL.MODE\).txt "$saveWhere"ALL.FOLDER.TREE\(FULL.MODE\).txt "$saveWhere"ALL.FOLDER.TREE\(PART.MODE\).txt "$saveWhere"ALL.FILENAME.sfv
#刪除暫存檔
rm "$saveWhere"ALL.FILE.MAP.txt
rm "$saveWhere"ALL.FILENAME.txt
rm "$saveWhere"ALL.TREE\(PART.MODE\).txt
rm "$saveWhere"ALL.TREE\(FULL.MODE\).txt
rm "$saveWhere"ALL.FOLDER.TREE\(PART.MODE\).txt
rm "$saveWhere"ALL.FOLDER.TREE\(FULL.MODE\).txt
rm "$saveWhere"ALL.FILENAME.sfv
rm -f $findList
#rm -f $newNameList
rm -f $tempList
echo -e ""
echo "* All the jobs are finished *"
echo -e ""
Copy the Code
Pay attention of the variables of
dirWhere=batch compressd files source
saveWhere=where are the compressed files saving as
programWhere=where are the test7 to test9 binary files saving as
findList=a specified file script will add source (file/folder) name into automatically
newNameList= the new item name of the compressed source
tempList=a specified file adding some temp information
password=password of the rar achiver
par2_method=only can select the mode 10 or 11
rarVersion=the tag name of rar version
information_mode=only can select the mode 01 or 02
And, not only fill out the absolate path of the variables of dirWhere to tempList ,but assign the file name belonging with findList、 newNameList and tempList.
Final, add the new name inforamtion into the file belonging with newNameList.
Give an example and a illustration to everyone :
1. If we hope to move the 10 folders (these name are 01、02、03... 10 )and 1 file (the name is 11) I want to do compress to the specified location(source locaion = $dirWhere = /home/id/compress/) and specify the name of the compress files of each item are c-01、c-02、c-03 ... c-11 and I want to save all the compressed volume and par2 reapir files to the another location(target location = $saveWhere = /home/id/save/)
2.I create the shell script and test7~test9 (I will describe at next part) at another location(/home/id/script/)
3.Use - P.A.S.S.W.O.R.D as the compress files' password
4.par2_method is a very complex part, I like the mode 10 but , because of people don't understand the throey of mode 10 nad 11, I suggest people use mode 11.
5.The version of rar we use to do compress is 5.11, so we use the word - r5d11fin to be the version tag.
6.Use the mode 01 to get the sparate information of every single item.
7.Edit the variable conents of shell script, like the below parameters:
dirWhere= /home/id/compress/
saveWhere=location /home/id/save/
programWhere=/home/id/script/
findList=/home/id/script/source
newNameList= /home/id/script/target
tempList=/home/id/script/temp
password="P.A.S.S.W.O.R.D"
par2_method="10"
rarVersion="r5d11fin"
information_mode="01"
8.I also need to edit the file of (newNameList) - /home/id/script/target and improte the content liking the below block
c-01
c-02
c-03
c-04
c-05
c-06
c-07
c-08
c-09
c-10
Copy the Code
Give everyone another example picture to illustrate the information_mode 2
Like the above example, the files (and folders) of left putty window belog to $dirWhere and the location of center window belong to $saveWhere.
We arrived the result to know the content of target file are G201405.No-01、G201405.No-02、G201405.No-03 ... by the illustration of deduction .
The illustration is the saame as the above example using the information mode2m people can see the center region ((G201405.No-01).to.(G201405.No-91).information.v2.packet.rar) of the illusration.
B : Other source code
The test7 ~ test9 can help the script to calculate some parameters and load the parameters into par2 to build par2 files, before running the script ,people must compiler these source codes youself and move the program into the location belong to $programWhere
How to compiler the source code, people can refer this command.
After adding the shell script (script.sh) and move test7~test9 to the same folder, change the script's premission then we can use the scirpt to add a job.
---release notes---
2014/10/10 version 1.1b
1.Add new variable $dataWhere, fill out the information in real patch of data source,then the script will auto move the datato $dirWhere and move back the data after finish the cimpression.
2.The contents of test7 to test9 are the same.
2014/10/10 version 1.2b
1.Fix the variable of the part of moving files and allow the special character file/folder name!!
2.The contents of test7 to test9 are the same.
Source code of script
#!/bin/sh
# improte the variables
dataWhere='/home/userid/data/'
dirWhere='/home/userid/a/'
saveWhere="/home/userid/save/"
programWhere="/home/userid/software/"
findList="/home/userid/software/source"
newNameList="/home/userid/software/target"
tempList="/home/userid/software/templist"
password="P.A.S.S.W.O.R.D"
#10.old ,11-new
par2_method="10"
rarVersion="r5d11fin"
#mode01=unpacket,mode02=packet
information_mode="02"
#collect data
while read fileName
do
mv $dataWhere"$fileName" $dirWhere
done < $findList
echo "* The data is finish collected *"
# check for bad situation 1
if [ ! -f "$newNameList" ] ; then
echo 'error, list does not exist'
exit
fi
# assign the path to te working directory
cd "$dirWhere"
# search and get the the name of working directory
find . -maxdepth 1 > $tempList
cat $tempList | sort > $findList
sed -i "s/\.\///g" $findList
sed -i '/^\.$/d' $findList
cd "$programWhere"
#rm -f $tempList
# check for the bad situation 2
lineTotal_01=` cat "$findList" | wc -l `
lineTotal_02=` cat "$newNameList" | wc -l`
if [ $lineTotal_01 -ne $lineTotal_02 ] ; then
echo 'error, the list numbers does not match with the folders and files nambers of working directory。'
exit
fi
#start to enter the working loop
# 初始化用來當迴圈「計數器」的變數的值
loopTimes=1
echo -e ""
echo "* The core of this scrip start working *"
echo -e ""
# 迴圈開始
while read fileName
do
echo "loopTimes=$loopTimes"
#mapping新檔名
# 新檔名 = 取出「更名列表清單」特定行數的資料
# 行數 = 迴圈計數器的值(由1開始,依序增加讀取)
lineNum=` echo $loopTimes"p" `
newFileName=` sed -n $lineNum $newNameList `
mkdir $saveWhere$newFileName
sleep 2
# 壓縮->修名->重新命名
#壓縮
if [ $loopTimes -ne 0 ] ; then
echo "$fileName" > "$dirWhere"comment
echo -e "" >> "$dirWhere"comment
#echo "* This shell script is created by bdmv.info * " >> "$dirWhere"comment
#echo "* People are only allowed to use the script for nonprofit reserch *" >> "$dirWhere"comment
rar a -m0 -hp"$password" -v1073741824b -rr3 -ep1 -z"$dirWhere"comment $saveWhere"$newFileName"/"$newFileName".$rarVersion.rr3 "$dirWhere""$fileName"
echo -e "\n\n"
echo "echo 1.rar壓縮"
echo -e "\n\n"
sleep 2
fi
#取得壓縮檔完整數量
rarFileNumber=` ls -laR $saveWhere"$newFileName" |grep "^-" |awk 'END{print NR}' `
rarFileSize=` ls -l $saveWhere"$newFileName" | awk '{ sum += $5} END { printf("%.3f\n",sum/1024/1024/1024)}'`