GNU Parallel s, i GNU StowZiua Libertat, ii Programelor s, i 30 de ani de GNU ın Balt, i
Radu Dumbraveanu
Universitatea de Stat “A. Russo” din Balt, i, Republica Moldova
Balt, i, 2013
GNU Parallel
GNU parallel este un program cu interfat, a ın linia de comanda(CLI) care permite utilizatorului executarea ın paralel a uneicomenzi sau a unui set de comenzi (script shell).
Sintaxa:
parallel [optiuni] [comanda [argumente]] < lista de argumente
sau
parallel [optiuni] [comanda [argumente]] ::: argumente
GNU Parallel
GNU parallel este un program cu interfat, a ın linia de comanda(CLI) care permite utilizatorului executarea ın paralel a uneicomenzi sau a unui set de comenzi (script shell).
Sintaxa:
parallel [optiuni] [comanda [argumente]] < lista de argumente
sau
parallel [optiuni] [comanda [argumente]] ::: argumente
Exemple clasice de utilizare
(echo ls; echo grep root /etc/passwd) | parallel
parallel traceroute ::: foss.org.my debian.org freenetproject.org
parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org
find . −type f | parallel gzip
ls ∗.gz | parallel −X mv {} archive
Exemple clasice de utilizare
(echo ls; echo grep root /etc/passwd) | parallel
parallel traceroute ::: foss.org.my debian.org freenetproject.org
parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org
find . −type f | parallel gzip
ls ∗.gz | parallel −X mv {} archive
Exemple clasice de utilizare
(echo ls; echo grep root /etc/passwd) | parallel
parallel traceroute ::: foss.org.my debian.org freenetproject.org
parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org
find . −type f | parallel gzip
ls ∗.gz | parallel −X mv {} archive
Exemple clasice de utilizare
(echo ls; echo grep root /etc/passwd) | parallel
parallel traceroute ::: foss.org.my debian.org freenetproject.org
parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org
find . −type f | parallel gzip
ls ∗.gz | parallel −X mv {} archive
Exemple clasice de utilizare
(echo ls; echo grep root /etc/passwd) | parallel
parallel traceroute ::: foss.org.my debian.org freenetproject.org
parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org
find . −type f | parallel gzip
ls ∗.gz | parallel −X mv {} archive
Dificultat, i?
Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:
The program ’parallel’ is currently not installed. You caninstall it by typing:sudo apt−get install moreutils
Nu urmat, i acesta indicat, ie; comanda parallel din pachetulmoreutils este cu totul altceva.
Dificultat, i?
Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:
The program ’parallel’ is currently not installed. You caninstall it by typing:sudo apt−get install moreutils
Nu urmat, i acesta indicat, ie; comanda parallel din pachetulmoreutils este cu totul altceva.
Dificultat, i?
Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:
The program ’parallel’ is currently not installed. You caninstall it by typing:sudo apt−get install moreutils
Nu urmat, i acesta indicat, ie; comanda parallel din pachetulmoreutils este cu totul altceva.
Modalitat, i de instalare
Putem instala parallel ın felul urmator:
1 sudo apt−get install parallel
sau
1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz22 tar xf parallel−20130922.tar.bz23 cd parallel−201309224 ./configure5 make6 make install
20130922 = versiunea parallel din 22.09.2013
Modalitat, i de instalare
Putem instala parallel ın felul urmator:
1 sudo apt−get install parallel
sau
1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz22 tar xf parallel−20130922.tar.bz23 cd parallel−201309224 ./configure5 make6 make install
20130922 = versiunea parallel din 22.09.2013
Modalitat, i de instalare
Putem instala parallel ın felul urmator:
1 sudo apt−get install parallel
sau
1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz22 tar xf parallel−20130922.tar.bz23 cd parallel−201309224 ./configure5 make6 make install
20130922 = versiunea parallel din 22.09.2013
Modalitat, i de instalare
Daca nu sıntet, i root atunci:creat, i dosarele ~/bin s, i ~/share apoi
1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz22 tar xf parallel−20130922.tar.bz23 cd parallel−201309224 ./configure −−prefix=$HOME5 make6 make install
Modalitat, i de instalare
Sau daca dorit, i sa utilizat, i parallel fara a-l instala (de exemplupentru a experimenta):
1 wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel2 chmod 755 parallel3 cp parallel sem4 mv parallel sem ˜/bin/
parallel este un script Perl, nu ai ce compila.Mai multa informat, ie pehttps://github.com/razrichter/gnu_parallel.
Modalitat, i de instalare
Sau daca dorit, i sa utilizat, i parallel fara a-l instala (de exemplupentru a experimenta):
1 wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel2 chmod 755 parallel3 cp parallel sem4 mv parallel sem ˜/bin/
parallel este un script Perl, nu ai ce compila.Mai multa informat, ie pehttps://github.com/razrichter/gnu_parallel.
Dificultat, i?
Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:
1 /bin/bash: {}: command not found
sau
1 /bin/bash: :::: command not found
Solut, ii:
I adaugat, i opt, iunea --gnu (parallel --gnu ...);
I sau sudo rm /etc/parallael/config;
I sau http://stackoverflow.com/questions/16448887/
gnu-parallel-not-working-at-all
Dificultat, i?
Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:
1 /bin/bash: {}: command not found
sau
1 /bin/bash: :::: command not found
Solut, ii:
I adaugat, i opt, iunea --gnu (parallel --gnu ...);
I sau sudo rm /etc/parallael/config;
I sau http://stackoverflow.com/questions/16448887/
gnu-parallel-not-working-at-all
Dificultat, i?
Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:
1 /bin/bash: {}: command not found
sau
1 /bin/bash: :::: command not found
Solut, ii:
I adaugat, i opt, iunea --gnu (parallel --gnu ...);
I sau sudo rm /etc/parallael/config;
I sau http://stackoverflow.com/questions/16448887/
gnu-parallel-not-working-at-all
Dificultat, i?
Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:
1 /bin/bash: {}: command not found
sau
1 /bin/bash: :::: command not found
Solut, ii:
I adaugat, i opt, iunea --gnu (parallel --gnu ...);
I sau sudo rm /etc/parallael/config;
I sau http://stackoverflow.com/questions/16448887/
gnu-parallel-not-working-at-all
Exemple de utilizare (divizarea fis, ierelor mari pe blocuri)
cat fisiermare.sql | parallel −k −−pipe grep −i ”text”
cat bigfiledump.sql | parallel −k −−pipe −−block 2M \grep −i ”text”
cat bigfiledump.sql | parallel −k −−pipe −−block 2M \−−sshlogin server.example.com,server2.example.net \grep −i ”text”
Exemple de utilizare (divizarea fis, ierelor mari pe blocuri)
cat fisiermare.sql | parallel −k −−pipe grep −i ”text”
cat bigfiledump.sql | parallel −k −−pipe −−block 2M \grep −i ”text”
cat bigfiledump.sql | parallel −k −−pipe −−block 2M \−−sshlogin server.example.com,server2.example.net \grep −i ”text”
Modalitat, i de transmitere a parametrilor
Cum transmitem parametrii la comanda care trebuie executata ınparalel?
1) Prin intrarea standard:
1 seq 5 | parallel echo
2 13 24 35 46 5
Modalitat, i de transmitere a parametrilor
Cum transmitem parametrii la comanda care trebuie executata ınparalel?1) Prin intrarea standard:
1 seq 5 | parallel echo
2 13 24 35 46 5
Modalitat, i de transmitere a parametrilor
Accesul la parametri:
1 seq 5 | parallel echo ”parm={}”
2 param=13 param=24 param=35 param=46 param=5
Modalitat, i de transmitere a parametrilor
Mai mult, i parametri deodata:
1 seq 10 | parallel −X echo
2 1 2 3 4 53 6 7 8 9 10
Modalitat, i de transmitere a parametrilor
2) Prin fis, iere:
1 parallel wget < adresse.lst
1 cat adrese.lst
2 foss.org.my3 debian.org4 freenetproject.org
Modalitat, i de transmitere a parametrilor
3) Prin linia de comanda:
1 parallel echo ::: 1 2 3 4
1 12 23 34 4
sau
1 parallel unzip ::: ./∗.zip
Modalitat, i de transmitere a parametrilor
3) Prin linia de comanda:
1 parallel echo ::: 1 2 3 4
1 12 23 34 4
sau
1 parallel unzip ::: ./∗.zip
Specificarea numarul de procese paralele
1) Numarul de procese = numarul de procesoare:
1 seq 10 20 | parallel −j+0 sleep
1 ps f −o pid,command
2 PID COMMAND3 31379 −bash4 32270 \ ps f −o pid,command5 30463 −bash6 30579 \ mc7 30581 \ bash −rcfile .bashrc8 32239 \ /usr/bin/perl −w /usr/bin/parallel −j+0 sleep9 32268 \ sleep 10
10 32269 \ sleep 11
Specificarea numarul de procese paralele
2) Numarul maximal de procese, explicit:
1 seq 10 20 | parallel −j 5 sleep
1 ps f −o pid,command
2 PID COMMAND3 31379 −bash4 32523 \ ps f −o pid,command5 30463 −bash6 30579 \ mc7 30581 \ bash −rcfile .bashrc8 32487 \ /usr/bin/perl −w /usr/bin/parallel −j 5 sleep9 32518 \ sleep 10
10 32519 \ sleep 1111 32520 \ sleep 1212 32521 \ sleep 1313 32522 \ sleep 14
Specificarea numarul de procese paralele
3) Numarul de sarcini per procesor, explicit:
1 $ seq 10 20 | parallel −j 150% sleep
1 ps f −o pid,command
2 PID COMMAND3 31379 −bash4 32636 \ ps f −o pid,command5 30463 −bash6 30579 \ mc7 30581 \ bash −rcfile .bashrc8 32602 \ /usr/bin/perl −w /usr/bin/parallel −j 150% sleep9 32632 \ sleep 10
10 32633 \ sleep 1111 32634 \ sleep 12
Contraexemplu
1 cd dosarcu66925fisiere2 /usr/bin/time cp −r ∗ /tmp
3 1.77user 49.62system 6:49.01elapsed 12%CPU4 7692784inputs+7367600outputs (0major+320minor)
1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j+0 cp −r {} /tmp
3 2.40user 37.61system 6:30.38elapsed 10%CPU4 7402904inputs+7367600outputs (21major+27017minor)
1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j 150% cp −r {} /tmp
3 2.51user 50.74system 12:55.95elapsed 6%CPU4 7579504inputs+7367608outputs (27major+28524minor)
Contraexemplu
1 cd dosarcu66925fisiere2 /usr/bin/time cp −r ∗ /tmp
3 1.77user 49.62system 6:49.01elapsed 12%CPU4 7692784inputs+7367600outputs (0major+320minor)
1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j+0 cp −r {} /tmp
3 2.40user 37.61system 6:30.38elapsed 10%CPU4 7402904inputs+7367600outputs (21major+27017minor)
1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j 150% cp −r {} /tmp
3 2.51user 50.74system 12:55.95elapsed 6%CPU4 7579504inputs+7367608outputs (27major+28524minor)
Contraexemplu
1 cd dosarcu66925fisiere2 /usr/bin/time cp −r ∗ /tmp
3 1.77user 49.62system 6:49.01elapsed 12%CPU4 7692784inputs+7367600outputs (0major+320minor)
1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j+0 cp −r {} /tmp
3 2.40user 37.61system 6:30.38elapsed 10%CPU4 7402904inputs+7367600outputs (21major+27017minor)
1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j 150% cp −r {} /tmp
3 2.51user 50.74system 12:55.95elapsed 6%CPU4 7579504inputs+7367608outputs (27major+28524minor)
Adrese web
Pagina web
http://www.gnu.org/software/parallel
Manual s, i exemple
http://www.gnu.org/software/parallel/man.html
Surselehttp://savannah.gnu.org/git/?group=parallel
Adrese web
Tutorialehttp://savannah.gnu.org/projects/parallel
Video (partea I s, i II)
http://www.youtube.com/watch?v=OpaiGYxkSuQ
http://www.youtube.com/watch?v=P40akGWJ_gY
Miscelaneuhttp://static.usenix.org/publications/login/2011-02/pdfs/Tange.pdf
http://wiki.scinethpc.ca/wiki/images/7/7b/Tech-talk-gnu-parallel.pdf
http://www.luga.de/Angebote/Vortraege/GNU_Parallel_LIT_2011/GNU_
Parallel_LIT_2011.pdf
http://www.admin-magazine.com/HPC/Articles/
GNU-Parallel-Multicore-at-the-Command-Line-with-GNU-Parallel
Adrese web
Comparat, ie cu alte comenzi (xargs, find -exec, make -j etc)
http://www.gnu.org/software/parallel/man.html#summary_table
GNU Stow
GNU Stow este un caz particular de manager de pachete.
Instalare:
1 sudo apt−get install stow
Instalarea pachetelor
Cum instalam un pachet cu ajutorul lui GNU Stow?
1 sudo make install prefix=/usr/local/stow/numepachet2 cd /usr/local/stow3 sudo stow numepachet
Pachetele care nu folosesc GNU AutoConf pot sa nu accepteparametrul “prefix”, atunci este nevoie de modificari ın fis, ierulMakefile.
Instalarea pachetelor
Cum instalam un pachet cu ajutorul lui GNU Stow?
1 sudo make install prefix=/usr/local/stow/numepachet2 cd /usr/local/stow3 sudo stow numepachet
Pachetele care nu folosesc GNU AutoConf pot sa nu accepteparametrul “prefix”, atunci este nevoie de modificari ın fis, ierulMakefile.
Eliminarea pachetelor
1 cd /usr/local/stow/2 sudo stow −D numepachet
Exemplu: mai multe versiuni ale unuia s, i aceluias, i pachetın aceeas, i distribut, ie
Mai multe versiuni ale unuia s, i aceluias, i pachet ımpreuna.
1 cd numepachet−3.02 sudo make install prefix=/usr/local/stow/numepachet−3.0
3 cd numepachet−3.1−beta4 sudo make install prefix=/usr/local/stow/numepachet−3.1−beta
Exemplu: mai multe versiuni ale unuia s, i aceluias, i pachetın aceeas, i distribut, ie
Putem accesa fis, iere ambelor pachete prin
1 /usr/local/stow/numepachet−3.0/bin/fisierbinar
s, i
2 /usr/local/stow/numepachet−3.1−beta/bin/fisierbinar
Iar daca executam
1 cd /usr/local/stow2 sudo stow numepachet−3.0
atunci lansınd ın linia de comanda doar fisierbinar vom accesa/usr/local/stow/numepachet-3.0/bin/fisierbinar.
Adrese web
Pagina web
http://www.gnu.org/software/stow/
Manual s, i exemple
http://www.gnu.org/software/stow/manual/