bash - Output to the same file sequence -
suppose have myscript.sh below:
#!/bin/bash $1 > bla.txt bla.txt > temp.txt ... cat temp.txt >> finalouput.txt
then run parallel below:
parallel myscript.sh {} ::: {1..3}
does write output in order? finaloutput.txt
have results of 1
first, 2
, , 3
.
note: outputting separate files merging them in required order once parallel complete, wondering if avoid step.
the processes run in parallel. not there no guarantee finish in order, there's not guarantee can have multiple processes writing same file , end useful.
if going writing same file multiple processes, should implement sort of locking prevent corruption. example:
while ! mkdir finaloutput.lock; sleep 1 done cat temp.txt >> finaloutput.txt rmdir finaloutput.lock
if order matters, should each script write unique file, , assemble final output in correct order after parallel jobs have finished.
#!/bin/bash $1 > bla.txt bla.txt > temp-$1.txt ... cat temp.txt >> finalouput.txt
and after parallel
has finished:
cat temp-*.txt > finaloutput.txt
Comments
Post a Comment