Tech Tip: Running rsync over multiple threads

The following command can be used to run rsync over multiple threads in parallel. This can tremendously speedup file copy from one disk or computer node on a network to the next.

In the example below, the main variables used are:

$thread=24 -- Use 24 threads. Adjust this depending on the capacity of the systems involved. Set to 0 for xargs to maximize this value.

$src=/src/ -- This represents the source path of the directory of contents you wish to copy. It's important to pay attention to the ending /.

$dest=/dest/ -- This is the destination path for the files to be copied.

$threads=24; $src=/src/; $dest=/dest/ rsync -aL -f"+ */" -f"- *" $src $dest && (cd $src && find . -type f | xargs -n1 -P$threads -I% rsync -az % $dest/% )