Table of Contents
Introduction
Remote Sync (Rsync) is common across all the Linux distribution, it can be RPM-based, Debian based Linux distribution or it can be BSD based operating system, Solaris and Mac OS as well. If you are good to work with anyone of the operating systems it’s more than enough to go through the below guide.
Syntax to run a Rsync Command
# rsync options source destination
# rsync -avz root@192.168.107.201:/root/ /root/1. Install Remote Sync (Rsync) Package
In a minimal installed Operating system, rsync command will not be available. we need to install the package to start with Rsync.
# yum install rsync -y # RHEL, CentOS, Oracle Linux
# sudo apt-get install rsync # Ubuntu, Debian
# zypper install rsync # OpenSuse/SLE Linux
# brew install rsync # MacOS
# pkg install rsync # FreeBSD2. Copying from Source to Destination
Sync files from source to a remote server with archive, verbose and compress.
# rsync -avz /root/ root@192.168.107.101:/root- -a – archive mode
- -v – Verbose mode
- -z – Compress the files while in transfer
3. No action just Dry Run
Before starting with sync just do a dry run to check what are the files will be synced.
# rsync -avz --dry-run root@192.168.107.201:/root/* /root/[root@corcls1 ~]# rsync -avz --dry-run root@192.168.107.201:/root/* /root/
receiving incremental file list
configfile1.conf
configfile2.conf
configfile3.conf
configfile4.conf
configfile5.conf
largefile.iso
largefile1.iso
logfile1.log
logfile2.log
logfile3.log
logfile4.log
logfile5.log
textfile1.txt
textfile2.txt
textfile3.txt
textfile4.txt
textfile5.txt
sent 75 bytes received 424 bytes 998.00 bytes/sec
total size is 314,572,800 speedup is 630,406.41 (DRY RUN)
[root@corcls1 ~]#Good for the small number of files, large filesystem dry run will consume the equal time of actual file sync.
4. Syncing multiple Directories
Copy multiple different directories from the remote server to the source.
# rsync -avc root@192.168.107.101:/root/somefiles :/root/linuxsysadmins .Separate each directory with “:“
- -c – Skip based on checksum
5. Syncing only Specific files
Sync only specific files from a remote server to the destination.
# rsync -avz root@192.168.107.101:/root/somefiles/{test1.txt,test2.txt} .Mention all your specific files inside the curly brace with comma-separated.
6. Files with White space
Copying a filename with whitespace using rsync from remote to source servers.
# rsync -av root@192.168.107.101:'/root/somefiles/file\ with\ space.txt' .Use backslash in every white space.
7. Showing the Progress
Transfer and show the progress for each file. This will help while we transfer large size files.
# rsync -avz --progress /root/ root@192.168.107.101:/root8. Read a file and Sync
Remote Syncing from source to a destination only required listed files from a file.
Create a file and list out all the files need to be copied on the remote server.
# vim list.txt[root@docker ~]# cat list.txt
./somefiles/somelarge_file.iso
./somefiles/test10.txt
./somefiles/test3.txt
./somefiles/test5.txt
./somefiles/test7.txt
./somefiles/test9.txt
[root@docker ~]## rsync -avz --progress --files-from=/root/list.txt /root/ root@192.168.107.101:/root/Output for Reference
[root@docker ~]# rsync -avz --progress --files-from=/root/list.txt /root/ root@192.168.107.101:/root/
building file list …
8 files to consider
./
somefiles/
somefiles/somelarge_file.iso
1,073,741,824 100% 114.47MB/s 0:00:08 (xfr#1, to-chk=5/8)
somefiles/test10.txt
10,485,760 100% 9.59MB/s 0:00:01 (xfr#2, to-chk=4/8)
somefiles/test3.txt
10,485,760 100% 70.42MB/s 0:00:00 (xfr#3, to-chk=3/8)
somefiles/test5.txt
10,485,760 100% 40.32MB/s 0:00:00 (xfr#4, to-chk=2/8)
somefiles/test7.txt
25,165,824 100% 52.40MB/s 0:00:00 (xfr#5, to-chk=1/8)
somefiles/test9.txt
10,485,760 100% 18.59MB/s 0:00:00 (xfr#6, to-chk=0/8)
sent 1,109,971 bytes received 136 bytes 105,724.48 bytes/sec
total size is 1,140,850,688 speedup is 1,027.69
[root@docker ~]# 9. Remote Sync (Rsync) through different Port
Using Rsync from local server to remote server with a non-standard (22) SSH port.
# rsync -avz -e "ssh -p 5254" /root/ root@192.168.107.101:/root/- -e – Specify the remote shell to use
- -p – SSH Port or remote server.
10. Copy with Bandwidth Limit
Sync by limiting the bandwidth, This will be helpful in case your server is a most critical production system and needs to free from chocking the network.
$ rsync -avz --progress --bwlimit=56KB /home/babin/ubuntu-18.04.2-live-server-amd64.iso root@192.168.107.200:/root- –bwlimit – To Limit the bandwidth
sending incremental file list
ubuntu-18.04.2-live-server-amd64.iso
3,768,320 0% 55.02kB/s 4:23:4611. Remove from Source (Remote server)
Here, I’m trying to sync the files from my remote server to my local desktop machine. Once the files completed syncing it will remove from the remote server.
# rsync -rvz --remove-source-files root@192.168.107.200:/home/ansible/* /home/babin/backup/- –remove-source-files – Remove the files from the remote server.
12. Limiting the size of a single file size
While syncing we can limit each file size to a maximum or minimum size. If you set a maximum size it will not transfer any file above the specified size. The same thing applied for minimum size as well.
--min-size=SIZE
--max-size=SIZE List the files in the source server
[root@corcls1 ~]# ls -lthr
total 1.4G
-rw-r--r--. 1 root root 1.0G Aug 8 19:52 largefile.iso
-rw-r--r--. 1 root root 300M Aug 8 19:53 largefile2.iso
-rw-r--r--. 1 root root 100M Aug 8 19:54 largefile4.iso
[root@corcls1 ~]#From the above list, we need to transfer only the 100 MB file. Let us use –max-size=150MB to transfer the file not larger than 150 MB.
# rsync -avz --progress --max-size=150M /root/* root@192.168.107.201:/root/Here we can see only one file transferred from source to destination. The file “largefile4.iso” is 100 MB in size.
[root@corcls1 ~]# rsync -avz --progress --max-size=150M /root/* root@192.168.107.201:/root/
sending incremental file list
largefile4.iso
104,857,600 100% 78.16MB/s 0:00:01 (xfr#1, to-chk=0/3)
sent 102,115 bytes received 35 bytes 40,860.00 bytes/sec
total size is 1,493,172,224 speedup is 14,617.45
[root@corcls1 ~]#Verify the transferred file on the destination server.
[root@corcls2 ~]# ls -lthr
total 101M
-rw-------. 1 root root 1.8K Aug 2 02:00 anaconda-ks.cfg
-rw-r--r--. 1 root root 100M Aug 8 19:54 largefile4.iso
[root@corcls2 ~]#13. Ignore empty directories during Remote Sync
These are the file and directories on the remote server.
[root@corcls2 ~]# ls -lthr
total 300M
-rw-r--r--. 1 root root 0 Aug 8 20:44 logfile5.log
-rw-r--r--. 1 root root 0 Aug 8 20:44 logfile4.log
-rw-r--r--. 1 root root 0 Aug 8 20:44 logfile3.log
-rw-r--r--. 1 root root 0 Aug 8 20:44 logfile2.log
-rw-r--r--. 1 root root 0 Aug 8 20:44 logfile1.log
-rw-r--r--. 1 root root 0 Aug 8 20:44 textfile1.txt
-rw-r--r--. 1 root root 0 Aug 8 20:44 textfile5.txt
-rw-r--r--. 1 root root 0 Aug 8 20:44 textfile4.txt
-rw-r--r--. 1 root root 0 Aug 8 20:44 textfile3.txt
-rw-r--r--. 1 root root 0 Aug 8 20:44 textfile2.txt
-rw-r--r--. 1 root root 0 Aug 8 20:48 configfile5.conf
-rw-r--r--. 1 root root 0 Aug 8 20:48 configfile4.conf
-rw-r--r--. 1 root root 0 Aug 8 20:48 configfile3.conf
-rw-r--r--. 1 root root 0 Aug 8 20:48 configfile2.conf
-rw-r--r--. 1 root root 0 Aug 8 20:48 configfile1.conf
-rw-r--r--. 1 root root 100M Aug 8 20:49 largefile.iso
-rw-r--r--. 1 root root 200M Aug 8 20:50 largefile1.iso
drwxr-xr-x. 2 root root 6 Aug 8 22:01 somedir1
drwxr-xr-x. 2 root root 6 Aug 8 22:01 someemptydir
[root@corcls2 ~]#Ignore all the empty directories
# rsync -avz --prune-empty-dirs root@192.168.107.201:/root/* /root/Use –prune-empty-dirs or “-m”.
[root@corcls1 ~]# rsync -avz --prune-empty-dirs root@192.168.107.201:/root/* /root/
receiving file list … done
configfile1.conf
configfile2.conf
configfile3.conf
configfile4.conf
configfile5.conf
largefile.iso
largefile1.iso
logfile1.log
logfile2.log
logfile3.log
logfile4.log
logfile5.log
textfile1.txt
textfile2.txt
textfile3.txt
textfile4.txt
textfile5.txt
sent 347 bytes received 306,967 bytes 68,292.00 bytes/sec
total size is 314,572,800 speedup is 1,023.62
[root@corcls1 ~]#
[root@corcls1 ~]# ls -lthr
total 300M
-rw-r--r--. 1 root root 0 Aug 8 20:44 logfile5.log
-rw-r--r--. 1 root root 0 Aug 8 20:44 logfile4.log
-rw-r--r--. 1 root root 0 Aug 8 20:44 logfile3.log
-rw-r--r--. 1 root root 0 Aug 8 20:44 logfile2.log
-rw-r--r--. 1 root root 0 Aug 8 20:44 logfile1.log
-rw-r--r--. 1 root root 0 Aug 8 20:44 textfile1.txt
-rw-r--r--. 1 root root 0 Aug 8 20:44 textfile5.txt
-rw-r--r--. 1 root root 0 Aug 8 20:44 textfile4.txt
-rw-r--r--. 1 root root 0 Aug 8 20:44 textfile3.txt
-rw-r--r--. 1 root root 0 Aug 8 20:44 textfile2.txt
-rw-r--r--. 1 root root 0 Aug 8 20:48 configfile5.conf
-rw-r--r--. 1 root root 0 Aug 8 20:48 configfile4.conf
-rw-r--r--. 1 root root 0 Aug 8 20:48 configfile3.conf
-rw-r--r--. 1 root root 0 Aug 8 20:48 configfile2.conf
-rw-r--r--. 1 root root 0 Aug 8 20:48 configfile1.conf
-rw-r--r--. 1 root root 100M Aug 8 20:49 largefile.iso
-rw-r--r--. 1 root root 200M Aug 8 20:50 largefile1.iso
[root@corcls1 ~]# The somedir1 and someemptydir are not transferred to the destination.
14. Log everything in Destination server
While transferring files just we need to log all entries in the remote system.
# rsync -avz --remote-option=--log-file=/tmp/rsyslogstatus.log /root/ root@192.168.107.201:/root/Logging will help to debug if any disconnectivity happened in the middle of any sync.
[root@corcls2 ~]# ls -lthr /tmp/rsyslogstatus.log
-rw-r--r--. 1 root root 1.1K Aug 9 01:09 /tmp/rsyslogstatus.log
[root@corcls2 ~]#
[root@corcls2 ~]# cat /tmp/rsyslogstatus.log
2019/08/09 01:09:01 [14336] receiving file list
2019/08/09 01:09:01 [14336] .d..t…… ./
2019/08/09 01:09:01 [14336] >f+++++++++ configfile1.conf
2019/08/09 01:09:01 [14336] >f+++++++++ configfile2.conf
2019/08/09 01:09:01 [14336] >f+++++++++ configfile3.conf
2019/08/09 01:09:01 [14336] >f+++++++++ configfile4.conf
2019/08/09 01:09:01 [14336] >f+++++++++ configfile5.conf
2019/08/09 01:09:02 [14336] >f+++++++++ largefile.iso
2019/08/09 01:09:04 [14336] >f+++++++++ largefile1.iso
2019/08/09 01:09:04 [14336] >f+++++++++ logfile1.log
2019/08/09 01:09:04 [14336] >f+++++++++ logfile2.log
2019/08/09 01:09:04 [14336] >f+++++++++ logfile3.log
2019/08/09 01:09:04 [14336] >f+++++++++ logfile4.log
2019/08/09 01:09:04 [14336] >f+++++++++ logfile5.log
2019/08/09 01:09:04 [14336] >f+++++++++ textfile1.txt
2019/08/09 01:09:04 [14336] >f+++++++++ textfile2.txt
2019/08/09 01:09:04 [14336] >f+++++++++ textfile3.txt
2019/08/09 01:09:04 [14336] >f+++++++++ textfile4.txt
2019/08/09 01:09:04 [14336] >f+++++++++ textfile5.txt
2019/08/09 01:09:04 [14336] sent 350 bytes received 307304 bytes total size 314578933
[root@corcls2 ~]#15. Append a file without replacing it
Appending to a file without overwriting the entire file. For instance, Copy a log file from source to destination and write the update from source to destination.
In below example, we have copied /var/log/messages from remote server 192.168.107.201 to our source server 192.168.107.200 under /root directory. Then to sync the content of the source and destination file we are using –append option.
# rsync --append root@192.168.107.201:/var/log/messages /root/This will not overwrite the file again and again instead it will append the new contents.
16. Syncing with Include and exclude
We need to sync only the messages logs from one of the remote servers to source. In our example, we are syncing all messages log and excluding all other files.
# rsync -avz --include 'messages*' --exclude '*' root@192.168.107.201:/var/log/ /root/17. Use Remote Shell while doing a Sync
For a secure synchronization, we can use a remote shell by forcing to use the SSH. This invokes a secure file transfer.
# rsync -avze ssh root@192.168.107.201:/home/linuxsysadmins/ /root/backup18. Recursively back-up all changed files
In our example, we are syncing all log files “/var/log/” from source server 192.168.107.201 to destination server 192.168.107.200 under /root/log/. While performing the second sync if any changes happened on the log files on the source (192.168.107.201) a backup copy of existing files in the destination will be copied under /root/backup/. This help to keep our old files in a backup location.
# rsync -btr --backup-dir=/root/backup/ root@192.168.107.201:/var/log/ /root/logs/- -b – –backup – makes a backup of existing files
- -t – –times – Preserve modification of times
- -r – Copy the files recursively
19. Use as a Backup tool
Use Rsync as a Backup tool for large file system over the network
# rsync -avzPr --delete -e ssh root@192.168.107.201:/home/* /backup/users_files/20. Getting Help of Remote Sync commands
To get to know more about Rsync run
# man rsync
# rsync --helpThat’s it we have seen a few of Remote Sync (Rsync) examples which we are used in the production environment.
Conclusion
Remote Sync used to sync files securely on any nix servers, which can be used as a backup tool for the small environment as well. Subscribe for our newsletter and provide your feedback in the below comment section.
I found this article quite useful! Very well written and quite clear too. Thank you very much!