I already tried using Nextcloud as a backup solution which will sync my complete home directory into a Nextcloud instance. This is very practical if you want to have access to your files when you don’t have your laptop with you or when it gets lost. Of course there are some drawbacks with this approach: Remote files are unencrypted (so that the Nextcloud file explorer can see them), accidental deletions results in data loss on both sides and lack of file versioning or snapshots (depends on your Nextcloud configuration).
An other issue is the performance in case of syncing many small files. This is a problem in case of the Nextcloud Desktop client because it creates a separate http connection (WebDav) for every single file. I created a bug report on Github to start a discussion for finding a better and faster approach.
In this post I’m going to compare the performance of some Linux file synchronization clients and further describe how to automate the benchmarking.
The test results
First of all, here are my test results. This graph shows the time in seconds needed to synchronize ~8MB of local-to-remote and ~1MB of remote-to-local files (972 single files):
Since my connection speed is not the bottleneck in this test, Nextcloud will need up to 8 minutes to index and up-/download the files. Considering the very small file size in total, this is not a very good result. I’m not sure what technique Dropbox is using but I guess it’s a proprietary, more enhanced protocol. Rclone is a really interesting tool which supports various cloud services. One limitation is that rclone does not support bidirectional file synchronization yet, so it is not really comparable to the other clients.
In this part I’ll sketch my setup I used for the benchmark. It will run in a container on Archlinux using systemd-nspawn and as a user called test. In the next steps, we’ll install the AUR handler pacaur und the syncing desktop clients.
pacman -S arch-install-scripts btrfs subvol create /var/lib/container/archlinux-base mkdir /etc/systemd/nspawn pacstrap /var/lib/container/archlinux-base base base-devel systemctl enable --now systemd-networkd systemd-resolved systemd-nspawn -nD /var/lib/machines/archlinux-nextcloudcli --template=/var/lib/container/archlinux-base systemctl start systemd-nspawn@archlinux-nextcloudcli machinectl shell root@archlinux-nextcloudcli /bin/bash -c "systemctl enable --now systemd-networkd systemd-resolved" machinectl shell root@archlinux-nextcloudcli useradd -m test passwd test vim /etc/sudoers # add user test su test cd curl "https://aur.archlinux.org/cgit/aur.git/snapshot/cower.tar.gz" | tar xz -C . curl "https://aur.archlinux.org/cgit/aur.git/snapshot/pacaur.tar.gz" | tar xz -C . cd cower makepkg -si --skipinteg cd ../pacaur makepkg -si pacaur -S nextcloud-client rclone dropbox-cli
In our test scenario we’ll sync the openssh source code to the server and also test the bi-directional synchronisation performance by putting the mosh source code into the Nextcloud instance.
su test cd wget https://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.7p1.tar.gz wget https://github.com/mobile-shell/mosh/archive/mosh-1.3.2.tar.gz tar -xvf openssh-7.7p1.tar.gz -C sync tar -xvf mosh-1.3.2.tar.gz -C sync_remote du -hs sync/openssh-7.7p1 sync_remote/mosh-mosh-1.3.2 # 8.3M sync/openssh-7.7p1 # 1.3M sync_remote/mosh-mosh-1.3.2 find sync/openssh-7.7p1 sync_remote/mosh-mosh-1.3.2 | wc -l # 972 nextcloudcmd -h -n -u test -p test123 sync_remote https://nextcloud.project-insanity.org/remote.php/webdav/
After these steps we’ll have the local folder sync and on the remote target the folder mosh-mosh-1.3.2. Further we have Nextcloud Desktop 2.3.3 (latest stable version), rclone and Dropbox installed.
Starting the tests
Following command starts the synchronisation with Nextcloud Client 2.3.3 (stable):
time nextcloudcmd -h -n -u test -p test123 /home/test/sync https://nextcloud.project-insanity.org/remote.php/webdav/
The prepended time command will return the running time of the command.
In the next step we’ll install Nextcloud Desktop 2.5.0 Beta 2 (which was the latest development version at this time), remove all sync databases and rerun the test:
pacaur -S nextcloud-desktop-git rm -r sync/._sync* sync/mosh-mosh-1.3.2 time nextcloudcmd -h -n -u test -p test123 /home/test/sync https://nextcloud.project-insanity.org/remote.php/webdav/
Starting the test with rclone is different. You have to setup your Nextcloud instance with the configuration wizard.
rclone config rm -r sync/._sync* sync/mosh-mosh-1.3.2 time rclone copy /home/test/sync remote:
In case of Dropbox, I had to copy the files into the synced directory and simultaneously copy the remote files to a local folder, to trigger the process. The filestatus command will show when it is finished.
dropbox-cli filestatus date && cp -r sync/openssh-7.7p1 Dropbox/ && cp -r Dropbox/mosh-mosh-1.3.2 /tmp/ date
This very small and limited test shows that commercial alternatives to Nextcloud such as Dropbox will offer faster and better desktop tools for file synchronization. It is not clear if Nextcloud developers will address this issue any time soon and to my knowledge there are no alternative clients with better performance. Let me know if you know other file synchronization tools for Linux which I can cover in future tests.