Keeping Nextcloud and a file server in sync automatically with rclone bisync

If you use a hosted Nextcloud and also run a file server on your local network, you know the problem: laptops sync quickly with the file server, but syncing with the Nextcloud takes a long time and is easily forgotten when you have to start it by hand. nc-bisync.sh automates exactly this step with rclone bisync. It runs unattended via cron and only raises an alarm when something looks wrong.

Download: nc-bisync.sh (comments and messages in the script are in German)

The setup

The file server is the hub between all devices:

Laptops  ⇄ (Unison, manual)  ⇄  File server  ⇄ (rclone bisync, automatic)  ⇄  Nextcloud
                                     │
                                     └─ rsnapshot (daily, versioned) → external drive

The laptops stay with Unison. On the local network it is fast, and its graphical interface helps with conflicts. rclone bisync takes over the slow link to the Nextcloud. rclone talks to WebDAV directly and needs no davfs mount. It also comes with safeguards that would otherwise require a wrapper script of your own.

What the script does

One directory pair per share. Each share on the file server is synced with the folder of the same name in Nextcloud, independently of the others. Pairs can be run individually or in groups, even at different frequencies. If needed, a single subfolder, such as a large project, becomes a pair of its own and is automatically excluded from the parent pair.

Archive only on demand. A subfolder archiv inside a pair is never synced in a normal run, only with --archiv. Internally it is a separate pair with its own state. That is more robust than a folder that a filter sometimes includes and sometimes excludes.

Protection against mass damage. Three built-in mechanisms guard against larger damage:

--check-access: Each root contains a file named RCLONE_TEST. If it is missing on one side, because a path is wrong or a drive is not mounted, the run aborts. This also prevents an empty bind mount from being pushed to the Nextcloud as “everything deleted”.
--max-delete 10: If more than 10 % of the files would be deleted, nothing happens at all.
--backup-dir1: Files overwritten or deleted on the file server end up in a daily folder and are cleaned up after 30 days.

Conflicts. If a file was changed on both sides, the newer version wins (--conflict-resolve newer). The older one is kept under a new name.

Robustness. With --resilient and --recover, bisync usually survives interrupted runs without a manual --resync. A lock file prevents overlapping runs.

Notifications. On errors you get a message by email or via ntfy. The script distinguishes whether a resync is needed, the delete limit was hit or the access check failed. Every run is logged with its duration in summary.log.

Filters. Temporary files from Unison, Office, macOS and Windows are never synced.

Setup

The script does not run as root but as the user who owns the files on the file server. Otherwise files newly fetched from the Nextcloud would belong to root and could no longer be written by the other users of the share. sudo is only needed for the installation.

1. Install rclone

The script needs rclone 1.66 or newer. Many distributions ship older packages, so it is better to use the official install script:

curl https://rclone.org/install.sh | sudo bash
rclone version

2. Set up the Nextcloud remote

In Nextcloud, create an app password under Settings → Security. Then set up the remote as the user the script will run as later (without sudo):

rclone config
# n) New remote
# name> nc
# Storage> webdav
# url> https://cloud.example.com/remote.php/dav/files/USERNAME/
# vendor> nextcloud
# user> USERNAME
# pass> (app password)

To test:

rclone lsd nc:

3. Install and adapt the script

sudo install -m 755 nc-bisync.sh /usr/local/bin/nc-bisync.sh
sudo install -d -o USER /var/lib/nc-bisync /var/log/nc-bisync
sudo install -d -o USER -m 700 /mnt/USB-Crucial/.bisync-backup

Mode 755 makes the script executable for all users, but only root can change it. The second line creates the state and log directories and hands them over to the user the script runs as. The third creates the backup directory (BACKUP_ROOT); mode 700, because overwritten files from all shares end up there and only the script’s user should see them.

In the configuration block at the top, enter the pairs, one per line with name, local path and path in Nextcloud:

PAIRS=(
    "administration|/mnt/shares/administration|nc:Administration"
    "bibliothek|/mnt/shares/bibliothek|nc:Bibliothek"
    "dokumente|/mnt/shares/dokumente|nc:Dokumente"
    "fotos|/mnt/shares/fotos|nc:Fotos"
    "projekte|/mnt/shares/projekte|nc:Projekte"
    "unternehmen|/mnt/shares/unternehmen|nc:Unternehmen"
)

Further settings:

Variable	Meaning	Example
`ARCHIVE_SUBDIR`	Subfolder that only runs with `--archiv`	`archiv`
`BACKUP_ROOT`	Storage for overwritten files	`/mnt/USB-Crucial/.bisync-backup`
`MAX_DELETE`	Delete limit in percent	`10`
`NOTIFY_MAIL`	Recipient for error messages (optional)	`admin@example.com`
`NTFY_URL`	ntfy topic for push messages (optional)	`https://ntfy.sh/my-topic`

Make sure BACKUP_ROOT is outside all pairs. nc-bisync.sh --list shows which pairs are configured and which have an archive subfolder.

4. Group permissions on the file server

The script creates new files with umask 002 so that they stay writable for the group. To make sure they also get the right group rather than the user’s primary group, the synced directories should have the setgid bit. The setgid bit on a directory makes files and subfolders newly created in it automatically inherit its group instead of getting their creator’s primary group.

For the pairs from the example above, set it on all existing directories like this:

for d in administration bibliothek dokumente fotos projekte unternehmen; do
    sudo find "/mnt/shares/$d" -type d -exec chmod g+s {} +
done

You can tell it is set by an s in the group permissions:

$ ls -ld /mnt/shares/dokumente
drwxrws--- 10 stefan familie 4,0K 21. Jul 13:19 /mnt/shares/dokumente

5. Adjust Unison on the laptops

bisync detects changes by modification time and size. So that a Unison run from a laptop does not make every file look changed, add this to every Unison profile:

times = true

6. Initial sync

Test with --dry-run first, then run the initial sync:

nc-bisync.sh --init --dry-run
nc-bisync.sh --init

--init creates the RCLONE_TEST files in every pair and every existing archive subfolder and syncs everything with --resync. Where they differ, the newer file wins. With a lot of data it makes sense to set up the pairs one at a time, for example nc-bisync.sh --init fotos.

In a dry run the script changes nothing: it does not create the RCLONE_TEST files either and therefore skips the access check.

7. Schedule

Add the cron jobs with crontab -e to the crontab of the user the script runs as. Because each pair has its own lock, frequently changed shares can run more often than rarely changed ones:

# frequently changed: every 2 hours
0 */2 * * *  /usr/local/bin/nc-bisync.sh dokumente fotos projekte unternehmen administration
# rarely changed: at night
30 2 * * *   /usr/local/bin/nc-bisync.sh bibliothek

With many files, the bottleneck is not the transfer but listing everything over WebDAV, and that happens in full on every run. So start with two hours and check the run times after a few days:

tail /var/log/nc-bisync/summary.log

As a rule of thumb, the interval should be about three times as long as a run. Do not sync too often, because some providers throttle large numbers of WebDAV requests.

Day-to-day use

Command	When
`nc-bisync.sh`	sync all pairs, without archive
`nc-bisync.sh fotos dokumente`	sync only the pairs named
`nc-bisync.sh --archiv projekte`	after archiving files in `projekte`
`nc-bisync.sh --dry-run fotos`	see beforehand what would happen
`nc-bisync.sh --archiv --force projekte`	when the delete limit was hit while archiving
`nc-bisync.sh --list`	show the configured pairs

Always pass rclone options that take a value as --option=value; otherwise the script takes the value for the name of a pair.

Archiving files

Move the files within the share into the subfolder archiv, then run the sync for that share:

nc-bisync.sh --archiv projekte

The script syncs the archive first and only then the rest of the share. That way the files are already in the Nextcloud archive before they disappear from the rest of the share there. If the archive run fails, the rest of the share is not touched at all.

If you move a lot of files at once, the delete limit may kick in. In that case, take a quick look at the log and run again with --force.

To add an archive later, create the folder archiv locally and run nc-bisync.sh --init NAME once.

Individual subfolders as a pair of their own

If a share becomes too large for frequent runs, a subfolder can get a pair of its own:

"tango|/mnt/shares/projekte/Tango|nc:Projekte/Tango"

The script then automatically excludes Tango from the pair projekte. Because this changes its filter, nc-bisync.sh --init projekte tango is needed once. The subfolder must sit in the corresponding place in Nextcloud, otherwise the script aborts with a message.

When a notification arrives

Resync needed: Check the cause in the log mentioned. After that, nc-bisync.sh --init NAME usually fixes the state, ideally with --dry-run first.
Too many deletions: Check whether the deletions are intended. If so, run with --force.
Access check failed: An RCLONE_TEST file is missing. Check that the path is correct, the Nextcloud is reachable and the drive is mounted.

Limits

rclone bisync is not a backup. The sync also carries over accidental changes; only mass deletions are caught. If ransomware encrypts files on a laptop, for example, they end up in the Nextcloud via Unison and bisync, because files are overwritten, not deleted. The real protection therefore comes from the versioned rsnapshot backups and Nextcloud’s file versions. The retention period should be longer than the time it takes to notice damage.