bfsync is a file-synchronization tool which allows to keep a collection of big files synchronized on many machines. There are two types of repositories: master repositories and checkouts. Master repositories only contain the history information. Usually they should be stored on a central server which all computers that should view/edit this data can reach (at least some of the time). Master repositories are created using bfsync-init(1). The other repository type is a checkout. A checkout can be created using bfsync-clone(1). Besides the history information checkouts contain the actual file contents, so a checkout can be huge (houndreds of gigabytes) whereas a master repository is usually small.

To view/edit the data in a checked out repository, the repository must be mounted using bfsyncfs(1). Once mounted, bfsync can be called with the commands given below. For instance bfsync commit/push/pull can be used to modify the local history and resynchronize the history with the master history. Other commands like bfsync get/put can be used to transfer data between checkouts.

For transfer, bfsync needs to be installed on every system that needs to be accessed. The actual transfer is done using ssh. Transfer has only be tested with ssh keys; it’s highly recommended to use ssh-agent to avoid entering your password over and over again.

It is possible to use bfsync for backups, in that case you only use a subset of what bfsync can do to get a versioned FUSE filesystem with deduplication. See the section BACKUP WALKTHROUGH for a description.

For sub-commands like bfsync pull or bfsync commit, the options depend on the command. In these cases, options are documented on the sub-command manual page.

Every bfsync checkout has a file called "config", which can be used to set configuration variables for this checkout.

The configuration keys in the default group can be set simultaneously, by using

The configuration keys in the expire group can be set simultaneously, for instance by using

The configuration keys in the sql-export group can be set simultaneously, for instance by using

Shared memory is used by bfsync to access the Berkeley DB database contents from different processes: the bfsync FUSE filesystem process, bfsyncfs, and the python frontend, bfsync. Under Linux, the amount of shared memory usually is limited by three system-wide kernel parameters:

These limits need to be large enough to allow bfsync to allocate the required amount of shared memory. The amount of shared memory required mainly depends on the cache size. Bfsync will use somewhat more shared memory than the cache size, but setting the limits too high is usually not a problem.

Example: If you’re using three bfsync filesystems with 256 MB cache per filesystem, you can do so if shmall is 2 GB and shmmax is 512 MB. shmmni is usually not an issue, because bfsync doesn’t use may segments (about 4 per filesystem).

To display your current limits, you can use:

To adjust shared memory settings at boot time, create a file called /etc/sysctl.d/90-bfsync-shm.conf:

Note that if you have other programs that also need shared memory, you need to coordinate the settings of all shared memory using programs. Its also not a problem if your limits are too high, so if the system wide limit for shmall is already 8 GB, there is no need to adjust it.

After creating this files, the settings will be loaded at boot time. To activate the shared memory configuration without rebooting, you can use

bfsync allows independent modifications of the data/history contained in different checkouts. Upon push, bfsync will check that the master history doesn’t contain new commits that are unknown to the local checkout. If two clients modify the repository independently, the first client that uses bfsync push will simply reintegrate its changes into the master history, and the new master history will be this client’s history.

However, if the second client tries a bfsync push, the push will be refused. To resolve the situation, the second client can use bfsync-pull(1). Once it is detected that merging both histories is necessary, a merge algorithm will be used. For non-conflicting changes, everything will be merged automatically.

Non-conflicting changes could be:

However, there are situations where the merge algorithm can’t merge both histories automatically:

In any case, after the merge decisions are made (if any), the merge algorithm will use them to modify the local history so that it can be executed without conflicts after the master history. After this step, the modified local commits will be based on the master history. This means that then, bfsync push will succeed, and the modified changes of client 2 can be pushed to the master history.

Note that the master history is always linear, so the history branch that was present before the merge algorithm was used will no longer be visible in the history after the pull. The merged history will simply contain the old history (before client 1 and client 2 made their changes), the changes made on client 1, an extra merge commit (if necessary to resolve merge issues), and the modified changes of client 2.

First, we create and setup repositories on three computers: server, client1 and client2. The server will hold the master repository (which manages the history, but nothing else). It is stored under ~/repos/big.bfsync. All computers will contain a checkout, so that the actual contents of the files can be kept there.

As second step, we add a music file on client1. Of course it’s possible to add more files in one step; you can also use rsync, mc or a file manager to copy files into the repository. Whenever files are added or otherwise changed, we need to commit and push the changes to the server, so that it contains the canonical index of files.

So far, we have added the file to the repository on client1, but the contents of the file are only present on client1, and not in the other repos. To change this, we can transfer the file to the server.

As last step, we’ll transfer the file to client2. Of course we could use the same commands that we used to get the file to the server, but let’s assume that client2 is behind a firewall, and that it’s not possible to ssh to client2 directly. Fortunately, besides uploading files to another host (bfsync put), it’s also possible to download data from another host (bfsync get).

Since bfsync implements file level deduplication and versioning of files, it can be used to do backups. Backups typically contain lots of files (like 5.000.000 files). Therefore you can only use a subset of the available commands for backups, since some commands do not work well if the number of files is that large.

Commands like commit, get, put, push, pull, check and gc should work fine for backup usage. Also deleting old backups using expire and maybe copy-expire is supported. However, advanced functions like merges might never be supported for backups - for typical backup scenarios this is not an issue.

The first step for backups is to set up repositories. All steps should be done as root. For this example, we assume that our backup harddisk is mounted to /backup.

The cache size is important for backups: if it is too small, the backup will take a lot more time. However, since the cache is stored in shared memory, a overly large cache may use too much of the system memory. As a rule of thumb, 100 megabytes of cache should be used for every 1.000.000 files that are stored in the backup. More is better, if you can afford it.

Now that everything is initialized, we can backup some data. For this example we backup /home.

The rsync options we use here are:

We have the initial full backup. Now one day later, we only need to backup changes (which will be a lot faster than the initial backup), like this:

Now, we’ve created the first incremental backup. This usually uses a lot less additional disk space than the initial full backup, since usually only few files will be changed. To access an individual backup, you can use

To automate the process, a script which runs the rsync and commit steps every night can be used. Removing the contents of old backups is currently not supported, but will be available in the future.

The commandline for creating a backup of the root filesystem is:

If you backup more than one filesystem every day, you only need to commit once, that is first rsync all filesystems and commit as last step.

The repository format is not (yet) stable across bfsync versions. In some cases upgrades from an old to the current bfsync version can be done automatically by running

This is the easiest way. If this way doesn’t work, you need to manually convert your old repositories to a new version. There is currently no easy way to preserve the history when manually updating from an old version of bfsync. But if you need only preserve the repository content, you can use the following steps:

To avoid retransfer if you have the data on other machines, the following steps can be used:

Repeat these steps on all machines that contain checkouts of your repository. You can delete the old format checkouts after verifying with bfsync-check(1) that all files you need are there. You do not need to install old and new bfsync versions on the satellite systems. Only the new bfsync is required to perform the checkout & collect steps.

bfsyncfs(1).