Free off-site versioned backups using Duplicacy

The best backups are automated (hence frequent), versioned (so you can recover deleted or hacked files) and off-site (no worries about fire or theft).

To set them up you’ll obviously need somewhere to store them, and the good news is if you have an Office 365 subscription you already have 1 TB (a thousand gigabytes) of free OneDrive storage that is probably largely unused. (Other cloud providers can also be used.)

By the way – did you know that if you have all your files synchronised to OneDrive, you can restore older versions up to 29 days ago by going to https://onedrive.live.com and selecting the settings (gear wheel symbol) at the top right, then Options > Restore your OneDrive > Select a date?

Restore OneDrive

Unfortunately, standard backup software like Microsoft File History and Apple Time Machine can’t use this sort of cloud storage. They also struggle to back up large files that are constantly in use, like mail archives and log files. This is where Duplicacy comes in – it’s fast, efficient and secure, available for Windows, OSX and Linux and the command line version is free for personal use. The catch is it’s a bit tricky to set up and the documentation is sparse, so that’s what this post is about.

This post is specifically about Windows backups – for backups from Linux see our later post.

Initial set up

First, download the latest version of the command line executable appropriate to your operating system. For Windows 10 64-bit at the time of writing this is duplicacy_win_x64_2.1.1.exe.  The first sentence of the Quick Start quide says “Once you have the Duplicacy executable on your path…”, so we’ll start with that.

On Windows right-click on the Windows start button and open a Windows Powershell (Admin). You will need administrator privileges to install the program and also to run it later in the background, so make sure you do all of the following as an admin user.

You can use the following Powershell commands (or File Manager) to copy the Duplicacy executable we just downloaded to C:\Windows\system32. For convenience, we’ll rename it to “duplicacy.exe” as well.

cd ~\Downloads
ren duplicacy_win_x64_2.1.1.exe duplicacy.exe
mv duplicacy.exe \Windows\system32

If you now type this simple command

duplicacy

you should see a version number and a list of options. If not, enter this command to find what your “path” environment variable is set to, and copy the duplicacy.exe file to one of those locations.

$Env:Path

The next step in the Quick Start guide is “…change to the directory that you want to back up” (which they confusingly call the “repository”). Let’s use our home directory.

cd ~

Next we need to create a directory on OneDrive for storing our backups, and obtain authorization to use it. Using a web browser, log in to onedrive.com and select Files > New > Folder and give this folder a suitable name, such as “Duplicacy”.

OneDrive File > New > Folder

You don’t want to synchronise this new folder back to your PC, so open your OneDrive settings (taskbar > cloud symbol > More > Settings):

OneDrive settings

Then click “Choose folders” and make sure the new backup folder is NOT selected.

Duplicacy deselected

Next, visit https://duplicacy.com/one_start and click “Download my credentials as one-token.json” (links for other storage providers are here). Save this to your home folder (e.g. C:\Users\myname).

Finally you need to choose a “repository id” to identify this computer. A good choice would be the computer name from Start > Settings > System > About > Device name. Let’s say it’s “My-PC”.

Putting this all together, we can now initialise Duplicacy with the following command:

duplicacy init My-PC one://Duplicacy

You will be asked “Enter the path of the OneDrive token file (downloadable from https://duplicacy.com/one_start):” and the answer will be the filename and location you used above, for example:

one-token.json

Making backups

To specify the files to back up, you must create a text file called filters in the .duplicacy folder that you should now find in the root of your repository.  The rules are specified here and can be quite complex but here’s an example set of rules that will work on most Windows machines:

# Exclude files and folders that begin with a dot
-.*
# Exclude the AppData folder and some others
-AppData/
-Downloads/
-NTUSER.DAT*
-ntuser.dat*
-MicrosoftEdgeBackups/
# Back up everything else for this user

When you’re happy that the right files are being included, run the command

duplicacy backup

to start your first backup. Depending on the speed of your internet connection this could take several days to complete! Don’t worry, it will be much faster after the first time and it’s OK to interrupt it. You will probably want to (temporarily) prevent your computer going to sleep until it’s done by going to Start > Settings > System > Power & Sleep > Sleep then selecting PC goes to sleep after: Never.

If you see an error message saying a file is “in use” or “locked by another user” you can take advantage of a feature that is only available only on Windows and only for  administrators called “Volume Shadow Copy”. It cleverly locks the file, takes a quick copy then unlocks it again. You can do it like this:

duplicacy backup -vss

To schedule regular backups once an hour, as an administrator run Task Scheduler (under Windows Administrative Tools in the Windows Start menu) and select Create Task… at the right. Give the task a name and  select Run whether user is logged in or not (this prevents annoying pop-up windows every time it runs).

Duplicacy task pane

In the Trigger tab, under Begin the task select At startup. Then under Advance settings for Repeat task every select 1 hour for a duration of: Indefinitely.

Duplicacy task trigger pane

Under the Action tab,  for Program/script enter duplicacy with argument backup and Start in (optional): C:\Users\<your home folder>.

Duplicacy backup actions tab

Under the Conditions tab, under Network, select the option Start only if the following network connection is available: Any connection.

When you click OK so save the task, you will be prompted for an Administrator’s username and password.

Pruning old backups

Duplicacy includes a prune command for keeping the backup size under control. Their example scheme is quite sensible and can be run as a daily task:

duplicacy prune -keep 0:360 -keep 30:180 -keep 7:30 -keep 1:7

It basically means keep no backups older than a year, only monthly backups after 6 months, only weekly backups after a month, and only daily backups after a week (and hourly backups until then).

Restoring files or directories

If the worst happens and you accidentally delete some files, you need to use the Duplicacy restore command. First you have to find which revision number you want. If you know roughly when the problem occurred, the command

duplicacy list

will show you a list of revisions and the dates and times they were made. Alternatively, you can use the “duplicacy history” command to see when changes were made to a particular file or directory, or the “duplicacy diff” command to compare two snapshots or two revisions of a file. Once you know the revision number, you can use the command

duplicacy restore -r <revision> [pattern]

to  restore  a folder or file to its original location. For example, if you wanted to restore revision 42 of folder Music the command would be

duplicacy restore -r 42 +Music/*

There are other options shown in the documentation. By default, any existing files in the folder you are restoring to will not be overwritten or deleted.