Automating Proxmox ZFS backups with pve-zsync

Today we have a quick how-to guide on Proxmox VE’s new feature: pve-zsync.  The pve-zsync features allows one to take automated backups of ZFS / virtual machine volumes and copy those backups to another node (more info here.) After having to recover from a major crash years ago, we now make more backups and also do VM level backups of our web hosting nodes. The reason for this is simple, recovering and moving applications when you have snapshots of virtual machines is a trivial task. Since we do use Proxmox in our infrastructure, we have been doing a few articles on the nice free hyper-converged platform. This is a guide we would suggest for anyone using two or more Proxmox hosts and ZFS on Linux for storage. Proxmox integrates ZFS on Linux fairly well so it is available to users on the newest versions out of the box and pve-zsync will allow you to replicate data across two hosts.

Getting ready for pve-zsync

In this guide we are going to be backing-up ZFS backed VMs from fmt-pve-07 to fmt-pve-01. We have some extra capacity in our Fremont colocation facility so this guide is essentially moving data from a mirrored set of NVMe drives to another set of mirrored NVMe drives in a second server. We have this guide on setting up ZFS mirrored disks in Proxmox. Here is a quick diagram of what we are going to setup:

The first step is to create a target/ destination for the ZFS send operations to land. If you have this setup already then great. If not, then you will need to do a bit of prep work on your destination server. We are going to use “zfs create -s -V 400g” to create a sparse (thin provisioned) 400GB volume for a backup target.

One can see the “fmt-01-nvme1/thinbackup2” entry in our zfs list as having only 64K used. We use sparse/ thin provisioning because we are not exactly sure how big our volume will need to be.

Automating Proxmox ZFS backups with pve-zsync

The next thing we need to do is login to our source server and create a pve-zsync job. We use the following command to install pve-zsync on the server:

apt-get install pve-zsync

We use the command:

pve-zsync create --source 100 --dest 10.0.104.201:fmt-01-nvme1/thinbackup2 --verbose --maxsnap 7

Bereaking down that command:

• pve-zsync – the command we are using.
• create – we are making a new recurring job.
• –source 100 – we are going to use the volume associated with Proxmox VM number 100 and use that to snapshot and send to the other server
• –dest 10.0.104.201:fmt-01-nvme1/thinbackup2 – tells pve-zsync to use the private (non routable) IP of our 10Gb network to send backups to the volume we created on our destination server. We could have used the server name but we instead use the static 10Gb storage network IP address for fmt-pve-01.
• –verbose – we want to see what is going on
• –maxsnap 7 – this tells pve-zsync the number of snapshots we want to keep. We have 7 in this example. Keep in mind the default is to snapshot every 15 minutes (we will see this later) so if you wanted a full day’s worth of 15min increment snapshots, you would want this to be 96.
• –name STHwp1bak just a name given to the job so you can see it easily later on

One can see the command run and the initial snapshot taking place.

We have a 10Gb dedicated storage network that s also being used for Ceph and other traffic on both nodes. Still with NVMe storage we see 300MB/s speeds. If you were using 1GbE and had a larger dataset, this initial sync can take a long time. If you did not have a dedicated storage network, this would also put a lot of stress on your NICs handling VM traffic.

When we use ZFS list on the target machine after compeltion, we can see fmt-01-nvme1/thinbackup2 has increased by approximately 40.5GB in size due to the sync occuring.

If you want to change settings, you can edit /etc/cron.d/pve-zsync and edit the cron job that is taking snapshots.

One can see the pve-zsync list and pve-zsync status has both of these working. It also shows the last time that the last sync took place.

Conclusion

Overall, this is a simple guide. There are a few features we would like to see added:

• The ability set and retain different time frames of snapshots (e.g. one monthly snapshot for 12 months, one daily snapshot for 30 days and etc.)
• The ability to use servers without root login access. Proxmox to Proxmox works well. We can also use Proxmox to rsync.net as a backup method. Some other NAS units such as FreeNAS will be a bit harder to use with existing setups where one does not have a root password login.
• The ability to specify multiple backup targets using the same set of snapshots of a VM.

It is a great feature and if you have spare capacity in your Proxmox network, or a backup service like rsync.net with ZFS and root access in a jail, this works very well.