| CARVIEW |
According to the documentation “It instructs libvirt to avoid termination of the VM if the guest OS shuts down while the backup is still running. The VM is in that scenario reset and paused instead of terminated allowing the backup to finish. Once the backup finishes the VM process is terminated.”
Added support for this in virtnbdbackup 2.40.
]]>In the past, it was possible to clone existing projects on the opensuse build service to build the images by yourself, but i couldn’t find any templates for SLES 16.
Naturally, there are several ways to build images, and the tooling around involves kiwi-ng, opensuse build service, or packer recipes etc.. (existing packer recipes wont work anymore, as Yast has been replaced by a new installer, called agma). All pretty complicated, …
So my current take on creating a vagrant image for SLE16 has been the following:
- Spin up an QEMU virtual machine
- Manually install the system, all in default except for one special setting: In the Network connection details, “Edit Binding settings” and set the Interface to not bind a particular MAC address or interface. This will make the system pick whatever network device naming scheme is applied during boot.
- After installation has finished, shutdown.
Two guestfs-tools that can now be used to modify the created qcow2 image:
- run virt-sysrpep on the image to wipe settings that might cause troubles:
virt-sysprep -a sles16.qcow2- create a simple shellscript that setups all vagrant related settings:
#!/bin/bash
useradd vagrant
mkdir -p /home/vagrant/.ssh/
chmod 0700 /home/vagrant/.ssh/
echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIF
o9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9W
hQ== vagrant insecure public key" > /home/vagrant/.ssh/authorized_keys
chmod 0600 /home/vagrant/.ssh/authorized_keys
chown -R vagrant:vagrant /home/vagrant/
# apply recommended ssh settings for vagrant boxes
SSHD_CONFIG=/etc/ssh/sshd_config.d/99-vagrant.conf
if [[ ! -d "$(dirname ${SSHD_CONFIG})" ]]; then
SSHD_CONFIG=/etc/ssh/sshd_config
# prepend the settings, so that they take precedence
echo -e "UseDNS no\nGSSAPIAuthentication no\n$(cat ${SSHD_CONFIG})" > ${SSHD_CONFIG}
else
echo -e "UseDNS no\nGSSAPIAuthentication no" > ${SSHD_CONFIG}
fi
SUDOERS_LINE="vagrant ALL=(ALL) NOPASSWD: ALL"
if [ -d /etc/sudoers.d ]; then
echo "$SUDOERS_LINE" >| /etc/sudoers.d/vagrant
visudo -cf /etc/sudoers.d/vagrant
chmod 0440 /etc/sudoers.d/vagrant
else
echo "$SUDOERS_LINE" >> /etc/sudoers
visudo -cf /etc/sudoers
fi
mkdir -p /vagrant
chown -R vagrant:vagrant /vagrant
systemctl enable sshd- use virt-customize to upload the script into the qcow image:
virt-customize -a sle16.qcow2 --upload vagrant.sh:/tmp/vagrant.sh- execute the script via:
virt-customize -a sle16.qcow2 --run-command "/tmp/vagrant.sh"After this, use the create-box.sh from the vagrant-libvirt project to create an box image:
https://github.com/vagrant-libvirt/vagrant-libvirt/blob/main/tools/create_box.sh
and add the image to your environment:
create_box.sh sle16.qcow2 sle16.box
vagrant box add --name my/sles16 test.boxthe resulting box is working well within my CI environment as far as i can tell.
]]>Also, the regular “query-block” qmp command doesn’t list the created bitmaps as usual.
If the virtual machine version is set to “9.2+pve”, everything seems to work out of the box.
I’ve released Version 0.50 with some small changes so its compatible with the newer machine versions.
]]>Unfortunately this means there are currently no official vagrant images for Debian trixie, for reasons
Of course there are various boxes floating around on hashicorp’s vagrant cloud, but either they do not fit my needs (too big) or i don’t consider them trustworthy enough…
Building the images using the existing toolset is quite straight forward. The required scripts are maintained in the Debian Vagrant images repository.
With a few additional changes applied and following the instructions of the README, you can build the images yourself.
For me, the built images work like expected.
]]>To be able to use the new feature, you need to enable a special flag for the LVM volume group. This example shows the general workflow for a fresh setup.
1) Create the volume group with the snapshot-as-volume-chain feature turned on:
pvesm add lvm lvmthick --content images --vgname lvm --snapshot-as-volume-chain 12) From this point on, you can create virtual machines right away, BUT those virtual machines disks must use the QCOW image format for their disk volumes. If you use the RAW format, you wont be able to create snapshots, still.
VMID=401
qm create $VMID --name vm-lvmthick
qm set $VMID -scsi1 lvmthick:2,format=qcow2So, why would it make sense to format the LVM volume as QCOW?
Snapshots on LVM thick provisioned devices are, as everybody knows, a very I/O intensive task. Besides each snapshot, a special -cow Device is created that tracks the changed block regions and the original block data for each change to the active volume. This will waste quite some space within your volume group for each snapshot.
Formatting the LVM volume as QCOW image, makes it possible to use the QCOW backing-image option for these devices, this is the way PVE 9 handles these kind of snapshots.
Creating a snapshot looks like this:
qm snapshot $VMID id
snapshotting 'drive-scsi1' (lvmthick3:vm-401-disk-0.qcow2)
Renamed "vm-401-disk-0.qcow2" to "snap_vm-401-disk-0_id.qcow2" in volume group "lvm"
Rounding up size to full physical extent 1.00 GiB
Logical volume "vm-401-disk-0.qcow2" created.
Formatting '/dev/lvm/vm-401-disk-0.qcow2', fmt=qcow2 cluster_size=131072 extended_l2=on preallocation=metadata compression_type=zlib size=1073741824 backing_file=snap_vm-401-disk-0_id.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16So it will rename the current active disk and create another QCOW formatted LVM volume, but pointing it to the snapshot image using the backing_file option.
Neat.
]]>Usually the incremental backup feature using bitmaps was limited to qcow2 based images, as there was no way to store the bitmaps persistently within raw devices. This basically ruled out proper incremental backups for direct attached luns, etc.
In the past, there were some discussions how to implement this, mostly by using a separate metadata qcow image, holding the bitmap information persistently.
These approaches have been discussed again lately and required features were implemented
In order to be able to use the feature, you need to configure the virtual machines and its disks in a special way:
Lets assume you have a virtual machine that uses a raw device
/tmp/datafile.raw
1) Create an qcow image (same size as the raw image):
# point the data-file to a temporary file, as create will overwrite whatever it finds here
qemu-img create -f qcow2 /tmp/metadata.qcow2 -o data_file=/tmp/TEMPFILE,data_file_raw=true ..
rm -f /tmp/TEMPFILE2) Now use the amend option to point the qcow image to the right raw device using the data-file option:
qemu-img amend /tmp/metadata.qcow2 -o data_file=/tmp/datafile.raw,data_file_raw=true3) Reconfigure the virtual machine configuration to look like this:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap'/>
<source file='/tmp/metadata.qcow2'>
<dataStore type='file'>
<format type='raw'/>
<source file='/tmp/datafile.raw'/>
</dataStore>
</source>
<target dev='vda' bus='virtio'/>
</disk>Now its possible to create persistent checkpoints:
virsh checkpoint-create-as vm6 --name test --diskspec vda,bitmap=test
Domain checkpoint test createdand the persistent bitmap will be stored within the metadata image:
qemu-img info /tmp/tmp.16TRBzeeQn/vm6-sda.qcow2
[..]
bitmaps:
[0]:
flags:
[0]: auto
name: test
granularity: 65536Hoooray.
]]>Usually, during backup, Qemu will use a so called copy-before-write filter so
that data for new guest writes is sent to the backup target first, the guest
write blocks until this operation is finished.
If the backup target is flaky, or becomes unavailable during backup operation, this could lead to high I/O wait times or even complete VM lockups.
To fix this, a so called “fleecing” image is introduced during backup being used as temporary cache for write operations by the guest. This image can be placed on the same storage as the virtual machine disks, so is independent from the backup target performance.
The documentation on which steps are required to get this going, using the Qemu QMP protocol is, lets say.. lacking..
The following examples show the general functionality, but should be enhanced
to use transactions where possible. All commands are in qmp-shell command
format.
Lets start with a full backup:
# create a new bitmap
block-dirty-bitmap-add node=disk1 name=bitmap persistent=true
# add the fleece image to the virtual machine (same size as original disk required)
blockdev-add driver=qcow2 node-name=fleecie file={"driver":"file","filename":"/tmp/fleece.qcow2"}
# add the backup target file to the virtual machine
blockdev-add driver=qcow2 node-name=backup-target-file file={"driver":"file","filename":"/tmp/backup.qcow2"}
# enable the copy-before-writer for the first disk attached, utilizing the fleece image
blockdev-add driver=copy-before-write node-name=cbw file=disk1 target=fleecie
# "blockdev-replace": make the copy-before-writer filter the major device (use "query-block" to get path parameter value, qdev node)
qom-set path=/machine/unattached/device[20] property=drive value=cbw
# add the snapshot-access filter backing the copy-before-writer
blockdev-add driver=snapshot-access file=cbw node-name=snapshot-backup-source
# create a full backup
blockdev-backup device=snapshot-backup-source target=backup-target-file sync=full job-id=test
[ wait until block job finishes]
# remove the snapshot access filter from the virtual machine
blockdev-del node-name=snapshot-backup-source
# switch back to the regular disk
qom-set path=/machine/unattached/device[20] property=drive value=node-disk1
# remove the copy-before-writer
blockdev-del node-name=cbw
# remove the backup-target-file
blockdev-del node-name=backup-target-file
# detach the fleecing image
blockdev-del node-name=fleecieAfter this process, the temporary fleecing image can be deleted/recreated. Now lets go for a incremental backup:
# add the fleecing and backup target image, like before
blockdev-add driver=qcow2 node-name=fleecie file={"driver":"file","filename":"/tmp/fleece.qcow2"}
blockdev-add driver=qcow2 node-name=backup-target-file file={"driver":"file","filename":"/tmp/backup-incremental.qcow2"}
# add the copy-before-write filter, but utilize the bitmap created during full backup
blockdev-add driver=copy-before-write node-name=cbw file=disk1 target=fleecie bitmap={"node":"disk1","name":"bitmap"}
# switch device to the copy-before-write filter
qom-set path=/machine/unattached/device[20] property=drive value=cbw
# add the snapshot-access filter
blockdev-add driver=snapshot-access file=cbw node-name=snapshot-backup-source
# merge the bitmap created during full backup to the snapshot-access device so
# the backup operation can access it. (you better use an transaction here)
block-dirty-bitmap-add node=snapshot-backup-source name=bitmap
block-dirty-bitmap-merge node=snapshot-backup-source target=bitmap bitmaps=[{"node":"disk1","name":"bitmap"}]
# create incremental backup (you better use an transaction here)
blockdev-backup device=snapshot-backup-source target=backup-target-file job-id=test sync=incremental bitmap=bitmap
[ wait until backup has finished ]
[ cleanup like before ]
# clear the dirty bitmap (you better use an transaction here)
block-dirty-bitmap-clear node=disk1 name=bitmapOr, use a simple reproducer by directly passing qmp commands via stdio:
#!/usr/bin/bash
qemu-img create -f raw disk 1M
qemu-img create -f raw fleece 1M
qemu-img create -f raw backup 1M
qemu-system-x86_64 -drive node-name=disk,file=disk,format=file -qmp stdio -nographic -nodefaults <<EOF
{"execute": "qmp_capabilities"}
{"execute": "block-dirty-bitmap-add", "arguments": {"node": "disk", "name": "bitmap"}}
{"execute": "blockdev-add", "arguments": {"node-name": "fleece", "driver": "file", "filename": "fleece"}}
{"execute": "blockdev-add", "arguments": {"node-name": "backup", "driver": "file", "filename": "backup"}}
{"execute": "blockdev-add", "arguments": {"node-name": "cbw", "driver": "copy-before-write", "file": "disk", "target": "fleece", "bitmap": {"node": "disk", "name": "bitmap"}}}
{"execute": "query-block"}
{"execute": "qom-set", "arguments": {"path": "/machine/unattached/device[4]", "property": "drive", "value": "cbw"}}
{"execute": "blockdev-add", "arguments": {"node-name": "snapshot", "driver": "snapshot-access", "file": "cbw"}}
{"execute": "block-dirty-bitmap-add", "arguments": {"node": "snapshot", "name": "tbitmap"}}
{"execute": "block-dirty-bitmap-merge", "arguments": {"node": "snapshot", "target": "tbitmap", "bitmaps": [{"node": "disk", "name": "bitmap"}]}}
[..]
{"execute": "quit"}
EOFSmall utility to scan virtual machine backups on PBS via clamav.
]]>It seems golang shared libraries don’t play well with programs that fork().
As a result, the Plugin was only usable if nbdkit was run in foreground mode (-f), making it impossible to use nbdkit’s’ captive modes, which are quite useful.. Lessons learned.
]]>Lately, i wanted to access VM backups from a Proxmox Backup Server via network (not by using the proxmox-backup-client map function..)
For example, to test-boot a virtual machine snapshot directly from a backup. NBD suits that usecase quite well, so i quickly put a nbdkit plugin together that can be used for this.
The available golang bindings for the proxmox backup client API, made that quite easy.
As nbdkit already comes with a neat COW plugin, its only been a few lines of go code resulting in: pbsnbd
]]>