hammer2: no space left on device on Dragonfly BSD

The Issue

hammer2 does not actually delete a file when you rm or unlink it. Since recovery of the file is possible (this is the design of hammer2), there will still be an entry taking up data. It's similar to how git works.

[root@ ~]# df -h
Filesystem         Size   Used  Avail Capacity  Mounted on
vbd0s1a           16.6G  12.4G  4314M    75%    /

Even with 75% usage listed here, the filesystem could still have filled up. If you are using it as your root filesystem, then attempts to clean up data may fail. If the kernel panics over this, you will see something like this.

Rebooting will probably not succeed.

The Fix

If you have a recent enough version of the rescue ramdisk installed, on bootup you can press 'r' and access the rescue ramdisk. Your provider will have to offer some sort of remote interface for interacting with the operating system before it boots, like VNC or IPMI. You can then mount your filesystem using:

[root@ ~]# mkdir /tmp/fs
[root@ ~]# mount_hammer2 -o local /dev/vbd0s1a /tmp/fs

If you receive an error that /sbin/hammer2 is not found, then your rescue ramdisk is not up to date enough. In that scenario, download the latest 5.2 iso from dragonflybsd.org and boot from the cd-rom on your virtual machine or physical device. Just login as root instead of installer.

If the mount does succeed, then all you have to do is run the following twice:

[root@ ~]# /sbin/hammer2 bulkfree /tmp/fs

If you do not have enough memory on your machine, you may need to mount swap. Add your swap partition to the /etc/fstab and then do:

[root@ ~]# swapon -a

Once you have ran the bulkfree command twice, the usage reported by df -h will be correct. However, there is a chance on reboot that a core dump will be placed in /var/crash/ so be prepared to have plenty of space free in case that happens. You should also delete any files you can and run the bulkfree operation twice afterwards to clear up additional space.