Wrap Dangerous Commands in pre-post ZFS or BRTFS Snapshots
Diggin’ Dotfiles CollectionPermalink
It’s time to dig the dotfiles, and to dig in to them! This post is a part of a blog post collection called Diggin’ Dotfiles where I unearth some gems from my personal dotfiles repo erikw/dotfiles.
The following articles are a part of this series:
- Manage shell configurations across different systems
- Wrap Dangerous Commands in pre-post ZFS or BRTFS Snapshots
- Finger of god: using sudo with TouchID on macOS
- Generate permutation shell aliases
Dangerous Commands in the Daily Life of a SysadminPermalink
We’re all the sysadmin of our machines nowadays and as we go on, we will often sway around with potentially very dangerous commands that has the potential to break the system.
The most common dangerous operation would probably be to install or upgrade some packages with a (systems) package manager, or maybe upgrade the OS itself even. Other commands are directly destructible and non-reversible like for example dd(1). For those running an OS that is working with rolling releases like FreeBSD or Arch Linux, it pays of to keep up with the development and upgrade frequent but small instead of seldom and big. This means however, executing dangerous commands becomes part of everyday life.
Does
$ sudo pacman -Syyu
$ # -or-
$ sudo freebsd-upgrade installgive you shivers? It should! But it must not…
Is there a way to make this safer? Yes!
Copy-on-write Snappshotting to the RescuePermalink
COW (Copy-on-write is a space-efficient storage management technique where copies are made in a soft fashion and only the diff from the source is recorded.
) is a feature that some file systems offer that let’s you very easily make a whole (system) disk snapshot for free in terms of storage. Of course nothing is for free, but the idea is that the current live data and a snapshot at some time in the past share data that are not modified. Put in another way, the difference on disk between the live and a snapshot is only those files that are different. Thus this has the potential to be quite storage efficient
Two file systems that I’ve been using a lot the last 10 years are Brtfs (A Linux native file system with features like copy-on-write and easy logical volume management.
) on my Arch (Probably the best Linux distribution you can get.
) Linux system and ZFS (A well honored filesystem and volume manager popular on BSD-derived operating systems.
) on my FreeBSD one. The both provide full file system snapshots and tools to view, restore, diff etc. these snapshots with each other and the current live data.
snp & znp: Snapshot Wrappers for CommandsPermalink
Over time I whipped up two scripts for the respective file systems that basically
- makes a
pre-snapshot of the full system. - executes the supplied system command.
- makes a
post-snapshot of the full system.
Additionally the output of the command is logged to /var/local/log/ with a timestamp.
The script can be executed by just being a “prefix” to the normal command you would do:
$ sudo snp echo "not-so-dangerous-command"
$ sudo snp pacman -Syyu # a bit more dangerous command mitigated
$ sudo znp freebsd-upgrade install # quite serious...
$ # super dangerous if a space is added before * by mistake
$ sudo znp rm -rf abc* With this, we can in case of issues easily do a diff between these two snapshots and see what really happened on the file system level when those packages got upgraded or whatever the command did. Furthermore an issue might arise only after a few days and you don’t have access to the output of that command anymore. Then we just go to the log in /var/local/log/ to refresh the mind and the proceed to inspect the logged snapshot numbers listed in the logs.
I can tell you, over the course of the years these snapshots have saved me (and my systems) many times! I made it a habit to use these snapshots frequently as they are cheap anyways. You should figure out a way to clean up snapshots by setting up a retention policy scheme as well. Additionally if you’re getting in to snapshotting, why not make snapshots all the time? For BSD there’s the excellent zfstools that let’s you do exactly that, snapshots on a (cron) time schedule.
snpPermalink
Hosted on Gist at erikw/snp.
| #!/usr/bin/env bash | |
| # Runs a command wrapped in btrfs snapper pre-post snapshots. | |
| # Usage: $ snp <commands> | |
| # e.g.: $ snp pacman -Syyu | |
| # Requirements: snapper (https://wiki.archlinux.org/title/snapper) | |
| # The latest version of this script is hosted at https://gist.github.com/erikw/5229436 | |
| log_path="/var/local/log/snp" | |
| date=$(date "+%Y-%m-%d-%H%M%S") | |
| log_file="${log_path}/snp_${date}.log" | |
| ! [ -d $log_path ] && mkdir -p $log_path | |
| # Log stdout and stderr. Reference: http://stackoverflow.com/questions/3173131/redirect-copy-of-stdout-to-log-file-from-within-bash-script-itself | |
| exec > >(tee -a "$log_file") | |
| exec 2> >(tee -a "$log_file" >&2) | |
| cmd="$@" | |
| echo "> Logging to: ${log_file}" | |
| snapshot_nbr=$(snapper create --type=pre --cleanup-algorithm=number --print-number --description="${cmd}") | |
| echo "> New pre snapshot with number ${snapshot_nbr}." | |
| echo -e "> Running command \"${cmd}\".\n" | |
| eval "$cmd" | |
| snapshot_nbr=$(snapper create --type=post --cleanup-algorithm=number --print-number --pre-number="$snapshot_nbr") | |
| echo -e "\n> New post snapshot with number ${snapshot_nbr}." | |
| # Snapper has a --command option nowadays. But it works worse, the output from the command is not printed separately from the snaptshot number, just becomes a mess. | |
| #echo "> Running command \"${cmd}\"." | |
| #snapshot_nbr=$(snapper create --command "${cmd}" --print-number --cleanup-algorithm=number --description="${cmd}" | tail -1) | |
| #echo -e "\n> New pre-post snapshot with numbers ${snapshot_nbr}." |
znpPermalink
Hosted on Gist at erikw/znp.
| #!/usr/bin/env bash | |
| # Runs a command wrapped in ZFS pre-post snapshots. The whole data pool is recursively snapshotted. | |
| # Analogous to my snp script for BTRFS: https://gist.github.com/erikw/5229436 | |
| # Usage: $ znp <commands> | |
| # e.g.: $ znp pgk upgrade | |
| # e.g.: $ znp portmaster -aG | |
| # e.g.: $ znp freebsd-upgrade install | |
| zfs_pool=zroot | |
| log_path="/var/local/log/znp" | |
| date=$(date "+%Y-%m-%d-%H%M%S") | |
| snapshot_prefix="znp_${date}" | |
| log_file="${log_path}/znp_${date}.log" | |
| ! [ -d $log_path ] && mkdir -p $log_path | |
| # Log stdout and stderr. Reference: http://stackoverflow.com/questions/3173131/redirect-copy-of-stdout-to-log-file-from-within-bash-script-itself | |
| exec > >(tee -a "$log_file") | |
| exec 2> >(tee -a "$log_file" >&2) | |
| cmd="$@" | |
| echo "> Logging to: ${log_file}" | |
| # snapper create --type=pre --cleanup-algorithm=number --print-number --description="${cmd}" | |
| snapshot_pre="${zfs_pool}@${snapshot_prefix}_pre" | |
| zfs snapshot -r $snapshot_pre | |
| echo "> New pre snapshot created: ${snapshot_pre}" | |
| echo -e "> Running command \"${cmd}\".\n" | |
| eval "$cmd" | |
| ecode="$?" | |
| echo -e "\n> Command exit code: ${ecode}" | |
| snapshot_post="${zfs_pool}@${snapshot_prefix}_post" | |
| zfs snapshot -r $snapshot_post | |
| echo "> New post snapshot created: ${snapshot_post}" | |
| echo "> See the diff between them using:" | |
| # Tip: add -t to xargs to print every individual command. | |
| echo "zfs list -t snapshot -o name | grep ${snapshot_prefix} | cut -d@ -f1 | uniq | xargs -I{} -n1 sh -c 'zfs diff -F {}@${snapshot_prefix}_pre {}@${snapshot_prefix}_post 2>/dev/null'" | |
| echo "> Destroy these snapshots with:" | |
| echo "zfs destroy -r ${snapshot_pre} && zfs destroy -r ${snapshot_post}" | |
| echo "> Destroy all snapshots created with znp:" | |
| echo "zfs list -H -o name -t snapshot | grep "${zfs_pool}@znp_" | xargs -n1 zfs destroy -r" | |
| echo "> Destroy the 10 oldest snapshot pairs created with znp:" | |
| echo "zfs list -H -o name -t snapshot | grep "${zfs_pool}@znp_" | head -20 | xargs -n1 zfs destroy -r" |
Leave a comment
Your email address will not be published. Required fields are marked *