update draft

This commit is contained in:
John Bowdre 2024-07-20 18:34:05 -05:00
parent 981571fbc4
commit 6e1640368e
2 changed files with 341 additions and 10 deletions

View file

@ -160,6 +160,8 @@ identity_policies []
policies ["packer"]
```
The token will only be displayed this once so I make sure to copy it somewhere safe.
Within the `packer` secrets engine, I have two secrets which each have a number of subkeys.
`proxmox` contains values related to the Proxmox environment:
@ -701,9 +703,8 @@ local "proxmox_token_secret" {
And the next `locals {}` block leverages other expressions to:
- dynamically set `local.build_date` to the current time (l. 70),
- combine individual string variables, like `local.iso_checksum` and `local.iso_path` (ll. 73-74),
- define a shutdown command to clean up sudoers includes and shutdown the VM at the end of the build (ll. 75),
- capture the keypair generated by the SSH key plugin (ll. 76-77),
- and use the [`templatefile()` function](https://developer.hashicorp.com/packer/docs/templates/hcl_templates/functions/file/templatefile) to process the `cloud-init` config file and insert appropriate variables (ll. 78-101)
- capture the keypair generated by the SSH key plugin (ll. 75-76),
- and use the [`templatefile()` function](https://developer.hashicorp.com/packer/docs/templates/hcl_templates/functions/file/templatefile) to process the `cloud-init` config file and insert appropriate variables (ll. 77-100)
```hcl
# torchlight! {"lineNumbers":true, "lineNumbersStart":69}
@ -711,9 +712,8 @@ locals {
build_date = formatdate("YYYY-MM-DD hh:mm ZZZ", timestamp()) # [tl! ~~]
build_description = "Ubuntu Server 22.04 LTS template\nBuild date: ${local.build_date}\nBuild tool: ${local.build_tool}"
build_tool = "HashiCorp Packer ${packer.version}"
iso_checksum = "${var.iso_checksum_type}:${var.iso_checksum_value}" # [tl! ~~:2]
iso_checksum = "${var.iso_checksum_type}:${var.iso_checksum_value}" # [tl! ~~:1]
iso_path = "${local.proxmox_iso_path}/${var.iso_file}"
shutdown_command = "sudo sh -c 'rm -f /etc/sudoers.d/*; /usr/sbin/shutdown -P now'"
ssh_private_key_file = data.sshkey.install.private_key_path # [tl! ~~:1]
ssh_public_key = data.sshkey.install.public_key
data_source_content = { # [tl! ~~:23]
@ -1122,12 +1122,12 @@ The post install scripts run after the `cloud-init` installation has completed,
if [ -f /etc/cloud/cloud.cfg.d/99-installer.cfg ]; then
sudo rm /etc/cloud/cloud.cfg.d/99-installer.cfg
echo 'Deleting subiquity cloud-init config'
echo '>> Deleting subiquity cloud-init config...'
fi
if [ -f /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg ]; then
sudo rm /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
echo 'Deleting subiquity cloud-init network config'
echo '>> Deleting subiquity cloud-init network config...'
fi
```
3. `install-ca-certs.sh` to install any trusted CA certs which were in the `certs/` folder of the Packer environment and copied to `/tmp/certs/` in the guest:
@ -1170,8 +1170,8 @@ The post install scripts run after the `cloud-init` installation has completed,
# disables multipathd
set -eu
echo '>> Disabling multipathd...'
sudo systemctl disable multipathd
echo 'Disabling multipathd'
```
5. `prune-motd.sh` to remove those noisy, promotional default messages that tell you to enable cockpit or check out Ubuntu Pro or whatever:
```shell
@ -1181,7 +1181,6 @@ The post install scripts run after the `cloud-init` installation has completed,
set -eu
echo '>> Pruning default MOTD...'
if awk -F= '/^ID/{print $2}' /etc/os-release | grep -q rhel; then
if [ -L "/etc/motd.d/insights-client" ]; then
sudo unlink /etc/motd.d/insights-client
@ -1211,6 +1210,7 @@ The post install scripts run after the `cloud-init` installation has completed,
# configures pam_mkhomedir to create home directories with 750 permissions
set -eu
echo '>> Configuring pam_mkhomedir...'
sudo sed -i 's/optional.*pam_mkhomedir.so/required\t\tpam_mkhomedir.so umask=0027/' /usr/share/pam-configs/mkhomedir
```
8. `update-packages.sh` to install any available package updates and reboot:
@ -1238,5 +1238,336 @@ The post install scripts run after the `cloud-init` installation has completed,
fi
```
After the reboot, the process picks back up with the pre-final scripts.
#### Build Script
##### Pre-Final
1. `cleanup-cloud-init.sh` performs a [`clean`](https://cloudinit.readthedocs.io/en/latest/reference/cli.html#clean) action to get the template ready to be re-used:
```shell
# torchlight! {"lineNumbers":true}
#!/usr/bin/env bash
# cleans up cloud-init state
set -eu
echo '>> Cleaning up cloud-init state...'
sudo cloud-init clean -l
```
2. `cleanup-packages.sh` uninstalls packages and kernel versions which are no longer needed:
```shell
# torchlight! {"lineNumbers":true}
#!/usr/bin/env bash
# cleans up unneeded packages to reduce the size of the image
set -eu
if awk -F= '/^ID/{print $2}' /etc/os-release | grep -q debian; then
echo '>> Cleaning up unneeded packages...'
sudo apt-get -y autoremove --purge
sudo apt-get -y clean
elif awk -F= '/^ID/{print $2}' /etc/os-release | grep -q rhel; then
if which dnf &>/dev/null; then
echo '>> Cleaning up unneeded packages...'
sudo dnf -y remove linux-firmware
sudo dnf -y remove "$(dnf repoquery --installonly --latest-limit=-1 -q)"
sudo dnf -y autoremove
sudo dnf -y clean all --enablerepo=\*;
else
echo '>> Cleaning up unneeded packages...'
sudo yum -y remove linux-firmware
sudo package-cleanup --oldkernels --count=1
sudo yum -y autoremove
sudo yum -y clean all --enablerepo=\*;
fi
fi
3. `build/linux/22-04-lts/hardening.sh` is a build-specific script to perform basic hardening tasks toward the CIS Level 2 server benchmark. It doesn't have a lot of fancy logic because it is *only intended to be run during this package process* when it's making modifications from a known state. It's long so I won't repost it here, and I may end up writing a separate post specifically about this hardening process, but you're welcome to view the full script for [Ubuntu 22.04 here](https://github.com/jbowdre/packer-proxmox-templates/blob/main/builds/linux/ubuntu/22-04-lts/hardening.sh).
4. `zero-disk.sh` fills a file with zeroes until the disk runs out of space, and then removes it, resulting in a reduced template image size:
```shell
# torchlight! {"lineNumbers":true}
#!/usr/bin/env bash
# zeroes out free space to reduce disk size
set -eu
echo '>> Zeroing free space to reduce disk size...'
sudo sh -c 'dd if=/dev/zero of=/EMPTY bs=1M || true; sync; sleep 1; sync'
sudo sh -c 'rm -f /EMPTY; sync; sleep 1; sync'
```
5. `generalize.sh` performs final steps to get the template ready for cloning, including removing the `sudoers.d` configuration which allowed passwordless elevation during the setup:
```shell
# torchlight! {"lineNumbers":true}
#!/usr/bin/env bash
# prepare a VM to become a template.
set -eu
echo '>> Clearing audit logs...'
sudo sh -c 'if [ -f /var/log/audit/audit.log ]; then
cat /dev/null > /var/log/audit/audit.log
fi'
sudo sh -c 'if [ -f /var/log/wtmp ]; then
cat /dev/null > /var/log/wtmp
fi'
sudo sh -c 'if [ -f /var/log/lastlog ]; then
cat /dev/null > /var/log/lastlog
fi'
sudo sh -c 'if [ -f /etc/logrotate.conf ]; then
logrotate -f /etc/logrotate.conf 2>/dev/null
fi'
sudo rm -rf /var/log/journal/*
sudo rm -f /var/lib/dhcp/*
sudo find /var/log -type f -delete
echo '>> Clearing persistent udev rules...'
sudo sh -c 'if [ -f /etc/udev/rules.d/70-persistent-net.rules ]; then
rm /etc/udev/rules.d/70-persistent-net.rules
fi'
# check for only RHEL releases
if awk -F= '/^ID=/{print $2}' /etc/os-release | grep -q rhel; then
echo '>> Clearing RHSM subscription...'
sudo subscription-manager unregister
sudo subscription-manager clean
fi
echo '>> Clearing temp dirs...'
sudo rm -rf /tmp/*
sudo rm -rf /var/tmp/*
# check for RHEL-like releases (RHEL and Rocky)
if awk -F= '/^ID/{print $2}' /etc/os-release | grep -q rhel; then
sudo rm -rf /var/cache/dnf/*
sudo rm -rf /var/log/rhsm/*
fi
echo '>> Clearing host keys...'
sudo rm -f /etc/ssh/ssh_host_*
echo '>> Removing Packer SSH key...'
sed -i '/packer_key/d' ~/.ssh/authorized_keys
echo '>> Clearing machine-id...'
sudo truncate -s 0 /etc/machine-id
if [ -f /var/lib/dbus/machine-id ]; then
sudo rm -f /var/lib/dbus/machine-id
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id
fi
echo '>> Clearing shell history...'
unset HISTFILE
history -cw
echo > ~/.bash_history
sudo rm -f /root/.bash_history
echo '>> Clearing sudoers.d...'
sudo rm -f /etc/sudoers.d/*
```
### Packer Build
At this point, I should (in theory) be able to kick off the build from my laptop with a Packer command - but first I'll need to set up some environment variables so that Packer will be able to communicate with my Vault server:
```shell
export VAULT_ADDR="https://vault.tailnet-name.ts.net/" # [tl! .cmd:1]
export VAULT_TOKEN="hvs.CAES[...]GSFQ"
```
Okay, now I can run the Ubuntu 22.04 build from the top-level of my Packer directory like so:
```shell
packer init builds/linux/ubuntu/22-04-lts # [tl! .cmd:1]
packer build -on-error=cleanup -force builds/linux/ubuntu/22-04-lts
proxmox-iso.linux-server: output will be in this color. # [tl! .nocopy:start]
==> proxmox-iso.linux-server: Creating CD disk... # [tl! collapse:15]
proxmox-iso.linux-server: xorriso 1.5.6 : RockRidge filesystem manipulator, libburnia project.
proxmox-iso.linux-server: xorriso : NOTE : Environment variable SOURCE_DATE_EPOCH encountered with value 315532800
proxmox-iso.linux-server: Drive current: -outdev 'stdio:/tmp/packer684761677.iso'
proxmox-iso.linux-server: Media current: stdio file, overwriteable
proxmox-iso.linux-server: Media status : is blank
proxmox-iso.linux-server: Media summary: 0 sessions, 0 data blocks, 0 data, 174g free
proxmox-iso.linux-server: xorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA 119 rules
proxmox-iso.linux-server: Added to ISO image: directory '/'='/tmp/packer_to_cdrom2909484587'
proxmox-iso.linux-server: xorriso : UPDATE : 2 files added in 1 seconds
proxmox-iso.linux-server: xorriso : UPDATE : 2 files added in 1 seconds
proxmox-iso.linux-server: ISO image produced: 186 sectors
proxmox-iso.linux-server: Written to medium : 186 sectors at LBA 0
proxmox-iso.linux-server: Writing to 'stdio:/tmp/packer684761677.iso' completed successfully.
proxmox-iso.linux-server: Done copying paths from CD_dirs
proxmox-iso.linux-server: Uploaded ISO to local:iso/packer684761677.iso
==> proxmox-iso.linux-server: Force set, checking for existing artifact on PVE cluster
==> proxmox-iso.linux-server: No existing artifact found
==> proxmox-iso.linux-server: Creating VM
==> proxmox-iso.linux-server: No VM ID given, getting next free from Proxmox
==> proxmox-iso.linux-server: Starting VM
==> proxmox-iso.linux-server: Waiting 4s for boot
==> proxmox-iso.linux-server: Typing the boot command
==> proxmox-iso.linux-server: Waiting for SSH to become available... # [tl! .nocopy:end]
```
It'll take a few minutes while Packer waits on SSH, and while *I* wait on that I can look at the Proxmox console for the VM to follow along with the installer's progress:
![Proxmox VM console showing the installer progress](proxmox-console-progress.png)
That successful SSH connection signifies the transition from the `source {}` block to the `build {}` block, so it starts with uploading any certs and the `join-domain.sh` script before getting into running those post-install tasks:
```shell
==> proxmox-iso.linux-server: Connected to SSH! # [tl! .nocopy:start **:2]
==> proxmox-iso.linux-server: Uploading certs => /tmp
==> proxmox-iso.linux-server: Uploading scripts/linux/join-domain.sh => /home/john/join-domain.sh
proxmox-iso.linux-server: join-domain.sh 5.59 KiB / 5.59 KiB [========================================================================================================] 100.00% 0s
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/wait-for-cloud-init.sh # [tl! **:start]
proxmox-iso.linux-server: >> Waiting for cloud-init...
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/cleanup-subiquity.sh
proxmox-iso.linux-server: >> Deleting subiquity cloud-init config...
proxmox-iso.linux-server: >> Deleting subiquity cloud-init network config...
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/install-ca-certs.sh
proxmox-iso.linux-server: >> Installing certificates...
proxmox-iso.linux-server: No certs to install.
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/disable-multipathd.sh
proxmox-iso.linux-server: >> Disabling multipathd... # [tl! **:end]
==> proxmox-iso.linux-server: Removed /etc/systemd/system/multipath-tools.service.
==> proxmox-iso.linux-server: Removed /etc/systemd/system/sockets.target.wants/multipathd.socket.
==> proxmox-iso.linux-server: Removed /etc/systemd/system/sysinit.target.wants/multipathd.service.
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/prune-motd.sh # [tl! **:3]
proxmox-iso.linux-server: >> Pruning default MOTD...
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/persist-cloud-init-net.sh
proxmox-iso.linux-server: >> Preserving network settings...
proxmox-iso.linux-server: manual_cache_clean: True
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/configure-pam_mkhomedir.sh # [tl! **:3]
proxmox-iso.linux-server: >> Configuring pam_mkhomedir...
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/update-packages.sh
proxmox-iso.linux-server: >> Checking for and installing updates...
proxmox-iso.linux-server: Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
proxmox-iso.linux-server: Hit:2 http://us.archive.ubuntu.com/ubuntu jammy InRelease
proxmox-iso.linux-server: Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
proxmox-iso.linux-server: Hit:4 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
proxmox-iso.linux-server: Reading package lists...
proxmox-iso.linux-server: Reading package lists...
proxmox-iso.linux-server: Building dependency tree...
proxmox-iso.linux-server: Reading state information...
proxmox-iso.linux-server: Calculating upgrade...
proxmox-iso.linux-server: The following packages have been kept back:
proxmox-iso.linux-server: python3-update-manager update-manager-core
proxmox-iso.linux-server: 0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
proxmox-iso.linux-server: >> Rebooting! # [tl! ** .nocopy:end]
```
There's a brief pause during the reboot, and then things pick back up with the hardening script and then the cleanup tasks:
```shell
==> proxmox-iso.linux-server: Pausing 30s before the next provisioner... # [tl! .nocopy:start]
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/cleanup-cloud-init.sh # [tl! **:3]
proxmox-iso.linux-server: >> Cleaning up cloud-init state...
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/cleanup-packages.sh
proxmox-iso.linux-server: >> Cleaning up unneeded packages...
proxmox-iso.linux-server: Reading package lists...
proxmox-iso.linux-server: Building dependency tree...
proxmox-iso.linux-server: Reading state information...
proxmox-iso.linux-server: 0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/builds/linux/ubuntu/22-04-lts/hardening.sh # [tl! **:1]
proxmox-iso.linux-server: >>> Beginning hardening tasks...
proxmox-iso.linux-server: [...]
proxmox-iso.linux-server: >>> Hardening script complete!
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/zero-disk.sh # [tl! **:1]
proxmox-iso.linux-server: >> Zeroing free space to reduce disk size...
==> proxmox-iso.linux-server: dd: error writing '/EMPTY': No space left on device
==> proxmox-iso.linux-server: 25905+0 records in
==> proxmox-iso.linux-server: 25904+0 records out
==> proxmox-iso.linux-server: 27162312704 bytes (27 GB, 25 GiB) copied, 10.7024 s, 2.5 GB/s
==> proxmox-iso.linux-server: Provisioning with shell script: /home/john/projects/packer-proxmox-templates/scripts/linux/generalize.sh # [tl! **:10]
proxmox-iso.linux-server: >> Clearing audit logs...
proxmox-iso.linux-server: >> Clearing persistent udev rules...
proxmox-iso.linux-server: >> Clearing temp dirs...
proxmox-iso.linux-server: >> Clearing host keys...
proxmox-iso.linux-server: >> Removing Packer SSH key...
proxmox-iso.linux-server: >> Clearing machine-id...
proxmox-iso.linux-server: >> Clearing shell history...
proxmox-iso.linux-server: >> Clearing sudoers.d...
==> proxmox-iso.linux-server: Stopping VM
==> proxmox-iso.linux-server: Converting VM to template
proxmox-iso.linux-server: Deleted generated ISO from local:iso/packer152219352.iso
Build 'proxmox-iso.linux-server' finished after 10 minutes 52 seconds. # [tl! **:5]
==> Wait completed after 10 minutes 52 seconds
==> Builds finished. The artifacts of successful builds are:
--> proxmox-iso.linux-server: A template was created: 105 # [tl! .nocopy:end]
```
That was a *lot* of prep work but now that everything is in place it only takes about eleven minutes to create a fresh Ubuntu 22.04 template, and that template is fully up-to-date and hardened to about 95% of the CIS Level 2 benchmark. This will save me a lot of time as I build new VMs in my homelab.
### Wrapper Script
But having to export the Vault variables and run the Packer commands manually is a bit of a chore. So I put together a couple of helper scripts to help streamline things. This will really come in handy as I add new OS variants and schedule automated builds with GitHub Actions.
First, I made a `vault-env.sh` script to hold my Vault address and the token for Packer.
{{% notice note "Sensitive Values!" %}}
The `VAULT_TOKEN` variable is a sensitive value and should be protected. This file should be added to `.gitignore` to ensure it doesn't get inadvertently committed to a repo.
{{% /notice %}}
```shell
# torchlight! {"lineNumbers":true}
#!/usr/bin/env bash
set -eu
export VAULT_ADDR="https://vault.tailnet-name.ts.net/"
export VAULT_TOKEN="hvs.CAES[...]GSFQ"
```
This `build.sh` scripts expects a single argument: the name of the build to create. It then checks to see if the `VAULT_TOKEN` environment variable is already set; if not, it tries to sources it from `vault-env.sh`. And then it kicks off the appropriate build.
```shell
# torchlight! {"lineNumbers":true}
#!/usr/bin/env bash
# Run a single packer build
#
# Specify the build as an argument to the script. Ex:
# ./build.sh ubuntu2204
set -eu
if [ $# -ne 1 ]; then
echo """
Syntax: $0 [BUILD]
Where [BUILD] is one of the supported OS builds:
ubuntu2204 ubuntu2404
"""
exit 1
fi
if [ ! "${VAULT_TOKEN+x}" ]; then
source vault-env.sh || ( echo "No Vault config found"; exit 1 )
fi
build_name="${1,,}"
build_path=
case $build_name in
ubuntu2204)
build_path="builds/linux/ubuntu/22-04-lts/"
;;
ubuntu2404)
build_path="builds/linux/ubuntu/24-04-lts/"
;;
*)
echo "Unknown build; exiting..."
exit 1
;;
esac
packer init "${build_path}"
packer build -on-error=cleanup -force "${build_path}"
```
Then I can kick off a build with just:
```shell
./build.sh ubuntu2204 # [tl! .cmd]
proxmox-iso.linux-server: output will be in this color. # [tl! .nocopy:6]
==> proxmox-iso.linux-server: Creating CD disk...
proxmox-iso.linux-server: xorriso 1.5.6 : RockRidge filesystem manipulator, libburnia project.
proxmox-iso.linux-server: xorriso : NOTE : Environment variable SOURCE_DATE_EPOCH encountered with value 315532800
proxmox-iso.linux-server: Drive current: -outdev 'stdio:/tmp/packer2372067848.iso'
[...]
```
### Up Next...
Being able to generate a template on-demand is pretty cool, but the next stage of this project is to integrate it with a GitHub Actions workflow so that the templates can be built automatically on a schedule or as the configuration gets changed. But this post is long enough (and I've been poking at it for long enough) so that explanation will have to wait for another time.
(If you'd like a sneak peak of what's in store, take a self-guided tour of [the GitHub repo](https://github.com/jbowdre/packer-proxmox-templates).)
Stay tuned!

Binary file not shown.

After

Width:  |  Height:  |  Size: 464 KiB