vPotato/virtuallypotato
1
0
Fork 0
mirror of https://github.com/jbowdre/virtuallypotato.git synced 2024-11-27 09:22:18 +00:00
Code Issues Projects Releases Packages Wiki Activity

update draft

Browse source
This commit is contained in:
John Bowdre 2022-12-10 17:02:58 -06:00
parent 4b3150c259
commit bbc6dec2c9
1 changed files with 44 additions and 44 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

88
content/posts/k8s-on-vsphere-node-template-with-packer/index.md
View file

@ -1,8 +1,8 @@
--- ---
title: "K8s on vSphere: Building a Node Template With Packer" # Title of the blog post. title: "K8s on vSphere: Building a Node Template With Packer" # Title of the blog post.
date: 2022-12-03T10:41:17-08:00 # Date of post creation. date: 2022-12-10T17:00:00-06:00 # Date of post creation.
# lastmod: 2022-12-03T10:41:17-08:00 # Date when last modified # lastmod: 2022-12-03T10:41:17-08:00 # Date when last modified
description: "Using HashiCorp Packer to automatically build Kubernetes node templates" # Description used for search engine. description: "Using HashiCorp Packer to automatically build Kubernetes node templates on vSphere." # Description used for search engine.
featured: false # Sets if post is a featured post, making appear on the home page side bar. featured: false # Sets if post is a featured post, making appear on the home page side bar.
draft: true # Sets whether to render this page. Draft of true will not be rendered. draft: true # Sets whether to render this page. Draft of true will not be rendered.
toc: true # Controls if a table of contents should be generated for first-level links automatically. toc: true # Controls if a table of contents should be generated for first-level links automatically.
@ -80,8 +80,6 @@ After quite a bit of experimentation, I've settled on a preferred way to organiz
├── data ├── data
│   ├── meta-data │   ├── meta-data
│   └── user-data.pkrtpl.hcl │   └── user-data.pkrtpl.hcl
├── packer_cache
│   └── ssh_private_key_packer.pem
├── scripts ├── scripts
│   ├── cleanup-cloud-init.sh │   ├── cleanup-cloud-init.sh
│   ├── cleanup-subiquity.sh │   ├── cleanup-subiquity.sh
@ -101,9 +99,8 @@ After quite a bit of experimentation, I've settled on a preferred way to organiz
└── variables.pkr.hcl └── variables.pkr.hcl
``` ```
- The `certs` folder holds the Base64-encoded PEM-formatted certificate of my [internal Certificate Authority](/ldaps-authentication-tanzu-community-edition/#prequisite) which will be automatically installed in the provisioned VM's trusted certificate store. I could also include additional root or intermediate certificates as needed, just as long as the file names end in `*.cer` (we'll see why later). - The `certs` folder holds the Base64-encoded PEM-formatted certificate of my [internal Certificate Authority](/ldaps-authentication-tanzu-community-edition/#prequisite) which will be automatically installed in the provisioned VM's trusted certificate store.
- The `data` folder stores files used for [generating the `cloud-init` configuration](#user-datapkrtplhcl) that will be used for the OS installation and configuration. - The `data` folder stores files for [generating the `cloud-init` configuration](#user-datapkrtplhcl) that will automate the OS installation and configuration.
- `packer_cache` keeps the SSH private key that Packer will use for logging in to the provisioned VM post-install.
- The `scripts` directory holds a [collection of scripts](#post_install_scripts) used for post-install configuration tasks. Sure, I could just use a single large script, but using a bunch of smaller ones helps keep things modular and easy to reuse elsewhere. - The `scripts` directory holds a [collection of scripts](#post_install_scripts) used for post-install configuration tasks. Sure, I could just use a single large script, but using a bunch of smaller ones helps keep things modular and easy to reuse elsewhere.
- `variables.pkr.hcl` declares [all of the variables](#variablespkrhcl) which will be used in the Packer build, and sets the default values for some of them. - `variables.pkr.hcl` declares [all of the variables](#variablespkrhcl) which will be used in the Packer build, and sets the default values for some of them.
- `ubuntu-k8s.auto.pkrvars.hcl` [assigns values](#ubuntu-k8sautopkrvarshcl) to those variables. This is where most of the user-facing options will be configured, such as usernames, passwords, and environment settings. - `ubuntu-k8s.auto.pkrvars.hcl` [assigns values](#ubuntu-k8sautopkrvarshcl) to those variables. This is where most of the user-facing options will be configured, such as usernames, passwords, and environment settings.
@ -131,24 +128,32 @@ packer {
} }
} }
``` ```
As I mentioned above, I'll be using the official [`vsphere` plugin](https://github.com/hashicorp/packer-plugin-vsphere) to handle the provisioning on my vSphere environment. I'll also make use of the [`sshkey` plugin](https://github.com/ivoronin/packer-plugin-sshkey) to easily handle the SSH keys. As I mentioned above, I'll be using the official [`vsphere` plugin](https://github.com/hashicorp/packer-plugin-vsphere) to handle the provisioning on my vSphere environment. I'll also make use of the [`sshkey` plugin](https://github.com/ivoronin/packer-plugin-sshkey) to dynamically generate SSH keys for the build process.
#### `data` block
This section would be used for loading information from various data sources, but I'm only using it for the `sshkey` plugin (as mentioned above).
```text
// BLOCK: data
// Defines data sources.
data "sshkey" "install" {
type = "ed25519"
name = "packer_key"
}
```
This will generate an ECDSA keypair, and the public key will include the identifier `packer_key` to make it easier to manage later on. Using this plugin to generate keys means that I don't have to worry about storing a private key somewhere in the build directory.
#### `locals` block #### `locals` block
Locals are a type of Packer variable which aren't explicitly declared in the `variables.pkr.hcl` file. They only exist within the context of a single build (hence the "local" name). Typical Packer variables are static and don't support string manipulation; locals, however, do support expressions that can be used to change their value on the fly. This makes them very useful when you need to combine variables (like a datastore name, path, filename) into a single string (such as in the highlighted line): Locals are a type of Packer variable which aren't explicitly declared in the `variables.pkr.hcl` file. They only exist within the context of a single build (hence the "local" name). Typical Packer variables are static and don't support string manipulation; locals, however, do support expressions that can be used to change their value on the fly. This makes them very useful when you need to combine variables into a single string or concatenate lists of SSH public keys (such as in the highlighted lines):
```text {hl_lines=[10,17]}
```text {hl_lines=[13]}
// BLOCK: locals // BLOCK: locals
// Defines the local variables. // Defines local variables.
data "sshkey" "install" {
}
locals { locals {
ssh_public_key = data.sshkey.install.public_key ssh_public_key = data.sshkey.install.public_key
ssh_private_key_file = data.sshkey.install.private_key_path ssh_private_key_file = data.sshkey.install.private_key_path
build_tool = "HashiCorp Packer ${packer.version}" build_tool = "HashiCorp Packer ${packer.version}"
build_date = formatdate("YYYY-MM-DD hh:mm ZZZ", timestamp()) build_date = formatdate("YYYY-MM-DD hh:mm ZZZ", timestamp())
build_description = "Kubernetes Ubuntu 20.04 Node template\nBuild date: ${local.build_date}\nBuild tool: ${local.build_tool}" build_description = "Kubernetes Ubuntu 20.04 Node template\nBuild date: ${local.build_date}\nBuild tool: ${local.build_tool}"
shutdown_command = "sudo -S -E shutdown -P now"
iso_paths = ["[${var.common_iso_datastore}] ${var.iso_path}/${var.iso_file}"] iso_paths = ["[${var.common_iso_datastore}] ${var.iso_path}/${var.iso_file}"]
iso_checksum = "${var.iso_checksum_type}:${var.iso_checksum_value}" iso_checksum = "${var.iso_checksum_type}:${var.iso_checksum_value}"
data_source_content = { data_source_content = {
@ -156,7 +161,7 @@ locals {
"/user-data" = templatefile("data/user-data.pkrtpl.hcl", { "/user-data" = templatefile("data/user-data.pkrtpl.hcl", {
build_username = var.build_username build_username = var.build_username
build_password = bcrypt(var.build_password) build_password = bcrypt(var.build_password)
build_key = var.build_key ssh_keys = concat([local.ssh_public_key], var.ssh_keys)
vm_guest_os_language = var.vm_guest_os_language vm_guest_os_language = var.vm_guest_os_language
vm_guest_os_keyboard = var.vm_guest_os_keyboard vm_guest_os_keyboard = var.vm_guest_os_keyboard
vm_guest_os_timezone = var.vm_guest_os_timezone vm_guest_os_timezone = var.vm_guest_os_timezone
@ -178,7 +183,6 @@ You'll notice that most of this is just mapping user-defined variables (with the
```text ```text
// BLOCK: source // BLOCK: source
// Defines the builder configuration blocks. // Defines the builder configuration blocks.
source "vsphere-iso" "ubuntu-k8s" { source "vsphere-iso" "ubuntu-k8s" {
// vCenter Server Endpoint Settings and Credentials // vCenter Server Endpoint Settings and Credentials
@ -232,13 +236,12 @@ source "vsphere-iso" "ubuntu-k8s" {
boot_wait = var.vm_boot_wait boot_wait = var.vm_boot_wait
boot_command = var.vm_boot_command boot_command = var.vm_boot_command
ip_wait_timeout = var.common_ip_wait_timeout ip_wait_timeout = var.common_ip_wait_timeout
shutdown_command = local.shutdown_command shutdown_command = var.vm_shutdown_command
shutdown_timeout = var.common_shutdown_timeout shutdown_timeout = var.common_shutdown_timeout
// Communicator Settings and Credentials // Communicator Settings and Credentials
communicator = "ssh" communicator = "ssh"
ssh_username = var.build_username ssh_username = var.build_username
ssh_password = var.build_password
ssh_private_key_file = local.ssh_private_key_file ssh_private_key_file = local.ssh_private_key_file
ssh_clear_authorized_keys = var.build_remove_keys ssh_clear_authorized_keys = var.build_remove_keys
ssh_port = var.communicator_port ssh_port = var.communicator_port
@ -282,7 +285,6 @@ This block brings everything together and executes the build. It calls the `sour
```text ```text
// BLOCK: build // BLOCK: build
// Defines the builders to run, provisioners, and post-processors. // Defines the builders to run, provisioners, and post-processors.
build { build {
sources = [ sources = [
"source.vsphere-iso.ubuntu-k8s" "source.vsphere-iso.ubuntu-k8s"
@ -304,9 +306,10 @@ build {
provisioner "shell" { provisioner "shell" {
execute_command = "bash {{ .Path }}" execute_command = "bash {{ .Path }}"
expect_disconnect = true
scripts = var.pre_final_scripts scripts = var.pre_final_scripts
} }
} }
``` ```
So you can see that the `ubuntu-k8s.pkr.hcl` file primarily focuses on the structure and form of the build, and it's written in such a way that it can be fairly easily adapted for building other types of VMs. Very few things in this file would have to be changed since so many of the properties are derived from the variables. So you can see that the `ubuntu-k8s.pkr.hcl` file primarily focuses on the structure and form of the build, and it's written in such a way that it can be fairly easily adapted for building other types of VMs. Very few things in this file would have to be changed since so many of the properties are derived from the variables.
@ -328,7 +331,6 @@ Most of these carry descriptions with them so I won't restate them outside of th
// Defines the input variables. // Defines the input variables.
// vSphere Credentials // vSphere Credentials
variable "vsphere_endpoint" { variable "vsphere_endpoint" {
type = string type = string
description = "The fully qualified domain name or IP address of the vCenter Server instance. ('vcenter.lab.local')" description = "The fully qualified domain name or IP address of the vCenter Server instance. ('vcenter.lab.local')"
@ -353,7 +355,6 @@ variable "vsphere_insecure_connection" {
} }
// vSphere Settings // vSphere Settings
variable "vsphere_datacenter" { variable "vsphere_datacenter" {
type = string type = string
description = "The name of the target vSphere datacenter. ('Lab Datacenter')" description = "The name of the target vSphere datacenter. ('Lab Datacenter')"
@ -380,7 +381,6 @@ variable "vsphere_folder" {
} }
// Virtual Machine Settings // Virtual Machine Settings
variable "vm_name" { variable "vm_name" {
type = string type = string
description = "Name of the new VM to create." description = "Name of the new VM to create."
@ -496,7 +496,6 @@ variable "common_remove_cdrom" {
} }
// Template and Content Library Settings // Template and Content Library Settings
variable "common_template_conversion" { variable "common_template_conversion" {
type = bool type = bool
description = "Convert the virtual machine to template. Must be 'false' for content library." description = "Convert the virtual machine to template. Must be 'false' for content library."
@ -528,7 +527,6 @@ variable "common_content_library_skip_export" {
} }
// Snapshot Settings // Snapshot Settings
variable "common_snapshot_creation" { variable "common_snapshot_creation" {
type = bool type = bool
description = "Create a snapshot for Linked Clones." description = "Create a snapshot for Linked Clones."
@ -542,7 +540,6 @@ variable "common_snapshot_name" {
} }
// OVF Export Settings // OVF Export Settings
variable "common_ovf_export_enabled" { variable "common_ovf_export_enabled" {
type = bool type = bool
description = "Enable OVF artifact export." description = "Enable OVF artifact export."
@ -561,7 +558,6 @@ variable "common_ovf_export_path" {
} }
// Removable Media Settings // Removable Media Settings
variable "common_iso_datastore" { variable "common_iso_datastore" {
type = string type = string
description = "The name of the source vSphere datastore for ISO images. ('datastore-iso-01')" description = "The name of the source vSphere datastore for ISO images. ('datastore-iso-01')"
@ -599,7 +595,6 @@ variable "cd_label" {
} }
// Boot Settings // Boot Settings
variable "vm_boot_order" { variable "vm_boot_order" {
type = string type = string
description = "The boot order for virtual machines devices. ('disk,cdrom')" description = "The boot order for virtual machines devices. ('disk,cdrom')"
@ -634,11 +629,9 @@ variable "common_shutdown_timeout" {
} }
// Communicator Settings and Credentials // Communicator Settings and Credentials
variable "build_username" { variable "build_username" {
type = string type = string
description = "The username to login to the guest operating system. ('admin')" description = "The username to login to the guest operating system. ('admin')"
sensitive = true
} }
variable "build_password" { variable "build_password" {
@ -654,10 +647,11 @@ variable "build_password_encrypted" {
default = null default = null
} }
variable "build_key" { variable "ssh_keys" {
type = string type = list(string)
description = "The public key to login to the guest operating system." description = "List of public keys to be added to ~/.ssh/authorized_keys."
sensitive = true sensitive = true
default = []
} }
variable "build_remove_keys" { variable "build_remove_keys" {
@ -667,7 +661,6 @@ variable "build_remove_keys" {
} }
// Communicator Settings // Communicator Settings
variable "communicator_port" { variable "communicator_port" {
type = string type = string
description = "The port for the communicator protocol." description = "The port for the communicator protocol."
@ -691,7 +684,6 @@ variable "communicator_ssl" {
} }
// Provisioner Settings // Provisioner Settings
variable "cloud_init_apt_packages" { variable "cloud_init_apt_packages" {
type = list(string) type = list(string)
description = "A list of apt packages to install during the subiquity cloud-init installer." description = "A list of apt packages to install during the subiquity cloud-init installer."
@ -717,7 +709,6 @@ variable "pre_final_scripts" {
} }
// Kubernetes Settings // Kubernetes Settings
variable "k8s_version" { variable "k8s_version" {
type = string type = string
description = "Kubernetes version to be installed. Latest stable is listed at https://dl.k8s.io/release/stable.txt" description = "Kubernetes version to be installed. Latest stable is listed at https://dl.k8s.io/release/stable.txt"
@ -820,16 +811,20 @@ vm_boot_command = [
``` ```
Once the installer is booted and running, Packer will wait until the VM is available via SSH and then use these credentials to log in. (How will it be able to log in with those creds? We'll take a look at the `cloud-init` configuration in just a minute...) Once the installer is booted and running, Packer will wait until the VM is available via SSH and then use these credentials to log in. (How will it be able to log in with those creds? We'll take a look at the `cloud-init` configuration in just a minute...)
```text ```text
// Communicator Settings // Communicator Settings
communicator_port = 22 communicator_port = 22
communicator_timeout = "20m" communicator_timeout = "20m"
common_ip_wait_timeout = "20m" common_ip_wait_timeout = "20m"
common_shutdown_timeout = "15m" common_shutdown_timeout = "15m"
vm_shutdown_command = "sudo /usr/sbin/shutdown -P now"
build_remove_keys = false build_remove_keys = false
build_username = "admin" build_username = "admin"
build_password = "VMware1!" build_password = "VMware1!"
build_key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOpLvpxilPjpCahAQxs4RQgv+Lb5xObULXtwEoimEBpA builder" ssh_keys = [
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOpLvpxilPjpCahAQxs4RQgv+Lb5xObULXtwEoimEBpA builder"
]
``` ```
Finally, I'll create two lists of scripts that will be run on the VM once the OS install is complete. The `post_install_scripts` will be run immediately after the operating system installation. The `update-packages.sh` script will cause a reboot, and then the set of `pre_final_scripts` will do some cleanup and prepare the VM to be converted to a template. Finally, I'll create two lists of scripts that will be run on the VM once the OS install is complete. The `post_install_scripts` will be run immediately after the operating system installation. The `update-packages.sh` script will cause a reboot, and then the set of `pre_final_scripts` will do some cleanup and prepare the VM to be converted to a template.
@ -865,7 +860,7 @@ You can find an full example of this file [here](https://github.com/jbowdre/vsph
### `user-data.pkrtpl.hcl` ### `user-data.pkrtpl.hcl`
Okay, so we've covered the Packer framework that creates the VM; now let's take a quick look at the `cloud-init` configuration that will allow the OS installation to proceed unattended. Okay, so we've covered the Packer framework that creates the VM; now let's take a quick look at the `cloud-init` configuration that will allow the OS installation to proceed unattended.
See the bits that look `${ like_this }`? Those place-holders will take input from the [`locals` block of `ubuntu-k8s.pkr.hcl`](#locals-block) mentioned above. So that's how all the OS properties will get set, including the hostname, locale, LVM partition layout, username, password, and SSH key. See the bits that look `${ like_this }`? Those place-holders will take input from the [`locals` block of `ubuntu-k8s.pkr.hcl`](#locals-block) mentioned above. So that's how all the OS properties will get set, including the hostname, locale, LVM partition layout, username, password, and SSH keys.
```yaml ```yaml
#cloud-config #cloud-config
@ -1053,9 +1048,11 @@ autoinstall:
lock-passwd: false lock-passwd: false
sudo: ALL=(ALL) NOPASSWD:ALL sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash shell: /bin/bash
%{ if length( build_key ) > 0 ~} %{ if length( ssh_keys ) > 0 ~}
ssh_authorized_keys: ssh_authorized_keys:
- ${ build_key } %{ for ssh_key in ssh_keys ~}
- ${ ssh_key }
%{ endfor ~}
%{ endif ~} %{ endif ~}
``` ```
@ -1196,7 +1193,7 @@ Next I'll install the Kubernetes components and (crucially) `apt-mark hold` them
echo ".. install kubernetes version ${KUBEVERSION}" echo ".. install kubernetes version ${KUBEVERSION}"
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update && sudo apt-get install -y kubelet=${KUBEVERSION}-00 kubeadm=${KUBEVERSION}-00 kubectl=${KUBEVERSION}-00 sudo apt-get update && sudo apt-get install -y kubelet="${KUBEVERSION}"-00 kubeadm="${KUBEVERSION}"-00 kubectl="${KUBEVERSION}"-00
sudo apt-mark hold kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl
``` ```
@ -1240,7 +1237,7 @@ sudo sh -c 'rm -f /EMPTY; sync; sleep 1; sync'
``` ```
#### `generalize.sh` #### `generalize.sh`
Lastly, let's do a final run of cleaning up logs, temporary files, and unique identifiers that don't need to exist in a template: Lastly, let's do a final run of cleaning up logs, temporary files, and unique identifiers that don't need to exist in a template. This script will also remove the SSH key with the `packer_key` identifier since that won't be needed anymore.
```shell ```shell
#!/bin/bash -eu #!/bin/bash -eu
# Prepare a VM to become a template. # Prepare a VM to become a template.
@ -1274,6 +1271,9 @@ sudo rm -rf /var/tmp/*
echo '>> Clearing host keys...' echo '>> Clearing host keys...'
sudo rm -f /etc/ssh/ssh_host_* sudo rm -f /etc/ssh/ssh_host_*
echo '>> Removing Packer SSH key...'
sed -i '/packer_key/d' ~/.ssh/authorized_keys
echo '>> Clearing machine-id...' echo '>> Clearing machine-id...'
sudo truncate -s 0 /etc/machine-id sudo truncate -s 0 /etc/machine-id
if [ -f /var/lib/dbus/machine-id ]; then if [ -f /var/lib/dbus/machine-id ]; then