john/library-syncer
1
0
Fork 0
mirror of https://github.com/jbowdre/library-syncer.git synced 2024-11-23 10:12:18 +00:00
Code Issues Projects Releases Packages Wiki Activity

rebrand to "content-library-rsync"

Browse source
This commit is contained in:
John Bowdre 2023-02-21 09:46:47 -06:00
parent f6c7535e52
commit c1961daab8
6 changed files with 40 additions and 40 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

56
README.md
View file

@ -1,6 +1,6 @@
# library-syncer
# Content Library Rsync (CLR)
This project aims to ease some of the pains encountered when attempting to sync VM templates in a [VMware vSphere Content Library](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-254B2CE8-20A8-43F0-90E8-3F6776C2C896.html) to a large number of geographically-remote sites under less-than-ideal networking conditions.
This project aims to ease some of the pains encountered when attempting to sync VM templates in a [VMware vSphere Content Library](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-254B2CE8-20A8-43F0-90E8-3F6776C2C896.html) to a large number of geographically-remote sites under less-than-ideal networking conditions.
## Advantages
- Always have the latest templates at every remote site
@ -9,7 +9,7 @@ This project aims to ease some of the pains encountered when attempting to sync
- Per-site bandwidth limits to avoid saturating small pipes
## Overview
The solution leverages lightweight Docker containers in server and client roles. The server(s) would be deployed at the primary datacenter(s), and the clients at the remote sites. The servers make a specified library folder available for the clients to periodically synchronize using `rsync` over SSH, which allows for delta syncs so that bandwidth isn't wasted transferring large VMDK files when only small portions have changed.
The solution leverages lightweight Docker containers in server and client roles. The server(s) would be deployed at the primary datacenter(s), and the clients at the remote sites. The servers make a specified library folder available for the clients to periodically synchronize using `rsync` over SSH, which allows for delta syncs so that bandwidth isn't wasted transferring large VMDK files when only small portions have changed.
Once the sync has completed, each client runs a [Python script](client/build/update_library_manifests.py) to generate/update a Content Library JSON manifest which is then published over HTTP/HTTPS (courtesy of [Caddy](https://caddyserver.com/)). Traditional Content Libraries at the local site can connect to this as a [subscribed library](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-9DE2BD8F-E499-4F1E-956B-67212DE593C6.html) to make the synced items available within vSphere.
@ -22,25 +22,25 @@ The rough architecture looks something like this:
| | vSphere |
| +----------------+ | +--------------------+ |
| | | | | | |
| | library-syncer | | | subscribed content | |
| | CLR | | | subscribed content | |
+--+--->| +--+-->| | |
| | | client | | | library | |
| | | client | | | library | |
| | | | | | | |
| | +----------------+ | +--------------------+ |
| | | |
+-----------------+ | | +----------------+ | +--------------------+ |
| | | | | | | | | |
| library-syncer | | | | library-syncer | | | subscribed content | |
| CLR | | | | CLR | | | subscribed content | |
| +--+--+--->| +--+-->| | |
| server | | | | client | | | library | |
| server | | | | client | | | library | |
| | | | | | | | | |
+-----------------+ | | +----------------+ | +--------------------+ |
| | | |
| | +----------------+ | +--------------------+ |
| | | | | | | |
| | | library-syncer | | | subscribed content | |
| | | CLR | | | subscribed content | |
+--+--->| +--+-->| | |
| | client | | | library | |
| | client | | | library | |
| | | | | | |
| +----------------+ | +--------------------+ |
| +----------------------------+
@ -48,7 +48,7 @@ The rough architecture looks something like this:
## Prerequisites
### Docker and `docker-compose`
You'll need Docker (and preferably also `docker-compose`) on both the server and client VMs. Installing and configuring Docker is beyond the scope of this document as it will largely depend on what operating system you settle on for the Docker hosts.
You'll need Docker (and preferably also `docker-compose`) on both the server and client VMs. Installing and configuring Docker is beyond the scope of this document as it will largely depend on what operating system you settle on for the Docker hosts.
### SSH keypair for `rsync` user
The server image includes a `syncer` user account which the clients will use to authenticate over SSH. This account is locked down and restricted with `rrsync` to only be able to run `rsync` commands. All that you need to do is generate a keypair for the account to use:
@ -95,9 +95,9 @@ EOF
## Usage
### Server
#### Preparation
VM templates should be stored on the Docker host in its own folder under the `./data/library/` path. These should be in OVF format, _not_ OVA format, so that they can be made available in the vSphere inventory on the remote side.
VM templates should be stored on the Docker host in its own folder under the `./data/library/` path. These should be in OVF format, _not_ OVA format, so that they can be made available in the vSphere inventory on the remote side.
(For extra credit, you can export the `./data/library/` path as an NFS share and mount that as a datastore in vSphere. This would make it an easy target for a CI/CD pipeline to crank out new/updated templates on a regular schedule, and those would then be automatically available to the `library-syncer` clients without any additional effort. If you do this, you'll want to set the NFS `anonuid` option in `/etc/exports` to match the `syncer` UID to control how the permissions get squashed. *Just a thought.*)
(For extra credit, you can export the `./data/library/` path as an NFS share and mount that as a datastore in vSphere. This would make it an easy target for a CI/CD pipeline to crank out new/updated templates on a regular schedule, and those would then be automatically available to the CLR clients without any additional effort. If you do this, you'll want to set the NFS `anonuid` option in `/etc/exports` to match the `syncer` UID to control how the permissions get squashed. *Just a thought.*)
The server also needs the `id_syncer.pub` public key which was [generated earlier](#ssh-keypair-for-rsync-user). Place it in the `./data/ssh/` folder.
@ -108,17 +108,17 @@ Example folder structure:
│   ├── library
| | ├── Template_1
| | | ├── template_1.ovf
| | | └── template_1.vmdk
| | | └── template_1.vmdk
| | ├── Template_2
| | | ├── template_2.ovf
| | | └── template_2.vmdk
| | | └── template_2.vmdk
│   └── ssh
│   └── id_syncer.pub
└── docker-compose.yaml
```
#### Configuration
Strangely enough, the server side is a lot easier to configure than the client. The container just needs two volumes (one to hold the SSH key, and the other to hold the library content), and one network port on which to listen for incoming `rsync`-over-SSH connections from the clients.
Strangely enough, the server side is a lot easier to configure than the client. The container just needs two volumes (one to hold the SSH key, and the other to hold the library content), and one network port on which to listen for incoming `rsync`-over-SSH connections from the clients.
By default, the `syncer` account will be created with UID `31337`. You can override this with `SYNCER_UID` if you need to match an existing permission set.
@ -131,9 +131,9 @@ Here's an example `docker-compose.yaml` for the server:
version: '3'
services:
library-syncer-server:
container_name: library-syncer-server
container_name: clr-server
restart: unless-stopped
image: ghcr.io/jbowdre/library-syncer-server:latest
image: ghcr.io/jbowdre/clr-server:latest
environment:
- TZ=UTC
- SYNCER_UID=31337
@ -182,38 +182,38 @@ Some decisions need to be made on the client side, and most of those will be exp
| Variable | Example value (default)| Description |
|:--- |:--- |:--- |
| `TZ` | `America/Chicago` (`UTC`) | corresponding [TZ database name](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for the location to ensure sync schedule happens in local time |
| `SYNC_PEER` | `deb01.lab.bowdre.net` | FQDN or IP of the `library-syncer` server to which the client will connect |
| `SYNC_PEER` | `clr-server.example.com` | FQDN or IP of the CLR server to which the client will connect |
| `SYNC_PORT` | (`2222`)| SSH port for connecting to the server |
| `SYNC_SCHEDULE` | (`0 21 * * 5`) | `cron`-formatted schedule for when the client should initiate a sync (example syncs at 9PM on Friday night) |
| `SYNC_DELAY` | `true` (`false`) | if true, sleeps a random number of seconds before begining the sync |
| `SYNC_DELAY_MAX_SECONDS` | (`21600`) | maximum seconds to sleep (example will be delayed up to 6 hours) |
| `SYNC_MAX_BW` | `1.5m` (`0`) | `rsync` bandwidth limit; `1.5m` caps at 1.5MB/s, `0` is unlimited |
| `TLS_NAME` | `library.bowdre.net` | if set, the FQDN used for the client's web server; if not set, the library will be served strictly over HTTP |
| `SYNC_MAX_BW` | `1.5m` (`0`) | `rsync` bandwidth limit; `1.5m` caps at 1.5MB/s, `0` is unlimited |
| `TLS_NAME` | `library.example.com` | if set, the FQDN used for the client's web server; if not set, the library will be served strictly over HTTP |
| `TLS_CUSTOM_CERT` | `true` (`false`) | if `true`, the web server will expect to find a custom certificate *and private key* in the `./data/certs` volume |
| `LIBRARY_NAME` | (`Library`) | this name will show up in the generated Content Library JSON, but not anywhere else |
| `LIBRARY_BROWSE` | `true` (`false`) | enable directory browsing on the web server; otherwise you'll need to know the exact path of the item you're after |
Introducing a random sync delay might be useful if you have a bunch of remote sites and don't want them to attempt to sync all at once, but you're too lazy to manually customize the schedule for each one of them (no judgment!).
If you specify a `TLS_NAME` but don't set `TLS_CUSTOM_CERT`, the Caddy web server will automatically request and install a Let's Encrypt certificate for your specified FQDN. For this to work, the name must resolve in public DNS, and any firewalls must permit inbound traffic on port 80. Otherwise, the ACME validation will fail and you'll need to go back and try the `TLS_CUSTOM_CERT` route instead.
If you specify a `TLS_NAME` but don't set `TLS_CUSTOM_CERT`, the Caddy web server will automatically request and install a Let's Encrypt certificate for your specified FQDN. For this to work, the name must resolve in public DNS, and any firewalls must permit inbound traffic on port 80. Otherwise, the ACME validation will fail and you'll need to go back and try the `TLS_CUSTOM_CERT` route instead.
Here's a sample `docker-compose.yaml` for the client:
```yaml
version: '3'
services:
library-syncer-client:
container_name: library-syncer-client
container_name: clr-client
restart: unless-stopped
image: ghcr.io/jbowdre/library-syncer-client:latest
image: ghcr.io/jbowdre/clr-client:latest
environment:
- TZ=America/Chicago
- SYNC_PEER=deb01.lab.bowdre.net
- SYNC_PEER=clr-server.example.com
- SYNC_PORT=2222
- SYNC_SCHEDULE=0 21 * * 5
- SYNC_DELAY=true
- SYNC_DELAY_MAX_SECONDS=21600
- SYNC_MAX_KBPS=0
- TLS_NAME=library.lab.bowdre.net
- TLS_NAME=library.example.com
- TLS_CUSTOM_CERT=true
- LIBRARY_NAME=Library
- LIBRARY_BROWSE=true
@ -241,7 +241,7 @@ Watch the logs to see how it's going:
; docker logs library-syncer-client
[2022/08/07-02:53:23] Performing initial sync...
[2022/08/07-02:53:23] Sync sync starts NOW!
Warning: Permanently added '[deb01.lab.bowdre.net]:2222' (RSA) to the list of known hosts.
Warning: Permanently added '[clr-server.example.com]:2222' (RSA) to the list of known hosts.
receiving incremental file list
./
Harbor/
@ -292,11 +292,11 @@ Or hit the site root if `LIBRARY_BROWSE` is enabled:
![Directory browsing](res/browse.png)
### Subscribed library
The final piece of this puzzle to create a content library inside of vSphere to subscribe to the `library-syncer-client` library. This will (finally) make those templates available to deploy directly in vSphere.
The final piece of this puzzle to create a content library inside of vSphere to subscribe to the CLR client library. This will (finally) make those templates available to deploy directly in vSphere.
1. Log into the vSphere Client and navigate to **Menu > Content Libraries**.
2. Click **Create**, give your new library a good name, and click **Next**.
3. Click the button to make this a **Subscribed Content Library**, and enter the URL of the `library-syncer-client` library. The URL should end with `/lib.json`.
3. Click the button to make this a **Subscribed Content Library**, and enter the URL of the CLR client library. The URL should end with `/lib.json`.
![Library URL](res/library-url.png)
4. Select the option to download content immediately. At this point, content will just be transferred within a local site so bandwidth shouldn't be a concern. Click **Next**.
5. From this point, it's creating a library as usual. Click **Next** again unless you want to set a specific security policy, then select the datastore where the vSphere copy of the templates should be stored, then finally hit **Finish** to complete.

4
client/build/Dockerfile
View file

@ -1,6 +1,6 @@
FROM alpine:3.16
LABEL org.opencontainers.image.source="https://github.com/jbowdre/library-syncer"
LABEL org.opencontainers.image.source="https://github.com/jbowdre/content-library-rsync"
ENV CRONTAB_FILE=/var/spool/cron/crontabs/root
@ -22,7 +22,7 @@ COPY ./entrypoint.sh /
RUN chmod +x /entrypoint.sh \
&& chmod +x /syncer/sync.sh \
&& rm -rf /tmp/*
&& rm -rf /tmp/*
VOLUME ["/syncer/library", "/syncer/.ssh"]

6
client/build/update_library_manifests.py
View file

@ -79,7 +79,7 @@ def _make_item(directory, vcsp_type, name, files, description="", properties={},
item_id = "urn:uuid:%s" % identifier
else:
item_id = identifier
type_metadata = None
type_metadata = None
if vcsp_type == VCSP_TYPE_OVF:
# generate sample type metadata for OVF template so that subscriber can show OVF VM type
type_metadata_value = "{'id':'%s','version':'%s','libraryIdParent':'%s','isVappTemplate':'%s','vmTemplate':null,'vappTemplate':null,'networks':[],'storagePolicyGroups':null}" % (item_id, str(version), library_id, is_vapp_template)
@ -114,7 +114,7 @@ def _make_item(directory, vcsp_type, name, files, description="", properties={},
"properties": properties,
"selfHref": "%s/%s" % (urllib.parse.quote(directory), urllib.parse.quote(ITEM_FILE)),
"type": vcsp_type
}
}
def _make_items(items, version=1):
@ -287,7 +287,7 @@ def usage():
'''
The usage message for the argument parser.
'''
return """Usage: python update_library_manifests.py -n <library-name> -p <library-storage-path> --etag <true or false, default true>
return """Usage: python update_library_manifests.py -n <library-name> -p <library-storage-path> --etag <true or false, default true>
--skip-cert <true or fale, default true>
"""

8
client/docker-compose.yaml
View file

@ -1,18 +1,18 @@
version: '3'
services:
library-syncer-client:
container_name: library-syncer-client
container_name: clr-client
restart: unless-stopped
image: ghcr.io/jbowdre/library-syncer-client:latest
image: ghcr.io/jbowdre/clr-client:latest
environment:
- TZ=UTC
- SYNC_PEER=deb01.lab.bowdre.net
- SYNC_PEER=clr-server.example.com
- SYNC_PORT=2222
- SYNC_SCHEDULE=0 21 * * 5
- SYNC_DELAY=true
- SYNC_DELAY_MAX_SECONDS=21600
- SYNC_MAX_BW=0
- TLS_NAME=library.lab.bowdre.net
- TLS_NAME=library.example.com
- TLS_CUSTOM_CERT=true
- LIBRARY_NAME=Library
- LIBRARY_BROWSE=true

2
server/build/Dockerfile
View file

@ -1,6 +1,6 @@
FROM alpine:3.16
LABEL org.opencontainers.image.source="https://github.com/jbowdre/library-syncer"
LABEL org.opencontainers.image.source="https://github.com/jbowdre/content-library-rsync"
ENV SYNC_CMD='command="/usr/bin/rrsync -ro /syncer/library/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding'

4
server/docker-compose.yaml
View file

@ -1,9 +1,9 @@
version: '3'
services:
library-syncer-server:
container_name: library-syncer-server
container_name: clr-server
restart: unless-stopped
image: ghcr.io/jbowdre/library-syncer-server:latest
image: ghcr.io/jbowdre/clr-server:latest
environment:
- TZ=UTC
- SYNCER_UID=31337