|
|
@ -1,4 +1,4 @@
|
|
|
|
# Content Library Rsync (CLR)
|
|
|
|
# library-syncer
|
|
|
|
|
|
|
|
|
|
|
|
This project aims to ease some of the pains encountered when attempting to sync VM templates in a [VMware vSphere Content Library](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-254B2CE8-20A8-43F0-90E8-3F6776C2C896.html) to a large number of geographically-remote sites under less-than-ideal networking conditions.
|
|
|
|
This project aims to ease some of the pains encountered when attempting to sync VM templates in a [VMware vSphere Content Library](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-254B2CE8-20A8-43F0-90E8-3F6776C2C896.html) to a large number of geographically-remote sites under less-than-ideal networking conditions.
|
|
|
|
|
|
|
|
|
|
|
@ -22,25 +22,25 @@ The rough architecture looks something like this:
|
|
|
|
| | vSphere |
|
|
|
|
| | vSphere |
|
|
|
|
| +----------------+ | +--------------------+ |
|
|
|
|
| +----------------+ | +--------------------+ |
|
|
|
|
| | | | | | |
|
|
|
|
| | | | | | |
|
|
|
|
| | CLR | | | subscribed content | |
|
|
|
|
| | library-syncer | | | subscribed content | |
|
|
|
|
+--+--->| +--+-->| | |
|
|
|
|
+--+--->| +--+-->| | |
|
|
|
|
| | | client | | | library | |
|
|
|
|
| | | client | | | library | |
|
|
|
|
| | | | | | | |
|
|
|
|
| | | | | | | |
|
|
|
|
| | +----------------+ | +--------------------+ |
|
|
|
|
| | +----------------+ | +--------------------+ |
|
|
|
|
| | | |
|
|
|
|
| | | |
|
|
|
|
+-----------------+ | | +----------------+ | +--------------------+ |
|
|
|
|
+-----------------+ | | +----------------+ | +--------------------+ |
|
|
|
|
| | | | | | | | | |
|
|
|
|
| | | | | | | | | |
|
|
|
|
| CLR | | | | CLR | | | subscribed content | |
|
|
|
|
| library-syncer | | | | library-syncer | | | subscribed content | |
|
|
|
|
| +--+--+--->| +--+-->| | |
|
|
|
|
| +--+--+--->| +--+-->| | |
|
|
|
|
| server | | | | client | | | library | |
|
|
|
|
| server | | | | client | | | library | |
|
|
|
|
| | | | | | | | | |
|
|
|
|
| | | | | | | | | |
|
|
|
|
+-----------------+ | | +----------------+ | +--------------------+ |
|
|
|
|
+-----------------+ | | +----------------+ | +--------------------+ |
|
|
|
|
| | | |
|
|
|
|
| | | |
|
|
|
|
| | +----------------+ | +--------------------+ |
|
|
|
|
| | +----------------+ | +--------------------+ |
|
|
|
|
| | | | | | | |
|
|
|
|
| | | | | | | |
|
|
|
|
| | | CLR | | | subscribed content | |
|
|
|
|
| | | library-syncer | | | subscribed content | |
|
|
|
|
+--+--->| +--+-->| | |
|
|
|
|
+--+--->| +--+-->| | |
|
|
|
|
| | client | | | library | |
|
|
|
|
| | client | | | library | |
|
|
|
|
| | | | | | |
|
|
|
|
| | | | | | |
|
|
|
|
| +----------------+ | +--------------------+ |
|
|
|
|
| +----------------+ | +--------------------+ |
|
|
|
|
| +----------------------------+
|
|
|
|
| +----------------------------+
|
|
|
@ -97,7 +97,7 @@ EOF
|
|
|
|
#### Preparation
|
|
|
|
#### Preparation
|
|
|
|
VM templates should be stored on the Docker host in its own folder under the `./data/library/` path. These should be in OVF format, _not_ OVA format, so that they can be made available in the vSphere inventory on the remote side.
|
|
|
|
VM templates should be stored on the Docker host in its own folder under the `./data/library/` path. These should be in OVF format, _not_ OVA format, so that they can be made available in the vSphere inventory on the remote side.
|
|
|
|
|
|
|
|
|
|
|
|
(For extra credit, you can export the `./data/library/` path as an NFS share and mount that as a datastore in vSphere. This would make it an easy target for a CI/CD pipeline to crank out new/updated templates on a regular schedule, and those would then be automatically available to the CLR clients without any additional effort. If you do this, you'll want to set the NFS `anonuid` option in `/etc/exports` to match the `syncer` UID to control how the permissions get squashed. *Just a thought.*)
|
|
|
|
(For extra credit, you can export the `./data/library/` path as an NFS share and mount that as a datastore in vSphere. This would make it an easy target for a CI/CD pipeline to crank out new/updated templates on a regular schedule, and those would then be automatically available to the `library-syncer` clients without any additional effort. If you do this, you'll want to set the NFS `anonuid` option in `/etc/exports` to match the `syncer` UID to control how the permissions get squashed. *Just a thought.*)
|
|
|
|
|
|
|
|
|
|
|
|
The server also needs the `id_syncer.pub` public key which was [generated earlier](#ssh-keypair-for-rsync-user). Place it in the `./data/ssh/` folder.
|
|
|
|
The server also needs the `id_syncer.pub` public key which was [generated earlier](#ssh-keypair-for-rsync-user). Place it in the `./data/ssh/` folder.
|
|
|
|
|
|
|
|
|
|
|
@ -131,9 +131,9 @@ Here's an example `docker-compose.yaml` for the server:
|
|
|
|
version: '3'
|
|
|
|
version: '3'
|
|
|
|
services:
|
|
|
|
services:
|
|
|
|
library-syncer-server:
|
|
|
|
library-syncer-server:
|
|
|
|
container_name: clr-server
|
|
|
|
container_name: library-syncer-server
|
|
|
|
restart: unless-stopped
|
|
|
|
restart: unless-stopped
|
|
|
|
image: ghcr.io/jbowdre/clr-server:latest
|
|
|
|
image: ghcr.io/jbowdre/library-syncer-server:latest
|
|
|
|
environment:
|
|
|
|
environment:
|
|
|
|
- TZ=UTC
|
|
|
|
- TZ=UTC
|
|
|
|
- SYNCER_UID=31337
|
|
|
|
- SYNCER_UID=31337
|
|
|
@ -182,13 +182,13 @@ Some decisions need to be made on the client side, and most of those will be exp
|
|
|
|
| Variable | Example value (default)| Description |
|
|
|
|
| Variable | Example value (default)| Description |
|
|
|
|
|:--- |:--- |:--- |
|
|
|
|
|:--- |:--- |:--- |
|
|
|
|
| `TZ` | `America/Chicago` (`UTC`) | corresponding [TZ database name](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for the location to ensure sync schedule happens in local time |
|
|
|
|
| `TZ` | `America/Chicago` (`UTC`) | corresponding [TZ database name](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for the location to ensure sync schedule happens in local time |
|
|
|
|
| `SYNC_PEER` | `clr-server.example.com` | FQDN or IP of the CLR server to which the client will connect |
|
|
|
|
| `SYNC_PEER` | `deb01.lab.bowdre.net` | FQDN or IP of the `library-syncer` server to which the client will connect |
|
|
|
|
| `SYNC_PORT` | (`2222`)| SSH port for connecting to the server |
|
|
|
|
| `SYNC_PORT` | (`2222`)| SSH port for connecting to the server |
|
|
|
|
| `SYNC_SCHEDULE` | (`0 21 * * 5`) | `cron`-formatted schedule for when the client should initiate a sync (example syncs at 9PM on Friday night) |
|
|
|
|
| `SYNC_SCHEDULE` | (`0 21 * * 5`) | `cron`-formatted schedule for when the client should initiate a sync (example syncs at 9PM on Friday night) |
|
|
|
|
| `SYNC_DELAY` | `true` (`false`) | if true, sleeps a random number of seconds before begining the sync |
|
|
|
|
| `SYNC_DELAY` | `true` (`false`) | if true, sleeps a random number of seconds before begining the sync |
|
|
|
|
| `SYNC_DELAY_MAX_SECONDS` | (`21600`) | maximum seconds to sleep (example will be delayed up to 6 hours) |
|
|
|
|
| `SYNC_DELAY_MAX_SECONDS` | (`21600`) | maximum seconds to sleep (example will be delayed up to 6 hours) |
|
|
|
|
| `SYNC_MAX_BW` | `1.5m` (`0`) | `rsync` bandwidth limit; `1.5m` caps at 1.5MB/s, `0` is unlimited |
|
|
|
|
| `SYNC_MAX_BW` | `1.5m` (`0`) | `rsync` bandwidth limit; `1.5m` caps at 1.5MB/s, `0` is unlimited |
|
|
|
|
| `TLS_NAME` | `library.example.com` | if set, the FQDN used for the client's web server; if not set, the library will be served strictly over HTTP |
|
|
|
|
| `TLS_NAME` | `library.bowdre.net` | if set, the FQDN used for the client's web server; if not set, the library will be served strictly over HTTP |
|
|
|
|
| `TLS_CUSTOM_CERT` | `true` (`false`) | if `true`, the web server will expect to find a custom certificate *and private key* in the `./data/certs` volume |
|
|
|
|
| `TLS_CUSTOM_CERT` | `true` (`false`) | if `true`, the web server will expect to find a custom certificate *and private key* in the `./data/certs` volume |
|
|
|
|
| `LIBRARY_NAME` | (`Library`) | this name will show up in the generated Content Library JSON, but not anywhere else |
|
|
|
|
| `LIBRARY_NAME` | (`Library`) | this name will show up in the generated Content Library JSON, but not anywhere else |
|
|
|
|
| `LIBRARY_BROWSE` | `true` (`false`) | enable directory browsing on the web server; otherwise you'll need to know the exact path of the item you're after |
|
|
|
|
| `LIBRARY_BROWSE` | `true` (`false`) | enable directory browsing on the web server; otherwise you'll need to know the exact path of the item you're after |
|
|
|
@ -202,18 +202,18 @@ Here's a sample `docker-compose.yaml` for the client:
|
|
|
|
version: '3'
|
|
|
|
version: '3'
|
|
|
|
services:
|
|
|
|
services:
|
|
|
|
library-syncer-client:
|
|
|
|
library-syncer-client:
|
|
|
|
container_name: clr-client
|
|
|
|
container_name: library-syncer-client
|
|
|
|
restart: unless-stopped
|
|
|
|
restart: unless-stopped
|
|
|
|
image: ghcr.io/jbowdre/clr-client:latest
|
|
|
|
image: ghcr.io/jbowdre/library-syncer-client:latest
|
|
|
|
environment:
|
|
|
|
environment:
|
|
|
|
- TZ=America/Chicago
|
|
|
|
- TZ=America/Chicago
|
|
|
|
- SYNC_PEER=clr-server.example.com
|
|
|
|
- SYNC_PEER=deb01.lab.bowdre.net
|
|
|
|
- SYNC_PORT=2222
|
|
|
|
- SYNC_PORT=2222
|
|
|
|
- SYNC_SCHEDULE=0 21 * * 5
|
|
|
|
- SYNC_SCHEDULE=0 21 * * 5
|
|
|
|
- SYNC_DELAY=true
|
|
|
|
- SYNC_DELAY=true
|
|
|
|
- SYNC_DELAY_MAX_SECONDS=21600
|
|
|
|
- SYNC_DELAY_MAX_SECONDS=21600
|
|
|
|
- SYNC_MAX_KBPS=0
|
|
|
|
- SYNC_MAX_KBPS=0
|
|
|
|
- TLS_NAME=library.example.com
|
|
|
|
- TLS_NAME=library.lab.bowdre.net
|
|
|
|
- TLS_CUSTOM_CERT=true
|
|
|
|
- TLS_CUSTOM_CERT=true
|
|
|
|
- LIBRARY_NAME=Library
|
|
|
|
- LIBRARY_NAME=Library
|
|
|
|
- LIBRARY_BROWSE=true
|
|
|
|
- LIBRARY_BROWSE=true
|
|
|
@ -241,7 +241,7 @@ Watch the logs to see how it's going:
|
|
|
|
; docker logs library-syncer-client
|
|
|
|
; docker logs library-syncer-client
|
|
|
|
[2022/08/07-02:53:23] Performing initial sync...
|
|
|
|
[2022/08/07-02:53:23] Performing initial sync...
|
|
|
|
[2022/08/07-02:53:23] Sync sync starts NOW!
|
|
|
|
[2022/08/07-02:53:23] Sync sync starts NOW!
|
|
|
|
Warning: Permanently added '[clr-server.example.com]:2222' (RSA) to the list of known hosts.
|
|
|
|
Warning: Permanently added '[deb01.lab.bowdre.net]:2222' (RSA) to the list of known hosts.
|
|
|
|
receiving incremental file list
|
|
|
|
receiving incremental file list
|
|
|
|
./
|
|
|
|
./
|
|
|
|
Harbor/
|
|
|
|
Harbor/
|
|
|
@ -292,11 +292,11 @@ Or hit the site root if `LIBRARY_BROWSE` is enabled:
|
|
|
|
![Directory browsing](res/browse.png)
|
|
|
|
![Directory browsing](res/browse.png)
|
|
|
|
|
|
|
|
|
|
|
|
### Subscribed library
|
|
|
|
### Subscribed library
|
|
|
|
The final piece of this puzzle to create a content library inside of vSphere to subscribe to the CLR client library. This will (finally) make those templates available to deploy directly in vSphere.
|
|
|
|
The final piece of this puzzle to create a content library inside of vSphere to subscribe to the `library-syncer-client` library. This will (finally) make those templates available to deploy directly in vSphere.
|
|
|
|
|
|
|
|
|
|
|
|
1. Log into the vSphere Client and navigate to **Menu > Content Libraries**.
|
|
|
|
1. Log into the vSphere Client and navigate to **Menu > Content Libraries**.
|
|
|
|
2. Click **Create**, give your new library a good name, and click **Next**.
|
|
|
|
2. Click **Create**, give your new library a good name, and click **Next**.
|
|
|
|
3. Click the button to make this a **Subscribed Content Library**, and enter the URL of the CLR client library. The URL should end with `/lib.json`.
|
|
|
|
3. Click the button to make this a **Subscribed Content Library**, and enter the URL of the `library-syncer-client` library. The URL should end with `/lib.json`.
|
|
|
|
![Library URL](res/library-url.png)
|
|
|
|
![Library URL](res/library-url.png)
|
|
|
|
4. Select the option to download content immediately. At this point, content will just be transferred within a local site so bandwidth shouldn't be a concern. Click **Next**.
|
|
|
|
4. Select the option to download content immediately. At this point, content will just be transferred within a local site so bandwidth shouldn't be a concern. Click **Next**.
|
|
|
|
5. From this point, it's creating a library as usual. Click **Next** again unless you want to set a specific security policy, then select the datastore where the vSphere copy of the templates should be stored, then finally hit **Finish** to complete.
|
|
|
|
5. From this point, it's creating a library as usual. Click **Next** again unless you want to set a specific security policy, then select the datastore where the vSphere copy of the templates should be stored, then finally hit **Finish** to complete.
|
|
|
|