Compare commits

..

No commits in common. "c1961daab83446d8c52446e1f626a5783b24fee8" and "3a74c96264ffd504ad66273ab2015f3adb8fa7b9" have entirely different histories.

7 changed files with 40 additions and 65 deletions

View file

@ -1,4 +1,4 @@
# Content Library Rsync (CLR) # library-syncer
This project aims to ease some of the pains encountered when attempting to sync VM templates in a [VMware vSphere Content Library](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-254B2CE8-20A8-43F0-90E8-3F6776C2C896.html) to a large number of geographically-remote sites under less-than-ideal networking conditions. This project aims to ease some of the pains encountered when attempting to sync VM templates in a [VMware vSphere Content Library](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-254B2CE8-20A8-43F0-90E8-3F6776C2C896.html) to a large number of geographically-remote sites under less-than-ideal networking conditions.
@ -22,25 +22,25 @@ The rough architecture looks something like this:
| | vSphere | | | vSphere |
| +----------------+ | +--------------------+ | | +----------------+ | +--------------------+ |
| | | | | | | | | | | | | |
| | CLR | | | subscribed content | | | | library-syncer | | | subscribed content | |
+--+--->| +--+-->| | | +--+--->| +--+-->| | |
| | | client | | | library | | | | | client | | | library | |
| | | | | | | | | | | | | | | |
| | +----------------+ | +--------------------+ | | | +----------------+ | +--------------------+ |
| | | | | | | |
+-----------------+ | | +----------------+ | +--------------------+ | +-----------------+ | | +----------------+ | +--------------------+ |
| | | | | | | | | | | | | | | | | | | |
| CLR | | | | CLR | | | subscribed content | | | library-syncer | | | | library-syncer | | | subscribed content | |
| +--+--+--->| +--+-->| | | | +--+--+--->| +--+-->| | |
| server | | | | client | | | library | | | server | | | | client | | | library | |
| | | | | | | | | | | | | | | | | | | |
+-----------------+ | | +----------------+ | +--------------------+ | +-----------------+ | | +----------------+ | +--------------------+ |
| | | | | | | |
| | +----------------+ | +--------------------+ | | | +----------------+ | +--------------------+ |
| | | | | | | | | | | | | | | |
| | | CLR | | | subscribed content | | | | | library-syncer | | | subscribed content | |
+--+--->| +--+-->| | | +--+--->| +--+-->| | |
| | client | | | library | | | | client | | | library | |
| | | | | | | | | | | | | |
| +----------------+ | +--------------------+ | | +----------------+ | +--------------------+ |
| +----------------------------+ | +----------------------------+
@ -97,7 +97,7 @@ EOF
#### Preparation #### Preparation
VM templates should be stored on the Docker host in its own folder under the `./data/library/` path. These should be in OVF format, _not_ OVA format, so that they can be made available in the vSphere inventory on the remote side. VM templates should be stored on the Docker host in its own folder under the `./data/library/` path. These should be in OVF format, _not_ OVA format, so that they can be made available in the vSphere inventory on the remote side.
(For extra credit, you can export the `./data/library/` path as an NFS share and mount that as a datastore in vSphere. This would make it an easy target for a CI/CD pipeline to crank out new/updated templates on a regular schedule, and those would then be automatically available to the CLR clients without any additional effort. If you do this, you'll want to set the NFS `anonuid` option in `/etc/exports` to match the `syncer` UID to control how the permissions get squashed. *Just a thought.*) (For extra credit, you can export the `./data/library/` path as an NFS share and mount that as a datastore in vSphere. This would make it an easy target for a CI/CD pipeline to crank out new/updated templates on a regular schedule, and those would then be automatically available to the `library-syncer` clients without any additional effort. If you do this, you'll want to set the NFS `anonuid` option in `/etc/exports` to match the `syncer` UID to control how the permissions get squashed. *Just a thought.*)
The server also needs the `id_syncer.pub` public key which was [generated earlier](#ssh-keypair-for-rsync-user). Place it in the `./data/ssh/` folder. The server also needs the `id_syncer.pub` public key which was [generated earlier](#ssh-keypair-for-rsync-user). Place it in the `./data/ssh/` folder.
@ -131,9 +131,9 @@ Here's an example `docker-compose.yaml` for the server:
version: '3' version: '3'
services: services:
library-syncer-server: library-syncer-server:
container_name: clr-server container_name: library-syncer-server
restart: unless-stopped restart: unless-stopped
image: ghcr.io/jbowdre/clr-server:latest image: ghcr.io/jbowdre/library-syncer-server:latest
environment: environment:
- TZ=UTC - TZ=UTC
- SYNCER_UID=31337 - SYNCER_UID=31337
@ -182,13 +182,13 @@ Some decisions need to be made on the client side, and most of those will be exp
| Variable | Example value (default)| Description | | Variable | Example value (default)| Description |
|:--- |:--- |:--- | |:--- |:--- |:--- |
| `TZ` | `America/Chicago` (`UTC`) | corresponding [TZ database name](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for the location to ensure sync schedule happens in local time | | `TZ` | `America/Chicago` (`UTC`) | corresponding [TZ database name](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for the location to ensure sync schedule happens in local time |
| `SYNC_PEER` | `clr-server.example.com` | FQDN or IP of the CLR server to which the client will connect | | `SYNC_PEER` | `deb01.lab.bowdre.net` | FQDN or IP of the `library-syncer` server to which the client will connect |
| `SYNC_PORT` | (`2222`)| SSH port for connecting to the server | | `SYNC_PORT` | (`2222`)| SSH port for connecting to the server |
| `SYNC_SCHEDULE` | (`0 21 * * 5`) | `cron`-formatted schedule for when the client should initiate a sync (example syncs at 9PM on Friday night) | | `SYNC_SCHEDULE` | (`0 21 * * 5`) | `cron`-formatted schedule for when the client should initiate a sync (example syncs at 9PM on Friday night) |
| `SYNC_DELAY` | `true` (`false`) | if true, sleeps a random number of seconds before begining the sync | | `SYNC_DELAY` | `true` (`false`) | if true, sleeps a random number of seconds before begining the sync |
| `SYNC_DELAY_MAX_SECONDS` | (`21600`) | maximum seconds to sleep (example will be delayed up to 6 hours) | | `SYNC_DELAY_MAX_SECONDS` | (`21600`) | maximum seconds to sleep (example will be delayed up to 6 hours) |
| `SYNC_MAX_BW` | `1.5m` (`0`) | `rsync` bandwidth limit; `1.5m` caps at 1.5MB/s, `0` is unlimited | | `SYNC_MAX_BW` | `1.5m` (`0`) | `rsync` bandwidth limit; `1.5m` caps at 1.5MB/s, `0` is unlimited |
| `TLS_NAME` | `library.example.com` | if set, the FQDN used for the client's web server; if not set, the library will be served strictly over HTTP | | `TLS_NAME` | `library.bowdre.net` | if set, the FQDN used for the client's web server; if not set, the library will be served strictly over HTTP |
| `TLS_CUSTOM_CERT` | `true` (`false`) | if `true`, the web server will expect to find a custom certificate *and private key* in the `./data/certs` volume | | `TLS_CUSTOM_CERT` | `true` (`false`) | if `true`, the web server will expect to find a custom certificate *and private key* in the `./data/certs` volume |
| `LIBRARY_NAME` | (`Library`) | this name will show up in the generated Content Library JSON, but not anywhere else | | `LIBRARY_NAME` | (`Library`) | this name will show up in the generated Content Library JSON, but not anywhere else |
| `LIBRARY_BROWSE` | `true` (`false`) | enable directory browsing on the web server; otherwise you'll need to know the exact path of the item you're after | | `LIBRARY_BROWSE` | `true` (`false`) | enable directory browsing on the web server; otherwise you'll need to know the exact path of the item you're after |
@ -202,18 +202,18 @@ Here's a sample `docker-compose.yaml` for the client:
version: '3' version: '3'
services: services:
library-syncer-client: library-syncer-client:
container_name: clr-client container_name: library-syncer-client
restart: unless-stopped restart: unless-stopped
image: ghcr.io/jbowdre/clr-client:latest image: ghcr.io/jbowdre/library-syncer-client:latest
environment: environment:
- TZ=America/Chicago - TZ=America/Chicago
- SYNC_PEER=clr-server.example.com - SYNC_PEER=deb01.lab.bowdre.net
- SYNC_PORT=2222 - SYNC_PORT=2222
- SYNC_SCHEDULE=0 21 * * 5 - SYNC_SCHEDULE=0 21 * * 5
- SYNC_DELAY=true - SYNC_DELAY=true
- SYNC_DELAY_MAX_SECONDS=21600 - SYNC_DELAY_MAX_SECONDS=21600
- SYNC_MAX_KBPS=0 - SYNC_MAX_KBPS=0
- TLS_NAME=library.example.com - TLS_NAME=library.lab.bowdre.net
- TLS_CUSTOM_CERT=true - TLS_CUSTOM_CERT=true
- LIBRARY_NAME=Library - LIBRARY_NAME=Library
- LIBRARY_BROWSE=true - LIBRARY_BROWSE=true
@ -241,7 +241,7 @@ Watch the logs to see how it's going:
; docker logs library-syncer-client ; docker logs library-syncer-client
[2022/08/07-02:53:23] Performing initial sync... [2022/08/07-02:53:23] Performing initial sync...
[2022/08/07-02:53:23] Sync sync starts NOW! [2022/08/07-02:53:23] Sync sync starts NOW!
Warning: Permanently added '[clr-server.example.com]:2222' (RSA) to the list of known hosts. Warning: Permanently added '[deb01.lab.bowdre.net]:2222' (RSA) to the list of known hosts.
receiving incremental file list receiving incremental file list
./ ./
Harbor/ Harbor/
@ -292,11 +292,11 @@ Or hit the site root if `LIBRARY_BROWSE` is enabled:
![Directory browsing](res/browse.png) ![Directory browsing](res/browse.png)
### Subscribed library ### Subscribed library
The final piece of this puzzle to create a content library inside of vSphere to subscribe to the CLR client library. This will (finally) make those templates available to deploy directly in vSphere. The final piece of this puzzle to create a content library inside of vSphere to subscribe to the `library-syncer-client` library. This will (finally) make those templates available to deploy directly in vSphere.
1. Log into the vSphere Client and navigate to **Menu > Content Libraries**. 1. Log into the vSphere Client and navigate to **Menu > Content Libraries**.
2. Click **Create**, give your new library a good name, and click **Next**. 2. Click **Create**, give your new library a good name, and click **Next**.
3. Click the button to make this a **Subscribed Content Library**, and enter the URL of the CLR client library. The URL should end with `/lib.json`. 3. Click the button to make this a **Subscribed Content Library**, and enter the URL of the `library-syncer-client` library. The URL should end with `/lib.json`.
![Library URL](res/library-url.png) ![Library URL](res/library-url.png)
4. Select the option to download content immediately. At this point, content will just be transferred within a local site so bandwidth shouldn't be a concern. Click **Next**. 4. Select the option to download content immediately. At this point, content will just be transferred within a local site so bandwidth shouldn't be a concern. Click **Next**.
5. From this point, it's creating a library as usual. Click **Next** again unless you want to set a specific security policy, then select the datastore where the vSphere copy of the templates should be stored, then finally hit **Finish** to complete. 5. From this point, it's creating a library as usual. Click **Next** again unless you want to set a specific security policy, then select the datastore where the vSphere copy of the templates should be stored, then finally hit **Finish** to complete.

View file

@ -1,6 +1,6 @@
FROM alpine:3.16 FROM alpine:3.16
LABEL org.opencontainers.image.source="https://github.com/jbowdre/content-library-rsync" LABEL org.opencontainers.image.source="https://github.com/jbowdre/library-syncer"
ENV CRONTAB_FILE=/var/spool/cron/crontabs/root ENV CRONTAB_FILE=/var/spool/cron/crontabs/root

View file

@ -1,25 +0,0 @@
BSD 2-Clause License
Copyright (c) 2022, William Lam
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View file

@ -1,18 +1,18 @@
version: '3' version: '3'
services: services:
library-syncer-client: library-syncer-client:
container_name: clr-client container_name: library-syncer-client
restart: unless-stopped restart: unless-stopped
image: ghcr.io/jbowdre/clr-client:latest image: ghcr.io/jbowdre/library-syncer-client:latest
environment: environment:
- TZ=UTC - TZ=UTC
- SYNC_PEER=clr-server.example.com - SYNC_PEER=deb01.lab.bowdre.net
- SYNC_PORT=2222 - SYNC_PORT=2222
- SYNC_SCHEDULE=0 21 * * 5 - SYNC_SCHEDULE=0 21 * * 5
- SYNC_DELAY=true - SYNC_DELAY=true
- SYNC_DELAY_MAX_SECONDS=21600 - SYNC_DELAY_MAX_SECONDS=21600
- SYNC_MAX_BW=0 - SYNC_MAX_BW=0
- TLS_NAME=library.example.com - TLS_NAME=library.lab.bowdre.net
- TLS_CUSTOM_CERT=true - TLS_CUSTOM_CERT=true
- LIBRARY_NAME=Library - LIBRARY_NAME=Library
- LIBRARY_BROWSE=true - LIBRARY_BROWSE=true

View file

@ -1,6 +1,6 @@
FROM alpine:3.16 FROM alpine:3.16
LABEL org.opencontainers.image.source="https://github.com/jbowdre/content-library-rsync" LABEL org.opencontainers.image.source="https://github.com/jbowdre/library-syncer"
ENV SYNC_CMD='command="/usr/bin/rrsync -ro /syncer/library/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding' ENV SYNC_CMD='command="/usr/bin/rrsync -ro /syncer/library/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding'

View file

@ -1,9 +1,9 @@
version: '3' version: '3'
services: services:
library-syncer-server: library-syncer-server:
container_name: clr-server container_name: library-syncer-server
restart: unless-stopped restart: unless-stopped
image: ghcr.io/jbowdre/clr-server:latest image: ghcr.io/jbowdre/library-syncer-server:latest
environment: environment:
- TZ=UTC - TZ=UTC
- SYNCER_UID=31337 - SYNCER_UID=31337