mirror of
https://github.com/jbowdre/library-syncer.git
synced 2024-11-24 18:42:18 +00:00
Compare commits
2 commits
3a74c96264
...
c1961daab8
Author | SHA1 | Date | |
---|---|---|---|
c1961daab8 | |||
f6c7535e52 |
7 changed files with 65 additions and 40 deletions
38
README.md
38
README.md
|
@ -1,4 +1,4 @@
|
|||
# library-syncer
|
||||
# Content Library Rsync (CLR)
|
||||
|
||||
This project aims to ease some of the pains encountered when attempting to sync VM templates in a [VMware vSphere Content Library](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-254B2CE8-20A8-43F0-90E8-3F6776C2C896.html) to a large number of geographically-remote sites under less-than-ideal networking conditions.
|
||||
|
||||
|
@ -22,25 +22,25 @@ The rough architecture looks something like this:
|
|||
| | vSphere |
|
||||
| +----------------+ | +--------------------+ |
|
||||
| | | | | | |
|
||||
| | library-syncer | | | subscribed content | |
|
||||
| | CLR | | | subscribed content | |
|
||||
+--+--->| +--+-->| | |
|
||||
| | | client | | | library | |
|
||||
| | | client | | | library | |
|
||||
| | | | | | | |
|
||||
| | +----------------+ | +--------------------+ |
|
||||
| | | |
|
||||
+-----------------+ | | +----------------+ | +--------------------+ |
|
||||
| | | | | | | | | |
|
||||
| library-syncer | | | | library-syncer | | | subscribed content | |
|
||||
| CLR | | | | CLR | | | subscribed content | |
|
||||
| +--+--+--->| +--+-->| | |
|
||||
| server | | | | client | | | library | |
|
||||
| server | | | | client | | | library | |
|
||||
| | | | | | | | | |
|
||||
+-----------------+ | | +----------------+ | +--------------------+ |
|
||||
| | | |
|
||||
| | +----------------+ | +--------------------+ |
|
||||
| | | | | | | |
|
||||
| | | library-syncer | | | subscribed content | |
|
||||
| | | CLR | | | subscribed content | |
|
||||
+--+--->| +--+-->| | |
|
||||
| | client | | | library | |
|
||||
| | client | | | library | |
|
||||
| | | | | | |
|
||||
| +----------------+ | +--------------------+ |
|
||||
| +----------------------------+
|
||||
|
@ -97,7 +97,7 @@ EOF
|
|||
#### Preparation
|
||||
VM templates should be stored on the Docker host in its own folder under the `./data/library/` path. These should be in OVF format, _not_ OVA format, so that they can be made available in the vSphere inventory on the remote side.
|
||||
|
||||
(For extra credit, you can export the `./data/library/` path as an NFS share and mount that as a datastore in vSphere. This would make it an easy target for a CI/CD pipeline to crank out new/updated templates on a regular schedule, and those would then be automatically available to the `library-syncer` clients without any additional effort. If you do this, you'll want to set the NFS `anonuid` option in `/etc/exports` to match the `syncer` UID to control how the permissions get squashed. *Just a thought.*)
|
||||
(For extra credit, you can export the `./data/library/` path as an NFS share and mount that as a datastore in vSphere. This would make it an easy target for a CI/CD pipeline to crank out new/updated templates on a regular schedule, and those would then be automatically available to the CLR clients without any additional effort. If you do this, you'll want to set the NFS `anonuid` option in `/etc/exports` to match the `syncer` UID to control how the permissions get squashed. *Just a thought.*)
|
||||
|
||||
The server also needs the `id_syncer.pub` public key which was [generated earlier](#ssh-keypair-for-rsync-user). Place it in the `./data/ssh/` folder.
|
||||
|
||||
|
@ -131,9 +131,9 @@ Here's an example `docker-compose.yaml` for the server:
|
|||
version: '3'
|
||||
services:
|
||||
library-syncer-server:
|
||||
container_name: library-syncer-server
|
||||
container_name: clr-server
|
||||
restart: unless-stopped
|
||||
image: ghcr.io/jbowdre/library-syncer-server:latest
|
||||
image: ghcr.io/jbowdre/clr-server:latest
|
||||
environment:
|
||||
- TZ=UTC
|
||||
- SYNCER_UID=31337
|
||||
|
@ -182,13 +182,13 @@ Some decisions need to be made on the client side, and most of those will be exp
|
|||
| Variable | Example value (default)| Description |
|
||||
|:--- |:--- |:--- |
|
||||
| `TZ` | `America/Chicago` (`UTC`) | corresponding [TZ database name](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for the location to ensure sync schedule happens in local time |
|
||||
| `SYNC_PEER` | `deb01.lab.bowdre.net` | FQDN or IP of the `library-syncer` server to which the client will connect |
|
||||
| `SYNC_PEER` | `clr-server.example.com` | FQDN or IP of the CLR server to which the client will connect |
|
||||
| `SYNC_PORT` | (`2222`)| SSH port for connecting to the server |
|
||||
| `SYNC_SCHEDULE` | (`0 21 * * 5`) | `cron`-formatted schedule for when the client should initiate a sync (example syncs at 9PM on Friday night) |
|
||||
| `SYNC_DELAY` | `true` (`false`) | if true, sleeps a random number of seconds before begining the sync |
|
||||
| `SYNC_DELAY_MAX_SECONDS` | (`21600`) | maximum seconds to sleep (example will be delayed up to 6 hours) |
|
||||
| `SYNC_MAX_BW` | `1.5m` (`0`) | `rsync` bandwidth limit; `1.5m` caps at 1.5MB/s, `0` is unlimited |
|
||||
| `TLS_NAME` | `library.bowdre.net` | if set, the FQDN used for the client's web server; if not set, the library will be served strictly over HTTP |
|
||||
| `TLS_NAME` | `library.example.com` | if set, the FQDN used for the client's web server; if not set, the library will be served strictly over HTTP |
|
||||
| `TLS_CUSTOM_CERT` | `true` (`false`) | if `true`, the web server will expect to find a custom certificate *and private key* in the `./data/certs` volume |
|
||||
| `LIBRARY_NAME` | (`Library`) | this name will show up in the generated Content Library JSON, but not anywhere else |
|
||||
| `LIBRARY_BROWSE` | `true` (`false`) | enable directory browsing on the web server; otherwise you'll need to know the exact path of the item you're after |
|
||||
|
@ -202,18 +202,18 @@ Here's a sample `docker-compose.yaml` for the client:
|
|||
version: '3'
|
||||
services:
|
||||
library-syncer-client:
|
||||
container_name: library-syncer-client
|
||||
container_name: clr-client
|
||||
restart: unless-stopped
|
||||
image: ghcr.io/jbowdre/library-syncer-client:latest
|
||||
image: ghcr.io/jbowdre/clr-client:latest
|
||||
environment:
|
||||
- TZ=America/Chicago
|
||||
- SYNC_PEER=deb01.lab.bowdre.net
|
||||
- SYNC_PEER=clr-server.example.com
|
||||
- SYNC_PORT=2222
|
||||
- SYNC_SCHEDULE=0 21 * * 5
|
||||
- SYNC_DELAY=true
|
||||
- SYNC_DELAY_MAX_SECONDS=21600
|
||||
- SYNC_MAX_KBPS=0
|
||||
- TLS_NAME=library.lab.bowdre.net
|
||||
- TLS_NAME=library.example.com
|
||||
- TLS_CUSTOM_CERT=true
|
||||
- LIBRARY_NAME=Library
|
||||
- LIBRARY_BROWSE=true
|
||||
|
@ -241,7 +241,7 @@ Watch the logs to see how it's going:
|
|||
; docker logs library-syncer-client
|
||||
[2022/08/07-02:53:23] Performing initial sync...
|
||||
[2022/08/07-02:53:23] Sync sync starts NOW!
|
||||
Warning: Permanently added '[deb01.lab.bowdre.net]:2222' (RSA) to the list of known hosts.
|
||||
Warning: Permanently added '[clr-server.example.com]:2222' (RSA) to the list of known hosts.
|
||||
receiving incremental file list
|
||||
./
|
||||
Harbor/
|
||||
|
@ -292,11 +292,11 @@ Or hit the site root if `LIBRARY_BROWSE` is enabled:
|
|||
![Directory browsing](res/browse.png)
|
||||
|
||||
### Subscribed library
|
||||
The final piece of this puzzle to create a content library inside of vSphere to subscribe to the `library-syncer-client` library. This will (finally) make those templates available to deploy directly in vSphere.
|
||||
The final piece of this puzzle to create a content library inside of vSphere to subscribe to the CLR client library. This will (finally) make those templates available to deploy directly in vSphere.
|
||||
|
||||
1. Log into the vSphere Client and navigate to **Menu > Content Libraries**.
|
||||
2. Click **Create**, give your new library a good name, and click **Next**.
|
||||
3. Click the button to make this a **Subscribed Content Library**, and enter the URL of the `library-syncer-client` library. The URL should end with `/lib.json`.
|
||||
3. Click the button to make this a **Subscribed Content Library**, and enter the URL of the CLR client library. The URL should end with `/lib.json`.
|
||||
![Library URL](res/library-url.png)
|
||||
4. Select the option to download content immediately. At this point, content will just be transferred within a local site so bandwidth shouldn't be a concern. Click **Next**.
|
||||
5. From this point, it's creating a library as usual. Click **Next** again unless you want to set a specific security policy, then select the datastore where the vSphere copy of the templates should be stored, then finally hit **Finish** to complete.
|
||||
|
|
|
@ -1,6 +1,6 @@
|
|||
FROM alpine:3.16
|
||||
|
||||
LABEL org.opencontainers.image.source="https://github.com/jbowdre/library-syncer"
|
||||
LABEL org.opencontainers.image.source="https://github.com/jbowdre/content-library-rsync"
|
||||
|
||||
ENV CRONTAB_FILE=/var/spool/cron/crontabs/root
|
||||
|
||||
|
|
25
client/build/LICENSE
Normal file
25
client/build/LICENSE
Normal file
|
@ -0,0 +1,25 @@
|
|||
BSD 2-Clause License
|
||||
|
||||
Copyright (c) 2022, William Lam
|
||||
All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are met:
|
||||
|
||||
1. Redistributions of source code must retain the above copyright notice, this
|
||||
list of conditions and the following disclaimer.
|
||||
|
||||
2. Redistributions in binary form must reproduce the above copyright notice,
|
||||
this list of conditions and the following disclaimer in the documentation
|
||||
and/or other materials provided with the distribution.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
||||
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
||||
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
||||
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||||
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
||||
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
||||
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
||||
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
@ -1,18 +1,18 @@
|
|||
version: '3'
|
||||
services:
|
||||
library-syncer-client:
|
||||
container_name: library-syncer-client
|
||||
container_name: clr-client
|
||||
restart: unless-stopped
|
||||
image: ghcr.io/jbowdre/library-syncer-client:latest
|
||||
image: ghcr.io/jbowdre/clr-client:latest
|
||||
environment:
|
||||
- TZ=UTC
|
||||
- SYNC_PEER=deb01.lab.bowdre.net
|
||||
- SYNC_PEER=clr-server.example.com
|
||||
- SYNC_PORT=2222
|
||||
- SYNC_SCHEDULE=0 21 * * 5
|
||||
- SYNC_DELAY=true
|
||||
- SYNC_DELAY_MAX_SECONDS=21600
|
||||
- SYNC_MAX_BW=0
|
||||
- TLS_NAME=library.lab.bowdre.net
|
||||
- TLS_NAME=library.example.com
|
||||
- TLS_CUSTOM_CERT=true
|
||||
- LIBRARY_NAME=Library
|
||||
- LIBRARY_BROWSE=true
|
||||
|
|
|
@ -1,6 +1,6 @@
|
|||
FROM alpine:3.16
|
||||
|
||||
LABEL org.opencontainers.image.source="https://github.com/jbowdre/library-syncer"
|
||||
LABEL org.opencontainers.image.source="https://github.com/jbowdre/content-library-rsync"
|
||||
|
||||
ENV SYNC_CMD='command="/usr/bin/rrsync -ro /syncer/library/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding'
|
||||
|
||||
|
|
|
@ -1,9 +1,9 @@
|
|||
version: '3'
|
||||
services:
|
||||
library-syncer-server:
|
||||
container_name: library-syncer-server
|
||||
container_name: clr-server
|
||||
restart: unless-stopped
|
||||
image: ghcr.io/jbowdre/library-syncer-server:latest
|
||||
image: ghcr.io/jbowdre/clr-server:latest
|
||||
environment:
|
||||
- TZ=UTC
|
||||
- SYNCER_UID=31337
|
||||
|
|
Loading…
Reference in a new issue