diff --git a/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/api_user.png b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/api_user.png new file mode 100644 index 0000000..9fad192 Binary files /dev/null and b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/api_user.png differ diff --git a/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/dvportgroups.png b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/dvportgroups.png new file mode 100644 index 0000000..cec94f4 Binary files /dev/null and b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/dvportgroups.png differ diff --git a/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/index.md b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/index.md new file mode 100644 index 0000000..30add4b --- /dev/null +++ b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/index.md @@ -0,0 +1,462 @@ +--- +title: "Bulk Import vSphere dvPortGroups to phpIPAM" # Title of the blog post. +date: 2022-01-21T15:24:00-06:00 # Date of post creation. +# lastmod: 2022-01-21T15:24:00-06:00 # Date when last modified +description: "I wrote a Python script to interface with the phpIPAM API and import a large number of networks exported from vSphere for IP management." # Description used for search engine. +featured: false # Sets if post is a featured post, making appear on the home page side bar. +draft: true # Sets whether to render this page. Draft of true will not be rendered. +toc: false # Controls if a table of contents should be generated for first-level links automatically. +usePageBundles: true +# menu: main +# featureImage: "file.png" # Sets featured image on blog post. +# featureImageAlt: 'Description of image' # Alternative text for featured image. +# featureImageCap: 'This is the featured image.' # Caption (optional). +# thumbnail: "thumbnail.png" # Sets thumbnail image appearing inside card on homepage. +# shareImage: "share.png" # Designate a separate image for social media sharing. +codeLineNumbers: false # Override global value for showing of line numbers within code block. +series: Scripts +tags: + - vmware + - powercli + - python + - api + - phpipam +comment: true # Disable comment if false. +--- + +I [recently wrote](tanzu-community-edition-k8s-homelab/#a-real-workload---phpipam) about getting started with VMware's [Tanzu Community Edition](https://tanzucommunityedition.io/) and deploying [phpIPAM](https://phpipam.net/) as my first real-world Kubernetes workload. Well I've spent much of my time since then working on a script which would help to populate my phpIPAM instance with a list of networks to monitor. + +### Planning and Exporting +The first step in making this work was to figure out which networks I wanted to import. We've got hundreds of different networks in use across our production vSphere environments. I focused only on those which are portgroups on distributed virtual switches since those configurations are pretty standardized (being vCenter constructs instead of configured on individual hosts). These dvPortGroups bear a naming standard which conveys all sorts of useful information, and it's easy and safe to rename any dvPortGroups which _don't_ fit the standard (unlike renaming portgroups on a standard virtual switch). + +The standard naming convention is `[Site]-[Description] [Network Address]{/[Mask]}`. So the networks look something like this: +![Production dvPortGroups approximated in my testing lab environment](dvportgroups.png) + +Some networks have masks in the name, some don't. Most networks correctly include the network address with a `0` in the last octet, but some use an `x` instead. And the VLANs associated with the networks have a varying number of digits. These are all things that I needed to keep in mind as I worked on a solution which would make a true best effort at importing all of these. + +As long as the dvPortGroup names stick to this format I can parse the name to come up with the site/location as well as the IP space of the network. The dvPortGroup also carries information about the associated VLAN, which is useful information to have. And I can easily export this information with a simple PowerCLI query: + +```powershell +PS /home/john> get-vdportgroup | select Name, VlanConfiguration + +Name VlanConfiguration +---- ----------------- + +MGT-Home 192.168.1.0 +MGT-Servers 172.16.10.0 VLAN 1610 +BOW-Servers 172.16.20.0 VLAN 1620 +BOW-Servers 172.16.30.x VLAN 4 +BOW-Servers 172.16.40.0/26 VLAN 44 +DRE-Servers 172.16.50.0/28 VLAN 400 +DRE-Servers 172.16.60.0 VLAN 1660 +``` + +In my [homelab](/vmware-home-lab-on-intel-nuc-9/), I only have a single vCenter. In production, we've got a handful of vCenters, and each manages the hosts in a given region. So I can use information about which vCenter hosts a dvPortGroup to figure out which region a network is in. When I import this data into phpIPAM, I can use the vCenter name to assign [remote scan agents](https://github.com/jbowdre/phpipam-agent-docker) to networks based on the region that they're in. + +The vCenter can be found in the `Uid` property returned by `get-vdportgroup`: +```powershell +PS /home/john> get-vdportgroup | select Name, VlanConfiguration, UID + +Name VlanConfiguration Uid +---- ----------------- --- +MGT-Home 192.168.1.0 /VIServer=lab\john@vcsa.lab.bowdre.net:443/DistributedPortgroup=DistributedVirtualPortgroup-dvportgroup-27015/ +MGT-Servers 172.16.10.0 VLAN 1610 /VIServer=lab\john@vcsa.lab.bowdre.net:443/DistributedPortgroup=DistributedVirtualPortgroup-dvportgroup-27017/ +BOW-Servers 172.16.20.0 VLAN 1620 /VIServer=lab\john@vcsa.lab.bowdre.net:443/DistributedPortgroup=DistributedVirtualPortgroup-dvportgroup-28010/ +BOW-Servers 172.16.30.x VLAN 4 /VIServer=lab\john@vcsa.lab.bowdre.net:443/DistributedPortgroup=DistributedVirtualPortgroup-dvportgroup-28011/ +BOW-Servers 172.16.40.0/26 VLAN 44 /VIServer=lab\john@vcsa.lab.bowdre.net:443/DistributedPortgroup=DistributedVirtualPortgroup-dvportgroup-28012/ +DRE-Servers 172.16.50.0/28 VLAN 400 /VIServer=lab\john@vcsa.lab.bowdre.net:443/DistributedPortgroup=DistributedVirtualPortgroup-dvportgroup-28013/ +DRE-Servers 172.16.60.0 VLAN 1660 /VIServer=lab\john@vcsa.lab.bowdre.net:443/DistributedPortgroup=DistributedVirtualPortgroup-dvportgroup-28014/ +``` + +It's not pretty, but it'll do the trick. All that's left is to export this data into a handy-dandy CSV-formatted file that I can easily parse for import: + +```powershell +get-vdportgroup | select Name, VlanConfiguration, Uid | export-csv -NoTypeInformation ./networks.csv +``` +![My networks.csv export, including the networks which don't match the naming criteria and will be skipped by the import process.](/networks.csv.png) + +### Setting up phpIPAM +There are a few additional steps needed to enable API access to a freshly-installed phpIPAM installation. To start, I log in to my phpIPAM instance and navigate to the **Administration > Server Management > phpIPAM Settings** page, where I enabled both the *Prettify links* and *API* feature settings - making sure to hit the **Save** button at the bottom of the page once I do so. +![Enabling the API](/server_settings.png) + +Then I need to head to the **User Management** page to create a new user that will be used to authenticate against the API: +![New user creation](/new_user.png) + +And finally, I head to the **API** section to create a new API key with Read/Write permissions: +![API key creation](/api_user.png) + +### Script time +Well that's enough prep work; now it's time for the script. It's going to start by prompting the user to input required details like the fully-qualified host name of the phpIPAM server, the credentials and API key to use for the connection, and the CSV file from which to import the networks. + +![Answering the script's prompts](/script_initial.png) + +Notice that the script also prompts for a default set of DNS nameservers to be used. It will create a nameserver set in phpIPAM for each region (based on the vCenter) using these IPs. + +```python +# The latest version of this script can be found on Github: +# https://github.com/jbowdre/misc-scripts/blob/main/Python/phpipam-bulk-import.py + +import requests +from collections import namedtuple + +check_cert = True +created = 0 +remote_agent = False +mapping_set = namedtuple('mapping_set', ['name', 'id']) + +#for testing only +# from requests.packages.urllib3.exceptions import InsecureRequestWarning +# requests.packages.urllib3.disable_warnings(InsecureRequestWarning) +# check_cert = False + +def validate_input_is_not_empty(field, prompt): + while True: + user_input = input(f'\n{prompt}:\n') + if len(user_input) == 0: + print(f'[ERROR] {field} cannot be empty!') + continue + else: + return user_input + + +def get_unique_values_for_key(key, list_of_dict): + return set(sub[key] for sub in list_of_dict) + + +def get_id_from_sets(name, sets): + return [item.id for item in sets if name == item.name][0] + + +def auth_session(uri, auth): + # authenticate to the API endpoint and retrieve an auth token + print(f'Authenticating to {uri}...') + try: + req = requests.post(f'{uri}/user/', auth=auth, verify=check_cert) + except: + raise requests.exceptions.RequestException + if req.status_code != 200: + print(f'[ERROR] Authentication failure: {req.json()}') + raise requests.exceptions.RequestException + token = {"token": req.json()['data']['token']} + print('\n[AUTH_SUCCESS] Authenticated successfully!') + return token + + +def get_agent_sets(uri, token, vcenters): + # check for and create missing remote scan agents, and return + # a list of namedTuples mapping the names to IDs + agent_sets = [] + + def create_agent_set(uri, token, agentName): + import secrets + payload = { + 'name': agentName, + 'type': 'mysql', + 'code': secrets.base64.urlsafe_b64encode(secrets.token_bytes(24)).decode("utf-8") + } + req = requests.post(f'{uri}/tools/scanagents/', data=payload, headers=token, verify=check_cert) + id = req.json()['id'] + agent_set = mapping_set(agentName, id) + print(f'[AGENT_CREATE] Remote scan agent {agentName} created.') + return agent_set + + for vcenter in vcenters: + agentName = f'{vcenter}-agent' + req = requests.get(f'{uri}/tools/scanagents/?filter_by=name&filter_value={agentName}', headers=token, verify=check_cert) + if req.status_code == 200: + id = req.json()['data'][0]['id'] + print(f'[AGENT_FOUND] Remote scan agent {agentName} found.') + agent_set = mapping_set(agentName, id) + else: + agent_set = create_agent_set(uri, token, agentName) + agent_sets.append(agent_set) + return agent_sets + + +def get_section_sets(uri, token, sections, topSectionId): + # check for and create missing sections, and return a list of + # namedTuples mapping the names to IDs + section_sets = [] + + def create_section_set(uri, token, section, topSectionId): + payload = { + 'name': section, + 'masterSection': topSectionId, + 'permissions': '{"2":"2"}', + 'showVLAN': '1' + } + req = requests.post(f'{uri}/sections/', data=payload, headers=token, verify=check_cert) + id = req.json()['id'] + section_set = mapping_set(section, id) + print(f'[SECTION_CREATE] Section {section} created.') + return section_set + + for section in sections: + req = requests.get(f'{uri}/sections/{section}/', headers=token, verify=check_cert) + if req.status_code == 200: + id = req.json()['data']['id'] + print(f'[SECTION_FOUND] Section {section} found.') + section_set = mapping_set(section, id) + else: + section_set = create_section_set(uri, token, section, topSectionId) + section_sets.append(section_set) + return section_sets + + +def get_vlan_sets(uri, token, vlans): + # check for and create missing VLANs, and return a list of + # namedTuples mapping the VLAN numbers to IDs + vlan_sets = [] + + def create_vlan_set(uri, token, vlan): + payload = { + 'name': f'VLAN {vlan}', + 'number': vlan + } + req = requests.post(f'{uri}/vlan/', data=payload, headers=token, verify=check_cert) + id = req.json()['id'] + vlan_set = mapping_set(vlan, id) + print(f'[VLAN_CREATE] VLAN {vlan} created.') + return vlan_set + + for vlan in vlans: + if vlan is not '': + req = requests.get(f'{uri}/vlan/?filter_by=number&filter_value={vlan}', headers=token, verify=check_cert) + if req.status_code == 200: + id = req.json()['data'][0]['vlanId'] + print(f'[VLAN_FOUND] VLAN {vlan} found.') + vlan_set = mapping_set(vlan, id) + else: + vlan_set = create_vlan_set(uri, token, vlan) + vlan_sets.append(vlan_set) + return vlan_sets + + +def get_nameserver_sets(uri, token, vcenters, ips): + # check for and create missing nameservers, and return a list + # of namedTuples mapping the names to IDs + nameserver_sets = [] + + def create_nameserver_set(uri, token, ips, vcenter): + name = f'{vcenter}-nameserver' + payload = { + 'name': name, + 'namesrv1': ips, + 'description': f'Nameserver created for {vcenter}' + } + req = requests.post(f'{uri}/tools/nameservers/', data=payload, headers=token, verify=check_cert) + id = req.json()['id'] + nameserver_set = mapping_set(name, id) + print(f'[NAMESERVER_CREATE] Nameserver {name} created.') + return nameserver_set + + for vcenter in vcenters: + name = f'{vcenter}-nameserver' + req = requests.get(f'{uri}/tools/nameservers/?filter_by=name&filter_value={name}', headers=token, verify=check_cert) + if req.status_code == 200: + id = req.json()['data'][0]['id'] + print(f'[NAMESERVER_FOUND] Nameserver {name} found.') + nameserver_set = mapping_set(name, id) + else: + nameserver_set = create_nameserver_set(uri, token, ips, vcenter) + nameserver_sets.append(nameserver_set) + return nameserver_sets + + +def create_subnet(uri, token, network): + # create new subnets + + def update_nameserver_permissions(uri, token, network): + # update nameserver permissions so a subnet's parent section + # has access to the nameserver + nameserverId = network['nameserverId'] + sectionId = network['sectionId'] + req = requests.get(f'{uri}/tools/nameservers/{nameserverId}/', headers=token, verify=check_cert) + permissions = req.json()['data']['permissions'] + permissions = str(permissions).split(';') + if not sectionId in permissions: + permissions.append(sectionId) + if 'None' in permissions: + permissions.remove('None') + permissions = ';'.join(permissions) + payload = { + 'permissions': permissions + } + req = requests.patch(f'{uri}/tools/nameservers/{nameserverId}/', data=payload, headers=token, verify=check_cert) + + payload = { + 'subnet': network['subnet'], + 'mask': network['mask'], + 'description': network['name'], + 'sectionId': network['sectionId'], + 'scanAgent': network['agentId'], + 'nameserverId': network['nameserverId'], + 'vlanId': network['vlanId'], + 'pingSubnet': '1', + 'discoverSubnet': '1', + 'resolveDNS': '1', + 'DNSrecords': '1' + } + req = requests.post(f'{uri}/subnets/', data=payload, headers=token, verify=check_cert) + if req.status_code == 201: + network['subnetId'] = req.json()['id'] + update_nameserver_permissions(uri, token, network) + print(f"[SUBNET_CREATE] Created subnet {req.json()['data']}") + global created + created += 1 + elif req.status_code == 409: + print(f"[SUBNET_EXISTS] Subnet {network['subnet']}/{network['mask']} already exists.") + else: + print(f"[ERROR] Problem creating subnet {network['name']}: {req.json()}") + + +def import_networks(filepath): + # import the list of networks from the specified csv file + print(f'Importing networks from {filepath}...') + import csv + import re + ipPattern = re.compile('\d{1,3}\.\d{1,3}\.\d{1,3}\.[0-9xX]{1,3}') + networks = [] + with open(filepath) as csv_file: + reader = csv.DictReader(csv_file) + line_count = 0 + for row in reader: + network = {} + if line_count > 0: + if(re.search(ipPattern, row['Name'])): + network['subnet'] = re.findall(ipPattern, row['Name'])[0] + if network['subnet'].split('.')[-1].lower() == 'x': + network['subnet'] = network['subnet'].lower().replace('x', '0') + network['name'] = row['Name'] + network['section'] = row['Name'].split('-')[0] + if '/' in row['Name'][-3]: + network['mask'] = row['Name'].split('/')[-1] + else: + network['mask'] = '24' + try: + network['vlan'] = row['VlanConfiguration'].split('VLAN ')[1] + except: + network['vlan'] = '' + network['vcenter'] = f"{(row['Uid'].split('@'))[1].split(':')[0].split('.')[0]}" + networks.append(network) + line_count += 1 + print(f'\nProcessed {line_count} lines.') + print(f'Found {len(networks)} networks:\n') + print(networks) + return networks + + +def main(): + # gather inputs + import socket + import getpass + import os.path + + print("""\n\n + This script helps to add vSphere networks to phpIPAM for IP address management. It is expected + that the vSphere networks are configured as portgroups on distributed virtual switches and + named like '[Site]-[Purpose] [Subnet IP]{/[mask]}' (ex: 'LAB-Servers 192.168.1.0'). The following PowerCLI + command can be used to export the networks from vSphere: + + Get-VDPortgroup | Select Name, VlanConfiguration, Uid | Export-Csv -NoTypeInformation ./networks.csv + + Subnets added to phpIPAM will be automatically configured for monitoring either using the built-in + scan agent (default) or a new remote scan agent named for the source vCenter ('vcenter_name-agent'). + """) + input('Press enter to continue...') + + # make sure hostname resolves + while True: + hostname = input('\nFully-qualified domain name of the phpIPAM host:\n') + if len(hostname) == 0: + print('[ERROR] Hostname cannot be empty.') + continue + try: + test = socket.gethostbyname(hostname) + except: + print(f'[ERROR] Unable to resolve {hostname}.') + continue + else: + del test + break + + username = validate_input_is_not_empty('Username', f'Username with read/write access to {hostname}') + password = getpass.getpass(f'Password for {username}:\n') + apiAppId = validate_input_is_not_empty('App ID', f'App ID for API key (from https://{hostname}/administration/api/)') + topSectionName = validate_input_is_not_empty('Top-level Section Name', f'Top-level Section name (from https://{hostname}/administration/sections/)') + nameserver_ips = validate_input_is_not_empty('Nameserver IPs', f'Comma-separated list default nameserver IPs for DNS lookups from {hostname}') + + # make sure filepath is a path to an actual file + while True: + filepath = validate_input_is_not_empty('Filepath', 'Path to CSV-formatted export from vCenter') + if os.path.isfile(filepath): + break + else: + print(f'[ERROR] Unable to find file at {filepath}.') + continue + + agent = input('\nUse per-vCenter remote scan agents instead of a single local scanner? (y/N):\n') + try: + if agent.lower()[0] == 'y': + global remote_agent + remote_agent = True + except: + pass + + proceed = input(f'\n\nProceed with importing networks from {filepath} to {hostname}? (y/N):\n') + try: + if proceed.lower()[0] == 'y': + pass + else: + import sys + sys.exit("Operation aborted.") + except: + import sys + sys.exit("Operation aborted.") + del proceed + + # get collection of networks to import + networks = import_networks(filepath) + vcenters = get_unique_values_for_key('vcenter', networks) + sections = get_unique_values_for_key('section', networks) + vlans = get_unique_values_for_key('vlan', networks) + + # assemble variables + nameserver_ips = nameserver_ips.replace(',',';').replace(' ','') + uri = f'https://{hostname}/api/{apiAppId}' + auth = (username, password) + topSectionName = [topSectionName] + + # auth to phpIPAM + token = auth_session(uri, auth) + + # get lists of dictionaries matching names to IDs + nameserver_sets = get_nameserver_sets(uri, token, vcenters, nameserver_ips) + vlan_sets = get_vlan_sets(uri, token, vlans) + topSection_sets = get_section_sets(uri, token, topSectionName, None) + topSectionId = topSection_sets[0].id + section_sets = get_section_sets(uri, token, sections, topSectionId) + if remote_agent: + agent_sets = get_agent_sets(uri, token, vcenters) + + # create the networks + for network in networks: + network['nameserverId'] = get_id_from_sets(f"{network['vcenter']}-nameserver", nameserver_sets) + network['sectionId'] = get_id_from_sets(network['section'], section_sets) + network['topSectionId'] = topSectionId + if network['vlan'] is '': + network['vlanId'] = None + else: + network['vlanId'] = get_id_from_sets(network['vlan'], vlan_sets) + if remote_agent: + network['agentId'] = get_id_from_sets(f"{network['vcenter']}-agent", agent_sets) + else: + network['agentId'] = '1' + create_subnet(uri, token, network) + + print(f'\n[FINISH] Created {created} of {len(networks)} networks.') + + +if __name__ == "__main__": + main() +``` diff --git a/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/networks.csv.png b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/networks.csv.png new file mode 100644 index 0000000..fef4da6 Binary files /dev/null and b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/networks.csv.png differ diff --git a/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/new_user.png b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/new_user.png new file mode 100644 index 0000000..44c386f Binary files /dev/null and b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/new_user.png differ diff --git a/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/script_initial.png b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/script_initial.png new file mode 100644 index 0000000..b9c48fe Binary files /dev/null and b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/script_initial.png differ diff --git a/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/server_settings.png b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/server_settings.png new file mode 100644 index 0000000..425dfe0 Binary files /dev/null and b/content/posts/2022/bulk-import-vsphere-dvportgroups-to-phpipam/server_settings.png differ