Please see the first part for instructions on how to set up the needed Kubernetes cluster for the NSX Application Platform.
Now that the Kubernetes cluster is up, tested, and functional, we can go on and deploy the NSX Application Platform.
Based on my experience the VMware-provided public repo at harbor.vmware.com is quite unreliable and slow, which is why I prefer to use a private harbor registry. Harbor is an open-source project by the Linux Foundation with a heavy influence from VMware. You can find their official site here: Harbor (goharbor.io)
We can deploy the Harbor platform on our present Kubernetes Cluster – it is available as a Helm chart.
Harbor / Cloudnative-PG Prerequisites & Installation
You will need a PostgreSQL database to host the metadata and docker repository, as if found the included harbor-db image to be quite unreliable, often crashing the PostgreSQL process leading to database recoveries and unavailability.
For the harbor deployment you will need a valid certificate for HTTPS and OCI purposes. You can generate a certificate via Let’sEncrypt or use an own CA. Please keep in mind that you will need to trust the CA certificates on all nodes communicating with harbor (this includes the NSX Managers!).
Lastly, NSX only accepts Top-Level-Domains for the Harbor Repo! You will have to use a valid TLD as hostname and add a DNS entry for the instance. The signing of the certificate can still be done by an internal CA.
In my environment I added a A-record for harbor as “harbor.demo.com” and used a Windows CA to sign the certificate.
Let’s start with the installation of the PostgreSQL-Database. The easiest way I’ve found is the Cloudnative-PG Operator. The installation can be done quick and easy via helm chart.
sudo helm repo add cnpg https://cloudnative-pg.github.io/charts
sudo helm repo update
helm show values cnpg/cloudnative-pg >> cnpg-values.orig.json You can edit the default values, but I found the default values to be fully functional for me. I installed the operator without changing anything.
sudo helm install cnpg cnpg/cloudnative-pg -n cnpg-system --create-namespace [--values=cnpg-values.json]This will deploy a new namespace “cnpg-system” with a single operator pod. To add a database create a new descriptor file and apply it to the namespace. The operator will spin up pods, which will host the actual database.
cat > createdb.yaml << EOF
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: harbordb
spec:
instances: 3
storage:
size: 5Gi
EOF
kubectl apply -f createdb.yaml -n cnpg-systemThis will create the new 3 database server pods, as well as 3 services (read, write and r+w) which are reachable via the coreDNS service of the Kubernetes Cluster (svcname.namespace).
The credentials for the database access are stored in secrets, which you can also see in K9s:
You can read these via these commands:
kubectl get secrets harbordb-app -n cnpg-system --template={{.data.username}} | base64 -d
kubectl get secrets harbordb-app -n cnpg-system --template={{.data.password}} | base64 -dNote the Service DNS name (svcname.namespace – in my example harbordb-rw.cnpg-system), the Username and the Password, as we will need these values when configuring the harbor chart.
Let’s add the harbor repo and download the default values file:
sudo helm repo add harbor https://helm.goharbor.io
sudo helm repo update
helm show values harbor/harbor >> harbor-values.orig.json
cp harbor-values.orig.json harbor-values.jsonYou will have to edit the harbor file on a few key points. This mostly comes down to setting the default password, setting the certificate, the service frontend and the FQDN.
Choose your own liking – if you’re following my steps, set the values in their specific order as follows. You can also just use a file containing only these changed values, which will overwrite the default behavior:
nano harbor-values.json
...
expose:
type: loadBalancer
...
tls:
enabled: true
certSource: secret
secret:
secretName: "certsecret"
auto:
commonName: "https://harbor.demo.com"
externalURL: "https://harbor.demo.com"
harborAdminPassword: "changeme"
internalTLS:
enabled: true
registry:
relativeurls: true
database:
type: external
external:
host: "harbordb-rw.cnpg-system"
port: "5432"
username: "app"
password: "password"
coreDatabase: "app"Now, we can install the harbor chart:
sudo helm install harbor harbor/harbor -n harbor --values=harbor-values.jsonWait a few minutes for Harbor to initialize. In K9s it will look like this then:
Set the DNS entry to the displayed Load Balancer IP. You should be able to reach the Harbor UI via HTTPS.
Add the Napp repository to the Harbor Repo
In the Harbor Web UI, available at the Hostname set earlier, log in with the admin user. In the “Projects” tab, create a new project and give it an appropriate name like “napp”. Check the “Public” checkmark under “Access Level” and leave the Storage Quota at -1.
Now we can add the Napp binaries to the Repo. You will need to download the .tar-file directly from the NSX Enterprise Section at support.broadcom.com. At the time of writing the newest package is named “VMware-NSX-Application-Platform-4.1.2.1.0.23778081”
Unpack the file on a machine which has docker installed (the script will utilize the docker push command) with:
tar -xzf VMware-NSX-Application-Platform-4.1.2.1.0.23778081This will unpack all the binaries and will take a considerable amount of time. When the process is finished, you will have all the binaries and two shell scripts to upload the packages to harbor.
Edit the “upload_oci_artifacts_to_private_harbor.sh” file and enter your harbor details in the first 3 lines.
nano upload_oci_artifacts_to_private_harbor.shBe sure to set the correct URL of the harbor project created – if you followed my lead it will be “your_fqdn/napp”. You don’t need to specify a protocol. Also, set the username “admin” and your password, which you’ve set in the install.sh script.
Now we can mark the script as executable and run it:
sudo chmod +x upload_oci_artifacts_to_private_harbor.sh
./upload_oci_artifacts_to_private_harbor.shThis process will also take some time, so grab a coffee (or three) and watch out for the successful output at the end of the script:
{"saved":true} {"saved":true} {"saved":true} {"saved":true} {"saved":true} ...Now, check back at the Harbor UI. In the project, you should see 101 Repositories / artifacts.
Setting up Resources for the Napp Deployment
Now it’s finally time to log into NSX Manager and start the deployment. Go to “System” > “NSX Application Platform” and start with “Deploy NSX Application Platform”.
If you want to use VMware’s public repo, you can leave the default values – if you’ve set up your own Harbor instance in the previous steps, set the following values:
Helm Repository: oci://HARBOR-FQDN/PROJECTNAME/helm-charts
Docker Registry: HARBOR-FQDN/PROJECTNAME/clustering
Replace the FQDN and your project name. If you followed my example, it should be “napp”. Also, note that the Docker Registry needs no protocol whatsoever, only the Helm Repo will need HTTPS.
Save the URL and wait for the manager to load. If you get any issues here it may have to do with your Domain Name not being qualified (e.g. using internal TLDs) or certificate issues. For troubleshooting you can SSH on a manager node and check via the “docker login” and “helm registry login” commands if you can connect to your Harbor instance.
If everything works correctly, NSX will display the target version currently residing in the repository. In my case, it is 4.1.2-1.0-23778. Click Next.
Connecting the Cluster
Now we will need our Kubeconfig file. You can read it from a control plane node with this command:
cat /etc/kubernetes/admin.confSave the output to a text file (.conf or .yaml is fine) and upload it to NSX.
If you see a prompt wanting you to upload Kubernetes tools, your Kubernetes cluster doesn’t match the currently supported Kubernetes versions of the Napp deployment. As of writing supported versions are 1.23 through 1.27.
If the validation and connection succeed, you will now be able to select the storage class we created in the last post under the section “CSI / Container Storage Infrastructure”. Choose NFS-Client, as this is the default storage class created by our CSI plugin.
You will need to provide two FQDNs for the Interface and the messaging service for the NSX Management Cluster to interact with. Be sure to create Forward and Reverse DNS entries with corresponding IPs in your Kubernetes Load Balancer (MetalLB) range. The Napp Deployment will automatically assign the IPs to the pods.
You may also need to upload the newest Kubernetes Tools package, which supports the Kubernetes version. You can get this package at support.broadcom.com. (Note: The Kubernetes Tools 1.26.9 do support Kubernetes Version 1.27).
Now you get to choose the form factor, which decides if you will be able to install NSX Intelligence or not. Be sure that your nodes are appropriately sized.
When you click next, a series of Pre-Run-Check will be displayed. Be sure to fulfill them all. If you get a warning about the Time Synchronization, you can ignore it, as long as your nodes display the actual time when you run “timedatectl”.
The last step holding you back is the review screen. Check again for any issues and click “Deploy”.
Deployment workflow
The application platform will now go through the deployment steps. First, the Cert-Manager and projectcontour applications will be deployed as they are needed for the actual platform. If you run into any issues here, it will most likely be your CNI’s fault.
Using K9S, which we installed in the previous step we can watch the deployment and look into the logs and describe the events of the pods. In the first stage watch out for the “cert-manager-startupapi” and “projectcountour-envoy” pods for any issues.
If these first two steps succeed, your CNI is probably working correctly.
Next is the actual installation of the Napp. A good indicator here is the “postgresql-ha-postgresql” pod. If this is running ~5-10 minutes after the deployment and doesn’t create any error events when you describe the pod, you can lean back and enjoy the magic, as the rest should also work smoothly.
If everything works, you will be greeted by this nice overview.
From there on, you’re on your own – enjoy NSX Application Platform’s features😊