Skip to content

Deploy TKG packages in airgapped environments

Prerequisites

  • TKGS Supervisor cluster running
  • embedded Harbor running
  • shared services cluster running

Environment info

  • vCenter 7u3p (Build 22837322)
  • Supervisor cluster version v1.25.6+vmware.wcp.2
  • Guest Cluster version: v1.23.8---vmware.3-tkg.1

Install Packages

We are installing the following packages:

Package Version
cert-manager 1.7.2+vmware.1-tkg.1
contour 1.20.2+vmware.2-tkg.1
harbor 2.3.3+vmware.1-tkg.1

We deploy those packages on a Kubernetes cluster with version 1.23.8. This cluster has PodSecurityPolicies enabled. Because we don't care, we allow all pods with

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psp:privileged
rules:
- apiGroups: ['policy']
  resources: ['podsecuritypolicies']
  verbs:     ['use']
  resourceNames:
  - vmware-system-privileged
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: all:psp:privileged
roleRef:
  kind: ClusterRole
  name: psp:privileged
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts
  apiGroup: rbac.authorization.k8s.io
EOF

Deploy TKG Package Repository

Install kapp-controller

We are following the official docs here.

Run the following from a machine with access to the VMware public registry:

  1. list available versions:

    imgpkg tag list -i projects.registry.vmware.com/tkg/kapp-controller

  2. Copy the version of choice to your local registry

    imgpkg copy \
  -i projects.registry.vmware.com/tkg/kapp-controller:v0.30.0_vmware.1 \
  --to-repo 172.30.4.131/shared-services/kapp-controller \
  --registry-ca-cert-path ./ca.crt

    Alternatively, you can download the tar file to your filesystem

    imgpkg copy \
  -i projects.registry.vmware.com/tkg/kapp-controller:v0.41.7_vmware.1 \
  --to-tar  ./kapp-controller_v0.41.7_vmware.1

  3. Create the tanzu-package-repo-global namespace:

    kubectl create ns tanzu-package-repo-global

  4. create a secret to be able to pull images from the local registry with authentication

    kubectl create secret docker-registry embedded-harbor \
  --docker-server=172.30.4.131 \
  --docker-username=administrator@vsphere.local \
  --docker-password=VMware1! \
  -n tanzu-package-repo-global

  5. Copy the content of the kapp-controller manifest from here and make some changes:

  6. update the image: accordingly to point to your image stored in your local container registry

  7. add the following to Deployment.spec.template.spec

    ```shell
imagePullSecrets:
- name: embedded-harbor
```

  8. Switch kubectl context to your shared services cluster and apply the manifest

    kubectl apply -f kapp-controller.yaml

Add Package Repository to Cluster

We are following the official docs here.

Run the following from a machine with access to the VMware public registry:

  1. list available Package Repository versions:

    imgpkg tag list -i projects.registry.vmware.com/tkg/packages/standard/repo

  2. Copy your version of choice to your registry

    imgpkg copy \
  -b projects.registry.vmware.com/tkg/packages/standard/repo:v1.6.1 \
    --to-repo 172.30.4.131/shared-services/packages/standard/repo \
    --registry-ca-cert-path ./ca.crt

  3. Create a PackageRepository manifest and call it packagerepo-v1.6.1.yaml:

    apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
  name: tanzu-standard
  namespace: tanzu-package-repo-global
spec:
  fetch:
    imgpkgBundle:
      image: 172.30.4.131/shared-services/packages/standard/repo:v1.6.1
      secretRef:
        name: embedded-harbor

  4. Switch kubectl context to your shared services cluster and apply the manifest

    kubectl apply -f packagerepo-v1.6.1.yaml

Prepare user managed Tanzu Packages

  1. Create a common namespace used for all user managed Tanzu packages: 

    kubectl create ns tanzu-packages-user-managed

  2. Replicate the embedded-harbor secret from the tanzu-package-repo-global namespace to the tanzu-packages-user-managed namespace:

    kubectl create secret docker-registry embedded-harbor \
  --docker-server=172.30.4.131 \
  --docker-username=administrator@vsphere.local \
  --docker-password=VMware1! \
  -n tanzu-packages-user-managed

  3. To use the embedded-harbor in all cert-manager deployment's spec.template.spec.imagePullSecrets we have to create a ytt overlay and use that overlay in the PackageInstall. Create the overlay image-pull-secrets-overlay-deployment.yaml

    #@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.subset({"kind": "Deployment"}), expects="1+"
---
spec:
  template:
    spec:
      #@overlay/match missing_ok=True
      imagePullSecrets:
      - name: embedded-harbor

    and then create a Kubernetes secret:

    kubectl create secret generic image-pull-secret-overlay-deployment \
  --from-file=image-pull-secrets-overlay-deployment.yaml \
  -n tanzu-packages-user-managed

  4. We do the same for DaemonSets. Create the overlay image-pull-secrets-overlay-daemonset.yaml

    #@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.subset({"kind": "DaemonSet"}), expects="1+"
---
spec:
  template:
    spec:
      #@overlay/match missing_ok=True
      imagePullSecrets:
      - name: embedded-harbor

    and then create a Kubernetes secret:

    kubectl create secret generic image-pull-secret-overlay-daemonset \
  --from-file=image-pull-secrets-overlay-daemonset.yaml \
  -n tanzu-packages-user-managed

  5. We do the same for StatefulSets. Create the overlay image-pull-secrets-overlay-statefulsets.yaml

    #@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.subset({"kind": "StatefulSet"}), expects="1+"
---
spec:
  template:
    spec:
      #@overlay/match missing_ok=True
      imagePullSecrets:
      - name: embedded-harbor

    and then create a Kubernetes secret:

    kubectl create secret generic image-pull-secret-overlay-statefulsets \
  --from-file=image-pull-secrets-overlay-statefulsets.yaml \
  -n tanzu-packages-user-managed

Install cert-manager

We are following the official docs here.

  1. Create the cert-manager namespace:

    kubectl create ns cert-manager

  2. Create the embedded-harbor secrets used in imagePullSecrets in each pod:

    kubectl create secret docker-registry embedded-harbor \
  --docker-server=172.30.4.131 \
  --docker-username=administrator@vsphere.local \
  --docker-password=VMware1! \
  -n cert-manager

  3. List available version for cert-manager:

    kubectl get packages -n tanzu-packages-user-managed | grep cert-manager

  4. Create the manifest cert-manager.yaml:

    apiVersion: v1
kind: ServiceAccount
metadata:
  name: cert-manager-sa
  namespace: tanzu-packages-user-managed
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: cert-manager-sa
    namespace: tanzu-packages-user-managed
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: cert-manager
  namespace: tanzu-packages-user-managed
  annotations:
    ext.packaging.carvel.dev/fetch-0-secret-name: embedded-harbor
    ext.packaging.carvel.dev/ytt-paths-from-secret-name.0: image-pull-secret-overlay-deployment
spec:
  serviceAccountName: cert-manager-sa
  packageRef:
    refName: cert-manager.tanzu.vmware.com
    versionSelection:
      constraints: 1.7.2+vmware.1-tkg.1
  values:
  - secretRef:
      name: cert-manager-data-values
---
apiVersion: v1
kind: Secret
metadata:
  name: cert-manager-data-values
  namespace: tanzu-packages-user-managed
stringData:
  values.yml: |
    ---
    namespace: cert-manager

    and apply it to the cluster:

    kubectl apply -f cert-manager.yaml

Install Contour

The process is very simlar to installing cert-manager. We are following the official docs here using kubectl.

  1. Create the tanzu-system-ingress namespace:

    kubectl create ns tanzu-system-ingress

  2. Create the embedded-harbor secrets used in imagePullSecrets in each pod:

    kubectl create secret docker-registry embedded-harbor \
  --docker-server=172.30.4.131 \
  --docker-username=administrator@vsphere.local \
  --docker-password=VMware1! \
  -n tanzu-system-ingress

  3. List available version for contour:

    kubectl get packages -n tanzu-packages-user-managed | grep contour

  4. Create the manifest contour.yaml

    apiVersion: v1
kind: ServiceAccount
metadata:
  name: contour-sa
  namespace: tanzu-packages-user-managed
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: contour-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: contour-sa
    namespace: tanzu-packages-user-managed
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: contour
  namespace: tanzu-packages-user-managed
  annotations:
    ext.packaging.carvel.dev/fetch-0-secret-name: embedded-harbor
    ext.packaging.carvel.dev/ytt-paths-from-secret-name.0: image-pull-secret-overlay-deployment
    ext.packaging.carvel.dev/ytt-paths-from-secret-name.1: image-pull-secret-overlay-daemonset
spec:
  serviceAccountName: contour-sa
  packageRef:
    refName: contour.tanzu.vmware.com
    versionSelection:
      constraints: 1.20.2+vmware.2-tkg.1
  values:
  - secretRef:
      name: contour-data-values
---
apiVersion: v1
kind: Secret
metadata:
  name: contour-data-values
  namespace: tanzu-packages-user-managed
stringData:
  values.yml: |
    ---
    infrastructure_provider: vsphere
    namespace: tanzu-system-ingress
    contour:
      configFileContents: {}
      useProxyProtocol: false
      replicas: 2
      pspNames: "vmware-system-restricted"
      logLevel: info
    envoy:
      service:
        type: LoadBalancer
        annotations: {}
        nodePorts:
          http: null
          https: null
        externalTrafficPolicy: Cluster
        disableWait: false
      hostPorts:
        enable: false
        http: 80
        https: 443
      hostNetwork: false
      terminationGracePeriodSeconds: 300
      logLevel: info
      pspNames: null
    certificates:
      duration: 8760h
      renewBefore: 360h

    and apply it to the cluster:

    kubectl apply -f contour.yaml

Install Harbor

  1. Create the tanzu-system-registry namespace:

    kubectl create ns tanzu-system-registry

  2. Create the embedded-harbor secrets used in imagePullSecrets in each pod:

    kubectl create secret docker-registry embedded-harbor \
  --docker-server=172.30.4.131 \
  --docker-username=administrator@vsphere.local \
  --docker-password=VMware1! \
  -n tanzu-system-registry

  3. List available version for harbor:

    kubectl get packages -n tanzu-packages-user-managed | grep harbor

  4. Create the manifest harbor.yaml:

    apiVersion: v1
kind: ServiceAccount
metadata:
  name: harbor-sa
  namespace: tanzu-packages-user-managed
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: habor-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: harbor-sa
    namespace: tanzu-packages-user-managed
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: harbor
  namespace: tanzu-packages-user-managed
  annotations:
    ext.packaging.carvel.dev/fetch-0-secret-name: embedded-harbor
    ext.packaging.carvel.dev/ytt-paths-from-secret-name.0: image-pull-secret-overlay-deployment
    ext.packaging.carvel.dev/ytt-paths-from-secret-name.1: image-pull-secret-overlay-statefulsets
spec:
  serviceAccountName: harbor-sa
  packageRef:
    refName: harbor.tanzu.vmware.com
    versionSelection:
      constraints: 2.3.3+vmware.1-tkg.1
  values:
  - secretRef:
      name: harbor-data-values
---
apiVersion: v1
kind: Secret
metadata:
  name: harbor-data-values
  namespace: tanzu-packages-user-managed
stringData:
  values.yml: |
    namespace: tanzu-system-registry
    hostname: harbor.internal
    port:
      https: 443
    logLevel: info
    tlsCertificate:
      tls.crt: ""
      tls.key: ""
      ca.crt:
    tlsCertificateSecretName:
    enableContourHttpProxy: true
    harborAdminPassword: 'VMware1!'
    secretKey: 'aiGhooghu8uaS7zo'
    database:
      password: 'VMware1!'
      shmSizeLimit:
      maxIdleConns:
      maxOpenConns:
    exporter:
      cacheDuration:
    core:
      replicas: 1
      secret: 'VMware1!'
      xsrfKey: oopoo7iecae8wai5eejeethaingeip4W
    jobservice:
      replicas: 1
      secret: 'VMware1!'
    registry:
      replicas: 1
      secret: 'VMware1!'
    notary:
      enabled: true
    trivy:
      enabled: true
      replicas: 1
      gitHubToken: ""
      skipUpdate: false
    persistence:
      persistentVolumeClaim:
        registry:
          existingClaim: ""
          storageClass: "tkgs-storage-policy"
          subPath: ""
          accessMode: ReadWriteOnce
          size: 50Gi
        jobservice:
          existingClaim: ""
          storageClass: "tkgs-storage-policy"
          subPath: ""
          accessMode: ReadWriteOnce
          size: 10Gi
        database:
          existingClaim: ""
          storageClass: "tkgs-storage-policy"
          subPath: ""
          accessMode: ReadWriteOnce
          size: 10Gi
        redis:
          existingClaim: ""
          storageClass: "tkgs-storage-policy"
          subPath: ""
          accessMode: ReadWriteOnce
          size: 10Gi
        trivy:
          existingClaim: ""
          storageClass: "tkgs-storage-policy"
          subPath: ""
          accessMode: ReadWriteOnce
          size: 10Gi
    proxy:
      httpProxy:
      httpsProxy:
      noProxy: 127.0.0.1,localhost,.local,.internal
    pspNames: vmware-system-restricted,vmware-system-privileged
    network:
      ipFamilies: ["IPv4", "IPv6"]

    and apply it to the cluster:

    kubectl apply -f harbor.yaml

Upgrade Packages

We are updating the following packages:

Package Current Version Target Version
cert-manager 1.7.2+vmware.1-tkg.1 1.10.2+vmware.1-tkg.1
contour 1.20.2+vmware.2-tkg.1 1.23.5+vmware.1-tkg.1
harbor 2.3.3+vmware.1-tkg.1 2.5.3+vmware.1-tkg.1

Upgrade Tanzu Package Repository

In order to have newer Package versions available, we first have to upgrade the Tanzu Package Repository.

  1. list available Package Repository versions:

    imgpkg tag list -i projects.registry.vmware.com/tkg/packages/standard/repo

  2. Copy a new version to your local registry

    imgpkg copy \
  -b projects.registry.vmware.com/tkg/packages/standard/repo:v2.2.0 \
    --to-repo 172.30.4.131/shared-services/packages/standard/repo \
    --registry-ca-cert-path ./ca.crt

  3. Update the existing Package repository accordingly

    cat <<EOF | kubectl apply -f -
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
  name: tanzu-standard
  namespace: tanzu-package-repo-global
spec:
  fetch:
    imgpkgBundle:
      image: 172.30.4.131/shared-services/packages/standard/repo:v2.2.0
      secretRef:
        name: embedded-harbor
EOF

Upgrade cert-manager

  1. List available version for cert-manager:

    kubectl get packages -n tanzu-packages-user-managed | grep cert-manager

  2. Find a compatible cert-manager version with your Kubernetes version and update cert-manager:

    tanzu package installed update cert-manager \
  --version 1.10.2+vmware.1-tkg.1 \
  -n tanzu-packages-user-managed

Upgrade contour

  1. List available version for contour:

    kubectl get packages -n tanzu-packages-user-managed | grep contour

  2. Find a compatible contour version with your Kubernetes version and update contour:

    tanzu package installed update contour \
  --version 1.23.5+vmware.1-tkg.1 \
  -n tanzu-packages-user-managed

Upgrade harbor

  1. List available version for harbor:

    kubectl get packages -n tanzu-packages-user-managed | grep harbor

  2. Update harbor:

    tanzu package installed update harbor \
  --version 2.5.3+vmware.1-tkg.1 \
  -n tanzu-packages-user-managed

Things to note

Using local content library

when using local content library, the name when importing in vCenter has to be the same name as in item.json. Otherwise you get the following error when using the TKR:

unable to resolve that TKR due to could not resolve TKR/OSImage for controlPlane