Troubleshoot Google Distributed Cloud NFS and DataPlane v2 issues

This document details a manual procedure for Google Distributed Cloud if you have issues with NFS mounts with a stuck volume or Pod and you created your cluster with DataPlane v2 enabled.

This issue has been fixed for the following versions:

  • For minor version 1.16, version 1.16.4-gke.37 and higher.
  • For minor version 1.28, version 1.28.200-gke.111 and higher.

We recommend that you upgrade to a version where this issue is fixed. If you're unable to upgrade, use the procedures outlined in the following sections.

If you're using a version where this issue isn't fixed, you might encounter issues if you have workloads using ReadWriteMany volumes powered by storage drivers that are susceptible to this issue, such as (but not limited to):

  • Robin.io
  • Portworx (sharedv4 service volumes)
  • csi-nfs

NFS mounts on some storage architectures might become stuck when they're connected to an endpoint using a Kubernetes Service (ClusterIP) and DataPlane v2. This behavior is because of the limitations in how Linux kernel socket code interacts with Cillium's eBPF program. Containers might become blocked on I/O or even be unkillable, as the defunct NFS mount can't be unmounted.

You might experience this issue if you use RWX storage hosted on NFS servers that run on a Kubernetes node, including software-defined or hyperconverged storage solutions like as Ondat, Robin.io, or Portworx.

If you need additional assistance, reach out to Cloud Customer Care.

Review existing cluster configuration

Get some existing configuration values from your cluster. You use the values from these steps to create a kube-proxy manifest in the next section.

  1. Get the ClusterCIDR from cm/cilium-config:

    kubectl get cm -n kube-system cilium-config -o yaml | grep native-routing-cidr
    

    The following example output shows that you would use 192.168.0.0/16 as the ClusterCIDR:

    ipv4-native-routing-cidr: 192.168.0.0/16
    native-routing-cidr: 192.168.0.0/16
    
  2. Get the APIServerAdvertiseAddress and APIServerPort from the anetd DaemonSet:

    kubectl get ds -n kube-system  anetd -o yaml | grep KUBERNETES -A 1
    

    The following example output show that you would use 21.1.4.119 as the APIServerAdvertiseAddress and 443 as the APIServerPort:

    - name: KUBERNETES_SERVICE_HOST
      value: 21.1.4.119
    - name: KUBERNETES_SERVICE_PORT
      value: "443"
    
  3. Get the RegistryCredentialsSecretName from the anetd DaemonSet:

    kubectl get ds -n kube-system  anetd -o yaml | grep imagePullSecrets -A 1
    

    The following example output shows that you would use private-registry-creds as the RegistryCredentialsSecretName:

    imagePullSecrets:
      - name: private-registry-creds
    
  4. Get the Registry from the anetd DameonSet:

    kubectl get ds -n kube-system  anetd -o yaml | grep image
    

    The following example output shows that you would use gcr.io/gke-on-prem-release as the Registry:

    image: gcr.io/gke-on-prem-release/cilium/cilium:v1.12.6-anthos1.15-gke4.2.7
    
  5. Get the KubernetesVersion from the image tag for kube-apiserver in the cluster namespace of the admin cluster:

    KUBECONFIG=ADMIN_KUBECONFIG
    kubectl get sts -n CLUSTER_NAME kube-apiserver -o yaml | grep image
    

    Replace ADMIN_KUBECONFIG with the kubeconfig file for your admin cluster and CLUSTER_NAME with the name of your user cluster.

    The following example output shows that you would use v1.26.2-gke.1001 as the KubernetesVersion:

    image: gcr.io/gke-on-prem-release/kube-apiserver-amd64:v1.26.2-gke.1001
    imagePullPolicy: IfNotPresent
    

Prepare kube-proxy manifests

Use the values obtained in the previous section to create and apply a YAML manifest that will deploy kube-proxy to your cluster.

  1. Create a manifest named kube-proxy.yaml in the editor of your choice:

    nano kube-proxy.yaml
    
  2. Copy and paste the following YAML definition:

    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      labels:
        k8s-app: kube-proxy
      name: kube-proxy
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          k8s-app: kube-proxy
      template:
        metadata:
          annotations:
            scheduler.alpha.kubernetes.io/critical-pod: ""
          labels:
            k8s-app: kube-proxy
        spec:
          containers:
          - command:
            - kube-proxy
            - --v=2
            - --profiling=false
            - --iptables-min-sync-period=10s
            - --iptables-sync-period=1m
            - --oom-score-adj=-998
            - --ipvs-sync-period=1m
            - --ipvs-min-sync-period=10s
            - --cluster-cidr=ClusterCIDR
            env:
            - name: KUBERNETES_SERVICE_HOST
              value:APIServerAdvertiseAddress
            - name: KUBERNETES_SERVICE_PORT
              value: "APIServerPort"
            image: Registry/kube-proxy-amd64:KubernetesVersion
            imagePullPolicy: IfNotPresent
            name: kube-proxy
            resources:
              requests:
                cpu: 100m
                memory: 15Mi
            securityContext:
              privileged: true
            volumeMounts:
            - mountPath: /run/xtables.lock
              name: xtables-lock
            - mountPath: /lib/modules
              name: lib-modules
          imagePullSecrets:
          - name: RegistryCredentialsSecretName
          nodeSelector:
            kubernetes.io/os: linux
          hostNetwork: true
          priorityClassName: system-node-critical
          serviceAccount: kube-proxy
          serviceAccountName: kube-proxy
          tolerations:
          - effect: NoExecute
            operator: Exists
          - effect: NoSchedule
            operator: Exists
          volumes:
          - hostPath:
              path: /run/xtables.lock
              type: FileOrCreate
            name: xtables-lock
          - hostPath:
              path: /lib/modules
              type: DirectoryOrCreate
            name: lib-modules
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: system:kube-proxy
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:node-proxier
      subjects:
        - kind: ServiceAccount
          name: kube-proxy
          namespace: kube-system
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: kube-proxy
        namespace: kube-system
    

    In this YAML manifest, set the following values:

    • APIServerAdvertiseAddress: the value of KUBERNETES_SERVICE_HOST, such as 21.1.4.119.
    • APIServerPort: the value of KUBERNETES_SERVICE_PORT, such as 443.
    • Registry: the prefix of the Cilium image, such as gcr.io/gke-on-prem-release.
    • RegistryCredentialsSecretName: the image pull secret name, such as private-registry-creds.
  3. Save and close the manifest file in your editor.

Prepare anetd patch

Create and prepare an update for anetd:

  1. Create a manifest named cilium-config-patch.yaml in the editor of your choice:

    nano cilium-config-patch.yaml
    
  2. Copy and paste the following YAML definition:

    data:
      kube-proxy-replacement: "disabled"
      kube-proxy-replacement-healthz-bind-address: ""
      retry-kube-proxy-healthz-binding: "false"
      enable-host-reachable-services: "false"
    
  3. Save and close the manifest file in your editor.

Deploy kube-proxy and reconfigure anetd

Apply your configuration changes to your cluster. Create backups of your existing configuration before you apply the changes.

  1. Back up your current anetd and cilium-config configuration:

    kubectl get ds -n kube-system anetd > anetd-original.yaml
    kubectl get cm -n kube-system cilium-config > cilium-config-original.yaml
    
  2. Apply kube-proxy.yaml using kubectl:

    kubectl apply -f kube-proxy.yaml
    
  3. Check that the Pods are Running:

    kubectl get pods -n kube-system -o wide | grep kube-proxy
    

    The following example condensed output shows that the Pods are running correctly:

    kube-proxy-f8mp9    1/1    Running   1 (4m ago)    [...]
    kube-proxy-kndhv    1/1    Running   1 (5m ago)    [...]
    kube-proxy-sjnwl    1/1    Running   1 (4m ago)    [...]
    
  4. Patch the cilium-config ConfigMap using kubectl:

    kubectl patch cm -n kube-system cilium-config --patch-file cilium-config-patch.yaml
    
  5. Edit anetd using kubectl:

    kubectl edit ds -n kube-system anetd
    

    In the editor that opens up, edit the spec of anetd. Insert the following as the first item under initContainers:

    - name: check-kube-proxy-rules
      image: Image
      imagePullPolicy: IfNotPresent
      command:
      - sh
      - -ec
      - |
        if [ "$KUBE_PROXY_REPLACEMENT" != "strict" ]; then
          kube_proxy_forward() { iptables -L KUBE-FORWARD; }
          until kube_proxy_forward; do sleep 2; done
        fi;
      env:
      - name: KUBE_PROXY_REPLACEMENT
        valueFrom:
          configMapKeyRef:
            key: kube-proxy-replacement
            name: cilium-config
            optional: true
      securityContext:
        privileged: true
    

    Replace Image with the same image used in the other Cilium containers in the anetd DaemonSet, such as gcr.io/gke-on-prem-release/cilium/cilium:v1.12.6-anthos1.15-gke4.2.7.

  6. Save and close the manifest file in your editor.

  7. To apply these changes, reboot of all nodes in your cluster. To minimize disruption, you can attempt to drain each node prior to the reboot. However, Pods using RWX volumes may be stuck in a Terminating state due to broken NFS mounts that block the drain process.

    You can force delete blocked Pods and allow the Node to correctly drain:

    kubectl delete pods -–force -–grace-period=0 --namespace POD_NAMESPACE POD_NAME
    

    Replace POD_NAME with the Pod you are trying to delete and POD_NAMESPACE with its namespace.

What's next

If you need additional assistance, reach out to Cloud Customer Care.