Cloud Vault with Kubernetes auth service account

HashiCorp Cloud Platform (HCP)
,
1

Hello all,

I have the paid cloud version of the vault and I’m trying to get my EKS cluster login properly to the vault. Specifically, I want to use cert-manager and external-secrets.

I configure my vault backend like this:

resource "vault_kubernetes_auth_backend_config" "connect_sa_with_vault" {
  backend                = vault_auth_backend.kubernetes.path
  kubernetes_host        = var.kubernetes_host
}

And for external-secrets I do:

spec:
      provider:
        vault:
          server: ${var.vault_address}
          path: "secret"
          namespace: "admin/${var.environment}"
          version: "v2"
          auth:
            kubernetes:
              mountPath: ${vault_auth_backend.kubernetes.path}
              role: ${vault_kubernetes_auth_backend_role.sa_vault_role.role_name}
              serviceAccountRef:
                name: ${kubernetes_service_account_v1.vault_client.metadata[0].name}
                namespace: ${kubernetes_namespace_v1.vault_namespace.metadata[0].name}

However, I keep getting 403 permission denied. I have tried to pass the “kubernetes_ca_cert” field as well without success.

On the other hand, if I create a static secret and use it instead of serviceAccountRef, and also use the token_reviewer_jwt and the kubernetes_ca_cert coming from such a token, then it works ok.

The service account has also the cluster role “system:auth-delegator”.

Does anyone know what I am missing?

Thank you in advance and regards

2

Without a deeper dive into your environment, I would guess you are not passing the namespace in? 11 times out of 10 that’s the cause of 403s for me when working with HCP Vault Dedicated.

Check out the HCP Vault Dedicated tutorials here:

In that collection Kubernetes with HCP Vault Dedicated and Deploy the Kubernetes Vault Secrets Operator with HCP Vault Dedicated may help.

3

Hello @jonathanfrappier,

Thanks for your response! I pass the namespace, for example in the external-secrets snippet I put in the first message there is a field of “namespace” which is the Vault’s namespace. I know it works because I also mentioned that with static token/secret it works.

What else would you need to see in order to help me? I can place as much code as needed, I didn’t want the first message to be very long so I do not scare people away.

4

If static secrets are working, the connection to Vault should be okay. Is it possible you have the path to the other secrets engines configured incorrectly?

The link to the tutorial I posted above has a number of examples to compare your environment with.

5

Hello,

I’ve managed to solved it. At the end it was a missing audience:

Instead of:

              audiences: ["vault://${kubernetes_namespace_v1.issuer_namespace.metadata[0].name}/${kubernetes_service_account_v1.issuer_sa.metadata[0].name}"]

I had to use:

              audiences: ["https://kubernetes.default.svc","vault://${kubernetes_namespace_v1.issuer_namespace.metadata[0].name}/${kubernetes_service_account_v1.issuer_sa.metadata[0].name}"]

Still talking with cert-manager people to understand why.

Thanks for the help and regards

1 Like
6

Glad you got it sorted out!