A workaround in the meantime is to delete the StatefulSet pod a second time if it's stuck in pending due to missing PVC.

/assign

/assign msau42

Hey @msau42 thank you for having opened the issue and the temporary workaround

any news on that bug ? ??

I think this need to be fixed, but not just workaround. a second delete of pod is not a graceful operation.

I would be happy to help guide someone to come up with a possible design and implementation for a fix.

/help

@msau42:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

when recreate pod, check if the PVC was label with deletion. maybe we could delete the PVC immediately by force and create a new PVC before Pod creation.

Hey @msau42,

I would like to work on this issue. As far as I understand StatefulSetController check the different state of the pods and take some action according to pod states when syncing statefulset.

May be I add the pod Pending state here and then check if the pod's pvc exist or not, if not then create the missing pvc.

wdyt?

the most hard part is that finding the status of pvc(creating pod process), and the deletion of pvc(pvc-protection finalizer) is in different goroutines. there will be concourrent problem.

@msau42 @kow3ns I have sent a PR #79164.
I have ran it in my cluster, it works.

Hey @msau42,

I would like to work on this issue. As far as I understand StatefulSetController check the different state of the pods and take some action according to pod states when syncing statefulset.

May be I add the pod Pending state here and then check if the pod's pvc exist or not, if not then create the missing pvc.

wdyt?

Alternative solution might be like https://github.com/mucahitkurt/kubernetes/tree/fix/enable-statefulset-pvc-recreation-for-pending-pod

I think you mean mucahitkurt@1ab4b97

@mucahitkurt if I understand your code well, if a Pod is in PodPending state you try to recreate the PVC. What if a Pod is just restarted by k8s on the same host ? would the recreation just fail ? what if the previous PVC is still in terminating (same name in our case) ?

@mucahitkurt we may need know if the pod keeps in pending or just pending transitorily.

1. If it keeps in pending, we need try to recreate the pvc ( there also may be many other reasons lead to pending)
2. If it is pending transitorily, we should do nothing

@mucahitkurt if I understand your code well, if a Pod is in PodPending state you try to recreate the PVC. What if a Pod is just restarted by k8s on the same host ? would the recreate just fail ? what if the previous PVC is still in terminating (same name in our case) ?

@cscetbon I'll test these scenarios on my local cluster, but CreatePersistentVolumeClaims does not directly create the PVC, firstly it checks the pod's pvc is missing or not, if the pvc is not found then it tries to create the pvc. So If a statefulset pod is restarting and there is no missing pvc this solution won't do anything.

If the previous PVC is still terminating, I expect that pvc lister returns this pvc and CreatePersistentVolumeClaims won't try to recreate the pvc.

@cwdsuzhou actualy CreatePersistentVolumeClaims try to create the PVC if it's missing, if the pod is in pending state and reason is another thing other than missing PVC, this solution's cost will be querying the same pvcs again and again (spc.pvcLister.PersistentVolumeClaims(claim.Namespace).Get(claim.Name)). So if a pod in pending state and there is no missing pvc the solution won't do anything that has side effect.

I wonder if we actually need to check pod state? Regardless of pod state, can we just check if any of its pvcs are missing?

@msau42 what if we check that it exists and it does in deleting state ?

I wonder if we actually need to check pod state? Regardless of pod state, can we just check if any of its pvcs are missing?

@msau42 ,

Most probably there won't be any side effect checking the missing PVCs with CreatePersistentVolumeClaims method regarless of pod state, because as far as I understand this method works in a idempotent way.

Checking pod's PVCs existence at every sync operation can create some performance effect.

I don't know about the different scenarios that we need to check the existence of pod's PVCs at every statefulset sync. Do you have any scenario?

We could potentially add logic in the statefulset controller to check if the PVC is terminating. However, it uses an informer so the race may still exist.

An alternative is to get PVC directly from API server instead of cache to avoid the race. The logic would be straightforward and the code change is small.

We'll only do this when creating a Pod with an existing PVC. Therefore, the chance of hitting this case would be pretty low, and the impact to API server would be small too.

We could potentially add logic in the statefulset controller to check if the PVC is terminating. However, it uses an informer so the race may still exist.

An alternative is to get PVC directly from API server instead of cache to avoid the race. The logic would be straightforward and the code change is small.

We'll only do this when creating a Pod with an existing PVC. Therefore, the chance of hitting this case would be pretty low, and the impact to API server would be small too.

@janetkuo
Do you have any concern with my solution alternative that has no race condition I think?

/sub

Is this issue fixed in k8s version 1.20? I used to hit this issue 100% of the times in 1.19, in 1.20 pod remains in pending until the pvc gets recreated, without any PVC not found events, once the pvc is recreated pod goes to running.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

/remove-lifecycle rotten

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Is the deletion order of Pod -> PVC -> PV still prone to this bug/race-condition? I still see this behavior in 1.20.5

Yes, PVC should be marked deleted before Pod.

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@cofyc Is this issue completely resolved or still need some fixes after your change?

/remove-lifecycle rotten
not resolved completely, but the impact is minor (I guess).

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

@k8s-triage-robot: Closing this issue.