Persistent Disk and duplicate resource issue
The issue in this knowledge base relates to a Google cloud error related to duplicate resources of volumes that prevent launches new instances.
The issue is caused by resources not being completely deleted the last time the cloud app was running.
In this example we are seeing volumes not being stopped correctly during the termination process.
The termination process is an important stage of the life cycle of a server
The server life cycle can be seen in the attached image named server states.
We were able to find an example of this in the audit entries of the server the volume was connected to an image named server audit entry is attached
On this image you can see how the instance was terminated.
As you can see the instance is terminated before our stop stage has completed this means that the decommission stage hasn’t completed below is an explanation of what the purpose if the decommissions stage is.
“Since each cloud infrastructure handles the shutdown process differently, RightScale added an intermediary phase before a cloud instance is permanently terminated. The purpose of the 'decommissioning' state is to provide you with a brief window to gracefully terminate your machine. The 'decommissioning' state is a RightScale-specific concept that is only supported when you terminate servers that you've defined using ServerTemplates. Each RightScale server instance that is terminated via the RightScale Dashboard or API will go through the 'decommissioning' state.”
https://docs.rightscale.com/cm/management_guide/server_states.html
One of the roles of the decommission stage is to make sure that all resources can be deleted safely, due to it skipping this stage we believe Volumes are not able to be delated when the volume is a persistent disk this is due to the nature of persistent disks.
“A persistent volume is not destroyed when an instance is deleted; instead, it must be unmounted and detached from the relevant instance.”
https://cloud.google.com/sdk/gcloud/reference/compute/disks/delete
The issue is caused by Google taking a long time to complete the deletion of the process as well as Rightscale reusing resource ID’s.
We are working on changing our process, but we are not able to resolve the issue on googles side.
The current workaround for this issue is to delete the volume on googles side as we are no longer able contact the volume. We would like to work towards finding a permanent solution to this issue.
