Overview/Background
In certain situations, executing konvoy up on a cluster that is currently deployed and managed by Konvoy will cause virtual machine (EC2) instances and/or volumes (EBS) to be replaced. This will generally happen when the instance is not in a Running state and konvoy up is executed. Note that when an instance is shutdown or rebooted it will NOT be in a Running state, and these instances will be replaced (and the new instances started) if a konvoy up is executed during this time. When konvoy up does not see the instance state as Running:
- The EC2 instance will be terminated and replaced with a new instance
- The EBS volumes associated with that instance will be unattached and new EBS volumes are created and attached in their place In situations where stateful workloads are running in the cluster, this may cause issues due to the EBS volume where data resides being unattached.
Solution
If you do not wish for instances to be replaced, ensure that all EC2 instances are in a Running state before executing konvoy up. When instances are replaced, their EBS volumes will become unattached and available to attach to another instance. As such, to recover the data from the unattached volumes, you may wish to reattach the EBS volumes to the new EC2 instances in AWS.