[BUG] Model is getting stuck in deploying state

[gaurav7830]

What is the bug?
Model is getting stuck in deploying state while registering it on the cluster. We have seen cases where the model is not found on the few nodes.

Scenario

1. Model stuck in DEPLOYING state.
2. Call model undeploy api on the cluster returning the following response.

    "NodeId": {
        "stats": {
            "ModelId": "not_found"
        }
    },
    "NodeId": {
        "stats": {
            "ModelId": "not_found"
        }
    },
    "NodeId": {
        "stats": {
            "ModelId": "undeployed"
        }
    },
    "NodeId": {
        "stats": {
            "ModelId": "not_found"
        }
    },
    "NodeId": {
        "stats": {
            "ModelId": "undeployed"
        }
    },
    "NodeId": {
        "stats": {
            "ModelId": "not_found"
        }
    },
    "NodeId": {
        "stats": {
            "ModelId": "not_found"
        }
    },
    "NodeId": {
        "stats": {
            "ModelId": "undeployed"
        }
    },
    "NodeId": {
        "stats": {
            "ModelId": "not_found"
        }
    },
    "NodeId": {
        "stats": {
            "ModelId": "not_found"
        }
    }
}

3. Called GetModel api and it returning model state as DEPLOYING.

What is the expected behavior?
Model should be undeployed.

[ylwu-amzn]

@Zhangxunmt I know you have some suggestion to enhance this part. Please help take a look.