Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stack rollback stuck in UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS #26618

Closed
malikalimoekhamedov opened this issue Aug 3, 2023 · 1 comment
Labels
@aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud bug This issue is a bug. needs-triage This issue or PR still needs to be triaged.

Comments

@malikalimoekhamedov
Copy link

malikalimoekhamedov commented Aug 3, 2023

Describe the bug

One of my CloudFormation stacks is stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS state. The stack is not nested.

I can't do anything with it besides waiting. This is a blocking issue. Is there anything I can do through, say, AWS CLI?
I looked online, and they say you should contact AWS to unblock the state. However, I'm on a basic support plan, so I can't file a case.

Expected Behavior

UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS state should turn to UPDATE_COMPLETED.

Current Behavior

The stack update status is stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS.

Reproduction Steps

I'm not sure the code below can cast any light on this issue, but I'll provide it anyway.

I'm trying to update the stack with a new custom resource:

import {
  AwsCustomResource,
  AwsCustomResourcePolicy,
  PhysicalResourceId,
} from 'aws-cdk-lib/custom-resources';
import { Construct } from 'constructs';
import { DataWarehouseButlerProps } from './data-warehouse-butler.types';
import { RetentionDays } from 'aws-cdk-lib/aws-logs';

/**
 * A custom AWS resource managing the schema and the contents of the data warehouse
 * upon CUD operations.
 *
 * @author Malik Alimoekhamedov
 * @category Constructs
 */
class DataWarehouseButler extends Construct {
  constructor(scope: Construct, id: string, props: DataWarehouseButlerProps) {
    super(scope, id);

    if (!props.shouldRestoreFromSnapshot) {
      new AwsCustomResource(this, 'Init', {
        resourceType: 'Custom::DataWarehouseButlerInit',
        functionName: 'data-warehouse-butler-init',
        logRetention: RetentionDays.ONE_DAY,
        vpc: props.vpc,
        vpcSubnets: { subnets: props.vpc.isolatedSubnets },
        policy: AwsCustomResourcePolicy.fromSdkCalls({
          resources: AwsCustomResourcePolicy.ANY_RESOURCE,
        }),
        onCreate: {
          service: 'RedshiftData', //TODO: Switch to @aws-sdk/client-redshift-data when SDK v3 support kicks in.
          action: 'batchExecuteStatement', //TODO: Switch to BatchExecuteStatementCommand when SDK v3 support kicks in.
          physicalResourceId: PhysicalResourceId.of(props.workgroupName),
          parameters: {
            WorkgroupName: props.workgroupName,
            Database: props.dbName,
            Sqls: [
              `
              CREATE TABLE public.domain_statistics (
                domain character varying(1000) NOT NULL ENCODE raw distkey,
                created_at date NOT NULL ENCODE az64,
                redirect_to character varying(1000) ENCODE lzo,
                ip_address character varying(16) ENCODE lzo,
                domain_manager character varying(36) ENCODE lzo,
                desktop_ranking integer ENCODE az64,
                PRIMARY KEY ("domain", created_at)
              ) DISTSTYLE KEY
              SORTKEY
                (domain);
              `,
              `
              CREATE TABLE public.domain_country_statistics (
                domain character varying(1000) NOT NULL ENCODE raw distkey,
                created_at date NOT NULL ENCODE az64,
                country character varying(2) NOT NULL ENCODE lzo,
                http_status_code character varying(3) ENCODE lzo,
                desktop_ranking integer ENCODE az64,
                desktop_visits double precision ENCODE raw,
                mobile_visits double precision ENCODE raw,
                desktop_visitors double precision ENCODE raw,
                mobile_visitors double precision ENCODE raw,
                PRIMARY KEY ("domain", created_at, country)
              ) DISTSTYLE KEY
              SORTKEY
                (domain);
              `,
              /* `
              -- Load saved dump
              COPY domain_statistics FROM 's3://${props.dataDumpS3BucketName}/domain_statistics/'
              IAM_ROLE default
              FORMAT AS PARQUET;
              `,
              `
              -- Load saved dump
              COPY domain_country_statistics FROM 's3://${props.dataDumpS3BucketName}/domain_country_statistics/'
              IAM_ROLE default
              FORMAT AS PARQUET;
              `, */
            ],
          },
        },
      });
    } else {
      new AwsCustomResource(this, 'Restore', {
        resourceType: 'Custom::DataWarehouseButlerRestore',
        functionName: 'data-warehouse-butler-restore',
        logRetention: RetentionDays.ONE_DAY,
        vpc: props.vpc,
        vpcSubnets: { subnets: props.vpc.isolatedSubnets },
        policy: AwsCustomResourcePolicy.fromSdkCalls({
          resources: AwsCustomResourcePolicy.ANY_RESOURCE,
        }),
        onCreate: {
          service: 'RedshiftServerless',
          action: 'restoreFromSnapshot',
          physicalResourceId: PhysicalResourceId.of(props.workgroupName),
          parameters: {
            workgroupName: props.workgroupName,
            namespaceName: props.namespaceName,
            snapshotName: props.finalSnapshotName,
          },
        },
      });
    }
  }
}

export { DataWarehouseButler };

Possible Solution

Maybe some --force flag issues together with an AWS CLI command? There's nothing I can do from within the web console to unblock the situation.

Additional Information/Context

No response

CDK CLI Version

2.89

Framework Version

No response

Node.js Version

18

OS

MacOS Ventura 13.5

Language

Typescript

Language Version

5.1.6

Other information

No response

@malikalimoekhamedov malikalimoekhamedov added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 3, 2023
@github-actions github-actions bot added the @aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud label Aug 3, 2023
@github-actions
Copy link

github-actions bot commented Aug 3, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud bug This issue is a bug. needs-triage This issue or PR still needs to be triaged.
Projects
None yet
Development

No branches or pull requests

1 participant