Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove firehose logging of vpc flow logs #7994

Merged
merged 1 commit into from
Sep 20, 2024
Merged

Remove firehose logging of vpc flow logs #7994

merged 1 commit into from
Sep 20, 2024

Conversation

dms1981
Copy link
Contributor

@dms1981 dms1981 commented Sep 20, 2024

A reference to the issue / Description of it

#7607

How does this PR fix the problem?

Remove streaming of flow logs through aws data firehose. A following PR will instate logging directly to S3

How has this been tested?

Tested with local PR

Deployment Plan / Instructions

Deploy through CI

Checklist (check x in [ ] of list items)

  • I have performed a self-review of my own code
  • All checks have passed
  • I have made corresponding changes to the documentation
  • Plan and discussed how it should be deployed to PROD (If needed)

Additional comments (if any)

{Please write here}

@dms1981 dms1981 requested a review from a team as a code owner September 20, 2024 12:15
@github-actions GitHub Actions
Copy link
Contributor

Trivy Scan Success

Show Output ```hcl

Trivy will check the following folders:
terraform/environments/core-vpc

Running Trivy in terraform/environments/core-vpc
2024-09-20T12:17:03Z INFO [db] Need to update DB
2024-09-20T12:17:03Z INFO [db] Downloading DB... repository="ghcr.io/aquasecurity/trivy-db:2"
2024-09-20T12:17:05Z INFO [vuln] Vulnerability scanning is enabled
2024-09-20T12:17:05Z INFO [misconfig] Misconfiguration scanning is enabled
2024-09-20T12:17:05Z INFO Need to update the built-in policies
2024-09-20T12:17:05Z INFO Downloading the built-in policies...
74.86 KiB / 74.86 KiB [-----------------------------------------------------------] 100.00% ? p/s 0s2024-09-20T12:17:05Z INFO [secret] Secret scanning is enabled
2024-09-20T12:17:05Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-09-20T12:17:05Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.55/docs/scanner/secret#recommendation for faster secret detection
2024-09-20T12:17:06Z INFO [terraform scanner] Scanning root module file_path="."
2024-09-20T12:17:06Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="aws_iam_role.member-delegation" value="cty.NilVal"
2024-09-20T12:17:06Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="aws_iam_role_policy.member-delegation" value="cty.NilVal"
2024-09-20T12:17:06Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.dns-zone" value="cty.NilVal"
2024-09-20T12:17:06Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.dns_zone_extend" value="cty.NilVal"
2024-09-20T12:17:06Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.dns_zone_extend_private" value="cty.NilVal"
2024-09-20T12:17:06Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.resource-share" value="cty.NilVal"
2024-09-20T12:17:06Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.vpc" value="cty.NilVal"
2024-09-20T12:17:06Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.vpc_attachment" value="cty.NilVal"
2024-09-20T12:17:06Z ERROR [terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable. block="module.vpc_nacls" value="cty.NilVal"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-sns-enable-topic-encryption" range="monitoring.tf:9-15"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:48"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z INFO [terraform executor] Ignore finding rule="aws-iam-no-policy-wildcards" range="iam.tf:48"
2024-09-20T12:17:06Z INFO Number of language-specific files num=0
2024-09-20T12:17:06Z INFO Detected config files num=4
trivy_exitcode=0

</details> #### `Checkov Scan` Success
<details><summary>Show Output</summary>

```hcl

*****************************

Checkov will check the following folders:
terraform/environments/core-vpc

*****************************

Running Checkov in terraform/environments/core-vpc
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
2024-09-20 12:17:10,302 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-pagerduty-integration?ref=0179859e6fafc567843cd55c0b05d325d5012dc4:None (for external modules, the --download-external-modules flag is required)
2024-09-20 12:17:10,303 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-aws-data-firehose?ref=2e58c8fd0b43ca8461dfd0c8cc5f43a1a9c49987:None (for external modules, the --download-external-modules flag is required)
2024-09-20 12:17:10,303 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-member-vpc?ref=c0475531b5c9f45a78a94fea441cd9ce91ad3c59:None (for external modules, the --download-external-modules flag is required)
2024-09-20 12:17:10,303 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/terraform-aws-observability-platform-tenant?ref=fbbe5c8282786bcc0a00c969fe598e14f12eea9b:None (for external modules, the --download-external-modules flag is required)
terraform scan results:

Passed checks: 169, Failed checks: 0, Skipped checks: 36


checkov_exitcode=0

CTFLint Scan Success

Show Output 
*****************************

Setting default tflint config...
Running tflint --init...
Installing "terraform" plugin...
Installed "terraform" (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.9.1)
tflint will check the following folders:
terraform/environments/core-vpc

*****************************

Running tflint in terraform/environments/core-vpc
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

Trivy Scan Success

Show Output 
*****************************

Trivy will check the following folders:
terraform/environments/core-vpc

*****************************

Running Trivy in terraform/environments/core-vpc
2024-09-20T12:17:03Z	INFO	[db] Need to update DB
2024-09-20T12:17:03Z	INFO	[db] Downloading DB...	repository="ghcr.io/aquasecurity/trivy-db:2"
2024-09-20T12:17:05Z	INFO	[vuln] Vulnerability scanning is enabled
2024-09-20T12:17:05Z	INFO	[misconfig] Misconfiguration scanning is enabled
2024-09-20T12:17:05Z	INFO	Need to update the built-in policies
2024-09-20T12:17:05Z	INFO	Downloading the built-in policies...
74.86 KiB / 74.86 KiB [-----------------------------------------------------------] 100.00% ? p/s 0s2024-09-20T12:17:05Z	INFO	[secret] Secret scanning is enabled
2024-09-20T12:17:05Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-09-20T12:17:05Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.55/docs/scanner/secret#recommendation for faster secret detection
2024-09-20T12:17:06Z	INFO	[terraform scanner] Scanning root module	file_path="."
2024-09-20T12:17:06Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="aws_iam_role.member-delegation" value="cty.NilVal"
2024-09-20T12:17:06Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="aws_iam_role_policy.member-delegation" value="cty.NilVal"
2024-09-20T12:17:06Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.dns-zone" value="cty.NilVal"
2024-09-20T12:17:06Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.dns_zone_extend" value="cty.NilVal"
2024-09-20T12:17:06Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.dns_zone_extend_private" value="cty.NilVal"
2024-09-20T12:17:06Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.resource-share" value="cty.NilVal"
2024-09-20T12:17:06Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.vpc" value="cty.NilVal"
2024-09-20T12:17:06Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.vpc_attachment" value="cty.NilVal"
2024-09-20T12:17:06Z	ERROR	[terraform evaluator] Failed to expand block. Invalid "for-each" argument. Must be known and iterable.	block="module.vpc_nacls" value="cty.NilVal"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-sns-enable-topic-encryption" range="monitoring.tf:9-15"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-cloudwatch-log-group-customer-key" range="../../modules/r53-resolver-logs/main.tf:13-17"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:48"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:31-50"
2024-09-20T12:17:06Z	INFO	[terraform executor] Ignore finding	rule="aws-iam-no-policy-wildcards" range="iam.tf:48"
2024-09-20T12:17:06Z	INFO	Number of language-specific files	num=0
2024-09-20T12:17:06Z	INFO	Detected config files	num=4
trivy_exitcode=0

@dms1981 dms1981 added this pull request to the merge queue Sep 20, 2024
Merged via the queue into main with commit 66499b4 Sep 20, 2024
12 checks passed
@dms1981 dms1981 deleted the feature/7607-stop-vpc-logging-via-firehose branch September 20, 2024 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ASTRobinson ASTRobinson ASTRobinson approved these changes

Assignees
No one assigned
Labels
core networking
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dms1981 @ASTRobinson