IaC local execution experimental release #1 #1601

rontalx · 2021-01-31T17:28:08Z

What does this PR do?

This PR adds a new --experimental flag to be used with snyk iac test.
It is part of a wider project called Local Execution.
It adds new logic which does issue scanning locally with a provided .wasm file that will be manually given to Snykers for a dog-fooding release.
This is intentionally not exposed yet under --help.
This flag currently supports the following functionality:

Single K8s file scanning
Sarif & JSON support
Severity threshold support
Yaml MultiDoc support

It does not support yet:

TF Scanning
Directory Support
Issue to Line Number support

Running Locally

In order to run locally, checkout the branch, build and run the local built version.
make sure to have a folder called .iac-data in your CWD where you're running the CLI from, and have the policy.wasm & data.json files in it (can be found in the attachments of: https://snyksec.atlassian.net/browse/CC-601)

snyk iac test ./k8s-configs/multi-file.yaml --experimental

TODO

Adding E2E integration tests. (WIP)

github-actions · 2021-01-31T17:29:55Z

	Warnings
⚠️	Looks like you added a new Tap test. Consider making it a Jest test instead. See files like `test/*.spec.ts` for examples. Files found: `test/smoke/.iac-data/data.json` `test/smoke/.iac-data/policy.wasm` `test/smoke/spec/iac/snyk_test_local_exec_spec.sh`

	Messages
📖	You are modifying something in test/smoke directory, yet you are not on the branch starting with smoke/. You can prefix your branch with smoke/ and Smoke tests will trigger for this PR.

Generated by 🚫 dangerJS against f8bd3f3

package.json

aron

👏 This looks great from my side. Only high level comments I have is that we should better document the entry-point in index.ts to be clear about what this new code path is and that the current code makes many uses of the term "legacy" but nowhere do we say what is legacy or why.

aron · 2021-02-02T14:38:58Z

src/cli/commands/test/iac-local-execution/index.ts

+
+const readFileContentsAsync = util.promisify(fs.readFile);
+
+export default async function legacyWrapper(pathToScan: string, options) {


Why is this a legacy wrapper? What are legacy results? Can we add types for options and the return value?

maybe the naming is inaccurate...
I didn't want the old structure and types to dictate how we re-build the new experience.
So this basically just does some "adapter" logic for the results to "plug&play" with the rest of the CLI, which made it easier to achieve lots of extra functionality like the --json & --sarif support.
got ideas for better naming?

Oh I see, yes let's avoid legacy as the rest of the CLI will continue to use it. How about we refer to the output of this function as the "cli results" and the legacy-adapter.ts as the cli-results-formatter with a comment about how it takes the internal data model and formats it for consumption by the wider snyk cli helpers.

We can just call this function execute() I think or test() with a comment above to explain why it's doing the formatting.

Thoughts?

removed all use of legacy and renamed.

src/cli/commands/test/iac-local-execution/index.ts

aron · 2021-02-02T14:41:08Z

src/cli/commands/test/iac-local-execution/index.ts

+  }
+
+  return [
+    { filePath: pathToScan, fileType: getFileType(pathToScan) as IacFileTypes },


Can we avoid this cast? Or document why it is needed.

aron · 2021-02-02T14:42:54Z

src/cli/commands/test/iac-local-execution/index.ts

+          docId: yamlDocuments.length > 1 ? docId : undefined,
+        });
+      } else {
+        throw new Error('Invalid K8s File!');


Can we include the missing fields here to help with debugging.

If we throw in one of the iterations, are we ok w/ throwing away all the process?

@aviadatsnyk
for now yes, as it supports a single file scan.
when we add directory support (upcoming follow-up work), probably partial results are the right approach, which is also how it works now if I remember correctly..
Anyway, ones of our goal along this migration to the local-exec model, is to align with the experience we currently have today for the iac remote-processing cli flow.

src/cli/commands/test/iac-local-execution/index.ts

src/cli/commands/test/iac-local-execution/policy-engine.ts

src/cli/commands/test/index.ts

test/smoke/spec/iac/snyk_test_local_exec_spec.sh

src/cli/commands/test/iac-local-execution/legacy-adapter.ts

aron

👍 Nice work, very excited to see this used internally.

src/cli/commands/test/iac-local-execution/index.ts

src/cli/commands/test/index.ts

src/cli/commands/test/iac-local-execution/policy-engine.ts

src/cli/commands/test/iac-local-execution/index.ts

maxjeffos · 2021-02-02T19:03:31Z

src/cli/commands/test/iac-local-execution/results-formatter.ts

+//   }
+// }
+
+function iacLocalFileScanToFormattedResult(


Should return a TestResult type (defined in src/lib/snyk-test/legacy.ts).

absolutely correct, but for now adding a TODO on these as it's a typing rabbit-hole due to lack of accurate typing down the road in the rest of the CLI..

maxjeffos · 2021-02-02T19:04:28Z

src/cli/commands/test/iac-local-execution/results-formatter.ts

+//   CloudConfigFileTypes,
+// } from '@snyk/cloud-config-parser';
+
+export function formatResults(


Should return a TestResult[] type (defined in src/lib/snyk-test/legacy.ts).

src/lib/snyk-test/common.ts

src/cli/commands/test/iac-local-execution/index.ts

almog27 · 2021-02-03T09:00:30Z

src/cli/commands/test/iac-local-execution/index.ts

+  const fileDataToScan = await parseFileContentsForPolicyEngine(
+    filePathsToScan,
+  );
+  const scanResults = await policyEngine.scanFiles(fileDataToScan);


maybe worth align this with the evaluate of CCPE (I saw that there is an internal evaluate function there - but maybe there it should be evaluateData or something similar)

will consider that for next release.
although I don't want to have our old types & terminology restrict us here, as it's an opportunity to rethink naming as well.

aviadhahami · 2021-02-03T10:29:51Z

package.json

@@ -58,6 +58,7 @@
  "author": "snyk.io",
  "license": "Apache-2.0",
  "dependencies": {
+    "@open-policy-agent/opa-wasm": "git+https://github.com/open-policy-agent/npm-opa-wasm.git#f4a21fe6f4d70706f85106dc6ea867983747e040",


why are we binding to a git commit and not the npm distro?
https://www.npmjs.com/package/@open-policy-agent/opa-wasm

#1601 (comment)

intentionally.
the package is not mature, and the latest tagged release is missing functionality that we are dependent on.
therefore, until they do a new release, we're targeting a specific commit-hash.
see:
open-policy-agent/npm-opa-wasm#31

aviadhahami · 2021-02-03T10:30:33Z

src/cli/commands/test/iac-local-execution/index.ts

+// without these files this method would fail.
+// if you're interested in trying out the experimental local execution model for IaC scanning, please reach-out.
+export async function test(pathToScan: string, options) {
+  // TODO: add support for proper typing of old TestResult interface.


this is a future todo?

yes, as it is a typing rabbit-hole, and I prefer not having the old types in the CLI dictate how we build the new experience.
+I want to focus on the bare-minimum of fields for the new experience data-structures, and not jam tons of fields just because the main CLI flow requires them.
so this will be typed, but soon :)

aviadhahami · 2021-02-03T10:32:50Z

src/cli/commands/test/iac-local-execution/index.ts

+  ];
+}
+
+const REQUIRED_K8S_FIELDS = ['apiVersion', 'kind', 'metadata'];


can we hoist this?

aviadhahami · 2021-02-03T10:36:23Z

src/cli/commands/test/iac-local-execution/index.ts

+          docId: yamlDocuments.length > 1 ? docId : undefined,
+        });
+      } else {
+        throw new Error('Invalid K8s File!');


If we throw in one of the iterations, are we ok w/ throwing away all the process?

aviadhahami · 2021-02-03T10:36:58Z

src/cli/commands/test/iac-local-execution/policy-engine.ts

+import * as fs from 'fs';
+
+const LOCAL_POLICY_ENGINE_DIR = `.iac-data`;
+const LOCAL_POLICY_ENGINE_WASM_PATH = `${LOCAL_POLICY_ENGINE_DIR}/policy.wasm`;


please use path.sep instead of / :)

aviadhahami · 2021-02-03T10:41:31Z

src/cli/commands/test/iac-local-execution/results-formatter.ts

+}
+
+//
+// function getFileTypeForLineNumber(


generally I 100% agree.
but keeping these intentionally, as this is logic that will be entered in one of the next follow-up PR's and is commented out due to a blocker.
wrote a comment on it here:
https://github.com/snyk/snyk/pull/1601/files#diff-2e4fd6421898cc5c5947d03530196bee66469cd25276fcdfcc0ba4381268b2c9R44

otherwise it will just get re-implemented soon.

aviadhahami · 2021-02-03T10:43:25Z

src/cli/commands/test/iac-local-execution/results-formatter.ts

+  return Object.values(groupedData);
+}
+
+const SEVERITIES = [SEVERITY.LOW, SEVERITY.MEDIUM, SEVERITY.HIGH];


hoist please

aviadhahami · 2021-02-03T10:43:45Z

src/cli/commands/test/iac-local-execution/types.ts

+import { SEVERITY } from '../../../../lib/snyk-test/common';
+import { IacFileInDirectory } from '../../../../lib/types';
+
+// eslint-disable-next-line


wanted to do aliasing, but the linter doesn't like it.
replacing with:

export type IacFileMetadata = IacFileInDirectory;

which the linter is ok with.

aviadhahami · 2021-02-03T10:46:08Z

test/smoke/spec/iac/snyk_test_local_exec_spec.sh

@@ -0,0 +1,50 @@
+#shellcheck shell=sh


why not #!/bin/sh?

we're mainly adapting the standards used here.
@JackuB do you have an answer for that?

aviadhahami · 2021-02-03T10:47:09Z

test/smoke/spec/iac/snyk_test_local_exec_spec.sh

+  After snyk_logout
+
+  Describe "k8s single file scan"
+    It "finds issues in k8s file"


why are we testing this way?

it has lots of benefits for E2E integration tests which we can easily cover most of the core flows of our product.

it's a new testing framework Hammer introduced to the CLI and we are ones of the first to adopt it out.

our old test-coverage for test iac are mainly E2E tests which are written in Tap, and are deprecated and no longer maintained.

We will probably add later-on some Jest unit-tests, but for now we're fine with that coverage.

almog27

Exciting!

aviadhahami

LGTM.
Please squash commit messages :)

github-actions · 2021-02-03T16:48:51Z

Expected release notes (by @rontalx)

features:
iac experimental single k8s file (f8bd3f3)

I hereby acknowledge these release notes are 🥙 AWESOME 🥙

snyksec · 2021-02-04T10:18:43Z

🎉 This PR is included in version 1.441.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Revert "Merge pull request #1601 from snyk/feat/iac-experimental-local-exec"

rontalx force-pushed the feat/iac-experimental-local-exec branch 5 times, most recently from 067a821 to 913c536 Compare February 1, 2021 16:52

rontalx changed the title ~~feat: iac experimental test - single k8s file~~ Iac local execution experimental release #1 Feb 2, 2021

rontalx changed the title ~~Iac local execution experimental release #1~~ IaC local execution experimental release #1 Feb 2, 2021

rontalx marked this pull request as ready for review February 2, 2021 10:20

rontalx requested review from a team as code owners February 2, 2021 10:20

rontalx requested a review from a team February 2, 2021 10:29

rontalx force-pushed the feat/iac-experimental-local-exec branch from 913c536 to 806b5e6 Compare February 2, 2021 10:34

rontalx commented Feb 2, 2021

View reviewed changes

package.json Outdated Show resolved Hide resolved

rontalx force-pushed the feat/iac-experimental-local-exec branch 4 times, most recently from fb7cad4 to 51e3e76 Compare February 2, 2021 14:20

rontalx commented Feb 2, 2021

View reviewed changes

package.json Outdated Show resolved Hide resolved

aron approved these changes Feb 2, 2021

View reviewed changes

rontalx force-pushed the feat/iac-experimental-local-exec branch 2 times, most recently from c91706d to 6ef4f48 Compare February 2, 2021 17:02

aron approved these changes Feb 2, 2021

View reviewed changes

src/cli/commands/test/iac-local-execution/index.ts Outdated Show resolved Hide resolved

src/cli/commands/test/iac-local-execution/index.ts Outdated Show resolved Hide resolved

maxjeffos approved these changes Feb 2, 2021

View reviewed changes

almog27 reviewed Feb 3, 2021

View reviewed changes

aviadhahami reviewed Feb 3, 2021

View reviewed changes

almog27 approved these changes Feb 3, 2021

View reviewed changes

rontalx requested a review from aviadhahami February 3, 2021 16:17

aviadhahami approved these changes Feb 3, 2021

View reviewed changes

feat: iac experimental single k8s file

f8bd3f3

rontalx force-pushed the feat/iac-experimental-local-exec branch from 87b270b to f8bd3f3 Compare February 3, 2021 16:48

rontalx merged commit 40e7136 into master Feb 3, 2021

rontalx deleted the feat/iac-experimental-local-exec branch February 3, 2021 17:27

snyksec added the released label Feb 4, 2021

github-actions bot mentioned this pull request Feb 4, 2021

Revert "Merge pull request #1601 from snyk/feat/iac-experimental-local-exec" #1615

Merged

JackuB added a commit that referenced this pull request Feb 4, 2021

Merge pull request #1615 from snyk/revert/fix-git-less-install

1606fbd

Revert "Merge pull request #1601 from snyk/feat/iac-experimental-local-exec"

rontalx mentioned this pull request Feb 8, 2021

re-release iac experimental single k8s file - with NPM dependency #1621

Merged

This was referenced Apr 26, 2023

[Snyk] Fix for 25 vulnerabilities aliscco/snyk#1053

Open

[Snyk] Fix for 26 vulnerabilities aliscco/snyk#1060

Open

[Snyk] Fix for 26 vulnerabilities aliscco/snyk#1099

Open

[Snyk] Fix for 26 vulnerabilities aliscco/snyk#1100

Open

This was referenced Apr 28, 2023

[Snyk] Fix for 25 vulnerabilities aliscco/snyk#1142

Open

[Snyk] Fix for 10 vulnerabilities aliscco/snyk#1157

Open

[Snyk] Fix for 24 vulnerabilities aliscco/snyk#1172

Open

snyk-bot mentioned this pull request Apr 29, 2023

[Snyk] Fix for 23 vulnerabilities aliscco/snyk#1175

Open


		const readFileContentsAsync = util.promisify(fs.readFile);

		export default async function legacyWrapper(pathToScan: string, options) {

IaC local execution experimental release #1 #1601

IaC local execution experimental release #1 #1601

Conversation

rontalx commented Jan 31, 2021 • edited Loading

What does this PR do?

Running Locally

github-actions bot commented Jan 31, 2021 • edited Loading

aron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rontalx Feb 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rontalx Feb 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

almog27 left a comment

Choose a reason for hiding this comment

aviadhahami left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 3, 2021 • edited by rontalx Loading

snyksec commented Feb 4, 2021

rontalx commented Jan 31, 2021 •

edited

Loading

github-actions bot commented Jan 31, 2021 •

edited

Loading

rontalx Feb 3, 2021 •

edited

Loading

rontalx Feb 3, 2021 •

edited

Loading

github-actions bot commented Feb 3, 2021 •

edited by rontalx

Loading