test: integrate runtime-tests as a helper library

[TarikGul]

closes: #495

This integrates sidecar-runtime-test into Substrate-api-sidecar. I also used this script written by @niklasad1 to help automate the tests for each chain.

TODO:

• Fix and organize types inside of runtime-tests
• Setup package.json
  • Lint, test, and build should ignore runtime-tests for sidecar
  • Lint, test, and build should ignore sidecar for runtime-tests
• Adjust readme for publishing and releasing
• Setup a readme for runtime-tests (Should explain Python env, and dependencies)

[emostov]

Would be cool if we had our own nodes - but I supposed you could not check the addresses into source control.

[niklasad1]

I think we could input the URLs as either CLI args or set env variables for them instead.

[TarikGul]

Yea I agree having our own nodes would be very nice. Would be a nice addition for the tools team in general.

[niklasad1]

Great,

In another PR: would be super to have a job that runs this on chore(release) PRs in the CI.

[dvdplm]

Not sure I understand what this does. What is the expected input here? It walks all cli arguments, filters out anything that starts with "/", sets the config[cmd] as it goes? How is it used?

[TarikGul]

Yea this can be a bit confusing.

1. Parses out all inputs that start with /. (This is so we dont save things such as the path of the node.js binary into the config)

2. For everything else If the input doesn't start with --, then we return "" (This sets the accumulator in this case cmd to false). This will resolve to a falsey value which will stop the reducer from saving it into the config.

3. argv.push(arg) is used to save any local config options in order to pass it into the jest binary such as --config=./runtime-tests/jest.config.js.

4. If the arg is equal to -- and has the name of chain then it will set it equal to the accumulator or cmd and then the arg that follows will be the argument for the command itself.

[dvdplm]

Hmm, I wonder if we're trying to be a bit too clever for our own good here; feels like this should be easier to do? ("no it's just you being thick" is a valid answer here!).

I was actually wondering something more basic: what is this args processing for? Is this right (or close to right)?

"Process the command line arguments used to launch the test session. Merge any config options provided at the command line with the on disk config."

[TarikGul]

Yea so you actually make a great point and I was able to incredibly simplify it. This method is way too clever for what it is suppose to be. I basically just wrote out a custom parser which doesn't need to be done either.

I chose to do it this way because there is no straight forward way to pass custom CLI commands to jest other than setting an entry file, and calling the binary after adding a custom configuration to the disk. But that being said I also could have just used a arg-parser and then directly pass it in same as I did. Which is exactly what I changed it too. I added argparse now and have it just be the same as any other node argument parsing tool. No loops, or complicated logic anymore.

Good catch. Thanks

[dvdplm]

I'm used to seeing reduce() return the accumulator at every iteration (including the last) and assign the return value to a variable, e.g. let sum = [1,2,3].reduce((acc, x) => x*x). Instead here it seems like you're not using the return value of the whole reduce and relying instead on side-effects like this one. Should it be a for-loop then?

[TarikGul]

So in this case I am using the reducer a little differently than normal, the accumulator in this case is being used to identify falsey values, vs truthy commands. I stayed away from the for loop because I didnt want to introduce messy iterator manipulation in order to parse the args, and I think the reducer itself allows for more customization for the future in case we wanted to add arguments to the script.

[TarikGul]

@emostov Before we merge this, can you pull this branch down and test two things as a sanity check as well.

1. yarn test:init-runtime-tests - Lets make sure you like messaging as well as the stdout output.

2. yarn lint - Sanity check to make sure the linting issue was something local to my machine and not the setup.

[emostov]

LGTM

For follow up based on convos

• Scan sidecar process stdout on startup to tell if sidecar is connected to the archive node before starting tests (right now we have a 30sec timeout)
• Save response from each endpoint and compare those to the responses received while the test is running. We can use these to document changes to the user api contract

[emostov]

@emostov Before we merge this, can you pull this branch down and test two things as a sanity check as well.

1. yarn test:init-runtime-tests - Lets make sure you like messaging as well as the stdout output.
2. yarn lint - Sanity check to make sure the linting issue was something local to my machine and not the setup.

@TarikGul Ran it locally and it all checks out ✅

[TarikGul]

Ready to merge.

Per my conversation with @emostov, we think it could be a great idea to turn this runtime-tests into a full e2e test for the /blocks endpoint. This would be done in a Part 2 extension of this PR.

Part 2

Currently the runtime tests only checks to make sure we receive a successful response back from the node. I do this by checking the block height of the response against the expected block height. But this could be done better by checking against a full response. Therefore simulating a full user request-response cycle. This would mean efficiently storing the endpoints we currently have logged for each runtime per chain, and then matching the JSON responses from the node against our stored responses.

[dvdplm]

❤️