Adding mixed cluster, rolling upgrade, restart upgrade bwc tests #158

VachaShah · 2021-08-05T22:15:09Z

Signed-off-by: Vacha vachshah@amazon.com

Description

Adding bwc tests and the supporting test framework for bwc. This is part of automating backwards compatibility support for the OpenSearch project. The bwc scenarios covered are:

Mixed cluster
Rolling Upgrade
Restart Upgrade

To run the mixed cluster test: ./gradlew adBwcCluster#mixedClusterTask -DmixedCluster=123 -Dtests.security.manager=false

To run the rolling upgrade test: ./gradlew adBwcCluster#rollingUpgradeClusterTask -DrollingUpgradeCluster=123 -Dtests.security.manager=false

To run the restart upgrade test: ./gradlew adBwcCluster#fullRestartClusterTask -DfullRestartCluster=123 -Dtests.security.manager=false

These tests will be hooked up to CI as part of a separate PR.

Issues Resolved

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Vacha <vachshah@amazon.com>

codecov-commenter · 2021-08-05T22:32:58Z

Codecov Report

Merging #158 (ef71fb1) into main (6843b29) will increase coverage by 0.06%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main     #158      +/-   ##
============================================
+ Coverage     70.30%   70.36%   +0.06%     
- Complexity     2970     2975       +5     
============================================
  Files           268      269       +1     
  Lines         14051    14056       +5     
  Branches       1409     1410       +1     
============================================
+ Hits           9878     9890      +12     
+ Misses         3554     3551       -3     
+ Partials        619      615       -4

Flag	Coverage Δ
plugin	`70.36% <ø> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...pensearch/ad/settings/AnomalyDetectorSettings.java	`100.00% <0.00%> (ø)`
src/main/java/org/opensearch/ad/util/MathUtil.java	`100.00% <0.00%> (ø)`
...opensearch/ad/indices/AnomalyDetectionIndices.java	`51.87% <0.00%> (+0.31%)`	⬆️
src/main/java/org/opensearch/ad/MemoryTracker.java	`77.38% <0.00%> (+1.13%)`	⬆️
.../opensearch/ad/cluster/ADClusterEventListener.java	`90.90% <0.00%> (+1.81%)`	⬆️
...search/ad/transport/AnomalyDetectorJobRequest.java	`100.00% <0.00%> (+13.51%)`	⬆️

saratvemulapalli · 2021-08-10T22:01:59Z

build.gradle

@@ -184,6 +185,12 @@ integTest {
        }
    }

+    if (System.getProperty("tests.rest.suite") == null) {
+        filter {


Looks like the tests are explicitly excluded in integTests, if BWC tests do not have any dependencies we should just let it run as part of the integtests which will be part of gradle build and will run in CI for every PR commit.

Ref: AD CI https://github.com/opensearch-project/anomaly-detection/blob/main/.github/workflows/CI.yml

They are currently excluded from the integTest as they need some additional setup to be run in CI, if I include them with integTest right now without the changes, the workflow would fail. I figured that we could exclude currently and then hook them to CI (as part of integTest) as part of the next task. LMK what you think.

saratvemulapalli · 2021-08-10T22:08:33Z

build.gradle

@@ -259,6 +266,97 @@ testClusters.integTest {
    }
 }

+String bwcVersion = "1.13.0.0";


Lets test with last known latest release, I believe it was 1.13.2
Ref: https://opendistro.github.io/for-elasticsearch-docs/version-history/

For job-scheduler and anomaly-detection, the latest releases appear to be 1.13.0.0. https://github.com/opendistro-for-elasticsearch/job-scheduler/releases and https://github.com/opendistro-for-elasticsearch/anomaly-detection/releases

Got it thank you.

saratvemulapalli · 2021-08-10T22:13:38Z

build.gradle

+testClusters {
+    "${baseName}" {
+        testDistribution = "ARCHIVE"
+        versions = ["7.10.2","1.0.0"]


We should have a way to define these versions as this is not scalable for each release.
We should open an issue for this, hopefully the plugin owners can help contribute it for us if not we can pick it up.

Yeah this is temporary, we will definitely need a way to define versions for each release. I will open an issue for this.

saratvemulapalli

This is awesome, just love this.
Dropped a few comments.

saratvemulapalli · 2021-08-10T22:19:39Z

build.gradle

+                return new RegularFile() {
+                    @Override
+                    File getAsFile() {
+                        return fileTree(bwcFilePath + "job-scheduler/" + bwcVersion).getSingleFile()


Similar comment here.
We should be able to predictably get older releases of opensearch project plugins from an artifact repository.
Can you open an issue in opensearch-build to track this?

Tracking in issue: opensearch-project/opensearch-build#184

saratvemulapalli · 2021-08-10T22:20:48Z

build.gradle

+                }
+            }
+        }))
+        setting 'repositories.url.allowed_urls', 'http://snapshot.test*'


Looks like this is needed for snapshot tests and not needed for mixed cluster

saratvemulapalli · 2021-08-10T22:23:01Z

build.gradle

+            includeTestsMatching "org.opensearch.ad.bwc.*IT"
+        }
+    }
+    systemProperty 'tests.rest.suite', 'old_cluster'


rest.suite seems ambiguous, what do you think of renaming it to rest.bwcsuite?

Suggested change

systemProperty 'tests.rest.suite', 'old_cluster'

systemProperty 'tests.rest.bwcsuite', 'old_cluster'

That makes sense! Renamed it to tests.rest.bwcsuite.

saratvemulapalli · 2021-08-10T23:05:45Z

build.gradle

+    nonInputProperties.systemProperty('tests.clustername', "${-> testClusters."${baseName}".getName()}")
+}
+
+task "${baseName}#mixedClusterTask"(type: StandaloneRestIntegTestTask) {


Couple of comments:

I see build.gradle is creeping up with lots of changes.
Can all of this logic be abstracted in bwc.gradle and consumed here?

Can you add comments to each of the steps, it would be helpful for other plugins owners to understand why these steps are required.

I have created an issue for this Split the build.gradle to make it cleaner #167, will work on that.

Added comments in the steps for all types of upgrades.

Signed-off-by: Vacha <vachshah@amazon.com>

saratvemulapalli

Overall the changes look good to me.
I'll take another pass and would also want another reviewer from AD team since they own this code base.

saratvemulapalli · 2021-08-12T20:17:42Z

build.gradle

+        })
+    ]
+
+//Creates a test cluster with 3 nodes of the old version.


super minor nit: space after comment escape string.

Suggested change

//Creates a test cluster with 3 nodes of the old version.

// Creates a test cluster with 3 nodes of the old version.

super minor nit: space after comment escape string.

Updated for all the comments.

Signed-off-by: Vacha <vachshah@amazon.com>

saratvemulapalli

The changes look good to me.
Thanks @VachaShah!

…nsearch-project#158) * Adding mixed cluster bwc tests Signed-off-by: Vacha <vachshah@amazon.com> * Adding anomaly detector assertions for bwc Signed-off-by: Vacha <vachshah@amazon.com> * Adding rolling upgrade and full restart upgrade bwc tests Signed-off-by: Vacha <vachshah@amazon.com>

* Adding mixed cluster bwc tests Signed-off-by: Vacha <vachshah@amazon.com> * Adding anomaly detector assertions for bwc Signed-off-by: Vacha <vachshah@amazon.com> * Adding rolling upgrade and full restart upgrade bwc tests Signed-off-by: Vacha <vachshah@amazon.com>

Using opensearch-project/anomaly-detection#158 as a template, created those changes as a seperate gradle file. Had trouble with the dependency and import statements so reordered how the buildscripts were setup. Running into trouble with the applying the 'opensearch.testclusters' dependency because org/gradle/jvm/toolchain/JavaInstallation was not found Signed-off-by: Peter Nied <petern@amazon.com>

Using opensearch-project/anomaly-detection#158 as a template, created those changes as a seperate gradle file. Gradlew execution of tests has been implemented, but untested. The tests themselves have been started but do not have the actual security plugin setup/validation logic that we would like to use. Scan for TODO within the files. Signed-off-by: Peter Nied <petern@amazon.com>

Adding mixed cluster bwc tests

7f3aa3a

Signed-off-by: Vacha <vachshah@amazon.com>

VachaShah force-pushed the bwc_tests_support branch from 1c3c7b6 to 7f3aa3a Compare August 5, 2021 22:16

VachaShah mentioned this pull request Aug 5, 2021

Support for bwc tests for plugins opensearch-project/OpenSearch#1051

Merged

5 tasks

VachaShah requested a review from saratvemulapalli August 5, 2021 22:19

VachaShah force-pushed the bwc_tests_support branch from 750e0cb to e2ed764 Compare August 7, 2021 03:38

VachaShah marked this pull request as ready for review August 10, 2021 20:59

saratvemulapalli reviewed Aug 10, 2021

View reviewed changes

VachaShah mentioned this pull request Aug 10, 2021

Automate publishing plugins zips to artifacts repository opensearch-project/opensearch-build#184

Closed

Adding anomaly detector assertions for bwc

88b07f4

Signed-off-by: Vacha <vachshah@amazon.com>

VachaShah force-pushed the bwc_tests_support branch from e2ed764 to 88b07f4 Compare August 11, 2021 20:23

VachaShah requested a review from saratvemulapalli August 12, 2021 16:33

VachaShah changed the title ~~Adding mixed cluster bwc tests~~ Adding mixed cluster, rolling upgrade, restart upgrade bwc tests Aug 12, 2021

VachaShah mentioned this pull request Aug 12, 2021

Split the build.gradle to make it cleaner #167

Open

saratvemulapalli reviewed Aug 12, 2021

View reviewed changes

saratvemulapalli requested a review from ylwu-amzn August 12, 2021 20:39

Adding rolling upgrade and full restart upgrade bwc tests

8d53957

Signed-off-by: Vacha <vachshah@amazon.com>

VachaShah force-pushed the bwc_tests_support branch from ef71fb1 to 8d53957 Compare August 12, 2021 20:55

VachaShah requested a review from saratvemulapalli August 12, 2021 21:54

saratvemulapalli approved these changes Aug 12, 2021

View reviewed changes

saratvemulapalli added enhancement New feature or request backwards-compatibility v1.1.0 labels Aug 12, 2021

saratvemulapalli merged commit fa081d4 into opensearch-project:main Aug 12, 2021

This was referenced Aug 16, 2021

[META] Add BWC tests to bwc_test.py opensearch-project/opensearch-build#232

Closed

Add bwc tests to run with CI #181

Merged

VachaShah deleted the bwc_tests_support branch August 17, 2021 16:58

VachaShah mentioned this pull request Aug 23, 2021

Add backwards compatibility tests for automation #188

Closed

peternied mentioned this pull request Mar 7, 2022

Backwards Compatibility Tests Draft opensearch-project/security#1666

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding mixed cluster, rolling upgrade, restart upgrade bwc tests #158

Adding mixed cluster, rolling upgrade, restart upgrade bwc tests #158

VachaShah commented Aug 5, 2021 •

edited

Loading

codecov-commenter commented Aug 5, 2021 •

edited

Loading

saratvemulapalli Aug 10, 2021

VachaShah Aug 10, 2021

saratvemulapalli Aug 10, 2021

VachaShah Aug 11, 2021

saratvemulapalli Aug 12, 2021

saratvemulapalli Aug 10, 2021

VachaShah Aug 10, 2021

saratvemulapalli left a comment

saratvemulapalli Aug 10, 2021

VachaShah Aug 10, 2021

saratvemulapalli Aug 10, 2021

VachaShah Aug 11, 2021

saratvemulapalli Aug 10, 2021

VachaShah Aug 11, 2021

saratvemulapalli Aug 10, 2021

VachaShah Aug 12, 2021

saratvemulapalli left a comment

saratvemulapalli Aug 12, 2021

VachaShah Aug 12, 2021

saratvemulapalli left a comment

	systemProperty 'tests.rest.suite', 'old_cluster'
	systemProperty 'tests.rest.bwcsuite', 'old_cluster'

	//Creates a test cluster with 3 nodes of the old version.
	// Creates a test cluster with 3 nodes of the old version.

Adding mixed cluster, rolling upgrade, restart upgrade bwc tests #158

Adding mixed cluster, rolling upgrade, restart upgrade bwc tests #158

Conversation

VachaShah commented Aug 5, 2021 • edited Loading

Description

Issues Resolved

Check List

codecov-commenter commented Aug 5, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saratvemulapalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saratvemulapalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saratvemulapalli left a comment

Choose a reason for hiding this comment

VachaShah commented Aug 5, 2021 •

edited

Loading

codecov-commenter commented Aug 5, 2021 •

edited

Loading