[BUG] Performance tests for 1.3 unable to detect indexing performance degradation

[kotwanikunal]

Describe the bug

• For 1.3, we ran performance tests where the results were in line or better than the previous tests for 1.2.4 in similar/same configuration(s)
• [BUG] Indexing Performance Degraded in OpenSearch 1.3.+ #2916 was raised where we noticed a drop in indexing performance
• The performance tests were unable to detect this

To Reproduce

• Steps outlined here - [BUG] Indexing Performance Degraded in OpenSearch 1.3.+ #2916

Expected behavior

• Performance tests should have detected the anomalous behavior

Plugins

• Default distro bundle

Screenshots

• N/A

Host/Environment (please complete the following information):

• [BUG] Indexing Performance Degraded in OpenSearch 1.3.+ #2916

Additional context

• N/A

[reta]

@kotwanikunal we have this benchmarking issue already opened [1], I think the automation of the benchmark runs would have been a life saver for such kind of issues.

[1] #1276

[mattweber]

Something along the lines of Lucene nightly benchmarks or Elasticsearch Benchmarks would be great to have for OpenSearch.

[reta]

Something along the lines of Lucene nightly benchmarks or Elasticsearch Benchmarks would be great to have for OpenSearch.

Exactly, unfortunately, the CI is not open (yet), it is a bit limiting towards contributions

[kotwanikunal]

@reta @mattweber From my understanding on #2916, there should have been a huge performance difference in the numbers that we got when running the performance tests with 1.2.4 and 1.3.0 with the bundled JDK.

The testing that we ran for 1.3.0 release has the results here - opensearch-project/opensearch-build#889 (comment)
But we did not see any drop or anomaly in the result. I am also investigating on why did that happen.

[mattweber]

@kotwanikunal probably need to run more than the nyc_taxis data set to test various gc profiles. The type of docs I was indexing in #2916 are large with a lot of free text so the analysis process tends to generate a lot of objects that need to be collected.

[kotwanikunal]

On further discussions with the benchmarking team, the current available workloads do not provide support for very large documents. nyc_taxis is the recommended workload to test with these limitations.
I have scheduled runs for additional 1.2.4 and 1.3.0 tests to account for any variances between the reported numbers with nyc_taxis. I will be posting the results here post test completion.

In the meanwhile, exploring further options to reproduce the reported performance drops using the current performance test framework.

cc: @travisbenedict

[kotwanikunal]

Update: The recently run tests on 1.2.4 seem to echo similar metrics as opensearch-project/opensearch-build#889 (comment). 1.3.0 seems to perform equivalent and even better in the runs with the same config.

Added in a feature request to the benchmarking team to add larger documents with free text to further strengthen the test outcomes and notice any anomalies earlier in the feature building stages.

Ref: opensearch-project/opensearch-benchmark-workloads#43

[mattweber]

@kotwanikunal in the metrics you link to, look at the GC (ms) Old column. Going from 0 in every 1.2.4 test to really anything else could be an indicator something has changed. Old GC are typically the "bad" ones that cause all the issues.

[dblock]

@kotwanikunal Do I understand it correctly that in order to identify the performance regression we will need opensearch-project/opensearch-benchmark-workloads#43? In which case we probably want to reopen this issue, and only close it when we have performance test automation that notices this regression.

[kotwanikunal]

@dblock I have reopened the issue in that case. I'll wait for updates on opensearch-benchmark-workloads#43

[saratvemulapalli]

@kotwanikunal this issue is tagged 2.1.0.
Is this something you are taking care of ? We are code freeze for 2.1, ring the bells if this is a blocker.

[kotwanikunal]

@saratvemulapalli This issue is open because we are waiting on opensearch-project/opensearch-benchmark-workloads#43. This affects the performance tests generally, and not just for v2.1.0.
The feature request is still pending and might not be available anytime soon.

@dblock Any concerns if we drop the version label from this issue given its general nature?

[dblock]

@kotwanikunal No concerns dropping the label, except that we still have this problem and I hope it gets prioritized ;)

[kotwanikunal]

Based on the new public dashboard and infra, it seems like we have the right mechanisms in place to track and detect any further issues. Closing this issue out.