[BUG] Integration tests for dashboards failing with the image/latest we are currently using #4241

Divyaasm · 2023-11-28T01:02:21Z

Is your feature request related to a problem? Please describe

Integtests dashboards jenkins workflow uses the docker image latest (v4 == v4.3 at the time of writing this)

This is failing for a few of the dashboard plugins with the following error
Sending interrupt signal to process
script returned exit code 143.
Sample build

Plugins failed:

opensearch-dashboards
alertingDashboards
anomalyDetectionDashboards
customImportMapDashboards
indexManagementDashboards
observabilityDashboards
securityAnalyticsDashboards
securityDashboards

The issue is seen with the new image which has been recently uploaded, but the the previous versions of the docker image are working fine. (previous version == 4.2)

Describe the solution you'd like

Debug the reason for the failing tests or revert the image latest to use the previous version for integ-testing

Describe alternatives you've considered

No response

Additional context

No response

The text was updated successfully, but these errors were encountered:

peterzhuamazon · 2024-01-24T23:06:05Z

Will take a look on whether the changes of the user and home cause this issue.

peterzhuamazon · 2024-01-25T23:37:08Z

Using OSD tar on 2.12 7163 build on x64 as an example.

Using 4.3 image will keep stucking on this step of without-security test for dashboards-maps:


  Running:  plugins/custom-import-map-dashboards/documentsLayer.spec.js                     (2 of 4)


  Documents layer

In 4.2 it passes with this:


  Running:  plugins/custom-import-map-dashboards/documentsLayer.spec.js                     (2 of 4)


  Documents layer
    ✓ Add new documents layer with configuration (110069ms)
    ✓ Open saved map with documents layer


  2 passing (2m)

peterzhuamazon · 2024-01-26T17:46:37Z

Seems like issues with the chromium browser, or compatibility between Cypress 9.5.4 and Chromium version over 114:

% docker exec -it osd-rockylinux-4.2 chromium-browser --version
Chromium 114.0.5735.198 Fedora Project

% docker exec -it osd-rockylinux-4.3 chromium-browser --version
Chromium 118.0.5993.117 Fedora Project

The cypress test will stuck on the Chromium version > 114.

I even use the very old dockerfile that build 4.2 to build a new image here: https://github.com/opensearch-project/opensearch-build/blob/35048a49164337843efcf128a2b4245dc1559b8c/docker/ci/dockerfiles/current/test.rockylinux8.opensearch-dashboards.x64.arm64.dockerfile

And it still stuck with the same issue, version is:

% docker exec -it osd-rockylinux-4.2-rebuild chromium-browser --version
Chromium 120.0.6099.199 Fedora Project

peterzhuamazon · 2024-01-26T17:50:47Z

The 4.2 Rebuild Logs are here:
opensearch_dashboards.log
opensearch.log
cypress.log

peterzhuamazon · 2024-01-26T18:32:42Z

I switch back to electron and the test passes.


$ env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=false --browser electron --spec 'cypress/integration/plugins/custom-import-map-dashboards/*'

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        9.5.4                                                                          │
  │ Browser:        Electron 94 (headless)                                                         │
  │ Node Version:   v18.16.0 (/usr/share/opensearch/.nvm/versions/node/v18.16.0/bin/node)          │
  │ Specs:          4 found (plugins/custom-import-map-dashboards/add_saved_object.spec.js, plugin │
  │                 s/custom-import-map-dashboards/documentsLayer.spec.js, plugins/custom-import-m │
  │                 ap-dashboards/import_vector_map_tab.spec.js, plugins/custom-import-map-dashboa │
  │                 rds/opensearchMapLayer...)                                                     │
  │ Searched:       cypress/integration/plugins/custom-import-map-dashboards/*                     │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  plugins/custom-import-map-dashboards/add_saved_object.spec.js                   (1 of 4)


  Add flights dataset saved object
    ✓ check if maps saved object of flights dataset can be found and open (76695ms)


  1 passing (1m)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 minute, 20 seconds                                                             │
  │ Spec Ran:     plugins/custom-import-map-dashboards/add_saved_object.spec.js                    │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Video)

  -  Started processing:  Compressing to 32 CRF
  -  Finished processing: /tmp/tmpjmuih98k/customImportMapDashboards/cypress/videos/p    (8 seconds)
                          lugins/custom-import-map-dashboards/add_saved_object.spec.j
                          s.mp4


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  plugins/custom-import-map-dashboards/documentsLayer.spec.js                     (2 of 4)


  Documents layer
    ✓ Add new documents layer with configuration (114497ms)
    ✓ Open saved map with documents layer (5851ms)


  2 passing (2m)

peterzhuamazon · 2024-01-26T18:45:50Z

There is no way to install chromium 114 now as the latest available on fedora project repo is already 120.
The 120 chromium will also cause this issue on security enabled test:


  Running:  plugins/custom-import-map-dashboards/documentsLayer.spec.js                     (2 of 4)


  0 passing (2ms)

Warning: We failed processing this video.

This error will not alter the exit code.

TimeoutError: operation timed out
    at afterTimeout (/usr/share/opensearch/.cache/Cypress/9.5.4/Cypress/resources/app/packages/server/node_modules/bluebird/js/release/timers.js:46:19)
    at Timeout.timeoutTimeout [as _onTimeout] (/usr/share/opensearch/.cache/Cypress/9.5.4/Cypress/resources/app/packages/server/node_modules/bluebird/js/release/timers.js:76:13)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)

peterzhuamazon · 2024-01-26T19:32:26Z

Firefox issues: cypress-io/cypress#18919


yarn run v1.22.21
$ env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=false --browser firefox --spec 'cypress/integration/plugins/custom-import-map-dashboards/*'

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        9.5.4                                                                          │
  │ Browser:        Firefox 115 (headless)                                                         │
  │ Node Version:   v18.16.0 (/usr/share/opensearch/.nvm/versions/node/v18.16.0/bin/node)          │
  │ Specs:          4 found (plugins/custom-import-map-dashboards/add_saved_object.spec.js, plugin │
  │                 s/custom-import-map-dashboards/documentsLayer.spec.js, plugins/custom-import-m │
  │                 ap-dashboards/import_vector_map_tab.spec.js, plugins/custom-import-map-dashboa │
  │                 rds/opensearchMapLayer...)                                                     │
  │ Searched:       cypress/integration/plugins/custom-import-map-dashboards/*                     │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  plugins/custom-import-map-dashboards/add_saved_object.spec.js                   (1 of 4)


  Add flights dataset saved object
Still waiting to connect to Firefox, retrying in 1 second (attempt 18/62)

peterzhuamazon · 2024-01-26T19:38:00Z


Cypress failed to make a connection to Firefox.

This usually indicates there was a problem opening the Firefox browser.

Error: could not find CRI target
    at lazyAssLogic (/usr/share/opensearch/.cache/Cypress/9.5.4/Cypress/resources/app/node_modules/lazy-ass/index.js:110:14)
    at lazyAss (/usr/share/opensearch/.cache/Cypress/9.5.4/Cypress/resources/app/node_modules/lazy-ass/index.js:115:28)
    at findStartPage (/usr/share/opensearch/.cache/Cypress/9.5.4/Cypress/resources/app/packages/server/lib/browsers/protocol.js:56:28)
    at runMicrotasks (<anonymous>:null:null)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
~

peterzhuamazon · 2024-01-26T19:50:41Z

Chrome 121 show the same issue as chromium120.

peterzhuamazon · 2024-01-26T20:25:28Z

Approaches:

Upgrade Cypress from 9.5.4 to latest version (Long Term)
- Pros:
  - Resolve the issue from the root cause
  - Update to latest Cypress for more features and functions
- Cons:
  - Significant amount of work on both the core changes and all plugins to support it, already tried by Craig and Security Team contractor for several times, both stale: Upgrade Cypress to v12 opensearch-dashboards-functional-test#408
  - Will not be done for 2.12.0 release
Change FTrepo to use Electron instead of Chromium
- Pros:
  - Quick Fix
  - Proves working
- Cons:
  - Directly change FTrepo which cause all users of FTrepo force to change from Chromium to Electron
  - Have issues with Electron not showing all the issues that Chromium shows
(Recommended) Change opensearch-build to define BROWSER_PATH=electron env vars to override FTrepo settings
- Pros:
  - Quick Fix
  - Proves working
  - Will not change FTrepo and only change opensearch-build related workflows
- Cons:
  - Have issues with Electron not showing all the issues that Chromium shows
Change to other browsers apart from Electron/Chromium
- Chrome has the same error as Chromium
- Firefox have issues being detected by Cypress 9.5.4
- Edge is basically using the Chromium core

bbarani · 2024-01-26T20:34:06Z

@kavilla @joshuarrrr @ashwin-pc @seraphjiang @AMoo-Miki We are thinking of using electron for 2.12.0 instead of Chromium due to the issues mentioned above. Can you provide your inputs?

peterzhuamazon · 2024-01-29T22:49:28Z

Related to #4115.

Override browser default from chromium to electron as Cypress 9.5.4 does not run with chromium 114+ #4383

Divyaasm added bug Something isn't working untriaged Issues that have not yet been triaged labels Nov 28, 2023

peterzhuamazon self-assigned this Nov 28, 2023

peterzhuamazon added the release label Dec 1, 2023

peterzhuamazon mentioned this issue Dec 1, 2023

[Meta] OpenSearch Release Improvements #4216

Open

32 tasks

jordarlu removed the untriaged Issues that have not yet been triaged label Dec 5, 2023

jordarlu unassigned peterzhuamazon Dec 5, 2023

peterzhuamazon changed the title ~~Integration tests for dashboards failing with the image/latest we are currently using~~ [BUG] Integration tests for dashboards failing with the image/latest we are currently using Jan 25, 2024

peterzhuamazon mentioned this issue Jan 26, 2024

Reactor and enable integ-tests for all distributions in OSD #4374

Merged

peterzhuamazon self-assigned this Jan 26, 2024

peterzhuamazon added docker integtest labels Jan 26, 2024

peterzhuamazon closed this as completed Jan 26, 2024

peterzhuamazon reopened this Jan 26, 2024

github-actions bot added the untriaged Issues that have not yet been triaged label Jan 26, 2024

peterzhuamazon added test ci-test-automation v2.12.0 and removed untriaged Issues that have not yet been triaged labels Jan 26, 2024

This was referenced Jan 29, 2024

Support ppc64le OpenSearch distribution for POWER architecture. #3122

Open

Override browser default from chromium to electron as Cypress 9.5.4 does not run with chromium 114+ #4383

Merged

peterzhuamazon closed this as completed in #4383 Jan 30, 2024

peterzhuamazon mentioned this issue Feb 6, 2024

[RELEASE] Release version 2.12.0 #4115

Closed

88 tasks

peterzhuamazon mentioned this issue Feb 20, 2024

[Enhancement] Properly lock docker ci images chromium version #4459

Closed

SuZhou-Joe mentioned this issue Jun 13, 2024

feat: freeze browser version to 112 opensearch-project/opensearch-dashboards-functional-test#1408

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Integration tests for dashboards failing with the image/latest we are currently using #4241

[BUG] Integration tests for dashboards failing with the image/latest we are currently using #4241

Divyaasm commented Nov 28, 2023 •

edited by peterzhuamazon

Loading

peterzhuamazon commented Jan 24, 2024

peterzhuamazon commented Jan 25, 2024 •

edited

Loading

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024 •

edited

Loading

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024 •

edited

Loading

bbarani commented Jan 26, 2024 •

edited

Loading

peterzhuamazon commented Jan 29, 2024 •

edited

Loading

[BUG] Integration tests for dashboards failing with the image/latest we are currently using #4241

[BUG] Integration tests for dashboards failing with the image/latest we are currently using #4241

Comments

Divyaasm commented Nov 28, 2023 • edited by peterzhuamazon Loading

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Additional context

peterzhuamazon commented Jan 24, 2024

peterzhuamazon commented Jan 25, 2024 • edited Loading

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024 • edited Loading

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024

peterzhuamazon commented Jan 26, 2024 • edited Loading

bbarani commented Jan 26, 2024 • edited Loading

peterzhuamazon commented Jan 29, 2024 • edited Loading

Divyaasm commented Nov 28, 2023 •

edited by peterzhuamazon

Loading

peterzhuamazon commented Jan 25, 2024 •

edited

Loading

peterzhuamazon commented Jan 26, 2024 •

edited

Loading

peterzhuamazon commented Jan 26, 2024 •

edited

Loading

bbarani commented Jan 26, 2024 •

edited

Loading

peterzhuamazon commented Jan 29, 2024 •

edited

Loading