[FEATURE] Migrate Plugin install into an GitHub action

[peternied]

Is your feature request related to a problem?
The plugin-install workflow user to separate paths to configure and run OpenSearch from the min build. This would be really useful if it was reusable for this codebase as well as others. Being that we are thinking more multiplatform this is also an opportunity where we should could invest in combining these workflows.

What solution would you like?
In the plugin-install workflow the there all the downloading, configuring, starting of OpenSearch would be replaced with a step like the following:

   - id: start-opensearch
     uses: uses: ./.github/actions/start-opensearch
     with:
         platform: ${{ matrix.platform }}
         opensearch-version: 3.0.0
         plugin-zip-path: ./security-dashboards-plugin.zip
         plugin-start-script: plugins/opensearch-security/tools/install_demo_configuration.sh

This would the be reusable for workflows within this Security codebase, but also for opensearch-project/security-dashboards-plugin#1171 where this kind of workflow would simplify the extra work needed

Do you have any additional context?
Started a prototype with peternied@1b880f4

[peternied]

FYI @scrawfor99 we talked about this offline to support the dashboard integration test workflows on linux/windows/mac

[peternied]

Note; I think this action could be made more generic so it was usable by all plugin teams, but it might be good to start building it here before sharing it out to all the other plugins via opensearch-build or another repo

[stephen-crawford]

To start things off I have been working on creating a more streamlined workflow for the plugin install. My plan is to get it working on both JDK versions for all three OS' and then I will start splitting out steps in actions and files for drop-in functionality along the lines of the prototype you linked. For a preview checkout https://github.com/scrawfor99/security/tree/CombinedInstallWorkflows

[stephen-crawford]

Update 2: Still working on streamlining the workflow and adding macOS install. Because the GHA macOS runner does not come with docker (Linux does) we cannot simply reuse the Linux commands so I am having to write a new workflow similar to the Windows install.

[peternied]

Maybe for now we just thrown an error if the OS is mac? OpenSearch doesn't yet support the mac platform so there are no guarentees it will work or we could get support to get it running.

[stephen-crawford]

Update 3: This (https://github.com/scrawfor99/security/tree/CombinedInstallWorkflows) is a refactoring of the plugin_install.yml that I made before. This version has a streamlined workflow with Linux and Windows environments being supported. The MacOS code is there but you cannot use it as it does not work yet so I removed any calls to it.
From this code I will create an actions suite for installing any given plugin on a given OS using a given opensearch version.

[stephen-crawford]

Update 4: I further cleaned up some of the install logic and then added the basic setup for the action-ification of a generic plugin install process (https://github.com/scrawfor99/security/tree/ActionfiedWorkflows) . I next need to look at the way actions work and decide whether it is best to have the user assemble the plugin beforehand and provide it built in the repo or whether we want to have the building be part of the action. As is, it expects:

platform:
description: 'What platform is this action running on, e.g. "windows-latest" ? '
required: true

opensearch-version:
description: 'The version of OpenSearch that should be used, e.g "3.0.0"'
required: true

plugin-name:
description: 'The the name of the plugin to use, such as security-dashboards-plugin'
required: true

plugin-zip-path:
description: 'The relative path to where the assembled plugin zip is stored'
required: true

plugin-start-script:
description: 'The file name for the configuration script for the plugin such as install_demo_configurations'
required: true

I have not tested it but using these inputs the security-plugin install file I wrote should work for both Windows and Linux. However, this is maybe a little bit more challenging to use then simply requiring a path to a plugin folder and parsing a lot of this information. It is just a pain do to the interaction of the shells, github actions, and the actual logic.

Once I get the security plugin to be install-able using the action instead of the workflow, then I will look at how dashboards works and figure out what changes are needed for us to be able to run OpenSearch with dashboards installed and a connected plugin interface i.e. security dashboards. Ultimately, the idea will be that the action will prompt for whether you are going to run dashboards with additional plugins and then ask you to provide paths to the plugins and the dashboard components for them.

[peternied]

Just took a peek at the plugin-install.yml [1] this is great work. Absolutely on the right track!

[1] https://github.com/scrawfor99/security/blob/ActionfiedWorkflows/.github/workflows/plugin_install.yml

[stephen-crawford]

Just took a peek at the plugin-install.yml [1] this is great work. Absolutely on the right track!

[1] https://github.com/scrawfor99/security/blob/ActionfiedWorkflows/.github/workflows/plugin_install.yml

Thank you :). I also am hoping that when we add MacOS this will make doing so pretty straightforward.

[stephen-crawford]

Update 5: Today is mostly being spent looking at the rolling upgrade issue. That being said, checkout out the linked repo above should show a cleared plugin_install with the steps migrated to an action file which will then be called as the third step.

[stephen-crawford]

Update 6:

Run curl https://localhost:9200 -u 'admin:admin' -k -v
  curl https://localhost:9200 -u 'admin:admin' -k -v
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    JAVA_HOME_17.0.5_x64: /opt/hostedtoolcache/jdk/17.0.5/x64
    JAVA_HOME: /opt/hostedtoolcache/jdk/17.0.5/x64
    JAVA_HOME_17_0_5_X64: /opt/hostedtoolcache/jdk/17.0.5/x64
*   Trying 127.0.0.1:9200...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.0 (OUT), TLS header, Unknown (21):
} [5 bytes data]
* TLSv1.3 (OUT), TLS alert, decode error (562):
} [2 bytes data]
* error:0A000126:SSL routines::unexpected eof while reading

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (35) error:0A000126:SSL routines::unexpected eof while reading
Error: Process completed with exit code 35.

This is the final error that prevents the workflow using a 5-parameter action to successfully run the Linux install test. I am not sure what the solution is and do not know what is different in the code between the functional fully-workflow-based install and the action-based version. I will hopefully fix this then address the remaining Windows steps.

[peternied]

It seems as if as it seems like it was attempting to use TLSv1.3 and the output could not be decoded both weird and bad! A couple steps that you could take to figure it out,

1. Retry the workflow - it might have been a network blip?
2. I'd recommend disabling fast-fail so you get as many test results as possible maybe its only a jdk11 thing?
3. Try printing out the curl version before using the command, maybe also add it to a version of the workflow that does work, see an example below
4. Add the command to dump the logs from docker into the workflow, maybe the certs aren't loading correctly or something else has gone wrong with the docker setup script

Curl version info

% curl -V                                                                                                                                                                                             ~/git/opensearch
curl 7.85.0 (x86_64-pc-linux-gnu) libcurl/7.85.0 OpenSSL/1.0.2zf zlib/1.2.11 c-ares/1.17.0 nghttp2/1.41.0
Release-Date: 2022-08-31
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets

Get opensearch logs

security/.github/workflows/plugin_install.yml

Lines 62 to 64 in f75be3e

	- name: Get Docker Logs
	if: always()
	run: docker logs ops

[stephen-crawford]

Update 6: I spent the day working on the refactoring requested by @peternied a half-working version is present on the linked PR and a full process can be seen over at the commit history of https://github.com/scrawfor99/security/tree/LoweredPermissionsInstall.

I also started creating a side-by-side run action for Opensearch and Dashboards. That can be found at https://github.com/scrawfor99/security/tree/DashboardsInstallAction2.4. This has been put on hold while I have been working on the refactoring.

The plan is to finish the refactoring and then continue to do the dashboards install in the style of the refactor. After that, I will be creating a dual front and backend test with OS, Security, OS Dash, Sec Dash all running.

I am tagging @RyanL1997 because yesterday we went over a lot of this and I know he is also working on some code towards these designs.

NOTE: As hinted at by the branch name I am still trying to get the lowered permissions script running on GHA.

[stephen-crawford]

Update 7: https://github.com/opensearch-project/security/pull/2239/files#diff-8518d56d53ee081f6bbcd0a88a46182f8089f57be5dab4360ee71a808e9963abR100 seems ready for final touches and merge. I have also created a branch of OS core and OS dashboards just for working on the dashboards actions. Currently implemented 2 actions on https://github.com/scrawfor99/OpenSearch-Dashboards/tree/DashboardsSetupActions that are running as I type this. They may be looping though so am looking into that.

[stephen-crawford]

This issue is fully resolved with PR #2271