[7.x] [Fleet][EPM] Save installed package assets in ES (#83391) #85186
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Summary Store package assets (from Registry or local upload) in Elasticsearch. Related to proposal [issue](elastic#83426) & [document](https://docs.google.com/document/d/18XoS6CSl9UxxPPBt9LXuJngf1Jv-4tl3jY6l19U1yH8) * New `epm-packages-assets` saved objects are stored on `.kibana` index, like our existing saved object `epm-packages` * Asset id is uuid v5 based on the package name, package version & file path. See 1974324 * Add a list of IDs of all the installed assets, to `epm-packages` saved object. Like the existing `installed_` properties. [Example](https://github.com/elastic/kibana/pull/83391/files#diff-fa07cac51b6a49bf1e4824bc2250c9a77dac6c7d6b0a56020f559ef1ff9be25fR491-R512) from a test <details><summary>Mapping for new Saved Object</summary> https://github.com/elastic/kibana/blob/37f7b6ded747edb5cc487661b801c5e1c0a102a7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fsaved_objects%2Findex.ts#L329-L339 </details> <details><summary>Additional property on existing <code>epm-packages</code> Saved Object</summary> https://github.com/elastic/kibana/blob/c4f27ab25715a225c710727a37f5f105364e2f72/x-pack/plugins/fleet/server/saved_objects/index.ts#L306-L312 I don't think the saved object changes are strictly required. It can be removed without changing much about how things work - Pros: - Preserves accurate record of the assets added at installation time. Separates what assets are currently available for package-version from what was installed. They _should_ be the same, but things happen. - Avoids a query to get the installed assets before operating on them - Cons: - size/noise? Could be tens or hundreds of ids - migration? </details> ### More details **When are saved objects added?** During installation, after all other actions have succeeded, just before marking the save object as installed, we commit all the files from the package to ES https://github.com/elastic/kibana/blob/37f7b6ded747edb5cc487661b801c5e1c0a102a7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2F_install_package.ts#L193-L198 **When are documents removed from the index?** In the `removeInstallation` function which is called in response to a `DELETE /api/fleet/epm/packages/pkgkey` https://github.com/elastic/kibana/blob/37f7b6ded747edb5cc487661b801c5e1c0a102a7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2Fremove.ts#L72 or a failed package (re-)installation https://github.com/elastic/kibana/blob/bf068739acce044ac27902253e8fc31df621f081/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2Finstall.ts#L145 **How are we using these assets?** We're not, currently. Here's an example showing how we could update [`getFileHandler`](https://github.com/elastic/kibana/blob/514b50e4c2d7a3be79d77e73838ff57b6cf1304a/x-pack%2Fplugins%2Ffleet%2Fserver%2Froutes%2Fepm%2Fhandlers.ts#L101) to check for local assets before reaching out to the Registry if we wished. It's not DRY, but it does work ```typescript const esDocRoot = `http://elastic:changeme@localhost:9200/${PACKAGE_ASSETS_INDEX_NAME}/_doc`; const escapedDocId = encodeURIComponent(`${pkgName}-${pkgVersion}/${filePath}`); const esRes = await fetch(`${esDocRoot}/${escapedDocId}`); const esJson = await esRes.json(); if (esJson.found) { const asset: PackageAsset = esJson._source; const body = asset.data_utf8 || Buffer.from(asset.data_base64, 'base64'); return response.ok({ body, headers: { 'content-type': asset.media_type, // should add our own `cache-control` header here // kibana default is prevents caching: `private, no-cache, no-store, must-revalidate` // elastic#83631 }, }); } ``` ### Checklist _updated tests to include new saved object output, no tests added yet_ - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
💚 Build SucceededMetrics [docs]Distributable file count
Page load bundle
Saved Objects .kibana field count
To update your PR or re-run it, just comment with: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backports the following commits to 7.x: