[Fleet][EPM] Save installed package assets in ES

[jfsiii]

Summary

Store package assets (from Registry or local upload) in Elasticsearch. Related to proposal issue & document

• New epm-packages-assets saved objects are stored on .kibana index, like our existing saved object epm-packages
• Asset id is uuid v5 based on the package name, package version & file path. See 1974324
• Add a list of IDs of all the installed assets, to epm-packages saved object. Like the existing installed_ properties. Example from a test

Mapping for new Saved Object

kibana/x-pack/plugins/fleet/server/saved_objects/index.ts

Lines 329 to 339 in 37f7b6d

	mappings: {
	properties: {
	package_name: { type: 'keyword' },
	package_version: { type: 'keyword' },
	install_source: { type: 'keyword' },
	asset_path: { type: 'keyword' },
	media_type: { type: 'keyword' },
	data_utf8: { type: 'text', index: false },
	data_base64: { type: 'binary' },
	},
	},

Additional property on existing epm-packages Saved Object

kibana/x-pack/plugins/fleet/server/saved_objects/index.ts

Lines 306 to 312 in c4f27ab

	package_assets: {
	type: 'nested',
	properties: {
	id: { type: 'keyword' },
	type: { type: 'keyword' },
	},
	},

I don't think the saved object changes are strictly required. It can be removed without changing much about how things work

• Pros:
  - Preserves accurate record of the assets added at installation time. Separates what assets are currently available for package-version from what was installed. They should be the same, but things happen.
  - Avoids a query to get the installed assets before operating on them
• Cons:
  - size/noise? Could be tens or hundreds of ids
  - migration?

More details

When are saved objects added?
During installation, after all other actions have succeeded, just before marking the save object as installed, we commit all the files from the package to ES

kibana/x-pack/plugins/fleet/server/services/epm/packages/_install_package.ts

Lines 193 to 198 in 37f7b6d

	const packageAssetRefs: PackageAssetReference[] = packageAssetResults.saved_objects.map(
	(result) => ({
	id: result.id,
	type: ASSETS_SAVED_OBJECT_TYPE,
	})
	);

When are documents removed from the index?

In the removeInstallation function which is called in response to a DELETE /api/fleet/epm/packages/pkgkey

kibana/x-pack/plugins/fleet/server/services/epm/packages/remove.ts

Line 72 in 37f7b6d

await removeArchiveEntries({ savedObjectsClient, refs: installation.package_assets });

or a failed package (re-)installation

kibana/x-pack/plugins/fleet/server/services/epm/packages/install.ts

Line 145 in bf06873

await removeInstallation({ savedObjectsClient, pkgkey, callCluster });

How are we using these assets?
We're not, currently. Here's an example showing how we could update getFileHandler to check for local assets before reaching out to the Registry if we wished. It's not DRY, but it does work

const esDocRoot = `http://elastic:changeme@localhost:9200/${PACKAGE_ASSETS_INDEX_NAME}/_doc`;
const escapedDocId = encodeURIComponent(`${pkgName}-${pkgVersion}/${filePath}`);
const esRes = await fetch(`${esDocRoot}/${escapedDocId}`);
const esJson = await esRes.json();
if (esJson.found) {
  const asset: PackageAsset = esJson._source;
  const body = asset.data_utf8 || Buffer.from(asset.data_base64, 'base64');
  return response.ok({
    body,
    headers: {
      'content-type': asset.media_type,
      // should add our own `cache-control` header here
      // kibana default is prevents caching: `private, no-cache, no-store, must-revalidate`
      // https://github.com/elastic/kibana/issues/83631
    },
  });
}

Checklist

updated tests to include new saved object output, no tests added yet

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[jfsiii]

Went with two fields because I thought we wanted binary type for the binary assets vs text but perhaps one field is possible/desirable?

[jfsiii]

Also considered mime_type. Happy to hear any comments / suggestions.

[nchaulet]

@jfsiii How are we going to consume these assets later? Do we want to use a generated id like pkgName:pkgVersion:path ?

[jfsiii]

How are we going to consume these assets later? Do we want to use a generated id like pkgName:pkgVersion:path ?

@nchaulet good question, perhaps we should discuss in #83426 so others see & comment?

I believe combining the pkgName, pkgVersion, path, and installSource properties should give only one result.

[nchaulet]

Looks good to me

[jfsiii]

@elasticmachine merge upstream

[neptunian]

I left a comment #83426 (comment) to see if there was some concern about having so many saved objects. If not, I'm happy for this to go in.

[jfsiii]

@elasticmachine merge upstream

[joshdover]

Does that mean you'd also suggest biasing towards using Saved Objects vs ES documents?

Yes!

Getting migrations seemed like a Good Thing ™️ but I didn't know about the impact that number (typically hundreds or thousands but could be tens of thousands) or size of documents.

Migrations only get applied to objects that have an actual migration function registered, so this should not really be an issue, even with 10k objects.

Distribution of the file sizes

Most of these look mostly small (it appears 75% of the packages are less than 500kb). Are there any known reasons these will get larger in the future?

Also this text from the proposal:

Or “why not store only the compressed/zip file?” Primarily because the zip file is a binary asset and Elasticsearch stores text. Despite the name, the ‘binary’ type in Elasticsearch is still a string. The archive would have to be converted to a Base64 encoded string which increases the size by about 30%. So a 1MB zip becomes a 1.3MB string and a 15MB zip becomes a nearly 20MB string.

Have we actually verified that a base64 encoded zip file is larger than the unencoded strings of all the individual files? It's possible that the compression on the zip outweighs the 30% increase from base64 encoding.

From a long-term maintenance standpoint, I think it's better if we can limit the number of documents in Elasticsearch to just the zip'd packages. This will make removing & reinstalling any corrupted packages much simpler for our support team in the case that something breaks. I also think the runtime of unzipping a 1MB file is quite small and I don't imagine this to be a common operation (maybe I'm wrong?) that we need to optimize for.

[jfsiii]

Getting migrations seemed like a Good Thing ™️ but I didn't know about the impact that number (typically hundreds or thousands but could be tens of thousands) or size of documents.

Migrations only get applied to objects that have an actual migration function registered, so this should not really be an issue, even with 10k objects.

I could have worded this more clearly. I meant that migrations seemed like a good reason to use them but, migrations aside, I didn't know if the number & size of objects we'd be creating would be a concern. Doesn't seem so.

Most of these look mostly small (it appears 75% of the packages are less than 500kb). Are there any known reasons these will get larger in the future?

That's size of the individual files. Here are the current archive sizes

screenshot

The distribution might say the same, but I expect we will see an increase in packages at the larger end as we open to the public and get different types of integrations (e.g. example logs

Have we actually verified that a base64 encoded zip file is larger than the unencoded strings of all the individual files? It's possible that the compression on the zip outweighs the 30% increase from base64 encoding.

In my mind it's less about total size store on disk as it is minimum & median request. It's less taxing to transfer & process several smaller strings vs one large one. Especially because (de)serialization is CPU-intensive and often blocking. In my experience with web performance, time spent in JSON.parse/stringify and other string operations is a primary culprit.

This will make removing & reinstalling any corrupted packages much simpler for our support team in the case that something breaks

This should always be possible via a single action, whether an http call, a "delete by" query, or action in the UI.

I also think the runtime of unzipping a 1MB file is quite small and I don't imagine this to be a common operation (maybe I'm wrong?) that we need to optimize for.

We'll use the local assets on the package details page to avoid hitting the registry for installed assets. We don't have to have them stored individually to do this. We could simply keep the unzipped buffers in memory as we do now. It's more about what I mentioned above about keeping the transaction small.

[joshdover]

You're correct that we decided to store the PDFs and PNGs created by Reporting in a separate index because of file size concerns. Currently, Reporting uses a new index per week to allow users to implement their own retention policy for how long to keep these reports around.

One difference here is that Reporting outputs could potentially grow indefinitely, whereas package assets are more finite. It seems unlikely that a user would install tens of thousands of packages (or that we'd ever even have that many in our registry). This makes file size potentially less of a concern, but I think we should verify it.

Could we do a simple test where we create 10,000 of these objects in the .kibana index with 1MB of zeros and measure the performance of Kibana and some basic queries (_search, GET /.kibana/_doc/<id>, etc.)? We have some load tests we could run to see if there is a performance impact compared to master.

In my mind it's less about total size store on disk as it is minimum & median request. It's less taxing to transfer & process several smaller strings vs one large one. Especially because (de)serialization is CPU-intensive and often blocking. In my experience with web performance, time spent in JSON.parse/stringify and other string operations is a primary culprit.

Makes sense, and I haven't spent time looking at how Fleet processes this data. If we store the decompressed output of files, wouldn't we still need to parse the JSON on the responses from ES? Even so, your point about minimum & median request times checks out. I thought the size distribution charts were the entire packages, not the files. I agree that decompressing & parsing 10MB+ during a request is not going to be a good experience.

[jfsiii]

@elasticmachine merge upstream

[jfsiii]

@elasticmachine merge upstream

[jfsiii]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: ec00e42

Metrics [docs]

Distributable file count

id	before	after	diff
default	46897	47662	+765

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	374.9KB	375.2KB	+311.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
epm-packages	15	18	+3
epm-packages-assets	-	8	+8
total			+11

History

• 💔 Build #92228 failed 78d2483
• 💚 Build #92064 succeeded 3a16056
• 💚 Build #91686 succeeded 4d42ee7
• 💚 Build #90738 succeeded 98dfeb5
• 💚 Build #90514 succeeded 4baccd5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[paul-tavares]

LGTM John. Thanks again for teh changes to the endpoint generator

[jfsiii]

How large of documents are we talking about for packages?

@kobelb I missed this question. I believe I addressed it in comments to @joshdover but here's some data on them

Zip files (not stored by this PR) ~50% from 100-200kB. ~30% 500kB-1MB. (1) 10MB file (aws)

Individual assets (stored by this PR). ~80% under 5kB. ~90% under 10kb. ~5% over 100kB. ~1% over 500kB. Only 30 files or 0.3% over 1MB

[jfsiii]

@joshdover I didn't get a chance to run the load tests with 10,000 Saved Objects, but I did run them with every currently available package installed. That resulted in 1951 saved objects

$ curl --user elastic:changeme "localhost:5601/cwl/api/saved_objects/_find?type=epm-packages-assets&per_page=5000" | jq '.total'
1951

This command

$ curl --user elastic:changeme "localhost:5601/cwl/api/saved_objects/_find?type=epm-packages-assets&per_page=5000" | jq '.saved_objects[] | .attributes | {name:.package_name, version:.package_version, source: .install_source, asset_path: .asset_path}' 

shows the assets installed (only name, version & path) https://gist.github.com/jfsiii/9b2c25fe204a7ebc1912692ed11c13e3

Here are the results of the load tests. They look quite similar to me
master.zip
pr.zip

[jfsiii]

I'm merging this so it's available for @skh, @neptunian, and others to use before Feature Freeze.

I believe I've addressed all the feedback from @kobelb, @joshdover, @legrego and others but please comment if there's anything remaining you'd like addressed. We'll add more tests and perhaps instrumentation in follow up PRs.

[legrego]

Do you need to add this new type to allSavedObjectTypes so that your users are granted access to these new saved objects? It's probably not strictly necessary since you still require superuser privileges for Fleet, and superusers can do everything regardless of what you specify here

kibana/x-pack/plugins/fleet/server/plugin.ts

Lines 104 to 112 in aa07f5c

	const allSavedObjectTypes = [
	OUTPUT_SAVED_OBJECT_TYPE,
	AGENT_POLICY_SAVED_OBJECT_TYPE,
	PACKAGE_POLICY_SAVED_OBJECT_TYPE,
	PACKAGES_SAVED_OBJECT_TYPE,
	AGENT_SAVED_OBJECT_TYPE,
	AGENT_EVENT_SAVED_OBJECT_TYPE,
	ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE,
	];