[Search Sessions] Save all sessions, with persisted flag #89570

lizozom · 2021-01-28T14:49:47Z

Summary

Depends on #87966 (@kibana-app and @elastic/logs-ui no need to review)

This PR introduces two changes to the Search Sessions feature architecture:

Previously, our code was holding async search ids in memory until they are saved by the user. This PR instead, immediately saves any search session as a saved object.
Previously ,we were saving all async searches for 1m and extending them as the browser polls for results. This PR instead stores async searches (that are a part of a session) for 7d by default.

Problems solved

There was a small risk of data loss if a server is shut down before a user saves a search session
There was a risk of race conditions while extending the async searches
There is no API for extending async_searches on behalf of the user, meaning we had to store the user's credientials somehow to extend on their behalf. This resolves this problem too.
The previous implementation had a risk of using too much memory. We don't use server memory anymore in this PR.

Limitations and things to be aware of.

Elasticsearch now relies on Kibana to clear any unsaved sessions. Otherwise data will remain in the .async_search index for 7 days. This is resolved by using the monitoring task to clear out abandoned search sessions.
- Running sessions, will be killed and cleared after 1m of inactivity (set in notTouchedTimeout).
- Completed sessions will be cleared after 5m (Set in completedTimeout), to allow users to "Send to backround" on sessions that have already finished running.
Since savedObjectsClient does not expose an atomic updateOrCreate function using upset, we had to implement a logic using retries. This might result in less than optimal performance.
There will be many more search sessions saved objects now. To handle that, this PR adds support for saved object pagination (which was previously postponed to 7.13)

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

…d state

…-all-sessions

…a into sessions/save-all-sessions

…-all-sessions

lizozom · 2021-02-03T13:00:57Z

@elastic/kibana-app and @elastic/logs-ui please ignore the review request.
It happened due to merging another PR into this one. I will rebase after #87966 is merged.

…-all-sessions

kibanamachine · 2021-02-03T19:51:28Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 07523f3

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`data`	614	613	-1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`dataEnhanced`	150.8KB	150.6KB	-265.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`data`	798.4KB	797.9KB	-500.0B
`dataEnhanced`	35.9KB	33.6KB	-2.4KB
total			-2.9KB

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
`search-session`	8	10	+2

History

💚 Build #103426 succeeded 592c81b
💔 Build #103379 failed c890574
💚 Build #103335 succeeded 19fa7e1
💚 Build #102998 succeeded d61fa45
💚 Build #102993 succeeded fc29c7f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

lukasolson

LGTM!

@lukasolson

* [data.search] Add search session methods to search service contract * Fix types * Fix tests and switch to cancel * Update docs * Fix types/tests * Fix tests * Update status of SO before cancelling search requests * Add API integration test * Fix types * Update expiration route to use config defaultExpiration * Fix test * Update docs * New logic for extend * Remove declare module * Search Sessions: Unskip Flaky Functional Test * Review feedback * fix ts * Save all search sessions and then manage them based on their persisted state * Get default search session expiration from config * randomize sleep time * fix test * Remove test that is no longer valid * fix test * Make sure we poll, and dont persist, searches not in the context of a session * Added keepalive unit tests * fix ts * code review @lukasolson * ts * More tests, rename onScreenTimeout to completedTimeout * lint * lint * Delete async seaches * Support saved object pagination Fix get search status tests * better PersistedSearchSessionSavedObjectAttributes ts * test titles * Fix undefined bug * Remove runAt from monitoring task Increase testing trackingInterval (caused bug) * support workload histograms that take into account overdue tasks * Update touched when changing session status to complete \ error * removed test * Updated management test data * Rename configs * delete tap first add comments * Use DataRequestHandlerContext in maps * ts * Fixed ts Co-authored-by: Lukas Olson <olson.lukas@gmail.com> Co-authored-by: Timothy Sullivan <tsullivan@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Anton Dosov <anton.dosov@elastic.co> Co-authored-by: Gidi Meir Morris <github@gidi.io>

@lukasolson

…0236) * [data.search] Add search session methods to search service contract * Fix types * Fix tests and switch to cancel * Update docs * Fix types/tests * Fix tests * Update status of SO before cancelling search requests * Add API integration test * Fix types * Update expiration route to use config defaultExpiration * Fix test * Update docs * New logic for extend * Remove declare module * Search Sessions: Unskip Flaky Functional Test * Review feedback * fix ts * Save all search sessions and then manage them based on their persisted state * Get default search session expiration from config * randomize sleep time * fix test * Remove test that is no longer valid * fix test * Make sure we poll, and dont persist, searches not in the context of a session * Added keepalive unit tests * fix ts * code review @lukasolson * ts * More tests, rename onScreenTimeout to completedTimeout * lint * lint * Delete async seaches * Support saved object pagination Fix get search status tests * better PersistedSearchSessionSavedObjectAttributes ts * test titles * Fix undefined bug * Remove runAt from monitoring task Increase testing trackingInterval (caused bug) * support workload histograms that take into account overdue tasks * Update touched when changing session status to complete \ error * removed test * Updated management test data * Rename configs * delete tap first add comments * Use DataRequestHandlerContext in maps * ts * Fixed ts Co-authored-by: Lukas Olson <olson.lukas@gmail.com> Co-authored-by: Timothy Sullivan <tsullivan@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Anton Dosov <anton.dosov@elastic.co> Co-authored-by: Gidi Meir Morris <github@gidi.io> Co-authored-by: Lukas Olson <olson.lukas@gmail.com> Co-authored-by: Timothy Sullivan <tsullivan@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Anton Dosov <anton.dosov@elastic.co> Co-authored-by: Gidi Meir Morris <github@gidi.io>

lukasolson and others added 25 commits January 11, 2021 20:39

[data.search] Add search session methods to search service contract

c22024f

Merge branch 'master' into search-session-api

1046c54

Fix types

7040716

Fix tests and switch to cancel

eeddf84

Update docs

f09eb38

Merge branch 'master' into search-session-api

e6124af

Fix types/tests

e160019

Fix tests

3c08c3d

Update status of SO before cancelling search requests

0cd92ed

Add API integration test

f1a1d0b

Merge branch 'master' into search-session-api

552e545

Merge branch 'master' into search-session-api

ee6e337

Fix types

32315b7

Update expiration route to use config defaultExpiration

542de95

Fix test

5b22249

Update docs

66420b4

New logic for extend

8a5e3d3

Remove declare module

71582c7

Search Sessions: Unskip Flaky Functional Test

03f413a

Review feedback

65840b3

Merge branch 'master' of github.com:elastic/kibana into pr/87966

fdb940b

fix ts

b2bce4b

Merge branch 'master' into flaky/search-sessions

2e09f56

Save all search sessions and then manage them based on their persiste…

cb1de74

…d state

Get default search session expiration from config

2b3b90e

lizozom changed the title ~~Sessions/save all sessions~~ [Search Sessions] Save all sessions, with persisted flag Jan 28, 2021

lizozom self-assigned this Jan 28, 2021

lizozom added NeededFor:AppServices v7.12.0 v8.0.0 labels Jan 28, 2021

Liza K added 2 commits February 3, 2021 12:03

Merge branch 'master' of github.com:elastic/kibana into sessions/save…

cf45bb3

…-all-sessions

Merge branch 'sessions/save-all-sessions' of github.com:lizozom/kiban…

19fa7e1

…a into sessions/save-all-sessions

lizozom mentioned this pull request Feb 3, 2021

[Search Sessions] Use sync config #90138

Merged

9 tasks

Liza K added 6 commits February 3, 2021 12:53

Merge branch 'master' of github.com:elastic/kibana into pr/87966

5c564c2

Merge branch 'pr/87966' into sessions/save-all-sessions

2b5c2aa

Use DataRequestHandlerContext in maps

3eb8f01

ts

6462da6

Merge branch 'pr/87966' into sessions/save-all-sessions

4b89dbf

Merge branch 'master' of github.com:elastic/kibana into sessions/save…

c890574

…-all-sessions

lizozom requested a review from a team February 3, 2021 12:59

lizozom requested a review from a team as a code owner February 3, 2021 12:59

Liza K added 2 commits February 3, 2021 16:49

Fixed ts

592c81b

Merge branch 'master' of github.com:elastic/kibana into sessions/save…

07523f3

…-all-sessions

lizozom requested review from lukasolson and removed request for a team February 3, 2021 16:01

lukasolson approved these changes Feb 3, 2021

View reviewed changes

lizozom merged commit 7fbcf68 into elastic:master Feb 3, 2021

lizozom mentioned this pull request Feb 3, 2021

[7.x] [Search Sessions] Save all sessions, with persisted flag (#89570) #90236

Merged

This was referenced Feb 9, 2021

Extend async search keep alive elastic/elasticsearch#67877

Merged

[Discuss][Search Sessions] Synchronizing async_search keepslive updates #88599

Closed

This was referenced Feb 9, 2021

[Search Sessions] Adds "Kibana / Search Sessions" privilege #90750

Closed

[Search Session] Control "Kibana / Search Sessions" management section by privileges #90818

Merged

Dosant mentioned this pull request Feb 18, 2021

Failing test: X-Pack API Integration Tests.x-pack/test/api_integration/apis/search/session·ts - apis search search session should sync search ids into not persisted session #90332

Closed

chriscressman added the Team:EnterpriseSearch label Jan 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Search Sessions] Save all sessions, with persisted flag #89570

[Search Sessions] Save all sessions, with persisted flag #89570

lizozom commented Jan 28, 2021 •

edited

Loading

lizozom commented Feb 3, 2021

kibanamachine commented Feb 3, 2021

lukasolson left a comment

[Search Sessions] Save all sessions, with persisted flag #89570

[Search Sessions] Save all sessions, with persisted flag #89570

Conversation

lizozom commented Jan 28, 2021 • edited Loading

Summary

Problems solved

Limitations and things to be aware of.

Checklist

For maintainers

lizozom commented Feb 3, 2021

kibanamachine commented Feb 3, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

Saved Objects .kibana field count

History

lukasolson left a comment

Choose a reason for hiding this comment

lizozom commented Jan 28, 2021 •

edited

Loading