Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] [RAC] Add alerts count table to Security Solutions alerts page #106358

Merged
merged 8 commits into from
Jul 29, 2021

Conversation

machadoum
Copy link
Member

@machadoum machadoum commented Jul 21, 2021

Summary

issue: https://github.com/elastic/security-team/issues/1273

Add a table that shows the count of alerts for each stacked by field. The table is hidden under the "Count" Tab.

Changes:

  • Refactor AlertHistogram to allow AlertsCount to reuse its code.
  • Create AlertsCount panel.
  • Create a new query for AlertsCount that fetches 10.000 items.
  • Add AlertsCount panel to alerts page.

Screenshot 2021-07-21 at 15 45 59

Checklist

Delete any items that are not applicable to this PR.

@machadoum machadoum force-pushed the rac-alerts-table-issue-1273-2 branch from 5b21bc7 to 511be2b Compare July 21, 2021 07:52
@machadoum machadoum changed the title Rac alerts table issue 1273 2 [Security Solution] [RAC] Add alerts count table to Security Solutions alerts page Jul 21, 2021
@machadoum machadoum marked this pull request as ready for review July 21, 2021 09:35
@machadoum machadoum requested a review from a team as a code owner July 21, 2021 09:35
@machadoum machadoum self-assigned this Jul 21, 2021
@machadoum machadoum added release_note:enhancement Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting Security Solution Threat Hunting Team v7.15.0 v8.0.0 labels Jul 21, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

@machadoum machadoum force-pushed the rac-alerts-table-issue-1273-2 branch 3 times, most recently from cfe3fc1 to ff91116 Compare July 21, 2021 13:01
@mdefazio
Copy link
Contributor

Running into some issues getting this PR going, so apologies if these are obvious.

Will there be 10,000 results in the 'Count' table and it would scroll? Or is that just the query and we only show top 5 results?

Do we need to say 'Trend' or 'Count'. I would think the histogram might be clear that it's alert count breakdown over time. Whereas we may need a title for the table (again, suggesting Top 5 {Rules, Hosts...} or something there). Ideally we would also be able to combine the dropdowns into a single that changes both.

I know I'm reiterating some conversations here, but just getting them down for posterity. And I think it's likely worthwhile to bring this back into design for the 2nd iteration of this. So I'm mainly trying to figure out a more simple first solution to this section.

cc/ @paulewing

@michaelolo24
Copy link
Contributor

@elasticmachine merge upstream

@machadoum machadoum force-pushed the rac-alerts-table-issue-1273-2 branch from 93bbd91 to 3affd8d Compare July 22, 2021 13:43
@machadoum
Copy link
Member Author

machadoum commented Jul 22, 2021

Will there be 10,000 results in the 'Count' table and it would scroll? Or is that just the query and we only show top 5 results?

Yes, we are displaying a max of 10,000 items and allowing the user to scroll. However, it might be harmful performance-wise to show 10.000 items. We are investigating if we can easily hide rows that are not visible. @paulewing Would it be possible to set a lower limit like "top 5" or "top 100" rows?

@snide snide requested a review from a team July 26, 2021 12:38
@machadoum machadoum requested a review from dhurley14 July 27, 2021 13:27
@machadoum machadoum force-pushed the rac-alerts-table-issue-1273-2 branch from 3affd8d to ef20841 Compare July 27, 2021 13:33
@machadoum
Copy link
Member Author

@dhurley14 I implemented the feature using /routes/signals/query_signals_route.ts API as we agreed. It only required a small change on BE. Could you check if this feature implementation doesn't impact your work?

value: AlertsStackByField;
}

export type AlertsStackByField =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe in a follow-up PR can you add a comment saying why these fields specifically? Thanks!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I asked myself the same question. "Why these fields?".
The only answer I got is that these values are hardcoded since the first implementation. The original PR that introduced this feature lists them #53742.

Maybe someone else could help us here. 😵‍💫

Copy link
Contributor

@michaelolo24 michaelolo24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally and works great. Switching between the field types was smooth, 👍🏾

Copy link
Contributor

@dhurley14 dhurley14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Contributor

@mdefazio mdefazio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one nit as far as using numbers and not Eui variables.

One change I'd like to propose is to use popovers instead of the form dropdowns. My reason for this is primarily to reduce the amount of exposed dropdowns and treat 'configuration' options the same across the page. These are based off new mockups, so I apologize for the late updates.

Here's the proposed update:
image

And with the popover selector:
image

The button label that triggers the popover would then be updated with the selected field value.

I do not have the search option shown for the popover, but I think if the field options list grows any further, we may want to consider that.

If you feel this is better suited for another PR, I'm fine with that and can approve this (with the small nit) and repost in a new PR.

@machadoum machadoum force-pushed the rac-alerts-table-issue-1273-2 branch from e965f85 to 51db368 Compare July 28, 2021 11:22
@machadoum
Copy link
Member Author

machadoum commented Jul 28, 2021

If you feel this is better suited for another PR, I'm fine with that and can approve this (with the small nit) and repost in a new PR.

Hi @mdefazio, thanks for taking a look at this PR.

It would be better to implement the popover in another PR. So, we can unblock this one. Btw, I replaced the pixel size padding for an EuiVariable.

Soon, I will create another PR and add you as a reviewer. It would be helpful if you could send me the link to the Figma mockup.

Copy link
Contributor

@mdefazio mdefazio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Popover styling edits can be done in a separate PR. Thanks!

@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky


Test Failures

Kibana Pipeline / general / Firefox UI Functional Tests.test/functional/apps/discover/_field_data·ts.discover app discover tab field data the search term should be highlighted in the field data

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 8 times on tracked branches: https://github.com/elastic/kibana/issues/100437

[00:00:00]       │
[00:08:19]         └-: discover app
[00:08:19]           └-> "before all" hook in "discover app"
[00:08:19]           └-> "before all" hook in "discover app"
[00:08:19]           └-: discover tab
[00:08:19]             └-> "before all" hook in "discover tab"
[00:08:19]             └-> "before all" hook in "discover tab"
[00:08:19]               │ debg Cleaning all saved objects { space: undefined }
[00:08:19]               │ info deleting batch of 1 objects
[00:08:20]               │ succ deleted 1 objects
[00:08:20]               │ debg resolved import for test/functional/fixtures/kbn_archiver/discover.json to /dev/shm/workspace/parallel/7/kibana/test/functional/fixtures/kbn_archiver/discover.json
[00:08:20]               │ info importing 2 saved objects { space: undefined }
[00:08:20]               │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/gAlLx6F1QrimShUB80Mg3Q] update_mapping [_doc]
[00:08:21]               │ succ import success
[00:08:21]               │ info [test/functional/fixtures/es_archiver/logstash_functional] Loading "mappings.json"
[00:08:21]               │ info [test/functional/fixtures/es_archiver/logstash_functional] Loading "data.json.gz"
[00:08:21]               │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [logstash-2015.09.22] creating index, cause [api], templates [], shards [1]/[0]
[00:08:21]               │ info [test/functional/fixtures/es_archiver/logstash_functional] Created index "logstash-2015.09.22"
[00:08:21]               │ debg [test/functional/fixtures/es_archiver/logstash_functional] "logstash-2015.09.22" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:08:21]               │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [logstash-2015.09.20] creating index, cause [api], templates [], shards [1]/[0]
[00:08:21]               │ info [test/functional/fixtures/es_archiver/logstash_functional] Created index "logstash-2015.09.20"
[00:08:21]               │ debg [test/functional/fixtures/es_archiver/logstash_functional] "logstash-2015.09.20" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:08:21]               │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [logstash-2015.09.21] creating index, cause [api], templates [], shards [1]/[0]
[00:08:21]               │ info [test/functional/fixtures/es_archiver/logstash_functional] Created index "logstash-2015.09.21"
[00:08:21]               │ debg [test/functional/fixtures/es_archiver/logstash_functional] "logstash-2015.09.21" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:08:31]               │ info progress: 13542
[00:08:31]               │ info [test/functional/fixtures/es_archiver/logstash_functional] Indexed 4633 docs into "logstash-2015.09.22"
[00:08:31]               │ info [test/functional/fixtures/es_archiver/logstash_functional] Indexed 4757 docs into "logstash-2015.09.20"
[00:08:31]               │ info [test/functional/fixtures/es_archiver/logstash_functional] Indexed 4614 docs into "logstash-2015.09.21"
[00:08:31]               │ debg replacing kibana config doc: {"defaultIndex":"logstash-*","discover:searchFieldsFromSource":true}
[00:08:32]               │ debg applying update to kibana config: {"timepicker:timeDefaults":"{ \"from\": \"2015-09-18T06:31:44.000Z\", \"to\": \"2015-09-23T18:31:44.000Z\"}"}
[00:08:33]               │ debg navigating to discover url: http://localhost:6171/app/discover#/
[00:08:33]               │ debg navigate to: http://localhost:6171/app/discover#/
[00:08:33]               │ debg browser[log] "^ A single error about an inline script not firing due to content security policy is expected!"
[00:08:33]               │ debg ... sleep(700) start
[00:08:34]               │ debg ... sleep(700) end
[00:08:34]               │ debg returned from get, calling refresh
[00:08:35]               │ debg browser[log] "^ A single error about an inline script not firing due to content security policy is expected!"
[00:08:35]               │ debg currentUrl = http://localhost:6171/app/discover#/
[00:08:35]               │          appUrl = http://localhost:6171/app/discover#/
[00:08:35]               │ debg TestSubjects.find(kibanaChrome)
[00:08:35]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:08:38]               │ debg ... sleep(501) start
[00:08:38]               │ debg ... sleep(501) end
[00:08:39]               │ debg in navigateTo url = http://localhost:6171/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:%272015-09-18T06:31:44.000Z%27,to:%272015-09-23T18:31:44.000Z%27))&_a=(columns:!(_source),filters:!(),index:%27logstash-*%27,interval:auto,query:(language:kuery,query:%27%27),sort:!(!(%27@timestamp%27,desc)))
[00:08:39]               │ debg --- retry.tryForTime error: URL changed, waiting for it to settle
[00:08:39]               │ debg ... sleep(501) start
[00:08:40]               │ debg ... sleep(501) end
[00:08:40]               │ debg in navigateTo url = http://localhost:6171/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:%272015-09-18T06:31:44.000Z%27,to:%272015-09-23T18:31:44.000Z%27))&_a=(columns:!(_source),filters:!(),index:%27logstash-*%27,interval:auto,query:(language:kuery,query:%27%27),sort:!(!(%27@timestamp%27,desc)))
[00:08:40]             └-: field data
[00:08:40]               └-> "before all" hook for "search php should show the correct hit count"
[00:08:40]               └-> search php should show the correct hit count
[00:08:40]                 └-> "before each" hook: global before each for "search php should show the correct hit count"
[00:08:40]                 │ debg QueryBar.setQuery(php)
[00:08:40]                 │ debg TestSubjects.click(queryInput)
[00:08:40]                 │ debg Find.clickByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:40]                 │ debg Find.findByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:40]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/gAlLx6F1QrimShUB80Mg3Q] update_mapping [_doc]
[00:08:42]                 │ debg TestSubjects.getAttribute(queryInput, value, tryTimeout=120000, findTimeout=10000)
[00:08:42]                 │ debg TestSubjects.find(queryInput)
[00:08:42]                 │ debg Find.findByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:43]                 │ debg QueryBar.submitQuery
[00:08:43]                 │ debg TestSubjects.click(queryInput)
[00:08:43]                 │ debg Find.clickByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:43]                 │ debg Find.findByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:44]                 │ debg isGlobalLoadingIndicatorVisible
[00:08:44]                 │ debg TestSubjects.exists(globalLoadingIndicator)
[00:08:44]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:08:44]                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:08:44]                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:08:45]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/gAlLx6F1QrimShUB80Mg3Q] update_mapping [_doc]
[00:08:47]                 │ debg isGlobalLoadingIndicatorVisible
[00:08:47]                 │ debg TestSubjects.exists(globalLoadingIndicator)
[00:08:47]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:08:48]                 │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:08:49]                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:08:49]                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:08:49]                 │ debg TestSubjects.getVisibleText(discoverQueryHits)
[00:08:49]                 │ debg TestSubjects.find(discoverQueryHits)
[00:08:49]                 │ debg Find.findByCssSelector('[data-test-subj="discoverQueryHits"]') with timeout=10000
[00:08:49]                 └- ✓ pass  (9.1s) "discover app discover tab field data search php should show the correct hit count"
[00:08:49]               └-> the search term should be highlighted in the field data
[00:08:49]                 └-> "before each" hook: global before each for "the search term should be highlighted in the field data"
[00:08:49]                 │ debg QueryBar.setQuery(php)
[00:08:49]                 │ debg TestSubjects.click(queryInput)
[00:08:49]                 │ debg Find.clickByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:49]                 │ debg Find.findByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:49]                 │ debg TestSubjects.getAttribute(queryInput, value, tryTimeout=120000, findTimeout=10000)
[00:08:49]                 │ debg TestSubjects.find(queryInput)
[00:08:49]                 │ debg Find.findByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:49]                 │ debg QueryBar.submitQuery
[00:08:49]                 │ debg TestSubjects.click(queryInput)
[00:08:49]                 │ debg Find.clickByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:49]                 │ debg Find.findByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:50]                 │ debg isGlobalLoadingIndicatorVisible
[00:08:50]                 │ debg TestSubjects.exists(globalLoadingIndicator)
[00:08:50]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:08:50]                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:08:50]                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:08:52]                 │ debg TestSubjects.moveMouseTo(field-extension)
[00:08:52]                 │ debg TestSubjects.find(field-extension)
[00:08:52]                 │ debg Find.findByCssSelector('[data-test-subj="field-extension"]') with timeout=10000
[00:08:52]                 │ debg TestSubjects.click(fieldToggle-extension)
[00:08:52]                 │ debg Find.clickByCssSelector('[data-test-subj="fieldToggle-extension"]') with timeout=10000
[00:08:52]                 │ debg Find.findByCssSelector('[data-test-subj="fieldToggle-extension"]') with timeout=10000
[00:08:53]                 │ debg TestSubjects.find(docTable)
[00:08:53]                 │ debg Find.findByCssSelector('[data-test-subj="docTable"]') with timeout=10000
[00:08:53]                 │ debg Firefox stdout: 1627473444430	Marionette	WARN	TimedPromise timed out after 500 ms: stacktrace:
[00:08:53]                 │      TimedPromise/<@chrome://marionette/content/sync.js:228:19
[00:08:53]                 │      TimedPromise@chrome://marionette/content/sync.js:213:10
[00:08:53]                 │      interaction.flushEventLoop@chrome://marionette/content/interaction.js:418:10
[00:08:53]                 │      webdriverClickElement@chrome://marionette/content/interaction.js:179:31
[00:08:53]                 │
[00:08:53]                 │ info Taking screenshot "/dev/shm/workspace/parallel/7/kibana/test/functional/screenshots/failure/discover app discover tab field data the search term should be highlighted in the field data.png"
[00:08:54]                 │ info Current URL is: http://localhost:6171/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:%272015-09-18T06:31:44.000Z%27,to:%272015-09-23T18:31:44.000Z%27))&_a=(columns:!(extension),filters:!(),index:%27logstash-*%27,interval:auto,query:(language:kuery,query:php),sort:!(!(%27@timestamp%27,desc)))
[00:08:54]                 │ info Saving page source to: /dev/shm/workspace/parallel/7/kibana/test/functional/failure_debug/html/discover app discover tab field data the search term should be highlighted in the field data.html
[00:08:54]                 └- ✖ fail: discover app discover tab field data the search term should be highlighted in the field data
[00:08:54]                 │      StaleElementReferenceError: The element reference of <mark> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
[00:08:54]                 │       at Object.throwDecodedError (node_modules/selenium-webdriver/lib/error.js:550:15)
[00:08:54]                 │       at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:565:13)
[00:08:54]                 │       at Executor.execute (node_modules/selenium-webdriver/lib/http.js:491:26)
[00:08:54]                 │       at processTicksAndRejections (internal/process/task_queues.js:95:5)
[00:08:54]                 │       at Task.exec (test/functional/services/remote/prevent_parallel_calls.ts:28:20)
[00:08:54]                 │ 
[00:08:54]                 │ 

Stack Trace

StaleElementReferenceError: The element reference of <mark> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
    at Object.throwDecodedError (node_modules/selenium-webdriver/lib/error.js:550:15)
    at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:565:13)
    at Executor.execute (node_modules/selenium-webdriver/lib/http.js:491:26)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at Task.exec (test/functional/services/remote/prevent_parallel_calls.ts:28:20) {
  remoteStacktrace: 'WebDriverError@chrome://marionette/content/error.js:181:5\n' +
    'StaleElementReferenceError@chrome://marionette/content/error.js:442:5\n' +
    'get@chrome://marionette/content/element.js:239:13\n' +
    'evaluate.fromJSON@chrome://marionette/content/evaluate.js:242:26\n' +
    'dispatch/</req<@chrome://marionette/content/listener.js:101:29\n' +
    'dispatch/<@chrome://marionette/content/listener.js:100:15\n'
}

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 2323 2331 +8

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 6.4MB 6.4MB +5.9KB
Unknown metric groups

References to deprecated APIs

id before after diff
securitySolution 800 808 +8

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @machadoum

@machadoum machadoum merged commit 12271f3 into elastic:master Jul 29, 2021
machadoum added a commit to machadoum/kibana that referenced this pull request Jul 29, 2021
…s alerts page (elastic#106358)

* Add alerts count table to Security Solutions alerts page

* Refactor alerts_histogram_panel to reuse logic on alerts_count

* Improve AlertsCount mobile layout

# Conflicts:
#	x-pack/plugins/translations/translations/zh-CN.json
machadoum added a commit that referenced this pull request Jul 29, 2021
…s alerts page (#106358) (#107124)

* Add alerts count table to Security Solutions alerts page

* Refactor alerts_histogram_panel to reuse logic on alerts_count

* Improve AlertsCount mobile layout

# Conflicts:
#	x-pack/plugins/translations/translations/zh-CN.json
streamich pushed a commit to vadimkibana/kibana that referenced this pull request Aug 8, 2021
…s alerts page (elastic#106358)

* Add alerts count table to Security Solutions alerts page

* Refactor alerts_histogram_panel to reuse logic on alerts_count

* Improve AlertsCount mobile layout
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:enhancement Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting Security Solution Threat Hunting Team v7.15.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants