Skip to content

Commit

Permalink
[Security Solution][CTI] Investigation time enrichment UI (#103383)
Browse files Browse the repository at this point in the history
* Add pure fn and consuming hook to fetch event enrichment

It's not being invoked yet, but I've added a placeholder where it's
going.

* Move existing enrichment tests to new spec file

This is a rough copy/paste, I'll clean up as I flesh out the new tests.

* Move test constants into tests that use them

* style: declare FC function as an FC

* Extract some inline parsing logic into a helper function

And test it!

* Solidifying enrichment types on the backend

* Declares an enum for our types
* Sets type during indicator match rule enrichment
* Sets type during investigation-time enrichment

* WIP: Enrichment rows are rendered on the alerts summary

There are lots of TODOs here, but this implements the following:

* Fetching investigation-time enrichments from the backend
* Parsing existing enrichments from timeline data
* Merging the two enrichment types together, and rendering them in rows
  as specified

Much of the data-fetching is hardcoded, and this broke the existing
pattern with SummaryView/SummaryRow so that got a little messy; I may
end up just using my own EuiTable but we'll see.

Threat Intel tab is currently broken; that's up next.

* Updates ThreatDetailsView to accept an array of enrichments

The investigation-time enrichments are a little messy because they
contain all the non-ECS fields that indicators contain; other than that,
this is looking good.

Still need to add the new header, and potentially sort the fields.

* Sort our details fields

This promotes sanity for the user.

* Add "view threat intel data" button

This simply opens the threat intel tab.

* Implement header for threat details sections

* Add a basic jest "unit" test around ThreatSummaryView

* Fix remaining tests for components we modified

This also addresses a bug where we were not properly sorting new
enrichments by first_seen; this is covered under the tests that were
fixed.

* Filter out duplicate investigation-time enrichments

Because the enrichment endpoint is dumb and doesn't know about the
existing event or its enrichments, we need to merge these together on
the client to reduce noise and redundant data.

* Add inspect button to investigation enrichments

* Massages the response into the format that the inspect component uses
* Moves stateful fetching of query and persisting in redux to new, more
  specialized hook
* Moves existing enrichment hook to a more suitable location in
  containers/

* Fix failing unit tests

* indicator match rule now specifies `matched.type` as coming from the
  rule
* Inspecting the enrichment query requires use of the redux store, which
  was not previously mocked

* Fix existing CTI cypress tests

This covers the basics of the Alert Summary and Threat Intel tabs; the
investigation-time enrichment functionality is up next.

* Adds a cypress test exercising investigation time enrichment

* Loads more indicators (filebeat data, `threat_indicator2` archive)
  AFTER the rule has executed
* Asserts that those indicators are also found on the alert summary.

* Populate event enrichment call with actual alert fields

This was previously hardcoded during development.

* Add a new field to our suspicious event to trigger enrichment

The existing myhash field will generate an alert due to the way the rule
is written, but the alert had no other fields that would match the
investigation time enrichment. This gives it a source.ip, and updates
the indicator to match.

* Only fetch enrichments data if there are valid event fields

If none of the alert's fields would be relevant to the enrichment query,
then we don't make the request at all.

* Update enrichments matched.typed in integration tests

This field was updated to reflect the source of the match, in this case:
indicator match rules.

* Ensure draggable fields are unique in a multi-match scenario

If a given field matched multiple indicators, then the previous
contextId was not unique as it was based on field/value that matched.
Adding provider to the mix would fix it, except that we're not
guaranteed to have a provider.

I've added both provider (if present) and an index value to the key to
ensure that it's unique.

* Simplify types

This field can never be null, as we always set it in our response.

* Move helper functioons out of shared location and into consuming component

These are unlikely to be used elsewhere.

* Clean up data parsing logic using reduce

This obviates the need for our filter/guard function and the extra loop
that it entails. We have to specify the return value of our reduce fn,
however, but that's mostly equivalent to our type guard.

* Move our general function into a general location

* Extract the concept of "enrichment identifiers"

This was already partially codified with 'buildEnrichmentId,' which is
used to dedup enrichments; this extends the idea to all fields that
could uniquely identify a given indicator.

* Use existing constant as the source of our enrichments query

This is now used by both the overview card and the enrichment query.

* Codify our default enrichment lookback as constants

* Remove unnecessary flexbox

The generic SummaryView component previously had to deal with
multi-valued CTI fields, representing the multiple values coming from
the multiple nested objects with that field.

However, with the new UI we no longer have that constraint, and so the
default columnar style, and the corresponding overriding styles, are no
longer necessary.

* Filter out partial responses in the event enrichment observable

The UI does not currently handle these. We need to test the behavior of
long-running queries with this filter, but this should simplify the
behavior to complete/error until we handle partial responses.

* Display placeholders while event enrichment is loading

Displays a loading spinner in the Threat Intel tab title, and some
loading lines where the enrichments summary is.

* Update our indicator data to be within the last 30 days

This fixes our cypress test, but it's going to start failing again in 30
days. However, by that time I'll have implemented the absolute data
picker, which will allow for a more comprehensive test in addition to us
sidestepping this issue.

* Fix type error with our details tabs

The name prop on a Tab will be rendered as a node, so both strings and
elements are acceptable. This relaxes the types to inherit from the
component itself.

* Fix failing jest tests

The addition of our filtering of the search observable broke this test,
since we now need to implement the search observable.

Rather than do that, we'll instead mock our local hook as that's more
likely to change.
  • Loading branch information
rylnd committed Jun 30, 2021
1 parent 0fb21c4 commit 569c209
Show file tree
Hide file tree
Showing 40 changed files with 2,662 additions and 650 deletions.
18 changes: 10 additions & 8 deletions x-pack/plugins/security_solution/common/cti/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { INDICATOR_DESTINATION_PATH } from '../constants';

export const MATCHED_ATOMIC = 'matched.atomic';
export const MATCHED_FIELD = 'matched.field';
export const MATCHED_ID = 'matched.id';
export const MATCHED_TYPE = 'matched.type';
export const INDICATOR_MATCH_SUBFIELDS = [MATCHED_ATOMIC, MATCHED_FIELD, MATCHED_TYPE];

Expand All @@ -18,11 +19,12 @@ export const INDICATOR_MATCHED_TYPE = `${INDICATOR_DESTINATION_PATH}.${MATCHED_T

export const EVENT_DATASET = 'event.dataset';
export const EVENT_REFERENCE = 'event.reference';
export const EVENT_URL = 'event.url';
export const PROVIDER = 'provider';
export const FIRSTSEEN = 'first_seen';

export const INDICATOR_DATASET = `${INDICATOR_DESTINATION_PATH}.${EVENT_DATASET}`;
export const INDICATOR_EVENT_URL = `${INDICATOR_DESTINATION_PATH}.event.url`;
export const INDICATOR_EVENT_URL = `${INDICATOR_DESTINATION_PATH}.${EVENT_URL}`;
export const INDICATOR_FIRSTSEEN = `${INDICATOR_DESTINATION_PATH}.${FIRSTSEEN}`;
export const INDICATOR_LASTSEEN = `${INDICATOR_DESTINATION_PATH}.last_seen`;
export const INDICATOR_PROVIDER = `${INDICATOR_DESTINATION_PATH}.${PROVIDER}`;
Expand All @@ -37,13 +39,10 @@ export const CTI_ROW_RENDERER_FIELDS = [
INDICATOR_PROVIDER,
];

export const SORTED_THREAT_SUMMARY_FIELDS = [
INDICATOR_MATCHED_FIELD,
INDICATOR_MATCHED_TYPE,
INDICATOR_PROVIDER,
INDICATOR_FIRSTSEEN,
INDICATOR_LASTSEEN,
];
export enum ENRICHMENT_TYPES {
InvestigationTime = 'investigation_time',
IndicatorMatchRule = 'indicator_match_rule',
}

export const EVENT_ENRICHMENT_INDICATOR_FIELD_MAP = {
'file.hash.md5': 'threatintel.indicator.file.hash.md5',
Expand All @@ -58,6 +57,9 @@ export const EVENT_ENRICHMENT_INDICATOR_FIELD_MAP = {
'registry.path': 'threatintel.indicator.registry.path',
};

export const DEFAULT_EVENT_ENRICHMENT_FROM = 'now-30d';
export const DEFAULT_EVENT_ENRICHMENT_TO = 'now';

export const CTI_DEFAULT_SOURCES = [
'Abuse URL',
'Abuse Malware',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import { IEsSearchResponse } from 'src/plugins/data/public';

import {
CtiEnrichment,
CtiEventEnrichmentRequestOptions,
CtiEventEnrichmentStrategyResponse,
CtiQueries,
Expand Down Expand Up @@ -99,11 +100,63 @@ export const buildEventEnrichmentRawResponseMock = (): IEsSearchResponse => ({
},
});

export const buildEventEnrichmentMock = (
overrides: Partial<CtiEnrichment> = {}
): CtiEnrichment => ({
'@timestamp': ['2021-05-28T18:33:52.993Z'],
'agent.ephemeral_id': ['d6b14f65-5bf3-430d-8315-7b5613685979'],
'agent.hostname': ['rylastic.local'],
'agent.id': ['ff93aee5-86a1-4a61-b0e6-0cdc313d01b5'],
'agent.name': ['rylastic.local'],
'agent.type': ['filebeat'],
'agent.version': ['8.0.0'],
'ecs.version': ['1.6.0'],
'event.category': ['threat'],
'event.created': ['2021-05-28T18:33:52.993Z'],
'event.dataset': ['threatintel.abusemalware'],
'event.ingested': ['2021-05-28T18:33:55.086Z'],
'event.kind': ['enrichment'],
'event.module': ['threatintel'],
'event.reference': [
'https://urlhaus-api.abuse.ch/v1/download/15b012e6f626d0f88c2926d2bf4ca394d7b8ee07cc06d2ec05ea76bed3e8a05e/',
],
'event.type': ['indicator'],
'fileset.name': ['abusemalware'],
'input.type': ['httpjson'],
'matched.atomic': ['5529de7b60601aeb36f57824ed0e1ae8'],
'matched.field': ['file.hash.md5'],
'matched.id': ['31408415b6d5601a92d29b86c2519658f210c194057588ae396d55cc20b3f03d'],
'matched.index': ['filebeat-8.0.0-2021.05.28-000001'],
'matched.type': ['investigation_time'],
'related.hash': [
'5529de7b60601aeb36f57824ed0e1ae8',
'15b012e6f626d0f88c2926d2bf4ca394d7b8ee07cc06d2ec05ea76bed3e8a05e',
'768:NXSFGJ/ooP6FawrB7Bo1MWnF/jRmhJImp:1SFXIqBo1Mwj2p',
],
'service.type': ['threatintel'],
tags: ['threatintel-abusemalware', 'forwarded'],
'threatintel.indicator.file.hash.md5': ['5529de7b60601aeb36f57824ed0e1ae8'],
'threatintel.indicator.file.hash.sha256': [
'15b012e6f626d0f88c2926d2bf4ca394d7b8ee07cc06d2ec05ea76bed3e8a05e',
],
'threatintel.indicator.file.hash.ssdeep': [
'768:NXSFGJ/ooP6FawrB7Bo1MWnF/jRmhJImp:1SFXIqBo1Mwj2p',
],
'threatintel.indicator.file.hash.tlsh': [
'FFB20B82F6617061C32784E2712F7A46B179B04FD1EA54A0F28CD8E9CFE4CAA1617F1C',
],
'threatintel.indicator.file.size': [24738],
'threatintel.indicator.file.type': ['html'],
'threatintel.indicator.first_seen': ['2021-05-28T18:33:29.000Z'],
'threatintel.indicator.type': ['file'],
...overrides,
});

export const buildEventEnrichmentResponseMock = (
overrides: Partial<CtiEventEnrichmentStrategyResponse> = {}
): CtiEventEnrichmentStrategyResponse => ({
...buildEventEnrichmentRawResponseMock(),
enrichments: [],
enrichments: [buildEventEnrichmentMock()],
inspect: { dsl: ['{"mocked": "json"}'] },
totalCount: 0,
...overrides,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
*/

import { IEsSearchResponse } from 'src/plugins/data/public';
import { EVENT_ENRICHMENT_INDICATOR_FIELD_MAP } from '../../../cti/constants';
import { Inspect } from '../../common';
import { RequestBasicOptions } from '..';

Expand All @@ -18,9 +19,24 @@ export interface CtiEventEnrichmentRequestOptions extends RequestBasicOptions {
}

export type CtiEnrichment = Record<string, unknown[]>;
export type EventFields = Record<string, unknown>;

export interface CtiEnrichmentIdentifiers {
id: string | undefined;
field: string | undefined;
value: string | undefined;
type: string | undefined;
provider: string | undefined;
}

export interface CtiEventEnrichmentStrategyResponse extends IEsSearchResponse {
enrichments: CtiEnrichment[];
inspect?: Inspect;
inspect: Inspect;
totalCount: number;
}

export type EventField = keyof typeof EVENT_ENRICHMENT_INDICATOR_FIELD_MAP;
export const validEventFields = Object.keys(EVENT_ENRICHMENT_INDICATOR_FIELD_MAP) as EventField[];

export const isValidEventField = (field: string): field is EventField =>
validEventFields.includes(field as EventField);
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { getFirstElement } from './data_retrieval';

describe('getFirstElement', () => {
it('returns undefined if array is undefined', () => {
expect(getFirstElement(undefined)).toEqual(undefined);
});

it('returns undefined if array is empty', () => {
expect(getFirstElement([])).toEqual(undefined);
});

it('returns the first element if present', () => {
expect(getFirstElement(['hi mom'])).toEqual('hi mom');
});

it('returns the first element of multiple', () => {
expect(getFirstElement(['hi mom', 'hello world'])).toEqual('hi mom');
});
});
15 changes: 15 additions & 0 deletions x-pack/plugins/security_solution/common/utils/data_retrieval.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

/**
* Retrieves the first element of the given array.
*
* @param array the array to retrieve a value from
* @returns the first element of the array, or undefined if the array is undefined
*/
export const getFirstElement: <T = unknown>(array: T[] | undefined) => T | undefined = (array) =>
array ? array[0] : undefined;
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { newThreatIndicatorRule } from '../../objects/rule';
import { cleanKibana, reload } from '../../tasks/common';
import { esArchiverLoad, esArchiverUnload } from '../../tasks/es_archiver';
import { loginAndWaitForPageWithoutDateRange } from '../../tasks/login';
import {
JSON_LINES,
TABLE_CELL,
TABLE_ROWS,
THREAT_CONTENT,
THREAT_DETAILS_VIEW,
THREAT_INTEL_TAB,
THREAT_SUMMARY_VIEW,
TITLE,
} from '../../screens/alerts_details';
import { TIMELINE_FIELD } from '../../screens/rule_details';
import { goToRuleDetails } from '../../tasks/alerts_detection_rules';
import { expandFirstAlert, goToManageAlertsDetectionRules } from '../../tasks/alerts';
import { createCustomIndicatorRule } from '../../tasks/api_calls/rules';
import {
openJsonView,
openThreatIndicatorDetails,
scrollJsonViewToBottom,
} from '../../tasks/alerts_details';

import { ALERTS_URL } from '../../urls/navigation';
import { addsFieldsToTimeline } from '../../tasks/rule_details';

describe('CTI Enrichment', () => {
before(() => {
cleanKibana();
esArchiverLoad('threat_indicator');
esArchiverLoad('suspicious_source_event');
loginAndWaitForPageWithoutDateRange(ALERTS_URL);
goToManageAlertsDetectionRules();
createCustomIndicatorRule(newThreatIndicatorRule);
reload();
});

after(() => {
esArchiverUnload('threat_indicator');
esArchiverUnload('suspicious_source_event');
});

beforeEach(() => {
loginAndWaitForPageWithoutDateRange(ALERTS_URL);
goToManageAlertsDetectionRules();
goToRuleDetails();
});

it('Displays enrichment matched.* fields on the timeline', () => {
const expectedFields = {
'threat.indicator.matched.atomic': newThreatIndicatorRule.atomic,
'threat.indicator.matched.type': 'indicator_match_rule',
'threat.indicator.matched.field': newThreatIndicatorRule.indicatorMappingField,
};
const fields = Object.keys(expectedFields) as Array<keyof typeof expectedFields>;

addsFieldsToTimeline('threat.indicator.matched', fields);

fields.forEach((field) => {
cy.get(TIMELINE_FIELD(field)).should('have.text', expectedFields[field]);
});
});

it('Displays persisted enrichments on the JSON view', () => {
const expectedEnrichment = [
{ line: 4, text: ' "threat": {' },
{
line: 3,
text:
' "indicator": "{\\"first_seen\\":\\"2021-03-10T08:02:14.000Z\\",\\"file\\":{\\"size\\":80280,\\"pe\\":{},\\"type\\":\\"elf\\",\\"hash\\":{\\"sha256\\":\\"a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3\\",\\"tlsh\\":\\"6D7312E017B517CC1371A8353BED205E9128223972AE35302E97528DF957703BAB2DBE\\",\\"ssdeep\\":\\"1536:87vbq1lGAXSEYQjbChaAU2yU23M51DjZgSQAvcYkFtZTjzBht5:8D+CAXFYQChaAUk5ljnQssL\\",\\"md5\\":\\"9b6c3518a91d23ed77504b5416bfb5b3\\"}},\\"type\\":\\"file\\",\\"event\\":{\\"reference\\":\\"https://urlhaus-api.abuse.ch/v1/download/a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3/\\",\\"ingested\\":\\"2021-03-10T14:51:09.809069Z\\",\\"created\\":\\"2021-03-10T14:51:07.663Z\\",\\"kind\\":\\"enrichment\\",\\"module\\":\\"threatintel\\",\\"category\\":\\"threat\\",\\"type\\":\\"indicator\\",\\"dataset\\":\\"threatintel.abusemalware\\"},\\"matched\\":{\\"atomic\\":\\"a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3\\",\\"field\\":\\"myhash.mysha256\\",\\"id\\":\\"84cf452c1e0375c3d4412cb550bd1783358468a3b3b777da4829d72c7d6fb74f\\",\\"index\\":\\"filebeat-7.12.0-2021.03.10-000001\\",\\"type\\":\\"indicator_match_rule\\"}}"',
},
{ line: 2, text: ' }' },
];

expandFirstAlert();
openJsonView();
scrollJsonViewToBottom();

cy.get(JSON_LINES).then((elements) => {
const length = elements.length;
expectedEnrichment.forEach((enrichment) => {
cy.wrap(elements)
.eq(length - enrichment.line)
.should('have.text', enrichment.text);
});
});
});

it('Displays threat indicator details on the threat intel tab', () => {
const expectedThreatIndicatorData = [
{ field: 'event.category', value: 'threat' },
{ field: 'event.created', value: '2021-03-10T14:51:07.663Z' },
{ field: 'event.dataset', value: 'threatintel.abusemalware' },
{ field: 'event.ingested', value: '2021-03-10T14:51:09.809069Z' },
{ field: 'event.kind', value: 'enrichment' },
{ field: 'event.module', value: 'threatintel' },
{
field: 'event.reference',
value:
'https://urlhaus-api.abuse.ch/v1/download/a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3/(opens in a new tab or window)',
},
{ field: 'event.type', value: 'indicator' },
{ field: 'file.hash.md5', value: '9b6c3518a91d23ed77504b5416bfb5b3' },
{
field: 'file.hash.sha256',
value: 'a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3',
},
{
field: 'file.hash.ssdeep',
value: '1536:87vbq1lGAXSEYQjbChaAU2yU23M51DjZgSQAvcYkFtZTjzBht5:8D+CAXFYQChaAUk5ljnQssL',
},
{
field: 'file.hash.tlsh',
value: '6D7312E017B517CC1371A8353BED205E9128223972AE35302E97528DF957703BAB2DBE',
},
{ field: 'file.size', value: '80280' },
{ field: 'file.type', value: 'elf' },
{ field: 'first_seen', value: '2021-03-10T08:02:14.000Z' },
{
field: 'matched.atomic',
value: 'a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3',
},
{ field: 'matched.field', value: 'myhash.mysha256' },
{
field: 'matched.id',
value: '84cf452c1e0375c3d4412cb550bd1783358468a3b3b777da4829d72c7d6fb74f',
},
{ field: 'matched.index', value: 'filebeat-7.12.0-2021.03.10-000001' },
{ field: 'matched.type', value: 'indicator_match_rule' },
{ field: 'type', value: 'file' },
];

expandFirstAlert();
openThreatIndicatorDetails();

cy.get(THREAT_INTEL_TAB).should('have.text', 'Threat Intel (1)');
cy.get(THREAT_DETAILS_VIEW).within(() => {
cy.get(TABLE_ROWS).should('have.length', expectedThreatIndicatorData.length);
expectedThreatIndicatorData.forEach((row, index) => {
cy.get(TABLE_ROWS)
.eq(index)
.within(() => {
cy.get(TABLE_CELL).eq(0).should('have.text', row.field);
cy.get(TABLE_CELL).eq(1).should('have.text', row.value);
});
});
});
});

describe('with additional indicators', () => {
before(() => {
esArchiverLoad('threat_indicator2');
});

after(() => {
esArchiverUnload('threat_indicator2');
});

it('Displays matched fields from both indicator match rules and investigation time enrichments on Alerts Summary tab', () => {
const indicatorMatchRuleEnrichment = {
field: 'myhash.mysha256',
value: 'a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3',
};
const investigationTimeEnrichment = {
field: 'source.ip',
value: '192.168.1.1',
};
const expectedMatches = [indicatorMatchRuleEnrichment, investigationTimeEnrichment];

expandFirstAlert();

cy.get(THREAT_SUMMARY_VIEW).within(() => {
cy.get(TABLE_ROWS).should('have.length', expectedMatches.length);
expectedMatches.forEach((row, index) => {
cy.get(TABLE_ROWS)
.eq(index)
.within(() => {
cy.get(TITLE).should('have.text', row.field);
cy.get(THREAT_CONTENT).should('have.text', row.value);
});
});
});
});
});
});
Loading

0 comments on commit 569c209

Please sign in to comment.