Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cases] Migrate user actions connector ID V2 #112710

Merged

Conversation

jonathan-buttner
Copy link
Contributor

The original PR was reverted because it broke master: #108272 (comment)

I updated the tests in the last commit: 09ec155


Background: #105677

Initial PR: #104221

This is a follow on PR for the work to migrate references to saved objects that are stored in fields outside of the references field for user actions. The connector ID can be stored in a few types of user actions:

Creation of a case user action

{
...
  connector: {
    id: <connector id to be migrated>,
    ...
  }
}

Updating the connector within a case

{
  id: <id to be migrated>
  ...
}

Pushing a case

{
  connector_id: <id to be migrated>
  ...
}

User actions holds a JSON encoded version of the above objects in the fields named oldValue and newValue to indicate what the original value was and what it is being changed to.

Performance Concern

The easiest solution would be to have the backend decode the user actions oldValue and newValue fields and place the connector ID back into them and re-encode the fields before they are returned to the UI. This would have avoiding needing to change any code in the UI. We had some performance concerns with doing that approach for every user action when the case detail view is requested.

Instead the approach we took was to move the ids to new_val_connector_id and old_val_connector_id. This required a number of changes in the UI to look for that field instead of using the encoded values for the id.

Notable Changes

  • Backend
    • Migration for user actions saved objects to extract the connector id and move it to the references field
    • The json encoding work was moved down into the service layer
    • The service layer handles remove the connector id field and moving it to the references field
    • When returning the user actions the connector id is placed in the new_val_connector_id and old_val_connector_id fields to correspond with the newValue and oldValue
  • Frontend
    • Refactored the places where we were accessing the connector id from the encoded value to leverage both the encoded value and the appropriate *_val_connector_id
    • Added more type checking for places where we called parseString on the encoded value

Testing

I added unit tests for the transformation and migration code in the backend and frontend where I could. I also added integration tests that run the migration on older 7.13 data.

To test this PR:

  • Create a case
  • Add/remove/update the connector
  • Push the case
  • Add comments and other attachments
  • Ensure that the user actions are all displaying the type of action taken by the user correctly

@jonathan-buttner jonathan-buttner added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Threat Hunting Security Solution Threat Hunting Team auto-backport Deprecated - use backport:version if exact versions are needed v7.16.0 Team:Threat Hunting:Cases labels Sep 21, 2021
@jonathan-buttner jonathan-buttner requested a review from a team as a code owner September 21, 2021 17:53
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

@jonathan-buttner jonathan-buttner changed the title [Cases] Migrate user actions connector ID [Cases] Migrate user actions connector ID V2 Sep 21, 2021
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
cases 331 334 +3

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
cases 412 425 +13

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
cases 308.4KB 308.8KB +424.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
cases 79.6KB 80.2KB +621.0B
Unknown metric groups

API count

id before after diff
cases 454 469 +15

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jonathan-buttner jonathan-buttner merged commit 70f635b into elastic:master Sep 22, 2021
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Sep 22, 2021
* Making progress

* Fleshing out the extraction logic

* Finishing migration logic and starting more tests

* Finishing migration unit tests

* Making progress on services

* Finishing transform to es schema

* Finishing transform functionality and unit tests

* reverting migration data updates

* Cleaning up type errors

* fixing test error

* Working migration tests

* Refactoring retrieval of connector fields

* Refactoring connector id in and tests in frontend

* Fixing tests and finished refactoring parse string

* Fixing integration test

* Fixing integration tests

* Removing some duplicate code and updating test name

* Fixing create connector user action bug

* Addressing feedback and logging error

* Moving parsing function to common

* Fixing type errors

* Fixing type errors

* Addressing feedback

* Fixing lint errors

* Adjusting import for user action changes

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@kibanamachine
Copy link
Contributor

💚 Backport successful

Status Branch Result
7.x

This backport PR will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request Sep 22, 2021
* Making progress

* Fleshing out the extraction logic

* Finishing migration logic and starting more tests

* Finishing migration unit tests

* Making progress on services

* Finishing transform to es schema

* Finishing transform functionality and unit tests

* reverting migration data updates

* Cleaning up type errors

* fixing test error

* Working migration tests

* Refactoring retrieval of connector fields

* Refactoring connector id in and tests in frontend

* Fixing tests and finished refactoring parse string

* Fixing integration test

* Fixing integration tests

* Removing some duplicate code and updating test name

* Fixing create connector user action bug

* Addressing feedback and logging error

* Moving parsing function to common

* Fixing type errors

* Fixing type errors

* Addressing feedback

* Fixing lint errors

* Adjusting import for user action changes

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
lykkin pushed a commit to lykkin/kibana that referenced this pull request Sep 28, 2021
* Making progress

* Fleshing out the extraction logic

* Finishing migration logic and starting more tests

* Finishing migration unit tests

* Making progress on services

* Finishing transform to es schema

* Finishing transform functionality and unit tests

* reverting migration data updates

* Cleaning up type errors

* fixing test error

* Working migration tests

* Refactoring retrieval of connector fields

* Refactoring connector id in and tests in frontend

* Fixing tests and finished refactoring parse string

* Fixing integration test

* Fixing integration tests

* Removing some duplicate code and updating test name

* Fixing create connector user action bug

* Addressing feedback and logging error

* Moving parsing function to common

* Fixing type errors

* Fixing type errors

* Addressing feedback

* Fixing lint errors

* Adjusting import for user action changes

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes Team:Threat Hunting Security Solution Threat Hunting Team v7.16.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants