[Reporting] APM integration for baseline performance measurements #59967

tsullivan · 2020-03-11T23:33:09Z

Summary

Refer to #43548 for information about Kibana's APM integrations.

Adds APM transactions and spans around the slow parts of Reporting code.

This lets us see a trace of where the flow is taking the most time when screenshots are captured. Here is an example of capturing a Dashboard with "print_layout" mode. You can see the multiple calls to get_screenshots:

Checklist

Delete any items that are not applicable to this PR.

Need some findings on this issue: Kibana crash: TypeError: request.abort is not a function #59712
~~[ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~[ ] Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios
~~[ ] This was checked for keyboard-only and screenreader accessibility~~
~~[ ] This renders correctly on smaller devices using a responsive layout. (You can test this in your browser~~
~~[ ] This was checked for cross-browser compatibility, including a check against IE11~~

For maintainers

This was checked for breaking API changes and was labeled appropriately

src/core/server/elasticsearch/cluster_client.ts

tsullivan · 2020-03-12T01:02:01Z

x-pack/legacy/plugins/reporting/export_types/common/lib/screenshots/get_screenshots.ts

    const item = elementsPositionAndAttributes[i];
-    const base64EncodedData = await asyncDurationLogger(


asyncDurationLogger was hiding a type error, because we're resolving a string, not a Buffer

More context about this: since we'll have APM to measure the duration of all these modules, we no longer need this helper function. After having removed it, I noticed the surrounding code treated the helper function return value as a Buffer but it is actually a string of base64 content.

elasticmachine · 2020-03-12T21:47:00Z

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

x-pack/legacy/plugins/reporting/export_types/printable_pdf/server/lib/tracker.ts

vigneshshanmugam

Amazing job on doing the full custom instrumentation using the APM Node.js agent. Love to see we are dog fooding more.

I just have one small suggestion, It just personal opinion as Span can be null | Span at any point in time. You could have a generic tracer utility

// utility
function startTrace() {
   const span = apm.startSpan(...);
   return () {
        span && span.end()
   }
}


// in actual code
const endTrace = startTrace(name, type)

endTrace()

Helps to avoid the if check everywhere.

x-pack/legacy/plugins/reporting/export_types/common/lib/screenshots/open_url.ts

x-pack/legacy/plugins/reporting/export_types/printable_pdf/server/lib/tracker.ts

...ck/legacy/plugins/reporting/export_types/common/lib/screenshots/get_element_position_data.ts

src/core/server/elasticsearch/cluster_client.ts

…nce-measurement

tsullivan · 2020-04-29T19:21:38Z

x-pack/legacy/plugins/reporting/export_types/png/server/execute_job/index.ts

@@ -54,13 +54,14 @@ export const executeJobFactory: QueuedPngExecutorFactory = async function execut
          job.layout
        );
      }),
-      map(({ buffer, warnings }) => {
+      map(({ base64, warnings }) => {


I had to rename this - it was very unclear that the return payload actually is a string

tsullivan · 2020-04-30T23:58:23Z

@elasticmachine merge upstream

tsullivan · 2020-05-05T23:44:12Z

Will merge this after the 7.8 branch is cut

kibanamachine · 2020-05-07T21:09:05Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: e4495c4

History

💚 Build #45714 succeeded dd9c55b
💔 Build #45678 failed 6b5c392
💔 Build #44683 failed 7618e42
💔 Build #44205 failed e79e875
💔 Build #43854 failed 3db397f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…astic#59967) * apm stuff * fix cluster_client * fix snapshot * tracker utility for generate_pdf * call apm.startSpan instead of txn.startSpan * Fix async call to end transaction * fix typescript * remove captuureErrors * restore accidental removal * add startTrace lib * fix import * fix imports * ts fix * fix generate_png to not format base64 to buffer and back to base64 * 💅 * revert change to cluster client * fix unused translation Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…or-part-mvp-2 * 'master' of github.com:elastic/kibana: (259 commits) SavedObjects bulkCreate API should return migrationVersion and strip the type & namespace from the id (elastic#65150) Drilldown count tooltip (elastic#65105) plugins logs start with "plugins." prefix (elastic#65710) [ML] Fix pagination reset on search query update. (elastic#65668) [SIEM] Add types to the mappings objects so extra keys cannot be introduced [apm] Update machine learning flyout and service maps docs (elastic#65517) change api endpoint and throw error (elastic#65790) [Maps] remove SLA percentage metric (elastic#65718) [Reporting] APM integration for baseline performance measurements (elastic#59967) fix(NA): noParse regex for windows on kbn optimizer (elastic#65755) [ML] DFA: ensure at least one field is included in analysis before job can be created (elastic#65320) [Data plugin] cleanup - remove unused getRoutes / routes from indexPattern object (elastic#65683) Removed skip to enable test. (elastic#65575) [Lens] Type safe migrations (elastic#65576) [Canvas] Fix nav link behavior in Canvas (elastic#65590) [Event log] Fix flaky test (elastic#65658) [Alerting] changes preconfigured actions config from array to object (elastic#65397) remove immediate functions from esqueue worker cycles (elastic#65375) [Metrics UI] Fix isAbove to only display when threshold set (elastic#65540) draft search profiler accessibility tests (elastic#62357) ... # Conflicts: # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_form/pipeline_form_fields.tsx

…9967) (#65797) * apm stuff * fix cluster_client * fix snapshot * tracker utility for generate_pdf * call apm.startSpan instead of txn.startSpan * Fix async call to end transaction * fix typescript * remove captuureErrors * restore accidental removal * add startTrace lib * fix import * fix imports * ts fix * fix generate_png to not format base64 to buffer and back to base64 * 💅 * revert change to cluster client * fix unused translation Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

tsullivan changed the title ~~Reporting/performance measurement~~ [Reporting] APM integration for baseline performance measurements Mar 11, 2020

tsullivan force-pushed the reporting/performance-measurement branch from 5fca169 to 1b2a777 Compare March 11, 2020 23:39

tsullivan commented Mar 11, 2020

View reviewed changes

src/core/server/elasticsearch/cluster_client.ts Outdated Show resolved Hide resolved

tsullivan commented Mar 12, 2020

View reviewed changes

tsullivan force-pushed the reporting/performance-measurement branch from 1b2a777 to 4a5af92 Compare March 12, 2020 01:04

tsullivan added 3 commits March 12, 2020 09:29

fix cluster_client

2521d76

apm stuff

912a4ad

fix snapshot

a865bcc

tsullivan force-pushed the reporting/performance-measurement branch from 4a5af92 to a865bcc Compare March 12, 2020 16:48

tsullivan added 3 commits March 12, 2020 13:15

tracker utility for generate_pdf

0fc8813

call apm.startSpan instead of txn.startSpan

cbd094b

Fix async call to end transaction

c037d4d

tsullivan marked this pull request as ready for review March 12, 2020 21:22

tsullivan requested a review from a team as a code owner March 12, 2020 21:22

tsullivan requested review from vigneshshanmugam, joelgriffith, a team and poffdeluxe March 12, 2020 21:22

fix typescript

0aa13e6

tsullivan added (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead release_note:skip Skip the PR/issue when compiling release notes Team:Reporting Services v8.0.0 labels Mar 12, 2020

tsullivan commented Mar 12, 2020

View reviewed changes

x-pack/legacy/plugins/reporting/export_types/printable_pdf/server/lib/tracker.ts Show resolved Hide resolved

vigneshshanmugam reviewed Mar 12, 2020

View reviewed changes

x-pack/legacy/plugins/reporting/export_types/common/lib/screenshots/open_url.ts Outdated Show resolved Hide resolved

x-pack/legacy/plugins/reporting/export_types/printable_pdf/server/lib/tracker.ts Show resolved Hide resolved

vigneshshanmugam requested a review from watson March 12, 2020 22:08

pgayvallet reviewed Mar 16, 2020

View reviewed changes

...ck/legacy/plugins/reporting/export_types/common/lib/screenshots/get_element_position_data.ts Outdated Show resolved Hide resolved

src/core/server/elasticsearch/cluster_client.ts Outdated Show resolved Hide resolved

mshustov mentioned this pull request Mar 16, 2020

Kibana crash: TypeError: request.abort is not a function #59712

Closed

tsullivan added the blocked label Mar 30, 2020

tsullivan added 5 commits March 30, 2020 10:40

add startTrace lib

33debce

fix import

cc57a7a

Merge remote-tracking branch 'elastic/master' into reporting/performa…

498a629

…nce-measurement

fix imports

fd63584

Merge branch 'master' into reporting/performance-measurement

2b564ff

joelgriffith approved these changes Apr 27, 2020

View reviewed changes

tsullivan added 4 commits April 28, 2020 10:54

Merge branch 'master' into reporting/performance-measurement

890e5f2

ts fix

3db397f

Merge branch 'master' into reporting/performance-measurement

e193ecf

fix generate_png to not format base64 to buffer and back to base64

e79e875

tsullivan commented Apr 29, 2020

View reviewed changes

elasticmachine and others added 5 commits April 30, 2020 19:58

Merge branch 'master' into reporting/performance-measurement

7618e42

Merge branch 'master' into reporting/performance-measurement

0867800

💅

4ac5ce0

revert change to cluster client

6b5c392

fix unused translation

dd9c55b

tsullivan added the v7.9.0 label May 5, 2020

Merge branch 'master' into reporting/performance-measurement

e4495c4

tsullivan removed the blocked label May 7, 2020

tsullivan merged commit 6bf0890 into elastic:master May 7, 2020

tsullivan mentioned this pull request May 8, 2020

[7.x] [Reporting] APM integration for baseline performance measurements (#59967) #65797

Merged

tsullivan deleted the reporting/performance-measurement branch May 8, 2020 19:27

tsullivan mentioned this pull request May 8, 2020

[Reporting] Improvements needed for screenshot capture flow #59396

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Reporting] APM integration for baseline performance measurements #59967

[Reporting] APM integration for baseline performance measurements #59967

tsullivan commented Mar 11, 2020 •

edited

Loading

tsullivan Mar 12, 2020 •

edited

Loading

tsullivan May 5, 2020

elasticmachine commented Mar 12, 2020

vigneshshanmugam left a comment

tsullivan Apr 29, 2020

tsullivan commented Apr 30, 2020

tsullivan commented May 5, 2020

kibanamachine commented May 7, 2020

		const item = elementsPositionAndAttributes[i];
		const base64EncodedData = await asyncDurationLogger(

[Reporting] APM integration for baseline performance measurements #59967

[Reporting] APM integration for baseline performance measurements #59967

Conversation

tsullivan commented Mar 11, 2020 • edited Loading

Summary

Checklist

For maintainers

tsullivan Mar 12, 2020 • edited Loading

Choose a reason for hiding this comment

tsullivan May 5, 2020

Choose a reason for hiding this comment

elasticmachine commented Mar 12, 2020

vigneshshanmugam left a comment

Choose a reason for hiding this comment

tsullivan Apr 29, 2020

Choose a reason for hiding this comment

tsullivan commented Apr 30, 2020

tsullivan commented May 5, 2020

kibanamachine commented May 7, 2020

💚 Build Succeeded

History

tsullivan commented Mar 11, 2020 •

edited

Loading

tsullivan Mar 12, 2020 •

edited

Loading