[Reporting] Improvements needed for screenshot capture flow

[tsullivan]

The Reporting plugin has the ability to take a URL to a Kibana page, and a few fields of metadata, and return an observable that resolves to a screenshot of the page, in the form of buffer data.

To execute this task, several things happen in within the pipeline of the observable:

1. openUrl: Open a browser page to the URL using the Puppeteer library
   • Authentication is provided via Basic authentication in the request headers. In order not to reveal the authentication headers outside of the server origin, Reporting intercepts each request made by the browser to check if authentication is required for the request.
   • Reporting knows if it has found the expected page by waiting for the .application selector to resolve.
2. skipTelemetry(): Attempt to have the browser instance avoid Telemetry calls by injecting data to into localStorage that telemetry does not need to be sent [1]
3. getNumberOfItems(): Read data from the DOM of the Kibana page to get a count of how many visualization items are on the page
   • It needs that info to know how many visualizations should "check in" with custom events to signal to Reporting that they are done rendering.
4. driver.setViewport()Set the browser viewport and "zoom" based on the metadata
5. waitForElementsToBeInDOM: Wait for all visualization elements to be in the DOM
6. injectCss(): Inject custom CSS that hacks the Kibana page into a Report style
   • hides the navigation and page menus
   • hides interactive elements such as buttons and filter inputs
7. layout.positionElements(): Conditional step: if the visualization elements need to be re-arranged for "print mode," do it here
8. waitForRenderComplete(): Wait for all visualization elements to be done rendering
9. getElementPositionAndAttributes: Find (a) the x/y position of each visualization, and (b) title from the DOM for each visualization, that will be captured on the page
   • Note: the element position / data can be a set of multiple items for "print" mode of a dashboard
10. getTimeRange: Read the time range from the filter bar DOM so it can be added to the screenshot title [2]
11. getScreenshots: Capture screenshots of each visualization on the page, using the element position info and return it, along with the date range and titles info

[1] This is a problematic arrangement because it forces Reporting to be aware of Telemetry and how to talk to it by manipulating Telemetry data in localStorage - also, it doesn't work properly.
[2] Steps 9 and 10 happen in parallel

Once the buffer data is available to the higher level code that runs the capture flow, it gets converted to base64 data, and the Reporting job document gets updated with the base64 and the "status" and "completed_at" fields updated.

Between the first and last step, there are parts that do things to the page to prepare it for reports, and parts that check the progress of the preparations by waiting for CSS selectors.

Plan to improve performance.

We need a plan to improve performance in Reporting for these steps.

Benchmark the current state of performance
We can define a "typical" dashboard as one of the sample data dashboards. We'll run many tests of generating a PNG report, while capture verbose logs with timestamps. Based on that information, we can find the average time taken by each part of the steps in the pipeline.

Benchmarks will help us focus on fixing the parts where the most time gets wasted, and gives us something to compare against after the improvements are done.

Tweak the pipeline
Based on the results of the benchmarking, we can make small changes to the pipeline to chase some improvements in the timing. Presumably, running more things in parallel could be something done as part of this.

Add a "Screenshot Mode Service" to Kibana core
There are steps in the pipeline that can be moved away to other parts of Kibana that "prepare" the page for Kibana. These parts wouldn't be needed if the preparations happen automatically when Reporting opens the page. That would be a huge performance boost for a few reasons:

1. No wasted rendering that gets replaced with the Reporting styles
2. No "interaction" components loading or fetching data in the background, which consumes resources.

Screenshot Mode Service RFC: #59084

This issue will be updated with some benchmark results.

[tsullivan]

Here's a starting data point showing a trace of the different methods that get called when capturing a screenshot. This data is shown in APM, being integrated in this PR: [eCommerce] Revenue Dashboard. In this test, 5 PNGs were generated. According to this data, all of the transactions took between 24 and 28 seconds to complete.

Here's some data about the timing of generating a PNG report. Almost the entire time is spent generating the screenshots:

Drilling into the transaction of just the screenshot pipeline, we can see most of the time is waiting for the URL to open and waiting for the Kibana visualizations to complete rendering:

[tsullivan]

Here's a data point of the timings of generating a 2-page PDF of the Canvas eCommerce sample data worksheet.

Here is the overall timing of generating the PDF. Unlike the PNG report where the entire generation time is capturing screenshots, there is about 1 extra second of processing time to convert the PNG buffers to a multi-page PDF:

Here is a drill-in of the screenshots pipeline. You can see that 2 pages were opened, and they each took about the same amount of time to render:

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[tsullivan]

This issue description talks a lot about what is happening inside the screenshot pipeline, but we need more info about what happens in PDF generation after the screenshots are captured.

Here is an explanation:

1. The screenshots observable returns PNG buffer data (Screenshot[] type) to generate_pdf
2. Each PNG buffer is converted to base64 to pass into the PdfMaker library
3. We tell pdfMaker to generate the PDF buffer
4. We convert the PDF buffer to base64 to model it for the elasticsearch document

[tsullivan]

Closing via:

• [Canvas] Reduce report generation time by re-using headless browser page in background #63301
• [Reporting] APM integration for baseline performance measurements #59967
• [rfc][skip-ci][reporting] Rendering API RFC #64372