[D&D] Add Visualization service

[ashwin-pc]

Visualization service (Design)

Background

In order for the drag and drop experience (Design, Proposal) to render correctly, each visualization type needs the UI to accept different properties. In addition, the way each visualization type needs to process the data varies between each visualization type. To support these needs and provide a common interface for each visualization type to declare itself to the drag and drop experience, we have designed the visualization service. This service will support as much flexibility as possible while remaining consistent with the general UX for the drag and drop experience. There are two key problems that this service needed to solve for:

• Allow a visualization type to define all the features it needs so that a user can easily generate the visualization.
• Allow a visualization type to define how the given inputs can be translated into a useful visualization using the data from Opensearch core

Tenets

• As flexible as possible for various visualization, data, and aggregation types.
• Consistent UI and performance across the various visualization types.
• Simple interface with options to expand.

Key terms

• Expressions plugin: Opensearch Dashboards plugin to define a pipeline of functions that pipe the output of the previous function to the input of the next and who’s final output can be piped to one of the registered renderers registered.
• Contributions: How a visualization can contribute to the UI as needed to support user experience for the visualization.

Anatomy of the workspace:

Proposed solution

Using Config and Expressions (Recommended)

This approach uses a config object that is an instance of the WizardVisualization class to register the visualization with the service. The config has 3 primary functions:

1. Define the visualization metadata
2. Provide contributions to configure the workspace view according to the visualization type’s needs
3. Define the expression pipeline to handle the data fetching/transformation and render the visualization

The config object interface:

interface WizardVisulatization {
  name: string;
  title: string;
  description: string;
  icon: string;
  stage: "beta" | "production";
  options: any;
  contributions: {
    containers?: { // Define new or override existing view containers
      name: string;
      title: string;
      location: "panel" | "toolbar";
      render: (schemas: ContainerSchema[]) => {}; // recieves an array of items to render within the container
    };
    items: {
      "container-name": ContainerSchema[]; // schema that is used to render the container. Each container is responsible for deciding that for consistency
    };
  };
  pipeline: Expression;
}

Contributions:

Visualizations each have a unique set of inputs and properties that the user can define to render the desired visual. To support a wide variety of visualizations while keeping the user interface and experience consistent, we are proposing a contribution architecture. The principle here is that the drag and drop experience will offer a variety of ways in which a visualization type can contribute to the user interface called contributions.

The contributions are a way by which the visualization type can define the elements it needs in the user interface to function correctly. The contributions also serve as an abstraction to guard the user interface from becoming unwieldy by enforcing schemas that the visual elements need to adhere to.

In the initial design there are only two types of contributions:

• Containers
• Items

Containers: They are visual elements within which “Items” can be defined. They define a schema which each item within the container will adhere to and are responsible for rendering the items from the provided Item object.

e.g.

interface ToolbarSchema {
  name: string;
  icon: string;
  action: string;
}

const container = {
  name: "toolbar",
  title: "Toolbar",
  localtion: "toolbar",
  render: (items: ToolbarSchema[]) => (
    <Container>
      {items.map(({ icon, name, action }) => (
        <ToolbarIcon key={name} icon={icon} onclick={dispatchEvent(action)} />
      ))}
    </Container>
  ),
}

Some containers are provided by default such as the toolbar and config and style panels. The visualization config allows the visualization type to add a container or override the way an existing container behaves (by contributing a container with the same name as the default container).

Most visualization types can work without contributing containers and using just the default containers. The ability to contribute containers is so that unique visualizations whose requirements aren't met with the existing containers and their schemas can still function using a custom schema that suits their need.

Items: They are elements that are rendered within a container. They are objects that adhere to the schema of the container they are going into. Each Item contains all the metadata the container needs to render the element correctly and function properly.

e.g.

const toolbarItem: ToolbarSchema = {
  name: "fullscreen",
  icon: "expand",
  action: "launch_fullscreen",
}

Callouts:

• Every visualization type is required to define the UI items it needs. No item is defined by default for visualizations.
• Items require a container to render but while the other way is not strictly necessary, it is an anti pattern and not advised unless necessary.

Expressions:

While contributions are responsible for the UI, expression are used to define the pipeline used by the visualization type to use the inputs from the user to fetch data and render the visualization. It uses the existing expressions plugin to define this pipeline. The plugin allows the user to define expression functions, register them and define an expressions pipeline to render the visualization. The power fo the expressions plugin is that it allows us to define any javascript function as an expressions function and chain expression functions together that outputs to an expression renderer. This allows each visualization type to define its any custom pipeline using both their own custom expression functions and existing expression functions.

Read more about expressions: Expressions

Alternatives

There are two things that the proposed approach aims to solve, allow a visualization type to configure the UI to meet its needs and define a pipeline on how to handle data to render the visualization. For each of these we have some alternatives we explored.

UI: Custom elements

This is a simpler approach in which we allow the visualization type to directly render the required UI elements directly onto the workspace. We provide hook points into which the visualization type can render its controls directly in the DOM. It can use existing components exposed by the drag and drop plugin or create custom components for its own use case. This means that elements such as the toolbar and side panels are rendered by the visualization type and not the drag and drop tool.

e.g.

interface WizardVisulatization {
  name: string;
  title: string;
  description: string;
  icon: string;
  stage: "beta" | "production";
  options: any;
  ui: {
    toolbar: () => <Toolbar />,
    configPanel: () => <ConfigPanel />,
    // ...
  }
  pipeline: Expression;
}

While this is simpler and offers a lot more flexibility, it has its own drawbacks. Specifically;

• It’s harder to maintain UI consistency since each visualization type can define any custom markup that suits their needs. This is still possible with the proposed approach but has to be done intentionally. Visualization developers may choose the path of least resistance and choose to build their own UI’s instead of using the built in options and patterns.
• Performance can also be easily impacted since there are very few guardrails for how these components can be rendered

Pipeline: Lifecycle Method

In this approach we have lifecycle hooks for the workspace that the selected visualization can register with and use to fetch aggregation for the fields and visualize them. This is a predefined lifecycle which has hooks that the visualization can register to use for fetching pre/post processing and rendering data.

e.g.

interface WizardVisulatization {
  name: string;
  title: string;
  description: string;
  icon: string;
  stage: "beta" | "production";
  options: any;
  // ...
  lifecycle: {
      setup: (deps: StartDependencies) => void
      prefetch: (args) => void
      fetch: (args) => void
      postProcess: (args) => void
      render: (args) => void
  };
}

The main difference between this and Expressions is that this pipeline is predefined allowing for less flexibility. All visualizations must follow the lifecycle pattern though all lifecycle methods are not strictly necessary for every visualization type. We can also have different lifecycles for each the editing view and the embeddable view.

This solution would be easier to implement and wouldn’t add any plugin overhead, but like the previous option has some significant disadvantages. Namely;

• It’s less flexibility in the pipeline for more unique visualizations and use-cases.
• It required more reinvention since the expressions plugin is already built to handle this use-case and is used through out the various visualization apps in OpenSearch Dashboards.

FAQ

Why contributions?

Contributions are an abstraction that allows us to create a very flexible workspace that has a coherent UX since each visualization is only allowed to edit the view using each containers schemas.

Why expressions?

Expression pipeline is a chain of functions that pipe its output to the input of the next function. Functions can be configured using arguments provided by the user. The final output of the expression pipeline can be rendered using one of the renderers registered in expressions plugin. This allows us to customize the lifecycle of a visualization.

Standard pipeline functions will be provided such a function to take workspace data and produce a data request for aggregating data. Each visualization can define additional functions to manipulate the data or otherwise customize the visualization workspace or embeddable. The visualization will also have to define a pipeline to render the visualization given the input data from the sub panel (config) and other inputs into the workspace.

What does the flow look like for a developer creating a viz type?

A new visualization type can be created a a part of Dashboards core, or as a separate plugin. In either case, the type will depend on the drag and drop plugin and use this service to register the new visualization type. When a user launches the drag and drop experience, they will be able to select the newly registered visualization from the visualization type picker.

What is the performance impact of the proposed solution?

There should be no impact to performance since that is one of the guiding principles behind the abstractions. Each visualization type is only allowed to alter the state of the experience when its type is selected. Even then since all the rendering is in the hands of the container and the expressions pipeline, it is their responsibility to keep the performance of the experience in check. Since most visualizations will use the default containers and similar expressions pipeline, there should be no impact to even custom visualizations that follow the standard pattern.