[RFC] Reusing Discover Queries to Build Artifacts such as Visualizations, Reports, and Alerts

[brijos]

Summary

This RFC proposes creating an extensible jump-off point within OpenSearch Discover, allowing analysts to seamlessly transition from their initial query to various workflows, such as building visualizations, creating reports, and generating alert monitors. The primary goal is to provide a consistent query language experience across these workflows, eliminating the need for analysts to learn different languages or interfaces for each feature.

Motivation

OpenSearch Discover currently serves as the starting point for analysts to refine and validate their queries. However, once the query is finalized, analysts must navigate to different areas of the application to leverage the query for specific features, such as building visualizations or creating alerts. This process can be disjointed and inefficient, especially when the desired features support different query languages or interfaces.

By establishing Discover as an extensible jump-off point, analysts can initiate various workflows directly from their initial query, streamlining the overall experience and improving productivity. Additionally, maintaining a consistent query language experience across these workflows will reduce the learning curve and eliminate the need for analysts to adapt to different languages or interfaces for each feature.

Proposed Solution

1. Extensible Jump-off Point: Enhance OpenSearch Discover to serve as an extensible jump-off point, allowing analysts to initiate various workflows directly from their initial query. This could include options to build visualizations, create reports, generate alert monitors, and potentially other workflows based on community feedback and requirements.
2. Consistent Query Language Experience: Ensure that the query language used in Discover (e.g., DQL, PPL, SQL) is consistently supported across the initiated workflows. Analysts should be able to use their preferred query language throughout the entire process without having to learn a new language or interface for each feature.
3. Query Translation (potential option): Implement a query translation mechanism to translate queries from one language to another (e.g., DQL to PPL, PPL to Query DSL) when necessary. This optional feature could enable analysts to switch between languages seamlessly without having to rewrite their queries manually.
4. Workflow Integration: Integrate the initiated workflows seamlessly within the OpenSearch interface, providing a cohesive and intuitive experience for analysts as they move between different areas of the tool.
5. Documentation and Examples: Update the documentation and provide examples to guide analysts on using the extensible jump-off point and consistent query language experience across different workflows.

Considerations

1. Inconsistent Language Support across Features: Currently, different OpenSearch features support different query languages or interfaces, which can create friction and inconsistency for analysts. Addressing this inconsistency is crucial for providing a seamless query language experience across workflows.
2. User Interface Design: Integrating the extensible jump-off point and consistent query language experience may require adjustments to the user interface (UI) to ensure a cohesive and intuitive experience for analysts.
3. Performance Implications: Extending query language support across multiple workflows may have performance implications, especially for complex queries or large datasets. Thorough testing and optimization will be required to ensure acceptable performance levels.
4. Backward Compatibility: Ensuring backward compatibility with existing queries and workflows is crucial to minimize disruption for existing users.
5. Community Feedback: Gathering feedback from the OpenSearch community, particularly power users and contributors, will be valuable in identifying potential issues, edge cases, and additional requirements for the extensible jump-off point and consistent query language experience.

Next Steps

1. Gather feedback and input from the OpenSearch community on this proposal.
2. Conduct a detailed analysis of the technical requirements and implications of the proposed solution.

Please provide your feedback, suggestions, or concerns regarding this proposal.