[Discover] Remaining work on Fields API

[majagrubic]

7.12

• Unknown/unmapped fields should show in source mode in the sidebar (Discover doesn't show unmapped/unknown fields when in _source mode #91697)
• Default column behavior is broken when switching between reading from source and fields API (fixed with: [Discover] Adapt default column behavior #89826)
• Fields filter doesn't work ([Discover] Fields filter doesn't work #88193); broken on current master as well (PR open)
• Unmapped fields are not shown ([Discover] Unmapped fields are not shown #88192)
• Unknown/unmapped fields should show in source mode in the sidebar (Discover doesn't show unmapped/unknown fields when in _source mode #91697)
• Support for nested fields (ES PR was merged, but not verified on Discover side)
• Date nanons are not properly displayed (ES issue) -> workaround is that Kibana will request explicit date format with every date field (see [Search Source] Specify strict_date_optional_time_nanos as a format #90415 (comment) for details)
• Grouping multifields in the expanded documents view ([Discover] Group multifields in document table view #85941)
• Multi fields in the sidebar: removal button needs to be implemented (removing works only from the grid, not the field itself)
• Fixing fieldsFromSource in the saved search Saved Objects ([Discover] Fix search source behavior #91459)
• Do not include unmapped fields in a search request when source filtering is enabled; might break ES otherwise [Search Source] Fields API: Do not request unmapped fields if source filtering is used #91907
• Disable source fetching ui setting on 7.12 and 7.x branch (to default to loading via fields API) @flash1293 Enable fields API by default #91986 Enable fields API by default #91988
• Unmapped fields not shown in single doc viewer and context view Discover: "Show unmapped fields" flag does not carry over to single doc viewer and context viewer #91688
• Context view retains wrong default column when toggling the UI setting ([Discover] When switching from Fields API to _source, default column is confused in context view #91493)
• Context view uses wrong default column when navigating there from new data grid (Discover: Context view shows _source column if data grid is enabled #91633)
• Exclude unmapped fields if source filtering is enabled [Search Source] Fields API: Do not request unmapped fields if source filtering is used #91907 (important for 7.12? I guess so) @majagrubic [Search Source] Do not request unmapped fields if source filters are provided #91921 (comment)
• Date nanos in context view doesn't work Discover: Context view doesn't work for date nanos if fields API is enabled #92005 (we definitely need this for 7.12) @flash1293
• "Show unmapped fields" on by default (without visible switch) @majagrubic PR [Discover] Always show unmapped fields #91735
• Source column not shown correctly in embeddable Discover: Source column in embeddable not shown correctly #91982 @flash1293 PR Discover: Handle source column differences in embeddable as well #91987
• Field aliases don't properly Discover doesn't show field aliases #21705
• Add back "Hide missing fields" switch @wylieconlon PR [Discover] Always show the "hide missing fields" toggle #91889
• Reconstruct object root fields as columns in the data grid (https://github.com/elastic/kibana-team/issues/256#issuecomment-717435122) [Discover] Missing data on top-level object fields when switching from _source-based to fields-based data #91953 @wylieconlon PR [Discover] Show correct data for top level object columns #91954
• Default column not respected ([Discover] Default column not respected when using fields API #92018) @majagrubic

In progress

• Testing

Later

• Rollup index pattern doesn't work with source fetching Discover: Rollup index pattern doesn't work with source fetching #92362
• search_after API does not accept data_nanos format of fields API necessarily (thus Discover context view might break) (requires Formatted sort values for search_after elasticsearch#69192)
• Add a server-side config value that will serve as a "master switch" and an override of the advanced setting toggle Discover: Add yml setting to disable new fields API fetching #91835
• Export to CSV still used _source implementation
• What happens with shared URLs that contain _source in the URL? We'd need to handle this scenario -> will go through the same migration system than saved objects do already
• create single Visualize button that would in case of one visualizable multi-fields visualize that instead of the parent field

After removal of _source mode

• related to previous point: remove _source from meta fields in Advanced Settings (and remove _source from default columns)
• Remove _source from existing saved searches' columns ([Discover] Remove _source from columns in saved searches #91812)

The 7.12 experience

There are two modes, controlled by the "Read fields from _source" advanced setting (discover:searchFieldsFromSource). If that setting is "off" (default in 7.12) fields will be read using the fields API (also called fields mode below), if it's "on" _source will be used (also called source mode below).

• Document/_source column:
  • source mode _source is an available column representing the _source of the field.
  • fields mode When the Fields API is used, the _source column is filtered from all views (in saved searches, default columns via advanced settings, ...). If _source was the only displayed column, it's replaced by the "Document" column showing all available fields in a flattened way
  • fields mode You can't mix the "document" column with other columns (as you can with source mode).
• Runtime fields (defined on the mapping):
  • source mode will show up in the field list, but will always appear empty inside the documents in any of the discover apps, unless they are of type date or date_nanos in which case they will work fine even in source mode.
  • fields mode will work as any other indexed field
• Unmapped fields:
  • source mode Unmapped fields will be shown (as they have been so far)
  • fields mode If fields API is used (the default), will be shown in the sidebar and in the documents. If source filtering (via the index pattern) is used, they won't show up anymore. In contrast to before unmapped leafs will be shown in the sidebar (see Discover: Leaves of disabled object fields are shown separately in the fields list #92019)
  • Future plans: We plan to remove unmapped fields completely in 8.0 and instead other technologies to access that data (like runtime fields) need to be used.
• Object fields:
  • source mode Object fields (root and leaf fields) work as beforehand¹, only columns with array of objects will now be shown in one line instead of multi line JSON
  • fields mode Object roots won't be shown in the field list anymore (unless you have an old saved search containing one under "selected fields" so you can remove it). Leaf fields will now be correctly detected as having data, even when the document has an array of objects.
  • fields mode In the table root fields (which can only happen via manipulating the URL or opening existing saved searches which had them added) will be shown in a similar way to the "Document" field, i.e. a summary representation of that object:
    
  • fields mode Leaf fields will be working for all objects and if the document contained an array of objects, show the values of all objects in that array flattened:
    
• Nested fields:
  • source mode Nested fields behave as beforehand. They behaved the same as object fields so far (see the footnote for more details).
  • fields mode The root of a nested field can be added from the field list and will show the structured JSON of that field in the document. Nested field columns now show the JSON correctly for all documents. Earlier it wouldn't show for documents that had only one object (instead of an array of objects) indexed into that field.
  • fields mode If you have a saved search with a nested leaf field as a column (which could have been possible earlier, if there was a document having only one nested document and not an array) this will show as empty now. We want to have this behavior going forward since we assume the nested mapping type is only used when you index an array of objects, since it otherwise does not make much sense to be used.

¹ The sidebar shows the object field (aka root) with an unknown icon if at least one document loaded has an array of objects for that object field, otherwise the root field won't be shown. The leaf fields will be shown with their correct field icons. Leaf fields that do not appear in any document or only in documents that have arrays of objects for this object field, will NOT appear in the field list unless "Hide missing fields" is switched off, e.g. in the screenshot below user.age which only appears in the first document that has an array of users will not appear in the field list by default. In the table root fields as columns will show the JSON of the fields content ONLY if that document contained an array of objects and will be empty otherwise. Leaf fields as columns will ONLY show values if the document did NOT contain an array of objects and stay empty otherwise:

Known issues 7.12

• Environments that are running CCS with mixed versions of ES will not show all fields in Discover, but only the fields from the ES versions that support the Fields API.
• Surrounding document view does fail with date_nanos with a custom formatter ([Discover] Surrounding document views breaks on date nanos with custom formatter #92595)
• Rollup v1 index patterns are not working in Discover when in source mode (Discover: Rollup index pattern doesn't work with source fetching #92362)
• Runtime fields not working in CSV export ([Reporting CSV] Show Runtime Fields in CSV Export #81079)

[timroes]

EOD Update 2021-02-17

Closed

• Requesting date fields explicitly merged ([Search Source] Request date fields explicitly #91478)
• Saved Search ending up in Saved object merged ([Discover] Fix search source behavior #91459)

New

• Context view retains wrong default column when toggling the UI setting ([Discover] When switching from Fields API to _source, default column is confused in context view #91493)
• Context view uses wrong default column when navigating there from new data grid (Discover: Context view shows _source column if data grid is enabled #91633)
• Toggling "Show unmapped fields" is not re-rendering the table (Discover: Toggling "Show unmapped fields" is not re-rendering the table #91644)
• Unmapped fields not shown in single doc viewer and context view (Discover: "Show unmapped fields" flag does not carry over to single doc viewer and context viewer #91688)
• Single document viewer is not using fields API yet (PR started: [Discover][Doc] Implement usage of fields search when discover:searchFieldsFromSource is set to false #91627)

[flash1293]

EOD Update 2021-02-18

Closed

• Always show unmapped fields in single doc/surrounding docs view merged ([Discover] Always request unmapped fields in a single doc view and a context view #91825)
• Fix handling of _source column merged (Discover: Add handling for source column #91815)
• Show unmapped fields in sidebar when using _source merged ([Discover] Show unmapped fields when reading from source #91719)

Added to 7.12 list

• "Show unmapped fields" on by default (without visible switch)
• Add back "Hide missing fields" switch @wylieconlon
• Disable source fetching ui setting on 7.12 and 7.x branch (to default to loading via fields API)
• Timeboxed: Reconstruct object root fields as columns in the data grid

PRs:
https://github.com/elastic/kibana/pulls?q=is%3Apr+is%3Aclosed+label%3AFeature%3ADiscover+sort%3Aupdated-desc++merged%3A%3E%3D2021-02-18+

[flash1293]

Update 2021-02-19

New

• Don't request all fields if not necessary [Discover] Only request fields that are shown to user when using new fields API #91897 (valid?)
• Exclude unmapped fields if source filtering is enabled [Search Source] Fields API: Do not request unmapped fields if source filtering is used #91907
• Source column not shown correctly Discover: Source column in embeddable not shown correctly #91982
• Date nanos doesn't work on context view Discover: Context view doesn't work for date nanos if fields API is enabled #92005
• Default column not respected ([Discover] Default column not respected when using fields API #92018)
• Leaves of enabled: false object types are shown as separate fields Discover: Leaves of disabled object fields are shown separately in the fields list #92019

Closed

• Exclude unmapped fields if source filtering is enabled [Search Source] Fields API: Do not request unmapped fields if source filtering is used #91907
• Disable source fetching ui setting on 7.12 and 7.x branch
• Date nanos doesn't work on context view

PRs:
https://github.com/elastic/kibana/pulls?q=is%3Apr+is%3Aclosed+label%3AFeature%3ADiscover+sort%3Aupdated-desc++merged%3A%3E%3D2021-02-19+

[timroes]

Update 2021-02-22

New

• Filter list shows all fields, although they are not present in the field list -> confusing behavior

Closed

Merged PRs

[flash1293]

Update 2021-02-23

New

• Rollup index pattern doesn't work with source fetching Discover: Rollup index pattern doesn't work with source fetching #92362
  • -> We will discuss if it makes sense to fix this (since rollup v1 in Discover did never make much sense), but even if fixed we won't fix it in 7.12 anymore.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[timroes]

Going to close this, since we've introduced the fields API across Discover and will track other issues that might come up individually.