Snapshot and restore doesn't compute Index math expressions

[32bitbradley]

Kibana version: 7.10.0

Elasticsearch version: 7.10.0

Server OS version: Centos 7

Browser version: Chrome

Browser OS version: Windows 10

Original install method (e.g. download page, yum, from source, etc.): Yum

Describe the bug:

Snapshot and restore doesn't compute Index math expressions for 'Data streams and indices'. Instead, the whole expression is passed as a string/fails to compile and therefore doesn't match any indices, causing the snapshot to fail.

For example, todays date being 08/12/2020, the expression <cowrie-honeypot-{now/d-1d}> is being pased as literal string/failing to compute, resulting in "cowrie-honeypot-{now/d-1d}>" rather than "cowrie-honeypot-07/12/2020"

Steps to reproduce:

1. Using the Kibana Snapshot and restore GUI, create a policy using an index template that has math expressions, such as '<cowrie-honeypot-{now/d-1d}>'
2. Save and run the policy either manually or on the schedule.
3. Snapshot will fail, and details output will show that the expression is not being compiled when searching for indexes.

Expected behavior:

The maths expression shpuld be compiled when looking for indexes, for example '<cowrie-honeypot-{now/d-1d}>' to 'cowrie-honeypot-07/12/2020'

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Snapshot failure details:

{
  "type": "index_not_found_exception",
  "reason": "no such index [<cowrie-honeypot-{now/d-1d}>]",
  "resource.type": "index_or_alias",
  "resource.id": "<cowrie-honeypot-{now/d-1d}>",
  "index_uuid": "_na_",
  "index": "<cowrie-honeypot-{now/d-1d}>",
  "stack_trace": "[<cowrie-honeypot-{now/d-1d}>] IndexNotFoundException[no such index [<cowrie-honeypot-{now/d-1d}>]]\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:945)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:882)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:838)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.dataStreamNames(IndexNameExpressionResolver.java:146)\n\tat org.elasticsearch.snapshots.SnapshotsService$2.execute(SnapshotsService.java:409)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository$1.execute(BlobStoreRepository.java:371)\n\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)\n\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702)\n\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219)\n\tat org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"
}

Any additional context:

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[cjcenizal]

Hey @elastic/es-core-features looking at the error in the PR description, this looks like a problem at the API level. Can you confirm?

[dakrone]

Yes, I believe this is an API-level issue. I also think it's related to #62164

[hendry-lim]

Did a quick test:

Version	working?
7.9.2	❌
7.14.0	✔️
7.15.0-SNAPSHOT 83f2fe8a	✔️