Migrate security routes to a new Elasticsearch client. #84528

azasypkin · 2020-11-30T15:22:52Z

We should eventually remove all dependencies on the Legacy ES Client, and in this PR I migrated all our routes to use new ES client.

Related to: #83910

x-pack/plugins/security/server/plugin.ts

@@ -185,6 +185,7 @@ export class Plugin {
      taskManager,
    });

+    core.getStartServices().then(([{ elasticsearch }]) => elasticsearch.client);


azasypkin · 2020-11-30T17:19:21Z

x-pack/plugins/security/server/routes/api_keys/get.test.ts

      if (apiResponse) {
-        mockScopedClusterClient.callAsCurrentUser.mockImplementation(apiResponse);
+        mockContext.core.elasticsearch.client.asCurrentUser.security.getApiKey.mockImplementation(


note: it's not really easy to use these mocks as the signature requires many other fields in addition to body.

elasticmachine · 2020-11-30T17:27:23Z

Pinging @elastic/kibana-security (Team:Security)

legrego · 2020-12-01T11:54:51Z

@thomheymann thanks for taking this review! ❤️

kibanamachine · 2020-12-02T13:06:24Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request
Commit: db31c6a
Flaky suites:
- xpack-kibana-ciGroup3

Test Failures

Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/ml/data_frame_analytics/cloning·ts.machine learning data frame analytics jobs cloning supported by UI form classification job supported by the form runs the clone analytics job and displays it correctly in the job list

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: machine learning
[00:00:00]           └-> "before all" hook
[00:00:00]           └-> "before all" hook
[00:00:00]             │ debg creating role ft_ml_source
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added role [ft_ml_source]
[00:00:00]             │ debg creating role ft_ml_source_readonly
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added role [ft_ml_source_readonly]
[00:00:00]             │ debg creating role ft_ml_dest
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added role [ft_ml_dest]
[00:00:00]             │ debg creating role ft_ml_dest_readonly
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added role [ft_ml_dest_readonly]
[00:00:00]             │ debg creating role ft_ml_ui_extras
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added role [ft_ml_ui_extras]
[00:00:00]             │ debg creating role ft_default_space_ml_all
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added role [ft_default_space_ml_all]
[00:00:00]             │ debg creating role ft_default_space_ml_read
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added role [ft_default_space_ml_read]
[00:00:00]             │ debg creating role ft_default_space_ml_none
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added role [ft_default_space_ml_none]
[00:00:00]             │ debg creating user ft_ml_poweruser
[00:00:00]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added user [ft_ml_poweruser]
[00:00:00]             │ debg created user ft_ml_poweruser
[00:00:00]             │ debg creating user ft_ml_poweruser_spaces
[00:00:00]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added user [ft_ml_poweruser_spaces]
[00:00:00]             │ debg created user ft_ml_poweruser_spaces
[00:00:00]             │ debg creating user ft_ml_viewer
[00:00:00]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [.ds-ilm-history-5-000001] creating index, cause [initialize_data_stream], templates [ilm-history], shards [1]/[0]
[00:00:00]             │ info [o.e.c.m.MetadataCreateDataStreamService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] adding data stream [ilm-history-5] with write index [.ds-ilm-history-5-000001] and backing indices []
[00:00:00]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ds-ilm-history-5-000001][0]]])." previous.health="YELLOW" reason="shards started [[.ds-ilm-history-5-000001][0]]"
[00:00:00]             │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] moving index [.ds-ilm-history-5-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [ilm-history-ilm-policy]
[00:00:00]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added user [ft_ml_viewer]
[00:00:00]             │ debg created user ft_ml_viewer
[00:00:00]             │ debg creating user ft_ml_viewer_spaces
[00:00:00]             │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] moving index [.ds-ilm-history-5-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [ilm-history-ilm-policy]
[00:00:00]             │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] moving index [.ds-ilm-history-5-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [ilm-history-ilm-policy]
[00:00:00]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added user [ft_ml_viewer_spaces]
[00:00:00]             │ debg created user ft_ml_viewer_spaces
[00:00:00]             │ debg creating user ft_ml_unauthorized
[00:00:01]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added user [ft_ml_unauthorized]
[00:00:01]             │ debg created user ft_ml_unauthorized
[00:00:01]             │ debg creating user ft_ml_unauthorized_spaces
[00:00:01]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] added user [ft_ml_unauthorized_spaces]
[00:00:01]             │ debg created user ft_ml_unauthorized_spaces
[00:38:34]           └-: data frame analytics
[00:38:34]             └-> "before all" hook
[00:41:28]             └-: jobs cloning supported by UI form
[00:41:28]               └-> "before all" hook
[00:41:28]               └-> "before all" hook
[00:41:28]                 │ debg applying update to kibana config: {"dateFormat:tz":"UTC"}
[00:41:28]                 │ debg SecurityPage.forceLogout
[00:41:28]                 │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=100
[00:41:28]                 │ debg --- retry.tryForTime error: .login-form is not displayed
[00:41:29]                 │ debg Redirecting to /logout to force the logout
[00:41:29]                 │ debg Waiting on the login form to appear
[00:41:29]                 │ debg Waiting for Login Page to appear.
[00:41:29]                 │ debg Waiting up to 100000ms for login page...
[00:41:29]                 │ debg browser[INFO] http://localhost:61131/logout?_t=1606910984340 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:41:29]                 │
[00:41:29]                 │ debg browser[INFO] http://localhost:61131/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:41:29]                 │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=2500
[00:41:32]                 │ debg browser[INFO] http://localhost:61131/login?msg=LOGGED_OUT 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:41:32]                 │
[00:41:32]                 │ debg browser[INFO] http://localhost:61131/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:41:32]                 │ debg --- retry.tryForTime error: .login-form is not displayed
[00:41:33]                 │ERROR browser[SEVERE] http://localhost:61131/internal/spaces/_active_space - Failed to load resource: the server responded with a status of 401 (Unauthorized)
[00:41:33]                 │ debg browser[INFO] http://localhost:61131/38519/bundles/core/core.entry.js 12:194343 "Detected an unhandled Promise rejection.
[00:41:33]                 │      Error: Unauthorized"
[00:41:33]                 │ERROR browser[SEVERE] http://localhost:61131/38519/bundles/core/core.entry.js 5:3002 
[00:41:33]                 │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=2500
[00:41:33]                 │ debg TestSubjects.exists(loginForm)
[00:41:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="loginForm"]') with timeout=2500
[00:41:33]                 │ debg Waiting for Login Form to appear.
[00:41:33]                 │ debg Waiting up to 100000ms for login form...
[00:41:33]                 │ debg TestSubjects.exists(loginForm)
[00:41:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="loginForm"]') with timeout=2500
[00:41:33]                 │ debg TestSubjects.setValue(loginUsername, ft_ml_poweruser)
[00:41:33]                 │ debg TestSubjects.click(loginUsername)
[00:41:33]                 │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:41:33]                 │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:41:33]                 │ debg TestSubjects.setValue(loginPassword, mlp001)
[00:41:33]                 │ debg TestSubjects.click(loginPassword)
[00:41:33]                 │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:41:33]                 │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:41:33]                 │ debg TestSubjects.click(loginSubmit)
[00:41:33]                 │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:41:33]                 │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:41:33]                 │ debg Waiting for login result, expected: chrome.
[00:41:33]                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"] .app-wrapper:not(.hidden-chrome)') with timeout=20000
[00:41:34]                 │ proc [kibana]   log   [12:09:48.952] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:41:36]                 │ debg browser[INFO] http://localhost:61131/app/home 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:41:36]                 │
[00:41:36]                 │ debg browser[INFO] http://localhost:61131/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:41:36]                 │ debg Finished login process currentUrl = http://localhost:61131/app/home#/
[00:41:36]                 │ debg Waiting up to 20000ms for logout button visible...
[00:41:36]                 │ debg TestSubjects.exists(userMenuButton)
[00:41:36]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenuButton"]') with timeout=2500
[00:41:36]                 │ debg TestSubjects.exists(userMenu)
[00:41:36]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=2500
[00:41:38]                 │ debg --- retry.tryForTime error: [data-test-subj="userMenu"] is not displayed
[00:41:39]                 │ debg TestSubjects.click(userMenuButton)
[00:41:39]                 │ debg Find.clickByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:41:39]                 │ debg Find.findByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:41:39]                 │ debg TestSubjects.exists(userMenu)
[00:41:39]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=120000
[00:41:39]                 │ debg TestSubjects.exists(userMenu > logoutLink)
[00:41:39]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"] [data-test-subj="logoutLink"]') with timeout=2500
[00:41:39]               └-: classification job supported by the form
[00:41:39]                 └-> "before all" hook
[00:41:39]                 └-> "before all" hook
[00:41:39]                   │ info [ml/bm_classification] Loading "mappings.json"
[00:41:39]                   │ info [ml/bm_classification] Loading "data.json.gz"
[00:41:39]                   │ info [ml/bm_classification] Skipped restore for existing index "ft_bank_marketing"
[00:41:39]                   │ debg Searching for 'index-pattern' with title 'ft_bank_marketing'...
[00:41:39]                   │ debg  > Found '1ade3b90-3497-11eb-9fd5-b9884d59120c'
[00:41:39]                   │ debg Index pattern with title 'ft_bank_marketing' already exists. Nothing to create.
[00:41:39]                   │ debg Creating data frame analytic job with id 'bm_1_1606908494529'...
[00:41:39]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [.ml-config] creating index, cause [auto(bulk api)], templates [.ml-config], shards [1]/[1]
[00:41:39]                   │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] updating number_of_replicas to [0] for indices [.ml-config]
[00:41:39]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [.ml-annotations-6] creating index, cause [api], templates [], shards [1]/[1]
[00:41:39]                   │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] updating number_of_replicas to [0] for indices [.ml-annotations-6]
[00:41:39]                   │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [.ml-config/7QXBDri1RRWoyzU9NB4wmw] update_mapping [_doc]
[00:41:39]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [.ml-notifications-000001] creating index, cause [auto(bulk api)], templates [.ml-notifications-000001], shards [1]/[1]
[00:41:39]                   │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] updating number_of_replicas to [0] for indices [.ml-notifications-000001]
[00:41:40]                   │ debg Waiting up to 5000ms for 'bm_1_1606908494529' to exist...
[00:41:40]                   │ debg Fetching data frame analytics job 'bm_1_1606908494529'...
[00:41:40]                   │ debg navigating to ml url: http://localhost:61131/app/ml
[00:41:40]                   │ debg navigate to: http://localhost:61131/app/ml
[00:41:40]                   │ debg browser[INFO] http://localhost:61131/app/ml?_t=1606910995716 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:41:40]                   │
[00:41:40]                   │ debg browser[INFO] http://localhost:61131/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:41:40]                   │ debg ... sleep(700) start
[00:41:41]                   │ debg ... sleep(700) end
[00:41:41]                   │ debg returned from get, calling refresh
[00:41:42]                   │ERROR browser[SEVERE] http://localhost:61131/38519/bundles/core/core.entry.js 12:193396 TypeError: Failed to fetch
[00:41:42]                   │          at _callee3$ (http://localhost:61131/38519/bundles/core/core.entry.js:6:43940)
[00:41:42]                   │          at s (http://localhost:61131/38519/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:1751466)
[00:41:42]                   │          at Generator._invoke (http://localhost:61131/38519/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:1751219)
[00:41:42]                   │          at Generator.forEach.e.<computed> [as throw] (http://localhost:61131/38519/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:1751823)
[00:41:42]                   │          at fetch_asyncGeneratorStep (http://localhost:61131/38519/bundles/core/core.entry.js:6:38998)
[00:41:42]                   │          at _throw (http://localhost:61131/38519/bundles/core/core.entry.js:6:39406)
[00:41:42]                   │ debg browser[INFO] http://localhost:61131/app/ml?_t=1606910995716 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:41:42]                   │
[00:41:42]                   │ debg browser[INFO] http://localhost:61131/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:41:42]                   │ debg currentUrl = http://localhost:61131/app/ml
[00:41:42]                   │          appUrl = http://localhost:61131/app/ml
[00:41:42]                   │ debg TestSubjects.find(kibanaChrome)
[00:41:42]                   │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:41:42]                   │ debg ... sleep(501) start
[00:41:43]                   │ debg ... sleep(501) end
[00:41:43]                   │ debg in navigateTo url = http://localhost:61131/app/ml/overview
[00:41:43]                   │ debg --- retry.try error: URL changed, waiting for it to settle
[00:41:43]                   │ debg ... sleep(501) start
[00:41:44]                   │ debg ... sleep(501) end
[00:41:44]                   │ debg in navigateTo url = http://localhost:61131/app/ml/overview
[00:41:44]                   │ debg TestSubjects.exists(statusPageContainer)
[00:41:44]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:41:46]                   │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:41:47]                   │ debg TestSubjects.exists(mlApp)
[00:41:47]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlApp"]') with timeout=2000
[00:41:47]                   │ debg TestSubjects.click(~mlMainTab & ~dataFrameAnalytics)
[00:41:47]                   │ debg Find.clickByCssSelector('[data-test-subj~="mlMainTab"][data-test-subj~="dataFrameAnalytics"]') with timeout=10000
[00:41:47]                   │ debg Find.findByCssSelector('[data-test-subj~="mlMainTab"][data-test-subj~="dataFrameAnalytics"]') with timeout=10000
[00:41:47]                   │ debg TestSubjects.exists(~mlMainTab & ~dataFrameAnalytics & ~selected)
[00:41:47]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="mlMainTab"][data-test-subj~="dataFrameAnalytics"][data-test-subj~="selected"]') with timeout=120000
[00:41:47]                   │ debg TestSubjects.exists(mlPageDataFrameAnalytics)
[00:41:47]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlPageDataFrameAnalytics"]') with timeout=120000
[00:41:47]                   │ debg TestSubjects.exists(~mlAnalyticsTable)
[00:41:47]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="mlAnalyticsTable"]') with timeout=60000
[00:41:47]                   │ debg TestSubjects.exists(mlAnalyticsTable loaded)
[00:41:47]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsTable loaded"]') with timeout=30000
[00:41:48]                   │ debg TestSubjects.exists(~mlAnalyticsTable)
[00:41:48]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="mlAnalyticsTable"]') with timeout=60000
[00:41:48]                   │ debg TestSubjects.exists(mlAnalyticsTable loaded)
[00:41:48]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsTable loaded"]') with timeout=30000
[00:41:48]                   │ debg TestSubjects.find(mlAnalyticsTableContainer)
[00:41:48]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsTableContainer"]') with timeout=10000
[00:41:48]                   │ debg TestSubjects.find(mlAnalyticsTableContainer)
[00:41:48]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsTableContainer"]') with timeout=10000
[00:41:48]                   │ debg TestSubjects.find(~mlAnalyticsTable)
[00:41:48]                   │ debg Find.findByCssSelector('[data-test-subj~="mlAnalyticsTable"]') with timeout=10000
[00:41:48]                   │ debg TestSubjects.exists(mlAnalyticsJobDeleteButton)
[00:41:48]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsJobDeleteButton"]') with timeout=2500
[00:41:51]                   │ debg --- retry.tryForTime error: [data-test-subj="mlAnalyticsJobDeleteButton"] is not displayed
[00:41:51]                   │ debg TestSubjects.click(~mlAnalyticsTable > ~row-bm_1_1606908494529 > euiCollapsedItemActionsButton)
[00:41:51]                   │ debg Find.clickByCssSelector('[data-test-subj~="mlAnalyticsTable"] [data-test-subj~="row-bm_1_1606908494529"] [data-test-subj="euiCollapsedItemActionsButton"]') with timeout=10000
[00:41:51]                   │ debg Find.findByCssSelector('[data-test-subj~="mlAnalyticsTable"] [data-test-subj~="row-bm_1_1606908494529"] [data-test-subj="euiCollapsedItemActionsButton"]') with timeout=10000
[00:41:51]                   │ debg TestSubjects.exists(mlAnalyticsJobDeleteButton)
[00:41:51]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsJobDeleteButton"]') with timeout=5000
[00:41:51]                   │ debg TestSubjects.click(mlAnalyticsJobCloneButton)
[00:41:51]                   │ debg Find.clickByCssSelector('[data-test-subj="mlAnalyticsJobCloneButton"]') with timeout=10000
[00:41:51]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsJobCloneButton"]') with timeout=10000
[00:41:51]                   │ debg TestSubjects.exists(mlAnalyticsCreationContainer)
[00:41:51]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsCreationContainer"]') with timeout=120000
[00:41:52]                 └-> opens the existing job in the data frame analytics job wizard
[00:41:52]                   └-> "before each" hook: global before each
[00:41:52]                   │ debg === TEST STEP === should open the wizard with a proper header
[00:41:52]                   │ debg TestSubjects.getVisibleText(mlDataFrameAnalyticsWizardHeaderTitle)
[00:41:52]                   │ debg TestSubjects.find(mlDataFrameAnalyticsWizardHeaderTitle)
[00:41:52]                   │ debg Find.findByCssSelector('[data-test-subj="mlDataFrameAnalyticsWizardHeaderTitle"]') with timeout=10000
[00:41:52]                   │ debg TestSubjects.exists(mlAnalyticsCreateJobWizardConfigurationStep active)
[00:41:52]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardConfigurationStep active"]') with timeout=120000
[00:41:52]                   └- ✓ pass  (58ms) "machine learning data frame analytics jobs cloning supported by UI form classification job supported by the form opens the existing job in the data frame analytics job wizard"
[00:41:52]                 └-> navigates through the wizard, checks and sets all needed fields
[00:41:52]                   └-> "before each" hook: global before each
[00:41:52]                   │ debg === TEST STEP === should have correct init form values for config step
[00:41:52]                   │ debg TestSubjects.exists(mlAnalyticsCreation-classification-option selectedJobType)
[00:41:52]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsCreation-classification-option selectedJobType"]') with timeout=120000
[00:41:52]                   │ debg TestSubjects.exists(~mlAnalyticsCreateJobWizardDependentVariableSelect)
[00:41:52]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="mlAnalyticsCreateJobWizardDependentVariableSelect"]') with timeout=5000
[00:41:52]                   │ debg TestSubjects.exists(mlAnalyticsCreateJobWizardDependentVariableSelect loaded)
[00:41:52]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardDependentVariableSelect loaded"]') with timeout=30000
[00:41:52]                   │ debg comboBox.getComboBoxSelectedOptions, comboBoxSelector: ~mlAnalyticsCreateJobWizardDependentVariableSelect > comboBoxInput
[00:41:52]                   │ debg TestSubjects.find(~mlAnalyticsCreateJobWizardDependentVariableSelect > comboBoxInput)
[00:41:52]                   │ debg Find.findByCssSelector('[data-test-subj~="mlAnalyticsCreateJobWizardDependentVariableSelect"] [data-test-subj="comboBoxInput"]') with timeout=10000
[00:41:52]                   │ debg TestSubjects.getAttribute(mlAnalyticsCreateJobWizardTrainingPercentSlider, value, tryTimeout=120000, findTimeout=10000)
[00:41:52]                   │ debg TestSubjects.find(mlAnalyticsCreateJobWizardTrainingPercentSlider)
[00:41:52]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardTrainingPercentSlider"]') with timeout=10000
[00:41:52]                   │ debg TestSubjects.exists(mlAnalyticsCreationDataGrid loaded)
[00:41:52]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsCreationDataGrid loaded"]') with timeout=5000
[00:41:52]                   │ debg TestSubjects.exists(mlAnalyticsCreateJobWizardIncludesTable)
[00:41:52]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardIncludesTable"]') with timeout=8000
[00:41:53]                   │ debg TestSubjects.exists(mlAnalyticsCreateJobWizardIncludesSelect)
[00:41:53]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardIncludesSelect"]') with timeout=120000
[00:41:53]                   │ debg === TEST STEP === should continue to the additional options step
[00:41:53]                   │ debg TestSubjects.clickWhenNotDisabled(mlAnalyticsCreateJobWizardContinueButton)
[00:41:53]                   │ debg Find.clickByCssSelectorWhenNotDisabled('[data-test-subj="mlAnalyticsCreateJobWizardContinueButton"]') with timeout=10000
[00:41:53]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardContinueButton"]') with timeout=10000
[00:41:53]                   │ debg TestSubjects.exists(mlAnalyticsCreateJobWizardAdvancedStep active)
[00:41:53]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardAdvancedStep active"]') with timeout=120000
[00:41:53]                   │ debg === TEST STEP === should have correct init form values for additional options step
[00:41:53]                   │ debg TestSubjects.getAttribute(mlAnalyticsCreateJobWizardPredictionFieldNameInput, value, tryTimeout=120000, findTimeout=10000)
[00:41:53]                   │ debg TestSubjects.find(mlAnalyticsCreateJobWizardPredictionFieldNameInput)
[00:41:53]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardPredictionFieldNameInput"]') with timeout=10000
[00:41:53]                   │ debg === TEST STEP === should continue to the details step
[00:41:53]                   │ debg TestSubjects.clickWhenNotDisabled(mlAnalyticsCreateJobWizardContinueButton)
[00:41:53]                   │ debg Find.clickByCssSelectorWhenNotDisabled('[data-test-subj="mlAnalyticsCreateJobWizardContinueButton"]') with timeout=10000
[00:41:53]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardContinueButton"]') with timeout=10000
[00:41:54]                   │ debg TestSubjects.exists(mlAnalyticsCreateJobWizardDetailsStep active)
[00:41:54]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardDetailsStep active"]') with timeout=120000
[00:41:54]                   │ debg === TEST STEP === should have correct init form values for details step
[00:41:54]                   │ debg TestSubjects.getAttribute(mlAnalyticsCreateJobFlyoutJobIdInput, value, tryTimeout=120000, findTimeout=10000)
[00:41:54]                   │ debg TestSubjects.find(mlAnalyticsCreateJobFlyoutJobIdInput)
[00:41:54]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobFlyoutJobIdInput"]') with timeout=10000
[00:41:54]                   │ debg TestSubjects.getAttribute(mlDFAnalyticsJobCreationJobDescription, value, tryTimeout=120000, findTimeout=10000)
[00:41:54]                   │ debg TestSubjects.find(mlDFAnalyticsJobCreationJobDescription)
[00:41:54]                   │ debg Find.findByCssSelector('[data-test-subj="mlDFAnalyticsJobCreationJobDescription"]') with timeout=10000
[00:41:54]                   │ debg TestSubjects.getAttribute(mlAnalyticsCreateJobFlyoutDestinationIndexInput, value, tryTimeout=120000, findTimeout=10000)
[00:41:54]                   │ debg TestSubjects.find(mlAnalyticsCreateJobFlyoutDestinationIndexInput)
[00:41:54]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobFlyoutDestinationIndexInput"]') with timeout=10000
[00:41:54]                   │ debg TestSubjects.setValueWithChecks(mlAnalyticsCreateJobFlyoutJobIdInput, bm_1_1606908494529_clone)
[00:41:54]                   │ debg TestSubjects.click(mlAnalyticsCreateJobFlyoutJobIdInput)
[00:41:54]                   │ debg Find.clickByCssSelector('[data-test-subj="mlAnalyticsCreateJobFlyoutJobIdInput"]') with timeout=10000
[00:41:54]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobFlyoutJobIdInput"]') with timeout=10000
[00:41:56]                   │ debg TestSubjects.getAttribute(mlAnalyticsCreateJobFlyoutJobIdInput, value, tryTimeout=120000, findTimeout=10000)
[00:41:56]                   │ debg TestSubjects.find(mlAnalyticsCreateJobFlyoutJobIdInput)
[00:41:56]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobFlyoutJobIdInput"]') with timeout=10000
[00:41:56]                   │ debg TestSubjects.setValueWithChecks(mlAnalyticsCreateJobFlyoutDestinationIndexInput, user-bm_1_1606908494529_clone)
[00:41:56]                   │ debg TestSubjects.click(mlAnalyticsCreateJobFlyoutDestinationIndexInput)
[00:41:56]                   │ debg Find.clickByCssSelector('[data-test-subj="mlAnalyticsCreateJobFlyoutDestinationIndexInput"]') with timeout=10000
[00:41:56]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobFlyoutDestinationIndexInput"]') with timeout=10000
[00:41:57]                   │ debg TestSubjects.getAttribute(mlAnalyticsCreateJobFlyoutDestinationIndexInput, value, tryTimeout=120000, findTimeout=10000)
[00:41:57]                   │ debg TestSubjects.find(mlAnalyticsCreateJobFlyoutDestinationIndexInput)
[00:41:57]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobFlyoutDestinationIndexInput"]') with timeout=10000
[00:41:57]                   │ debg === TEST STEP === should continue to the create step
[00:41:57]                   │ debg TestSubjects.clickWhenNotDisabled(mlAnalyticsCreateJobWizardContinueButton)
[00:41:57]                   │ debg Find.clickByCssSelectorWhenNotDisabled('[data-test-subj="mlAnalyticsCreateJobWizardContinueButton"]') with timeout=10000
[00:41:57]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardContinueButton"]') with timeout=10000
[00:41:58]                   │ debg TestSubjects.exists(mlAnalyticsCreateJobWizardCreateStep active)
[00:41:58]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardCreateStep active"]') with timeout=120000
[00:41:58]                   └- ✓ pass  (5.4s) "machine learning data frame analytics jobs cloning supported by UI form classification job supported by the form navigates through the wizard, checks and sets all needed fields"
[00:41:58]                 └-> runs the clone analytics job and displays it correctly in the job list
[00:41:58]                   └-> "before each" hook: global before each
[00:41:58]                   │ debg === TEST STEP === should have enabled Create button on a valid form input
[00:41:58]                   │ debg TestSubjects.isEnabled(mlAnalyticsCreateJobWizardCreateButton)
[00:41:58]                   │ debg TestSubjects.find(mlAnalyticsCreateJobWizardCreateButton)
[00:41:58]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardCreateButton"]') with timeout=10000
[00:41:58]                   │ debg === TEST STEP === should create a clone job
[00:41:58]                   │ debg TestSubjects.click(mlAnalyticsCreateJobWizardCreateButton)
[00:41:58]                   │ debg Find.clickByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardCreateButton"]') with timeout=10000
[00:41:58]                   │ debg Find.findByCssSelector('[data-test-subj="mlAnalyticsCreateJobWizardCreateButton"]') with timeout=10000
[00:41:58]                   │ debg TestSubjects.exists(analyticsWizardCardManagement)
[00:41:58]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="analyticsWizardCardManagement"]') with timeout=120000
[00:41:58]                   │ debg Waiting up to 5000ms for 'bm_1_1606908494529_clone' to exist...
[00:41:58]                   │ debg Fetching data frame analytics job 'bm_1_1606908494529_clone'...
[00:41:58]                   │ debg === TEST STEP === should finish analytics processing
[00:41:58]                   │ debg Waiting up to 60000ms for 'bm_1_1606908494529_clone' to have training_docs_count > 0...
[00:41:58]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:41:58]                   │ debg --- retry.waitForWithTimeout error: expected data frame analytics job 'bm_1_1606908494529_clone' to have training_docs_count > 0 (got 0)
[00:41:59]                   │ info [o.e.x.m.a.TransportStartDataFrameAnalyticsAction] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [bm_1_1606908494529_clone] Starting data frame analytics from state [null]
[00:41:59]                   │ info [o.e.x.c.m.u.MlIndexAndAlias] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] About to create first concrete index [.ml-state-000001] with alias [.ml-state-write]
[00:41:59]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [.ml-state-000001] creating index, cause [api], templates [.ml-state], shards [1]/[1]
[00:41:59]                   │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] updating number_of_replicas to [0] for indices [.ml-state-000001]
[00:41:59]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] moving index [.ml-state-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [ml-size-based-ilm-policy]
[00:41:59]                   │ info [o.e.x.c.m.u.MlIndexAndAlias] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] About to create first concrete index [.ml-stats-000001] with alias [.ml-stats-write]
[00:41:59]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [.ml-stats-000001] creating index, cause [api], templates [.ml-stats], shards [1]/[1]
[00:41:59]                   │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] updating number_of_replicas to [0] for indices [.ml-stats-000001]
[00:41:59]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] moving index [.ml-state-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [ml-size-based-ilm-policy]
[00:41:59]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:41:59]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:41:59]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] moving index [.ml-stats-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [ml-size-based-ilm-policy]
[00:41:59]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] moving index [.ml-state-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [ml-size-based-ilm-policy]
[00:41:59]                   │ info [o.e.x.m.d.DataFrameAnalyticsManager] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [bm_1_1606908494529_clone] Creating destination index [user-bm_1_1606908494529_clone]
[00:41:59]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] moving index [.ml-stats-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [ml-size-based-ilm-policy]
[00:41:59]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [user-bm_1_1606908494529_clone] creating index, cause [api], templates [], shards [1]/[1]
[00:41:59]                   │ info [o.e.x.m.d.DataFrameAnalyticsManager] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [bm_1_1606908494529_clone] Started reindexing
[00:41:59]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] moving index [.ml-stats-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [ml-size-based-ilm-policy]
[00:41:59]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:41:59]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:00]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:00]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:00]                   │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [bm_1_1606908494529_clone] Started loading data
[00:42:00]                   │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [bm_1_1606908494529_clone] Started analyzing
[00:42:00]                   │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-centos-tests-xxl-1606906578403241444] [bm_1_1606908494529_clone] Waiting for result processor to complete
[00:42:00]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:00]                   │ debg Waiting up to 120000ms for analytics state to be stopped...
[00:42:00]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:00]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:00]                   │ debg --- retry.waitForWithTimeout error: expected analytics state to be stopped but got analyzing
[00:42:01]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:01]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:01]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:01]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:01]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:01]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:02]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:02]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:02]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:02]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:02]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:02]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:03]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:03]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:03]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:03]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:03]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:03]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:04]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:04]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:04]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:04]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:04]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:04]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:05]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:05]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:05]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:05]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:05]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:05]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:06]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:06]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:06]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:06]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:06]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:06]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:07]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:07]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:07]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:07]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:07]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:07]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:08]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:08]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:08]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:09]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:09]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:09]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:09]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:09]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:09]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:10]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:10]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:10]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:10]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:10]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:10]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:11]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:11]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:11]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:11]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:11]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:11]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:12]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:12]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:12]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:12]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:12]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:12]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:13]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:13]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:13]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:13]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:13]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:13]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:14]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:14]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:14]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:14]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:14]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:14]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:15]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:15]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:15]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:15]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:15]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:15]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:16]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:16]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:16]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:16]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:16]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:16]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:17]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:17]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:17]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:17]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:17]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:17]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:18]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:18]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:18]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:18]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:18]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:18]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:19]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:19]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:19]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:19]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:19]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:19]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:20]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:20]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:20]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:20]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:20]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:20]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:21]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:21]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:21]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:21]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:21]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:21]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:22]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:22]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:22]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:22]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:22]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:22]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:23]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:23]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:23]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:23]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:23]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:23]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:24]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:24]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:24]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:24]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:24]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:24]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:25]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:25]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:25]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:25]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:25]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:25]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:26]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:26]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:26]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:26]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:26]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:26]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:27]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:27]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:27]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:27]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:27]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:27]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:28]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:28]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:28]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:28]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:28]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:28]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:29]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:29]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:29]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:29]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:29]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:29]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:30]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:30]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:30]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:30]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:30]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:30]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:31]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:31]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:31]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:31]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:31]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:31]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:32]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:32]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:32]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:32]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:32]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:32]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:33]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:33]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:33]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:33]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:33]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:33]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:34]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:34]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:34]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:34]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:34]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:34]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:35]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:35]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:35]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:35]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:35]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:35]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:36]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:36]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:36]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:36]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:36]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:37]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:37]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:37]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:37]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:38]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:38]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:38]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:38]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:38]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:38]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:39]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:39]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:39]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:39]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:39]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:39]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:40]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:40]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:40]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:40]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:40]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:40]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:41]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:41]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:41]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:41]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:41]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:41]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:42]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:42]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:42]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:42]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:42]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:42]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:43]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:43]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:43]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:43]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:43]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:43]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:44]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:44]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:44]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:44]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:44]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:44]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:45]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:45]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:45]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:45]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:45]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:45]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:46]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:46]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:46]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:46]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:46]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:46]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:47]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:47]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:47]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:47]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:47]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:47]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:48]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:48]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:48]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:48]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:48]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:48]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:49]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:49]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:49]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:49]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:49]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:49]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:50]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:50]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:50]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:50]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:50]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:50]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:51]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:51]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:51]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:51]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:51]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:51]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:52]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:52]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:52]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:52]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:52]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:52]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:53]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:53]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:53]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:53]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:53]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:53]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:54]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:54]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:54]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:54]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:54]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:54]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:55]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:55]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:55]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:55]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:55]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:55]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:56]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:56]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:56]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:56]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:56]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:56]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:57]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:57]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:57]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:57]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:57]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:57]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:58]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:58]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:58]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:58]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:58]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:58]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:59]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:59]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:59]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:42:59]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:42:59]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:42:59]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:00]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:00]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:00]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:00]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:00]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:00]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:01]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:01]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:01]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:01]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:01]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:01]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:02]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:02]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:02]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:02]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:02]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:02]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:03]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:03]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:03]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:03]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:03]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:03]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:04]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:04]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:04]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:04]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:04]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:04]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:05]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:05]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:05]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:06]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:06]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:06]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:06]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:06]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:06]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:07]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:07]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:07]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:07]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:07]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:07]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:08]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:08]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:08]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:08]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:08]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:08]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:09]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:09]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:09]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:09]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:09]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:09]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:10]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:10]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:10]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:10]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:10]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:10]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:11]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:11]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:11]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:11]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:11]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:11]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:12]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:12]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:12]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:12]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:12]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:12]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:13]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:13]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:13]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:13]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:13]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:13]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:14]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:14]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:14]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:14]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:14]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:14]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:15]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:15]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:15]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:15]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:15]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:15]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:16]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:16]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:16]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:16]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:16]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:16]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:17]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:17]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:17]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:17]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:17]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:17]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:18]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:18]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:18]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:18]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:18]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:18]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:19]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:19]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:19]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:19]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:19]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:19]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:20]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:20]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:20]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:20]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:20]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:20]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:21]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:21]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:21]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:21]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:21]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:21]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:22]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:22]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:22]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:22]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:22]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:22]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:23]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:23]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:23]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:23]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:23]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:23]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:24]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:24]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:24]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:24]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:24]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:24]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:25]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:25]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:25]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:25]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:25]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:25]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:26]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:26]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:26]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:26]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:26]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:26]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:27]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:27]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:27]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:27]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:27]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:27]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:28]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:28]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:28]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:28]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:28]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:28]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:29]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:29]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:29]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:29]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:29]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:29]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:30]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:30]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:30]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:30]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:30]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:30]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:31]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:31]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:31]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:31]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:31]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:31]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:32]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:32]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:32]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:32]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:32]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:32]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:33]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:33]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:33]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:34]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:34]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:34]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:34]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:34]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:34]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:35]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:35]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:35]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:35]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:35]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:35]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:36]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:36]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:36]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:36]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:36]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:36]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:37]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:37]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:37]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:37]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:37]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:37]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:38]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:38]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:38]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:38]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:38]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:38]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:39]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:39]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:39]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:39]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:39]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:39]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:40]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:40]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:40]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:40]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:40]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:40]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:41]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:41]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:41]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:41]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:41]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:41]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:42]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:42]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:42]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:42]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:42]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:42]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:43]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:43]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:43]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:43]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:43]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:43]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:44]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:44]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:44]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:44]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:44]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:44]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:45]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:45]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:45]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:45]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:45]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:45]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:46]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:46]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:46]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:46]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:46]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:46]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:47]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:47]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:47]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:47]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:47]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:47]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:48]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:48]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:48]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:48]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:48]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:48]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:49]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:49]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:49]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:49]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:49]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:49]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:50]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:50]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:50]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:50]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:50]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:50]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:51]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:51]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:51]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:51]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:51]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:51]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:52]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:52]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:52]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:52]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:52]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:52]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:53]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:53]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:53]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:53]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:53]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:53]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:54]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:54]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:54]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:54]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:54]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:54]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:55]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:55]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:55]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:55]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:55]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:55]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:56]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:56]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:56]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:56]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:56]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:56]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:57]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:57]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:57]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:57]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:57]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:57]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:58]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:58]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:58]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:58]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:58]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:58]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:59]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:59]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:59]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:43:59]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:43:59]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:43:59]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:00]                   │ debg Fetching analytics state for job bm_1_1606908494529_clone
[00:44:00]                   │ debg Fetching data frame analytics job stats for job bm_1_1606908494529_clone...
[00:44:00]                   │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:00]                   │ info Taking screenshot "/dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/screenshots/failure/machine learning data frame analytics jobs cloning supported by UI form classification job supported by the form runs the clone analytics job and displays it correctly in the job list.png"
[00:44:01]                   │ info Current URL is: http://localhost:61131/app/ml/data_frame_analytics/new_job?index=1ade3b90-3497-11eb-9fd5-b9884d59120c&jobId=bm_1_1606908494529
[00:44:01]                   │ info Saving page source to: /dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/failure_debug/html/machine learning data frame analytics jobs cloning supported by UI form classification job supported by the form runs the clone analytics job and displays it correctly in the job list.html
[00:44:01]                   └- ✖ fail: machine learning data frame analytics jobs cloning supported by UI form classification job supported by the form runs the clone analytics job and displays it correctly in the job list
[00:44:01]                   │      timed out waiting for analytics state to be stopped -- last error: Error: expected analytics state to be stopped but got analyzing
[00:44:01]                   │     at /dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/services/ml/api.ts:299:19
[00:44:01]                   │     at runAttempt (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:38:15)
[00:44:01]                   │     at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:77:21)
[00:44:01]                   │     at retryForTruthy (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_truthy.ts:38:3)
[00:44:01]                   │     at Retry.waitForWithTimeout (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:57:7)
[00:44:01]                   │     at Object.waitForAnalyticsState (/dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/services/ml/api.ts:291:7)
[00:44:01]                   │     at Object.waitForAnalyticsCompletion (/dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/services/ml/data_frame_analytics.ts:62:7)
[00:44:01]                   │     at Context.<anonymous> (/dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/apps/ml/data_frame_analytics/cloning.ts:213:11)
[00:44:01]                   │     at Object.apply (/dev/shm/workspace/parallel/13/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
[00:44:01]                   │   Error: timed out waiting for analytics state to be stopped -- last error: Error: expected analytics state to be stopped but got analyzing
[00:44:01]                   │       at /dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/services/ml/api.ts:299:19
[00:44:01]                   │       at runAttempt (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:38:15)
[00:44:01]                   │       at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:77:21)
[00:44:01]                   │       at retryForTruthy (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_truthy.ts:38:3)
[00:44:01]                   │       at Retry.waitForWithTimeout (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:57:7)
[00:44:01]                   │       at Object.waitForAnalyticsState (test/functional/services/ml/api.ts:291:7)
[00:44:01]                   │       at Object.waitForAnalyticsCompletion (test/functional/services/ml/data_frame_analytics.ts:62:7)
[00:44:01]                   │       at Context.<anonymous> (test/functional/apps/ml/data_frame_analytics/cloning.ts:213:11)
[00:44:01]                   │       at Object.apply (/dev/shm/workspace/parallel/13/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
[00:44:01]                   │       at onFailure (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_truthy.ts:50:13)
[00:44:01]                   │       at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:68:13)
[00:44:01]                   │       at retryForTruthy (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_truthy.ts:38:3)
[00:44:01]                   │       at Retry.waitForWithTimeout (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:57:7)
[00:44:01]                   │       at Object.waitForAnalyticsState (test/functional/services/ml/api.ts:291:7)
[00:44:01]                   │       at Object.waitForAnalyticsCompletion (test/functional/services/ml/data_frame_analytics.ts:62:7)
[00:44:01]                   │       at Context.<anonymous> (test/functional/apps/ml/data_frame_analytics/cloning.ts:213:11)
[00:44:01]                   │       at Object.apply (/dev/shm/workspace/parallel/13/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
[00:44:01]                   │ 
[00:44:01]                   │

Stack Trace

Error: timed out waiting for analytics state to be stopped -- last error: Error: expected analytics state to be stopped but got analyzing
    at /dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/services/ml/api.ts:299:19
    at runAttempt (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:38:15)
    at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:77:21)
    at retryForTruthy (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_truthy.ts:38:3)
    at Retry.waitForWithTimeout (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:57:7)
    at Object.waitForAnalyticsState (test/functional/services/ml/api.ts:291:7)
    at Object.waitForAnalyticsCompletion (test/functional/services/ml/data_frame_analytics.ts:62:7)
    at Context.<anonymous> (test/functional/apps/ml/data_frame_analytics/cloning.ts:213:11)
    at Object.apply (/dev/shm/workspace/parallel/13/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
    at onFailure (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_truthy.ts:50:13)
    at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:68:13)
    at retryForTruthy (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_truthy.ts:38:3)
    at Retry.waitForWithTimeout (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:57:7)
    at Object.waitForAnalyticsState (test/functional/services/ml/api.ts:291:7)
    at Object.waitForAnalyticsCompletion (test/functional/services/ml/data_frame_analytics.ts:62:7)
    at Context.<anonymous> (test/functional/apps/ml/data_frame_analytics/cloning.ts:213:11)
    at Object.apply (/dev/shm/workspace/parallel/13/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)

Metrics [docs]

✅ unchanged

History

💚 Build #90950 succeeded a5b198c
💚 Build #90736 succeeded 964f4ce

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

thomheymann

Verified that all instances of callAsCurrentUser have been migrated and new implementation follow migration guidelines.

LGTM!

azasypkin · 2020-12-02T15:43:30Z

7.x/7.11.0: 6665f27

azasypkin added chore Team:Security Team focused on: Auth, Users, Roles, Spaces, Audit Logging, and more! v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:NP Migration v7.11.0 labels Nov 30, 2020

azasypkin mentioned this pull request Nov 30, 2020

[Meta] Switch to the new Elasticsearch client #83910

Closed

74 tasks

azasypkin commented Nov 30, 2020

View reviewed changes

x-pack/plugins/security/server/plugin.ts Outdated

@@ -185,6 +185,7 @@ export class Plugin {

taskManager,

});

core.getStartServices().then(([{ elasticsearch }]) => elasticsearch.client);

This comment was marked as resolved.

Sign in to view

azasypkin commented Nov 30, 2020

View reviewed changes

azasypkin marked this pull request as ready for review November 30, 2020 17:27

azasypkin requested a review from a team as a code owner November 30, 2020 17:27

Migrate security routes to a new Elasticsearch client.

a5b198c

azasypkin force-pushed the issue-xxx-new-es-client branch from 964f4ce to a5b198c Compare December 1, 2020 08:04

azasypkin requested a review from thomheymann December 1, 2020 10:02

Merge branch 'master' into issue-xxx-new-es-client

db31c6a

thomheymann approved these changes Dec 2, 2020

View reviewed changes

azasypkin merged commit c1e7f69 into elastic:master Dec 2, 2020

azasypkin deleted the issue-xxx-new-es-client branch December 2, 2020 14:00

azasypkin mentioned this pull request Dec 2, 2020

[7.x] Migrate security routes to a new Elasticsearch client. (#84528) #84767

Merged

azasypkin added a commit to azasypkin/kibana that referenced this pull request Dec 2, 2020

Migrate security routes to a new Elasticsearch client. (elastic#84528)

0c7f0a5

azasypkin added the backported label Dec 2, 2020

jportner mentioned this pull request Jan 15, 2021

Kibana user can't change own password in cloud deployments #88484

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate security routes to a new Elasticsearch client. #84528

Migrate security routes to a new Elasticsearch client. #84528

azasypkin commented Nov 30, 2020 •

edited

Loading

This comment was marked as resolved.

azasypkin Nov 30, 2020 •

edited

Loading

elasticmachine commented Nov 30, 2020

legrego commented Dec 1, 2020

kibanamachine commented Dec 2, 2020

Standard Out

Stack Trace

thomheymann left a comment

azasypkin commented Dec 2, 2020

Migrate security routes to a new Elasticsearch client. #84528

Migrate security routes to a new Elasticsearch client. #84528

Conversation

azasypkin commented Nov 30, 2020 • edited Loading

This comment was marked as resolved.

azasypkin Nov 30, 2020 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Nov 30, 2020

legrego commented Dec 1, 2020

kibanamachine commented Dec 2, 2020

💛 Build succeeded, but was flaky

Test Failures

Standard Out

Stack Trace

Metrics [docs]

History

thomheymann left a comment

Choose a reason for hiding this comment

azasypkin commented Dec 2, 2020

azasypkin commented Nov 30, 2020 •

edited

Loading

azasypkin Nov 30, 2020 •

edited

Loading