[BUG]: Opensearch dashboard creation through terraform

[Souvikpanda6]

What is the bug?

creation of opensearch dashboards through terraform with proper visualizations and index-patterns to load. Basically my desire is to automate creation of the opensearch dashboards using terraform with a payload specified and automate the deployment of same to other clusters.

For that i will need to create opensearch dashboards for a cluster using a specified payload which will have all visualizations and also create the visualizations required in the dashboard.
I was able to create dashboard using this -
resource "opensearch_dashboard_object" "example_dashboard" {
body = jsonencode([
{
"_id": "dashboard:${var.dashboards_folder_name}",
"_type": "_doc",
"_source": {
"type": "dashboard",
"dashboard": {
"title": "Test Dashboard",
"hits": 0,
"description": "",
"panelsJSON": "[{"version":"1.3.2","gridData":{"x":0,"y":0,"w":9,"h":32,"i":"af43a9f2-f1df-49dd-8cd4-8b7da376f352"},"panelIndex":"af43a9f2-f1df-49dd-8cd4-8b7da376f352","embeddableConfig":{"title":"[impact] SMS Service","hidePanelTitles":false},"title":"[impact] SMS Service","panelRefName":"panel_0"},{"version":"1.3.2","gridData":{"x":9,"y":0,"w":7,"h":32,"i":"452f2f5e-c34a-4768-8934-32f58fdd211f"},"panelIndex":"452f2f5e-c34a-4768-8934-32f58fdd211f","embeddableConfig":{"title":"[impact] MDI Connector","hidePanelTitles":false},"title":"[impact] MDI Connector","panelRefName":"panel_1"},{"version":"1.3.2","gridData":{"x":16,"y":0,"w":8,"h":32,"i":"b2d0e573-eabb-4a31-896b-8be53523f69b"},"panelIndex":"b2d0e573-eabb-4a31-896b-8be53523f69b","embeddableConfig":{"title":"[impact] Streaming API","hidePanelTitles":false},"title":"[impact] Streaming API","panelRefName":"panel_2"},{"version":"1.3.2","gridData":{"x":24,"y":0,"w":7,"h":32,"i":"fb66628f-2d57-4b40-b663-f0a5e21d67f7"},"panelIndex":"fb66628f-2d57-4b40-b663-f0a5e21d67f7","embeddableConfig":{},"panelRefName":"panel_3"},{"version":"1.3.2","gridData":{"x":31,"y":0,"w":8,"h":32,"i":"7b3d2ab0-8d86-48ba-9f70-f7e9445ffdd6"},"panelIndex":"7b3d2ab0-8d86-48ba-9f70-f7e9445ffdd6","embeddableConfig":{},"panelRefName":"panel_4"},{"version":"1.3.2","gridData":{"x":39,"y":0,"w":8,"h":32,"i":"8b268192-c9bd-469c-9fc8-70b7d44cb941"},"panelIndex":"8b268192-c9bd-469c-9fc8-70b7d44cb941","embeddableConfig":{},"panelRefName":"panel_5"},{"version":"1.3.2","gridData":{"x":0,"y":32,"w":9,"h":22,"i":"36d2b578-7f12-4de5-9fb0-7f786f9d4ba1"},"panelIndex":"36d2b578-7f12-4de5-9fb0-7f786f9d4ba1","embeddableConfig":{},"panelRefName":"panel_6"}]",
"optionsJSON": "{"hidePanelTitles":false,"useMargins":true}",
"version": 1,
"timeRestore": true,
"timeTo": "now",
"timeFrom": "now-4w",
"refreshInterval": {
"pause": true,
"value": 0
},
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{"query":{"language":"kuery","query":""},"filter":[]}"
}
},
"references": [
{
"name": "panel_0",
"type": "visualization",
"id": "4a1feb90-f5ac-11ee-9eaf-7b883d41eea5"
},
{
"name": "panel_1",
"type": "visualization",
"id": "7bfcc970-f643-11ee-9e29-8b89be297bc7"
},
{
"name": "panel_2",
"type": "visualization",
"id": "82569500-f646-11ee-a047-af7b2ba9c6be"
},
{
"name": "panel_3",
"type": "visualization",
"id": "fbc8b610-f647-11ee-9eaf-7b883d41eea5"
},
{
"name": "panel_4",
"type": "visualization",
"id": "7a612130-f64b-11ee-9e29-8b89be297bc7"
},
{
"name": "panel_5",
"type": "visualization",
"id": "b5a57150-f64c-11ee-a047-af7b2ba9c6be"
},
{
"name": "panel_6",
"type": "visualization",
"id": "df704550-f72d-11ee-a047-af7b2ba9c6be"
}
],
"migrationVersion": {
"dashboard": "7.9.3"
}
}
}
])
}
But this doesn't load the visualizations so i created each. of the visualizations presnt in the dashboard using something like the-
resource "opensearch_dashboard_object" "source_ip_addresses" {
body = jsonencode([{
"_id": "b5a57150-f64c-11ee-a047-af7b2ba9c6be",
"_source":{
"type": "visualization",
"visualization": {
"title": "[imapct] FSM Connector",
"visState": "{"title":"[imapct] FSM Connector","type":"table","aggs":[{"id":"1","enabled":true,"type":"count","params":{"customLabel":"Outbound events"},"schema":"metric"},{"id":"2","enabled":true,"type":"terms","params":{"field":"context_account_id.keyword","orderBy":"1","order":"desc","size":10000,"otherBucket":false,"otherBucketLabel":"Other","missingBucket":false,"missingBucketLabel":"Missing","customLabel":"Accounts"},"schema":"bucket"}],"params":{"perPage":"","showPartialRows":false,"showMetricsAtAllLevels":false,"sort":{"columnIndex":null,"direction":null},"showTotal":false,"totalFunc":"sum","percentageCol":""}}",
"uiStateJSON": "{"vis":{"params":{"sort":{"columnIndex":null,"direction":null}}}}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{"query":{"query":"","language":"kuery"},"filter":[]}"
},
"savedSearchRefName": "search_0"
},
"references": [
{
"name": "search_0",
"type": "search",
"id": "856cee00-f64c-11ee-9eaf-7b883d41eea5"
}
]
}}])
}
also included the index pattern required in the code.
on running terraform apply these resources are getting created but the visualizations doesn't load to the dashboard,
on checking in the UI the name of the visualizations exists, but doesn't load it doesn't have anything to load it says-
" Could not locate that visualization{id of that particular}"

How can one reproduce the bug?

while trying to create visualizations and dashboard with opensearch

What is the expected behavior?

A clear and concise description of what you expected to happen.

What is your host/environment?

Operating system, version.

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[prudhvigodithi]

[Triage]
Thanks @Souvikpanda6, from my understanding, you are able to create the dashboard but not the visualization's, if so can you please post the exact error thrown on the dashboard ? Also can you check this, in past @PhilippReinke is able to create visualization's using opensearch_dashboard_object #184 (comment)
Thanks
@getsaurabh02

[prudhvigodithi]

Also @Souvikpanda6 here are some more example for opensearch_dashboard_object
https://github.com/rblcoder/terraform-opensearch-samples/blob/main/tenant/main.tf#L48-L81
https://github.com/rblcoder/terraform-opensearch-samples/blob/b626744b81073f57a9c508bcbd1e850742842ab8/tenant/search/main.tf#L2
https://github.com/rblcoder/terraform-opensearch-samples/blob/b626744b81073f57a9c508bcbd1e850742842ab8/tenant/index_pattern/main.tf#L25
Thanks to our maintainer @rblcoder for the examples.

[rblcoder]

Thank you, @prudhvigodithi , another discussion which might help
#56 (comment)

[Souvikpanda6]

[Triage] Thanks @Souvikpanda6, from my understanding, you are able to create the dashboard but not the visualization's, if so can you please post the exact error thrown on the dashboard ? Also can you check this, in past @PhilippReinke is able to create visualization's using opensearch_dashboard_object #184 (comment) Thanks @getsaurabh02

I shows it creates the dashboard visualizations but when i open the UI go to visualizations it shows the visualization, But on opening it it says - Could not locate that visualization{id of that particular} also in the dashboard UI it shows the same for all the components. can you please help me in this ihave attached the exact visualization code i used. Or am i missing something like depends on ?? But my concern is it should load to visualizations when checked in the visualise Tab

[Souvikpanda6]

I tried this I gets created on the terminal -
resource "opensearch_dashboard_object" "source_ip_addresses" {
depends_on = [
opensearch_dashboard_object.sb1_logs_index_pattern
]
body = jsonencode([{
"_id": "b5a57150-f64c-11ee-a047-af7b2ba9c6be",
"_source":{
"type": "visualization",
"visualization": {
"title": "[imapct] FSM Connector",
"visState": "{"title":"[imapct] FSM Connector","type":"table","aggs":[{"id":"1","enabled":true,"type":"count","params":{"customLabel":"Outbound events"},"schema":"metric"},{"id":"2","enabled":true,"type":"terms","params":{"field":"context_account_id.keyword","orderBy":"1","order":"desc","size":10000,"otherBucket":false,"otherBucketLabel":"Other","missingBucket":false,"missingBucketLabel":"Missing","customLabel":"Accounts"},"schema":"bucket"}],"params":{"perPage":"","showPartialRows":false,"showMetricsAtAllLevels":false,"sort":{"columnIndex":null,"direction":null},"showTotal":false,"totalFunc":"sum","percentageCol":""}}",
"uiStateJSON": "{"vis":{"params":{"sort":{"columnIndex":null,"direction":null}}}}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{"query":{"query":"","language":"kuery"},"filter":[]}"
},
"savedSearchRefName": "search_0"
},
"references": [
{
"name": "search_0",
"type": "search",
"id": "856cee00-f64c-11ee-9eaf-7b883d41eea5"
}
]
}}])
}
But same thing on the UI i says -- Could not locate that visualization(b5a57150-f64c-11ee-a047-af7b2ba9c6be)
Please help me if i am missing something

[Souvikpanda6]

Thanks for helping got this fixed. issue was simple but unknown -
changing id to - "_id": "visualization:b5a57150-f64c-11ee-a047-af7b2ba9c6be".

One request to add example for "opensearch_dashboard_object" for creation of dashboards.
This will be helpful -
resource "opensearch_dashboard_object" "example_dashboard" {
body = jsonencode([
"_id": "dashboard:${var.dashboards_folder_name}",
"_type": "_doc",
"_source": {
"type": "dashboard",
"dashboard": {
"title": "Test Dashboard",
"hits": 0,
"description": "",
"panelsJSON": "[]",
"optionsJSON": "{"hidePanelTitles":false,"useMargins":true}",
"version": 1,
"timeRestore": true,
"timeTo": "now",
"timeFrom": "now-4w",
"refreshInterval": {
"pause": true,
"value": 0
},
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{"query":{"language":"kuery","query":""},"filter":[]}"
}
},
"references": [],
"migrationVersion": {
"dashboard": "7.9.3"
}
}
}
])
}

[prudhvigodithi]

Thanks for helping got this fixed. issue was simple but unknown -
changing id to - "_id": "visualization:b5a57150-f64c-11ee-a047-af7b2ba9c6be".

Thanks @Souvikpanda6 I see the issue is-reopened again, is there anything else missing except form your documentation request ?

[Souvikpanda6]

Thanks for helping got this fixed. issue was simple but unknown -
changing id to - "_id": "visualization:b5a57150-f64c-11ee-a047-af7b2ba9c6be".

Thanks @Souvikpanda6 I see the issue is-reopened again, is there anything else missing except form your documentation request ?

No thanks

[Souvikpanda6]

Hi I have one more question. With the above payload the resources are created, but the last updated at timestamp is not visible in the UI how to acheive that ??

[dblock]

[Catch All Triage - 1, 2, 3]

[Souvikpanda6]

any updates on how to acheive that ??

[Souvikpanda6]

Also the above works with the provider as -
provider "opensearch" {
url = "http://localhost:9200"
}
considering i port forwarded to 9200.
while trying to create the same dashboard and visualization with the same payload through terraform gives 400 to an AWS opensearch custom domain. with - https://url-domain/_dashboards
How can i fix that to work ??

[prudhvigodithi]

Hi I have one more question. With the above payload the resources are created, but the last updated at timestamp is not visible in the UI how to acheive that ??

Hey @Souvikpanda6 I would suggest to please open a new issue will all the details, including the latest comment #205 (comment).
Thank you

[Souvikpanda6]

Hey @prudhvigodithi . i have created a new issue for this. -[ https://github.com//issues/210)]. Thanks

[prudhvigodithi]

Thanks @Souvikpanda6 closing this and we can discuss more on the created issue.