Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Logstash to Opensearch - "Failed to install template", "Got response code '400'" #179

Closed
Direwave opened this issue Nov 22, 2022 · 2 comments
Labels
bug Something isn't working

Comments

@Direwave
Copy link

[Description]
Hello there ! I'm trying to forward some OpenEDR logs (json) with logstash and can't get rid of an error : "Failed to install template". I'm using a very basic configuration with logstash 7.10 and opensearch 2.3.0. I've also tried logstash 6.7 and checked the rights of the role i'm using, but i can't figure it out.

[logstash.conf]

input {
file {
path => "C:\ProgramData\edrsvc\log\output_events*"
}
}

filter {

}
output {
elasticsearch {
hosts => ["https://172.16.3.71:9200"]
user => "admin"
password => "admin"
ssl => "true"
ssl_certificate_verification => "false"
index => "openedr-%{+YYYY.MM.dd}"
}
}

[ERROR]

Using JAVA_HOME defined java: C:\logstash-7.10.0\jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/C:/Users/user/AppData/Local/Temp/jruby-5568/jruby296237807069400059jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to C:/logstash-7.10.0/logs which is now configured via log4j2.properties
[2022-11-22T11:31:55,221][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [mswin32-x86_64]"}
[2022-11-22T11:31:55,455][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-11-22T11:31:57,190][INFO ][org.reflections.Reflections] Reflections took 32 ms to scan 1 urls, producing 23 keys and 47 values
[2022-11-22T11:31:57,893][WARN ][logstash.outputs.elasticsearch][main] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
[2022-11-22T11:31:58,190][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://admin:xxxxxx@172.16.3.71:9200/]}}
[2022-11-22T11:31:58,518][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://admin:xxxxxx@172.16.3.71:9200/"}
[2022-11-22T11:31:58,565][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>2}
[2022-11-22T11:31:58,596][WARN ][logstash.outputs.elasticsearch][main] DEPRECATION WARNING: Connecting to an OSS distribution of Elasticsearch using the default distribution of Logstash will stop working in Logstash 8.0.0. Please upgrade to the default distribution of Elasticsearch, or use the OSS distribution of Logstash {:url=>"https://admin:xxxxxx@172.16.3.71:9200/"}
[2022-11-22T11:31:58,611][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://172.16.3.71:9200"]}
[2022-11-22T11:31:58,658][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>2, :ecs_compatibility=>:disabled}
[2022-11-22T11:31:58,705][INFO ][logstash.outputs.elasticsearch][main] Index Lifecycle Management is set to 'auto', but will be disabled - Your Elasticsearch cluster is before 7.0.0, which is the minimum version required to automatically run Index Lifecycle Management
[2022-11-22T11:31:58,705][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["C:/logstash-7.10.0/config/logstash-sample.conf"], :thread=>"#<Thread:0x5f1c5e1 run>"}
[2022-11-22T11:31:58,721][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"template"=>"logstash-", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true, "ignore_above"=>256}}}}}, {"float_fields"=>{"match"=>"", "match_mapping_type"=>"float", "mapping"=>{"type"=>"float", "doc_values"=>true}}}, {"double_fields"=>{"match"=>"", "match_mapping_type"=>"double", "mapping"=>{"type"=>"double", "doc_values"=>true}}}, {"byte_fields"=>{"match"=>"", "match_mapping_type"=>"byte", "mapping"=>{"type"=>"byte", "doc_values"=>true}}}, {"short_fields"=>{"match"=>"", "match_mapping_type"=>"short", "mapping"=>{"type"=>"short", "doc_values"=>true}}}, {"integer_fields"=>{"match"=>"", "match_mapping_type"=>"integer", "mapping"=>{"type"=>"integer", "doc_values"=>true}}}, {"long_fields"=>{"match"=>"", "match_mapping_type"=>"long", "mapping"=>{"type"=>"long", "doc_values"=>true}}}, {"date_fields"=>{"match"=>"", "match_mapping_type"=>"date", "mapping"=>{"type"=>"date", "doc_values"=>true}}}, {"geo_point_fields"=>{"match"=>"", "match_mapping_type"=>"geo_point", "mapping"=>{"type"=>"geo_point", "doc_values"=>true}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "doc_values"=>true}, "@Version"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true}, "geoip"=>{"type"=>"object", "dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip", "doc_values"=>true}, "location"=>{"type"=>"geo_point", "doc_values"=>true}, "latitude"=>{"type"=>"float", "doc_values"=>true}, "longitude"=>{"type"=>"float", "doc_values"=>true}}}}}}}}
[2022-11-22T11:31:58,768][INFO ][logstash.outputs.elasticsearch][main] Installing elasticsearch template to _template/logstash
[2022-11-22T11:31:58,830][ERROR][logstash.outputs.elasticsearch][main] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'https://172.16.3.71:9200/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in perform_request'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in perform_request_to_url'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:319:in block in perform_request'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:414:in with_connection'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in perform_request'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326:in block in Pool'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:352:in template_put'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in template_install'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:31:in install'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in install_template'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/common.rb:218:in install_template'", "C:/logstash-7.10.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/common.rb:49:in block in setup_after_successful_connection'"]}
[2022-11-22T11:31:59,440][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.72}
[2022-11-22T11:32:00,033][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/logstash-7.10.0/data/plugins/inputs/file/.sincedb_3d2466290cb76da98e60057f5ee5f764", :path=>["C:\ProgramData\edrsvc\log\output_events\*"]}
[2022-11-22T11:32:00,049][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-11-22T11:32:00,111][INFO ][filewatch.observingtail ][main][87c71efbba5db0908882e3f95a9f202ad9f973d24a78f7bf850e368a09d48467] START, creating Discoverer, Watch with file and sincedb collections
[2022-11-22T11:32:00,127][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-11-22T11:32:00,440][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[HOST]

  • Opensearch on Debian 10
  • Logstash on Windows 10
@Direwave Direwave added bug Something isn't working untriaged labels Nov 22, 2022
@dlvenable
Copy link
Member

This might be similar to #176.

@dblock
Copy link
Member

dblock commented Feb 13, 2024

Closing as dup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants