DogAPI reports vague error: Cannot determine local hostname via hostname -f

[tschellhorn]

It happens that on a node that is short on memory, the chef run was failing to converge and reporting an error:

Cannot determine local hostname via hostname -f

The problem happens at https://github.com/DataDog/dogapi-rb/blob/master/lib/dogapi/common.rb#L163 where it doesn't report the exception that is thrown. The actual issues was due to the following error:

Cannot allocate memory - hostname

[yannmh]

Hi @tschellhorn,

Sorry to hear that you are experiencing issues with dogapi-rb and thanks for the detailed feedback.

I think we can do a better job, here, at handling hostname resolution exceptions: we could simply log a warning message with the exception message, and fallback with no hostname.
On a side note, a short term workaround could be to manually set up a hostname during the initialize call, to avoid any hostname resolution c.f. https://github.com/DataDog/dogapi-rb/blob/master/lib/dogapi/facade.rb#L26.

Please let me know what you think about it. We'll consider the appropriate changes for the upcoming release.

Yann

[tschellhorn]

I came across the error on one of our nodes running Chef with the Datadog cookbook. I tracked the problem down to the Dogapi. I looked at the code for how the dd-handler is included in in the Chef cookbook recipe and there is no hostname that is passed over so I don't think I would be able to set the hostname explicitly in the DogAPI object. (https://github.com/DataDog/chef-handler-datadog/blob/master/lib/chef/handler/datadog.rb#L22)

Fixing this for the next milestone is okay with me. I just wanted to point out the minor issue to help other users in the future.

[degemer]

DataDog/chef-datadog#308 should fix this issue if you set a hostname in Chef. (released in chef-datadog 2.5.0)
As @yannmh said, it would be nice to have a better hostname resolution, we'll see what we can do.

[degemer]

We could raise a more explicit error DatadogHostnameError for instance. We'll see if we can change it for 2.0, since it doesn't seem to be a blocker.

[cabrinha]

I'm seeing this behavior a lot in my Puppetserver logs:

2019-08-07T03:58:50.518Z ERROR [qtp1381296136-16228] [puppetserver] Puppet Report processor failed: Cannot determine local hostname via hostname -f
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/dogapi-1.36.0/lib/dogapi/common.rb:170:in `find_localhost'
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/dogapi-1.36.0/lib/dogapi/facade.rb:72:in `initialize'
/etc/puppetlabs/code/environments/master/modules/datadog_agent/lib/puppet/reports/datadog_reports.rb:114:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:53:in `block in processors'
org/jruby/RubyArray.java:1801:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:285:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:187:in `do_save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:266:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1801:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:47:in `handleRequest'

I can't tell why this is failing. I don't have the memory issue like the original comment in this issue. Also, running hostname -f on the server returns fine. Is there another way to pass hostname into the dogapi gem? https://github.com/DataDog/dogapi-rb/blob/master/lib/dogapi/common.rb#L168

I loaded up an irb shell and ran the command, and got a seemingly fine result.

root@infra-puppetserver6-07ee34fd19e6807a4:~# irb
irb(main):001:0> %x[hostname -f]
=> "infra-puppetserver6-07ee34fd19e6807a4\n"
irb(main):002:0> %x[hostname -f].strip
=> "infra-puppetserver6-07ee34fd19e6807a4"
irb(main):003:0>

I'm running Puppet 6 with the latest version of the DataDog module, 2.7.0.

@degemer or @yannmh could you take a look please?

[gzussa]

@cabrinha What we are discussing here may help in your context. #179.

@degemer @yannmh did the chef fix fixed the issue? I am not sure to understand from this thread if we can pass along the hostname now or not. What this issue open to better handle exceptions?

[cabrinha]

@cabrinha What we are discussing here may help in your context. #179.

@degemer @yannmh did the chef fix fixed the issue? I am not sure to understand from this thread if we can pass along the hostname now or not. What this issue open to better handle exceptions?

Thanks for the reply, but the hostname binary is available in my case.

It does return the short name, however. Could that be the issue I'm having? Is DogAPI RB expecting the FQDN, including the TLD?

If so, we should allow users to set that themselves.

[github-actions]

Thanks for your contribution!

This issue has been automatically marked as stale because it has not had activity in the last 30 days. Note that the issue will not be automatically closed, but this notification will remind us to investigate why there's been inactivity. Thank you for participating in the Datadog open source community.

If you would like this issue to remain open:

1. Verify that you can still reproduce the issue in the latest version of this project.

2. Comment that the issue is still reproducible and include updated details requested in the issue template.

[gzussa]

Reopening this so @yannmh and @degemer can provide some guidance. cc @dabcoder as well