fluent-bit Crash at startup

[yonchin]

Error is as follows：

`
[2018/05/09 02:06:41] [ info] [engine] started (pid=1)
[2018/05/09 02:06:41] [ warn] [filter_kube] merge_json_log is deprecated, enabling 'merge_log' option instead
[2018/05/09 02:06:41] [ info] [filter_kube] https=1 host=kubernetes.default.svc.cluster.local port=443
[2018/05/09 02:06:41] [ info] [filter_kube] local POD info OK
[2018/05/09 02:06:41] [ info] [filter_kube] testing connectivity with API server...
[2018/05/09 02:06:41] [ info] [filter_kube] API server connectivity OK
[engine] caught signal (SIGSEGV)
Fluent-Bit v0.13.0
Copyright (C) Treasure Data

#0 0x5584b6257911 in msgpack_pack_object() at lib/msgpack-2.1.3/src/objectc.c:85
#1 0x5584b6046dae in pack_map_content() at plugins/filter_kubernetes/kubernetes.c:321
#2 0x5584b60475fc in cb_kube_filter() at plugins/filter_kubernetes/kubernetes.c:491
#3 0x5584b5fe79aa in flb_filter_do() at src/flb_filter.c:86
#4 0x5584b5fe53fc in flb_input_dbuf_write_end() at include/fluent-bit/flb_input.h:642
#5 0x5584b5fe6f5c in flb_input_dyntag_append_raw() at src/flb_input.c:894
#6 0x5584b600b27d in process_content() at plugins/in_tail/tail_file.c:290
#7 0x5584b600c0c8 in flb_tail_file_chunk() at plugins/in_tail/tail_file.c:651
#8 0x5584b6009ab7 in in_tail_collect_static() at plugins/in_tail/tail.c:129
#9 0x5584b5fe749b in flb_input_collector_fd() at src/flb_input.c:995
#10 0x5584b5fee1de in flb_engine_handle_event() at src/flb_engine.c:296
#11 0x5584b5fee1de in flb_engine_start() at src/flb_engine.c:515
#12 0x5584b5f8e4c6 in main() at src/fluent-bit.c:824
#13 0x7f420d38f2e0 in ???() at ???:0
#14 0x5584b5f8c949 in ???() at ???:0
#15 0xffffffffffffffff in ???() at ???:0
`

[edsiper]

which specific Fluent Bit Docker image are you running ?

[yonchin]

@edsiper fluent/fluent-bit:0.13 or 0.13.0

[StevenACoffman]

@yonchin Can you provide your configuration? Are you installing this from the fluent/fluent-bit-kubernetes-logging repository?

[yonchin]

@StevenACoffman Yes, I am based on the fluent/fluent-bit-kubernetes-logging repository.

Here is my configuration：

`
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020

@INCLUDE input-kubernetes.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE output-elasticsearch.conf

input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/.log
Exclude_Path /var/log/containers/kube.log,/var/log/containers/fluent*.log,/var/log/containers/kibana*.log,/var/log/containers/ingress*.log,/var/log/containers/calico*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10

filter-kubernetes.conf: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Annotations Off
K8S-Logging.Parser On

output-elasticsearch.conf: |
[OUTPUT]
Name es
Match *
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
Logstash_Format On
Retry_Limit False

parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
# Command | Decoder | Field | Optional Action
# =============|==================|=================
Decode_Field_As escaped log
`

[bat9r]

Same situation, fluent-bit restarts randomly.
Kubernetes 1.8.11 ( 3 masters 2cpu 7ram, 9 agentpool 8 cpu 56 ram (Azure, ACS))

[2018/05/15 18:22:28] [ info] [engine] started (pid=1)
[2018/05/15 18:22:28] [ warn] [filter_kube] merge_json_log is deprecated, enabling 'merge_log' option instead
[2018/05/15 18:22:28] [ info] [filter_kube] https=1 host=kubernetes.default.svc.cluster.local port=443
[2018/05/15 18:22:28] [ info] [filter_kube] local POD info OK
[2018/05/15 18:22:28] [ info] [filter_kube] testing connectivity with API server...
[2018/05/15 18:22:28] [ info] [filter_kube] API server connectivity OK
[engine] caught signal (SIGSEGV)
Fluent-Bit v0.13.0
Copyright (C) Treasure Data
 #0  0x565228286911      in  msgpack_pack_object() at lib/msgpack-2.1.3/src/objectc.c:85
#1  0x565228075dae      in  pack_map_content() at plugins/filter_kubernetes/kubernetes.c:321
#2  0x5652280765fc      in  cb_kube_filter() at plugins/filter_kubernetes/kubernetes.c:491
#3  0x5652280169aa      in  flb_filter_do() at src/flb_filter.c:86
#4  0x5652280143fc      in  flb_input_dbuf_write_end() at include/fluent-bit/flb_input.h:642
#5  0x565228015f5c      in  flb_input_dyntag_append_raw() at src/flb_input.c:894
#6  0x56522803a27d      in  process_content() at plugins/in_tail/tail_file.c:290
#7  0x56522803b0c8      in  flb_tail_file_chunk() at plugins/in_tail/tail_file.c:651
#8  0x565228038ab7      in  in_tail_collect_static() at plugins/in_tail/tail.c:129
#9  0x56522801649b      in  flb_input_collector_fd() at src/flb_input.c:995
#10 0x56522801d1de      in  flb_engine_handle_event() at src/flb_engine.c:296
#11 0x56522801d1de      in  flb_engine_start() at src/flb_engine.c:515
#12 0x565227fbd4c6      in  main() at src/fluent-bit.c:824
#13 0x7feee22ad2e0      in  ???() at ???:0
#14 0x565227fbb949      in  ???() at ???:0
#15 0xffffffffffffffff  in  ???() at ???:0

[azweb76]

Im seeing the same problem with a very similar setup. On 0.13.1.

[azweb76]

setting Merge_JSON_Log Off seemed to fix this issue

[fruuf]

I enabled debug to see which log file caused the error and viewed the logs manual. they all contained lines that started with valid json joined with another message, like {"ok":true}2018-05-28 08:55:07 connect().... I prevented those messages, in my case adding > /dev/null to my curl command.

[edsiper]

@fruuf would you please attach a full log file that can help to replicate the problem ?

[edsiper]

Fixed

[rolandjitsu]

I think I stumbled upon a similar issue. FB kept restarting and I could not track down the reason. But after I removed all the logs which it was tailing, it seems to be ok.

[edsiper]

@rolandjitsu would you please open a new issue on github and attach your config and info about versions: OS & Fluent Bit ?

[rolandjitsu]

@edsiper just did, see #1362.