out_stackdriver: fix project_id length for testing #2611
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Yen-Cheng Chou <ycchou@google.com>
StevenYCChou
requested review from
edsiper,
fujimotos and
koleini
as code owners
4 tasks
|
Thanks! |
xmcqueen
pushed a commit
to xmcqueen/fluent-bit
that referenced
this pull request
Oct 6, 2020
Signed-off-by: Yen-Cheng Chou <ycchou@google.com>
Kingyo
added a commit
to Kingyo/fluent-bit
that referenced
this pull request
Nov 25, 2020
* ra: fix typo of comment Signed-off-by: Takahiro YAMASHITA <nokute78@gmail.com> * build: add an option for OSS-Fuzz builds (fluent#2502) This will make things a lot easier from the OSS-Fuzz side and also make it easier to construct new fuzzers. * aws: utils: fix mem leak in flb_imds_request (fluent#2532) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * io: fix EINPROGRESS check, also check that errno is not zero (fluent#2497) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * upstream: new 'destroy_queue' to defer connections context destroy (fluent#2497) A connection context might be destroyed while the event loop still has some pending event to be processed, in some cases a network exception. Destroying the context might lead to a corruption. The following patch implements a new queue to store temporary the connection context so the 'destroy' process is defered until all events from the event loop has been processed. Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * engine: do upstream cleanup after the event loop finished processing events (fluent#2497) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_stackdriver: parse tag that matches kubernetes default regex Signed-off-by: Jeff Luo <jeffluoo@google.com> * test: runtime: out_stackdriver: add unit test for tag regex matching Signed-off-by: Jeff Luo <jeffluoo@google.com> * upstream: expose flb_upstream_conn_pending_destroy() Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * tests: runtime: filter_modify: fix sleep() usage and others Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_stackdriver: fix header guard Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_stackdriver: add missing ctype.h header on http_request Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_stackdriver: add cast for regex_match() variable Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_azure_blob: add support for blockblob type and new blob_type property Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * upstream: fix deletion of destroy_queue Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * io: remove unnecessary errno message Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * upstream: remove unnecessary errno message Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_syslog: move configuration to a config_map Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * pack: print timestamp formatting timespec fields Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * filter_lua: fix handling of integer/double values (fluent#1932 fluent#1647) This patch makes to recognize the preferred numeric data type returned by the Lua script in the record fields. Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * filter_lua: new 'time_as_table' property for timestamp handling (fluent#641 fluent#2519 fluent#2015) Long time ago we got some reports that using timestamps as double/floats might lost precision when the values are converted back from Lua. Actually there is no exact way to have 100% precision in doubles in our use case 'C > Lua > C'. Community suggested that we might workaround this with another solution. This patch considering backward compatibility, implements a new configuration property called 'time_as_table', which passes the timestamp as a Lua table with the following keys ['sec'] => timestamp seconds ['nsec'] => timestamp nanoseconds for users looking for 100% timestamp precision and specifically when dealing with nanoseconds, this option ensures timestamp integrity. If the option is enabled the user just need to adjust the script to use the new format if they touch or use the timestamp value. By default the option 'time_as_table' is disabled and in the future we might consider to enable it by default. Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_file: use flb_time to format timestamp Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * pack: remove unused variable Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * tests: internal: mp: include 'flb_compat.h' <unistd.h> is not POSIX standard and not provided by Microsoft Visual C++. Include "flb_compat.h" to avoid the build failure. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * utils: add `flb_utils_hex2int()` function mk_utils_hex2int() is included in mk_server. For this reason, this function only exists when FLB_HTTP_SERVER is enabled. Bring the function to flb_utils.c, so that we can use the functionality regardless of the compile-time option. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * Port Azure BLOB Output plugin to Windows With this patch out_azure_blob can be compiled and linked on Windows Server. This patch also fixes CI builds on AppVeyor. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * pack: Use PRIu64/PRId64 to format 64 bit integers. Do not use `%lu` to format 64-bit int; A long can be 32-bit depending on the system, so this line can implicitly truncate the target value. This should fix the "timestamps are truncated inproperly" bug on out_datadog. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * tests: internal: fuzzer: added three new fuzzer for msgpack handling (fluent#2541) Signed-off-by: davkor <david@adalogics.com> * tests: runtime: out_cloudwatch_logs: Disable net.keepalive in test (fluent#2533) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * in_forward: fix header declaration (copy paste) (fluent#2559) Use FLB_IN_FW_CONFIG_H instead the MQTT one * ci: travis: remove deprecated key (fluent#2454) See: https://blog.travis-ci.com/2018-11-19-required-linux-infrastructure-migration Signed-off-by: Zero King <l2dy@icloud.com> * out_es: support nanosecond timestamp precision (fluent#2544) Starting in Elasticsearch 7, a "date_nanos" data type was added, increasing timestamp precision from milliseconds to nanoseconds. This patch adds a "Time_Key_Nanos" option which tells the ElasticSearch output plugin to send 9 decimal places instead of 3 to ElasticSearch. Tests are included, and a patch to document the new option will be submitted shortly. Signed-off-by: Neal Turett <nturett@evoforge.org> * doc: Add missing plugins to README (fluent#2534) Signed-off-by: zihengCat <zihenglv@gmail.com> * lib: sqlite: upgrade from 3.31.0 to 3.33.0 (fluent#2552) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * random: Introduce a random generator (fluent#2555) This adds flb_randombytes(), which fills the given buffer with random numbers using each OS's built-in Crypt API. The most common use scenario is: unsigned char buf[64] if (get_randombytes(buf, 64)) { flb_error("cannot get random bytes"); } This function supports both UNIX and Windows. You can use this function without caring the underlying OS. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * lib: mbedtls: upgrade from 2.16.5 to 2.24.0 Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * lib: mbedtls: comment out link_to_source call Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * lib: mbedtls: comment out Python3 check and 3dparty dir Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * pack: gelf: validate incoming object is a map (oss-fuzz 25754) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * tests: internal: fuzzer: remove test message Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * lib: luajit: upgrade from 2.1.0-beta3 to 2.1.0-dd5032e Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * build: adjust luajit path Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * pack: gelf: format timestamp as seconds.milliseconds Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * tests: internal: gelf: adjust test case for expected records and timestamp Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * in_cpu: normalize per-process CPU stats by number of cores (fluent#2543) Signed-off-by: yang-padawan <25978390+yang-padawan@users.noreply.github.com> * pack: json_sds: validate unpacking Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * utils: fix bad handling of invalid utf-8 bytes (oss-fuzz 25785) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * strptime: Add a fallback macro for `timezone` (fluent#2493) According to the UNIX standard: The external variable timezone shall be set to the difference, in seconds, between Coordinated Universal Time (UTC) and local standard time FreeBSD is incompatible with this standard. In particular, since it exposes a function symbol `char* timezone(int, int)`, expressions like `-(timezone)` causes a compile error. Fix it by adding a compat macro for FreeBSD. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * out_forward: always initialize salt with random numbers (fluent#2575) There is an initialization bug that leaves the shared key salt being uninitialized when SSL is not enabled. This might allow attackers to guess the shared key by looking at the hash. Let's always initialize the salt buffer securely using flb_randombytes(). Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * out_gelf: port the plugin to Windows (fluent#2574) With this patch, the GELF Output plugin can be compiled and linked on Windows. This commit also fixes a few bugs in GELF plugin: * The message header was not constructed properly. In particular, 11-12th bytes were filled in the reversed order. * The message id generation was bogus (e.g. it did "tv_nsec*1000000 + tm.tm.tv_nsec" to generate a timestamp) Fix these glitches and add detailed documentation to each function. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * out_pgsql: add support for CockroachDB (fluent#2512) The SQL query will change when new option `cockroachdb` it's set to true Signed-off-by: Jonathan Gonzalez V <jonathan.gonzalez@2ndquadrant.com> Co-authored-by: Jonathan Gonzalez V <jonathan.gonzalez@2ndquadrant.com> * lib: chunkio: upgrade to v1.0.6 (dev changes) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * fstore: new interface to manage local files storage Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * random: rename function flb_randombytes() to flb_random_bytes() Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * tests: internal: random: fix function name Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_forward: remove unused variable and adjust random api name Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * build: on FLB_SMALL mode, do not use 'strip' flag for non-gcc compiler Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * strptime: initialize 'len' variable Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * pack: gelf: initialize 'val_len' variable Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * str: use memcpy to silent gcc warning Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * io: tls: remove unused conditional and initialize variable Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * filter_lua: initialize variable Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * filter_parser: initialize variable Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * filter_kubernetes: validate metadata is a map Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * in_systemd: initialize variable Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_azure_blob: add cast Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_gelf: initialize variable Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_syslog: initialize variables Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * tests: internal: signv4: initialize variable Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * in stdin: add buffer_size parameter (fluent#2364) * in sdtin: add buffer_size parameter Signed-off-by: Martin Dojcak <martin.dojcak@lablabs.io> * lib: mbedtls: force compiler c99 mode Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * tests: internal: disable gelf test in Win32 Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * aws: s3 key format option Signed-off-by: Meghna Prabhu <meghnapr@amazon.com> * signv4: Support S3 Signed Payload Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_es: update with new signv4 api Signed-off-by: Wesley Pettit <wppttt@amazon.com> * aws: add new S3 local buffering library Signed-off-by: Wesley Pettit <wppttt@amazon.com> * aws: s3 object key formatting and request signing Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_s3: new output plugin for Amazon S3 Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_s3: use new fstore API plus other adjustments - use new fstore API for buffer management - register plugin flags - config: change configmap for TIME and SIZE value types - config: rename chunk_buffer_dir -> store_dir - fix leaks on exceptions - use field to 'lock' chunks, instead of temporary node deletion - others adjustments note: still work in process. Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * aws: fix broken s3 key test Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_kinesis_firehose: new high performance core plugin for Kinesis Firehose (fluent#2572) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * in_tail: implement large-file support for Windows Previously it was not possible to handle files larger than 2GB on Windows. The fix is archived by migrating each vairable and function to be explicitly 64-bit ready. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * in_tail: do a small cleanup of the Windows port We have added a number of helpers to make it easier to support Windows (e.g. `flb_pipefd_t` as a portable type for pipes). This introduces these helpers to in_tail. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * fstore: return on memory exception (CID 309436) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * fstore: release context on exception (CID 309435) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * fstore: fix error message parameter (CID 309433) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * signv4: fix sds_printf parameters (CID 306765) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_azure_blob: fix use-after-free on exception (CID 306763 306762) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_azure_blob: do not use http context before validation (CID 306663) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_stackdriver: release pattern on exception (CID 305802) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * in_tail: fix debug message params (CID 305255) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_cloudwatch_logs: add support for custom sts endpoint Signed-off-by: Meghna Prabhu <meghnapr@amazon.com> * out_es: add support for custom STS endpoint Signed-off-by: Meghna Prabhu <meghnapr@amazon.com> * aws: add support for custom STS endpoint in flb_aws_sts_provider Signed-off-by: Meghna Prabhu <meghnapr@amazon.com> * aws: add support for custom sts endpoint in flb_eks_provider Signed-off-by: Meghna Prabhu <meghnapr@amazon.com> * aws: sts: set custom_endpoint flag correctly Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_s3: Add sts_endpoint and role_arn options Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_kinesis_firehose: Add sts_endpoint option Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_cloudwatch_logs: truncate events that are larger than 256KiB Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_cloudwatch_logs: fix off by one error in payload creation Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_cloudwatch_logs: Use documented max payload size and other fixes Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_kinesis_firehose: fix type of event_bytes (CID 309457) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_kinesis_firehose: Use correct printf format for time_t (CID 309456) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * fstore: fix null dereference (CID 309455) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * aws: util: fix double free issues in get_s3_key (CID 309453 & 309443) (fluent#2599) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_s3: fix possible null dereference (CID 309448) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_s3: always cast fstore meta_buf to char * (CID 309442) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_s3: fix mem leak (CID 309438) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * lib: jansson: add jansson library for upcoming Avro support (fluent#2568) Signed-off-by: xmcqueen <bmcqueen@linkedin.com> * doc: add users: 'Microsoft', 'Linked In' and 'Trend Micro' Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_stackdriver: use the correct project ID The value for project ID of monitored resources of a LogEntry should be project ID, not the project number, which is in numeric format. The value of the project ID of Cloud Logging's Monitored Resources are documented in https://cloud.google.com/logging/docs/api/v2/resource-list. The project ID to be retrived from instance metadata is documented in https://cloud.google.com/compute/docs/storing-retrieving-metadata#default. Signed-off-by: Yen-Cheng Chou <ycchou@google.com> * pack: fix json floating point format regression (fluent#2592) The recent change to the JSON floating point formatting to use "%.16g" caused a regression where values that have no fractional part are formatted as integers. For example, "10.0" gets formatted as "10". This patch uses the same approach as https://github.com/ohler55/oj/blob/v3.10.13/ext/oj/dump_strict.c#L100-L101, which is used in Fluentd. It checks if the double value is equal to the integer part, and if so, will use "%.1f" as the format to ensure the decimal part is still rendered (with a single decimal place of ".0"). This prevents downstream datastores from having data type conflicts. This was tested by building locally and running through different value using the dummy input plugin and stdout output plugin with json_lines formatting. Will include example outputs of tests in Pull Request. Signed-off-by: Joey Paskhay <joey.paskhay@gmail.com> * aws_util: fix failing sanitizer builds (fluent#2604) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_s3: use correct printf format for MAX_UPLOAD_ERRORS (CID 309440) (fluent#2602) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * out_stackdriver: fix project_id length for testing (fluent#2611) Signed-off-by: Yen-Cheng Chou <ycchou@google.com> * out_http: on error, print HTTP response payload if available (fluent#2593) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * config: new flag to determinate if the ingestion is active Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * engine: upon exit, turn ingestion flag off Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * input: if ingestion is disable, do not resume it Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * input: chunk: if ingestion is disable, do not resume it Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * lib: rename worker thread to 'flb-pipeline' Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * log: rename worker thread to 'flb-logger' Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * bin: spawn pipeline using library mode (fluent#1496 fluent#2610) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * task: fix counter of running tasks, use 'users' counter (fluent#2411) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * input_chunk: drop oldest buffer chunks when reaching the limit Signed-off-by: Jeff Luo <jeffluoo@google.com> * output: add new buffer option: storage.total_limit_size Signed-off-by: Jeff Luo <jeffluoo@google.com> * router: add function to get routes_mask by tag Signed-off-by: Jeff Luo <jeffluoo@google.com> * scheduler: clean up request in request_wait queue Signed-off-by: Jeff Luo <jeffluoo@google.com> * in_tail: add checking to avoid deleting fd with value -1 Signed-off-by: Jeff Luo <jeffluoo@google.com> * pipe: add checking to avoid closing fd with value -1 Signed-off-by: Jeff Luo <jeffluoo@google.com> * task: creat task_routes by routes_mask Signed-off-by: Jeff Luo <jeffluoo@google.com> * test: internal: input_chunk: add unit tests for buffering mechanism Signed-off-by: Jeff Luo <jeffluoo@google.com> * in_tail: inotify: use proper logger API Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * lib: new flb_loop() to wait for main thread exit Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * bin: use new flb_loop() for clear exit Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * out_gelf: port the random seed generation to Windows (fluent#2614) Windows does not have /dev/urandom. For this reason, it was always using a less secure value (= UNIX time) as an entropy source. Use flb_randombytes() to use a good entropy source, and thus, reduce the possibility of message collision. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * in_random: use the new API to generate a random seed (fluent#2613) Simplify the random seed generation by using flb_random_bytes() instead of trying to read entropy sources manually. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net> * aws: util: initialize key buffer to zero Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> * pack: on JSON formatting, remove duplicated keys (fluent#1835
Signed-off-by: Yen-Cheng Chou ycchou@google.com
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
Documentation
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.