Allow to pass config overrides via the Settings Struct #8449
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sometime a custom beat that get executed want to override system defaults instead of relying on code defined by libbeat. This is the case with beatless, the queue has limits and flush values that make sense when the beat is run on the edge but not on on AWS lambda. Note that settings is passed via liberal common.Config no type checking is done. I went that route because many parts of beats doesn't expose the config struct outside of the package. This is similar to explicitely defining them in the yaml.
|
This will need to go in 6.5 for beatless. |
ph
added a commit
to ph/beats
that referenced
this pull request
Oct 1, 2018
* Allow to pass config overrides via the Settings Struct Sometime a custom beat that get executed want to override system defaults instead of relying on code defined by libbeat. This is the case with beatless, the queue has limits and flush values that make sense when the beat is run on the edge but not on on AWS lambda. Note that settings is passed via liberal common.Config no type checking is done. I went that route because many parts of beats doesn't expose the config struct outside of the package. This is similar to explicitely defining them in the yaml. (cherry picked from commit 4a03bb3)
ph
added
v6.5.0
and removed
needs_backport
Merged
ph
added a commit
that referenced
this pull request
Oct 4, 2018
* Allow to pass config overrides via the Settings Struct Sometime a custom beat that get executed want to override system defaults instead of relying on code defined by libbeat. This is the case with beatless, the queue has limits and flush values that make sense when the beat is run on the edge but not on on AWS lambda. Note that settings is passed via liberal common.Config no type checking is done. I went that route because many parts of beats doesn't expose the config struct outside of the package. This is similar to explicitely defining them in the yaml. (cherry picked from commit 4a03bb3)
ph
added a commit
that referenced
this pull request
Oct 24, 2018
Cherry-pick of PR #8485 to 6.x branch. Original message: ## This PR provide the following 1. Plugin infra for developing providers and functions 2. A local stdin provider only used for testing, I will remove it in the final version. 3. AWS provider and function types for: - Cloudwatch logs - SQS - Kinesis - Api web gateway proxy 4. License checker 5. Packaging of artifact 6. Runners 7. CLI infrastructure 8. CLI to push a cloudwatch logs function. 9. CLI to delete any function 10. Processors support. 11. Types to validate value from the users and the lambda function. ## What it doesn't provides - ECS and full event extraction. (NOT for v1) - Specifying the AWS credentials in the configuration - CLI for SQS, Kinesis, API - Robust CLI interaction with the API, rollback on failure / versioning. - Removal of not supported outputs - Removal of seccomp check - Integration tests - Updated build task to produce containing the user executable beat and the linux beats. - Concurrency / memory settings ## Vendored - aws/aws-lambda-go - aws/aws-sdk-go-v2 --- **Note:** The PR look much bigger because of the vendored files. **Notes 2:** Make sure you add this to your config YAMl, I wan't to make the following the default options but it would have been really messy to do it in this PR. ``` # Disable anything that could be send to disk path.data: /tmp path.logs: /tmp/logs logging.to_stderr: true logging.to_files: false #logging.level: debug setup.template.enabled: true queue.mem: events: 50 flush.min_events: 1 flush.timeout: 0.5s ``` ## related work - #8449 - #8443
ph
added a commit
to ph/beats
that referenced
this pull request
Oct 30, 2018
* Allow to pass config overrides via the Settings Struct Sometime a custom beat that get executed want to override system defaults instead of relying on code defined by libbeat. This is the case with beatless, the queue has limits and flush values that make sense when the beat is run on the edge but not on on AWS lambda. Note that settings is passed via liberal common.Config no type checking is done. I went that route because many parts of beats doesn't expose the config struct outside of the package. This is similar to explicitely defining them in the yaml. (cherry picked from commit 4a03bb3)
ph
added a commit
that referenced
this pull request
Oct 30, 2018
…ings Struct (#8815) Cherry-pick of PR #8449 to 6.4 branch. Original message: Sometime a custom beat that get executed want to override system defaults instead of relying on code defined by libbeat. This is the case with beatless, the queue has limits and flush values that make sense when the beat is run on the edge but not on on AWS lambda. Note that settings is passed via liberal common.Config no type checking is done. I went that route because many parts of beats doesn't expose the config struct outside of the package. This is similar to explicitely defining them in the yaml. **Notes:** this part of the code is pretty hard to unit test, we have to do some refactoring to make it happen :(
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
…he Settings Struct (elastic#8815) Cherry-pick of PR elastic#8449 to 6.4 branch. Original message: Sometime a custom beat that get executed want to override system defaults instead of relying on code defined by libbeat. This is the case with beatless, the queue has limits and flush values that make sense when the beat is run on the edge but not on on AWS lambda. Note that settings is passed via liberal common.Config no type checking is done. I went that route because many parts of beats doesn't expose the config struct outside of the package. This is similar to explicitely defining them in the yaml. **Notes:** this part of the code is pretty hard to unit test, we have to do some refactoring to make it happen :(
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sometime a custom beat that get executed want to override system
defaults instead of relying on code defined by libbeat.
This is the case with beatless, the queue has limits and flush values
that make sense when the beat is run on the edge but not on on AWS lambda.
Note that settings is passed via liberal common.Config no type checking is
done. I went that route because many parts of beats doesn't expose the
config struct outside of the package.
This is similar to explicitely defining them in the yaml.
Notes: this part of the code is pretty hard to unit test, we have to do some refactoring to make it happen :(