Beatless initial PR #8485

ph · 2018-09-28T15:18:08Z

This PR provide the following

Plugin infra for developing providers and functions
A local stdin provider only used for testing, I will remove it in the
final version.
AWS provider and function types for:

Cloudwatch logs
SQS
Kinesis
Api web gateway proxy

License checker
Packaging of artifact
Runners
CLI infrastructure
CLI to push a cloudwatch logs function.
CLI to delete any function
Processors support.
Types to validate value from the users and the lambda function.

What it doesn't provides

ECS and full event extraction. (NOT for v1)
Specifying the AWS credentials in the configuration
CLI for SQS, Kinesis, API
Robust CLI interaction with the API, rollback on failure / versioning.
Removal of not supported outputs
Removal of seccomp check
Integration tests
Updated build task to produce containing the user executable beat and
the linux beats.
Concurrency / memory settings

Vendored

aws/aws-lambda-go
aws/aws-sdk-go-v2

Note: The PR look much bigger because of the vendored files.

Notes 2:

Make sure you add this to your config YAMl, I wan't to make the following the default options but it would have been really messy to do it in this PR.

# Disable anything that could be send to disk
path.data: /tmp
path.logs: /tmp/logs
logging.to_stderr: true
logging.to_files: false
#logging.level: debug
setup.template.enabled: true
queue.mem:
  events: 50
  flush.min_events: 1
  flush.timeout: 0.5s

related work

houndci-bot · 2018-09-28T15:18:25Z

libbeat/feature/registry.go

 // feature are grouped by namespace, a namespace is a kind of plugin like outputs, inputs, or queue.
 // The feature name must be unique.
-type registry struct {
+type FeatureRegistry struct {


type name will be used as feature.FeatureRegistry by other packages, and that stutters; consider calling this Registry

+1 for naming the type 'Registry'. Let's replace the global 'Registry' variable with func GlobalRegistry() *Registry.

Yeah much better, it will lead to less fragile code. :)

houndci-bot · 2018-09-28T15:18:25Z

libbeat/feature/registry.go

@@ -28,26 +28,26 @@ import (

 type mapper map[string]map[string]Featurable

-// Registry implements a global registry for any kind of feature in beats.
+// Registry implements a global FeatureRegistry for any kind of feature in beats.


comment on exported type FeatureRegistry should be of the form "FeatureRegistry ..." (with optional leading article)

The comment indeed reads somewhat funny.

ph · 2018-10-02T17:19:21Z

@kvch @ruflin go ahead for first round of review, I am still working on the CLI so expect them to changes but I see no reason to not start reviewing this.

ph · 2018-10-02T17:35:53Z

I think I will migrate the to the cloud formation api instead this look like a more solid way of doing it.

ph · 2018-10-03T12:47:59Z

@urso @ruflin @kvch Do not takes too much times into reviewing the CLI code in the cloudwatchlogs.go I've migrated everything to the cloudformation API which provided a lot more stable experience.

kvch · 2018-10-03T14:56:38Z

x-pack/beatless/provider/local/local.go

+	"github.com/elastic/beats/x-pack/beatless/provider"
+)
+
+var stdinName = "stdin"


kvch · 2018-10-03T15:01:32Z

x-pack/beatless/provider/local/local.go

+		scanner.Split(bufio.ScanLines)
+
+		for scanner.Scan() {
+			if scanner.Err() != nil {


It's enough to call scanner.Err once, if you rewrite the code a little:

if err := scanner.Err(); err != nil { errChan <- err return }

kvch · 2018-10-03T15:21:05Z

x-pack/beatless/provider/aws/op_create_lambda.go

+	"github.com/elastic/beats/libbeat/logp"
+)
+
+var handlerName = "beatless"


Why is this variable here? Although it's in the same package, but it's used in x-pack/beatless/provider/aws/cloudwatch_logs.go.
This also could be const.

kvch · 2018-10-03T15:27:58Z

x-pack/beatless/provider/aws/executor.go

+	"github.com/elastic/beats/libbeat/logp"
+)
+
+var errNeverRun = errors.New("executer was never executed")


Please group the variables together.

var ( errNeverRun = errors.New("executer was never executed") errCannotAdd = errors.New("cannot add to an already executed executer") errAlreadyExecuted = errors.New("executer already executed") )

kvch · 2018-10-03T15:33:32Z

x-pack/beatless/provider/aws/executor.go

+
+func newExecutor(log *logp.Logger, context *executorContext) *executor {
+	if log == nil {
+		log = logp.NewLogger("executer")


This probably should be "executor".

kvch · 2018-10-03T15:37:53Z

x-pack/beatless/provider/aws/executor.go

+		}
+	}
+	if err == nil {
+		e.log.Debug("all operations successful")


"all operations have been successful"?

houndci-bot · 2018-10-04T15:01:19Z

metricbeat/module/php_fpm/url.go

+// specific language governing permissions and limitations
+// under the License.
+
+package php_fpm


don't use an underscore in package name

houndci-bot · 2018-10-04T15:01:19Z

metricbeat/module/logstash/node_stats/data_test.go

+
+// +build !integration
+
+package node_stats


don't use an underscore in package name

houndci-bot · 2018-10-04T15:01:19Z

libbeat/common/file/interval_rotator.go

+	)
+}
+
+// OrderIntervalLogs, when given a log filename in the form [prefix]-[formattedDate]-n


comment on exported function OrderIntervalLogs should be of the form "OrderIntervalLogs ..."

houndci-bot · 2018-10-04T15:01:19Z

dev-tools/cmd/dashboards/export_dashboards.go

@@ -48,11 +49,14 @@ func makeURL(url, path string, params url.Values) string {
 	return strings.Join([]string{url, path, "?", params.Encode()}, "")
 }

-func Export(client *http.Client, conn string, dashboard string, out string) error {
+func Export(client *http.Client, conn string, spaceID string, dashboard string, out string) error {


exported function Export should have comment or be unexported

ph · 2018-10-04T17:27:12Z

depends on #8573

ph · 2018-10-04T17:32:39Z

@ruflin As discussed I have added the following command beatless function package this will generate a package that you can push manually or via another CLI to lambda. By default it will create a package.zip in the working folder, but you can override that with -o

Also, this PR is a lot more robust when deploying, updating or removing a cloudwatch function, it require permission on cloudformation, but it will make the operation atomic, and will take into consideration the changes that you add.

I have updated the configuration to correctly expose lambda specific settings:

beatless.provider.aws.functions:
  # Define the list of function availables, each function required to have a unique name.
  - name: fn_cloudwatch_logs
    type: cloudwatch_logs

    # Description of the method to help identify them when you run multiples functions.
    description: "listen to cloudwatch logs"
    #
    # Concurrency, is the reserved number of instance for that function.
    # Default is unreserved.
    #
    # Note: There is a hard limit of 1000 of running function of any kind per account.
    # concurrency: 5
    #
    # The maximum memory allocated for this function, the configured size must be a factor of 64.
    # There is a hard limit of 3008MiB for each function. Default is 128MiB.
    # memory_size: 128MiB
    #
    # Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
    # dead_letter_config.target_arn:
    #
    # Optional fields that you can specify to add additional information to the
    # output. Fields can be scalar values, arrays, dictionaries, or any nested
    # combination of these.
    #fields:
    #  env: staging

    # List of cloudwatch log group registered to that function.
    triggers:
      - log_group_name: /aws/lambda/beatless-cloudwatch_logs
        filter_pattern: mylog_
    #
    # Define custom processors for this function.
    # processors:
    #   - dissect:
    #       tokenizer: "%{key1} %{key2}"

Once the PR #8573 is merged it will be a PR the same size that you had before.

guidelines.

ph · 2018-10-19T12:49:23Z

I've rebased this PR on top of the feature branch you can now create artifact of beatless and push the function using theses artifacts.

## This PR provides the following 1. Plugin infra for developing providers and functions 2. A local stdin provider only used for testing, I will remove it in the final version. 3. AWS provider and function types for: - Cloudwatch logs - SQS - Kinesis - Api web gateway proxy 4. License checker 5. Packaging of artifact 6. Runners 7. CLI infrastructure 8. CLI to push a cloudwatch logs function. 9. CLI to delete any function 10. Processors support. 11. Types to validate value from the users and the lambda function. ## What it doesn't provides - ECS and full event extraction. (NOT for v1) - Specifying the AWS credentials in the configuration - CLI for SQS, Kinesis, API - Robust CLI interaction with the API, rollback on failure / versioning. - Removal of not supported outputs - Removal of seccomp check - Integration tests - Updated build task to produce containing the user executable beat and the linux beats. - Concurrency / memory settings

## This PR provides the following 1. Plugin infra for developing providers and functions 2. A local stdin provider only used for testing, I will remove it in the final version. 3. AWS provider and function types for: - Cloudwatch logs - SQS - Kinesis - Api web gateway proxy 4. License checker 5. Packaging of artifact 6. Runners 7. CLI infrastructure 8. CLI to push a cloudwatch logs/sqs function. 9. CLI to delete any function 10. Processors support. 11. Types to validate value from the users and the lambda function. ## What it doesn't provides - ECS and full event extraction. (NOT for v1) - Specifying the AWS credentials in the configuration - CLI for SQS, Kinesis, API - Robust CLI interaction with the API, rollback on failure / versioning. - Removal of not supported outputs - Removal of seccomp check - Integration tests - Updated build task to produce containing the user executable beat and the linux beats. - Concurrency / memory settings

1. Plugin infra for developing providers and functions 2. A local stdin provider only used for testing, I will remove it in the final version. 3. AWS provider and function types for: - Cloudwatch logs - SQS - Kinesis - Api web gateway proxy 4. License checker 5. Packaging of artifact 6. Runners 7. CLI infrastructure 8. CLI to push a cloudwatch logs/sqs function. 9. CLI to delete any function 10. Processors support. 11. Types to validate value from the users and the lambda function. - ECS and full event extraction. (NOT for v1) - Specifying the AWS credentials in the configuration - CLI for SQS, Kinesis, API - Robust CLI interaction with the API, rollback on failure / versioning. - Removal of not supported outputs - Removal of seccomp check - Integration tests - Updated build task to produce containing the user executable beat and the linux beats. - Concurrency / memory settings (cherry picked from commit 3ef0b4e)

Cherry-pick of PR #8485 to 6.x branch. Original message: ## This PR provide the following 1. Plugin infra for developing providers and functions 2. A local stdin provider only used for testing, I will remove it in the final version. 3. AWS provider and function types for: - Cloudwatch logs - SQS - Kinesis - Api web gateway proxy 4. License checker 5. Packaging of artifact 6. Runners 7. CLI infrastructure 8. CLI to push a cloudwatch logs function. 9. CLI to delete any function 10. Processors support. 11. Types to validate value from the users and the lambda function. ## What it doesn't provides - ECS and full event extraction. (NOT for v1) - Specifying the AWS credentials in the configuration - CLI for SQS, Kinesis, API - Robust CLI interaction with the API, rollback on failure / versioning. - Removal of not supported outputs - Removal of seccomp check - Integration tests - Updated build task to produce containing the user executable beat and the linux beats. - Concurrency / memory settings ## Vendored - aws/aws-lambda-go - aws/aws-sdk-go-v2 --- **Note:** The PR look much bigger because of the vendored files. **Notes 2:** Make sure you add this to your config YAMl, I wan't to make the following the default options but it would have been really messy to do it in this PR. ``` # Disable anything that could be send to disk path.data: /tmp path.logs: /tmp/logs logging.to_stderr: true logging.to_files: false #logging.level: debug setup.template.enabled: true queue.mem: events: 50 flush.min_events: 1 flush.timeout: 0.5s ``` ## related work - #8449 - #8443

## This PR provides the following 1. Plugin infra for developing providers and functions 2. A local stdin provider only used for testing, I will remove it in the final version. 3. AWS provider and function types for: - Cloudwatch logs - SQS - Kinesis - Api web gateway proxy 4. License checker 5. Packaging of artifact 6. Runners 7. CLI infrastructure 8. CLI to push a cloudwatch logs function. 9. CLI to delete any function 10. Processors support. 11. Types to validate value from the users and the lambda function. ## What it doesn't provides - ECS and full event extraction. (NOT for v1) - Specifying the AWS credentials in the configuration - CLI for SQS, Kinesis, API - Robust CLI interaction with the API, rollback on failure / versioning. - Removal of not supported outputs - Removal of seccomp check - Integration tests - Updated build task to produce containing the user executable beat and the linux beats. - Concurrency / memory settings

ph added in progress Pull request is currently in progress. Functionbeat labels Sep 28, 2018

ph changed the title ~~CLI improvements~~ [WIP] CLI improvements Sep 28, 2018

houndci-bot reviewed Sep 28, 2018

View reviewed changes

ph added the blocked label Sep 28, 2018

ph force-pushed the fix/beatless-cli-improved branch from 136c136 to 56d7d16 Compare October 2, 2018 17:16

ph added review and removed blocked in progress Pull request is currently in progress. labels Oct 2, 2018

ph changed the title ~~[WIP] CLI improvements~~ Beatless initial PR Oct 2, 2018

ph assigned ruflin and kvch Oct 2, 2018

ph assigned urso Oct 3, 2018

kvch reviewed Oct 3, 2018

View reviewed changes

ph force-pushed the fix/beatless-cli-improved branch from 9095553 to 908966b Compare October 4, 2018 15:00

houndci-bot reviewed Oct 4, 2018

View reviewed changes

ph force-pushed the fix/beatless-cli-improved branch from 908966b to b5d6a42 Compare October 4, 2018 15:09

ph force-pushed the feature-beatless branch from ff2a5f2 to 97800c9 Compare October 4, 2018 15:11

ph force-pushed the fix/beatless-cli-improved branch from b5d6a42 to a57b46d Compare October 4, 2018 17:25

ph force-pushed the fix/beatless-cli-improved branch from 8a92d96 to edd2dfa Compare October 5, 2018 13:39

ph added 14 commits October 19, 2018 08:40

MINOR: Added doc

6d799dc

MINOR: typo

d9a32f4

MINOR: cleanups

d7c0415

Fix fmt

3f932ae

hound comment

d80ad14

CHANGE: make update, deploy and remove top level command.

fd4ab64

use new uuid library

423a85c

uuid again

39ed4cf

better formatting for the reference file.

38c77b7

use a function instead of a method on a struct

bf69342

Fix a few issue with logging

417d211

return directly

ea2c451

Ensure capitalization on all logger message are following the

72cb211

guidelines.

rewording

0f84be5

ph force-pushed the fix/beatless-cli-improved branch from 2b8742c to 0f84be5 Compare October 19, 2018 12:42

Better logging messages

406b0fa

urso approved these changes Oct 19, 2018

View reviewed changes

ph merged commit 26e6adb into elastic:feature-beatless Oct 19, 2018

ph mentioned this pull request Oct 24, 2018

Cherry-pick #8485 to 6.x: Beatless initial PR #8736

Merged

ph added the v6.5.0 label Oct 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Beatless initial PR #8485

Beatless initial PR #8485

ph commented Sep 28, 2018 •

edited

Loading

houndci-bot Sep 28, 2018

urso Oct 15, 2018

ph Oct 16, 2018

houndci-bot Sep 28, 2018

urso Oct 18, 2018

ph commented Oct 2, 2018

ph commented Oct 2, 2018

ph commented Oct 3, 2018

kvch Oct 3, 2018

kvch Oct 3, 2018

kvch Oct 3, 2018

kvch Oct 3, 2018 •

edited

Loading

kvch Oct 3, 2018

kvch Oct 3, 2018

houndci-bot Oct 4, 2018

houndci-bot Oct 4, 2018

houndci-bot Oct 4, 2018

houndci-bot Oct 4, 2018

ph commented Oct 4, 2018

ph commented Oct 4, 2018

ph commented Oct 19, 2018

Beatless initial PR #8485

Beatless initial PR #8485

Conversation

ph commented Sep 28, 2018 • edited Loading

This PR provide the following

What it doesn't provides

Vendored

related work

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Oct 2, 2018

ph commented Oct 2, 2018

ph commented Oct 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch Oct 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Oct 4, 2018

ph commented Oct 4, 2018

ph commented Oct 19, 2018

ph commented Sep 28, 2018 •

edited

Loading

kvch Oct 3, 2018 •

edited

Loading