Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Add Base64 encoded HMAC & UUID template functions to httpjson input #27873

Merged
merged 7 commits into from
Sep 20, 2021

Conversation

legoguy1000
Copy link
Contributor

@legoguy1000 legoguy1000 commented Sep 10, 2021

What does this PR do?

Add Base64 encoded HMAC & UUID template functions to httpjson input

Why is it important?

In order to create an Akamai Agent integration.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 10, 2021
@legoguy1000 legoguy1000 marked this pull request as ready for review September 10, 2021 18:41
@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 10, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-09-20T09:21:34.833+0000

  • Duration: 89 min 41 sec

  • Commit: c08213a

Test stats 🧪

Test Results
Failed 0
Passed 8206
Skipped 1201
Total 9407

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 8206
Skipped 1201
Total 9407

@legoguy1000 legoguy1000 changed the title Init commit [Filebeat] Add Akamai EdgeGrid auth to httpjson input Sep 11, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 13, 2021
@mergify
Copy link
Contributor

mergify bot commented Sep 14, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b akamai-edgegrid upstream/akamai-edgegrid
git merge upstream/master
git push upstream akamai-edgegrid

@P1llus
Copy link
Member

P1llus commented Sep 14, 2021

Hi @legoguy1000 , thanks for your awesome PR's as always! I will discuss this one, as we might not want to use any additional libraries for this, and rather add in the functionality it requires to httpjson itself, it might be that our recent changes we made would work with this actually

@legoguy1000
Copy link
Contributor Author

legoguy1000 commented Sep 14, 2021

I think the alternative to adding the EdgeGrid Library is to add another HMAC function that returns Base64 encoded signatures instead of Hex. See f746b13

Then do below to implement EdgeGrid and possibly remove the EdgeGrid Module

- type: httpjson
  enabled: true
  config_version: 2
  interval: 1m
  request.url: XXXXXX
  request.transforms:
    - set:
        target: url.params.from
        value: '[[.cursor.timestamp]]'
        default: '[[ (now (parseDuration "-24h")).Unix ]]'
    - set:
        target: header.XTimestamp
        value: '[[ formatDate (now) "20060102T15:04:05-0700" ]]'
    - set:
        target: header.XSignatureBase
        value: '[[ sprintf "EG1-HMAC-SHA256 client_token=%s;access_token=%s;timestamp=%s;nonce=%d;" "a" "b" (.header.Get "XTimestamp") now.UnixNano  ]]'
    - set:
        target: header.XSignatureKey
        value: '[[ hmacBase64 "sha256" "c" (.header.Get "XTimestamp") ]]'
    - set:
        target: header.XSignature
        value: '[[ hmacBase64 "sha256" (.header.Get "XSignatureKey") "GET\t" .url.Host "\t" .url.Path "?" .url.RawQuery "\t\t\t" (.header.Get "XSignatureBase") ]]'
    - set:
        target: header.Authorization
        value: 'EG1-HMAC-SHA256 [[ sprintf "%ssignature=%s" (.header.Get "XSignatureBase") (.header.Get "XSignature") ]]'

It appears to work and generates the below headers

xtimestamp	20210914T18:33:08+0000
xsignaturekey	L/1kX9gfY7GIFly7G3z3vkj3YUzFWpCHqqDR7HimwpM=
xsignaturebase	EG1-HMAC-SHA256 client_token=a;access_token=b;timestamp=20210914T18:33:08+0000;nonce=1631644388219193225;
xsignature	i4Ln1LOulRMhwJ8v8vAa5s+XfqcQ2fCkirJYHuGzLSQ=
authorization	EG1-HMAC-SHA256 EG1-HMAC-SHA256 client_token=a;access_token=b;timestamp=20210914T18:33:08+0000;nonce=1631644388219193225;signature=i4Ln1LOulRMhwJ8v8vAa5s+XfqcQ2fCkirJYHuGzLSQ=

@legoguy1000
Copy link
Contributor Author

I think I'm going to also add a UUID template function for the nonce field as there is no way currently to generate randomness.

@legoguy1000 legoguy1000 changed the title [Filebeat] Add Akamai EdgeGrid auth to httpjson input [Filebeat] Add Base64 encoded HMAC & UUID template functions to httpjson input Sep 17, 2021
@legoguy1000 legoguy1000 force-pushed the akamai-edgegrid branch 2 times, most recently from 3b36486 to 8d1f1f2 Compare September 17, 2021 11:47
@legoguy1000
Copy link
Contributor Author

@marc-gr @P1llus I think we're good to go now.

Copy link
Contributor

@marc-gr marc-gr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 thanks for taking the time to change it with our feedback 😄 !

@marc-gr
Copy link
Contributor

marc-gr commented Sep 20, 2021

/test

1 similar comment
@marc-gr
Copy link
Contributor

marc-gr commented Sep 20, 2021

/test

@marc-gr marc-gr merged commit 942d537 into elastic:master Sep 20, 2021
@marc-gr marc-gr added backport-v7.16.0 Automated backport with mergify enhancement labels Sep 20, 2021
mergify bot pushed a commit that referenced this pull request Sep 20, 2021
…son input (#27873)

* Init commit

* Update docs and tests

* Add Base64 HMAC

* Remove EdgeGrid Library

* Add UUID and Base64 HMAC template functions

* don't upgrade google/uuid library

* Format value_tpl.go

Co-authored-by: Marc Guasch <marc.guasch@elastic.co>
(cherry picked from commit 942d537)
@legoguy1000 legoguy1000 deleted the akamai-edgegrid branch September 20, 2021 11:05
marc-gr pushed a commit that referenced this pull request Sep 20, 2021
…son input (#27873) (#28001)

* Init commit

* Update docs and tests

* Add Base64 HMAC

* Remove EdgeGrid Library

* Add UUID and Base64 HMAC template functions

* don't upgrade google/uuid library

* Format value_tpl.go

Co-authored-by: Marc Guasch <marc.guasch@elastic.co>
(cherry picked from commit 942d537)

Co-authored-by: Alex Resnick <adr8292@gmail.com>
v1v added a commit to v1v/beats that referenced this pull request Sep 20, 2021
* upstream/master: (658 commits)
  Add complete k8s metadata through composable provider (elastic#27691)
  Revert "Fix issue where --insecure didn't propogate to Fleet Server ES connection (elastic#27969)" (elastic#27997)
  Remove deprecated kafka fields (elastic#27938)
  [Filebeat] Add Base64 encoded HMAC & UUID template functions to httpjson input (elastic#27873)
  Improve httpjson template function join (elastic#27996)
  Remove kubernetes.container.image alias (elastic#27898)
  [Elastic Agent] Golden files for program tests (elastic#27862)
  [Elastic Agent] Disable modules.d in metricbeat (elastic#27860)
  libbeat/common/seccomp: provide default policy for linux arm64 (elastic#27955)
  Fix logger statement in aws-s3 input (elastic#27982)
  Fix wrong merge (elastic#27976)
  Fix issue where --insecure didn't propogate to Fleet Server ES connection (elastic#27969)
  Forward-port 7.14.2 changelog to master (elastic#27975)
  [Filebeat] Removing duplicate modules (aliases) Observability (elastic#27919)
  Fix path in vagrant windows script (elastic#27966)
  [Filebeat] Removing duplicate modules (aliases) and Cyberark (elastic#27915)
  No changelog for 8.0.0-alpha2 (elastic#27961)
  Add write access to 'url.value' from 'request.transforms'. (elastic#27937)
  Docker: remove deprecated fields (elastic#27933)
  Filebeat: Make all filesets disabled in default configuration (elastic#27762)
  ...
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
…son input (elastic#27873)

* Init commit

* Update docs and tests

* Add Base64 HMAC

* Remove EdgeGrid Library

* Add UUID and Base64 HMAC template functions

* don't upgrade google/uuid library

* Format value_tpl.go

Co-authored-by: Marc Guasch <marc.guasch@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.16.0 Automated backport with mergify enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants