Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create object history API #2074

Merged
merged 12 commits into from
Dec 6, 2023
Merged

Create object history API #2074

merged 12 commits into from
Dec 6, 2023

Conversation

Donnype
Copy link
Contributor

@Donnype Donnype commented Nov 30, 2023

Changes

Adds an API representing XTDB's entity-history. This can be used to infer when objects were first and last seen (CC @Rieven). This would boil down to filtering the transaction history by "doc is not None" (since these are deletes, not PUT's) and filtering on valid_time. Then the valid time of the first transaction is the "first_seen" and the valid time of the last transaction is the "last_seen".

Demo

The API:

image

Example transaction log json where I added then deleted Network|internet, then added it again:

[
  {
    "txTime": "2023-11-23T08:56:35+00:00",
    "txId": 0,
    "validTime": "2023-11-23T08:56:34+00:00",
    "contentHash": "df4fe77b0026ef481131a1e2a7d69416e1f98f57",
    "doc": {
      "object_type": "Network",
      "Network/primary_key": "Network|internet",
      "Network/name": "internet",
      "xt/id": "Network|internet"
    }
  },
  {
    "txTime": "2023-11-30T14:21:57+00:00",
    "txId": 155,
    "validTime": "2023-11-30T14:21:57+00:00",
    "contentHash": "0000000000000000000000000000000000000000",
    "doc": null
  },
  {
    "txTime": "2023-11-30T14:22:07+00:00",
    "txId": 159,
    "validTime": "2023-11-30T14:22:07+00:00",
    "contentHash": "df4fe77b0026ef481131a1e2a7d69416e1f98f57",
    "doc": {
      "object_type": "Network",
      "Network/primary_key": "Network|internet",
      "Network/name": "internet",
      "xt/id": "Network|internet"
    }
  }
]

Code Checklist

  • All the commits in this PR are properly PGP-signed and verified;
  • This PR only contains functionality relevant to the issue; tickets have been created for newly discovered issues.
  • I have written unit tests for the changes or fixes I made.
  • For any non-trivial functionality, I have added integration and/or end-to-end tests.
  • I have performed a self-review of my code and refactored it to the best of my abilities.

Communication

  • I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
  • I have made corresponding changes to the documentation, if necessary.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.


Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Signed-off-by: Donny Peeters <donny@bitestreams.com>
@Donnype Donnype requested a review from a team as a code owner November 30, 2023 15:09
Signed-off-by: Donny Peeters <donny@bitestreams.com>
Signed-off-by: Donny Peeters <donny@bitestreams.com>
Signed-off-by: Donny Peeters <donny@bitestreams.com>
@Donnype
Copy link
Contributor Author

Donnype commented Dec 1, 2023

Done @underdarknl

@Donnype Donnype added octopoes Issues related to octopoes backend labels Dec 4, 2023
@Donnype Donnype self-assigned this Dec 4, 2023
@Donnype Donnype added the report label Dec 4, 2023
Copy link
Contributor

@ammar92 ammar92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good 👍 Just a few remarks, should be good to go soon

octopoes/octopoes/models/transaction.py Outdated Show resolved Hide resolved
octopoes/octopoes/xtdb/client.py Outdated Show resolved Hide resolved
octopoes/octopoes/core/service.py Show resolved Hide resolved
octopoes/octopoes/connector/octopoes.py Show resolved Hide resolved
octopoes/octopoes/api/router.py Show resolved Hide resolved
@Donnype Donnype mentioned this pull request Dec 6, 2023
… model with snake casing.

Signed-off-by: Donny Peeters <donny@bitestreams.com>
Signed-off-by: Donny Peeters <donny@bitestreams.com>
… feature/object-history-api

# Conflicts:
#	octopoes/octopoes/xtdb/client.py
#	octopoes/tests/integration/test_api_connector.py
Signed-off-by: Donny Peeters <donny@bitestreams.com>
@Donnype
Copy link
Contributor Author

Donnype commented Dec 6, 2023

Done @ammar92

@stephanie0x00
Copy link
Contributor

Checklist for QA:

  • I have checked out this branch, and successfully ran a fresh make reset.
  • I confirmed that there are no unintended functional regressions in this branch:
    • I have managed to pass the onboarding flow
    • Objects and Findings are created properly
    • Tasks are created and completed properly
  • I confirmed that the PR's advertised feature or hotfix works as intended.

What works:

There appears to be no impact in the UI. Onboarding flow works, adding an extra URL object works and this is also scanned.

What doesn't work:

N/a.

Bug or feature?:

N/a.

@underdarknl underdarknl merged commit 6201fd5 into main Dec 6, 2023
31 checks passed
@underdarknl underdarknl deleted the feature/object-history-api branch December 6, 2023 15:07
jpbruinsslot added a commit that referenced this pull request Dec 6, 2023
* main: (25 commits)
  Create object history API (#2074)
  Bump actions/github-script from 6 to 7 (#2076)
  Installation manual for Windows (2) (#2096)
  Update howdoesitwork.rst (#2091)
  Add benchmarking script to the scheduler (#2071)
  Add fix-poetry-merge-conflict makefile command (#2088)
  Bump sphinx-rtd-theme from 1.2.2 to 2.0.0 (#2080)
  Lower quality level so the CI check doesn't fail (#2086)
  Update xtdb version in octopoes CI docker compose and docker-compose.release-example.yml (#2085)
  Name test nodes by testname instead of uuid (#2087)
  Upgrade to Pydantic v2 (#1912)
  Docs: add dependency installation commands for RHEL based systems (#2059)
  Fix/2072 (#2082)
  Feature/service to systems reports rocky (#2073)
  Update scheduler python packages (#2062)
  Add uvicorn back as non-dev dependency (#2053)
  Bump `cryptography` (#2070)
  Filter tree objects with depth=1 for Findings  (#1982)
  Bump aiohttp from 3.8.6 to 3.9.0 in /boefjes (#2061)
  Translations update from Hosted Weblate (#2057)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend octopoes Issues related to octopoes report
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants