Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[full-ci] enhancement: add support for natural language kql date ranges #7263

Merged
merged 11 commits into from
Sep 15, 2023

Conversation

fschade
Copy link
Contributor

@fschade fschade commented Sep 11, 2023

Description

follow up pr for the ocis kql integration, as the spec describes kql is able to select date ranges by writing natural language.

The following ranges are described in the spec and therefor supported by the ocis implementation:

  • Mtime:today - range: start of today till end of today
  • Mtime:yesterday - range: start of yesterday till end of yesterday
  • Mtime:"this week" - range: start of this week till end of this week
  • Mtime:"this month" - range: start of this month till end of this month
  • Mtime:"last month" - range: start of last month till end of last month
  • Mtime:"this year" - range: start of this year till end of this year
  • Mtime:"last year" - range: start of last year till end of last year

the ast will extended as follow:

	&ast.Ast{
		Nodes: []ast.Node{
			&ast.DateTimeNode{
				Key:      "Mtime",
				Operator: &ast.OperatorNode{Value: ">="},
				Value:    mustParseTime(t, "2023-09-10"),
			},
			&ast.OperatorNode{Value: kql.BoolAND},
			&ast.DateTimeNode{
				Key:      "Mtime",
				Operator: &ast.OperatorNode{Value: "<="},
				Value:    mustParseTime(t, "2023-09-10 23:59:59.999999999"),
			},
		},
	},
	&ast.Ast{
		Nodes: []ast.Node{
			&ast.DateTimeNode{
				Key:      "Mtime",
				Operator: &ast.OperatorNode{Value: ">="},
				Value:    mustParseTime(t, "2023-09-09"),
			},
			&ast.OperatorNode{Value: kql.BoolAND},
			&ast.DateTimeNode{
				Key:      "Mtime",
				Operator: &ast.OperatorNode{Value: "<="},
				Value:    mustParseTime(t, "2023-09-09 23:59:59.999999999"),
			},
		},
	},

and so on....

following queries are now exactly covered as described in the spec:

* 2.1.2 AND Operator 
* 2.1.6 NOT Operator
* 2.1.8 OR Operator
* 2.1.12 Parentheses
* 2.3.5 Date Tokens
* 3.1.11 Implicit Operator
* 3.1.12 Parentheses
* 3.1.2 AND Operator
* 3.1.6 NOT Operator
* 3.1.8 OR Operator
* 3.2.3 Implicit Operator for Property Restriction
* 3.3.1.1.1 Implicit AND Operator
* 3.3.5 Date Tokens

beside that it also includes:

  • structured table tests

Related Issue

Motivation and Context

implement the spec

How Has This Been Tested?

  • unit tests
  • acceptance tests
  • e2e tests

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Technical debt
  • Tests only (no source changes)

Checklist:

  • Code changes
  • Unit tests added
  • Acceptance tests added
  • Documentation ticket raised:

@fschade fschade marked this pull request as ready for review September 15, 2023 07:28
@sonarcloud
Copy link

sonarcloud bot commented Sep 15, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 7 Code Smells

88.7% 88.7% Coverage
0.0% 0.0% Duplication

@fschade fschade merged commit 0f2b2b9 into master Sep 15, 2023
3 checks passed
@delete-merged-branch delete-merged-branch bot deleted the human-date-time-tokens branch September 15, 2023 09:31
ownclouders pushed a commit that referenced this pull request Sep 15, 2023
…es (#7263)

* enhancement: add more kql spec tests and simplify ast normalization

* enhancement: kql parser error if query starts with AND

* enhancement: add kql docs and support for date and time only dateTimeRestriction queries

* enhancement: add the ability to decide how kql nodes get connected

connecting nodes (with edges) seem straight forward when not using group, the default connection for nodes with the same node is always OR. THis only applies for first level nodes, for grouped nodes it is defined differently. The KQL docs are saying, nodes inside a grouped node, with the same key are connected by a AND edge.

* enhancement: explicit error handling for falsy group nodes and queries with leading binary operator

* enhancement: use optimized grammar for kql parser and toolify pigeon

* enhancement: simplify error handling

* fix: kql implicit 'AND' and 'OR' follows the ms html spec instead of the pdf spec

* enhancement: add support for natural language kql date queries

* enhancement: structure kql parser tests into logical clusters

* fix: time-range error naming
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants