Skip to content

Commit

Permalink
Add dp3 client subcommand
Browse files Browse the repository at this point in the history 
This adds a cool interpreter to the dp3 client tool. Going forward we
will make this the favored interactive usage experience.

The interpreter supports running queries with the language, importing
data files, and querying for stat ranges.
  • Loading branch information
@wkalt
wkalt committed Apr 10, 2024
1 parent bc23937 commit f688894
Show file tree
Hide file tree
Showing 10 changed files with 487 additions and 158 deletions.
114 changes: 24 additions & 90 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,105 +25,39 @@ The following instructions will start dp3 with a data directory on local disk.
./dp3 server --data-dir data
```

3. Import your data
3. Start the interpreter and type help to get started

```
./dp3 import --producer my-robot example-data/fix.mcap
```
[/mnt/nvme/projects/dp3] (main) $ ./dp3 client
__ _ _____
\ \ __| |_ __|___ /
\ \ / _` | '_ \ |_ \
/ / | (_| | |_) |__) |
/_/ \__,_| .__/____/
|_|
Type "help" for help.
5. Query the data
dp3 # help
The dp3 client is an interactive interpreter for dp3. dp3 is a
multimodal log database for low-latency playback and analytics.
```
./dp3 export --producer my-robot -t /fix --json
```
The client supports interaction via either queries or slash commands. The
supported slash commands are:
```json
{"topic":"/fix","sequence":1193398,"log_time":1479513060.001367422,"publish_time":1479513060.001367422,"data":{"header":{"seq":115877,"stamp":1479513060.000963926,"frame_id":"/imu"},"status":{"status":0,"service":0},"latitude":37.39954376220703,"longitude":-122.10643005371094,"altitude":-8.820882797241211,"position_covariance":[0,0,0,0,0,0,0,0,0],"position_covariance_type":0}}
{"topic":"/fix","sequence":1193405,"log_time":1479513060.003966743,"publish_time":1479513060.003966743,"data":{"header":{"seq":115878,"stamp":1479513060.003520011,"frame_id":"/imu"},"status":{"status":0,"service":0},"latitude":37.39954376220703,"longitude":-122.10643005371094,"altitude":-8.820837020874023,"position_covariance":[0,0,0,0,0,0,0,0,0],"position_covariance_type":0}}
{"topic":"/fix","sequence":1193411,"log_time":1479513060.006441819,"publish_time":1479513060.006441819,"data":{"header":{"seq":115879,"stamp":1479513060.005985975,"frame_id":"/imu"},"status":{"status":0,"service":0},"latitude":37.39954376220703,"longitude":-122.10643005371094,"altitude":-8.82080078125,"position_covariance":[0,0,0,0,0,0,0,0,0],"position_covariance_type":0}}
```
\h [topic] to print help text. If topic is blank, prints this text.
\statrange to run a statrange query
\import to import data to the database
6. Inspect multigranular statistics
Available help topics are:
query: Show examples of query syntax.
statrange: Explain the \statrange command.
import: Explain the \import command.
Any input aside from "help" that does not start with a backslash is interpreted
as a query. Queries are terminated with a semicolon.
dp3 #
```

$ ./dp3 statrange --start 2010-01-01 --end 2020-01-01 -t /fix --producer my-robot --granularity 6000
| Start | End | Schema | Type | Field | Name | Value |
|---------------------------|---------------------------|----------|--------|---------------------------|------------------|-------------------------|
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | header.seq | mean | 63119.5 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | header.seq | min | 1 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | header.seq | max | 126238 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | header.seq | sum | 7.968079441e+09 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | text | header.frame_id | min | /imu |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | text | header.frame_id | max | /imu |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | status.status | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | status.status | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | status.status | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | status.status | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | status.service | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | status.service | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | status.service | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | status.service | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | latitude | mean | 37.399634673678946 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | latitude | min | 37.39691925048828 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | latitude | max | 37.402809143066406 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | latitude | sum | 4.721255081935883e+06 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | longitude | mean | -122.10455175078349 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | longitude | min | -122.1085205078125 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | longitude | max | -122.10015869140625 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | longitude | sum | -1.5414234403915405e+07 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | altitude | mean | -10.400682659429746 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | altitude | min | -14.843596458435059 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | altitude | max | -6.972412109375 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | altitude | sum | -1.3129613775610924e+06 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[0] | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[0] | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[0] | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[0] | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[1] | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[1] | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[1] | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[1] | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[2] | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[2] | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[2] | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[2] | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[3] | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[3] | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[3] | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[3] | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[4] | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[4] | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[4] | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[4] | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[5] | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[5] | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[5] | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[5] | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[6] | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[6] | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[6] | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[6] | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[7] | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[7] | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[7] | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[7] | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[8] | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[8] | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[8] | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance[8] | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance_type | mean | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance_type | min | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance_type | max | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | float | position_covariance_type | sum | 0 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | int | | messageCount | 126238 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | int | | byteCount | 1.514856e+07 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | int | | minObservedTime | 1.4795127703096174e+18 |
| 2016-11-18T14:56:00-08:00 | 2016-11-18T16:00:00-08:00 | 312f98d | int | | maxObservedTime | 1.4795130859038692e+18 |
```

See `dp3 -h` for an up to date listing of commands.

## Background
Multimodal log data may be characterized by,
* High frequencies
Expand Down
Loading

0 comments on commit f688894

Please sign in to comment.