Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Major] Dataloader: Just-In-Time tabularization #1529

Merged
merged 141 commits into from
Jun 26, 2024
Merged

[Major] Dataloader: Just-In-Time tabularization #1529

merged 141 commits into from
Jun 26, 2024

Conversation

ourownstory
Copy link
Owner

@ourownstory ourownstory commented Feb 9, 2024 •
edited
Loading

Complete re-write of dataloader: Instead of tabularizing samples ahead of batching, samples are tabularized and features computed during batch creation. Trades Memory use for Compute.

This is an early implementation that works, but that needs to be compute-optimized.

SimonWittner and others added 30 commits December 13, 2023 09:37
@SimonWittner
minimal pytest
34909cb
@SimonWittner
move_func_getitem
687c085
@SimonWittner
slicing
5215340
@SimonWittner
predict_mode
c70fae2
@SimonWittner
typos
b78d5e0
@SimonWittner
lr-finder
beae5bb
@SimonWittner
drop_missing
8427ffc
@SimonWittner
predict_v2
ff05b2a
@SimonWittner
predict_v3
c408e95
@SimonWittner
samples
df29f33
@SimonWittner
lagged regressor n_lags
29fe999
@SimonWittner
preliminary: events, holidays
2f584c2
@SimonWittner
adjustes pytests
fca7adf
@SimonWittner
selective forecasting
139a97f
@SimonWittner
black
30aa303
@SimonWittner
ruff
381c912
@SimonWittner
lagged_regressors
660934c
@ourownstory
Note down df path to TimeDataset
51fa0a6
@ourownstory
complete notes on TimeDataset, move meta
da74f87
@ourownstory
Big rewrite with real and pseudocode
97fbe07
@ourownstory
Merge branch 'main' into dataloader-fly-TimeDataset
3cfa2be
@ourownstory
Merge branch 'main' into dataloader-fly-TimeDataset
b1f084b
@ourownstory
create_target_start_end_mask
bdf529c
@ourownstory
boolean mask
c814115
@ourownstory
combine masks into map
7119419
@ourownstory
notes for nan check
66bb911
@ourownstory
bypass NAN filter
fe382c1
@ourownstory
rework index to point at prediction origin, not first forecast.
8ec4f9f
@ourownstory
tabularize: converted time and lags to single sample extraction
23d6100
@ourownstory
convert lagged regressors
49af45b
@MaiBe-ctrl MaiBe-ctrl force-pushed the dataloader-jit branch 2 times, most recently from 3f39771 to 60260bd Compare June 22, 2024 03:06
ourownstory
ourownstory commented Jun 22, 2024
View reviewed changes
Copy link
Owner Author

@ourownstory ourownstory left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's consolidate our tests for holidays and events!

tests/test_unit.py Outdated Show resolved Hide resolved
tests/test_hdays_utils.py Outdated Show resolved Hide resolved
tests/test_hdays_utils.py Outdated Show resolved Hide resolved
tests/test_integration.py Outdated Show resolved Hide resolved
tests/test_unit.py Outdated Show resolved Hide resolved
@ourownstory ourownstory added this to the 1.0.0rc10 milestone Jun 26, 2024
MaiBe-ctrl
MaiBe-ctrl approved these changes Jun 26, 2024
View reviewed changes
Copy link
Collaborator

@MaiBe-ctrl MaiBe-ctrl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job!

@MaiBe-ctrl MaiBe-ctrl merged commit 1bfa633 into main Jun 26, 2024
9 of 11 checks passed
@MaiBe-ctrl MaiBe-ctrl deleted the dataloader-jit branch June 26, 2024 23:35
@joejoe2 joejoe2 mentioned this pull request Jun 27, 2024
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MaiBe-ctrl MaiBe-ctrl MaiBe-ctrl approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.0.0rc10
Development

Successfully merging this pull request may close these issues.
3 participants
@ourownstory @MaiBe-ctrl @SimonWittner