Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Size of task prototype #52

Open
sebffischer opened this issue Oct 13, 2023 · 3 comments
Open

Size of task prototype #52

sebffischer opened this issue Oct 13, 2023 · 3 comments
Assignees
Labels

Comments

@sebffischer
Copy link
Sponsor Member

When creating a graph learner that has as input a task with tf columns, the data_prototype that is saved in the learner's state after training contains the arg and value vectors as well as the evaluator and other metadata defined in tf.
This unnecessarily blows up the size of learner states in a way that was not intended.

I think this should be fixed in tf, i.e. 0-lentgh tf vectors should drop discardable metadata.

@sebffischer sebffischer added bug Something isn't working and removed bug Something isn't working labels Oct 27, 2023
@sebffischer sebffischer added the tf label Nov 24, 2023
@sebffischer sebffischer added this to the 0.1.0 milestone Nov 24, 2023
@sebffischer
Copy link
Sponsor Member Author

sebffischer commented Dec 8, 2023

With recent PRs in mlr3 and mlr3misc, this problem should be mostly mitigated:

@sebffischer
Copy link
Sponsor Member Author

As we now decided to not merge the warning, we should do something about this.
E.g. in mlr3 when creating the data prototype during $train(), it should be possible to add a function that leanifies each column, e.g. stored in mlr_reflections$data_leanifier$tf. This function would then remove the srcref attribute from the functional columns to avoid overly large object sizes when installing with sourcerefs. In the resample() case this is no problem because the prototype is not kept in the learner state.

@sebffischer
Copy link
Sponsor Member Author

sebffischer commented Feb 6, 2024

@sebffischer sebffischer self-assigned this Feb 6, 2024
@sebffischer sebffischer removed this from the 0.1.0 milestone Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant