Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid TensorFlow empty inputs in central interface #44481

Open
valsdav opened this issue Mar 20, 2024 · 6 comments
Open

Avoid TensorFlow empty inputs in central interface #44481

valsdav opened this issue Mar 20, 2024 · 6 comments

Comments

@valsdav
Copy link
Contributor

valsdav commented Mar 20, 2024

Empty Tensors passed to the TensorFlow session may cause crashes on some architectures while returning silently empty results on others. See #44333

An explicit check on input tensors in the TensorFlow interface can prevent the issue once for all here:

  • do not run the inference if input.NumElements() == 0
  • Print a warning to the framework

A PR will follow, any thoughts about this?

@valsdav
Copy link
Contributor Author

valsdav commented Mar 20, 2024

assign ml

@cmsbuild
Copy link
Contributor

New categories assigned: ml

@valsdav,@wpmccormack you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

cms-bot internal usage

@cmsbuild
Copy link
Contributor

A new Issue was created by @valsdav.

@smuzaffar, @Dr15Jones, @rappoccio, @sextonkennedy, @antoniovilela, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor

  • Print a warning to the framework

How necessary a warning is? Is the expectation that users would also avoid TF inference calls when input is empty, or that only the central TF interface does that?

I mean, an empty input (no objects) sounds completely valid input from physics side, so I'm wondering if the TF inference call could be just omitted silently in those cases?

@valsdav
Copy link
Contributor Author

valsdav commented Mar 22, 2024

My expectation is that checks should be put in producers to avoid calling the inference for empty inputs, and that the guard in the central TF code is a more a protection.

Returning empty results would be correct for empty inputs, but if we do it silently we wouldn't notice problems (like in the DeepTau case) when the producer is having unexpected empty inputs and not having protections. What do you think about this?

I can do a quick survey of the producers using TF to check is protections are in place and add them also there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants