Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

-max-response-body-size should be used to limit body download #325

Closed
nil0x42 opened this issue Jul 22, 2021 · 1 comment · Fixed by #346 or #347
Closed

-max-response-body-size should be used to limit body download #325

nil0x42 opened this issue Jul 22, 2021 · 1 comment · Fixed by #346 or #347
Assignees
Labels
Status: Completed Nothing further to be done with this issue. Awaiting to be closed. Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.

Comments

@nil0x42
Copy link

nil0x42 commented Jul 22, 2021

Describe the bug
From now, -max-response-body-size is only used to write a truncated file.
However, IMHO it should be used at network level (only read() up-to -max-response-body-size). So httpx doesn't eat all bandwtidth if input has lot of URLs with multi-GB content.

Doing this would then save CPU-time on -match-regex, and save network bandwitdh overall when max-response-body-size is used

@Mzack9999 Mzack9999 added the Type: Discussion Some ideas need to be planned and disucssed to come to a strategy. label Jul 26, 2021
@ehsandeep ehsandeep added Status: Available No one has claimed responsibility for resolving this issue Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. and removed Type: Discussion Some ideas need to be planned and disucssed to come to a strategy. labels Aug 3, 2021
@Mzack9999 Mzack9999 added Status: In Progress This issue is being worked on, and has someone assigned. and removed Status: Available No one has claimed responsibility for resolving this issue labels Aug 6, 2021
@Mzack9999 Mzack9999 linked a pull request Aug 7, 2021 that will close this issue
@ehsandeep ehsandeep added Status: Completed Nothing further to be done with this issue. Awaiting to be closed. and removed Status: In Progress This issue is being worked on, and has someone assigned. labels Aug 7, 2021
@ehsandeep
Copy link
Member

Hi @nil0x42,

Now response-size-to-read will consider user input in bytes to limit response read size, additionally, now you can also control how much data you want to write to file as well with response-size-to-save flag.

As default, both flags are set to default, i.e unlimited.

@ehsandeep ehsandeep linked a pull request Aug 7, 2021 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Completed Nothing further to be done with this issue. Awaiting to be closed. Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
4 participants