-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
multiple bug fixes + performance improvements #5148
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit d3131f7.
Closed
tarunKoyalwar
changed the title
feat error enhancements
multiple bug fixes + performance improvements
May 23, 2024
Mzack9999
approved these changes
May 23, 2024
ehsandeep
approved these changes
May 24, 2024
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bug fix 🐛
Http req @timeout annotation not working
The
@timeout
annotation for HTTP requests is not working in the latest release. This issue is likely a side effect of recent bug fixes, including those for goroutine leaks and nuclei hang/stuck issues. The problem seems to stem from the context inhttpclientpool
being always overwritten. Adding selective overwrite has resolved the issueMax Host Error causes input to be skipped
Nuclei was skipping active domains or IPs due to errors returned while executing a template on closed or filtered ports. These errors were being matched across the maxhosterrors set of errors. This issue was partly addressed by updating the definitions of errors that should be skipped in maxhosterrors. In addition, the hosterrorcache has been refactored to consider the address as host:port instead of host. This change avoids counting legitimate port closed errors against the host.
Performance Improvements 🚀
For complete context, refer projectdiscovery/fastdialer#283. In summary, errors were being stacked when a domain, such as [hackerone.com](http://hackerone.com), resolved to multiple IPs. This issue occurred during attempts to connect to closed ports, significantly reducing the scan speed, particularly in the middle or end of the scan.
After several iterations in fastdialer and changes in retryablehttp, we have addressed this issue. We implemented a singleflight parallel dial on all IPs during the first dial of a given address. This approach effectively prevents the significant speed drop in the middle or end of the scan. As a result, we have observed remarkable speed improvements of 40-70%, depending on the target and concurrency levels.
Enhancements 🧨
Improving Error Ecosystem in Nuclei
This PR, implementing the
errkit
package fromprojectdiscovery/errkit
, enhances error accuracy and user-friendliness. For more details, refer to errkit.In the context of nuclei, we have reduced vague errors like
context deadline exceeded
and improved the-elog
output format to includeaddress
,kind
andattrs
(optional)the overall result of this change can be verified using
-elog
output , sorted , uniq errorsBefore
refer https://gist.github.com/tarunKoyalwar/8ddb1fc5a9e21dbd90ac26d2330c705b#file-old_unique_errors-txt for complete output
After
Other Improvements
nuclei/postgres
js moduleThreadSafeNucleiEngine
(closes 使用sdk包创建的并发安全的引擎,因共享资源的释放无法正常执行完全部任务 #5202)NUCLEI_ARGS="host-error-stats" cmdutil ./nuclei -u https://hackerone.com -stats -elog newupdate2.log -c 200
-timeout
network
andjavascript
protocolsResponseHeaderTimeout
to10 Sec