[Threadless] OSError: [WinError 10038] an operation was attempted on something that is not a socket

[lubosmato]

Describe the bug
When starting proxy.py in threadless mode on Windows, OS error is raised. Threaded mode works fine.

Here is a stack trace:
(num workers option is specified only to see the error just once, it occurs on all workers and floods stdout)

C:\>python -m proxy --threadless --num-workers 1
2021-01-28 07:43:24,739 - pid:5480 [I] load_plugins:334 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2021-01-28 07:43:24,740 - pid:5480 [I] listen:113 - Listening on ::1:8899
2021-01-28 07:43:24,744 - pid:5480 [I] start_workers:136 - Started 1 workers
Process Threadless-1:1:
Traceback (most recent call last):
  File "D:\Python38\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "D:\Python38\lib\site-packages\proxy\core\acceptor\threadless.py", line 175, in run
    self.run_once()
  File "D:\Python38\lib\site-packages\proxy\core\acceptor\threadless.py", line 146, in run_once
    with self.selected_events() as (readables, writables):
  File "D:\Python38\lib\contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "D:\Python38\lib\site-packages\proxy\core\acceptor\threadless.py", line 73, in selected_events
    ev = self.selector.select(timeout=1)
  File "D:\Python38\lib\selectors.py", line 323, in select
    r, w, _ = self._select(self._readers, self._writers, [], timeout)
  File "D:\Python38\lib\selectors.py", line 314, in _select
    r, w, x = select.select(r, w, w, timeout)
OSError: [WinError 10038] an operation was attempted on something that is not a socket

I couldn't reproduce it on WSL so I assume this is only Windows problem.

To Reproduce
Steps to reproduce the behavior:

1. Run python -m proxy --threadless on Windows
2. See the error

Expected behavior
Proxy shouldn't raise this error I guess.

Version information

• OS: Windows:

OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.19041 N/A Build 19041
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Workstation
OS Build Type:             Multiprocessor Free
Processor(s):              1 Processor(s) Installed.
                           [01]: AMD64 Family 23 Model 113 Stepping 0 AuthenticAMD ~3893 Mhz
BIOS Version:              American Megatrends Inc. F2, 7. 7. 2020
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume1
System Locale:             cs;Čeština
Input Locale:              cs;Čeština
Time Zone:                 (UTC+01:00) Praha, Bratislava, Budapešť, Bělehrad, Lublaň
Total Physical Memory:     65 482 MB
Available Physical Memory: 48 516 MB
Virtual Memory: Max Size:  75 210 MB
Virtual Memory: Available: 54 568 MB
Virtual Memory: In Use:    20 642 MB
Hotfix(s):                 10 Hotfix(s) Installed.
                           [01]: KB4586876
                           [02]: KB4561600
                           [03]: KB4570334
                           [04]: KB4577266
                           [05]: KB4580325
                           [06]: KB4584229
                           [07]: KB4586864
                           [08]: KB4593175
                           [09]: KB4598481
                           [10]: KB4598242
Network Card(s):           4 NIC(s) Installed.
                           [01]: Realtek Gaming 2.5GbE Family Controller
                                 Connection Name: Ethernet
                                 DHCP Enabled:    Yes
                                 DHCP Server:     192.168.0.1
                                 IP address(es)
                                 [01]: 192.168.0.109
                                 [02]: xxxxx
                           [02]: Intel(R) Dual Band Wireless-AC 3168
                                 Connection Name: Wi-Fi
                                 Status:          Media disconnected
                           [03]: Bluetooth Device (Personal Area Network)
                                 Connection Name: Síťové připojení Bluetooth
                                 Status:          Media disconnected
                           [04]: Hyper-V Virtual Ethernet Adapter
                                 Connection Name: vEthernet (WSL)
                                 DHCP Enabled:    No
                                 IP address(es)
                                 [01]: 172.31.160.1
                                 [02]: xxxxx
Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.    

• Python: Python 3.8.5 x64
• proxy.py: 2.3.1

Additional context
Running proxy.py in WSL is without any problem.

mato@mato-pc:/mnt/c/Windows/system32$ python3 --version
Python 3.8.5
mato@mato-pc:/mnt/c/Windows/system32$ pip3 list | grep proxy.py
proxy.py               2.3.1
mato@mato-pc:/mnt/c/Windows/system32$ python3 -m proxy --threadless
2021-01-28 07:53:20,437 - pid:84 [I] load_plugins:334 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2021-01-28 07:53:20,437 - pid:84 [I] listen:113 - Listening on ::1:8899
2021-01-28 07:53:20,447 - pid:84 [I] start_workers:136 - Started 16 workers

[lubosmato]

The problem here is that syscall select supports only sockets on Windows platform and proxy.py uses select for multiprocessing.connection.PipeConnection.

Alternative to select syscall on Windows: https://tinyclouds.org/iocp-links.html

[abhinavsingh]

@lubosmato Thank you for digging into it. One of the reasons I never enabled --threadless as default due to lack of testing across OS (especially Windows). IIUC, this is a problem only with --threadless mode and proxy.py works file in default threaded mode.

I'll check the possibilities at my end, but due to time constraints currently at my end, I won't be able to jump into it anytime soon. I'll appreciate Windows users to chip in with suggestion/proposals/PR and get us past this :)

[webknjaz]

FTR --threadless also causes (seemingly similar) problems on macOS (although the root cause may differ). Weirdly enough, it sometimes works without any issues with --threadless on all the platforms and it's absolutely unclear why that is.

[abhinavsingh]

@webknjaz Thank you for context about macOS. Frankly I haven't seen a problem on my MacBook yet, having said that Python version changes have occurred including Big Sur upgrades. I have started proxy.py locally on my laptop and will give it a dry run for a week. I have done this before and never faced any issues on mac. Will get back on it.

Regarding Windows, yes, I never get a chance to test on Windows. I will certainly need some help to get it fixed for Windows. May be one of the Windows users can help dig into it and I can guide as needed. At my end, I'll try to get a Windows VM up on VirtualBox and see how it goes.

PS: I am kind of swamped right now due to upcoming month end release (for another project). Give me sometime, will definitely get back to pending issues on GitHub. Thank you 🙏

[abhinavsingh]

PS: It's about a month I am running proxy.py locally on macOS BigSur with --threadless. I haven't observed anything noticeable.

Only occasion when I had to stop proxy.py was while uploading ipa via XCode. Reason for which was uploaded ipa was huge in size and requires bumping up buffer values to work reliably.

I am still curious to understand where and why does --threadless fail (outside of Windows).

On Windows, definitely we needs an alternative strategy or may be windows can stay on threaded mode :(

[abhinavsingh]

@lubosmato @webknjaz I wanted to make --threadless as default (in v2.4.0) but seemingly it's not safe to do so at this stage. Let's do it in another future release :)

From what I observe and realize, most proxy.py users are using --threadless mode now. At least, if they have discovered it :)

[webknjaz]

@abhinavsingh TL;DR This mostly seems problematic under the combination of Python 3.6 and macOS or Windows.

We use --threadless in tests on the master branch https://github.com/aio-libs/aiohttp/blob/31b90c1/tests/test_proxy_functional.py#L44, it's running against a matrix of Python 3.7-3.10 and Ubuntu/macOS/Windows.

But we had to only use it under Ubuntu on the 3.8 branch because it also runs tests under Python 3.6 and crashes under macOS/Windows with unclear root cause.
https://github.com/aio-libs/aiohttp/blob/v3.8.0/tests/test_proxy_functional.py#L53-L54
I think maybe since both share the I/O loop and the context manager is synchronous, some internals time out. Maybe spawning proxy.py as a subprocess would work, though. I haven't tested that.

[abhinavsingh]

Thanks for bringing version specific behavior to my notice. I am thinking if this is the case, then may be proxy.py can disable --threadless for offending versions while keeping it default for where it works reliably. IIUC, it can be set as default for 3.7+ on all platform (except Win).

Solving for Windows will be a separate story I feel. Either we'll need to change/re-invent how pipes are currently being used to pass descriptors, or avoid multi-accept on Windows (this will remove needs of pipe), or simply have an alternate implementation (may be iocp) for windows at some point.

[webknjaz]

FWIW for some reason it works under Python 3.7+ on Windows in our CI. No idea why... Maybe we do something to the ioloop that "fixes" it, dunno.

[abhinavsingh]

Per pypistats I think it is safe to set default --threadless flags for Python 3.8+. Not for windows though, as it may need further investigation. Issue reported by @lubosmato is from Python 3.8.5 version.

Distribution across python versions:

• 88% using 3.8
• 8% using using 3.9
• 2% using 3.10 -- Mostly CI/CD integrations or new adopters?
• 1.5% using 2.7 -- Mostly dead projects or still pinned to pre-v2 version of proxy.py
• 0.5% using 3.6 and 3.7 each -- Mostly CI/CD integrations?

Maybe we do something to the ioloop that "fixes" it, dunno.

@webknjaz This is interesting, if that's the case, we could apply same patch within proxy.py core.

[abhinavsingh]

Python has a ProactorEventLoop which uses IOCP internally. May be we can look into it at some point.

[abhinavsingh]

Update

I checked on a Windows system. ProactorEventLoop was being used by default. But still we are getting:

• an operation was attemped on something that is not a socket (for threadless remote mode)
• oserror (for threadless local mode)

Only --threaded mode works on Windows (default when running proxy from command line).

PS: Threaded uses asyncio internally to manage connection lifecycle. Threaded implies one-thread-per-connection.