import psutil raises IOError on Linux when /proc/stat is inaccessible #715
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For safety reasons, reloading a module does not reinitialize c
extensions. For this reason, `del <c_extension>` in `__init__.py`
yields the following on reimport:
```
Traceback (most recent call last):
File "/src/test/_linux.py", line 479, in test_psutil_is_reloadable
imp.reload(psutil)
File "psutil/__init__.py", line 89, in <module>
from . import _psutil_linux
ImportError: cannot import name _psutil_linux
```
As none of the other platform-specific extensions are removed from
the module dict post-import, we simply omit the `del` for linux.
sethp-jive
force-pushed
the
fix/procfs-import
branch
3 times, most recently
from
eb73ec6
to
d24f52a
Compare
The very excellent work that went in under giampaolo#558 gave us a psutil that can be used on systems with a moved /proc. However, on some systems (e.g. test harnesses, containers) may not have any files mounted at /proc at all. In these cases, psutil fails at import time with an IOError trying to access /proc/stat. This patch removes the import-time dependency on a live procfs mounted at /proc, and validates the behavior of some of the details we were hoping to read at import time (i.e. cpu % stats).
sethp-jive
force-pushed
the
fix/procfs-import
branch
from
d24f52a
to
b4e9adf
Compare
sethp-jive
changed the title
| @@ -1522,8 +1525,8 @@ def cpu_times_percent(interval=None, percpu=False): | |||
| def calculate(t1, t2): | |||
| nums = [] | |||
| all_delta = sum(t2) - sum(t1) | |||
| for field in t1._fields: | |||
| field_delta = getattr(t2, field) - getattr(t1, field) | |||
| for field in t2._fields: | |||
There was a problem hiding this comment.
why did you change from t1 to t2?
There was a problem hiding this comment.
t1 is the least-recent set of data points, so if the underlying /proc/stat file changes to provide a different number of fields (e.g. if we go from not having fields to having some fields) then t2 will have the latest and most correct set of fields.
The danger is that t1 will be missing some values, hence the default attr provided on the next line.
giampaolo
added a commit
that referenced
this pull request
Dec 14, 2015
|
I reorganized the import machinery so that we no longer crash at import time if |
|
Indeed I will – give me a few minutes and I should have it ready. |
mrjefftang
added a commit
to mrjefftang/psutil
that referenced
this pull request
Jan 5, 2016
* giampaolo/master: (148 commits) update doc add DEVNOTES.rst Add import-time tests for psutil one import per line use with lock: set global cpu vars to None if they can't be determined at import time add more tests giampaolo#717: ignore everything after the first occurrence of '\x00' instead of replacing '\x00' for the whole string fix giampaolo#717: [Linux] Process.open_files fails if deleted files still visible. giampaolo#715: don't crash at import time if cpu_times() fail for some reason. safety measure for ZombieProcess exc giampaolo#718: process_iter() thread safety giampaolo#708: use buffering for open() only on Python 2; on Python 3 this does not have any effect so it's better to let python decide what to do little speedup for system connections fix giampaolo#708 [Linux]: speedup psutil.net_connections() and psutil.Process.connections() linux refactoring: use a wrapper around open() for binary files update doc raise no memory err if malloc() fails giampaolo#714: [OpenBSD] return shared virtual mem add test for vmem total on freebsd ...
giampaolo
added a commit
that referenced
this pull request
Oct 12, 2018
nlevitt
added a commit
to nlevitt/psutil
that referenced
this pull request
Apr 9, 2019
* origin/master: (182 commits) giampaolo#1394 / windows / process exe(): convert errno 0 into ERROR_ACCESS_DENIED; errno 0 occurs when the Python process runs in 'Virtual Secure Mode' pre-release fix win num_handles() test update readme fix giampaolo#1111: use a lock to make Process.oneshot() thread safe pdate HISTORY giampaolo#1373: different approach to oneshot() cache (pass Process instances around - which is faster) use PROCESS_QUERY_LIMITED_INFORMATION also for username() Linux: refactor _parse_stat_file() and return a dict instead of a list (+ maintainability) fix giampaolo#1357: do not expose Process' memory_maps() and io_counters() methods if not supported by the kernel giampaolo#1376 Windows: check if variable is NULL before free()ing it enforce lack of support for Win XP fix giampaolo#1370: improper usage of CloseHandle() may lead to override the original error code resulting in raising a wrong exception update HISTORY (Windows) use PROCESS_QUERY_LIMITED_INFORMATION access rights (giampaolo#1376) update HISTORY revert 5398c48; let's do it in a separate branch giampaolo#1111 make Process.oneshot() thread-safe sort HISTORY give CREDITS to @EccoTheFlintstone for giampaolo#1368 fix ionice set not working on windows x64 due to LENGTH_MISMATCH (giampaolo#1368) make flake8 happy give CREDITS to @amanusk for giampaolo#1369 / giampaolo#1352 and update doc Add CPU frequency support for FreeBSD (giampaolo#1369) giampaolo#1359: add test case for cpu_count(logical=False) against lscpu utility disable false positive mem test on travis + osx fix PEP8 style mistakes give credits to @koenkooi for giampaolo#1360 Fix giampaolo#1354 [Linux] disk_io_counters() fails on Linux kernel 4.18+ (giampaolo#1360) giampaolo#1350: give credits to @amanusk FreeBSD adding temperature sensors (WIP) (giampaolo#1350) pre release sensors_temperatures() / linux: convert defaultdict to dict fix giampaolo#1004: Process.io_counters() may raise ValueError fix giampaolo#1307: [Linux] disk_partitions() does not honour PROCFS_PATH refactor hasattr() checks as global constants giampaolo#1197 / linux / cpu_freq(): parse /proc/cpuinfo in case /sys/devices/system/cpu fs is not available fix giampaolo#1277 / osx / virtual_memory: 'available' and 'used' memory were not calculated properly travis / osx: set py 3.6 travis: disable pypy; se py 3.7 on osx skip test on PYPY + Travis fix travis fix giampaolo#715: do not print exception on import time in case cpu_times() fails. fix different travis failures give CREDITS for giampaolo#1320 to @truthbk [aix] improve compilation on AIX, better support for gcc/g++ + fix cpu metrics (giampaolo#1320) give credits to @alxchk for giampaolo#1346 (sunOS) Fix giampaolo#1346 (giampaolo#1347) giampaolo#1284, giampaolo#1345 - give credits to @amanusk Add parsing for /sys/class/thermal (giampaolo#1345) Fix decoding error in tests catch UnicodeEncodeError on print() use memory tolerance in occasionally failing test Fix random 0xC0000001 errors when querying for Connections (giampaolo#1335) Correct capitalization of PyPI (giampaolo#1337) giampaolo#1341: move open() utilities/wrappers in _common.py Refactored ps() function in test_posix (giampaolo#1341) fix giampaolo#1343: document Process.as_dict() attrs values giampaolo#1332 - update HISTORY make psutil_debug() aware of PSUTIL_DEBUG (giampaolo#1332) also include PYPY (or try to :P) travis: add python 3.7 build add download badge remove failing test assertions remove failing test make test more robust pre release pre release set version to 5.4.7 OSX / SMC / sensors: revert giampaolo#1284 (giampaolo#1325) setup.py: add py 3.7 fix giampaolo#1323: [Linux] sensors_temperatures() may fail with ValueError fix failing linux tests giampaolo#1321 add unit tests giampaolo#1321: refactoring make disk_io_counters more robust (giampaolo#1324) fix typo Fix DeprecationWarning: invalid escape sequence (giampaolo#1318) remove old test update is_storage_device() docstring fix giampaolo#1305 / disk_io_counters() / Linux: assume SECTOR_SIZE is a fixed 512 giampaolo#1313 remove test which no longer makes sense disk_io_counters() - linux: mimic iostat behavior (giampaolo#1313) fix wrong reference link in doc disambiguate TESTFN for parallel testing fix giampaolo#1309: add STATUS_PARKED constant and fix STATUS_IDLE (both on linux) give CREDITS to @sylvainduchesne for giampaolo#1294 retain GIL when querying connections table (giampaolo#1306) Update index.rst (giampaolo#1308) fix giampaolo#1279: catch and skip ENODEV in net_if_stat() appveyor: retire 3.5, add 3.7 revert file renaming of macos files; get them back to 'osx' prefix winmake: add upload-wheels cmd Rename OSX to macOS (giampaolo#1298) apveyor: reset py 3.4 and remove 3.7 (not available yet) try to fix occasional children() failure on Win: https://ci.appveyor.com/project/giampaolo/psutil/build/job/je3qyldbb86ff66h appveyor: remove py 3.4 and add 3.7 giampaolo#1284: give credits to @amanusk + some minor adjustments little refactoring Osx temps (giampaolo#1284) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello! I found a few issues trying to take advantage of the new relocatable procfs, predominately that trying to import the psutils module on a system with no /proc/stat file fails. Additionally, I found that attempting to reload the psutils module (which is very useful for testing import-time issues) on linux caused some trouble.
Please see the individual commit messages for more details. Thanks!