python 3 support

[giampaolo]

From thomas.s...@gmail.com on February 18, 2010 23:18:39

Hi,

currently there is some effort in packaging this for fedora, and I'd like
to know, if you support python3 builds of psutil?

I needed to run 2to3 on the sources, but it seems like psutil works under
python3 after that.

Do you intend to use two different sources - one for python2 and one for
python3 - or do you prefer running 2to3 on them and thus support python2/3
in one tarball?

  Thomas

P.S. live session in python3:
(Maybe is process.name broken, or I did something wrong...)

$ python3
Python 3.1.1 ( r311 :74480, Jan 14 2010, 15:34:21) 
[GCC 4.4.2 20091222 (Red Hat 4.4.2-20)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.get_pid_list()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 42, 43, 45,
46, 47, 48, 114, 221, 226, 227, 230, 251, 252, 253, 437, 439, 440, 560,
561, 562, 589, 648, 867, 908, 1036, 1037, 1038, 1085, 1088, 1089, 1136,
1137, 1138, 1139, 1140, 1141, 1243, 1320, 1343, 1344, 1355, 1372, 1382,
1383, 1404, 1426, 1428, 1493, 1523, 1535, 1537, 1546, 1547, 1568, 1584,
1595, 1604, 1613, 1642, 1650, 1661, 1674, 1675, 1676, 1677, 1678, 1679,
1710, 1711, 1729, 1735, 1769, 1781, 1786, 1803, 1811, 1820, 1821, 1871,
1912, 1915, 1917, 1921, 1922, 1924, 1926, 1927, 1928, 1930, 1932, 1937,
1939, 1943, 1946, 1948, 1950, 1958, 1960, 1962, 1964, 1968, 1972, 1976,
1997, 2032, 2067, 2088, 2096, 2101, 2114, 2146, 2157, 2200, 2205, 2206,
2221, 2223, 2259, 2284, 2337, 2339, 2341, 2342, 2345, 2355, 2360, 2361,
2371, 2407, 2462, 2464, 2469, 2475, 2495, 2661, 2866, 3111, 3120, 3180, 3218]
>>> p = psutil.Process(2088)
>>> p.name
'pulseaudio'
>>> p.cmdline
['/usr/bin/pulseaudio', '--start']
>>> p.uid
500
>>> p.gid
500
>>> p.username
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Process' object has no attribute 'username'
>>> p.usernmae
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Process' object has no attribute 'usernmae'
>>> rss, vms =  p.get_memory_info()
>>> rss
4886528
>>> vms
458084352
>>>

Original issue: http://code.google.com/p/psutil/issues/detail?id=75

[giampaolo]

From jlo...@gmail.com on February 18, 2010 14:49:39

Hi Thomas, 

We haven't tested Python 3 at all actually - so it's good to hear that after running
2to3 on it everything works OK. If that's the case I see no objection to packing a
Python 3 build by running 2to3 on the current sources for now. I'd like to avoid
maintaining two separate code branches in any case. 

As far as your live session I'm not sure why you're getting the error... what version
of psutil are you running this with?

Status: Accepted
Labels: Portability

[giampaolo]

From thomas.s...@gmail.com on February 18, 2010 15:10:57

It's 0.1.2.

Running:
- dead_process.py
- leak_test.py
- test_cpu_percent.py
- test_memory_leaks.py
- process_listing.py

worked with python3


test_psutil.py did not work, because there is no test_support from the test module.


Great to hear, that you want to have python3 working. In case there won't be
something working, I'm sure, someone will file a bug report in fedora, till then, I'd
assume, that this is working, because the tests are mostly working :)

[giampaolo]

From thomas.s...@gmail.com on February 18, 2010 15:11:48

Ah, forgot to add the bug url (just in case ;)): https://bugzilla.redhat.com/show_bug.cgi?id=554714

[giampaolo]

From billiej...@gmail.com on February 18, 2010 16:15:59

I'm very glad to hear psutil is going to be packaged for Fedora but if I were you I'd
wait a little bit more for the 0.1.3 version to come out (which will be very soon)
which includes new functionnalities, fixes and a lot richer test suite.

Porting psutil to Python 3 is an exciting challenge and I'd like to do that, but
honestly I'm not sure on how to proceed exactly, whether letting the user run 2to3,
run 2to3 during the setup process or modify the code so that it works with both
versions (not sure if this would actually be possible).
In any case I agree with Jay that maintaining two separate code branches is evil and
should be avoided if possible.

I'm surprised as it seems the 2to3 tool made almost everything for us already.
Linux implementation is pure Python code, but Windows, OSX and FreeBSD
implementations are pure C: the porting process couldn't be as easy if the Python 3 C
API changed substantially as the 2to3 tool is not gonna take care of that.

I'll try to look into this soon and see if this can be implemented in a reasonable
amount of time, otherwise I'd leave this for 0.1.4 version.

PS - "username" functionnality didn't work out since it will be included in 0.1.3
version. It's my fault as I included it in the documentation too soon. =)

[giampaolo]

From thomas.s...@gmail.com on February 18, 2010 16:39:31

I believe, if you apply 2to3 on the sources, you require at least 2.6, so this
shouldn't be a problem to support both from one code basis without the need to run 2to3.

Running 2to3 from the user would be bad, but currently there are some other packages,
that run 2to3 on the sources while building and work after that pretty well. So this
is definetely a possibility, if you don't want to bump the requirement to 2.6.

For the C API changes, you might be interessted in: 
http://dmalcolm.livejournal.com/3935.html As long as 0.1.2 works, I don't see a problem to ultimately wait for 0.1.3 for the
packaging, but I'm 'just' the reviewer not the requestee ;)

It's good to hear, that you will support python3, if it's a bit experimental by now,
I can live with that ;)

(By the way: I still don't know, what from test import test_support does and after
deleting anything, the suite mostly runs just fine (exept for the username etc ;))
When you'd delete all 'test_support' everywhere, we could run the testsuite
everytime, when building the package.)

[giampaolo]

From billiej...@gmail.com on February 20, 2010 12:30:12

Tried to run 2to3 against the trunk on Linux + Python 3.1 resulting in 5 test
failures all due to unicode/bytes problems which need to be addressed by hand. 

> For the C API changes, you might be interessted in:
> http://dmalcolm.livejournal.com/3935.html I've tried it but it hasn't been of much help. It just converted PyInt_FromLong to
PyLong_FromLong, while the Py_InitModule section remained the same.
Here it is explained how to do that: 
http://docs.python.org/dev/3.0/howto/cporting.html#module-initialization-and-state
 I think that aside from PyInt_FromLong the only thing we'll have to change will be
this one.

> (By the way: I still don't know, what from test import test_support does and after
> deleting anything, the suite mostly runs just fine (exept for the username etc ;))
> When you'd delete all 'test_support' everywhere, we could run the testsuite
> everytime, when building the package.)

I solved this in r499 where test_support module is no longer mandatory.
test_support is part of Python tests which are not installed on most Linux platforms
(e.g. on Ubuntu you have to install them via "apt-get install python-dev").

[giampaolo]

From g.rodola on February 21, 2010 13:01:39

I'm happy to announce that I succesfully managed to complete the Python 3.X porting
on all platforms by entirely working on the main trunk so it seems we won't have to
maintain a separate branch (hooray!).

Here is a summary to keep track of what has been done.


C modules initialization code
=============================

Problem described here: 
http://docs.python.org/dev/3.0/howto/cporting.html#module-initialization-and-state Fixed on Windows, OSX and BSD as r500 , r502 , r503 .

long/int unification
====================

Problem described here: 
http://docs.python.org/dev/3.0/howto/cporting.html#long-int-unification Just renamed PyInt_FromLong to PyLong_FromLong everywhere as r500 which worked for
both 2.X and 3.X.

bytes/string unification
========================

Part of the problem described here: 
http://docs.python.org/dev/3.0/howto/cporting.html#str-unicode-unification The problem was that process name, path, cmdargs and getcwd() values returned on
Python 3 was no longer strings but bytes instead.
This occurred only on Windows as it's the only platform which uses
PyUnicode_AsUTF8String().
I replaced it with PyUnicode_FromObject() for Python 3 only as r507 and everything
went fine.

2to3
====

( r504 ) I modified setup.py which now includes a (ugly) hack for Python 3 to tell
distutils to run 2to3 against the files which gets copied in the build directory
before installing.
This helps us to leave the source code that gets extracted from the tarball unchanged
so that it can be reused if the user wants to make another installation for Python 2.x.
The only effort is that 2to3 is required to be manually run against the test
directory when we want to run tests on Python 3.X.

test suite
==========

Various tests were failing on Python 3 because of string/bytes issues which I
addressed by using "if sys.version_info >= (3,): ..." clauses here and there. This
can be observed in r505 and r506 .


The only platform I didn't test is OSX. Jay could you please provide a feedback?

[giampaolo]

From billiej...@gmail.com on March 01, 2010 07:54:58

Labels: Milestone-0.1.3

[giampaolo]

From jlo...@gmail.com on March 02, 2010 10:42:36

Tests pass on OS X 10.4 for Python 3.0 and 3.1 with the exception of occasional
memory test failures caused by fluctuating memory usage - probably safe to ignore
those and call it good.

Labels: -Progress-3in4 Progress-4in4

[giampaolo]

From billiej...@gmail.com on March 02, 2010 11:53:39

Status: Fixed
Labels: -Type-Defect Type-Enhancement

[giampaolo]

From g.rodola on March 02, 2013 03:51:36

Updated csets after the SVN -> Mercurial migration: r311 == revision 
3bcd08a26f5e r499 == revision 0cef53d57a4f r500 == revision 9a5863028432 r504 
== revision 82a890fed2bb r505 == revision 924dd10a518e r507 == revision ca26ca6bdf6e