Joe Amenta's Blog

GSoC 2010 is officially over, and I feel that now is a good time to make plans for the future of lib3to2.

I will release “1.0″ when I have fully tested lib3to2 in both Python 3.1 and Python 2.7.  I will also release a version of lib3to2 specifically for Python 2.5/2.6/2.7/3.0/3.1, having already put the current lib2to3 trunk into a PyPI package called “two2three” to make lib3to2 more accessible to users of those versions of Python.  lib2to3, as released in each one of those versions of Python (except 2.5, which didn’t have it) has at least one bug that significantly affects lib3to2′s behavior.  Depending on how this ends up going, I may just end up making this the default, since most instances of Python in use will have the problem for quite a while.

This should hopefully make lib3to2 more widely used and thus generate more feedback.  I will not have the time to devote to testing lib3to2 on every piece of Python 3 code I can find, so I will need this feedback to help figure out where (and if) lib3to2 is deficient, beyond what I’ve already found.

I’ve spent the better part of today and some of Monday and Tuesday hunting down why so many tests in brain are failing.  While I haven’t completely followed the tracebacks of every single error or failure, I can say that for those that I have, the problem does not appear to be caused by lib3to2, but rather by what I can only explain as differences in the sqlite3 module between the Python3.1 version and the Python2.7 versions.  This suggests that there might still be some work to do with lib3to2.

However, I really don’t see a possible lib3to2-based solution for this.  I wouldn’t even know where to start with this, since sqlite3 is just a thin wrapper around _sqlite3, a C extension module included with Python.  I’m going to tentatively call this one finished, after I fix a bug involving print statements and parentheses.  It shows that even though lib3to2 is pretty good at syntax refactoring, there are some differences (often very subtle) between Python3 and Python2 that lib3to2 will fail to catch and will need manual refactoring (and it’s thanks to brain’s excellent test suite that I found those in the first place!).  Furthermore, lib3to2 is best used as an aid throughout the entire development process, as it will catch incompatibility problems early, before too much code starts building on incompatible code.

I’m also running 3to2′d code with Python 2.7, so there may be some additional problems with Python 2.6 and earlier.

On PyPI, I’ve run across a substantial third-party module written in Python 3 that is well-tested and demonstrates significant errors with lib3to2.  It’s called brain, and it has 202 test cases.  They all pass in Python 3.1, but less than a quarter of them pass in Python 2.7 after running 3to2 on them:

FAIL: 0 failures, 158 errors, 44 passed

This is a good real-world example of native Python 3 code that doesn’t get properly refactored in with lib3to2, and I’m going to really focus on this.

My new goal is to get that down to 0 errors.  Some failures might be OK: Python 3 and Python2 have semantic differences in some cases, especially regarding the differences Python2 str and Python3 bytes objects, that would justify leaving the code untouched.

System setup:

In order to efficiently test this, I have done the following tricks (on Linux):

• Downloaded the gzipped tarball and extracted it to /home/joe/brain-0.1.6
• Made a symbolic link to ../brain in /home/joe/brain-0.1.6/brain/test
• cd /home/joe/brain-0.1.6/brain/test
• Run 3to2 with the command 3to2 -j10 -w –no-diffs -fall -xprint -fprintfunction {brain/,internal/,public/,}*.py (-j10: run 10 separate processes to fully utilize my CPU to make this finish a lot sooner, -w: write to file, –no-diffs: don’t give me the diff output, since I’m just running the test anyway, {-fall, -xprint, -fprintfunction}: Run with the normal set of fixers, except use “from __future__ import print_function” instead of refactoring to print statement… for now, and {brain/,internal/,public/,}*.py: bash sorcery to get all the .py files in the current directory and in the public, internal, and brain directories.)
• Change all unicode literals from run.py into regular string literals (optparse barks at you otherwise) (I have two shell aliases for doing that: alias a=’sed -i s/u\’\”/\’\”/g run.py’; alias b=’sed -i ‘\”s/u”/”/g’\” run.py’… yes, about half an hour of my time was indeed spent just getting the right combination of shell quotes and escapes to play nicely with the regex quotes and escape)
• Then run python2.7 run.py func

I’ve been devoting much of my time to testing on the Python 3.1 standard library, and let me say that it is really frustrating.  I’ve been trying to make various sufficiently standalone modules pass their associated tests (also run with 3to2), and I’ve been successful in some ways.  In particular, Lib/site.py passes all the tests in Lib/test/test_site.py, which I’m really happy about.  However, in other cases like Lib/io.py, there’s such a strong reliance on the behavior pre-compiled builtin C modules that I can’t possibly get it to work 3to2′d.

And there’s so much that depends on the io module that I am having a really hard time hunting down lib3to2 bugs that exist beyond throwing basic SyntaxErrors.  Also, I’m sure there might be a few bugs in there that are being masked by failures resulting in comparing strings like ‘test output’ with u’test output’, which aren’t technically bugs with lib3to2, but rather with the level of specificity given by the testing code.

So as a result, I’m going to again spend the rest of the GSoC time again looking for well-tested third-party Python 3 code and trying it out with 3to2, since the Py3k standard library is a little bit too complicated a beast for me.

To that end, here’s what I’ve tested, and the results:

httplib2 from PyPI: All tests passed except for 1 error in iri2uri: iterations over bytes objects return an int value in py3k but a str of length 1 in py2k.  The specific error can be fixed by calling ord() on each value returned.
brain from PyPI: Less than a third of all functional test cases pass.

I have finished the fixer for keyword-only parameters.  This includes things like this:

def spam(arg1, arg2, *, kw_only_arg1, kw_only_arg2): pass

The param list would fix to: (arg1, arg2, **_3to2kwargs), and the body of the function would immediately look for ‘kw_only_arg1′ and ‘kw_only_arg2′ in the _3to2kwargs dict, assign to their respective names, and delete the values from the dict.

Special considerations are made for when the *args and the **kwargs params exist.  Check the test cases for exactly what happens.

Note that this fixer is VERY sensitive to incorrect code, so as always, make sure that code actually works in Python 3 before running it through 3to2.

I’ve managed to get myself back to coding after an unexpectedly long absence.  I built up a lot of inertia over all that time, so I’m glad to finally overcome it.

I did hit a snag related to my laptop’s Ubuntu installation; this results in me not being able to run the 3to2 tests right now.  I’ll just commit what I have (the recent change probably does what I think it does, but it’s disabled by default so that it doesn’t bother anyone) and work through this tomorrow.  Shouldn’t take more than a few minutes when I’m fully awake, but I’ve learned from experience that I shouldn’t try to do significant sysadmin stuff this late at night if it’s not time-critical.