So I wrote a fishing e-mail just now to python-dev asking for feedback on the idea of including lib3to2 in the Python 3.2 standard library. I’m going to write the PEP regardless (unless the right amount of people say that there is no possible way that lib3to2 will ever make it into the standard library); I just want to hear initial thoughts about the idea in general.
Stabilization of lib3to2 for inclusion in the standard library will involve:
- (Starting and) finishing the rewrite of fix_imports2.py
- Actively hunting down as much native (i.e., not simply 2to3′d) Python3 code as possible, then analyzing how 3to2 translates it
- Stress testing with crafted Python 3 code that should expose hidden bugs that may actually be relevant to real-world use cases (it’s rather easy to mess with 3to2 by screwing around with builtins and name binding… this is not what I’m referring to here.)
And, of course, moving to Py3k. I have entertained the idea of proposing inclusion in the Python 2 standard library, and I disagree with the idea; here’s why.
It’s possible that Python 2 does not get a new release when a new version of Python 3 is released (or ever, but that’s another topic), and thus backwards-incompatible features that can potentially be added to Python 3 do not get accompanying warnings in the Python 2 version of lib3to2, leading to code that refactors without major warnings in Python 2′s lib3to2 but fails at runtime.
There’s also a really good reason to include lib3to2 in the Python 3 standard library exclusively: if lib3to2 is provided with Python 3.x and warns about a complete list of backwards-incompatible Python 3 features that version 3.x includes, then lib3to2 is future-proof. New features in Python 3.y will not affect the correctness of lib3to2 provided with Python 3.x, and 3.y has its own lib3to2 to refactor code, again assuming that lib3to2 warns about a complete list of backwards-incompatible features.
This rationale provides the nail-in-the-coffin for not putting lib3to2 in the Python 2 standard library: when 3.y is released, the Python 2.x version of lib3to2 will become obsolete. Most likely, a user of Python 2.x will find a barrier to upgrading to Python 2.y even if the releases are synchronized indefinitely. That user will, however, need lib3to2 to warn about the 3.y features incompatible with their version of Python. The solution: use the latest lib3to2 for Python 2 from PyPI. This is a lot of trouble to deal with if there is a version of lib3to2 provided in the standard library, so for simplicity’s sake, don’t provide it with Python 2.
Sorry for the long-winded explanation.
OK, after taking a while to think about this and the feedback I got from my e-mail, I think it’s a bad idea at this point in time to push for standard library inclusion. Providing it with Python and freezing the code until the next Python release (apart from minor bugfixes) doesn’t sound like the right move, considering the fact that even if I were to finish everything that I can think of that needs finishing today and released a beta version, I’m not comfortable handing it off so soon to python-dev, basically saying “everything that this is intended to do works to the extent that it is intended to work.” My main issue is this: when writing lib3to2, I found a couple of issues with lib2to3 that required modifying parts of lib2to3 to get lib3to2 to work. Because lib2to3 is in the standard library, people who use 3to2 have only two options: either run it on a version of Python released after 2.6 / 3.0 or manually replace lib2to3 with the one in the sandbox.
3.2 is just too early of a target. I think I’m going to see if I can get a GSoC 2010 project for lib3to2 to finish up the outstanding rewrites, thoroughly test, and finally release a beta version. I can’t predict how it will go after I put a beta out there, but I know I’m going to be more confident in recommending it than in its current state. Right now, the blemish of fix_imports2 is annoying me to no end, and I’m not even sure what proportion of code is impacted by its status as “not working, do not use”.