So I wrote a fishing e-mail just now to python-dev asking for feedback on the idea of including lib3to2 in the Python 3.2 standard library. I’m going to write the PEP regardless (unless the right amount of people say that there is no possible way that lib3to2 will ever make it into the standard library); I just want to hear initial thoughts about the idea in general.
Stabilization of lib3to2 for inclusion in the standard library will involve:
- (Starting and) finishing the rewrite of fix_imports2.py
- Actively hunting down as much native (i.e., not simply 2to3′d) Python3 code as possible, then analyzing how 3to2 translates it
- Stress testing with crafted Python 3 code that should expose hidden bugs that may actually be relevant to real-world use cases (it’s rather easy to mess with 3to2 by screwing around with builtins and name binding… this is not what I’m referring to here.)
And, of course, moving to Py3k. I have entertained the idea of proposing inclusion in the Python 2 standard library, and I disagree with the idea; here’s why.
It’s possible that Python 2 does not get a new release when a new version of Python 3 is released (or ever, but that’s another topic), and thus backwards-incompatible features that can potentially be added to Python 3 do not get accompanying warnings in the Python 2 version of lib3to2, leading to code that refactors without major warnings in Python 2’s lib3to2 but fails at runtime.
There’s also a really good reason to include lib3to2 in the Python 3 standard library exclusively: if lib3to2 is provided with Python 3.x and warns about a complete list of backwards-incompatible Python 3 features that version 3.x includes, then lib3to2 is future-proof. New features in Python 3.y will not affect the correctness of lib3to2 provided with Python 3.x, and 3.y has its own lib3to2 to refactor code, again assuming that lib3to2 warns about a complete list of backwards-incompatible features.
This rationale provides the nail-in-the-coffin for not putting lib3to2 in the Python 2 standard library: when 3.y is released, the Python 2.x version of lib3to2 will become obsolete. Most likely, a user of Python 2.x will find a barrier to upgrading to Python 2.y even if the releases are synchronized indefinitely. That user will, however, need lib3to2 to warn about the 3.y features incompatible with their version of Python. The solution: use the latest lib3to2 for Python 2 from PyPI. This is a lot of trouble to deal with if there is a version of lib3to2 provided in the standard library, so for simplicity’s sake, don’t provide it with Python 2.
Sorry for the long-winded explanation.
I’m here in the Hyatt Regency in Downtown Atlanta for PyCon 2010. I’m going to be doing a poster session on Sunday… attached is the first draft of my poster slides (No images added for prettiness yet)
3to2 poster
Edit: Finished it: 3to2-poster
PyCon is just a couple of weeks away, and I haven’t gotten around to designing a poster for my session yet. I’m open to suggestions for what people would like to see (though I won’t hold out for them, of course).
I also haven’t written any 3to2 code since my last update — I haven’t heard anything from anyone using it yet. I’m still going to stick with the first beta release being when fix_imports2 works, which may be a (legitimate) reason that people aren’t using it enough to have feedback to give about it. I’m not coding as actively right now because of other priorities — but if I get feedback from users, I may be more inclined to prioritize 3to2 higher than it currently is.
I have decided to keep with the idea of having a fix_imports2 separate from fix_imports, so after finishing the usage parts of fix_imports_test, I have declared it finished and ready for general usage. It has replaced the old fix_imports.
fix_imports2 will be another rewrite, so I will go ahead and start on that eventually.
fix_imports_test is almost done – the actual imports are mostly successful, with just a few annoying bugs in how it handles things with commas, like:
import http.client, winreg
and
from http import client, spam, eggs
The next step is getting the actual usage of those imported modules to work.
The next step after that, of course, is the fix_imports2 stuff. I think that this will still be split out into a separate module – otherwise, special cases would pollute the heck out of the code for the other one, and it would make debugging a nightmare.
Finals week has started, and I already only have one more exam (on Friday).
I have done a little bit of work on the new fix_imports replacement, and I think that it will turn out to be much cleaner than the sloppy ones in the trunk right now.
Other updates:
I will be doing a poster session at PyCon 2010 about 3to2, so stay tuned for updates about that.
So I’ve gotten the first part of the rewrite for fixing code with imports working… (from) (http) (import) (client) goes to (import) (httplib) (as) (client)
I wrote this particular one for playing around with one of the complicated cases using the new pattern builder… I’ll toy around with it some more for the simple cases.
I’ve committed the first bit of a rewrite for fix_imports and fix_imports2. Short description: the plan is to use the 2to3 pattern matcher instead of a custom hack to improve speed, correctness, and simplicity.
Long description:
fix_imports2 literally scans the tree before anything happens, collecting relevant NAME leaves into a dict mapping top-level modules (e.g., urllib) to them.
fix_imports is another great example of bad code if anyone was looking for one.
I haven’t had the large chunk of time that it would take to do a rewrite until now. Hoping I can get it done by the end of Thanksgiving break, though that’s a little optimistic.
I’ve gotten the Extended Iterable Unpacking fixer to work in suites.
Also, I’m going to fill out an application to do a poster about 3to2 at PyCon 2010… here’s hoping!
More nothing, because my spending so much time on Statistics caused me to neglect other endeavours, so I’ve had to catch up on those things. I’m so busy…