Joe Amenta's Blog

I’ve been pulled away from GSoC so much over the past few weeks that I have actually fallen a little behind.  So, over the next few days, I plan to finish everything that is left.  Starting when I finish one last physics problem (which is, at the latest, midnight Friday) (approximately 24 hours from this post), I plan to dedicate the week-end to nothing but coding and the personal necessities.  My goal is to have the following completed by the end of the week-end:

• done.. fix_imports also fixes the usage of the names imported (at time of writing, it only fixed the import)
• done.. most of fix_imports2, a fixer that deals with multiple modules that were merged into a single module (at the minimum, the “from x import y” component)
• done.. fix_features, a fixer that will warn the user which code cannot be converted syntactically because of new py3k features
• not done.. (time-permitting) fix_print works for real (currently only adds “from __future__ import print_function”, which caps the python2.x version at 2.6 if there is a print)

I really think I can pull this off this week-end.  It will take just the right combination of Mountain Dew and my ADD meds.  And a DND message.

Also, barring catastrophic failure of reaching those goals, that will bring me very close to the end of the project.  The ample amounts of ahead-of-schedule time I had at the beginning have been brought in check by ever-increasingly difficult physics homework and the more-than-occasional free time to spend with Kristina.  By August 10 (suggested pencils-down date), barring acts of $DEITY, I should be prepared to do some more documentation, clean up the code, make sure my test suite is thorough, etc. to be prepared for the August 17 deadline.

Edit:

fix_imports2 should now be done, as far as (from x import y) is concerned.  It requires a heck of a lot more testing than my few test cases, though.  In any case, my weekend goal is now complete! (with the exception of fix_print refactoring, as I did not end up having the extra time to do it)  Final challenges are:

• fix_imports2 needs to work for (import x) and (from x import *) imports.
• The code needs to be more readable (ongoing process).
• Need to remind myself what I needed to do with itertools. Fixed in changeset e343ee635f0b
• Add in 3.1 features to fix_features.
• Test cases, test cases, test cases!!!  If anyone can help with this, I would really appreciate it!  Even just a link to a really comprehensive piece of py3k software to look at would help out tremendously.

Well, this sucks.  I found out, while trying to refactor stdin, that my testing framework does not test fixer interactions.  It started out simply: refactoring “print(‘omg’)”  just turned into “print(u’omg’)” (without adding from __future__ import print_function).  That was partially fixed.  However, this led me to test others, and there are other cases where I get the wrong output.  Fixes are on the way.

Edit: It works fine now; I just have to stay away from self.tree.was_changed.

And now, another milestone.  With set_literal being finished, I have now completed the 3to2 version of all 2to3 fixers that don’t deal with imports.  Now, all that’s left (which is really the hardest part) is the standard library (stdlib).  This will be split into two parts (one is easy, one is hard).

Part 1 (easy) is renaming every library module that was simply renamed in py3k.  This is easy, because all I have to do is change one line, e.g.: “_pickle” will be changed into “cPickle”, everywhere, consistently.

Part 2 (hard) involves modules in py3k that are amalgamations of multiple modules (or parts from multiple modules) in python2.x.  This is hard, because it will involve writing a hell of a lot of filters to determine whether (for example) the correct fix for “http.server” would be “BaseHTTPServer”, “CGIHTTPServer”, or “SimpleHTTPServer”, or some monstrous blend of all of them (would it ever?).

A (complete?) list of what the rest of my project will have to undo can be found in PEP 3108 — Standard Library Reorganization.  (There are many more things listed in this PEP than what I need to deal with: much of it involves things that were completely removed from the language for py3k, so I don’t have to deal with those)

As posted before, I will not yet be tackling new things added in py3k with no backwards-compatible syntactical equivalent.  That will be implemented first with warnings that point to what cannot be fixed (GSoC completion will require at least this by my current standard), then the option to raise the minimum version of python2.x supported, then (selective-case only) implementation of any features backported by third-parties that will require either the presence of that module on the host machine, or will fallback to that module distributed with lib3to2 (will require pythonx.y setup.py install).