Joe Amenta's Blog

I don’t know why, but I’ve been hitting roadblocks all day about fix_metaclass.  I’m not going to commit the trash I have, because I will probably start over completely on it.  I think I’m trying to rely too much on the pattern compiler; I should probably be mimicking 2to3′s fix_metaclass and use the PATTERN to match all classdefs, then call some of my own functions to narrow it down… but I keep trying to mess around with the PATTERN to match a class with a metaclass; I know it’s possible, I just can’t get it to work.  So tonight I’ll start over on it, and hopefully it’ll be done tomorrow.

Python 3.0 added a nice bucketload of new features that simply have no syntactical equivalent in Python 2.5, (See Appendix for a list) (table coming soon!).  Up to this point, I have not touched anything that requires more than a syntactic change.  However, this is an important topic, because developers using 3.0 are encouraged to use all of its new features and syntax, and the goal for any code refactoring tool is (or should be!) to work universally for all cases.  (See reasoning why fix_memoryview is required explicitly, even though not using the fixer would result in a SyntaxError in 2.5/2.6 if a memoryview is encountered)  Thus, I will address the feature issue here.

This blog post applies to all features present in any py3k branch of Python that have no backwards-compatible syntactic equivalent (“BCSE”), or an equivalent representation using syntax that already existed in Python 2.5 without an unreasonable amount of helper code, e.g., that recreates data types that are built-in to py3k (said features herein named “New Features”).   The end-game goal for lib3to2 is to make all code using any built-in functionality of all py3k branches work on python2.5 with minimal (or no) user intervention when 3to2-ified.  Obviously, this is a very difficult goal to achieve, given that new features added in py3k are always being added, and there is already enough to occupy a huge chunk of my time, projecting out towards Christmas 2009.  For the purposes of having a pretty good tool by the end of GSoC 2009, my goal is to make 3to2 refactor syntax where there is a BCSE, and (case-by-case) raise warnings that tell the user their options.  An example warning would look like this (using “memoryview” as the sample case):

Warning: memoryview does not have an exact syntactic equivalent in Python 2.5, and the fixer for memoryview -> buffer is disabled by default.  You may either enable the “memoryview” fixer and accept the consequences of using buffer (which only allows a subset of the operations available to memoryview), manually refactor your code to avoid using memoryview, or ignore this message.  Ignoring this message limits your code to work on the following Python version (at minimum): Python 2.7

That’s obviously rather verbose.  A short version might look like:

Warning: memoryview encountered, fixer disabled.  Enable fixer, manually refactor, or ignore (2.7 minimum required).

Comments on this blog post are welcome: I welcome and request alternatives to this structure.

Appendix

Features, comma-delimited, in the format feature(minimum python version supported): bin (2.6), str.format (2.6), multiprocessing (“2.5″)*, io (2.6), abc (2.6), numbers  (2.6), memoryview (2.7).  3.1 New Features not included.

* pypi has backports for the multiprocessing package back to 2.4, but it is not built in, so the syntax would require some additional fluff to make it work, which is a future goal, not for now.

I’m pretty pumped.  I just “finished” the print function fixer for 3to2.  I put “finished” in quotes (and only “working” instead of “complete” under its status text on the previous post) because it’s a sort of a placeholder right now: all it does is add “from __future__ import print_function” at the top of a refactored module that uses print(), so this will only successfully give you code certified for 2.6 and up, not the 2.5 branch as originally promised.  But still, this is a huge leap: now, most (if not all) of my own little py3k modules are working unmodified after running them through 3to2!  This is a good time for me to stop and explain how to get 3to2 working on your computer for those of you that do not know:

Find an SVN client (TortoiseSVN for Windows, subversion for others) , and check out (into separate folders) both http://svn.python.org/projects/sandbox/trunk/2to3 and http://svn.python.org/projects/python/trunk/.  Then, copy the folder (the entire folder!) lib2to3 (from the root of the 2to3 folder) into the Lib/ folder (from the root of the other folder).  Then, look up a great tutorial for how to build Python on your system (sorry, I’m not here to tell you how to do that… just how to get my stuff to work).

Once you’ve built your own custom Python build (with the most up-to-date version of lib2to3, that this project relies on [and I contributed a line or two of code here and there to make it work for my purposes]), then it’s time to get 3to2.  Find a Mercurial client (TortoiseHG for Windows, mercurial for others), and clone (into yet another separate folder) http://bitbucket.org/amentajo/lib3to2/.  Run “python2.7 setup.py build” (python2.7 being the path to your python2.7 executable you built earlier on), and then “python2.7 setup.py install” (you need administrator / root privileges for the latter command).  This will put “3to2″ on your sys path, and you will be able to simply run 3to2 like this:

3to2 /path/to/my_file.py # obviously, /path/to/my_file.py stands for a (relative or absolute) path to an existing py3k file that you need to refactor. (This will also work on an entire directory that contains .py files)

This will show you (in unified diff format) the changes that 3to2 will make to your file.  You can either redirect the stdout to a .patch file then apply that patch the normal way, or run 3to2 like this:

3to2 -w /path/to/my_file.py

with the -w command, and it will back up your original files, then fix them with the fixer suite.

One note: there is one fixer so far that must be run explicitly.  That is memoryview, which changes all memoryview() calls into buffer() calls.  To enable this, you must run 3to2 with the “-f memoryview” option.  There are two reasons for this.  The first is that buffer objects do not support all of the same methods that memoryview objects do, so if this fixer were enabled by default, it would fix code perfectly for some cases, and not for others.  The second reason is that memoryview was backported into python 2.7, so if the user wishes to run their fixed code on a python2.7 platform, they can benefit from keeping their memoryview() calls alone.

(Of course, if you’re using memoryview() or buffer(), you probably already know or can easily figure out that you can use -xmemoryview to turn that fixer off )

—

Having reached this milestone, I look both back and ahead: it feels like I’m almost all the way to a finished product.  What’s left as a high priority at this point is exception stuff, various import stuff, dict methods, the stdlib, magical metaclass mayhem, sys.intern, numliterals (0o0777, 0b011101, et al.), reduce, and set literals… plus with_statements, annotations, floor division, and some other stuff I’m too tired to think of, but was added in 3.0 that had no 2to3 fix.  Then, fix the print fixer to work for 2.5, i.e., actually perform logic and move stuff out of the function into a true print statement.  After that, I want to rework the stuff that was changed from lists into iterators in py3k to remove a direct list() call, so that list(range(…)) will (more) correctly be fixed into range(…) rather than list(xrange(…)), e.g.  After that’s done, I feel like more robust test suites are in order (for both 3to2 and 2to3).

OK, writing it all down like that in list form makes it look longer than it really is.  This doesn’t seem like it will take until the end of the Google Summer of Code program, at least not to me.  The stdlib stuff will take a good week or so, most of which will involve me learning what the modules do so that I can have a grasp on how they will be used and how they changed from 2.x to 3.x.  The print function -> statement is probably two or three days.  Exception stuff will probably take me another week, for the same reason that the stdlib stuff will: I have never used any more extended form of exception handling than “raise Exception()”, and “except Exception:”… so I don’t have much experience to go on.

So it looks like I just may get to the “Things to do if I finish early” part of my proposal.  That is, assuming I keep up the pace.

Here is a table consisting of the 2to3 fixes, whether they need a corresponding 3to2 fix, and their status (3to2 fix complete / doesn’t need a 3to2 fix, working (could do better, but the syntax is correct), incomplete (does not fix the syntax yet, but that fix is in progress), or not started (I haven’t touched it yet)):

#Out of fear of creating a monster post, the new features in 3.0 will get their own post above.

This particular post will be the post to watch for the summary of the status of lib3to2′s progress.