Greg Hewgill (ghewgill) wrote,
Greg Hewgill

python 2 to 3 upgrade and exception handling

I've just (partially) upgraded Psil to Python 3. I had originally been developing it in Python 2, but there are a few particular things about Python 3 that make it a better choice. There's no particular reason it needs to run on Python 2, so I decided to not develop two parallel versions, and not try to run the same code in both Python 2 and 3, but just upgrade wholesale to Python 3 and not look back.

There were in fact only a few simple changes that were required. The most obvious is the print() function, but I also ran into:

  • The result of map() is an iterator not a list (this affected some unit test code)
  • Renaming of raw_input() to input()
  • Use of next(it) instead of (and my use of a local variable called next)
  • Need to use f(*a) instead of apply(f, a)
  • Lack of callable(x) (use hasattr(x, "__call__"))
  • Exception syntax: except Exception as e instead of except Exception, e
  • reduce() now lives in the functools module
  • Some test code needed to handle Unicode strings properly
  • A difference in default exception handling (more on that below)

So I certainly got a pretty good coverage of the major differences.

The problem I had with exception handling was related to my unorthodox method of handling tail recursion. When I ran a program that used a lot of tail recursion, the memory usage immediately and quickly went through the roof! Clearly there was a memory leak, but Python is generally supposed to handle that for me with its garbage collector. The clue to solving this lay in an obscure warning in the documentation for sys.exc_info:

Warning: Assigning the traceback return value to a local variable in a function that is handling an exception will cause a circular reference. This will prevent anything referenced by a local variable in the same function or by the traceback from being garbage collected.

I had read that warning, but I wasn't using the traceback of sys.exc_info at all so I thought that shouldn't be a problem. However, Python 3 now automatically includes a __traceback__ attribute of every exception (see PEP 3134). Due to the way I was calling a function referenced within the exception object itself, the presence of the traceback was creating a huge chain of unfreeable function and exception references.

Fortunately, there was a simple solution:

        except TailCall as t:
            a = t
            a.__traceback__ = None

Setting the __traceback__ attribute explicitly to None releases the reference to previous stack frames and my code no longer leaks memory.

On the recommendation of the python-dev mailing list, I filed a documentation bug to clarify the warning quoted above.

Finally, I said at first that this was a partial upgrade, because I haven't even addressed the compiler part of Psil (that compiles Psil code to native Python). The modules and interfaces that I was using previously are either gone or changed in Python 3, so a slightly different approach is needed. More on that in a later post.


  • 2012 in review

    2012 has been fairly quiet. Maybe it just seems that way because I haven't actually written anything new in this blog since last year's annual…

  • 2011 in review

    2011 was the year that was pretty much defined by the earthquakes in Christchurch. Everything else seems insignificant, but it turns out I did…

  • 2010 in review

    2010 was a pretty interesting year. Went to Hawaii for three weeks with Amy. Didn't get to see flowing lava, but did get to swim with turtles!…

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded