Meet the Team: Brett Cannon

This post is part of the "Meet the Team" series of posts, which is meant to give a brief introduction to the Python core development team.

Name:	Brett Cannon
Location:	San Francisco, CA, USA
Home Page:	https://profiles.google.com/bcannon
Blog:	http://sayspy.blogspot.com

How long have you been using Python?

Since the fall of 2000

How long have you been a core committer?

Since April 2003 (shortly after PyCon 2003).

How did you get started as a core developer? Do you remember your first commit?

I became a core developer thanks to incessantly bugging people to commit patches for me (a trick that doesn't quite work as well as it used to; perk of getting in before Python's popularity spikein 2003/2004). Starting in August 2002 I revitalized the Python-Dev Summaries (which lasted for about 2.5 years). While writing the Summaries I would fairly regularly pick up on little issues that needed fixing. Since I was already talking on python-dev fairly regularly I simply asked folks to check my patches and commit them for me. One day Guido just asked why I didn't commit myself, I said I didn't have commit rights, and then he more or less said "you do now".

As for my first commit (changeset 28686), it was fixing some string escapement in time.strptime() (which happens to be my first contribution to Python itself).

Which parts of Python are you working on now?

I typically focus on the import machinery and making the Python language work well across all VMs.

What do you do with Python when you aren't doing core development work?

I managed to use Python a little bit in my PhD thesis by implementing some server-side stuff in Python. Otherwise all of my personal projects use Python as much as possible. And my future job at Google is going to be mostly in Python.

What do you do when you aren't programming?

I'm somewhat of a movie junkie with selective bits of TV tossed in (losing my television in the summer of 2000 to a heat wave was one of the best things that ever accidentally happened to me; marrying my wife has been the best thing I did on purpose =). Otherwise I read a lot; mostly magazines and websites, but with some book always under progress.

Meet the Team: Michael Foord

This post is part of the "Meet the Team" series of posts, which is meant to give a brief introduction to the Python core development team.

Name:	Michael Foord
Location:	Northampton UK
Home Page:	http://www.voidspace.org.uk/

How long have you been using Python?

I first started using Python as a hobby in 2002. I started using Python full time for work in 2006. When I started programming with Python it was with a group of guys who wanted to write a program to aggregate information from a Play By Email game. None of us had done any programming for a while and we had just decided on using Smalltalk when someone suggested we try Python. I quickly fell in love with Python.

How long have you been a core committer?

I became a core-committer at PyCon in 2009. It was originally because of my involvement with IronPython.

How did you get started as a core developer? Do you remember your first commit?

During the PyCon 2009 sprints I worked with Gregory Smith, another core developer, to incorporate some improvements to unittest contributed by Google.

Which parts of Python are you working on now?

After the initial work on unittest at the PyCon sprint I took on fixing other issues and making improvements to unittest, which was without a maintainer. I became the maintainer of unittest but also contribute to other parts of the standard library.

I'm involved in supporting Python in various other minor ways, such as looking after Planet Python, being a PSF member, helping out on the python.org webmaster alias and so on.

What do you do with Python when you aren't doing core development work?

For my day job I do web development for Canonical. I work on some of the web services infrastructure around the Canonical websites and also some of the services that integrate with Ubuntu itself. That's good fun and its a great team.

In my spare time I work on projects like unittest2 (a backport of the improvements of unittest for other platforms), mock (a testing library that provides mock objects and support for monkey patching in tests) and a whole bunch of other smaller stuff.

I'd like to write more, but having devoted the best part of two years to writing IronPython in Action I doubt I'll take on any large writing projects soon.

What do you do when you aren't programming?

I'm very involved in a church in Northampton (UK), which takes a lot of my time and I help with administration for a charity we run. This is one reason why working for Canonical is good - I can work from home and having put my roots down here I won't move anywhere else (I certainly don't stay for the weather). Needless to say there isn't much Python programming happening in Northampton. My first full time programming gig was with an amazing team in London, which was a two hour door to door commute each way. I managed four years of that, and really enjoyed the job, but having escaped the commute I'm not likely to ever go back.

I also enjoy gaming on the XBox. Unfortunately if I find a game I like I can get sucked into it for weeks so I have to be careful. I've avoided world of warcraft and eve online for this reason... I also organise a monthly geek meet in Northampton. There aren't enough Python programmers for a Python user group but we have a good collection of geeks of all sorts. We normally just get together in a pub and chew the fat or show off our latest gadgets.

A Python Launcher For Windows

Mark Hammond (author of pywin32 and long-time supporter of Python on Windows) has written PEP 397, which describes a new launcher for Python on Windows. Vinay Sanjip (author of the standard library logging module) has recently created an implementation of the launcher, downloadable from https://bitbucket.org/vinay.sajip/pylauncher/downloads

The launcher allows Python scripts (.py and .pyw files) on Windows to specify the version of Python which should be used, allowing simultaneous use of Python 2 and 3.

Windows users should consider downloading the launcher and testing it, to help the Python developers iron out any remaining issues. The launcher is packaged as a standalone application, and will support currently available versions of Python. The intention is that once the launcher is finalised, it will be included as part of Python 3.3 (although it will remain available as a standalone download for users of earlier versions).

Two versions of the launcher are available - launcher.msi which installs in the Program Files directory, and launchsys.msi which installs in Windows' System32 directory. (There are also 64-bit versions for 64-bit versions of Windows).

Some Details About the Launcher

The full specification of the behaviour of the launcher is given in PEP 397. To summarise the basic principles:

• The launcher supplies two executables - py.exe (the console version) and pyw.exe (the GUI version).
• The launcher is registered as the handler for .py (console) and .pyw (GUI) file extensions.
• When executing a script, the launcher looks for a Unix-style #! (shebang) line in the script. It recognises executable names python (system default python), python2 (default Python 2 release) and python3 (default Python 3 release). The precise details can easily be customised on a per-user or per-machine basis.
• When used standalone, the py.exe command launches the Python interactive interpreter. Command line switches are supported, so that py -2 launches Python 2, py -3 launches Python 3, and py launches the default version.

Simple Usage Instructions

When it is installed, the launcher associates itself with .py and .pyw scripts. Unless you do anything else, scripts will be run using the default Python on the machine, so you will see no change. One thing you might like to do, if you use the console a lot, is to add .py to your PATHEXT variable so that scripts don't get executed in a separate console.

To specify that a script must use Python 2, simply add:

#!/usr/bin/env python2

as the first line of the script. (This is a Unix-compatible form. If you don't need Unix compatibility, #!python2 will do).

If on the other hand, you want to specify that a script must use Python 3, add:

#!/usr/bin/env python3

as the first line.

You can also start the Python interpreter using any of the following commands:

# Default version of Python
py
# Python 2
py -2
# Python 3
py -3

For this to work, the py.exe executable must be on your path. This is automatic with the launchsys version of the installer, but the install directory (C:\Program Files\Python Launcher) must be added manually to PATH with launcher.msi.

Further Reading

The following email threads on python-dev cover some of the key discussions:

• Mark's initial announcement of the draft PEP: http://mail.python.org/pipermail/python-dev/2011-March/109509.html
• The second draft of the PEP: http://mail.python.org/pipermail/python-dev/2011-March/109786.html
• Vinay's initial query about a C implementation of the launcher: http://mail.python.org/pipermail/python-dev/2011-June/112145.html
• Vinay's announcement of his C implementation: http://mail.python.org/pipermail/python-dev/2011-July/112184.html
• Vinay's call for testers: http://mail.python.org/pipermail/python-dev/2011-July/112251.html

CPython 3.2.1 Released

On behalf of the python-dev team, release manager Georg Brandl has announced the final release of CPython 3.2.1. Windows installers and tarballs are available as of July 10, so please consider upgrading to this release.

The What's New document lists all of the new features in 3.2, and the Misc/NEWS file in the source lists each bug fixed.

If you find any issues with this release or any other, please report them to http://bugs.python.org/.

3.2.1 Release Candidate 2 Released

Following up a big month of releases in June, the second release candidate of the 3.2.1 line is now ready. Since the first release candidate on May 15, over 40 issues have been fixed. We encourage everyone to test their projects with this candidate to get one last look before the final release of 3.2.1.

What's fixed?

I/O

#1195 spent a few years witout a fix, but a simple addition to clear errors before calling fgets solves the problem of interrupting sys.stdin.read() with CTRL-D inside of input(). The io system saw a cleanup in #12175 with the readall method with None being the return value on a read() which returns None, and a ValueError is now raised when a file can't be opened.

Although this isn't new for RC2, #11272 is an important 3.2.1 fix to input() on Windows - the fixing of a trailing \r. The issue has been reported many times over and affects a many people (distutils upload command anyone?), so hopefully 3.2.1 does the trick for you.

Windows

3.2.0 brought a new feature for Windows: os.symlink support. With that feature came #12084, os.stat was improperly evaluating Windows symlinks, so the inner workings of the various stat functions were corrected.

A user noticed that os.path.isdir was slow, and the fact that it relied on os.stat contributed to that, especially when evaluating symlinks (which are generally twice as slow as regular files). While os.path.isdir isn't anyone's performance bottleneck, it's called numerous times on interpreter startup so changing it in #11583 to use GetFileAttributes gives a tiny speedup to build on.

subprocess

Creating a Popen object with unexpected arguments was causing an AttributeError, but that was reported in #12085 and was fixed by the reporter. Due to a change in 3.2.0, Popen wasn't correctly handling empty environment variables, specifically the env argument. #12383 was created for the issue and was promptly fixed.

...and more!

For a full list of changes through 3.2.1 RC2, check out the change log and download it now!

As always, please report any issues you find to http://bugs.python.org. We appreciate your help in making great Python releases.

June Releases - 2.6.7, 2.7.2, 3.1.4

June is a big month for Python releases, with an update coming out of all active branches.

2.6.7

A new source-only release of Python 2.6.7 is available, providing fixes to three security issues. Now that the 2.6 line is in security-mode, these releases will happen on an as-needed basis until October 2013 in source-only form. If you require binary installers, you should consider an upgrade to 2.7 or 3.2.

2.6.7 is the first release to contain a fix to the previously covered urllib vulnerability. Additionally, an smtpd DoS vulnerability (Issue #9129) and SimpleHTTPServer.list_directory XSS vulnerability (Issue #11442) are fixed.

2.7.2

The last minor version of the 2.x line, 2.7, received over 150 bug fixes since 2.7.1 in November 2010. 2.7.2 source and binary installers are available as of June 12, which include the security fixes mentioned in 2.6.7.

A number of crashes are fixed: a situation when Python incorrectly used non-Python managed memory while it was being modified by another thread, when deleting __abstractmethods__ from a class, accessing a memory-mapped file past its length, and several others.

A fix to getpass corrects a regression in regards to CTRL-C and CTRL-Z handling. multiprocessing received a number of fixes, including treating Windows services like frozen executables and a correction to a race condition when terminating multiprocessing.Pool workers. mmap was fixed to work with file sizes and offsets larger than 4 GB even on 32-bit builds, and a TypeError is now raised rather than segfaulting when trying to write to a non-writeable map.

For a full list of changes, see the 2.7.2 news file.

3.1.4

3.1.4 is the last bug-fix release of the 3.1.x line, sending 3.1 into security-mode as the 3.2 line carries on. 3.1.4 contains over 100 bug fixes since the 3.1.3 release in November 2010. As with 2.7.2, binary installers are available as of June 12, and 3.1.4 is the first 3.x release to contain the security fixes listed in 2.6.7.

3.1.4 corrects some problems with __dir__ lookups on objects, dates past 2038 in the Windows implementation of os.stat and os.utime, and a number of 64-bit cleanups. The io library saw a number of changes in returning None when nothing was read and raising appropriate exceptions in other spots. ctypes callback arguments were fixed on 64-bit Windows and a crash was also remedied.

For a full list of changes, see the 3.1.4 news file.

3.2.1

3.2.1 is currently in the release candidate phase, with one round already completed and a second release candidate expected soon. We would greatly appreciate 3.2 users trying out the release candidates to ensure we cover any issues you may be seeing. If you have any bugs to report, please file them on bugs.python.org.

New faulthandler module in Python 3.3 helps debugging

When a user reports that your program crashes or hangs, sometimes you can only help to try and collect more information and outline a scenario to reproduce the situation. Even with a reliable user scenario, as a developer you are often unable to reproduce the situation due to environment differences, e.g., operating system and compiler. If you are lucky, the user will be able to install debug tools, but most of time you will have to wait until another person is able to obtain more information from the same situation.

Fatal Errors

A new module introduced in Python 3.3 should help this problem: faulthandler. faulthandler provides the ability to dump the Python traceback on a fatal error such as a segmentation fault, division by zero, abort, or bus error. You can enable it inside your application using faulthandler.enable(), by providing the -X faulthandler option to the Python executable, or with the PYTHONFAULTHANDLER=1 environment variable. Output example:

Fatal Python error: Segmentation fault

Current thread 0x00007f7babc6b700:
  File "Lib/test/crashers/gc_inspection.py", line 29 in g
  File "Lib/test/crashers/gc_inspection.py", line 32 in <module>
Segmentation fault

Timeout

faulthandler can also dump the traceback after a timeout using faulthandler.dump_tracebacks_later(timeout). Call it again to restart the timer or call faulthandler.cancel_dump_tracebacks_later() to stop the timer. Output example:

Timeout (0:01:00)!
Current thread 0x00007f987d459700:
  File "Lib/test/crashers/infinite_loop_re.py", line 20 in <module>

Use the repeat=True option to dump the traceback each timeout seconds, or exit=True to immediatly exit the program in an unsafe fashion, e.g. don't flush files.

User Signal

If you have access to the host on which the program is running, you can use faulthandler.register(signal) to install a signal handler to dump the traceback when signal is received. On UNIX, for example, you can use the SIGUSR1 signal: kill -USR1 <pid> will dump the current traceback. This feature is not available on Windows. Output example:

Current thread 0x00007fdc3da74700:
  File "Lib/test/crashers/infinite_loop_re.py", line 19 in <module>

Another possibility is to explicitly call faulthandler.dump_traceback() in your program.

Security Issues and the Output File

faulthandler is disabled by default for security reasons, mainly because it stores the file descriptor of sys.stderr and writes the tracebacks into this file descriptor. If sys.stderr is closed and the file descriptor is reused, the file descriptor may be a socket, a pipe, a critical file or something else. By default, faulthandler writes the tracebacks to sys.stderr, but you can specify another file. For more information, see the faulthandler documentation.

Third-party Module for Older Python Versions

faulthandler is also maintained as a third-party module for Python 2.5 through 3.2 on PyPI. The major difference between the Python 3.3 module and the third-party module is the implementation of dump_tracebacks_later(): Python 3.3 uses a thread with a timeout on a lock, whereas the third party uses SIGALRM and alarm().

The lock timeout, which is a new feature of Python 3.3, has a microsecond resolution. The alarm() timer used on older versions has a resolution of one second, and the SIGALRM signal may interrupt the current system call which will fail with an EINTR error.

Early Success

The new faulthandler module has already helped with tracking down race conditions in our buildbots. We hope that it will also help you in your programs.