Introducing Electronic Contributor Agreements

We're happy to announce the new way to file a contributor agreement: on the web at http://www.python.org/psf/contrib/contrib-form/.

Through the use of Adobe's EchoSign, we got rid of the old hand-written, print out, scan or photograph, then fax or email of your form. It was a hassle for our contributors, and a hassle for our administrators. Faxes fail, mail gets lost, and sometimes pictures or scans turn out poorly. It was time to find a more user-friendly solution, and the Foundation is happy to finally offer this electronic form.


The new form is easy to fill out right on the site, guiding you through each of the required fields such as your name, bug tracker ID, address, and initial license. If you're signing the form on behalf of an organization, there's a check box to specify this, and then you are asked near the bottom to state your title in the organization. Lastly, your signature is either generated from your typed name, or you can draw your own or upload a signature file of your own.

Once you submit the form, you'll receive an email from echosign.com to verify the email address you entered. Once you click to confirm your address, the form will be emailed to the PSF and will be recorded.


We require all contributors to CPython to have a signed form, and we hope this makes it easier for potential contributors to join up and help make Python better. It's available just in time for PyCon and the CPython sprint that will be occurring March 18 through 21 in Santa Clara, California. Join us at the sprint, sign your contributor form, and help us fix some bugs or add some features!

Announcing defusedxml, Fixes for XML Security Issues

The following post was created on behalf of CPython contributor Christian Heimes using a subset of details found here.

Christian Heimes announces the release of his defusedxml and defusedexpat packages to address XML-related security issues which were reported to security@python.org over the last several months. Throughout the development of the patches, the security team has coordinated with other open source projects in order to make this announcement at 1500 UTC on Tuesday February 19.

Details will follow once releases of CPython have been organized.

Note: this post will be updated with more details as they switch from being private to publicly available, including links to the public bug reports on http://bugs.python.org.

defusedxml on PyPI: https://pypi.python.org/pypi/defusedxml
defusedexpat on PyPI: https://pypi.python.org/pypi/defusedexpat
"XML vulnerabilities" on bug tracker: http://bugs.python.org/issue17239

Synopsis

The results of an attack on a vulnerable XML library can be fairly dramatic. With just a few hundred Bytes of XML data an attacker can occupy several Gigabytes of memory within seconds. An attacker can also keep CPUs busy for a long time with a small to medium size request. Under some circumstances it is even possible to access local files on your server, to circumvent a firewall, or to abuse services to rebound attacks to third parties.

The attacks use and abuse less common features of XML and its parsers. The majority of developers are unacquainted with features such as processing instructions and entity expansions that XML inherited from SGML. At best they know about <!DOCTYPE> from experience with HTML but they are not aware that a document type definition (DTD) can generate an HTTP request or load a file from the file system.

None of the issues is new. They have been known for a long time. Billion laughs was first reported in 2003. Nevertheless some XML libraries and applications are still vulnerable and even heavy users of XML are surprised by these features. It's hard to say whom to blame for the situation. It's too short sighted to shift all blame on XML parsers and XML libraries for using insecure default settings. After all they properly implement XML specifications. Application developers must not rely that a library is always configured for security and potential harmful data by default.

Attack vectors

billion laughs / exponential entity expansion

The Billion Laughs attack -- also known as exponential entity expansion -- uses multiple levels of nested entities. The original example uses 9 levels of 10 expansions in each level to expand the string lol to a string of 3 * 10 ⁹ bytes, hence the name "billion laughs". The resulting string occupies 3 GB (2.79 GiB) of memory; intermediate strings require additional memory. Because most parsers don't cache the intermediate step for every expansion it is repeated over and over again. It increases the CPU load even more.

An XML document of just a few hundred bytes can disrupt all services on a machine within seconds.

Example XML:

<!DOCTYPE xmlbomb [
<!ENTITY a "1234567890" >
<!ENTITY b "&a;&a;&a;&a;&a;&a;&a;&a;">
<!ENTITY c "&b;&b;&b;&b;&b;&b;&b;&b;">
<!ENTITY d "&c;&c;&c;&c;&c;&c;&c;&c;">
]>
<bomb>&d;</bomb>

quadratic blowup entity expansion

A quadratic blowup attack is similar to a Billion Laughs attack; it abuses entity expansion, too. Instead of nested entities it repeats one large entity with a couple of ten thousand chars over and over again. The attack isn't as efficient as the exponential case but it avoids triggering countermeasures of parsers against heavily nested entities. Some parsers limit the depth and breadth of a single entity but not the total amount of expanded text throughout an entire XML document.

A medium-sized XML document with a couple of hundred kilobytes can require a couple of hundred MB to several GB of memory. When the attack is combined with some level of nested expansion an attacker is able to achieve a higher ratio of success.

<!DOCTYPE bomb [
<!ENTITY a "xxxxxxx... a couple of ten thousand chars">
]>
<bomb>&a;&a;&a;... repeat</bomb>

external entity expansion (remote)

Entity declarations can contain more than just text for replacement. They can also point to external resources by public identifiers or system identifiers. System identifiers are standard URIs. When the URI is a URL (e.g. a http:// locator) some parsers download the resource from the remote location and embed them into the XML document verbatim.

Simple example of a parsed external entity:

<!DOCTYPE external [
<!ENTITY ee SYSTEM "http://www.python.org/some.xml">
]>
<root>&ee;</root>

The case of parsed external entities works only for valid XML content. The XML standard also supports unparsed external entities with a NData declaration.

External entity expansion opens the door to plenty of exploits. An attacker can abuse a vulnerable XML library and application to rebound and forward network requests with the IP address of the server. It highly depends on the parser and the application what kind of exploit is possible. For example:

• An attacker can circumvent firewalls and gain access to restricted resources as all the requests are made from an internal and trustworthy IP address, not from the outside.
• An attacker can abuse a service to attack, spy on or DoS your servers but also third party services. The attack is disguised with the IP address of the server and the attacker is able to utilize the high bandwidth of a big machine.
• An attacker can exhaust additional resources on the machine, e.g. with requests to a service that doesn't respond or responds with very large files.
• An attacker may gain knowledge, when, how often and from which IP address a XML document is accessed.
• An attacker could send mail from inside your network if the URL handler supports smtp:// URIs.

external entity expansion (local file)

External entities with references to local files are a sub-case of external entity expansion. It's listed as an extra attack because it deserves extra attention. Some XML libraries such as lxml disable network access by default but still allow entity expansion with local file access by default. Local files are either referenced with a file:// URL or by a file path (either relative or absolute).

An attacker may be able to access and download all files that can be read by the application process. This may include critical configuration files, too.

<!DOCTYPE external [
<!ENTITY ee SYSTEM "file:///PATH/TO/simple.xml">
]>
<root>&ee;</root>

Python XML Libraries

vulnerabilities and features

kind	sax	etree	minidom	pulldom	xmlrpc
billion laughs	True	True	True	True	True
quadratic blowup	True	True	True	True	True
external entity expansion (remote)	True	False (3)	False (4)	True	untested
external entity expansion (local file)	True	False (3)	False (4)	True	untested
DTD retrieval	True	False	False	True	untested
gzip bomb	False	False	False	False	True
xpath support (7)	False	False	False	False	False
xsl(t) support (7)	False	False	False	False	False
xinclude support (7)	False	True (6)	False	False	False
C library	expat	expat	expat	expat	expat

1. Lxml is protected against billion laughs attacks and doesn't do network lookups by default.
2. libxml2 and lxml are not directly vulnerable to gzip decompression bombs but they don't protect you against them either.
3. xml.etree doesn't expand entities and raises a ParserError when an entity occurs.
4. minidom doesn't expand entities and simply returns the unexpanded entity verbatim.
5. genshi.input of genshi 0.6 doesn't support entity expansion and raises a ParserError when an entity occurs.
6. Library has (limited) XInclude support but requires an additional step to process inclusion.
7. These are features but they may introduce exploitable holes

How to avoid XML vulnerabilities

Best practices

• Don't allow DTDs
• Don't expand entities
• Don't resolve externals
• Limit parse depth
• Limit total input size
• Limit parse time
• Favor a SAX or iterparse-like parser for potential large data
• Validate and properly quote arguments to XSL transformations and XPath queries
• Don't use XPath expression from untrusted sources
• Don't apply XSL transformations that come untrusted sources

(based on Brad Hill's Attacking XML Security)

Related CVEs

CVE-2013-1664

Unrestricted entity expansion induces DoS vulnerabilities in Python XML libraries (XML bomb)

CVE-2013-1665

External entity expansion in Python XML libraries inflicts potential security flaws and DoS vulnerabilities

Acknowledgements

Brett Cannon (Python Core developer)

review and code cleanup

Antoine Pitrou (Python Core developer)

code review

Aaron Patterson, Ben Murphy and Michael Koziarski (Ruby community)

Many thanks to Aaron, Ben and Michael from the Ruby community for their report and assistance.

Thierry Carrez (OpenStack)

Many thanks to Thierry for his report to the Python Security Response Team on behalf of the OpenStack security team.

Carl Meyer (Django)

Many thanks to Carl for his report to PSRT on behalf of the Django security team.

Daniel Veillard (libxml2)

Many thanks to Daniel for his insight and assistance with libxml2.

semantics GmbH (http://www.semantics.de/)

Many thanks to my employer semantics for letting me work on the issue during working hours as part of semantics's open source initiative.

PandaBoard, Raspberry Pi coming to Buildbot fleet

Thanks to the Python Software Foundation, a PandaBoard arrived on Trent Nelson’s desk just in time for the holidays! Santa dropped off the present for python-dev this morning, and there’s a Raspberry Pi not far behind it.

On Raymond Hettinger’s recent thread about the memory layout of dictionaries, Barry Warsaw and Christian Heimes shared concerns about how things might look on ARM devices. Christian mentioned the Snakebite environment, run by Trent Nelson, but without any ARM machines in the environment, Trent offered to host the boxes if someone donates them.

Based on the thread’s suggestions and the low cost of the devices, the PSF authorized purchase of a PandaBoard ES, featuring a 1.2 GHz ARM Cortex A9, along with several accessories to get it running. The PSF already had a few Raspberry Pi devices on hand, which come with a 700 MHz ARMv6, so one was dispatched to Trent.

Thanks to the PSF for making the purchase, and thanks to Trent for offering to set up the machines and add them to the environment!

New Contributor Experience in Python and other FOSS Communities - A Survey

If you have joined the community of developers contributing to CPython since January 2010, I hope you can find a few spare minutes to participate in a survey being put on by Kevin Carillo. He’s a Ph. D. student at Victoria University of Wellington completing research on new contributors to free and open source projects. Kevin is interested in hearing from everyone from technical to non-technical contributors, whether you had positive or negative experiences.

The survey is available at https://limesurvey.sim.vuw.ac.nz/index.php?sid=65151&lang=en.

Kevin states, “The goal of this research is to understand how a person's experience as a newcomer to a Free/Open Source Software (FOSS) community influences that person's behavior and contributions within that community.” He estimates that the survey will take around 20 minutes to complete.

Throughout the nearly three year period since January 2010, over 40 committers were added and countless others contributed patches, reviews, and bug triage. Since the creation of the Core-Mentors group in March 2011, we’ve seen many first timers come through and have their work committed and released.

We hope that the mentorship group has helped introduce contributors in a positive manner, so we’re looking forward to the results of Kevin’s studies. One of the things Kevin hopes to find is whether or not formal mentorship programs work for introducing contributors. He also looks to find answers to how much community support of newcomers matters, whether formal joining processes involving sponsorship work, and if newcomer specific tasks are the way to go.

If you have the time, please fill out the survey. Python is participating in this survey along with several other groups including Debian, KDE, Gnome, openSUSE, and OpenHatch. Perhaps we can learn a few things and create an even better experience for new contributors!

Python Bug Day this Saturday

This Saturday, you have the opportunity of participating to the Python Bug Day. How would you like to be one of the contributors of Python? If you have ideas for improving parts of the official documentation, the standard library, the language itself, or if you have a patch waiting for a review that you would like to see committed, or if you just want to come and fix an existing bug, it’s your day!

Join us for an effort at closing some Python bugs and feature requests. Get quick feedback on your patches and bugfixes, learn how to submit and examine patches, and have fun chatting with the Python developers and other contributors. You don’t need to know the CPython codebase or process to join, just Python programming knowledge.

If you live in Montreal, come at Caravan to meet fellow hackers and take part in a physical sprint!

Please register to let us know how many people to expect. People from around the world are should join the #python-dev IRC channel to participate in the bug day.

This page contains all the information you need to get set up, see the list of bugs or learn about IRC: http://wiki.python.org/moin/PythonBugDay 

The goal of the bug day is to process bug reports in the Python bug tracker, trying to fix and close issues. 

 What to do:

• Grab a copy of the Python codebase from Mercurial, following instructions in the Developer's Guide, and compile it.
• If you have a problem that isn't in the bug tracker, announce it to the IRC channel, and if it's more than five minutes' work, create a bug report for it. See the bug reporting instructions to learn how to write bug reports.
• When you choose a bug to work on, announce it to the IRC channel (e.g. "I'm working on #123456.") or on the bug report itself. This avoids accidentally duplicating work.
• Consider providing a patch that fixes the problem, or at least a simple test case that demonstrates the bug. Please see the patch submission guidelines in the Developer's Guide before submitting a patch.
• Does the bug appear to be gone in the Python development version (the Mercurial branch "default", that will become 3.4), but not the 3.2, 3.3 or 2.7 maintenance branchs? Report that, too.
• If someone else has supplied a fix, see if this fix works for² you, and add your results to the bug.
• Read the text of proposed patches and assess them for correctness and code quality. This is usually the most time-consuming step in the bug fixing process, so reading patches is very useful.
• If there's a working fix, feel free to add a note asking for the fix to get committed. The bug tracker has a lot of items in it, and it's easy for bugs to be overlooked.
• Feature requests should be classified as type 'feature request' in the bug tracker.

If you need any help beforehand, feel free to ask on core-mentorship mailing-list

Updates to docs.python.org

If you haven't already noticed, several months ago we updated the Sphinx theme for documentation of versions Python 3.2 and beyond on docs.python.org. It's a more modern look, and it also serves as an indicator that you're looking at documentation for a newer version. Thanks go out to Georg Brandl for his work on Sphinx, Python's documentation, and this new theme!

PEP 430

Over the weekend, PEP 430 was approved, which changes the default documentation displayed at http://docs.python.org. See the PEP for full details, but the jist is that we're now promoting the current Python 3 release as the default when you go to the docs home page. However, as the majority use case is still for Python 2 documentation, navigating straight to an unversioned page will present you with the current Python 2 documentation. For example, an unversioned link such as http://docs.python.org/library/zipfile will bring up the 2.7.3 documentation.

Version Dropdown

Supporting that change is a new feature that adds a version dropdown to the top of all documentation pages. Not only does this help when users are brought to a page which they don't expect, but switching between versions is a common operation as more and more projects work to add support for Python 3. Issue 8040 is where you'll find discussion on the change and its patches, with the bulk of the work completed by Yury Selivanov with some help from Georg.

This dropdown is especially handy as you peruse the documentation and come to a page that you want to view in another version. Choosing another version while on any page will load that page's other version, where the latest release of that version is chosen, e.g., 2.7 currently points to 2.7.3. So, as you browse the 2.7.3 built-ins page, choosing 3.3 in the dropdown will bring you to the 3.3.0 built-ins page.

We hope these changes enhance your experience when browsing the Python documentation!

Python 3.3 Beta 2 Released

Release manager Georg Brandl announced on August 12 that the second beta of CPython 3.3 was released, complete with installers for both Mac and Windows. This release represents the final feature set, and the goal is to get it in the hands of users to iron out any last issues.

Following this beta will be two release candidates, coming August 25 and September 8. The final release is slated to happen on September 22.

The "What's New in Python 3.3" document is currently being finalized by curator and long time developer Raymond Hettinger. The document already contains many of the new changes, but keep an eye out for newer versions.

Here are some of the bigger changes:

• PEP 380, syntax for delegating to a subgenerator ("yield from")
• PEP 393, flexible string representation (doing away with the distinction between "wide" and "narrow" Unicode builds)
• A C implementation of the "decimal" module, with up to 80x speedup for decimal-heavy applications
• The import system (__import__) now based on importlib by default
• The new "lzma" module with LZMA/XZ support
• PEP 397, a Python launcher for Windows
• PEP 405, virtual environment support in core
• PEP 420, namespace package support
• PEP 3151, reworking the OS and IO exception hierarchy
• PEP 3155, qualified name for classes and functions
• PEP 409, suppressing exception context
• PEP 414, explicit Unicode literals to help with porting
• PEP 418, extended platform-independent clocks in the "time" module
• PEP 412, a new key-sharing dictionary implementation that significantly saves memory for object-oriented code
• PEP 362, the function-signature object
• The new "faulthandler" module that helps diagnosing crashes
• The new "unittest.mock" module
• The new "ipaddress" module
• The "sys.implementation" attribute
• A policy framework for the email package, with a provisional (see PEP 411) policy that adds much improved unicode support for email header parsing
• A "collections.ChainMap" class for linking mappings to a single unit
• Wrappers for many more POSIX functions in the "os" and "signal" modules, as well as other useful functions such as "sendfile()"
• Hash randomization, introduced in earlier bugfix releases, is now switched on by default

In total, almost 500 API items are new or improved in Python 3.3.

Be sure to check out this release at http://www.python.org/download/releases/3.3.0/ and report any issues to http://bugs.python.org.