PyPy2.7 and PyPy3.5 v5.9 dual release

The PyPy team is proud to release both PyPy2.7 v5.9 (an interpreter supporting Python 2.7 syntax), and a beta-quality PyPy3.5 v5.9 (an interpreter for Python 3.5 syntax). The two releases are both based on much the same codebase, thus the dual release. Note that PyPy3.5 supports Linux 64bit only for now.

This new PyPy2.7 release includes the upstream stdlib version 2.7.13, and PyPy3.5 includes the upstream stdlib version 3.5.3.

NumPy and Pandas now work on PyPy2.7 (together with Cython 0.27.1). Issues that appeared as excessive memory use were cleared up and other incompatibilities were resolved. The C-API compatibility layer does slow down code which crosses the python-c interface often, we have ideas on how it could be improved, and still recommend using pure python on PyPy or interfacing via CFFI. Many other modules based on C-API exentions now work on PyPy as well.

Cython 0.27.1 (released very recently) supports more projects with PyPy, both on PyPy2.7 and PyPy3.5 beta. Note version 0.27.1 is now the minimum version that supports this version of PyPy, due to some interactions with updated C-API interface code.

We optimized the JSON parser for recurring string keys, which should decrease memory use to 50% and increase parsing speed by up to 15% for large JSON files with many repeating dictionary keys (which is quite common).

CFFI, which is part of the PyPy release, has been updated to 1.11.1, improving an already great package for interfacing with C. CFFI now supports complex arguments in API mode, as well as char16_t and char32_t and has improved support for callbacks.

Please let us know if your use case is slow, we have ideas how to make things faster but need real-world examples (not micro-benchmarks) of problematic code.

Work sponsored by a Mozilla grant continues on PyPy3.5; numerous fixes from CPython were ported to PyPy. Of course the bug fixes and performance enhancements mentioned above are part of both PyPy2.7 and PyPy3.5 beta.

As always, this release fixed many other issues and bugs raised by the growing community of PyPy users. We strongly recommend updating.

You can download the v5.9 releases here:

We would like to thank our donors for the continued support of the PyPy project.

We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on pypy, or general help with making RPython’s JIT even better.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7 and CPython 3.5. It’s fast (PyPy and CPython 2.7.x performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

The PyPy 2.7 release supports:

  • x86 machines on most common operating systems (Linux 32/64 bits, Mac OS X 64 bits, Windows 32 bits, OpenBSD, FreeBSD)

  • newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux,

  • big- and little-endian variants of PPC64 running Linux,

  • s390x running Linux

Highlights of the PyPy2.7, cpyext, and RPython changes (since 5.8 released June, 2017)

See also issues that were resolved

Note that these are also merged into PyPy 3.5

  • New features and cleanups

    • Add support for PyFrozenSet_New, PyObject_HashNotImplemented, PyObject_Print(NULL, ...), PyObject_RichCompareBool(a, a, ...), PyType_IS_GC (does nothing), PyUnicode_FromFormat

    • ctypes char_p and unichar_p indexing now CPython compatible

    • gcdump now reports largest object

    • More complete support in the _curses CFFI module

    • Add cPickle.Unpickler.find_global (issue 1853)

    • Fix PyErr_Fetch + PyErr_NormalizeException with no exception set

    • Simplify gc.get_referrers() to return the opposite of gc.get_referents()

    • Update RevDB to version pypy2.7-v5.6.2

    • Previously, instance.method would return always the same bound method object, when gotten from the same instance (as far as is and id() can tell). CPython doesn’t do that. Now PyPy, like CPython, returns a different bound method object every time. For type.method, PyPy2 still returns always the same unbound method object; CPython does it for built-in types but not for user-defined types

    • Link to disable PaX protection for the JIT when needed

    • Update build instructions and an rarely used Makefile

    • Recreate support for using leakfinder in cpyext tests which had suffered bit-rot, disable due to many false positives

    • Add more functionality to sysconfig

    • Added _swappedbytes_ support for ctypes.Structure

    • Better support the inspect module on frames

  • Bug Fixes

    • Fix issue 2592 - cpyext PyListObject.pop, pop_end must return a value

    • Implement PyListOjbect.getstorage_copy

    • Fix for reversed(dictproxy) issue 2601

    • Fix for duplicate names in ctypes’ _fields__, issue 2621

    • Update built-in pyexpat module on win32 to use UTF-8 version not UTF-16

    • gc.get_objects now handles objects with finalizers more consistently

    • Fixed memory leak in SSLContext.getpeercert returning validated certificates and SSLContext.get_ca_certs(binary_mode=True) (_get_crl_dp) CPython issue 29738

  • Performance improvements:

    • Improve performance of bytearray.extend by rewriting portions in app-level

    • Optimize list accesses with constant indexes better by retaining more information about them

    • Add a jit driver for array.count and array.index

    • Improve information retained in a bridge wrt array

    • Move some dummy CAPI functions and Py*_Check functions from RPython into pure C macros

    • In the fast zip(intlist1, intlist2) implementation, don’t wrap and unwrap all the ints

    • Cache string keys that occur in JSON dicts, as they are likely to repeat

  • RPython improvements

    • Do not preallocate a RPython list if we only know an upper bound on its size

    • Issue 2590: fix the bounds in the GC when allocating a lot of objects with finalizers

    • Replace magical NOT RPYTHON comment with a decorator

    • Implement socket.sendmsg()/.recvmsg() for py3.5

    • Add memory_pressure for _SSLSocket objects

  • Degredations

    • Disable vmprof on win32, due to upstream changes that break the internal _vmprof module

Highlights of the PyPy3.5 release (since 5.8 beta released June 2017)

  • New features

    • Add support for _PyNamespace_New, PyMemoryView_FromMemory, Py_EnterRecursiveCall raising RecursionError, PyObject_LengthHint, PyUnicode_FromKindAndData, PyDict_SetDefault, PyGenObject, PyGenObject, PyUnicode_Substring, PyLong_FromUnicodeObject

    • Implement PyType_FromSpec (PEP 384) and fix issues with PEP 489 support

    • Support the new version of os.stat() on win32

    • Use stat3() on Posix

    • Accept buffer objects as filenames, except for oslistdir`

    • Make slices of array memoryview s usable as writable buffers if contiguous

    • Better handling of '%s' formatting for byte strings which might be utf-8 encoded

    • Update the macros Py_DECREF and similar to use the CPython 3.5 version

    • Ensure that mappingproxy is recognised as a mapping, not a sequence

    • Enable PGO for CLang

    • Rework cppyy packaging and rename the backend to _cppyy

    • Support for libressl 2.5.4

    • Mirror CPython classmethod __reduce__ which fixes pickling test

    • Use utf-8 for readline history file

    • Allow assigning '__class__' between ModuleType and its subclasses

    • Add async slot functions in cpyext

  • Bug Fixes

    • Try to make openssl CFFI bindings more general and future-proof

    • Better support importlib by only listing built-in modules in sys.builtin

    • Add memory_pressure to large CFFI allocations in _lzma, issue 2579

    • Fix for reversed(mapping object) issue 2601

    • Fixing regression with non-started generator receiving non-None, should always raise TypeError

    • itertools.islice: use same logic as CPython, fixes 2643

  • Performance improvements:

  • The following features of Python 3.5 are not implemented yet in PyPy:

    • PEP 442: Safe object finalization

Please update, and continue to help us make PyPy better.

Cheers