We have released PyPy 5.0, about three months after PyPy 4.0.1. We encourage all users of PyPy to update to this version. Apart from the usual bug fixes, there is an ongoing effort to improve the warmup time and memory usage of JIT-related metadata. The exact effects depend vastly on the program you’re running and can range from insignificant to warmup being up to 30% faster and memory dropping by about 30%.
We also merged a major upgrade to our C-API layer (cpyext), simplifying the interaction between c-level objects and PyPy interpreter level objects. As a result, lxml (prerelease) with its cython compiled component passes all tests on PyPy. The new cpyext is also much faster.
vmprof has been a go-to profiler for PyPy on linux for a few releases and we’re happy to announce that thanks to the cooperation with jetbrains, vmprof now works on Linux, OS X and Windows on both PyPy and CPython.
You can download the PyPy 5.0 release here:
We would like to thank our donors for the continued support of the PyPy project.
We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on pypy, or general help with making RPython’s JIT even better.
While not applicable only to PyPy, cffi is arguably our most significant contribution to the python ecosystem. PyPy 5.0 ships with cffi-1.5.2 which now allows embedding PyPy (or cpython) in a C program.
What is PyPy?¶
PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It’s fast (PyPy and CPython 2.7.x performance comparison) due to its integrated tracing JIT compiler.
We also welcome developers of other dynamic languages to see what RPython can do for them.
This release supports x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD), newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux, and the big- and little-endian variants of PPC64 running Linux.
Other Highlights (since 4.0.1 released in November 2015)¶
Support embedding PyPy in a C-program via cffi and static callbacks in cffi. This deprecates the old method of embedding PyPy
Refactor vmprof to work cross-operating-system, deprecate using buggy libunwind on Linux platforms. Vmprof even works on Windows now.
Support more of the C-API type slots, like tp_getattro, and fix C-API macros, functions, and structs such as _PyLong_FromByteArray(), PyString_GET_SIZE, f_locals in PyFrameObject, Py_NAN, co_filename in PyCodeObject
Use a more stable approach for allocating PyObjects in cpyext. (see blog post). Once the PyObject corresponding to a PyPy object is created, it stays around at the same location until the death of the PyPy object. Done with a little bit of custom GC support. It allows us to kill the notion of “borrowing” inside cpyext, reduces 4 dictionaries down to 1, and significantly simplifies the whole approach (which is why it is a new feature while technically a refactoring) and allows PyPy to support the populart lxml module (as of the next release) with no PyPy specific patches needed
Make the default filesystem encoding ASCII, like CPython
Use hypothesis in test creation, which is great for randomizing tests
Backport always using os.urandom for uuid4 from cpython and fix the JIT as well (issue #2202)
More completely support datetime, optimize timedelta creation
Fix for issue #2185 which caused an inconsistent list of operations to be generated by the unroller, appeared in a complicated DJango app
Fix an elusive issue with stacklets on shadowstack which showed up when forgetting stacklets without resuming them
Fix entrypoint() which now acquires the GIL
Fix direct_ffi_call() so failure does not bail out before setting CALL_MAY_FORCE
Fix (de)pickling long values by simplifying the implementation
Fix RPython rthread so that objects stored as threadlocal do not force minor GC collection and are kept alive automatically. This improves perfomance of short-running Python callbacks and prevents resetting such object between calls
Support floats as parameters to itertools.isslice()
Check for the existence of CODESET, ignoring it should have prevented PyPy from working on FreeBSD
Fix for corner case (likely shown by Krakatau) for consecutive guards with interdependencies
Fix applevel bare class method comparisons which should fix pretty printing in IPython
Updates to numpy 1.10.2 (incompatibilities and not-implemented features still exist)
Support dtype=((‘O’, spec)) union while disallowing record arrays with mixed object, non-object values
Remove all traces of micronumpy from cpyext if –withoutmod-micronumpy option used
Support indexing filtering with a boolean ndarray
Support partition() as an app-level function, together with a cffi wrapper in pypy/numpy, this now provides partial support for partition()
Optimize global lookups
Improve the memory signature of numbering instances in the JIT. This should massively decrease the amount of memory consumed by the JIT, which is significant for most programs. Also compress the numberings using variable- size encoding
Optimize string concatenation
Use INT_LSHIFT instead of INT_MUL when possible
Improve struct.unpack by casting directly from the underlying buffer. Unpacking floats and doubles is about 15 times faster, and integer types about 50% faster (on 64 bit integers). This was then subsequently improved further in optimizeopt.py.
Optimize two-tuple lookups in mapdict, which improves warmup of instance variable access somewhat
Reduce all guards from int_floordiv_ovf if one of the arguments is constant
Identify permutations of attributes at instance creation, reducing the number of bridges created
Greatly improve re.sub() performance
Refactor and improve exception analysis in the annotator
Remove unnecessary special handling of space.wrap().
Support list-resizing setslice operations in RPython
Tweak the trace-too-long heuristic for multiple jit drivers
Refactor bookkeeping (such a cool word - three double letters) in the annotater
Refactor wrappers for OS functions from rtyper to rlib and simplify them
Simplify backend loading instructions to only use four variants
Simplify GIL handling in non-jitted code
Refactor naming in optimizeopt
Change GraphAnalyzer to use a more precise way to recognize external functions and fix null pointer handling, generally clean up external function handling
Remove pure variants of
getfield_gc_*operations from the JIT by determining purity while tracing
Simplify bootstrapping in cpyext
Refactor rtyper debug code into python.rtyper.debug
Seperate structmember.h from Python.h Also enhance creating api functions to specify which header file they appear in (previously only pypy_decl.h)
Fix tokenizer to enforce universal newlines, needed for Python 3 support
Please update, and continue to help us make PyPy better.
The PyPy Team