asyncio in OpenStack

Warning

The project of replacing eventlet with trollius using aioeventlet in OpenStack is abandonned. It might done “later” when Python 2 support will be removed from OpenStack which is not going to happen in a near future.

First part (in progress): add support for trollius coroutines

Prepare OpenStack (Oslo Messaging) to support trollius coroutines using yield: explicit asynchronous programming. Eventlet is still supported, used by default, and applications and libraries don’t need to be modified at this point.

Already done:

  • Write the trollius project: port asyncio to Python 2
  • Stabilize trollius API
  • Add trollius dependency to OpenStack
  • Write the aioeventlet project to provide the asyncio API on top of eventlet

To do:

  • Stabilize aioeventlet API
  • Add aioeventlet dependency to OpenStack
  • Write an aioeventlet executor for Oslo Messaging: rewrite greenio executor to replace greenio with aioeventlet

Second part (to do): rewrite code as trollius coroutines

Switch from implicit asynchronous programming (eventlet using greenthreads) to explicit asynchronous programming (trollius coroutines using yield). Need to modify OpenStack Common Libraries and applications. Modifications can be done step by step, the switch will take more than 6 months.

The first application candidate is Ceilometer. The Ceilometer project is young, developers are aware of eventlet issues and like Python 3, and Ceilometer don’t rely so much on asynchronous programming: most time is spent into waiting the database anyway.

The goal is to port Ceilometer to explicit asynchronous programming during the cycle of OpenStack Kilo.

Some applications may continue to use implicit asynchronous programming. For example, nova is probably the most complex part beacuse it is and old project with a lot of legacy code, it has many drivers and the code base is large.

To do:

  • Ceilometer: add trollius dependency and set the trollius event loop policy to aioeventlet
  • Ceilometer: change Oslo Messaging executor from “eventlet” to “aioeventlet”
  • Redesign the service class of Oslo Incubator to support aioeventlet and/or trollius. Currently, the class is designed for eventlet. The service class is instanciated before forking, which requires hacks on eventlet to update file descriptors.
  • In Ceilometer and its OpenStack depedencencies: add new functions which are written with explicit asynchronous programming in mind (ex: trollius coroutines written with yield).
  • Rewrite Ceilometer endpoints (RPC methods) as trollius coroutines.

Questions:

  • What about WSGI? aiohttp is not compatible with trollius yet.
  • The quantity of code which need to be ported to asynchronous programming is unknown right now.
  • We should be prepared to see deadlocks. OpenStack was designed for eventlet which implicitly switch on blocking operations. Critical sections may not be protected with locks, or not the right kind of lock.
  • For performances, blocking operations can be executed in threads. OpenStack code is probably not thread-safe, which means new kinds of race conditions. But the code executed in threads will be explicitly scheduled to be executed in a thread (with loop.run_in_executor()), so regressions can be easily identified.
  • This part will take a lot of time. We may need to split it into subparts to have milestones, which is more attractive for developers.

Last part (to do): drop eventlet

Replace aioeventlet event loop with trollius event loop, drop aioeventlet and drop eventlet at the end.

This change will be done on applications one by one. This is no need to port all applications at once. The work will start on Ceilometer, as a follow up of the second part.

To do:

  • Port remaining code to trollius
  • Write a “trollius” executor for Oslo Messaging
  • Ceilometer: Add a blocking call to loop.run_forever() in the main() function
  • Ceilometer: Replace “aioeventlet” executor with “trollius” executor
  • Ceilometer: Use the standard trollius event loop policy
  • Ceilometer: drop the eventlet dependency

Questions:

  • Putting a blocking call to loop.run_forever() may need to redesign Ceilometer, this part is unclear to me right now.

Optimization, can be done later:

  • Oslo Messaging: watch directly the underlying file descriptor of sockets, instead of using a busy loop polling the notifier
  • Ceilometer: use libraries supporting directly trollius to be able to run parallel tasks (ex: send multiple requests to a database)

openstack-dev mailing list

History of asyncio in OpenStack

Threads and asyncio specs:

Maybe the good one, aioeventlet project:

OpenStack Kilo Summit, November 3-7, 2014, at Paris:

New try with a greenio executor for Oslo Messaging:

First try with a trollius executor for Oslo Messaging:

First discussion around asyncio and OpenStack:

History of asynchronous programming in OpenStack

In the past, the Nova project used Tornado, then Twisted and it is now using eventlet which also became the defacto standard in OpenStack