5. Upgrading¶

5.1. Development¶

(Fill this out whenever you introduce breaking changes to MongoEngine)

URLField’s constructor no longer takes verify_exists

5.2. 0.15.0¶

5.3. 0.14.0¶

This release includes a few bug fixes and a significant code cleanup. The most important change is that QuerySet.as_pymongo no longer supports a coerce_types mode. If you used it in the past, a) please let us know of your use case, b) you’ll need to override as_pymongo to get the desired outcome.

This release also makes the EmbeddedDocument not hashable by default. If you use embedded documents in sets or dictionaries, you might have to override __hash__ and implement a hashing logic specific to your use case. See #1528 for the reason behind this change.

5.4. 0.13.0¶

This release adds Unicode support to the EmailField and changes its structure significantly. Previously, email addresses containing Unicode characters didn’t work at all. Starting with v0.13.0, domains with Unicode characters are supported out of the box, meaning some emails that previously didn’t pass validation now do. Make sure the rest of your application can accept such email addresses. Additionally, if you subclassed the EmailField in your application and overrode EmailField.EMAIL_REGEX, you will have to adjust your code to override EmailField.USER_REGEX, EmailField.DOMAIN_REGEX, and potentially EmailField.UTF8_USER_REGEX.

5.5. 0.12.0¶

This release includes various fixes for the BaseQuerySet methods and how they are chained together. Since version 0.10.1 applying limit/skip/hint/batch_size to an already-existing queryset wouldn’t modify the underlying PyMongo cursor. This has been fixed now, so you’ll need to make sure that your code didn’t rely on the broken implementation.

Additionally, a public BaseQuerySet.clone_into has been renamed to a private _clone_into. If you directly used that method in your code, you’ll need to rename its occurrences.

5.6. 0.11.0¶

This release includes a major rehaul of MongoEngine’s code quality and introduces a few breaking changes. It also touches many different parts of the package and although all the changes have been tested and scrutinized, you’re encouraged to thoroughly test the upgrade.

First breaking change involves renaming ConnectionError to MongoEngineConnectionError. If you import or catch this exception, you’ll need to rename it in your code.

Second breaking change drops Python v2.6 support. If you run MongoEngine on that Python version, you’ll need to upgrade it first.

Third breaking change drops an old backward compatibility measure where from mongoengine.base import ErrorClass would work on top of from mongoengine.errors import ErrorClass (where ErrorClass is e.g. ValidationError). If you import any exceptions from mongoengine.base, change it to mongoengine.errors.

5.7. 0.10.8¶

This version fixed an issue where specifying a MongoDB URI host would override more information than it should. These changes are minor, but they still subtly modify the connection logic and thus you’re encouraged to test your MongoDB connection before shipping v0.10.8 in production.

5.8. 0.10.7¶

QuerySet.aggregate_sum and QuerySet.aggregate_average are dropped. Use QuerySet.sum and QuerySet.average instead which use the aggregation framework by default from now on.

5.9. 0.9.0¶

The 0.8.7 package on pypi was corrupted. If upgrading from 0.8.7 to 0.9.0 please follow:

python -m pip uninstall pymongo
python -m pip uninstall mongoengine
python -m pip install pymongo==2.8
python -m pip install mongoengine

5.10. 0.8.7¶

Calling reload on deleted / nonexistent documents now raises a DoesNotExist exception.

5.11. 0.8.2 to 0.8.3¶

Minor change that may impact users:

DynamicDocument fields are now stored in creation order after any declared fields. Previously they were stored alphabetically.

5.12. 0.7 to 0.8¶

There have been numerous backwards breaking changes in 0.8. The reasons for these are to ensure that MongoEngine has sane defaults going forward and that it performs the best it can out of the box. Where possible there have been FutureWarnings to help get you ready for the change, but that hasn’t been possible for the whole of the release.

Warning

Breaking changes - test upgrading on a test system before putting live. There maybe multiple manual steps in migrating and these are best honed on a staging / test system.

5.12.1. Python and PyMongo¶

MongoEngine requires python 2.6 (or above) and pymongo 2.5 (or above)

5.12.2. Data Model¶

5.12.2.1. Inheritance¶

The inheritance model has changed, we no longer need to store an array of types with the model we can just use the classname in _cls. This means that you will have to update your indexes for each of your inherited classes like so:

# 1. Declaration of the class
class Animal(Document):
    name = StringField()
    meta = {
        'allow_inheritance': True,
        'indexes': ['name']
    }

# 2. Remove _types
collection = Animal._get_collection()
collection.update({}, {"$unset": {"_types": 1}}, multi=True)

# 3. Confirm extra data is removed
count = collection.find({'_types': {"$exists": True}}).count()
assert count == 0

# 4. Remove indexes
info = collection.index_information()
indexes_to_drop = [key for key, value in info.items()
                   if '_types' in dict(value['key'])]
for index in indexes_to_drop:
    collection.drop_index(index)

# 5. Recreate indexes
Animal.ensure_indexes()

5.12.2.2. Document Definition¶

The default for inheritance has changed - it is now off by default and _cls will not be stored automatically with the class. So if you extend your Document or EmbeddedDocuments you will need to declare allow_inheritance in the meta data like so:

class Animal(Document):
    name = StringField()

    meta = {'allow_inheritance': True}

Previously, if you had data in the database that wasn’t defined in the Document definition, it would set it as an attribute on the document. This is no longer the case and the data is set only in the document._data dictionary:

>>> from mongoengine import *
>>> class Animal(Document):
...    name = StringField()
...
>>> cat = Animal(name="kit", size="small")

# 0.7
>>> cat.size
u'small'

# 0.8
>>> cat.size
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Animal' object has no attribute 'size'

The Document class has introduced a reserved function clean(), which will be called before saving the document. If your document class happens to have a method with the same name, please try to rename it.

def clean(self):
pass

5.12.2.3. ReferenceField¶

ReferenceFields now store ObjectIds by default - this is more efficient than DBRefs as we already know what Document types they reference:

# Old code
class Animal(Document):
    name = ReferenceField('self')

# New code to keep dbrefs
class Animal(Document):
    name = ReferenceField('self', dbref=True)

To migrate all the references you need to touch each object and mark it as dirty eg:

# Doc definition
class Person(Document):
    name = StringField()
    parent = ReferenceField('self')
    friends = ListField(ReferenceField('self'))

# Mark all ReferenceFields as dirty and save
for p in Person.objects:
    p._mark_as_changed('parent')
    p._mark_as_changed('friends')
    p.save()

An example test migration for ReferenceFields is available on github.

Note

Internally mongoengine handles ReferenceFields the same, so they are converted to DBRef on loading and ObjectIds or DBRefs depending on settings on storage.

5.12.2.4. UUIDField¶

UUIDFields now default to storing binary values:

# Old code
class Animal(Document):
    uuid = UUIDField()

# New code
class Animal(Document):
    uuid = UUIDField(binary=False)

To migrate all the uuids you need to touch each object and mark it as dirty eg:

# Doc definition
class Animal(Document):
    uuid = UUIDField()

# Mark all UUIDFields as dirty and save
for a in Animal.objects:
    a._mark_as_changed('uuid')
    a.save()

An example test migration for UUIDFields is available on github.

5.12.2.5. DecimalField¶

DecimalFields now store floats - previously it was storing strings and that made it impossible to do comparisons when querying correctly.:

# Old code
class Person(Document):
    balance = DecimalField()

# New code
class Person(Document):
    balance = DecimalField(force_string=True)

To migrate all the DecimalFields you need to touch each object and mark it as dirty eg:

# Doc definition
class Person(Document):
    balance = DecimalField()

# Mark all DecimalField's as dirty and save
for p in Person.objects:
    p._mark_as_changed('balance')
    p.save()

Note

DecimalFields have also been improved with the addition of precision and rounding. See DecimalField for more information.

An example test migration for DecimalFields is available on github.

5.12.2.6. Cascading Saves¶

To improve performance document saves will no longer automatically cascade. Any changes to a Document’s references will either have to be saved manually or you will have to explicitly tell it to cascade on save:

# At the class level:
class Person(Document):
    meta = {'cascade': True}

# Or on save:
my_document.save(cascade=True)

5.12.2.7. Storage¶

Document and Embedded Documents are now serialized based on declared field order. Previously, the data was passed to mongodb as a dictionary and which meant that order wasn’t guaranteed - so things like $addToSet operations on EmbeddedDocument could potentially fail in unexpected ways.

If this impacts you, you may want to rewrite the objects using the doc.mark_as_dirty('field') pattern described above. If you are using a compound primary key then you will need to ensure the order is fixed and match your EmbeddedDocument to that order.

5.12.3. Querysets¶

5.12.3.1. Attack of the clones¶

Querysets now return clones and should no longer be considered editable in place. This brings us in line with how Django’s querysets work and removes a long running gotcha. If you edit your querysets inplace you will have to update your code like so:

# Old code:
mammals = Animal.objects(type="mammal")
mammals.filter(order="Carnivora")       # Returns a cloned queryset that isn't assigned to anything - so this will break in 0.8
[m for m in mammals]                    # This will return all mammals in 0.8 as the 2nd filter returned a new queryset

# Update example a) assign queryset after a change:
mammals = Animal.objects(type="mammal")
carnivores = mammals.filter(order="Carnivora") # Reassign the new queryset so filter can be applied
[m for m in carnivores]                        # This will return all carnivores

# Update example b) chain the queryset:
mammals = Animal.objects(type="mammal").filter(order="Carnivora")  # The final queryset is assgined to mammals
[m for m in mammals]                                               # This will return all carnivores

5.12.3.2. Len iterates the queryset¶

If you ever did len(queryset) it previously did a count() under the covers, this caused some unusual issues. As len(queryset) is most often used by list(queryset) we now cache the queryset results and use that for the length.

This isn’t as performant as a count() and if you aren’t iterating the queryset you should upgrade to use count:

# Old code
len(Animal.objects(type="mammal"))

# New code
Animal.objects(type="mammal").count()

5.12.3.3. .only() now inline with .exclude()¶

The behaviour of .only() was highly ambiguous, now it works in mirror fashion to .exclude(). Chaining .only() calls will increase the fields required:

# Old code
Animal.objects().only(['type', 'name']).only('name', 'order')  # Would have returned just `name`

# New code
Animal.objects().only('name')

# Note:
Animal.objects().only(['name']).only('order')  # Now returns `name` *and* `order`

5.12.4. Client¶

PyMongo 2.4 came with a new connection client; MongoClient and started the depreciation of the old Connection. MongoEngine now uses the latest MongoClient for connections. By default operations were safe but if you turned them off or used the connection directly this will impact your queries.

5.12.4.1. Querysets¶

5.12.4.1.1. Safe¶

safe has been depreciated in the new MongoClient connection. Please use write_concern instead. As safe always defaulted as True normally no code change is required. To disable confirmation of the write just pass {“w”: 0} eg:

# Old
Animal(name="Dinasour").save(safe=False)

# new code:
Animal(name="Dinasour").save(write_concern={"w": 0})

5.12.4.1.2. Write Concern¶

write_options has been replaced with write_concern to bring it inline with pymongo. To upgrade simply rename any instances where you used the write_option keyword to write_concern like so:

# Old code:
Animal(name="Dinasour").save(write_options={"w": 2})

# new code:
Animal(name="Dinasour").save(write_concern={"w": 2})

5.12.5. Indexes¶

Index methods are no longer tied to querysets but rather to the document class. Although QuerySet._ensure_indexes and QuerySet.ensure_index still exist. They should be replaced with ensure_indexes() / ensure_index().

5.12.6. SequenceFields¶

SequenceField now inherits from BaseField to allow flexible storage of the calculated value. As such MIN and MAX settings are no longer handled.

5.13. 0.6 to 0.7¶

5.13.1. Cascade saves¶

Saves will raise a FutureWarning if they cascade and cascade hasn’t been set to True. This is because in 0.8 it will default to False. If you require cascading saves then either set it in the meta or pass via save eg

# At the class level:
class Person(Document):
    meta = {'cascade': True}

# Or in code:
my_document.save(cascade=True)

Note

Remember: cascading saves do not cascade through lists.

5.13.2. ReferenceFields¶

ReferenceFields now can store references as ObjectId strings instead of DBRefs. This will become the default in 0.8 and if dbref is not set a FutureWarning will be raised.

To explicitly continue to use DBRefs change the dbref flag to True

class Person(Document):
    groups = ListField(ReferenceField(Group, dbref=True))

To migrate to using strings instead of DBRefs you will have to manually migrate

# Step 1 - Migrate the model definition
class Group(Document):
    author = ReferenceField(User, dbref=False)
    members = ListField(ReferenceField(User, dbref=False))

# Step 2 - Migrate the data
for g in Group.objects():
    g.author = g.author
    g.members = g.members
    g.save()

5.13.3. item_frequencies¶

In the 0.6 series we added support for null / zero / false values in item_frequencies. A side effect was to return keys in the value they are stored in rather than as string representations. Your code may need to be updated to handle native types rather than strings keys for the results of item frequency queries.

5.13.4. BinaryFields¶

Binary fields have been updated so that they are native binary types. If you previously were doing str comparisons with binary field values you will have to update and wrap the value in a str.

5.14. 0.5 to 0.6¶

Embedded Documents - if you had a pk field you will have to rename it from _id to pk as pk is no longer a property of Embedded Documents.

Reverse Delete Rules in Embedded Documents, MapFields and DictFields now throw an InvalidDocument error as they aren’t currently supported.

Document._get_subclasses - Is no longer used and the class method has been removed.

Document.objects.with_id - now raises an InvalidQueryError if used with a filter.

FutureWarning - A future warning has been added to all inherited classes that don’t define allow_inheritance in their meta.

You may need to update pyMongo to 2.0 for use with Sharding.

5.15. 0.4 to 0.5¶

There have been the following backwards incompatibilities from 0.4 to 0.5. The main areas of changed are: choices in fields, map_reduce and collection names.

5.15.1. Choice options:¶

Are now expected to be an iterable of tuples, with the first element in each tuple being the actual value to be stored. The second element is the human-readable name for the option.

5.15.2. PyMongo / MongoDB¶

map reduce now requires pymongo 1.11+- The pymongo merge_output and reduce_output parameters, have been depreciated.

More methods now use map_reduce as db.eval is not supported for sharding as such the following have been changed:

sum()

average()

item_frequencies()

5.15.3. Default collection naming¶

Previously it was just lowercase, it’s now much more pythonic and readable as it’s lowercase and underscores, previously

class MyAceDocument(Document):
    pass

MyAceDocument._meta['collection'] == myacedocument

In 0.5 this will change to

class MyAceDocument(Document):
    pass

MyAceDocument._get_collection_name() == my_ace_document

To upgrade use a Mixin class to set meta like so

class BaseMixin(object):
    meta = {
        'collection': lambda c: c.__name__.lower()
    }

class MyAceDocument(Document, BaseMixin):
    pass

MyAceDocument._get_collection_name() == "myacedocument"

Alternatively, you can rename your collections eg

from mongoengine.connection import _get_db
from mongoengine.base import _document_registry

def rename_collections():
    db = _get_db()

    failure = False

    collection_names = [d._get_collection_name()
                        for d in _document_registry.values()]

    for new_style_name in collection_names:
        if not new_style_name:  # embedded documents don't have collections
            continue
        old_style_name = new_style_name.replace('_', '')

        if old_style_name == new_style_name:
            continue  # Nothing to do

        existing = db.collection_names()
        if old_style_name in existing:
            if new_style_name in existing:
                failure = True
                print "FAILED to rename: %s to %s (already exists)" % (
                    old_style_name, new_style_name)
            else:
                db[old_style_name].rename(new_style_name)
                print "Renamed:  %s to %s" % (old_style_name,
                                              new_style_name)

    if failure:
        print "Upgrading  collection names failed"
    else:
        print "Upgraded collection names"

5.15.4. mongodb 1.8 > 2.0 +¶

It’s been reported that indexes may need to be recreated to the newer version of indexes. To do this drop indexes and call ensure_indexes on each model.