######### Upgrading ######### Development *********** (Fill this out whenever you introduce breaking changes to MongoEngine) URLField's constructor no longer takes `verify_exists` 0.15.0 ****** 0.14.0 ****** This release includes a few bug fixes and a significant code cleanup. The most important change is that `QuerySet.as_pymongo` no longer supports a `coerce_types` mode. If you used it in the past, a) please let us know of your use case, b) you'll need to override `as_pymongo` to get the desired outcome. This release also makes the EmbeddedDocument not hashable by default. If you use embedded documents in sets or dictionaries, you might have to override `__hash__` and implement a hashing logic specific to your use case. See #1528 for the reason behind this change. 0.13.0 ****** This release adds Unicode support to the `EmailField` and changes its structure significantly. Previously, email addresses containing Unicode characters didn't work at all. Starting with v0.13.0, domains with Unicode characters are supported out of the box, meaning some emails that previously didn't pass validation now do. Make sure the rest of your application can accept such email addresses. Additionally, if you subclassed the `EmailField` in your application and overrode `EmailField.EMAIL_REGEX`, you will have to adjust your code to override `EmailField.USER_REGEX`, `EmailField.DOMAIN_REGEX`, and potentially `EmailField.UTF8_USER_REGEX`. 0.12.0 ****** This release includes various fixes for the `BaseQuerySet` methods and how they are chained together. Since version 0.10.1 applying limit/skip/hint/batch_size to an already-existing queryset wouldn't modify the underlying PyMongo cursor. This has been fixed now, so you'll need to make sure that your code didn't rely on the broken implementation. Additionally, a public `BaseQuerySet.clone_into` has been renamed to a private `_clone_into`. If you directly used that method in your code, you'll need to rename its occurrences. 0.11.0 ****** This release includes a major rehaul of MongoEngine's code quality and introduces a few breaking changes. It also touches many different parts of the package and although all the changes have been tested and scrutinized, you're encouraged to thoroughly test the upgrade. First breaking change involves renaming `ConnectionError` to `MongoEngineConnectionError`. If you import or catch this exception, you'll need to rename it in your code. Second breaking change drops Python v2.6 support. If you run MongoEngine on that Python version, you'll need to upgrade it first. Third breaking change drops an old backward compatibility measure where `from mongoengine.base import ErrorClass` would work on top of `from mongoengine.errors import ErrorClass` (where `ErrorClass` is e.g. `ValidationError`). If you import any exceptions from `mongoengine.base`, change it to `mongoengine.errors`. 0.10.8 ****** This version fixed an issue where specifying a MongoDB URI host would override more information than it should. These changes are minor, but they still subtly modify the connection logic and thus you're encouraged to test your MongoDB connection before shipping v0.10.8 in production. 0.10.7 ****** `QuerySet.aggregate_sum` and `QuerySet.aggregate_average` are dropped. Use `QuerySet.sum` and `QuerySet.average` instead which use the aggregation framework by default from now on. 0.9.0 ***** The 0.8.7 package on pypi was corrupted. If upgrading from 0.8.7 to 0.9.0 please follow: :: python -m pip uninstall pymongo python -m pip uninstall mongoengine python -m pip install pymongo==2.8 python -m pip install mongoengine 0.8.7 ***** Calling reload on deleted / nonexistent documents now raises a DoesNotExist exception. 0.8.2 to 0.8.3 ************** Minor change that may impact users: DynamicDocument fields are now stored in creation order after any declared fields. Previously they were stored alphabetically. 0.7 to 0.8 ********** There have been numerous backwards breaking changes in 0.8. The reasons for these are to ensure that MongoEngine has sane defaults going forward and that it performs the best it can out of the box. Where possible there have been FutureWarnings to help get you ready for the change, but that hasn't been possible for the whole of the release. .. warning:: Breaking changes - test upgrading on a test system before putting live. There maybe multiple manual steps in migrating and these are best honed on a staging / test system. Python and PyMongo ================== MongoEngine requires python 2.6 (or above) and pymongo 2.5 (or above) Data Model ========== Inheritance ----------- The inheritance model has changed, we no longer need to store an array of :attr:`types` with the model we can just use the classname in :attr:`_cls`. This means that you will have to update your indexes for each of your inherited classes like so: :: # 1. Declaration of the class class Animal(Document): name = StringField() meta = { 'allow_inheritance': True, 'indexes': ['name'] } # 2. Remove _types collection = Animal._get_collection() collection.update({}, {"$unset": {"_types": 1}}, multi=True) # 3. Confirm extra data is removed count = collection.find({'_types': {"$exists": True}}).count() assert count == 0 # 4. Remove indexes info = collection.index_information() indexes_to_drop = [key for key, value in info.items() if '_types' in dict(value['key'])] for index in indexes_to_drop: collection.drop_index(index) # 5. Recreate indexes Animal.ensure_indexes() Document Definition ------------------- The default for inheritance has changed - it is now off by default and :attr:`_cls` will not be stored automatically with the class. So if you extend your :class:`~mongoengine.Document` or :class:`~mongoengine.EmbeddedDocuments` you will need to declare :attr:`allow_inheritance` in the meta data like so: :: class Animal(Document): name = StringField() meta = {'allow_inheritance': True} Previously, if you had data in the database that wasn't defined in the Document definition, it would set it as an attribute on the document. This is no longer the case and the data is set only in the ``document._data`` dictionary: :: >>> from mongoengine import * >>> class Animal(Document): ... name = StringField() ... >>> cat = Animal(name="kit", size="small") # 0.7 >>> cat.size u'small' # 0.8 >>> cat.size Traceback (most recent call last): File "", line 1, in AttributeError: 'Animal' object has no attribute 'size' The Document class has introduced a reserved function `clean()`, which will be called before saving the document. If your document class happens to have a method with the same name, please try to rename it. def clean(self): pass ReferenceField -------------- ReferenceFields now store ObjectIds by default - this is more efficient than DBRefs as we already know what Document types they reference:: # Old code class Animal(Document): name = ReferenceField('self') # New code to keep dbrefs class Animal(Document): name = ReferenceField('self', dbref=True) To migrate all the references you need to touch each object and mark it as dirty eg:: # Doc definition class Person(Document): name = StringField() parent = ReferenceField('self') friends = ListField(ReferenceField('self')) # Mark all ReferenceFields as dirty and save for p in Person.objects: p._mark_as_changed('parent') p._mark_as_changed('friends') p.save() `An example test migration for ReferenceFields is available on github `_. .. Note:: Internally mongoengine handles ReferenceFields the same, so they are converted to DBRef on loading and ObjectIds or DBRefs depending on settings on storage. UUIDField --------- UUIDFields now default to storing binary values:: # Old code class Animal(Document): uuid = UUIDField() # New code class Animal(Document): uuid = UUIDField(binary=False) To migrate all the uuids you need to touch each object and mark it as dirty eg:: # Doc definition class Animal(Document): uuid = UUIDField() # Mark all UUIDFields as dirty and save for a in Animal.objects: a._mark_as_changed('uuid') a.save() `An example test migration for UUIDFields is available on github `_. DecimalField ------------ DecimalFields now store floats - previously it was storing strings and that made it impossible to do comparisons when querying correctly.:: # Old code class Person(Document): balance = DecimalField() # New code class Person(Document): balance = DecimalField(force_string=True) To migrate all the DecimalFields you need to touch each object and mark it as dirty eg:: # Doc definition class Person(Document): balance = DecimalField() # Mark all DecimalField's as dirty and save for p in Person.objects: p._mark_as_changed('balance') p.save() .. note:: DecimalFields have also been improved with the addition of precision and rounding. See :class:`~mongoengine.fields.DecimalField` for more information. `An example test migration for DecimalFields is available on github `_. Cascading Saves --------------- To improve performance document saves will no longer automatically cascade. Any changes to a Document's references will either have to be saved manually or you will have to explicitly tell it to cascade on save:: # At the class level: class Person(Document): meta = {'cascade': True} # Or on save: my_document.save(cascade=True) Storage ------- Document and Embedded Documents are now serialized based on declared field order. Previously, the data was passed to mongodb as a dictionary and which meant that order wasn't guaranteed - so things like ``$addToSet`` operations on :class:`~mongoengine.EmbeddedDocument` could potentially fail in unexpected ways. If this impacts you, you may want to rewrite the objects using the ``doc.mark_as_dirty('field')`` pattern described above. If you are using a compound primary key then you will need to ensure the order is fixed and match your EmbeddedDocument to that order. Querysets ========= Attack of the clones -------------------- Querysets now return clones and should no longer be considered editable in place. This brings us in line with how Django's querysets work and removes a long running gotcha. If you edit your querysets inplace you will have to update your code like so: :: # Old code: mammals = Animal.objects(type="mammal") mammals.filter(order="Carnivora") # Returns a cloned queryset that isn't assigned to anything - so this will break in 0.8 [m for m in mammals] # This will return all mammals in 0.8 as the 2nd filter returned a new queryset # Update example a) assign queryset after a change: mammals = Animal.objects(type="mammal") carnivores = mammals.filter(order="Carnivora") # Reassign the new queryset so filter can be applied [m for m in carnivores] # This will return all carnivores # Update example b) chain the queryset: mammals = Animal.objects(type="mammal").filter(order="Carnivora") # The final queryset is assgined to mammals [m for m in mammals] # This will return all carnivores Len iterates the queryset ------------------------- If you ever did `len(queryset)` it previously did a `count()` under the covers, this caused some unusual issues. As `len(queryset)` is most often used by `list(queryset)` we now cache the queryset results and use that for the length. This isn't as performant as a `count()` and if you aren't iterating the queryset you should upgrade to use count:: # Old code len(Animal.objects(type="mammal")) # New code Animal.objects(type="mammal").count() .only() now inline with .exclude() ---------------------------------- The behaviour of `.only()` was highly ambiguous, now it works in mirror fashion to `.exclude()`. Chaining `.only()` calls will increase the fields required:: # Old code Animal.objects().only(['type', 'name']).only('name', 'order') # Would have returned just `name` # New code Animal.objects().only('name') # Note: Animal.objects().only(['name']).only('order') # Now returns `name` *and* `order` Client ====== PyMongo 2.4 came with a new connection client; MongoClient_ and started the depreciation of the old :class:`~pymongo.connection.Connection`. MongoEngine now uses the latest `MongoClient` for connections. By default operations were `safe` but if you turned them off or used the connection directly this will impact your queries. Querysets --------- Safe ^^^^ `safe` has been depreciated in the new MongoClient connection. Please use `write_concern` instead. As `safe` always defaulted as `True` normally no code change is required. To disable confirmation of the write just pass `{"w": 0}` eg: :: # Old Animal(name="Dinasour").save(safe=False) # new code: Animal(name="Dinasour").save(write_concern={"w": 0}) Write Concern ^^^^^^^^^^^^^ `write_options` has been replaced with `write_concern` to bring it inline with pymongo. To upgrade simply rename any instances where you used the `write_option` keyword to `write_concern` like so:: # Old code: Animal(name="Dinasour").save(write_options={"w": 2}) # new code: Animal(name="Dinasour").save(write_concern={"w": 2}) Indexes ======= Index methods are no longer tied to querysets but rather to the document class. Although `QuerySet._ensure_indexes` and `QuerySet.ensure_index` still exist. They should be replaced with :func:`~mongoengine.Document.ensure_indexes` / :func:`~mongoengine.Document.ensure_index`. SequenceFields ============== :class:`~mongoengine.fields.SequenceField` now inherits from `BaseField` to allow flexible storage of the calculated value. As such MIN and MAX settings are no longer handled. .. _MongoClient: http://blog.mongodb.org/post/36666163412/introducing-mongoclient 0.6 to 0.7 ********** Cascade saves ============= Saves will raise a `FutureWarning` if they cascade and cascade hasn't been set to True. This is because in 0.8 it will default to False. If you require cascading saves then either set it in the `meta` or pass via `save` eg :: # At the class level: class Person(Document): meta = {'cascade': True} # Or in code: my_document.save(cascade=True) .. note:: Remember: cascading saves **do not** cascade through lists. ReferenceFields =============== ReferenceFields now can store references as ObjectId strings instead of DBRefs. This will become the default in 0.8 and if `dbref` is not set a `FutureWarning` will be raised. To explicitly continue to use DBRefs change the `dbref` flag to True :: class Person(Document): groups = ListField(ReferenceField(Group, dbref=True)) To migrate to using strings instead of DBRefs you will have to manually migrate :: # Step 1 - Migrate the model definition class Group(Document): author = ReferenceField(User, dbref=False) members = ListField(ReferenceField(User, dbref=False)) # Step 2 - Migrate the data for g in Group.objects(): g.author = g.author g.members = g.members g.save() item_frequencies ================ In the 0.6 series we added support for null / zero / false values in item_frequencies. A side effect was to return keys in the value they are stored in rather than as string representations. Your code may need to be updated to handle native types rather than strings keys for the results of item frequency queries. BinaryFields ============ Binary fields have been updated so that they are native binary types. If you previously were doing `str` comparisons with binary field values you will have to update and wrap the value in a `str`. 0.5 to 0.6 ********** Embedded Documents - if you had a `pk` field you will have to rename it from `_id` to `pk` as pk is no longer a property of Embedded Documents. Reverse Delete Rules in Embedded Documents, MapFields and DictFields now throw an InvalidDocument error as they aren't currently supported. Document._get_subclasses - Is no longer used and the class method has been removed. Document.objects.with_id - now raises an InvalidQueryError if used with a filter. FutureWarning - A future warning has been added to all inherited classes that don't define :attr:`allow_inheritance` in their meta. You may need to update pyMongo to 2.0 for use with Sharding. 0.4 to 0.5 ********** There have been the following backwards incompatibilities from 0.4 to 0.5. The main areas of changed are: choices in fields, map_reduce and collection names. Choice options: =============== Are now expected to be an iterable of tuples, with the first element in each tuple being the actual value to be stored. The second element is the human-readable name for the option. PyMongo / MongoDB ================= map reduce now requires pymongo 1.11+- The pymongo `merge_output` and `reduce_output` parameters, have been depreciated. More methods now use map_reduce as db.eval is not supported for sharding as such the following have been changed: * :meth:`~mongoengine.queryset.QuerySet.sum` * :meth:`~mongoengine.queryset.QuerySet.average` * :meth:`~mongoengine.queryset.QuerySet.item_frequencies` Default collection naming ========================= Previously it was just lowercase, it's now much more pythonic and readable as it's lowercase and underscores, previously :: class MyAceDocument(Document): pass MyAceDocument._meta['collection'] == myacedocument In 0.5 this will change to :: class MyAceDocument(Document): pass MyAceDocument._get_collection_name() == my_ace_document To upgrade use a Mixin class to set meta like so :: class BaseMixin(object): meta = { 'collection': lambda c: c.__name__.lower() } class MyAceDocument(Document, BaseMixin): pass MyAceDocument._get_collection_name() == "myacedocument" Alternatively, you can rename your collections eg :: from mongoengine.connection import _get_db from mongoengine.base import _document_registry def rename_collections(): db = _get_db() failure = False collection_names = [d._get_collection_name() for d in _document_registry.values()] for new_style_name in collection_names: if not new_style_name: # embedded documents don't have collections continue old_style_name = new_style_name.replace('_', '') if old_style_name == new_style_name: continue # Nothing to do existing = db.collection_names() if old_style_name in existing: if new_style_name in existing: failure = True print "FAILED to rename: %s to %s (already exists)" % ( old_style_name, new_style_name) else: db[old_style_name].rename(new_style_name) print "Renamed: %s to %s" % (old_style_name, new_style_name) if failure: print "Upgrading collection names failed" else: print "Upgraded collection names" mongodb 1.8 > 2.0 + =================== It's been reported that indexes may need to be recreated to the newer version of indexes. To do this drop indexes and call ``ensure_indexes`` on each model.