PYTHON / DATABASES WITH PYTHON
MongoDB documents and PyMongo
Model data as BSON documents, navigate PyMongo's client/database/collection handles, and predict which Python types survive a round trip.
What you will learn
- Derive database and collection handles from one MongoClient with no network I/O
- Predict BSON round trips: tuple becomes list, datetime loses microseconds
- Read an ObjectId: 12 bytes, 24 hex chars, embedded creation timestamp
- Serialize ObjectId and datetime with bson.json_util, not the json module
Understanding MongoDB documents and PyMongo
A MongoDB document is an ordered sequence of field/value pairs stored in a binary format called BSON, and a collection is just a bag of such documents with no declared schema. PyMongo's job is translation: on the way out it encodes a Python dict into BSON bytes, and on the way in it decodes BSON back into a dict. The container hierarchy mirrors that with three cheap handle objects, MongoClient -> Database -> Collection, none of which touch the network when you create them. That is why client["shop"]["orders"] never fails even if the server is down or the name is misspelled; the namespace only comes into existence when a write actually lands.
Because BSON, not Python, defines the storage types, the encode/decode cycle is lossy in specific ways worth memorising. BSON has one array type, so a tuple goes out and a list comes back; it has no set type at all, so a set raises InvalidDocument; its UTC datetime is a 64-bit millisecond count, so microseconds are truncated and naive datetimes are treated as UTC. Small ints become int32, large ones int64, and Python floats become BSON doubles, which is why money belongs in bson.decimal128.Decimal128 rather than float. The useful mental model: the dict you hold is a decoded snapshot, and only BSON-representable values can make the trip.
Every document needs a unique _id within its collection, and it is a real indexed field, not hidden metadata. If you omit it, PyMongo generates an ObjectId on the client before sending the document: 12 bytes made of a 4-byte Unix timestamp, a 5-byte per-process random value, and a 3-byte counter. That means ids are roughly time-ordered but never a dense sequence like an SQL AUTO_INCREMENT, and you can recover the creation second with ObjectId.generation_time without storing an extra field. You may also supply your own _id (an int, a string, a subdocument) when you have a natural key, and the server will reject a duplicate.
from bson import ObjectId, decode, encode
import datetime
doc = {
"_id": ObjectId("64b7f1b2c9e77a1f3c8d4e5a"),
"sku": "TS-1001",
"price": 19.99,
"stock": 42,
"tags": ("cotton", "summer"),
"added": datetime.datetime(2023, 7, 19, 12, 0, 0, 123456),
"supplier": {"name": "Acme", "rating": 4.5},
}
raw = encode(doc)
back = decode(raw)
print(len(raw), "bytes")
for key, value in back.items():
print(f"{key:9} {type(value).__name__:8} {value!r}")A MongoDB document is a BSON object that PyMongo surfaces as a dict, so BSON's type system and its _id rule, not Python's, decide what you can store.
Worked examples
Taking an ObjectId apart
Shows that an ObjectId is 12 raw bytes carrying its own creation timestamp, and that validity can be checked without a server.
from bson import ObjectId
oid = ObjectId("507f1f77bcf86cd799439011")
print(len(oid.binary))
print(oid.binary.hex())
print(oid.generation_time)
print(ObjectId.is_valid("not-an-id"))Example explained
Line 1oid.binary is the 12-byte value actually stored in the document; the 24-character hex string is only its printable form.
Line 2generation_time decodes the first 4 bytes as a Unix timestamp and returns a timezone-aware UTC datetime.
Line 3is_valid checks the string shape offline, which is exactly what you want when validating an id coming from a URL.
Line 4Nothing here contacts MongoDB: ids are minted and parsed entirely in the driver.
Documents are not plain JSON
Demonstrates why json.dumps chokes on a document and how bson.json_util represents BSON types as Extended JSON.
import json
from datetime import datetime, timezone
from bson import ObjectId
from bson.json_util import dumps
doc = {
"_id": ObjectId("64b7f1b2c9e77a1f3c8d4e5a"),
"name": "widget",
"created": datetime(2023, 7, 19, 12, 0, tzinfo=timezone.utc),
}
try:
json.dumps(doc)
except TypeError as exc:
print("json:", exc)
print(dumps(doc))Example explained
Line 1json.dumps only knows the JSON types, and ObjectId is not one of them, so it raises TypeError.
Line 2json_util.dumps emits Extended JSON, wrapping BSON-only types in $oid and $date so the type survives the trip.
Line 3The matching json_util.loads turns that text back into real ObjectId and datetime objects.
Line 4Field order in the output follows the dict order, because BSON documents are ordered.
Handles cost nothing
Creating a client, database, and collection object performs no I/O and no creation on the server.
from pymongo import MongoClient
client = MongoClient("mongodb://localhost:27017/", connect=False)
db = client["shop"]
orders = db.orders
same = db["orders"]
print(orders.full_name)
print(orders is same, orders == same)
print(db["order-items"].full_name)
print(client.get_database("shop").name)Example explained
Line 1connect=False proves the point: this script runs with no MongoDB server anywhere.
Line 2db.orders and db["orders"] build two distinct Collection objects that compare equal, since a handle is just a (database, name) pair.
Line 3Bracket syntax is required for names Python cannot parse as attributes, such as order-items.
Line 4full_name is the server-side namespace shop.orders, which is what appears in logs and error messages.
Important notes
One document is capped at 16 MB and about 100 levels of nesting; large blobs belong in GridFS or object storage with only a reference in the document.
Field names cannot contain a null byte, and names with dots or a leading $ make query paths and update operators ambiguous even where the server accepts them.
Common mistakes
Assuming client["shop"]["orders"] created something: a typo in either name is invisible until the first write, which then silently populates a brand-new empty namespace nobody queries.
Reusing the same dict for two insert_one calls; PyMongo adds the generated _id into your dict in place, so the second insert raises DuplicateKeyError.
Storing prices as Python floats and naive local datetimes, then wondering why totals drift by a cent and date ranges are off by the UTC offset.
Try it yourself
Change, predict, then run
Build a dict containing a set of tags and call bson.encode on it, catching bson.errors.InvalidDocument and printing the message; then convert the set to a sorted list and assert that decode(encode(doc)) equals the fixed document.
Open the Python workspaceCheck your understanding
You store {"_id": 1, "tags": ("new", "sale")} and later read the document back, where type(doc["tags"]) is list. What explains this?
- BSON has a single array type, so the tuple is encoded as an array and arrays always decode into Python lists
- PyMongo rejects tuples during validation and substitutes an empty list, which the server then fills in
- MongoDB stores the tuple's repr as a string and PyMongo parses it back into a list on read
- The server converts tuples to lists because tuples are immutable and cannot be updated in place
Show answer
Encoding is lossy whenever two Python types share one BSON type: both list and tuple become a BSON array, and decoding has only one sensible target, list. The immutability answer is tempting but wrong about where the conversion happens; the server never sees Python objects at all, since the driver has already turned the document into BSON bytes before sending it.