← Back to API docs

Datastore API

The Datastore API is a managed JSON document store with a bounded, App Engine Datastore–style query engine — multi-field filters, sorting, aggregates, and atomic transactions — without exposing raw SQL. Each namespace is an isolated, strongly consistent store, so reads always see the latest committed writes. All paths below are relative to a datastore instance you provision in the console.

Authentication and base URL

Send requests to https://api.altengine.net with an organization API key as a bearer token. Reads require a read grant, writes a write grant, and deletes a full grant — see Authentication.

Authorization: Bearer ae_yourkeyid.your-api-key-secret

Every path is scoped to an instance: /v1/datastore/{instance}/…. Data is organized as instance → namespace → collection → document. A namespace is an isolated store (up to 10 GB) that you create on demand — ideal for one-database-per-tenant designs; a collection is a named group of documents within it. Namespaces and collections are created automatically on first write.

Datastore requests are rate-limited per instance to a sustained 100 requests per second (6,000 per minute) by default, with a matching burst; you can set a lower limit per instance in the console (Datastore → your instance → Settings). Over the limit, a request returns 429 with a RATE_LIMITED code; back off and retry.

Endpoints

Documents

A document is any JSON object, addressed by a key that is unique within its collection. Objects may nest freely; there is no schema to declare up front.

POST /v1/datastore/app/namespaces/acme/collections/users/documents
{
  "documents": [
    { "key": "u1", "data": { "name": "Ada", "age": 36, "addr": { "city": "SF" } } }
  ]
}
# → { "keys": ["u1"] }

Putting a document with an existing key replaces it. Send up to 500 documents per request.

Keys: strings or numbers

Keys are strings — you can use any UTF-8 string (1–512 bytes, no NUL byte): a slug, an email, a UUID, whatever you like. A JSON number is also accepted and is stored as its decimal string, so 5 and "5" address the same document. (There is no separate integer-key type — every key is ultimately a string.)

{ "documents": [
  { "key": "[email protected]", "data": { "plan": "pro" } },
  { "key": 1042,              "data": { "sku": "A-1042" } }   // stored as "1042"
] }

Auto-generated keys

Omit key and the instance generates one for you; the strategy is set per instance in the console (Datastore → your instance → Settings). The generated keys come back in the keys array, in input order.

Get, batch get, and delete

# Get one document
GET /v1/datastore/app/namespaces/acme/collections/users/documents/u1
# → { "document": { "key": "u1", "data": { … } } }

# Get many documents in one request (found ones only; up to 500 keys)
POST /v1/datastore/app/namespaces/acme/collections/users/documents/batchGet
{ "keys": ["u1", "u2", "u3"] }
# → { "documents": [ { "key": "u1", "data": { … } }, … ] }

# Batch delete by key (requires a full grant)
POST /v1/datastore/app/namespaces/acme/collections/users/documents/delete
{ "keys": ["u1", "u2"] }
# → { "deleted": 2 }

Caching pattern: run a query with "keys_only": true to fetch just the matching keys (a small, cheap payload), keep a client-side cache of documents you've already loaded, then batchGet only the keys you're missing.

Query

Post a structured query to a collection. where filters, order sorts, limit bounds the page, and cursor pages forward. Fields are dot-paths into the document (nested paths like addr.city work); the reserved selectors __key__, __created__, and __updated__ sort by key and by write time.

POST /v1/datastore/app/namespaces/acme/collections/orders/query
{
  "where": [
    { "field": "status", "op": "=",  "value": "open" },
    { "field": "total",  "op": ">",  "value": 100 }
  ],
  "order": [{ "field": "__updated__", "dir": "desc" }],
  "limit": 25
}
# → { "documents": [ { "key": "…", "data": { … } } ], "cursor": "…"|null }

To page, pass the returned cursor back on the next request; when it is null there are no more results. Supported operators: =, !=, <, <=, >, >=, and in (a non-empty array). Add "keys_only": true to return just the matching keys ({ "keys": [ … ], "cursor": … }) instead of full documents — a lighter response you can pair with batchGet for client-side caching. A query page is capped at 500 documents (default 25). Repeated identical queries are served from a fast in-memory cache that a write to the same collection immediately invalidates, so you never read stale data.

Joins

Enrich each result with a referenced document from another collection via join. Each join reads the value at local_field and looks up that key in the foreign collection, attaching the result under as. The lookup is by the foreign collection's key (its primary key), so a join is always index-served — never a full scan — and bounded by the page size. Up to 5 joins per query.

POST /v1/datastore/app/namespaces/acme/collections/orders/query
{
  "where": [{ "field": "status", "op": "=", "value": "open" }],
  "order": [{ "field": "total", "dir": "desc" }],
  "join": [
    { "as": "customer", "collection": "users", "local_field": "user_id" }
  ]
}
# → { "documents": [
#      { "key": "o1", "data": { "user_id": "u1", … },
#        "joins": { "customer": { "key": "u1", "data": { "name": "Ada" } } } }
#    ], "cursor": … }

When the referenced document doesn't exist (or the local field is absent), joins.<as> is null. Joins are only for where/order queries, not keys_only ones.

Indexes

Queries are index-served: a query that a declared index cannot satisfy returns 400 with an INDEX_REQUIRED code naming the field to index — this keeps every query fast and cost-bounded (no accidental full-collection scans). A plain listing (no filter, no sort) needs no index.

Declare an index on one or more fields. Single-field indexes are enough to filter on several fields at once — the engine seeks one and filters the rest. A composite index (multiple fields) additionally serves a query that filters on the leading field(s) and sorts by the next, and backs grouped aggregates. Add "unique": true to enforce uniqueness of the field value within the collection.

# Declare indexes
POST /v1/datastore/app/namespaces/acme/collections/orders/indexes
{ "fields": ["status"] }                 # single-field
# and, for filter-by-status + sort-by-total:
{ "fields": ["status", "total"] }        # composite

# List / drop
GET    /v1/datastore/app/namespaces/acme/collections/orders/indexes
DELETE /v1/datastore/app/namespaces/acme/collections/orders/indexes/{id}

You can also manage indexes visually in the console (Datastore → your instance → Indexes). Declaring an index on a populated collection builds it once over the existing documents.

Auto-indexing (on by default). If a query or aggregate needs an index that doesn't exist yet, the engine creates the one it suggests and runs your request — so you never hit INDEX_REQUIRED while building. The one-time build is billed as writes (one row per existing document), and the response includes "auto_indexed": { "fields": [ … ] } so it's never silent. Turn it off per instance (console → Datastore → your instance → Settings → uncheck Automatically create missing indexes) to instead get a 400 INDEX_REQUIRED naming the field to add — the right choice in production, where you want to decide exactly which indexes exist rather than have queries create them for you.

Aggregates & metrics

Compute grouped metrics without reading every document. metrics lists the aggregates (count, sum, avg, min, max), group lists the dimension fields, and where filters first. Grouped and filtered fields must be index-served, same as queries.

POST /v1/datastore/app/namespaces/acme/collections/orders/aggregate
{
  "group": ["region"],
  "metrics": [
    { "fn": "count",              "as": "orders" },
    { "fn": "sum", "field": "total", "as": "revenue" }
  ],
  "order": [{ "field": "revenue", "dir": "desc" }]
}
# → { "groups": [ { "group": { "region": "west" }, "metrics": { "orders": 2, "revenue": 240 } } ] }

Cost & indexing tradeoffs

Datastore bills on three axes — reads, writes, and stored data — and indexing is the lever that moves all three. A minute spent here keeps your bill predictable; see the pricing page for exact rates.

So indexing is a two-sided lever, and the goal is the middle:

• Under-indexing makes queries scan (or fail with INDEX_REQUIRED): slow, and read-expensive because you pay for every row examined.
• Over-indexing makes every write more expensive and grows storage. An index on a field you never filter, sort, or group by is pure cost with no benefit.

Rule of thumb: index what you query — and nothing more. Declare an index for each field (or field combination) you actually filter, sort, or group on. Prefer a composite index when you filter and sort together — one composite is cheaper to maintain than several single-field indexes. Drop indexes you've stopped querying. And because reads are priced by rows examined, a selective filter on an indexed field is not just faster — it's directly cheaper than a broad one that scans the collection.

This is also the tradeoff behind auto-indexing (see Indexes above): leaving it on is convenient in development but can create indexes you didn't plan — each one adding write and storage cost — so in production many teams turn it off and declare indexes deliberately.

Transactions

Run several operations atomically within one namespace — they all commit or none do. Transactions within a namespace are strongly consistent: everything commits together or the whole transaction aborts. Operations run in order:

POST /v1/datastore/app/namespaces/acme/transaction
{
  "operations": [
    { "op": "check",  "collection": "accounts", "key": "a1", "exists": true },
    { "op": "mutate", "collection": "accounts", "key": "a1", "increment": { "balance": -30 } },
    { "op": "put",    "collection": "ledger",   "data": { "delta": -30 } }
  ]
}
# → { "keys": [null, "a1", "…"] }

Atomic increment is race-free — concurrent increments to the same field never lose an update.

Backups & restore

Every namespace has continuous point-in-time recovery for the last 30 days at no extra charge. From the console (Datastore → your instance → Backups) you can copy a namespace's current snapshot and restore the namespace to a past timestamp or a saved snapshot — useful for undoing a bad deploy or bulk edit.

Limits

Errors

Errors use standard HTTP status codes and the same JSON shape as the rest of the API. Beyond the common codes, Datastore adds: