Caching

Two-Layer Cache

Sub-millisecond reads. Cross-process sharing. Stale-while-revalidate.

L1 catches repeated reads in microseconds. L2 keeps all instances in sync. SWR keeps responses fast during TTL refresh.

Incoming Request

GET /v1/{service}/{method}

↓

L1 — In-Memory (cachetools)

LRU cache · per-process · <1ms lookup · max 1000 entries · TTL 60s

HIT

Return immediately

MISS

Check L2 →

↓ on L1 miss

L2 — Redis DB 0 (stale-while-revalidate)

Shared across all instances · ~5ms lookup · TTL 300s · SWR grace 60s

HIT (fresh)

Return + populate L1

HIT (stale)

Serve stale + bg refresh

MISS

Call handler →

↓ on L2 miss

Service Handler

@service_method(verb="GET") → adapter → external API

↓

Store result in L1 (60s TTL) + L2 (300s TTL + 60s SWR grace) → Return response

Cache-Control: max-age=300, stale-while-revalidate=60

Set on all cacheable responses

What & Why

Octopus uses a two-layer cache to eliminate redundant external API calls across repeated reads. Most high-frequency reads — listing repositories, fetching calendar events, querying issues — are identical within a short burst window. Without a cache, each of those hits a rate-limited third-party API.

Layer	Scope	Latency	TTL
L1 — cachetools LRU	Single process	<1ms	60s
L2 — Redis DB 0	All instances	~5ms	300s + 60s SWR

Only read methods are cached. Methods decorated with @service_method(verb="GET") or verb="QUERY" are eligible. Create, update, and delete methods (POST, PATCH, PUT, DELETE) are never cached — they bypass both layers entirely.

Stale-while-revalidate keeps p99 latency flat even during TTL refresh. When a Redis entry has expired but is within the 60-second grace window, Octopus serves the stale value instantly and fires a background task to refresh the cache. The caller never waits for the refresh.

---

Cache Key Anatomy

Tenant-scoped. Param-canonical. Collision-free.

Keys are deterministic: the same call with the same params always maps to the same key, across all instances.

prefix

cache:

tenant_id

acme:

multi-tenant isolation

service

github:

method

list_repositories:

params_hash

a1b2c3d4

MD5 / SHA256

Full key example

cache:acme:github:list_repositories:a1b2c3d4

Param Canonicalization

{"org": "acme", "limit": 10}

→

sort keys

{"limit": 10, "org": "acme"}

→

JSON serialize

'{"limit": 10, "org": "acme"}'

→

MD5/SHA256

a1b2c3d4...

Tenant Isolation

Tenant A (acme)

cache:acme:github:list_repositories:a1b2c3d4

Tenant B (globex)

cache:globex:github:list_repositories:a1b2c3d4

Same method + same params — different keys. Tenant A can never read Tenant B's cached data.

Stale-While-Revalidate Timeline

Responses stay fast during TTL refresh — no blocking refresh cycle.

Entries expire at t=300s. Stale entries are still served until t=360s while a background task refreshes the cache.

t=0

t=300s

t=360s

→

Fresh

Serve from cache instantly

Stale

Serve stale + bg refresh

Evicted

Full cache miss

0s – 300s · Fresh

Serve from cache

Entry exists in Redis and is within its TTL. Return immediately. No external API call. L1 may also serve it in <1ms if populated.

300s – 360s · Stale

Serve stale + background refresh

TTL has expired but the entry is within the 60-second SWR grace window. Serve the stale value to the caller immediately. Spawn a background task to refresh from the external API and update Redis.

After 360s · Evicted

Full cache miss

Entry has been evicted from Redis. The request falls through to the service handler, calls the external API, and stores the fresh result back in L1 + L2.

Cache Bypass Rules

Not all requests are cached. Octopus respects standard Cache-Control semantics and method verb annotations.

Condition	L1 lookup	L2 lookup	L1 store	L2 store
Normal GET request	Yes	Yes	Yes	Yes
Cache-Control: no-cache header	Skip	Skip	Yes	Yes
Cache-Control: no-store header	Skip	Skip	Skip	Skip
POST / PATCH / PUT / DELETE method	Never	Never	Never	Never
@service_method(verb="POST") or mutating verb	Never	Never	Never	Never

no-cache forces a fresh fetch but still stores the result — useful for getting a guaranteed fresh value while keeping the cache warm for subsequent callers. no-store opts out of caching entirely.

---

Cache Invalidation

Method	How
TTL expiry	Automatic — entries expire after 300s and are evicted after the 60s SWR grace window.
Explicit flush	DELETE /v1/cache/{service} — clears all cache entries for a service across all tenants.
Process restart	L1 (in-memory) is wiped. L2 (Redis) persists across restarts.

There is no event-based invalidation. The system uses an eventual consistency model: mutations do not automatically evict related cache entries. If you need immediate consistency after a write, issue a DELETE /v1/cache/{service} call or use Cache-Control: no-cache on the next read.

---

Reference

Headers, endpoints, and environment variables

Request Headers

Header	Effect
Cache-Control: no-cache	Skip L1 + L2 lookup; store result after fetch
Cache-Control: no-store	Skip lookup and storage entirely
Cache-Control: max-age=N	Respected on responses; not used to override TTL on requests

Cache Invalidation Endpoint

DELETE/v1/cache/{service}

Clears all cached entries for the specified service. Requires a valid auth token. Use after bulk writes or when forcing a cache flush for a specific integration.

# Example

curl -X DELETE /v1/cache/github

-H "Authorization: Bearer $TOKEN"

Environment Variables

Variable	Default	Description
CACHE_TTL	300	L2 Redis TTL in seconds (fresh window)
CACHE_GRACE	60	SWR grace window in seconds (stale-while-revalidate duration)
CACHE_MAX_SIZE_L1	1000	Maximum entries in the per-process LRU cache (L1)
CACHE_REDIS_DB	0	Redis database index for L2 cache (DB 1 is used by Sagittarius)