The S3 Gateway, Part 1 — Why We Built and Didn’t Buy

← Blog/Engineering//1 min read

“S3-compatible” is the most stretched phrase in object storage. Almost every product can show a bucket create and an object upload. The phrase is only worth something if it carries one property all the way down: a success response means the thing happened. When the gateway says 200 to “enable versioning,” the bucket is versioned. When it says 200 to “set retention,” the object cannot be deleted until the date passes. When it reports usage, those are real numbers.

That property is what every tool above the storage layer quietly depends on. Your backup tool believes deletes are recoverable because versioning said yes. Your compliance tooling believes retention is set because the call succeeded. Your billing dashboard believes the usage figures. If a gateway’s yes doesn’t mean yes, every one of those layers is wrong and nobody finds out until a restore or an audit.

So that became the first rule of the XNS S3 Gateway: the endpoint existing and the endpoint working are the same statement. If something isn’t built yet, the gateway says so with an honest error. It never accepts a request it can’t honor.


The Relayer’s original S3 layer was a frozen third-party fork that could not take the product where storage customers are in 2026. We replaced it from scratch rather than patch it, for three reasons.

1. The roadmap customers expect. Every capability a storage buyer asks for now — object versioning, WORM/Object Lock with compliance-mode retention, real IAM with temporary credentials and OIDC federation, multi-tenant isolation, checksum negotiation with modern SDKs — has to be designed in, not bolted on. Building our own gateway meant each of those could be built to the same rules, by the team that answers for it.

2. Trust has to be designed in. The “a yes means yes” property above can’t be patched into a codebase after the fact; every handler has to be written against it. Starting clean meant writing every handler to one rule and testing every handler against that rule.

3. We own every line. A gateway is a security boundary. Owning the code means we can audit it, patch it the day a new vulnerability class makes the news, and ship the fix on our schedule, with no upstream to wait on and no inherited code nobody on the team has read.


Clean-room, in Go, from the protocol down. The gateway shares no code with what came before. That sounds like machismo; it’s actually the cheap option once you accept reason #2 — if every handler has to be written against “success implies enforcement,” you are writing every handler anyway.

Target the tools, not the spec. The S3 “spec” is really the union of what AWS’s own SDKs do, and the failure surface of every S3-compatible product is the gap between the documented API and the defaults real tools ship with. So the compatibility target is behavioral: AWS CLI v2, boto3, rclone, restic, kopia, MinIO’s own mc client, Terraform’s AWS provider, and the Ceph s3tests conformance suite — run against the gateway, defaults on. Modern SDK defaults are aggressive (chunked uploads with trailing checksums, automatic integrity checking, virtual-hosted addressing), and a gateway that passes yesterday’s spec fails today’s client out of the box. Part 2 is largely the war stories from this.

Contract-driven, with the contract in version control. The gateway’s compatibility surface is documented in a code-verified operation matrix: today, 70 routes — 64 real handlers plus exactly 6 deliberate 501 stubs (website hosting and cross-region replication, three verbs each). The route count is machine-checked — a router test asserts the table size, so the documentation and the binary can’t drift apart silently. Each row in the contract carries a status, and the status legend is the design philosophy in miniature:

StatusMeaning
✅ WorkingReal handler with persistence and, where it applies, enforcement. Behavior matches AWS S3 for the operations in our test suite.
🟠 PartialReal handler; a documented sub-feature diverges from AWS. The divergence is written down per-row.
⛔ StubRoute registered; returns a well-formed 501 in the S3 XML error envelope. Deliberate, documented.
❌ Not exposedNo route; clients get 404. Also deliberate, also documented.

There is no “returns 200 but does nothing” row, because we don’t ship that state. Where a feature isn’t ready, the answer is an honest 501 or an honest 404 — and every stub row in the contract requires a test that asserts the current behavior, so even our divergences can’t change silently.

Fail-closed as posture, not feature. The policy engine denies on any unsupported condition operator or unresolvable principal rather than skipping the check. Authentication-layer JWT audience enforcement refuses to start the server rather than run open. The credential tables use database-level row security, so a misconfigured query returns zero rows instead of every tenant’s secrets. The uniform rule: when a component can’t establish that access is allowed, the answer is no.

And underneath: the gateway is the top of a stack, not the product. Worth one paragraph of context. The gateway translates S3 into the language of a distributed storage network: objects are erasure-coded into 80 data + 40 parity sectors across 120 independent provider nodes spread across roughly 48 countries, with continuous health monitoring and automatic repair. Losing 40 of the 120 nodes simultaneously still recovers everything. The storage layer is always-encrypted before sharding, independent of any S3-level SSE configuration. That’s the asset that justifies this much gateway effort — the durability story deserved a protocol surface whose answers can be trusted. The deep dive on the storage layer is its own post.


Part 2 is the build itself: the SDK defaults that broke first, the listing query that went from 1.8 seconds to 3.5 milliseconds, the memory ceilings we designed out, the test habit we adopted (assert state after the call, never just the status code), and the full inventory of what shipped: IAM with 27 AWS query-protocol verbs on the S3 port, STS with verified OIDC federation, per-tenant isolation, WORM, and the operational surface.

If you want the receipts now, the full compatibility contract, route by route, is published at S3-Compatibility Chart – How Does XNS Stand Up?.

I’m proud of what we’ve built, but the proof is in the sauce, give it a try!


© Copyright - SCP, Corp | Xa Net Services and Affiliates