Review of ATProto data model change
2026-11-02 21:13
This blog post is a review of a new floats proposal that Bryan Newbold blogged about. It’s very different from mine and hence deserves a deeper look. For the rest of this post I expect you to have read both blog posts.
My first reaction was: no way. You would inherit all the problems JSON has with numbers, which is part of the reason we are discussing all this. But after some thinking I realized that the proposal is very different from how I’ve been thinking about this problem space over the past few years, and that it could work and would be worth doing.
My previous thinking
When I was thinking about float support in ATProto over the past few months, it became clear to me that what I really want is to store (almost) arbitrary JSON as CBOR records. At the moment the ATProto data model only supports integers, so only a fraction of all JSON numbers can be used.
Usually (besides JS) programming languages have a distinction between integers and floats deeply integrated into their native type system, which makes sense as they behave very differently. That distinction was also integrated into the IPLD data model, from which the ATProto data model is derived. So the obvious choice for me was to add a float type there as well and call it a day.
For IPLD, CBOR is the serialization format to use. We made sure that our subset called DAG-CBOR serializes things deterministically so that it can be used for content-addressing. The current IETF ATP WG drafts also make CBOR core to ATProto. When you want to verify something, you take the CBOR and hash it. You would use the JSON representation as a starting point and maybe for debugging. So there needs to be a bidirectional mapping, and baking it into the JSON number formatting is fine.
I think that’s a coherent view and it makes sense. It’s based on the assumption that you produce the data with some programming language and the source of truth is always the CBOR serialization, nowhere else.
What changed
When @bnewbold.net posted a new proposal to the IETF AT WG mailing list, it got me thinking. Although I knew about the requirement to transform JSON back to the original CBOR without a schema, I was under the impression that you always control the JSON generation. My proposal works in that case.
But the requirements are different. JSON should be a first-class primitive and CBOR is no longer the single source of truth. It’s still a bit more nuanced: no one plans to hash the JSON directly, as that wouldn’t reliably produce the same hashes for the same data for various reasons (especially floats). By first-class primitive I mean that the JSON to CBOR conversion also needs to work with third-party systems that are not under your control.
My roots are in the database world at the storage level. Although I’m also a huge fan of JSON (hello CouchDB), the distinction between floats and integers has always felt clear and obvious to me. I would build systems around those primitives. I want my arrays of numbers to have a single type.
A different data model
The new proposal is about changing the data model ATProto uses. Instead of having only integers, or having integers and floats as I proposed, you’d have a single number type similar to JSON. As mentioned, it totally breaks my mental model of how I think about such things.
But the more I thought about it, the more I realized that it’s just a continuation of what already happens even in DAG-CBOR. There, an integer is always stored in its shortest possible representation. So if you define a 64-bit integer with a value of 87 in your favourite strictly typed programming language, it’s serialized as an 8-bit integer. That conversion is lossless, hence when you request your 64-bit value again, you get what you expect. Storing it minimally encoded is just an implementation detail that doesn’t matter when interacting with the data.
So if you think about the concept of a number as a union of integers and floats, you can do the same. How it’s serialized is an implementation detail. You store your 64-bit float value 31.0 and get it serialized. It doesn’t matter whether it’s stored as a CBOR 64-bit float or as an 8-bit integer as long as the conversion is lossless. For content-addressing it’s important that the serialization is deterministic. Ideally a single value has only a single way to be serialized. The new proposal achieves this by forbidding floats that can be represented as integers without loss from being stored as floats.
What are the implications
The change to the data model might seem minor, but it’s a huge change. So far the data model has been a strict subset of IPLD, which I’ve been happy about. Having worked for so many years on IPLD, it’s been great to see such a huge application of it in the wild. I expected the Bluesky team to be open to my proposal and not to put too much weight on the current implementation. So I should also be open to breaking with IPLD too, if it makes for a better system.
For me the good news is that the serialized CBOR will remain compatible with IPLD. It would just be a stricter subset. This means all IPLD-compatible software can still work with ATProto’s data, you would just need special encoders if you want to produce ATProto-compatible data.
What does it mean for DRISL? I don’t know yet. I lean towards adapting it to this change. For me it never really made sense to have both DRISL and DAG-CBOR, which are almost identical. This change would make DRISL a stricter subset of DAG-CBOR.
Thanks
I’d like to thank everyone who was involved in the discussion, as well as the folks from Bluesky for thinking this through and providing a proposal that makes it possible to store (almost) arbitrary JSON data as ATProto records without further processing.