Protobuf-py: A Complete, Fast, and Idiomatic Protobuf Library for Python
Protobuf-py: Protobuf for Python, without compromises
We built protobuf-py to give Python developers a complete Protocol Buffers implementation that feels native. Unlike Google's package, which feels like a C++ binding, or betterproto, which sacrifices spec coverage, our library offers full compatibility with proto2, proto3, and editions. By keeping message data as plain Python objects and using an optional Rust accelerator, we achieve performance that beats the C engine in real-world production workloads while maintaining readable, typed code.
For years you took both or neither: the clumsy API and the speed shipped together, but we proved those constraints could all be true at once.
- jtbaker
Buf.build, never again. Had a proto evangelist get us ingratiated into their system a few years ago and then they introduced a bunch of rent seeking behavior just to be able to to the code generation, and getting it ripped out was a huge PITA. https://github.com/betterproto/python-betterproto2 was a decent (async, type hinted) python implementation last I checked.
- esrauch
Engineer who works on Google Protobuf here, commenting as myself and not as an official statement.
It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable with any compatible thing that can understand Protobuf encoding including this one.
Kudos to Buf for making something that I'm sure a ton of people will find useful and which takes conformance so seriously.
Just to chime in with some context about Google's own implementations here though (since that's a lot of the discussion otherwise).
Google definitely takes Protobuf seriously including for the long term: you can't really understand how engrained it is within the Google stack without seeing it for yourself. It's not just RPC layer, it's storage, logging, FFI. Html templating is driven off Protobuf messages. Systems which interact with bank XML based systems uses Protobuf schemas. Internally it's widely used for in-memory library api types even without any direct/obvious connection to serialization just because it makes internal details like logging easier. This extremely large surface does create constraints and use-cases to balance. You can see Buf's reported numbers reflect that it is faster for a usecase they expect is typical, but at scale users do fall into th […]
- tommek4077
Optimizing for the last microsecond and then use Python?
Why?
- usrnm
After gogoproto I'm hesitant to depend on another non-standard implementation, getting off gogo was a pain. This thing may be better than the one from Google (gogo definitely was), but can we be sure that it will still be around in 10 years?
- parhamn
Whats the case for protobuf these days? I loved it for a while. Don't dislike it particularly now (but I've rolled off).
- Performance? Fastest JSON lib in most languages are as fast if not faster
- Cross language schema generation? So many tools for this these days, they do one thing do them right (depending on your choice of 'right' for things like unions/enums/etc)
- The wire protocol? Seems to get in the way vs http2/3. Need special considerations for your proxies (be it nginx or cloudflare). Forced certs, etc are annoying too.
I feel like to most shops these days its mostly a schema manager? Protobuf is super bloated for that use case.