protobuf vs gRPC

130
votes

I try to understand protobuf and gRPC and how I can use both. Could you help me understand the following:

  • Considering the OSI model what is where, for example is Protobuf at layer 4?
  • Thinking through a message transfer how is the "flow", what is gRPC doing what protobuf misses?
  • If the sender uses protobuf can the server use gRPC or does gRPC add something which only a gRPC client can deliver?
  • If gRPC can make synchronous and asynchronous communication possible, Protobuf is just for the marshalling and therefore does not have anything to do with state - true or false?
  • Can I use gRPC in a frontend application communicating instead of REST or GraphQL?

I already know - or assume I do - that:

Protobuf

  • Binary protocol for data interchange
  • Designed by Google
  • Uses generated "Struct" like description at client and server to un-/-marshall message

gRPC

  • Uses protobuf (v3)
  • Again from Google
  • Framework for RPC calls
  • Makes use of HTTP/2 as well
  • Synchronous and asynchronous communication possible

I again assume its an easy question for someone already using the technology. I still would thank you to be patient with me and help me out. I would also be really thankful for any network deep dive of the technologies.

3
networkingprotocol-buffersgrpc
I feel a need to throw in that RPC has been around since the 1980s, lest someone be mistaken for thinking the idea was first conceived by engineers under Google. – John Mee
I assumed that's clear for everyone having a computer science degree? Does my question convey a different image? Should I adjust something? – lony

3 Answers

111
votes

Protocol buffers is (are?) an Interface Definition Language and serialization library:

  • You define your data structures in its IDL i.e. describe the data objects you want to use
  • It provides routines to translate your data objects to and from binary, e.g. for writing/reading data from disk

gRPC uses the same IDL but adds syntax "rpc" which lets you define Remote Procedure Call method signatures using the Protobuf data structures as data types:

  • You define your data structures
  • You add your rpc method definitions
  • It provides code to serve up and call the method signatures over a network
  • You can still serialize the data objects manually with Protobuf if you need to

In answer to the questions:

  1. gRPC works at layers 5, 6 and 7. Protobuf works at layer 6.
  2. When you say "message transfer", Protobuf is not concerned with the transfer itself. It only works at either end of any data transfer, turning bytes into objects
  3. Using gRPC by default means you are using Protobuf. You could write your own client that uses Protobuf but not gRPC to interoperate with gRPC, or plugin other serializers to gRPC - but using gRPC would be easier
  4. True
  5. Yes you can
79
votes

Actually, gRPC and Protobuf are 2 completely different things. Let me simplify:

  • gRPC manages the way a client and a server can interact (just like a web client/server with a REST API)
  • protobuf is just a serialization/deserialization tool (just like JSON)

gRPC has 2 sides: a server side, and a client side, that is able to dial a server. The server exposes RPCs (ie. functions that you can call remotely). And you have plenty of options there: you can secure the communication (using TLS), add authentication layer (using interceptors), ...

You can use protobuf inside any program, that has no need to be client/server. If you need to exchange data, and want them to be strongly typed, protobuf is a nice option (fast & reliable).

That being said, you can combine both to build a nice client/server sytem: gRPC will be your client/server code, and protobuf your data protocol.

PS: I wrote this paper to show how one can build a client/server with gRPC and protobuf using Go, step by step.

8
votes

grpc is a framework build by google and it is used in production projects from google itself and #HyperledgerFabric is built with grpc there are many opensource applications built with grpc

protobuff is a data representation like json this is also by google in fact they have some thousands of proto file are generated in their production projects

grpc

  • gRPC is an open-source framework developed by google
  • It allows us to create Request & Response for RPC and handle rest by the framework
  • REST is CRUD oriented but grpc is API oriented(no constraints)
  • Build on top of HTTP/2
  • Provides >>>>> Auth, Loadbalancing, Monitoring, logging
  • [HTTP/2]
    • HTTP1.1 has released in 1997 a long time ago
    • HTTP1 opens a new TCP connection to a server at each request
    • It doesn't compress headers
    • NO server push, it just works with Req, Res
    • HTTP2 released in 2015 (SPDY)
    • Supports multiplexing
    • client & server can push messages in parallel over the same TCP connection
    • Greatly reduces latency
    • HTTP2 supports header compression
    • HTTP2 is binary
      • proto buff is binary so it is a great match for HTTP2
  • [TYPES]
    • Unary
    • client streaming
    • server streaming
    • Bi directional streaming
    • grpc servers are Async by default
    • grpc clients can be sync or Async

protobuff

  • Protocol buffers are language agnostic
  • Parsing protocol buffers(binary format) is less CPU intensive
  • [Naming]
    • Use camel case for message names
    • underscore_seperated for fields
    • Use camelcase for Enums and CAPITAL_WITH_UNDERSCORE for value names
  • [Comments]
    • Support //
    • Support /* */
  • [Advantages]
    • Data is fully Typed
    • Data is fully compressed (less bandwidth usage)
    • Schema(message) is needed to generate code and read the code
    • Documentation can be embedded in the schema
    • Data can be read across any language
    • Schema can evolve any time in a safe manner
    • faster than XML
    • code is generated for you automatically
    • Google invented proto buff, they use 48000 protobuf messages & 12000.proto files
    • Lots of RPC frameworks, including grpc use protocol buffers to exchange data