api2 – Our Solution to Facilitate API Development

api2 streamlines development, and has a place in most Go API development

Pavel Dolgov - Lead Core Engineer

Building and extending HTTP APIs is a part of developing Go backends which we need to do almost daily. It is a repetitive task which requires a lot of toil (routine work) if done in a standard way. We needed to come up with a more systematic approach to avoid spending time on routine work. Boris, one of the Relayer developers, wrote a library to simplify and speed up API development. The goal was to hide encoding/decoding, binding handlers to routes and other routine work to let developer focus on business logic, while leaving options to configure and customize what is done by the library. As a result, a developer just needs to define data structures for request and response types and a server handler with business logic, and the tool automatically generates code to make HTTP client and server for them. Therefore, making remote calls and serving them is as simple for the developer as internal calls between packages inside the same program on one machine.

Virtually any HTTP server developer needs to solve this problem, so a wide group of users might find the library helpful. 

api2 has some interesting additional features, such as automatic OpenAPI spec generation (swagger compatible), Protobuf format support, support for custom encoding, and more. It became our internal standard on backend and is used by all parts of the project.

Here is more detailed description of the api2 library:

Package api2 provides types and functions used to define interfaces of client-server API and facilitate creation of server and client for it. You define a common structure (GetRoutes, see below) in Go and api2 makes both HTTP client and HTTP server for you. You do not have to do JSON encoding-decoding yourself or duplicate schema information (data types and path) in client.

Usage

Organize your code in services. Each service provides some domain specific functionality. It is a Go type whose methods correspond to exposed RPCs of the API. Each method has the following signature:

func(ctx, *Request) (*Response, error)

Let’s define a service Foo with method Bar.

type Foo struct {
 ...
}
type BarRequest struct {
 // These fields are stored in JSON format in body.
 Name string `json:"name"`
 // These fields are GET parameters.
 UserID int `query:"user_id"`
 // These fields are headers.
 FileHash string `header:"file_hash"`
 // These fields are cookies.
 Foo string `cookie:"foo"`
 // These fields are skipped.
 SkippedField int `json:"-"`
}
type BarResponse struct {
 // These fields are stored in JSON format in body.
 FileSize int `json:"file_size"`
 // These fields are headers.
 FileHash string `header:"file_hash"`
 // These fields are skipped.
 SkippedField int `json:"-"`
}
func (s *Foo) Bar(ctx context.Context, req *BarRequest) (*BarResponse, error) {
 ...
}

A field must not have more than one of tags: json, query, header, cookie. Fields in query, header and cookie parts are encoded and decoded with fmt.Sprintf and fmt.Sscanf. Strings are not decoded with fmt.Sscanf, but passed as is. Types implementing encoding.TextMarshaler and encoding.TextUnmarshaler are encoded and decoded using it. If no JSON field is in the struct, then HTTP body is skipped.

If you need the top-level type matching body JSON to not be a struct, but of some other kind (e.g. slice or map), you should provide a field in your struct with tag use_as_body:"true":

type FooRequest struct {
	// Body of the request is JSON array of strings: ["abc", "eee", ...].
	Body []string `use_as_body:"true"`

	// You can add 'header', 'query' and 'cookie' fields here, but not 'json'.
}

If you use use_as_body:"true", you can also set is_protobuf:"true" and put a protobuf type (convertible to proto.Message) in that field. It will be sent over wire as protobuf binary form.

Now let’s write the function that generates the table of routes:

func GetRoutes(s *Foo) []api2.Route {
	return []api2.Route{
		{
			Method:    http.MethodPost,
			Path:      "/v1/foo/bar",
			Handler:   s.Bar,
			Transport: &api2.JsonTransport{},
		},
	}
}

You can add multiple routes with the same path, but in this case their HTTP methods must be different so that they can be distinguished.

If Transport is not set, DefaultTransport is used which is defined as &api2.JsonTransport{}.

In the server you need a real instance of service Foo to pass to GetRoutes. Then just bind the routes to http.ServeMux and run the server:

// Server.
foo := NewFoo(...)
routes := GetRoutes(foo)
api2.BindRoutes(http.DefaultServeMux, routes)
log.Fatal(http.ListenAndServe(":8080", nil))

The server is running. It serves foo.Bar function on path /v1/foo/bar with HTTP method Post.

Now let’s create the client:

// Client.
routes := GetRoutes(nil)
client := api2.NewClient(routes, "http://127.0.0.1:8080")
barRes := &BarResponse{}
err := client.Call(context.Background(), barRes, &BarRequest{
	...
})
if err != nil {
	panic(err)
}
// Server's response is in variable barRes.

Note that you don’t have to pass a real service object to GetRoutes on client side. You can pass nil, it is sufficient to pass all needed information about request and response types in the routes table, that is used by client to find a proper route.

You can make GetRoutes accepting an interface instead of a concrete Service type. In this case you can not get method handlers by s.Bar, because this code panics if s is nil interface. As a workaround api2 provides function Method(service pointer, methodName) which you can use:

type Service interface {
	Bar(ctx context.Context, req *BarRequest) (*BarResponse, error)
}

func GetRoutes(s Service) []api2.Route {
	return []api2.Route{
		{Method: http.MethodPost, Path: "/v1/foo/bar", Handler: api2.Method(&s, "Bar"), Transport: &api2.JsonTransport{}},
	}
}

If you have function GetRoutes in package foo as above you can generate static client for it in file client.go located near the file in which GetRoutes is defined:

api2.GenerateClient(foo.GetRoutes)

You can find an example in directory example. To build and run it:

$ go get github.com/starius/api2/example/...
$ server &
$ client
test
87672h0m0s

Code generation code is located in directory example/gen. To regenerate file client.go run:

$ go generate github.com/starius/api2/example

Real World Example

Finally, a simple code sample from the real codebase is provided. It is taken from a bigger commit which switches one of our internal services to api2. One of the handlers is rewritten using api2:

- func (s *server) handleHostToken(w http.ResponseWriter, req *http.Request) {
- if err := s.checkRequestAuth(req); err != nil {
- http.Error(w, err.Error(), http.StatusUnauthorized)
- return
- }
- if req.Method != http.MethodGet {
- http.Error(w, http.StatusText(http.StatusMethodNotAllowed), http.StatusMethodNotAllowed)
- return
- }
- hpk := strings.TrimPrefix(req.URL.Path, "/hosttoken/")
- token := s.wallet.GenerateHostToken(hostdb.HostPublicKey(hpk))
- writeJSON(w, token)
+ func (s *server) HostToken(ctx context.Context, req *HostTokenRequest) (*HostTokenResponse, error) {
+ token := s.wallet.GenerateHostToken(hostdb.HostPublicKey(req.HostPublicKey))
+ return &HostTokenResponse{
+ Token: token,
+ }, nil
+ }

First of all, it can be seen how many lines of code were saved by switching to api2. More importantly, the method signature became much clearer: instead of http.ResponseWriter and http.Request arguments which would be the same for all handlers, we have meaningful types HostTokenRequest and HostTokenResponse describing actual request and response of this specific endpoint. Parsing and decoding is done for us before the handler is called.  This simplifies testing; we can call server methods directly in tests, like normal Go functions, without having to think about encoding. If we also want to test the HTTP part, we can use an auto generated client as well. Finally, the code readability is greatly improved: business logic (in our case just a single GenerateHostToken call) is not mixed with utility code, repeating over and over again in all the handlers, there is simply no such utility code anymore.

Print Friendly, PDF & Email
© Copyright - SCP, Corp | Xa Net Services and Affiliates