What is the client/server model?
How to create an HTTP client.
How to send HTTP requests.
How to add specific headers to your requests.
Header
Client/Server
HTTP
Imagine that you have developed a Go Program that is managing your pictures. This program is handling your library of photos on your computer. You might want to share those pictures with other family members; to do that, you can send them an email with all your pictures in it. This solution might be impossible if you have taken ten thousand pictures.
\nYou can upload the content automatically to your favorite social network. The operation could become time-consuming if you have to do it one picture at a time. The other solution might be to plug your program directly to the social network systems to upload the pictures programmatically.
\nWe can do this with the help of the API exposed by the social network. You can push your pictures in a for loop by calling their API directly.
\nIn this case, you will consume an API. You are the client. The social network represents the Server.
\n\nReceiving and processing payments is a difficult task. The vast majority of e-commerce websites will use a payment provider. The website will call the payment provider to initiate card payments. In this context, the website is the client, and the payment provider is the server.
\n\nIf you want to integrate into your website a widget that displays the current temperature in San Francisco, you can build an API Client that will request a meteorological API (Server). Then parse the response and finally display it on your website homepage.
\n\nCalling an API means making an HTTP(s) request to a web server by following a precise documentation.
\nThe terms client and server are important, and you must memorize them.
\nAs a client we use (or consume) an API.
The server is a computer program that is designed to accept and respond to API Calls of clients.
REST stands for Representational State Transfer. Roy Fielding created the term in his doctorate thesis in 2000. Fielding wanted to develop a “model for the modern Web architecture”
Note that Fielding has defined the concept of REST, but we used those architectural constraints before his thesis. REST is now widely used among the web community; we even invented an adjective for defining APIs that follow this architectural style: RESTful.
\nThe API consumer and its producer (the developers behind it) can benefit from these constraints. The API producers follow the guidelines, but they are also free to ignore constraints. On the other side, APIs consumers can build software quicker because, at the technical level, APIs are built following the same standards. They are plenty of libraries (modules) that are freely available to build either the server part or the client part.
\nWhat are exactly those constraints?
\nCommunication is stateless : it means that the server doesn’t keep information about sessions, each request is treated individually. The server does not remember your previous requests. Therefore each request might contain all the necessary information for the server to handle it.
Cacheable responses : the responses sent by the server can be cached to be used for later requests.
Uniform interface : REST APIs always have the same interface for consumers. This uniform interface is built following three sub-constraints :
\nIdentification of ressources : Each ressource of your service (an order, a product, a photo) is addressed by an unique ressource identifier (URI).
\nhttps://maximilien-andile.com/product/2 represent the URL of the product number 2 resource.Manipulation of resources through representations : when you use an API, you are manipulating a representation of the resource. The resource can be represented as JSON, XML, YAML...
Self-descriptive messages : messages that are exchanged between the client and server must contain all the necessary information to be correctly handled, and this information should be transmitted in a way that clients can easily understand it and servers. To do so, APIs builders use HTTP methods (the norm HTTP 1.1 defines eight methods), HTTP Headers, the body of the request but also the query string.
Hypermedia as the engine of application state (HATEOAS): the web server can send links that points to the resource itself or the linked resources. For instance, if a consumer sends a request to https://maximilien-andile.com/product/2 to get information about product 2, the web server can also send to the client links to the product variants (color, size, ...).
Layered system : when building an API, you can have a specific system to handle authentication of your requests, another component responsible for retrieving the data in databases and formatting the response... From the client’s point of view, those layers should be invisible.
Code on demand : this last constraint is optional, and in reality, it’s not used intensively in the industry. It allows a web server to return scripts to clients. Clients can then execute those scripts in their system. The advertisement industry uses this feature a lot; they expose APIs to distribute Javascript or HTML banners to their clients.
You can find an overview of HTTP in the section [sec:What-is-HTTP].
\n\nThe client is the party that will send the request to the server.
\nWe will use the standard Go client from net/http1. To initialize an HTTP client, simply create a variable of type http.Client :
// consuming-api/simple/main.go\npackage main\n\nimport (\n "fmt"\n "io/ioutil"\n "net/http"\n "time"\n)\n\nfunc main() {\n c := http.Client{Timeout: time.Duration(1) * time.Second}\n resp, err := c.Get("https://www.google.com")\n if err != nil {\n fmt.Printf("Error %s", err)\n return\n }\n defer resp.Body.Close()\n body, err := ioutil.ReadAll(resp.Body)\n fmt.Printf("Body : %s", body)\n}When you create an HTTP client, you can specify the following options : Transport (type : http.RoundTripper): You can custom the way your HTTP requests will be executed by setting this field to a type that implements the type interface http.RoundTripper.
DefaultTransport is used. If you are curious, its source code is located here: https://golang.org/src/net/http/transport.goCheckRedirect (type func(req *Request, via []*Request)): we can use this field to define a function that will be called each time your request is redirected. A redirection is triggered when the server sends a special response. The response received has a special Code (HTTP Status Code) (see the section about HTTP basics to get more information about redirections).
Jar (type CookieJar): with this field, you can add cookies to your request. Cookies are passed to the server. The server himself can add cookies to the request. Those “inbound cookies” will be added to the jar.
A cookie is a piece of data stored on the client side by web browsers.
\nYou can set cookies via javascript code
The server can also set them via a specific header.
Timeout(type time.Duration): your client will open a connection to the server via HTTP; the server may take some time to answer your request. This field will define a maximum time to wait for the response. If you do not specify a timeout, there is no timeout by default. This can be dangerous for the user-perceived performance of your service. It’s better, in my opinion, to display an error than to make the user wait indefinitely. Note that this time includes:
Connection time (the time needed to connect to the distant server)
\nTime taken by redirects (if any)
Time is taken to read the response body.
We build an HTTP Client with the DefaultTransport, no CheckRedirect function, no cookies but a timeout set to 1 second.
\nWith that client, we can send requests to the web server. In our example, we make an http request with the GET method to the URL https://www.google.com :
\nresp, err := c.Get("https://www.google.com")\nif err != nil {\n fmt.Errorf("Error %s", err)\n return\n}You have a similar API to use the HTTP POST method :
myJson := bytes.NewBuffer([]byte(`{"name":"Maximilien"}`))\nresp, err := c.Post("https://www.google.com", "application/json", myJson)\nif err != nil {\n fmt.Errorf("Error %s", err)\n return\n}Note that in the case of the POST method, in addition to the URL, we must define the body of the request and the content-type of the data. The body of the request is the data we send. Why not simply put a JSON string? Why do we use a bytes.Buffer ? That’s because this argument must implement the interface io.Reader !
To make an HEAD request the syntax is close :
resp, err = c.Head("https://www.google.com")\nif err != nil {\n fmt.Errorf("Error %s", err)\n return\n}\nfmt.Println(resp.Header)Note than a HEAD request does not return a body. However, it returns the headers received in response. (this method is used to test URL, to check if they are valid, accessible or recently modified
Let’s get back to the end of our script :
\ndefer resp.Body.Close()\nbody, err := ioutil.ReadAll(resp.Body)\nfmt.Printf("Body : %s", body)We close the body when the function returns with defer resp.Body.Close().resp.Body is a stream of data read by the http client. Do not forget to add this closing instruction; otherwise, the client might not reuse a potential persistent connection to the server (cf. https://golang.org/pkg/net/http/#Client.Do).
Request Headers, or simply headers, are fields that are attached to the request. The client (your program) can use those fields to transmit additional pieces of information to the server about the request or himself
The specification of HTTP 1/1 allows you to use several header fields. Here is a shortlist of headers that are habitually used : Content-Type : it indicates to the server what type of media you will transmit in the request
Value example :
\napplication/json; charset=utf-8 => for JSON data encoded in UTF-8
Content-Length : this is the size of the message sent (in bytes)
Value Example :
\n42
User-Agent : the name and the version of the program that sends the request.
Value Example :
\ncurl/7.16.3 libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 If you use curl to perform your request
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36If you use chrome on a MacBook
Accept : specify which media types are acceptable for the response
Value Example
\n*/* All media types are accepted
application/json : is you accept only JSON
Accept-Encoding : specify which types of encoding are acceptable in the response.
Value Example
\ngzip gzipped (compressed) content is accepted
Authorization : in this header, the sender can specify its credentials (API key, username/password, JWT .…)
Value Example
\nBearer cGFydDJibGEcGFydDJibGEcGFydDJibGEcGFydDJibGE=.cGFydDJcGFydDJibGEcGFydDJibGEcGFydDJibGEibGE=.eW9sbcGFydDJibGEcGFydDJibGEcGFydDJibGEw==
To add a header, you have to build the request. It’s more verbose but building the method allows you to gain more control.
\n// consuming-api/request-building/main.go\npackage main\n\nimport (\n "fmt"\n "io/ioutil"\n "net/http"\n "time"\n)\n\nfunc main() {\n\n c := http.Client{Timeout: time.Duration(1) * time.Second}\n req, err := http.NewRequest("GET", "http://www.google.fr", nil)\n if err != nil {\n fmt.Printf("error %s", err)\n return\n }\n req.Header.Add("Accept", `application/json`)\n resp, err := c.Do(req)\n if err != nil {\n fmt.Printf("error %s", err)\n return\n }\n defer resp.Body.Close()\n body, err := ioutil.ReadAll(resp.Body)\n fmt.Printf("Body : %s", body)\n}We have to create a client and then use the http.NewRequest method which has the following signature :
(method, url string, body io.Reader) (*Request, error)The first parameter is the method which can be one of the following :
\nOPTIONS
GET
HEAD
POST
PUT
DELETE
TRACE
CONNECT
Then you provide the URL. The body is the third parameter of this method. In our case, we set it to nil because we do not have a body to send. The method will not send the request but will return an http.Request. With req.Header.Add (where req is a struct of type htttp.Request) you can append a header to the request :
req.Header.Add("Accept", `application/json`)Note that the Header field of a request is under the hood, an element of type map[string][]string a map where the keys are strings and the values are slices of strings.
The request is sent with c.Do(req).
Go will add some headers automatically to your requests :
\nFor a standard GET request :
\nc.Get("http://localhost:8091/status")Accept-Encoding : gzip
User-Agent: Go-http-client/1.1
For a standard POST request :
\nc.Post("http://localhost:8091/status", "application/json", bytes.NewBuffer([]byte("42")))Accept-Encoding : gzip
User-Agent: Go-http-client/1.1
Content-Length: 2
Content-Type: application/json
In this section, We will build an HTTP client that will send requests to the Github API. Github allows you to share code and collaborate on coding projects with other developers.
\nGithub exposes an API to search for projects, to get and update issues of a specific project... The possibilities that are open by the API of GitHub are very large.
\nThe documentation of the Github API is available on this website:
\nhttps://developer.github.com/v3In this example, we will use the v3 API.
\nThe base URL of the API is https://api.github.com/ ; let’s try to do an HTTP GET request on it.
\n// consuming-api/github/main.go \n\nc := http.Client{Timeout: time.Duration(1) * time.Second}\nreq, err := http.NewRequest("GET", "https://api.github.com/", nil)\nif err != nil {\n fmt.Printf("error %s", err)\n return\n}\nreq.Header.Add("Accept", `application/json`)\nresp, err := c.Do(req)\nif err != nil {\n fmt.Printf("error %s", err)\n return\n}\ndefer resp.Body.Close()\nbody, err := ioutil.ReadAll(resp.Body)\nif err != nil {\n fmt.Printf("error %s", err)\n return\n}\nfmt.Printf("Body : %s \\n ", body)\nfmt.Printf("Response status : %s \\n", resp.Status)When we launch our script, we get the following output :
\nBody : {"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/clie\nnt_id}",\n...\n}\n\n Response status: 200 OKThe response body is JSON. Note that the three dots means that I have not printed the full body of the response. The status code of our response is “200 OK” which means that our request was successful.
\nThe body content suggests that we can ping the GitHub API using other routes. To get the current user, we can request https://api.github.com/user. Let’s try it :
\nreq, err := http.NewRequest("GET", "https://api.github.com/user", nil)The response is not very glamourous :
\nBody : {"message":"Requires authentication","documentation_url":"https://developer.github.com/v3/users/#get-the-authenticated-user"}\n\nResponse status: 401 UnauthorizedThe status code is “401 Unauthorized”. It means that “the request requires user authentication. The response MUST include a WWW-Authenticate header field (section 14.47) containing a challenge applicable to the requested resource.”
Our requested resource (seeing details about the current user) requires that we prove our identity. To do that, we must register to Github. If you already have an account on Github, just go to that web page https://github.com/settings/tokens; if you do not have an account, you can create one if you want. Note that you can follow the tutorial without having a Github account. The idea is to have an idea on how to use an API.
\nMost of the time, API providers require you to register before using their API to control who has access to the data.
\n\nNavigate to https://github.com/settings/tokens. On this page, click on the button “Generate new token” :
\nYou will then be asked to give a name to your personal token and select the scopes that Github will attach to your new personal token. Selecting scopes allow you to restrict the usage of your personal token. In our example, we will just select “Gist”. By publishing a Gist, you can share code with other users. We will use the Github API to create Gists.
\nOnce you got your personal token, you can inject it into your requests. You might ask, where do I put it? If we follow the HTTP 1/1 RFC, then we have to inject it into an “Authenticate header field”. You must always read the API’s documentation because there is no unified way to inject authorization tokens.
\nThe Github API documentation specifies that we should inject the token into the Header Authorization :
\nLet’s try it. We will make the same call as before, but this time we will use an authentication header as suggested by the GitHub docs :
\n// current user\nreq, err = http.NewRequest("GET", "https://api.github.com/user", nil)\n// TODO : handle error\nreq.Header.Add("Accept", `application/json`)\n// add header for authentication\nreq.Header.Add("Authorization", fmt.Sprintf("token %s", os.Getenv("TOKEN"))\n// ...The most important line of this listing is :
\nreq.Header.Add("Authorization", fmt.Sprintf("token %s", os.Getenv("GITHUB_TOKEN"))Note that we are not saving the access token directly in our script. This is a very bad idea because if you share your code, you do not want your fellow developers to access your Github account.
\nInstead, we are using os.Getenv(\"GITHUB_TOKEN\"). The Getenv method from the os standard package will retrieve the environment variable names TOKEN and parse it. Note that to handle your configuration, you can also use a configuration file. There are very popular packages that handle configuration for you (see chapter [chap:Application-Configuration]).
To pass an environment variable to the program, just prepend GITHUB_TOKEN=XXX to your command :
$ GITHUB_TOKEN=aabbcc go run main.goThe first thing to do is to read the documentation. You can infer that if we want to create something, it will be a POST; the URI might contain gist, but that’s all. I made a screenshot of the documentation page :
\nThe endpoint is specified as follow :
\nThe HTTP verb to use is POST
The input parameters are
\nfiles (which is an object)
description : a string
public which is a boolean value
Let’s implement this call :
\n// consuming-api/github-post-gist/main.go \n\nreq, err := http.NewRequest("POST", "https://api.github.com/gists", bytes.NewBuffer(gistRequestJson))\nif err != nil {\n fmt.Printf("%s", err)\n return\n}\nreq.Header.Add("Accept", `application/json`)\n// add header for authentication\nreq.Header.Add("Authorization", fmt.Sprintf("token %s", os.Getenv("TOKEN")))\n\nresp, err := c.Do(req)\nif err != nil {\n fmt.Printf("Error %s", err)\n return\n}I didn’t inject the requested parameters to see what kind of error message we will have.
\nBody : {"message":"Invalid request.\\n\\nFor 'links/0/schema', nil is not an object.","documentation_url":"https://developer.github.com/v3/gists/#create-a-gi\nst"}\n Response status: 422 Unprocessable EntityWe’ve got an error; there might be an input validator somewhere in the Github server that controls the user input.
\nThe parameters will be transmitted in the body of the request. Parameters will be encoded in JSON. To do that, we will create a struct that will hold the parameters.
\ntype GistRequest struct {\n Files map[string]File `json:"files"`\n Description string `json:"description"`\n Public bool `json:"public"`\n}\ntype File struct {\n Content string `json:"content"`\n}Those struct types will hold the parameters of our request :
\nfiles := map[string]File{\n "main.go": File{"test"}}\ngistRequest := GistRequest{\n Files: files,\n Description: "this is a test",\n Public: false}The variable gistRequest needs to be transformed into a valid JSON string :
gistRequestJson, err := json.Marshal(gistRequest)\nif err != nil {\n fmt.Printf("%s", err)\n return\n}Then we just have to populate the body of our request with our brand new JSON. There is just a little difficulty here : json.Marshal returns a slice of bytes. The signature of NewRequest is :
func NewRequest(method, url string, body io.Reader) (*Request, error)The body needs to implement the type interface io.Reader.
To do that, there is a simple way, using the standard package bytes :
bytes.NewBuffer(gistRequestJson)The returned value of bytes.NewBuffer implements io.Reader.
We can now launch the request :
\nreq, err := http.NewRequest("POST", "https://api.github.com/gists", bytes.NewBuffer(gistRequestJson))\nreq.Header.Add("Accept", `application/json`)\nreq.Header.Add("Authorization", fmt.Sprintf("token %s", os.Getenv("TOKEN")))\nresp, err := c.Do(req)\n//...The response is a success :
\nBody : {"url":"https://api.github.com/gists/c960d211532f7c35aeb0c854892bf108",...}\n Response status : 201 CreatedThe Gist has been created, as you can see in figure 1. The URI of this new resource is returned in the body response :
\nhttps://api.github.com/gists/c960d211532f7c35aeb0c854892bf108. It’s not always the case because APIs do not strictly follow the RFC’s recommendations. Do not be surprised if you do not get the URI in the response. Concerning the response status, it’s also not always “201 Created” but instead “200 OK”.
\nMost of the time, you have to adapt your client to the API you are integrating. The response status code is not always the same; the authentication method can also be different.
\nWe have created a resource. Now we can try to update it. Most of the time, you update a resource with an HTTP PUT or PATCH request.
\nLet’s take a look at the API documentation (2).
\nThe request use a PATCH method. The URL is /gist/:gist_id,meaning that we have to replace gist_id by the id of the gist, we want to modify. In the REST API world, each resource has an identifier, here in the case of a gist, the id is denoted gist_id. In our case the gist_id is c960d211532f7c35aeb0c854892bf108. So the request URL will be:
"https://api.github.com/gists/c960d211532f7c35aeb0c854892bf108"The parameters are the same as the POST request (to create the gist). The following listing will create the request body :
\n// consuming-api/github-patch-gist/main.go \n\nfiles := map[string]File{\n "main.go": File{"test updated"}}\ngistRequest := GistRequest{\n Files: files,\n Description: "this is a test",\n Public: false}\ngistRequestJson, err := json.Marshal(gistRequest)\nif err != nil {\n fmt.Printf("%s", err)\n return\n}Here we are just updating the content of main.go that will become “test updated”. The next thing to do is to build or http client, create the request, execute it and display the result
\n// consuming-api/github-patch-gist/main.go \n\nc := http.Client{Timeout: time.Duration(4) * time.Second}\n\nreq, err := http.NewRequest("PATCH", "https://api.github.com/gists/79c9cec21a116f6ee166fd73ba750565", bytes.NewBuffer(gistRequestJson))\nif err != nil {\n fmt.Printf("%s", err)\n return\n}\nreq.Header.Add("Accept", `application/json`)\n// add header for authentication\nreq.Header.Add("Authorization", fmt.Sprintf("token %s", os.Getenv("TOKEN")))\n\nresp, err := c.Do(req)\nif err != nil {\n fmt.Printf("Error %s", err)\n return\n}\ndefer resp.Body.Close()\nbody, err := ioutil.ReadAll(resp.Body)\nfmt.Printf("Body : %s \\n ", body)\nfmt.Printf("Response status : %s \\n", resp.Status)The execution result is :
\nBody : {"url":"https://api.github.com/gists/c960d211532f7c35aeb0c854892bf108",...}\n Response status : 200 OKA response status of 200 indicates that the server has correctly handled the request.
\nOn the GitHub gist website, our gist has been updated! (see figure 3)
\nDeleting a resource using a web service is a common task. The http request method to do that is typically DELETE. The Github API documentation is pretty straightforward (see figure 4)
\nThe code is simpler than the previous request because we do not have to include a body :
\n// consuming-api/github-gist-delete/main.go \n\nc := http.Client{Timeout: time.Duration(4) * time.Second}\nreq, err := http.NewRequest("DELETE", "https://api.github.com/gists/79c9cec21a116f6ee166fd73ba750565", nil)\nif err != nil {\n fmt.Printf("Error %s", err)\n return\n}\nreq.Header.Add("Accept", `application/json`)\n// add header for authentication\nreq.Header.Add("Authorization", fmt.Sprintf("token %s", os.Getenv("TOKEN")))\n\nresp, err := c.Do(req)\nif err != nil {\n fmt.Printf("Error %s", err)\n return\n}\ndefer resp.Body.Close()\nbody, err := ioutil.ReadAll(resp.Body)\nif err != nil {\n fmt.Printf("Error %s", err)\n return\n}\nfmt.Printf("Body : %s \\n ", body)\nfmt.Printf("Response status : %s \\n", resp.Status)The execution result shows an empty body and a response status code equal to 204 No Content :
\nBody :\n\nResponse status : 204 No ContentThe response status code is 2... meaning our request has been a success. We have deleted our gist.
\n\nTrue or False. The server is responsible for sending requests to the client.
Give two architectural constraints of a REST API (as defined in the Roy Fielding thesis).
How to create a basic HTTP client with a 3 seconds timeout?
In which header can you put user credentials?
If you send a POST request with the default Go, http client, which headers are already set?
True or False. The server is responsible for sending requests to the client.
\nFalse
The server receives and handles requests; the client sends requests.
Give two architectural constraints of a REST API (as defined in the Roy Fielding thesis).
\nStateless
Responses are cacheable
Uniform interface
Layered systems
Code on demand
How to create a basic HTTP client with a 3 seconds timeout?
\nc := http.Client{Timeout:1 * time.Second}In which header can you put user credentials?
\nAuthorization headerIf you send a POST request with the default Go, http client, which headers are already set?
\nUser-Agent
Content-Length
Content-Type
Accept-Encoding
In the client/server model, the client requests a resource (or a service). The server provides access to this resource.
The server receives and handles requests; the client sends requests.
Servers are also called producers. Clients are also called consumers.
Roy Fielding has developed the term REST API in his doctoral thesis
Roy Fielding defined some architectural constraints used to build RESTful APIs.
c := http.Client{Timeout:1 * time.Second}. Then you can use the client to send requests :\n// Send simple GET and POST requests\nres, err := c.Get("http://localhost:8091/status")\n// ....\nres, err = c.Post("http://localhost:8091/status", "application/json", bytes.NewBuffer([]byte("42")))\n\n// Build more complex requests\nreq, err := http.NewRequest("HEAD", "http://localhost:8091/status", nil)\n// ....\nreq.Header.Add("Accept", `application/json`)\n// send the requests\nc.Do(req)By default, the client has no timeout set. It’s important in your application to always configure this option (set the field Timeout with an appropriate duration).
You can add additional headers to the request with request.Header.Add
If you are curious, I suggest you take a look at the source code here : https://golang.org/src/net/http/client.go.↩︎
Previous
\n\t\t\t\t\t\t\t\t\tBenchmarks
\n\t\t\t\t\t\t\t\tNext
\n\t\t\t\t\t\t\t\t\tProgram Profiling
\n\t\t\t\t\t\t\t\t