Unit testing http calls

What is Unit Testing?

Unit testing is a software testing technique in which individual units or components of a software application are tested in isolation from the rest of the application to ensure that each unit works correctly. It involves writing test cases for individual units of code, executing these tests, and evaluating the results to ensure that the code behaves as expected.

The purpose of unit testing is to detect and isolate defects in individual units of code before they are integrated into the more extensive system. By catching errors early in the development process, fixing them is easier and less expensive. Additionally, unit testing helps ensure that code changes do not introduce new errors into the system.

HTTP Calls

HTTP (Hypertext Transfer Protocol) calls refer to the communication between a client and a server over the internet using the HTTP protocol. When a client makes an HTTP call to a server, it sends a request message to the server and waits for a response message.

Example Call

func DirectGet() ([]byte, error) {
    res, err := http.Get("https://httpbin.org/get")
    if err != nil {
        println("Failed to make request err: ", err)
        return nil, err
    }
    resByte, err := io.ReadAll(res.Body)
    if err != nil {
        println("Failed to read body err: ", err)
        return nil, err
    }
    defer res.Body.Close()
    return resByte, nil
}

To write a unit test for DirectGet the method there is no other way than to make an actual HTTP call.

Let’s change this to use a Client

HTTP Call with a client:

func Client() ([]byte, error) {
   url := "https://httpbin.org/get"
   req, err := http.NewRequest("GET", url, nil)
   if err != nil {
       return nil, err
   }
   c := &http.Client{Timeout: time.Second * 10}
   res, err := c.Do(req)
   if err != nil {
       return nil, err
   }
   resByte, err := io.ReadAll(res.Body)
   if err != nil {
       return nil, err
   }
   defer res.Body.Close()
   return resByte, nil
}

http.Client does have a method called Do.

Do(req http.Request) (http.Response, error) But we are not injecting the client into the method instead we are creating the client so it still can’t be mocked. We need to use an interface to mock the client.

Let’s add an interface with the Do method

type Requester interface {
    Do(req *http.Request) (*http.Response, error)
}

Also, a struct to pass the client.

type HTTPRequest struct {
    Client Requester
}

Let's refactor the method now to use the struct.

func (h *HTTPRequest) GetHTTPBin() ([]byte, error) {
   url := "https://httpbin.org/get"
   req, err := http.NewRequest("GET", url, nil)
   if err != nil {
       return nil, err
   }
   res, err := h.Client.Do(req)
   if err != nil {
       return nil, err
   }
   resByte, err := io.ReadAll(res.Body)
   if err != nil {
       return nil, err
   }
   defer res.Body.Close()
   if res.StatusCode < 200 || res.StatusCode > 299 {
       return resByte, errors.New("http status not okay")
   }
   return resByte, nil
}

Now call the method as

    client := &http.Client{Timeout: time.Second * 10}
    r := HTTPRequest{
        Client: client,
    }
    r.GetHTTPBin()

Writing Unit Test for GetHTTPBin method.

We will need a struct to mock the Do Request, also other structs to capture requests, and also provide mock responses with code and error.

type mockReq struct {
   method  string
   url     string
   payload []byte
}
type mockRes struct {
   response   []byte
   statusCode int
   err        error
}
// Mock client will be used to make the request
type mockClient struct {
   req mockReq
   res mockRes
}

Now the mock Do method

func (m *mockClient) Do(req *http.Request) (*http.Response, error) {
   m.req = mockReq{
       method: req.Method,
       url:    req.URL.String(),
   }
   if req.Body != nil {
       b, err := req.GetBody()
       if err != nil {
           return nil, err
       }
       payload, err := io.ReadAll(b)
       if err != nil {
           return nil, err
       }
       m.req.payload = payload
   }
   r := io.NopCloser(bytes.NewReader(m.res.response))
   res := &http.Response{
       StatusCode: m.res.statusCode,
       Body:       r,
   }
   return res, m.res.err
}

Let’s prepare for a successful response.

Mock Client

client := &mockClient{
           res: mockRes{
               statusCode: 200,
               response:   []byte(`{"message": "I am okay"}`)},
       }

Now let's use the client with the HTTPRequester.

 req := HTTPRequest{
           Client: client,
       }

Call the method with the mocked client.

res, err := req.GetHTTPBin()

This shouldn't return an error so we can check.

assert.NoError(t, err, "unexpected error %v", err) (using github.com/stretchr/testify/assert for assertions)

Method and URL Check for the request

assert.Equal(t, client.req.method, "GET")
assert.Equal(t, client.req.url, "https://httpbin.org/get")

Also, let's say we need to check the failure. To mock error from the client we can create a client as:

client := &mockClient{
           res: mockRes{err: errors.New("something went wrong")},
       }

To mock an error as an HTTP request not okay we can

client := &mockClient{
           res: mockRes{statusCode: 400},
       }

This is okay when we have only one HTTP request, let’s say we have multiple HTTP calls from a method and need to mock each of them. The above mock will return the last payload and the response would be the same for each request.

Let's separate making HTTP calls to make it easier.

func (h *HTTPRequest) makeHTTPCall(method, url string, payload []byte) ([]byte, error) {
    client := h.Client
    req, err := http.NewRequest(method, url, bytes.NewBuffer(payload))
    if err != nil {
        return nil, err
    }
    res, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    body, err := ioutil.ReadAll(res.Body)
    if err != nil {
        return nil, err
    }
    defer res.Body.Close()

    if res.StatusCode < 200 || res.StatusCode > 299 {
        return nil, fmt.Errorf("http status code %d", res.StatusCode)
    }
    return body, nil
}

When we have to make multiple requests from the same method.

func (h *HTTPRequest) getUUID() (string, error) {
   url := "https://httpbin.org/uuid"
   resp, err := h.makeHTTPCall("GET", url, nil)
   if err != nil {
       return "", err
   }
   type uuid struct {
       UUID string `json:"uuid"`
   }
   var u uuid
   err = json.Unmarshal(resp, &u)
   if err != nil {
       return "", err
   }
   return u.UUID, nil
}
func (h *HTTPRequest) postUUID(uuid string) error {
   url := "https://httpbin.org/post"
   body := map[string]string{"id": uuid}
   bt, _ := json.Marshal(body)
   _, err := h.makeHTTPCall("POST", url, bt)
   return err
}

func (h *HTTPRequest) normalGet() ([]byte, error) {
   url := "https://httpbin.org/get"
   return h.makeHTTPCall("GET", url, nil)
}

// GetHTTPBin gets data from http bin
func (h *HTTPRequest) MultipleHTTP() ([]byte, error) {
   id, err := h.getUUID()
   if err != nil {
       return nil, err
   }
   err = h.postUUID(id)
   if err != nil {
       return nil, err
   }
   return h.normalGet()
}

MultipleHTTP() method makes three requests first, gets UUID, next use the response from there to make a post request, and lastly get request.
So we would need to mock three HTTP calls here.

Let’s modify our mock client to have multiple req/res.

type mockClient struct {
   called int
   req    map[int]mockReq
   res    map[int]mockRes
}

Now we have called to check how many times the request is made

Map of request and response to track multiple req and res.

Last we need the Do method to capture requests properly and pass the response for each request.

func (m *mockClient) Do(req *http.Request) (*http.Response, error) {
   m.called++

   r := mockReq{
       method: req.Method,
       url:    req.URL.String(),
   }
   if req.Body != nil {
       b, err := req.GetBody()
       if err != nil {
           return nil, err
       }
       payload, err := io.ReadAll(b)
       if err != nil {
           return nil, err
       }
       r.payload = payload
   }
   m.req[m.called] = r

   mockResponse := m.res[m.called]
   respBody := io.NopCloser(bytes.NewReader(mockResponse.response))
   res := &http.Response{
       StatusCode: mockResponse.statusCode,
       Body:       respBody,
   }
   return res, mockResponse.err
}

We have all set up for mocks, now let's prepare for the test.

First, we need a mock client with proper responses.

client := &mockClient{
           res: map[int]mockRes{
               1: mockRes{
                   statusCode: 200,
                   response:   []byte(`{"uuid": "3c95e984-b50c-471b-8f67-c2ace3809b06"}`),
               },
               2: mockRes{
                   statusCode: 200,
                   response:   []byte(`{"message": "all good from post"}`),
               },
               3: mockRes{
                   statusCode: 200,
                   response:   []byte(`{"message": "all good from get"}`),
               },
           },
       }

So for the map, the key is which HTTP request is being made.

Now call MultipleHTTP()method and check the responses

 _, err := req.MultipleHTTP()
       assert.NoError(t, err, "unexpected error %v", err)

       // testing first http call
       assert.Equal(t, "GET", client.req[1].method)
       assert.Equal(t, "https://httpbin.org/uuid", client.req[1].url)

       // testing second http call
       assert.Equal(t, "POST", client.req[2].method)
       assert.Equal(t, "https://httpbin.org/post", client.req[2].url)
       assert.Equal(t, []byte(`{"id":"3c95e984-b50c-471b-8f67-c2ace3809b06"}`), client.req[2].payload) 

       // testing third http call
       assert.Equal(t, "GET", client.req[3].method)
       assert.Equal(t, "https://httpbin.org/get", client.req[3].url)

Failure cases can be tested in a similar way as above. Fail the first request.

client := &mockClient{
           req: map[int]mockReq{},
           res: map[int]mockRes{
               1: {
                   statusCode: 400,
                   response:   []byte(`{"message": "something went wrong"}`),
               },
               2: {
                   statusCode: 200,
                   response:   []byte(`{"message": "all good from post"}`),
               },
               3: {
                   statusCode: 200,
                   response:   []byte(`{"message": "all good from get"}`),
               },
           },
       }

Now we can test the HTTP call is made only once.

 assert.Equal(t, 1, client.called, "expected to have called once.")

Similarly, we can fail any one of the requests.

You can find all the code here.

Samit's Blog

Samit's Blog

Unit testing multiple HTTP calls in GoLang

What is Unit Testing?

HTTP Calls

Example Call

HTTP Call with a client: