Unit testing http calls

Unit testing http calls

What is Unit Testing?

Unit testing is a software testing technique in which individual units or components of a software application are tested in isolation from the rest of the application to ensure that each unit works correctly. It involves writing test cases for individual units of code, executing these tests, and evaluating the results to ensure that the code behaves as expected.

The purpose of unit testing is to detect and isolate defects in individual units of code before they are integrated into the more extensive system. By catching errors early in the development process, fixing them is easier and less expensive. Additionally, unit testing helps ensure that code changes do not introduce new errors into the system.

HTTP Calls

HTTP (Hypertext Transfer Protocol) calls refer to the communication between a client and a server over the internet using the HTTP protocol. When a client makes an HTTP call to a server, it sends a request message to the server and waits for a response message.

Example Calls

Without Interface


func DirectGet(baseURL string) ([]byte, error) {
    url := baseURL + "/get"
    res, err := http.Get(url)
    if err != nil {
        println("Failed to make request err: ", err)
        return nil, err
    }
    defer res.Body.Close()

    if res.StatusCode != http.StatusOK {
        println("Failed to make request err: ", err)
        return nil, fmt.Errorf("bad response status %s", res.Status)
    }
    resByte, err := io.ReadAll(res.Body)
    if err != nil {
        println("Failed to read body err: ", err)
        return nil, err
    }
    return resByte, nil
}

To write a unit test for DirectGet method we need to use a mock server using httptest.

type mockServer struct {
    // return response
    respStatusCode int
    respBody       []byte

    // capture request
    reqPath   string
    reqMethod string
}

func (m *mockServer) createServer() *httptest.Server {
    return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        m.reqMethod = r.Method
        m.reqPath = r.URL.Path

        w.WriteHeader(m.respStatusCode)
        w.Write(m.respBody)
    }))
}

func TestDirectGet(t *testing.T) {
    t.Run("no error on request", func(t *testing.T) {
        mock := mockServer{
            respStatusCode: http.StatusOK,
            respBody:       []byte(`{"success": true}`),
        }
        mockServer := mock.createServer()
        defer func() { mockServer.Close() }()

        _, err := DirectGet(mockServer.URL)
        if err != nil {
            t.Fatalf("Failed to make request err: %v", err)
        }

        if mock.reqMethod != "GET" {
            t.Errorf("Expected method GET, got %s", mock.reqMethod)
        }

        if mock.reqPath != "/get" {
            t.Errorf("Expected path /, got %s", mock.reqPath)
        }
    })

    t.Run("error on response status code", func(t *testing.T) {
        mock := mockServer{
            respStatusCode: http.StatusBadRequest,
            respBody:       []byte(`{"success": false}`),
        }
        mockServer := mock.createServer()
        defer func() { mockServer.Close() }()

        _, err := DirectGet(mockServer.URL)
        if err == nil {
            t.Error("Expected error, got nil")
        }
    })
}

Here we have a mock server created, which will create an actual server, intercept requests, and send back the response as required. The only catch is your method should take the server’s base URL as the parameter. You can test with different cases like checking the HTTP status or body.

HTTP Call with a client:

func Client(baseURL string) ([]byte, error) {
    url := baseURL + "/get"
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        return nil, err
    }

    c := &http.Client{Timeout: time.Second * 10}

    res, err := c.Do(req)
    if err != nil {
        return nil, err
    }
    resByte, err := io.ReadAll(res.Body)
    if err != nil {
        return nil, err
    }
    defer res.Body.Close()
    return resByte, nil
}

http.Client does have a method called Do.

Do(req http.Request) (http.Response, error) But we are not injecting the client into the method instead we are creating the client so we will need to do the same as above, create a mock server, and pass the request to the mock server.

With Interface

type Requester interface {
    Do(req *http.Request) (*http.Response, error)
}

Also, a struct to pass the client.

type HTTPRequest struct {
    Client Requester
}

Let's refactor the method now to use the struct.

func (h *HTTPRequest) GetHTTPBin(ctx context.Context) ([]byte, error) {
    url := "https://httpbin.org/get"
    req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
    if err != nil {
        return nil, err
    }
    res, err := h.Client.Do(req)
    if err != nil {
        return nil, err
    }
    resByte, err := io.ReadAll(res.Body)
    if err != nil {
        return nil, err
    }
    defer res.Body.Close()
    if res.StatusCode < 200 || res.StatusCode > 299 {
        return resByte, fmt.Errorf("http status code %d", res.StatusCode)
    }
    return resByte, nil
}

Now call the method as

    client := &http.Client{Timeout: time.Second * 10}
    r := HTTPRequest{
        Client: client,
    }
    r.GetHTTPBin()

Writing Unit Test for GetHTTPBin method.

We will need a struct to mock the Do Request, other structs to capture requests, and also provide mock responses with code and error.

type mockReq struct {
   method  string
   url     string
   payload []byte
}
type mockRes struct {
   response   []byte
   statusCode int
   err        error
}
// Mock client will be used to make the request
type mockClient struct {
   req mockReq
   res mockRes
}

Now the mock Do method

func (m *mockClient) Do(req *http.Request) (*http.Response, error) {
   m.req = mockReq{
       method: req.Method,
       url:    req.URL.String(),
   }
   if req.Body != nil {
       b, err := req.GetBody()
       if err != nil {
           return nil, err
       }
       payload, err := io.ReadAll(b)
       if err != nil {
           return nil, err
       }
       m.req.payload = payload
   }
   r := io.NopCloser(bytes.NewReader(m.res.response))
   res := &http.Response{
       StatusCode: m.res.statusCode,
       Body:       r,
   }
   return res, m.res.err
}

Let’s prepare for a successful response.

Mock Client

client := &mockClient{
           res: mockRes{
               statusCode: 200,
               response:   []byte(`{"message": "I am okay"}`)},
       }

Now let's use the client with the HTTPRequester.

 req := HTTPRequest{
           Client: client,
       }

Call the method with the mocked client.

res, err := req.GetHTTPBin()

This shouldn't return an error so we can check.

assert.NoError(t, err, "unexpected error %v", err) (using github.com/stretchr/testify/assert for assertions)

Method and URL Check for the request

assert.Equal(t, client.req.method, "GET")
assert.Equal(t, client.req.url, "https://httpbin.org/get")

Also, let's say we need to check the failure. To mock error from the client we can create a client as:

client := &mockClient{
           res: mockRes{err: errors.New("something went wrong")},
       }

To mock an error as an HTTP request not okay we can

client := &mockClient{
           res: mockRes{statusCode: 400},
       }

This is okay when we have only one HTTP request, let’s say we have multiple HTTP calls from a method and need to mock each of them. The above mock will return the last payload and the response will be the same for each request.

Let's separate making HTTP calls to make it easier.


func (h *HTTPRequest) makeHTTPCall(method, url string, payload []byte) ([]byte, error) {
    client := h.Client
    req, err := http.NewRequest(method, url, bytes.NewBuffer(payload))
    if err != nil {
        return nil, err
    }
    res, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    body, err := io.ReadAll(res.Body)
    if err != nil {
        return nil, err
    }
    defer res.Body.Close()

    if res.StatusCode < 200 || res.StatusCode > 299 {
        return nil, fmt.Errorf("http status code %d", res.StatusCode)
    }
    return body, nil
}

When we have to make multiple requests from the same method.

func (h *HTTPRequest) getUUID() (string, error) {
   url := "https://httpbin.org/uuid"
   resp, err := h.makeHTTPCall("GET", url, nil)
   if err != nil {
       return "", err
   }
   type uuid struct {
       UUID string `json:"uuid"`
   }
   var u uuid
   err = json.Unmarshal(resp, &u)
   if err != nil {
       return "", err
   }
   return u.UUID, nil
}
func (h *HTTPRequest) postUUID(uuid string) error {
   url := "https://httpbin.org/post"
   body := map[string]string{"id": uuid}
   bt, _ := json.Marshal(body)
   _, err := h.makeHTTPCall("POST", url, bt)
   return err
}

func (h *HTTPRequest) normalGet() ([]byte, error) {
   url := "https://httpbin.org/get"
   return h.makeHTTPCall("GET", url, nil)
}

// GetHTTPBin gets data from http bin
func (h *HTTPRequest) MultipleHTTP() ([]byte, error) {
   id, err := h.getUUID()
   if err != nil {
       return nil, err
   }
   err = h.postUUID(id)
   if err != nil {
       return nil, err
   }
   return h.normalGet()
}

MultipleHTTP() method makes three requests first, gets UUID, next use the response from there to make a post request, and lastly get request.
So we would need to mock three HTTP calls here.

Let’s modify our mock client to have multiple req/res.

type mockClient struct {
   called int
   req    map[int]mockReq
   res    map[int]mockRes
}

Now we have called to check how many times the request is made

Map of request and response to track multiple req and res.

Last we need the Do method to capture requests properly and pass the response for each request.

func (m *mockClient) Do(req *http.Request) (*http.Response, error) {
   m.called++

   r := mockReq{
       method: req.Method,
       url:    req.URL.String(),
   }
   if req.Body != nil {
       b, err := req.GetBody()
       if err != nil {
           return nil, err
       }
       payload, err := io.ReadAll(b)
       if err != nil {
           return nil, err
       }
       r.payload = payload
   }
   m.req[m.called] = r

   mockResponse := m.res[m.called]
   respBody := io.NopCloser(bytes.NewReader(mockResponse.response))
   res := &http.Response{
       StatusCode: mockResponse.statusCode,
       Body:       respBody,
   }
   return res, mockResponse.err
}

We have all set up for mocks, now let's prepare for the test.

First, we need a mock client with proper responses.

client := &mockClient{
           res: map[int]mockRes{
               1: mockRes{
                   statusCode: 200,
                   response:   []byte(`{"uuid": "3c95e984-b50c-471b-8f67-c2ace3809b06"}`),
               },
               2: mockRes{
                   statusCode: 200,
                   response:   []byte(`{"message": "all good from post"}`),
               },
               3: mockRes{
                   statusCode: 200,
                   response:   []byte(`{"message": "all good from get"}`),
               },
           },
       }

So for the map, the key is which HTTP request is being made.

Now call MultipleHTTP()method and check the responses

 _, err := req.MultipleHTTP()
       assert.NoError(t, err, "unexpected error %v", err)

       // testing first http call
       assert.Equal(t, "GET", client.req[1].method)
       assert.Equal(t, "https://httpbin.org/uuid", client.req[1].url)

       // testing second http call
       assert.Equal(t, "POST", client.req[2].method)
       assert.Equal(t, "https://httpbin.org/post", client.req[2].url)
       assert.Equal(t, []byte(`{"id":"3c95e984-b50c-471b-8f67-c2ace3809b06"}`), client.req[2].payload) 

       // testing third http call
       assert.Equal(t, "GET", client.req[3].method)
       assert.Equal(t, "https://httpbin.org/get", client.req[3].url)

Failure cases can be tested in a similar way as above. Fail the first request.

client := &mockClient{
           req: map[int]mockReq{},
           res: map[int]mockRes{
               1: {
                   statusCode: 400,
                   response:   []byte(`{"message": "something went wrong"}`),
               },
               2: {
                   statusCode: 200,
                   response:   []byte(`{"message": "all good from post"}`),
               },
               3: {
                   statusCode: 200,
                   response:   []byte(`{"message": "all good from get"}`),
               },
           },
       }

Now we can test the HTTP call is made only once.

 assert.Equal(t, 1, client.called, "expected to have called once.")

Similarly, we can fail any one of the requests.

You can find all the code here.