Skip to content →

How Not to Mock Web Requests for Unit Testing

You have a web application and you want to write some unit tests for one of the endpoints. But you don’t want to make actual network calls when you run your tests, for a few reasons.

  1. You want your unit tests to be fast. You don’t want to add network latency to the time it takes your tests to complete. Especially if you have implemented continuous integration that triggers builds and tests when you push a commit.
  2. You don’t want to skew metrics. If you are keeping track of how many requests are made to your server, you don’t want “test requests” to be included in those metrics.
  3. You don’t want any unintended side effects. For example, if your server modifies a database, or connects with another piece of infrastructure. You don’t want that to happen when you run unit tests.

So, how do you test the functionality of how your app responds to a web request without making an actual request and returning a response. One approach is to mock the requests and responses. Let’s look at an example.

Here is your app. It’s a simple “Hello, World” API and has a little bit of logic to it. If the user includes a header with a key of name, then the API will return “Hello, $name”. Otherwise, it will return “Hello, World”.

from flask import Flask, request


app = Flask(__name__)

@app.route('/')
def hello():
    name = request.headers.get('name')
    if name:
        return f'Hello, {name}'
    return 'Hello, World'


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Let’s run it!

$ python app.py
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
....

# open up another terminal...
$ curl localhost:5000
Hello, World
$ curl -H 'name: Lee' localhost:5000
Hello, Lee

Now, let’s write a test.

import requests

URL = 'http://localhost:5000'

def test_hello():
    r = requests.get(URL)
    assert r.text == 'Hello, World'    

    headers = {'name': 'Lee'}
    r = requests.get(URL, headers=headers)
    assert r.text == 'Hello, Lee' 

I’ll run the test using pytest.

$ pytest
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x103df4490>: Failed to establish a new connection: [Errno 61] Connection refused')

I get a NewConnectionError because my server isn’t running. This is another reason why mocking can be beneficial. You shouldn’t have to have your server running to execute unit tests. Why not? Because unit tests are supposed to test small chunks of logic during the development process. A faulty network should not prevent a developer from iterating on the logic of a function.

For now I will forget about that and just start the server in another terminal and run the tests again.

$ python app.py
# server starts listening on port 5000
# open up another terminal...

$ pytest
test_app.py .                                                                                                                             [100%]

1 passed in 0.05s

It feels good to pass a test, but this is not an ideal workflow for two reasons.

  1. My server has to be running whenever I execute my unit tests
  2. I’m making actual network requests to my server that may have undesired side effects and can skew metrics

So now let’s introduce mocking. Let’s do it the wrong way. Let’s mock the requests module to return something that we want our function to return.

import requests
from unittest.mock import Mock

URL = 'http://localhost:5000'

requests = Mock()

def test_hello():

    requests.get.return_value.text = 'Hello, World'
    r = requests.get(URL)
    assert r.text == 'Hello, World'    

    requests.get.return_value.text = 'Hello, Lee'
    headers = {'name': 'Lee'}
    r = requests.get(URL, headers=headers)
    assert r.text == 'Hello, Lee'

There is our updated test, with a few changes.

  • Mocked the requests module using a unittest.mock.Mock object
  • Set the r.text to our expected string

Since we mocked the requests module, we no longer make web requests to our URL, so we can kill our server! If you’re following along go ahead and kill your server and then run the tests.

$ pytest
test_app.py .                                                                                                                             [100%]
1 passed in 0.04s

Awesome, we passed our test and we eliminated the need to keep our server up and running! Before we clock out for the day, let’s make sure that our tests do what they are supposed to do. Let’s break our function and run the tests again. Make a slight change to the function in your app.

...
def hello():
    name = request.headers.get('name')
    if name:
        return f'Hey what is up, {name}?' # note the change here
    return 'Hello, World'
...

Alright, now let’s run our tests and watch them fail!

$ pytest
test_app.py .                                                                                                                             [100%]
1 passed in 0.04s

That feeling when your tests pass but you’re not happy…

If it isn’t obvious, the tests succeeded because we hard-coded a mocked response and didn’t change our tests. We could go update our tests to return “Hey, what is up, Lee?” instead of “Hello, Lee”, but then we’ve completely lost the purpose of testing. Why even have tests if we are just comparing hard-coded strings without our tests?

So let’s do better. Let’s make our tests valuable, but also mock the requests so we aren’t running unit tests on a live server. One way to do this is to create a separate function, (you could call it a “handler”), that contains the core logic of our app but isn’t part of the flask route. Then we can run our tests on that function instead. Let me show you what I mean.

from flask import Flask, request


app = Flask(__name__)

def hello_handler(name=None):
    if name:
        return f'Hello, {name}'
    return 'Hello, World'

@app.route('/')
def hello():
    name = request.headers.get('name')
    return hello_handler(name)


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Notice the changes.

  • Created a function called hello_handler that contains the core logic of how to handle the request.
  • In hello() we call the handler

The app still functions the same.

$ python app.py
....

# open up another terminal...
$ curl localhost:5000
Hello, World
$ curl -H 'name: Derrick' localhost:5000
Hello, Derrick

Now let’s revisit the tests.

from app import hello_handler

def test_hello_handler():

    r = hello_handler()
    assert r == 'Hello, World'    

    name = 'Lee'
    r = hello_handler(name)
    assert r == 'Hello, Lee'

We made quite a few changes.

  • No more requests, so we definitely aren’t making web requests and don’t need our server up and running
  • No more Mock. We actually aren’t mocking anything!
  • We are importing and testing hello_handler() not hello(). Since these are unit tests, which are supposed to test small chunks of logic in a function, it makes sense to test the function that contains the logic.

Let’s run the tests.

$ pytest
test_app.py .                                                                                                                             [100%]
1 passed in 0.04s

Now let’s try to break the tests by changing our logic.

...
def hello_handler(name=None):
    if name:
        return f'Hey, {name}, how is it going?' # note the change here
    return 'Hello, World'
...

And run the tests again.

$ pytest
AssertionError: assert 'Hey, Lee, how is it going?' == 'Hello, Lee'
  - Hey, Lee, how is it going?
  + Hello, Lee

And it fails!

So now we can feel confident that when we change the logic of our function, the tests will reflect those changes.

Conclusion

Instead of mocking requests for unit tests, we broke up our application into a “handler” function. The handler handles the core logic of the app. We then call the handler when a request is made to our endpoint. In other words, we have decoupled our logic from the HTTP endpoint so we can write simple unit tests without being concerned about the network.

Published in Today I Learned