Unit test helper

Currently, Pait provides a simple unit test support through TestHelper, which runs tests by automatically adding URLs, HTTP methods and other parameters by route functions. And when getting the result, it will get the most matching response model from response_modle_list for simple validation, thus reducing the amount of code required for developers to write test cases.

1.Usage of TestHelper

The sample code used this time is to expand the sample code on the home page, the main change is to add a parameter named return_error_resp in the route function, when return_error_resp is True it will return a response that does not match the response model, the code is as follows:

FlaskStarletteSanicTornado

# flake8: noqa: E402
from typing import Type

from flask import Flask, Response, jsonify
from pydantic import BaseModel, Field

from pait.app.flask import pait
from pait.field import Json
from pait.model.response import JsonResponseModel


class DemoResponseModel(JsonResponseModel):
    class ResponseModel(BaseModel):
        uid: int = Field()
        user_name: str = Field()

    description: str = "demo response"
    response_data: Type[BaseModel] = ResponseModel


@pait(response_model_list=[DemoResponseModel])
def demo_post(
    uid: int = Json.t(description="user id", gt=10, lt=1000),
    username: str = Json.t(description="user name", min_length=2, max_length=4),
    return_error_resp: bool = Json.i(description="return error resp", default=False),
) -> Response:
    if return_error_resp:
        return jsonify()
    return jsonify({"uid": uid, "user_name": username, "a": 123})


app.run(port=8000)

# flake8: noqa: E402
from typing import Type

from pydantic import BaseModel, Field
from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route

from pait.app.starlette import pait
from pait.field import Json
from pait.model.response import JsonResponseModel


class DemoResponseModel(JsonResponseModel):
    class ResponseModel(BaseModel):
        uid: int = Field()
        user_name: str = Field()

    description: str = "demo response"
    response_data: Type[BaseModel] = ResponseModel


@pait(response_model_list=[DemoResponseModel])
async def demo_post(
    uid: int = Json.t(description="user id", gt=10, lt=1000),
    username: str = Json.t(description="user name", min_length=2, max_length=4),
    return_error_resp: bool = Json.i(description="return error resp", default=False),
) -> JSONResponse:
    if return_error_resp:
        return JSONResponse({})
    return JSONResponse({"uid": uid, "user_name": username})


import uvicorn
uvicorn.run(app)

# flake8: noqa: E402
from typing import Type

from pydantic import BaseModel, Field
from sanic.app import Sanic
from sanic.response import HTTPResponse, json

from pait.app.sanic import pait
from pait.field import Json
from pait.model.response import JsonResponseModel


class DemoResponseModel(JsonResponseModel):
    class ResponseModel(BaseModel):
        uid: int = Field()
        user_name: str = Field()

    description: str = "demo response"
    response_data: Type[BaseModel] = ResponseModel


@pait(response_model_list=[DemoResponseModel])
async def demo_post(
    uid: int = Json.t(description="user id", gt=10, lt=1000),
    username: str = Json.t(description="user name", min_length=2, max_length=4),
    return_error_resp: bool = Json.i(description="return error resp", default=False),
) -> HTTPResponse:
    if return_error_resp:
        return json({})
    return json({"uid": uid, "user_name": username})



import uvicorn
uvicorn.run(app)

# flake8: noqa: E402
from typing import Type

from pydantic import BaseModel, Field
from tornado.web import Application, RequestHandler

from pait.app.tornado import pait
from pait.field import Json
from pait.model.response import JsonResponseModel


class DemoResponseModel(JsonResponseModel):
    class ResponseModel(BaseModel):
        uid: int = Field()
        user_name: str = Field()

    description: str = "demo response"
    response_data: Type[BaseModel] = ResponseModel


class DemoHandler(RequestHandler):
    @pait(response_model_list=[DemoResponseModel])
    def post(
        self,
        uid: int = Json.t(description="user id", gt=10, lt=1000),
        username: str = Json.t(description="user name", min_length=2, max_length=4),
        return_error_resp: bool = Json.i(description="return error resp", default=False),
    ) -> None:
        if return_error_resp:
            self.write({})
        else:
            self.write({"uid": uid, "user_name": username})

app.listen(8000)

from tornado.ioloop import IOLoop
IOLoop.instance().start()

Then can write test cases with TestHelper, first need to import TestHelper and the test client for the corresponding web framework and also initialize the test framework:.

FlaskStarletteSanicTornado

Note

Since Flask automatically registers the OPTIONS method in the route that registers the POST method, which interferes with the autodiscovery of TestHelper's HTTP methods, need to block the OPTIONS method with apply_block_http_method_set.

from pait.extra.config import apply_block_http_method_set

###########################################################
# Block the OPTIONS method that Flask automatically adds  #
###########################################################
from pait.g import config

config.init_config(apply_func_list=[apply_block_http_method_set({"OPTIONS"})])


#############
# unit test #
#############
from typing import Generator

import pytest
from flask.ctx import AppContext
from flask.testing import FlaskClient

from pait.app.flask import FlaskTestHelper


@pytest.fixture
def client() -> Generator[FlaskClient, None, None]:
    # Flask provides a way to test your application by exposing the Werkzeug test Client
    # and handling the context locals for you.
    client: FlaskClient = app.test_client()
    # Establish an application context before running the tests.
    ctx: AppContext = app.app_context()
    ctx.push()
    yield client  # this is where the testing happens!
    ctx.pop()

Note

When using with TestClient(app) as client, Starlette automatically calls the app's startup and shutdown methods, which is a good habit to get into when using with TestClient(app) as client, even though it's not used in this test case.

from typing import Generator

#############
# unit test #
#############
import pytest
from starlette.testclient import TestClient

from pait.app.starlette import StarletteTestHelper


@pytest.fixture
def client() -> Generator[TestClient, None, None]:
    with TestClient(app) as client:
        yield client

from typing import Generator

#############
# unit test #
#############
import pytest
from sanic_testing.testing import SanicTestClient

from pait.app.sanic import SanicTestHelper


@pytest.fixture
def client() -> Generator[SanicTestClient, None, None]:
    yield app.test_client

Note

Currently I don't know how to execute Tornado test cases via Pytest, so I used Tornado's AsyncHTTPTestCase for initialization. If you know how to execute Tornado test cases via Pytest, feel free to give feedback via issue.

from tornado.testing import AsyncHTTPTestCase

#############
# unit test #
#############
from pait.app.tornado import TornadoTestHelper


class TestTornado(AsyncHTTPTestCase):
    def get_app(self) -> Application:
        return app

    def get_url(self, path: str) -> str:
        """Returns an absolute url for the given path on the test server."""
        return "%s://localhost:%s%s" % (self.get_protocol(), self.get_http_port(), path)

After writing the initialization code for the test case, it is time to write the test case code, first it will be demonstrated how to write a test case through TestHelper with the following code:

FlaskStarletteSanicTornado

def test_demo_post_route_by_call_json(client: FlaskClient) -> None:
    test_helper = FlaskTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test"},
    )
    assert test_helper.json() == {"uid": 11, "user_name": "test"}

def test_demo_post_route_by_call_json(client: TestClient) -> None:
    test_helper = StarletteTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test"},
    )
    assert test_helper.json() == {"uid": 11, "user_name": "test"}

def test_demo_post_route_by_call_json(client: SanicTestClient) -> None:
    test_helper = SanicTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test"},
    )
    assert test_helper.json() == {"uid": 11, "user_name": "test"}

class TestTornado(AsyncHTTPTestCase):
    ...

    def test_demo_post_route_by_call_json(self) -> None:
        test_helper = TornadoTestHelper(
            client=self,
            func=DemoHandler.post,
            body_dict={"uid": 11, "username": "test"},
        )
        assert test_helper.json() == {"uid": 11, "user_name": "test"}

In this test case, TestHelper will be initialized, TestHelper initialization requires the Web framework corresponding to the test client, the route function, as well as the route function of some of the request parameters, after the initialization is complete, you can get the request response through the TestHelper.

While executing the test, TestHelper automatically discovers the URL and HTTP method of the route function. So when calling the json method, TestHelper will automatically initiate a post request and gets the response result. Then it serializes the response Body into a Python dict object and returns it. However, when the route function is bound to more than one request method, TestHelper will not be able to do this automatically, and need to specify the corresponding HTTP method when calling the json method, using the following method:

FlaskStarletteSanicTornado

def test_demo_post_route_by_use_method(client: FlaskClient) -> None:
    test_helper = FlaskTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test"},
    )
    assert test_helper.json(method="POST") == {"uid": 11, "user_name": "test"}

def test_demo_post_route_by_use_method(client: TestClient) -> None:
    test_helper = StarletteTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test"},
    )
    assert test_helper.json(method="POST") == {"uid": 11, "user_name": "test"}

def test_demo_post_route_by_use_method(client: SanicTestClient) -> None:
    test_helper = SanicTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test"},
    )
    assert test_helper.json(method="POST") == {"uid": 11, "user_name": "test"}

class TestTornado(AsyncHTTPTestCase):
    ...

    def test_demo_post_route_by_use_method(self) -> None:
        test_helper = TornadoTestHelper(
            client=self,
            func=DemoHandler.post,
            body_dict={"uid": 11, "username": "test"},
        )
        assert test_helper.json(method="POST") == {"uid": 11, "user_name": "test"}

In addition, when writing test cases, may need a response object, rather than response data, in order to validate data such as status codes, Header, etc. This can be done by calling the HTTP method of the TestHelper and getting the response object. The following code makes a request to the route function via the post method and returns the response object of the Web framework's test client, which is then asserted:

FlaskStarletteSanicTornado

def test_demo_post_route_by_raw_web_framework_response(client: FlaskClient) -> None:
    test_helper = FlaskTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test"},
    )
    resp = test_helper.post()
    assert resp.status_code == 200
    assert resp.json == {"uid": 11, "user_name": "test"}

def test_demo_post_route_by_raw_web_framework_response(client: TestClient) -> None:
    test_helper = StarletteTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test"},
    )
    resp = test_helper.post()
    assert resp.status_code == 200
    assert resp.json() == {"uid": 11, "user_name": "test"}

def test_demo_post_route_by_raw_web_framework_response(client: SanicTestClient) -> None:
    test_helper = SanicTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test"},
    )
    resp = test_helper.post()
    assert resp.status_code == 200
    assert resp.json == {"uid": 11, "user_name": "test"}

class TestTornado(AsyncHTTPTestCase):
    ...

    def test_demo_post_route_by_raw_web_framework_response(self) -> None:
        test_helper = TornadoTestHelper(
            client=self,
            func=DemoHandler.post,
            body_dict={"uid": 11, "username": "test"},
        )
        resp = test_helper.post()
        assert resp.code == 200
        assert resp.body.decode() == '{"uid": 11, "user_name": "test"}'

Although in this case TestHelper is not much different from the way it is used with the Web framework's corresponding test client. However, when TestHelper gets the response from the route function, it will pick the best matching response model from the response_model_list of the route function. If one of the response object's HTTP status code, header and response data does not match the response model, an error is thrown and the test case is aborted, for example:

FlaskStarletteSanicTornado

def test_demo_post_route_by_test_helper_check_response_error(client: FlaskClient) -> None:
    test_helper = FlaskTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test", "return_error_resp": True},
    )
    assert test_helper.json() == {"uid": 11, "user_name": "test"}

def test_demo_post_route_by_test_helper_check_response_error(client: TestClient) -> None:
    test_helper = StarletteTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test", "return_error_resp": True},
    )
    assert test_helper.json() == {"uid": 11, "user_name": "test"}

def test_demo_post_route_by_test_helper_check_response_error(client: SanicTestClient) -> None:
    test_helper = SanicTestHelper(
        client=client,
        func=demo_post,
        body_dict={"uid": 11, "username": "test", "return_error_resp": True},
    )
    assert test_helper.json() == {"uid": 11, "user_name": "test"}

class TestTornado(AsyncHTTPTestCase):
    ...

    def test_demo_post_route_by_test_helper_check_response_error(self) -> None:
        test_helper = TornadoTestHelper(
            client=self,
            func=DemoHandler.post,
            body_dict={"uid": 11, "username": "test", "return_error_resp": True},
        )
        assert test_helper.json() == {"uid": 11, "user_name": "test"}

After executing the test case, TestHelper will find that the response result of the route function does not match the response model of the route function, then it will throw an exception, interrupt the test case and output the result as follows:

> raise exc
E pait.app.base.test_helper.CheckResponseException: maybe error result:
E >>>>>>>>>>
E check json content error, exec: 2 validation errors for ResponseModel
E uid
E   field required (type=value_error.missing)
E user_name
E   field required (type=value_error.missing)
E
E >>>>>>>>>>
E by response model:<class 'docs_source_code.unit_test_helper.flask_test_helper_demo.DemoResponseModel'>

The output shows that the exception thrown is CheckResponseException. Meanwhile, according to the exception message, can understand that the response model for this validation is DemoResponseModel, and it found that the response data is missing the uid field and user_name field during the validation process.

2.Parameter introduction

The parameters of TestHelper are divided into three types: initialization parameters, request-related parameters, and response-related result parameters. Among them, the initialization parameters are described as follows:

Parameters	Description
client	The test client for the Web framework
func	The route function to be tested

There are multiple request parameters, which for most web frameworks encapsulate a layer of calls, but for using frameworks such as Tornado that don't encapsulate the test client much, request parameters provide some convenience, and these parameters are described as follows:

• body_dict: Json data of the request.
• cookie_dict: Cookie data of the request.
• file_dict: File data of the request.
• form_dict: Form data of the request.
• header_dict: Header data of the request.
• path_dict: Path data of the request.
• query_dict: Query data of the request.

In addition to this, TestHelper has some parameters related to response result validation, such as strict_inspection_check_json_content. By default, the strict_inspection_check_json_content parameter has a value of True, which will allow TestHelper to perform strict checks on the data structure of the response result, as in the following example:

a = {
    "a": 1,
    "b": {
        "c": 3
    }
}
b = {
    "a": 2,
    "b": {
        "c": 3,
        "d": 4
    }
}

In this example, a and b have different data structures, where the a represents the data structure of the response model and the b is the data structure of the response body. When TestHelper performs the validation, it will throw an error because it detects an extra structure ['b']['d'] in the b. However, it is possible to set the parameter strict_inspection_check_json_content to False, so that TestHelper will only validate fields that appear in the response model, and will not check fields outside the response model.

In addition to the above, TestHelper has several other parameters, as follows:

parameter	description
target_pait_response_class	If the value is not null, then TestHelper will filter a batch of eligible response_models from the response_model_list by target_pait_response_class to perform the calibration. This value is usually the parent class of the response model, and the default value of None means no match.
enable_assert_response	Indicates whether TestHelper will assert the response result, default value is True.