Early detection of mismatches between your API specs and implementation: Exploring Specmatic’s API Coverage Report

The problem

Incomplete API specifications impact feature awareness for consumers and undocumented features cannot be verified against API design standards through linters. Moreover such specifications can neither be leveraged for stubbing / service virtualization nor for contract testing.

Key Benefits of the Specmatic API Coverage Report

Specmatic’s API coverage report makes it easy to identify and fix mismatches between an OpenAPI specification and an application’s implementation.

1. Identify mismatches between OpenAPI specifications and implementations early in your development cycle.
2. Detect paths and methods which are implemented in the application, but not documented in the API specification and vice-versa.
3. Fail your CI builds based on API Coverage Report success criteria to prevent mismatches between API specifications and implementations from propagating further.

Check out the sample code

Sample Java project:
https://github.com/znsio/specmatic-order-api-java

Sample Python projects:
https://github.com/znsio/specmatic-order-api-python
https://github.com/znsio/specmatic-order-bff-python
https://github.com/znsio/specmatic-order-bff-python-sanic

Sample NodeJS project
https://github.com/znsio/specmatic-order-bff-nodejs

Download Specmatic: https://github.com/znsio/specmatic/releases

Generating an API coverage report

If you have an application which has a route and an OpenAPI specification in which this route is documented, Specmatic can leverage the OpenAPI specification as an executable contract and run contract tests against the application. If the implementation of the route matches the specification, then Specmatic will report the contract test as passed. 

What if you document a new path in the specification but fail to actually implement it in the application? Now, any API consumers who use this spec will get a 404 error when they hit this unimplemented path. It could also happen that you add a new route in the application but fail to document it in the specification. The OpenAPI specification is now incomplete and it does not accurately describe the application’s endpoints. An incomplete specification is a huge problem and has serious implications. 

1. The effectiveness of your contract test is compromised. 
2. Any API consumers who use this spec will never become aware of the missing path. This will also impact those API consumers who wish to leverage this spec as a stub. You won’t be able to detect API compatibility breakage. 
3. You won’t be able to enforce organizational API standards using linters on the undocumented endpoints. 

This is exactly the problem that the Specmatic API coverage report solves. Specmatic can detect an application’s endpoints and generate a comprehensive API coverage report. The API coverage report helps us to identify any mismatch between the OpenAPI specification and the application early in the development lifecycle. You can also prevent these issues from propagating any further by failing the CI pipeline build if the API coverage report detects any mismatch.

Code walkthrough: API Coverage report in action

In this example, we have a Springboot application with a bunch of controllers with routes implemented in them. We also have an OpenAPI specification for this app with all the endpoints described.

When running a contract test, the key aspect to note is the Endpoints API property, through which Specmatic will determine all the endpoints implemented in the app.

Since this is a Springboot application, we use the mappings endpoint provided by the spring actuator library. When running the example test you’ll see there’s one test which has failed as well in the build.

API Coverage Report Walkthrough

The API coverage report lists all the endpoints with their methods and response codes. For every endpoint, it reports the coverage and also indicates if the endpoint is missing in the spec or not implemented in the application. It also reports how many times a particular endpoint was called during the test execution. There are three main issues identified in this example. 

1. The /health endpoint is missing in the spec. 
2. The GET method of the products/{id} endpoint is also missing in the spec.
3. The products/{id}/inventory endpoint is present in the specification, but not implemented in the app. The corresponding test for this endpoint has also failed. 

Below this, you can see that the success criteria for the API coverage report has not been met.

Let’s dig into this further. The success criteria for the API coverage report is defined in the report configuration section of the specmatic.json file. 

• You can define a minimum overall API coverage required for the build to succeed. 
• You can define the maximum number of endpoints that can be found missing in the specification. 

Specmatic can be configured to enforce the above success criteria and fail the build if either any of them are not met. 

With this in mind, let’s revisit the API coverage report and try to fix these issues.

First, let us implement the missing endpoint which is present in the specification, but not available in the app.

After implementing the missing endpoint, when we re-run the test you’ll see that where it previously failed it is now passing.

But the build will still fail because the success criteria of the API coverage report have not been met. Total API coverage is still 79%, whereas the required threshold is set to 100%. Also, there are two endpoints in the application that have not been documented in the specification and the maximum threshold for missing endpoints in the specification is zero.

Let us add the missing GET method for the product/{id} endpoint to the specification and try running the test again. Please note that apart from adding the GET method, we also need to include examples for the same, without which Specmatic would still consider it as missing in spec.

In the coverage report, you’ll now see that this GET method is reported as covered.

The build fails again because the success criteria still hasn’t been met. The API coverage now is 83%, less than the threshold of 100%, and there is still one missing endpoint in the specification, and that’s the health endpoint. 

It’s possible you want to exclude certain monitoring endpoints like health and heartbeat from the coverage report. This can be done in the report configuration section of Specmatic’s config file by adding the URL in the excluded endpoints array.

Running the test again you’ll see all tests are passing. The API coverage is 100% and there are no missing endpoints in the specification. The success criteria of the API coverage report have been met, and the build passes as well. 

This should give you an idea of what a powerful tool the API coverage report is and how it helps ensure that your OpenAPI specifications accurately describe your application. 

The OpenAPI coverage report is available now for:

• Java Springboot
• Node.js
• Python

Continuous Integration Build

As mentioned earlier, we can fail the CI builds based on API coverage success criteria. Here is an example Github action that has failed on a pull request pre-merge check because an endpoint was added to the implementation without updating the API specification.

Check out the sample code

Sample Java project:
https://github.com/znsio/specmatic-order-api-java

Sample Python projects:
https://github.com/znsio/specmatic-order-api-python
https://github.com/znsio/specmatic-order-bff-python
https://github.com/znsio/specmatic-order-bff-python-sanic

Sample NodeJS project
https://github.com/znsio/specmatic-order-bff-nodejs

Download Specmatic: https://github.com/znsio/specmatic/releases