TMForum ODA CTK API specification conformance testing with Specmatic

Introduction

TMForum is a collaborative platform for service providers, technology suppliers, and other stakeholders in the telco space to share best practices, develop standards, and drive innovation. Their Open API program is a global initiative to enable seamless connectivity, interoperability and portability across complex ecosystem services. And in this context, TMForum provides a Conformance Test Kit (CTK) to self-certify implementation of their Open API specifications. This is a great idea. However we recently discovered some major issues with the TMForum Conformance Test Kit (CTK) v5.0.0 and will demonstrate how using Specmatic can address these problems.

The Problem with CTK5

By running the CTK5 test, we discovered that it is possible for the API implementation to successfully pass the conformance verification and yet to be out of sync with the OpenAPI specification that is on their website. When we investigated further, we realized that the CTK contains a copy of the specification on TMForum’s Open API Table. And the Cypress test suite itself is a repetition of several aspects that are already specified in OpenAPI. With this duplication, comes the possibility of divergence and that is exactly what we see when comparing specifications on TMForum Open API Table (for example, the Party API, TMF 632 v5.0.0, published on 19-Sep-23) with copies of the same that ship with the CTK.

Furthermore, the request payload that users of the CTK are supposed to modify in accordance with their test data setup, is not validated against the OpenAPI specification that is shipped with the CTK5. This may allow users to inadvertently provide an incorrect (OpenAPI schema invalid) request payload and yet successfully pass the conformance test.

CTK5 also does not verify request parameters, error response schema and properties of fields such as optionality, nullability that are defined in the OpenAPI specification in the request.

Introducing Specmatic

To overcome these challenges, we replaced the Cypress tests with Specmatic contract tests. Specmatic leverages the primary source of truth, the API specification on the TMForum website, to generate tests in a completely no-code manner. By validating payloads against the specification, Specmatic ensures that the application remains in sync with the defined API structure and behavior.

Benefits of Specmatic

There are several advantages of adopting Specmatic for conformance testing:

1. Elimination of Duplicate Specifications: With Specmatic, there is only one source of truth—the API specification on the TMForum website. This eradicates the confusion caused by multiple specifications and ensures consistency throughout the testing process.

2. Validated Request Payloads: Specmatic validates payloads against the specification before executing tests, preventing out-of-sync requests. This ensures that the implementation adheres to the defined API structure and accurately handles incoming data.

3. Comprehensive Verification: Unlike the CTK, Specmatic verifies not only the response schema but also other crucial aspects such as 4xx error responses, optionality, nullability, and more. This comprehensive validation guarantees a higher degree of conformance and increases the reliability of API implementations.

4. Simplified Architecture: By generating tests based on the OpenAPI specification, Specmatic eliminates the need for duplicating logic already present in the specification. This simplification streamlines the testing process and reduces maintenance efforts.

5. End-to-End Testing and Fault Injection: Specmatic goes beyond conformance testing by allowing API dependencies to be stubbed out. This enables full end-to-end testing in isolation and even facilitates fault injection to ensure the application can handle various scenarios, such as timeouts and outages.

Demonstration and Comparison

By running a side-by-side comparison of the issues identified in CTK 5 and how Specmatic effectively addresses them, we can see how Specmatic’s contract tests detect incompatible changes made to the application, validate request payloads, and comprehensively verify 4xx error responses. Additionally, Specmatic provides an API coverage summary, offering an overview of the percentage of APIs that adhere to the specification.

Conclusion

Specmatic presents a superior alternative to the current Cypress based test setup in CTK for conformance testing. By leveraging the primary API specification as the single source of truth, validating request payloads, and generating tests based on the OpenAPI spec, Specmatic ensures accurate conformance of API implementations. With its simplified architecture, comprehensive verification, and additional testing capabilities, we believe Specmatic can resolve current problems and greatly enhance the abilities of the CTK.