| Approach |
Contract Driven Development (CDD) |
Code First, Consumer Driven Contract Testing |
Bi-Directional Contract Testing (BDCT) |
| OpenAPI support |
YES |
NO (OpenAPIs and Pact are designed with different ends in mind) |
PARTIAL, only as Provider Spec (OAS as the provider spec) Has several limitations – no remote refs / single document approach, need to dereference “$Ref”, and more |
| AsyncAPI support |
YES Full Contract Testing vs Async Mocking on a paid plan |
NO |
NO Plugins may provide limited support. |
| Suitable for Consumer-first development style (Frontend before backend) |
YES Consumer Team can author the API Specification, leverage it as mock / stub and then share it with Provider team as a guideline |
YES Pact requires starting with Consumer development first by mocking the Provider and then leveraging these mocks to generate the Contract for the Provider |
YES same as Pact OSS |
| Suitable for Provider-first development style (Backend before frontend, public facing APIs, etc.) |
YES Provider Team can author the API Specification, leverage as tests and then share it with Consumer team to stub the Provider |
NO Pact is better suited for Consumer First Development Style |
YES Provider has to publish their OAS to PactFlow server, they will only get feedback after Consumer has publish their Contract and PactFlow has completed Cross Contract Verification |
| Suitable for API-first development style |
YES Consumer and Provider Teams collaboratively author API Specification and only then begin development |
NO It does not support Contract / API Spec first Development Style |
YES Consumers still cannot use the API Specification as Stub, they will only have to Cross Contract validate with the Specification |
| Parallel development of Consumer and Provider |
YES This is one of the primary capabilities of Specmatic, to allow teams to independently develop and confidently deploy microservices |
NO Consumer development has to be ahead of Provider development |
YES |
| Backward Compatibility Verification #NOCODE |
YES This is one of the flagship capabilities in Specmatic. Contract vs Contract comparison, no need to implement the changes in Provider or Consumer Code |
NO Compatibility can only be possibly detected after an actual change happens in Provider code, two version of Pact JSONs cannot be compared to see if the changes are compatible |
NO Same as Pact OSS |
| Example generation and validation for OpenAPI |
YES |
NO |
NO |
| Proxy based OpenAPI generation, validation |
YES |
NO |
PARTIAL bring your own tools |
| GUI based visual #NOCODE testing |
YES Specmatic Studio unlocks this (paid) |
NO |
NO |
| Contract Conformance Report |
YES |
NO |
NO |
| Dashboarding & service dependency mapping |
YES comprehensive with Insights |
NO |
PARTIAL |
| API specifications as Test #NOCODE |
YES Specmatic can generate and run tests, both happy path and boundary conditions, all with just your OpenAPI specifications |
NO Pact JSONs are the ones that are run as test, OpenAPI cannot be run as tests |
YES WITH CAVEAT – External Tool Dependency with Bring Your Own testing tool such as Dredd – Also several limitations / considerations |
| API specifications as Mock #NOCODE |
YES OpenAPI is all you need to run a stub server, no need to define a stub, just the expectation data |
NO Pact DSL necessary to define Stubs |
NO When leveraging Bring Your Own mocks, the frameworks such as Mountebank, Wiremock, etc. have to be setup |
| Smart Mocks (Stale mock detection / Expectation validation against API specification) |
YES / Immediate Feedback Expectations that are not as per API Spec are immediately rejected with proper error message |
NO / Not Applicable Contracts are generated based on the expectation, stale stub detection can only happen when pact json is run as test against Provider |
NO / Not Applicable PactFlow allows Bring Your Own (BYO) mocks, even with Mountebank the approach has no immediate verification of stub data with Specification. Mountebank imposters have to be converted to Pact Files which are then uploaded to PactFlow wh |
| Unfragmented Devex / Minimal learning curve |
YES You just need 1. API Specification of your choice such as OpenAPI 2. Specmatic – No additional DSLs |
NO Need to pick up several tools to achieve a basic level of coverage 1. Pact 2. Pact DSL for Mocking 3. Pact Broker 4. API Specification such as OpenAPI 5. Tool to run API Specification as Test Has significant learning curve |
NO The Bring Your Own tools approach adds to the list of Pact OSS 1. PactFlow 2. Pact DSL for Mocking AND / OR other tools like Mountebank 3. API Specification such as OpenAPI 4. Ready API, Dredd, etc. to run API Specification as Test 5. PactFlow Broker S |
| Can all stakeholders participate / author Contracts / API Specifications |
YES Devs, architects, test engineers, practically anyone who can author an OpenAPI Spec. |
NO Developer/SDET only because they have to generate to Pact JSON / Contract only after they have written a mock with Pact DSL |
PARTIAL Only Provider Spec / OAS can be written by anyone Consumer Contract has to be written / generated by Consumer Developer / Testers |
| Suitable for adoption in Greenfield Projects |
YES Specmatic help the team become productive quickly with Contract Driven Development |
YES |
NO Better suited for “Retrofitting contract-tests onto existing systems” |
| Suitable for adoption in Brownfield Projects |
YES Specmatic can immediately provide value on existing projects and helps the team slowly adopt Contract Driven Development |
NO Better suited for “Greenfield projects” |
YES WITH CAVEAT – Involves Guarantee Tradeoffs |
| Minimal / No Leakage for Compatibility Issues |
YES Specmatic keeps a tight check on both the Consumer and Provider and immediately providers feedback on deviation from API Specification thereby preventing Compatibility Issues |
YES While Code First, Consumer Driven Contract has a narrow set of use cases where it is suitable, within that narrow area it is able to prevent / reduce compatibility issues. |
NO BiDirectional Contract Testing has lower guarantees in comparison to Consumer Driven Contract / Pact OSS |
| Low cost of maintenance |
YES Since Specmatic is working off of your API Specifications there is no maintenance overhead. You just need to maintain your Stub / Expectation data. |
NO In comparison to Specmatic which only needs API Specifications, you need to maintain the Pact JSONs. |
NO Has higher cost of maintenance than even Pact OSS |
| Open Source |
YES License – MIT |
YES |
NO – Involves Proprietary Code |
| Language and Platform agnostic |
YES Is able to work completely as a standalone executable |
NO Pact DSL is language specific. While Pact supports several languages, if your language is not supported, you will have to try alternate approaches |
YES WITH CAVEAT – With Bring Your Own Tools approach this should be possible, however it comes at additional effort. Example: If you are using Mountebank instead of Pact Mocking, imposters have to be converted to Pact Files |
| Unlimited Team Size |
YES |
YES |
NO Team sizes vary based on the Pricing Plan |
| Broker Less |
YES Requires no server setup |
NO Pact Broker has to be run on a server where Pacts are uploaded, requires maintenance |
NO PactFlow involves cloud based Pact Broker, on-prem options seem to be available with enterprise pricing plan |
|
Specmatic.io |
Pact.io |
Pactflow.io |
