Comparison: Specmatic vs Pact.io and Pactflow.io

Specmatic is unique in its approach towards leveraging API Specification as Executable Contracts. However putting ourselves in the shoes of someone evaluating a process or a tool, it always helps to have detailed comparisons with other tools that may have some similarity. This is the first post in a series where we will be covering techniques and tools that try to address the problem of identifying compatibility issues between microservices early.

In this post we will be be looking at Pact which largely popularised Contract Testing. Pact.io or OpenSource Pact is tool that I have personally used and it makes for a great addition to the Dev experience if your development style / process aligns with the workflow of the tool (Consumer Driven Contract Testing). Pactflow.io supports another technique called Bi-Directional Contract Testing (BDCT) which is an interesting approach. Specmatic and Contract Driven Development differ significantly from Pact. We built Specmatic out of our own necessity where Pact did not fit the bill for us. We have tried to cover as many areas of comparison as possible, however we would love to hear if you have any other criteria on which you would like to see additional details.

Platform	Specmatic.in	Pact.io	Pactflow.io Paid
Approach	Contract Driven Development (CDD)	Code First, Consumer Driven Contract Testing	Bi-Directional Contract Testing (BDCT)
OpenAPI Support	YES	NO OpenAPIs and Pact are designed with different ends in mind	PARTIAL, only supports OAS as Provider Spec, Several Limitations – No Remote Refs / Single document approach, Need to dereference “$Ref”, and more
Suitable for Consumer First Development Style (Frontend Before Backend)	YES Consumer Team can author the API Specification, leverage it as mock / stub and then share it with Provider team as a guideline	YES Pact requires starting with Consumer development first by mocking the Provider and then leveraging these mocks to generate the Contract for the Provider	YES same as Pact OSS
Suitable for Provider First Development Style (Backend Before Frontend, public facing APIs, ec.)	YES Provider Team can author the API Specification, leverage as tests and then share it with Consumer team to stub the Provider	NO Pact is better suited for Consumer First Development Style	YES Provider has to publish their OAS to PactFlow server, they will only get feedback after Consumer has publish their Contract and PactFlow has completed Cross Contract Verification
Suitable for API First Development Style	YES Consumer and Provider Teams collaboratively author API Specification and only then being development	NO It does not support Contract / API Spec first Development Style	YES Consumers still cannot use the API Specification as Stub, they will only have to Cross Contract validate with the Specification
Parallel Development of Consumer and Provider	YES This is one of the primary capabilities of Specmatic, to allow teams to independently develop and confidently deploy microservices	NO Consumer development has to be ahead of Provider development	YES
Backward Compatibility Verification #NOCODE	YES This is one of the flagship capabilities in Specmatic. Contract vs Contract comparison, no need to implement the changes in Provider or Consumer Code	NO Compatibility can only be possibly detected after an actual change happens in Provider code, two version of Pact JSONs cannot be compared to see if the changes are compatible	NO Same as Pact OSS
API Specifications as Test #NOCODE	YES Specmatic can generate and run tests, both happy path and boundary conditions, all with just your OpenAPI specifications	NO Pact JSONs are the ones that are run as test, OpenAPI cannot be run as tests	YES WITH CAVEAT – External Tool Dependency with Bring Your Own testing tool such as Dredd – Also several limitations / considerations
API Specifications as Stub #NOCODE	YES OpenAPI is all you need to run a stub server, no need to define a stub, just the expectation data	NO Pact DSL necessary to define Stubs	NO When leveraging Bring Your Own mocks, the frameworks such as Mountebank, Wiremock, etc. have to be setup
Smart Mocks (Stale Stub Detection / Expectation Validation against API Specification)	YES Immediate Feedback, expectations that are not as per API Spec are immediately rejected with proper error message	NO / not Applicable. Contracts are generated based on the expectation, stale stub detection can only happen when pact json is run as test against Provider	NO / not Applicable. PactFlow allows Bring Your Own (BYO) mocks, even with Mountebank the approach has no immediate verification of stub data with Specification. Mountebank imposters have to be converted to Pact Files which are then uploaded to PactFlow where it is compared with Provider’s OAS. Almost 3 levels of indirection before you know whether the stubs are stale.
Unfragmented Devex / Minimal learning curve	YES You just need API Specification of your choice such as OpenAPI and Specmatic – no additional DSLs	NO Need to pick up several tools to achieve a basic level of coverage – Pact, Pact DSL for Mocking, Pact Broker, API Specification such as OpenAPI, Tool to run API Specification as Test. Has significant learning curve.	NO The Bring Your Own tools approach adds to the list of Pact OSS – PactFlow, Pact DSL for Mocking AND / OR other tools like Mountebank, API Specification such as OpenAPI, Ready API, Dredd, etc. to run API Specification as Test, PactFlow Broker Server. If teams within an organization are leveraging a mix and match of these tools that makes it hard for people to move across teams.
Can all stakeholders participate / author Contracts / API Specifications	YES Devs, architects, test engineers, practically anyone who can author an OpenAPI Spec.	NO Developer/SDET only because they have to generate to Pact JSON / Contract only after they have written a mock with Pact DSL	PARTIAL Only Provider Spec / OAS can be written by anyone. Consumer Contract has to be written / generated by Consumer Developer / Testers.
Suitable for adoption in Greenfield Projects	YES Specmatic helps team become productive quickly with Contract Driven Development	YES	NO Better suited for “Retrofitting contract-tests onto existing systems”
Suitable for adoption in Brownfield Projects	YES Specmatic can immediately provide value on existing projects and helps the team slowly adopt Contract Driven Development	NO Better suited for “Greenfield projects”	YES WITH CAVEAT – Involves Guarantee Tradeoffs
Minimal / No Leakage for Compatibility Issues	YES Specmatic keeps a tight check on both the Consumer and Provider and immediately provides feedback where they deviate from API Specification thereby preventing compatibility Issues	YES While Code First, Consumer Driven Contract has a narrow set of use cases where it is suitable, within that narrow area it is able to prevent / reduce compatibility issues.	NO BiDirectional Contract Testing has lower guarantees in comparison to Consumer Driven Contract / Pact OSS
Low Cost of Maintenance	YES Since Specmatic is leveraging your API Specifications there is no maintenance overhead. You just need to maintain your Stub / Expectation data.	NO In comparison to Specmatic which only needs API Specifications, you need to maintain the Pact JSONs.	NO Has higher cost of maintenance than even Pact OSS
Open Source	YES License – MIT	YES	NO – Involves Proprietary Code
Language and Platform Agnostic	YES Is able to work completely as a standalone executable	NO Pact DSL is language specific. While Pact supports several languages, if your language is not supported, you will have to try alternate approaches.	YES WITH CAVEAT – With Bring Your Own Tools approach this should be possible, however it comes at additional effort. Example: If you are using Mountebank instead of Pact Mocking, imposters have to be converted to Pact Files
Unlimited Team Size	YES	YES	NO Team sizes vary based on the Pricing Plan
Broker Less	YES Requires no server setup	NO Pact Broker has to be run on a server where Pacts are uploaded, requires maintenance.	NO PactFlow involves cloud based Pact Broker, on-prem options seem to be available with enterprise pricing plan

Related Posts

1 thought on “Comparison: Specmatic vs Pact.io and Pactflow.io”