Comparison: Specmatic vs Pact.io and Pactflow.io

By Hari Krishnan

Share this page

Specmatic is unique in its approach towards leveraging API Specification as Executable Contracts. However putting ourselves in the shoes of someone evaluating a process or a tool, it always helps to have detailed comparisons with other tools that may have some similarity. This is the first post in a series where we will be covering techniques and tools that try to address the problem of identifying compatibility issues between microservices early.

In this post we will be be looking at Pact which largely popularised Contract Testing. Pact.io or OpenSource Pact is tool that I have personally used and it makes for a great addition to the Dev experience if your development style / process aligns with the workflow of the tool (Consumer Driven Contract Testing). Pactflow.io supports another technique called Bi-Directional Contract Testing (BDCT) which is an interesting approach. Specmatic and Contract Driven Development differ significantly from Pact. We built Specmatic out of our own necessity where Pact did not fit the bill for us. We have tried to cover as many areas of comparison as possible, however we would love to hear if you have any other criteria on which you would like to see additional details.

Platform	Specmatic.io	Pact.io Pact OSS	Pactflow.io Paid
Approach	Contract Driven Development (CDD)	Code First, Consumer Driven Contract Testing	Bi-Directional Contract Testing (BDCT)
OpenAPI support	YES	NO (OpenAPIs and Pact are designed with different ends in mind)	PARTIAL, only as Provider Spec (OAS as the provider spec) Has several limitations – no remote refs / single document approach, need to dereference “$Ref”, and more
AsyncAPI support	YES Full Contract Testing vs Async Mocking on a paid plan	NO	NO Plugins may provide limited support.
Suitable for Consumer-first development style (Frontend before backend)	YES Consumer Team can author the API Specification, leverage it as mock / stub and then share it with Provider team as a guideline	YES Pact requires starting with Consumer development first by mocking the Provider and then leveraging these mocks to generate the Contract for the Provider	YES same as Pact OSS
Suitable for Provider-first development style (Backend before frontend, public facing APIs, etc.)	YES Provider Team can author the API Specification, leverage as tests and then share it with Consumer team to stub the Provider	NO Pact is better suited for Consumer First Development Style	YES Provider has to publish their OAS to PactFlow server, they will only get feedback after Consumer has publish their Contract and PactFlow has completed Cross Contract Verification
Suitable for API-first development style	YES Consumer and Provider Teams collaboratively author API Specification and only then begin development	NO It does not support Contract / API Spec first Development Style	YES Consumers still cannot use the API Specification as Stub, they will only have to Cross Contract validate with the Specification
Parallel development of Consumer and Provider	YES This is one of the primary capabilities of Specmatic, to allow teams to independently develop and confidently deploy microservices	NO Consumer development has to be ahead of Provider development	YES
Backward Compatibility Verification #NOCODE	YES This is one of the flagship capabilities in Specmatic. Contract vs Contract comparison, no need to implement the changes in Provider or Consumer Code	NO Compatibility can only be possibly detected after an actual change happens in Provider code, two version of Pact JSONs cannot be compared to see if the changes are compatible	NO Same as Pact OSS
Example generation and validation for OpenAPI	YES	NO	NO
Proxy based OpenAPI generation, validation	YES	NO	PARTIAL bring your own tools
GUI based visual #NOCODE testing	YES Specmatic Studio unlocks this (paid)	NO	NO
Contract Conformance Report	YES	NO	NO
Dashboarding & service dependency mapping	YES comprehensive with Insights	NO	PARTIAL
API specifications as Test #NOCODE	YES Specmatic can generate and run tests, both happy path and boundary conditions, all with just your OpenAPI specifications	NO Pact JSONs are the ones that are run as test, OpenAPI cannot be run as tests	YES WITH CAVEAT – External Tool Dependency with Bring Your Own testing tool such as Dredd – Also several limitations / considerations
API specifications as Mock #NOCODE	YES OpenAPI is all you need to run a stub server, no need to define a stub, just the expectation data	NO Pact DSL necessary to define Stubs	NO When leveraging Bring Your Own mocks, the frameworks such as Mountebank, Wiremock, etc. have to be setup
Smart Mocks (Stale mock detection / Expectation validation against API specification)	YES / Immediate Feedback Expectations that are not as per API Spec are immediately rejected with proper error message	NO / Not Applicable Contracts are generated based on the expectation, stale stub detection can only happen when pact json is run as test against Provider	NO / Not Applicable PactFlow allows Bring Your Own (BYO) mocks, even with Mountebank the approach has no immediate verification of stub data with Specification. Mountebank imposters have to be converted to Pact Files which are then uploaded to PactFlow wh
Unfragmented Devex / Minimal learning curve	YES You just need 1. API Specification of your choice such as OpenAPI 2. Specmatic – No additional DSLs	NO Need to pick up several tools to achieve a basic level of coverage 1. Pact 2. Pact DSL for Mocking 3. Pact Broker 4. API Specification such as OpenAPI 5. Tool to run API Specification as Test Has significant learning curve	NO The Bring Your Own tools approach adds to the list of Pact OSS 1. PactFlow 2. Pact DSL for Mocking AND / OR other tools like Mountebank 3. API Specification such as OpenAPI 4. Ready API, Dredd, etc. to run API Specification as Test 5. PactFlow Broker S
Can all stakeholders participate / author Contracts / API Specifications	YES Devs, architects, test engineers, practically anyone who can author an OpenAPI Spec.	NO Developer/SDET only because they have to generate to Pact JSON / Contract only after they have written a mock with Pact DSL	PARTIAL Only Provider Spec / OAS can be written by anyone Consumer Contract has to be written / generated by Consumer Developer / Testers
Suitable for adoption in Greenfield Projects	YES Specmatic help the team become productive quickly with Contract Driven Development	YES	NO Better suited for “Retrofitting contract-tests onto existing systems”
Suitable for adoption in Brownfield Projects	YES Specmatic can immediately provide value on existing projects and helps the team slowly adopt Contract Driven Development	NO Better suited for “Greenfield projects”	YES WITH CAVEAT – Involves Guarantee Tradeoffs
Minimal / No Leakage for Compatibility Issues	YES Specmatic keeps a tight check on both the Consumer and Provider and immediately providers feedback on deviation from API Specification thereby preventing Compatibility Issues	YES While Code First, Consumer Driven Contract has a narrow set of use cases where it is suitable, within that narrow area it is able to prevent / reduce compatibility issues.	NO BiDirectional Contract Testing has lower guarantees in comparison to Consumer Driven Contract / Pact OSS
Low cost of maintenance	YES Since Specmatic is working off of your API Specifications there is no maintenance overhead. You just need to maintain your Stub / Expectation data.	NO In comparison to Specmatic which only needs API Specifications, you need to maintain the Pact JSONs.	NO Has higher cost of maintenance than even Pact OSS
Open Source	YES License – MIT	YES	NO – Involves Proprietary Code
Language and Platform agnostic	YES Is able to work completely as a standalone executable	NO Pact DSL is language specific. While Pact supports several languages, if your language is not supported, you will have to try alternate approaches	YES WITH CAVEAT – With Bring Your Own Tools approach this should be possible, however it comes at additional effort. Example: If you are using Mountebank instead of Pact Mocking, imposters have to be converted to Pact Files
Unlimited Team Size	YES	YES	NO Team sizes vary based on the Pricing Plan
Broker Less	YES Requires no server setup	NO Pact Broker has to be run on a server where Pacts are uploaded, requires maintenance	NO PactFlow involves cloud based Pact Broker, on-prem options seem to be available with enterprise pricing plan
	Specmatic.io	Pact.io	Pactflow.io

Comparison: Specmatic vs WireMock

API mocking is only effective if the mocks are truly representative of the provider / services they are emulating. Deviations between mocks and providers can

Updates,

By Naresh Jain

Contract Testing using OpenAPI Specs as Executable Contracts

Demonstration video showing OpenAPI specifications being leveraged as executable contracts with Specmatic Sample projects with OpenAPI https://docs.specmatic.io/sample_projects.html#openapi

Contract Testing with Specmatic - Markus Oberlehner

In the Media, Updates,

By John

Stubbing an HTTP end-point with Specmatic

Markus Oberlehner gives a brief overview on the topic of Contract Testing with Specmatic and his experience using it for stubbing an HTTP end-point.

Updates,

By Joel Rosario

TMForum ODA CTK API specification conformance testing with Specmatic

We recently discovered some major issues with the TMForum Conformance Test Kit (CTK) v5.0.0 and will demonstrate how using Specmatic can address these problems.

Demonstration, Feature Spotlight, Updates,

By Jaydeep Kulkarni

Early detection of mismatches between your API specs and implementation: Exploring Specmatic’s API Coverage Report

Specmatic’s API coverage report helps identify any mismatches between an OpenAPI specification and an application's endpoints early in the development lifecycle. The report lists all

Dave Farley pointing to the Specmatic tool for easy microservice testing, featured on Dave Farley's Continuous Delivery channel.

In the Media, Updates,

By John

“Easy Microservice Testing” – Specmatic featured on Continuous Delivery channel

Dave Farley is a respected author and widely recognised proponent of Continuous Delivery. In this video on his Continuous Delivery YouTube channel he discusses why