API Blueprint & Testing

Mission: Test backend responses against what is defined in a blueprint. This is the blueprint validation.

With API Blueprint in CI the goal is to minimize number of ambiguous transactions in blueprint and support complex tests using scenarios.

Terms

Test Case

An HTTP transaction to be tested. All the data needed to complete an HTTP transaction. A Well-defined Transaction.

Scenario

A scenario is a sequence of steps. Where a step is either another scenario, a test case or a freeform text.

Background

A setup block referencing a scenario.

Test Case

Implicit Test Case

An implicit test case is a test case created as a by-product of API Blueprint reference style description of an API.

Question

Can an API blueprint be, by default, composed of implicit test cases or do they need to be explicitly stated?

Collision points

• Parameters of an HTTP Request

  • URI parameters
  • URI Query parameters
  • HTTP message headers
  • message-body properties

• Ambiguous Transactions

  A transaction with multiple defined responses.

• Stateful Server

  Possible need for some sort of set-up or ordering of test cases

Proposal

Proposed steps to resolve collision points:

• Parameters of an HTTP Request

  Parser warnings, when there is no example in parameter's description.
  HTTP-message headers and a message-body properties are already addressed by API Blueprint specification-by example.

• Ambiguous Transactions

Parser warnings + introducing possible keyword to denote a response that shouldn't be tested.

E.g.:

+ Response 201
+ Possible Response 401
+ Possible Response 503

Explicit Test Case

Inside out approach where somebody writes tests for an API and the reference-style documentation is the by-product.

Scenario

Ad-hoc Scenario

A scenario written after your API is described in an API blueprint. References to test cases and / or resources and actions.

Scenario-driven Blueprint

A freetext scenario. A scenario discussing and API and thus implicitly describing the API.

Background

A technicality, a reference to another scenario.

# Scenario A
 ...
# Scenario B
 ...
# My Resource [/resource]
## Retrieve [GET]
### Transaction Example 1
+ Background [Scenario A][]
+ Request A
+ Response 200
+ Possible Response 200
+ Possible Response 200
### Transaction Example 2
+ Background [Scenario B][]
+ Request B
+ Response 200

Notes

Transaction Example

As of API Blueprint Format 1A the API Blueprint support "under the hood" transactions examples. With implicit "pairing" planned in its parser.

The upcoming API Blueprint revision should consider introducing explicit support for defining transaction.