Skip to content

DESIGN.md

Latest commit

 

History

History
104 lines (66 loc) · 4.3 KB

DESIGN.md

File metadata and controls

104 lines (66 loc) · 4.3 KB

Design

Goals

aep-e2e-validator covers the gap of validation of runtime functionality defined in aep.dev.

Design Details

Code

The code is written Go, leveraging the shared functionality in the aep-lib-go repository.

The code is organized under individual rules, which can be referenced by the AEP that they are validating, along with the semantic behavior (e.g. aep-133-create). Each individual test is located inside of it's own file.

Each test is composed of precondition, setup, test, and teardown function calls and hooks, mapped to a common struct.

The various common exit codes (precondition failed, teardown failed, etc.) are defined in the pkg/validator/exit_codes.go file.

Interface

The tester will be available as a binary, and will be invoked as follows:

aep-e2e-validator validate --config <path-to-openapi-spec> --collection <collection-name>

All collections can be validated using the --all-collections flag:

aep-e2e-validator validate --config <path-to-openapi-spec> --all-collections

--all-collections is mutually exclusive with --collection.

By default all tests will be run for a given collection. This can be overridden using the --tests flag, which will allow the user to specify a list of tests to run.

aep-e2e-validator validate --config <path-to-openapi-spec> --collection <collection-name> --tests aep-133-create,aep-133-duplicate-creation-check

Testing Approach

The validator, by necessity, must operate on the resources that are exposed by an API. Therefore, the validator should be run against a staging or testing API.

The test suite runs in a single pass, due to the prerequisites for the state of the backing data (e.g. to delete a resource, a resource must already exist). If a specific test fails and success is a prerequisite for the next test, execution will stop there.

Testing on a collection

The e2e test can be run on a per-collection basis. The API payload is generated by examining the openapi schema of the resource, and constructing a payload that only contains the required values.

Testing child collections

Some collections are children of a separate collection, requiring a parent resource to be specified in order to be tested properly.

In this case, the user specifies a parent resource directly:

aep-e2e-validator validate --config "http://localhost:8000/openapi.json" --collection books --parent "shelves/horror"

Individual tests

Many of the tests require changes to the backend to run. For these tests, the preconditions of the test (e.g. a resource doesn't exist) will be verified before the test is run. If the precondition is not met, the test will fail with exit code 2 (precondition not met).

At the end of the test, the mutations applied on the collection will be attempted to be reversed (e.g. if a resource was created, it will be deleted). If that is not possible, the test fails with exit code 3 (teardown failed).

Failure exit codes

  1. Test failed.
  2. Precondition not met.
  3. Teardown failed.
  4. Setup failed

Tests to be created

To start with, the following tests will be created:

  • aep-132-list-resources-limit-1: Attempt to list resources with a limit of 1.
  • aep-132-list-resources-page-token: Verify a list request that does not
  • aep-133-create: Create a resource and verify it was created.
  • aep-133-duplicate-creation-check: Attempt to create a resource with the same ID twice, and verify it fails. enumerate the full list returns a page token, and the page token can be used to submit a subsequent request to list the rest of the resources.
  • aep-134-update-resource: Update a resource and verify it was updated.
  • aep-135-delete-resource: Delete a resource and verify it was deleted.
  • aep-135-delete-nonexistent-resource: Attempt to delete a non-existent resource and verify it returns 404 not found.

Global setup

When running the test suite over and over again, improper cleanup may result in garbage being collected in the collection-under-test.

To avoid the friction with having to clean up the collection manually, a global setup and teardown will be run once before and after the test suite is run.

The global setup and teardown would include, but not be limited to:

  • List, then delete all resources in the collection.