Add test suite package#413
Conversation
✅ Deploy Preview for designtokensorg ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
This would fix issue #386 |
drwpow
left a comment
drwpow
left a comment
There was a problem hiding this comment.
These look great! And I see that the pnpm --recursive run test already picked up the new tests with the "test" npm script. I’ll wait for this to rebased off main, but otherwise LGTM and can’t wait to ship!
| @@ -0,0 +1,105 @@ | |||
| /** | |||
| * Schema validation tests using the VS Code JSON language service. | |||
There was a problem hiding this comment.
💯💯💯 I can’t believe we’ll get these (well, once the issue mentioned is fixed upstream)!!
| errorDetails: string; | ||
| } | ||
|
|
||
| /** Options for customising test generation. */ |
There was a problem hiding this comment.
Use US English "customizing" (and then remove "customising" from terminology.txt)
Introduce the test-suite package with positive and negative test fixtures for format and resolver schemas, and add AJV-based and VS Code validation tests to the schemas package. - Add test-suite package with 250 test fixtures organized by token type (colors, dimensions, borders, typography, etc.) - Add schema validation tests using AJV (vitest) - Add VS Code JSON language service tests (currently skipped pending upstream fix for $id-based $ref resolution) - Add test dependencies to schemas package (ajv, vitest, etc.) - Add test-suite to pnpm workspace - Add build:schemas step to CI workflow (required before tests)
dgtlntv
force-pushed
the
pr/02-test-suite
branch
from
14b10a2 to
2f3e66a
Compare
Rebased it and addressed @kaelig feedback as well! Once the fix lands for the VSCode JSON language service I would create a followup PR that un-skips the VSCode tests. |
3 participants
(This PR is stacked on top of #412. So the diff also includes the schema package changes from that PR. The new changes in this PR are focused on the test suite package and the schema tests that consume it.)
Changes
This PR introduces a test suite as a separate package. I originally created these fixtures to test the JSON schema while working on it, but I think they may also be useful to third-party design token tool developers. At the moment, every design token tool developer has to write their own fixtures to test things like token parsers. Being able to use official fixtures could make that easier.
For the DTCG, this gives the fixtures a dual purpose: they let us test the JSON schemas we publish, and they provide official test fixtures for the wider community.
This PR also adds tests to the
schemaspackage that use the fixtures to test the bundled schemas. These tests currently run withajv. I also added avscode-json-languageservicetest setup so we can verify that the JSON schemas work in VS Code as well. Those tests are currently skipped because they do not pass yet. We need to wait for the upstream fix mentioned in #412.The test setup is structured so it should be easy to add more JSON schema validators in the future if we want to compare behavior across multiple validators.
The test suite package itself is structured similarly to the JSON-LD Working Group test suites. Specifically it uses manifest files instead of relying only on folder structure, so each fixture has an explicit id, expected result, purpose, input path, and feature tags. That should make the fixtures easier to consume programmatically, easier to review, and easier to extend over time without forcing every implementation to infer test meaning from filenames alone.
I know this PR is quite big and probably hard to review. LMK if I should split it up in a certain way.
How to Review