Testing

Status: current. This page describes the suites as they stand after Phase 7, the last phase of the refactor plan.

Deckard has one suite per kind of failure. Each catches something no other suite can, so a change is verified by the set, not by one suite.

The suites

npm run check-types and npm run lint are not suites, but they are the per-commit gate. check-types runs tsc over the root config and over src/webview/tsconfig.json, which checks page code against the DOM. That is what checks the pages’ scripts: until Phase 6 step 7, when the last template page was gone, test:ui also ran checkWebviewScripts.js, which pulled each template page’s inline script out by regex and type-checked it with strict: false; it is retired and deleted. lint runs ESLint over the source (.ts and .tsx), the tests, and the scripts, and dependency-cruiser over the imports (layers.md). bench:index measures indexing speed and asserts nothing; among its timings is how long the Stats, Task Board, and search page snapshots take to build, which decides whether a page’s HTML carries its first snapshot (Q3 in webviews.md). LAYOUT_TIMING=1 node test/ui/checkLayout.js times each surface’s first render in Chrome in real time instead of checking its layout, and scripts/measure-page-bundles.js uses it, with jsdom and a VSIX, to compare one bundle per page with a shared one (webviews.md).

compile-tests builds incrementally, with its build info in out/.tsbuildinfo, so a second test:* script recompiles only what changed, about 3 seconds rather than 13, and clearing out/ starts it afresh. Every page links its style sheets from dist/webview/, and every page that runs a script loads its bundle from there too, so a suite that draws a page builds them first: test:unit, test:ui, test:contrast, test:e2e, test:layout, test:visual, and test:dom each run npm run build:webview after compile-tests, and pretest runs node esbuild.js, which builds the host and the pages. node scripts/check-bundle-inputs.js checks what each bundle took in, from the metafiles the last build wrote to out/: the host only the third-party packages it took in when the check arrived, a page’s script only Preact, and a page’s sheet nothing (webviews.md).

Which suites run where

A suite runs under test:unit when none of the modules it imports reaches vscode. test/harness/importGraph.js reads that from dependency-cruiser each time, and .vscode-test.mjs asks it for the rest, so every test runs exactly once and a suite moves to the fast tier as soon as the refactor cuts its last path to vscode. After Phase 7, 126 suites run under plain mocha and 95 in the host; the import-graph inventory says what keeps the rest there.

Those host suites run with no workspace open, so a write to the workspace’s or a folder’s settings has nowhere to go. The suites in src/test/scopes/ test where a setting is written and which level’s value wins, so .vscode-test.mjs runs them twice more, labeled single-folder and multi-root, each in a workspace it makes empty in the temporary directory at the start of the run. listHostSuites lists only src/test/*.test.ts, so the run without a workspace never reaches them, and npx vscode-test --label multi-root runs one of the three alone. src/test/scopes/scopes.ts sets a value at the user, workspace, or folder level as a reader would, and records the context keys Deckard sets.

The shared harness

The Node harnesses share three modules in test/harness/, so a page or a file can move without each harness changing:

Tests that assert on CSS text read the sheets as written under src/webview, through src/test/sheets.ts: readSheet and themeSheet read one file, expandSheet puts each @import in place as esbuild does, linkedSheets lists the sheets a page’s shell links in cascade order, and pageSheets and withSheets give every rule a page draws with, the latter after its HTML. verifyWebviews.js holds the zen sheet, as written, to be the last layer of every loaded page.

src/test/pages.ts is the page catalog: every webview page and how to render it. Every mocha suite that builds a page calls its renderPage(id, options), and test/ui/pages.js takes its pages from it too, so a builder’s signature changes in one file. Each page renders against the one stand-in webview of src/test/pageWebview.ts, which names no vscode type: its extensionUri is the repository, as a stand-in URI a builder joins onto with with as it would a vscode.Uri, and its asWebviewUri hands out every file of the repository as vscode-webview://deckard/<path>, the address the page loader reads back. A page that loads its bundle from dist/webview therefore runs in every harness, and gets the same URIs on every machine. The e2e stub’s asWebviewUri maps dist/webview the same way. Every Node harness that runs the compiled sources outside VS Code loads that stub with its install(), which makes require('vscode') resolve to it: the e2e suites and the checks that draw surfaces through support.js, and the index bench. test/ui/pages.js installs none, since the catalog it renders from reaches no vscode. A page builder reads no settings: it is given the theme and zen mode it draws in, as chrome, which a host reads as it writes the page and the catalog’s renderPage takes as an option (Corpo with zen off when a suite names none). So the suites that only render catalog pages run under test:unit, and test/ui/pages.js passes the theme and zen mode it draws each page in the same way. A host test that hands the extension’s folder to VS Code itself makes it with vscode.Uri.file(REPOSITORY_ROOT). openWebviewPage(html, state, options) in src/test/webviewPage.ts runs a page in jsdom for a mocha suite, and test/e2e/support.js mounts one for an e2e suite. Both pass messages as JSON, as VS Code does. options.savedState is what the page’s getState returns from the start, as in a webview VS Code restored. webview-saved-state.test.ts uses it to pin what each page draws from every state shape a release has saved (the persisted-formats inventory, rows 20 to 24), and searchPage.e2e.js and taskBoard.e2e.js pin what the hosts reopen on.

Visual baselines are kept per platform in test/ui/visual-baseline/<platform>/, because the operating system rasterizes fonts, and macOS and Linux differ in every glyph’s edge. Chrome takes each screenshot at a moment it picks, so the harness ends every transition and animation at once and hides the caret: a page is compared where it settles, and an unchanged page draws identically run after run. The one exception is Synthwave’s Task Board, by status, by tag, and with a card’s menu open, whose columns still come to rest a strip of a card apart now and then, about 0.04% of the page; checkVisual.js names those three surfaces and lets them differ by 0.1% of the page rather than 0.01% until that is fixed. On macOS alone, Related Notes’ native select sometimes draws its chevron flipped, which no darwin baseline can settle, so a darwin-only list in checkVisual.js lets sidebarNotes and sidebarNotesUntagged differ by 0.1% there; Linux, the gate of record, never reads it. When a change in looks is meant, record it with npm run test:visual -- --update and commit the new baselines.

The layout, visual, and DOM checks draw the same surfaces, listed in test/ui/surfaces.js: a page, the snapshot it is drawn from, and the size it is drawn at. There are 21. Besides the Task Board, the calendars, Related Notes, Stats, and the search page, they cover the Dashboard (Home, Tags, Home being arranged, and Home’s search with its builder open), Help, a guide page shown in Help, the Notes Graph’s controls, and the Related Notes debug page, and two pages with a menu open, so a floating layer at the body’s level is drawn too. A surface may name messages its host sends after the state, such as Help’s guide page; drive, the [event, selector] pairs a reader’s actions dispatch, such as opening a menu; css of its own, such as the Notes Graph’s, which hides the canvas because its simulation settles differently on each run; and pageOptions its page renders with, such as the entry the debug page diagnoses. A driven surface hovers something only its action draws, so the layout check fails if the action stops reaching it. The guide page is a fixture, test/ui/fixtures/guide-daily-notes.html, captured in the extension host from renderGuidePage, which renders through VS Code’s markdown.api.render. The debug page runs no script, so its policy names no script-src; the harness adds one that admits only its own scripts, by the page’s nonce. Its tables wrap their headers, and one still too wide scrolls in its own box, so the page is held to its width like any other.

The pages’ shared core (src/webview/shared) is held to the template script it replaced by src/test/webview-shared.test.ts: src/test/sharedBundle.ts bundles core modules as esbuild builds a page, and the suite draws each component and requires the DOM the template’s helper drew, node for node, with the tip, the key sheet, and the undo toast opened as the template opened them. src/test/webview-tasks.test.ts does the same for the task row, its title, and its badges; src/test/webview-query-editor.test.ts drives the shared search box through the clicks, keys, and typing the template’s box was driven through, and requires at each step the DOM it drew, the searches it ran, and focus where it left it; and src/test/webview-search.test.ts draws every note excerpt of the sample corpus (test/fixtures/sample-corpus, once the Story Tour), the development notes, and the fixtures from its block tokens and through markdown-it as the sanitizer left it, and opens the tag menu the template opened. Each compares with normalizeBody from test/harness/domSnapshot.js, the normalization test:dom uses. Until step 7 the template’s side ran beside the component, from getComponentScript and getQueryEditorScript. Step 7 deleted those, so the suites first recorded what the template drew, ran, and returned at every comparison into src/test/fixtures/template/<suite>.json, which src/test/templateRecords.ts reads; a case with no recording fails rather than passing, and the recordings are never rewritten, since the template they record is gone. The task dates in them are local times, so they hold in every time zone, and a floating layer is compared with its neighboring text joined, since its recording is HTML. None of the four reaches vscode, so they run under test:unit: webview-search.test.ts reads the sample corpus through src/test/indexCorpus.ts, which takes the date tokens and file names from src/domain/notes/sampleNotes.ts, as the Work Sample’s command does. Since step 6 removed sanitize-html, the excerpts and titles take the template’s HTML from src/test/legacyMarkdown.ts, markdown-it with its renderer writing what the sanitizer kept, which agreed with rendering.ts on every excerpt and title the suites draw before that file went. It needs no @testing-library/preact, and openWebviewPage stays the page harness. Catalog pages that read their first snapshot from the shell say so with readsInertState, and renderPage(id, { state }) builds the shell with it, so a suite can draw a page both ways.

The DOM goldens in test/ui/dom-baseline/<surface>[+zen].html were recorded from the pages before Phase 6 rewrote any markup, and are re-recorded (npm run test:dom -- --update) only by a commit that names the difference it accepts. The layout check’s probe clones the body before it measures anything, and test/harness/domSnapshot.js normalizes the clone, so a golden holds only what a selector or a reader could tell apart. Scripts, style sheets, comments, and the probe’s output are dropped, and the nonce reads NONCE. Attributes and class names are sorted, a boolean attribute is written as "", and a style attribute is written from its declarations, sorted. A form control’s value, checkedness, and selectedness are read from the control and carried as data-dom-*, since a component sets properties where a template wrote attributes. Text is merged and its whitespace collapsed, but a text node of only whitespace is kept, and a difference of spacing alone fails as one, because a space between two inline elements is visible. The golden is HTML that parses back to the normalized body, with a line break inside each tag, so a diff reads line by line. The DOM is Chrome’s, laid out at each surface’s size, because the Task Board draws only the cards on screen. The goldens were recorded on macOS, and CI runs test:dom on Linux in a job of its own, so a golden holds on both; a failed run uploads the DOM each surface drew. Before any Chrome starts, the goldens’ names are checked against the surfaces: a golden nothing draws fails at once, and a surface without one is named.

Three checks used to find elements by reading markup out of the page scripts’ string literals, which compiled TSX does not have. Each now reads the drawn DOM as well, through test/harness/domGoldens.js, which parses the goldens; test:dom keeps the goldens equal to what the pages draw. The contrast check’s element shapes are the union of test/ui/contrast-shapes.json, the shapes it read from each page’s text before Phase 6, frozen with every attribute a rule asks about so states no surface draws stay in the matrix, and the shapes each surface draws. The drawn shapes found the calendar’s day and week buttons, which the text could not name; both are written background: none, which the model once read as no background declared, handing them the theme’s button fill, and reads now as transparent, as the browser does, so contrast-baseline.json is empty. The model reads no icon and settles a hovered element’s color by its state before its specificity, so the calendar’s icons and a hovered day outside the month are left to the rendered check. verifyWebviews.js holds content rows to a shared surface in both the page text, which still holds the body of a page whose host builds it (Help and the debug page), and the drawn DOM, and does not parse or nonce-check a page’s application/json state. The “no native title” test reads both, and an ESLint rule forbids title on a control or an element with tabIndex in .tsx, for the states no surface draws. The drawn DOM found one it could not read before, the Notes Graph’s tag filter checkboxes, which carried a native title from the page’s script until they took the page’s tip instead. The “starts busy” test reads the loaded page’s DOM.

The page-driving suites can record every state they reach, for the states no surface draws: an open menu, a builder row, a column removed. With DECKARD_DOM_RECORD=<dir>, openWebviewPage writes the page’s normalized body after each send and click, and test/e2e/support.js after each message the host delivers and each reader action, keyed by suite file, mount, and step (test/harness/domRecorder.js). Run the suites on the phase base and on a branch, each into its own folder, then node test/ui/diffDomRecords.js <base> <branch> lists every state that differs. A few states draw the wall clock, such as Stats’ last refresh, so a second base run passed as a third folder sets aside the states that differ from run to run. The records are not committed. openWebviewPage’s clockStep option makes performance.now() move on a fixed amount per reading, so a page that animates by the clock, such as the Notes Graph, draws the same frames every run. Its clock option gives the page a clock the test moves by hand (src/test/manualClock.ts): the window’s timers and Date.now wait until page.clock.advance(ms), which runs every timer due by then in the order each fell due, so a test of a 400 ms pause takes no 400 ms and can pin the pause to the millisecond. installManualClock(globalThis) does the same for code that runs in Node, such as the debouncers. Each timer path keeps one test on the real clock, so a bug only real timers show still fails somewhere.

Drawing pages at once

The layout, rendered-contrast, and visual checks each open a headless Chrome for every surface in every pass, some 336 pages a check. Opened one at a time, test:layout took 24 minutes and test:visual 19 on a 16-thread Mac. They now draw through a small pool, test/ui/chromePool.js: UI_CONCURRENCY=<n> Chromes at once, or, unset, half the machine’s logical cores, at least one and at most four. UI_CONCURRENCY=1 draws one page at a time, as the checks always did. Each Chrome gets a profile of its own under the temporary folder, incognito, since a headless Chrome with a profile on disk writes the page out and then never exits, and runs in a process group of its own, so a wedged Chrome is killed with its helpers and a check never kills a Chrome it did not start. A wedge is still retried once per page, then fails that surface with the same message. What each page prints is held until every page before it has printed, so a run reports the same surfaces, in the same order, with the same lines and summary, as a run one at a time. Files are named by pass and surface, so no two pages share one.

A Chrome costs some five seconds of CPU to start, draw one page, and quit, so the pool runs out of cores before it runs out of pages: on that Mac, four at once drew 48 pages in 55 seconds against 86 one at a time, and eight in 51, no faster. Time in each Chrome is virtual, and it runs on while Chrome loads the page’s frame, so the busier the machine, the later in virtual time a page finishes: its probe, written about 0.25 seconds in when one page is drawn at a time, came as late as 2.3 seconds of the 3-second budget with eight at once, and 0.7 with four. That is why the default stops at four, and why the page the layout and contrast checks build no longer reads the probe at a fixed 0.4 seconds but waits for it. Measured on that Mac on one afternoon, the unchanged checks took 20.8 minutes for layout and rendered contrast and 11.2 for the visual check, one page at a time; with four at once, npm run test:layout took 12.3 minutes and npm run test:visual 6.9, the build each runs first included, and every surface had the same verdict. A wedged Chrome now holds up one of the four rather than the whole run. test:dom and LAYOUT_TIMING=1 still draw one page at a time.

Shards

UI_SHARD=<k>/<n> makes the layout, rendered-contrast, and visual checks take every n-th of the sixteen passes, starting with the k-th (test/ui/passes.js), so the shards of one n make every pass exactly once; CI runs each check in two. The matrix is split, never sampled: 162a7c7c failed only in Corpo with zen. Before any Chrome starts, the visual check compares the baselines’ names with every surface in every pass, whatever the shard: a baseline nothing draws fails in the first second, and a surface with none is named at once and still drawn, recorded, and, under --ci, failed by the shard that draws it. VISUAL_CHANGED=<dir> copies there only what a failure needs: each image drawn with no baseline or differing from it, under its baseline’s name, and its diff.

Unit suites at once

test:unit runs each suite file in one of a few worker processes (mocha’s parallel mode): one fewer than the logical cores, at least two, at most four on CI and eight elsewhere, or UNIT_JOBS=<n>. On a 16-thread Mac the 162 files took 50 seconds one after another and 16 at once. UNIT_SERIAL=1 runs them in one process, one after another, as the nightly run does, so a suite that leans on what another file left behind, or on the order files run in, still shows. On CI (CI set) a .only left in a suite fails the unit, host, and end-to-end runs rather than running one test and passing.

When to run what

A refactor phase lands only on a clean full run with no pixel changes. A visual diff during a refactor means behavior changed, which is a bug in the refactor.

Continuous integration

ci.yml runs once for each commit: on every pull request, and, through workflow_call, as the first job of a release. A push runs nothing of its own; a branch that wants CI opens a pull request, a draft if it is not ready, or runs the workflow by hand. Its work is split between jobs that run at once, so a run takes as long as its slowest job, about 6 to 8 minutes where the one job it replaced took 18 to 27:

The Chrome checks draw in a pinned Chrome, CHROME_VERSION in ci.yml: Chrome for Testing, installed by browser-actions/setup-chrome and passed as CHROME_PATH, at the version the Linux baselines were drawn in, since the runner’s own Chrome moves with each image. Raise it in the commit that re-records the Linux baselines. The host suites run in a pinned VS Code, VSCODE_VERSION, which .vscode-test.mjs reads (today’s stable when unset), and its download is cached by system and version. With --ci, a surface with no baseline fails the run rather than being recorded; a failed visual shard uploads as its artifact only the images that differ or had none, under their baselines’ names, and their diffs, and a failed DOM check the DOM each surface drew. Dependabot (.github/dependabot.yml) opens the updates, to dev, for every package and every action.

nightly.yml runs the same set on dev each night, and by hand on any branch, the slow way: one Linux job with the unit suites in one process (UNIT_SERIAL=1) and each Chrome check over all sixteen passes, the unit and host suites on Windows and macOS, and the host suites in VS Code 1.134.0, the oldest engines allows, where CI runs a recent one. It catches what the split run can hide: a suite that leans on another file’s state or on the order files run in, a test that fails only on a later date, and an API newer than engines promises. A scheduled workflow runs from master’s copy of the file.

prepare-release.yml cuts a release, the version and the changelog’s section, when the pull request from dev is given the release label, or when run by hand on dev, not on every push, since each cut commit starts a full CI run of its own.

release.yml calls the whole workflow as a reusable workflow before it builds and publishes, so a merge whose CI run was cancelled is not released untested. Then it runs three jobs, each with only the permissions it needs: build packages the VSIX once and signs its provenance (gh attestation verify <vsix> --repo doctorallen/deckard checks it); github-release creates the release with that file; marketplace publishes the same file from the marketplace environment, which only master may deploy to and which the Microsoft Entra credential trusts. Every action is pinned to a full commit SHA, with its version in a comment, and codeql.yml scans the code and the workflows on each pull request, after each merge, and weekly.

A page’s host-controller tests run its PageController in a WebviewHost attached to the FakeSurface of src/test/fakeWebview.ts, around a FakeWebview that has only postMessage and onDidReceiveMessage. A test sends what the page would with await surface.webview.send(message), which resolves when the handler has finished, and reads what the host posted in surface.webview.posted. stats-host.test.ts is the first; webviews.md says how each page adds its own. navigation.e2e.js pins, against the hosts themselves, which lines and tags each page’s openSource and openTag open, so a host moved onto the shared handlers is held to what it accepted before.