@exercode/problem-utils

💯 A set of utilities for judging programs on Exercode (https://exercode.willbooster.com/).

Packages

Package	Purpose
@exercode/problem-utils	CLI, validation, result types, and stdio/command/GUI/evaluation presets; no browser or AI SDK dependency
@exercode/problem-utils-browser	Puppeteer Chromium lifecycle, browser judging, and screenshots
@exercode/problem-utils-llm	LLM judging and AI SDK providers

Install only the packages the problem uses. Import llmJudgePreset from @exercode/problem-utils-llm. Browser judges receive Puppeteer's native Page:

import { DecisionCode } from '@exercode/problem-utils';
import { browserJudgePreset } from '@exercode/problem-utils-browser';

await browserJudgePreset({
  testCases: [
    [
      'heading',
      async (page) => ({
        decisionCode:
          (await page.$$eval('h1', (elements) => elements.length)) === 1
            ? DecisionCode.ACCEPTED
            : DecisionCode.WRONG_ANSWER,
      }),
    ],
  ],
});

The browser preset serves the submitted directory, runs checks in order on one page, prints each result, stops on the first non-accepted result, and closes the browser even when a check throws. Checks return learner-facing verdicts; uncaught harness errors propagate to the caller. Use timeoutMs, viewport, contextOptions, and launchOptions to set problem-specific requirements. screenshotOnFailure attaches a full-page image to a non-accepted result; capture failures are recorded in stderr without replacing the verdict. launchBrowser, createBrowserPage, and captureScreenshot are also exported for harnesses that manage their own HTTP server or test loop. Use await createBrowserPage(browser) (or pass a browser context) to create a native Puppeteer page with the same dialog handling as the presets. These pages dismiss dialogs by default. Registering a dialog listener takes over handling: the listener must call dialog.accept() or dialog.dismiss(), even when it only inspects the message. Asynchronous handlers retain control until they handle the dialog.

The browser package depends on puppeteer. Before running browser judges or PDF export, install its version-matched Chrome headless shell explicitly when package installation scripts are disabled:

bun run exercode-browser browsers install chrome-headless-shell

exercode-browser runs this package's Puppeteer CLI, independently of an application's E2E test version. Docker/CI should install unzip before this command and Chrome's OS libraries at image build/setup time. Install the fonts required by the course content in that environment.

HTML comparison

htmlJudgePreset({ solutionDirectoryPath, requiredFiles?, compareDom?, textNormalizationPattern? }) compares the submitted page with the specified model-answer directory. It reports snapshot_body first, then screenshot, stopping on the first difference. Set compareDom: false for exercises that grade only the rendered appearance. The DOM comparison ignores comments and normalizes text whitespace and attribute order. textNormalizationPattern overrides the regular-expression source used to replace text-node matches with spaces before trimming; it defaults to "\\s+". Set it when a course requires a different text comparison rule. Screenshots render formatted HTML at 800×600, with CSS animations disabled and fonts loaded; a difference includes both PNG files. HTML decoding honors a BOM or declared HTTP/meta charset, defaulting to UTF-8 when none is declared; formatted responses explicitly use UTF-8. If either document cannot be decoded or formatted, both are rendered raw. Each check uses fresh, separate browser contexts for both answers. Pixel comparison requires deterministic page content; JavaScript timers, random content, and animated images are not frozen. Missing required files are reported before starting Chromium.

For isolated judging, keep shared assets inside the problem directory. Both directories can use the nearest ancestor's assets directory, through assets/ or directly from the served root. Local assets are merged with that shared directory; submission files and links take precedence, followed by local assets. A directory symlink overrides that entire directory rather than merging shared files into its target. Source files and linked directories are left unchanged. The temporary served directories and browser are closed after judging. The directory helpers leave process signals to the host. Hosts must provide and remove a per-run TMPDIR when a harness terminates before disposal; isolated CLI checks do this automatically for SIGINT, SIGTERM, and SIGKILL. Long-lived hosts can await disposal in their own shutdown handlers. captureHtmlBodySnapshot, captureHtmlScreenshotPair, and createHtmlServedDirectory expose the same operations for custom checks. The screenshot pair takes two { page, url } targets and returns PNGs in that order, formatting both documents or neither. Give those pages matching viewport options and separate fresh browser contexts. See the HTML example.

Custom browser pages and lifecycle hooks

browserJudgePreset accepts directoryPath to serve an assembled exercise directory, entryPath to select its initial page, and navigationOptions for native Puppeteer navigation settings. initializePage runs before navigation, so it can register console and page-error listeners. afterTests runs after the checks, including a failing verdict, while the page remains open. An exception in initialization, navigation, or a check propagates to the caller and skips afterTests; browser cleanup still runs.

Spring Boot courses

Import springBootJudgePreset from @exercode/problem-utils/presets/springBoot and pass problemDirectoryPath plus an async evaluate callback. The preset builds the problem's Maven project offline, starts its Spring Boot JAR on port 59000, and invokes the problem's judge.ts --evaluate. It preserves the course build, startup, and evaluation limits (90, 60, and 60 seconds) and emitted screenshot metadata. The host provides Java, Maven, Bun, cached dependencies, an exclusive execution slot, and interrupted-run cleanup. Evaluators can build their request URLs with buildSpringBootUrl from the same core subpath. Browser evaluation can use launchBrowser and captureTomcatScreenshots from the browser package; the Spring Boot preset itself adds no browser dependency to core.

JavaScript and Tomcat courses

javascriptJudgePreset(problemDirectoryPath, options) from @exercode/problem-utils-browser runs main.mjs or main.js in Chromium and compares console output with test_cases/*.out. The .in files contain browser setup JavaScript. Set initializeAndVerifyDom: true to call the setup's initializeTest and verifyDom hooks, and waitForConsoleIdle: true for exercises whose asynchronous console output must settle before comparison.

javascriptDomJudgePreset(problemDirectoryPath) supports DOM exercises whose .in files define initializeTest, verifyDom, or window.test. Setup scripts retain global declarations for submitted code and verification hooks. The preset exposes top-level function declarations to those callbacks, captures console output, and stops at the first failing case. Both JavaScript presets require a host-enforced overall timeout: the console-idle heuristics do not bound programs that keep emitting output.

tomcatJudgePreset and buildTomcatUrl are available from @exercode/problem-utils/presets/tomcat, without browser dependencies. The preset accepts problemDirectoryPath, jspDirectory ('' or 'WEB-INF/jsp'), optional forbiddenTexts, and an asynchronous evaluate callback. It builds the problem's pom.xml with Maven offline, serves the judge application on port 59000, and invokes the same judge.ts with --evaluate. The host must provide Maven, GNU time, GNU timeout, Bun, CATALINA_HOME, and exclusive access to that port. Build and evaluation limits are 60 and 30 seconds respectively.

Browser evaluations can use native Puppeteer pages together with captureTomcatScreenshots or verifyTomcatHtml from the browser package. The latter compares document markup with whitespace removed and records screenshots for the verdict. verifyTomcatPath(page, endpoint) waits up to five seconds for the endpoint's path and parsed document, reporting the expected and current paths in Japanese if navigation fails; query strings do not affect the check. All interrupted-run cleanup remains the host's responsibility. requirePageElement(page, selector) immediately returns a native Puppeteer element handle or reports the missing selector in Japanese; callers use the handle's native actions for course-specific interactions.

PDF export

import { markdownToPdf } from '@exercode/problem-utils-browser/pdf';

const pdf = await markdownToPdf(markdown, {
  assetDirectoryPath: import.meta.dirname,
  pdfOptions: { format: 'A4' },
});
await Bun.write('material.pdf', pdf);

PDF assets are served on IPv4 loopback, and encoded paths cannot escape the asset directory. Intentional asset symlinks remain usable. Core also exports startLocalHttpServer for callers that need the same local-only server; await its startup before using its address.

The PDF entry point removes YAML mapping frontmatter, renders Markdown with syntax highlighting and CJK-friendly emphasis, and resolves relative images against assetDirectoryPath, and waits for fonts and images before printing. Missing or invalid images leave browser placeholders without preventing the document from exporting. Pass mermaidScriptPath pointing to a Mermaid browser bundle to render diagrams, css to customize styling, and pdfOptions for native Puppeteer PDF settings. The defaults use screen media, A4 paper when no custom dimensions are supplied, printed backgrounds, and margins of 30 mm top/bottom, 40 mm right, and 20 mm left. PDF rendering dependencies are loaded through this subpath; importing the browser judging entry point does not load them.

CLI

The package ships an exercode-problem command for problem authors (run it with bun x in a repository that depends on @exercode/problem-utils):

# Judge all model answers of all problems (directories containing problem.md or <id>.problem.md) under a directory.
# model_answers/* must be fully accepted; model_answers.fails/* must fail at least one test case.
bun x exercode-problem                # everything under the current directory
bun x exercode-problem courses/foo    # everything under courses/foo
bun x exercode-problem --only a_plus --skip gui_ --concurrency 2

# Judge one answer directory of the problem in the current directory.
bun x exercode-problem judge model_answers/python
bun x exercode-problem judge model_answers/python '{ "language": "python" }'

# Debug one answer directory of the problem in the current directory.
bun x exercode-problem debug model_answers/python '{ "stdin": "1 2" }'

judge and debug run a custom judge.ts / debug.ts when the problem has one, and apply stdioJudgePreset / stdioDebugPreset otherwise, mirroring the Exercode server. The debug fallback applies only to standard problems: a problem with a custom judge.ts needs its own debug.ts, and exercode-problem debug fails with a message otherwise (the server likewise reports debug as unsupported there).

The all-problem check judges serially by default because time limits are measured in wall-clock time; pass --concurrency <n> to parallelize when the checked problems are not timing-sensitive.

A standard stdin/stdout problem must NOT commit a judge.ts or debug.ts that is identical to the default stdio harness: the absence of judge.ts marks the problem as standard, and committed copies would drift from the server's defaults. The CLI rejects such files; a file kept intentionally (e.g. to demonstrate the default harness) can add an explanatory comment to be treated as custom.

Validators

The CLI also validates learning-material files without running any program, mirroring the checks the Exercode importer applies:

# Validate problem directories (problem.md frontmatter, test_cases, model_answers, templates, judge.ts / debug.ts).
bun x exercode-problem validate-problem <problemDir>...
# Validate a course directory (course.yaml, lecture materials with embedded questions, problem references).
bun x exercode-problem validate-course <courseDir> [--problems-dir <dir>]
# Validate a contest (*.contest.yaml) file.
bun x exercode-problem validate-contest <contestYamlPath> [--problems-dir <dir>]

Each target prints OK or NG followed by its errors and warnings; the command exits 1 when any target has an error. --problems-dir points to the directory holding the referenced problems; a course is always searched at any depth, since Exercode links a material only to the problems inside its course, so for a course the option must name a directory inside it. The validators are also exported (validateProblemDirectory, validateCourseDirectory, validateContestFile, validateMaterialFile).

Agent skills

The skills/ directory holds skills for AI coding agents that author and review Exercode learning content: generate-learning-content (entry point), generate-course-materials, generate-judge-problems, generate-judge-contest, review-learning-content, and setup-exercode-course-repository. Agents working in this repository load them through the symlinks under .claude/skills/; install them elsewhere with the skills CLI:

bun x skills add WillBooster/exercode-problem-utils --agent claude-code --agent codex

Test cases

A problem keeps its test cases under test_cases/. A test case id is the shared name of the following entries, and each entry is optional:

Entry	Meaning
<id>.in	Standard input. Omit it (or leave it empty) when the program reads nothing.
<id>.out	Expected standard output.
<id>.fin/	Files copied into the working directory before the run (input files).
<id>.fout/	Expected output files, compared with the files of the same relative paths in the working directory.
_shared.fin/	Files copied into the working directory before every test case.
<id>.json	Configuration for a custom judge.ts that reads it itself; the presets ignore it (the Judge server lists it as a test case of the custom judge).

A test case whose id contains example (the judge server's rule, e.g. example_1 or 01_example_small) is an example shown to learners; every other case is hidden.

Standard output and text files are compared as space-separated tokens: consecutive white spaces count as one separator, and an expected token that contains a decimal point and parses as a finite number (e.g. 3.14, but not 1, 1e-3 or 1.0e309) accepts a value within an absolute or relative error of 1e-6. A file is text when it is valid UTF-8 without NUL bytes; other files (e.g. images) must match byte for byte. A received file larger than 8 MiB counts as not produced, an expected file larger than 8 MiB is an authoring error (the case is reported as a runtime error), and a file larger than 1 MiB is left out of the reported pair. When a file differs, the result carries <name>_expected.<ext> and <name>_received.<ext> so Exercode can show both (Exercode decides per test case whether a learner may see them, as it does for expected stdout).

How a missing expectation is treated depends on the harness:

• stdioJudgePreset (the default for problems without judge.ts) requires <id>.out or a non-empty <id>.fout/ for every test case, so a standard problem cannot accept a run without checking it. A problem whose problem.md declares requiredOutputFilePaths or isManualScoringRequired is exempt, because every test case is judged by those instead; code rules (requiredRegExpsInCode etc.) and requiredSubmissionFilePaths check the submission once, in addition to the output comparison, and do not exempt.
• commandJudgePreset without a test option checks whatever expectations exist, and a test case with neither only has to run within the limits. A test option replaces that comparison; it receives testCase.output, testCase.fileOutputPath and cwd and can call the exported judgeAgainstExpectations (or compareStdoutAsSpaceSeparatedTokens and compareExpectedOutputFiles). A custom readTestCases may return any test case type with id (plus optional input and fileInputPath); the default verdict judges any case that exposes a string output or fileOutputPath, which the default reader's CommandTestCase does.
• guiCommandJudgePreset passes the expectations to the problem's test, which decides everything.
• llmJudgePreset runs no program, so it copies no .fin/; it hands <id>.in as the prompt input and the whole entry (including fileOutputPath) to the problem's test.
• stdioDebugPreset by default copies the answer directory to a temporary directory and builds and runs it there together with _shared.fin/ and the first example case's .fin/ (the case's files win; hidden cases' inputs are never handed to learner code), so the answer directory is left untouched (its node_modules, and those of its ancestors, are linked into the copy rather than duplicated); files the program writes are reported only through requiredOutputFilePaths. Callers that already own a disposable answer directory can pass { disposableWorkingDirectory: true } as the second argument to build and run directly there; they must remove it themselves, including when the harness is terminated.
• evaluationJudgePreset does not use test_cases/.

readTestCases is exported for harnesses that enumerate test_cases/ themselves.

Measurements

stdioJudgePreset, stdioDebugPreset, and commandJudgePreset (with its default runner; a custom runCommand decides whether its results carry cpuTimeSeconds) run a program under GNU time (/usr/bin/time on Linux, gtime on macOS) and report its wall time (timeSeconds), user plus system CPU time (cpuTimeSeconds), and peak resident set size (memoryBytes, at least the footprint of the GNU timeout wrapper the program runs under, about 1 MiB) in every test case result; guiCommandJudgePreset reports the wall time and the peak resident set size, and llmJudgePreset the wall time only. Time limits are judged by wall time. The CPU time is recorded even for a run that exceeded its limit, so a judge server sharing CPUs between programs can tell a program that used up its limit (its CPU time, summed over its threads, reaches the limit, so it would exceed it on any single CPU) from one that may only have waited for a CPU (its CPU time stays below the limit) and re-run just the latter alone.

browserJudgePreset forwards measurements returned by each check; it does not measure elapsed time, CPU time, or memory automatically. Its timeoutMs sets the page default timeout for navigation, waiting and shared screenshot capture. Other native operations follow Puppeteer’s own timeout behavior; this option does not bound the complete check. Checks that require their own measurements or time-limit verdicts must provide them explicitly. The hosting judge can independently limit the overall harness process.