I don't just review the agent's code anymore. I ask for proof

When an agent tells me "done, it works," I don't believe it. Not because it's lying, but because "it works" is a conclusion, and what I need is the evidence.

At work I use Playwright for that, a lot. I ask the agent to open the browser through the Playwright MCP, do what a user would do, and leave me a screenshot, a video or a trace. More than once, watching the video, I found what the agent hadn't told me.

But that conversation gets lost. Whatever worked while exploring has to end up as a file in e2e/ that runs by itself on every deploy. That's what this post is about: going from "show me once" to "show me every time."

I built it on this blog: six reader flows that run in CI, on every PR against the build and after every deploy against the live site. I approved the list of flows. One agent wrote the code and another reviewed it, the one that reviews this site. The site's repo is private, but the site isn't. You can run everything here yourself against davidquinta.tech.

A flow is what the reader does, and what has to happen#

A flow isn't "test the blog." It's a short story with an outcome you can check. For example: I'm reading a post in English, I switch to Spanish, and I have to land on that post's translation, not the index. And if I go back to the portfolio after that, it has to stay in Spanish.

import { expect, test } from '@playwright/test'
 
const POST_EN = '/blog/i-made-my-dog-a-nintendogs/'
const POST_ES = '/es/blog/le-hice-un-nintendogs-a-mi-perro/'
 
test('switching to Spanish on a post opens its translation, and the portfolio follows', async ({ page }) => {
  await page.goto(POST_EN)
  await page.getByRole('group', { name: 'Language' }).getByRole('link', { name: 'ES' }).click()
 
  await expect(page).toHaveURL(POST_ES)
  await expect(page.locator('html')).toHaveAttribute('lang', 'es')
 
  await page.getByRole('link', { name: 'Portafolio', exact: true }).click()
  await expect(page.locator('html')).toHaveAttribute('lang', 'es')
})

This flow didn't come out of nowhere. It already happened on this blog: a hand-written /blog/ was sending Spanish readers to the English blog. I tell that story in the subagents post. A good regression flow is a bug you already had, written down so it doesn't come back.

The selectors go by role and name (getByRole), not by CSS classes. If I change the markup tomorrow, the test still works. And it also checks that the link has an accessible name, because if a screen reader can't find it, neither can the test.

Careful with the name, though: by default getByRole matches the text as a case-insensitive substring. The first time it ran, it failed with strict mode violation: on the Spanish page, "Portafolio" matched the nav link and also "VER PORTAFOLIO", which looks like a button but is a link. Hence the exact: true.

Against the real build#

The flows don't run against next dev. They run against the same files the reader gets: on a PR, the static export in out/; everywhere else, production.

import { defineConfig } from '@playwright/test'
 
const local = process.env.LOCAL === '1'
 
export default defineConfig({
  testDir: 'e2e',
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,
  reporter: [['list'], ['html', { open: 'never' }]],
  use: {
    baseURL: local ? 'http://127.0.0.1:4173' : (process.env.BASE_URL ?? 'https://www.davidquinta.tech'),
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    trace: 'retain-on-failure',
  },
  webServer: local
    ? {
        command: 'python3 -m http.server 4173 --bind 127.0.0.1 --directory out',
        url: 'http://127.0.0.1:4173/',
        reuseExistingServer: !process.env.CI,
      }
    : undefined,
})

forbidOnly keeps a forgotten test.only from letting CI pass after running a single test. The retries: 1 in CI is for the network, not for covering up unstable tests: if a flow keeps needing the retry, the report marks it as flaky and it has to be fixed.

The test leaves the evidence, not the agent#

Those three lines, screenshot, video and trace, are the "show me every time." When something fails, I don't have to ask the agent what happened. I have the screenshot, the video and the trace, which is a recording of every step with the DOM, the network and the console.

To check that this actually helps, we broke the site on purpose: I made the language button on posts always go to the blog index. The flow failed like this:

Error: expect(page).toHaveURL(expected) failed
Expected: "http://127.0.0.1:4173/es/blog/le-hice-un-nintendogs-a-mi-perro/"
Received: "http://127.0.0.1:4173/es/blog/"

Open image full size

×Close image

The trace of the broken flow. The test expected the post and ended up on the index.

And that's the difference from an agent's "done": anyone can open that trace with npx playwright show-trace and see exactly where it broke. In CI, the trace stays inside the report uploaded as the run's artifact.

What a click won't show you#

Some flows aren't about clicks. The video in Dante's post has to wait until it's on screen to start, and if the reader pauses it, it has to stay paused even when they scroll.

test('the clip waits until it is on screen, and a reader can pause it', async ({ page }) => {
  await page.goto(POST_ES)
  const video = page.locator('video.post-video')
  const paused = () => video.evaluate((v: HTMLVideoElement) => v.paused)
 
  // Below the fold, and still not playing a second later.
  await expect(video).not.toBeInViewport()
  await page.waitForTimeout(1000)
  expect(await paused()).toBe(true)
 
  await video.scrollIntoViewIfNeeded()
  await expect.poll(paused).toBe(false)
 
  // The reader pauses it, scrolls away and comes back.
  await video.evaluate((v: HTMLVideoElement) => v.pause())
  await page.evaluate(() => window.scrollTo({ top: 0, behavior: 'instant' }))
  await expect(video).not.toBeInViewport()
  await video.scrollIntoViewIfNeeded()
  await expect(video).toBeInViewport()
  await page.waitForTimeout(1000)
  expect(await paused()).toBe(true)
})

expect.poll retries until the video starts, without a fixed sleep. The waitForTimeout calls are fixed, on purpose: there are three in the whole file, and all three check that something doesn't happen, which is impossible without giving it time to happen. Everything else waits by polling. And in this test, before each wait, toBeInViewport confirms where the video is, because if it's off screen, "still paused" proves nothing.

That video had already failed once. It started on its own and there was no way to pause it, which breaks WCAG 2.2.2. axe, which runs in the site's audit, didn't catch it: its autoplay rules look at audio, and this video has no sound. The agent that reviews the site found it, in the video's PR. Now there's a test for it.

The other flows do the same with things a click won't show: that closing an enlarged image with Escape puts focus back on the image, that a post can be read with JavaScript turned off (javaScriptEnabled: false), and that a link to a section doesn't leave the heading hidden under the nav bar, on desktop and on a phone:

for (const viewport of [
  { width: 1280, height: 800 },
  { width: 390, height: 844 },
]) {
  test(`a link to a section lands just below the sticky nav at ${viewport.width}px`, async ({ page }) => {
    await page.setViewportSize(viewport)
    // The site scrolls smoothly, so a fresh load is still on its way to the
    // anchor. Under reduced motion it jumps, and the position can be read.
    await page.emulateMedia({ reducedMotion: 'reduce' })
    await page.goto(`${POST_ES}#la-cara-no-salio`)
    const navBottom = await page.locator('nav').first().evaluate((n) => n.getBoundingClientRect().bottom)
    const top = () => page.locator('#la-cara-no-salio').evaluate((h) => h.getBoundingClientRect().top)
 
    await expect.poll(top).toBeGreaterThanOrEqual(navBottom)
    expect(await top()).toBeLessThanOrEqual(navBottom + 24)
  })
}

The range matters as much as the minimum: if the heading lands 300 px lower, that's not right either.

Four times the evidence proved me wrong#

A test or a review caught all four, not a reader. That's what they're there for.

The test that ran without the option. To test the video with reduced motion, the agent wrote test.use({ reducedMotion: 'reduce' }), an option that doesn't exist in the repo's version of Playwright. The runner silently ignored it, the test failed, and my first suspect was the site. TypeScript does catch it:

error TS2353: Object literal may only specify known properties,
and 'reducedMotion' does not exist in type 'Fixtures<...>'.

But Playwright runs tests in TypeScript without checking the types. That's why CI runs tsc before the flows: it's the only step in CI that catches an option that doesn't exist.

The clock. Dante's game has a Playwright test in its repo that throws him the ball and waits for him to bring it back. It passed on my machine; on GitHub Actions it failed all three times. The runner has no GPU, the game ran at less than one frame per second and, since each frame advances at most 0.05 s of game time, only about 3 seconds of game time went by in the test's 120 seconds. The fix: in tests, a slow frame gets split into several 0.05 s steps. In the browser the game doesn't change.

The codec that was there after all. The agent wrote in the config that Playwright's Chromium on Linux doesn't play H.264 and switched the tests to Chrome. That was true for the old Chromium, but since 1.57 Playwright runs on Chrome for Testing builds. The agent didn't check, and I didn't question it. A CI run with both browsers played the video in both, so we took the switch to Chrome out. If I ask the agent for evidence, I have to ask myself for it too.

The two tests that couldn't fail. Before publishing this, the agent that reviews the site put the flows to the test: it removed the site's anchor offset and measured where the video was at each step. Two of the six passed anyway. The anchor one measured too early: the site scrolls smoothly, so right after loading the heading was still on its way, far down the page, and "below the nav" was always true. The video one used mouse.wheel, which doesn't wait for the scroll to finish, so the video ended up off screen right when the test checked that it was still paused. Off screen it never starts, so the test passed even if the site forgot the reader's pause. The agent fixed them, broke the site on purpose, and both failed the way they should. A test you've never seen fail tells you nothing.

In CI: on every PR and every deploy#

on:
  pull_request:
  deployment_status:
  workflow_dispatch:
 
jobs:
  flows:
    if: github.event_name != 'deployment_status' || (github.event.deployment_status.state == 'success' && github.event.deployment.environment == 'Production')
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      # Playwright runs TypeScript without type-checking it, so a wrong option
      # name in a test is only caught here.
      - run: npm run typecheck
      - run: npx playwright install --with-deps chromium
      - if: github.event_name == 'pull_request'
        run: npm run build
      - name: Flows
        run: npx playwright test
        env:
          LOCAL: ${{ github.event_name == 'pull_request' && '1' || '' }}
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14

On a PR, it builds the site and runs the flows against that build, so a regression shows up before the merge. After every production deploy, it runs them against the live site, in case something slipped through or the deploy changed something the build doesn't show. I don't run them daily: the site is static, and between deploys there's nothing new to test. The deploy signal comes for free: Vercel reports each deploy to GitHub, and the if only lets a finished production one through. If it fails, the report with the screenshots, videos and traces stays as the run's artifact.

The six flows take about 15 seconds in CI. The whole job, with the browser install and the build, takes about two minutes.

Try it#

The snippets in this post run as is against davidquinta.tech: copy the first test into e2e/ along with the config, install @playwright/test, download the browser with npx playwright install chromium and run npx playwright test. If you find something broken on the site, let me know. And if you want the next post, there's the RSS feed.