Using Tests as a Check on Codex's Own Work

Tests are Codex's feedback loop

When you give Codex a task and context, you're asking it to produce code that satisfies some implicit contract. Tests make that contract explicit. A test that says "when I call add(2, 3), it returns 5" is a specification. When Codex sees your test suite, it learns what the code is supposed to do. When you run tests against Codex's output, you get concrete evidence of whether Codex succeeded or failed.

This is more powerful than any task description. A test is unambiguous. Either the code passes the test or it doesn't.

Test patterns that help Codex succeed

Unit tests for the modified function. If Codex is changing a function, include tests for that function. For example, if the task is "make the validateEmail() function case-insensitive," include tests like:

test("validateEmail should accept uppercase addresses", () => {
  expect(validateEmail("[email protected]")).toBe(true);
  expect(validateEmail("[email protected]")).toBe(true);
});

Codex reads this test and understands exactly what behavior is expected. It will write code that makes the test pass.

Boundary and edge-case tests. These are the tests Codex is most likely to miss on its own. Include them explicitly:

test("parseDate should handle leap years", () => {
  expect(parseDate("2024-02-29")).not.toThrow();
});

test("getUserById should return null for non-existent users", () => {
  expect(getUserById(99999)).toBeNull();
});

Codex will see these tests and either write code that handles the edge case or produce a diff that fails the test, giving you a chance to ask for a revision.

Integration tests that verify the changed module still works with dependent modules. If you're changing a core module, include tests that verify consumers of that module still work:

test("order processing should still work after caching layer", () => {
  const order = createOrder({...});
  const result = processOrder(order);
  expect(result.status).toBe("completed");
});

This tells Codex not just "make this function work" but "make this function work in the context of the system."

Performance tests. If the task is performance-related ("reduce latency of this query"), include a performance test:

test("getUserById should complete in under 50ms", async () => {
  const start = Date.now();
  await getUserById(123);
  const duration = Date.now() - start;
  expect(duration).toBeLessThan(50);
});

Codex sees this and optimizes for meeting the constraint.

Using test output to guide revisions

When Codex produces a diff and tests fail, the test output is your communication channel. If a test says AssertionError: Expected 5, but got undefined, you can tell Codex "The function is returning undefined when it should return a number. Check that you're returning the result of the calculation."

Codex reads test output and tries to fix the issue. This creates a loop: task -> Codex produces diff -> tests fail -> you show Codex the failure -> Codex revises. Often, one revision fixes multiple test failures because Codex identifies the root cause.

A worked example: test-driven Codex revision

You ask Codex to "add a retry mechanism to the fetchUser() API call. If the call fails with a 5xx error, retry up to 3 times with exponential backoff."

You provide tests:

test("fetchUser should retry on 500 error", async () => {
  mockFetch.mockRejectedValueOnce({ status: 500 });
  mockFetch.mockResolvedValueOnce({ data: { id: 1, name: "Alice" } });

  const result = await fetchUser(1);
  expect(result.name).toBe("Alice");
  expect(mockFetch).toHaveBeenCalledTimes(2); // initial + 1 retry
});

test("fetchUser should not retry on 4xx errors", async () => {
  mockFetch.mockRejectedValueOnce({ status: 404 });

  const result = await fetchUser(1);
  expect(result).toBeNull();
  expect(mockFetch).toHaveBeenCalledTimes(1); // no retry
});

test("fetchUser should respect exponential backoff timing", async () => {
  // First attempt fails, second fails, third succeeds
  mockFetch.mockRejectedValueOnce({ status: 500 });
  mockFetch.mockRejectedValueOnce({ status: 500 });
  mockFetch.mockResolvedValueOnce({ data: { id: 1 } });

  const start = Date.now();
  await fetchUser(1);
  const duration = Date.now() - start;

  // First retry at ~100ms, second at ~200ms
  expect(duration).toBeGreaterThan(250);
});

Codex produces a diff. You run tests and see:

✓ fetchUser should retry on 500 error
✗ fetchUser should not retry on 4xx errors
  Expected: 1 call
  Actual: 2 calls
✓ fetchUser should respect exponential backoff timing

One test failed. Codex retried on 404 when it shouldn't. You show Codex the test output and say "The second test is failing. Your code is retrying on 404 errors, but it should only retry on 5xx errors. Check your error handling logic."

Codex revises the diff to check the error status code and only retry on 5xx. You run tests again, they all pass, and you approve the diff.

Test coverage and Codex

Higher test coverage doesn't always mean Codex will succeed more, but tests for the specific function Codex is touching definitely help. If you ask Codex to modify an untested function, Codex has to infer what it should do from the function signature and how it's called elsewhere. If the function has tests, Codex has an explicit specification.

Common mistake

Writing tests that are too high-level or too vague. "Test that the caching works" is not actionable; Codex doesn't know what "works" means. "Test that getUserById(1) returns cached data within 100ms on the second call when Redis is running" is specific and actionable. Write tests as if they're specifications, because for Codex, they are.