Selenium: Automating the Browser

Selenium drives a real browser — opening pages, clicking, typing, reading back content — through WebDriver, a W3C standard that major browsers implement. Because it operates a genuine browser rather than a simulation, it exercises the same JavaScript, rendering, and network behaviour a user would trigger. It is the long-standing default for browser automation, and the ecosystem around it is larger than any alternative's.

What sits between your code and the page

Four layers, and knowing them explains most of the confusing errors.

Your script calls the Selenium client library in Python, Java, JavaScript, or another language. That library sends HTTP requests in the WebDriver format to a driver process — chromedriver, geckodriver — which translates them into the browser's own automation protocol and controls the browser, which renders the page.

Two consequences follow. First, every command is a network round trip, which is why a hundred small interactions are slower than they look and why chatty scripts benefit from doing more per call. Second, the driver version must match the browser version; a browser that auto-updates past its driver produces a startup failure that reads like a configuration error and is really a version skew. Selenium Manager now handles this automatically in current versions, and it remains the first thing to check when a suite that worked yesterday will not start today.

Flakiness has four sources, and only one is timing

A test that passes locally and fails in CI is the defining Selenium complaint. Retrying hides it; fixing it requires knowing which of four things happened.

Timing. The element was not ready. This is the majority, and the fix is explicit waits on a condition rather than a guessed duration:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

wait = WebDriverWait(driver, 10)
button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[data-testid='submit']")))
button.click()

time.sleep(5) is wrong in both directions at once: too slow when the page is ready in 200ms, and still too short on the day CI is loaded. Wait for the condition you actually need — visible, clickable, present, or text-changed — and note that these are different. An element can be present in the DOM but not visible, or visible but covered by an overlay that intercepts the click.

Selectors. The element moved or was renamed. Covered below.

State. The test depended on data another test created, or on its own previous run. Tests that share state fail in parallel and pass alone, which is the most misleading symptom in the set. Each test should create what it needs and clean up after itself.

Environment. Different viewport in headless mode, different timezone or locale in CI, a slower machine, an animation that runs on one and not the other. Pin the window size explicitly rather than inheriting whatever the CI container defaults to.

StaleElementReferenceException deserves a specific mention because it confuses everyone the first time: you held a reference to an element, the page re-rendered, and the reference now points at a node no longer in the DOM. Re-find the element after any navigation or update rather than caching it across an interaction.

Selectors that survive

Selector choice determines how much maintenance the suite costs over its life.

Best is a dedicated test attribute — data-testid — because it exists only for tests and changes only when someone means to change it. Next best are accessible roles and labels, which double as an accessibility check. Then a stable ID.

Avoid anything that encodes the page's current shape. Long CSS descendant chains, absolute XPath, generated class names from a CSS framework, and index-based positions all break on the next redesign for reasons unrelated to behaviour. Text content is workable if the product is not localised and fragile the moment it is.

The other structural fix is the page object pattern: keep selectors in one class per page, and let tests call methods like login_page.sign_in(user). When the markup changes you edit one file instead of forty.

Where Selenium still fits

Its advantages are breadth: the widest browser and language support, the largest ecosystem, remote-grid execution across many machines, and a decade of accumulated answers to whatever you hit. Newer tools have better defaults for auto-waiting and debugging, which matters most on greenfield test suites. For an established suite, broad browser coverage, or a language the alternatives do not support well, Selenium remains the correct answer.

Common mistakes

Fixed sleeps instead of conditions. Simultaneously the slowest and the least reliable option.

Waiting for presence when you need clickability. Present, visible, and clickable are three different states.

Caching element references across a re-render. The source of every stale element exception.

Tests that share state. Pass alone, fail in parallel, and waste days.