Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 1 addition & 10 deletions docs/1-introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,7 @@ gem "ferrum"

## Docker

:::note
When running in docker as root
:::

```ruby
Ferrum::Browser.new(dockerize: true)
```

Essentially it just sets CLI flags for a browser to make it start. On CI, you can just set `FERRUM_CHROME_DOCKERIZE=true` environment variable, and it will be
passed to all browser instances.
See [Docker](/docs/ferrum/docker).

## Quick Start

Expand Down
24 changes: 5 additions & 19 deletions docs/2-customization.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,9 +96,6 @@ There is simply no network service behind it, so every navigation returns `net::
`about:blank`.
:::

Ferrum shipped `--no-crashpad` as a default from 0.18.0 in an attempt at this. It never did anything, and has been
removed.

### Operating system differences

On macOS one handler starts rather than two, and it exits when the browser does, so nothing accumulates.
Expand All @@ -115,22 +112,11 @@ $ ps -Ao pid=,ppid=,comm= | grep crashpad

### Why this matters most in Docker

The handler is not a problem outside a container, and killing the browser is not what deals with it. The handler is
independent of Chrome's process group, so the signal Ferrum sends during teardown never reaches it — and does not
need to. The handler watches the browser and exits by itself once Chrome is gone. Verified by sending `TERM` and
`KILL` to the process group, and `KILL` and `SIGUSR1` to the browser pid directly: in every case it terminated on its
own within a second.

What is left behind is an exit status. Because the handler double-forks away from Chrome, its parent is pid 1, and
pid 1 is what reaps it — systemd or launchd, silently, so nothing is left over and no cleanup is needed.

In a container your own process is usually pid 1, and it does not wait on children it never spawned. Nothing reaps
the handlers, so every browser leaves two more `<defunct>` entries behind, and they are never reclaimed.
**They accumulate, until the process table fills up.**

Run the container with an init — `docker run --init`, `init: true` in Compose, or tini as the entrypoint. It is worth
doing regardless, since every other process in the image has the same problem, and there is no flag that avoids the
need for it.
The handler is harmless outside a container. It is not in Chrome's process group, so the signal Ferrum sends on
teardown never reaches it — and does not need to: it watches the browser and exits by itself once Chrome is gone.
What is left behind is an exit status, and because the handler double-forks away from Chrome its parent is pid 1.
Collecting it is PID 1's job which doesn't exist in a container, so the handlers pile up as `<defunct>` entries.
See [Docker](/docs/ferrum/docker).

## Examples

Expand Down
131 changes: 131 additions & 0 deletions docs/22-docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
---
sidebar_position: 22
---

# Docker

Chrome runs a bit differently in a container, in ways that are not obvious and that surface as
slow leaks rather than errors. This page collects everything Ferrum-specific about it.

:::note
When running in docker as root
:::

```ruby
Ferrum::Browser.new(dockerize: true)
```

Essentially it just sets CLI flags for a browser to make it start. On CI, you can just set `FERRUM_CHROME_DOCKERIZE=true`
environment variable, and it will be passed to all browser instances.

## An init is required

:::warning
If your Ruby process is pid 1, run the container with an init: `docker run --init`, `init: true` in Compose, or
`ENTRYPOINT ["/sbin/tini", "--"]` in the image. This is a requirement, not a nicety.
:::

Chrome is a process tree — a browser process plus renderer, GPU, zygote and utility processes — and it delegates the
last step of its own cleanup to the operating system.

When Chrome shuts down it tells its children to go and then exits about 10ms later. Its children take another
20-30ms to finish, whilst their parent is gone, so there's no one to collect them. This is not avoidable
and not specific to Ferrum: it happens whether the browser is asked to quit politely or killed outright, and whether
the signal goes to the browser alone or to its whole process group. Measured on Chrome 151, roughly eight to eleven
processes per browser finish after the browser itself has exited.

Unix answers this by handing orphans to pid 1, whose job is to collect them. On a normal machine that is systemd or
launchd and it happens instantly. In a container started as `CMD ["bundle", "exec", "..."]` there is no init at
all: *your* process is pid 1, it never inherited that job, and it collects nothing.

Two things then go wrong, both measured on the same image with and without `--init`:

| | no init | with an init |
|---|---|---|
| processes left after each `#quit` | 9 `<defunct>`, never reclaimed | 0 |
| time `#quit` takes | 2.02s | 0.06s |

The leak is the obvious cost. Each `<defunct>` entry holds a pid and a process-table slot forever, so eventually
`fork` starts failing, new browsers will not start, and page loads time out for no visible reason:

```console
$ docker exec app ps -A
PID TTY TIME CMD
41 ? 00:00:00 [chrome] <defunct>
42 ? 00:00:00 [chrome] <defunct>
62 ? 00:00:01 [chrome] <defunct>
...
```

The two seconds are less obvious. Ferrum waits for Chrome's process group to empty before returning, and a zombie
still answers `kill(0, ...)` — so with nothing reaping them the group never looks empty and every `#quit` waits out
its full kill timeout.

An init fixes both, and is worth having regardless: every other process your image spawns has the same problem.

## Kubernetes

There is no `init: true` in a pod spec, so you have two options.

**Bake an init into the image.** `ENTRYPOINT ["/sbin/tini", "--"]`. Scoped to your container, behaves the same
everywhere, and is the better default.

**Or set `shareProcessNamespace: true` on the pod.** Kubernetes then makes the `pause` container pid 1 for the whole
pod, and `pause` is an init — its `SIGCHLD` handler loops on `waitpid(-1, WNOHANG)`, which is exactly the job that
is otherwise missing. Note this is not mentioned on the Kubernetes documentation page for the feature; it is
visible in `pause.c`.

It has real trade-offs beyond reaping, though: every container in the pod can see the other processes, their
environment variables through `/proc/$pid/environ`, and their filesystems through `/proc/$pid/root`. Your process
also stops being pid 1, so anyone doing `kill -HUP 1` now signals `pause`.

:::note
`initContainers` are not an init system.
:::

## What Ferrum cleans up, and what it does not

Ferrum spawns exactly one process: the browser. On `#quit` it signals the browser's process group, escalating from
`TERM` to `KILL`, and waits on the browser itself — the one process it is the parent of, and therefore the only one
nobody else can collect.

Chrome's renderer, GPU and utility processes are Ferrum's *grand*children. When they outlive the browser the kernel
reparents them to pid 1, and collecting them is PID 1 job by definition. Ferrum deliberately does not do it:
`waitpid(-1)` would also collect child processes your own application spawned, stealing their exit statuses. A
library cannot take that liberty, which is why the answer here is an init rather than something Ferrum can fix.

`chrome_crashpad_handler` is a whole new story — Chrome double-forks it into its own session before any of this
happens, so it is not even in the process group Ferrum signals. See [the crashpad handler](/docs/ferrum/customization#the-crashpad-handler),
including why the flag that appears to disable it must not be used.

## Running Chrome in a separate container

If Chrome lives in its own container and Ferrum connects over a websocket, Ferrum did not spawn that browser and
couldn't kill it:

```ruby
browser = Ferrum::Browser.new(ws_url: "ws://chrome:3000/")
```

`#quit` closes the connection and nothing more. Whether the remote browser then shuts down is up to that service —
most reclaim a session when its connection drops, and most have a timeout for when they do not. To ask the browser
itself to exit rather than relying on that, use `#close`, which sends the CDP `Browser.close` command:

```ruby
browser.close
browser.quit
```

## Diagnosing

Check whether the processes piling up are dead or alive:

```console
$ docker exec app ps -Ao pid=,ppid=,stat=,comm=
```

`STAT` of `Z`, or `<defunct>` in the command, means the process is finished and merely uncollected. That is the
missing-init problem above.

Live processes accumulating instead means browsers are never being shut down at all — usually a `#quit` that is not
called, or a remote browser service holding sessions open. An init will not help with those.
Loading