`treecript`: Process Tree Metrics Transcriptor

Originally named Execution Process Metrics Collector

A set of Python programs to monitor, collect, and digest metrics of a given Linux process or command line, and its descendants. Initially developed for ELIXIR STEERS.

Repository Structure

treecript/
├── treecript/               # Core Python package — all program logic lives here
├── installation/            # Constraints and requirements files for reproducible installs
├── legacy/                  # Deprecated Bash scripts kept for historical reference
├── sample-series/           # Real metrics from a WfExS workflow execution, used in documentation examples
├── sample-charts/           # Pre-generated charts from the sample series, embedded in this README
├── sample-work-to-measure/  # Example scripts showing how to set up and run a measurement
├── sample_cpuinfo/          # Example /proc/cpuinfo files for testing the TDP finder programs
├── onboarding/              # Full worked example with metrics, charts and a step-by-step walkthrough
├── sample/                  # Legacy single-process sample from 2018 (pre-treecript era)
└── tests/                   # Unit tests

Directory	Description
`treecript/`	Core Python package — aggregator, collector, parser, plotter, TDP finder
`installation/`	Per-version constraints files and requirements for reproducible installs
`legacy/`	Deprecated Bash scripts superseded by the Python programs
`sample-series/`	Real metrics collected from a WfExS workflow run, used throughout this README as examples
`sample-charts/`	Pre-generated chart outputs (SVG/PDF/PNG) from the sample series
`sample-work-to-measure/`	Ready-to-use scripts to download and run example workloads to measure
`sample_cpuinfo/`	Example `/proc/cpuinfo` files (Intel and AMD) for testing `cpuinfo-tdp-finder` and `modelname-tdp-finder` without needing a real machine
`onboarding/`	Self-contained worked example: a full metrics collection, chart generation and aggregation walkthrough for new users
`sample/`	Legacy single-process sample from 2018, predating the current process-tree approach
`tests/`	Unit tests for the core collector module

Installation

Prerequisites (singularity)

Linux OS
Singularity or Apptainer.

Prerequisites (native)

Linux OS (Ubuntu recommended)
Python 3.9 or newer
Git

Not sure which installation method to use?

I want to...	Use
Keep things simple, already have either Apptainer or Singularity installed and you only want to gather metrics	Option 1 — Singularity
Keep things simple and already have Python installed	Option 2 — pip + venv
Already use conda or manage multiple projects/environments	Option 3 — Conda
Work on an HPC or shared cluster environment (e.g. BSC)	Option 1 — Singularity or Option 3 — Conda
Deploy a single, dependency-free executable to a machine where you can't install Python, pip, or containers directly (e.g. copying onto compute nodes)	Option 4 — Standalone binary (PyInstaller)
Work on a machine with a corporate or university firewall	Either — both have firewall notes in their respective sections

Choosing a constraints file (only needed for option 2, option 3 or option 4)

The repository ships per-version constraints files under the installation/ directory to ensure a working set of dependencies. Pick the one that matches your setup (including the Python version):

Situation	Constraints file to use
Native Linux, Python 3.x	`installation/constraints-3.x.txt`
Ubuntu 22.04 on WSL (Windows)	`installation/constraints-3.x_Ubuntu-22.04-wsl.txt`
Ubuntu 24.04 on WSL (Windows)	`installation/constraints-3.x_Ubuntu-24.04-wsl.txt`

WSL = Windows Subsystem for Linux — Ubuntu running inside Windows rather than directly on hardware. If you are running Ubuntu natively on your machine, use the plain constraints file. To check:
uname -r  # if the output contains "microsoft" or "WSL", you are on WSL

To check your Python version:

python3 --version

Option 1: Singularity

Alternative A: Fetch a pre-built image (only for metrics gathering)

Pre-build singularity images are listed at https://github.com/inab/treecript/pkgs/container/treecript .

They can be fetched using Apptainer/Singularity just using singularity pull subcommand:

# Replace exec by the tag of the version you want to use 
singularity pull oras://ghcr.io/inab/treecript:exec

# The name of the created file depends on the tag
ls -l treecript-exec.sif

Alternative B: Building from source

When no file is locally, but you already have either Apptainer or Singularity installed:

# TREECRIPT_VER can be either a branch, a tag or a commit hash
TREECRIPT_VER=7c0a20a688518e43952d7bd7080bc34b863e32da

# If you don't have the recipe, you can fetch it using either curl or wget
mkdir -p treecript_SIF_build/installation
cd treecript_SIF_build/installation
curl -O https://raw.githubusercontent.com/inab/treecript/${TREECRIPT_VER}/installation/Singularity.def
cd ..

If you have already checked out the code

# Alternatively, you can use a checked out copy
cd treecript
TREECRIPT_VER=$(git rev-parse HEAD)

At last, you can build the SIF image with next command:

singularity build --build-arg treecript_checkout="${TREECRIPT_VER}" \
  treecript-${TREECRIPT_VER}.sif installation/Singularity.def

Option 2: pip + virtual environment (venv)

Use this if you already have Python installed on your system and don't use conda. This is the lightest option — it creates an isolated Python environment using only tools that come built into Python, with no additional software required.

# 1. Create a virtual environment
python3 -m venv TREECRIPT

# 2. Activate it
source TREECRIPT/bin/activate

# 3. Upgrade pip and wheel
pip install --upgrade pip wheel

# 4. Download the constraints file for your Python version (adjust filename as needed)
wget https://raw.githubusercontent.com/inab/treecript/exec/installation/constraints-3.10.txt

# 5. Install treecript with constraints
pip install -c constraints-3.10.txt 'treecript [analytics,docker] @ git+https://github.com/inab/treecript.git@exec'

Network issues? If you are behind a corporate or university firewall (e.g. Fortiguard), add --no-check-certificate to the wget command.

To deactivate the environment:

deactivate

To reactivate later:

source TREECRIPT/bin/activate

Option 3: Conda environment

Use this if you already work with Anaconda or Miniconda, or if you prefer conda for managing environments across multiple projects. Conda handles both Python and system-level dependencies, which makes it particularly well suited for HPC or shared computing environments.

Installing Miniconda (if not already installed)

Miniconda is a minimal conda installer — it gives you the conda command and Python without bundling hundreds of extra packages like the full Anaconda distribution does.

# Download installer (add --no-check-certificate if behind a firewall)
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh

# Run the installer
bash miniconda.sh
# - Accept the license
# - Accept the default install location
# - Answer "yes" when asked to update your shell profile

# Apply changes to current session
source ~/.bashrc

# Verify
conda --version

Network issues? If repo.anaconda.com is blocked by your network, use Miniforge instead — it is functionally identical to Miniconda but downloads from GitHub and defaults to the conda-forge channel, which is actually a better fit for the scientific packages treecript needs:
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh

Creating the treecript conda environment

# 1. Create a clean environment with Python 3.10
conda create -n treecript python=3.10 -y

# 2. Activate it
conda activate treecript

# 3. Download the constraints file (adjust filename for your Python version / OS)
wget https://raw.githubusercontent.com/inab/treecript/exec/installation/constraints-3.10.txt
# or for WSL Ubuntu 22.04:
# wget https://raw.githubusercontent.com/inab/treecript/exec/installation/constraints-3.10_Ubuntu-22.04-wsl.txt

# 4. Install treecript and all dependencies in one shot
pip install -c constraints-3.10.txt 'treecript [analytics,docker] @ git+https://github.com/inab/treecript.git@exec'

To deactivate:

conda deactivate

To remove the environment entirely:

conda deactivate
conda remove -n treecript --all -y

Option 4: Standalone binary (PyInstaller)

Use this if you need to run execution-metrics-collector on a machine where you cannot install Python, pip, or a container runtime — for example, a compute node on an HPC cluster that only allows copying pre-built files. PyInstaller bundles the Python interpreter, treecript, and all its dependencies into a single, dependency-free executable.

Alternative A: Use a pre-built binary

A pre-built, ready-to-use binary of execution-metrics-collector is published under this repository's Packages — the same place the Singularity images live (see Option 1). You can find it directly at ghcr.io/inab/treecript:exec-binary-linux-x86_64.

What it is: a monolithic, single-file executable (built with PyInstaller inside a manylinux_2_28 container, see Alternative B below) that bundles the Python interpreter, treecript, and all its dependencies. It requires glibc >= 2.28 on the target machine and nothing else — no Python, no pip, no virtual environment, no container runtime.

Binaries are pushed and pulled as OCI artifacts using oras, the same underlying tool singularity pull oras://... uses for the container images above.

Downloading it:

# Install oras once, if you don't have it: https://oras.land/docs/installation
# Replace the tag with the version you want (mirrors the treecript git ref/tag)
oras pull ghcr.io/inab/treecript:exec-binary-linux-x86_64

chmod +x execution-metrics-collector
./execution-metrics-collector

Using it on an HPC cluster: since the binary has no dependencies, it can be pulled and run directly on the login node, or inside an interactive Slurm allocation:

# Option A: pull and run directly on the login node
oras pull ghcr.io/inab/treecript:exec-binary-linux-x86_64
chmod +x execution-metrics-collector
./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2

# Option B: interactive job via Slurm (salloc) — recommended for anything
# that needs dedicated compute resources rather than sharing the login node
salloc -n 1 -c 4 -t 00:30:00   # adjust cores/time/partition to your cluster
# once the allocation starts and you're placed on a compute node:
./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2
exit   # ends the salloc session when done

Before relying on this binary on a new target machine, double-check glibc compatibility — see Checking glibc compatibility before deploying below. Since it was built against manylinux_2_28 (glibc 2.28), it will run on any machine with glibc 2.28 or newer, but not on an older one:

ldd --version | head -1   # on the target machine; must report glibc >= 2.28

If your target machine has an older glibc, or a different CPU architecture, this pre-built binary won't work for you — build your own following Alternative B below.

Publishing a new version (maintainers):

# 1. Log in to GitHub Container Registry (needs a PAT with write:packages scope)
echo "$GITHUB_TOKEN" | oras login ghcr.io -u your-github-username --password-stdin

# 2. Build the binary (see "Building the binary" below), then push it as an
#    OCI artifact — no Dockerfile/container involved, just the raw file
oras push ghcr.io/inab/treecript:exec-binary-linux-x86_64 \
  dist/execution-metrics-collector:application/octet-stream

# 3. Confirm it shows up under the repository's Packages tab

Use a version tag that reflects what the binary actually is (e.g. exec-binary-linux-x86_64, or include the treecript ref/commit if you publish multiple versions), so users can tell binaries apart without downloading each one.

Alternative B: Building it yourself

Why a container is needed

The binary must be built on a machine whose glibc version is equal to or older than the target machine's glibc, because glibc is forward-compatible but not backward-compatible: a binary built against a newer glibc will refuse to run on a system with an older one, failing with an error such as:

/lib64/libc.so.6: version `GLIBC_2.35' not found (required by execution-metrics-collector)

Development laptops typically run a fairly recent Linux distribution (e.g. Ubuntu 22.04, glibc 2.35), while HPC login/compute nodes often run older, more conservative distributions (e.g. glibc 2.28–2.34). Building directly on the laptop would therefore very likely produce a binary that cannot run on the cluster.

The safe approach is to build inside a manylinux container, which ships a deliberately old glibc chosen to be compatible with virtually any modern Linux target:

# Check your target machine's glibc version:
ssh your-user@your-hpc-login-node "ldd --version | head -1"

# Check your build machine's glibc version:
ldd --version | head -1

If the target's glibc is older than your laptop's, build inside a manylinux_2_28 (or older, e.g. manylinux2014) container instead of building natively.

Note: the Python interpreters preinstalled inside manylinux images (under /opt/python/cpXXX-cpXXX) are built without a shared libpython — they are meant for compiling wheels, not for running applications. PyInstaller requires a shared libpythonX.Y.so, so the container build compiles its own Python from source with --enable-shared. It also needs openssl-devel (and other -devel headers) installed before compiling Python, otherwise Python's ssl module is silently skipped and pip cannot reach PyPI.

Building the binary

Save the following as Dockerfile in a directory that also contains your chosen installation/constraints-3.x.txt file:

FROM quay.io/pypa/manylinux_2_28_x86_64

ARG PYVER=3.13.9
ARG TREECRIPT_GIT_URL=https://github.com/inab/treecript.git
ARG TREECRIPT_REF=exec

ENV PYSHARED_PREFIX=/opt/python-shared
ENV LD_LIBRARY_PATH=${PYSHARED_PREFIX}/lib

# Build-time dependencies for compiling Python (openssl-devel is required,
# otherwise Python ends up without the ssl module and pip cannot reach PyPI)
RUN dnf install -y \
        openssl-devel bzip2-devel libffi-devel zlib-devel xz-devel \
        ncurses-devel readline-devel sqlite-devel tk-devel gdbm-devel \
    && dnf clean all

# Compile Python with --enable-shared, required by PyInstaller
RUN curl -O https://www.python.org/ftp/python/${PYVER}/Python-${PYVER}.tgz \
    && tar xzf Python-${PYVER}.tgz \
    && cd Python-${PYVER} \
    && ./configure --enable-shared --prefix=${PYSHARED_PREFIX} --with-openssl=/usr \
    && make -j"$(nproc)" \
    && make altinstall \
    && cd .. && rm -rf Python-${PYVER} Python-${PYVER}.tgz

ENV PYBIN=${PYSHARED_PREFIX}/bin/python3.13
WORKDIR /src

RUN ${PYBIN} -m venv /opt/venv
ENV PATH=/opt/venv/bin:$PATH

COPY constraints-3.12.txt /src/constraints-3.12.txt

RUN /opt/venv/bin/pip install --upgrade pip wheel \
    && /opt/venv/bin/pip install -c /src/constraints-3.12.txt \
       "treecript[analytics,docker] @ git+${TREECRIPT_GIT_URL}@${TREECRIPT_REF}" \
    && /opt/venv/bin/pip install pyinstaller

CMD ["/opt/venv/bin/pyinstaller", "-F", "-n", "execution-metrics-collector", \
     "/opt/venv/bin/execution-metrics-collector"]

Build the image once (this recompiles Python from source, so it takes several minutes the first time; subsequent builds reuse the cached layer as long as PYVER doesn't change):

docker build -t treecript-builder \
  --build-arg TREECRIPT_GIT_URL=https://github.com/inab/treecript.git \
  --build-arg TREECRIPT_REF=exec \
  -f Dockerfile .

Then generate the binary itself (fast — reuses the already-built image):

docker run --rm -v "$PWD/dist:/src/dist" treecript-builder

The resulting dist/execution-metrics-collector is a single, self-contained executable (--onefile / -F build).

Why only execution-metrics-collector? The whole point of this standalone binary is to run metrics collection on a machine with no Python available — typically an HPC compute node. The other treecript programs (plotGraph, tdp-finder, metrics-aggregator) are analysis tools meant to run afterwards on a regular machine (your laptop, a login node with Python, etc.), where a normal Option 2 or Option 3 install is simpler and works just as well — see the full workflow example below. It's technically possible to build any of them the same way (just point pyinstaller at a different entry point, e.g. /opt/venv/bin/plotGraph instead of /opt/venv/bin/execution-metrics-collector).

Checking glibc compatibility before deploying

Before copying the binary anywhere, confirm the minimum glibc version it actually requires:

objdump -T dist/execution-metrics-collector | grep GLIBC_ \
  | sed 's/.*GLIBC_\([0-9.]*\).*/\1/' | sort -V | tail -1

This should print a version at or below the target machine's ldd --version.

Troubleshooting

Symptom	Cause	Fix
`version 'GLIBC_2.XX' not found` when running the binary	Built on a machine/container with newer glibc than the target	Rebuild inside an older `manylinux` base image (e.g. `manylinux2014` instead of `manylinux_2_28`)
`Python was built without a shared library` during `pyinstaller` build	Using the manylinux-provided Python instead of a `--enable-shared` build	Compile Python from source with `--enable-shared`, as shown above
`SSL module is not available` / pip cannot reach PyPI while building the image	`openssl-devel` (and related `-devel` headers) missing before compiling Python	Install `openssl-devel` (and the other `-devel` packages listed above) before the `./configure && make` step
Files generated by `docker run` (`dist/`, `build/`, `*.spec`) cannot be deleted from the host	The container runs as `root`, so generated files are root-owned	`sudo rm -rf build dist *.spec`, or delete them from inside another container run as root, or add `--user "$(id -u):$(id -g)"` to `docker run` to avoid the issue going forward

Full workflow example: HPC metrics collection + local analysis

This walks through a complete, real scenario: building a monolithic binary on a laptop, using it to collect metrics on an HPC cluster where no Python environment is available, and then bringing those metrics back to the laptop for plotting, TDP lookup, and energy aggregation — all done locally, since only metrics collection strictly requires running on the HPC node itself.

# ── 1. On the laptop: build the binary (see Alternative B above) ──────────
docker build -t treecript-builder -f Dockerfile .
docker run --rm -v "$PWD/dist:/src/dist" treecript-builder

# Confirm the glibc floor is compatible with the target cluster
objdump -T dist/execution-metrics-collector | grep GLIBC_ \
  | sed 's/.*GLIBC_\([0-9.]*\).*/\1/' | sort -V | tail -1

# ── 2. Copy the binary to the HPC login/compute node ───────────────────────
scp dist/execution-metrics-collector your-user@your-hpc-login-node:~/

# ── 3. On the HPC node: collect metrics for the workload you care about ────
ssh your-user@your-hpc-login-node
chmod +x execution-metrics-collector

# Option A: direct run on the login node (fine for short/light workloads;
# check your cluster's usage policy before running heavier jobs here)
./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2

# Option B: interactive job via Slurm (salloc) — recommended for anything
# that needs dedicated compute resources rather than sharing the login node
salloc -n 1 -c 4 -t 00:30:00   # adjust cores/time/partition to your cluster
# once the allocation starts and you're placed on a compute node:
./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2
exit   # ends the salloc session when done

# This creates ~/my_metrics/<timestamp>-<pid>/ with the raw CSV/TSV metrics
# (from whichever of the two runs above you used)

# ── 4. Bring the metrics directory back to the laptop ──────────────────────
# (run from the laptop)
scp -r your-user@your-hpc-login-node:~/my_metrics/<timestamp>-<pid> ./my_metrics/

# ── 5. On the laptop: set up a normal treecript environment for analysis ───
# (analysis tools — plotGraph, tdp-finder, metrics-aggregator — don't need
#  to run on the HPC node itself; a regular venv/conda install is enough)
python3 -m venv TREECRIPT
source TREECRIPT/bin/activate
pip install --upgrade pip wheel
pip install -c constraints-3.12.txt 'treecript[analytics,docker] @ git+https://github.com/inab/treecript.git@exec'

# ── 6. Get a CPU dataset for TDP lookups (once) ─────────────────────────────
git clone https://github.com/JosuaCarl/cpu-spec-dataset cpu-spec-dataset_Josua

# ── 7. Plot time series charts ──────────────────────────────────────────────
plotGraph ./my_metrics/<timestamp>-<pid>/ ./my_charts/

# ── 8. Find the HPC node's CPU TDP ──────────────────────────────────────────
# Use the model name reported by `lscpu` (or /proc/cpuinfo) on the HPC node,
# e.g. "Intel(R) Xeon(R) Platinum 8480+"
tdp-finder ./my_metrics/<timestamp>-<pid>/ cpu-spec-dataset_Josua/dataset/*.csv
# Model [Intel(R) Xeon(R) Platinum 8480+] => TDP [MaxTDP] => 350.0 W

# ── 9. Aggregate and estimate energy consumption ────────────────────────────
metrics-aggregator ./my_metrics/<timestamp>-<pid>/ ./my_agg/ 350.0

Tip: if tdp-finder prints Unable to match a valid processor row for some of the CSV sources you passed, that's normal — different sources use slightly different naming conventions for the same CPU. tdp-finder tries each file in order and stops at the first match; passing several sources increases the odds of a hit. The official vendor CSVs (e.g. intel-cpus.csv) tend to be the most reliable matches.

Verifying the installation

⚠️ Not applicable to the Option 4 binary itself. This check requires a Python environment with treecript importable, which the standalone binary from Option 4 intentionally does not provide (that's the whole point of a monolithic executable — see the note right below instead). This check does apply to the local analysis environment you create in Option 4's full workflow example step 5 (the python3 -m venv TREECRIPT used to run plotGraph, tdp-finder and metrics-aggregator locally) — it's a completely separate, ordinary Option 2/3-style install, just used alongside the binary rather than instead of it.

Run this after Option 2 (pip + venv) or Option 3 (Conda) — or after setting up the local analysis environment from Option 4's workflow example — to confirm all dependencies are working correctly:

python -c "
import psutil; print('psutil OK:', psutil.__version__)
import docker; print('docker OK:', docker.__version__)
import pandas; print('pandas OK:', pandas.__version__)
import networkx; print('networkx OK:', networkx.__version__)
import matplotlib; print('matplotlib OK:', matplotlib.__version__)
import adjustText; print('adjustText OK:', adjustText.__version__)
import treecript; print('treecript OK')
"

For the Option 4 binary itself, there is no Python environment to verify this way — simply run the binary, e.g. ./execution-metrics-collector with no arguments, and confirm it prints its usage message.

Quick Start (Singularity/Apptainer)

At the moment, it is limited to metrics gathering

# 1. Collect metrics for a command in background
my_command --arg1 --arg2 &

# bash puts the PID of the last background process in $! variable
# This way is needed because the 
singularity exec treecript-exec.sif process-metrics-collector $! ~/my_metrics

Quick Start (other options)

# 1. Collect metrics for a command
execution-metrics-collector ~/my_metrics my_command --arg1 --arg2

# 2. Plot time series charts
plotGraph ~/my_metrics/2025_01_01-00_00-12345/ ~/my_charts/

# 3. Find your CPU's TDP
tdp-finder ~/my_metrics/2025_01_01-00_00-12345/ cpu-spec-dataset_Josua/dataset/*.csv

# 4. Aggregate and estimate energy consumption
metrics-aggregator ~/my_metrics/2025_01_01-00_00-12345/ ~/my_agg/ 28.0

Programs Reference

Collecting metrics

execution-metrics-collector runs a command and monitors it and all its child processes:

execution-metrics-collector {base_metrics_directory} {command} {args...}

Internally this launches the command, captures its PID, and calls process-metrics-collector with a sampling period of 1 second:

process-metrics-collector {pid} {base_metrics_directory} [sample_period]

Example:

execution-metrics-collector ~/metrics python myscript.py --input data.txt

Plotting time series charts

plotGraph generates line charts for each monitored process, comparing time series of CPU, memory, I/O and other metrics:

plotGraph {metrics_directory} {output_directory}

Example:

plotGraph ~/metrics/2025_01_01-00_00-12345/ ~/charts/

Finding CPU TDP

Three programs are available depending on what information you have:

`tdp-finder` — from a metrics directory

tdp-finder {metrics_directory} {csv_files...}

Example:

tdp-finder ~/metrics/2025_01_01-00_00-12345/ cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv

Use -q for quiet mode (outputs only the TDP value, useful for scripting):

tdp-finder -q ~/metrics/2025_01_01-00_00-12345/ cpu-spec-dataset_Josua/dataset/*.csv
# Output: 28.0

`cpuinfo-tdp-finder` — from `/proc/cpuinfo`

Does not require a metrics directory:

cpuinfo-tdp-finder /proc/cpuinfo cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv

Or from a saved copy of /proc/cpuinfo — the sample_cpuinfo/ directory contains example files for Intel and AMD processors you can use for testing:

cpuinfo-tdp-finder sample_cpuinfo/cpuinfo-amd.txt cpu-spec-dataset_Josua/dataset/*.csv

`modelname-tdp-finder` — from a processor model string

modelname-tdp-finder "11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz" cpu-spec-dataset_Josua/dataset/*.csv
modelname-tdp-finder "AMD EPYC 7742 64-Core Processor" cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv

Digesting metrics

metrics-aggregator digests the collected time series and estimates energy consumption per process subtree. It requires the CPU TDP value in Watts.

metrics-aggregator {metrics_directory} {output_directory} {TDP_watts} [command_filter]

The optional command_filter argument filters results to show only processes whose command matches the string (e.g. "docker run" to focus on Docker steps).

Example using the included sample series:

metrics-aggregator sample-series/Wetlab2Variations_metrics/2025_05_20-02_19-14001/ dest_directory 28.0 "docker run"

The output directory will contain:

A table of energy consumption per task (stdout)
graph.pdf / graph.svg — process call graph as a tree
spiral-graph.pdf / spiral-graph.svg — process call graph as a spiral
consumptions.pdf / consumptions.svg — barplot of task energy and duration
timeline.pdf / timeline.svg — lollipop chart of task start, duration, and end

CPU Dataset Setup

The TDP programs require one or more CPU specification datasets to look up processor TDP values. Three sources are supported:

Recommended — JosuaCarl fork (better column names):

git clone https://github.com/JosuaCarl/cpu-spec-dataset cpu-spec-dataset_Josua

Alternative — original felixsteinke repo:

git clone https://github.com/felixsteinke/cpu-spec-dataset

CPUBenchmark scrape (good coverage for AMD server CPUs):

python -m treecript.tdp_sources cpumark_table.csv

You can pass multiple sources to the TDP programs and they will be tried in order:

tdp-finder ~/metrics/dir/ cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv

Tip: if tdp-finder logs Unable to match a valid processor row for one of your CSV sources, that's usually just that particular source's naming convention not matching your /proc/cpuinfo model string exactly (e.g. extra vendor symbols, or [Dual CPU] prefixes for multi-socket listings). This is expected when passing several sources — tdp-finder simply moves on to the next file until it finds a match. The intel-cpus.csv / ampere-cpus.csv sources (official vendor data) tend to match most reliably for their respective vendors.

Output Files Reference

Each execution-metrics-collector run creates a subdirectory named after the start timestamp and PID. It contains:

File	Description
`reference_pid.txt`	PID of the root process being monitored
`sampling-rate-seconds.txt`	Sampling rate in seconds (usually 1)
`pids.txt`	Table of all spawned processes with timestamps and parent PIDs
`agg_metrics.tsv`	Time series of aggregated metrics across all processes
`metrics-{pid}_{create_time}.csv`	Per-process time series metrics
`command-{pid}_{create_time}.txt`	Linearized command line for each process
`command-{pid}_{create_time}.json`	JSON representation of the command line
`cpu_details.json`	Physical CPU information from `/proc/cpuinfo`
`core_affinity.json`	Processor-to-core-to-CPU mapping derived from `/proc/cpuinfo`

Per-process metrics (`metrics-{pid}_{create_time}.csv`)

Column	Description
`Time`	Sample timestamp
`PID`	Process ID
`Virt`	Virtual memory size (matches `top` VIRT)
`Res`	Resident set size — non-swapped physical memory (matches `top` RES)
`CPU`	CPU utilization as a percentage (can exceed 100% for multithreaded processes)
`Memory`	RSS memory as a percentage of total physical system memory
`TCP connections`	Number of open TCP connections
`Thread Count`	Number of threads (non-cumulative)
`User`	Time spent in user mode (seconds)
`System`	Time spent in kernel mode (seconds)
`Children_User`	User time of child processes (always 0 on Windows/macOS)
`Children_System`	System time of child processes (always 0 on Windows/macOS)
`IO`	Time waiting for blocking I/O (Linux only)
`uss`	Unique Set Size — memory freed if this process terminated now
`swap`	Memory swapped out to disk
`processor_num`	Number of unique CPU processors used
`core_num`	Number of unique CPU cores used
`cpu_num`	Number of unique physical CPUs used
`processor_ids`	IDs of CPU processors used (space-separated)
`core_ids`	IDs of CPU cores used (space-separated)
`cpu_ids`	IDs of physical CPUs used (space-separated)
`process_status`	Process status string (e.g. `sleeping`, `running`)
`read_count`	Cumulative number of read syscalls
`write_count`	Cumulative number of write syscalls
`read_bytes`	Bytes physically read from disk (cumulative)
`write_bytes`	Bytes physically written to disk (cumulative)
`read_chars`	Bytes passed to read syscalls (cumulative, Linux only)
`write_chars`	Bytes passed to write syscalls (cumulative, Linux only)

Aggregated metrics (`agg_metrics.tsv`)

Each row is a 1-second sample across all monitored processes combined:

Column	Description
Timestamp	Sample time
Number of PIDs	Processes monitored at that moment
Threads	Total thread count
Processors	Number of distinct CPU processors in use
Cores	Number of distinct CPU cores in use
Physical CPUs	Number of distinct physical CPUs in use
CPU IDs	IDs of physical CPUs (space-separated)
User memory	Total user memory across all processes
Swap memory	Total swap memory across all processes
Read ops	Total read operations
Write ops	Total write operations
Read bytes	Bytes physically read
Write bytes	Bytes physically written
Read chars	Bytes passed to read syscalls
Write chars	Bytes passed to write syscalls

Legacy

The legacy/ directory contains older Bash-based scripts that predate the current Python implementation. They are kept for historical reference but are no longer maintained or recommended.

`execution-metrics-collector.sh`

The original Bash wrapper for launching a command and monitoring it. It runs the command in the background, captures the PID, and calls process-metrics-collector directly:

./legacy/execution-metrics-collector.sh {base_metrics_directory} {command} {args...}

This has been superseded by execution-metrics-collector, which provides the same functionality in a more portable and maintainable way. The sample series included in this repository was originally collected using this script:

~/projects/treecript/legacy/execution-metrics-collector.sh \
  ~/projects/treecript/Wetlab2Variations_metrics \
  python WfExS-backend.py -L workflow_examples/local_config.yaml \
  staged-workdir offline-exec 01a1db90-1508-4bad-beb7-7f7989838542

`plotGraph.sh`

The original gnuplot-based visualization script. It reads the collected CSV files and generates .pdf charts using gnuplot (requires apt install gnuplot). It has been superseded by plotGraph, which generates richer charts without requiring gnuplot.

./legacy/plotGraph.sh {metrics_csv_files...}

`plot-metrics.sh`

An earlier helper script for plotting individual metric files. Also superseded by plotGraph.

These scripts are no longer actively maintained. For all new usage, prefer the Python equivalents.

License

Licensed under GNU GPL v3.

This repository is a fork and evolution of chamilad/process-metrics-collector.

Name		Name	Last commit message	Last commit date
.github		.github
data		data
installation		installation
legacy		legacy
onboarding		onboarding
sample-charts		sample-charts
sample-series/Wetlab2Variations_metrics/2025_05_20-02_19-14001		sample-series/Wetlab2Variations_metrics/2025_05_20-02_19-14001
sample-work-to-measure		sample-work-to-measure
sample/27819-2018_05_03_1521		sample/27819-2018_05_03_1521
sample_cpuinfo		sample_cpuinfo
tests		tests
treecript		treecript
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
RO-Crate-REPRESENTATION.md		RO-Crate-REPRESENTATION.md
cpuinfo-tdp-finder.py		cpuinfo-tdp-finder.py
execution-metrics-collector.py		execution-metrics-collector.py
metrics-aggregator.py		metrics-aggregator.py
modelname-tdp-finder.py		modelname-tdp-finder.py
plotGraph.py		plotGraph.py
process-metrics-collector.py		process-metrics-collector.py
pyproject.toml		pyproject.toml
setup.py		setup.py
tdp-finder.py		tdp-finder.py

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

treecript: Process Tree Metrics Transcriptor

Table of Contents

Repository Structure

Installation

Prerequisites (singularity)

Prerequisites (native)

Not sure which installation method to use?

Choosing a constraints file (only needed for option 2, option 3 or option 4)

Option 1: Singularity

Alternative A: Fetch a pre-built image (only for metrics gathering)

Alternative B: Building from source

Option 2: pip + virtual environment (venv)

Option 3: Conda environment

Installing Miniconda (if not already installed)

Creating the treecript conda environment

Option 4: Standalone binary (PyInstaller)

Alternative A: Use a pre-built binary

Alternative B: Building it yourself

Why a container is needed

Building the binary

Checking glibc compatibility before deploying

Troubleshooting

Full workflow example: HPC metrics collection + local analysis

Verifying the installation

Quick Start (Singularity/Apptainer)

Quick Start (other options)

Programs Reference

Collecting metrics

Plotting time series charts

Finding CPU TDP

tdp-finder — from a metrics directory

cpuinfo-tdp-finder — from /proc/cpuinfo

modelname-tdp-finder — from a processor model string

Digesting metrics

CPU Dataset Setup

Output Files Reference

Per-process metrics (metrics-{pid}_{create_time}.csv)

Aggregated metrics (agg_metrics.tsv)

Legacy

execution-metrics-collector.sh

plotGraph.sh

plot-metrics.sh

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`treecript`: Process Tree Metrics Transcriptor

`tdp-finder` — from a metrics directory

`cpuinfo-tdp-finder` — from `/proc/cpuinfo`

`modelname-tdp-finder` — from a processor model string

Per-process metrics (`metrics-{pid}_{create_time}.csv`)

Aggregated metrics (`agg_metrics.tsv`)

`execution-metrics-collector.sh`

`plotGraph.sh`

`plot-metrics.sh`