firm-microsim-paper

Synthetic firm-level microsimulation of the UK VAT registration threshold.

This repository builds an open, firm-level synthetic population of UK businesses — calibrated to official ONS and HMRC aggregates — that resolves the turnover distribution at the individual-firm level around the VAT registration threshold, where published statistics only report coarse bands. It supports bunching estimation, static revenue costings, and dynamic firm re-optimisation around the threshold notch.

Layout

firm-microsim-paper/
├── data/                 # official inputs + generated output (see data/README.md)
│   ├── raw/              # pristine ONS + HMRC source workbooks
│   ├── processed/        # derived band tables, by vintage (2023-24, 2024-25)
│   └── synthetic/        # generated synthetic population (regenerated, not committed)
├── src/firm_microsim/    # installable package
│   ├── config.py         # vintage, VAT threshold, paths, hyperparameters
│   ├── generate.py       # synthetic-population generator
│   ├── static/           # static threshold-reform results
│   ├── bunching/         # reduced-form bunching estimator
│   ├── notch/            # structural notch diagnostics
│   ├── dynamic/          # conditional behavioural costing
│   └── analysis/         # paper table and diagnostic scripts
├── pyproject.toml        # package metadata, dependencies, console scripts
├── paper/                # Quarto/LaTeX manuscript sources and checked PDF
├── results/              # generated figures + calibration_accuracy.txt
└── requirements.txt

Method (data generation)

A two-stage synthetic-population pipeline, parameterised by a single VAT threshold:

1. Draw base firms from the ONS business structure — sample continuous turnover strictly inside each published band, sector-conditional employment, and intermediate inputs. These within-band values are explicit modelling assumptions, not recovered administrative microdata.

2. Calibrate firm weights by multi-objective optimisation (Adam, symmetric relative-error loss) on three declared universes (issues #37, #25):

   • the ONS VAT/PAYE enterprise frame (~2.72M enterprises): the population total and the employment-band totals;
   • the HMRC VAT-registered subset (~2.18M traders): trader counts by turnover band and by trade sector, and net VAT liability by band. Each frame firm enters these rows with a registration propensity — the HMRC count in its turnover band divided by the frame's mass in that band (0.89 below the threshold; 0.51–0.94 above it, the remainder being PAYE-only or exempt-sector enterprises outside the VAT net; the open ONS "5000+" band is drawn log-uniform on [£5m, £50m), issue #40). HMRC negative/zero-turnover traders are appended before calibration as an out-of-frame stratum;
   • the DBT unregistered stratum (~2.86M businesses registered for neither VAT nor PAYE, from the Business Population Estimates by SIC division, scripts/etl_bpe_tables.py): drawn exponential with each division's BPE mean turnover, weights frozen in calibration, registrable below the threshold and out of scope (exempt) above it. With the stratum present the OBR £1k-bin counts enter as levels — on frame + stratum below the threshold, frame alone above — so the frame's near-threshold density is the residual between the OBR chart and the stratum.

   Within closed ONS bands above the first, turnover is drawn from a truncated power law whose exponent is the log-log slope of neighbouring band densities (monotone within bands, approximately continuous across edges); the bottom band is uniform.

   VAT scope and registration flags are then assigned by seeded weighted selection per band so registered totals match HMRC to within one weight.

The result is ~5.8M firm rows (2.72M frame draws, 2.86M unregistered-stratum businesses, 0.22M appended traders) weighted to ~2.72M frame enterprises, of which ~2.18M are VAT-registered, plus the stratum at fixed weight. Because the population is calibrated to the HMRC aggregates, agreement with them is an internal consistency check, not external validation.

The official target surface is also being mirrored into PolicyEngine Ledger and Populace. This repository keeps the paper's archived CSV inputs and generator for reproducibility, and includes a Populace/Ledger comparison command so the pinned migration snapshot can be audited without silently changing the published paper population.

Data vintages — single version, one-line switch

The pipeline is single-version: there is one VAT_THRESHOLD, not separate 85k/90k scripts. Two coherent official-data vintages are available and selected with a single switch (see data/README.md):

Vintage	Data	Threshold	Role
2023-24 (default)	ONS 2024 + HMRC 2023-24	£85,000	Paper baseline
2024-25	ONS 2025 + HMRC 2024-25	£90,000	Latest gov data

Usage

Install the package in editable mode, then run the package entry points:

uv sync --extra dev --locked
uv run python -m pytest

uv run firm-microsim          # ALL DATA: every vintage + calibration report + figures
uv run firm-microsim-static   # ALL STATIC RESULTS: threshold-reform figures

firm-microsim with no arguments runs the full data build — it generates synthetic_firms_<vintage>.csv for every vintage, writes results/calibration_accuracy.txt, and renders the descriptive figures. Single steps are still available:

firm-microsim --vintage 2024-25             # one vintage only (£90k)
firm-microsim --threshold 88 --seed 7 --output my_run.csv
firm-microsim-report                        # calibration report only
firm-microsim-figures                       # descriptive figures only
firm-microsim-populace-ledger               # Populace/Ledger comparison

import firm_microsim
df = firm_microsim.generate()                      # baseline
df = firm_microsim.generate(vintage="2024-25")     # latest
df, report = firm_microsim.generate(return_report=True)

Output is written to data/synthetic/synthetic_firms.csv (sic_code, annual_turnover_k, annual_input_k, vat_liability_k, employment, weight, vat_scope, vat_registered, in_frame, unregistered). in_frame marks ONS-frame enterprises; unregistered marks the DBT stratum; vat_scope marks firms in the VAT net (registered above the threshold, registrable below it); vat_registered marks HMRC-count-matched traders. Static and behavioural costings use in-scope firms only.

The processed ONS tables are rebuilt from the raw workbooks by scripts/etl_ons_tables.py (Table 8 for turnover sizebands, Table 3 for employment sizebands, both counting enterprises); --check fails if a checked CSV differs from the extract, and tests/test_data_universes.py runs it.

Paper

The manuscript is rendered with Quarto, while preserving the existing LaTeX section files and template:

cd paper
quarto render index.qmd --to pdf

The PolicyBench-style web paper is generated from the same LaTeX sections and rendered with PolicyEngine design tokens:

cd paper
python3 build_site.py

The deployable static site is written to paper/site/. Quarto writes PDF render artifacts to paper/out/; the checked-in submission PDF is paper/main.pdf. The legacy direct-LaTeX entrypoint paper/main.tex remains available for debugging and uses the same shared frontmatter/body inputs.

The current Vercel deployment is https://firm-microsim-paper.vercel.app/.

Citation and licensing

Code in this repository is released under the MIT License. Cite the paper and software using CITATION.cff. The archived ONS, HMRC, and OBR source files remain subject to their original public-sector terms; the MIT licence does not relicense third-party data. A submission release should be tagged and archived with a DOI, which can then be added to CITATION.cff and the manuscript without changing the computational results.

Calibration accuracy

The population is calibrated to five official ONS + HMRC target groups; the validator scores each dimension as accuracy = max(0, 1 − |synthetic − target| / |target|), HMRC dimensions on VAT-registered rows and ONS dimensions on frame rows. The displayed error is the clipped complement of that score, not a signed relative error. Overall is the simple mean over the five calibrated dimensions below. Reproduce with:

firm-microsim-report

Calibrated dimension	85k (2023-24)	90k (2024-25)
HMRC turnover bands (registered rows; hold by construction)	100.0%	100.0%
ONS population (frame rows)	99.7%	99.8%
Employment bands (frame rows)	99.8%	99.9%
Sector distribution (registered rows)	93.3%	92.2%
VAT liability by band (6 calibrated bands, registered rows)	97.1%	98.0%
Overall (5 calibrated dimensions)	98.0%	98.0%

VAT liability by sector is not a calibration target — it is reported as an informational diagnostic only, and neither is the below-threshold (£1-to-Threshold) liability band: its HMRC total is remitted by voluntary registrants (input-reclaim traders averaging ~£2,150 net) whom the standard-rate-on-value-added liability model does not represent, so calibrating it against the whole below-threshold population distorts near-threshold weights (see the paper's data section). The model draws per-firm input shares (mean value-added share ≈ 40%) and sets net liability v = 0.20 × (turnover − input) — the standard rate applied to value added — but does not yet calibrate the input/output tax structure by sector, so per-sector net liability is structurally unhittable and is gated off via Config.calibrate_vat_liability_sector = False. Restoring it after input/output calibration is tracked in issues #1 and #2. An earlier build set v = turnover − input (no 0.20 factor); the correction and its consequences are documented in #15 and the paper's Section 5.

Fast iteration builds

firm-microsim --fast runs the full pipeline on a stratified sample (~20% of rows: 30% inside the £15k–£150k analysis window, 5% outside, per-stratum floors), carrying the thinned mass as base weights so every calibration target remains a true total. A vintage builds in ~15 seconds instead of ~13 minutes; headline aggregates reproduce the full build within ~0.3% and local bunching statistics within ~5%. Use for development only — release artifacts are full-size. Generator-seed sensitivity of the full build is recorded in results/seed_sensitivity.txt (E ±165, raise ±£1.1m / taper ±£1.1m, base ±£0.63bn across seeds — the scope flag is a seeded draw; reproduce with scripts/seed_sensitivity.py).

Populace/Ledger migration check

firm-microsim-populace-ledger reports the current migration comparison. The checked reference run used the 2024-25 Ledger target surface from PolicyEngine/ledger#67 at cd98b5cb7b1604fbf7750689a429bbc356e5603a and Populace's experimental UK firm generator from PolicyEngine/populace#223 at fa20daf75ff023e5e88731a140f456f58e0b864e. Both upstream PRs merged on June 30, 2026: Ledger at merge commit ac643afa0c1d45fc4abd0268dc5aa7c843440b38, and Populace at merge commit 8271d767244161631253ad1d9ad792a82e2b96b4. The reference population uses 1,000 calibration iterations:

firm-microsim-populace-ledger \
  --output results/populace_ledger_comparison.txt \
  --json-output results/populace_ledger_provenance.json

When populace-build is installed from the Populace source tree, the same command can recompute the table and paper-CSV parity from Ledger consumer facts:

firm-microsim-populace-ledger \
  --facts-jsonl /path/to/uk_firm_consumer_facts.jsonl \
  --iterations 1000 \
  --output results/populace_ledger_comparison.txt \
  --json-output results/populace_ledger_provenance.json

The current reference comparison shows exact parity between the Ledger-backed targets and the paper's processed 2024-25 numeric inputs: six normalized source tables checked, zero mismatches, max numeric difference 0. It does not exactly replicate the paper's generated synthetic population: Populace's shared optimizer landed at 93.8% overall accuracy under its own validator versus the paper's then-90.5% (2024-25 scores 98.0% on the current two-universe build, results/calibration_accuracy.txt), but that overall pair is not like-for-like: HMRC turnover-band accuracy uses different band sets, and sector distribution reflects different calibration-target definitions. The directly comparable rows are ONS population, employment bands, and VAT liability by turnover band. Note: the pinned Populace snapshot predates this repo's net-liability correction (issue #15) and inherits the same v = turnover − input mis-scaling; the target-parity result is unaffected (it concerns input tables, not generated rows), but the Populace generator needs the same fix upstream. The Populace/Ledger path remains a migration check rather than a silent replacement for the paper's archived generator/results.

Figures

Figures follow the project house style: single clean panels (no embedded titles, source notes, or logos — captions and side-by-side layouts are composed in LaTeX), teal palette, saved as snake_case PNGs to results/ at 300 dpi. They are produced by the in-package firm_microsim.figures module and generated for both vintages (two full sets, suffixed _85k / _90k):

firm-microsim-figures          # regenerate every figure, both vintages

results/ then contains:

Figure	85k (2023-24)	90k (2024-25)	Source
All UK firms by turnover band	firms_by_turnover_band_85k.png	firms_by_turnover_band_90k.png	ONS
VAT-registered firms by turnover band	vat_firms_by_turnover_band_85k.png	vat_firms_by_turnover_band_90k.png	HMRC
Full-range turnover distribution	turnover_distribution_85k.png	turnover_distribution_90k.png	synthetic

The turnover-distribution figures require the matching synthetic CSV (data/synthetic/synthetic_firms_<vintage>.csv); generate it first with firm-microsim --vintage <vintage> --output synthetic_firms_<vintage>.csv.

Note on ONS counts: firm-count figures sum the per-SIC rows only — the ONS band tables include a Total summary row that must be excluded, or every firm is counted twice (a bug present in earlier drafts that doubled the ONS panel).

Static threshold reform results

The firm_microsim.static module costs VAT-threshold reforms mechanically (turnover held fixed; only registration status changes), reproducing the paper's static results. Run:

firm-microsim-static          # -> results/{vat_threshold_revenue_impact,revenue_impact_2025_26,firms_impact_2025_26}.png

• vat_threshold_revenue_impact.png — the £85k→£90k anchor reform vs HMRC's published costing, by fiscal year. Built on the £85k / 2023-24 vintage — the pre-reform basis HMRC actually had at the 6 March 2024 costing. Each year differences revenue under the £90k policy and under the counterfactual threshold path (85/85/87/89/92k) on data-year turnover: mechanical −193/−197/−120/−41/+88 vs HMRC −150/−185/−125/−50/+65 £m; with 43% voluntary retention −110/−112/−68/−24/+88. HMRC's figures lie between the two conventions in every release year. See results/static_sweep.txt for the deregistration-gap sensitivity.
• revenue_impact_2025_26.png / firms_impact_2025_26.png — the forward static sweep of registration thresholds (£70k–£120k) vs the current £90k baseline, on the £90k / 2024-25 vintage.

Two vintages, two exercises, one convention. The anchor reform uses the £85k vintage, where the affected [85,90)k band sits above the £85k registration threshold and is cleanly populated with in-scope registered firms. The forward sweep uses the current £90k vintage. Both are direct mechanical reclassifications of in-scope firms on data-year turnover (a fixed cross-section has no growth process; ageing the near-threshold stock across the threshold would treat bunched firms as crossers); liabilities are aged by the fiscal-year factor. A rise releases every in-scope registrant in the vacated band (they no longer need to register); the statutory £2k deregistration gap and the LLAT 43% retention share are reported as sensitivities. Voluntary registrants stay registered. Cuts draw in the unregistered stratum. results/static_sweep.txt holds the machine-readable table.