L3/L4 dataflow layer is serial in the driver; naive Ray fan-out measured 3.1x slower

[rahlk]

Is your feature request related to a problem? Please describe.

The L3/L4 dataflow layer runs serially in the driver process while only the symbol table is distributed over Ray. On odoo full (6,086 modules) the driver sat at 99% of a single core with 12.3 GB resident for ~7 minutes after the Ray phase completed. On erpnext it is ~60% of L4 wall time.

The obvious fix — fan build_function_pdgs out over Ray — was implemented and measured to be 3.1× slower, so this issue records the negative result and points at the real target.

Describe the solution you'd like

Not stated in the original issue.

Describe alternatives you've considered

Not stated in the original issue.

Additional context

What was tried (and reverted)

build_function_pdgs fanned out one Ray task per contiguous slice of modules (4 slices/core), oracle selected by name (ORACLE_SYNTACTIC / ORACLE_SCALPEL) because a factory closure cannot be pickled, results merged in submission order to preserve key insertion order.

Correctness was fine. Output equivalence on erpnext L4: identical callables (8,182), identical cfg (150,164) and cdg (83,043), ddg within 0.05% (1,804,611 vs 1,805,564), same prov mix, key order preserved. The residual delta sat inside the #146 noise floor (serial-vs-serial differed on 3 callables, serial-vs-parallel on 2).

Performance was not. Interleaved A/B on erpnext L4, --ray, same machine:

variant	wall	load at start
parallel dataflow	855s	9.57
serial dataflow (baseline)	272s	14.93

The parallel variant lost badly while running under lower load, so this is not a measurement artifact. Isolated micro-benchmarks did show phase 1 itself parallelizing well (flask 13.3s → 4.6s; erpnext-scale probe 310s → 19s), which means the loss is integration overhead, not the algorithm — most plausibly per-task worker startup and module imports (codeanalyzer + jedi + scalpel + pydantic) paid across ~80 tasks per run, on top of Ray's runtime_env provisioning that _ensure_ray configures.

Where the time actually is

Phase profile of L4 on erpnext (228.6s total run):

phase	time	parallelism
build_function_pdgs (×2: syntactic + Scalpel)	44.5s	embarrassingly parallel
compute_summaries	33.8s	whole-program fixpoint, serial
assemble_sdg	28.8s	global assembly, serial
callsite/nested-def wiring	~8.4s	parallel after a broadcast
emit_l3_body + emit_l4	13.5s	serial projection

So ~48% of the dataflow layer is inherently serial whole-program work. Even a free phase-1 parallelization caps out at roughly a 15–20% cut to L4 wall time — which is why the integration overhead was able to swallow it whole.

CAVEATS

• Any retry must be measured interleaved (A/B/A/B) with load recorded per run. Three separate measurement errors were made on this task — concurrent runs, falling-load ordering, and a micro-benchmark whose 310s serial figure cannot be reconciled with a 272s total run. Back-to-back ordering is not sufficient on this machine.
• Worker-pool reuse (long-lived Ray actors that import once and process many slices) is the only variant likely to beat serial, since it amortizes the per-task import cost that appears to dominate.
• The unexplained micro-benchmark discrepancy is worth resolving on its own: build_function_pdgs with the Scalpel oracle measured 310s standalone but the entire serial CLI run is 272s. Something differs between the standalone PyApplication and the CLI's, likely how often make_alias_oracle degrades to the type-based fallback.

DEFINITION OF DONE

• Either a measured net win on erpnext and odoo full under interleaved A/B, or a decision recorded that the dataflow layer stays serial and effort moves to the serial 48%.
• If parallelized: byte-equivalent output (modulo L2 call graph is not byte-reproducible: Jedi overload resolution for open() varies run to run #146) on flask, requests, erpnext.
• The compute_summaries / assemble_sdg serial tail is profiled at odoo-full scale, since it — not phase 1 — is the dominant cost at scale.