perf: stream sampled profiles instead of loading every line #682

Closed

JooaoPedro15 wants to merge 1 commit from fix/stream-profile-input into main

JooaoPedro15 commented

2026-11-08 16:45:39 +00:00

(Migrated from github.com)

scripts/prof.py reads every line into memory before it aggregates samples, even when skip is zero. Large captures can therefore use much more memory than the report needs.

Stream the input in one pass when skip is zero. For nonzero skip, count physical lines and rewind before streaming the selected range. This preserves the existing slice behavior, symbol lookup, filters and report output.

The branch is rebased on upstream main at 8b2e1d3. The current CI runner discovers the new tests/test_profile_stream.py suite automatically.

python3 -I -B tests/run_all.py --strict profile_stream passed on macOS. The suite covers output, duplicate frames, skip and filter behavior, empty input, and symbol resolution.
git diff --check upstream/main...HEAD passed.
The earlier macOS measurement on 100,000 synthetic samples produced identical output. Peak Python allocations measured by tracemalloc fell from 11.16 MiB to 0.02 MiB.

With nonzero skip, the lower memory use requires a second pass over the input. AI assistance from OpenAI Codex is recorded in the commit trailers.

`scripts/prof.py` reads every line into memory before it aggregates samples, even when `skip` is zero. Large captures can therefore use much more memory than the report needs. Stream the input in one pass when `skip` is zero. For nonzero `skip`, count physical lines and rewind before streaming the selected range. This preserves the existing slice behavior, symbol lookup, filters and report output. The branch is rebased on upstream `main` at `8b2e1d3`. The current CI runner discovers the new `tests/test_profile_stream.py` suite automatically. - `python3 -I -B tests/run_all.py --strict profile_stream` passed on macOS. The suite covers output, duplicate frames, skip and filter behavior, empty input, and symbol resolution. - `git diff --check upstream/main...HEAD` passed. - The earlier macOS measurement on 100,000 synthetic samples produced identical output. Peak Python allocations measured by `tracemalloc` fell from 11.16 MiB to 0.02 MiB. With nonzero `skip`, the lower memory use requires a second pass over the input. AI assistance from OpenAI Codex is recorded in the commit trailers.

NotRexed commented

2026-11-09 05:29:55 +00:00

(Migrated from github.com)