Repository navigation
Sched: make the cost estimate of a task with many candidate processors cheap - #791
Open
AnnaLena77 wants to merge 4 commits into
Open
AnnaLena77 wants to merge 4 commits into
AnnaLena77 wants to merge 4 commits into
Conversation
…per task Under a uniform-execution acceleration (MPI) every task re-sorted all of its candidate processors with a comparison sort whose key (processor_order_key) formats a short_name string on every comparison, and schedule_one! first built the candidate vector with a quadratic 'proc in input_procs' scan and ran estimate_task_costs!, whose result the uniform selection discards. Unscoped @Spawn therefore cost O(P log P) string constructions per task and grew with the processor count: ~85 ms/task at 832 processors (see JuliaParallel#771). * sort_processors_canonical!: compute the keys once per processor, then sortperm. * compatible_processors_ordered: cache the canonical order of a candidate set next to the compatible set (CompatEntry), so it is built once per set. * place_processors_uniform!: the per-task part only -- owner preference (binary search over the owner-sorted canonical order, three in-place reversals) and the thunk-id rotation. No scratch, no per-candidate boxing. * schedule_one!: under uniform execution start from the cached order and skip estimate_task_costs!; otherwise drop the redundant quadratic membership scan (the candidates come from a Set). * select_processors_uniform! keeps its contract (Datadeps and proc grids call it) and is now sort_processors_canonical! + place_processors_uniform!. The resulting order is identical to the old algorithm's, which uniformity across ranks depends on: test/uniform-placement.jl compares it against a copy of the old implementation on randomized candidate sets (partial scopes, absent preferred ranks, all rotations; 5770 assertions) and test/mpi/spawn.jl still passes (2 ranks x 2 threads). Measured with the new benchmark/suites/tasks.jl (suite 'tasks', ntasks=256), 4 ranks x 16 threads per node, Fulda (Opteron 6272), master vs this commit, median of 7 samples, same suite and nodes for both: unscoped spawn, 64 tasks 64 procs (1 node) 244.3 ms -> 11.7 ms (21x) 128 procs (2 nodes) 547.2 ms -> 12.2 ms (45x) 256 procs (4 nodes) 1257.9 ms -> 14.0 ms (90x) allocations of that benchmark 0.75 M -> 13 k, 1.74 M -> 19 k, 4.0 M -> 18.7 k The unscoped cost no longer grows with the processor count (about 0.2 ms/task at 256 processors). Pinned (ExactScope) benchmarks have a single candidate (the old cost estimate returned immediately for it) and are within noise; test/allocations.jl is unchanged (task spawn 183 allocs vs a bound of 500).
do_tasks ends every enqueue by 'kicking' the doorbell of every other processor of the scheduler, to make it try to steal. A runner whose processor may not steal (stealing_permitted false: every MPIProcessor) wakes up, finds that out and goes back to sleep, so under MPI each enqueued task woke ranks x threads runners on every rank: 256 wake-ups per task at 256 processors, about 832 at the application's size. The wake-ups are task switches on threads that could be running tasks, plus the allocations of the runner loop. Skip processors for which stealing_permitted is false. A kick only makes a runner try to steal, so skipping them cannot change which task runs where. Measured with benchmark/suites/tasks.jl on Fulda (Opteron 6272), 4 ranks x 16 threads per node, median of 7, before = 83475c6 (previous commit), 1/2/4 nodes = 4/8/16 ranks: unscoped (64 tasks) 11.0 / 12.8 / 15.9 ms -> 7.3 / 9.8 / 10.4 ms pinned, round robin (256) 44.6 / 58.9 / 47.8 ms -> 38.3 / 39.3 / 39.8 ms dependent hops (64) 55.4 / 66.2 / 64.8 ms -> 41.8 / 43.5 / 42.8 ms fan-in tree (256 leaves) 95.5 / 98.4 / 134.2 ms -> 87.9 / 90.4 / 94.0 ms allocations, pinned round robin 64.6 k / 83.1 k / 67.5 k -> 40.7 k / 41.1 k / 42.9 k allocations per pinned task in a clean loop (4 ranks x 16 threads): 183.7 -> 159.0 Every one of the 27 benchmark/size cells is faster or equal; allocations fall by 20-65 %. No semantic change: the skipped runners would not have stolen anything.
…eping it compat_entry inserted with BasicLFUCache.get!, which for a present key returns the existing value. After the processor list changed without a cache flush (here: a one-rank communicator's processors joining the context after the world's tasks had warmed the cache) the entry for that key no longer matched its procs, so every following lookup missed, recomputed the compatible set and re-sorted it, and the stale entry stayed. Measured on 4 ranks x 16 threads: a suite that registers the extra processors mid-run made unscoped tasks 5.3x slower with 10x the allocations (7.5 -> 40.1 ms per 64 tasks); without the registration they stay at 1.15x. Replace the entry on a miss.
…s cheap
An unscoped task on a process with many threads is placed by estimate_task_costs!,
which evaluated every candidate thread with dictionaries keyed by the abstractly
typed Processor and a dynamic dispatch on every element. The per-task cost grew
with the thread count (the maintainers' MPI notes list 'per-task cost grows with the
thread count of one process' as an unexplained observation): 126 us at 8 threads
and 553 us at 64 on the Fulda nodes, for an empty task, in a single process.
* Costs live in a vector aligned with the candidates instead of a
Dict{Processor,Float64}: boxing a key per insertion and a lookup per comparison
in the sort are gone. The dictionary is only filled when the caller asks for it
(estimate_task_costs, used by Datadeps and the tests); the scheduler passes nothing.
* What is the same for every candidate is computed once: with no Chunk argument the
transfer-cost table is not built at all, and the per-worker pressure and
transfer-rate tables are looked up once per run of candidates of one worker.
* The two per-candidate loops are function barriers specialised on the candidates'
common concrete type (every thread of a process), so get_parent, root_worker_id and
the keyed lookups are no longer dynamic calls. Mixed candidates keep the generic
path. Equal-cost candidates are still shuffled and then stably sorted, so ties
keep their random order as before.
Measured in one process without MPI on Fulda (Opteron 6272), unscoped empty task,
us per task, 8 / 16 / 32 / 64 threads:
before 125.7 / 158.0 / 251.6 / 552.7
after 101.0 / 110.0 / 151.0 / 331.2 (-20 % / -30 % / -40 % / -40 %)
The cost still rises with the thread count (at 64 threads the scheduler lock and
the dispatch of the task dominate), but much more slowly.
test/scheduler.jl 'Scheduler algorithms' (cost values, per-processor pressure,
per-processor transfer rate, signatures; 82 assertions) passes unchanged and
test/allocations.jl is unchanged (task spawn 183 allocations, bound 500).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #790.
This branch contains the three commits of #790 (placement constants) followed by one new commit, "Sched: make the cost estimate of a task with many candidate processors cheap". Please review only that last commit. Once #790 is merged, this PR will shrink to that single commit. I'm opening it now so the review can run in parallel.
An unscoped task on a process with many threads is placed by
estimate_task_costs!, which evaluated every candidate thread with dictionaries keyed by the abstractly typedProcessorand a dynamic dispatch on every element. The per-task cost therefore grew with the thread count of one process: 126 µs at 8 threads and 553 µs at 64 for an empty task.Dict{Processor,Float64}, so there is no key boxing per insertion and no lookup per comparison in the sort. The dictionary is only filled when the caller asks for it (estimate_task_costs, used by Datadeps and the tests); the scheduler passes nothing.Chunkargument the transfer-cost table is not built, and the per-worker pressure and transfer-rate tables are looked up once per run of candidates of one worker.get_parent,root_worker_idand the keyed lookups are no longer dynamic calls. Mixed candidates keep the generic path. Equal-cost candidates are still shuffled and then stably sorted, so ties keep their random order.Measured in one process without MPI (AMD Opteron 6272), unscoped empty task, µs per task at 8 / 16 / 32 / 64 threads:
That is -20 / -30 / -40 / -40 %. The cost still rises with the thread count (at 64 threads the scheduler lock and the dispatch dominate), but much more slowly.
test/scheduler.jl"Scheduler algorithms" (cost values, per-processor pressure, transfer rate, signatures) passes unchanged, andtest/allocations.jlis unchanged. On a laptop (Julia 1.13) the Allocations, Thunk, Scheduler, Processors, Uniform Placement and Options suites pass.AI disclosure: I prepared this change with the help of Claude (Anthropic) and reviewed it myself.
🤖 Generated with Claude Code