SysadminNews

Nvidia Vera: why the scheduler prefers the same sibling

On this page
  1. The cost is in the transition
  2. Read the benchmark with its conditions
  3. What an independent comparison would need
  4. Source

Nvidia’s August 31 scheduler proposal prefers PE0 when both siblings of an Olympus core are available. It is a consistent placement choice, not a claim that PE0 is intrinsically faster than PE1.

Illustrative non-overlapping work bursts on symmetric PE0/PE1 siblings. A stable choice leaves the other sibling idle longer; the diagram is not a measured scheduling trace.
Illustrative non-overlapping work bursts on symmetric PE0/PE1 siblings. A stable choice leaves the other sibling idle longer; the diagram is not a measured scheduling trace. Chart : PeopleAreGeek. Data source.
View full-size image

The cost is in the transition

Andrea Righi’s patch cover letter describes two symmetric processing elements sharing one core. Returning to full single-thread resources after the other sibling becomes idle takes a qualification interval. Repeatedly switching the active sibling can therefore matter even when useful work barely overlaps.

The proposal first selects an idle core, then the preferred available sibling within that core. The generic behaviour requires an architecture-provided SD_ASYM_PACKING SMT domain. It is not enabled indiscriminately on every machine with SMT.

The timeline illustrates sequential bursts: keeping them on PE0 lets PE1 stay idle. It does not plot measured durations, and it does not imply that all workloads should be pinned manually to PE0.

Read the benchmark with its conditions

The author reports approximately 9.4 to 10.1 TFLOP/s for an 88-thread single-precision GEMM on 88 physical cores of one NUMA node in a two-node Vera system. From those rounded figures, (10.1 / 9.4 - 1) × 100 is about 7.4%.

That is a calculation from Nvidia’s experiment, not our benchmark and not a universal SMT gain. It must not be combined with results from earlier scheduler fixes as though they used one baseline. The previous article mixed this proposal with the earlier asymmetric-capacity work and a roughly 2× figure; that account has been replaced.

What an independent comparison would need

Keep workload, thread count, affinity, NUMA placement, compiler options and kernel baseline identical. Repeat the test and retain both throughput and variance. A task that saturates both siblings continuously is not the same case as intermittent single-thread bursts.

For production planning, first establish whether your exact kernel and architecture activate this policy. A discussion on the mailing list is not a released distribution feature. Manually pinning an unrelated server’s processes based on this article could defeat its scheduler’s other placement decisions.

The general lesson is narrower and useful: logical CPUs can have equal nominal capacity while the history of their activity changes immediately available resources. Measuring only utilization averages can miss those transitions.

Source

Replaced the conflated March/asymmetric-capacity narrative with the August proposal’s actual mechanism; distinguished symmetric PEs and vendor-only benchmark results.