| Age | Commit message (Collapse) | Author |
|---|
|
A collapse is two jobs. One reads a PTE table under mmap_lock and decides
whether the range is worth collapsing. The other allocates, isolates,
copies and flushes, and wants the lock given up first.
collapse_single_pmd() did both, so the boundary between them was somewhere
in the middle of a function.
Give each half its own function:
- collapse_scan_pmd() scans one table. It picks the side:
collapse_scan_anon(), the anonymous scan that used to carry the
collapse_scan_pmd() name, or collapse_scan_file().
- collapse_pmd() does the collapse the scan asked for, and is handed
what the scan returned. SCAN_SUCCEED means there is something to
collapse. SCAN_PTE_MAPPED_HUGEPAGE means the page cache already
holds the PMD folio and only the PTE table is left to retract.
Anything else is why there is nothing to do. It calls
collapse_anon(), formerly mthp_collapse(), or collapse_file().
For now collapse_single_pmd() stays, as the two of them with the
mmap_lock drop in between, so its callers see what they saw before.
collapse_control_init() sets a control up before its first scan.
What the scan found and the collapse needs travels in collapse_control.
For an anonymous table that is the orders and the referenced and
swapped-out counts. collapse_anon() and collapse_huge_page() now read
them from there instead of taking them as arguments.
For a file it is the file itself and the offset in it. A file collapse
works on the page cache and never sees a VMA, so the scan takes the
reference while it still has one and the collapse gives it back.
The file scan moves under mmap_lock with the anonymous one, where before
the lock was given up first. The lock is now held over the page cache
walk, an RCU walk over one table's worth of slots with no PTL, and taken
fewer times. collapse_scan_mm_slot() ends its walk whenever the lock was
dropped, so a refused file table used to cost khugepaged an unlock, a
trip back through khugepaged_do_scan(), a relock and a VMA lookup. Now
only a table that goes on to be collapsed does.
Link: https://lore.kernel.org/20261014171145.3292736-10-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
Tests scattered through the collapse path decide what a collapse is
allowed to do by asking whether khugepaged started it. Between them they
settle:
- which VMAs are eligible, and how hard to try for a folio;
- how many empty, swapped-out or shared PTEs a window may contain, and
whether a sub-PMD window is held to a stricter rule than a PMD;
- whether a range has to look used, and whether a MADV_FREE'd page is
left alone;
- whether the PMD is mapped as part of the request, and whether dirty
pages are worth writing back and retrying.
None of those is a fact about khugepaged. Each is something the caller
decided before asking, and the collapse code should not have to look up
who called to find out.
Add struct collapse_policy for the caller to fill: khugepaged from its
own settings, MADV_COLLAPSE from the fact that a user asked explicitly.
Every test becomes a read of a field, and cc->is_khugepaged goes, having
no reader left.
The PTE limits come as two sets, one for a PMD-sized window and one for
anything smaller, so that the helpers pick a set for the order and read
it. khugepaged takes no swapped-out or shared PTE into a sub-PMD window,
and empty PTEs only when the knob says all or nothing; the warning for a
knob value in between moves to where khugepaged fills its policy. The
fields only one side reads say which: anon_ or file_. David Hildenbrand
asked for both.
khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per
call. That is the one change in behaviour. The max_ptes_* limits and the
defrag setting behind the allocation mask are sampled once per pass rather
than on every table. A table scanned early in a pass and one scanned late
are then treated alike.
collapse_file() also drops a NULL check on the collapse_control. It has
one call site, reached only from collapse_single_pmd(), which dereferences
cc unconditionally, so the check was already dead.
Link: https://lore.kernel.org/20261014171145.3292736-6-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
khugepaged.c holds both the users of collapse and the machinery that
performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's
entry point and the collapse itself all sit in one file and reach into
each other freely. Nothing marks where a user ends and the engine
begins.
Start drawing that line.
Add mm/collapse.h for what the two sides have to agree on:
- enum scan_result - what the engine hands back;
- struct collapse_control - the state a request carries.
And two constants move with them:
- KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT;
- KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER.
Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix.
No functional change.
Link: https://lore.kernel.org/20261014171145.3292736-5-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
