mm: read from the page cache without folio referencesnoref-filemap-read-with-llm

Problem
-------

Small reads of one file from many threads contend on the folio refcount:
filemap_get_read_batch() takes a reference (lock cmpxchg) and
filemap_read() drops it (lock decl), so every read writes the struct
page line twice.  Matthew raised it in 2024 from a customer workload
[1].  On my laptop (i7-8565U, 4 cores / 8 threads), with 8 threads doing
16-byte pread() of one page, that line takes 99.92% of the load HITMs.

Idea
----

  A folio removed from the page cache goes back to the page allocator
  only after an RCU grace period.

Then a page that xas_load() finds in i_pages under rcu_read_lock() holds
that file's data until rcu_read_unlock(), and filemap_read() copies from
it straight to the user buffer with page faults disabled.  No reference,
no on-stack bounce buffer, no seqcount, no check after the copy, and no
limit on the size of the read.

The earlier attempts ([2], and Kiryl's "fast short reads" [3]) copied
without a reference into a stack buffer and validated afterwards.  They
stopped at what the validation cannot see: free, reuse and re-insert at
the same index [4], migration ("__folio_migrate_mapping() does not use
page_cache_delete()", Hugh [5]), and reading memory that is no longer
page cache at all: secretmem (Pedro), unaccepted memory (Kiryl), "or
even worse MMIO with undesired read-effects" (David [6]; all three in
the thread of [3]).

Deferring the free removes those cases instead of detecting them.  It
has been suggested several times and not tried; Matthew's objection was
"it just feels too risky to have so much memory waiting to be reused"
[7], and that is the part of this series that is least solved (patches 7
and 8, and problem 1 below).

Design decisions
----------------

 - The final free is deferred, not the removal and not the reference.
   PG_rcu_free is set where a folio leaves i_pages (page_cache_delete(),
   page_cache_delete_batch(), replace_page_cache_folio(),
   __folio_migrate_mapping(), collapse_file(),
   shmem_delete_from_page_cache()). __free_pages_prepare() tests it,
   queues the page and returns false, so whoever drops the last
   reference does not need to know.

   So the guarantee is "the memory is not reused", not "the refcount
   stays above zero". Reclaim, migration and split still freeze the
   count under a reader, folio_try_get() stays conditional, and the
   lockless lookup protocol is unchanged. Deferring the page cache's own
   reference instead would be the stronger guarantee, but reclaim frees
   from a frozen count without a put, so there is nothing to defer
   there.

 - Queueing: per CPU, a pending list and one batch whose grace period is
   in flight (queue_rcu_work(), WQ_MEM_RECLAIM workqueue). A per-CPU
   spinlock, irqsave, makes "cut pending into the batch" and "start the
   grace period" one step; without it a page can join a batch whose
   grace period has already started. A page waits one to two grace
   periods. This is the shape of kfree_rcu() batching, without its
   shrinker.

 - A CPU that frees faster than grace periods end must not fill the
   lists. Above 1024 pages on a CPU's pending list a second work item
   takes the list, calls synchronize_rcu_expedited() and frees it
   (patch 7); only that work waits. The loops that remove page cache
   reschedule when the scheduler has asked for it, so that the work gets
   the CPU before the next tick (patch 8).

 - Not deferred: folios of an inode that has I_FREEING, so that rm of a
   big file does not send all of it through the lists. The rule this
   creates: a lockless reader that relies on the deferral must hold a
   reference to mapping->host. An open file does. Also not covered:
   hugetlb (frees to its pool) and the swap cache. filemap_read() reads
   neither.

 - Reuse without a free breaks the guarantee as well.
   page_cache_pipe_buf_try_steal() takes a folio out of the page cache
   and fuse puts it into another file. Patch 4 removes the steal; fuse
   and virtio_console copy instead.

 - Read path: per page, xas_load(), then uptodate, readahead mark,
   dropbehind, i_size, copy. It never reads folio_size() or anything
   else of the head that a split rewrites: a split leaves the bytes in
   place, so the page at the looked-up index is right whatever the folio
   looks like by then. Page faults are disabled for the whole RCU read
   section, and the copy is plain copy_page_to_iter():
   copy_page_to_iter_nofault() would run the hardened usercopy check,
   which reads the compound order.

   FOLIO_BATCH_SIZE pages per rcu_read_lock(). On any reason to stop
   (user fault, folio missing or not uptodate, readahead mark, EOF,
   fatal signal, flush_dcache_folio() needed) the existing path,
   unchanged, continues the read from there.

   folio_mark_accessed() needs a reference, so the fast path is taken
   only where it would change nothing (a test of the flags, for both LRU
   implementations). A hot folio gets there after a few reads through
   the old path.

Numbers
-------

My laptop only, ext4, mitigations off, CONFIG_HARDENED_USERCOPY off.
The laptop throttles on temperature, so compare cost per read, not
reads per second.  16-byte pread() at offset 0 of a one-page file, one
fd per thread:

                          kernel insn   kernel cycles   reads/s
                            per read       per read
  base,   1 thread             973         452-454      7.9-8.1 M
  base,   8 threads          988-989      2937-2975    9.9-10.4 M
  base,   4 threads,
          one per core       979-980      1318-1401     8.9-11.6 M
  series, 1 thread             873         408-409      8.8-8.9 M
  series, 8 threads            864         725-726       30-33 M
  series, 4 threads,
          one per core       873-876       417-438       25-26 M

With the series one thread per core costs 2-7% more than one thread
alone; the 8-thread figure is two SMT siblings sharing a core.  Load
HITMs (perf c2c, 8 threads): 30538 -> 101.

All the cores of this laptop are on one die and share one L3.  A
machine with more CPUs, or more than one socket, should gain more;
nothing bigger has been measured.

Kernel cycles per read, base -> series:

 - 16 bytes, a file per thread, so nothing is shared: 8 threads
   827 -> 730-732, 1 thread 449-456 -> 409-411.  The base pays for SMT
   too; the rest of its 2937-2975 comes from sharing the file.

 - A read into a user page that is not present costs one extra failed
   exception per read(); the old path then takes the real fault and
   finishes the read.  16 bytes, 1 thread: 4117-4188 -> 6543-6593.
   64 KiB with 16 faults: 57478-58073 -> 59648-59822.

 - Large reads stay on the fast path.  64 KiB, 5% fewer instructions:
   1 thread 7334-7346 -> 6917-6929; 8 threads, one file, 17948-18051
   -> 10925-11146.  1 MiB, 6% more instructions: 1 thread
   166544-171752 -> 161315-166941.  1 MiB at 8 threads is not
   comparable in cycles (the clock differed between runs).  In time
   per read: with a file per thread the same, 637-721 us ->
   616-660 us; with one file 257-280 us -> 215-230 us, two runs.

Memory waiting for a grace period.  N processes each fallocate() and
hole-punch 64 MiB of a memfd in a loop; together they fill and free
about 22 GiB/s from N = 4 on.  Mean of what waits, sampled every 2 ms,
in MiB:

   N   patches 1-6   patches 1-8
   1       148           6.6
   4       565            15
   8       503            70
  16       540           177

With patches 1-6 a page waits 20-25 ms and the peaks are 1.0 to
1.2 GiB from N = 4 on.  With patches 7 and 8 the peaks are 59, 128,
249 and 870 MiB, the expedite work runs 4600 times a second at N = 4,
and the processes fill and free as fast as before.  The same loop on
ext4 files (pwrite(), ftruncate(0), 7-9 GiB/s): 124 -> 12 MiB at N = 4.

Kiryl's reproducer (that loop in four processes next to one that
touches 1400 MiB of anonymous memory 20 times), in a 2 GiB VM without
swap: with patches 1-6 the OOM killer takes the large process with 4
and with 8 CPUs, and not with 1 or 2.  With patches 1-8 it runs to the
end with 1, 2, 4 and 8 CPUs and never enters direct reclaim.

The series' HITM count and the figures below are from earlier builds
of the series, which copied with copy_page_to_iter_nofault().

 - Slower frees: a loop of fadvise(DONTNEED) freeing 850 MiB/s had at
   most 28 MiB waiting (sampled every 12 ms).  Truncating a 256 MiB
   file had 208 MiB waiting, gone within 25 ms.  On this laptop from
   boot until the test runs: 3323 pages deferred, 3.29 M not deferred
   because their inode was being evicted.

Testing
-------

evict-race: 4 readers check every byte they read from a clean file
while one thread evicts it with fadvise(DONTNEED) in a loop and reuses
the freed pages at once, and two more recycle page cache and anonymous
memory.  0 bad reads in 30 s runs of about 10^8 reads each.  With a
debug patch that frees PG_rcu_free pages immediately, the same test
gets bytes that are not the file's (12 and 138 bad reads in two runs),
some of them from other users of the freed page, so it does detect a
broken guarantee.  That control was run on an earlier build of the
series; the debug patch has not been redone for this one.

partial-fault: reads of a cached file into buffers with some pages
zapped by MADV_DONTNEED, so that copies fault partway through a page,
with every byte checked.  0 bad in 1.15 M reads.

punch-wait and the VM: the hole-punch loop of "Numbers" with a sampler
of the vmstat counters, and Kiryl's reproducer as the only program of
an initramfs under qemu/KVM.

Those, a gcc build of one x86-64 config, and running the laptop on it,
are all the testing there is.  That config has CONFIG_DEBUG_VM, lazy
preemption and preemptible RCU, and no lockdep, no
CONFIG_DEBUG_ATOMIC_SLEEP and no KASAN.  The last commit builds
without warnings.  An earlier form of the first six was compiled at
each commit, and with clang at the last; patch 7 has not been built
without patch 8.

Known problems, and what I'd like eyes on
-----------------------------------------

 1. The allocator cannot see the waiting pages.  They have refcount 0,
    are not PageBuddy, are on no pcp list and in no counter.

    - It ran a machine out of memory (Kiryl's reproducer, under
      "Numbers").  should_reclaim_retry() computes "available" from
      the LRU counters and NR_FREE_PAGES, the waiting pages are in
      neither, and it decides for the OOM killer while hundreds of MiB
      wait.  Patches 7 and 8 keep the lists short; they do not make
      the allocator see what is on them.  The limit of 1024 pages is a
      guess.  With no idle CPU (N = 16 above) 177 MiB still waits on
      average and 870 MiB at the peak.  kswapd and direct reclaim
      overshoot for the same reason, and compaction's source pages
      reach the free lists late.
    - alloc_contig_range() fails.  The sources of its migration are
      still waiting when test_pages_isolated() runs, so it returns
      -EBUSY for a range that had page cache in it, and cma_alloc()
      goes on to the next range.  That is a regression for CMA as the
      series stands.  hugetlb's deferred free has the same shape, and
      test_pages_isolated() calls wait_for_freed_hugetlb_folios() for
      it.
    - Nothing bounds what waits when a grace period is slow.

    I would rather expedite than wait in the allocator: a
    synchronize_rcu() there throttles page-out.  alloc_contig_range()
    is different, it needs those very pages: a
    get_state_synchronize_rcu() cookie per batch, and
    cond_synchronize_rcu_expedited() plus the free from
    test_pages_isolated().  Not written.  A zone counter for what
    waits is not written either.

 2. One more page flag.  With MAXSMP, NUMA balancing and the page idle
    flag it takes one of MGLRU's two refs bits (arithmetic from
    page-flags-layout.h, not built).  Is there a bit that is dead once
    a folio has left the page cache?

 3. splice.  SPLICE_F_MOVE of page cache pages into fuse is gone; does
    anyone depend on it?  And one more way in is closed only by
    accident: vmsplice(SPLICE_F_GIFT) of a mapped page cache page, then
    truncate, leaves the pipe with the only reference,
    generic_pipe_buf_try_steal() takes the page, and fuse_check_folio()
    refuses it only because PG_rcu_free is not in its list of allowed
    flags (with a dump_page()).  That needs an explicit test.

 4. s390: __make_folio_secure() freezes the refcount.  A reference
    blocks it, an RCU reader does not.  What a reader copies from a
    page that goes secure under it is not worked out.

 5. Cost outside the read path is not measured.  Every deferred free
    takes an irqsave spinlock, and the free itself moves to a kworker
    on whichever CPU ran the RCU callback: cold pages, CPU time not
    charged to the task that freed, and bad-page reports and
    page_owner free stacks that name the kworker.  Reclaim-heavy and
    truncate-heavy loads need numbers.

 6. Arguments that are only arguments.  The ordering from the i_pages
    store to the start of the grace period (store; the remover's
    reference drop; the final dec-and-test; the per-CPU lock; the cut;
    queue_rcu_work()) is argued from the documented guarantees, with no
    litmus test.  Not every ->migrate_folio() was read for "the
    destination is not uptodate before its data is there".

 7. Not tested: xfstests (generic/095 and generic/741 failed with the
    2024 patch [8]), anything that splits folios under the fast path,
    memory pressure other than Kiryl's reproducer in a VM, lockdep,
    KASAN, a kernel linked by clang, 32-bit, CONFIG_HARDENED_USERCOPY,
    any machine but this one.  And any preemption model but lazy with
    preemptible RCU: without preemptible RCU a task that punches holes
    in a loop reports no quiescent state until RCU asks for one, about
    HZ/10 into the grace period, so more would wait (from the code,
    not run).

 8. trace_mm_filemap_get_pages and folio_mark_accessed() are not
    reached by reads the fast path serves, so tools that count either
    see fewer page cache hits.

 9. For RCU.  The deferral keeps its own per-CPU pending list and one
    batch per grace period, started with queue_rcu_work().  That is
    the shape of kfree_rcu(), and RCU's segmented callback lists
    already do the same thing in general.  call_rcu_hurry() per folio
    would need no lock and no workqueue, and would lose the batched
    free_unref_folios(); not tried.  Is there a better way to hook
    in?

    Patch 7 calls synchronize_rcu_expedited() from a work item every
    time a CPU has 1024 pages waiting, 4600 times a second with four
    hole-punching processes.  Its kernel-doc says not to use it in a
    loop.  What the IPIs cost a task that frees nothing is not
    measured.  Is there a way to get a short grace period without
    them?

10. For the scheduler.  With lazy preemption the kworker that runs the
    expedite work waits for the tick while the task that queued it
    keeps the CPU: 0.84 ms to start, against 20 us once that task
    makes way.  Patch 8 calls schedule() in the page cache removal
    loops when TIF_NEED_RESCHED_LAZY is set, which is the
    cond_resched() that lazy preemption took out, and it acts on any
    lazy request.  An unbound workqueue does as well while a CPU is
    idle and nothing when none is.  I am not sure this is the
    direction; I would rather have the freeing task do something
    synchronous that leaves the scheduler alone.

Tools
-----

The code, the test programs, the review behind the list above and the
first draft of this text were done with an LLM coding assistant (Claude
Code) under my direction.  The measurements were run on my machine.

Patches
-------

  1  mm: free PG_rcu_free folios only after an RCU grace period
  2  mm: free folios removed from the page cache only after an RCU
     grace period
  3  mm: don't defer the free of page cache folios of an inode being
     evicted
  4  splice: don't steal page cache folios
  5  mm/filemap: read from the page cache without folio references
  6  mm: count pages whose free waits for an RCU grace period
  7  mm: expedite the grace period when a CPU has many deferred frees
  8  mm: make way for the expedite work in the page cache removal
     loops

Base: 7.3-rc6 plus fixes, 0c2669a9f4a1.

[1] https://lore.kernel.org/all/Zduto30LUEqIHg4h@casper.infradead.org/
[2] https://lore.kernel.org/all/CAHk-=wjOogaW0yLoUqQ0WfQ=etPA4cOFLy56VYCnHVU_DOMLrg@mail.gmail.com/
[3] https://lore.kernel.org/all/20251017141536.577466-1-kirill@shutemov.name/
[4] https://lore.kernel.org/all/Zds8T9O4AYAmdS9d@casper.infradead.org/
[5] https://lore.kernel.org/all/b8e56515-3903-068c-e4bd-fc0ca5c30d94@google.com/
[6] https://lore.kernel.org/all/dcdfb58c-5ba7-4015-9446-09d98449f022@redhat.com/
[7] https://lore.kernel.org/all/Zdz9p_Kn0puI1KEL@casper.infradead.org/
[8] https://lore.kernel.org/all/ZkNQiWpTOZDMp3kS@bombadil.infradead.org/

0 files changed, 0 insertions, 0 deletions