| Age | Commit message (Collapse) | Author |
|---|
|
Convert to const struct mnt_idmap.
A mount's idmapping is immutable. The only thing that is allowed to be
modified afterwards is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only ever reads
from the idmapping. This is the same model that struct cred uses and the
idmapping is also rather sensitive.
So make the idmap argument const wherever we can. The conversion is done
from the bottom up so callers can continue to pass a non-const pointer
to a const parameter until the conversion is finished.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-10-54ccd48e100b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Convert to const struct mnt_idmap.
A mount's idmapping is immutable. The only thing that is allowed to be
modified afterwards is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only ever reads
from the idmapping. This is the same model that struct cred uses and the
idmapping is also rather sensitive.
So make the idmap argument const wherever we can. The conversion is done
from the bottom up so callers can continue to pass a non-const pointer
to a const parameter until the conversion is finished.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-9-54ccd48e100b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Convert to const struct mnt_idmap.
A mount's idmapping is immutable. The only thing that is allowed to be
modified afterwards is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only ever reads
from the idmapping. This is the same model that struct cred uses and the
idmapping is also rather sensitive.
So make the idmap argument const wherever we can. The conversion is done
from the bottom up so callers can continue to pass a non-const pointer
to a const parameter until the conversion is finished.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-8-54ccd48e100b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Convert to const struct mnt_idmap.
A mount's idmapping is immutable. The only thing that is allowed to be
modified afterwards is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only ever reads
from the idmapping. This is the same model that struct cred uses and the
idmapping is also rather sensitive.
So make the idmap argument const wherever we can. The conversion is done
from the bottom up so callers can continue to pass a non-const pointer
to a const parameter until the conversion is finished.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-7-54ccd48e100b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Convert to const struct mnt_idmap.
A mount's idmapping is immutable. The only thing that is allowed to be
modified afterwards is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only ever reads
from the idmapping. This is the same model that struct cred uses and the
idmapping is also rather sensitive.
So make the idmap argument const wherever we can. The conversion is done
from the bottom up so callers can continue to pass a non-const pointer
to a const parameter until the conversion is finished.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-6-54ccd48e100b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Convert to const struct mnt_idmap.
A mount's idmapping is immutable. The only thing that is allowed to be
modified afterwards is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only ever reads
from the idmapping. This is the same model that struct cred uses and the
idmapping is also rather sensitive.
So make the idmap argument const wherever we can. The conversion is done
from the bottom up so callers can continue to pass a non-const pointer
to a const parameter until the conversion is finished.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-5-54ccd48e100b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Convert to const struct mnt_idmap.
A mount's idmapping is immutable. The only thing that is allowed to be
modified afterwards is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only ever reads
from the idmapping. This is the same model that struct cred uses and the
idmapping is also rather sensitive.
So make the idmap argument const wherever we can. The conversion is done
from the bottom up so callers can continue to pass a non-const pointer
to a const parameter until the conversion is finished.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-4-54ccd48e100b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Convert to const struct mnt_idmap.
A mount's idmapping is immutable. The only thing that is allowed to be
modified afterwards is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only ever reads
from the idmapping. This is the same model that struct cred uses and the
idmapping is also rather sensitive.
So make the idmap argument const wherever we can. The conversion is done
from the bottom up so callers can continue to pass a non-const pointer
to a const parameter until the conversion is finished.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-3-54ccd48e100b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
struct mnt_idmap is immutable once alloc_mnt_idmap() has set it up. The
only thing that changes afterwards is the reference count. Model
mnt_idmap_{get,put}() after struct cred's reference count handling and
take a const pointer.
This model is forced upon us because ceph stores the idmapping in mds
requests and need to be able to take and put references.
mnt_idmap_get() keeps returning a non-const pointer for now because
struct vfsmount still stores a non-const pointer. We can only change
that at the end of the series once every user of mnt_idmap() has been
ported.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-2-54ccd48e100b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
map_id_down(), map_id_up() and map_id_range_up() search the extents
of an idmapping and never modify it. Let all relevant helpers pass a
const struct uid_gid_map. Callers can now pass pass struct mnt_idmap as
const and pass down &idmap->uid_map and &idmap->gid_map.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-1-54ccd48e100b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
The ASC() macro used to define an additional sense code with no additional
sense code qualifier conflicts with the ASC macro definition of the file
arch/sparc/include/asm/ns87303.h, leading to compilation errors. Avoid this
conflict by renaming the macro as SCSI_ASC() and the ASC_DEF() macro to
SCSI_ASC_DEF() to be consistent.
Reported-by: kernel test robot <lkp@intel.com>
Fixes: 72c54bd0cbb2 ("scsi: core: Define all additional sense codes and their qualifiers")
Closes: https://lore.kernel.org/oe-kbuild-all/202609180743.kyprDg18-lkp@intel.com/
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260918062959.2245533-1-dlemoal@kernel.org
Signed-off-by: Martin K. Petersen (Oracle) <mkp@kernel.org>
|
|
Add a backend operation to enable or disable direction-specific CRC
handling. On input data paths, the backend handles CRC data generated by
the frontend. On output data paths, the backend generates CRC data for
the frontend to verify.
Reviewed-by: David Lechner <dlechner@baylibre.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@intel.com>
Signed-off-by: Janani Sunil <janani.sunil@analog.com>
Signed-off-by: Jonathan Cameron <jonathan.cameron@oss.qualcomm.com>
|
|
Add parent-child relationship between iio channels by creating a parent
pointer field in iio_chan_spec struct and exposing a sysfs attribute that
returns the parent channel prefix.
Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
Signed-off-by: Jonathan Cameron <jonathan.cameron@oss.qualcomm.com>
|
|
Add debugfs_reg64_access function pointer field into iio_info and modify
file operation callbacks to favor 64-bit variant when it is available.
Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
Signed-off-by: Jonathan Cameron <jonathan.cameron@oss.qualcomm.com>
|
|
Add IIO_FREQUENCY channel type to support Direct Digital Synthesizer (DDS)
devices that handles frequency values written directly to its buffered
interface.
Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
Signed-off-by: Jonathan Cameron <jonathan.cameron@oss.qualcomm.com>
|
|
Merge Linux 7.3-rc2 to resolve libata and mpi3mr merge conflicts.
Signed-off-by: Martin K. Petersen (Oracle) <mkp@kernel.org>
Conflicts:
drivers/ata/libata-scsi.c
drivers/scsi/mpi3mr/mpi3mr_transport.c
|
|
tpacket_snd sends skbs with frags pointing into its ring slots. Slots
are released when skb->destructor is called.
A call to skb_orphan calls skb->destructor before the skb is freed.
This can cause the slot to be reused while still linked into the skb.
Switch to standard zerocopy completion (ubuf_info) so the slot is only
released once all references to the payload are freed or copied.
Restore skb->destructor to standard sock_wfree.
The ubuf_info completion callback can be called with a NULL skb, but
only from net_zcopy_put and related API, used by zerocopy implementations
that hold their own reference on the uarg, such as MSG_ZEROCOPY. This
uarg is only ever completed from skb_zcopy_clear, so skb is always set.
To prevent userspace from aliasing in-flight state on shared ring
slots, allocate tpacket_uarg per packet, rather than per slot. This
adds a small allocation to the transmit path. Use standard kmalloc to
allow backporting to stable kernels.
The uarg holds an sk_wmem_alloc reference, rather than an sk_refcnt
reference. packet_free_tx_ring waits on sk_wmem_alloc before freeing
the ring pages. Always allocate vec->deferred for tx_ring so page-backed
rings also wait on sk_wmem_alloc when skb_copy_ubufs drops page refs
before calling tpacket_ubuf_complete.
Drop the tx_ring.pg_vec test that tpacket_destruct_skb performed before
accessing the slot. The sk_wmem_alloc reference now guarantees that the
slot is valid. The test is also not sufficient by itself, as it reads
pg_vec without pg_vec_lock, so it can race with packet_set_ring.
As a result a slot is released when its payload is copied, which can
be before transmission (e.g., in skb_orphan_frags_rx). If copied
before skb_tx_timestamp() is called, no slot timestamp is recorded,
similar to when skb_orphan() was called early in the datapath before
this patch.
Revert the now unused previous skb_zcopy_.._nouarg infra.
Depends on commit 992cc9f94ca9 ("net/packet: defer vmalloc TX_RING
free until skbs finish").
Reported-by: Katherine Leaver <kleaver@janestreet.com>
Reported-by: Bjoern Doebel <doebel@amazon.de>
Closes: https://lore.kernel.org/netdev/20260909085542.3370986-1-doebel@amazon.de/
Fixes: 5cd8d46ea156 ("packet: copy user buffers before orphan or clone")
Cc: stable@vger.kernel.org
Signed-off-by: Willem de Bruijn <willemb@google.com>
Link: https://patch.msgid.link/20260919004748.1463985-3-willemdebruijn.kernel@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
BPF programs that manage their own objects have no way to run their own
logic once an RCU grace period has elapsed. bpf_obj_drop() defers a
free, but returning an index to an allocator or unpinning a resource
once readers are done has no equivalent. sched_ext's BPF library works
around this today by pushing freed nodes onto a list and having a
userspace thread call membarrier(MEMBARRIER_CMD_GLOBAL) and then run a
BPF program to reclaim them.
Add:
int bpf_call_rcu(struct bpf_rcu_head *rh, void *map,
int (*callback)(struct bpf_map *map, void *key,
void *value));
@rh is a struct bpf_rcu_head embedded in a value of @map, so the
callback runs as callback(map, key, value) for the element it lives in
and needs no cookie. A head can only be armed once, which bounds
outstanding work by the number of elements.
struct bpf_rcu_head holds the callback state inline rather than a
pointer to it, as bpf_timer, bpf_wq and bpf_task_work do, because there
is nothing to cancel and so nothing that has to outlive the map value.
That avoids an allocation and a state machine on the arming path at the
cost of 48 bytes per element.
An RCU callback cannot be cancelled, so everything it touches has to
stay alive until it runs:
- The callback is the program's text, so arming takes a program
reference as bpf_timer, bpf_wq and bpf_task_work do, dropped once
the callback returns. bpf_prog_inc_not_zero() also fails the arm
with -EBADF once the program is dying.
- The map is held by that reference through used_maps. An inner map
is not, so bpf_rcu_head is rejected in one.
- The field is only accepted in BPF_MAP_TYPE_ARRAY, whose elements
are never freed individually. A hash element can be deleted and
recycled while a callback is queued on it.
- The head is disarmed before the callback runs so it can be armed
again from there, which takes a new program reference before the
running callback drops its own. Arming therefore fails with -EPERM
once the map is held by neither a process nor bpffs.
bpf_iter hands a program a writable pointer to the live element, which
would let it overwrite a queued head, so bpf_iter_attach_map() rejects
maps carrying one.
The callback is verified non-sleepable even when the caller is
sleepable, and RCU invokes it with BH disabled.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260922200208.3203834-2-puranjay@kernel.org
|
|
The verifier marks ALU instructions that include at least
one arena operand with needs_zext: These instructions are
fixed up after verification to be ALU32 instructions to
ensure that the result is a valid offset into an arena.
However, different code paths may provide two non-arena
64-bit arguments to the same instruction. The result of
the operation in that code path is wrong, since it is
now unexpectedly truncated to 32 bits and zero-extended.
Add logic to the verifier to ensure every instruction either
always has at least one PTR_TO_ARENA argument, or never does.
Since needs_zext already tracks the first scenario, add a
prevent_zext field in bpf_insn_aux to track the latter.
Reject instructions that use arena arguments and have prevent_zext
set, or do not have arena arguments and have needs_zext set.
Fixes: 6082b6c328b5 ("bpf: Recognize addr_space_cast instruction in the verifier.")
Reported-by: Nicholas Carlini <nicholas@carlini.com>
Suggested-by: Nicholas Carlini <nicholas@carlini.com>
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260922172028.6269-8-emil@etsalapatis.com
|
|
The skb_pointer_if_linear() function checks whether a
memory region of length len starting at offset off into
the skb is in the linear area, and returns a pointer to
the region if so. The check currently subtracts between
skb_headlen and offset of the check, and since skb_headlen
is unsigned the subtraction can underflow. This causes the
bounds check to spuriously pass and generate an arbitrary
pointer of the form *(skb->data + off).
The only user of this helper is currently skb-backed BPF
dynptr code. Returning the wrong pointer leads to the
dynptr erroneously being backed with invalid memory.
Ensure the subtraction cannot underflow, and fail the check if
it would. Use u64 arithmetic to also prevent overflow when
calculating (skb_headlen(skb) - off) since off is unsigned.
Fixes: 6f5a630d7c57 ("bpf, net: Introduce skb_pointer_if_linear().")
Reported-by: Nicholas Carlini <nicholas@carlini.com>
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Link: https://patch.msgid.link/20260922172028.6269-2-emil@etsalapatis.com
|
|
Tracepoint consumers use a ruleset ID and version to identify the
successful landlock_add_rule(2) call prefix used to create a domain.
LANDLOCK_MAX_NUM_RULES bounds distinct stored rules, not successful
calls: re-adding already-present rights for an object or port succeeds
without increasing num_rules. Because every successful call increments
the version, these calls can wrap the 32-bit counter and give different
prefixes the same trace identity.
Widen the counter and its trace fields to 64 bits so the counter cannot
wrap in practice, while preserving the successful-call semantics.
Saturating would alias all subsequent histories, while rejecting a call
at the limit would change otherwise valid syscall behavior solely for
trace metadata.
Cc: Günther Noack <gnoack@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Fixes: 63747c94774d ("landlock: Add landlock_add_rule_fs and landlock_add_rule_net tracepoints")
Reviewed-by: Günther Noack <gnoack@google.com>
Link: https://patch.msgid.link/20260922132615.1025945-1-mic@digikod.net
Signed-off-by: Mickaël Salaün <mic@digikod.net>
|
|
While the likes of acpi_dev_uid_to_integer() and acpi_dev_uid_match()
already have stubs for !ACPI, the basic acpi_device_uid() getter does
not. Although calls in portable drivers should already be conditional
on a valid acpi_device in a manner which would usually lead to compile-
time elimination anyway, add a stub for the sake of completeness.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
[ rjw: Subject tweak ]
Link: https://patch.msgid.link/8325277ff0484ab869fecf662a7ea356cd00bcde.1790090794.git.robin.murphy@arm.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
|
|
Replace acpi_get_first_physical_node() that is slated for removal
with acpi_bus_get_primary_device() that takes a reference to the
device it is about to return.
This addresses a potential use-after-free that may occur if the
device returned by acpi_get_first_physical_node() is removed right
after dropping its ACPI companion's physical_node_lock in that
function.
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Reviewed-by: Hans de Goede <johannes.goede@oss.qualcomm.com>
Link: https://patch.msgid.link/2287208.Mh6RI2rZIc@rafael.j.wysocki
|
|
Corrently acpi/acpi_bus.h is included in linux/acpi.h only if
CONFIG_ACPI is set, but it contains declarations that are useful
otherwise too.
For consistency, modify linux/acpi.h to always include acpi/acpi_bus.h
and update the latter to avoid exposing too much when CONFIG_ACPI is
not set.
Since acpi_has_method() is called from some code build when
CONFIG_ACPI is unset, add a stub for it to the !CONFIG_ACPI
block in acpi/acpi_bus.h.
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Reviewed-by: Sudeep Holla <sudeep.holla@kernel.org>
Link: https://patch.msgid.link/3717753.iIbC2pHGDl@rafael.j.wysocki
|
|
When doing in-place conversion from PRIVATE to SHARED, immediately inform
arch code of the conversion for all allocated pages/folios, e.g. so that
arch code can put hardware metadata tables in the correct state. Eagerly
updating the table for to SHARED conversions avoids having to implement
on-demand updates, e.g. when faulting in host userspace mappings. Skip the
entire flow if the arch doesn't implement conversion callbacks, as getting
folios from the filemap is noticeably expensive, especially when converting
large chunks of memory.
Deliberately don't eagerly update the metadata table on conversions from
SHARED to PRIVATE, because assigning a page to a VM (versus "returning" it
to the host) requires the exact GFN associated with the page, i.e would
require walking the memslot bindings. And because KVM *must* do on-demand
metadata updates when getting a PFN for KVM-internal usage, as that's the
only time a relevant memslot binding is guaranteed to exist.
Note! Inform arch code of the conversion within the protection of the
invalidation sequence, to ensure that any existing mappings are dropped
before hardware is updated, and to ensure that new mappings can't be
established until after the conversion is complete.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-16-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
When converting memory to private in guest_memfd, it is necessary to ensure
that the pages are not currently being accessed by any other part of the
kernel or userspace to avoid any current user writing to guest private
memory.
guest_memfd checks for any outstanding references to determine whether a
page is still in use. The only expected references after unmapping the
range requested for conversion are those that are held by guest_memfd
itself.
A folio will have outstanding references if it is present in a per-CPU
lru_add fbatch. guest_memfd does not actually participate in LRU, but
freshly-allocated folios are still added to the lru_add fbatch for batch
LRU statistics processing.
A folio may also have extra refcounts if it is on the mlock fbatch.
These two known usages of the folio are handled by draining both the
lru_add and mlock fbatches for the folio. After draining, if the refcount
is still elevated, then there are truly outstanding references.
If the page may be DMA-pinned, DMA is using it and hence there are
outstanding references. Checking if the page may be DMA-pinned can have
false positives, but that is only with a significant number of refcounts,
at which point draining LRU is not going to move the needle - it can still
be concluded that the folio has outstanding references.
If the page is still mapped after guest_memfd tried to unmap it earlier in
the conversion process, it also has outstanding references.
Exit early to avoid unnecessary draining in these two cases. If an
outstanding reference is detected, stop scanning, record the failing
offset, and return immediately.
Track the drain status to avoid repeatedly draining LRU caches across
multiple folios while scanning the requested range.
Update the kvm_memory_attributes2 structure to include an error_offset
field. This allows KVM to report the exact offset where a conversion
failed. If the safety check fails, return -EAGAIN and copy the error_offset
back to userspace so that it can potentially retry the operation or handle
the failure gracefully.
Report error_offset if preallocating memory to track attributes fails with
-ENOMEM as well, in which case the start offset of the range is returned.
Update documentation to document the error_offset field and the possible
-EAGAIN error.
Suggested-by: David Hildenbrand <david@kernel.org>
Co-developed-by: Vishal Annapurve <vannapurve@google.com>
Signed-off-by: Vishal Annapurve <vannapurve@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-15-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Add a new ioctl (and matching struct), KVM_SET_MEMORY_ATTRIBUTES2, using
the same base ioctl number (0xd2), but with R/W semantics for the kernel
instead of just read semantics. "Officially" documenting that KVM writes
to the payload will allow KVM to support partial/incremental conversions,
instead of all-or-nothing updates (which requires complex unwinding), by
recording the failing offset if an error occurs.
Opportunistically add a new struct as well, even though KVM could squeeze
the error offset into "struct kvm_memory_attributes", as there's no cost to
doing so in practice. Pad the struct with a pile of extra space to try and
avoid ending up with "struct kvm_memory_attributes3" in the future. Use
the same layout for the fields common to version 1 of the struct, e.g. to
ease upgrading userspace, and to provide flexibility if KVM ever adds
support for KVM_SET_MEMORY_ATTRIBUTES2 at VM scope.
Introduce KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES to advertise the
availability of the KVM_SET_MEMORY_ATTRIBUTES2 ioctl.
Update the KVM API documentation to define the new ioctl and its behavior,
and add the necessary UAPI definitions and capability checks.
The process of setting memory attributes has a clear point of no return
because, for CoCo VMs, zapping stage 2 page tables is a destructive
operation. Unlike regular VMs, where re-faulting pages into the stage 2
page tables merely incurs a performance penalty, CoCo guests must
(re-):accept pages after every fault. To preserve CoCo security guarantees,
guests will not accept pages they did not explicitly request faults
for. Consequently, during memory conversions, any operation that could
cause the process to abort must be completed before the stage 2 page tables
are zapped.
Zap only the ranges that are not already in the requested state to avoid
inadvertently destroying (CoCo) data. ARM CCA guests will try to mark the
entire DRAM as private at boot. If there are no shared pages at all, the
to-private conversion can be skipped, but the existence of a single shared
page would require the conversion process to proceed, and if it proceeds,
zapping both shared and private pages would destroy data and break the
guest.
Suggested-by: Michael Roth <michael.roth@amd.com>
Suggested-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Co-developed-by: Vishal Annapurve <vannapurve@google.com>
Signed-off-by: Vishal Annapurve <vannapurve@google.com>
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-14-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Move the kvm_arch_has_private_mem() stub and a few guest_memfd function
definitions/declarations "down" in kvm_host.h to utilize existing #ifdefs,
and so that related code is clustered together.
No functional change intended.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-10-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Stub in global variable to enable in-place guest_memfd private<=>shared
memory conversion, which will eventually be exposed to userspace via a
module param, and wire up the __kvm_is_private_gfn() static call to the
guest_memfd version when in-place conversion is enabled, i.e. when gmem is
the sole authority on private vs. shared memory.
Co-developed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Fuad Tabba <tabba@google.com>
Cc: Xiaoyao Li <xiaoyao.li@intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-9-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Introduce a generic kvm_is_private_gfn() interface using a static call to
determine if a GFN is private. This allows the implementation for checking
a GFN's private/shared status to be set at runtime.
In preparation for choosing implementations between a guest_memfd lookup
and the existing VM attribute lookup, rename the existing
VM-attribute-based check to kvm_vm_is_private_gfn() to emphasize that it
looks up VM attributes.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-8-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Rename kvm_mem_is_private() to kvm_is_private_gfn() to prepare for in-place
conversion, where there will be two lookup functions,
kvm_vm_is_private_gfn() and kvm_gmem_is_private_gfn().
This renaming allows consistent prefixing of "vm" vs "gmem" for
kvm_*_is_private_gfn(), as opposed to kvm_gmem_mem_is_private(), which
looks like a typo.
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-7-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Rename memory attribute APIs to add a "vm_" in the name in anticipation of
moving PRIVATE tracking into guest_memfd, to allow in-place conversion
between SHARED and PRIVATE. At that point, there will effectively be two
(potential) sources of memory attributes: the VM and guest_memfd.
kvm_vm_set_mem_attributes() already has "vm" in the name to indicate that
it is a VM ioctl. Rename it to kvm_set_vm_mem_attributes() to show that it
is setting the VM's memory attributes. (Drop the VM-ioctl scoping since it
is a helper local to the file.) Update the accompanying trace function to
match.
No functional change intended.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-6-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
defined
Explicitly guard reporting support for KVM_MEMORY_ATTRIBUTE_PRIVATE based
on kvm_arch_has_private_mem being #defined in anticipation of tracking
PRIVATE vs. SHARED state per-guest_memfd, not per-VM (to allow in-place
conversion).
guest_memfd support for memory attributes is expected to be unconditional
to avoid yet more macros (all architectures that support guest_memfd are
expected to use per-gmem attributes at some point), at which point
enumerating support KVM_MEMORY_ATTRIBUTE_PRIVATE based solely on memory
attributes being supported by KVM at-large would result in a system-scope
check (NULL @kvm) over-reporting support on arm64.
Give architectures full control over overriding the default definition of
kvm_arch_has_private_mem() by removing the coupling with
CONFIG_KVM_VM_MEMORY_ATTRIBUTES.
In a later patch, kvm_arch_has_private_mem() will be defined based on
whether architectural features are compiled in, and made orthogonal to
CONFIG_KVM_VM_MEMORY_ATTRIBUTES.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-5-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Rename the per-VM memory attributes Kconfig to make it explicitly about
per-VM attributes in anticipation of adding memory attributes support to
guest_memfd, at which point it will be possible (and desirable) to have
memory attributes without the per-VM support, even in x86.
No functional change intended.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-4-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Add and use kvm_arch_has_gmem_convert() to guard guest_memfd's invocation
of arch hooks related to converting memory between private and shared, as
only one half of the x86 CoCo duo needs the runtime hooks (any pre-work is
pure overhead for TDX). At this exact moment, the overhead is negligible,
but that will change when in-place conversion comes along, at which point
to-shared conversions will "need" to find all affected folios prior to
calling into arch code. In quotes because very technically that work could
be pushed to arch code, but that would bleed guest_memfd details into arch
code and would be far worse than adding yet another kvm_arch_has... hook.
Opportunistically provide the kvm_arch_gmem_make_private() declaration, and
rely on dead-code elimination to eliminate the call to non-existent code
when CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT=n.
Reported-by: Binbin Wu <binbin.wu@linux.intel.com>
Closes: https://lore.kernel.org/all/1ec08cd8-3072-4753-ad5e-cd34956647f8@linux.intel.com
Suggested-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-1-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
KVM currently expects guest_memfd PFN lookups to return a refcounted struct
page, which callers hold across fault handling.
Drop the page's refcount before returning from kvm_gmem_get_pfn() to
prepare for the in-place conversions series.
CoCo shared-to-private conversion handling must inspect folio refcounts to
ensure exclusive ownership by guest_memfd. A concurrent guest page fault
taking a temporary reference on the folio causes conversions to fail due to
an elevated refcount.
While this refcount is also taken on host userspace page faults, that
refcount is taken on behalf of the host userspace page tables. This
refcount will be dropped when conversions unmaps the page. Either way, once
there's an mmap() or userspace mapping, the pages are open to way more
refcounts, transient or not. This patch focuses on just dropping refcounts
before handing KVM a page.
guest_memfd already notifies KVM of page invalidations, so callers within
KVM only need to respect the MMU invalidation protocol to safely rely on
guest_memfd for page presence.
Since the page refcounts are dropped, don't return the struct page pointer.
Not returning the struct page from the guest_memfd PFN lookup moves KVM
closer toward supporting memory backends that are not backed by struct
page.
Here are some notes on the cleanup in the callers of kvm_gmem_get_pfn():
kvm_release_faultin_page() in ARM's gmem_abort() originally also serves to
set the page dirty and accessed under some conditions. The dirty and
accessed flags don't matter for guest_memfd anyway, so it is safe to just
drop the call to kvm_release_faultin_page().
For ARM's kvm_translate_vncr(), the local page pointer must be initialized
to NULL so that the shared cleanup path that releases faulted-in pages
safely no-ops for guest_memfd.
For x86, no additional changes are required in the MMU fault path because
the page fault tracking structure is zero-initialized at the start of page
fault handling, ensuring the refcounted page pointer is already NULL.
Reported-by: Yan Zhao <yan.y.zhao@intel.com>
Closes: https://lore.kernel.org/all/anZ4W9o5pTWIEgMY@yzhao56-desk.sh.intel.com/
Co-developed-by: Yan Zhao <yan.y.zhao@intel.com>
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Tested-by: Michael Roth <michael.roth@amd.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Co-developed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Link: https://patch.msgid.link/20260826-gmem-no-return-page-v4-5-3bb9c1ddb4e3@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Replace open coded krealloc() and remove superfluous member of struct
rc_map - we don't need to track both the size in bytes and entries.
Signed-off-by: Sean Young <sean@mess.org>
Acked-by: Hans Verkuil <hverkuil+cisco@kernel.org>
|
|
Jisheng Zhang <jszhang@kernel.org> says:
Fix two issues reported by sashiko.
https://sashiko.dev/#/patchset/20260901043123.5401-1-jszhang@kernel.org?part=1
Link: https://patch.msgid.link/20260909143957.9284-1-jszhang@kernel.org
|
|
Including linux/kmod.h alone results in 1.5 MB of preprocessed output, even
though it provides only a few functions and macros.
The header currently depends on:
* __printf() -> linux/compiler_attributes.h,
* ENOSYS -> linux/errno.h,
* bool -> linux/types.h.
Include only these files, reducing the preprocessed output to 10 kB.
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
Backmerging drm/drm-next into drm-misc-next to get fixes from v7.3-rc4.
Required for commit 2ab510e63197 ("drm/sched: Fix virtual runtime race").
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
|
|
Currently /proc/<pid>/coredump_filter determines what types of memory
are included in a coredump produced by <pid>. This is fairly static. The
coredump server has no easy way to configure what memory to dump even
though it can figure out all the necessary details to make an informed
decision.
Add a new COREDUMP_MEMORY_TYPES feature bit. If the coredump server
raises it the kernel will dump memory types raised in the
coredump_ack->memory_types member. Zero is valid and causes the creation
of a coredump that just includes the program headers and notes but no
memory apart from the mappings that are always dumped.
struct coredump_req gains @memory_types which is set to the default
memory types that are included in the coredump. This can be overridden by
raising bits in coredump_ack->memory_types. It also gains
@memory_types_mask which contains a bitmask of all memory types the
kernel knows about. A coredump server may only raise bits in
coredump_ack->memory_types that are raised in
coredump_req->memory_types_mask.
struct coredump_ack grows too. If COREDUMP_MEMORY_TYPES is raised in
@mask the kernel dumps the memory types set in the @memory_types mask.
Zero is valid and dumps no memory apart from the mappings that are
always dumped. A coredump server wanting to add or drop memory types
instead of outright replacing it should simply copy
coredump_req->memory_types and then mask off or raise types as needed.
@memory_types must be zero if COREDUMP_MEMORY_TYPES isn't raised.
COREDUMP_MEMORY_TYPES requires COREDUMP_KERNEL and an ack of at least
COREDUMP_ACK_SIZE_VER1 bytes.
Link: https://patch.msgid.link/20260821-work-coredump-filter-v1-1-91f9a73ef03e@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
When the coredump server raises COREDUMP_RECORDS send the coredump in
records. A record consists of a struct coredump_record_header and data.
A header and the bytes it describes go out in one iovec.
A hole is flushed through __dump_emit() like before. So zeroes still are
sent on the socket as actual data records. Making holes cheap is
COREDUMP_SPARSE's job.
Link: https://patch.msgid.link/20260820-work-coredump-sparse-v2-16-ba32dd718c51@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
A coredump with a lot of unpopulated mappings sends useless amounts of
zero data to userspace. This is nonsensical. While __dump_skip() can
seek over them when the target is a regular file a socket cannot do
this. COREDUMP_RECORDS put the zeroes in records but it didn't get rid
of them.
Add a COREDUMP_SPARSE feature bit and a COREDUMP_RECORD_ZERO record
type. A zero record is a bare header that tells userspace how many zero
bytes were skipped.
So a hole crosses the socket as one header no matter how long it is. The
coredump server can recreate this sparsely. Zero records only exist
inside a record stream. COREDUMP_SPARSE requires COREDUMP_RECORDS.
Link: https://patch.msgid.link/20260820-work-coredump-sparse-v2-14-ba32dd718c51@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Currently a coredump sent over a socket is raw data. The kernel knows
things about the data it's sending that are useful for a coredump
server. For example, it knows where the unpopulated parts of a mapping
are. We can't communicate this to userspace currently though.
Add a COREDUMP_RECORDS feature bit and a struct coredump_record_header
so userspace can negotiate that feature. Instead of a byte stream it
gets a header plus data. Reassembling the records yields the same
coredump that would have been sent without them.
The record itself is also versioned and thus extensible with the same
protocol as the ack-req sync.
A record stream ends explicitly. A COREDUMP_RECORD_END record closes it,
carries no data and reports the size of the coredump. It is only sent
once the whole coredump has been written. If it's missing the coredump
should be treated as truncated.
This just adds the infrastructure.
Link: https://patch.msgid.link/20260820-work-coredump-sparse-v2-13-ba32dd718c51@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Right now coredump state handling is messy. The binfmt->core_dump::
methods return 1 when the coredump method did anything at all which
means that a partial write counts as having dumped core. This is fine as
the state is really only used to indicate that a coredump event occurred
in the exit status of the task. That should obviously be indicated even
if the actual writeout of the coredump failed.
But it means the coredump method of the binary formats is different from
all the other coredump helpers. And there's no way to communicate to
userspace that a coredump was truncated. We'll add support for that in a
second. For now, clean this up.
Add a flag member into struct coredump_params. Let the coredump method
raise COREDUMP_STATE_STARTED. This is what coredump_finish() will end up
using to splice in the coredump bit into the exit status.
This allows us to let the return value mean success or failure and align
it with the other coredump helpers. We also start raising
COREDUMP_STATE_TRUNCATED. This will be used in the next patches to
communicate truncation to userspace via the coredump socket.
No functional changes.
Link: https://patch.msgid.link/20260820-work-coredump-sparse-v2-12-ba32dd718c51@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
The various dump helpers return one and zero. Every caller just does a
boolean test. Convert them to return an actual bool.
While at it, drop the externs.
No functional changes.
Link: https://patch.msgid.link/20260820-work-coredump-sparse-v2-10-ba32dd718c51@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
The coredump server negotiates a set of COREDUMP_* options with the
kernel. The core dump path cannot see them though.
Move the mask into struct coredump_params to make them available.
No functional change.
Link: https://patch.msgid.link/20260820-work-coredump-sparse-v2-8-ba32dd718c51@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
COREDUMP_REQ_SIZE_VER0 and COREDUMP_ACK_SIZE_VER0 define the initial
struct sizes.
Assert that both published sizes still match their structs. While at it
fix some issues with docs.
Link: https://patch.msgid.link/20260820-work-coredump-sparse-v2-7-ba32dd718c51@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
underestimation bug
The scheduler scales LLC capacity by the fraction of cache-sharing CPUs
covered by a domain:
llc_bytes = cache_size * span_weight / shared_weight
During CPU teardown, sched_cpu_deactivate() rebuilds scheduler domains
before cacheinfo_cpu_pre_down() removes the CPU from shared_cpu_map. The
new domains therefore use the old sharing weight. The later call to
sched_update_llc_bytes() looks up the departing CPU's sd_llc, which has
already been detached, and returns without correcting the surviving CPUs.
On a Ryzen 5 7535U with twelve logical CPUs sharing a 16 MiB LLC,
offlining one SMT sibling left the remaining CPUs with:
llc_bytes = floor(16777216 * 11 / 12) = 15379114 bytes
The correct capacity is still 16777216 bytes. On systems with active
cache-aware scheduling, an underestimated capacity can cause
exceed_llc_capacity() to reject aggregation for a process whose footprint
would fit. Unchanged cpuset partitions sharing the physical cache can
also retain stale capacity when a CPU comes online in another partition.
Pass the cache-sharing mask already retained by cacheinfo to the
scheduler update. Refresh every surviving CPU using its own LLC domain
so that each partition receives the correct share. This also preserves
the correction needed as cache-sharing maps grow during boot.
Keep the existing CPU-hotplug and scheduler-domain synchronization. The
update remains on the hotplug path; no steady-state scheduling operation
or persistent allocation is added.
Fixes: 7030513a0877 ("sched/cache: Calculate the LLC size and store it in sched_domain")
Signed-off-by: Davi Chaves Azevedo <davichazbh@gmail.com>
Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Chen Yu <yu.c.chen@intel.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: <stable@kernel.org> # v7.2.x
Link: https://patch.msgid.link/6751d93e15889e624796c74db0bfe66603d60b1b.1790035273.git.tim.c.chen@linux.intel.com
|
|
Add a sched_cache_grp pointer to task_struct so that scheduler code
can access the cache group directly via the task, without going
through mm->sched_cache_grp. This decouples the scheduler's hot-path
accesses from the mm_struct.
Each task holds its own refcount on the sched_cache_group, separate
from the reference held by its mm_struct. The reference is acquired
in copy_mm() (fork) and exec_mmap() (exec), and released in exit_mm().
This fixes the use-after-free when account_mm_sched() reaches the group
through a task whose mm is being switched, as reported by Hyunwoo:
https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Convert all scheduler code in fair.c and exit.c to use
p->sched_cache_grp instead of p->mm->sched_cache_grp.
Keep the fork/exec/exit reference management out of the generic mm
paths: add sched_cache_fork(), sched_cache_fork_cleanup(),
sched_cache_exec_mmap() and sched_cache_exit_mm() in
kernel/sched/cache_sched.c (with empty stubs for !CONFIG_SCHED_CACHE),
so fs/exec.c, kernel/fork.c and kernel/exit.c each call one helper
instead of open-coding the refcounting under #ifdef. Also add
sched_cache_group_get() and task_cache_group_get().
Fixes: df0d98475954 ("sched/cache: Introduce infrastructure for cache-aware load balancing")
Closes: https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Closes: https://lore.kernel.org/all/343a7e07-7fad-4979-9c9b-82ec038c293c@linux.dev/
Reported-by: Hyunwoo Kim <imv4bel@gmail.com>
Reported-by: Zenghui Yu (Huawei) <zenghui.yu@linux.dev>
Co-developed-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: <stable@kernel.org> #7.2.x
Link: https://patch.msgid.link/ae7081dc54736bf115215f9867abb2711a7403fb.1790035273.git.tim.c.chen@linux.intel.com
|
