linux-next.git/include/linux/memcontrol.h, branch akpm-base

mm: memcontrol: use obj_cgroup APIs to charge the LRU pages

2022-06-27T20:45:37+00:00

We will reuse the obj_cgroup APIs to charge the LRU pages. Finally, page->memcg_data will have 2 different meanings. - For the slab pages, page->memcg_data points to an object cgroups vector. - For the kmem pages (exclude the slab pages) and the LRU pages, page->memcg_data points to an object cgroup. In this patch, we reuse obj_cgroup APIs to charge LRU pages. In the end, The page cache cannot prevent long-living objects from pinning the original memory cgroup in the memory. At the same time we also changed the rules of page and objcg or memcg binding stability. The new rules are as follows. For a page any of the following ensures page and objcg binding stability: - the page lock - LRU isolation - lock_page_memcg() - exclusive reference Based on the stable binding of page and objcg, for a page any of the following ensures page and memcg binding stability: - objcg_lock - cgroup_mutex - the lruvec lock - the split queue lock (only THP page) If the caller only want to ensure that the page counters of memcg are updated correctly, ensure that the binding stability of page and objcg is sufficient. Link: https://lkml.kernel.org/r/20220621125658.64935-10-songmuchun@bytedance.com Signed-off-by: Muchun Song Reviewed-by: Michal Koutný Acked-by: Roman Gushchin Cc: Johannes Weiner Cc: Michal Hocko Cc: Shakeel Butt Cc: Waiman Long Cc: Xiongchun Duan Signed-off-by: Andrew Morton

mm: memcontrol: introduce memcg_reparent_ops

2022-06-27T20:45:36+00:00

In the previous patch, we know how to make the lruvec lock safe when LRU pages are reparented. We should do something like following. memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) relocate from current memcg to its parent // Move all the pages from the lruvec list to the parent lruvec list. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); Apart from the page lruvec lock, the deferred split queue lock (THP only) also needs to do something similar. So we extract the necessary three steps in the memcg_reparent_objcgs(). memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) relocate memcg_reparent_ops->relocate(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Now there are two different locks (e.g. lruvec lock and deferred split queue lock) need to use this infrastructure. In the next patch, we will use those APIs to make those locks safe when the LRU pages reparented. Link: https://lkml.kernel.org/r/20220621125658.64935-9-songmuchun@bytedance.com Signed-off-by: Muchun Song Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutný Cc: Roman Gushchin Cc: Shakeel Butt Cc: Waiman Long Cc: Xiongchun Duan Signed-off-by: Andrew Morton

mm: memcontrol: make all the callers of {folio,page}_memcg() safe

2022-06-27T20:45:36+00:00

When we use objcg APIs to charge the LRU pages, the page will not hold a reference to the memcg associated with the page. So the caller of the {folio,page}_memcg() should hold an rcu read lock or obtain a reference to the memcg associated with the page to protect memcg from being released. So introduce get_mem_cgroup_from_{page,folio}() to obtain a reference to the memory cgroup associated with the page. In this patch, make all the callers hold an rcu read lock or obtain a reference to the memcg to protect memcg from being released when the LRU pages reparented. We do not need to adjust the callers of {folio,page}_memcg() during the whole process of mem_cgroup_move_task(). Because the cgroup migration and memory cgroup offlining are serialized by @cgroup_mutex. In this routine, the LRU pages cannot be reparented to its parent memory cgroup. So {folio,page}_memcg() is stable and cannot be released. This is a preparation for reparenting the LRU pages. Link: https://lkml.kernel.org/r/20220621125658.64935-8-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Roman Gushchin Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutný Cc: Shakeel Butt Cc: Waiman Long Cc: Xiongchun Duan Signed-off-by: Andrew Morton

mm: thp: make split queue lock safe when LRU pages are reparented

2022-06-27T20:45:36+00:00

Similar to the lruvec lock, we use the same approach to make the split queue lock safe when LRU pages are reparented. Link: https://lkml.kernel.org/r/20220621125658.64935-7-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Roman Gushchin Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutný Cc: Shakeel Butt Cc: Waiman Long Cc: Xiongchun Duan Signed-off-by: Andrew Morton

mm: memcontrol: make lruvec lock safe when LRU pages are reparented

2022-06-27T20:45:36+00:00

The diagram below shows how to make the folio lruvec lock safe when LRU pages are reparented. folio_lruvec_lock(folio) rcu_read_lock(); retry: lruvec = folio_lruvec(folio); // The folio is reparented at this time. spin_lock(&lruvec->lru_lock); if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) // Acquired the wrong lruvec lock and need to retry. // Because this folio is on the parent memcg lruvec list. spin_unlock(&lruvec->lru_lock); goto retry; // If we reach here, it means that folio_memcg(folio) is stable. memcg_reparent_objcgs(memcg) // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); // Move all the pages from the lruvec list to the parent lruvec list. spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); After we acquire the lruvec lock, we need to check whether the folio is reparented. If so, we need to reacquire the new lruvec lock. On the routine of the LRU pages reparenting, we will also acquire the lruvec lock (will be implemented in the later patch). So folio_memcg() cannot be changed when we hold the lruvec lock. Since lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after we hold the lruvec lock, lruvec_memcg_debug() check is pointless. So remove it. This is a preparation for reparenting the LRU pages. Link: https://lkml.kernel.org/r/20220621125658.64935-5-songmuchun@bytedance.com Signed-off-by: Muchun Song Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutný Cc: Roman Gushchin Cc: Shakeel Butt Cc: Waiman Long Cc: Xiongchun Duan Signed-off-by: Andrew Morton

mm: memcontrol: prepare objcg API for non-kmem usage

2022-06-27T20:45:35+00:00

Pagecache pages are charged at the allocation time and holding a reference to the original memory cgroup until being reclaimed. Depending on the memory pressure, specific patterns of the page sharing between different cgroups and the cgroup creation and destruction rates, a large number of dying memory cgroups can be pinned by pagecache pages. It makes the page reclaim less efficient and wastes memory. We can convert LRU pages and most other raw memcg pins to the objcg direction to fix this problem, and then the page->memcg will always point to an object cgroup pointer. Therefore, the infrastructure of objcg no longer only serves CONFIG_MEMCG_KMEM. In this patch, we move the infrastructure of the objcg out of the scope of the CONFIG_MEMCG_KMEM so that the LRU pages can reuse it to charge pages. We know that the LRU pages are not accounted at the root level. But the page->memcg_data points to the root_mem_cgroup. So the page->memcg_data of the LRU pages always points to a valid pointer. But the root_mem_cgroup dose not have an object cgroup. If we use obj_cgroup APIs to charge the LRU pages, we should set the page->memcg_data to a root object cgroup. So we also allocate an object cgroup for the root_mem_cgroup. Link: https://lkml.kernel.org/r/20220621125658.64935-4-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Johannes Weiner Reviewed-by: Michal Koutný Acked-by: Roman Gushchin Cc: Michal Hocko Cc: Shakeel Butt Cc: Waiman Long Cc: Xiongchun Duan Signed-off-by: Andrew Morton

mm: rename unlock_page_lruvec{_irq, _irqrestore} to lruvec_unlock{_irq, _irqrestore}

2022-06-27T20:45:35+00:00

It is weird to use folio_lruvec_lock() variants and unlock_page_lruvec() variants together, e.g. locking folio and unlocking page. So rename unlock_page_lruvec{_irq, _irqrestore} to lruvec_unlock{_irq, _irqrestore}. Link: https://lkml.kernel.org/r/20220621125658.64935-3-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Roman Gushchin Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutný Cc: Shakeel Butt Cc: Waiman Long Cc: Xiongchun Duan Signed-off-by: Andrew Morton

mm: memcontrol: remove dead code and comments

2022-06-27T20:45:35+00:00

Patch series "Use obj_cgroup APIs to charge the LRU pages", v6. With the following patchsets applied, all the kernel memory is charged with the new APIs of obj_cgroup: commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects instead of pages") commit b4e0b68fbd9d ("mm: memcontrol: use obj_cgroup APIs to charge kmem pages") But user memory allocations (LRU pages) pinning memcgs for a long time - it exists at a larger scale and is causing recurring problems in the real world: page cache doesn't get reclaimed for a long time, or is used by the second, third, fourth, ... instance of the same job that was restarted into a new cgroup every time. Unreclaimable dying cgroups pile up, waste memory, and make page reclaim very inefficient. We can convert LRU pages and most other raw memcg pins to the objcg direction to fix this problem, and then the LRU pages will not pin the memcgs. This patchset aims to make the LRU pages to drop the reference to memory cgroup by using the APIs of obj_cgroup. Finally, we can see that the number of the dying cgroups will not increase if we run the following test script. #!/bin/bash dd if=/dev/zero of=temp bs=4096 count=1 cat /proc/cgroups | grep memory for i in {0..2000} do mkdir /sys/fs/cgroup/memory/test$i echo $$ > /sys/fs/cgroup/memory/test$i/cgroup.procs cat temp >> log echo $$ > /sys/fs/cgroup/memory/cgroup.procs rmdir /sys/fs/cgroup/memory/test$i done cat /proc/cgroups | grep memory rm -f temp log This patch (of 11): Since no-hierarchy mode is deprecated after commit bef8620cd8e0 ("mm: memcg: deprecate the non-hierarchical mode") so parent_mem_cgroup() cannot return a NULL except root memcg, however, root memcg cannot be offline, so it is safe to drop the check of returned value of parent_mem_cgroup(). Remove those dead code. The comments in memcg_offline_kmem() above memcg_reparent_list_lrus() are out of date since commit 5abc1e37afa0 ("mm: list_lru: allocate list_lru_one only when needed") There is no ordering requirement between memcg_reparent_list_lrus() and memcg_reparent_objcgs(), so remove those outdated comments. Link: https://lkml.kernel.org/r/20220621125658.64935-1-songmuchun@bytedance.com Link: https://lkml.kernel.org/r/20220621125658.64935-2-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Roman Gushchin Cc: Johannes Weiner Cc: Waiman Long Cc: Michal Hocko Cc: Shakeel Butt Cc: Xiongchun Duan Cc: Michal Koutný Signed-off-by: Andrew Morton

mm: memcontrol: introduce mem_cgroup_ino() and mem_cgroup_get_from_ino()

2022-06-27T20:45:17+00:00

Patch series "mm: introduce shrinker debugfs interface", v5. The only existing debugging mechanism is a couple of tracepoints in do_shrink_slab(): mm_shrink_slab_start and mm_shrink_slab_end. They aren't covering everything though: shrinkers which report 0 objects will never show up, there is no support for memcg-aware shrinkers. Shrinkers are identified by their scan function, which is not always enough (e.g. hard to guess which super block's shrinker it is having only "super_cache_scan"). To provide a better visibility and debug options for memory shrinkers this patchset introduces a /sys/kernel/debug/shrinker interface, to some extent similar to /sys/kernel/slab. For each shrinker registered in the system a directory is created. As now, the directory will contain only a "scan" file, which allows to get the number of managed objects for each memory cgroup (for memcg-aware shrinkers) and each numa node (for numa-aware shrinkers on a numa machine). Other interfaces might be added in the future. To make debugging more pleasant, the patchset also names all shrinkers, so that debugfs entries can have meaningful names. This patch (of 5): Shrinker debugfs requires a way to represent memory cgroups without using full paths, both for displaying information and getting input from a user. Cgroup inode number is a perfect way, already used by bpf. This commit adds a couple of helper functions which will be used to handle memcg-aware shrinkers. Link: https://lkml.kernel.org/r/20220601032227.4076670-1-roman.gushchin@linux.dev Link: https://lkml.kernel.org/r/20220601032227.4076670-2-roman.gushchin@linux.dev Signed-off-by: Roman Gushchin Acked-by: Muchun Song Cc: Dave Chinner Cc: Kent Overstreet Cc: Hillf Danton Cc: Christophe JAILLET Cc: Roman Gushchin Signed-off-by: Andrew Morton

net: set proper memcg for net_init hooks allocations

2022-06-17T02:48:31+00:00

__register_pernet_operations() executes init hook of registered pernet_operation structure in all existing net namespaces. Typically, these hooks are called by a process associated with the specified net namespace, and all __GFP_ACCOUNT marked allocation are accounted for corresponding container/memcg. However __register_pernet_operations() calls the hooks in the same context, and as a result all marked allocations are accounted to one memcg for all processed net namespaces. This patch adjusts active memcg for each net namespace and helps to account memory allocated inside ops_init() into the proper memcg. Link: https://lkml.kernel.org/r/f9394752-e272-9bf9-645f-a18c56d1c4ec@openvz.org Signed-off-by: Vasily Averin Acked-by: Roman Gushchin Acked-by: Shakeel Butt Cc: Michal Koutný Cc: Vlastimil Babka Cc: Michal Hocko Cc: Florian Westphal Cc: David S. Miller Cc: Jakub Kicinski Cc: Paolo Abeni Cc: Eric Dumazet Cc: Johannes Weiner Cc: Kefeng Wang Cc: Linux Kernel Functional Testing Cc: Muchun Song Cc: Naresh Kamboju Cc: Qian Cai Signed-off-by: Andrew Morton