diff options
Diffstat (limited to 'Documentation/x86/resctrl.rst')
| -rw-r--r-- | Documentation/x86/resctrl.rst | 1447 |
1 files changed, 0 insertions, 1447 deletions
diff --git a/Documentation/x86/resctrl.rst b/Documentation/x86/resctrl.rst deleted file mode 100644 index 387ccbcb558f..000000000000 --- a/Documentation/x86/resctrl.rst +++ /dev/null @@ -1,1447 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 -.. include:: <isonum.txt> - -=========================================== -User Interface for Resource Control feature -=========================================== - -:Copyright: |copy| 2016 Intel Corporation -:Authors: - Fenghua Yu <fenghua.yu@intel.com> - - Tony Luck <tony.luck@intel.com> - - Vikas Shivappa <vikas.shivappa@intel.com> - - -Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT). -AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). - -This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo -flag bits: - -=============================================== ================================ -RDT (Resource Director Technology) Allocation "rdt_a" -CAT (Cache Allocation Technology) "cat_l3", "cat_l2" -CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2" -CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc" -MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local" -MBA (Memory Bandwidth Allocation) "mba" -SMBA (Slow Memory Bandwidth Allocation) "" -BMEC (Bandwidth Monitoring Event Configuration) "" -=============================================== ================================ - -Historically, new features were made visible by default in /proc/cpuinfo. This -resulted in the feature flags becoming hard to parse by humans. Adding a new -flag to /proc/cpuinfo should be avoided if user space can obtain information -about the feature from resctrl's info directory. - -To use the feature mount the file system:: - - # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps]] /sys/fs/resctrl - -mount options are: - -"cdp": - Enable code/data prioritization in L3 cache allocations. -"cdpl2": - Enable code/data prioritization in L2 cache allocations. -"mba_MBps": - Enable the MBA Software Controller(mba_sc) to specify MBA - bandwidth in MBps - -L2 and L3 CDP are controlled separately. - -RDT features are orthogonal. A particular system may support only -monitoring, only control, or both monitoring and control. Cache -pseudo-locking is a unique way of using cache control to "pin" or -"lock" data in the cache. Details can be found in -"Cache Pseudo-Locking". - - -The mount succeeds if either of allocation or monitoring is present, but -only those files and directories supported by the system will be created. -For more details on the behavior of the interface during monitoring -and allocation, see the "Resource alloc and monitor groups" section. - -Info directory -============== - -The 'info' directory contains information about the enabled -resources. Each resource has its own subdirectory. The subdirectory -names reflect the resource names. - -Each subdirectory contains the following files with respect to -allocation: - -Cache resource(L3/L2) subdirectory contains the following files -related to allocation: - -"num_closids": - The number of CLOSIDs which are valid for this - resource. The kernel uses the smallest number of - CLOSIDs of all enabled resources as limit. -"cbm_mask": - The bitmask which is valid for this resource. - This mask is equivalent to 100%. -"min_cbm_bits": - The minimum number of consecutive bits which - must be set when writing a mask. - -"shareable_bits": - Bitmask of shareable resource with other executing - entities (e.g. I/O). User can use this when - setting up exclusive cache partitions. Note that - some platforms support devices that have their - own settings for cache use which can over-ride - these bits. -"bit_usage": - Annotated capacity bitmasks showing how all - instances of the resource are used. The legend is: - - "0": - Corresponding region is unused. When the system's - resources have been allocated and a "0" is found - in "bit_usage" it is a sign that resources are - wasted. - - "H": - Corresponding region is used by hardware only - but available for software use. If a resource - has bits set in "shareable_bits" but not all - of these bits appear in the resource groups' - schematas then the bits appearing in - "shareable_bits" but no resource group will - be marked as "H". - "X": - Corresponding region is available for sharing and - used by hardware and software. These are the - bits that appear in "shareable_bits" as - well as a resource group's allocation. - "S": - Corresponding region is used by software - and available for sharing. - "E": - Corresponding region is used exclusively by - one resource group. No sharing allowed. - "P": - Corresponding region is pseudo-locked. No - sharing allowed. - -Memory bandwidth(MB) subdirectory contains the following files -with respect to allocation: - -"min_bandwidth": - The minimum memory bandwidth percentage which - user can request. - -"bandwidth_gran": - The granularity in which the memory bandwidth - percentage is allocated. The allocated - b/w percentage is rounded off to the next - control step available on the hardware. The - available bandwidth control steps are: - min_bandwidth + N * bandwidth_gran. - -"delay_linear": - Indicates if the delay scale is linear or - non-linear. This field is purely informational - only. - -"thread_throttle_mode": - Indicator on Intel systems of how tasks running on threads - of a physical core are throttled in cases where they - request different memory bandwidth percentages: - - "max": - the smallest percentage is applied - to all threads - "per-thread": - bandwidth percentages are directly applied to - the threads running on the core - -If RDT monitoring is available there will be an "L3_MON" directory -with the following files: - -"num_rmids": - The number of RMIDs available. This is the - upper bound for how many "CTRL_MON" + "MON" - groups can be created. - -"mon_features": - Lists the monitoring events if - monitoring is enabled for the resource. - Example:: - - # cat /sys/fs/resctrl/info/L3_MON/mon_features - llc_occupancy - mbm_total_bytes - mbm_local_bytes - - If the system supports Bandwidth Monitoring Event - Configuration (BMEC), then the bandwidth events will - be configurable. The output will be:: - - # cat /sys/fs/resctrl/info/L3_MON/mon_features - llc_occupancy - mbm_total_bytes - mbm_total_bytes_config - mbm_local_bytes - mbm_local_bytes_config - -"mbm_total_bytes_config", "mbm_local_bytes_config": - Read/write files containing the configuration for the mbm_total_bytes - and mbm_local_bytes events, respectively, when the Bandwidth - Monitoring Event Configuration (BMEC) feature is supported. - The event configuration settings are domain specific and affect - all the CPUs in the domain. When either event configuration is - changed, the bandwidth counters for all RMIDs of both events - (mbm_total_bytes as well as mbm_local_bytes) are cleared for that - domain. The next read for every RMID will report "Unavailable" - and subsequent reads will report the valid value. - - Following are the types of events supported: - - ==== ======================================================== - Bits Description - ==== ======================================================== - 6 Dirty Victims from the QOS domain to all types of memory - 5 Reads to slow memory in the non-local NUMA domain - 4 Reads to slow memory in the local NUMA domain - 3 Non-temporal writes to non-local NUMA domain - 2 Non-temporal writes to local NUMA domain - 1 Reads to memory in the non-local NUMA domain - 0 Reads to memory in the local NUMA domain - ==== ======================================================== - - By default, the mbm_total_bytes configuration is set to 0x7f to count - all the event types and the mbm_local_bytes configuration is set to - 0x15 to count all the local memory events. - - Examples: - - * To view the current configuration:: - :: - - # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config - 0=0x7f;1=0x7f;2=0x7f;3=0x7f - - # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config - 0=0x15;1=0x15;3=0x15;4=0x15 - - * To change the mbm_total_bytes to count only reads on domain 0, - the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary - (in hexadecimal 0x33): - :: - - # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config - - # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config - 0=0x33;1=0x7f;2=0x7f;3=0x7f - - * To change the mbm_local_bytes to count all the slow memory reads on - domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b - in binary (in hexadecimal 0x30): - :: - - # echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config - - # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config - 0=0x30;1=0x30;3=0x15;4=0x15 - -"max_threshold_occupancy": - Read/write file provides the largest value (in - bytes) at which a previously used LLC_occupancy - counter can be considered for re-use. - -Finally, in the top level of the "info" directory there is a file -named "last_cmd_status". This is reset with every "command" issued -via the file system (making new directories or writing to any of the -control files). If the command was successful, it will read as "ok". -If the command failed, it will provide more information that can be -conveyed in the error returns from file operations. E.g. -:: - - # echo L3:0=f7 > schemata - bash: echo: write error: Invalid argument - # cat info/last_cmd_status - mask f7 has non-consecutive 1-bits - -Resource alloc and monitor groups -================================= - -Resource groups are represented as directories in the resctrl file -system. The default group is the root directory which, immediately -after mounting, owns all the tasks and cpus in the system and can make -full use of all resources. - -On a system with RDT control features additional directories can be -created in the root directory that specify different amounts of each -resource (see "schemata" below). The root and these additional top level -directories are referred to as "CTRL_MON" groups below. - -On a system with RDT monitoring the root directory and other top level -directories contain a directory named "mon_groups" in which additional -directories can be created to monitor subsets of tasks in the CTRL_MON -group that is their ancestor. These are called "MON" groups in the rest -of this document. - -Removing a directory will move all tasks and cpus owned by the group it -represents to the parent. Removing one of the created CTRL_MON groups -will automatically remove all MON groups below it. - -All groups contain the following files: - -"tasks": - Reading this file shows the list of all tasks that belong to - this group. Writing a task id to the file will add a task to the - group. If the group is a CTRL_MON group the task is removed from - whichever previous CTRL_MON group owned the task and also from - any MON group that owned the task. If the group is a MON group, - then the task must already belong to the CTRL_MON parent of this - group. The task is removed from any previous MON group. - - -"cpus": - Reading this file shows a bitmask of the logical CPUs owned by - this group. Writing a mask to this file will add and remove - CPUs to/from this group. As with the tasks file a hierarchy is - maintained where MON groups may only include CPUs owned by the - parent CTRL_MON group. - When the resource group is in pseudo-locked mode this file will - only be readable, reflecting the CPUs associated with the - pseudo-locked region. - - -"cpus_list": - Just like "cpus", only using ranges of CPUs instead of bitmasks. - - -When control is enabled all CTRL_MON groups will also contain: - -"schemata": - A list of all the resources available to this group. - Each resource has its own line and format - see below for details. - -"size": - Mirrors the display of the "schemata" file to display the size in - bytes of each allocation instead of the bits representing the - allocation. - -"mode": - The "mode" of the resource group dictates the sharing of its - allocations. A "shareable" resource group allows sharing of its - allocations while an "exclusive" resource group does not. A - cache pseudo-locked region is created by first writing - "pseudo-locksetup" to the "mode" file before writing the cache - pseudo-locked region's schemata to the resource group's "schemata" - file. On successful pseudo-locked region creation the mode will - automatically change to "pseudo-locked". - -When monitoring is enabled all MON groups will also contain: - -"mon_data": - This contains a set of files organized by L3 domain and by - RDT event. E.g. on a system with two L3 domains there will - be subdirectories "mon_L3_00" and "mon_L3_01". Each of these - directories have one file per event (e.g. "llc_occupancy", - "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these - files provide a read out of the current value of the event for - all tasks in the group. In CTRL_MON groups these files provide - the sum for all tasks in the CTRL_MON group and all tasks in - MON groups. Please see example section for more details on usage. - -Resource allocation rules -------------------------- - -When a task is running the following rules define which resources are -available to it: - -1) If the task is a member of a non-default group, then the schemata - for that group is used. - -2) Else if the task belongs to the default group, but is running on a - CPU that is assigned to some specific group, then the schemata for the - CPU's group is used. - -3) Otherwise the schemata for the default group is used. - -Resource monitoring rules -------------------------- -1) If a task is a member of a MON group, or non-default CTRL_MON group - then RDT events for the task will be reported in that group. - -2) If a task is a member of the default CTRL_MON group, but is running - on a CPU that is assigned to some specific group, then the RDT events - for the task will be reported in that group. - -3) Otherwise RDT events for the task will be reported in the root level - "mon_data" group. - - -Notes on cache occupancy monitoring and control -=============================================== -When moving a task from one group to another you should remember that -this only affects *new* cache allocations by the task. E.g. you may have -a task in a monitor group showing 3 MB of cache occupancy. If you move -to a new group and immediately check the occupancy of the old and new -groups you will likely see that the old group is still showing 3 MB and -the new group zero. When the task accesses locations still in cache from -before the move, the h/w does not update any counters. On a busy system -you will likely see the occupancy in the old group go down as cache lines -are evicted and re-used while the occupancy in the new group rises as -the task accesses memory and loads into the cache are counted based on -membership in the new group. - -The same applies to cache allocation control. Moving a task to a group -with a smaller cache partition will not evict any cache lines. The -process may continue to use them from the old partition. - -Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID) -to identify a control group and a monitoring group respectively. Each of -the resource groups are mapped to these IDs based on the kind of group. The -number of CLOSid and RMID are limited by the hardware and hence the creation of -a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID -and creation of "MON" group may fail if we run out of RMIDs. - -max_threshold_occupancy - generic concepts ------------------------------------------- - -Note that an RMID once freed may not be immediately available for use as -the RMID is still tagged the cache lines of the previous user of RMID. -Hence such RMIDs are placed on limbo list and checked back if the cache -occupancy has gone down. If there is a time when system has a lot of -limbo RMIDs but which are not ready to be used, user may see an -EBUSY -during mkdir. - -max_threshold_occupancy is a user configurable value to determine the -occupancy at which an RMID can be freed. - -Schemata files - general concepts ---------------------------------- -Each line in the file describes one resource. The line starts with -the name of the resource, followed by specific values to be applied -in each of the instances of that resource on the system. - -Cache IDs ---------- -On current generation systems there is one L3 cache per socket and L2 -caches are generally just shared by the hyperthreads on a core, but this -isn't an architectural requirement. We could have multiple separate L3 -caches on a socket, multiple cores could share an L2 cache. So instead -of using "socket" or "core" to define the set of logical cpus sharing -a resource we use a "Cache ID". At a given cache level this will be a -unique number across the whole system (but it isn't guaranteed to be a -contiguous sequence, there may be gaps). To find the ID for each logical -CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id - -Cache Bit Masks (CBM) ---------------------- -For cache resources we describe the portion of the cache that is available -for allocation using a bitmask. The maximum value of the mask is defined -by each cpu model (and may be different for different cache levels). It -is found using CPUID, but is also provided in the "info" directory of -the resctrl file system in "info/{resource}/cbm_mask". Intel hardware -requires that these masks have all the '1' bits in a contiguous block. So -0x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9 -and 0xA are not. On a system with a 20-bit mask each bit represents 5% -of the capacity of the cache. You could partition the cache into four -equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000. - -Memory bandwidth Allocation and monitoring -========================================== - -For Memory bandwidth resource, by default the user controls the resource -by indicating the percentage of total memory bandwidth. - -The minimum bandwidth percentage value for each cpu model is predefined -and can be looked up through "info/MB/min_bandwidth". The bandwidth -granularity that is allocated is also dependent on the cpu model and can -be looked up at "info/MB/bandwidth_gran". The available bandwidth -control steps are: min_bw + N * bw_gran. Intermediate values are rounded -to the next control step available on the hardware. - -The bandwidth throttling is a core specific mechanism on some of Intel -SKUs. Using a high bandwidth and a low bandwidth setting on two threads -sharing a core may result in both threads being throttled to use the -low bandwidth (see "thread_throttle_mode"). - -The fact that Memory bandwidth allocation(MBA) may be a core -specific mechanism where as memory bandwidth monitoring(MBM) is done at -the package level may lead to confusion when users try to apply control -via the MBA and then monitor the bandwidth to see if the controls are -effective. Below are such scenarios: - -1. User may *not* see increase in actual bandwidth when percentage - values are increased: - -This can occur when aggregate L2 external bandwidth is more than L3 -external bandwidth. Consider an SKL SKU with 24 cores on a package and -where L2 external is 10GBps (hence aggregate L2 external bandwidth is -240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20 -threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3 -bandwidth of 100GBps although the percentage value specified is only 50% -<< 100%. Hence increasing the bandwidth percentage will not yield any -more bandwidth. This is because although the L2 external bandwidth still -has capacity, the L3 external bandwidth is fully used. Also note that -this would be dependent on number of cores the benchmark is run on. - -2. Same bandwidth percentage may mean different actual bandwidth - depending on # of threads: - -For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4 -thread, with 10% bandwidth' can consume upto 10GBps and 40GBps although -they have same percentage bandwidth of 10%. This is simply because as -threads start using more cores in an rdtgroup, the actual bandwidth may -increase or vary although user specified bandwidth percentage is same. - -In order to mitigate this and make the interface more user friendly, -resctrl added support for specifying the bandwidth in MBps as well. The -kernel underneath would use a software feedback mechanism or a "Software -Controller(mba_sc)" which reads the actual bandwidth using MBM counters -and adjust the memory bandwidth percentages to ensure:: - - "actual bandwidth < user specified bandwidth". - -By default, the schemata would take the bandwidth percentage values -where as user can switch to the "MBA software controller" mode using -a mount option 'mba_MBps'. The schemata format is specified in the below -sections. - -L3 schemata file details (code and data prioritization disabled) ----------------------------------------------------------------- -With CDP disabled the L3 schemata format is:: - - L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... - -L3 schemata file details (CDP enabled via mount option to resctrl) ------------------------------------------------------------------- -When CDP is enabled L3 control is split into two separate resources -so you can specify independent masks for code and data like this:: - - L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... - L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... - -L2 schemata file details ------------------------- -CDP is supported at L2 using the 'cdpl2' mount option. The schemata -format is either:: - - L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... - -or - - L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... - L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... - - -Memory bandwidth Allocation (default mode) ------------------------------------------- - -Memory b/w domain is L3 cache. -:: - - MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... - -Memory bandwidth Allocation specified in MBps ---------------------------------------------- - -Memory bandwidth domain is L3 cache. -:: - - MB:<cache_id0>=bw_MBps0;<cache_id1>=bw_MBps1;... - -Slow Memory Bandwidth Allocation (SMBA) ---------------------------------------- -AMD hardware supports Slow Memory Bandwidth Allocation (SMBA). -CXL.memory is the only supported "slow" memory device. With the -support of SMBA, the hardware enables bandwidth allocation on -the slow memory devices. If there are multiple such devices in -the system, the throttling logic groups all the slow sources -together and applies the limit on them as a whole. - -The presence of SMBA (with CXL.memory) is independent of slow memory -devices presence. If there are no such devices on the system, then -configuring SMBA will have no impact on the performance of the system. - -The bandwidth domain for slow memory is L3 cache. Its schemata file -is formatted as: -:: - - SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... - -Reading/writing the schemata file ---------------------------------- -Reading the schemata file will show the state of all resources -on all domains. When writing you only need to specify those values -which you wish to change. E.g. -:: - - # cat schemata - L3DATA:0=fffff;1=fffff;2=fffff;3=fffff - L3CODE:0=fffff;1=fffff;2=fffff;3=fffff - # echo "L3DATA:2=3c0;" > schemata - # cat schemata - L3DATA:0=fffff;1=fffff;2=3c0;3=fffff - L3CODE:0=fffff;1=fffff;2=fffff;3=fffff - -Reading/writing the schemata file (on AMD systems) --------------------------------------------------- -Reading the schemata file will show the current bandwidth limit on all -domains. The allocated resources are in multiples of one eighth GB/s. -When writing to the file, you need to specify what cache id you wish to -configure the bandwidth limit. - -For example, to allocate 2GB/s limit on the first cache id: - -:: - - # cat schemata - MB:0=2048;1=2048;2=2048;3=2048 - L3:0=ffff;1=ffff;2=ffff;3=ffff - - # echo "MB:1=16" > schemata - # cat schemata - MB:0=2048;1= 16;2=2048;3=2048 - L3:0=ffff;1=ffff;2=ffff;3=ffff - -Reading/writing the schemata file (on AMD systems) with SMBA feature --------------------------------------------------------------------- -Reading and writing the schemata file is the same as without SMBA in -above section. - -For example, to allocate 8GB/s limit on the first cache id: - -:: - - # cat schemata - SMBA:0=2048;1=2048;2=2048;3=2048 - MB:0=2048;1=2048;2=2048;3=2048 - L3:0=ffff;1=ffff;2=ffff;3=ffff - - # echo "SMBA:1=64" > schemata - # cat schemata - SMBA:0=2048;1= 64;2=2048;3=2048 - MB:0=2048;1=2048;2=2048;3=2048 - L3:0=ffff;1=ffff;2=ffff;3=ffff - -Cache Pseudo-Locking -==================== -CAT enables a user to specify the amount of cache space that an -application can fill. Cache pseudo-locking builds on the fact that a -CPU can still read and write data pre-allocated outside its current -allocated area on a cache hit. With cache pseudo-locking, data can be -preloaded into a reserved portion of cache that no application can -fill, and from that point on will only serve cache hits. The cache -pseudo-locked memory is made accessible to user space where an -application can map it into its virtual address space and thus have -a region of memory with reduced average read latency. - -The creation of a cache pseudo-locked region is triggered by a request -from the user to do so that is accompanied by a schemata of the region -to be pseudo-locked. The cache pseudo-locked region is created as follows: - -- Create a CAT allocation CLOSNEW with a CBM matching the schemata - from the user of the cache region that will contain the pseudo-locked - memory. This region must not overlap with any current CAT allocation/CLOS - on the system and no future overlap with this cache region is allowed - while the pseudo-locked region exists. -- Create a contiguous region of memory of the same size as the cache - region. -- Flush the cache, disable hardware prefetchers, disable preemption. -- Make CLOSNEW the active CLOS and touch the allocated memory to load - it into the cache. -- Set the previous CLOS as active. -- At this point the closid CLOSNEW can be released - the cache - pseudo-locked region is protected as long as its CBM does not appear in - any CAT allocation. Even though the cache pseudo-locked region will from - this point on not appear in any CBM of any CLOS an application running with - any CLOS will be able to access the memory in the pseudo-locked region since - the region continues to serve cache hits. -- The contiguous region of memory loaded into the cache is exposed to - user-space as a character device. - -Cache pseudo-locking increases the probability that data will remain -in the cache via carefully configuring the CAT feature and controlling -application behavior. There is no guarantee that data is placed in -cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict -“locked” data from cache. Power management C-states may shrink or -power off cache. Deeper C-states will automatically be restricted on -pseudo-locked region creation. - -It is required that an application using a pseudo-locked region runs -with affinity to the cores (or a subset of the cores) associated -with the cache on which the pseudo-locked region resides. A sanity check -within the code will not allow an application to map pseudo-locked memory -unless it runs with affinity to cores associated with the cache on which the -pseudo-locked region resides. The sanity check is only done during the -initial mmap() handling, there is no enforcement afterwards and the -application self needs to ensure it remains affine to the correct cores. - -Pseudo-locking is accomplished in two stages: - -1) During the first stage the system administrator allocates a portion - of cache that should be dedicated to pseudo-locking. At this time an - equivalent portion of memory is allocated, loaded into allocated - cache portion, and exposed as a character device. -2) During the second stage a user-space application maps (mmap()) the - pseudo-locked memory into its address space. - -Cache Pseudo-Locking Interface ------------------------------- -A pseudo-locked region is created using the resctrl interface as follows: - -1) Create a new resource group by creating a new directory in /sys/fs/resctrl. -2) Change the new resource group's mode to "pseudo-locksetup" by writing - "pseudo-locksetup" to the "mode" file. -3) Write the schemata of the pseudo-locked region to the "schemata" file. All - bits within the schemata should be "unused" according to the "bit_usage" - file. - -On successful pseudo-locked region creation the "mode" file will contain -"pseudo-locked" and a new character device with the same name as the resource -group will exist in /dev/pseudo_lock. This character device can be mmap()'ed -by user space in order to obtain access to the pseudo-locked memory region. - -An example of cache pseudo-locked region creation and usage can be found below. - -Cache Pseudo-Locking Debugging Interface ----------------------------------------- -The pseudo-locking debugging interface is enabled by default (if -CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl. - -There is no explicit way for the kernel to test if a provided memory -location is present in the cache. The pseudo-locking debugging interface uses -the tracing infrastructure to provide two ways to measure cache residency of -the pseudo-locked region: - -1) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data - from these measurements are best visualized using a hist trigger (see - example below). In this test the pseudo-locked region is traversed at - a stride of 32 bytes while hardware prefetchers and preemption - are disabled. This also provides a substitute visualization of cache - hits and misses. -2) Cache hit and miss measurements using model specific precision counters if - available. Depending on the levels of cache on the system the pseudo_lock_l2 - and pseudo_lock_l3 tracepoints are available. - -When a pseudo-locked region is created a new debugfs directory is created for -it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single -write-only file, pseudo_lock_measure, is present in this directory. The -measurement of the pseudo-locked region depends on the number written to this -debugfs file: - -1: - writing "1" to the pseudo_lock_measure file will trigger the latency - measurement captured in the pseudo_lock_mem_latency tracepoint. See - example below. -2: - writing "2" to the pseudo_lock_measure file will trigger the L2 cache - residency (cache hits and misses) measurement captured in the - pseudo_lock_l2 tracepoint. See example below. -3: - writing "3" to the pseudo_lock_measure file will trigger the L3 cache - residency (cache hits and misses) measurement captured in the - pseudo_lock_l3 tracepoint. - -All measurements are recorded with the tracing infrastructure. This requires -the relevant tracepoints to be enabled before the measurement is triggered. - -Example of latency debugging interface -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In this example a pseudo-locked region named "newlock" was created. Here is -how we can measure the latency in cycles of reading from this region and -visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS -is set:: - - # :> /sys/kernel/tracing/trace - # echo 'hist:keys=latency' > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/trigger - # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable - # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure - # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable - # cat /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/hist - - # event histogram - # - # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active] - # - - { latency: 456 } hitcount: 1 - { latency: 50 } hitcount: 83 - { latency: 36 } hitcount: 96 - { latency: 44 } hitcount: 174 - { latency: 48 } hitcount: 195 - { latency: 46 } hitcount: 262 - { latency: 42 } hitcount: 693 - { latency: 40 } hitcount: 3204 - { latency: 38 } hitcount: 3484 - - Totals: - Hits: 8192 - Entries: 9 - Dropped: 0 - -Example of cache hits/misses debugging -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In this example a pseudo-locked region named "newlock" was created on the L2 -cache of a platform. Here is how we can obtain details of the cache hits -and misses using the platform's precision counters. -:: - - # :> /sys/kernel/tracing/trace - # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable - # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure - # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable - # cat /sys/kernel/tracing/trace - - # tracer: nop - # - # _-----=> irqs-off - # / _----=> need-resched - # | / _---=> hardirq/softirq - # || / _--=> preempt-depth - # ||| / delay - # TASK-PID CPU# |||| TIMESTAMP FUNCTION - # | | | |||| | | - pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0 - - -Examples for RDT allocation usage -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1) Example 1 - -On a two socket machine (one L3 cache per socket) with just four bits -for cache bit masks, minimum b/w of 10% with a memory bandwidth -granularity of 10%. -:: - - # mount -t resctrl resctrl /sys/fs/resctrl - # cd /sys/fs/resctrl - # mkdir p0 p1 - # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata - # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata - -The default resource group is unmodified, so we have access to all parts -of all caches (its schemata file reads "L3:0=f;1=f"). - -Tasks that are under the control of group "p0" may only allocate from the -"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. -Tasks in group "p1" use the "lower" 50% of cache on both sockets. - -Similarly, tasks that are under the control of group "p0" may use a -maximum memory b/w of 50% on socket0 and 50% on socket 1. -Tasks in group "p1" may also use 50% memory b/w on both sockets. -Note that unlike cache masks, memory b/w cannot specify whether these -allocations can overlap or not. The allocations specifies the maximum -b/w that the group may be able to use and the system admin can configure -the b/w accordingly. - -If resctrl is using the software controller (mba_sc) then user can enter the -max b/w in MB rather than the percentage values. -:: - - # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata - # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata - -In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w -of 1024MB where as on socket 1 they would use 500MB. - -2) Example 2 - -Again two sockets, but this time with a more realistic 20-bit mask. - -Two real time tasks pid=1234 running on processor 0 and pid=5678 running on -processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy -neighbors, each of the two real-time tasks exclusively occupies one quarter -of L3 cache on socket 0. -:: - - # mount -t resctrl resctrl /sys/fs/resctrl - # cd /sys/fs/resctrl - -First we reset the schemata for the default group so that the "upper" -50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by -ordinary tasks:: - - # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata - -Next we make a resource group for our first real time task and give -it access to the "top" 25% of the cache on socket 0. -:: - - # mkdir p0 - # echo "L3:0=f8000;1=fffff" > p0/schemata - -Finally we move our first real time task into this resource group. We -also use taskset(1) to ensure the task always runs on a dedicated CPU -on socket 0. Most uses of resource groups will also constrain which -processors tasks run on. -:: - - # echo 1234 > p0/tasks - # taskset -cp 1 1234 - -Ditto for the second real time task (with the remaining 25% of cache):: - - # mkdir p1 - # echo "L3:0=7c00;1=fffff" > p1/schemata - # echo 5678 > p1/tasks - # taskset -cp 2 5678 - -For the same 2 socket system with memory b/w resource and CAT L3 the -schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is -10): - -For our first real time task this would request 20% memory b/w on socket 0. -:: - - # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata - -For our second real time task this would request an other 20% memory b/w -on socket 0. -:: - - # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata - -3) Example 3 - -A single socket system which has real-time tasks running on core 4-7 and -non real-time workload assigned to core 0-3. The real-time tasks share text -and data, so a per task association is not required and due to interaction -with the kernel it's desired that the kernel on these cores shares L3 with -the tasks. -:: - - # mount -t resctrl resctrl /sys/fs/resctrl - # cd /sys/fs/resctrl - -First we reset the schemata for the default group so that the "upper" -50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0 -cannot be used by ordinary tasks:: - - # echo "L3:0=3ff\nMB:0=50" > schemata - -Next we make a resource group for our real time cores and give it access -to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on -socket 0. -:: - - # mkdir p0 - # echo "L3:0=ffc00\nMB:0=50" > p0/schemata - -Finally we move core 4-7 over to the new group and make sure that the -kernel and the tasks running there get 50% of the cache. They should -also get 50% of memory bandwidth assuming that the cores 4-7 are SMT -siblings and only the real time threads are scheduled on the cores 4-7. -:: - - # echo F0 > p0/cpus - -4) Example 4 - -The resource groups in previous examples were all in the default "shareable" -mode allowing sharing of their cache allocations. If one resource group -configures a cache allocation then nothing prevents another resource group -to overlap with that allocation. - -In this example a new exclusive resource group will be created on a L2 CAT -system with two L2 cache instances that can be configured with an 8-bit -capacity bitmask. The new exclusive resource group will be configured to use -25% of each cache instance. -:: - - # mount -t resctrl resctrl /sys/fs/resctrl/ - # cd /sys/fs/resctrl - -First, we observe that the default group is configured to allocate to all L2 -cache:: - - # cat schemata - L2:0=ff;1=ff - -We could attempt to create the new resource group at this point, but it will -fail because of the overlap with the schemata of the default group:: - - # mkdir p0 - # echo 'L2:0=0x3;1=0x3' > p0/schemata - # cat p0/mode - shareable - # echo exclusive > p0/mode - -sh: echo: write error: Invalid argument - # cat info/last_cmd_status - schemata overlaps - -To ensure that there is no overlap with another resource group the default -resource group's schemata has to change, making it possible for the new -resource group to become exclusive. -:: - - # echo 'L2:0=0xfc;1=0xfc' > schemata - # echo exclusive > p0/mode - # grep . p0/* - p0/cpus:0 - p0/mode:exclusive - p0/schemata:L2:0=03;1=03 - p0/size:L2:0=262144;1=262144 - -A new resource group will on creation not overlap with an exclusive resource -group:: - - # mkdir p1 - # grep . p1/* - p1/cpus:0 - p1/mode:shareable - p1/schemata:L2:0=fc;1=fc - p1/size:L2:0=786432;1=786432 - -The bit_usage will reflect how the cache is used:: - - # cat info/L2/bit_usage - 0=SSSSSSEE;1=SSSSSSEE - -A resource group cannot be forced to overlap with an exclusive resource group:: - - # echo 'L2:0=0x1;1=0x1' > p1/schemata - -sh: echo: write error: Invalid argument - # cat info/last_cmd_status - overlaps with exclusive group - -Example of Cache Pseudo-Locking -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked -region is exposed at /dev/pseudo_lock/newlock that can be provided to -application for argument to mmap(). -:: - - # mount -t resctrl resctrl /sys/fs/resctrl/ - # cd /sys/fs/resctrl - -Ensure that there are bits available that can be pseudo-locked, since only -unused bits can be pseudo-locked the bits to be pseudo-locked needs to be -removed from the default resource group's schemata:: - - # cat info/L2/bit_usage - 0=SSSSSSSS;1=SSSSSSSS - # echo 'L2:1=0xfc' > schemata - # cat info/L2/bit_usage - 0=SSSSSSSS;1=SSSSSS00 - -Create a new resource group that will be associated with the pseudo-locked -region, indicate that it will be used for a pseudo-locked region, and -configure the requested pseudo-locked region capacity bitmask:: - - # mkdir newlock - # echo pseudo-locksetup > newlock/mode - # echo 'L2:1=0x3' > newlock/schemata - -On success the resource group's mode will change to pseudo-locked, the -bit_usage will reflect the pseudo-locked region, and the character device -exposing the pseudo-locked region will exist:: - - # cat newlock/mode - pseudo-locked - # cat info/L2/bit_usage - 0=SSSSSSSS;1=SSSSSSPP - # ls -l /dev/pseudo_lock/newlock - crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock - -:: - - /* - * Example code to access one page of pseudo-locked cache region - * from user space. - */ - #define _GNU_SOURCE - #include <fcntl.h> - #include <sched.h> - #include <stdio.h> - #include <stdlib.h> - #include <unistd.h> - #include <sys/mman.h> - - /* - * It is required that the application runs with affinity to only - * cores associated with the pseudo-locked region. Here the cpu - * is hardcoded for convenience of example. - */ - static int cpuid = 2; - - int main(int argc, char *argv[]) - { - cpu_set_t cpuset; - long page_size; - void *mapping; - int dev_fd; - int ret; - - page_size = sysconf(_SC_PAGESIZE); - - CPU_ZERO(&cpuset); - CPU_SET(cpuid, &cpuset); - ret = sched_setaffinity(0, sizeof(cpuset), &cpuset); - if (ret < 0) { - perror("sched_setaffinity"); - exit(EXIT_FAILURE); - } - - dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR); - if (dev_fd < 0) { - perror("open"); - exit(EXIT_FAILURE); - } - - mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, - dev_fd, 0); - if (mapping == MAP_FAILED) { - perror("mmap"); - close(dev_fd); - exit(EXIT_FAILURE); - } - - /* Application interacts with pseudo-locked memory @mapping */ - - ret = munmap(mapping, page_size); - if (ret < 0) { - perror("munmap"); - close(dev_fd); - exit(EXIT_FAILURE); - } - - close(dev_fd); - exit(EXIT_SUCCESS); - } - -Locking between applications ----------------------------- - -Certain operations on the resctrl filesystem, composed of read/writes -to/from multiple files, must be atomic. - -As an example, the allocation of an exclusive reservation of L3 cache -involves: - - 1. Read the cbmmasks from each directory or the per-resource "bit_usage" - 2. Find a contiguous set of bits in the global CBM bitmask that is clear - in any of the directory cbmmasks - 3. Create a new directory - 4. Set the bits found in step 2 to the new directory "schemata" file - -If two applications attempt to allocate space concurrently then they can -end up allocating the same bits so the reservations are shared instead of -exclusive. - -To coordinate atomic operations on the resctrlfs and to avoid the problem -above, the following locking procedure is recommended: - -Locking is based on flock, which is available in libc and also as a shell -script command - -Write lock: - - A) Take flock(LOCK_EX) on /sys/fs/resctrl - B) Read/write the directory structure. - C) funlock - -Read lock: - - A) Take flock(LOCK_SH) on /sys/fs/resctrl - B) If success read the directory structure. - C) funlock - -Example with bash:: - - # Atomically read directory structure - $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl - - # Read directory contents and create new subdirectory - - $ cat create-dir.sh - find /sys/fs/resctrl/ > output.txt - mask = function-of(output.txt) - mkdir /sys/fs/resctrl/newres/ - echo mask > /sys/fs/resctrl/newres/schemata - - $ flock /sys/fs/resctrl/ ./create-dir.sh - -Example with C:: - - /* - * Example code do take advisory locks - * before accessing resctrl filesystem - */ - #include <sys/file.h> - #include <stdlib.h> - - void resctrl_take_shared_lock(int fd) - { - int ret; - - /* take shared lock on resctrl filesystem */ - ret = flock(fd, LOCK_SH); - if (ret) { - perror("flock"); - exit(-1); - } - } - - void resctrl_take_exclusive_lock(int fd) - { - int ret; - - /* release lock on resctrl filesystem */ - ret = flock(fd, LOCK_EX); - if (ret) { - perror("flock"); - exit(-1); - } - } - - void resctrl_release_lock(int fd) - { - int ret; - - /* take shared lock on resctrl filesystem */ - ret = flock(fd, LOCK_UN); - if (ret) { - perror("flock"); - exit(-1); - } - } - - void main(void) - { - int fd, ret; - - fd = open("/sys/fs/resctrl", O_DIRECTORY); - if (fd == -1) { - perror("open"); - exit(-1); - } - resctrl_take_shared_lock(fd); - /* code to read directory contents */ - resctrl_release_lock(fd); - - resctrl_take_exclusive_lock(fd); - /* code to read and write directory contents */ - resctrl_release_lock(fd); - } - -Examples for RDT Monitoring along with allocation usage -======================================================= -Reading monitored data ----------------------- -Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would -show the current snapshot of LLC occupancy of the corresponding MON -group or CTRL_MON group. - - -Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group) ------------------------------------------------------------------------- -On a two socket machine (one L3 cache per socket) with just four bits -for cache bit masks:: - - # mount -t resctrl resctrl /sys/fs/resctrl - # cd /sys/fs/resctrl - # mkdir p0 p1 - # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata - # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata - # echo 5678 > p1/tasks - # echo 5679 > p1/tasks - -The default resource group is unmodified, so we have access to all parts -of all caches (its schemata file reads "L3:0=f;1=f"). - -Tasks that are under the control of group "p0" may only allocate from the -"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. -Tasks in group "p1" use the "lower" 50% of cache on both sockets. - -Create monitor groups and assign a subset of tasks to each monitor group. -:: - - # cd /sys/fs/resctrl/p1/mon_groups - # mkdir m11 m12 - # echo 5678 > m11/tasks - # echo 5679 > m12/tasks - -fetch data (data shown in bytes) -:: - - # cat m11/mon_data/mon_L3_00/llc_occupancy - 16234000 - # cat m11/mon_data/mon_L3_01/llc_occupancy - 14789000 - # cat m12/mon_data/mon_L3_00/llc_occupancy - 16789000 - -The parent ctrl_mon group shows the aggregated data. -:: - - # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy - 31234000 - -Example 2 (Monitor a task from its creation) --------------------------------------------- -On a two socket machine (one L3 cache per socket):: - - # mount -t resctrl resctrl /sys/fs/resctrl - # cd /sys/fs/resctrl - # mkdir p0 p1 - -An RMID is allocated to the group once its created and hence the <cmd> -below is monitored from its creation. -:: - - # echo $$ > /sys/fs/resctrl/p1/tasks - # <cmd> - -Fetch the data:: - - # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy - 31789000 - -Example 3 (Monitor without CAT support or before creating CAT groups) ---------------------------------------------------------------------- - -Assume a system like HSW has only CQM and no CAT support. In this case -the resctrl will still mount but cannot create CTRL_MON directories. -But user can create different MON groups within the root group thereby -able to monitor all tasks including kernel threads. - -This can also be used to profile jobs cache size footprint before being -able to allocate them to different allocation groups. -:: - - # mount -t resctrl resctrl /sys/fs/resctrl - # cd /sys/fs/resctrl - # mkdir mon_groups/m01 - # mkdir mon_groups/m02 - - # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks - # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks - -Monitor the groups separately and also get per domain data. From the -below its apparent that the tasks are mostly doing work on -domain(socket) 0. -:: - - # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy - 31234000 - # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy - 34555 - # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy - 31234000 - # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy - 32789 - - -Example 4 (Monitor real time tasks) ------------------------------------ - -A single socket system which has real time tasks running on cores 4-7 -and non real time tasks on other cpus. We want to monitor the cache -occupancy of the real time threads on these cores. -:: - - # mount -t resctrl resctrl /sys/fs/resctrl - # cd /sys/fs/resctrl - # mkdir p1 - -Move the cpus 4-7 over to p1:: - - # echo f0 > p1/cpus - -View the llc occupancy snapshot:: - - # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy - 11234000 - -Intel RDT Errata -================ - -Intel MBM Counters May Report System Memory Bandwidth Incorrectly ------------------------------------------------------------------ - -Errata SKX99 for Skylake server and BDF102 for Broadwell server. - -Problem: Intel Memory Bandwidth Monitoring (MBM) counters track metrics -according to the assigned Resource Monitor ID (RMID) for that logical -core. The IA32_QM_CTR register (MSR 0xC8E), used to report these -metrics, may report incorrect system bandwidth for certain RMID values. - -Implication: Due to the errata, system memory bandwidth may not match -what is reported. - -Workaround: MBM total and local readings are corrected according to the -following correction factor table: - -+---------------+---------------+---------------+-----------------+ -|core count |rmid count |rmid threshold |correction factor| -+---------------+---------------+---------------+-----------------+ -|1 |8 |0 |1.000000 | -+---------------+---------------+---------------+-----------------+ -|2 |16 |0 |1.000000 | -+---------------+---------------+---------------+-----------------+ -|3 |24 |15 |0.969650 | -+---------------+---------------+---------------+-----------------+ -|4 |32 |0 |1.000000 | -+---------------+---------------+---------------+-----------------+ -|6 |48 |31 |0.969650 | -+---------------+---------------+---------------+-----------------+ -|7 |56 |47 |1.142857 | -+---------------+---------------+---------------+-----------------+ -|8 |64 |0 |1.000000 | -+---------------+---------------+---------------+-----------------+ -|9 |72 |63 |1.185115 | -+---------------+---------------+---------------+-----------------+ -|10 |80 |63 |1.066553 | -+---------------+---------------+---------------+-----------------+ -|11 |88 |79 |1.454545 | -+---------------+---------------+---------------+-----------------+ -|12 |96 |0 |1.000000 | -+---------------+---------------+---------------+-----------------+ -|13 |104 |95 |1.230769 | -+---------------+---------------+---------------+-----------------+ -|14 |112 |95 |1.142857 | -+---------------+---------------+---------------+-----------------+ -|15 |120 |95 |1.066667 | -+---------------+---------------+---------------+-----------------+ -|16 |128 |0 |1.000000 | -+---------------+---------------+---------------+-----------------+ -|17 |136 |127 |1.254863 | -+---------------+---------------+---------------+-----------------+ -|18 |144 |127 |1.185255 | -+---------------+---------------+---------------+-----------------+ -|19 |152 |0 |1.000000 | -+---------------+---------------+---------------+-----------------+ -|20 |160 |127 |1.066667 | -+---------------+---------------+---------------+-----------------+ -|21 |168 |0 |1.000000 | -+---------------+---------------+---------------+-----------------+ -|22 |176 |159 |1.454334 | -+---------------+---------------+---------------+-----------------+ -|23 |184 |0 |1.000000 | -+---------------+---------------+---------------+-----------------+ -|24 |192 |127 |0.969744 | -+---------------+---------------+---------------+-----------------+ -|25 |200 |191 |1.280246 | -+---------------+---------------+---------------+-----------------+ -|26 |208 |191 |1.230921 | -+---------------+---------------+---------------+-----------------+ -|27 |216 |0 |1.000000 | -+---------------+---------------+---------------+-----------------+ -|28 |224 |191 |1.143118 | -+---------------+---------------+---------------+-----------------+ - -If rmid > rmid threshold, MBM total and local values should be multiplied -by the correction factor. - -See: - -1. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update: -http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html - -2. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update: -http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf - -3. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual: -https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html - -for further information. |