1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
|
Using XSTATE features in user space applications
================================================
The x86 architecture supports floating-point extensions which are
enumerated via CPUID. Applications consult CPUID and use XGETBV to
evaluate which features have been enabled by the kernel XCR0.
Up to AVX-512 and PKRU states, these features are automatically enabled by
the kernel if available. Features like AMX TILE_DATA (XSTATE component 18)
are enabled by XCR0 as well, but the first use of related instruction is
trapped by the kernel because by default the required large XSTATE buffers
are not allocated automatically.
The purpose for dynamic features
--------------------------------
Legacy userspace libraries often have hard-coded, static sizes for
alternate signal stacks, often using MINSIGSTKSZ which is typically 2KB.
That stack must be able to store at *least* the signal frame that the
kernel sets up before jumping into the signal handler. That signal frame
must include an XSAVE buffer defined by the CPU.
However, that means that the size of signal stacks is dynamic, not static,
because different CPUs have differently-sized XSAVE buffers. A compiled-in
size of 2KB with existing applications is too small for new CPU features
like AMX. Instead of universally requiring larger stack, with the dynamic
enabling, the kernel can enforce userspace applications to have
properly-sized altstacks.
Using dynamically enabled XSTATE features in user space applications
--------------------------------------------------------------------
The kernel provides an arch_prctl(2) based mechanism for applications to
request the usage of such features. The arch_prctl(2) options related to
this are:
-ARCH_GET_XCOMP_SUPP
arch_prctl(ARCH_GET_XCOMP_SUPP, &features);
ARCH_GET_XCOMP_SUPP stores the supported features in userspace storage of
type uint64_t. The second argument is a pointer to that storage.
-ARCH_GET_XCOMP_PERM
arch_prctl(ARCH_GET_XCOMP_PERM, &features);
ARCH_GET_XCOMP_PERM stores the features for which the userspace process
has permission in userspace storage of type uint64_t. The second argument
is a pointer to that storage.
-ARCH_REQ_XCOMP_PERM
arch_prctl(ARCH_REQ_XCOMP_PERM, feature_nr);
ARCH_REQ_XCOMP_PERM allows to request permission for a dynamically enabled
feature or a feature set. A feature set can be mapped to a facility, e.g.
AMX, and can require one or more XSTATE components to be enabled.
The feature argument is the number of the highest XSTATE component which
is required for a facility to work.
When requesting permission for a feature, the kernel checks the
availability. The kernel ensures that sigaltstacks in the process's tasks
are large enough to accommodate the resulting large signal frame. It
enforces this both during ARCH_REQ_XCOMP_SUPP and during any subsequent
sigaltstack(2) calls. If an installed sigaltstack is smaller than the
resulting sigframe size, ARCH_REQ_XCOMP_SUPP results in -ENOSUPP. Also,
sigaltstack(2) results in -ENOMEM if the requested altstack is too small
for the permitted features.
Permission, when granted, is valid per process. Permissions are inherited
on fork(2) and cleared on exec(3).
The first use of an instruction related to a dynamically enabled feature is
trapped by the kernel. The trap handler checks whether the process has
permission to use the feature. If the process has no permission then the
kernel sends SIGILL to the application. If the process has permission then
the handler allocates a larger xstate buffer for the task so the large
state can be context switched. In the unlikely cases that the allocation
fails, the kernel sends SIGSEGV.
AMX TILE_DATA enabling example
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Below is the example of how userspace applications enable
TILE_DATA dynamically:
1. The application first needs to query the kernel for AMX
support::
#include <asm/prctl.h>
#include <sys/syscall.h>
#include <stdio.h>
#include <unistd.h>
#ifndef ARCH_GET_XCOMP_SUPP
#define ARCH_GET_XCOMP_SUPP 0x1021
#endif
#ifndef ARCH_XCOMP_TILECFG
#define ARCH_XCOMP_TILECFG 17
#endif
#ifndef ARCH_XCOMP_TILEDATA
#define ARCH_XCOMP_TILEDATA 18
#endif
#define MASK_XCOMP_TILE ((1 << ARCH_XCOMP_TILECFG) | \
(1 << ARCH_XCOMP_TILEDATA))
unsigned long features;
long rc;
...
rc = syscall(SYS_arch_prctl, ARCH_GET_XCOMP_SUPP, &features);
if (!rc && (features & MASK_XCOMP_TILE) == MASK_XCOMP_TILE)
printf("AMX is available.\n");
2. After that, determining support for AMX, an application must
explicitly ask permission to use it::
#ifndef ARCH_REQ_XCOMP_PERM
#define ARCH_REQ_XCOMP_PERM 0x1023
#endif
...
rc = syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, ARCH_XCOMP_TILEDATA);
if (!rc)
printf("AMX is ready for use.\n");
Note this example does not include the sigaltstack preparation.
Dynamic features in signal frames
---------------------------------
Dynamically enabled features are not written to the signal frame upon signal
entry if the feature is in its initial configuration. This differs from
non-dynamic features which are always written regardless of their
configuration. Signal handlers can examine the XSAVE buffer's XSTATE_BV
field to determine if a features was written.
Dynamic features for virtual machines
-------------------------------------
The permission for the guest state component needs to be managed separately
from the host, as they are exclusive to each other. A coupled of options
are extended to control the guest permission:
-ARCH_GET_XCOMP_GUEST_PERM
arch_prctl(ARCH_GET_XCOMP_GUEST_PERM, &features);
ARCH_GET_XCOMP_GUEST_PERM is a variant of ARCH_GET_XCOMP_PERM. So it
provides the same semantics and functionality but for the guest
components.
-ARCH_REQ_XCOMP_GUEST_PERM
arch_prctl(ARCH_REQ_XCOMP_GUEST_PERM, feature_nr);
ARCH_REQ_XCOMP_GUEST_PERM is a variant of ARCH_REQ_XCOMP_PERM. It has the
same semantics for the guest permission. While providing a similar
functionality, this comes with a constraint. Permission is frozen when the
first VCPU is created. Any attempt to change permission after that point
is going to be rejected. So, the permission has to be requested before the
first VCPU creation.
Note that some VMMs may have already established a set of supported state
components. These options are not presumed to support any particular VMM.
Signal Frame Layout and Portability
-----------------------------------
The signal frame is designed to be self-describing and portable. This is
especially important for checkpoint/restore tools like CRIU, which may restore
a process on a different host than where it was checkpointed. A signal frame
created on a machine with fewer CPU features can be successfully restored on a
machine with more CPU features, but not vice-versa.
Signal Frame Software Reserved Bytes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
On CPUs supporting XSAVE, bytes 464..511 in the 512-byte FXSAVE/FXRSTOR frame
are reserved for software use and contain ``struct _fpx_sw_bytes`` (defined in
``<uapi/asm/sigcontext.h>``)::
struct _fpx_sw_bytes {
__u32 magic1;
__u32 extended_size;
__u64 xfeatures;
__u32 xstate_size;
__u32 padding[7];
};
- ``magic1``: Set to ``FP_XSTATE_MAGIC1`` (``0x46505853U``) if an extended
xstate context is present; 0 for a legacy frame.
- ``extended_size``: The total size allocated on the stack for the frame,
measured from the ``fpstate`` pointer. In 32-bit signal frames, this also
includes the 112-byte legacy FPU state prefix of ``struct _fpstate_32``.
- ``xfeatures``: The mask of xstate features saved in the frame.
- ``xstate_size``: The actual size of the xstate context for the enabled
features (including the 512-byte FXSAVE area and the 64-byte XSAVE header).
The kernel uses ``xstate_size`` in conjunction with the pointer to the xstate
context to locate the ``FP_XSTATE_MAGIC2`` (``0x46505845U``) marker right after
the xstate context (at ``xstate_context + xstate_size``). In 64-bit signal frames,
the ``fpstate`` pointer points directly to the xstate context. In 32-bit signal
frames (including 32-bit compat tasks on 64-bit kernels), the ``fpstate``
pointer points to ``struct _fpstate_32``, which contains the 112-byte legacy
FPU state followed by the 512-byte FXSR state (and any extended xstate). Since
there is no standalone UAPI structure defined for just the 112-byte legacy
state, the xstate context starts at ``fpstate + 112`` (and ``extended_size``
spans the entire allocation from ``fpstate``).
Portability Constraints
^^^^^^^^^^^^^^^^^^^^^^^
Signal frame portability is constrained by the architectural XSAVE layout.
Restoration is supported only if the destination host supports all features
present in the frame and uses matching component offsets and sizes for them.
While layout compatibility is generally maintained across CPUs from the same
vendor, differences can occur across vendors or if the XSAVE space of a
deprecated feature (e.g. MPX) is repurposed for a newer feature (e.g. APX).
|
