[off list]

On Tuesday 30 July 2024 08:19:11 GMT-7 you wrote:
> sub rsp, 512 ; allocate space for the floating points
> fxsave [rsp] ; save the floating points

Insufficient space and wrong instruction for all modern CPUs because FXSAVE does
not save the AVX registers and everything added since then. You need to use
the XSAVE instruction, not FXSAVE, and the space to be used needs to be
obtained from CPUID leaf 0xd. But XSAVE is not present in all processors, so
you need to use CPUID to detect it in the first place too, falling back to
FXSAVE. Don't forget to align the buffer to 512 bytes too.

You don't need to save all registers and the flags register, if you can assumes
that the function calls are ABI-compliant. You only need to save the callee-
save (preserved) registers in that case.

> push r10 ; Put r10's value on the stack...
> add rsp, 8 ; ...still there but hidden

No, not on Windows. There is no red zone below the stack where some values are
preserved no matter what. The moment that you added to RSP, the value you had
pushed there must be assumed to have disappeared.

https://en.wikipedia.org/wiki/Red_zone_(computing)
https://stackoverflow.com/questions/38042188/where-exactly-is-the-red-zone-on-x86-64

> jmp qword ptr[rsp - (512 + 17*8)] ; Jump back to caller

Ditto. And this far down the stack, it's beyond the red zone even on the
System V ABI, which means the data there would be clobbered by a signal
delivery to this thread.

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
  Principal Engineer - Intel DCAI Platform & System Engineering