According to the Standard, when using std::atomic<float>::fetch_add etc.,
operations that would result in a value of range, an unspecified value
results rather than undefined behavior. This is like how overflowing a
std::atomic<int> is well-defined unlike just int.

The Standard says that such float fetch_add operations may use a different
floating-point environment than the calling function.

If the calling thread has floating-point exceptions enabled, and one of the
atomic operations would trigger in an addition / subtraction, what should
happen?

MSVC, libc++ and libstdc++ all will throw the exception.