[vector.modifiers] says that inserting an element at position `p`
invalidates all iterators, references and pointers from position `p`
onwards.
I'm wondering why there's such a requirement for pointers (and
references) [...]
This kind of Library provision, saying that iterators, pointers, and references to elements are invalidated upon condition X, seems overly broad even if you don't care about using pointers in an "index-like" way, because it means you can't safely use an element type that stores (and properly updates across moves/copies) the 'this' pointer as a member. I suspect that the real intent of this kind of provision is that it only applies to iterators, pointers, and references obtained through the container's API.