On 2/12/23 09:05, Sebastian Wittmeier via Std-Proposals wrote:
> Hi Phil,
>
> can you ellaborate on the reliability issues of RtlAllocateHeap()?
>
> Are you talking about out-of-memory conditions or heap corruption with
> HEAP_NO_SERIALIZE in multi-threading scenarios or about performance in
> those cases?
>
> Are there reliability issues even without HEAP_NO_SERIALZE? Then
> RtlAllocateHeap() violates the Windows API specification.

No I do not use HEAP_NO_SERIALZE so it should be "thread-safe".

> Or you want C++ to provide facilities to provide ways to avoid/replace
> RtlAllocateHeap() for reliability or compliance reasons for the whole
> application, even for static and dynamic libraries, which have been
> directly linked to it?

In brave new world that's exactly what I would hope for. A memory pool
is not very complicated to implement using the following possible
characteristics:
- thread-local for faster and safer accesses;
- type-oriented to group block sizes together and to create less
fragmentation;
- allocation rate to extrapolate future memory page allocation size;
- possible "reallocation";
- possible destruction of heap pages with or without calling destructors;
- ...

So anyway there's a lot of factors that can be considered and I would
have hoped to replace low-level memory pools with custom-ones. But the
only way to achieve this is by standardizing linkers.

Linkers are very simple to implement. Vendors could keep their favorite
name mangling but it really is a simple hash table of symbols and
addresses with some initialization function.

1) Let's suppose we have all these new standards already in place then
it would be possible for an app to redirect low-level calls to
RtlAllocateHeap() and RtlFreeHeap() to some other custom defined using
another local hash table to redirect these calls.

2) Another solution would be to get Microsoft and other vendors to call
another handler using a function similar to set_new_handler.

3) Use something similar to LD_PRELOAD under Linux (or AppInit_DLLs
under Windows but deactivated under secure boot).

Either option 1) or 2), vendors will have to modify their core dynamics
libraries. That is why I'm addressing this group.

If you search engine search on Rtl*Heap() functions then this problem
lasted for a long time and all related bug reports were simply
dismissed. I think it's time to turn the page on those issues.


Thanks,

> If C++ would provide a set_new_handle() function for connecting to an
> allocation routine in an application-wide way, what would prevent the
> static or dynamic library to call set_new_handle and link to
> RtlAllocateHeap?
>
> To really enforce it one would have to introduce ways into C++ to have
> translation units with higher privilege level that may call certain
> functions from the standard library (set_new_handler), other translation
> units may not.
>
> Also one would have to prevent the other translation units from being
> compiled with different settings or different standard library versions
> and prevent them from calling the Windows API directly?
>
> And you want to specifically prevent ntdll.dll from being compiled on
> this higher translation unit privilege level?
>
> I think alternative approaches would be either:
>
> - Have RtlAllocateHeap() give better (?) guarantees in heavily
> multi-threaded environments.
>
> - Ask Microsoft to provide facilities to customize RtlAllocateHeap()
> behavior (e.g. for registering custom allocation routines)
>
> If the set_new_handler is extended, then one would have to
>
> - Define a standard for nice behaving dynamic and static libraries
> that should neither call certain Windows API functions (like
> RtlAllocateHeap() or the new set_new_handler) directly nor depend on
> another library calling those API functions
>
>
> Best,
>
> Sebastian
>
> -----Ursprüngliche Nachricht-----
> *Von:* Phil Bouchard via Std-Proposals <std-proposals_at_[hidden]>
> *Gesendet:* Sa 11.02.2023 23:05
> *Betreff:* [std-proposals] set_new_handler extension
> *An:* Std-Proposals_at_[hidden];
> *CC:* Phil Bouchard <boost_at_[hidden]>;
> Greetings,
>
> From my experience the underlying Windows RtlAllocateHeap() and
> RtlFreeHeap() aren't reliable in extensive multithreading usages.
>
> One solution is to use thread-local memory pools, which are not only
> faster but safer.
>
> Regarding the C++ standards there is only the following function that
> can be use upon memory allocation failures:
>
> https://en.cppreference.com/w/cpp/memory/new/set_new_handler
>
> Why can't we have a handler that would be called before it even
> tries to
> allocate anything? This way we could implement our thread-local
> allocation routines through the C++ standards and not relying on some
> linker hacks magic.
>
>
> Regards,
>
> --
> Email Signature
> Logo <https://www.fornux.com/>
> *Phil Bouchard* facebook icon
> <https://www.linkedin.com/in/phil-bouchard-5723a910/>
> CTO
> T: (819) 328-4743
> E: phil_at_[hidden]| www.fornux.com <http://www.fornux.com>
> 8 rue de la Baie| Gatineau (Qc), J8T 3H3 Canada
>
> Banner <https://goglobalawards.org/> Le message ci-dessus, ainsi que
> les
> documents l'accompagnant, sont destinés uniquement aux personnes
> identifiées et peuvent contenir des informations privilégiées,
> confidentielles ou ne pouvant être divulguées. Si vous avez reçu ce
> message par erreur, veuillez le détruire.
> This communication (and/or the attachments) is intended for named
> recipients only and may contain privileged or confidential information
> which is not to be disclosed. If you received this communication by
> mistake please destroy all copies.
>
> --
> Std-Proposals mailing list
> Std-Proposals_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/std-proposals
>
>

-- 
Logo <https://www.fornux.com/>  
*Phil Bouchard*  facebook icon
<https://www.linkedin.com/in/phil-bouchard-5723a910/> 
CTO
T: (819) 328-4743
E: phil_at_[hidden]| www.fornux.com <http://www.fornux.com>
8 rue de la Baie| Gatineau (Qc), J8T 3H3 Canada
Banner <https://goglobalawards.org/> Le message ci-dessus, ainsi que les
documents l'accompagnant, sont destinés uniquement aux personnes
identifiées et peuvent contenir des informations privilégiées,
confidentielles ou ne pouvant être divulguées. Si vous avez reçu ce
message par erreur, veuillez le détruire.
This communication (and/or the attachments) is intended for named
recipients only and may contain privileged or confidential information
which is not to be disclosed. If you received this communication by
mistake please destroy all copies.