Date: Fri, 8 Mar 2019 06:07:42 -1000
On Fri, Mar 8, 2019 at 5:45 AM Tom Honermann <tom_at_[hidden]> wrote:
> On 3/8/19 2:28 AM, Gabriel Dos Reis via Modules wrote:
> > I prefer the suggestion of a zip file containing the required files much
> better.
>
> The zip approach has advantages in terms of being able to reproduce
> preprocesor issues (assuming the issue doesn't have to do with resolving
> header names to files of course).
>
> However, the zip approach doesn't work well for test case reduction
> tools like Delta [1] or C-reduce [2] that need to mutate the files. Use
> with them would require exploding the zip. If the zip file contains
> files at absolute paths (e.g., under /usr/include), then the files can't
> be exploded to their natural locations thus requiring additional
> modifications to the compiler invocation to use alternate include paths
> - which can affect test case reduction.
>
Actually, I would expect it to be *easier* for them. Those tools will be
able to read the files in the zip (or other format) directly without ever
writing them to disk. There are plenty of libraries for that. If this
format is accepted as stdin to compilers, those tools should be able to
synthesize a new stream in memory and then shove it in to the compiler
without writing a single file to the filesystem until it has found the
minimal repro to output. What do you expect will make this harder?
>
> At Coverity, we do a fair amount of test case reduction in this way, so
> it is quite important to us.
>
> Tom.
>
> [1]: http://delta.tigris.org/
> [2]: http://embed.cs.utah.edu/creduce/
>
> >
> > — Gaby
> >
> >> On Mar 7, 2019, at 10:23 PM, Tom Honermann <tom_at_[hidden]> wrote:
> >>
> >>> On 3/5/19 11:11 AM, Ben Craig wrote:
> >>> I will concede that static analysis tools and other tools that try to
> parse C++ probably don't need a textual inclusion format, since they most
> likely need to be able to parse and understand the pragmas anyway. If
> module mapping is sufficiently straightforward, then those tools can do
> module lookup the same as a compiler. Those tools already need to do
> include lookup in the same way that compilers do.
> >>>
> >>> I think the textual inclusion format will still be very useful to
> distribution and caching tools though, as they don't need to understand the
> code. Those tools frequently lean on the compiler's preprocessor today,
> and don't know how to do include lookups.
> >> Another use case is reproducing issues encountered in the field. Static
> analysis tools like Coverity need to emulate other compilers. Today, when
> we fail to parse a TU that the emulated compiler accepts, we ask customers
> to send us preprocessed output for reproduction and analysis purposes. We
> ask for preprocessed output because that is much simpler to handle than the
> entire collection of included header files that must then be arranged
> according to some specific compiler invocation and set of include paths.
> We need a solution for this that works for modules as well. Clang's
> -frewrite-imports option so far seems to do the job for us and it uses
> #pragma directives in similar manner to those described here. I strongly
> favor specifying a common set of #pragma directives for this purpose.
> >>
> >> Tom.
> >>
> >>>> -----Original Message-----
> >>>> From: Nathan Sidwell <nathanmsidwell_at_[hidden]> On Behalf Of Nathan
> >>>> Sidwell
> >>>> Sent: Tuesday, March 5, 2019 6:04 AM
> >>>> To: Ben Craig <ben.craig_at_[hidden]>; modules_at_[hidden]; WG21
> Tooling
> >>>> Study Group SG15 <tooling_at_[hidden]>
> >>>> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules with
> old
> >>>> bad build systems
> >>>>
> >>>>> On 3/4/19 10:02 AM, Ben Craig wrote:
> >>>>> I do mean textual inclusion, though I can be convinced otherwise.
> Textual
> >>>> inclusion (with extra generated pragmas) should make it much easier
> to keep
> >>>> tools like distcc and cppcheck happy in the short term. I suspect
> that those
> >>>> tools don't want to crack open a BMI to figure out which other BMIs
> need to
> >>>> be found.
> >>>>> Tools that (think they can) parse C++ will still need to understand
> these
> >>>> pragmas in order to provide the right macro, visibility, and
> reachability
> >>>> behaviors, so some work will still be required on their part, but at
> least they
> >>>> won't need to understand new binary formats.
> >>>>
> >>>> Correct, tools consuming such #pragma-marked flattened source will
> need to
> >>>> understand modules at a fundamental level. As such, why not implement
> >>>> the same mechanisms to find module source as the compiler?
> >>>> That'll give them more information to perform code analysis with.
> >>>>
> >>>> nathan
> >>>>
> >>>>>>> On 3/2/19 1:03 PM, Ben Craig wrote:
> >>>>>>>
> >>>>>>> Some quick notes on this implementation strategy:
> >>>>>>> * Uses TEXTUAL inclusion
> >>>>>>> * Compiler assumes that the build system knows nothing of BMIs
> >>>>>>> * Compiler needs to be able to do module mapping with minimal input
> >>>>>>> from users.
> >>>>>> Do you literally mean textual inclusion or do you really mean
> >>>>>> dynamically produce an internal-only BMI object?
> >>>>>>
> >>>>>> nathan
> >>>>>>
> >>>>>> --
> >>>>>> Nathan Sidwell
> >>>> --
> >>>> Nathan Sidwell
> >>> _______________________________________________
> >>> Tooling mailing list
> >>> Tooling_at_[hidden]
> >>>
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.open-std.org%2Fmailman%2Flistinfo%2Ftooling&data=02%7C01%7Cgdr%40microsoft.com%7C1cdb6c203df64eb7b84408d6a38ea3ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636876230368116236&sdata=ugzbdk5BI%2Bppnt5nS0xDISJehZ3mizhZTBWDrVZZ%2BA8%3D&reserved=0
> >>
> >> _______________________________________________
> >> Modules mailing list
> >> Modules_at_[hidden]
> >> Subscription:
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fmodules&data=02%7C01%7Cgdr%40microsoft.com%7C1cdb6c203df64eb7b84408d6a38ea3ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636876230368116236&sdata=UPJ70IqxSq6PeLpRdMc7IZCotDOeyV6n7QNxOXjYFQo%3D&reserved=0
> >> Link to this post:
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fmodules%2F2019%2F03%2F0218.php&data=02%7C01%7Cgdr%40microsoft.com%7C1cdb6c203df64eb7b84408d6a38ea3ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636876230368116236&sdata=%2BsOB4DmF0Lf6EPmsPij3i%2BsDHDZS3ujHN4S9rVjMer4%3D&reserved=0
> > _______________________________________________
> > Modules mailing list
> > Modules_at_[hidden]
> > Subscription: http://lists.isocpp.org/mailman/listinfo.cgi/modules
> > Link to this post: http://lists.isocpp.org/modules/2019/03/0220.php
>
>
> _______________________________________________
> Modules mailing list
> Modules_at_[hidden]
> Subscription: http://lists.isocpp.org/mailman/listinfo.cgi/modules
> Link to this post: http://lists.isocpp.org/modules/2019/03/0226.php
>
> On 3/8/19 2:28 AM, Gabriel Dos Reis via Modules wrote:
> > I prefer the suggestion of a zip file containing the required files much
> better.
>
> The zip approach has advantages in terms of being able to reproduce
> preprocesor issues (assuming the issue doesn't have to do with resolving
> header names to files of course).
>
> However, the zip approach doesn't work well for test case reduction
> tools like Delta [1] or C-reduce [2] that need to mutate the files. Use
> with them would require exploding the zip. If the zip file contains
> files at absolute paths (e.g., under /usr/include), then the files can't
> be exploded to their natural locations thus requiring additional
> modifications to the compiler invocation to use alternate include paths
> - which can affect test case reduction.
>
Actually, I would expect it to be *easier* for them. Those tools will be
able to read the files in the zip (or other format) directly without ever
writing them to disk. There are plenty of libraries for that. If this
format is accepted as stdin to compilers, those tools should be able to
synthesize a new stream in memory and then shove it in to the compiler
without writing a single file to the filesystem until it has found the
minimal repro to output. What do you expect will make this harder?
>
> At Coverity, we do a fair amount of test case reduction in this way, so
> it is quite important to us.
>
> Tom.
>
> [1]: http://delta.tigris.org/
> [2]: http://embed.cs.utah.edu/creduce/
>
> >
> > — Gaby
> >
> >> On Mar 7, 2019, at 10:23 PM, Tom Honermann <tom_at_[hidden]> wrote:
> >>
> >>> On 3/5/19 11:11 AM, Ben Craig wrote:
> >>> I will concede that static analysis tools and other tools that try to
> parse C++ probably don't need a textual inclusion format, since they most
> likely need to be able to parse and understand the pragmas anyway. If
> module mapping is sufficiently straightforward, then those tools can do
> module lookup the same as a compiler. Those tools already need to do
> include lookup in the same way that compilers do.
> >>>
> >>> I think the textual inclusion format will still be very useful to
> distribution and caching tools though, as they don't need to understand the
> code. Those tools frequently lean on the compiler's preprocessor today,
> and don't know how to do include lookups.
> >> Another use case is reproducing issues encountered in the field. Static
> analysis tools like Coverity need to emulate other compilers. Today, when
> we fail to parse a TU that the emulated compiler accepts, we ask customers
> to send us preprocessed output for reproduction and analysis purposes. We
> ask for preprocessed output because that is much simpler to handle than the
> entire collection of included header files that must then be arranged
> according to some specific compiler invocation and set of include paths.
> We need a solution for this that works for modules as well. Clang's
> -frewrite-imports option so far seems to do the job for us and it uses
> #pragma directives in similar manner to those described here. I strongly
> favor specifying a common set of #pragma directives for this purpose.
> >>
> >> Tom.
> >>
> >>>> -----Original Message-----
> >>>> From: Nathan Sidwell <nathanmsidwell_at_[hidden]> On Behalf Of Nathan
> >>>> Sidwell
> >>>> Sent: Tuesday, March 5, 2019 6:04 AM
> >>>> To: Ben Craig <ben.craig_at_[hidden]>; modules_at_[hidden]; WG21
> Tooling
> >>>> Study Group SG15 <tooling_at_[hidden]>
> >>>> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules with
> old
> >>>> bad build systems
> >>>>
> >>>>> On 3/4/19 10:02 AM, Ben Craig wrote:
> >>>>> I do mean textual inclusion, though I can be convinced otherwise.
> Textual
> >>>> inclusion (with extra generated pragmas) should make it much easier
> to keep
> >>>> tools like distcc and cppcheck happy in the short term. I suspect
> that those
> >>>> tools don't want to crack open a BMI to figure out which other BMIs
> need to
> >>>> be found.
> >>>>> Tools that (think they can) parse C++ will still need to understand
> these
> >>>> pragmas in order to provide the right macro, visibility, and
> reachability
> >>>> behaviors, so some work will still be required on their part, but at
> least they
> >>>> won't need to understand new binary formats.
> >>>>
> >>>> Correct, tools consuming such #pragma-marked flattened source will
> need to
> >>>> understand modules at a fundamental level. As such, why not implement
> >>>> the same mechanisms to find module source as the compiler?
> >>>> That'll give them more information to perform code analysis with.
> >>>>
> >>>> nathan
> >>>>
> >>>>>>> On 3/2/19 1:03 PM, Ben Craig wrote:
> >>>>>>>
> >>>>>>> Some quick notes on this implementation strategy:
> >>>>>>> * Uses TEXTUAL inclusion
> >>>>>>> * Compiler assumes that the build system knows nothing of BMIs
> >>>>>>> * Compiler needs to be able to do module mapping with minimal input
> >>>>>>> from users.
> >>>>>> Do you literally mean textual inclusion or do you really mean
> >>>>>> dynamically produce an internal-only BMI object?
> >>>>>>
> >>>>>> nathan
> >>>>>>
> >>>>>> --
> >>>>>> Nathan Sidwell
> >>>> --
> >>>> Nathan Sidwell
> >>> _______________________________________________
> >>> Tooling mailing list
> >>> Tooling_at_[hidden]
> >>>
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.open-std.org%2Fmailman%2Flistinfo%2Ftooling&data=02%7C01%7Cgdr%40microsoft.com%7C1cdb6c203df64eb7b84408d6a38ea3ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636876230368116236&sdata=ugzbdk5BI%2Bppnt5nS0xDISJehZ3mizhZTBWDrVZZ%2BA8%3D&reserved=0
> >>
> >> _______________________________________________
> >> Modules mailing list
> >> Modules_at_[hidden]
> >> Subscription:
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fmodules&data=02%7C01%7Cgdr%40microsoft.com%7C1cdb6c203df64eb7b84408d6a38ea3ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636876230368116236&sdata=UPJ70IqxSq6PeLpRdMc7IZCotDOeyV6n7QNxOXjYFQo%3D&reserved=0
> >> Link to this post:
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fmodules%2F2019%2F03%2F0218.php&data=02%7C01%7Cgdr%40microsoft.com%7C1cdb6c203df64eb7b84408d6a38ea3ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636876230368116236&sdata=%2BsOB4DmF0Lf6EPmsPij3i%2BsDHDZS3ujHN4S9rVjMer4%3D&reserved=0
> > _______________________________________________
> > Modules mailing list
> > Modules_at_[hidden]
> > Subscription: http://lists.isocpp.org/mailman/listinfo.cgi/modules
> > Link to this post: http://lists.isocpp.org/modules/2019/03/0220.php
>
>
> _______________________________________________
> Modules mailing list
> Modules_at_[hidden]
> Subscription: http://lists.isocpp.org/mailman/listinfo.cgi/modules
> Link to this post: http://lists.isocpp.org/modules/2019/03/0226.php
>
Received on 2019-03-08 17:07:58