C++ Logo

sg15

Advanced search

Re: [Tooling] [isocpp-modules] Round2: Path to modules with old bad build systems

From: Gabriel Dos Reis <gdr_at_[hidden]>
Date: Tue, 5 Mar 2019 17:18:45 +0000


> On Mar 5, 2019, at 9:10 AM, Ben Craig <ben.craig_at_[hidden].com> wrote:
>
> Here are two use cases (that I've mentioned before) for a flattened file format that doesn't need to be parse-able by third party tools
> 1) Distributed build systems like distcc and FASTBuild that currently preprocess a file and send the file over the network to be compiled. These tools don't currently have a way to reach back over the network to pick up artifacts on the host machine.
> 2) Object caching tools, like ccache that use the preprocessed output of a file as a key in a map from preprocessed output to built object file.

I acknowledge all those scenarios. My real question is why the compiler has to go through the trouble of emitting such text, as opposed to being invoked on the other end with say -ftreat-imports-as-glorified headers? Tools like ccache couldn’t care less where the preprocessed file contains these magical pragmas or not.

— Gaby

>
>> -----Original Message-----
>> From: Gabriel Dos Reis <gdr_at_microsoft.com>
>> Sent: Tuesday, March 5, 2019 11:04 AM
>> To: Ben Craig <ben.craig_at_[hidden]>
>> Cc: modules_at_lists.isocpp.org; WG21 Tooling Study Group SG15
>> <tooling_at_open-std.org>; Nathan Sidwell <nathan_at_acm.org>
>> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules with old
>> bad build systems
>>
>> If the flattened file format doesn’t need to be parse-able by third party tools,
>> then why go through the trouble of generating it in the first place - requiring
>> additional language extensions at the source level?
>>
>> — Gaby
>>
>>> On Mar 5, 2019, at 8:43 AM, Ben Craig <ben.craig_at_[hidden]i.com> wrote:
>>>
>>> If the flattened file format doesn't need to be parse-able by third party
>> tools, then the compiler can do whatever it wants with the text format, so
>> long as the compiler itself can consume the resulting file on a different
>> machine. Abandoning the parse-able requirement means that you could
>> instead do...
>>>
>>> #pragma module begin(foo)
>>> /* base64 encoded implicitly generated BMI */ #pragma module end(foo)
>>>
>>> No cross-vendor spec would be necessary for such an approach.
>>>
>>>> -----Original Message-----
>>>> From: Modules <modules-bounces_at_[hidden]> On Behalf Of
>> Gabriel
>>>> Dos Reis via Modules
>>>> Sent: Tuesday, March 5, 2019 10:19 AM
>>>> To: modules_at_lists.isocpp.org
>>>> Cc: Gabriel Dos Reis <gdr_at_microsoft.com>; WG21 Tooling Study Group
>>>> SG15 <tooling_at_open-std.org>; Nathan Sidwell <nathan_at_acm.org>
>>>> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules with
>>>> old bad build systems
>>>>
>>>>
>>>>
>>>>> On Mar 5, 2019, at 8:12 AM, Ben Craig <ben.craig_at_ni.com> wrote:
>>>>>
>>>>> I will concede that static analysis tools and other tools that try
>>>>> to parse C++
>>>> probably don't need a textual inclusion format, since they most
>>>> likely need to be able to parse and understand the pragmas anyway.
>>>> If module mapping is sufficiently straightforward, then those tools
>>>> can do module lookup the same as a compiler. Those tools already
>>>> need to do include lookup in the same way that compilers do.
>>>>
>>>> I haven’t yet seen a mapping that isn’t straightforward :-)
>>>>
>>>>>
>>>>> I think the textual inclusion format will still be very useful to
>>>>> distribution
>>>> and caching tools though, as they don't need to understand the code.
>>>> Those tools frequently lean on the compiler's preprocessor today, and
>>>> don't know how to do include lookups.
>>>>
>>>> I don’t know there is an actual spec of such flattened text file that
>>>> isn’t at least as involved as the module spec - if not more involved.
>>>>
>>>> I worry that we may be creating a more complex problem than the issue
>>>> we are set to address.
>>>>
>>>> — Gaby
>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Nathan Sidwell <nathanmsidwell_at_gmail.com> On Behalf Of
>> Nathan
>>>>>> Sidwell
>>>>>> Sent: Tuesday, March 5, 2019 6:04 AM
>>>>>> To: Ben Craig <ben.craig_at_ni.com>; modules_at_lists.isocpp.org; WG21
>>>>>> Tooling Study Group SG15 <tooling_at_open-std.org>
>>>>>> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules
>>>>>> with old bad build systems
>>>>>>
>>>>>>> On 3/4/19 10:02 AM, Ben Craig wrote:
>>>>>>> I do mean textual inclusion, though I can be convinced otherwise.
>>>>>>> Textual
>>>>>> inclusion (with extra generated pragmas) should make it much easier
>>>>>> to keep tools like distcc and cppcheck happy in the short term. I
>>>>>> suspect that those tools don't want to crack open a BMI to figure
>>>>>> out which other BMIs need to be found.
>>>>>>>
>>>>>>> Tools that (think they can) parse C++ will still need to
>>>>>>> understand these
>>>>>> pragmas in order to provide the right macro, visibility, and
>>>>>> reachability behaviors, so some work will still be required on
>>>>>> their part, but at least they won't need to understand new binary
>> formats.
>>>>>>
>>>>>> Correct, tools consuming such #pragma-marked flattened source will
>>>>>> need to understand modules at a fundamental level. As such, why
>>>>>> not implement the same mechanisms to find module source as the
>> compiler?
>>>>>> That'll give them more information to perform code analysis with.
>>>>>>
>>>>>> nathan
>>>>>>
>>>>>>>>> On 3/2/19 1:03 PM, Ben Craig wrote:
>>>>>>>>>
>>>>>>>>> Some quick notes on this implementation strategy:
>>>>>>>>> * Uses TEXTUAL inclusion
>>>>>>>>> * Compiler assumes that the build system knows nothing of BMIs
>>>>>>>>> * Compiler needs to be able to do module mapping with minimal
>>>>>>>>> input from users.
>>>>>>>>
>>>>>>>> Do you literally mean textual inclusion or do you really mean
>>>>>>>> dynamically produce an internal-only BMI object?
>>>>>>>>
>>>>>>>> nathan
>>>>>>>>
>>>>>>>> --
>>>>>>>> Nathan Sidwell
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Nathan Sidwell
>>>>> _______________________________________________
>>>>> Modules mailing list
>>>>> Modules_at_lists.isocpp.org
>>>>> Subscription:
>>>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-&amp;data=02%7C01%7Cgdr%40microsoft.com%7Cbb5ce952bffc482e87b108d6a18d66a7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636874026040110157&amp;sdata=B7tMwaCrtynz2Hru8w%2B%2FilQOK7FRTTFRGCvUnzQRBvI%3D&amp;reserved=0
>> 3A__nam06.safelinks
>>>>> .protection.outlook.com_-3Furl-3Dhttps-253A-252F-
>> 252Furldefense.proo
>>>>> fpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D3A-5F-
>> 5Fnam06.safelinks.
>>>>> p-26amp-3Bdata-3D02-257C01-257Cgdr-2540microsoft.com-
>> 257Cb10248c40d1
>>>>> 04406a58008d6a1899f80-257C72f988bf86f141af91ab2d7cd011db47-
>> 257C1-257
>>>>> C0-257C636874009800024740-26amp-3Bsdata-
>> 3Dp7A5pBCX07tWsIhQL6SmfcP6iu
>>>>> nVFWqvUY1vKEw69t8-253D-26amp-3Breserved-
>> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LM
>>>>> TVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
>> UCZRX0Vl1g&m=OVMNvDsyTU
>>>>> HgfVyzp-kJjPXybT2k8OAlWfIf-It-W20&s=ceh-
>> yGW5IKak5viXjbJplZ1eOuUh8QuG
>>>>> -ECUU_y4QqY&e=
>>>>> rotection.outlook.com_-3Furl-3Dhttp-253A-252F-252Flists.isocpp.org-2
>>>>> 52
>>>>> Fmailman-252Flistinfo.cgi-252Fmodules-26amp-3Bdata-3D02-257C01-
>>>> 257Cgdr
>>>>> -2540microsoft.com-257Ca5f3d89c3d7d4346707f08d6a1854be8-
>>>> 257C72f988bf86
>>>>> f141af91ab2d7cd011db47-257C1-257C0-257C636873991217371177-
>> 26amp-
>>>> 3Bsdat
>>>>> a-3DRwfoGFvU8Df-252F2llvXjbpAiIIG-252B0RxL4Su8ewJeoypCw-253D-
>>>> 26amp-3Br
>>>>> eserved-
>>>> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y
>>>>> 8mub81SfUi-
>>>>
>> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
>>>> s=
>>>>> ofQHenjonX7d85yaK2-6F3x_myxVyMcitwJo9rUo8gI&e=
>>>>> Link to this post:
>>>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-&amp;data=02%7C01%7Cgdr%40microsoft.com%7Cbb5ce952bffc482e87b108d6a18d66a7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636874026040110157&amp;sdata=B7tMwaCrtynz2Hru8w%2B%2FilQOK7FRTTFRGCvUnzQRBvI%3D&amp;reserved=0
>> 3A__nam06.safelinks
>>>>> .protection.outlook.com_-3Furl-3Dhttps-253A-252F-
>> 252Furldefense.proo
>>>>> fpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D3A-5F-
>> 5Fnam06.safelinks.
>>>>> p-26amp-3Bdata-3D02-257C01-257Cgdr-2540microsoft.com-
>> 257Cb10248c40d1
>>>>> 04406a58008d6a1899f80-257C72f988bf86f141af91ab2d7cd011db47-
>> 257C1-257
>>>>> C0-257C636874009800024740-26amp-3Bsdata-
>> 3Dp7A5pBCX07tWsIhQL6SmfcP6iu
>>>>> nVFWqvUY1vKEw69t8-253D-26amp-3Breserved-
>> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LM
>>>>> TVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
>> UCZRX0Vl1g&m=OVMNvDsyTU
>>>>> HgfVyzp-kJjPXybT2k8OAlWfIf-It-W20&s=ceh-
>> yGW5IKak5viXjbJplZ1eOuUh8QuG
>>>>> -ECUU_y4QqY&e=
>>>>> rotection.outlook.com_-3Furl-3Dhttp-253A-252F-252Flists.isocpp.org-2
>>>>> 52
>>>>> Fmodules-252F2019-252F03-252F0145.php-26amp-3Bdata-3D02-257C01-
>>>> 257Cgdr
>>>>> -2540microsoft.com-257Ca5f3d89c3d7d4346707f08d6a1854be8-
>>>> 257C72f988bf86
>>>>> f141af91ab2d7cd011db47-257C1-257C0-257C636873991217371177-
>> 26amp-
>>>> 3Bsdat
>>>>> a-3DOuOtS6D49Sy5EhRl8VUpv-252ByE5esxGZ7dvPvf-252Be1zr1c-253D-
>>>> 26amp-3Br
>>>>> eserved-
>>>> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y
>>>>> 8mub81SfUi-
>>>>
>> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
>>>> s=
>>>>> sg8E8Gx-jIbRDIRalv20OlbIkHaNJbyC3TwX8-ueRDU&e=
>>>> _______________________________________________
>>>> Modules mailing list
>>>> Modules_at_[hidden]
>>>> Subscription:
>>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__nam06.safelinks&amp;data=02%7C01%7Cgdr%40microsoft.com%7Cbb5ce952bffc482e87b108d6a18d66a7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636874026040110157&amp;sdata=ypcf1hy9wb9Y5IsJTmgg0OJiClBGm2rlS4ryjKxaEiI%3D&amp;reserved=0.
>>>> protection.outlook.com_-3Furl-3Dhttps-253A-252F-
>> 252Furldefense.proofp
>>>> oint.com-252Fv2-252Furl-253Fu-253Dhttp-2D-26amp-3Bdata-3D02-
>> 257C01-25
>>>> 7Cgdr-2540microsoft.com-257Cb10248c40d104406a58008d6a1899f80-
>> 257C72f9
>>>> 88bf86f141af91ab2d7cd011db47-257C1-257C0-257C636874009800024740-
>> 26amp
>>>> -3Bsdata-3DvJxYGAlfPqfe-252BB6HOBD4JCjOZSEP9Sc1fKDJt8ehEu4-253D-
>> 26amp
>>>> -3Breserved-
>> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjix
>>>> A&r=y8mub81SfUi-UCZRX0Vl1g&m=OVMNvDsyTUHgfVyzp-
>> kJjPXybT2k8OAlWfIf-It-
>>>> W20&s=LGVTU9QiQe6-6qXpL9XYoNaa4f0qPN5ydsuaoCSBxQU&e=
>>>>
>> 3A__lists.isocpp.org_mailman_listinfo.cgi_modules&d=DwIGaQ&c=I_0YwoK
>>>> y7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
>>>>
>> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
>>>> s=gmG6dYi-y2pNIFyGrUsRGlDKCKRKWf4WQQlvzQDWM-s&e=
>>>> Link to this post:
>>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__nam06.safelinks&amp;data=02%7C01%7Cgdr%40microsoft.com%7Cbb5ce952bffc482e87b108d6a18d66a7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636874026040110157&amp;sdata=ypcf1hy9wb9Y5IsJTmgg0OJiClBGm2rlS4ryjKxaEiI%3D&amp;reserved=0.
>>>> protection.outlook.com_-3Furl-3Dhttps-253A-252F-
>> 252Furldefense.proofp
>>>> oint.com-252Fv2-252Furl-253Fu-253Dhttp-2D-26amp-3Bdata-3D02-
>> 257C01-25
>>>> 7Cgdr-2540microsoft.com-257Cb10248c40d104406a58008d6a1899f80-
>> 257C72f9
>>>> 88bf86f141af91ab2d7cd011db47-257C1-257C0-257C636874009800034748-
>> 26amp
>>>> -3Bsdata-3D123o9Z3tNcVY-252FTsR6VbHITcMCHsPMk3459OhqCqpGwk-
>> 253D-26amp
>>>> -3Breserved-
>> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjix
>>>> A&r=y8mub81SfUi-UCZRX0Vl1g&m=OVMNvDsyTUHgfVyzp-
>> kJjPXybT2k8OAlWfIf-It-
>>>> W20&s=Kjdm4tjilk-j7xq9maoXFFiZ0PeL2iBPixAyNRRBc4s&e=
>>>>
>> 3A__lists.isocpp.org_modules_2019_03_0147.php&d=DwIGaQ&c=I_0YwoKy
>>>> 7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
>>>>
>> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
>>>> s=51GVUSm-w3aQdhBiPoTvrNnOE6QlopXYmzwNAYDSU5o&e=

Received on 2019-03-05 18:18:49