Date: Tue, 19 Mar 2019 18:34:44 +0000
* outputs = > doesn't seem useful to me ?
What is the reasoning behind that?
* logical-provides => exported-modules ( there can not be more than 1)
True, but it can provides other files and other languages (e.g. Fortran) allow more than 1 – as Ben explained before. Furthermore, if people decide to have more than one module interface specification per file, the build system will already be set. I don’t want us to be repeating this painful discussion again.
From: Modules <modules-bounces_at_[hidden]> On Behalf Of Corentin
Sent: Monday, March 4, 2019 6:53 PM
To: WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
Cc: modules_at_[hidden]; brad.king_at_[hidden]
Subject: Re: [isocpp-modules] [Tooling] Dependency information for module-aware build tools
Quick suggestions:
outputs = > doesn't seem useful to me ?
logical-provides => exported-modules ( there can not be more than 1)
requires => required-modules
Version => only one field should be enough (with a separator)
As for utf8 I would maybe not care about it? It would make the file json-like rather than json, but it might be better to system-encode it? It's not meant to be shared
And pretending filename are Unicode consistently leads to pain, unfortunately.
I think the TR should address these issues at some point :)
Do we want to add a field for partitions specifically?
I would support multiple files, ie
{
files : [
"foo.cpp": {
"requires" : ...
}
]
}
I would add a field (globally for the file), that gives the basepath for relative paths
By default, I would make relatives paths relatives to the JSON file?
Maybe add a global list of include paths since build systems often have to extract that from the compiler?
On Tue, 5 Mar 2019 at 02:35 Steve Downey <sdowney_at_[hidden]<mailto:sdowney_at_[hidden]>> wrote:
I believe the intent is for the compiler to emit where it was told, implicitly or explicitly, where the output is to go, and what it believes are the dependencies. The problem is make, where exact spelling counts.
On Mon, Mar 4, 2019, 20:30 Mathias Stearn <redbeard0531+isocpp_at_[hidden]<mailto:redbeard0531%2Bisocpp_at_[hidden]>> wrote:
On Mon, Mar 4, 2019, 7:09 PM Ben Boeckel <ben.boeckel_at_[hidden]<mailto:ben.boeckel_at_[hidden]>> wrote:
Personally, my gut reaction is that this belongs at the build executor
level, not the compiler.
I don't think the build executor should need to know how to tell which changes within a BMI should or should not trigger downstream rebuilds.
Currently, most of these do mtime-based
detection. Easy to implement at the expense of potential excess work.
Also easy to force the executor to redo something. Smarter executors might do
content hashing detection of changes (I believe that bazel and similar
tools effectively do this). Really smart ones might support a
`content_hash = somecmd` to compute it for any output rule (possibly
with some built-in). This would also potentially work with things like
abidiff for object files and shared libraries too. I don't know how one
would implement `rebuild_hash` semantics without backdating mtimes in
existing executors which seems really finicky.
At least for ninja it is easy. You just have an intermediate output.hash file that everything depends on and you have a restat=1 rule that updates it only when needed. I've done this and it works quite well https://github.com/RedBeard0531/mongo_module_ninja/commit/f19916bbb1d2f7c8b18d39f38559de72ba486cd2#diff-2955d5257f635de1df53f55a171ca5c7<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRedBeard0531%2Fmongo_module_ninja%2Fcommit%2Ff19916bbb1d2f7c8b18d39f38559de72ba486cd2%23diff-2955d5257f635de1df53f55a171ca5c7&data=02%7C01%7Cgdr%40microsoft.com%7C45a9da6f16504d4a386d08d6ac8b4d96%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636886111646706313&sdata=3Xy5CS%2Blh5XdzWVDQUVXxqJsAyjnk9ogf09QkBpLN44%3D&reserved=0>.
I don't know that we can shoehorn this information into the output of
the scan step and expect it to actually be implemented before we know
what actually affects BMI output (cf. Clang assuming all flags affect
BMI output and not even "trying" to guess).
Sure. I was thinking that this would be in the output of every compiler invocation, whether scanning, extracting a BMI, or compiling to an object file. Each stage would output information relevant to itself, and needed for later stages. You will need to get deps during compilation, not just the earlier stages, since it is legal to #include in the private module fragment, which hopefully would never even be lexed by earlier invocations.
Also, I didn't notice this at first, but it looks like your proposed scan output is telling the build system where the compiler will be storing its outputs. It seems better to have it list the logical outputs it will generate, with later stages being told *exactly* where each file should be written. We already have a large problem with one case where the compiler decides on its own where to write files (split dwarf .dwo files) so I really don't want to introduce more cases like it. The build system tells the compiler where to put its output, not the other way around.
_______________________________________________
Modules mailing list
Modules_at_[hidden]pp.org<mailto:Modules_at_[hidden]>
Subscription: http://lists.isocpp.org/mailman/listinfo.cgi/modules<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fmodules&data=02%7C01%7Cgdr%40microsoft.com%7C45a9da6f16504d4a386d08d6ac8b4d96%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636886111646716322&sdata=9uE4r7mMdihTjUfVP8Zo%2B7hMPYW51%2BegqMrefxCjSVg%3D&reserved=0>
Link to this post: http://lists.isocpp.org/modules/2019/03/0128.php<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fmodules%2F2019%2F03%2F0128.php&data=02%7C01%7Cgdr%40microsoft.com%7C45a9da6f16504d4a386d08d6ac8b4d96%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636886111646726330&sdata=anMhUWUvf7TSTw%2B%2FuKTl9FT3ZLLUTIcN8v2fVb9ZIiQ%3D&reserved=0>
_______________________________________________
Tooling mailing list
Tooling_at_[hidden]rg<mailto:Tooling_at_[hidden]>
http://www.open-std.org/mailman/listinfo/tooling<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.open-std.org%2Fmailman%2Flistinfo%2Ftooling&data=02%7C01%7Cgdr%40microsoft.com%7C45a9da6f16504d4a386d08d6ac8b4d96%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636886111646726330&sdata=DjMfLTtgzUfllgYKYgEUG60ezS6PRhq1SFOGsuupkDM%3D&reserved=0>
What is the reasoning behind that?
* logical-provides => exported-modules ( there can not be more than 1)
True, but it can provides other files and other languages (e.g. Fortran) allow more than 1 – as Ben explained before. Furthermore, if people decide to have more than one module interface specification per file, the build system will already be set. I don’t want us to be repeating this painful discussion again.
From: Modules <modules-bounces_at_[hidden]> On Behalf Of Corentin
Sent: Monday, March 4, 2019 6:53 PM
To: WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
Cc: modules_at_[hidden]; brad.king_at_[hidden]
Subject: Re: [isocpp-modules] [Tooling] Dependency information for module-aware build tools
Quick suggestions:
outputs = > doesn't seem useful to me ?
logical-provides => exported-modules ( there can not be more than 1)
requires => required-modules
Version => only one field should be enough (with a separator)
As for utf8 I would maybe not care about it? It would make the file json-like rather than json, but it might be better to system-encode it? It's not meant to be shared
And pretending filename are Unicode consistently leads to pain, unfortunately.
I think the TR should address these issues at some point :)
Do we want to add a field for partitions specifically?
I would support multiple files, ie
{
files : [
"foo.cpp": {
"requires" : ...
}
]
}
I would add a field (globally for the file), that gives the basepath for relative paths
By default, I would make relatives paths relatives to the JSON file?
Maybe add a global list of include paths since build systems often have to extract that from the compiler?
On Tue, 5 Mar 2019 at 02:35 Steve Downey <sdowney_at_[hidden]<mailto:sdowney_at_[hidden]>> wrote:
I believe the intent is for the compiler to emit where it was told, implicitly or explicitly, where the output is to go, and what it believes are the dependencies. The problem is make, where exact spelling counts.
On Mon, Mar 4, 2019, 20:30 Mathias Stearn <redbeard0531+isocpp_at_[hidden]<mailto:redbeard0531%2Bisocpp_at_[hidden]>> wrote:
On Mon, Mar 4, 2019, 7:09 PM Ben Boeckel <ben.boeckel_at_[hidden]<mailto:ben.boeckel_at_[hidden]>> wrote:
Personally, my gut reaction is that this belongs at the build executor
level, not the compiler.
I don't think the build executor should need to know how to tell which changes within a BMI should or should not trigger downstream rebuilds.
Currently, most of these do mtime-based
detection. Easy to implement at the expense of potential excess work.
Also easy to force the executor to redo something. Smarter executors might do
content hashing detection of changes (I believe that bazel and similar
tools effectively do this). Really smart ones might support a
`content_hash = somecmd` to compute it for any output rule (possibly
with some built-in). This would also potentially work with things like
abidiff for object files and shared libraries too. I don't know how one
would implement `rebuild_hash` semantics without backdating mtimes in
existing executors which seems really finicky.
At least for ninja it is easy. You just have an intermediate output.hash file that everything depends on and you have a restat=1 rule that updates it only when needed. I've done this and it works quite well https://github.com/RedBeard0531/mongo_module_ninja/commit/f19916bbb1d2f7c8b18d39f38559de72ba486cd2#diff-2955d5257f635de1df53f55a171ca5c7<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRedBeard0531%2Fmongo_module_ninja%2Fcommit%2Ff19916bbb1d2f7c8b18d39f38559de72ba486cd2%23diff-2955d5257f635de1df53f55a171ca5c7&data=02%7C01%7Cgdr%40microsoft.com%7C45a9da6f16504d4a386d08d6ac8b4d96%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636886111646706313&sdata=3Xy5CS%2Blh5XdzWVDQUVXxqJsAyjnk9ogf09QkBpLN44%3D&reserved=0>.
I don't know that we can shoehorn this information into the output of
the scan step and expect it to actually be implemented before we know
what actually affects BMI output (cf. Clang assuming all flags affect
BMI output and not even "trying" to guess).
Sure. I was thinking that this would be in the output of every compiler invocation, whether scanning, extracting a BMI, or compiling to an object file. Each stage would output information relevant to itself, and needed for later stages. You will need to get deps during compilation, not just the earlier stages, since it is legal to #include in the private module fragment, which hopefully would never even be lexed by earlier invocations.
Also, I didn't notice this at first, but it looks like your proposed scan output is telling the build system where the compiler will be storing its outputs. It seems better to have it list the logical outputs it will generate, with later stages being told *exactly* where each file should be written. We already have a large problem with one case where the compiler decides on its own where to write files (split dwarf .dwo files) so I really don't want to introduce more cases like it. The build system tells the compiler where to put its output, not the other way around.
_______________________________________________
Modules mailing list
Modules_at_[hidden]pp.org<mailto:Modules_at_[hidden]>
Subscription: http://lists.isocpp.org/mailman/listinfo.cgi/modules<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fmodules&data=02%7C01%7Cgdr%40microsoft.com%7C45a9da6f16504d4a386d08d6ac8b4d96%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636886111646716322&sdata=9uE4r7mMdihTjUfVP8Zo%2B7hMPYW51%2BegqMrefxCjSVg%3D&reserved=0>
Link to this post: http://lists.isocpp.org/modules/2019/03/0128.php<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fmodules%2F2019%2F03%2F0128.php&data=02%7C01%7Cgdr%40microsoft.com%7C45a9da6f16504d4a386d08d6ac8b4d96%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636886111646726330&sdata=anMhUWUvf7TSTw%2B%2FuKTl9FT3ZLLUTIcN8v2fVb9ZIiQ%3D&reserved=0>
_______________________________________________
Tooling mailing list
Tooling_at_[hidden]rg<mailto:Tooling_at_[hidden]>
http://www.open-std.org/mailman/listinfo/tooling<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.open-std.org%2Fmailman%2Flistinfo%2Ftooling&data=02%7C01%7Cgdr%40microsoft.com%7C45a9da6f16504d4a386d08d6ac8b4d96%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636886111646726330&sdata=DjMfLTtgzUfllgYKYgEUG60ezS6PRhq1SFOGsuupkDM%3D&reserved=0>
Received on 2019-03-19 19:34:48