On Aug 11, 2019, at 10:34 AM, David Blaikie via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>> wrote:

Given that at least Clang's module binary format is designed for lazy loading - copying this data around rather than being able to share constant memory mapped files between multiple compilations on the same system seems like it would be a loss (or at best "no better" - the compiler would read all the data into memory and then treat it the same as if it'd mapped it off disk).

Same for MSVC.



What sort of filesystem race conditions do you have in mind?

On Sun, Aug 11, 2019 at 8:37 AM Lyberta via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>> wrote:
Another extension I would like to propose is to use module mapper to
transfer BMIs directly without needing file system.

In p1184 4.2 change the syntax to something like this:

Query from compiler to build system:

IMPORT $module

Reply from build system:

OK PATH $bmipath

Then compiler uses file system as usual.

Or:

OK BLOB $bmicontents

Then build system transfers raw bytes of BMI straight into compiler. The
format of $bmicontents could be something simple like unsigned 64 bit
integer holding the size (assuming BMIs larger than 4 GiB) followed by
raw bytes.

Or:

ERROR $msg

Next, query from compiler to build system:

EXPORT $module

Reply from build system:

OK DEFAULT

Then compiler does its default behavior which can be set via command
line flags.

Or:

OK PATH $bmipath

Then compiler writes the BMI to the given file system path. The response
can be bikeshedded.

Or:

OK BLOB

Then compiler transfers BMI directly to the build system. The response
can be bikeshedded.

I think this design avoids file system races and can make distributed
builds faster because file system is usually slower than sockets and
keeping data in RAM.

Later we can expand this to a general purpose communication channel and
avoid file system altogether by transferring source files and object
files via sockets.

_______________________________________________
SG15 mailing list
SG15_at_[hidden]<mailto:SG15_at_[hidden]>
https://lists.isocpp.org/mailman/listinfo.cgi/sg15<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fsg15&data=02%7C01%7Cgdr%40microsoft.com%7C3dc94ff8fa7240ce502908d71e8230db%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637011416832497154&sdata=8R1oL%2BBOsLBYJ2X3yhVsHaVADDg6%2BKUxH2BlwzGp%2FG4%3D&reserved=0>
_______________________________________________
SG15 mailing list
SG15_at_[hidden]<mailto:SG15_at_[hidden]pp.org>
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fsg15&amp;data=02%7C01%7Cgdr%40microsoft.com%7C3dc94ff8fa7240ce502908d71e8230db%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637011416832527138&amp;sdata=TqDa3vvey6DHBH7Dm6HPhzo5gF6kJwnpX3yhiJpgOqU%3D&amp;reserved=0