Received: from CY4PR21MB0837.namprd21.prod.outlook.com (2603:10b6:301:2a::42) by MWHPR21MB0848.namprd21.prod.outlook.com with HTTPS via MWHPR1001CA0029.NAMPRD10.PROD.OUTLOOK.COM; Fri, 24 May 2019 02:44:15 +0000 Received: from MWHPR21CA0027.namprd21.prod.outlook.com (2603:10b6:300:129::13) by CY4PR21MB0837.namprd21.prod.outlook.com (2603:10b6:903:b8::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1943.3; Fri, 24 May 2019 02:44:13 +0000 Received: from BL2NAM06FT012.Eop-nam06.prod.protection.outlook.com (2a01:111:f400:7e55::207) by MWHPR21CA0027.outlook.office365.com (2603:10b6:300:129::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1943.4 via Frontend Transport; Fri, 24 May 2019 02:44:13 +0000 Received: from www.open-std.org (93.90.116.65) by BL2NAM06FT012.mail.protection.outlook.com (10.152.107.7) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.20.1949.0 via Frontend Transport; Fri, 24 May 2019 02:44:12 +0000 Received: from open-std.org (vps-93-90-116-65 [127.0.0.1]) by www.open-std.org (Postfix) with ESMTP id 5200D9DB181; Fri, 24 May 2019 04:44:09 +0200 (CEST) Received: from mx0b-00010702.pphosted.com (mx0b-00010702.pphosted.com [148.163.158.57]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by www.open-std.org (Postfix) with ESMTP id 3F7D69DB17F for ; Fri, 24 May 2019 04:44:07 +0200 (CEST) Received: from pps.filterd (m0098779.ppops.net [127.0.0.1]) by mx0b-00010702.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4O26TbJ024472; Thu, 23 May 2019 21:12:49 -0500 Received: from nam03-dm3-obe.outbound.protection.outlook.com (mail-dm3nam03lp2054.outbound.protection.outlook.com [104.47.41.54]) by mx0b-00010702.pphosted.com with ESMTP id 2sp3f18n23-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 23 May 2019 21:12:48 -0500 Received: from DM6PR04MB4620.namprd04.prod.outlook.com (20.176.105.209) by DM6PR04MB5819.namprd04.prod.outlook.com (20.179.49.12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1922.16; Fri, 24 May 2019 02:12:46 +0000 Received: from DM6PR04MB4620.namprd04.prod.outlook.com ([fe80::d5fc:915e:dc2c:ffe2]) by DM6PR04MB4620.namprd04.prod.outlook.com ([fe80::d5fc:915e:dc2c:ffe2%3]) with mapi id 15.20.1922.018; Fri, 24 May 2019 02:12:46 +0000 From: Ben Craig To: WG21 Tooling Study Group SG15 , Gabriel Dos Reis , Anna Gringauze , Lukasz Mendakiewicz , Cameron DaCamara Subject: Re: [Tooling] BMI distribution and reading BMI data Thread-Topic: BMI distribution and reading BMI data Thread-Index: AdURxRLMIa05M4YDQEC2B1T129nTiQAERpo/ Sender: "tooling-bounces@open-std.org" Date: Fri, 24 May 2019 02:12:46 +0000 Message-ID: References: List-Help: List-Subscribe: , List-Unsubscribe: , In-Reply-To: Reply-To: WG21 Tooling Study Group SG15 Accept-Language: en-US Content-Language: en-US X-MS-Exchange-Organization-AuthSource: BL2NAM06FT012.Eop-nam06.prod.protection.outlook.com X-MS-Has-Attach: yes X-Auto-Response-Suppress: All X-MS-Exchange-Organization-Network-Message-Id: bc566f4d-0ad2-4fc6-b267-08d6dff1b4ca X-MS-TNEF-Correlator: X-MS-Exchange-Organization-RecordReviewCfmType: 0 received-spf: None (protection.outlook.com: ni.com does not designate permitted sender hosts) x-originating-ip: [40.90.243.52] dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nio365.onmicrosoft.com; s=selector1-nio365-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NrtDR4pMd/oLhVKibEIWT1o0fVKK822XbAUVGUWoTZQ=; b=ft8QkXUF7gqZZ5oSu34b0U0c0G8C9bOXA2g4e/z9x4/szBoodDSqAlSzDkEKOt7kivoQpaVOyT0TnJ+hbGVEOW9Vcf/TVyHpl4MGFECPdN5lBxE0/+OuS5j/q62uRiEoIyiauqDeBMzt8Yhov6A4teOHRsIXipP2qPYCGoJ5PIk= x-ms-publictraffictype: Email x-ms-exchange-organization-originalclientipaddress: 93.90.116.65 x-ms-exchange-organization-originalserveripaddress: 10.152.107.7 errors-to: tooling-bounces@open-std.org x-proofpoint-virus-version: vendor=fsecure engine=2.50.10434:, , definitions=2019-05-23_18:, , signatures=0 x-proofpoint-spam-details: rule=inbound_policy_notspam policy=inbound_policy score=30 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=30 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905240012 X-Microsoft-Antispam-Mailbox-Delivery: wl:1;pcwl:1;ucf:0;jmr:0;ex:0;auth:0;dest:I;ENG:(750119)(520011016)(520004050)(702028)(944504077)(944701077); X-Microsoft-Antispam-Message-Info: =?Windows-1252?Q?kQuy7mDkOptIFS70cesCA6VtGdazDugYtdxKJ0meEbY20CsviE2UEXtx?= =?Windows-1252?Q?4ZanGl5NOTR7MdVpCJm+uviWAyUmajrW2jirflbgl7prItkLEHbK3/ks?= =?Windows-1252?Q?YYA5b7l4hrEoOhgIx1V2dHxGJxT4lWuHG29fVx8vYucJM7pNyDho2cDw?= =?Windows-1252?Q?e6nl1tmQPZ/2s7o9V1X1g1JV2hxhp/ni2OvPllkwLBXJruj3Q6Vg9fta?= =?Windows-1252?Q?V8VK3AvwHR0bqJhFsZ/t3sIw2ORggIubzheImMHQ+kXENMG+M6YAz70X?= =?Windows-1252?Q?hE9pARsoZ2zk8fzQ3GbVDgn+cGUVOJkYnXEwwJX3eK2C+HZlh/Qe5k9g?= =?Windows-1252?Q?BHpdO2i8WL+Qqn29BK56NvMn1dBvASP4U2wFkYwXu9+uu0RCpXr28etn?= =?Windows-1252?Q?YhuPrARq8YCtgZmyVv3e/nac7NAqtXYynn2vVUTOCvyYTn86ByJwmhso?= =?Windows-1252?Q?3MS/J9xqmONqbOb50oXpF+8GISTQRdbK5VSEA02XK6rLVRHm4e1jkMqG?= =?Windows-1252?Q?YsVJNClr7qmmwUVOrttEazkm5LsANDoGe86wzTpySlIRvz/jDcxnEuiI?= =?Windows-1252?Q?gz+eFcD1OtjPLtlElPOwJxTKLpyH7Y05T2aqrTPhkQzU5SvnttW+Z9wE?= =?Windows-1252?Q?09DdgXNW8KgWyKiPwFFJM7Lf60utHuqwgKTQioNqmCpqCJFjvKnxCfdG?= =?Windows-1252?Q?gvbvmHr0iA5luZwyhHuef1mmTq4EV5UDJSMkGtttCtlfwjiz42CF+V0k?= =?Windows-1252?Q?vO91Tzi/tHalRXerCBSF+NuiQRkIvgnn6IDhY5Cu4vlu8c8jdzKknoiW?= =?Windows-1252?Q?7SBW/D/w994TKn8pDpsm4RVDthbHe09oLq/0W1sRRQnQfgn3MpWGwxpE?= =?Windows-1252?Q?CEs6/wExLuSN1yNOAO7qMzl8AKsZt/+8+lixAM2+hgARbSKbtljl1gjX?= =?Windows-1252?Q?ZlaJWX4JGq1qJJqraTGBb2aZj8xb334KxRSq+/LYhhkIu8FoLlOmrZgc?= =?Windows-1252?Q?YD9W6Z2Kqn7spOIQkol0d/2i+BY3A7Jr/jzzaHU/Tl/1imbiAl6QMZOR?= =?Windows-1252?Q?eyqO27vaFj3+whkaYJwNnS5A9PAJiLYdkmoBwU+l13ik39C7DsqnAN7L?= =?Windows-1252?Q?mHl8abSeF6X/sd51r2P8jjEZhl6mQqJ1c2B+a6YVJ20ycRxSuwvChRRE?= =?Windows-1252?Q?Icz8EbwbwcfpSkK3Q/Typ02noxcXhSt8vtP8UaK9RBBNR848c7y8PGFY?= =?Windows-1252?Q?HIP7Yw7ij+U9A4jmeS7XflAY9cl1LVw6SuBGsVPX?= Content-Type: multipart/mixed; boundary="_004_DM6PR04MB4620EA8609D9EF49043FF9368B020DM6PR04MB4620namp_" MIME-Version: 1.0 --_004_DM6PR04MB4620EA8609D9EF49043FF9368B020DM6PR04MB4620namp_ Content-Type: multipart/alternative; boundary="_000_DM6PR04MB4620EA8609D9EF49043FF9368B020DM6PR04MB4620namp_" --_000_DM6PR04MB4620EA8609D9EF49043FF9368B020DM6PR04MB4620namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable What are the current restrictions with regards to static libraries and link= time optimization? Do those restrictions also apply to modules and link ti= me optimizations? Get Outlook for Android _____ From: tooling-bounces@open-std.org on behalf= of Olga Arkhipova Sent: Thursday, May 23, 2019 7:30:57 PM To: Tooling@isocpp.open-std.org; Gabriel Dos Reis; Anna Gringauze; Lukasz M= endakiewicz; Cameron DaCamara Subject: [EXTERNAL] [Tooling] BMI distribution and reading BMI data Hi all, I=92d like to discuss the BMI usage and distribution topics =96 can we do o= n tomorrow=92s SG15 meeting? Or later? Thanks, Olga BMI distribution Currently, built modules (BMI) are very similar to static libraries from bu= ild perspective: 1. They are specific to the compiler version (i.e. can only be used b= y the compiler binary compatible with the one which produced them) 2. If they depend on other modules, their BMIs need to be present too= for successful build. 3. A number of compiler switches which were used to build the module = should match the compiler switches for the source which uses this module. So distribution of BMIs currently has similar limitations as the distributi= on of built static libraries: =95=95=95=95=95=95=95 has strict requirements on the compiler and other use= d libraries versions =95=95=95=95=95=95=95 limited to the platforms, architectures and #defines = it is built for. The BMI distribution definitely has performance advantage for the builds wh= ich meet all restrictions and requirements, i.e. the same ones which can us= e built static libraries. If BMI's restriction of the specific compiler and exact command line can be= weakened somehow, or at least some data can be extracted from all BMIs, th= e performance advantage of the BMI distribution can be wider. Scenarios where extracting at least some data from BMIs is needed VS instellisense (EDG) Visual Studio and VS Code support not only MSVC, but also clang and gcc. VS is using EDG compiler as intellisense engine, which currently supports M= SVC, Clang and gcc modes. As performance of EDG compilation is very critica= l, ideally, EDG should be able to use modules already built by MSVC, clang = and gcc. Linters (as-you-type code analysis) require additional data specified in th= e source (annotations, pragmas, attributes, contracts). Ideally, this infor= mation should be always be present in BMI, independent on whether the code = has been compiled for analysis or code generation. o Alternative: producing a new BMI for linters and IntelliSense, making i= t slower. Note: MSVC will have an option to include the original input source (not TU= -expanded) into the IFCs. Clang-cl and clang-gcc Currently, clang-cl is (almost) ABI compatible with cl. I believe the same = is true for clang and gcc. Should Clang-cl be able to use modules produced by MSVC and vice versa? Should Clang-gcc be able to use modules produced by gcc and vice versa? Build systems If BMIs are distributed together with their sources (like modules for MS st= andard libs) build systems might want to check if the available BMIs are ac= tually compatible with the current build settings and if not, produce a dif= ferent BMI from the source Static analysis (background code analysis, code analysis at build) Static analysis often requires additional data computed from the source whi= ch is normally not stored in the BMI. Such additional information is produc= ed by static analysis tools during a separate analysis phase of the module = and needs to be stored into a different BMI file. o Alternative 1: Always adding extra info which is not always needed is a= n unjustifiable performance expense. o Alternative 2: Adding this information to already created BMI file crea= tes build system complications. The format of additional information is not defined at this point =96 the t= ools decide how to read and write the data. We recommend storing the data i= n a way that can be consumed by other tools and compilers. Other scenarios? BMI data to extract At minimum, the following data should be extractable from any BMI =95=95=95=95=95=95=95 Module source file name/location (as it was during th= e module build) =95=95=95=95=95=95=95 Compilation options used to build BMI, especially the= ones which have to match in the source using this module (#defines, etc.) =95=95=95=95=95=95=95 Referenced modules and their BMIs (unless included in= to compilation options) =95=95=95=95=95=95=95 Static analysis data o Imported header units and the referenced header o Additional information stored by static analysis tools o Anything that is possible to extract from header files using a compiler= frontend =96 i.e. types, symbols, ASTs, source information, annotations (d= eclspecs), pragmas, attributes, etc (for consumption by code analysis) Should we encourage the compiler vendors to provide a way to do this for th= e BMIs they produce? --_000_DM6PR04MB4620EA8609D9EF49043FF9368B020DM6PR04MB4620namp_ Content-Type: text/html; charset="Windows-1252" Content-ID: <14703E42E5D90548A981534098C4AF28@namprd21.prod.outlook.com> Content-Transfer-Encoding: quoted-printable
What are the current restrictions with regards to static libraries and link= time optimization? Do those restrictions also apply to modules and link ti= me optimizations?

From: tooling-bounces@open-= std.org <tooling-bounces@open-std.org> on behalf of Olga Arkhipova &l= t;olgaark@microsoft.com>
Sent: Thursday, May 23, 2019 7:30:57 PM
To: Tooling@isocpp.open-std.org; Gabriel Dos Reis; Anna Gringauze; L= ukasz Mendakiewicz; Cameron DaCamara
Subject: [EXTERNAL] [Tooling] BMI distribution and reading BMI data<= /font>
 

Hi all,

I=92d like to discuss the BMI usage and distribution= topics =96 can we do on tomorrow=92s  SG15 meeting? Or later?

Thanks,

Olga

 

 

BMI distribution

 

Currently, built modules (BMI) are very similar to s= tatic libraries from build perspective:

1.&= nbsp;      They are specific to the = compiler version (i.e. can only be used by the compiler binary compatible w= ith the one which produced them)

2.&= nbsp;      If they depend on other m= odules, their BMIs need to be present too for successful build.

3.&= nbsp;      A number of compiler swit= ches which were used to build the module should match the compiler switches= for the source which uses this module.

 

So distribution of BMIs currently has similar limita= tions as the distribution of built static libraries:

=B7   &nbs= p;   has strict requirements on the compiler and other used= libraries versions

=B7   &nbs= p;   limited to the platforms, architectures and #defines i= t is built for.

 

The BMI distribution definitely has performance adva= ntage for the builds which meet all restrictions and requirements, i.e. the= same ones which can use built static libraries.

 

If BMI's restriction of the specific compiler and ex= act command line can be weakened somehow, or at least some data can be extr= acted from all BMIs, the performance advantage of the BMI distribution can = be wider.

 

Scenarios where extracting at least some data fro= m BMIs is needed

 

VS instellisense (EDG)

Visual Studio and VS Co= de support not only MSVC, but also clang and gcc.

VS is using EDG compile= r as intellisense engine, which currently supports MSVC, Clang and gcc mode= s. As performance of EDG compilation is very critical, ideally, EDG should = be able to use modules already built by MSVC, clang and gcc.

 

Linters (as-you-type co= de analysis) require additional data specified in the source (a= nnotations, pragmas, attributes, contracts). Ideally, this information shou= ld be always be present in BMI, independent on whether the code has been co= mpiled for analysis or code generation.

o   Alternat= ive: producing a new BMI for linters and IntelliSense, making it slower.

 

Note: MSVC will have an= option to include the original input source (not TU-expanded) into the IFC= s.

 

Clang-cl and clang-gcc

Currently, clang-cl is = (almost) ABI compatible with cl. I believe the same is true for clang and g= cc.

Should Clang-cl be able= to use modules produced by MSVC and vice versa?

Should Clang-gcc be abl= e to use modules produced by gcc and vice versa?

 

Build systems

If BMIs are distributed= together with their sources (like modules for MS standard libs) build syst= ems might want to check if the available BMIs are actually compatible with = the current build settings and if not, produce a different BMI from the source

 

Static= analysis (background code analysis, code analysis at build)

Static analysis often r= equires additional data computed from the source which is normally not stor= ed in the BMI. Such additional information is produced by static analysis t= ools during a separate analysis phase of the module and needs to be stored into a different BMI file.

o &= nbsp; Alternative 1: Always adding extra info which is not a= lways needed is an unjustifiable performance expense.

o &= nbsp; Alternative 2: Adding this information to already crea= ted BMI file creates build system complications.

The format of additiona= l information is not defined at this point =96 the tools decide how to read= and write the data. We recommend storing the data in a way that can be con= sumed by other tools and compilers.

 

 

Other scenarios?

 

BMI data to extract

 

At minimum, the following data should be extractable= from any BMI

 

=B7   &nbs= p;   Module source file name/location (as it was during the= module build)

=B7   &nbs= p;   Compilation options used to build BMI, especially the = ones which have to match in the source using this module (#defines, etc.)

=B7   &nbs= p;   Referenced modules and their BMIs (unless included int= o compilation options)

=B7   &nbs= p;   Static analysis data

o &= nbsp; Imported header units and the referenced header

o &= nbsp; Additional information stored by static analysis tools=

o &= nbsp; Anything that is possible to extract from header files= using a compiler frontend =96 i.e. types, symbols, ASTs, source informatio= n, annotations (declspecs), pragmas, attributes, etc (for consumption by co= de analysis)

 

Should we encourage the compiler vendors to provide = a way to do this for the BMIs they produce?

 

--_000_DM6PR04MB4620EA8609D9EF49043FF9368B020DM6PR04MB4620namp_-- --_004_DM6PR04MB4620EA8609D9EF49043FF9368B020DM6PR04MB4620namp_ Content-Type: text/plain; name="ATT00001.txt" Content-Description: ATT00001.txt Content-Disposition: attachment; filename="ATT00001.txt"; size=430; creation-date="Fri, 24 May 2019 02:44:15 GMT"; modification-date="Fri, 24 May 2019 02:44:15 GMT" Content-ID: <1DA5CFD8B53B4447A4DD84F1376E3C59@namprd21.prod.outlook.com> Content-Transfer-Encoding: base64 X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NClRvb2xpbmcg bWFpbGluZyBsaXN0DQpUb29saW5nQGlzb2NwcC5vcGVuLXN0ZC5vcmcNCmh0dHBzOi8vbmFtMDYu c2FmZWxpbmtzLnByb3RlY3Rpb24ub3V0bG9vay5jb20vP3VybD1odHRwJTNBJTJGJTJGd3d3Lm9w ZW4tc3RkLm9yZyUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRnRvb2xpbmcmYW1wO2RhdGE9MDIlN0Mw MSU3Q29sZ2FhcmslNDBtaWNyb3NvZnQuY29tJTdDYmM1NjZmNGQwYWQyNGZjNmIyNjcwOGQ2ZGZm MWI0Y2ElN0M3MmY5ODhiZjg2ZjE0MWFmOTFhYjJkN2NkMDExZGI0NyU3QzElN0MwJTdDNjM2OTQy NjI2NTU2MDgyOTg2JmFtcDtzZGF0YT0wRUNxaWFtSnhPRmN6eGdMM3RiTXFyR2NPOFdSYjZWY1Ql MkZRNnNMa3AyeXMlM0QmYW1wO3Jlc2VydmVkPTANCg== --_004_DM6PR04MB4620EA8609D9EF49043FF9368B020DM6PR04MB4620namp_--