Hey guys,It had been a while since I attended one of these calls and I’m amazed at the incredible work that’s been done. Quick question: what’s the preferred means of discussion/feedback? We used to have a slack channel but it seems abandoned now. I was curious to know more about the function differentiation mechanisms in Python among other things.
David Lindelöf, Ph.D.
+41 (0)79 415 66 41Follow me on Twitter:On 14 January 2021 at 22:33:45, Michael Wong via SG19 (sg19@lists.isocpp.org) wrote:
--On Wed, Jan 13, 2021 at 3:21 PM Michael Wong <fraggamuffin@gmail.com> wrote:SG19 Machine Learning 2 hours. This session will focus on Differential Calculus and reinforcement learning but with updates from all the others optionally.
Link to Automatic differentiation proposal:
Hi,
Michael Wong is inviting you to a scheduled Zoom meeting.
Topic: SG19 monthly Dec 2020-Feb 2021
Time: Jan 14, 2020 02:00 PM Eastern Time (US and Canada)
Every month on the Second Thu, until Feb 11, 2021, 3 occurrence(s)
Dec 10, 2020 02:00 PM ET 1900 UTC
Jan 14, 2021 02:00 PM ET 1900 UTC
Feb 11, 2021 02:00 PM ET 1900 UTC
Please download and import the following iCalendar (.ics) files to your
calendar system.
Monthly:
https://iso.zoom.us/meeting/tJctf-2tpzotGNHL5pZqwtjELee0mcG2zzCi/ics?icsToken=98tyKuCrrjMuH92UtxuCRowqAoqgLO_xmH5ajY11sEr1OTFEdgnTGudHYr98N4rKJoin from PC, Mac, Linux, iOS or Android:
https://iso.zoom.us/j/93084591725?pwd=K3QxZjJlcnljaE13ZWU5cTlLNkx0Zz09
Password: 035530Or iPhone one-tap :
US: +13017158592,,93084591725# or +13126266799,,93084591725#
Or Telephone:
Dial(for higher quality, dial a number based on your current location):
US: +1 301 715 8592 or +1 312 626 6799 or +1 346 248 7799 or +1
408 638 0968 or +1 646 876 9923 or +1 669 900 6833 or +1 253 215 8782
or 877 853 5247 (Toll Free)
Meeting ID: 930 8459 1725
Password: 035530
International numbers available: https://iso.zoom.us/u/agewu4X97Or Skype for Business (Lync):
https://iso.zoom.us/skype/93084591725Agenda:
1. Opening and introductions
The ISO Code of conduct:
https://www.iso.org/files/live/sites/isoorg/files/store/en/PUB100397.pdf
The IEC Code of Conduct:
https://basecamp.iec.ch/download/iec-code-of-conduct-for-delegates-and-experts/ISO patent policy.
https://isotc.iso.org/livelink/livelink/fetch/2000/2122/3770791/Common_Policy.htm?nodeid=6344764&vernum=-2The WG21 Practices and Procedures and Code of Conduct:
https://isocpp.org/std/standing-documents/sd-4-wg21-practices-and-procedures1.1 Roll call of participants
Richard Dosselman, Phil ratzloff, Andrew Lumsdaine, Ayenem. David Lindelof, Cyril Khazan, Eugenio Bargiacchi, Jens Maurer, Joe Sachs, Kevin Deweese, Larry Lewis, marco foco, Ozan Irsoy, Scott Mcmillan, vassil vassilev, Will WrayMichael Wong. William Moses1.2 Adopt agenda
LA1.3 Approve minutes from previous meeting, and approve publishing
previously approved minutes to ISOCPP.org1.4 Action items from previous meetings
2. Main issues (125 min)
2.1 General logistics
Meeting plan, focus on one paper per meeting but does not preclude other
paper updates:Dec 10, 2020 02:00 PM ET1900 UTC Stats DONE
Jan 14, 2021 02:00 PM ET 1900 UTCReinforcement Learning and Diff
Calculus
Feb 11, 2021 02:00 PM 1900 UTC ET GraphISO meeting status
future C++ Std meetings
2.2 Paper reviews
2.2.1: ML topics
2.2.1.1 Differential Calculs:
WM:Enzyme using IR and not ASTdifferentiation is first class in other languagescant train the network without one part of loss function isnt AD compatiblecannot fo BPso vast set of C++ codebases in MLnumercalsymbolicAD or algorithmic using chain rulewill generate the dervative functioncan do non-closed form expressionsimpl exists:1. use operator overloading, create a new differential double type and it isAOT compilation, will not work ifipow is not there already2. source transformation like Tapenade2 styles:1. forward mode (1 input and multiple output)2. reverse mode (multiple inputs and one output, gradienst)Dual numbersa+eB is a dual numberneeds to be nativeop overloading approach(adept) and source rewriters(OpenAD) rewritign original program to use that specific suset that is differentiable , JAX does thisAD with language/compiler is efficientseveral proof of concept: enzyme and CLADEnzyme can work wit hdifferent FEsfor all functsions, will build backward passnow we can optimize this looks like hand compile derivativescombining AD with optimizationrunnign AD after optimizationloop invariant code motion assuming no aliasing between out and inusing restrictif you do AD then Code motion, it is O(Nsquare)if you do code motion then AD then it is O(N)ADBench from MS testd Enzymeshows this caseC++ proposalsupport AD at a low level enables AD at high levelso if u import eigen libraryAL: compose different library, compiler basedwill add burden on compiler implstart with minimal set that compiler differentiatethen custom derivatives and generic functionsimpl complexity: reverse mode in clad is 2000 linesperformance numbers with clad? comingless intrusive like library are evaluated and have less performancecould be niche which make it tough selluse reflection to inspect ADTreflection does not work on statement and expressionsP2040 reflxpr touched on expression, but SG7 seems to be opposed to exposing teh ASTP2237Library solution would serve no purpose to community on existing codepeople moving from TF topeople moving from C++ to Tf, TF to PTwhat about portable format for modules by Gaby Dos Reis some library overload the explicit conversion operathave we exhausted library apporach, as they are oldbut are there modern expression template techniqueYes we explored lirbary, even latest one, like AOki which use shadow languageeve library is higher bar and favors small libraryif we dont standardize it, someone will put it in llvmdo not see modules supporting this, Expression template needs ADL and leaves out whole set of already written C++ codealso issue with scalability like a chain of differentiation, and the space is largeand will miss optimization opportunitymight be mitigated by CSEEis there a small set of features that could be provided by compiler that could make an efficient library implementation? I think nonethat library could be close to compiler implmay be CSSE of templatesyes I spent time to understand doing it as a libr with small set of features from teh compiler, but could not find any that workcontinue in this direction, publish the paper2.2.1.2 Reinforcement Learning Larry Lewis Jorge Silva
Reinforcement Learning proposal:
LL::RI is dependent on LA, ML, NN, -> optimizer, AD, data loaders, LA, tensorsmay be better to focus on tensorsmdspan and owning mdarray; but they are not enough for a good tensor library, no common tensor operations, just indexing, stridesNN needs matrix multNN needs AD to prevent handcoding BP, especially for CN when doing RGDtheano has ADLA syntax and LA BLASLA syntax adds a definition for arithmetic operators to create matrix and vector types, currently in LEWG , rebased to C++ 20 using requires clauses great for mathematics which reduces interfacetensorflow/theano tensor looks more like pythonmay have to work US NL physics tensorsHeterogeneous tensorflow is currently built on top of SYCL, CUDAreview xtensor, pytorch, and TF tensor opsoptimzers, weights by Andrew LumsdaineP1416 by xtensor peopleP1415wg21.linkData tables from Nvidia building data frame capability xdateframe?2.2.1.3 Graph Proposal Phil Ratsloff et al
P1709R1: Graph Proposal for Machine Learning
P1709R3:
https://docs.google.com/document/d/1kLHhbSTX7j0tPeTYECQFSNx3R35Mu3xO5_dyYdRy4dM/edit?usp=sharinghttps://docs.google.com/document/d/1QkfDzGyfNQKs86y053M0YHOLP6frzhTJqzg1Ug_vkkE/edit?usp=sharing
<http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2119r0.html>
split and reduce for C++26common api?Concept of algorithmreducing the number of types and number of functions leveraging what is in rangeslooking at BGL concept are just handfullSTL is just into concepts and algorithms2.2.1.4: Stats paper
Stats review Richard Dosselman et al
P1708R3: Math proposal for Machine Learning: 3rd review
PXXXX: combinatorics: 1st Review
> std.org/jtc1/sc22/wg21/docs/papers/2020/p1708r2
> above is the stats paper that was reviewed in Prague
> http://wiki.edg.com/bin/view/Wg21prague/P1708R2SG19
>
> Review Jolanta Polish feedback.
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2119r0.htmlComninatoricspython has factorials, perm/comb is missing from our math libraryhad identified the min P1415 overview documentapplications beyond SG19need large numerical type for factorialspython has builtin wide integer typeunbounded template T for a numeric type, we have failed to specify for the complex of T type for float and long doublewhat operations are declared on that type, can't just say plug in any Tthese could also be in C without the templates, may also consider how to do it in C2.2.3 any other proposal for reviews?
2.3 Other Papers and proposals
P1416R1: SG19 - Linear Algebra for Data Science and Machine Learning
https://docs.google.com/document/d/1IKUNiUhBgRURW-UkspK7fAAyIhfXuMxjk7xKikK4Yp8/edit#heading=h.tj9hitg7dbtrP1415: Machine Learning Layered list
https://docs.google.com/document/d/1elNFdIXWoetbxjO1OKol_Wj8fyi4Z4hogfj5tLVSj64/edit#heading=h.tj9hitg7dbtr2.2.2 SG14 Linear Algebra progress:
Different layers of proposal
https://docs.google.com/document/d/1poXfr7mUPovJC9ZQ5SDVM_1Nb6oYAXlK_d0ljdUAtSQ/edit2.5 Future F2F meetings:
2.6 future C++ Standard meetings:
https://isocpp.org/std/meetings-and-participation/upcoming-meetingsNone
3. Any other business
New reflector
http://lists.isocpp.org/mailman/listinfo.cgi/sg19
Old Reflector
https://groups.google.com/a/isocpp.org/forum/#!newtopic/sg19
<https://groups.google.com/a/isocpp.org/forum/?fromgroups=#!forum/sg14>Code and proposal Staging area
4. Review
4.1 Review and approve resolutions and issues [e.g., changes to SG's
working draft]4.2 Review action items (5 min)
5. Closing process
5.1 Establish next agenda
TBD
5.2 Future meeting
Feb 11, 2021 02:00 PM 1900 UTC ET Graph paper
SG19 mailing list
SG19@lists.isocpp.org
https://lists.isocpp.org/mailman/listinfo.cgi/sg19