Welcome back David.

On Fri, Jan 15, 2021 at 11:26 AM David Lindelof <lindelof_at_[hidden]> wrote:

> Hey guys,
>
> It had been a while since I attended one of these calls and I’m amazed at
> the incredible work that’s been done. Quick question: what’s the preferred
> means of discussion/feedback? We used to have a slack channel but it seems
> abandoned now. I was curious to know more about the function
> differentiation mechanisms in Python among other things.
>
The best way is just to initiate a question her on the SG19 reflector. Then
someone familiar with the paper or the author will answer it. In specific
cases, they might invite you to joint authorship so you can actively help
to shape the future:)

Thank you.

>
>
> David Lindelöf, Ph.D.
> +41 (0)79 415 66 41
> http://davidlindelof.com
> Follow me on Twitter:
> http://twitter.com/dlindelof
>
>
> On 14 January 2021 at 22:33:45, Michael Wong via SG19 (
> sg19_at_[hidden]) wrote:
>
>
>
> On Wed, Jan 13, 2021 at 3:21 PM Michael Wong <fraggamuffin_at_[hidden]>
> wrote:
>
>> SG19 Machine Learning 2 hours. This session will focus on Differential
>> Calculus and reinforcement learning but with updates from all the others
>> optionally.
>>
>> Link to Automatic differentiation proposal:
>>
>>
>> https://docs.google.com/document/d/175wIm8o4BNGti0WLq8U6uZORegKVjmnpfc-_E8PoGS0/edit?ts=5fff27cd#heading=h.9ogkehmdmtel
>>
>> Hi,
>>
>> Michael Wong is inviting you to a scheduled Zoom meeting.
>>
>> Topic: SG19 monthly Dec 2020-Feb 2021
>> Time: Jan 14, 2020 02:00 PM Eastern Time (US and Canada)
>> Every month on the Second Thu, until Feb 11, 2021, 3 occurrence(s)
>> Dec 10, 2020 02:00 PM ET 1900 UTC
>> Jan 14, 2021 02:00 PM ET 1900 UTC
>> Feb 11, 2021 02:00 PM ET 1900 UTC
>> Please download and import the following iCalendar (.ics) files to
>> your
>> calendar system.
>> Monthly:
>>
>> https://iso.zoom.us/meeting/tJctf-2tpzotGNHL5pZqwtjELee0mcG2zzCi/ics?icsToken=98tyKuCrrjMuH92UtxuCRowqAoqgLO_xmH5ajY11sEr1OTFEdgnTGudHYr98N4rK
>>
>> Join from PC, Mac, Linux, iOS or Android:
>> https://iso.zoom.us/j/93084591725?pwd=K3QxZjJlcnljaE13ZWU5cTlLNkx0Zz09
>> Password: 035530
>>
>> Or iPhone one-tap :
>> US: +13017158592,,93084591725# or +13126266799,,93084591725#
>> Or Telephone:
>> Dial(for higher quality, dial a number based on your current
>> location):
>> US: +1 301 715 8592 or +1 312 626 6799 or +1 346 248 7799 or +1
>> 408 638 0968 or +1 646 876 9923 or +1 669 900 6833 or +1 253 215 8782
>> or 877 853 5247 (Toll Free)
>> Meeting ID: 930 8459 1725
>> Password: 035530
>> International numbers available: https://iso.zoom.us/u/agewu4X97
>>
>> Or Skype for Business (Lync):
>> https://iso.zoom.us/skype/93084591725
>>
>> Agenda:
>>
>> 1. Opening and introductions
>>
>> The ISO Code of conduct:
>> https://www.iso.org/files/live/sites/isoorg/files/store/en/PUB100397.pdf
>> The IEC Code of Conduct:
>>
>> https://basecamp.iec.ch/download/iec-code-of-conduct-for-delegates-and-experts/
>>
>> ISO patent policy.
>>
>> https://isotc.iso.org/livelink/livelink/fetch/2000/2122/3770791/Common_Policy.htm?nodeid=6344764&vernum=-2
>>
>> The WG21 Practices and Procedures and Code of Conduct:
>>
>> https://isocpp.org/std/standing-documents/sd-4-wg21-practices-and-procedures
>>
>> 1.1 Roll call of participants
>>
> Richard Dosselman, Phil ratzloff, Andrew Lumsdaine, Ayenem. David
> Lindelof, Cyril Khazan, Eugenio Bargiacchi, Jens Maurer, Joe Sachs, Kevin
> Deweese, Larry Lewis, marco foco, Ozan Irsoy, Scott Mcmillan, vassil
> vassilev, Will Wray
> Michael Wong. William Moses
>
>> 1.2 Adopt agenda
>>
> LA
>
> 1.3 Approve minutes from previous meeting, and approve publishing
>> previously approved minutes to ISOCPP.org
>>
>> 1.4 Action items from previous meetings
>>
>> 2. Main issues (125 min)
>>
>> 2.1 General logistics
>>
>> Meeting plan, focus on one paper per meeting but does not preclude other
>> paper updates:
>>
>> Dec 10, 2020 02:00 PM ET1900 UTC Stats DONE
>> Jan 14, 2021 02:00 PM ET 1900 UTCReinforcement Learning and Diff
>> Calculus
>> Feb 11, 2021 02:00 PM 1900 UTC ET Graph
>>
>> ISO meeting status
>>
>> future C++ Std meetings
>>
>> 2.2 Paper reviews
>>
>> 2.2.1: ML topics
>>
>> 2.2.1.1 Differential Calculs:
>>
>>
>> https://docs.google.com/document/d/175wIm8o4BNGti0WLq8U6uZORegKVjmnpfc-_E8PoGS0/edit?ts=5fff27cd#heading=h.9ogkehmdmtel
>>
>>
>> WM:
> Enzyme using IR and not AST
> differentiation is first class in other languages
> cant train the network without one part of loss function isnt AD compatible
> cannot fo BP
> so vast set of C++ codebases in ML
>
> numercal
> symbolic
> AD or algorithmic using chain rule
> will generate the dervative function
> can do non-closed form expressions
> impl exists:
> 1. use operator overloading, create a new differential double type and it
> is
> AOT compilation, will not work ifipow is not there already
> 2. source transformation like Tapenade
> 2 styles:
> 1. forward mode (1 input and multiple output)
> 2. reverse mode (multiple inputs and one output, gradienst)
>
> Dual numbers
> a+eB is a dual number
>
> needs to be native
> op overloading approach(adept) and source rewriters(OpenAD) rewritign
> original program to use that specific suset that is differentiable , JAX
> does this
>
> AD with language/compiler is efficient
> several proof of concept: enzyme and CLAD
> Enzyme can work wit hdifferent FEs
> for all functsions, will build backward pass
> now we can optimize this looks like hand compile derivatives
> combining AD with optimization
>
> runnign AD after optimization
> loop invariant code motion assuming no aliasing between out and in
> using restrict
>
> if you do AD then Code motion, it is O(Nsquare)
> if you do code motion then AD then it is O(N)
> ADBench from MS testd Enzyme
> shows this case
>
> C++ proposal
> support AD at a low level enables AD at high level
> so if u import eigen library
>
> AL: compose different library, compiler based
> will add burden on compiler impl
> start with minimal set that compiler differentiate
> then custom derivatives and generic functions
>
> impl complexity: reverse mode in clad is 2000 lines
>
>
> performance numbers with clad? coming
>
> less intrusive like library are evaluated and have less performance
>
> could be niche which make it tough sell
> use reflection to inspect ADT
> reflection does not work on statement and expressions
> P2040 reflxpr touched on expression, but SG7 seems to be opposed to
> exposing teh AST
> P2237
>
> Library solution would serve no purpose to community on existing
> codepeople moving from TF to
>
>
> people moving from C++ to Tf, TF to PT
>
> what about portable format for modules by Gaby Dos Reis some library
> overload the explicit conversion operat
>
> have we exhausted library apporach, as they are old
> but are there modern expression template technique
> Yes we explored lirbary, even latest one, like AOki which use shadow
> language
>
> eve library is higher bar and favors small library
>
> if we dont standardize it, someone will put it in llvm
>
> do not see modules supporting this, Expression template needs ADL and
> leaves out whole set of already written C++ code
> also issue with scalability like a chain of differentiation, and the space
> is large
> and will miss optimization opportunity
> might be mitigated by CSEE
>
> is there a small set of features that could be provided by compiler that
> could make an efficient library implementation? I think nonethat library
> could be close to compiler impl
> may be CSSE of templates
> yes I spent time to understand doing it as a libr with small set of
> features from teh compiler, but could not find any that work
>
> continue in this direction, publish the paper
>
>
>
> 2.2.1.2 Reinforcement Learning Larry Lewis Jorge Silva
>>
>> Reinforcement Learning proposal:
>>
> LL:
> :
> RI is dependent on LA, ML, NN, -> optimizer, AD, data loaders, LA, tensors
> may be better to focus on tensors
> mdspan and owning mdarray; but they are not enough for a good tensor
> library, no common tensor operations, just indexing, strides
> NN needs matrix mult
> NN needs AD to prevent handcoding BP, especially for CN when doing RGD
> theano has AD
> LA syntax and LA BLAS
> LA syntax adds a definition for arithmetic operators to create matrix and
> vector types, currently in LEWG , rebased to C++ 20 using requires clauses
> great for mathematics which reduces interface
> tensorflow/theano tensor looks more like python
> may have to work US NL physics tensors
> Heterogeneous tensorflow is currently built on top of SYCL, CUDA
> review xtensor, pytorch, and TF tensor ops
> optimzers, weights by Andrew Lumsdaine
> P1416 by xtensor people
> P1415
> wg21.link
>
> Data tables from Nvidia building data frame capability xdateframe?
>
>
>
> 2.2.1.3 Graph Proposal Phil Ratsloff et al
>>
>> P1709R1: Graph Proposal for Machine Learning
>>
>> P1709R3:
>>
>> https://docs.google.com/document/d/1kLHhbSTX7j0tPeTYECQFSNx3R35Mu3xO5_dyYdRy4dM/edit?usp=sharing
>>
>>
>> https://docs.google.com/document/d/1QkfDzGyfNQKs86y053M0YHOLP6frzhTJqzg1Ug_vkkE/edit?usp=sharing
>>
>> <http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2119r0.html>
>>
>
> split and reduce for C++26
> common api?
> Concept of algorithm
> reducing the number of types and number of functions leveraging what is in
> ranges
> looking at BGL concept are just handfull
> STL is just into concepts and algorithms
>
>
> 2.2.1.4: Stats paper
>>
>> Stats review Richard Dosselman et al
>>
>> P1708R3: Math proposal for Machine Learning: 3rd review
>>
>> PXXXX: combinatorics: 1st Review
>>
>> > std.org/jtc1/sc22/wg21/docs/papers/2020/p1708r2
>> > above is the stats paper that was reviewed in Prague
>> > http://wiki.edg.com/bin/view/Wg21prague/P1708R2SG19
>> >
>> > Review Jolanta Polish feedback.
>> > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2119r0.html
>>
>
>
> Comninatorics
> python has factorials, perm/comb is missing from our math library
> had identified the min P1415 overview document
> applications beyond SG19
> need large numerical type for factorials
> python has builtin wide integer type
>
> unbounded template T for a numeric type, we have failed to specify for the
> complex of T type for float and long double
> what operations are declared on that type, can't just say plug in any T
>
> these could also be in C without the templates, may also consider how to
> do it in C
>
>
>
>
> 2.2.3 any other proposal for reviews?
>>
>> 2.3 Other Papers and proposals
>>
>> P1416R1: SG19 - Linear Algebra for Data Science and Machine Learning
>>
>> https://docs.google.com/document/d/1IKUNiUhBgRURW-UkspK7fAAyIhfXuMxjk7xKikK4Yp8/edit#heading=h.tj9hitg7dbtr
>>
>> P1415: Machine Learning Layered list
>>
>> https://docs.google.com/document/d/1elNFdIXWoetbxjO1OKol_Wj8fyi4Z4hogfj5tLVSj64/edit#heading=h.tj9hitg7dbtr
>>
>> 2.2.2 SG14 Linear Algebra progress:
>> Different layers of proposal
>>
>> https://docs.google.com/document/d/1poXfr7mUPovJC9ZQ5SDVM_1Nb6oYAXlK_d0ljdUAtSQ/edit
>>
>> 2.5 Future F2F meetings:
>>
>> 2.6 future C++ Standard meetings:
>> https://isocpp.org/std/meetings-and-participation/upcoming-meetings
>>
>> None
>>
>> 3. Any other business
>>
>> New reflector
>>
>> http://lists.isocpp.org/mailman/listinfo.cgi/sg19
>>
>> Old Reflector
>> https://groups.google.com/a/isocpp.org/forum/#!newtopic/sg19
>> <https://groups.google.com/a/isocpp.org/forum/?fromgroups=#!forum/sg14>
>>
>> Code and proposal Staging area
>>
>> 4. Review
>>
>> 4.1 Review and approve resolutions and issues [e.g., changes to SG's
>> working draft]
>>
>> 4.2 Review action items (5 min)
>>
>> 5. Closing process
>>
>> 5.1 Establish next agenda
>>
>> TBD
>>
>> 5.2 Future meeting
>>
>>
>> Feb 11, 2021 02:00 PM 1900 UTC ET Graph paper
>>
> --
> SG19 mailing list
> SG19_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg19
>
>