Date: Thu, 10 Sep 2020 13:24:41 -0400
On Thu, Sep 10, 2020 at 12:31 AM Tom Honermann via SG16 <
sg16_at_[hidden]> wrote:
> On 9/9/20 1:38 PM, Alisdair Meredith via SG16 wrote:
>
> Related to Tom’s pending paper, I am wondering if anyone with experience
> can report on how non-ASCII/unicode toolchains cope with open source
> projects, which are usually under source control that is encoded in something
> very much like ASCII/UTF-8?
>
> Do they maintain a parallel source tree that is transcoded?
>
> Generally, no.
> Do they maintain separate transcoded packages to distribute for their environment?
>
> For releases, I can't imagine why not. If you meant development source ,
then see the answer above.
> Are such environments naturally hesitant to use open source due to such
> obstacles, so the question rarely arises?
>
> No. The problems with source code encoding are mostly well understood and
not too difficult to work around.
Important note: If the source code and text resources use only characters
in Latin-1.
> Some other variation as my imagination is quite limited?
>
> The following mostly reflects my own experience working on z/OS 10+ years
> ago, but I'll conclude with some links that may be more representative of
> current practice. In general, I expect that existing practice is quite
> varied.
>
> Perhaps Hubert can add to this.
>
I can only add that open source on z/OS is still alive and well and people
seem to be able to use Git with tagged files just fine. Again, I am talking
about files that are Latin-1 compatible.
> The company that I worked for back then developed an application that was
> ported to Windows, a number of Linux and UNIX platforms, and the z/OS POSIX
> environment (called UNIX System Services back then). Most of our
> developers worked on Windows; I was part of a small team that supported the
> application on other platforms, including z/OS. Most automation was driven
> from Windows at the time, partially because we used Microsoft SourceSafe as
> our version control system (and enjoyed the downtime every weekend when
> access to SourceSafe was disabled while the database was rebuilt due to
> corruption that accumulated each week).
>
> The application was primarily C++ with a little Java added for flavor.
> The C++ code was built using IBM's xlC C++ compiler which required source
> files to be encoded in an EBCDIC code page.
>
> In order to build on z/OS and the other POSIX based platforms, we had a
> batch file called TarSrc.bat that used pax (not tar!) to create an archive
> of the project that was then transferred to the appropriate build system.
> The z/OS version of pax
> <https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.bpxa500/r4paxsh.htm>
> supports non-standard options (see the -o from=<codeset> and -o
> to=<codeset> options) to perform encoding conversion when creating or
> extracting archives. A shell script run on the build system then used pax
> to extract the project while converting files from ISO8859-1 (I think) to
> IBM-1047 (EBCDIC). The script would then un-convert the binary files
> contained in the package that should not have been converted. I believe
> that script was also responsible for performing line ending conversions
> (which also had to be careful not to mutate binary files). The build was
> then performed using a z/OS port of GNU make. Few people worked directly
> on z/OS. Those that did worked out their own system for updating source
> files on z/OS and then transferring them to a Windows system for checkin.
> This wasn't as horrible as it sounds (actually it was, I don't know why I
> just said that), TarSrc.bat, the shell script, and the build system
> supported incremental builds. The build system also allowed for one build
> environment to be shadowed by another such that portions of the source
> could be built with local modifications without having to perform a full
> build.
>
> We eventually moved from SourceSafe to ClearCase hosted on a UNIX
> platform. This was a major improvement. Sadly though, there was no z/OS
> ClearCase client so while we were able to eschew TarSrc.bat on most UNIX
> systems after that, we were still stuck with it for z/OS. In a desperate
> attempt to make development more uniform, I attempted to create a set of
> shell scripts that provided the appearance of a ClearCase client on z/OS.
> This was called KludgeCase. It relied on the ability of a UNIX system to
> export a ClearCase view over NFS and the ability for the z/OS NFS client to
> perform dynamic file conversions based on NFS mount options. The scripts
> that emulated the ClearCase cleartool utility on z/OS would transparently
> run remote commands on the UNIX system hosting the ClearCase view and
> (un)mount the view via NFS as necessary. This sufficed for files to be
> checked out and edited from z/OS just as on any other POSIX system. The
> problem was that it was very slow and I never found a good way to deal with
> binary files.
>
Yes, the NFS + symlink tree + binary patch-up and variations using pax,
etc. show up organically. Software developers are very good at
problem-solving.
> After IBM bought Rational, they added a z/OS ClearCase client that, as far
> as I could tell, was, and remains, completely useless. It seems to be
> designed around remote build processes (not unlike ye olde TarSrc.bat) and
> assumes that a z/OS developer's goal in life is to not actually interact
> with a z/OS system at all. But perhaps I'm being too harsh and critical; I
> haven't actually used it. I just knew that what I wanted was the same
> experience I was used to on other POSIX systems.
>
> Third party packages were treated the same as our own source code in most
> cases. We checked in the source and TarSrc.bat did the rest. Well, except
> for the part where we would have to go update that shell script to
> un-convert binary files that shouldn't have been converted.
>
> Today, git and github rule the roost. I don't have any experience using
> either on z/OS, but it is clear that others do. A z/OS port of git
> <https://www.rocketsoftware.com/product-categories/mainframe/git-for-zos>
> is available complete with instructions for how to use it with github
> with transparent file conversions
> <https://forum.rocketsoftware.com/t/using-git-for-z-os-with-github/654>.
> The solution appears to depend on use of git attributes to specify encoding
> of files.
>
> The z/OS filesystem allows tagging files with an encoding. The
> _BPX_AUTOCVT
> <https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxb200/bpxkenv.htm>
> or _BPXK_CCSIDS
> <https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxb200/bpxkenv.htm>
> environment variables can then be used to enable implicit file
> conversions. (There are a few other interesting Unicode related
> environment variables on those linked pages for anyone that is interested).
>
> Open source software has been used on z/OS since basically as long as open
> source has been a thing. Autoconf has supported z/OS since before I
> started using it (and since it was called OS/390; before the rename to
> z/OS). I'm not sure if CMake has been ported to z/OS or not.
>
Such porting (and also for ninja) is known to be possible.
> Tom.
>
> AlisdairM
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
>
sg16_at_[hidden]> wrote:
> On 9/9/20 1:38 PM, Alisdair Meredith via SG16 wrote:
>
> Related to Tom’s pending paper, I am wondering if anyone with experience
> can report on how non-ASCII/unicode toolchains cope with open source
> projects, which are usually under source control that is encoded in something
> very much like ASCII/UTF-8?
>
> Do they maintain a parallel source tree that is transcoded?
>
> Generally, no.
> Do they maintain separate transcoded packages to distribute for their environment?
>
> For releases, I can't imagine why not. If you meant development source ,
then see the answer above.
> Are such environments naturally hesitant to use open source due to such
> obstacles, so the question rarely arises?
>
> No. The problems with source code encoding are mostly well understood and
not too difficult to work around.
Important note: If the source code and text resources use only characters
in Latin-1.
> Some other variation as my imagination is quite limited?
>
> The following mostly reflects my own experience working on z/OS 10+ years
> ago, but I'll conclude with some links that may be more representative of
> current practice. In general, I expect that existing practice is quite
> varied.
>
> Perhaps Hubert can add to this.
>
I can only add that open source on z/OS is still alive and well and people
seem to be able to use Git with tagged files just fine. Again, I am talking
about files that are Latin-1 compatible.
> The company that I worked for back then developed an application that was
> ported to Windows, a number of Linux and UNIX platforms, and the z/OS POSIX
> environment (called UNIX System Services back then). Most of our
> developers worked on Windows; I was part of a small team that supported the
> application on other platforms, including z/OS. Most automation was driven
> from Windows at the time, partially because we used Microsoft SourceSafe as
> our version control system (and enjoyed the downtime every weekend when
> access to SourceSafe was disabled while the database was rebuilt due to
> corruption that accumulated each week).
>
> The application was primarily C++ with a little Java added for flavor.
> The C++ code was built using IBM's xlC C++ compiler which required source
> files to be encoded in an EBCDIC code page.
>
> In order to build on z/OS and the other POSIX based platforms, we had a
> batch file called TarSrc.bat that used pax (not tar!) to create an archive
> of the project that was then transferred to the appropriate build system.
> The z/OS version of pax
> <https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.bpxa500/r4paxsh.htm>
> supports non-standard options (see the -o from=<codeset> and -o
> to=<codeset> options) to perform encoding conversion when creating or
> extracting archives. A shell script run on the build system then used pax
> to extract the project while converting files from ISO8859-1 (I think) to
> IBM-1047 (EBCDIC). The script would then un-convert the binary files
> contained in the package that should not have been converted. I believe
> that script was also responsible for performing line ending conversions
> (which also had to be careful not to mutate binary files). The build was
> then performed using a z/OS port of GNU make. Few people worked directly
> on z/OS. Those that did worked out their own system for updating source
> files on z/OS and then transferring them to a Windows system for checkin.
> This wasn't as horrible as it sounds (actually it was, I don't know why I
> just said that), TarSrc.bat, the shell script, and the build system
> supported incremental builds. The build system also allowed for one build
> environment to be shadowed by another such that portions of the source
> could be built with local modifications without having to perform a full
> build.
>
> We eventually moved from SourceSafe to ClearCase hosted on a UNIX
> platform. This was a major improvement. Sadly though, there was no z/OS
> ClearCase client so while we were able to eschew TarSrc.bat on most UNIX
> systems after that, we were still stuck with it for z/OS. In a desperate
> attempt to make development more uniform, I attempted to create a set of
> shell scripts that provided the appearance of a ClearCase client on z/OS.
> This was called KludgeCase. It relied on the ability of a UNIX system to
> export a ClearCase view over NFS and the ability for the z/OS NFS client to
> perform dynamic file conversions based on NFS mount options. The scripts
> that emulated the ClearCase cleartool utility on z/OS would transparently
> run remote commands on the UNIX system hosting the ClearCase view and
> (un)mount the view via NFS as necessary. This sufficed for files to be
> checked out and edited from z/OS just as on any other POSIX system. The
> problem was that it was very slow and I never found a good way to deal with
> binary files.
>
Yes, the NFS + symlink tree + binary patch-up and variations using pax,
etc. show up organically. Software developers are very good at
problem-solving.
> After IBM bought Rational, they added a z/OS ClearCase client that, as far
> as I could tell, was, and remains, completely useless. It seems to be
> designed around remote build processes (not unlike ye olde TarSrc.bat) and
> assumes that a z/OS developer's goal in life is to not actually interact
> with a z/OS system at all. But perhaps I'm being too harsh and critical; I
> haven't actually used it. I just knew that what I wanted was the same
> experience I was used to on other POSIX systems.
>
> Third party packages were treated the same as our own source code in most
> cases. We checked in the source and TarSrc.bat did the rest. Well, except
> for the part where we would have to go update that shell script to
> un-convert binary files that shouldn't have been converted.
>
> Today, git and github rule the roost. I don't have any experience using
> either on z/OS, but it is clear that others do. A z/OS port of git
> <https://www.rocketsoftware.com/product-categories/mainframe/git-for-zos>
> is available complete with instructions for how to use it with github
> with transparent file conversions
> <https://forum.rocketsoftware.com/t/using-git-for-z-os-with-github/654>.
> The solution appears to depend on use of git attributes to specify encoding
> of files.
>
> The z/OS filesystem allows tagging files with an encoding. The
> _BPX_AUTOCVT
> <https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxb200/bpxkenv.htm>
> or _BPXK_CCSIDS
> <https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxb200/bpxkenv.htm>
> environment variables can then be used to enable implicit file
> conversions. (There are a few other interesting Unicode related
> environment variables on those linked pages for anyone that is interested).
>
> Open source software has been used on z/OS since basically as long as open
> source has been a thing. Autoconf has supported z/OS since before I
> started using it (and since it was called OS/390; before the rename to
> z/OS). I'm not sure if CMake has been ported to z/OS or not.
>
Such porting (and also for ninja) is known to be possible.
> Tom.
>
> AlisdairM
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
>
Received on 2020-09-10 12:28:28