On Thu, Sep 10, 2020 at 12:31 AM Tom Honermann via SG16 <sg16@lists.isocpp.org> wrote:
On 9/9/20 1:38 PM, Alisdair Meredith via SG16 wrote:
Related to Tom’s pending paper, I am wondering if anyone with experience
can report on how non-ASCII/unicode toolchains cope with open source
projects, which are usually under source control that is encoded in something
very much like ASCII/UTF-8?

Do they maintain a parallel source tree that is transcoded?
Generally, no.
Do they maintain separate transcoded packages to distribute for their environment?
For releases, I can't imagine why not. If you meant development source , then see the answer above.
Are such environments naturally hesitant to use open source due to such
obstacles, so the question rarely arises?
No. The problems with source code encoding are mostly well understood and not too difficult to work around.
Important note: If the source code and text resources use only characters in Latin-1.
Some other variation as my imagination is quite limited?

The following mostly reflects my own experience working on z/OS 10+ years ago, but I'll conclude with some links that may be more representative of current practice.  In general, I expect that existing practice is quite varied.

Perhaps Hubert can add to this.

I can only add that open source on z/OS is still alive and well and people seem to be able to use Git with tagged files just fine. Again, I am talking about files that are Latin-1 compatible.

The company that I worked for back then developed an application that was ported to Windows, a number of Linux and UNIX platforms, and the z/OS POSIX environment (called UNIX System Services back then).  Most of our developers worked on Windows; I was part of a small team that supported the application on other platforms, including z/OS.  Most automation was driven from Windows at the time, partially because we used Microsoft SourceSafe as our version control system (and enjoyed the downtime every weekend when access to SourceSafe was disabled while the database was rebuilt due to corruption that accumulated each week).

The application was primarily C++ with a little Java added for flavor.  The C++ code was built using IBM's xlC C++ compiler which required source files to be encoded in an EBCDIC code page.

In order to build on z/OS and the other POSIX based platforms, we had a batch file called TarSrc.bat that used pax (not tar!) to create an archive of the project that was then transferred to the appropriate build system.  The z/OS version of pax supports non-standard options (see the -o from=<codeset> and -o to=<codeset> options) to perform encoding conversion when creating or extracting archives.  A shell script run on the build system then used pax to extract the project while converting files from ISO8859-1 (I think) to IBM-1047 (EBCDIC).  The script would then un-convert the binary files contained in the package that should not have been converted.  I believe that script was also responsible for performing line ending conversions (which also had to be careful not to mutate binary files).  The build was then performed using a z/OS port of GNU make.  Few people worked directly on z/OS.  Those that did worked out their own system for updating source files on z/OS and then transferring them to a Windows system for checkin.  This wasn't as horrible as it sounds (actually it was, I don't know why I just said that), TarSrc.bat, the shell script, and the build system supported incremental builds.  The build system also allowed for one build environment to be shadowed by another such that portions of the source could be built with local modifications without having to perform a full build.

We eventually moved from SourceSafe to ClearCase hosted on a UNIX platform.  This was a major improvement.  Sadly though, there was no z/OS ClearCase client so while we were able to eschew TarSrc.bat on most UNIX systems after that, we were still stuck with it for z/OS.  In a desperate attempt to make development more uniform, I attempted to create a set of shell scripts that provided the appearance of a ClearCase client on z/OS.  This was called KludgeCase.  It relied on the ability of a UNIX system to export a ClearCase view over NFS and the ability for the z/OS NFS client to perform dynamic file conversions based on NFS mount options.  The scripts that emulated the ClearCase cleartool utility on z/OS would transparently run remote commands on the UNIX system hosting the ClearCase view and (un)mount the view via NFS as necessary.  This sufficed for files to be checked out and edited from z/OS just as on any other POSIX system.  The problem was that it was very slow and I never found a good way to deal with binary files.

Yes, the NFS + symlink tree + binary patch-up and variations using pax, etc. show up organically. Software developers are very good at problem-solving.

After IBM bought Rational, they added a z/OS ClearCase client that, as far as I could tell, was, and remains, completely useless.  It seems to be designed around remote build processes (not unlike ye olde TarSrc.bat) and assumes that a z/OS developer's goal in life is to not actually interact with a z/OS system at all.  But perhaps I'm being too harsh and critical; I haven't actually used it.  I just knew that what I wanted was the same experience I was used to on other POSIX systems.

Third party packages were treated the same as our own source code in most cases.  We checked in the source and TarSrc.bat did the rest.  Well, except for the part where we would have to go update that shell script to un-convert binary files that shouldn't have been converted.

Today, git and github rule the roost.  I don't have any experience using either on z/OS, but it is clear that others do.  A z/OS port of git is available complete with instructions for how to use it with github with transparent file conversions.  The solution appears to depend on use of git attributes to specify encoding of files.

The z/OS filesystem allows tagging files with an encoding.  The _BPX_AUTOCVT or _BPXK_CCSIDS environment variables can then be used to enable implicit file conversions.  (There are a few other interesting Unicode related environment variables on those linked pages for anyone that is interested).

Open source software has been used on z/OS since basically as long as open source has been a thing.  Autoconf has supported z/OS since before I started using it (and since it was called OS/390; before the rename to z/OS).  I'm not sure if CMake has been ported to z/OS or not.

Such porting (and also for ninja) is known to be possible.



SG16 mailing list