Date: Thu, 10 Sep 2020 00:30:59 -0400
On 9/9/20 1:38 PM, Alisdair Meredith via SG16 wrote:
> Related to Tom’s pending paper, I am wondering if anyone with experience
> can report on how non-ASCII/unicode toolchains cope with open source
> projects, which are usually under source control that is encoded in something
> very much like ASCII/UTF-8?
>
> Do they maintain a parallel source tree that is transcoded?
> Do they maintain separate transcoded packages to distribute for their environment?
> Are such environments naturally hesitant to use open source due to such
> obstacles, so the question rarely arises?
> Some other variation as my imagination is quite limited?
The following mostly reflects my own experience working on z/OS 10+
years ago, but I'll conclude with some links that may be more
representative of current practice. In general, I expect that existing
practice is quite varied.
Perhaps Hubert can add to this.
The company that I worked for back then developed an application that
was ported to Windows, a number of Linux and UNIX platforms, and the
z/OS POSIX environment (called UNIX System Services back then). Most of
our developers worked on Windows; I was part of a small team that
supported the application on other platforms, including z/OS. Most
automation was driven from Windows at the time, partially because we
used Microsoft SourceSafe as our version control system (and enjoyed the
downtime every weekend when access to SourceSafe was disabled while the
database was rebuilt due to corruption that accumulated each week).
The application was primarily C++ with a little Java added for flavor.
The C++ code was built using IBM's xlC C++ compiler which required
source files to be encoded in an EBCDIC code page.
In order to build on z/OS and the other POSIX based platforms, we had a
batch file called TarSrc.bat that used pax (not tar!) to create an
archive of the project that was then transferred to the appropriate
build system. The z/OS version of pax
<https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.bpxa500/r4paxsh.htm>
supports non-standard options (see the -o from=<codeset> and -o
to=<codeset> options) to perform encoding conversion when creating or
extracting archives. A shell script run on the build system then used
pax to extract the project while converting files from ISO8859-1 (I
think) to IBM-1047 (EBCDIC). The script would then un-convert the
binary files contained in the package that should not have been
converted. I believe that script was also responsible for performing
line ending conversions (which also had to be careful not to mutate
binary files). The build was then performed using a z/OS port of GNU
make. Few people worked directly on z/OS. Those that did worked out
their own system for updating source files on z/OS and then transferring
them to a Windows system for checkin. This wasn't as horrible as it
sounds (actually it was, I don't know why I just said that), TarSrc.bat,
the shell script, and the build system supported incremental builds.
The build system also allowed for one build environment to be shadowed
by another such that portions of the source could be built with local
modifications without having to perform a full build.
We eventually moved from SourceSafe to ClearCase hosted on a UNIX
platform. This was a major improvement. Sadly though, there was no
z/OS ClearCase client so while we were able to eschew TarSrc.bat on most
UNIX systems after that, we were still stuck with it for z/OS. In a
desperate attempt to make development more uniform, I attempted to
create a set of shell scripts that provided the appearance of a
ClearCase client on z/OS. This was called KludgeCase. It relied on the
ability of a UNIX system to export a ClearCase view over NFS and the
ability for the z/OS NFS client to perform dynamic file conversions
based on NFS mount options. The scripts that emulated the ClearCase
cleartool utility on z/OS would transparently run remote commands on the
UNIX system hosting the ClearCase view and (un)mount the view via NFS as
necessary. This sufficed for files to be checked out and edited from
z/OS just as on any other POSIX system. The problem was that it was
very slow and I never found a good way to deal with binary files.
After IBM bought Rational, they added a z/OS ClearCase client that, as
far as I could tell, was, and remains, completely useless. It seems to
be designed around remote build processes (not unlike ye olde
TarSrc.bat) and assumes that a z/OS developer's goal in life is to not
actually interact with a z/OS system at all. But perhaps I'm being too
harsh and critical; I haven't actually used it. I just knew that what I
wanted was the same experience I was used to on other POSIX systems.
Third party packages were treated the same as our own source code in
most cases. We checked in the source and TarSrc.bat did the rest.
Well, except for the part where we would have to go update that shell
script to un-convert binary files that shouldn't have been converted.
Today, git and github rule the roost. I don't have any experience using
either on z/OS, but it is clear that others do. A z/OS port of git
<https://www.rocketsoftware.com/product-categories/mainframe/git-for-zos>
is available complete with instructions for how to use it with github
with transparent file conversions
<https://forum.rocketsoftware.com/t/using-git-for-z-os-with-github/654>.
The solution appears to depend on use of git attributes to specify
encoding of files.
The z/OS filesystem allows tagging files with an encoding. The
_BPX_AUTOCVT
<https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxb200/bpxkenv.htm>
or _BPXK_CCSIDS
<https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxb200/bpxkenv.htm>
environment variables can then be used to enable implicit file
conversions. (There are a few other interesting Unicode related
environment variables on those linked pages for anyone that is interested).
Open source software has been used on z/OS since basically as long as
open source has been a thing. Autoconf has supported z/OS since before
I started using it (and since it was called OS/390; before the rename to
z/OS). I'm not sure if CMake has been ported to z/OS or not.
Tom.
>
> AlisdairM
> Related to Tom’s pending paper, I am wondering if anyone with experience
> can report on how non-ASCII/unicode toolchains cope with open source
> projects, which are usually under source control that is encoded in something
> very much like ASCII/UTF-8?
>
> Do they maintain a parallel source tree that is transcoded?
> Do they maintain separate transcoded packages to distribute for their environment?
> Are such environments naturally hesitant to use open source due to such
> obstacles, so the question rarely arises?
> Some other variation as my imagination is quite limited?
The following mostly reflects my own experience working on z/OS 10+
years ago, but I'll conclude with some links that may be more
representative of current practice. In general, I expect that existing
practice is quite varied.
Perhaps Hubert can add to this.
The company that I worked for back then developed an application that
was ported to Windows, a number of Linux and UNIX platforms, and the
z/OS POSIX environment (called UNIX System Services back then). Most of
our developers worked on Windows; I was part of a small team that
supported the application on other platforms, including z/OS. Most
automation was driven from Windows at the time, partially because we
used Microsoft SourceSafe as our version control system (and enjoyed the
downtime every weekend when access to SourceSafe was disabled while the
database was rebuilt due to corruption that accumulated each week).
The application was primarily C++ with a little Java added for flavor.
The C++ code was built using IBM's xlC C++ compiler which required
source files to be encoded in an EBCDIC code page.
In order to build on z/OS and the other POSIX based platforms, we had a
batch file called TarSrc.bat that used pax (not tar!) to create an
archive of the project that was then transferred to the appropriate
build system. The z/OS version of pax
<https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.bpxa500/r4paxsh.htm>
supports non-standard options (see the -o from=<codeset> and -o
to=<codeset> options) to perform encoding conversion when creating or
extracting archives. A shell script run on the build system then used
pax to extract the project while converting files from ISO8859-1 (I
think) to IBM-1047 (EBCDIC). The script would then un-convert the
binary files contained in the package that should not have been
converted. I believe that script was also responsible for performing
line ending conversions (which also had to be careful not to mutate
binary files). The build was then performed using a z/OS port of GNU
make. Few people worked directly on z/OS. Those that did worked out
their own system for updating source files on z/OS and then transferring
them to a Windows system for checkin. This wasn't as horrible as it
sounds (actually it was, I don't know why I just said that), TarSrc.bat,
the shell script, and the build system supported incremental builds.
The build system also allowed for one build environment to be shadowed
by another such that portions of the source could be built with local
modifications without having to perform a full build.
We eventually moved from SourceSafe to ClearCase hosted on a UNIX
platform. This was a major improvement. Sadly though, there was no
z/OS ClearCase client so while we were able to eschew TarSrc.bat on most
UNIX systems after that, we were still stuck with it for z/OS. In a
desperate attempt to make development more uniform, I attempted to
create a set of shell scripts that provided the appearance of a
ClearCase client on z/OS. This was called KludgeCase. It relied on the
ability of a UNIX system to export a ClearCase view over NFS and the
ability for the z/OS NFS client to perform dynamic file conversions
based on NFS mount options. The scripts that emulated the ClearCase
cleartool utility on z/OS would transparently run remote commands on the
UNIX system hosting the ClearCase view and (un)mount the view via NFS as
necessary. This sufficed for files to be checked out and edited from
z/OS just as on any other POSIX system. The problem was that it was
very slow and I never found a good way to deal with binary files.
After IBM bought Rational, they added a z/OS ClearCase client that, as
far as I could tell, was, and remains, completely useless. It seems to
be designed around remote build processes (not unlike ye olde
TarSrc.bat) and assumes that a z/OS developer's goal in life is to not
actually interact with a z/OS system at all. But perhaps I'm being too
harsh and critical; I haven't actually used it. I just knew that what I
wanted was the same experience I was used to on other POSIX systems.
Third party packages were treated the same as our own source code in
most cases. We checked in the source and TarSrc.bat did the rest.
Well, except for the part where we would have to go update that shell
script to un-convert binary files that shouldn't have been converted.
Today, git and github rule the roost. I don't have any experience using
either on z/OS, but it is clear that others do. A z/OS port of git
<https://www.rocketsoftware.com/product-categories/mainframe/git-for-zos>
is available complete with instructions for how to use it with github
with transparent file conversions
<https://forum.rocketsoftware.com/t/using-git-for-z-os-with-github/654>.
The solution appears to depend on use of git attributes to specify
encoding of files.
The z/OS filesystem allows tagging files with an encoding. The
_BPX_AUTOCVT
<https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxb200/bpxkenv.htm>
or _BPXK_CCSIDS
<https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxb200/bpxkenv.htm>
environment variables can then be used to enable implicit file
conversions. (There are a few other interesting Unicode related
environment variables on those linked pages for anyone that is interested).
Open source software has been used on z/OS since basically as long as
open source has been a thing. Autoconf has supported z/OS since before
I started using it (and since it was called OS/390; before the rename to
z/OS). I'm not sure if CMake has been ported to z/OS or not.
Tom.
>
> AlisdairM
Received on 2020-09-09 23:34:30