Q/A: Which software packages do you want to see ported to z/OS?

[IgorTodorovskiIBM]

Which Open source packages do you want to see ported to z/OS (as a developer or consumer - please specify).

If you agree with a suggestion, please upvote it.

[v1gnesh]

Hello Igor,

Not from the above list, but would like to see the following:

ripgrep
https://github.com/burntsushi/ripgrep
As you'd say, it needs a native Rust toolchain first.
Hoping to hear more about that, since the C/C++ compilers are now "based on" LLVM.

rclone
https://github.com/rclone/rclone
A superset of rsync, but with more back-ends and features.
Adding in or verifying interop with z/OS NFS, zOS DSFS would be great.
Oh, and adding in support for the upcoming zOS Object Storage too.

textual
https://github.com/Textualize/textual
Not just one package this time but I think it's rich, and textual.
With the availability of tnz, once again, I'm hoping it will benefit from the pzazz of the above package.
What's more, I would really think about building the successor of Omegamon's E3270UI with the above.
Assuming tnz is at the core of the terminal app in zOSMF, adding in textual/rich will make delivering new-age 3270 apps more fun, by making the best of all the love that TUIs (terminal UIs) are getting these days (in Python, Go, or Rust).

[IgorTodorovskiIBM]

Based on the Guild meeting survey results, these were the tools that were voted for:

[MikeFultonDev]

@IgorTodorovskiIBM (or someone) - maybe update this to which ones are already done!

[MikeFultonDev]

A lot of people had interest in a lot of ports, which is great!
I'm happy to help get people started if they want. The ones with an asterisk (*) are already in ZOSOpenTools. The ones without an asterisk aren't, but it's pretty simple to set it up - just reach out and if you are the first one, you can be a committer on the repo.
You can send me an email to [email protected] or alternately respond in this discussion
Here's what was mentioned in the survey:

vim (*)
screen
less (*)
locate
top (*)
meld
git (*)
openssl (*)
curl (*)
Python,
bash (*)
gomtree
ninja (*)
gcc
PostgreSQL
gzip (*)
grafana
valgrind
Gin Web Framework
gradle
zpm (let's chat!)

Also, the following were mentioned. Kubernetes is being worked on by z/OS proper - IBM has a statement of direction around it. For Ansible, not sure if people want to use it to talk to z/OS (if so, it's already available), or if they want to run the Ansible client directly on z/OS (in which case, it would be interesting to understand the scenario).

[MikeFultonDev]

Of course, you can feel free to fork a repository, make some changes, test, and submit a pull request for the committers already working on it

[v1gnesh]

zpm (let's chat!)

Hi Mike, does this mean... z package manager? Something new?

[MikeFultonDev]

yes - zpm is technology that IBM ships in a product called IBM Z and Cloud Modernization Stack. It does indeed mean z package manager - it works with OCI artifacts as input and installs them into /usr/lpp/IBM/zpm
How zpm and 'zopen' overlap is an interesting question.

[v1gnesh]

Only today, I noticed that zopen can build oci images (since Aug 2022)!
So I guess that ties these two together.. i.e., zopen builds images and these are made available for deployment via zpm (or more generally the zOS operator?).

[MikeFultonDev]

@IgorTodorovskiIBM can you update the question to make it clear that additional software packages are good to hear about too and not just the ones we have created so far?

[v1gnesh]

What about the ones I've mentioned above.

Also, WireGuard proper for USS and zOS, please?
zOS CS supporting only IPSec and IKE even today is... limiting what's possible.

[v1gnesh]

@MikeFultonDev @IgorTodorovskiIBM
It'd also be useful to have zlib-ng and/or zstd support.
Well, port to begin with, and then native zOS support.

I do see that a zlib port is available; however, we may benefit from upgrades -
https://dougallj.wordpress.com/2022/08/20/faster-zlib-deflate-decompression-on-the-apple-m1-and-x86/

[MikeFultonDev]

@v1gnesh for the ones you mentioned, which ones would you like to work on? I can create 'port' repos for those and add you as a committer to get the ball rolling.

[v1gnesh]

I think most people will benefit from having ripgrep & wireguard.
ripgrep is written in Rust, and Rust supports targeting s390x.
However, I'm interested in these as a user, i.e., running Rust in zOS.
There are far, far better programmers than me who can do things far more efficiently than I can.

[v1gnesh]

Hello again,

Now that S3 is accessible through zOS, could something like s3fs be ported please, to support more seamless S3 usage?

Another question - would you be able to comment on where IBM is w.r.t zIIP offload-ability of python in general? And asyncio-based packages more specifically? Ex: https://github.com/aio-libs

[IgorTodorovskiIBM]

I wanted to bring this discussion back to the top to get more opinions on it., if you have any suggestions or comments around what tools you would like to see ported, please comment here. For a list of tools being ported, see the z/OS Open Tools list of repos: https://github.com/orgs/ZOSOpenTools/repositories.

Some of the new tools we're porting include gpg, sudo, emacs, htop, ruby, neovim, and ruby

[v1gnesh]

Go

direnv - Allows project-specific environment variables when working on multiple python/go/etc. projects with different toolchains. Automatic setting/unsetting of env variables.
rclone - "rsync for cloud storage"
caddy - A web server & more
wireguard-go - Userland implementation of WireGuard
NATS - A messaging system
Terraform - A Terraform provider for zOS would be great

C/C++

duckdb - An in-process SQL OLAP Database
DragonflyDB - A modern replacement for Redis and Memcached
Apache Arrow - This is/will be the foundation of advancements in data & analytics

They both do a lot of SIMD processing.
Will definitely need help to point to SIMD-related headers and what not.

Big stuff

eBPF - Adds "easy" programmability to the (linux) kernel
WireGuard - Should take the reins from OpenVPN, IKEv2, and IPsec

Would really like to see these on Z and zOS..

Rust

ripgrep - An upgrade from grep
Apart from this, there are a bunch of common *nix tools re-written in Rust.

I'll keep the Rust & Python list for later, seeing how ibm_python & ibm_rust comes to be.
I can try the first steps with a few of them. Then I'll surely need help :)

[ijmitch]

I just noticed podman mentioned earlier in this thread - it's part of the z/OS Container Platform - https://www.ibm.com/docs/en/zoscp/1.1.0?topic=platform-components-zos-container

[mfsysprog]

I thought that some components like podman were running from zcx, but it seems that podman is actually running natively on z/os

[AnthonyGiorgio]

Yup - zOSCP was the result of years of effort to get native containers running on z/OS.

[mfsysprog]

Yeah, I know, but I seemed to remember that some tools working with zOSCP were still running under zcx, since they were not yet ported to z/OS natively.

[t1m0thyj]

Would also love to see support for Rust on z/OS 😍

[v1gnesh]

Adding a bunch more unix-y ones:

https://gitlab.com/procps-ng/procps
https://gitlab.com/psmisc/psmisc
https://github.com/strace/strace

[AnthonyGiorgio]

I'd like to see the_silver_searcher ported.

[IgorTodorovskiIBM]

Ported! zopen install thesilversearcher

Then you should be able to run the ag command: ag pattern [path].

I like that it highlights the pattern in the search results.

[MikeFultonDev]

Nice!

[gngrossi]

Would these be considered part of coreutils?
watch
pstree
logrotate

[IgorTodorovskiIBM]

• I believe watch is part of procps (also includes top, pgrep and ps).
• pstree is part of psmisc
• I've never used logrotate before, but I believe the source for it is here: https://github.com/logrotate/logrotate

[gngrossi]

Any further consideration for logrotate and watch?
thanks

[IgorTodorovskiIBM]

Not yet, procps is on our list of todos.

https://github.com/logrotate/logrotate seems like a good one to port for a beginner contributor! If anyone is interested!

[gngrossi]

Regarding logrotate, one of the issues I've seen in our USS environment, is not having a good way to manage log file space. As an admin utility, "skulker" is run to remove files from the /tmp directories on each LPAR but each application owner has the responsibility to manage log files using their own scripts, etc.

Is their a best practice to to close a gap like this?
thanks

[MikeFultonDev]

Good question. When you are after a best practice I I am not sure if scope. IBM z/OS, z/OS community, or genera IT practice? Of course I guess you could argue they should all be the same! But am thinking that depending on your focus the discussion location might vary (IBM forum, discussion here in meta, or stackoverflow)

[v1gnesh]

@MikeFultonDev, I wonder if this initiative can gain more speed (to be fair, it's going very well already) by formalizing a sponsorship program (GitHub sponsors). Those who want tools ported can either dig in, or sponsor someone so that the work is sustainable.

[v1gnesh]

tree
https://mama.indstate.edu/users/ice/tree/
Source is at https://gitlab.com/OldManProgrammer/unix-tree and https://github.com/Old-Man-Programmer/tree.

[gngrossi]

I can re-install if needed.

RC=(0) [SYSA] bash-5.2$ which tree
/hewitt/zopentools/guild/tree-master/bin/tree

RC=(0) [SYSA] bash-5.2$ man tree
No manual entry for tree

[mfsysprog]

Can you do a which man?

[IgorTodorovskiIBM]

man tree is working for me. @gngrossi are you installing the pax manually or using zopen install? Do you have MANPATH pointing to tree's share/man directory?

[gngrossi]

RC=(0) [SYSA] bash-5.2$ which man
/hewitt/zopentools/guild/man-db-2.12.0/bin/man

/hewitt/zopentools/guild/tree-master
I sourced the .env file like the other tools...appears MANPATH is not set.

Installing via a pax file for now (our z/OS system doesn't have access to https://zosopentools.github.io/meta/api/zopen_releases.json)
Just tried with meta080.

[IgorTodorovskiIBM]

@gngrossi we need to update the official meta pax, but if you grab it from here: https://github.com/ZOSOpenTools/metaport/releases it includes the url change from github.io to githubusercontent.com

[daver1958]

I have a really fast way of syncing directories from 1 linux directory on one machine to another (not sure if its the fastest but I reckon its close).
It uses rsync to determine the files to be updated, then tar with zstd compression (to minimize the size of the files to be sent) and mbuffer to optimise the network connection. It uses mbuffer to zstd to tar on the receiving side. You have done a great job of porting most of that. So if you could port mbuffer that would be great.

[igortodorovski]

Interesting, I have not heard of mbuffer prior to your comment. I assume you are using it similarly to this answer: https://serverfault.com/a/435002 ? Have you compared the time with and without mbuffer?
Do you know if they have a public git repo? I see source distributions but no git repo in https://www.maier-komor.de/mbuffer.html

[daver1958]

Hi,
Sorry i don't know if it has a public git repo.
I have timed it on my lan at home and it is faster.
Here are the current settings for machines using 1gb nics .

send side: issue command from cd /zfs/zos25/z/

rsync --info=name --out-format="%n" -ainAXEtp /zfs/zos25/z/ [email protected]:/zfs/zos25/z | tar -I 'zstdmt -2' -cvf - -T - | mbuffer -s 128k -m 1G -O 10.2.1.xx:8080

receive side:
mbuffer -s 128k -m 1G -I8080 | tar -vx -C /zfs/zos25/z --use-compress-program=zstdmt

If you run out of CPU use single CPU zstd vs stdmt

[daver1958]

Took me a while to get it all working :)

[netskink]

20230907 Update

Package	Port Established	Notes
vim	Yes
screen	Yes
less	Yes
top	Yes
git	Yes
openssl	Yes
curl	Yes
Python 3	Yes
ninja	Yes
gzip	Yes
bash	Yes
locate
meld
gomtree
gcc
PostgreSQL
grafana
valgrind
Gin Web Framework
gradle
zpm		let's chat!

[MikeFultonDev]

thanks for the update @netskink !

[netskink]

You are welcome

[IgorTodorovskiIBM]

locate is also ported - part of findutils

[lnlyssg]

A true pipe dream request: John the Ripper - https://github.com/openwall/john

[AnthonyGiorgio]

We should port dos2unix/unix2dos. They are handy for working with files from Windows that have the wrong line endings.

[IgorTodorovskiIBM]

Now available: https://github.com/ZOSOpenTools/dos2unixport/releases

[IgorTodorovskiIBM]

This is another useful one that is Go based - https://github.com/git-lfs/git-lfs - provides large file storage on Git

[lbdyck]

How about dialog https://github.com/phracker/dialog

[daveyc]

nmap would be really useful.

[daver1958]

pyinstaller to turn python programs into executables.

[daver1958]

mc - would be great to be able to run that under uss.

[AnthonyGiorgio]

What about ncftp and nedit? I know that FTP as a protocol is dying, but it's still used for various things on z/OS. Having a current port of ncftp would be nice. Maybe we can port the xnedit fork of nedit.

Both of these have been ported in the past, although ncftp was version 2.x. I used nedit many years ago when I first got involved with z/OS, and it was handy for remotely editing files directly in the filesystem.

[gngrossi]

Using a telnet client to test connectivity (port open after a FW change) would be helpful.
For years, I have been using an alias to provide something close to it. alias telnet='ftp -e'
Recently been using openssl as well.

[IgorTodorovskiIBM]

Interesting, I didn't realize that z/OS does not provide a telnet client.

You can also use curl for this:

curl -v telnet://example.com:80

[gngrossi]

thanks...much appreciated

bash-5.2$ curl telnet://mvs-sysa:2222
SSH-2.0-OpenSSH_7.6

bash-5.2$ curl telnet://mvs-ste3:2222
SSH-2.0-OpenSSH_8.4

[gngrossi]

another way...

bash-5.2$ cat < /dev/tcp/mvs-ste1/2222
SSH-2.0-OpenSSH_8.4

[lbdyck]

This looks like it could be useful https://github.com/jacebrowning/env-diff

[fswarbrick]

I'm surprised netcat or socat is not yet on the list. Seems useful.

[IgorTodorovskiIBM]

@ccw-1 wrote a Go implementation of netcat, zos-nc: https://github.com/IBM/zos-nc . You can install it via zopen.

[fswarbrick]

Yeah, it's lacking in features... Specifically the one I am looking for allows nc to connect via an HTTP proxy, for ssh. i.e. ssh -o "ProxyCommand nc -X connect -x localhost:3128 %h %p" my_ssh_server.

[ccw-1]

This feature is added,

-X. [5,connect]
-x [host:port| https://....:3128]

I tested -X 5, it worked.
I don't have a proxy server to test -X connect

[fswarbrick]

If you saw my last message (now deleted), please ignore. I think I was testing my own attempt to add proxy usage, rather than yours!
I'm having cert issues getting yours, but I'll test it as soon as I can.

By the way, you might try installing Squid (a proxy) on a Windows or Linux machine. It's simple to get working.

[fswarbrick]

@ccw-1, I see you added additional code to get the HTTP proxy feature working. I was actually doing pretty much the same thing, but you go there first! It works for my use case (proxying SSH), but only on my PC. So I think that must be some issue with OpenSSH on z/OS. Oh well! I guess I'll continue using HTTPS for GitHub.
Thanks for your work!

[Tam-Lin]

It's a python package, but porting greenlet (https://pypi.org/project/greenlet/) would be really helpful. There are a number of other python packages with dependencies on greenlet, so having greenlet support z/OS would be really useful.