pipe/fork/exec and friends as custom functions

[vike2000]

Greetings!

Wanting to support external shell/binary cli use in calc; since system() doesn't catch and return output, as in cli calc '2+system("echo 3")'.

Hopefully the custom/pfe help-file synopsis outlines the added functions. If you think I should reproduce or reference more of the underlying syscalls' docs, I'll try.

I have very little experience in authoring PRs, and not too much experience of C (but some), nor of calc, so please bear with me.
I don't know how much of 3f106bf you want or whether 1d70a1f and 70d93c2 should be in separate PR(s) so please tell me what you think about that, if and how I should modify or recreate this PR.

Only built and tested on macOS 13.6.6 using Apple clang 14.0.3, part of Xcode 14.3.1, but my IDE is VSCode, w/ some clangd extension residue .gitignored.
Build req. readline and IDE support py38-pip installed as Macports ports. VSCode clangd support then installed w/ pip-3.8 install compiledb.

I've tried to adhere to perceived convention/style in .h/.c, albeit loosely vis-à-vis heavy tabbing for column alignment. In .cal and .sh I've allowed myself my own style - sorry; of course I'll reformat if requested.

As with everything else, naming and/or prefixing is open for discussion, especially in pfe, pread, and pwrite, being higher level utilities implemented first in calc/.cal and then in .h/.c.

I am a bit unsure whether I am responsible for any more free() than I do. I tried doing leaks(1) at some point but didn't know how to investigate from there.

Some of the functions use "addr" features (where support functions are named …ref…), some use optional arguments (valv_opt…) and type checking in combination, and some use all in combination.
Only a minuscule subset of possible usage alternatives there are tested.
We could also do more sanity checks after optional/type checking.

If these functions should try to be compatible with general file-handles like from fopen or calcs equivalents, I'd have to investigate that.

Grateful for any suggestions!

[lcn2]

Hello @vike2000,

Thank you for raising this issue.

There are some interesting concepts relating to "pipes, forks, and I/O with other processes, etc." in this pull request. We had considered similar concepts for calc v3.

Only built and tested on macOS 13.6.6 using Apple clang 14.0.3, part of Xcode 14.3.1, but my IDE is VSCode, w/ some clangd extension residue .gitignored.
Build req. readline and IDE support py38-pip installed as Macports ports. VSCode clangd support then installed w/ pip-3.8 install compiledb.

We would not want to impose an IDE such as VSCode, add clangd support, nor add things such as pp38-pip, python, and JSON files into calc however.

Before we would be able consider this pull request, we ask that those "extensions" be removed such that this pull request be a pure C addition only.

Please consider revising your pull request accordingly as this will make it easier for us to better evaluate the merits of your overall concept.

I have very little experience in authoring PRs, and not too much experience of C (but some), nor of calc, so please bear with me.

We are happy to be patient as you learn about PRs and explore C as it relates to calc.

There are some tricky concepts involving "pipes, forks, and I/O with other processes, etc.", especially when trying to do them in a portable way that works on a wide variety of systems.

[lcn2]

Hello @pmetzger,

You raise some valid concerns concerning this pull request as it initially was presented. Let us see how @vike2000 responds to this change request review and see if this general concept is something worth considering for, say, calc v3.

[vike2000]

Hello :)

I said I was unsure whether to add these "development files", so I understand your concerns and have now removed those files. Are my changes to .gitignore okay though?

[lcn2]

Hello @vike2000,

We might wish to consider adding the general concepts relating to "pipes, forks, and I/O with other processes, etc." to calc version 3. Maybe not in the initial form of this pull request (see our pullrequestreview-2172735939 above), but rather as a new TODO item for calc v3 (see that link for details).

The reason why we might do this is because the "pipes, forks, and I/O with other processes, etc." concepts would require a calc version 2.16 release (due to the new builtin functions that are required to be added to the calc language). This is happening when we are trying to get calc stable and ready for a code fork for calc v3. Moreover, such a major change, while very useful, might be better introduced into the newer code base (as pure C).

We mention this possibility: not because we dislike the overall concept you are suggesting (indeed we have considered how we might do this in the past) - we do appreciate the overall idea AND we APPRECIATE your interest in calc. We mention this because it might be better to transfer this discussion over in the calc v3 discussion thread.

What all that might mean is for us to add some "TODO" bullet point to the top level comment in the calc v3 discussion thread, and to figure out how to best introduce "pipes, forks, and I/O with other processes, etc." into calc.

Your ideas seem useful, and you are willing to be patient with us as we are willing to be patient as you explore the world of PRs and calc code, then exploring this in calc v3 world might produce the best outcome.

[lcn2]

Hello :)

I said I was unsure whether to add these "development files", so I understand your concerns and have now removed those files. Are my changes to .gitignore okay though?

Adding things to .gitignore to help ignore artifacts from environments such as your IDE is fine.

[lcn2]

BTW @vike2000,

In regards to the defer to calc v3 comment we made above. That does not represent a final decision, but rather our "thinking out load".

We will still ponder your pull request.

[vike2000]

There are some tricky concepts involving "pipes, forks, and I/O with other processes, etc.", especially when trying to do them in a portable way that works on a wide variety of systems.

I'm sure there are. Just recently I saw I had an #include <sys/_types/_timeval.h> and thought I'd have to see if that's a macOS specific header path.
Some year or two ago I had a cross-compilation setup for my macOS, a Linux via Docker, and Windows via Docker and VirtualBox for another project, but sadly my current machine is not in a well enough state for me to have a similar setup for this project at the moment. Perhaps in the future ;)
In part I have been thinking I could get away with this being custom functions - if only as a specific example implementation of such.
Another thought has been leaning more explicitly on POSIX but I haven't pursued that route yet.

[lcn2]

Hello @vike2000,

As with everything else, naming and/or prefixing is open for discussion, especially in pfe, pread, and pwrite, being higher level utilities implemented first in calc/.cal and then in .h/.c.

Over the lifetime of calc, a number of things have started off as calc scripts and custom functions. Often as you start to use some initial implementation of some new concept, one finds one needs additional functionality, additional parameters, and adjustments to the original concept. Gaining experience with the new concept before the implementation becomes "baked into the main code base" (as the express goes) is a very good idea as once it enters to main calc code base, it becomes harder to improve and expand it (as some people start using the original concept as is and become dependent on behavior that later experience suggests needs to change).

One path ("thinking out load" again) is for you to gain experience with concepts relating to "pipes, forks, and I/O with other processes, etc." and bring that experience to the calc v3 discussion thread.

Again, we appreciate your willingness to contribute to calc, to learn about PRs and to learn about calc internals. We mention this because we don't want you to become discouraged just because this PR readiest has not been immediately added to the calc code base. Overall the concepts relating to "pipes, forks, and I/O with other processes, etc." are worth doing well and that can take time.

[vike2000]

BTW @vike2000,

In regards to the defer to calc v3 comment we made above. That does not represent a final decision, but rather our "thinking out load".

We will still ponder your pull request.

No rush on my part. A calc v3 sounds cool (in that the project matures). I understand there are many things to consider in a project like this :)

[lcn2]

Hello @pmetzger,

As you may see, @vike2000 has revised their pull request and has focused on the core concepts relating to "pipes, forks, and I/O with other processes, etc.".

Please re-consider reevaluating this PR, keeping in mind that this is @vike2000's first calc PR and that @vike2000 is learning about both PRs and calc internals. That @vike2000 is willing to do that gives us some confidence.

Sure, there may be ways to improve on the implementation: but that is the way development goes sometimes. :-)

Also consider weighing in, if you please, on if the core concepts relating to "pipes, forks, and I/O with other processes, etc." are best defended to calc v3 or introduced as custom functions in a calc 2.16. We would value your consideration on those possibilities.

[pmetzger]

I'm finding reading large chunks of this very difficult to read because of unusual choices in code formatting. It would be best if these were cleaned up and made more conventional. Example:

[lcn2]

I'm finding reading large chunks of this very difficult to read because of unusual choices in code formatting. It would be best if these were cleaned up and made more conventional. Example:

Fair points, @pmetzger.

Perhaps @vike2000 you could adjust?

BTW: Even if we end up deferring this PR to calc v3, getting this PR into existing shape might be worth the effort so that we may reference it as concept for a calc v3 TODO.

[pmetzger]

@vike2000 There are a lot of other places with similar issues. I'd go through all the submitted code if I were you and try to make it as readable as possible.

[vike2000]

I'm finding reading large chunks of this very difficult to read because of unusual choices in code formatting.

I can understand that. I've pushed two fixes of the worst formatting.

There are a lot of other places with similar issues.

May I ask if you could point me to some examples of places you want fixed in the current set of files?

I also want to thank you for your time and willingness to help me try to get this PR to a good enough level this far - thank you both!

[lcn2]

Just a heads up, @vike2000:

Calc v2 is moving towards maintenance mode

The code base from which calc v3 will be formed is being finalized, and so a major new feature won't be available for calc v2 as that version is going to go into stable maintenance mode (fixing only critical bugs - no new features).

We released calc v2.15.1.0 is preparation for the above. (There may be a v2.15.1.1 before the calc v3 fork 🍴). Anyway that code base had to undergo an (some say unfortunate) change of white space (TAB removal from source: see issue #103 from that TAB discussion). As a result, your pull request you submitted has to be adjusted as the files underneath it changed.

Sorry (tm Canada 🇨🇦) to ask you to change your PR again. Think of this as good pull request practice 🤓.

Calc v3 and your ideas

We have added a TODO in the top comment of issue #103.

New projects often start out as evolving conversations as the TODO list in the opening comment gets edited and revised over time. Your concept was added to the bottom of the TODO (not out of priority, but because it was the most recent - later on as the project evolves certain TODO items will spin off into their own "enhancement project" issues and the TODO bullet point will become a "finish issue XYZZY" - for an example of arch a evolved "enhancement project" see issue 2239 of the temp-test-ioccc repo ).

You may notice slightly above this comment that GitHub added a notice that this pull request was referenced elsewhere. See the previous paragraph for details as to what and why.

How this pull request will be used

Back to this pull request. We highly recommend that you work with @pmetzger to get this pull request compatible with the evolved code format, and to address his suggestions about formatting.

Even though we will eventually reject this pull request, the reference from the link from the top comment of issue #103 to this (eventually closed PR) will remain. It would be a good thing (and good practice) to get this pull request into good shape for the later calc v3 code.

Working towards an new calc v3 enhancement project

In time (and after the calc v3 code fork) as work on the TODO list of issue #103 starts, a new Enhancement issue will be created with a title along the lines of this pull request. The top comment of that new calc v3 issue will include a link to this pull request as a starting position. Comments in that "new enhancement issue" will include ideas about the topic.

BTW: So far we have held off on making architecture comments and portability suggestions: that "new enhancement issue" is the proper place for those suggestions.

This is one reason why it would very welcome for this pull request to be polished and merge ready: this serve as a starting place for "new enhancement issue" for calc v3 project.

A bit of calc history and legacy

Calc has a multi-decade timeline with lots of users from various platforms and resources (some hardware platforms, some supercomputers, some cell phone apps, some mathematical, some cryptography, some space probes, etc.). And while backward compatibility is critical to those many users and developers and platforms, it makes it hard to make the sort of improvements we want to make to make calc better. Calc v3 will, for the first time in about 25 years, change the memory layout of its fundamental numeric objects because limiting numbers to 4 GB is now a problem for some tasks. (We know on one ongoing calculation that has been running for years: freezing calculation checkpoints from memory to disk and restoring memory into faster hardware. We plan a memory migration strategy for them from their calc v2 memory layout to the new calc v3 memory layout, for example).

So calc v3 was formed to make such changes while calc v2 will remain frozen 🥶 in maintenance mode. Topics such as this "new enhancement issue" will be ready for calc v3.

Once the fork for calc v3 occurs, projects like the "new enhancement issue" your pull request inspired will spring up. Meanwhile please 🙏 pardon our project planning dust as we get ready for the fork 🍴.

Sorry the the length

Sorry (tm Canada 🇨🇦) for the length and complexity of this comment. We wanted you, @vike2000, better understand what is happening and why.

[vike2000]

Ah, I see, and understand my PR inadvertently came at a time of great changes :) No worries - I'll bide my time, try to keep track of the progress of the project, and jump in when I feel the calling.

I am away from home this week and will have little time and bad access to my tools, but when I'm back I'll swiftly do a tab to space conversion. I'm also thinking I'll look at running clang-format or similar (or even do some massive tweaking of a gawk-script i've dabbled with for some miscellaneous color-coding and block balancing over the years).

So far we have held off on making architecture comments and portability suggestions: that "new enhancement issue" is the proper place for those suggestions.

I'm curiously awaiting following along in such improvement discussions :)

Thank you for getting me up to date, and history no less - I must say I hadn't realized calc had those magnitude of usages 🤩

[vike2000]

I just pushed u_pfe.c after a run of clang-format on it, with a custom config file (.gitignore'd).

The latest version I can install at the moment is 18 (via Macports port clang-18) so any config change requests to me will have to be compatible with that (as clang-format errs on unknown/bad config directive use).

Doing something similar to .cal-files seems to entail more or less quirks. I left it having realized clang-format is incompatible with the calc ## comment token...

[vike2000]

I managed to hack clang-format (v20) for basic calc support† and just pushed pfe.cal after a run of the former (using mostly v18 .clang-format config‡) on the latter.

†basic calc support

## instead of // line comments and only basic c lexing/parsing - disabling cxx - for e.g. interpreting delete(…) as the .cal function instead of the .cpp keyword;

‡v18 .clang-format config

with Language: Calc and, as clang-format interprets the calc keyword define as a …ReturnType…, disabling breaking after such;