segfault with extended globs

[jghub]

this code

i=0
v=a
s=
while ((i < 10000)); do
  ((++i))
  s+=$v
done
[[ $s == +($v) ]]

reliably segfaults with current ksh2020, the problem is in this case inherited from ksh93v- (which in turn inherited it, for a change, from ksh93u+, so this is not a regression but an original ksh93 bug I presume).

the extended glob pattern in the final test is causing the segfault.

[krader1961]

Bug reports are great and welcomed. But given your intense interest in a project you've repeatedly claimed has been "hijacked" I'd be more impressed if you actually provided an analysis of the bug and, ideally, a fix. If nothing else you could have provided a backtrace of the failure.

[krader1961]

A backtrace of the failure shows the stack being exceeded. Here are just the last few lines of the backtrace:

    frame #27214: 0x000000010fc020c5 ksh`parserep + 393
    frame #27215: 0x000000010fc00324 ksh`regnexec_parse + 4989
    frame #27216: 0x000000010fc00d20 ksh`regnexec_parse + 7545
    frame #27217: 0x000000010fbfeced ksh`regnexec + 1125
    frame #27218: 0x000000010fc17633 ksh`strngrpmatch + 387
    frame #27219: 0x000000010fb7d411 ksh`test_strmatch + 157
    frame #27220: 0x000000010fbd9904 ksh`sh_exec + 6164
    frame #27221: 0x000000010fbb48cd ksh`exfile + 1437
    frame #27222: 0x000000010fbb543c ksh`sh_main + 1779
    frame #27223: 0x00007fff654307fd libdyld.dylib`start + 1
    frame #27224: 0x00007fff654307fd libdyld.dylib`start + 1

Note the number of call frames. The three frames from #27214 to #27216 repeat until the stack size limit is exceeded. I would argue this is not a bug in the usual sense of an implementation bug; e.g., a buffer overflow or off by one error. But there is a strong argument to be made that it is a design bug in how the ksh regexp engine handles that pattern using recursion. Especially since, in this case, the shortest match is all that is needed and thus no recursion whatsoever is necessary even if recursion is otherwise required to match other inputs.

[krader1961]

Also, if you're using ksh for this type of problem you are using the wrong tool for the job. There are problems where ksh is the best tool. This isn't one of them. Google "if all you have is a hammer."

[jghub]

Google "if all you have is a hammer."

this coming from you... so you can be funny (even if only involuntarily).

but let's leave it at that here: segfault reported. this one of the "111" kind (present in ksh93u+, present in ksh93v-, present in ksh2020). put it on the list.

oh, and maybe count the "111" segfaults and user-facing operational bugs you know of. and than count the bugs in the "011" set (only present in 93v- and 2020 but working in 93u+: I am suspecting this is the largest set) and the :"110" and "010" ones (that would be real fixes in 2020). and the "001" (only buggy in 2020) ones.

[jghub]

you also wrote:

But there is a strong argument to be made that it is a design bug in how the ksh regexp engine handles that pattern using recursion. Especially since, in this case, the shortest match is all that is needed and thus no recursion whatsoever

what you state is wrong in multiple ways:

• we are not talking "regexp" matching here but "extended glob patterns". I am semi-confident that you understand there is a difference and you just used the word "regexp" carelessly. but to be on the safe side this needs to be pointed out.

• obviously, you don't understand extended glob patterns properly. +(a) means "only a's from start to end of the line", i.e. it is equivalent to the regexp ^a+$. so the matcher has to absorb the whole string to decide whether it matches. you can not restrict the match to the leading substring.

• ksh's regexp matcher (which it also supports, in case you don't know) does not have the problem:

i=0
v=a
s=
while ((i < 10000)); do
  ((++i))
  s+=$v
done
[[ $s =~ ^a+$ ]] && print match!

works just fine.

[jghub]

Bug reports are great and welcomed. But given your intense interest in a project you've repeatedly claimed has been "hijacked" I'd be more impressed if you actually provided an analysis of the bug and, ideally, a fix. If nothing else you could have provided a backtrace of the failure.

fake fact fighting cont'd:

• I don't have an "intense interest" in this project. I do have interest in availability of a fully functional ksh93 -- and ksh2020 is becoming something different and inferior in your hands.

• it was someone else who described what has happened to the ast project as "hijack". you might check this and also whether I've adopted and used that term at all afterwards (I don't think so).
  "hijack" is close but I'd rather describe the situation as a "hostile takeover" since the ksh community very obvious and nearly unanimously does not agree with your actions. and "hostile" is a good summary of your overall attitude anyway.

concerning the rest of your comment: nobody wants to impress you. nobody is impressed by you. you received a bug report. react any which inappropriate way you want.

[krader1961]

react any which inappropriate way you want.

Okay, I'm closing this since the problem is due to an algorithm chosen long ago by the old AST team. Their choice of a recursive algorithm for that pattern is a pretty common choice. It is not an implementation bug. We're not going to change the algorithm they implemented since doing so would be incredibly risky with respect to backward compatibility.

@jghub I'll grant that you know more about the behavior of the ksh language than I do (despite my using ksh for more than 15 years). Heck, you might be the leading expert on the planet on the topic of ksh behavior. But you are rude, argumentative, and obnoxious. Please, go away.

[jghub]

I'll grant that you know more about the behavior of the ksh language than I do (despite my using ksh for more than 15 years). Heck, you might be the leading expert on the planet on the topic of ksh behavior.

knowing more about KornShell language than you do is not difficult, unfortunately. so let's say I am just an average "real" ksh user while you are not and your grasp of the KornShell language and the ksh communities' needs is decidedly sub-standard (and manifestly inadequate for maintaining future ksh development).
if you would appreciate (and be willing to respect/follow) expert advice on finer details of core ksh (POSIX) behaviour I'd point you to @mirabilos -- if he could be bothered to suffer your ways gladly.

But you are rude, argumentative, and obnoxious. Please, go away.

I trust people will be able to judge by themselves to whom these attributes apply consistently in the first place.

[jghub]

@siteshwar:

I'd expect that you reopen this issue. a reproducible segfault is a bug. what else? krader is acting and arguing foolishly by closing it. he seems to draw satisfaction from this sort of actions.

I have reported the bug intentionally very concisely and did not want to get into another stupid argument with krader but --- see above. I also made it very clear that this is broken even in ksh93u+ (which is the rare case in this project not the regular one).

I agree with krader that this bug cannot be fixed easily (not by him, that's for sure) and it might be there to stay. in which case it needs a clear documentation in the manpage under KNOWN BUGS/LIMITATIONS in my view (and hint at using regexp as a workaround). but it needs to remain documented in this project as a BUG. I do hope you do agree and will reopen the issue accordingly.

[jghub]

@siteshwar:
I also would request that you break your silence and speak out how you view the state of affairs in this project. are you still caring or have you given up? I feel it is getting more alarming and ridiculous by the day. and there is a single person responsible for this sad situation: krader. it most definitely is not just a problem of "deficient communication skills" as you put it somewhere, I believe. many different people have voiced similar opinions by now. nobody has ever welcomed what krader is doing to ksh.

you know my position:
I believe he should do his thing in a private fork (where he might alter ksh beyond recognition without affecting anybody) but he definitely has no beneficial influence or purpose in the "official" ksh project. so he should not have commit rights. it was a grave error to grant him those rights (but apparently that was not obvious at the time). you should acknowledge that and correct this error.

what needs to happen seems rather clear: get together some sane and competent people and restart from ksh93u+ (presuming that nobody appears that is able to fully understand the ksh93v- code and to complete D. Korn's work to create a "ksh93v+").

"phase one" would be to replace the build system either by this meson/ninja thing (possible) or just by gmake/Cmake (preferable, probably). that is doable and would solve the single central issue of all the package maintainers out there as far as I have understood the situation: make ksh93u+ build easily and maintainable in the different distros with standard build tools.

considering all the breakage that still lurks in the ksh93v- heritage of ksh2020 and the additional booby traps planted by krader's actions of two years (case in point: what has happened regarding reverting the silly changes to cd in #1445? nothing, I bet) , considering the distinctly inferior performance of ksh2020 (#1449) while offering not a single new feature in comparison to ksh93u+ that I am aware of (I know that 93v- had new features but they do not yet work reliably and have already partly been removed by you like the "toupper", "tolower" string processing stuff...), it is very very obvious that ksh93u+ is a better choice as "the future of ksh" than ksh2020.

so the project now first and foremost should concentrate on 93u+ w/o for the time being touching the code at all (beyond what is necessary to make it build with gmake/Cmake).

I believe that almost everybody out there would be rather happy with this situation: ksh93u+ is already good enough for almost all purposes I'd say. but of course it does contain certain bugs (the one I reported here is one example). so "phase two" then would mean to start trying to fix known (or newly reported) real bugs of 93u+. but carefully and thoroughly. it goes without saying that your and krader's work at ksh2020 than could serve as one source where to look for successful fixes to problems that are common to 93u+ and 93v-.

it would be good if this strategy could be agreed on and further interference of krader be prevented.

so, what do you think?

[jghub]

@gordonwoodhull
@johrstrom
@kindofblue
@MarshM
@s-u
@tremaineeto

dear owners of the att/ast repository,

having been encouraged to do so by some followers of this project and after having spend a couple of months trying to provide constructive feedback completely unsuccessfully I'd like to bring to your attention the sorry state of this project.

I would appreciate if you would take the time and closely look at what is happening to this repository and to assess what damage is done to the reputation of D. Korn and ATT by kurtis rader (krader1961) who effectively has taken over command in this project and is severely misbehaving while also harming the code base.

please read, e.g., through the full history of the present issue and look at the full #1449 issue or this conclusion #1449 (comment)

there are also multiple instances of diverse people objecting against what is happening here over the full history of the project if you go back to the history of this project. #88 is just one early example for the complete misguidedness and ignorance of krader

a few further examples of the inadequacy of krader's personality as the de-facto lead (or even a contributor) to this project:

"Gah! Don’t get me started about the issues in the nameref code. It’s a steaming pile of caca." see here: https://gitter.im/ksh93/users

"> P.S., I am deliberately not including the mailing list address in this reply. Same as my earlier, private, replies in order to avoid making disagreements that are personality based public. That you are deliberately making them public tells me you are an asshole." see here:

https://groups.google.com/d/msg/korn-shell/TVpHd5rrI5g/8Cj6ir7NAwAJ

someone acting in this way should no longer have commit access to this repository in my view.

[jelmd]

@krader1961, thoughtless phrases commonly used to get rid of "pesky" people do not change anything. IMHO actually not @jghub but you should go away.

Well, some years ago I decided to only waste minimal time in this repo: just skim its archive from time to time, add some thumbs if it is really worth, and warn people, that this repo got hijacked if needed. Reasons are obvious. But since @jghub asked, and he really deserves to know, that he is not alone because of his admirable tenacity/power to get things into the right direction (I really appreciate it), and I got tired to explain the situation to people again and again, here my 10¢. If TLDR, just scroll down to the end.

Fact is, the repository got hijacked and is in bad state:

Once upon a time, there was a Advanced Software Technology (AST) department AT&T, where David Korn (dgk), Glenn Fowler (gsf), Kiem-Phong Vo (kpv) and others developed a lot of very smart stuff like ksh93. Around 1999 AT&T changed its mind and allowed AST to make it public. Since than features/patches etc. were discussed in personal mails or AT&T mailing lists, and occasionally gsf packaged the sources and published it on the AST web site (IIRC since 2000).

After dgk and gsf job's at AT&T got terminated in Oct 2013 (kpv already left some month ago)[1], dr.ek "took over" AST site maintenance and synced in gsf's packages like ast-open-2014-12-24.tgz [2], what many people erroneously took as a ksh stable release (ksh93v-) - note the trailing dash, i.e. "beta". This version is known to have so many bugs, that it "should not deemed release-worthy" [3]. Furthermore the lack of a public issue tracker and source code repository was mentioned as a major hindrance (Sep 2015).

At the end of 2015 the AST website went off the air (official statement: security issues) and finally dr.ek was so nice to push the AST leftovers to github in Jan 2016 [4]. Unfortunately after this, it became obvious that neither dgk/gsf/kpv/dr.ek nor anyone else at AT&T had/has time to devote to the repo.

So once dannyweldon contacted dr.ek (lkoutsofios) to ask about some things a while ago, he offered him commit access. He accepted and started to sort/tag issues submitted so far (~ Mar 2017). However, Danny's time was/is limited as well, and when siteshwar announced his plans to fork the AST repo, Danny thought some thing like "Oh nice that RedHat takes over a let a professional work full time at it" and suggested to ask dr.ek to give him commit access to the AST repo not knowing, that he is in bad company (i.e. Kurtis Rader (krader)). Finally he got it [5] and krader got access obviously from him ~ Nov 2017, possibly being still blended by his boring "I've worked for ..." in the fish project. They started immediately damaging the repo as well as ksh93 itself. Seeing this, Danny started to regret this accident wholeheartedly (and I guess he still does).

So both started to effectively kill most of the software of the AST repo, even so many people asked them to not do that but to fork and than start playing around. They even ignored the original owners of the repo[6]. In krader's case I guess, as usual he did not understood, what dr.ek is saying, because he rarely does its homework (i.e. carefully study the source, its background/context and related material) and therefore ignored it. In this special case krader did and still does not even know (and obviously siteshwar as well), that it was damn easy to build just the ksh93, that there is no need at all to build the whole AST tree.

Anyway, what I see is, that especially krader continues to not listen to people (only what he thinks rules), is misinformed about the software he tinkers with, ignores things he does not understand, obviously does not try to study and fully understand the required material (e.g. AST malloc) and prefers to rant about "how stupid and confusing the existing code is" [7]. Well, he actually does not even have a ksh93 background and continues to disparage ksh93 as a batch facility for his CLI calls [8][9], to kill very essential stuff, he does not understand [10][11][12] or he does not like [13][14], or to mess it up/replace it with other crap he has seen in other software [15]. One could think, his target is to kill ksh, to throw away all features, which actually make it different and so useful for real ksh users. Anyway, many people seem to have stopped wasting their time in such an user/developer unfriendly environment [16]. I'm still wondering, why krader tries to shoutdown others by hammering his working places into discussions all the time. In what dungeon have they put him that he did not realize, that there are other smart people working at, or got job offers from those companies as well and that there are also a lot of smart people work at small companies, universities, etc., there where most innovation actually happens? Anyway, in 2017 dgk mentioned, that he is still interested in ksh93 and wants to get back, when he gets some time for it (PM). I guess, he got scared off as well and now devotes his time to more valuable things.

And BTW, beside trying to remove very important, unique, very usable features of ksh93, he has shown several times, that he is completely unqualified as a maintainer of this [or any other] software. Insulting user which withstand his misinformation and lame argumentation, declaring builds as being a stable release, when it is known, that this is definitely not the case, closing issues from people he does not like (because they are right, or do not worship him), or removing important code just because he does not like the OS (and/or its vendor) using it [17], is not what I expect from a maintainer (some people already named his behavior that of an uneducated stubborn child and behaves like an elephant in the china shop[18]).

So yes, at least for me all this is exactly, what hijacking involves:

• got access by pretending to do something different as they actually have done
• neither krader nor siteshwar have any legitimization to call themselves "maintainer" of ksh93. Or did I miss any community poll or election or whatever regarding this?
• Instead of being honest and fork, they still hide what they produce under the coat of AT&T/AST and make people walk into this trap (as Debian did)
• most of the people trying to contribute/sail on the ksh93 ship got scared off especially by kraders rude, violent behavior and successfully killed the ksh93 community [19]
• It appears that at least krader is absolutely incapable to maintain software, especially this/a language (he seems to understand C and tooling around it pretty good, but almost nothing at a higher level like ksh93 features) and has no desire at all, to get it to its intended destination.
• krader seems to be very narrow-minded and has no desire to let ksh93 evolve or to enhance it with new features or to document, what he thinks is undocumented (instead he prefers to remove it).
• He doesn't care about breaking it, i.e. making it totally unusable for its users and existing ksh93 scripts.
• Last but not least, setting up a closed mailing list, so that only he and his believers can communicate, clearly shows krader's intention to keep out ksh93 community members, users, developers and keep this repo hijacked.

IMHO, this repo should be restored to its original state in Jan 2016, a clone be created as dr.ek suggested (/att/ksh93). Whether the master branch of it gets deduced from u+ or v- might be discussed there (I prefer u+ and migrate step by step changes from v- to it in a separate branch). I can live with siteshwar as a maintainer since he seems to have proper guidance and seems to loose his blindness wrt. krader, but krader itself should do, what he suggested several times to other people: he should create his own fork and if he likes submit his changes as normal Pull Requests (PR). IMHO he should never get commit access to the new ksh93 repo again (and neither to the AST repo too of course).

[jghub]

@jelmd: thanks a lot for speaking up! it seems high time to bring this bad development to and end. also many thanks for providing a lot of historic detailed background information I did not have -- although I had put the main pieces together already.

and further thanks for compiling additional extreme horrific examples of what krader has done to ksh ("kill non-forking subshells!?" that maybe so far is the craziest idea/deed(?) of them all).

I also would opt for restarting from u+ rather than v- for said reasons (but that is only an opinion based on personal experience with instability of v- and what people have said, not from a deeper analysis, so ...) and to start backporting fixes.

I have talked to a colleague: if these necessary steps are not taken here (depends on the reaction of the ATT people and siteshwar -- or absence thereof), we are determined to try to wrap 93u+ in a new build system (gmake, Cmake are candidates) and see our way from there. if that becomes necessary, maybe we can join forces regarding integration of patch sets (solaris, eg.?) etc.?

[jens-maus]

While I had mainly listened and monitored this repository in the past only, I also cannot stand the current situation anymore and would like to express my full support for @jelmd and @jghub and their idea to revoke direct commit access for @krader1961 to this att/ast repository.

While at the starting point of kraders' introduction to this repository I was still quite optimistic and found some of the changes he introduced also - at least - partly welcome, it now (after all these years) ended up in a complete mess which not only damages the legacies of David Korn, but also imposes some severe risk that the ksh shell (which has some clear advantages for non-interactive shell scripting) will not survive this if this situation is not resolved soon and the official ksh/ast repository not returned into the hands of the ksh community again.

Not only has @krader1961 introduced IMHO a quite uncommon build environment (meson+ninja) for such a legacy unix project (which introduces some unnecessary dependencies, e.g. python) and should have better used autoconf or cmake for such a C-only project which should be IMHO easily compilable on all different kinds of unix platforms from legacy platforms like solaris, hpux over complex x86 based linux system till resources limited environments like embedded systems. Furthermore, he introduced breaking changes or performance degradations without discussing these changes in advance in the ksh community and he is constantly ignoring concerns from other ksh community members that have a much broader knowledge about the scripting language or the functionality which makes ksh special.

But what I found most irritating is, that @krader1961 obviously severely lacks to have competences to act as a proper open source maintainer for such an important project where communication skills are absolutely necessary to bring together all different opinions and concerns. IMHO this project lacks a clear code of conduct which is common for all modern open source projects (which krader would have already violated multiple times). Furthermore, he is IMHO also missusing the AT&T brandmark by directly committing breaking changes to this branded repository rather than having forked it under his own account name and submitting all changes to that fork instead and then sending in pull requests to the original att/ast repository which someone else (someone from the original "att" group) would have to approve after having discussed this carefully in the ksh community. By directly committing to this repository external viewers might get the false impression that these changes are offcially approved by AT&T or the ksh community – which in fact both is not the case!

Thus, I would like to again express my deep support for the idea to revoke direct commit access for @krader1961 and restore this repository to the original ksh93u+ state. Then krader can fork this repository, integrate his changes into his own fork, rename it "kradersh" and perhaps submit each and every of his suggested pieces to the original att/ast repository via official pull requests which external reviewers can then check and integrate into a dedicated development branch after having discussed them carefully like this should have been done right from the start. And least not least, a detailed code of conduct should be added to the repository which defines under which conditions contributions and discusssions have to be done. Looking back, this would have probably helped to prevent this mess right from the start...

[siteshwar]

This issue has diverted from it's original topic. But since everyone is putting down their thoughts here, I will summarize my experience and view of this project. I will try not to be biased and put facts here :

• It was me (and not @krader1961) who first got commit access to this repository. I had made multiple attempts to reach @dgkorn before starting to commit here, but he never responded to me (except couple of private thanks on the patches that I posted on ast-developers mailing list). He never responded to me anytime after I started to commit to this repository either, so I have the right to believe previous maintainers have lost interest in helping with ksh.

• There was no voting for the new maintainers because there was no one to be voted for. Given the state of the source code that I started from, I have heard the best engineers at Red Hat saying that they don't understand it. No one was willing to touch it. This condition is still somewhat true today, despite of the major changes that we have made in last couple of years.

• I started by upstreaming internal Red Hat patches ( first one here ) to the beta branch of ast project. Initial set of those changes are now preserved in 2016-01-10-beta branch.

• @krader1961 made his first comment in ast repository more than 2 years ago. I agreed with his ideas and he got commit access to this repository through me.

• It was me and not @krader1961 who made the choice to use meson/ninja as the new build system. So at least he should not be blamed for choosing build system thats incompatible with old systems. I don't regret my choice and I am not going to use any other build system. No one (not even people who disagree with @krader1961) likes the old build system and all of us unanimously agree that it should have been changed.

• @krader1961 has made major changes to this codebase since he got access. This includes reformatting the code, dropping old (irrelevant) code, upstreaming patches from other distros (openSuse, Solaris etc.), writing framework for test cases etc. I support those changes. I was responsible for upstreaming (almost) all internal RHEL/Fedora patches and their respective test cases. I have also set up CI at Red Hat that runs internal test cases and also basic CI that builds every pull request on FreeBSD.

• I mostly care about fixing bugs and improving test coverage. So I am slightly less involved in this project since then, although I have done several talks about it.

• I don't agree with some of the changes made by @krader1961 recently for e.g. recently changed cd behaviour (_cd function, autoloading etc) breaks existing ksh scripts #1445. They were not reverted even if requested by users and downstream maintainers. If it continues this way, we will end up creating something that old users don't want and new users don't need.

What users need to understand ?

• This code is unmaintainable and it's hard to find people who are willing to work on it.

• @krader1961 is a brilliant engineer.

• His behavior has been inconsistent. Sometimes he is quite welcoming to new users/contributors, and sometimes not.

• He is not going to change his behavior.

What do I want from this project ?

• I would like to put new ksh in RHEL at some point and I want to do it with minimal hassle. In short, like other users I want new ksh to be as close as ksh93u+ in behavior.

• I will say it again. If there are changes made in behavior, they should not break compatibility with ksh93u+. If those changes have to be made, they should be minimal.

• I care about performance as long as it comes from maintainable code. If some of the performance changes were due to dropping subsystems like vmalloc or refactoring code to be more maintainable, I am fine with performance loss.

• My view of ksh is pretty much biased towards RHEL. Other vendors that I talked to wanted to get rid of ksh altogether. They regret using it.

How to go forward from here ?

• I have no problems if community wants to vote for new maintainers. I am willing to get my commit access revoked and will support the new maintainers as a package maintainer for RHEL/Fedora (as long as they agree with my goals).

• If I remain maintainer:

  • I am not going restart from ksh93u+. I will try to restore performance as much as I can, but will not revert to old subsystems like vmalloc, stdio (that was based on sfio) for performance.

  • I will revert any backward incompatible changes that may create issues with old scripts.

  • I will drink less beer.

• If @krader1961 wants to continue making changes in ksh behavior, it should be done in a separate fork.

• The users who are doing flame wars in this repository should try to make more meaningful changes in the source code.

[gordonwoodhull]

Hi all!

I am not knowledgeable about ksh but I have plenty of open source experience, and I am an admin on this repo since I work at AT&T. I'm going to help @lkoutsofios with this transition, however it goes.

It's a tough situation when the original maintainers abandon a project with no successors, and especially tough when they abandon it involuntarily.

It sounds like we are close to a consensus that the repo should be backed out to a prior version, and at least @krader1961 should fork if he wants to take it in a different direction. But there are a few conflicting proposals about what version this repo should be backed out to.

There shouldn't be any bad feelings in forking a repo when the original maintainers have left. It's just a shame that there wasn't any governance in place and we ended up with multiple maintainers with commit access but different goals.

Again, I'm not a stakeholder or knowledgeable about the technical issues, but in some ways backing out to a truly stable early version sounds cleanest to me. Cleanest in the sense that this fork would be the "AT&T Version". Then others can fork whatever other versions they want.

If I back anything out, I will of course leave other versions on branches. Nothing will be destroyed.

I'm open to all suggestions. I will follow the discussion and see if there are other stakeholders and other opinions.

(I am not going to become a long-term maintainer; just want to help resolve this conflict and restore the repo with my git skills.)

[jghub]

@gordonwoodhull:
I am very happy that my "SOS call" reached you (it was a last resort after months of trying otherwise) and that you care to intervene/help.

a first feedback: my best guess is that @siteshwar will have a somewhat different view from the rest of the followers of this project regarding how/whether to backout and start over. that is, I understand he does not want to do that but to go forward from the current state of the project -- which I would find problematic at the very least.

so I would agree completely with what you seem to propose: go back to last truly stable version (that most definitely is 93u+, impossible it could be 93v-) and make it 'longtime survivable' (new build system) as "the ATT ksh93" shell. what is @lkoutsofios opinion here? that would help a ton in my view. others should chime in to voice their opinions, of course.

case in point: some harm had already been done in recent months in different distros by ksh2020 displacing ksh93u+ as "the ksh" package due to confusion/unawareness on the side of the package maintainers. specifically this has happened at least in FreeBSD and Debian and OSX. don't know whether FreeBSD is aware now and will follow debian to again provide both (see below). OSX can't do that right now due to inability of compiling the old ksh93u+ code (with the nmake based built-system...) and provides only a "ksh" package containing ksh2020 while listing the old ksh93 package as "obsolete and replaced by ksh". the maintainer there clarified that he would be happy to provide the true ksh93u+ as well if I tell him where to find an easily buildable download .

and in debian it has only been reverted very recently after bringing the problem to the attention of the maintainer there (so that there now is a ksh93u+ package back and in parallel k2020 (which is no problem: if someone likes it better than the real thing: fine, go ahead...).

so it would definitely be beneficial if the "branding" question would be sorted out unambiguously: what is (and what is not) the ATT ksh shell.

overall it seems, ksh2020 should find its own place as a fork (if the project insists on maintaining commit access for krader..., otherwise, possibly, on a branch), get a different landing page (no longer claiming that the visitor is looking at the AST software...) and clearly describe itself as a ksh93v- descendant to be unambiguously distinguished from the ATT ksh93 shell (i.e.93u+) and, possibly, its carefully curated future bug-fix releases. my feeling is: a fork would be the best solution and most accurately reflect what has happened in ksh2020.

I will further discuss with colleagues and come back here if any new aspects/ideas pop up.

[jghub]

@siteshwar

we will not reach full consensus in our assessment but for the record:

• It was me and not @krader1961 who made the choice to use meson/ninja as the new build system. So
  at least he should not be blamed for choosing build system thats incompatible with old systems. I don't regret my choice and I am not going to use any other build system. No one (not even people who disagree with @krader1961) likes the old build system and all of us unanimously agree that it should have been changed.

I believe nobody really blamed krader (or you) for the choice of build system. it seems of tertiary importance ("as long as it builds...").

• @krader1961 has made major changes to this codebase since he got access. This includes reformatting the code, dropping old (irrelevant) code, upstreaming patches from other distros

define "irrelevant". problem starts here...

• I don't agree with some of the changes made by @krader1961 recently for e.g. recently changed cd behaviour (_cd function, autoloading etc) breaks existing ksh scripts #1445. They were not reverted even if requested by users and downstream maintainers. If it continues this way, we will end up creating something that old users don't want and new users don't need.

regarding the forseeable outcome ("old users don't want, new users don't need") we agree 100%.

What users need to understand ?

• This code is unmaintainable and it's hard to find people who are willing to work on it.

• @krader1961 is a brilliant engineer.

I challenge that assessment, given his actions and working style in this project. what user's definitely need to understand is that krader cannot deal with disagreement and cannot adjust.

• His behavior has been inconsistent. Sometimes he is quite welcoming to new users/contributors, and sometimes not.

I disagree: his behaviour is very consistent. he is consistently hostile to any criticism and disagreement. and this , for him, comprises simple bug reports (look at top of this issue just to get the idea....). his inability to compromise or step back is archetypical illustrated in #1445 (including the horrendous "issue closed"). a ksh project cannot afford in the long run to waste hours and hours of time to deciding the question "should builtin cd behaviour be maintained?"

• He is not going to change his behavior.

100% correct. this has very obvious consequences for this project which you, seemingly, try to avoid or do not want to acknowledge. at least I get this impression...

What do I want from this project ?

• I would like to put new ksh in RHEL at some point and I want to do it with minimal hassle. In short, like other users I want new ksh to be as close as ksh93u+ in behavior.

for you to decide. it seems not even to collide with what the rest of the community seems to want: maintain ksh93u+ and ksh2020 packages in parallel.

• I will say it again. If there are changes made in behavior, they should not break compatibility with ksh93u+. If those changes have to be made, they should be minimal.

this deserves the question "if ksh93u+ (plus the patch suite from RHEL and others) can be made to build easily and if no new relevant features are in ksh2020, what are the remaining advantages (vs. its disadvantages) of ksh2020?" k2020 heavily carries on its ksh93v- heritage and also on krader's changes (we will see whether it is only the memory management thing which brought down performance by up to more than a factor of 2 in comparison to 93u+, e.g.). so stability- and performance-wise I suspect ksh93u+ will continue to out-class ksh2020 while the latter has not yet managed (as far as I know) to make relevant (but broken) ksh93v- features available. correct me if I am wrong.

• I care about performance as long as it comes from maintainable code. If some of the performance changes were due to dropping subsystems like vmalloc or refactoring code to be more maintainable, I am fine with performance loss.

I am not. maintainability is not mutually exclusive with performance. performance is THE singular massive advantage of ksh over other shells. by no means should a factor of 2 slowdown (or even 30% in a mixed application and across the board) be accepted. bash and zsh still have a long way to go but they are becoming faster with every release (bash partly massively so). I don't believe that making ksh converge to bash performance by becoming slower is a good strategy or an irrelevant detail.

• My view of ksh is pretty much biased towards RHEL. Other vendors that I talked to wanted to get rid of ksh altogether. They regret using it.

might be. ksh2020 will not change this attitude.

How to go forward from here ?

• I have no problems if community wants to vote for new maintainers. I am willing to get my commit access revoked and will support the new maintainers as a package maintainer for RHEL/Fedora (as long as they agree with my goals).

I do not know exactly what you consider "the community". there are only a handful of people at the freenode #ksh channel and another (partly overlapping) handful following this project. so whose vote would you want and accept?

• If I remain maintainer:

  • I am not going restart from ksh93u+. I will try to restore performance as much as I can, but will not revert to old subsystems like vmalloc, stdio (that was based on sfio) for performance.

why state this so apodictically? there might be different options, even selectable at builtime #1449 (comment). if it is the final consensus to stay clear of vmalloc (why, exactly, if the built process would run seamlessly?) ok. but to make this a conditio-sine-qua-non?

• I will revert any backward incompatible changes that may create issues with old scripts.

that seems mandatory.

• I will drink less beer.

that seems optional. and difficult to pull off in the czech republic :).

• If @krader1961 wants to continue making changes in ksh behavior, it should be done in a separate fork.

s/If/Because/
s/should/must/

meaning: krader will not change his ways (you acknowledged so much) -- and I will go so far as to say he is unable to change them -- and because of that he must fork and have no longer commit access here.

• The users who are doing flame wars in this repository should try to make more meaningful changes in the source code.

I object against this description of what many users (at different times) have voiced here, notably regarding my own posts. if you like, I can point you to all my "discussions" with krader and you might decide who was dictating the "atmospheric conditions" or you can look at some of his rants against D.Korn and the code base: those are flame wars (remember the "steaming pile of caca"? where did any user succumb to that choice of words even if they describe krader's work much better than D.Korn's?) .

I have seen stupid flame wars elsewhere and I despise them. most everything I have read here was very sober and factual (apart from krader's contribution, to be sure).

and even if I cannot fix it myself, I can point out a bug (this issue) or maintain that cd behaviour should not be touched (#1445) or report a performance breakdown that went unnoticed by you and krader for the whole history of this project #1449. I do hope your statement is not to be interpreted as "shut up if you cannot fix it yourself"?

[jghub]

@siteshwar: shortly coming back to the actual starting point of this thread. thank you for reopening it. I'd also request to relabel it as a bug and to remove the "notabug" label (which is silly: it of course is exactly that: a bug). it also is not a god-given situation that this cannot be altered (just learned from mirabilos that mksh does handle the example just fine, for instance)

[siteshwar]

define "irrelevant". problem starts here...

Removing code that was related to supercomputers that became obsolete in the 90s. I can't find relevant commit on quick look.

I do not know exactly what you consider "the community". there are only a handful of people at the freenode #ksh channel and another (partly overlapping) handful following this project. so whose vote would you want and accept?

Ask @jelmd, he mentioned about this.

I do hope your statement is not to be interpreted as "shut up if you cannot fix it yourself"?

Most of the people involved in flame wars in this repository have zero code contributions. I just stated it as a fact as I mentioned at the beginning of my post.

[jghub]

define "irrelevant". problem starts here...

Removing code that was related to supercomputers that became obsolete in the 90s. I can't find relevant commit on quick look.

would be important to review and verify this claim. there are a multitude of instances where krader stated falsehoods/misconceptions as a justification for exterminating features/code. if 100% (rather than "probably") true in the end, sure. otherwise: just leave the dead code in, for the time being. it is a fringe issue.

I do not know exactly what you consider "the community". there are only a handful of people at the freenode #ksh channel and another (partly overlapping) handful following this project. so whose vote would you want and accept?

Ask @jelmd, he mentioned about this.

let's wait for him to reply if he feels like it.

I do hope your statement is not to be interpreted as "shut up if you cannot fix it yourself"?

Most of the people involved in flame wars in this repository have zero code contributions. I just stated it as a fact as I mentioned at the beginning of my post.

I still cannot parse this: I do have zero code contribution. nearly everybody has zero code contribution. krader has tons. so? does that make him right (both technically and socially)? does it invalidate the raised questions? does it make the reported observations/bug reports wrong? and make the people who spoke out disqualified for having a clear grasp of what is happening here? no, no, no, and no in my world.

and no code contributions very well might be better than most of those of krader. I iunderstand (and sympathise to a limited extent) with an "it is easier to criticise/demand than to fix" reaction to people coming at you with complaints, feature requests, and bug fix "demands". but that is decidedly not the situation here.

I reiterate that the "flame war" attribute is unjustified and it feels like a tactical defence move (sorry). so I reject it. there are very valid and crucial issues at stake. and people spell these out. it's far from being an ideologically driven flame war of some adolescent fan boys of one persuation or other. it just is disparaging and counterproductive to label the ongoing discussion as such.

maybe we better should focus at the real problems...

[siteshwar]

@gordonwoodhull Thanks for chiming in.

I'm open to all suggestions. I will follow the discussion and see if there are other stakeholders and other opinions.

There are few questions to answer:

• Most important question is who is going to be the lead maintainer if @krader1961 makes his changes in separate fork. Is there anyone watching this repository who is willing to volunteer ?

• The person that will be the lead maintainer should decide where to start from. If we choose to go back to ksh93u+ release, distributions maintainers should be made aware about it as some of them have moved to latest release.

• We need to clearly document goals of this project, so that people don't deviate from it.

• It should be clearly stated that it's a community project now and AT&T has least involvement in it.

• We need to have a policy to resolve conflicts.

[gordonwoodhull]

The authentic AT&T Version will be ksh93u+.

Since there is no one inside the company to maintain the project, that will be the end of the original fork, and we can archive it.

I’ll be glad to help port issues to another repo, if desired, and update the README to point to any forks which continue from that point.

If there is any volunteer to maintain a community fork, they could use their own account, or start e.g. a ksh organization. (A common solution in this situation.)

[siteshwar]

Since there is no one inside the company to maintain the project, that will be the end of the original fork, and we can archive it.

That would be the death of ksh. We will loose all the visibility we have gained in last couple of years and new fork will have to gain same level of visibility as we do now from scratch.

[gordonwoodhull]

If you are doing good work, people will follow. It has worked out fine for another project I maintain:

https://github.com/crossfilter/crossfilter

Exactly the same situation. The original company, Square, had ceased maintaining the project. I stepped in with a few other people, forked the project, cooperated with Square to link from their repo, and it has continued to flourish.

There is nothing special about the att organization. The code is what matters.

[jelmd]

Well, perhaps I was wrong and @siteshwar still needs some time - a maintainer should be able to describe "community". In the context I used it above, I think the definition given on wikipedia is sufficient: "a social unit that shares common values".

However, thanx @gordonwoodhull for helping to solve this repo problem. IMHO your plan is right. It gives distro/app maintainers as well as its users the reliability and confidence they need. It might be a hard decision, but I rather see this as necessary step to keep ksh93 alive. And yes, I like the idea of a central repo where at least issues and patches/PRs get collected, but it should run under a different label ...

But before the ast repo gets archived, I think it would be nice, if @lkoutsofios could setup a separate ksh93 repo with just the things required to build ksh93 as mentioned in [1] and when done to archive it as well. This would probably help people to fork and get to work much faster.

[kdudka]

I do not think that downstream consumers care too much whether they download release tarballs of ksh from att/ast or ksh-reloaded/ksh, as long as the software works well for them. For Red Hat it means basically two things:

1. Scripts written for ksh93 continue to work. This is not much compatible with the approach of @krader1961.
2. We are able to maintain it (build the software in predictable way and in reasonable time, we are able to fix bugs and security issues, etc.). This is not much compatible with the goals of some community members (requesting to keep support for decommissioned HW/SW, etc.).

If AT&T and the original community want to keep the brand solely for ksh93u+, it makes sense to me. We can rollback the master branch (while keeping ksh2020 in a non-default branch to not loose all the info recorded in issues and pull requests) and put there a README clearly stating that ksh93u+ is not being developed any more and list the existing forks with their pros and cons.

Would it make sense?

[gordonwoodhull]

Thanks all. I made a concrete proposal in #1466, let's discuss there.

[jghub]

principally, I agree with everything @gordonwoodhull has proposed/decided: thank you.

I disagree with @siteshwar that this would be the "death of ksh". rather to the contrary. going on without a clear cut would do decisive further (maybe mortal) harm to ksh. and the 'visibility' he talks about seems to be mostly spurious (or identical with the visibility here where there is near unanimous agreement that ksh2020 went off on a bad tangent right from starting with ksh93v-). maybe @siteshwar can reconcider his position to prevent fragmentation?

I also second @kdudka that downstream users are just interested in a predictably working true ksh93(u+..., for the time being) and the consequences this has for distros (not only RH). except that I am not sure (and have not seen statements to that extent around here that I can recall) that support of ancient HW/SW (beyond strict backward compatibility of ksh language-wise) is a crucial issue for a sizeable part of the community. so maybe there is no collision of interests regarding your second point at all, making it a non-issue?

@kdudka: regarding how exactly to make the transition, I am not sure whether I completely understand your proposal/intention. but the main idea seems to keep the present repo (instead of starting over from "clean" repo) and restructure it? maybe that would be good, but depends on what exactly is the plan.

I would like the idea of putting ksh2020 on a "dead end" branch (if that is the plan) to maintain easy access to the issues (however questionable their factual correctness, partly...) and fixes that are relevant for 93u+ (possibly not so many since most fixes related to ksh93v- and ksh2020 only problems...) . but by no means should that branch see then further activity in my view: ksh2020 development (= krader) needs to happen in a fork and the ksh2020 branch in the present repo should not accept pull requests but be in a "sealed/archived" state just for reference. development would exclusively happen for the ksh93u+ line. I would find this clear separation of ksh2020 and ksh93 mandatory for all the reasons that have let to the present need of a reset and in order to definitely avoid replication of the confusion out there between ksh93 proper and ksh2020. the distinction needs to be very clear.

it then seems now consensus (except with @siteshwar, possibly) that master should be rolled back to prestine 93u+. this would be good. after changing the build system (and nothing else!) that would provide the legacy ATT ksh93u+ shell for distros to use (modulo first including the known solid patches from different source but without overdoing it at this point?).

if that state is reached the many year hiatus in easy ksh93u+ availability would be at an end (which is the only real danger for ksh's survival in my view). that would probably be the time to put the result on a "ksh93u+-legacy" branch.

and further bug fixes could than happen on master. but with a policy similar to the one observed by the AST team of D. Korn et al.: carefully and one by one and with a very conservative attitude of integrating patches/pull request after thorough discussion and testing (in order to avoid duplication of the ksh2020 mess with a multitude of cuts and behavioural and feature changes/deletes nobody ever kept track of or cared to document beyond commit comments of questionable quality and reliability).

if this is of interest: @jens-maus has signalled principal willingness to look into the build system issue (considering Cmake so far as the way to go). I am confident he would do a good job. I would be able to help in testing (not so much with the Cmake stuff). any opinions whether this would be welcome? or whether to categorically stay with autoconf? or even meson (but most people do not seem to like that road)?

[siteshwar]

I disagree with @siteshwar that this would be the "death of ksh". rather to the contrary. going on without a clear cut would do decisive further (maybe mortal) harm to ksh.

We will see. I wish you good luck with your efforts. I will request to get my commit access revoked (it was going to happen anyways) and will move on.

[kdudka]

@kdudka: regarding how exactly to make the transition

It does not really matter as long as the references to the issues and pull requests (and the references inside them) stay valid.

after changing the build system (and nothing else!) that would provide the legacy ATT ksh93u+ shell for distros to use

The problem is that this task is more difficult than one would expect. If you want to keep support for all the obscure HW and SW at the same time (as repeatedly requested here), it is nearly impossible.

and further bug fixes could than happen on master. but with a policy similar to the one observed by the AST team of D. Korn et al.: carefully and one by one and with a very conservative attitude of integrating patches/pull request after thorough discussion and testing

I think you are underestimating the amount of work needed for this. Most of the users report just issues without patches. Someone has to debug them, write the patches, and review the patches. Given the shape that ksh93u+ is currently in, I think it requires at least one full-time employee dedicated to this.

if this is of interest: @jens-maus has signalled principal willingness to look into the build system issue (considering Cmake so far as the way to go). I am confident he would do a good job. I would be able to help in testing (not so much with the Cmake stuff). any opinions whether this would be welcome?

This would certainly be welcome and I wish you good luck with that!