Proposal for 'vim' of a PDS/PDSE member

[MikeFultonDev]

It would be nice to be able to edit a PDS(E) member (and perhaps a sequential dataset and/or GDG member) using vim.

To implement this, there are some considerations:

• How we determine whether the request is for a dataset or for a file? There are several options:

  • Use an option, e.g. --dataset to indicate it is a dataset and not a file we want to edit
  • Use the 'escape' of a file that starts with // is treated as a dataset and otherwise it is a file
  • Limit support to PDS(E)'s and if the name has parentheses in it, e.g. FULTONM.PROJ23.COBOL(PROG) to assume it is a dataset unless some option is specified to override it.
  • My opinion would be to use the 'escape' approach and then perhaps provide a one-line script people could use (or alias) called dvim that would call vim under the covers with the // syntax added in, e.g. dvim 'fultonm.proj23.cobol(prog)' would be a wrapper to vim "//'FULTONM.PROJ23.COBOL(PROG)'". This is also consistent with C services in USS as well as being consistent with ZOAU (for the dvim model).

• How we work with the underlying dataset:

  • Background: The implementation of vim uses POSIX I/O 'file descriptor' services (e.g. open, read, write, close) and not POSIX I/O 'file pointer' services (e.g. fopen, fread, fwrite, fclose), and also needs to have multiple active read file descriptors active concurrently. As such, providing 'native' dataset support in vim would be very complex.
  • There is however a simpler solution that would likely work well in practice: after detecting a dataset edit is requested:
    • Read the file into a temporary file in the file system (this is done by vim typically for backup/restore reasons anyway
    • Set the textwidth and colorcolumn based on the record length of the dataset so that people are less likely to exceed the record length of the dataset
    • The developer will then be editing a file (not a dataset) and the backups will be files not datasets
    • On save, the file will be written back to the dataset. At this point, we will record if there is a failure in the write (most typically because a record length is exceeded, but also possibly if the dataset is full). We would provide a diagnostic to indicate which line(s) were too long (perhaps just the first line that was too long? We could provide an option to truncate a record possibly - perhaps only if it is spaces?

[IgorTodorovskiIBM]

Use the 'escape' of a file that starts with // is treated as a dataset and otherwise it is a file

+1 - simple and effective.

I like it, so the initial scope would be limited to editing existing datasets? Curious how important will being able to create new datasets via vim be to users? We could also assume all dataset contents are in EBCDIC for the time being (and auto-convert to the ASCII since Vim is compiled in ascii mode)

[MikeFultonDev]

Good point about whether the dataset exists or not. I would say a good 'initial scope' would be the dataset exists. It's easy enough to create one with the attributes you want first (either through ISPF 3.2 or via dtouch ZOAU command).
Code page is interesting - 'EBCDIC' is a bit 'loose' - perhaps we want it to be the active code page for the MVS side of the house (which might be 1047 but could be 037 or some other variant?)

[AnthonyGiorgio]

It's easy to get the current POSIX locale, so we could use that as a baseline to interpret the dataset codepage. Since the user is running vim from USS, you'd expect to be able to tune the local from there.

[perry-ca]

We probably want to add a flag to vim so the user can set the locale, much like the LOCALE() option in the XL compiler. How does xedit handle code pages?

[lbdyck]

The likelihood of a GDG being a PDS is extremely rare, in fact I've never seen one in my 52 year career working with MVT 21.6 up thru z/OS 3.1. One of the (many) challenges with working with GDG datasets is that the dataset name appears to be be a PDS as the generation is where the member name would be - for example HLQ.DATASET(45) for absolute generation 45 or HLQ.DATASET(0) for the current generation or HLQ.DATASET(+1) to create a new generation.

[AnthonyGiorgio]

The neat thing about GDGs is that they aren't valid dataset names. You can't start a member name with a digit, +, or - character. ZOAU has some code that translates a GDG name to an absolute dataset name - perhaps zoslib could do something similar?

[AnthonyGiorgio]

Limit support to PDS(E)'s and if the name has parentheses in it, e.g. FULTONM.PROJ23.COBOL(PROG) to assume it is a dataset unless some option is specified to override it.

I don't like this. Filenames can have parentheses in them, and this seems like an arbitrary limitation without much upside.

[AnthonyGiorgio]

I feel that the baseline case for vim and datasets should be editing a parmlib member. Sometimes I want to make a quick change to BPXPRMxx, and I have to either drill down in VSCode+Zowe or fire up a 3270 and ISPF. It would be really handy to just to do:

vim "//'sys1.parmlib(bpxprm00)'"

[lbdyck]

Your comments on dsfs make sense - however I've had a number of IBM Ideas closed as delivered with dsfs as the solution so it would seem that IBM is pushing sites to configure it - which is actually very simple and since all RACF (or ACF2/Top Secret) security is in force there are no security issues (one hopes). In fact configuring dsfs was easier than configuring ssh :)

But providing the "//'hlq.dataset(mbr)'"` option still makes sense, if only for consistency among the z/OS shell commands.

[AnthonyGiorgio]

One thing I noticed about DSFS is that you can configure which HLQs are exported by the product. It's entirely possible that something like SYS1 won't be available by design. In that case, vim with the dataset path would still work.

[lbdyck]

sadly you're probably right - it would be a shame to prevent it but one never knows what auditors or secadmins will do - paranoia can run wild.

[lcarcaramo]

Its not that DSFS is bad per se, its just that there are too many thing that can get in the way from an accessibility perspective which would cause a lot of headaches in my opinion.

At the end of the day, the MVP should be, if you have z/OS Unix System Services setup with SSH access and vim, it should just work when opening a data set for editing.

[AnthonyGiorgio]

This is doubly true now because vim can come with your z/OS image if you select OEF during the Shopz install screen.

[v1gnesh]

We would provide a diagnostic to indicate which line(s) were too long (perhaps just the first line that was too long?
We could provide an option to truncate a record possibly - perhaps only if it is spaces?

Maybe another discussion altogether...
Does the LSP in Z Open Editor help with keeping syntax/format?
If so, can it be configured with rules to apply formatting, something may vary in each site.
Ex: JCLPrep or similar tools will have their own "opinions" of how to format, which may not make things easily readable.

Another question with editing in general is... when working with a .py file for example, even if a language plugin isn't in use, we can just tab once to get the required indentation. Can vim be made smart this way, to interpret tab as insert 'n' spaces until the column where this line is allowed to begin ?

[lcarcaramo]

For me, what is paramount is that vim uses the standard syntax that other unix commands use "//'MY.PDS(MEMBER)'" and that there is no dependency on DSFS to make vim as an editor for data sets as accessible as possible.

I also agree that vim should automatically truncate lines if they only contain spaces. Most of the time you don't care if there are extraneous trailing spaces and this is actually quite annoying when copying data from a file to a data set with the cp command.

As much as I would like vim to be fully ported to just work natively with data sets, I am ok with an implementation that uses temporary files, but my guess is that there is potential for some edge with creating the temporary file, but I guess that wouldn't be too different from what vim does with just regular files. Maybe to start, we could use the temporary file method, and then potentially look further into fully porting vim to just work natively with data sets as that sounds like it would be significantly more work and even potentially not viable. It also might also not be worth the effort potentially if the temporary file method is robust enough.

[lcarcaramo]

There is also no need to try to have vim automatically create data sets for you. Requiring the data set to exist is totally fine. ZOAU provides dtouch to help facilitate the creation of data sets.

[perry-ca]

Read the file into a temporary file in the file system (this is done by vim typically for backup/restore reasons anyway

Good idea. We will need to do something to ensure the file doesn't get modified somewhere else while the file is being edited in vim. May be open the file for read/write at the start (and ensuring a vim crash doesn't clobber the file) to prevent other people updating the file.

[perry-ca]

Does vim have a column mode? It's one thing to be able to open the files in vim. It's another to give the user a productive editing experience. I remember from my days of using xedit that column based movement was important.

[v1gnesh]

Am I (poorly) referring to the same thing in this comment?

[perry-ca]

I think they are different things. I like your ideas too.

What I'm thinking about is xedit had a mode where you can do editting based on columns instead of lines. You could add a column (i.e. add a space to every line at column X), delete a column and many other things you can do with lines but rotated 90 degrees. You could even do things like add a column for a range of lines or more commonly put a character in column X of every line in the range (eg. mark lines as comments). The column based functionality was very helpful when working with JCL, COBOL or other fixed column languages.

[v1gnesh]

Sounds like welcome ISPF tricks :)

[AnthonyGiorgio]

vim already has features that can work with columns.

[MikeFultonDev]

I am going to mark this as 'done' since we've got a good base implementation. If there are problems or enhancements, please open something new and feel free to reference this.