Overhaul spork's docs

[CFiggers]

This PR proposes a full overhaul of spork's doc/ folder.

Summary of changes:

• I propose adding a "folder structure" to the navigation sidebar that mirrors the structure of janet-lang.org.
  • In this PR, the "Spork" introduction and landing page is a top-level page at the top of the navigation tree.
  • Module-specific pages are now under a top-level "API" header page, just like we have on janet-lang.org (i.e., .mdz files relating to each module in spork are moved under doc/api/).
    • The "API" header page itself is a full index and a comprehensive generated function reference for all functions in the library. Much like on janet-lang.org, this page is convenient for Ctrl+F searching.
• I propose that every module in the spork library should have a doc page, however basic.
  • In this PR, I've created new pages for modules that didn't have one. Wherever comments exist in module source files those comments are copied/converted over to .mdz format.
  • Every module page has an auto-generated reference section at the bottom displaying its exported functions, created using Mendoza's @api-docs() macro.
    • Generating docs for each module in this way revealed entire modules missing from spork/init.janet, which could have repercussions elsewhere; this PR also proposes adding those in.
    • Some of the existing doc pages essentially consisted of hand-crafted versions of the reference entries that Mendoza's @api-docs() macro generates. In those cases (notably, the docs for generators, math, and pgp), the now-redundant hand-rolled function references are removed.
  • Each page is now (re)named to correspond exactly with the filename of its module. This aids identification when looking for docs related to a specific module, and follows the convention of janet-lang.org's module-specific pages.
• I propose adding a section to the README explaining how to build the docs site and serve it locally using Mendoza.
  • Without this information, the docs are essentially inaccessible to anyone not deeply familiar with Janet's extended (and less-recently updated) ecosystem (specifically, Mendoza).
• I propose small tweaks and polish throughout (mostly limited to the doc/ folder, but also catching some typos in two comment headers of actual modules).

If these additions and updates are judged to be good, my next thought after this would be that these improved and expanded docs should be incorporated (with adaptation, probably) into the spork-related section on janet-lang.org, or made into a GitHub-pages site hosted from this repo (similar to the way Mendoza hosts its docs on GitHub Pages) or even both.

I also see a lot of room for further improvements to be made—pretty much every page could use examples, explanation of use cases, rationale, etc.

Constructive criticism is quite welcome.

[CFiggers]

For anyone who is unfamiliar with Mendoza, you can install it with [sudo] jpm install mendoza and then run mdz build && mdz serve in the spork project root (either the current master in janet-lang/spork or in this PR's project root if you want to see my proposed changes). Then open a browser to http://localhost:8000 to view the generated static site for as long as the mdz serve process is running.

[sogaiu]

There is a lot to look at here and I think it's very helpful to have the instructions for being able to view the content in one's browser, thanks [1]!

The "API" header page itself is a full index and a comprehensive generated function reference for all functions in the library. Much like on janet-lang.org, this page is convenient for Ctrl+F searching.

The consistency and convenience aspects seem good.

Generating docs for each module in this way revealed entire modules missing from spork/init.janet,

TIL! It's this bit, right? (Just to make it a bit easier for others folks to see at a glance.)

made into a GitHub-pages site hosted from this repo (similar to the way Mendoza hosts its docs on GitHub Pages)

I think having an independent doc site has the benefit of decoupling updates from the update timing of janet-lang.org.

On a related note, I think there might be some benefit to just linking to spork docs from janet-lang.org instead of the current arrangement of including details there. May be that's a topic for another day.

I'm not a fan of having the info in two places for synchronization and maintenance reasons.

One concern I have though is whether this would overall be an increased maintenance burden -- not sure (^^;

---

[1] I did see similar instructions you had added in the PR, but I think it's good to have those surfaced in this discussion.

[CFiggers]

Thanks for having a look, Sogaiu.

One concern I have though is whether this would overall be an increased maintenance burden -- not sure (^^;

Since 90% of these docs are generated directly from the source, the maintenance burden of a GitHub pages site hosted out of the spork repo should be close to zero once the deployment pipeline is figured out. GitHub can automatically re-build and publish new static pages whenever there's updates to the repo, so at a minimum the generated function references should always stay in sync with the master branch. It could go stale if new modules aren't added over time, of course, but that's hardly going to be worse than the state of the docs in the current janet-lang/spork master, where half of the current modules don't even have a page, almost all lack @api-docs(), and there's no explanation provided for how to even read them.

On the subject more broadly of maintaining documentation... this isn't directly relevant to this PR, I'm just seizing the opportunity to write out some thoughts I've had for a while—just to be honest, the Janet ecosystem seems to be highly tolerant of incomplete or outdated documentation.

• Many of the more recently-added standard library functions are not added to JanetDocs.com and 404 when you click "Community Examples."
• At least one namespace recently introduced to the standard library does not have a module-specific page yet on janet-lang.org (there should be a page under "API" dedicated to the bundle/* functions, and right now there's not). bundle also really needs an article under the "Documentation" header, IMHO.
• The front page of janet-lang.org still references Janet 1.17 in the example CLI usage output, and still links to janet-lang/json, a repo that has long been moved into spork and declared a public archive. The front page of janet-lang.org doesn't mention spork anywhere and only just barely mentions jpm, and IMHO it should address both prominently.
• jpm has only a single page of explanatory docs anywhere that I can find (the one on janet-lang.org) and that one page doesn't give a comprehensive list of the valid declare-* options that can be used in project.janet files. The Mendoza-generated function reference for jpm on the site is overwhelming and has nothing to do with how jpm is typically used.
• Prior to this PR, spork's documentation is, if not in total disarray, at the very least highly incomplete, isn't hosted anywhere convenient, and is totally lacking explanation for how to read it.

(All of this is only related to documentation—there's more to say about load-bearing Janet projects that haven't had even maintenance commits in years... consider Circlet and Joy.)

Yet, despite all the (IMHO pretty clear) documentation deficiencies, the Janet world spins on. ¯\_(ツ)_/¯ Janeteers generally seem content reading the source to figure out how to use stuff. Which, on net, might be more feature than bug in certain respects, but it's undeniably forbidding to newcomers. We simply don't know how many people started trying to get into Janet and bounced off because, for example, setting up jpm is borderline-necessary to use Janet and yet can also be non-obvious, less-than-helpfully documented, and prone to repeated failure, even for generally experienced programmers.

But at any rate, the community's general tolerance for missing or outdated documentation is no reason not to improve it when and where we can.

This PR isn't a perfect solution, but it moves the ball down the field in what I feel is the right direction. And, if maintenance of these improved docs goes also by the wayside over time, it'll at least go there in a better state than they're currently in. :)

[bakpakin]

So the spork documentation has moved to the main website to be easier to maintain. I agree though with a lot of the other points here, we can always use more documentation.

https://janet-lang.org/api/spork/index.html

As for jpm problems, that is more related to JPM itself and people trying to customize their install without doing things the default way first and understanding how the paths are laid out. We are working to simplify that, mostly by removing a lot of configuration options.

EDIT: link to new location of source code: https://github.com/janet-lang/janet-lang.org/tree/master/content/api/spork

I don't personally see much value in keeping the docs here up to date - presumably we should delete them and leave a README pointing to the new location.

[sogaiu]

A bit tangential, but I'd like to put down some related thoughts here.

non-obvious, less-than-helpfully documented, and prone to repeated failure, even for generally experienced programmers.

I watched the live stream mentioned above and I think the main issue was that the operating system in question was guix (likely a similar story for nixos). Those systems do things in ways that are significantly different from most others [1] and IIRC it's not uncommon when packaging / installing to have to make "adjustments" (perhaps more than "ordinary" distributions might make).

The System Crafters guy (David Wilson), was clear about some of the difficulty likely being a result of using guix. I thought he was pretty fair in his assessment throughout the stream.

I believe another factor was that this was a live-stream (and allegedly certain steps had not been rehearsed). Some of the participants had helpful suggestions - e.g. @alternateved mentioning this issue. Even though that issue clearly spells out a solution, my impression was that closely reading the content was not a priority initially [2]. It took a while, but David got back to it at around 1:06:26 and an additional comment from alternateved about a non-root install plus a link seemed to be enough.

Having said the above, I don't disagree that documentation could continue to be improved and I consider this PR to be a plus (or perhaps several (^^; ).

---

[1] This is not a criticism -- I used to use both of those systems and can see a fair bit of value in them.

[2] Perhaps because you don't want your audience to be waiting...

[daviwil]

Hey folks! @alternateved pointed me to this issue so I just wanted to chime in.

First of all, love Janet! I've continued playing with it every day since the stream, planning to work on a couple projects with it to get a better feel for the language and dev workflow. @ianthehenry's Janet Guide has been super educational and enjoyable to read.

@sogaiu is correct that Guix was the main culprit in the trouble I faced getting jpm to work. I probably would have had less trouble if someone hadn't already packaged Janet for Guix because I would have just followed the steps to build Janet and jpm from source in the official docs. Those steps work perfectly on Guix and Debian, makes it really easy to get started.

I spent some time over the weekend trying to package jpm in Guix and got it mostly working but there's still some path issues to work out. The main problem is that the path where Janet gets installed is different than the path where jpm will be installed (they are two separate folder paths with their own root-level OS path structure (./lib, ./bin/, etc) so extra care has to be taken with the JANET_ path configuration.

Side note: it may be worth enabling JANET_TREE accept a delimited series of paths (if it doesn't already) so that package managers like Guix and Nix can generate an env variable containing the installation paths of all Janet dependencies that have been packaged for distribution. Any Janet application that gets built on Guix would be wrapped in a small shell script that sets JANET_TREE to the list of paths where its dependencies will be found.

All that said, I'm just going to move forward with my Janet experimentation using a local source build since I might end up sending some patches anyway.

Kudos on your work building Janet, @bakpakin! I started writing a Scheme-derived language and compiler called Mesche a couple years ago with a lot of the same goals, but Janet is much farther along (and more well-designed, IMO). I think I'll be able to build some cool things with it!

[sogaiu]

Regarding paths, guix, and janet, possibly of interest might be this discussion.

---

Edit: To spell things out a bit more, what I thought might be worth consideration were the opening two paragraphs:

I'm working on improving the Janet and jpm-installable module experience on Guix. Currently, we don't have any module packages, but this is something that I'd like change. Guix, like its cousin Nix, installs software differently than traditional operating systems. Essentially, rather than installing all packages into the same central location, each package gets its own directory hierarchy. Multiple views representing different subsets of packages can be created and used for system-wide installed software, software installed for each user, temporary development environments, etc.

For this use case, I don't think setting JANET_PATH would be sufficient since it only contains one directory. I think what I need is a search path that supports a colon separated list of directories where modules can be found. In order to get that, I'm thinking of adding a Guix specific patch to Janet in our package that add such a search path via a GUIX_JANET_PATH environment variable. We've used similar strategies with other software before to good effect.

I've seen some discussions regarding using janet / jpm with NixOS elsewhere (which I'd imagine might have some relevance for Guix), and I've been curious about what implications the following point in the second paragraph:

I don't think setting JANET_PATH would be sufficient since it only contains one directory.

might have for packaging on these two distributions...