docs: script to convert HTML manual pages to markdown

[neteler]

Script to convert recursively all .html files to .md (GitHub flavoured Markdown).
This is not only relevant for the conversion in GRASS-core but also for GRASS-Addons.

(see related #3849)

Suggestions needed for:

• path treatment of LUA file (e.g. on Windows)
• quality of generated markdown

[echoix]

Is this only to have the script in the repo? After that, is it supposed to be one time use or always used? (I assume that the docs in the repo will be in markdown instead of html soon..)

If it's supposed to be used only once, does it need to be part of the repo?

If ever we convert all of our html files to markdown (+formatting), I suggest to have an intermediate PR that only does the rename, to have the GitHub history +blame follow the file, instead of deleting and adding files (which would happen if the markdown formatting would have a lot of changes + renames)

[neteler]

Is this only to have the script in the repo? After that, is it supposed to be one time use or always used? (I assume that the docs in the repo will be in markdown instead of html soon..)

If it's supposed to be used only once, does it need to be part of the repo?

I see the following use cases:

• bulk conversion of all core manual pages (one time unless we need to optimize this script for better markdown output/modifications of HTML residuals, ...)
• bulk conversion of all addon manual pages (same as for core)
• helper script for addons in non-standard repos (see for example the "grass-gis-addons" tag: https://github.com/topics/grass-gis-addons). This should be offered long-term, IMHO.

If ever we convert all of our html files to markdown (+formatting), I suggest to have an intermediate PR that only does the rename, to have the GitHub history +blame follow the file, instead of deleting and adding files (which would happen if the markdown formatting would have a lot of changes + renames)

Yes, an idea is to do that with multiple PRs:

• convert HTML to MD, keep the HTML (PR 1, 2, ... n, i.e., submit in chunks to avoid too large PRs)
• have a (sort?) interim phase of both HTML and MD in parallel, esp. for quality control
• remove the HTML files in a different PR, keeping MD only

The "rename" comment I don't fully understand.

[echoix]

Is this only to have the script in the repo? After that, is it supposed to be one time use or always used? (I assume that the docs in the repo will be in markdown instead of html soon..)

If it's supposed to be used only once, does it need to be part of the repo?

I see the following use cases:

• bulk conversion of all core manual pages (one time unless we need to optimize this script for better markdown output/modifications of HTML residuals, ...)

• bulk conversion of all addon manual pages (same as for core)

• helper script for addons in non-standard repos (see for example the "grass-gis-addons" tag: https://github.com/topics/grass-gis-addons). This should be offered long-term, IMHO.

I see the value of having this available for use outside of a one-time use, like add-ons outside of the osgeo/grass-addons repo.

If ever we convert all of our html files to markdown (+formatting), I suggest to have an intermediate PR that only does the rename, to have the GitHub history +blame follow the file, instead of deleting and adding files (which would happen if the markdown formatting would have a lot of changes + renames)

Yes, an idea is to do that with multiple PRs:

• convert HTML to MD, keep the HTML (PR 1, 2, ... n, i.e., submit in chunks to avoid too large PRs)

• have a (sort?) interim phase of both HTML and MD in parallel, esp. for quality control

• remove the HTML files in a different PR, keeping MD only

The "rename" comment I don't fully understand.

I imagined that passing from html to markdown as the docs source would be done at once. Thus, no interim with both formats in the repo. Thus, to help navigating the history for the future, I was suggesting to have an interim commit in the main branch that renamed all html files to change the extension of html to md, without any content changes, and directly after, applying the conversion to md in these md files that are in fact html. Html can be used in md to some extent. However these two must be done right after the other, as I don't expect the html builds to be working in that interim commit. But I think it will greatly help navigating history by allowing to continue going back commits of the renamed file (instead of stopping there).

[wenzeslaus]

I was suggesting to have an interim commit in the main branch that renamed all html files to change the extension of html to md, without any content changes, and directly after, applying the conversion to md in these md files that are in fact html.

The history would be nice and the HTML==MD is nice trick, but...

However these two must be done right after the other, as I don't expect the html builds to be working in that interim commit.

...I'm afraid we can't just replace the server infrastructure for HTML with Markdown/mkdocs at the same time as merging the PR, so I think the change needs to be gradual in one way or another.

[echoix]

If the build process generates the html from the md, what changes would be needed on the web server infrastructure?

Maybe a test with a staging instance might help. But I'll keep thinking about what might be best here..

[neteler]

Valid point about the git history.

Suggestion:

• we git move the .html to .md (so history is kept) and commit
• we copy it back to .html (to keep them for a little while in parallel; these files do not have the git history), add and commit
• we replace HTML content of the now .md file with the markdown content and commit
• we compare both (quality control)
• eventually we drop the .html file

[echoix]

Valid point about the git history.

Suggestion:

• we git move the .html to .md (so history is kept) and commit
• we copy it back to .html (to keep them for a little while in parallel; these files do not have the git history), add and commit
• we replace HTML content of the now .md file with the markdown content and commit
• we compare both (quality control)
• eventually we drop the .html file

That's what I was thinking about if we needed to have html too in parallel (vs a clean cutoff, switching to Md at once).

[wenzeslaus]

Hardcoding utils below is simpler. For testing in combination with the other PR, assuming you wget right is a very safe assumption.

[neteler]

The question is: hardcoding yes, but how?
With a relative path, the script will not work in standard or other addon repos.

[wenzeslaus]

• we git move the .html to .md (so history is kept) and commit

• we copy it back to .html (to keep them for a little while in parallel; these files do not have the git history), add and commit

We could prepare that in one PR and then "Allow rebase merging" for couple minutes to merge that PR with its two commits. This would not break or workaround the CI.

[wenzeslaus]

The rename way seems also appealing because it is more natural: Even with several conversions done already here, I get plenty of HTML tags, some perhaps need to stay.

cd dist.x86_64-pc-linux-gnu/docs/mkdocs/source
grep -Eor '<[a-zA-Z][^>]*>' *.md | grep -Ev '<https?://[^>]+>'

...
r.in.wms.md:<div data-align="center" style="margin: 10px">
r.in.xyz.md:<sup>
r.li.cwed.md:<span class="small">
...

[neteler]

Using the latest version of this script, I still get a lot of warnings:

...
WARNING -  Doc file 'db.execute.md' contains a link 'topic_attribute_table.html', but the target is not found among documentation files.
WARNING -  Doc file 'db.execute.md' contains a link 'keywords.html#SQL', but the target 'keywords.html' is not found among documentation files. Did
           you mean 'keywords.md#SQL'?
WARNING -  Doc file 'db.in.ogr.md' contains a link 'database.html', but the target is not found among documentation files. Did you mean
           'database.md'?
WARNING -  Doc file 'db.in.ogr.md' contains a link 'topic_import.html', but the target is not found among documentation files. Did you mean
           'topic_import.md'?
WARNING -  Doc file 'db.in.ogr.md' contains a link 'keywords.html#attribute%20table', but the target 'keywords.html' is not found among documentation
           files. Did you mean 'keywords.md#attribute%20table'?
WARNING -  Doc file 'db.login.md' contains a link 'database.html', but the target is not found among documentation files. Did you mean 'database.md'?
WARNING -  Doc file 'db.login.md' contains a link 'topic_connection_settings.html', but the target is not found among documentation files.
...

The reason is (example, see KEYWORDS section where .md should be present rather than .html):

---
name: db.in.ogr
description: Imports attribute tables in various formats.
keywords: database, import, attribute table
---

# db.in.ogr

## NAME

***db.in.ogr*** - Imports attribute tables in various formats.

### KEYWORDS

[database](database.html),
[import](topic_import.html),
[attribute table](keywords.html#attribute%20table)

### SYNOPSIS

**db.in.ogr**
...

Seems we overlooked that in #3849? Any idea, @landam?

As the script of this PR runs before the parser is invoked a change in man/build_md.py or around might be needed?

[landam]

Seems we overlooked that in #3849? Any idea, @landam?

As the script of this PR runs before the parser is invoked a change in man/build_md.py or around might be needed?

For record, solved by #4740

[neteler]

Change in f91a111 to avoid undesired escaping of $ in plain text:

Example: dist.x86_64-pc-linux-gnu/docs/mkdocs/site/grass.html#examples

Before:

After: