Support for Markdown, Asciidoc and other document markup languages

[fsmunoz]

After having great success in using SASpy in Jupyter notebooks I started to wonder if I could use it in Markdown documents; the reasons are plenty and include the inability to produce configurable PDFs from Jupyterlab since there are no images, plus the different use-case I have (producing documentation that is easy to edit and maintain through git, etc)

One of the reasons this isn't directly possibly is because an <IPython.core.display.HTML object> will be present in non-browser settings.

I don't know enough (or even the basics) of ODS, so I suspect my approach is not the most direct, but I added a new "display" called "markdown" that uses BeatifulSoup + markdownify to output Markdown markup - very simple one since it wraps the img element.

With this change I could use SASpy to produce plots in the following systems:

• knitr , using a Rmd file with the python engine (which essentially means that the same Rmd file can actually have SAS, R and Python code in it)
• codebraid, using the jupyter kernel option.

While R has a SAS engine, using SASpy is, I think, the only way to use a remote SAS Viya instance, and this was the configuration I tested.

I think this could be worth exploring (and the changes needed are minimal and I can share them, albeit I haven't tested it thoroughly).

I'm attaching an example PDF for reference, built from an Rmd file - merely an example.
SAS.pdf

[tomweber-sas]

I have to say, I'm not100% up to speed on this. But I think I get where you're going. I don't use markdown much and haven't used those other packages you mentioned either, so I don't have any experience with this. Does the actual HTML, which is what plots and such are, not render in these environments? I just googled 'html in markdown', since I'm not that familiar with it and it seems to imply inline html works, and would be rendered, in markdown w/out trouble; so it says :) . I'm not sure how to try that.

Your comment about

One of the reasons this isn't directly possibly is because an <IPython.core.display.HTML object> will be present in non-browser settings.

Make me wonder if you were running w/ batch mode False in the case you were trying(https://sassoftware.github.io/saspy/advanced-topics.html#using-batch-mode). If you're running somewhere other than where Ipython Dispaly will work, then setting Batch to True will return you the actual HTML document for the plot, chart, graph ... instead of trying to call the Display method on it. In your pdf document, the heatmap method (in Batch mode) would simply return the html document itself, and then, would that have rendered?

[fsmunoz]

Thank you for your comment, and you've identified the main problem: just outputting the HTML will not work since it will not render. As an example, using Pandas and the pandas.DataFrame.to_markdown method like in the example will output a markdown table which will then be correctly viewed in a Markdown editor (or viewer, like GitHub) and also correctly parsed and converted into a tabular format of the desired format (namely in the example above, a PDF via LaTeX tables).

This to say that just outputting the HTML will not work, you can see an example of using batch mode and printing the HTML:

Just like in this comment (written in markdown) I have ![image](http://...), the same applies to documents - actually even more so since here if I use HTML it could be parsed by the browser, but it won't be parsed when converting Markdown to PDF.

One alternative option would be to return just the image data and let those writing the Markdown document to manually add the necessary syntax. The way I did it the correct markdown is returned by converting the HTML body to markdown, and the SGPlot HTML ends up like this:

 ![The SGPlot Procedure](data:image/png;base64,iVB.....)

The reason the Python code works is because reticulate does the necessary magic behind the scenes for matplotlib, which is not unlike what I've done.

If there is a way to produce a PDF document from a markdown source (or any other markup language) that is similar to my example then no changes would be needed. I haven't found a way though.

[tomweber-sas]

I've been trying to research this to be able to have an intelligent conversion. I've been playing with markdown cells in jupyter, as a means to see what works and what doesn't. Again, I don't have any experience with what you're trying to do here. I do see that html doesn't render in markdown. saspy output (plots, graphs, ...) are HTML5 with the images embedded; that's key to how it works in the various notebook server it supports. Getting the raw html, instead of some display object, is easy enough (Batch=True), and even extracting the "<img .../>" out of it is simple enough, yet still doesn't render via markup (that I can get to work):

I tried the ![The SGPlot Procedure](data:image/png;base64,iVB.....) you mentioned above, but it doesn't work. The only way I've seen an image render is if it is a .png file on disk: ! [Image] (./SGPlot.png)

So, I'm trying to figure out what you're looking for from saspy, in regards to this. I assume there's no Python packages, like all of the R things you've mentioned, that can do anything to help take html and convert it to anything markdown can support? I don't do anything like this, so I don't know if there are or aren't. Did you say you have something that works for you? What is that, if so?

FWIW, I did find a bit of a kludgy way to get this to be able to work. It's not a solution, but perhaps more of a brainstorming that may lead to something. I have an option in the config file, where you can change the output from HTML5 to HTML. I added that for someone trying to use SAS 9.3 (doesn't have html5) on their pc with local SAS. For that case, using HTML (images aren't embedded) happens to work, since the image files written out are linked in the html, and since they are all on the same machine, they render. If this were a remote connection to SAS, the image files wouldn't be found, nor displayed on the client.

But, with HTML, the images are written out a .png files, on the SAS server side (current dir is what I observed), and the name of the file is in the html returned. SASPy has a download method, so I just extracted the filename from the html and downloaded the .png file, and then using the ! [Image] (./SGPlot.png) in a markdown cell worked. If it were on the same machine, you wouldn't need to download, just reference the files, period, and they render.

I know that's a bit of a stretch, and it's not a solution, but maybe there's an idea in there that's part of something. Again, I'm just trying to even figure out the problem, as much as what you're looking for to solve it.

Thoughts?
Tom

[fsmunoz]

Thank you for your effort in this @tomweber-sas , I'll try to be as helpful as possible.

Markdown image inclusion

I think that the image snippet didn't work for you because I truncated it, due to the size. I'm attaching the SAS.md file that was the result of using $ R -e 'library(knitr);knit("SAS.Rmd", "SAS.md"): you can see that the data content is much larger, and it works (try $ pandoc SAS.md -o SAS.pdf or $ pandoc SAS.md -o SAS.html).

SAS.md

Including a file is also supported, and it's very often used: matplotlib and R code chunks generally use that approach, they save the file to a folder and add the appropriate code for including it (check the md file, the only plot which will be visible is the SAS one since the other files references are in my filesystem).

HTML to markdown

You mentioned

I assume there's no Python packages, like all of the R things you've mentioned, that can do anything to help take html and convert it to anything markdown can support?

Well, that's exactly the change that I did, so yes, you're right: with batch mode the logic can be done in Python by using BeautifulSoup and markdownify, as long as we can get access to the HTML (so, x here would be some_result['LST']:

from bs4 import BeautifulSoup as BS                                                                                                                                                                          
from markdownify import markdownify as md    

def mdHTML(x):                                                                                                                                                                                               
    soup = BS(x,features="lxml")                                                                                                                                                                             
    body = soup.find('body')                                                                                                                                                                                 
    return(md(str(body)))      

The difference being that this change at the SASpy level makes it transparent to use, whereas without it the logic must be repeated in each block (even if it's only a single function). Now that I think of it perhaps this is good enough and not really worth adding more complexity at the SASpy layer, even if it would - from an end-user PoV - be better... after some decades we're back to the Worse is Better debate :) More than a change, maybe this is more worth a blog post so that those that have a similar need can learn how to do it (and hopefully this discussion will become a reference in itself).

SASpy HTML and HTML5

I didn't saw that one, but yes, if running the code saves a file with a predictable name then it would be possible to work with that. The challenged would be the need to drop the HTML output and replace it for the Markdown equivalent, which ends up being similar to the previous approach (with an URI insted of data). As long as you can get the png file to the local filesystem it would work, the question would be if there should be some additional metadata returned (plot name, perhaps?).

Having a more general mechanism that essentially returns values (which could be the PNG image data or the PNG filename in the local filesystem) could be a way to address all needs without requiring parsing the HTML in the code. It would also work regardless of the markup language: Rst, Asciidoc, Markdown, all have slightly different formats, but as long as one can get the image data or path and the name of the plot, one can get it working with minimal effort.

In short...

Whether this is useful enough for inclusion I don't know... it's useful to me and I'm using it successfully. I can see three main options:

1. Keep it as is, don't touch the SASpy code and simply add a SASpy contrib example on how to do it (using the above approach) - I can gladly do this one.
2. Add a general mechanism in which to get the images without any HTML or markdown: completely agnostic and would work in any language by simply including the code between the necessary tags, wouldn't involve any HTML parsing at the destination.
3. Add a display option for each markdown language, or just the ones that are likely to be used (Markdown being an obvious example, Asciidoc another).

Maybe the first option is good enough, and considering one can define a function in the initial code chunk the impact is very small: the need to add the display logic is still left to the end user but it's simplified enough that I don't think it's an issue.

[tomweber-sas]

Ok, there a lot of stuff there. Let me try to hit things as I go.

For the ![The SGPlot Procedure](data:image/png;base64,iVB.....) case, I coded the entire 4k of data, I didn’t just type in that as is with the ‘....’, but It doesn’t render in Jupyter markdown cells. I believe that whatever pandoc command is, that you end up with it working in that case, so that’s cool. I just didn’t get anything to render in Jupyter markdown other than when using a .png file itself.

Seeing you’re source file, I see you’re just using the "```python" wrapper around the python code, which doesn’t end up rendering the results be they Ipython objects or HTML strings. I couldn’t get any variation of the html to work with that either, whether it be the whole document, of just the img part of it, even when wrapping the <img with <html> <img .../> </html>, so yeah, markdown just won’t work with anything other than the simplest html tag, it would seem.

The HTML instead of HTML5, isn’t a real solution, but rather an idea if you’re just trying to put together a one off kind of thing to just accomplish building a doc. Not a real solution. Trying to turn that in to a generalized use case isn’t really an answer.

To speak to your ‘In Short’ list:

1. Totally up for an example, use case, for what you’ve done, so others can use it, enhance it, ...

2. This seems (at the moment in my mind) a bit too big and pervasive. But, I don’t understand it enough to really see the whole picture and design in my mind. This isn’t how any of this works, so it doesn’t jump out as being the answer. Again, ignorance is fog that I don’t see the end implementation through, at the moment.

3. Having dedicated methods, which take in the existing HTML that is what the saspy methods return for all of these case, and transforms it into whatever is needed by the markup/down thing so it renders, is a perfectly reasonable enhancement. Guess that’s kinda like that pandas to_markdown() method, as it’s a dedicated transform method from the native thing (which doesn’t render on its own), to the other thing (which does). Having methods that do that is perfectly acceptable, and then provide the intermediate step to accomplish what you’re looking for without wholesale changes to all of saspy for this specific use case. Clearly these would require 3rd party modules (like the soup and markdown ones) which would make them being functionally dependent on you having those things installed. That would be documented, but not be a requirement for saspy itself. If you have them, you can use these methods, as opposed to they being requirements to be installed when saspy is installed. Though now that I re-read it, I think you're saying make these methods be other instances of the DISPLAY and HTML methods, which are the current rendering things for the different notebook servers I support? adding other options for these to make it 'just work' in those environments? I can picture that, though I'd have to actually play with it to really see if it make sense in all cases.

You mentioned defining your function up front so it’s there to be used, you can do things like that in the config file, as it’s a Python file that’s imported, so functions and things you code in it are available as soon as you import saspy. But, again, I think that speaks to number 3, where defining dedicated transform methods would be a perfectly reasonable, and useful, contribution. Then you have the methods to make the native HTML that’s already provided, turned into whatever it is you need for the scripting language you’re trying to use (markdown or whatever). That seems the most straightforward thing to me so far.

I hope that addresses your thoughts and even makes some sense. I'm thinking that exploring number 3 above, one way or another, is probably then next thing to do; in my mind anyway. What are your thoughts?

Thanks!
Tom

[fsmunoz]

I think we're on to something :)

Trying to be as concise as possible to avoid drowning you with details:

The SGPlot thing: it won't render in Jupyter, only in Markdown. That is exactly the change needed for it to work in Markdown, the move from HTML (which Jupyter renders) to Markdown (which gets correctly parsed by Markdown interpreters but will not be rendered in Jupyter). This is indeed the crux of the matter: when used in a Markdown document the results needs to be in markdown.

The md file is the result of parsing the Rmd with knitr; the Rmd file has more options in the code chunks and the knitr process uses that to execute the code in a single Python session

   ``{python, saspy-setup, warning=FALSE, echo="true", results="asis",fig.pos="!H", engine.path="/home/frmuno/anaconda3/bin/python3"}
   import saspy
   import pandas as pd
   sas = saspy.SASsession(cfgname='mycfg', results='Pandas')
   #sas.set_batch(True)
   hr_pd = pd.read_csv("./HR_comma_sep.csv")
   hr = sas.df2sd(hr_pd)  # the short form of: hr = sas.dataframe2sasdata(hr_pd)
   p = hr.columnInfo()
   print(p.to_markdown(tablefmt="github"))
    ``

When using Codebraid (a different system which isn't based on R but performs a similar function - it parses a .cmd file and produces a md one after running the code and merging the output) I can actually specify the jupyter kernel that will run the code, but this doesn't change the end result: HTML or iPython output will not work since Markdown processors won't be able to transform them into appropriate content (like going from an HTML table to a PDF table, or an HTML image to a png, etc).

Now, the reason you didn't saw anything in the python code is because (and perhaps I should've been clearer in this) I changed SASpy to add the necessary methods: this goes directly to the discussion on what option to follow. My local copy of SASpy has that mdHTML function defined and has the necessary plumbing done to allow a markdown display option to be used, which then chooses mdHTML and mdDISPLAY. It also tests, as you hinted, for the presence of the libraries (much like it already does for iPython, through a try/catch).

So, you essentially described my current working approach (again, apologies, I should've been clearer but when one's knee-deep in code it's easy to forget that context is very important).

On my side I can do two things right now:

1. Add that example that doesn't require any changes to SASpy. I think this in itself would be interesting. [edit: submitted https://github.com/Initial examples of using Rmd with SASpy saspy-examples#8]
2. Submit my changes to my fork so you can have a look. It's all trivially simple, and importantly enough not tested in almost any way (but assuming that the general principle doesn't change from plot to plot, it should work).

Point 2 above is without any expectation of merging, but merely to show what I've done so that any next step on this can have an additional reference point.