Handling PDFlatex compatibility

[appetrosyan]

Tectonic is based around XeLaTeX which is both the main criticism levied against tectonic, but also one of the biggest roadblocks to replacing latex distributions with it.

Motivation

XeLaTeX is far from standard and is not fully compatible with Pdflatex. For example, a paper could be easy to compile (with terrible hacky abuse of the engine) with pdflatex but not with xelatex.

XeLaTeX has a reputation for being buggy. While it does offer multiple advanced features, more often than not, we don't have a choice whether or not the god-awful template used by e.g. MDPI compiles with XeLaTeX.

Proposed solution

Allow the option to compile a file with a pdflatex compatibility mode.

The modifications are quite extensive, the custom fork of XeLaTeX would need to be supplemented with a custom fork of PDFLaTeX.

Possible drawbacks

None as I see them.

You could argue that this is an unnecessary complication for tectonic, but not having this mode is the sole reason why some people still have a TeXLive installation. It would also aid in its adoption as a full LaTeX solution for e.g. AucTeX.

[Neved4]

Hey @appetrosyan, I find your proposal interesting, may this best suited for Discussions? 🚀

[appetrosyan]

I'd like to.

At the very least we might end up with a thread with all of the incompatibilities.

[Neved4]

I really like the points you bring up so thank you 💯

First, I'll outline what I believe the main roadblocks are and then I'll go about engines and their usages.

It's true that some people that find issues to prevent adoption tectonic mostly lie in ~4 distinct camps:

1. A package they want is not working at all or producing the expected output due to outdated bundles, reliance on external tools, shell-escaping or versions mismatch (e.g., minted, biber, polyglossia, circledsteps, nicematrix, parskip, pgfornament, tikz... etc)

   — Currently, many of the common packages people resort to were already provided or updated, -Z shell-escape has helped with the external tools situation and the updating pace is better.

2. Packages that only support pdfLaTeX or LuaTeX, explicitly or implicitly (e.g., pdfx, attachfile... etc).

   — A solution would be to motivate maintainers to provide XeTeX/tectonic support for said packages and keep them up-to-date. There are reasons why they may not want to, and knowing why would help targeting specific engine changes to make tectonic more attractive to package authors.

3. Personal setups, with self-created packages and custom paths (e.g., $TEXINPUTS, $TEXMFHOME)

   — Supporting an environment variable would be awesome, the overall situation was quite eased with the -Z search-path=<path> option.

4. Older systems reliance. Some people found comfort in pdfLaTeX and its limitations (e.g., no fontspec, bad UTF-8 support, bad coverage with packaged fonts... etc) —same story for BibTEX not supporting UTF-8—. Having a features stripped engine that only plays nice with a limited set of (English-centric) inputs by default, that won't support TTF or OTF fonts, and would not be able to run advanced packages made it for some a predictable, vaguely "reproducible-like" format.

   — It's incredibly difficult to know what's the de facto standard engine for TeX Live. Is it pdfTeX? LuaTeX? XeTEX? The answer generally is it depends on what you're looking for.

   N.B.: Currently arXiv's submission process does not support processing with: XeTeX and its variants including LuaTeX, LyX, or PDFTeX. and doesn't support Biblatex's new format either. While tweaking the output to fool the engine is possible, those workarounds are frail at best. Into the bargain, arXiv users often freeze their TeX Live to match arXiv's (≈2 years behind current one) and thus preserve compatibility and save some headaches.

Background

TeX Live is convenient. It's packaged in a way that things play nice with each other, even those that were designed with quirks. It has a built-in package manager, distribution environment variables and rolling out a different.

Tectonic is not a small project to undertake, but the resources available are limited. Most contributions come from @pkgw and he needs to prioritise what to focus on.

The TeX ecosystem finds itself on a very interesting situation. It brings so many different users together. For me, pdfTeX is ancient and extremely limiting at best. Meanwhile, people at arXiv live and die for the former. Others, keep pdfTeX and XeTeX around for compatibility, those documents that only compile correctly with them. From time to time, there are also rendering differences between the engines. Finally, new users are drawn to newer engines (LuaTeX) and tools like (tectonic) for the important aspects they bring.

I personally do love other engines so I relate to the sentiment of having something as tectonic to be compatible with those same engines. I love LuaTeX, is my go-to for everything and the only engine actively maintained, other than tectonic. I do also love and use tectonic/XeTeX for more limited work, since I find the former output more polished typesetting than the later. pdfLaTeX seems very limited to me except for basic functionality, but for that it's fast and awesome.

There have been other attempts to alleviate the complexities of building a LaTeX document (e.g., arara, cluttex, latexmk, llmk and MikTeX) but no one compares to Tectonic. No other build system has as great of an UX nor tectonic's ambitions and vision for reproducible builds.

Moving forward

As for incompatibilities go I don't recall what was the best thread but this one comes to mind.

Secretly, I've been waiting until something like tectonic appears for LuaTeX; or even better, a wrapper where you can choose your engine (à la latexmk) with tectonic-like features: on-demand download with great UX and the power to choose any engine: pdfTEX/XeTEX/LuaTeX/tectonic. This would be a superb direction, something self-contained that ships multiple engines. I find doubtful whether this would be feasible within tectonic's scope and priorities but it's an awesome idea for a wrapper 🚀

Tectonic exists and it's a gem. It's not a pdfLaTeX or a LuaTeX compatible engine, but I use it whenever I can: the tradeoffs are worth it for how amazing it is.

I believe Tectonic's edge really is UX and reproducible builds, the more we can help making things work to bring people in, and the more we sharpen that edge the more people will love it.

Yes, TeX Live is convenient. Rolling a separate engine with its own bundle will bring out those quirks, that need to be fixed on a request, by hand, basis; but it also lays a series of exciting challenges in the pursue for better UX ⚡️

[appetrosyan]

I think it's worthwhile to outline a few solutions and stop-gap hacks which I think would be useful.

1. Package compatibility.

In most cases like with e.g. microstyping, the incompatibility is known, but not package wide, i.e. you can avoid using certain macros, and ahcnage a few options to make the package work. In most cases, it's enough to know which options one needs to avoid to produce an output. In my case, while some uses of microtyping were mandated by the journal (I'm uploading it to arxiv today, btw), but the one that broke the compilation with tectonic specifically wasn't.

Just knowing which ones don't work would have helped. But if we know which specific macros and in which option combinations fail on XeLaTeX, we can do more. Sometimes, one does not need an exact reproduction of the final paper: it's enough to produce the text with some accuracy, because one only needs to check the appearance roughly. In this case a flag like -Z skip-incompatible-macros, would have allowed using tectonic for prototyping, but submit via a PDFLaTeX distribution or TeXLive.

Finally, sometimes, the developers of said packages don't even know, that their reported incompatibility is not genuine. Tectonic's engine isn't exactly XeLaTeX, so this allows us to either patch the tectonic-specific package, and notify the developer, or provide a list of alternatives. Where specific packages are not mandated by journals and one needs something which looks similar, it might be useful to suggest that to the user.

Notice how almost all solutions revolve around providing more feedback to the users, and only one involves a small amount of coding (to ignore problematic macros).

Finally, one could genuinely patch the packages. At some point, there will be an extremely popular package which just needs to be reimplemented for tectonic. Shell escaping and external packages cna be implemented as statically linked modules using an internal API. This is quite a bit more involved, but this isn't something which is done unilaterally by tectonic developers, being in good contact with microtyping's author might give them the necessary jolt to start developing on a feature that they shelved temporarily (in fact this is what happened).

So to me, one should catalogue packages which are compatible/incompatible and in which option combinations. Provide feedback, alternatives, allow to skip over such incompatibilities by preprocessing the LaTeX and removing all mentions of the problematic packages. Finally, ask developers of said packages to provide support for tectonic directly.

[Neved4]

@appetrosyan I don't know the package microstyping, are you referring to microtype?
I'd personally love a list of —macro— incompatibilities between engines but haven't had much luck so far...

This issue seems related to what you raised: #158 and @pkgw was keen on knowing pdftex incompatibilities and open to the idea of making tectonic more compatible: #158 (comment), #158 (comment).

If some journals require a lot of microtype features (Table 1, p. 6) that generally rules XeTeX out, since it only supports protrusion. An issue to modify tectonic to support more or all of microtype's features sounds like a great idea in this direction.

I really like the idea you mentioned of a -Z skip-incompatible-macros or -Z pdflatex-compatible to group experimental changes targeted to produce a broader number of pdflatex documents.

Documenting incompatible packages seems a great idea too! 💯

[pkgw]

Great thread here! Do you mind if I convert this to a Discussion since I agree that that seems like a bit better format for what's going on here?

[appetrosyan]

Discussion sounds good.

[Neved4]

💯 go for it!