TeXLive 2021 update coming soon!

[pkgw]

I'm happy to let people know that the update to TeXLive 2021 will be coming soon! Just in time for TeXLive 2022!

The updates to the reference sources were initiated by @vlasakm in tectonic-typesetting/tectonic-staging#17. Thanks for instigating!

I have created a new bundle file and uploaded it as:

https://data1.fullyjustified.net/tlextras-2021.3r0.tar

The related updates to the bundle generation repo are in tectonic-typesetting/tectonic-texlive-bundles#8.

I've just submitted a pull request to update the engine implementations in #882. There are many thousands of lines of hand-transcribed updates, mainly in xdvipdfmx, which is always something that makes me feel anxious. But I have done a variety of tests and things seem to generally be working at least as well as they were before, and various issues like #862 and #879 should be fixed.

This new version tweaks the default bundle URL to include the "format version" of the underlying engine, which is bumped up to 32 for TeXLive 2021. Specifically, the default URL used by new builds will be:

https://relay.fullyjustified.net/default_bundle_v32.tar

as implemented by tectonic-typesetting/tectonic-relay-service#1. This means that once #882 is merged, builds of the master branch will start pulling down the new bundle, while older installations will keep using the TeXLive 2020 bundle. Nice!

[pkgw]

I've reviewed the diffs and caught a handful of transcription errors in the giant xdvipdfmx update. Also logged an overview of the changes in #882. Now doing some valgrind tests to look for new memory leaks ...

[pkgw]

Side note: updating to TeXLive 2022 should be much easier. Based on the preliminary changelog, the number of changes looks to be a lot more modest.

[pkgw]

I've done some more investigations and, yes, the delta to TeXLive 2022 is massively smaller than this update, so hopefully we should be able to catch up promptly. I still want to make a release targeting TL21 before diving into TL22 to do things more incrementally, but I don't see a need for a very long gap between the two if the technical aspects turn out to be as straightforward as they look.

[LeCyberDucky]

This is great! The effort you put into Tectonic is very much appreciated.

You mention, that new builds of the master branch will start using this new bundle. Do you plan on releasing new binaries accompanying this new bundle?

Edit: Oh, I just noticed https://github.com/tectonic-typesetting/tectonic/releases/tag/continuous. It seems like fresh binaries can be fetched from there. Excellent!

[pkgw]

That's right, the "continuous" release should provide binaries. I merged the PR this morning, and I plan to let things simmer on master for around a week, then make an official release if no issues arise.

[pkgw]

OK, the simmering has reveal something worth looking into: as per #859, it looks like TL21 doesn't work with shell-escape enabled because something in the very basic formats now requires pipe input, opening |extractbb --version. Note that this does not come up if shell-escape is disabled (the default).

As I wrote in the issue, I'd really like Tectonic to be able to work without needing external tools like extractbb to be installed. This may require some digging to figure out if this is something that we can just disable with a bundle patch, which might be a helpful stopgap measure.

[pkgw]

By the way, here's Knuth's commentary on the updates to the core TeX implementation, some of which make it into this update:

https://tug.org/TUGboat/tb42-1/tb130knuth-tuneup21.pdf

[pkgw]

I believe that the issue with shell-escape in #859 should now be fixed. I'm not aware of any other significant problems with the new TeXLive version, so I expect to make the next release soon if I can get confirmation of the fix.

[MTRNord]

Seems like something is wrong with this bundle sadly :/

PS C:\Users\marce\Documents\uni\uni-summer-2022\uni\infCN\homework\assignment_2\task1> tectonic -X build -k
note: "version 2" Tectonic command-line interface activated
note: connecting to https://data1.fullyjustified.net/tlextras-2021.3r0.tar
note: downloading index https://data1.fullyjustified.net/tlextras-2021.3r0.tar.index.gz
note: downloading SHA256SUM
note: generating format "latex"
note: downloading tectonic-format-latex.tex
note: downloading xelatex.ini
note: downloading latex.ltx
note: downloading texsys.cfg
note: downloading expl3.ltx
note: downloading expl3-code.tex
note: downloading cmr10.tfm
error: expl3-code.tex:22489: Invalid code (13), should be in the ranges 1..4, 6..8, 10..12
error: halted on potentially-recoverable error as specified

[pkgw]

Hm, are you running the very latest code? This issue should have been fixed in this commit:

8e14e86

[MTRNord]

I was running with the latest windows release version of tectonic and then manually setting this bundle in my workspace.

[MTRNord]

Tectonic 0.8.2 to be specific

[pkgw]

@MTRNord OK, yes, the new bundle requires newer code. It hasn't been released just yet, but version 0.9.0 should be out later today and include the fix.

[MTRNord]

Thanks that did it :)

[LeCyberDucky]

Related to this, I believe, I'm currently running into an error that looks something like this:

Extra \endgroup
! Extra \endgroup.
\document ->\endgroup 
                      \let \BeforeStartOfDocument \@firstofone \cpt@beforest...
l.7 \begin{document}

The internet tells me that this is likely due to my project making use of the xwatermark package, which appears to be using catoptions. Apparently, catoptions is incompatible with TeX Live 2021, which results in quite a frew broken packages. See:

https://tex.stackexchange.com/questions/566088/xwatermark-throws-error

https://www.overleaf.com/blog/tex-live-2021-now-available (Obsolete packages)

As it turns out, xwatermark has not been maintained for about a decade now, so I'll just switch to an alternative (draftwatermark perhaps, as mentioned in the above link?) in order to hopefully solve this.

I feel like patching the affected packages is unfeasible. Perhaps it would be possible to let tectonic detect the use of catoptions and tell the user to use something else?

[pkgw]

Yes, this is the kind of functionality that I'd like to have. The bundle could come with a data file of known incompatibilities and/or pitfalls, and we could wire in support for the engine to leverage that information. In this particular case, we could just have a table of filenames to warn about if they're being accessed. The other obvious candidates would be packages like inputenc and fontenc that shouldn't be used in XeTeX/Tectonic-land.

It's not an ideal user experience to have modes that are like "we'll let you do this but it definitely won't work!", but given the nature of the ecosystem, I think it would be the most sensible approach here.

[LeCyberDucky]

Yeah, that sounds good. Is anything related to this already in the works? Perhaps something like an issue tracking known incompatibilities?

[pkgw]

Nothing established. I've been trying to build up the bundle infrastructure to lay some groundwork for this kind of thing, like this list of package files and whether we can compile them naively or not. I think a wiki-like kind of framework could work well, but I'm not sure if it would be the easiest to kick-start ...

[pkgw]

Version 0.9.0 is now out, upgrading the default to TeXLive 2021.3! As alluded to upthread, the updated bundle won't work on older versions of Tectonic due to a small feature that's now needed to generate the LaTeX format.