feat: store grammar state in weakmap

[antfu]

In #712, we introduced GrammarState to resume the highlighting from the middle way. It solves the problem of customizing grammar context, but wasn't very useful for passable highlighting (for example, in a text editor, instead of highlighting the full context again over again, we can reuse the result from the previously highlighted lines) - because users need to run the parsing twice to get the GrammarState object. The existing API wasn't designed to be able to return extra property, making it hard to return the GrammarState in one-go, without breaking changes.

This PR overcharges the shiki.getLastGrammarState() function to be able to accept the return Hast Root or ThemedToken[][] to retrieve the GrammarState stored in a internal WeakMap. Make the usage more performent.

Along the way, we also made GrammarState work with multiple themes by storing multiple internal state for each theme.

The usage would be like:

const shiki = await getHighlighter(/*...*/)

const hast = shiki.codeToHast(/* ... */)

const grammarState = shiki.getLastGrammarState(hash) // pass the hast instead of code

close #803

[netlify]

✅ Deploy Preview for shiki-matsu ready!

Name	Link
🔨 Latest commit	9133728
🔍 Latest deploy log	https://app.netlify.com/sites/shiki-matsu/deploys/6703765aba5d4f00084b581b
😎 Deploy Preview	https://deploy-preview-804--shiki-matsu.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[netlify]

✅ Deploy Preview for shiki-next ready!

Name	Link
🔨 Latest commit	9133728
🔍 Latest deploy log	https://app.netlify.com/sites/shiki-next/deploys/6703765aba979000087e2155
😎 Deploy Preview	https://deploy-preview-804--shiki-next.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

Attention: Patch coverage is 94.01709% with 7 lines in your changes missing coverage. Please review.

Project coverage is 94.85%. Comparing base (ea4b8dd) to head (9133728).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
packages/core/src/textmate/grammar-state.ts	87.27%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #804      +/-   ##
==========================================
- Coverage   94.92%   94.85%   -0.08%     
==========================================
  Files          78       78              
  Lines        6504     6589      +85     
  Branches     1298     1317      +19     
==========================================
+ Hits         6174     6250      +76     
- Misses        321      330       +9     
  Partials        9        9              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[xsjcTony]

@antfu This seems to close #734 as well.
But instead of using highlighter.getLastGrammarState, I'll need to use highlighter.codeToTokens() and tokens.grammarState instead.
Not sure if this is intended or do I need to adjust my initial use case with getLastGrammarState?
Anyway at least there's a workaround now for #734, so I guess we can close that one as well?

[antfu]

Oh yeah, great to hear!