Expose Encoder and Decoder in TiktokenTokenizer

[razshare]

Fixes #7313

We are excited to review your PR.

So we can do the best job, please check:

• There's a descriptive title that will make sense to other developers some time from now.
• There's associated issues. All PR's should have issue(s) associated - unless a trivial self-evident change such as fixing a typo. You can use the format Fixes #nnnn in your description to cause GitHub to automatically close the issue(s) when your PR is merged.
• Your change description explains what the change does, why you chose your approach, and anything else that reviewers should know.
• You have included any necessary tests in the same PR.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.88%. Comparing base (5090327) to head (140aa42).
Report is 11 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7314      +/-   ##
==========================================
+ Coverage   68.87%   68.88%   +0.01%     
==========================================
  Files        1470     1470              
  Lines      274005   274005              
  Branches    28403    28401       -2     
==========================================
+ Hits       188717   188754      +37     
+ Misses      77970    77936      -34     
+ Partials     7318     7315       -3     

Flag	Coverage Δ
Debug	68.88% <100.00%> (+0.01%)	⬆️
production	63.30% <100.00%> (+0.01%)	⬆️
test	89.41% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs	78.28% <100.00%> (ø)
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs	99.39% <100.00%> (+0.40%)	⬆️

... and 8 files with indirect coverage changes

[tarekgh]

@razshare I replied on the issue. Let's discuss it there first before we continue here. Thanks a lot for your submission. I converted this PR to be draft for now till we finish the discussion.

[razshare]

@dotnet-policy-service agree