Missing o1 models support

[SeriousOne]

Are there plans to support o1 models: o1-preview and o1-mini? I suppose they use already existing encodings, so it is not a problem to add support for them.

[yethee]

Can you confirm that these models use the encoding o200k_base?

[SeriousOne]

Unfortunately, I haven't found any confirmation. When I asked ChatGPT and Perplexity, they both answered that this is cl100k_base. Sure, that's not reliable. I think it's better to wait until it is known for sure.

[iamarsenibragimov]

I've proposed support for the o1-mini and o1-preview models by mapping the "o1-" prefix to the "o200k_base" encoding in pull request #19.

This proposal is based on the recent update in tiktoken (see: openai/tiktoken@0.7.0...0.8.0#diff-088caf12b3c3144a63ff973b98bc3eec7e53f4c1d75d11d49720cbcd2744fd30R72), where these models were added.

[yethee]

Fixed in #19