add support for o1 models

[iamarsenibragimov]

Added support for OpenAI o1-mini and o1-preview models by mapping them to the o200k_base encoding. This update ensures compatibility with the latest tiktoken changes as introduced in version 0.8.0 of the official library.

Fixes compatibility issues with models not yet included in tiktoken-php.

[iamarsenibragimov]

This proposal is based on the recent update in tiktoken (see: openai/tiktoken@0.7.0...0.8.0#diff-088caf12b3c3144a63ff973b98bc3eec7e53f4c1d75d11d49720cbcd2744fd30R72), where these models were added.

[yethee]

Please add support for chatgpt-4o- prefix as well.

[iamarsenibragimov]

Please add support for chatgpt-4o- prefix as well.

I saw that they added it however I do not get what is chatgpt-4o- while we have already gpt-4o-. What do you think?

[yethee]

Models chatgpt-4o-* will not be matched with the prefix gpt-4o-.

For example:

(repl)> str_start_with('chatgpt-4o-latest', 'gpt-4o-')
(repl)> false

[iamarsenibragimov]

Sure, I was curious about the nature of both. Here is what I found

The difference between the models chatgpt-4o and gpt-4o is mostly in their intended use cases and some technical optimizations.

gpt-4o-* models are general-purpose language models designed for various tasks, including text generation, coding assistance, and other language-based functions. They are optimized for handling multimodal inputs like text, speech, and even video, making them versatile for different applications, such as content creation and translation.

chatgpt-4o-* models are specifically tailored for conversational use cases, like ChatGPT interactions. These models are designed to maintain more natural and fluid dialogues by taking into account the conversation history and context over long exchanges. This makes them more suited for customer support and chatbot functions, where maintaining continuity and relevance in conversations is critical.

The primary difference between these models lies in their optimization: chatgpt-4o-* is better at dialog-based tasks, whereas gpt-4o-* is geared toward handling a wider range of inputs and outputs, including multimodal data. However, the core architecture remains similar, with both being based on transformer neural networks​(
OpenAI Developer Forum

If you're working with chat-based systems, using chatgpt-4o models would give you an edge in managing dialogues, whereas gpt-4o would be preferable for broader applications requiring text and multimedia generation.

[iamarsenibragimov]

@yethee Could you kindly review and approve this pull request, as it covers the core changes for o1 model support? I will handle the addition of the chatgpt-4o-* models in a separate pull request to maintain clarity and ensure a structured update process. Thanks

[codecov]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

[yethee]

Thanks!

[yethee]

Sure, I was curious about the nature of both. Here is what I found

Another difference between them is the update lifecycle.

The chatgpt-4o-latest model version continuously points to the version of GPT-4o used in ChatGPT, and is updated frequently.

From: https://platform.openai.com/docs/models