fix: token speed should not be calculated based on state updates

[louis-jan]

Describe Your Changes

This PR addresses an issue where message rendering could impact token speed, which is not intended. The token speed calculation should not be affected by component rendering, but rather based on the received tokens.

Ensure that rendering of complicated message contents doesn’t reduce token speed (currently it does).

Token speed is now calculated on message receive, assuming each event yields one token.

The formula for calculating token speed is given by:

$$ \text{Token Speed} = \frac{\text{Eval Count}}{\text{Eval Duration}} $$

Where:

• Total Tokens is the total number of tokens processed.
• Duration is the time taken in seconds to process those tokens.

Screenshot demonstrate a case where convo is very long, but the token speed is not reduced (it's reduced to ~3 before where it should be ~8x)

Changes made

1. New State Management:

   • Added a new tokenSpeedAtom in ChatMessage.atom.ts to store token processing speed details using Jotai.

2. ModelHandler.tsx:

   • Imported tokenSpeedAtom.
   • Added functionality to calculate and update the token speed in the ModelHandler component. This involves using setTokenSpeed to track the speed at which tokens are processed for each message.

3. useSendChatMessage.ts:

   • Refactored to remove queuedMessageAtom and replace its function by setting tokenSpeedAtom to undefined when a new message is sent, resetting the state for new calculations.

4. index.tsx (SimpleTextMessage Component):

   • Simplified the logic by using useAtomValue to get the current token speed from tokenSpeedAtom.
   • Adjusted the UI to display the token speed only if the current message matches and tokenSpeed is above zero, using the new data structure.

5. Types:

   • Added token.d.ts defining the TokenSpeed type, which includes fields for message, tokenSpeed, tokenCount, and lastTimestamp, used to manage token speed calculations.

Overall, the feature updates the application with functionality to calculate, store, and display the speed at which tokens are processed for chat messages.

[github-actions]

Barecheck - Code coverage report

Total: 69.54%

Your code coverage diff: 0.22% ▴

Uncovered files and lines

File	Lines
web/helpers/atoms/ChatMessage.atom.ts	61-62, 64-65, 67, 70-71, 78-79, 81, 84-85, 88-89, 100, 103-104, 109, 112-113, 117, 120-121, 123-124, 127, 143-148, 150, 153-154, 156-157
web/hooks/useSendChatMessage.ts	92-96, 98, 100, 102, 110, 113, 118-119, 121-126, 129-130, 135, 142, 144-146, 149-150, 152-153, 155, 157-159, 161, 165, 167, 171, 174-175, 181-182, 186, 197, 200, 204, 207, 210, 218, 221, 227, 230-231, 234-238, 241, 244, 247, 250, 253, 258-259
web/screens/Thread/ThreadCenterPanel/SimpleTextMessage/index.tsx	53-57, 59-60, 63, 66, 74-77, 79, 83, 90-92, 98, 101-104, 109-110, 113, 163, 165-166, 171-172, 231, 236-238, 312, 335

[hiento09]

LGTM