core, standard tests, partner packages: add test for model params #21677
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
ccurme
changed the title
dosubot
bot
added
size:M
dosubot
bot
added
size:L
ccurme
changed the title
nfcampos
reviewed
May 16, 2024
| @@ -60,6 +63,15 @@ | |||
| from langchain_core.tools import BaseTool | |||
|
|
|||
|
|
|||
| class LangSmithParams(TypedDict, total=False): | |||
There was a problem hiding this comment.
why using both total=False and Optional?
There was a problem hiding this comment.
this is kind of an abuse of TypedDict because I expect fields to be missing.
here I include all parameters that we are standardizing (including those that can be None). this is mostly just to define them somewhere.
regarding total=False: the implementation of _get_ls_params on BaseChatModel is almost the same as _get_invocation_params() (it just adds ls_model_type="chat"). so there should be minimal changes in tracing to models on which we haven't implemented _get_ls_params yet. but providers generally don't have the ls_ params yet.
nfcampos
reviewed
May 16, 2024
nfcampos
reviewed
May 16, 2024
nfcampos
reviewed
May 16, 2024
ccurme
commented
May 16, 2024
nfcampos
reviewed
May 16, 2024
| **kwargs: Any, | ||
| ) -> LangSmithParams: | ||
| """Get standard params for tracing.""" | ||
| return LangSmithParams(ls_model_type="chat") |
There was a problem hiding this comment.
why do we want this default implementation, and why does it ignore stop words?
dqbd
reviewed
May 16, 2024
hinthornw
pushed a commit
that referenced
this pull request
Jun 20, 2024
…1677) 1. Adds `.get_ls_params` to BaseChatModel which returns ```python class LangSmithParams(TypedDict, total=False): ls_provider: str ls_model_name: str ls_model_type: Literal["chat"] ls_temperature: Optional[float] ls_max_tokens: Optional[int] ls_stop: Optional[List[str]] ``` by default it will only return ```python {ls_model_type="chat", ls_stop=stop} ``` 2. Add these params to inheritable metadata in `CallbackManager.configure` 3. Implement `.get_ls_params` and populate all params for Anthropic + all subclasses of BaseChatOpenAI Sample trace: https://smith.langchain.com/public/d2962673-4c83-47c7-b51e-61d07aaffb1b/r **OpenAI**: <img width="984" alt="Screenshot 2024-05-17 at 10 03 35 AM" src="https://github.com/langchain-ai/langchain/assets/26529506/2ef41f74-a9df-4e0e-905d-da74fa82a910"> **Anthropic**: <img width="978" alt="Screenshot 2024-05-17 at 10 06 07 AM" src="https://github.com/langchain-ai/langchain/assets/26529506/39701c9f-7da5-4f1a-ab14-84e9169d63e7"> **Mistral** (and all others for which params are not yet populated): <img width="977" alt="Screenshot 2024-05-17 at 10 08 43 AM" src="https://github.com/langchain-ai/langchain/assets/26529506/37d7d894-fec2-4300-986f-49a5f0191b03">
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
.get_ls_paramsto BaseChatModel which returnsby default it will only return
{ls_model_type="chat", ls_stop=stop}Add these params to inheritable metadata in
CallbackManager.configureImplement
.get_ls_paramsand populate all params for Anthropic + all subclasses of BaseChatOpenAISample trace: https://smith.langchain.com/public/d2962673-4c83-47c7-b51e-61d07aaffb1b/r
OpenAI:
Anthropic:
Mistral (and all others for which params are not yet populated):
