Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect model used for tool execution when both --default-model is used in cli to execute scripts that have model set. #645

Open
sangee2004 opened this issue Jul 17, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@sangee2004
Copy link
Contributor

gptscript version - v0.0.0-dev-6e92ee7a-dirty

Steps to reproduce the problem:

Pre Req - In my case I dont have gcloud authentication setup , so using github.com/gptscript-ai/gemini-vertexai-provider would fail.

Scenario 1:

  1. Execute script with default provider set to gemini vertexai provider for the following script - gptscript --disable-cache --debug-messages --default-model 'gemini-1.0-pro from github.com/gptscript-ai/gemini-vertexai-provider' test_answers_from_internet.gpt
Tools: github.com/gptscript-ai/answers-from-the-internet
Model: gpt-4o

Who won 2024 Superbowl?
  1. Script execution succeeds in this case since gpt-4o model gets used to make LLM calls to the Node GPTScript SDK when executing github.com/gptscript-ai/answers-from-the-internet tool.
gptscript --disable-cache --debug-messages --default-model 'gemini-1.0-pro from github.com/gptscript-ai/gemini-vertexai-provider' test_answers_from_internet.gpt     
15:55:52 WARNING: Changing the default model can have unknown behavior for existing tools. Use the model field per tool instead.
15:55:53 started  [main]
15:55:53 sent     [main]
15:55:53 messages [request={"model":"gpt-4o","messages":[{"role":"system","content":"\nYou are task oriented system.\nYou receive input from a user, process the input from the given instructions, and then output the result.\nYour objective is to provide consistent and correct results.\nYou do not need to explain the steps taken, only provide the result to the given instructions.\nYou are referred to as a tool.\nYou don't move to the next step until you have a result.\n\nWho won 2024 Superbowl?"}],"temperature":0,"tools":[{"type":"function","function":{"name":"answersFromTheInternet","description":"Uses Google to answer the provided question","parameters":{"properties":{"question":{"description":"the question to ask","type":"string"}},"type":"object"}}}]}]
         content  [1] content | Waiting for model response...
         content  [1] content | <tool call> answersFromTheInternet -> {"question":"Who won 2024 Superbowl?"}
15:55:54 messages [response={"role":"assistant","content":[{"toolCall":{"index":0,"id":"call_70LEmmgIsTXkaQYcb0tm05uC","function":{"name":"answersFromTheInternet","arguments":"{\"question\":\"Who won 2024 Superbowl?\"}"}}}],"usage":{"promptTokens":135,"completionTokens":23,"totalTokens":158}}]
15:55:54 started  [answers-from-the-internet(2)] [input={"question":"Who won 2024 Superbowl?"}]
15:55:54 sent     [answers-from-the-internet(2)]
15:55:54 messages [request={"command":["/usr/bin/env","npm","--prefix","/Users/sangeethahariharan/Library/Caches/gptscript/repos/401d974b425bbdd5637618c492beafbb644cad4c/tool.gpt/node21","run","tool"],"input":"{\"question\":\"Who won 2024 Superbowl?\"}"}]
         content  [2] content | 
         content  [2] content | > tool
         content  [2] content | > node --no-warnings --loader ts-node/esm src/server.ts
         content  [2] content | 
         content  [2] content | slow page: https://www.hollywoodreporter.com/news/general-news/super-bowl-2024-winner-1235822266/
### Sources:
         content  [2] content | - [Wikipedia](https://en.wikipedia.org/wiki/Super_Bowl_LVIII)
         content  [2] content | - [Fox Sports](https://www.foxsports.com/nfl/super-bowl)
         content  [2] content | - [The Hollywood Reporter](https://www.hollywoodreporter.com/news/general-news/super-bowl-2024-winne ...
         content  [2] content | 
         content  [2] content | ### Answer:
         content  [2] content | The Kansas City Chiefs won Super Bowl LVIII, defeating the San Francisco 49ers 25-22 in overtime. Pa ...
15:56:02 messages [response={"err":null,"output":"\n\u003e tool\n\u003e node --no-warnings --loader ts-node/esm src/server.ts\n\n### Sources:\n- [Wikipedia](https://en.wikipedia.org/wiki/Super_Bowl_LVIII)\n- [Fox Sports](https://www.foxsports.com/nfl/super-bowl)\n- [The Hollywood Reporter](https://www.hollywoodreporter.com/news/general-news/super-bowl-2024-winner-1235822266/)\n\n### Answer:\nThe Kansas City Chiefs won Super Bowl LVIII, defeating the San Francisco 49ers 25-22 in overtime. Patrick Mahomes was named the MVP of the game."}]
15:56:02 ended    [answers-from-the-internet(2)] [output=\u003e tool\n\u003e node --no-warnings --loader ts-node/esm src/server.ts\n\n### Sources:\n- [Wikipedia](https://e...]
15:56:02 continue [main]
15:56:02 sent     [main]
15:56:02 messages [request={"model":"gpt-4o","messages":[{"role":"system","content":"\nYou are task oriented system.\nYou receive input from a user, process the input from the given instructions, and then output the result.\nYour objective is to provide consistent and correct results.\nYou do not need to explain the steps taken, only provide the result to the given instructions.\nYou are referred to as a tool.\nYou don't move to the next step until you have a result.\n\nWho won 2024 Superbowl?"},{"role":"assistant","content":"","tool_calls":[{"id":"call_70LEmmgIsTXkaQYcb0tm05uC","type":"function","function":{"name":"answersFromTheInternet","arguments":"{\"question\":\"Who won 2024 Superbowl?\"}"}}]},{"role":"tool","content":"\n\u003e tool\n\u003e node --no-warnings --loader ts-node/esm src/server.ts\n\n### Sources:\n- [Wikipedia](https://en.wikipedia.org/wiki/Super_Bowl_LVIII)\n- [Fox Sports](https://www.foxsports.com/nfl/super-bowl)\n- [The Hollywood Reporter](https://www.hollywoodreporter.com/news/general-news/super-bowl-2024-winner-1235822266/)\n\n### Answer:\nThe Kansas City Chiefs won Super Bowl LVIII, defeating the San Francisco 49ers 25-22 in overtime. Patrick Mahomes was named the MVP of the game.","name":"answersFromTheInternet","tool_call_id":"call_70LEmmgIsTXkaQYcb0tm05uC"}],"temperature":0,"tools":[{"type":"function","function":{"name":"answersFromTheInternet","description":"Uses Google to answer the provided question","parameters":{"properties":{"question":{"description":"the question to ask","type":"string"}},"type":"object"}}}]}]
         content  [1] content | Waiting for model response...
         content  [1] content | The Kansas City Chiefs won Super Bowl LVIII, defeating the San Francisco 49ers 25-22 in overtime. Patrick Mahomes was named the MVP of the game.
15:56:03 messages [response={"role":"assistant","content":[{"text":"The Kansas City Chiefs won Super Bowl LVIII, defeating the San Francisco 49ers 25-22 in overtime. Patrick Mahomes was named the MVP of the game."}],"usage":{"promptTokens":299,"completionTokens":36,"totalTokens":335}}]
15:56:03 ended    [main] [output=The Kansas City Chiefs won Super Bowl LVIII, defeating the San Francisco 49ers 25-22 in overtime. Pa...]
15:56:03 usage    [total=493] [prompt=434] [completion=59]

OUTPUT:

The Kansas City Chiefs won Super Bowl LVIII, defeating the San Francisco 49ers 25-22 in overtime. Patrick Mahomes was named the MVP of the game.

Expected Behavior:
gemini provider should be used for the LLM calls to the Node GPTScript SDK when executing github.com/gptscript-ai/answers-from-the-internet tool.

@sangee2004 sangee2004 added the bug Something isn't working label Jul 17, 2024
@sangee2004 sangee2004 changed the title Incorrect model used for github.com/gptscript-ai/answers-from-the-internet tool execution. Incorrect model used for answers-from-the-internet tool execution. Jul 17, 2024
@sangee2004
Copy link
Contributor Author

The same issue is also seen when executing a chat enabled script with answers-from-the-internet tool with --diasble-tui or --debug-messages option.

This issue is not seen when executing the script in TUI.

Tools: github.com/gptscript-ai/answers-from-the-internet
Model: gpt-4o
chat: true

You are a good assistant.Wait for the user to ask a question and always return sources for the search result.

Pre Req -Do not have gcloud authentication setup , so using github.com/gptscript-ai/gemini-vertexai-provider would fail.

  1. gptscript --disable-cache --debug-messages --default-model 'gemini-1.0-pro from github.com/gptscript-ai/gemini-vertexai-provider' test_answers_from_internet_chat.gpt --> succeeds --> This is not expected behavior.

  2. gptscript --disable-cache --disable-tui --default-model 'gemini-1.0-pro from github.com/gptscript-ai/gemini-vertexai-provider' test_answers_from_internet_chat.gpt --> succeeds --> This is not expected behavior.

  3. gptscript --disable-cache --default-model 'gemini-1.0-pro from github.com/gptscript-ai/gemini-vertexai-provider' test_answers_from_internet_chat.gpt --> fails as we expect it to.

@sangee2004 sangee2004 changed the title Incorrect model used for answers-from-the-internet tool execution. Incorrect model used for tool execution when both --default-model is used in cli to execute scripts that have model set. Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant