partners/google-vertexai:fix _parse_response_candidate issue

[hsuyuming]

Description: enable _parse_response_candidate to support complex structure format.
Issue:
currently, if Gemini response complex args format, people will get "TypeError: Object of type RepeatedComposite is not JSON serializable" error from _parse_response_candidate.

response candidate example

content {
  role: "model"
  parts {
    function_call {
      name: "Information"
      args {
        fields {
          key: "people"
          value {
            list_value {
              values {
                string_value: "Joe is 30, his mom is Martha"
              }
            }
          }
        }
      }
    }
  }
}
finish_reason: STOP
safety_ratings {
  category: HARM_CATEGORY_HARASSMENT
  probability: NEGLIGIBLE
}
safety_ratings {
  category: HARM_CATEGORY_HATE_SPEECH
  probability: NEGLIGIBLE
}
safety_ratings {
  category: HARM_CATEGORY_SEXUALLY_EXPLICIT
  probability: NEGLIGIBLE
}
safety_ratings {
  category: HARM_CATEGORY_DANGEROUS_CONTENT
  probability: NEGLIGIBLE
}

error msg:

Traceback (most recent call last):
  File "/home/jupyter/user/abehsu/gemini_langchain_tools/example2.py", line 36, in <module>
    print(tagging_chain.invoke({"input": "Joe is 30, his mom is Martha"}))
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2053, in invoke
    input = step.invoke(
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 3887, in invoke
    return self.bound.invoke(
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 165, in invoke
    self.generate_prompt(
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 543, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 407, in generate
    raise e
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 397, in generate
    self._generate_with_cache(
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 576, in _generate_with_cache
    return self._generate(
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/site-packages/langchain_google_vertexai/chat_models.py", line 406, in _generate
    generations = [
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/site-packages/langchain_google_vertexai/chat_models.py", line 408, in <listcomp>
    message=_parse_response_candidate(c),
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/site-packages/langchain_google_vertexai/chat_models.py", line 280, in _parse_response_candidate
    function_call["arguments"] = json.dumps(
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/conda/envs/gemini_langchain_tools/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type RepeatedComposite is not JSON serializable

Twitter handle: @abehsu1992626

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Feb 5, 2024 11:47pm

[hsuyuming]

@lkuligin Can you help me review this PR when you get time? Thanks!

[hsuyuming]

Hi @hwchase17 :

May I know why you closed this PR?

[baskaryan]

Hi @hwchase17 :

May I know why you closed this PR?

not intentional, see #16796

[lkuligin]

can we make StubTextChatResponse an actual proto instead, please?

[lkuligin]

it looks a little bit excessive to me.
maybe smth like first_part_dict = proto.Message.to_dict(first_part) would be more readable / easier to grasp, wdyt?

[hsuyuming]

Hi @lkuligin :
This is the error i get, if i use the code you suggest.

(langchain) (base) jupyter@genai-common:~/user/abehsu/langchain$ python /home/jupyter/user/abehsu/gemini_langchain_tools/example2.py
> /opt/conda/envs/langchain/lib/python3.9/site-packages/vertexai/generative_models/_generative_models.py(1423)text()
-> if len(self.parts) > 1:
(Pdb) c
Traceback (most recent call last):
  File "/home/jupyter/user/abehsu/gemini_langchain_tools/example2.py", line 36, in <module>
    print(tagging_chain.invoke({"input": "Joe is 30, his mom is Martha"}))
  File "/home/jupyter/user/abehsu/langchain/libs/core/langchain_core/runnables/base.py", line 2053, in invoke
    input = step.invoke(
  File "/home/jupyter/user/abehsu/langchain/libs/core/langchain_core/runnables/base.py", line 3961, in invoke
    return self.bound.invoke(
  File "/home/jupyter/user/abehsu/langchain/libs/core/langchain_core/language_models/chat_models.py", line 165, in invoke
    self.generate_prompt(
  File "/home/jupyter/user/abehsu/langchain/libs/core/langchain_core/language_models/chat_models.py", line 543, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
  File "/home/jupyter/user/abehsu/langchain/libs/core/langchain_core/language_models/chat_models.py", line 407, in generate
    raise e
  File "/home/jupyter/user/abehsu/langchain/libs/core/langchain_core/language_models/chat_models.py", line 397, in generate
    self._generate_with_cache(
  File "/home/jupyter/user/abehsu/langchain/libs/core/langchain_core/language_models/chat_models.py", line 576, in _generate_with_cache
    return self._generate(
  File "/home/jupyter/user/abehsu/langchain/libs/partners/google-vertexai/langchain_google_vertexai/chat_models.py", line 380, in _generate
    generations = [
  File "/home/jupyter/user/abehsu/langchain/libs/partners/google-vertexai/langchain_google_vertexai/chat_models.py", line 382, in <listcomp>
    message=_parse_response_candidate(c),
  File "/home/jupyter/user/abehsu/langchain/libs/partners/google-vertexai/langchain_google_vertexai/chat_models.py", line 276, in _parse_response_candidate
    function_call_args_dict = proto.Message.to_dict(first_part)["function_call"]["args"]
  File "/opt/conda/envs/langchain/lib/python3.9/site-packages/proto/message.py", line 463, in to_dict
    cls.pb(instance),
  File "/opt/conda/envs/langchain/lib/python3.9/site-packages/proto/message.py", line 327, in pb
    raise TypeError(
TypeError: function_call {
  name: "Information"
  args {
    fields {
      key: "people"
      value {
        list_value {
          values {
            struct_value {
              fields {
                key: "name"
                value {
                  string_value: "Joe"
                }
              }
              fields {
                key: "age"
                value {
                  number_value: 30
                }
              }
            }
          }
          values {
            struct_value {
              fields {
                key: "name"
                value {
                  string_value: "Martha"
                }
              }
            }
          }
        }
      }
    }
  }
}
 is not an instance of Message

I find there are two of Parts class. one is from vertexai.generative_models._generative_models.Part, the other is from google.cloud.aiplatform_v1beta1.types.content.Part.

The response from gemini model is vertexai.generative_models._generative_models.Part, and this class is not an instance of Message. Any idea?

(Pdb) proto.Message.to_dict(response.candidates[0].content.parts[0])
*** TypeError: function_call {
  name: "Information"
  args {
    fields {
      key: "people"
      value {
        list_value {
          values {
            struct_value {
              fields {
                key: "name"
                value {
                  string_value: "Joe"
                }
              }
              fields {
                key: "age"
                value {
                  number_value: 30
                }
              }
            }
          }
          values {
            struct_value {
              fields {
                key: "relation"
                value {
                  string_value: "mother"
                }
              }
              fields {
                key: "name"
                value {
                  string_value: "Martha"
                }
              }
            }
          }
        }
      }
    }
  }
}
 is not an instance of Message

[hsuyuming]

This is the _parse_response_candidate i modified after suggestion

def _parse_response_candidate(response_candidate: "Candidate") -> AIMessage:
    try:
        content = response_candidate.text
    except ValueError:
        content = ""

    additional_kwargs = {}
    first_part = response_candidate.content.parts[0]
    if first_part.function_call:
        function_call = {"name": first_part.function_call.name}
        # dump to match other function calling llm for now
        function_call_args_dict = proto.Message.to_dict(first_part)["function_call"]["args"]
        function_call["arguments"] = json.dumps(
            {k: function_call_args_dict[k] for k in function_call_args_dict}
        )
        additional_kwargs["function_call"] = function_call
    return AIMessage(content=content, additional_kwargs=additional_kwargs)

[hsuyuming]

Hi @lkuligin :

I find a solution for it. Please help me review my PR again when you get time. Thank you! i learn a lot

[lkuligin]

nits: I'm not sure we need parametrize here. it might be slightly more readable without it.

[hsuyuming]

i use it for testing different response format from as the candidate.

[lkuligin]

@baskaryan take a look when you have time, please

[hsuyuming]

@baskaryan @efriis @hwchase17 Can you help me merge this PR when you get time please? Thank you

[hsuyuming]

@baskaryan I update the code base on your suggestion. Please let me know if you have other concern. Thank you!

[hsuyuming]

@baskaryan @efriis @hwchase17 Soeet to bother you guys. Is it possible to help us merge this fix? We are waiting for it for a period of time. Thank you for your help

[lkuligin]

@efriis could you add this one to the release, please?