Langserve Playground does not put out any text response when in chat mode (but it does when in default mode)

[joebah-joe]

So when I started this project, I initially deployed a Langchain app on google cloud run by using the python code sample from from https://codelabs.developers.google.com/codelabs/build-and-deploy-a-langchain-app-on-cloud-run#0 . I also set the playground to default mode. When I type in a question, and a little bit later the bot responded with an answer. Everything works as intended, no issue there.

However, I want to use chat mode. I understand that I am supposed to use 'dict' as an input_type for my chain, otherwise I'll get an 'Expected content-type' error as others have reported. So I did exactly that, but what happens is the "AI" line shows up in the chat, the bot thinks about it for a bit, and then leave the answer blank. See the screenshot below::

I tried several different ways to solve this, even creating my own custom input type like "class InputChat(BaseModel)" and set the chain's input_type to InputChat but the end result is the same, the bot thinks for a few seconds but does not output any text response. I'm not sure if this is an issue with Langserve, or if I coded it wrong. I would very much appreciate anyone's help with this.

Here is my server code that I deployed:

import os
from google.cloud.sql.connector import Connector
from langchain_google_vertexai import VertexAI
from langchain_google_vertexai import VertexAIEmbeddings
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import PromptTemplate, ChatPromptTemplate, MessagesPlaceholder
from langchain_community.vectorstores.pgvector import PGVector

from typing import List, Union
from langchain.pydantic_v1 import BaseModel, Field
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
from langchain.chat_models import ChatOpenAI

from fastapi import FastAPI
from fastapi.responses import RedirectResponse
from langserve import add_routes

app = FastAPI()


@app.get("/")
async def redirect_root_to_docs():
    return RedirectResponse("/docs")


# (0) Initialize chat history table




# (1) Initialize VectorStore
connector = Connector()


def getconn() -> pg8000.dbapi.Connection:
    conn: pg8000.dbapi.Connection = connector.connect(
        os.getenv("DB_INSTANCE_NAME", ""),
        "pg8000",
        user=os.getenv("DB_USER", ""),
        password=os.getenv("DB_PASS", ""),
        db=os.getenv("DB_NAME", ""),
    )
    return conn


vectorstore = PGVector(
    connection_string="postgresql+pg8000://",
    use_jsonb=True,
    engine_args=dict(
        creator=getconn,
    ),
    embedding_function=VertexAIEmbeddings(
        model_name="textembedding-gecko@003"
    )
)

# (2) Build retriever


def concatenate_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)


reports_retriever = vectorstore.as_retriever() | concatenate_docs



# (3) Create prompt template

template =     """You are a helpful financial analyst answering questions from investors. Use the retrieved analyst reports on the company's quarterly earnings to answer the questions

Reports: {reports}

Here is your question: {query}
Your answer: """

prompt_template = ChatPromptTemplate.from_template(template)    


# (4) Initialize LLM
llm = VertexAI(
    model_name="gemini-1.0-pro-001",
    temperature=0.2,
    max_output_tokens=500,
    top_k=40,
    top_p=0.95
)


# (5) Chain everything together
chain = (
    RunnableParallel({
        "reports": reports_retriever,
        "query": RunnablePassthrough()
    })
    | prompt_template
    | llm
    | StrOutputParser()
)


# add_routes(app, NotImplemented)

add_routes(app,
    chain.with_types(input_type=dict,output_type=str),         
    playground_type="chat",
           )


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8000)```

[protobon]

I encountered a similar issue with the playground's chat mode. The first message worked just fine, and the second message returned the 422 Unprocessable Entity Error.

I believe it is only an issue with the playground in chat mode because I had a conversation through the /invoke endpoint and it worked just fine.

I hope someone can help

[joebah-joe]

I believe it is only an issue with the playground in chat mode because I had a conversation through the /invoke endpoint and it worked just fine.

I hope someone can help

I see. Yes, it seems like the Chat mode requires different content-type solution for different configuration. Someone also had a the same issue (but he was using OpenAI and different chain and prompt template structure), and posted a solution that works for him (use input_type as 'dict'). However when I try to do the same as in the code I posed it doesn't work.

I tried to create my own class "Question" and use that an InputType but it didn't work either... also blank response from the AI.

class Question(BaseModel):
    messages: List[Union[HumanMessage, AIMessage, SystemMessage]] = Field()


# add_routes(app, NotImplemented)

add_routes(app,
    chain.with_types(input_type=Question),         
    playground_type="chat",
           )

Checking the Log on Google Cloud Run this was the error I Got:

TypeError: expected string or bytes-like object, got 'dict'"

So this is what's happening:

If I try to use Str as input type, I get the error that it expects text-event Stream but got JSON
If I try to use dict, or an object that is structured as dict, I get an error that it expects string or bytes-like object but got 'dict'

So if string and dict doesn't work, then I guess I should be using byte-like object as input type? But how to do that?