03-streaming.md

Streaming Tokens

Chat and Completions support the streaming of partial progress via server-sent events using the :stream parameter

(api/create-chat-completion {:model "gpt-3.5-turbo"
                             :messages [{:role "system" :content "You are a helpful assistant."}
                                        {:role "user" :content "Who won the world series in 2020?"}
                                        {:role "assistant" :content "The Los Angeles Dodgers won the World Series in 2020."}
                                        {:role "user" :content "Where was it played?"}]
                             :stream true})

Reading streamed tokens

Reading streamed tokens can be done either by providing your own callback function, or by taking from the returned core.async channel

Option 1 - Callback function

Provide your callback function in the :on-next parameter for example

(api/create-chat-completion {:model "gpt-3.5-turbo"
                             :messages [{:role "system" :content "You are a helpful assistant."}
                                        {:role "user" :content "Who won the world series in 2020?"}
                                        {:role "assistant" :content "The Los Angeles Dodgers won the World Series in 2020."}
                                        {:role "user" :content "Where was it played?"}]
                             :stream true
                             :on-next #(prn %)})

Option 2 - core.async channel

  (require '[clojure.core.async :as a])

  (def events (api/create-chat-completion {:model "gpt-3.5-turbo"
                                           :messages [{:role "system" :content "You are a helpful assistant."}
                                                      {:role "user" :content "Who won the world series in 2020?"}
                                                      {:role "assistant" :content "The Los Angeles Dodgers won the World Series in 2020."}
                                                      {:role "user" :content "Where was it played?"}]
                                           :stream true}))

  (a/go
    (loop []
      (let [event (a/<! events)]
        (when (not= :done event)

          ;; Do something with the event token
          (prn event)

          (recur)))))

If you need to ensure the core.async channel gets closed, pass the :stream/close? parameter. For more information see this PR

Example returned token

{:id "chatcmpl-6srv5jx3p4I9deNDzU7ucNXKoGS0L"
   :object "chat.completion.chunk"
   :created 1678534999
   :model "gpt-3.5-turbo-0301"
   :choices [{:delta {:content "The"} :index 0, :finish_reason nil}]}