Complete Unreal Engine plugin for the OpenAI API

This is an unofficial community-maintained library.

This plugin is a comprehensive Unreal Engine wrapper for the OpenAI API. It supports all OpenAI endpoints, including:

• Models
• Completions
• Chat
• Images (DALL·E 3, DALL·E 2)
• Vision
• Embeddings
• Batch
• Speech
• Audio
• Files
• Uploads
• Moderations
• Fine-tuning
• Functions
• Debugging requests

All requests are available in both C++ and Blueprints:

void AAPIOverview::CreateImage()
{
    Provider->SetLogEnabled(true);
    Provider->OnRequestError().AddUObject(this, &ThisClass::OnRequestError);
    Provider->OnCreateImageCompleted().AddLambda(
        [](const FImageResponse& Response, const FOpenAIResponseMetadata& Metadata)
        {
            auto* ArtTexture = UImageFuncLib::Texture2DFromBytes(Response.Data[0].B64_JSON);
            UE_LOGFMT(LogAPIOverview, Display, "{0}", Response.Data[0].B64_JSON);
        });

    FOpenAIImage Image;
    Image.Model = UOpenAIFuncLib::OpenAIImageModelToString(EImageModelEnum::DALL_E_3);
    Image.N = 1;
    Image.Prompt = "Bear with beard drinking beer";
    Image.Size = UOpenAIFuncLib::OpenAIImageSizeDalle3ToString(EImageSizeDalle3::Size_1024x1024);
    Image.Response_Format = UOpenAIFuncLib::OpenAIImageFormatToString(EOpenAIImageFormat::B64_JSON);
    Image.Quality = UOpenAIFuncLib::OpenAIImageQualityToString(EOpenAIImageQuality::Standard);
    Image.Style = UOpenAIFuncLib::OpenAIImageStyleToString(EOpenAIImageStyle::Natural);

    Provider->CreateImage(Image, Auth);
}

Supported Unreal Engine Versions

• Unreal Engine 5.5 (master), 5.4, 5.3, 5.2

Tutorials

• Plugin overview and usage example [English subtitles].
• How to use functions.
• DALLE 3 | Text To Speech | Vision [English subtitles].

Updates

• Add gpt-4o models

• Vision & Text To Speech Demo

• DALL·E 3 has been added. Details

• Text to speech has been added. Details

• Vision has been added. Details

• Functions support added! You can check the tutorial via link

Installation

Marketplace link

• Complete OpenAI API plugin

C++

1. Create a new C++ project in Unreal Engine.
2. Create a Plugins folder at the root of your project.
3. Add the OpenAI plugin to your Unreal Engine Plugins folder.
4. The easiest way to add plugins is by adding the current plugin as a submodule. Use the following command in the root folder of your project:

git submodule add https://github.com/life-exe/UnrealOpenAIPlugin Plugins/OpenAI

5. Alternatively, you can download the source code from the current repository and copy the files to your Plugins folder.
6. When complete, your Unreal Engine project structure should resemble the following:

7. Do the authentication steps
8. Generate Visual Studio project files.
9. Add OpenAI plugin to your YourProject.Build.cs:

public class YourProject : ModuleRules
{
    public YourProject(ReadOnlyTargetRules Target) : base(Target)
    {
        PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;

        PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "OpenAI" });

        PublicIncludePaths.AddRange(new string[] { "YourProject" });
    }
}

10. Build your project and run Editor.
11. In the Editor, navigate to Edit->Plugins. Find and activate the OpenAI plugin.
12. Restart the editor.
13. Make sure that your .uproject file contains plugin:

{
    "FileVersion": 3,
    "EngineAssociation": "5.x",
    "Category": "",
    "Description": "",
    "Modules": [
        {
            "Name": "ProjectName",
            "Type": "Runtime",
            "LoadingPhase": "Default",
            "AdditionalDependencies": [
                "Engine"
            ]
        }
    ],
    "Plugins": [
        {
            "Name": "OpenAI",
            "Enabled": true,
            "MarketplaceURL": "com.epicgames.launcher://ue/marketplace/product/97eaf1e101ab4f29b5acbf7dacbd4d16"
        }
    ]
}

14. Start using the plugin.

Blueprints

1. Create a blueprint project.
2. Create a Plugins folder in the root of your project.
3. Download precompiled plugin from the Releases.
4. Unzip plugin to the Plugins folder, specifically to Plugins/OpenAI.
5. Do the authentication steps
6. Run the Editor <YourProjectName>.uproject
7. In the Editor, navigate to Edit->Plugins. Find and activate the OpenAI plugin.
8. Restart the editor.
9. Make sure that your .uproject file contains plugin:

{
    "FileVersion": 3,
    "EngineAssociation": "5.x",
    "Category": "",
    "Description": "",
    "Modules": [
        {
            "Name": "ProjectName",
            "Type": "Runtime",
            "LoadingPhase": "Default",
            "AdditionalDependencies": [
                "Engine"
            ]
        }
    ],
    "Plugins": [
        {
            "Name": "OpenAI",
            "Enabled": true,
            "MarketplaceURL": "com.epicgames.launcher://ue/marketplace/product/97eaf1e101ab4f29b5acbf7dacbd4d16"
        }
    ]
}

10. Start using the plugin.

Authentication

1. Create an OpenAI account

2. Generate and store your API Key: sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

3. Record your Organization ID: org-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

4. Create a Project ID: proj_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

5. At the root of your Unreal Engine project, create a file named OpenAIAuth.ini (you can use setup_auth_files.bat in the plugin root folder. It will copy .ini files from template folder) with the following content:

APIKey=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
OrganizationID=org-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ProjectID=proj_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

6. Once completed, your Unreal Engine project structure might look like this:

7. Actually you can left OrganizationID empty. It doesn't affect on auth.

8. If you haven't created a ProjectID don't put ProjectID=proj_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx in the file.

9. Do the authentication steps for services if you need them.

Finally, compile your project and launch the Unreal Editor.

Examples of Usage

Widget example

This provides a basic example of request usage with Blueprints.

1. Make sure that plugin content activated:

2. Open the plugin content folder.
3. Open the level:

4. Run the game.
5. Test the API with the widget examples:

Feel free to modify requests in the widget code to test the API with different parameters:

Unreal Editor Chat-GPT

This is the Chat-GPT implementation with token streaming support.

1. Open the plugin content folder.
2. Navigate to the EditorWidgets folder.
3. Right-click on the ChatGPT editor utility widget:

3. Start chatting:

4. You can save the chat history to a file using the Dump button. The specific location where the history is saved can be checked in the chat or logs:

Unreal Editor Chat-GPT with OpenAI Functions

You can build your own services (addons, plugins) on top of ChatGPT:

Currently the plugin has two services available by default:

• Weather

• News

How to set up default services

I use free API for services. But you need to create accounts to get API keys.

1. Weather serivces: https://weatherstack.com. Once your account is created, you will find API Access Key on your dashboard.
2. News service: https://newsapi.org. Once your account is created, you will find API Key on your dashboard.
3. In the root of your Unreal Engine project, create a file called OnlineServicesAuth.ini (you can use setup_auth_files.bat in the plugin root folder. It will copy .ini files from template folder) with the following content:

WeatherstackAccessKey=dbe5fcdd54f2196d2cdc5194cf5
NewsApiOrgApiKey=1dec1793ed3940e880cc93b4087fcf96

4. Enter your APi keys for each service.
5. Compile your project, launch the Unreal Editor, launch ChatGPT editor widget, select the services that you want to use.
6. Ask for the weather or latest news somewhere. Example prompts:

What is the weather like in Oslo?

What is the latest news from Microsoft? (2 headlines)

7. Always check the logs when you encounter an error.

Blueprint Nodes Overview

This blueprint demonstrates all available nodes.

1. Open the plugin content folder:

2. Open BP_APIOverview blueprint.
3. Check the available nodes: functions and structs:

There are also several nodes that could be useful in other projects. Feel free to copy and paste the plugin code if you need them:

C++ example

1. Open Plugins\OpenAI\Source\OpenAI\Private\Sample\APIOverview.cpp actor
2. Uncomment the function that executes the request you want to test. Navigate to the function and adjust all the request parameters as needed.
3. Compile your project and open Unreal Editor.
4. Drop the AAPIOverview actor into any level.
5. Run the game.
6. Check the Output Log.

Plugin structure:

• OpenAIModule - core classes.
• OpenAIEditorModule - Chat-GPT editor utility widget.
• OpenAITestRunnerModule - unit tests.

Documentation

I highly recommend reading the OpenAI documentation for a better understanding of how all requests work:

• Overview
• API-Reference

In addition plugin code has its own documentation generated with help of Doxygen:

• Documentation

You can generate the plugin documentation locally by following these steps:

1. Update all submodules. You can use the batch script located at the root of the plugin folder: update_submodules.bat
2. Ensure that Doxygen is installed on your system.
3. Ensure that Python is installed on your system.
4. Generate the documentation using the batch script at the root of the plugin folder: generate_docs.bat
5. After the generation process, the documentation will be available in the Documentation folder at the root of the plugin: Documentation\html\index.html

Tests

Unit tests are available in the OpenAITestRunnerModule. You can initiate them using the Session Frontend:

There are also API tests that request a real endpoints. These have stress filters. They are mostly for my personal use to check API changes. But you can use test spec file to learn how to make requests, because testing in software development is a part of documentation:

Packaging

1. Verify that your .uproject file contains the following:

{
    "Name": "OpenAI",
    "Enabled": true
}

2. Package the project as you always do (kudos if you use a CI solution like Jenkins).
3. You can handle authentication in your project as you see fit. However, if you choose to use a file-based method, such as in a plugin, please remember to include the OpenAIAuth.ini and OnlineServicesAuth.ini files in your packaged folder:

• Build/Windows/<YourProjectName>/OpenAIAuth.ini
• Build/Windows/<YourProjectName>/OnlineServicesAuth.ini

If you are having problems loading the file, please check the logs to see where it might be located:

LogOpenAIFuncLib: Error: Failed loading file: C:/_Projects/UE5/OpenAICpp/Build/Windows/OpenAICpp/OpenAIAuth.ini

Limitations

• OpenAI hosts a variety of different models. Please check the models that are compatible with the particular request.

Miscellaneous

I would appreciate bug reports and pull-request fixes.

Enjoy! 🚀️