Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unreal engine beta #1498

Closed
wants to merge 123 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
123 commits
Select commit Hold shift + click to select a range
efda9bb
add new video streaming data doc, fix existing events doc
Kishan-Dhakan Nov 1, 2022
78bcb5c
fix old links
Kishan-Dhakan Nov 11, 2022
e506744
Small changes on review. Update the browsers so the Vercel build runs.
billy-the-fish Dec 2, 2022
bfd852b
Merge branch 'beta' of https://github.com/AgoraIO/Docs-Source into en…
billy-the-fish Dec 2, 2022
564df58
Reintegrate Broadcast streaming.
billy-the-fish Dec 2, 2022
97f4079
Merge pull request #818 from AgoraIO/enhance-broadcast-streaming-kish…
billy-the-fish Dec 2, 2022
dae84cf
Beta docs with new api for media push and latest broadcast streaming.
billy-the-fish Dec 2, 2022
a810d2d
Change header to English.
billy-the-fish Dec 5, 2022
804d4fa
Merge branch 'beta' of https://github.com/AgoraIO/Docs-Source into beta
shahriarpshuvo Dec 8, 2022
4df8ac2
unreal engine get started for video calling
hussain-khalid Dec 14, 2022
9187b9c
updates
hussain-khalid Dec 14, 2022
9c1df35
some updates
hussain-khalid Dec 14, 2022
183fa5e
commit
hussain-khalid Dec 16, 2022
120ef60
issue # 98
hussain-khalid Dec 18, 2022
d22809e
ILS and some updates for video calling
hussain-khalid Dec 20, 2022
daede85
commit
hussain-khalid Dec 20, 2022
180eaa0
updates
hussain-khalid Dec 20, 2022
b4bbc33
Added NCS docs
atovpeko Dec 21, 2022
9f26b1f
Updates on review.
billy-the-fish Dec 21, 2022
0cca870
Merge pull request #1047 from AgoraIO/939-add-docs
billy-the-fish Dec 21, 2022
bcfec80
Merge branch 'milestone39-beta-site' of https://github.com/AgoraIO/Do…
billy-the-fish Dec 21, 2022
5a1f4d2
Add AINS for beta docs.
billy-the-fish Dec 21, 2022
a87313f
call quality for unreal
hussain-khalid Dec 21, 2022
16497d3
some updates
hussain-khalid Dec 22, 2022
93b94b3
updates
hussain-khalid Dec 22, 2022
65e9835
updates
hussain-khalid Dec 22, 2022
de44f01
updates
hussain-khalid Dec 22, 2022
a3efe52
get started for voice calling
hussain-khalid Dec 22, 2022
07ccfa0
updates
hussain-khalid Dec 23, 2022
5403a82
some updates for ILS
hussain-khalid Dec 24, 2022
2c02e5b
updates
hussain-khalid Dec 24, 2022
e4a7a63
svgs
hussain-khalid Dec 24, 2022
4613bb0
commit
hussain-khalid Dec 26, 2022
c6814bb
Updates on review.
billy-the-fish Dec 26, 2022
c647f18
Merge pull request #1015 from AgoraIO/84-quickstart-for-unreal-engine
billy-the-fish Dec 26, 2022
4806c6f
Updates on review.
billy-the-fish Dec 26, 2022
d73e94b
Updates on review.
billy-the-fish Dec 26, 2022
f68c0f3
Merge pull request #1053 from AgoraIO/100-integrate-channel-quality-d…
billy-the-fish Dec 26, 2022
101eb61
product workflow doc for unreal
hussain-khalid Dec 27, 2022
d7c9888
Merge pull request #1055 from AgoraIO/973-add-the-unreal-implementati…
billy-the-fish Dec 27, 2022
9593d56
Merge branch 'milestone16-unrealengine' of https://github.com/AgoraIO…
billy-the-fish Dec 27, 2022
a973529
Updates on review.
billy-the-fish Dec 27, 2022
3df440e
Merge pull request #1060 from AgoraIO/101-implement-the-product-workf…
billy-the-fish Dec 27, 2022
5d9c815
Unreal Engine 4.x, Get Started and Call Quality.
billy-the-fish Dec 27, 2022
0d525ba
Unreal Engine 4.x, Get Started, Screen share and Call Quality.
billy-the-fish Dec 27, 2022
0a0fe39
Remove Broadcast Streaming.
billy-the-fish Dec 27, 2022
d1e76b5
Merge branch 'main' of https://github.com/AgoraIO/Docs-Source into beta
billy-the-fish Dec 27, 2022
7cdbbe5
audio and voice effect for unreal
hussain-khalid Dec 28, 2022
7ace043
updates
hussain-khalid Dec 28, 2022
10514b0
updates
hussain-khalid Dec 28, 2022
b35618f
updates
hussain-khalid Dec 28, 2022
9a75411
Merge branch 'milestone38-extensions-GA' of https://github.com/AgoraI…
billy-the-fish Dec 28, 2022
b2a4c16
updates after code testing
hussain-khalid Dec 28, 2022
bb477e9
geofencing doc for unreal
hussain-khalid Dec 28, 2022
89edfc0
encryption doc for unreal
hussain-khalid Dec 28, 2022
3d86234
Update to 3D Spatial Audio.
billy-the-fish Dec 28, 2022
ec7f151
Change experimental to beta for spatial audio.
billy-the-fish Dec 29, 2022
00675ce
Merge branch 'media-push-update-api' of https://github.com/AgoraIO/Do…
billy-the-fish Dec 29, 2022
cfb3a81
media player for unreal
hussain-khalid Dec 30, 2022
39464ee
updates
hussain-khalid Dec 30, 2022
773089b
Small changes on review.
billy-the-fish Dec 30, 2022
64d9839
Update to explain that there is no UIKit for unreal.
billy-the-fish Dec 30, 2022
1661def
Merge pull request #1068 from AgoraIO/107-geofencing-for-unrealengine
billy-the-fish Dec 30, 2022
c6da485
Updates on review.
billy-the-fish Dec 30, 2022
6703e14
updates
hussain-khalid Dec 30, 2022
aa068fc
Merge pull request #1069 from AgoraIO/106-secure-communication-with-c…
billy-the-fish Dec 30, 2022
11b596f
Merge pull request #1066 from AgoraIO/103-implement-audio-and-voice-e…
billy-the-fish Dec 30, 2022
bdd5480
Updates on review.
billy-the-fish Dec 30, 2022
0c7b0f6
Merge branch '102-play-media-files-to-a-channel-for-unrealengine' of …
billy-the-fish Dec 30, 2022
cf3e5cd
Updates on review.
billy-the-fish Dec 30, 2022
4dea14d
Merge pull request #1075 from AgoraIO/102-play-media-files-to-a-chann…
billy-the-fish Dec 30, 2022
e637faa
Integrate latest changes.
billy-the-fish Dec 30, 2022
db0b3c2
Integrate latest changes for Unreal Engine.
billy-the-fish Dec 30, 2022
0a7501d
Integrate latest changes for Unreal Engine. Try again to see all the …
billy-the-fish Dec 30, 2022
1cfe28a
Integrate latest changes for Unreal Engine. Try again to see all the …
billy-the-fish Jan 2, 2023
7dc3eb8
Integrate latest changes for Unreal Engine. Try again to see all the …
billy-the-fish Jan 2, 2023
302ca7e
cloud-proxy-doc-for-unreal
hussain-khalid Jan 2, 2023
23c811a
small fix
hussain-khalid Jan 2, 2023
30bbb06
:zap: implement metaindexable
shahriarpshuvo Jan 2, 2023
e42a982
:zap: add beta pages to search indexable
shahriarpshuvo Jan 5, 2023
650f510
:zap: add beta pages to search indexable
shahriarpshuvo Jan 5, 2023
02bab7a
multi-channel doc unreal
hussain-khalid Jan 7, 2023
642807c
Merge pull request #1083 from AgoraIO/1082-implement-cloud-proxy-doc-…
billy-the-fish Jan 10, 2023
3c4afab
Merge pull request #1094 from AgoraIO/1092-implement-multi-channel-li…
billy-the-fish Jan 10, 2023
3d6aba9
custom audio and video
hussain-khalid Jan 11, 2023
4405419
Merge branch 'beta' of https://github.com/AgoraIO/Docs-Source into mi…
billy-the-fish Jan 12, 2023
18e8ae6
Video SDK v4.1.0 for Unreal Engine, beta preview.
billy-the-fish Jan 12, 2023
b98fb61
Updates on review.
billy-the-fish Jan 12, 2023
2ad52d5
Merge pull request #1105 from AgoraIO/104-implement-custom-video-and-…
billy-the-fish Jan 12, 2023
2925e45
Video SDK v4.1.0 for Unreal Engine, beta preview 3.
billy-the-fish Jan 12, 2023
7c049e5
custom audio and video doc
hussain-khalid Jan 16, 2023
23e4c71
Update
saudsami Jan 17, 2023
dcfe835
Merge pull request #1125 from AgoraIO/1116-ncs-event-description-is-i…
billy-the-fish Jan 19, 2023
604274e
Merge branch '1126-best-practice-for-cloud-recording-media-push-and-m…
billy-the-fish Jan 19, 2023
c0f0f71
updates
hussain-khalid Jan 19, 2023
e2d37df
virtual background doc
hussain-khalid Jan 20, 2023
e9fa8a9
Add required audio profile setting and bluetooth microphone override …
maxxfrazer Jan 20, 2023
9e1ce43
Added a step to add SpatialAudio binary in Build Phases.
nirm2009 Jan 23, 2023
1568994
Small update on review.
billy-the-fish Jan 24, 2023
8c3ca4c
updates for voice calling
hussain-khalid Jan 24, 2023
84a54c9
Merge pull request #1118 from AgoraIO/105-implement-raw-video-and-aud…
billy-the-fish Jan 25, 2023
ea24dd2
Updates on review.
billy-the-fish Jan 25, 2023
200710e
Merge pull request #1139 from AgoraIO/spatial-ios
maxxfrazer Jan 25, 2023
d379141
Merge branch 'milestone16-unrealengine' of https://github.com/AgoraIO…
billy-the-fish Jan 25, 2023
bfd1458
Merge pull request #1145 from AgoraIO/110-link-check-and-final-check
billy-the-fish Jan 25, 2023
34772fd
1140 spatial audio updates for Android, Flutter and Web (#1167)
saudsami Feb 6, 2023
a3b13c7
Update supported version of Unreal Engine.
billy-the-fish Feb 9, 2023
45377a2
Unreal Engine Video SDK quick start validation code changes
Pankajg123 Feb 12, 2023
426e36a
Merge branch 'beta' of https://github.com/AgoraIO/Docs-Source into mi…
billy-the-fish Feb 13, 2023
dc2ec6f
Video SDK and Voice SDK v4.0.0 Beta 1 for Unreal Engine.
billy-the-fish Feb 14, 2023
ff09e9d
Language review
atovpeko Feb 16, 2023
2c1cb63
incorporated review suggestions.
Pankajg123 Feb 19, 2023
d616a8b
updates
hussain-khalid Feb 22, 2023
4771244
incorporated Hussain review coments.
Pankajg123 Feb 23, 2023
612c35b
Incorporated further suggestions.
Pankajg123 Feb 23, 2023
00e4a42
Merge branches '1152-unreal-engine-validate-the-code-in-these-docs' a…
hussain-khalid Feb 23, 2023
527e1a8
Update on review Get started for Unreal Engine.
billy-the-fish Feb 27, 2023
d0c6373
Merge pull request #1185 from AgoraIO/1152-unreal-engine-validate-the…
billy-the-fish Feb 27, 2023
1eab989
Merge branch 'staging' of https://github.com/AgoraIO/Docs-Source into…
billy-the-fish Jun 7, 2023
9c6b51c
Merge from staging.
billy-the-fish Jun 7, 2023
c06887f
Merge from staging.
billy-the-fish Jun 7, 2023
3790257
Merge from staging.
billy-the-fish Jun 7, 2023
adae677
Merge branch 'staging' of https://github.com/AgoraIO/Docs-Source into…
atovpeko Oct 30, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
custom audio and video doc
  • Loading branch information
hussain-khalid committed Jan 16, 2023
commit 7c049e502f305b2836b54616ff254f4002a85523
5 changes: 0 additions & 5 deletions shared/video-sdk/develop/_stream-raw-audio-and-video.mdx
Original file line number Diff line number Diff line change
@@ -10,10 +10,6 @@ In some scenarios, raw audio and video captured through the camera and microphon
<PlatformWrapper platform="windows">
This page is coming soon for Windows,
</PlatformWrapper>
<PlatformWrapper platform="unreal">
This page is coming soon for Unreal Engine,
</PlatformWrapper>
<PlatformWrapper notAllowed="unreal">
<PlatformWrapper notAllowed="windows">
<PlatformWrapper platform="web">
In Web SDK, <Vg k="COMPANY"/> provides extensions like <Link to="/video-calling/develop/ai-noise-suppression">AI Denoiser</Link>, <Link to="/video-calling/develop/virtual-background">Virtual Background</Link> and <Link to="/video-calling/develop/image-enhancement">Image Enhancement</Link> for pre-process and post-process phases.
@@ -62,4 +58,3 @@ This section contains information that completes the information in this page, o
<Reference/>
</PlatformWrapper>
</PlatformWrapper>
</PlatformWrapper>
Original file line number Diff line number Diff line change
@@ -6,6 +6,8 @@ import Unity from './unity.mdx';
import MacOS from './macos.mdx';
import Flutter from './flutter.mdx';
import ReactNative from './react-native.mdx';
import Unreal from './unreal.mdx';


<MacOS />
<Android />
@@ -15,3 +17,4 @@ import ReactNative from './react-native.mdx';
<Unity />
<Flutter />
<ReactNative />
<Unreal />
Original file line number Diff line number Diff line change
@@ -0,0 +1,308 @@

<PlatformWrapper platform="unreal">

### Handle the system logic

This sections describes the steps required to use the relevant libraries, declare the necessary variables, and set up access to the UI
elements.

1. **Import the required Unreal library**

To add the texture library for the local and remote video rendering, in `MyUserWidget.h`, add the following statement before `#include "MyUserWidget.generated.h"`:

``` cpp
#include "Engine/Texture2D.h"
```

2. **Define the variables to manage audio and video processing**

In `MyUserWidget.h`, add the following declaration to the `UMyUserWidget`:

``` java
UTexture2D* LocalRenderTexture;
FSlateBrush LocalRenderBrush;
UTexture2D* RemoteRenderTexture;
FSlateBrush RemoteRenderBrush;
agora::media::IMediaEngine* MediaEngine;
class VideoFrameEventHandler* handler;
TUniquePtr<FUpdateTextureRegion2D> UpdateTextureRegionProxy;
class AudioFrameEventHandler* audioHandler;
```

### Implement processing of raw video and audio data

To register and use video and audio frame observers in your <Vpl k="CLIENT" />, take the following steps:

1. **Set up the audio frame observer**

`IAudioFrameObserver` gives you access to each audio frame after it is captured or access to each audio frame before it is played back. To setup the `IAudioFrameObserver`, do the following:

1. In `MyUserWidget.h`, add the following class after `UMyUserWidget`:

```cpp
class AudioFrameEventHandler : public agora::media::IAudioFrameObserver
{
public:
AudioFrameEventHandler(UMyUserWidget* customAudioAndVideo)
{
agoraImplementation = customAudioAndVideo;
}
~AudioFrameEventHandler() {}
bool onPlaybackAudioFrameBeforeMixing(const char* channelId, agora::rtc::uid_t uid, agora::media::IAudioFrameObserverBase::AudioFrame& audioFrame) override;
bool onRecordAudioFrame(const char* channelId, agora::media::IAudioFrameObserverBase::AudioFrame& audioFrame) override;
bool onPlaybackAudioFrame(const char* channelId, agora::media::IAudioFrameObserverBase::AudioFrame& audioFrame) override;
bool onMixedAudioFrame(const char* channelId, agora::media::IAudioFrameObserverBase::AudioFrame& audioFrame) override;
bool onEarMonitoringAudioFrame(AudioFrame& audioFrame);
AudioParams getEarMonitoringAudioParams();
int getObservedAudioFramePosition() override;
agora::media::IAudioFrameObserverBase::AudioParams getPlaybackAudioParams() override;
agora::media::IAudioFrameObserverBase::AudioParams getRecordAudioParams() override;
agora::media::IAudioFrameObserverBase::AudioParams getMixedAudioParams() override;
private:
UMyUserWidget* agoraImplementation;
agora::media::IAudioFrameObserverBase::AudioParams audioParams;
};
```

2. In `MyUserWidget.cpp`, add the following callbacks before `setupVideoSDKEngine`:

```cpp
bool AudioFrameEventHandler::onPlaybackAudioFrameBeforeMixing(const char* channelId, rtc::uid_t uid, AudioFrame& audioFrame)
{
return true;
}
bool AudioFrameEventHandler::onRecordAudioFrame(const char* channelId, AudioFrame& audioFrame)
{
return true;
}
bool AudioFrameEventHandler::onPlaybackAudioFrame(const char* channelId, AudioFrame& audioFrame)
{
return true;
}
bool AudioFrameEventHandler::onMixedAudioFrame(const char* channelId, AudioFrame& audioFrame)
{
return true;
}
bool AudioFrameEventHandler::onEarMonitoringAudioFrame(AudioFrame& audioFrame)
{
return true;
}
agora::media::IAudioFrameObserverBase::AudioParams AudioFrameEventHandler::getEarMonitoringAudioParams()
{
return agoraImplementation->audioParams;
}
int AudioFrameEventHandler::getObservedAudioFramePosition()
{
return (int)(AUDIO_FRAME_POSITION::AUDIO_FRAME_POSITION_PLAYBACK |
AUDIO_FRAME_POSITION::AUDIO_FRAME_POSITION_RECORD |
AUDIO_FRAME_POSITION::AUDIO_FRAME_POSITION_BEFORE_MIXING |
AUDIO_FRAME_POSITION::AUDIO_FRAME_POSITION_MIXED);
}
agora::media::IAudioFrameObserverBase::AudioParams AudioFrameEventHandler::getPlaybackAudioParams()
{
return agoraImplementation->audioParams;
}
agora::media::IAudioFrameObserverBase::AudioParams AudioFrameEventHandler::getRecordAudioParams()
{
return agoraImplementation->audioParams;
}
agora::media::IAudioFrameObserverBase::AudioParams AudioFrameEventHandler::getMixedAudioParams()
{
return agoraImplementation->audioParams;
}
```

2. **Set up the video frame observer**

`IVideoFrameObserver` gives you access to each local video frame after it is captured and access to each remote video frame before it is played back. In this example, your read the captured video frames to create textures that you use to render the local and remote video frames. To set up `IVideoFrameObserver` in your <Vpl k = "CLIENT"/>, do the following:

1. In `MyUserWidget.h`, add the following class after `UMyUserWidget`:

``` cpp
class VideoFrameEventHandler : public agora::media::IVideoFrameObserver
{
public:
VideoFrameEventHandler(UMyUserWidget *customAudioAndVideo)
{
agoraImplementation = customAudioAndVideo;
}
~VideoFrameEventHandler() {}
virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) override;
virtual bool onPreEncodeVideoFrame(VideoFrame& videoFrame) override;
virtual bool onSecondaryCameraCaptureVideoFrame(VideoFrame& videoFrame) override;
virtual bool onSecondaryPreEncodeCameraVideoFrame(VideoFrame& videoFrame) override;
virtual bool onScreenCaptureVideoFrame(VideoFrame& videoFrame) override;
virtual bool onPreEncodeScreenVideoFrame(VideoFrame& videoFrame) override;
virtual bool onMediaPlayerVideoFrame(VideoFrame& videoFrame, int mediaPlayerId) override;
virtual bool onSecondaryScreenCaptureVideoFrame(VideoFrame& videoFrame) override;
virtual bool onSecondaryPreEncodeScreenVideoFrame(VideoFrame& videoFrame) override;
virtual agora::media::IVideoFrameObserver::VIDEO_FRAME_PROCESS_MODE getVideoFrameProcessMode() override;
virtual agora::media::base::VIDEO_PIXEL_FORMAT getVideoFormatPreference() override;
virtual bool getRotationApplied() override;
virtual bool getMirrorApplied() override;
virtual uint32_t getObservedFramePosition() override;
virtual bool isExternal() override;
virtual bool onRenderVideoFrame(const char* channelId, rtc::uid_t remoteUid, VideoFrame& videoFrame) override;
virtual bool onTranscodedVideoFrame(VideoFrame& videoFrame) override;
private:
UMyUserWidget* agoraImplementation;
};
```
2. In `MyUserWidget.cpp`, add the following callbacks before `NativeDestruct`:

```cpp
bool VideoFrameEventHandler::onPreEncodeVideoFrame(VideoFrame& videoFrame)
{
UE_LOG(LogTemp, Warning, TEXT("VideoFrameEventHandler::onPreEncodeVideoFrame"));
return true;
}
bool VideoFrameEventHandler::onSecondaryCameraCaptureVideoFrame(VideoFrame& videoFrame)
{
UE_LOG(LogTemp, Warning, TEXT("VideoFrameEventHandler::onPreEncodeVideoFrame"));
return true;
}
bool VideoFrameEventHandler::onSecondaryPreEncodeCameraVideoFrame(VideoFrame& videoFrame)
{
UE_LOG(LogTemp, Warning, TEXT("VideoFrameEventHandler::onSecondaryPreEncodeCameraVideoFrame"));
return true;
}
bool VideoFrameEventHandler::onScreenCaptureVideoFrame(VideoFrame& videoFrame)
{
UE_LOG(LogTemp, Warning, TEXT("VideoFrameEventHandler::onScreenCaptureVideoFrame"));
return true;
}
bool VideoFrameEventHandler::onPreEncodeScreenVideoFrame(VideoFrame& videoFrame)
{
UE_LOG(LogTemp, Warning, TEXT("VideoFrameEventHandler::onPreEncodeScreenVideoFrame"));
return true;
}
bool VideoFrameEventHandler::onMediaPlayerVideoFrame(VideoFrame& videoFrame, int mediaPlayerId)
{
return true;
}
bool VideoFrameEventHandler::onSecondaryScreenCaptureVideoFrame(VideoFrame& videoFrame)
{
UE_LOG(LogTemp, Warning, TEXT("VideoFrameEventHandler::onSecondaryScreenCaptureVideoFrame"));
return true;
}
bool VideoFrameEventHandler::onSecondaryPreEncodeScreenVideoFrame(VideoFrame& videoFrame)
{
UE_LOG(LogTemp, Warning, TEXT("VideoFrameEventHandler::onSecondaryPreEncodeScreenVideoFrame"));
return true;
}
agora::media::IVideoFrameObserver::VIDEO_FRAME_PROCESS_MODE VideoFrameEventHandler::getVideoFrameProcessMode()
{
return PROCESS_MODE_READ_WRITE;
}
agora::media::base::VIDEO_PIXEL_FORMAT VideoFrameEventHandler::getVideoFormatPreference()
{
return agora::media::base::VIDEO_PIXEL_RGBA;
}
bool VideoFrameEventHandler::getRotationApplied()
{
return true;
}
bool VideoFrameEventHandler::getMirrorApplied()
{
return true;
}
uint32_t VideoFrameEventHandler::getObservedFramePosition()
{
return agora::media::base::POSITION_POST_CAPTURER | agora::media::base::POSITION_PRE_RENDERER;
}
bool VideoFrameEventHandler::isExternal()
{
return true;
}
bool VideoFrameEventHandler::onRenderVideoFrame(const char* channelId, rtc::uid_t remoteUid, VideoFrame& videoFrame)
{
AsyncTask(ENamedThreads::GameThread, [=]()
{
UE_LOG(LogTemp, Warning, TEXT("UMyUserWidget::onRenderVideoFrame"));
if (agoraImplementation->RemoteRenderTexture == nullptr || !agoraImplementation->RemoteRenderTexture->IsValidLowLevel())
{
agoraImplementation->RemoteRenderTexture = UTexture2D::CreateTransient(videoFrame.width, videoFrame.height, PF_R8G8B8A8);
}
else
{
int yStride = videoFrame.yStride;
agoraImplementation->UpdateTextureRegionProxy = MakeUnique<FUpdateTextureRegion2D>(0, 0, 0, 0, videoFrame.width, videoFrame.height);
UTexture2D* tex = (UTexture2D*)agoraImplementation->RemoteRenderTexture;
uint8* raw = (uint8*)tex->GetPlatformData()->Mips[0].BulkData.Lock(LOCK_READ_WRITE);
memcpy(raw, videoFrame.yBuffer, videoFrame.height * videoFrame.width * 4);
tex->GetPlatformData()->Mips[0].BulkData.Unlock();
delete[] raw;
tex->UpdateTextureRegions(0, 1, agoraImplementation->UpdateTextureRegionProxy.Get(), yStride, (uint32)4, static_cast<uint8_t*>(raw));
agoraImplementation->RemoteRenderBrush.SetResourceObject(tex);
if (agoraImplementation->remoteView != nullptr)
{
agoraImplementation->remoteView->SetBrush(agoraImplementation->RemoteRenderBrush);
}
}
});
return true;
}
bool VideoFrameEventHandler::onCaptureVideoFrame(VideoFrame& videoFrame)
{
UE_LOG(LogTemp, Warning, TEXT("VideoFrameEventHandler::onCaptureVideoFrame"));
AsyncTask(ENamedThreads::GameThread, [=]()
{
if (agoraImplementation->LocalRenderTexture == nullptr || !agoraImplementation->LocalRenderTexture->IsValidLowLevel())
{
UE_LOG(LogTemp, Warning, TEXT("VideoFrameEventHandler::CreateTransient"));
agoraImplementation->LocalRenderTexture = UTexture2D::CreateTransient(videoFrame.width, videoFrame.height, PF_R8G8B8A8);
}
else
{
int yStride = videoFrame.yStride;
agoraImplementation->UpdateTextureRegionProxy = MakeUnique<FUpdateTextureRegion2D>(0, 0, 0, 0, videoFrame.width, videoFrame.height);
UTexture2D* tex = (UTexture2D*)agoraImplementation->LocalRenderTexture;
uint8* raw = (uint8*)tex->GetPlatformData()->Mips[0].BulkData.Lock(LOCK_READ_WRITE);
memcpy(raw, videoFrame.yBuffer, videoFrame.height * videoFrame.width * 4);
tex->GetPlatformData()->Mips[0].BulkData.Unlock();
tex->UpdateTextureRegions(0, 1, agoraImplementation->UpdateTextureRegionProxy.Get(), yStride, (uint32)4, static_cast<uint8_t*>(raw));
delete[] raw;
agoraImplementation->LocalRenderBrush.SetResourceObject(tex);
if (agoraImplementation->localView != nullptr)
{
UE_LOG(LogTemp, Warning, TEXT("VideoFrameEventHandler::New brush applied"));
agoraImplementation->localView->SetBrush(agoraImplementation->LocalRenderBrush);
}
}
});
return true;
}
```
Note that you must set the return value in `getVideoFrameProcessMode` to `PROCESS_MODE_READ_WRITE` in order for your raw data changes to take effect.

3. **Register the video and audio frame observers**

To receive callbacks declared in `IVideoFrameObserver` and `IAudioFrameObserver`, you must register the video and audio frame observers with the <Vg k="ENGINE" /> before joining a channel. To specify the format of audio frames captured by each `IAudioFrameObserver` callback, use the `setRecordingAudioFrameParameters`, `setMixedAudioFrameParameters` and `setPlaybackAudioFrameParameters` methods. To do this, in `MyUserWidget.cpp`, add the following at the end of `setupVideoSDKEngine`:

``` cpp
agoraEngine->queryInterface(AGORA_IID_MEDIA_ENGINE, (void**)&MediaEngine);
handler = new VideoFrameEventHandler(this);
int res = MediaEngine->registerVideoFrameObserver(handler);
audioHandler = new AudioFrameEventHandler(this);
MediaEngine->registerAudioFrameObserver(audioHandler);
audioHandler = new AudioFrameEventHandler(this);
MediaEngine->registerAudioFrameObserver(audioHandler);
// Set the format of the captured raw audio data.
int SAMPLE_RATE = 16000, SAMPLE_NUM_OF_CHANNEL = 1, SAMPLES_PER_CALL = 1024;
agoraEngine->setRecordingAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL,
RAW_AUDIO_FRAME_OP_MODE_READ_WRITE, SAMPLES_PER_CALL);
agoraEngine->setPlaybackAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL,
RAW_AUDIO_FRAME_OP_MODE_READ_WRITE, SAMPLES_PER_CALL);
agoraEngine->setMixedAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, SAMPLES_PER_CALL);
```

4. **Unregister the video and audio observers when you close the <Vpl k = "CLIENT"/>**

When you close the <Vpl k = "CLIENT"/>, you unregister the frame observers by calling the register frame observer method again with a `null` pointer. To do this, in `MyUserWidget.cpp`, add the following lines to `NativeDestruct` before `agoraEngine->unregisterEventHandler(this);`:

``` cpp
MediaEngine->registerAudioFrameObserver(nullptr);
MediaEngine->registerVideoFrameObserver(nullptr);
```

</PlatformWrapper>
Original file line number Diff line number Diff line change
@@ -6,6 +6,8 @@ import Unity from './unity.mdx';
import MacOS from './macos.mdx';
import Flutter from './flutter.mdx';
import ReactNative from './react-native.mdx';
import Unreal from './unreal.mdx';


<MacOS />
<Android />
@@ -15,3 +17,4 @@ import ReactNative from './react-native.mdx';
<Unity />
<Flutter />
<ReactNative />
<Unreal />
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
<PlatformWrapper platform="unreal">

1. Generate a temporary token in <Vg k="CONSOLE" />.

1. In your browser, navigate to the <Link target="_blank" to="{{Global.DEMO_BASIC_VIDEO_CALL_URL}}"><Vg k="COMPANY" /> web demo</Link> and update _App ID_, _Channel_, and _Token_ with the values for your temporary token, then click **Join**.

1. In Unreal Studio, open `MyUserWidget.h`, and update `appId`, `channelName` and `token` with the values for your temporary token.

1. In Unreal Studio, click **Play**. A moment later you see the <Vpl k = "CLIENT"/> running on your device.

If this is the first time you run the project, grant microphone and camera access to your app.

1. Press **Join** to see the video feed from the web app.

You see the local video in the local view and remote video in the remote view. This means your <Vpl k = "CLIENT"/> is rendering the capture video frames that you receive in the following callbacks:

- `onRenderVideoFrame`: Gets the remote user video frame data.

- `onCaptureVideoFrame`: Gets the captured video frame data.


1. Test processing of raw audio data.

Edit the `IAudioFrameObserver` callbacks by adding code that processes the raw audio data you receive in the following callbacks:

- `onRecordAudioFrame`: Gets the captured audio frame data

- `onPlaybackAudioFrame`: Gets the audio frame for playback

</PlatformWrapper>
Loading