Skip to content

Commit

Permalink
update read me
Browse files Browse the repository at this point in the history
  • Loading branch information
endink committed May 23, 2024
1 parent fa39b08 commit a2f5cad
Show file tree
Hide file tree
Showing 5 changed files with 61 additions and 22 deletions.
22 changes: 16 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ An Unreal Engine plugin that helps you use AI and ML techniques in your unreal e
<p align="center">
<a href=""><img src="https://img.shields.io/badge/Windows-Win 10/11 X64-brightgreen.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Android->=8.0 (Arm64)-brightgreen.svg"></a>
<a href=""><img src="https://img.shields.io/badge/VC++ Runtime->=14.36.32532-aff.svg"></a>
<a href=""><img src="https://img.shields.io/badge/MSVC->=14.38.33130-aff.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Windows SDK->=10.0.22621-orange.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Visual Sutdio->=2022.17.6.3-blue"></a>
<a href=""><img src="https://img.shields.io/badge/Visual Sutdio->=2022.17.9.6-blue"></a>
</p>
<p align="center">
<a href="https://opensource-labijie-com.translate.goog/Mediapipe4u-plugin/update_logs/?_x_tr_sl=zh-CN&_x_tr_tl=en&_x_tr_hl=zh-CN&_x_tr_pto=wapp">News</a>
Expand All @@ -41,19 +41,29 @@ An Unreal Engine plugin that helps you use AI and ML techniques in your unreal e

# What's news

Latest update: `20240520`
Latest update: `20240523`

- [new] :rainbow: Licenses is no longer required to use MediaPipe4U (all plugin) in the UE editor (you can still use all plugins even though the license file has expireds).
- [new] :rainbow: Unreal Engine version **5.4** support, I skipped UE 5.3, I'm sorry for that.
- [new] :rainbow: **Control Rig** supported (preview). Add a new animation blueprint node `ControlRigFromPose` that used to convert FK poses to Control Rig Blueprints pose.
- [new] :fire: Upgrade to latest goole mediapipe holistic task api, got better performace for holistic tracking.
- [fix] [#130](https://github.com/endink/Mediapipe4u-plugin/issues/130): When packaging a game, the mediapipe model file is missing.
- [improve] Optimize android performace, now Android platfrom can reach **15**-**20** fps (CPU: Qualcomm Snapdragon 870 ).
- [new] :rainbow: Add `StopAsync` to `MediaPipeHolisticComponent` to support asynchronous stop Mediapipe.
- [new] :rainbow: `MediaPipeHolisticComponent` Display "Raw mode" support for Android platform.
- [fix] :bug: [#130](https://github.com/endink/Mediapipe4u-plugin/issues/130): When packaging a game, the mediapipe model file is missing.
- [fix] :bug: [#124](https://github.com/endink/Mediapipe4u-plugin/issues/124): `BVH Recorder` crash if skeleton bone index was not used for compact space pose.
- [fix] :bug: The `AnchorWidgetInCanvas` function exceeds the size limit when scaling the frame image.
- [improve] :rose: Optimize android performace, now Android platfrom can reach **15**-**20** fps (CPU: Qualcomm Snapdragon 870 ).
- [improve] :rose: The Mediapipe4U solver is now adaptive to LOD changes and motion capture can also be done when the LOD changes.
- [remove] :nauseated_face: LLM plugin has been removed. Because of the high latency of the LLM and the frequent updates of the llama.cpp project for a long time, there is no stable version. So I remove it, and maybe it will come back in the future.
- [remove] :nauseated_face: Since this version, the 5.0.x version of UnrealEngine is no longer supported. Because 5.0.X requires VS 2019 and VS 2019 is no longer installed on my development PC. Sorry for that, but the older versions of the 5.0.x plugin are still available.
- [break change] :pill: `ASRCaptureComponent` change name to `SpeechCaptureComponent`;
- [upgrade]: :placard: Upgrade Windows SDK to 10.0.22621 (Windows 11 SDK)
- [upgrade]: :placard: Upgrade VC Runtime to 14.38.33130
- [other]: :placard: [Demo project](https://github.com/endink/MediaPipe4U-Demo ) upgrade to UE5.4, add ControlRig Demo, use [Windwalker Echo](https://www.unrealengine.com/marketplace/en-US/product/windwalker-echo-01) character instead `Kit boy` for facial capture demo.

# Known Issue:
- Google's new holistic task API doesn't support GPUs, so Android uses CPU inference, and Windows, as always, CPU inference.
- Sine Unreal Engine 5.4 , the 'OpenCV' engine built-in plugin no longer contains precompiled libraries (dll files), the first time of the engine launched, because M4U dependent on the 'OpenCV' plugin, the UE Editor will try to download the `OpenCV` source code over the internet and compile it, this process may take a long time, it feels like the UE loading is stuck at 75%, please be patient and wait it, for Chinese users, you may also need a VPN networking for that, you can do this manually: [#166](https://github.com/endink/Mediapipe4u-plugin/discussions/166).
- Sine Unreal Engine 5.4 , the `OpenCV` engine built-in plugin no longer contains precompiled libraries (dll files), the first time of the engine launched, because M4U dependent on the `OpenCV` plugin, the UE Editor will try to download the `OpenCV` source code over the internet and compile it, this process may take a long time, it feels like the UE loading is stuck at 75%, please be patient and wait it, for Chinese users, you may also need a VPN networking for that, you can do this manually: [#166](https://github.com/endink/Mediapipe4u-plugin/discussions/166).


## Release Notes
Expand Down
18 changes: 14 additions & 4 deletions README_CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
<p align="center">
<a href=""><img src="https://img.shields.io/badge/Windows-Win 10/11 X64-brightgreen.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Android->=8.0 (Arm64)-brightgreen.svg"></a>
<a href=""><img src="https://img.shields.io/badge/VC++ Runtime->=14.36.32532-aff.svg"></a>
<a href=""><img src="https://img.shields.io/badge/MSVC->=14.38.33130-aff.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Windows SDK->=10.0.22621-orange.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Visual Sutdio->=2022.17.6.3-blue"></a>
</p>
Expand All @@ -41,15 +41,25 @@

# 最新动态

最后更新:`20240520`
最后更新:`20240523`

- [new] :rainbow: 在编辑器中使用 MediaPipe4U 不再需要授权(即使授权文件过期,依然可以在编辑器中使用)。
- [new] :rainbow: Unreal Engine **5.4** 版本支持,我跳过了 5.3 版本,精力有限,实在抱歉。
- [new] :rainbow: 加入 **Control Rig** 支持 (预览质量). 新的动画蓝图节点 `ControlRigFromPose` 已经可用,可以将任意姿态转换到 ControlRig 蓝图中,方便使用 ControlRig 制作关节限制、全身 IK 等。
- [new] :fire: 对齐 google mediapipe 到最新版本,使用全新的 holistic task api, 整体追踪方案性能进一步提升,表情算解效果提升。
- [fix] #130: 当打包游戏时,mediapipe 模型文件丢失.
- [improve] 安卓平台性能优化, 现在,M4U 在安卓手机上可以跑到 **15**-**20** fps (CPU: 高通骁龙 870 ).
- [new] :rainbow: `MediaPipeHolisticComponent` 中加入 `StopAsync` 函数以支持异步停止 mediapipe.
- [new] :rainbow: Android 平台的 `MediaPipeHolisticComponent` 组件支持 "Raw 模式" 显示帧图像.
- [fix] :bug:[#130](https://github.com/endink/Mediapipe4u-plugin/issues/130): 当打包游戏时,mediapipe 模型文件丢失.
- [fix] :bug:[#124](https://github.com/endink/Mediapipe4u-plugin/issues/124): 当动画中的 CSPose 没有使用骨架(skeleton)的骨骼索引时,BVH Recoder 导致程序崩溃.
- [fix] :bug: `AnchorWidgetInCanvas` 函数缩放帧图像时大小超出限制.
- [improve] :rose: 安卓平台性能优化, 现在,M4U 在安卓手机上可以跑到 **15**-**20** fps (CPU: 高通骁龙 870 ).
- [improve] :rose: 算解器能够适应 LOD 变化,LOD 变化时依然能够求解姿态并进行动作捕捉.
- [remove] :nauseated_face: LLM 插件被移除. 由于大语言模型(LLM)本地推理需要苛刻的硬件资源,加之 llama.cpp 长期高频发布,没有稳定版本,暂时移除这个插件,或许将来还会回归。
- [remove] :nauseated_face: 从这个版本开始,5.0.x 的 Unreal Engine 版本将不再支持. 因为 5.0.X 需要 VS 2019 才能编译,我的开发机中已经不再安装 VS 2019, 非常抱歉 !旧版的 5.0.x 插件任然可用。
- [break change] :pill: `ASRCaptureComponent` 更名为 `SpeechCaptureComponent`;
- [upgrade]: :placard: Windows SDK 升级到 10.0.22621 (Windows 11 SDK)
- [upgrade]: :placard: Upgrade VC Runtime to 14.38.33130
- [other]: :placard: [Demo project](https://github.com/endink/MediaPipe4U-Demo ) 升级到 UE5.4, 添加 ControlRig Demo. 使用 [灵风女神厄科](https://www.unrealengine.com/marketplace/en-US/product/windwalker-echo-01) 角色代替“风筝男孩”作为表情捕捉的演示.

# 已知问题
- Google 新版 holistic task api 没有支持 GPU,所以安卓平台使用 CPU 推理,Windows 一如既往,使用 CPU 推理。
Expand Down
17 changes: 11 additions & 6 deletions docs/Install/requirement.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,16 +48,20 @@ MediaPipe4U 目前支持 Windows x64 和 Android arm-64 平台,但是功能支
|软件|版本|
|--------|--------------|
|操作系统|Windows10/11 x64|
|Unreal Engine |5.0.x, 5.1.x, 5.2.x|
|Visual Studio 2022 | 17.6.3 or later|
|Windows SDK|10.0.22621 or later|
|Visual C++ Redistributable|14.36.32532 or later|
|Unreal Engine |5.1.x, 5.2.x, 5.4.x|
|Visual Studio 2022 | 17.9.6 or later|
|Windows SDK| 10.0.22621 or later|
|MSVC |14.38.33130 or later|

{: .note}
> 通常 Visual Studio 2022 17.6.3 或更新的版本中包含了必要的 Visual C++ Redistributable
>
> 通常 Visual Studio 2022 17.9.6 或更新的版本中包含了必要的 MSVC 工具链
>
> 以 VS 2022 为例,可以在以下位置查看你的 MSVC 版本:
> `C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC`
>
> 仅支持 64 位 Windows,不支持 32 位 Windows。
**注意**:MSVC 14.39.X 和 UnrealEngine 5.4.1 可能存在冲突,建议使用 14.38.X 版本的 MSVC.

---

Expand All @@ -68,6 +72,7 @@ MediaPipe4U 目前支持 Windows x64 和 Android arm-64 平台,但是功能支
| 5.0 | r21e (21.4.7075529) | 26 or later |
| 5.1 | r25b (25.1.8937393) | 26 or later |
| 5.2 | r25b (25.1.8937393) | 26 or later |
| 5.4 | r25b (25.1.8937393) | 26 or later |

{: .note}
> 仅支持 Android ARM 64 (arm64-v8a) 平台,不支持其他安卓设备(例如 Android x86 )。
Expand Down
13 changes: 10 additions & 3 deletions docs/update_logs/en.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,24 @@ parent: 更新日志
[简体中文](./index.md)


### 2024-05-20
### 2024-05-23
{: .d-inline-block }
Current
{: .label .label-green }
- [new] :rainbow: Licenses is no longer required to use MediaPipe4U (all plugin) in the UE editor (you can still use M4U even though the license file has expireds).
- [new] :rainbow: Unreal Engine version **5.4** support, I skipped UE 5.3, I'm sorry for that.
- [new] :rainbow: **Control Rig** supported (preview). Add a new animation blueprint node `ControlRigFromPose` that used to convert FK poses to Control Rig Blueprints pose.
- [new] :fire: Upgrade to latest goole mediapipe holistic task api, got better performace for holistic tracking.
- [fix] [#130](https://github.com/endink/Mediapipe4u-plugin/issues/130): When packaging a game, the mediapipe model file is missing.
- [improve] Optimize android performace, now Android platfrom can reach **15**-**20** fps (CPU: Qualcomm Snapdragon 870 ).
- [new] :rainbow: Add `StopAsync` to `MediaPipeHolisticComponent` to support asynchronous stop Mediapipe.
- [new] :rainbow: `MediaPipeHolisticComponent` Display "Raw mode" support for Android platform.
- [fix] :bug: [#130](https://github.com/endink/Mediapipe4u-plugin/issues/130): When packaging a game, the mediapipe model file is missing.
- [fix] :bug: [#124](https://github.com/endink/Mediapipe4u-plugin/issues/124): `BVH Recorder` crash if skeleton bone index was not used for compact space pose.
- [fix] :bug: The `AnchorWidgetInCanvas` function exceeds the size limit when scaling the frame image.
- [improve] :rose: Optimize android performace, now Android platfrom can reach **15**-**20** fps (CPU: Qualcomm Snapdragon 870 ).
- [improve] :rose: The Mediapipe4U solver is now adaptive to LOD changes and motion capture can also be done when the LOD changes.
- [remove] :nauseated_face: LLM plugin has been removed. Because of the high latency of the LLM and the frequent updates of the llama.cpp project for a long time, there is no stable version. So I remove it, and maybe it will come back in the future.
- [remove] :nauseated_face: Since this version, the 5.0.x version of UnrealEngine is no longer supported. Because 5.0.X requires VS 2019 and VS 2019 is no longer installed on my development PC. Sorry for that, but the older versions of the 5.0.x plugin are still available.
- [break change] :pill: `ASRCaptureComponent` change name to `SpeechCaptureComponent`;

### 2023-10-10
- [new] :rainbow: **MediaPipe4U Mobile** Android packaging support, the development process is fully compatible with Windows.
Expand Down
13 changes: 10 additions & 3 deletions docs/update_logs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,24 @@ has_children: true
[English](./en.md)


### 2024-05-20
### 2024-05-23
{: .d-inline-block }
Current
{: .label .label-green }
- [new] :rainbow: 在编辑器中使用 MediaPipe4U 不再需要授权(即使授权文件过期,依然可以在编辑器中使用)。
- [new] :rainbow: Unreal Engine **5.4** 版本支持,我跳过了 5.3 版本,精力有限,实在抱歉。
- [new] :rainbow: 加入 **Control Rig** 支持 (预览质量). 新的动画蓝图节点 `ControlRigFromPose` 已经可用,可以将任意姿态转换到 ControlRig 蓝图中,方便使用 ControlRig 制作关节限制、全身 IK 等。
- [new] :fire: 对齐 google mediapipe 到最新版本,使用全新的 holistic task api, 整体追踪方案性能进一步提升,表情算解效果提升。
- [fix] [#130](https://github.com/endink/Mediapipe4u-plugin/issues/130): 当打包游戏时,mediapipe 模型文件丢失.
- [improve] 安卓平台性能优化, 现在,M4U 在安卓手机上可以跑到 **15**-**20** fps (CPU: 高通骁龙 870 ).
- [new] :rainbow: `MediaPipeHolisticComponent` 中加入 `StopAsync` 函数以支持异步停止 mediapipe.
- [new] :rainbow: Android 平台的 `MediaPipeHolisticComponent` 组件支持 "Raw 模式" 显示帧图像.
- [fix] :bug: [#130](https://github.com/endink/Mediapipe4u-plugin/issues/130): 当打包游戏时,mediapipe 模型文件丢失.
- [fix] :bug: [#124](https://github.com/endink/Mediapipe4u-plugin/issues/124): 当动画中的 CSPose 没有使用骨架(skeleton)的骨骼索引时,BVH Recoder 导致程序崩溃.
- [fix] :bug: `AnchorWidgetInCanvas` 函数缩放帧图像时大小超出限制.
- [improve] :rose: 安卓平台性能优化, 现在,M4U 在安卓手机上可以跑到 **15**-**20** fps (CPU: 高通骁龙 870 ).
- [improve] :rose: 算解器能够适应 LOD 变化,LOD 变化时依然能够求解姿态并进行动作捕捉.
- [remove] :nauseated_face: LLM 插件被移除. 由于大语言模型(LLM)本地推理需要苛刻的硬件资源,加之 llama.cpp 长期高频发布,没有稳定版本,暂时移除这个插件,或许将来还会回归。
- [remove] :nauseated_face: 从这个版本开始,5.0.x 的 Unreal Engine 版本将不再支持. 因为 5.0.X 需要 VS 2019 才能编译,我的开发机中已经不再安装 VS 2019, 非常抱歉 !旧版的 5.0.x 插件任然可用。
- [break change] :pill: `ASRCaptureComponent` 更名为 `SpeechCaptureComponent`;

### 2023-10-10
- [new] :rainbow: **MediaPipe4U Mobile** 安卓打包编译支持, 开发体验和 Windows 版本保持一致.
Expand Down

0 comments on commit a2f5cad

Please sign in to comment.