support RTL languages in axmol

[Arvant]

Hi.
Godot in new version(4) added Harfbuzz to engine core and now support RTL languages.
godotengine/godot#41100
https://godotengine.org/article/complex-text-layouts-progress-report-1/
I use reshaper for this problem but it not work well and the best way to do it is use Harfbuzz in Core of axmol.
have you any plan to do it?
@halx99
@rh101

[rh101]

@Arvant I'm just a contributor to the engine code, and not involved in planning core features.

Regardless of whether or not it is something planned in the engine, you can always have a go at implementing it yourself and submitting a PR to add that functionality.

[DelinWorks]

Axmol does support RTL, you can use label->setHorizontalAlignment(TextHAlignment::RIGHT); to toggle RTL, I use this with my reshaper and the lines are read from right to left (instead of left to right), also your reshaper has to deal with clamping, if you break lines the text will switch from RTL to LTR and that's the nature of it, don't use setDimenstion and clamping overflow with your labels when using RTL text, just implement a function to reverse the order of words when a line break occures. an example of this function.

unfortunately as with any engine in life that supports RTL languages some work has to be done on the CPU to shape those characters, I noticed in games where I select arabic language my performance degrades a bit when viewing arabic UI, so if you implement your reshaper or harfbuzz you should expect some performance loss. other than that you can use this useful site https://www.compart.com/en/unicode/, it will help immensely when building your own reshaper, just look at your languages character set and see which characters interconnect and what not, as for harfbuzz it's a bit complicated, I'd advice you to try to get it to work in a standalone c++ project and then try to glue it with axmol as an extension.

[armanhossiny]

half space Farsi

axmol(usingShapingEngine.hpp) :

cocos2dx v4(with out other class, only add one line code to label, but not working axmol):

Could I explain well?

[armanhossiny]

this half space

[armanhossiny]

I add this lines to shapingEngine.h
this all alphabet Arabic And Farsi

// ****** SPACES ARE IMPORTANT! DO NOT REMOVE!! ****** //
//glyphs.insert({ L'\u0621', Glyph(L" \uFE80") }); //1 HAMZA ء []
//glyphs.insert({ L'\u0622', Glyph(L" \uFE82") }); //2 ALEF_MADDA آ []
//glyphs.insert({ L'\u0623', Glyph(L" \uFE84") }); //3 ALEF_HAMZA_ABOVE أ []
//glyphs.insert({ L'\u0624', Glyph(L" \uFE86") }); //4 WAW_HAMZA ؤ []
//glyphs.insert({ L'\u0625', Glyph(L" \uFE88") }); //5 ALEF_HAMZA_BELOW إ []
//glyphs.insert({ L'\u0626', Glyph(L"\uFE8B\uFE8C\uFE8A") }); //6 YEH_HAMZA ئ []
//glyphs.insert({ L'\u0627', Glyph(L" \uFE8E") }); //7 ALEF ا []
//glyphs.insert({ L'\u0628', Glyph(L"\uFE91\uFE92\uFE90") }); //8 BEH ب
//glyphs.insert({ L'\u0629', Glyph(L"\uFE93\uFE94\uFE94") }); //9 TEH_MARBUTA ة []
//glyphs.insert({ L'\u062A', Glyph(L"\uFE97\uFE98\uFE96") }); //10 TEH ت
//glyphs.insert({ L'\u062B', Glyph(L"\uFE9B\uFE9C\uFE9A") }); //11 THEH ث
//glyphs.insert({ L'\u062C', Glyph(L"\uFE9F\uFEA0\uFE9E") }); //12 JEEM ج
//glyphs.insert({ L'\u062D', Glyph(L"\uFEA3\uFEA4\uFEA2") }); //13 HAH ح
//glyphs.insert({ L'\u062E', Glyph(L"\uFEA7\uFEA8\uFEA6") }); //14 KHAH خ
//glyphs.insert({ L'\u062F', Glyph(L"\uFEA9\uFEAA\uFEAA") }); //15 DAL د []
//glyphs.insert({ L'\u0630', Glyph(L"\uFEAB\uFEAC\uFEAC") }); //16 THAL ذ []
//glyphs.insert({ L'\u0631', Glyph(L"\uFEAD\uFEAE\uFEAE") }); //17 REH ر []
//glyphs.insert({ L'\u0632', Glyph(L"\uFEAF\uFEB0\uFEB0") }); //18 ZAIN ز []
glyphs.insert({ L'\u0698', Glyph(L" \uFB8B") }); //19 ZHEH ژ []
//glyphs.insert({ L'\u0633', Glyph(L"\uFEB3\uFEB4\uFEB2") }); //20 SEEN
//glyphs.insert({ L'\u0634', Glyph(L"\uFEB7\uFEB8\uFEB6") }); //21 SHEEN
//glyphs.insert({ L'\u0635', Glyph(L"\uFEBB\uFEBC\uFEBA") }); //22 SAD ص
//glyphs.insert({ L'\u0636', Glyph(L"\uFEBF\uFEC0\uFEBE") }); //23 DAD ض
//glyphs.insert({ L'\u0637', Glyph(L"\uFEC3\uFEC4\uFEC2") }); //24 TAH ط
//glyphs.insert({ L'\u0638', Glyph(L"\uFEC7\uFEC8\uFEC6") }); //25 ZAH ظ
//glyphs.insert({ L'\u0639', Glyph(L"\uFECB\uFECC\uFECA") }); //26 AIN ع
//glyphs.insert({ L'\u063A', Glyph(L"\uFECF\uFED0\uFECE") }); //27 GHAIN غ
//glyphs.insert({ L'\u0640', Glyph(L"\u0640\u0640\u0640") }); //28 TATWEEL ـ
//glyphs.insert({ L'\u0641', Glyph(L"\uFED3\uFED4\uFED2") }); //29 FEH ف
//glyphs.insert({ L'\u0642', Glyph(L"\uFED7\uFED8\uFED6") }); //30 QAF ق
//glyphs.insert({ L'\u0643', Glyph(L"\uFEDB\uFEDC\uFEDA") }); //31 KAF Arabic ك
//glyphs.insert({ L'\u0644', Glyph(L"\uFEDF\uFEE0\uFEDE") }); //32 LAM ل
//glyphs.insert({ L'\u0645', Glyph(L"\uFEE3\uFEE4\uFEE2") }); //33 MEEM م
glyphs.insert({ L'\u0646', Glyph(L"\uFEE7\uFEE8\uFEE6") }); //34 NOON ن
glyphs.insert({ L'\u0647', Glyph(L"\uFEEB\uFEEC\uFEEA") }); //35 HEH ه
//glyphs.insert({ L'\u0648', Glyph(L"\uFEED\uFEEE\uFEEE") }); //36 WAW و []
//glyphs.insert({ L'\u0649', Glyph(L"\uFEEF\uFEF0\uFEF0") }); //37 ALEF_MAKSURA [*]
//glyphs.insert({ L'\u064A', Glyph(L"\uFEF3\uFEF4\uFEF2") }); //38 YEH Arabic ي
glyphs.insert({ L'\u06CC', Glyph(L"\uFBFE\uFBFF\uFBFD") }); //39 YEH Farsi ی
glyphs.insert({ L'\u0686', Glyph(L"\uFB7C\uFB7D\uFB7B") }); //40 CHEH چ
glyphs.insert({ L'\u067E', Glyph(L"\uFB58\uFB59\uFB57") }); //41 Peh پ
glyphs.insert({ L'\u06AF', Glyph(L"\uFB94\uFB95\uFB93") }); //42 Gaf گ
glyphs.insert({ L'\u06A9', Glyph(L"\uFB90\uFB91\uFB8F") }); //43 Kaf ک
//glyphs.insert({ L'\u0020' , Glyph(L"\u0020\u0020\u0020") }); //44 Space
//glyphs.insert({ L'\u060C' , Glyph(L"\u060C\u060C\u060C") }); //45 Kama
glyphs.insert({ L'\u200C' , Glyph(L"\u200C \u200C")}); //46 half-space
//glyphs.insert({ L'\u0646 ' , Glyph(L"\u0646\u0646\u0646") }); //46 half-space
//glyphs.insert({ L'\u0627 ' , Glyph(L"\u0627\u0627\u0627") }); //46 half-space
//glyphs.insert({ L'\u0645' , Glyph(L"\u0645\u0645\u0645") }); //46 half-space
//glyphs.insert({ L'\u0647' , Glyph(L"\u0647\u0647\u0647") }); //46 half-space
//glyphs.insert({ L'\u0627 ' , Glyph(L"\u0627\u0627\u0627") }); //46 half-space
//glyphs.insert({ L'\u06CC' , Glyph(L"\u06CC\u06CC\u06CC") }); //46 half-space
//glyphs.insert({ L'\u2061' , Glyph(L"\u2061\u2061\u2061") }); //46 half-space
//glyphs.insert({ L'\u2062' , Glyph(L"\u2062\u2062\u2062") }); //46 half-space
//glyphs.insert({ L'\u2063' , Glyph(L"\u2063\u2063\u2063") }); //46 half-space
//glyphs.insert({ L'\u003A' , Glyph(L"\u003A\u003A\u003A") }); //47 :
//L"U200b"; , +U200c, +U200d, +Ufeff, +U200e, +U200f, +U2060, +U2061, +U2062, +U2063."
//glyphs.insert({ L'\u061B', Glyph(L"\u061B\u061B\u061B") }); //48 ؛
//glyphs.insert({ L'\u002E', Glyph(L"\u002E\u002E\u002E") }); //49 .
//glyphs.insert({ L'\u061F', Glyph(L"\u061F\u061F\u061F") }); //50 ؟
//glyphs.insert({ L'\u06F0', Glyph(L"\u06F0\u06F0\u06F0") }); //51 0
//glyphs.insert({ L'\u06F1', Glyph(L"\u06F1\u06F1\u06F1") }); //52 1
//glyphs.insert({ L'\u06F2', Glyph(L"\u06F2\u06F2\u06F2") }); //53 2
//glyphs.insert({ L'\u06F3', Glyph(L"\u06F3\u06F3\u06F3") }); //54 3
//glyphs.insert({ L'\u06F4', Glyph(L"\u06F4\u06F4\u06F4") }); //55 4
//glyphs.insert({ L'\u06F5', Glyph(L"\u06F5\u06F5\u06F5") }); //56 5
//glyphs.insert({ L'\u06F6', Glyph(L"\u06F6\u06F6\u06F6") }); //57 6
//glyphs.insert({ L'\u06F7', Glyph(L"\u06F7\u06F7\u06F7") }); //58 7
//glyphs.insert({ L'\u06F8', Glyph(L"\u06F8\u06F8\u06F8") }); //59 8
//glyphs.insert({ L'\u06F9', Glyph(L"\u06F9\u06F9\u06F9") }); //60 9
//glyphs.insert({ L'\u0028', Glyph(L"\u0028\u0028\u0028") }); //61 (
//glyphs.insert({ L'\u0029', Glyph(L"\u0029\u0029\u0029") }); //62 )

		glyphs.insert({ L'\u0640', Glyph(L"\u0640 \u0640") }); // ـ
		glyphs.insert({ L'\u0627', Glyph(L"  \uFE8E") }); // ا
		glyphs.insert({ L'\u0623', Glyph(L"  \uFE84") }); // أ
		glyphs.insert({ L'\u0625', Glyph(L"  \uFE88") }); // إ
		glyphs.insert({ L'\u0622', Glyph(L"  \uFE82") }); // آ
		glyphs.insert({ L'\u0628', Glyph(L"\uFE91\uFE92\uFE90") }); // ب
		glyphs.insert({ L'\u062A', Glyph(L"\uFE97\uFE98\uFE96") }); // ت
		glyphs.insert({ L'\u062B', Glyph(L"\uFE9B\uFE9C\uFE9A") }); // ث
		glyphs.insert({ L'\u062C', Glyph(L"\uFE9F\uFEA0\uFE9E") }); // ج
		glyphs.insert({ L'\u062D', Glyph(L"\uFEA3\uFEA4\uFEA2") }); // ح
		glyphs.insert({ L'\u062E', Glyph(L"\uFEA7\uFEA8\uFEA6") }); // خ
		glyphs.insert({ L'\u062F', Glyph(L"  \uFEAA") }); // د
		glyphs.insert({ L'\u0630', Glyph(L"  \uFEAC") }); // ذ
		glyphs.insert({ L'\u0631', Glyph(L"  \uFEAE") }); // ر
		glyphs.insert({ L'\u0632', Glyph(L"  \uFEB0") }); // ز
		glyphs.insert({ L'\u0633', Glyph(L"\uFEB3\uFEB4\uFEB2") }); // س
		glyphs.insert({ L'\u0634', Glyph(L"\uFEB7\uFEB8\uFEB6") }); // ش
		glyphs.insert({ L'\u0635', Glyph(L"\uFEBB\uFEBC\uFEBA") }); // ص
		glyphs.insert({ L'\u0636', Glyph(L"\uFEBF\uFEC0\uFEBE") }); // ض
		glyphs.insert({ L'\u0637', Glyph(L"\uFEC3\uFEC4\uFEC2") }); // ط
		glyphs.insert({ L'\u0638', Glyph(L"\uFEC7\uFEC8\uFEC6") }); // ظ
		glyphs.insert({ L'\u0639', Glyph(L"\uFECB\uFECC\uFECA") }); // ع
		glyphs.insert({ L'\u063A', Glyph(L"\uFECF\uFED0\uFECE") }); // غ
		glyphs.insert({ L'\u0641', Glyph(L"\uFED3\uFED4\uFED2") }); // ف
		glyphs.insert({ L'\u0642', Glyph(L"\uFED7\uFED8\uFED6") }); // ق
		glyphs.insert({ L'\u0643', Glyph(L"\uFEDB\uFEDC\uFEDA") }); // ك
		glyphs.insert({ L'\u0644', Glyph(L"\uFEDF\uFEE0\uFEDE") }); // ل
		glyphs.insert({ L'\u0645', Glyph(L"\uFEE3\uFEE4\uFEE2") }); // م
		glyphs.insert({ L'\u0646', Glyph(L"\uFEE7\uFEE8\uFEE6") }); // ن
		glyphs.insert({ L'\u0647', Glyph(L"\uFEEB\uFEEC\uFEEA") }); // ه
		glyphs.insert({ L'\u0648', Glyph(L"  \uFEEE") }); // و
		glyphs.insert({ L'\u064A', Glyph(L"\uFEF3\uFEF4\uFEF2") }); // ي
		glyphs.insert({ L'\u0626', Glyph(L"\uFE8B\uFE8C\uFE8A") }); // ئ
		glyphs.insert({ L'\u0649', Glyph(L"\u0649\u0649\uFEF0") }); // ى
		glyphs.insert({ L'\u0624', Glyph(L"  \uFE86") }); // ؤ
		glyphs.insert({ L'\u0629', Glyph(L"  \uFE94") }); // ة
		glyphs.insert({ L'\uFEFB', Glyph(L"  \uFEFC") }); // لا
		glyphs.insert({ L'\uFEF7', Glyph(L"  \uFEF8") }); // لأ
		glyphs.insert({ L'\uFEF9', Glyph(L"  \uFEFA") }); // لإ
		glyphs.insert({ L'\uFEF5', Glyph(L"  \uFED6") }); // لآ

[DelinWorks]

with ShapingEngine.hpp on my repo without any changes, u200C behaves normally..

Also u200C doesn't need to be added to glyphs since it's not a shaping or "shape"able character, it's rather a symbol so it would need to be treated accordingly using render with symbols, here
#define RENDER_SYMBOLS L".,<>(){}[]~!@#$%^&*?\"':;\\"
just add your half space as character in there and see if it works. you need to set render with symbols on,
#define RENDER_SYMBOLS L".,<>(){}[]~!@#$%^&*?\"':;\\\u200C"

you can make u200C shape letters for you but that's not practical, you could use ـ instead, unicode is u0640

without half-space L"اختبار"

with half-space L"ار\u200Cاختب" (I had to change the order since I didn't want to use render with symbols)

with full-space L"اختب ار"

Could you give me your font to check what's wrong?

[armanhossiny]

I use this font

https://fonts.google.com/specimen/Lalezar

and test this way, but not working
#define RENDER_SYMBOLS L".,<>(){}[]~!@#$%^&*?"':;\\u200C"

Tnx, again

[armanhossiny]

test your way

[armanhossiny]

and use this function:

ShapingEngine::render(wstr, true);

[DelinWorks]

can you list utf16 characters that have or might have this bug other than u200C?

[DelinWorks]

also how are you loading the fonts in both cocos2dx and axmol?

[armanhossiny]

using font in cocos2dx and axmol:
#define fntLalezar "fonts/Lalezar.ttf"

and use this define

[armanhossiny]

cocos2dx v4 not support farsi and I edit Class Buttom, label and ...:

this work in Windows and Android

but this not working in axmol

[DelinWorks]

@armanhossiny ok Fixed. I've made some changes in arabic-text-renderer-cpp-axmol, you can recopy the file.

It was because when using TTF fonts, a square is rendered instead of disregarding the zero width character itself which in this case is u200C, it can easily be fixed in axmol itself but it would cost performance since for every string change you would check for zero width characters, and given that the problem frustrates a few, it can just be a simple workaround to discard the zero width characters in the shaping engine itself from the beginning, the option _discardChars can be set true when you're certain that you'll use characters present in DISCARD_CHARS define, this is again for performance reasons.

ShapingEngine::Options::_discardChars = true;
auto text = std::wstring(L"اختب\u200Cار");
auto label = Label::createWithTTF(ShapingEngine::render(text, true), "bitsy-font-with-arabic.ttf", 48);

Keep in mind that the discard step is some kind of a "post-process" step, after the engine shapes the characters it will remove those characters set for discarding, and the shaping information of the characters left isn't lost and still retained. hence the reason I got the characters to separate with no u200C present.

[armanhossiny]

thank you very much, fixed :))))

[armanhossiny]

hi DelinWorks

when use mixed languages (english and arabic or farsi) don't show correct, e.x:

سلام و علیکم Arman

[DelinWorks]

About the 10 I will check now

[DelinWorks]

okay fixed, you can pull the latest changes.

auto label = Label::createWithTTF(ShapingEngine::render(text, false, true), "Lalezar-Regular.ttf", 48);
I've added a third parameter to the render functions and it's correct_num_order which is self explanatory.

works with arabic too

Also if you don't mind star the project if you found it useful, Thanks :)

[armanhossiny]

thank you dear DelinWorks fixed combine number and text and star to project :), really useful for my project

but combine Arabic and English not working

After the Farsi or Arabic text when the English text comes, in the rendering, the English text will appear first, then the Persian text will appear.

this correct image

but this not correct(always English first line , I want English end line)

[DelinWorks]

Sorry for late response @armanhossiny but can you give me the literal text you put in Visual Studio? I need the same order string since it matters

[DelinWorks]

Also can you open an issue in https://github.com/DelinWorks/arabic-text-renderer-cpp-axmol since this discussion doesn't have to do with anything in axmol, but rather an extension of axmol