Garbled characters when using OpenAI models to embed Chinese

[ted-b]

Garbled characters when using OpenAI models to embed Chinese.
But using --transformer-model intfloat/multilingual-e5-base it's normal.
endoing utf-8
What's the promble?

[yych42]

Hmm weird that the title is showing up just right. Might be a problem with indexing? But not sure why E5 works.

如果你可以分享一下有问题的文件的话，我可以尝试一下复现 (I can try to replicate the problem if you can share the file)

[yych42]

I managed to confirm the issue. The problem seems to be in xxx.tokens.json, where "\ufffd" pops up a lot when using openai.

Imo openai is not that much superior than e5 when it comes to search in Chinese, so not sure if this should be fixed now or after there's a plan for v0.2---this will be up to @freedmand

[ted-b]

Thanks for your reply. Looking forward to V0.2 improving this issue.