Design and document a stable C API

[DoumanAsh]

For Juman++ to be widely usable, we want to have a documented and stable C API and an option to have a dynamically linked library.
That library probably should use -fvisibility=hidden and explicit visibility on exported symbols on Unixes and __declspec(dllimport/dllexport) on Windows.

The minimal API should be:

1. Loading a model using a config file
2. Analyzing a sentence
3. Accessing the analysis top-1, top-N and dictionary data

Additional things (low priority):

1. Access to the lattice?

There should not be any exceptions passing that API boundary because of C.

One problem remains: what to do with strings.
Dictionary strings in Juman++ are never zero terminated (they are length prefixed). Probably the API should provide non-zero terminated strings by default (exposing some StringPiece-like C struct) and an easy way to convert such strings to zero-terminated ones.

[eiennohito]

YES! I need to do this! Definitely!

But right now I invite you to look into the main launcher https://github.com/ku-nlp/jumanpp/blob/master/src/jumandic/main/jumanpp.cc

[DoumanAsh]

Yes that's good starting point, though currently as I'm on windows I cannot use jumanpp.
Cuz #31

Though it gave me idea to write some cross-platform mmap for C++

[eiennohito]

Yes, if you can work on that it should be wonderful.
The C++ itself code should be compilable with MSVC 2017 (I haven't tried), or definitely by Windows build of clang/mingw64.

There are 3 pieces of unix api I use, mmap (a wrapper is designed to be easily implemented for Windows though), posix_memalign (in util/memory, on Linux I use it with 2MB alignment and madvice to force transparent huge pages) and unlink in tests.

[eiennohito]

The other thing is, on Windows, filesystem access should use W versions of API (and convert paths from/to utf-8).

[eiennohito]

@DoumanAsh @kou what do you think.
I've reused the issue to be more related to what actually should be done.

[DoumanAsh]

@eiennohito Should it be then just API in general? After all there will be also C++ API? 🤔

[eiennohito]

C++ API won't have stability guarantees (I will try not to break it much though) and won't support dynamic linking/systemwide installation as a library (basically, the current status-quo).

[DoumanAsh]

Ah that's understandable of course.

I guess the first thing we'd need to try to take a look at existing public API and see how it is to be used.
I understand goal of mimicking mecab API, but i think it wouldn't harm to do own thing if it can be better API.

I've somewhat busy lately but I'll try to take a look at existing API later on and give some inputs/suggestion on it.

[kou]

C++ API won't support dynamic linking/systemwide installation as a library (basically, the current status-quo).

I'm a bit confused.
Does it mean that C API will support dynamic link but C++ API doesn't support dynamic link? If it's true, does C API use C++ API by static link?

[eiennohito]

Yes, C API will support consuming Juman++ as a dynamic object. I plan to build it with a bit different set of compile flags.

In theory it is possible to build even the current version Juman++ with -fPIC and link everything as dynamic libraries, and things will probably even work, but there could be a lot of not-very-fun problems with C++, dynamic linking, initialization order of globals and other runtime stuff.
E.g. exposing C++ classes over dll boundaries on Windows can lead to serious problems.

The second problem is symbol visibility. We don't want to export our version of Eigen and fmtlib to outside world, because that would confuse other software using their own version of Eigen. On Unixes we should use -fvisibility=hidden for that and explicitly define exported symbols.

C API library will cleanly solve these problems by providing a single shared object with only C API marked as visible and all C++ state contained inside that object.

[kou]

Thanks. I understand what you explain.

[kou]

I'll try creating a C API and send feedback to Juman++ in a few months.

[eiennohito]

I maybe start doing that earlier, but I'll post here when I will begin doing it.

[eiennohito]

Nothing about C API yet, but I am writing a small tutorial on "How to use Juman++ to create your own morphological analyzers".

https://github.com/eiennohito/jumanpp-t9

[kou]

Sorry. I couldn't work on this this year...