Package SFML as a self contained dependency

[adinapoli]

It would be extremely cool, in the same vein of packages like hipmunk, to ship the complete self contained SFML + CSFML with this bindings, allowing end user to not bother with external dependencies.
Do you think it would be feasible?

[jeannekamikaze]

Sure. I suppose the description in the .cabal file should be changed then.

[adinapoli]

Brill!
Happy new year btw. I have been working on a small game using SFML + netwire and I'm almost ready to showcase it under the form on a blog post to attract attention on our bindings :)

A.

[jeannekamikaze]

Wait, for some reason I misread your initial post. I thought you wanted to merge SFML and SFML-Control into one big package, which is why I said the description in the .cabal file should then be changed, so as to explain what's going on. As far as my understanding goes, shipping SFML + CSFML is not an option. The reason is that CSFML is not C, it's just unmangled C++, and cabal cannot deal with C++ files, at least the last time I checked.

[jeannekamikaze]

I suppose we could get around with some Setup.hs hacking, compiling SFML and CSFML and then the bindings themselves. However, that would still require the user to previously install cmake, since it is needed to build SFML. I suppose it's doable, but do you think it's worth the hassle?

[adinapoli]

Hi :)
As a bit of an unrelated note, I think we are doing the right thing in keeping SFML and SFML-control separated. I'm using the latter in a small game experiment, and the performance overhead might be such to discourage its adoption for "serious" real time applications.

As regards the topic itself I'm just looking forward to the adoption of these bindings. Every application willing to use Haskell and SFML for game dev would need sooner or later to package the game as a standalone app, via static linking, and I'm not sure that our current setup will allow it (but I'm ignorant on the topic).
Furthermore, releasing the bindings + the bound library will allow us to version freeze sfml and csfml to a version we trust and know is working against the bindings.
That will be my main rationale :)

[jeannekamikaze]

Well, I have been modifying the .cabal file and added a Setup.hs to compile SFML and CSFML. Since static linking with CSFML is a no-go, I am faced with two problems:

1. Cabal's 'extra-lib-dirs' does not work with relative paths because there's no path to set a path relative to. Cabal suggests using ${pkgroot}, but I have no idea how to do that and the user guide does not even mention that variable.
2. If I hard-code full paths in extra-lib-dirs as a temporary solution, the demos compile but they don't run unless $LD_LIBRARY_PATH is overriden or the hard-coded paths to the SFML and CSFML libraries are added to $PATH. This also means that loading any code on ghci will royally fail. HSSFML should work on ghci.

Let's see if I find a way to get around these problems.

[adinapoli]

I still have to wrap my head around these problem, but as soon as I do I'll try to help with the troubleshooting. I hear you though, dealing with this stuff is always tricky :)

[jeannekamikaze]

The problem is that, no matter what you do, the user will ultimately have to copy the shared libs / dlls somewhere in their PATH. I can't do this from the build script because I don't know where to copy to. Plus, even though cabal creates its own .cabal directory, a vanilla installation of cabal / haskell platform does not set up any paths other than those for ghc and mingw on windows, which in any case would require root to touch.

I have come up with a mixed solution that would offer several ways to build the library:

1. Simply run 'cabal install SFML' off of hackage. The build script assumes that SFML/CSFML have already been installed, and that the include paths are available. If the latter is not true, one can still use --extra-include-dirs as mentioned in the installation manual. To compile the demos, one simply has to cabal install and then run as usual.
2. If the user does not have SFML/CSFML installed, or their version does not match ours, they can use cabal to build it. The user runs cabal unpack, then cabal install, and manually copies the shared libs that the build script will make available to somewhere in their PATH. To install the demos, a cabal install and then running as usual is enough.
3. The user manually compiles SFML/CSFML and then installs HSSFML by following the installation manual that is already available.

What do you think ?

Edit: of course, if we do it this way all those steps would be documented in the installation manual.

[adinapoli]

Hi Marc.
What if we choose a way more pragmatic and easy approach, namely enforcing upper and lower bounds for SFML and CSFML inside the Setup.hs file? This way we can at least make sure our bindings will target the correct versions of SFML/CSFML, and gives us freedom to experiment with more sophisticated approaches, whilst having at least a naive mechanism to enforce versioning. What you reckon? I'm generally open to any solution :)

[jeannekamikaze]

Hmm, how exactly do I check the version of an existing SFML installation ?

[adinapoli]

that's a very good question. Some possibilities on top of my mind:

a) We ask on the SFML forum for hints (parse the headers perhaps?)
b) We write a small C/haskell program that tries to use certain functionality of the library, and if it fails we assume the version is not correct. I don't know though how far we can get with this.
c) We generate a checksum for the installed library we want to target and we compare the results. If they diverge (e.g. we can ship the haskell package with precomputed MD5 hashes , one per platform) we abort the installation. Not sure how reliable this can be.

[jeannekamikaze]

Success! I have managed to statically link with SFML and CSFML on Linux. I had to edit some include files because cmake was creating libraries with mangled names. All the demos are working flawlessly, and the user's .cabal file need not worry about anything because SFML's .cabal file does everything for you. The only problem now is that one has to run "cabal copy" to get SFML/CSFML static libs copied over to cabal's installation directory. If I can find a way to hook the build process to do this automatically, then it'll be pretty much done.

[adinapoli]

Great job! Can I help with the Mac OS X side of things?

[jeannekamikaze]

Yeah, I can do linux and windows. I still have to figure out how to avoid the copy hook. I'll push the changes when I get everything working under linux.

[adinapoli]

fantastic, really happy of the outcome :)

[jeannekamikaze]

I have written a build hook that takes the C++ libs and the haskell lib, unpacks everything and repacks all the objects into one master static lib that overwrites the haskell static lib. Now, applications only have to link with the haskell SFML static lib, there are no C++ static libs lying around, and SFML code should work in ghci since everything is cointained inside the haskell lib. Plus, running 'cabal install' is now enough, no need to call "cabal copy" or anything like that.

Still have to fix some rough edges. I'll let you know when I'm done.

[adinapoli]

It sounds marvellous, looking forward to seeing the final code :)

[jeannekamikaze]

Ok, got everything working under Linux. My opinion is that HSSFML should allow two installation scenarios:

a) [default] The user runs 'cabal install'. HSSFML assumes that SFML is already installed in the system, like it does now.

b) The user passes a custom flag to 'cabal install'. This builds both HSSFML and the version of SFML that we ship with it.

The reason I wouldn't make b) the default is that it requires a c++ compiler and standard library, cmake, and all of SFML's dependencies: pthread, GL, GLU, GLEW, X11, Xrandr, Xi, openal, jpeg, freetype, and sndfile.

Also, I still have to think of a way to check version compatibility in case a).

What do you think?

[adinapoli]

Hi Marc,
I would be tempted to do b) as a default instead and here is why:
We are basically releasing the library with the implicit contract between us and the user that it will work as long as SFML and CSFML are to the correct version, and giving the freedom to use a custom version of SFML somewhat violates this contract.
Furthermore, as regards the dependencies introduced by option b), those dependencies would need to be installed anyway even with option a), because as far as I can tell having already a custom SFML version means those deps are already satisfied. In order words, is inevitable that if the user wants to use a graphic library he needs to install certain deps. I will tell you more, I would be surprised if the CMake script shipped with SFML doesn't install such deps for us, after all, is what a build system is made for.
We could then make b) the default and provide a flag like "--unsafe" to allow the installation of HSSFML with a custom version of SFML.

What you reckon?

[jeannekamikaze]

Ugh, right, you need those libraries anyway. Also, SFML's cmake does not install its own dependencies, you're expected to have them available.

I'll make b) the default and a) optional then. Let me test the build script on Windows and I'll push the changes.

Edit: some linux distributions (ubuntu) keep runtime libraries and header files in separate packages. For option b) you would also have to install the development packages (abc-dev), but I suppose it's still a non-issue.

[adinapoli]

It's a pretty standardised thing in linux to install the dev versions, yes :)

[jeannekamikaze]

What do you think about shipping pre-compiled libraries for windows? I'm finding that compiling SFML here is a bloody mess, especially if you have cygwin binaries and other stuff in your path. Additionally, SFML ships with pre-compiled DLLs for its dependencies, and those must be copied to the path, so just providing everything pre-compiled seems like a reasonable choice.

[adinapoli]

I would say; why we don't go further and we ship pre-compiled SFML for Mac OS X too? Probably we can't do that for Linux because distros might be slightly different (do they?) but when I think about "self-contained" I think "pre-compiled" :)

EDIT: Sorry, I think I misunderstood you. You meant pre-compiled HSSFML don't you?

[jeannekamikaze]

I meant pre-compiled SFML. How should we ship all of the dependencies then? They amount to a couple 50-60 MB.

[adinapoli]

very good point, let's think about that :)

[jeannekamikaze]

I just got this working on windows using precompiled DLLs for SFML and CSFML. The only issues at the moment are:

1. User must run 'cabal copy --destdir=somewhere/in/your/path' to copy the DLLs to somewhere where the system will find them. This is only required to run apps, not to compile them.
2. The precompiled DLLs amount to 110 MB because we must ship 32 and 64 bit binaries.

[adinapoli]

1. As long as it's properly documented, I don't think this is an issue.
2. I think is somewhat not avoidable; it might be a problem to have such a big package on hackage though.

This makes me wonder we should split things up, perhaps having an hierarchy of packages on hackage like:

• sfml-core (just the code)
• sfml-win32 (fat binaries for 32 bit)
• sfml-win64 (fat binaries for 64 bit)
• sfml-linux
• sfml-macos

and then use the appropriate package dependencies using ghc conditional flags, (ala c preprocessor directives) to
depend upon a particular version.

Example pseudocode:

build-depends:
sfml-core
#if DARWIN
sfml-mac
#endif

do you think it's feasible ?

[jeannekamikaze]

The problem is that depending on a particular package like sfml-win32 will only get it cabal installed, and the user must still run cabal copy. Also, having so many packages on hackage looks a bit messy.

I think that what we can do is a copy hook that wget's the DLLs from some online mirror. The copy hook would only download those libs matching your system. To deal with windows, we could ship a little wget.exe with HSSFML. What do you think?

[adinapoli]

uhm, this I think depends on different school of thoughts. From my experience, trying to have a core package with a series of backend packages is a very common practice, even in the Haskell world. To my knowledge, is very rare to fetch things from internet, also because wget is not standard (so is curl) on every system, and this will just complicate things further.
The main rationale behind the multi package approach is that you can bump core and, as long as you are not breaking the api, you can upload only the core package leaving the others untouched. Furthermore who is interested in developing just on linux, say, will have to fetch just two packages and few MB of data instead of tons of DLL he doesn't need.
This are my 2 cents, I guess we should ask for feedback to the Haskell community (#haskell or #haskell-game) before proceeding :)

[jeannekamikaze]

Fine by me. I just find it weird to have hackage packages devoted to storing DLLs, but if that's something conventional and they have no problems with hosting those kinds of packages then it is indeed the easiest solution.

The question, now, is: should we still compile SFML/CSFML from source on Linux? SFML/CSFML are available as precompiled shared libs for linux too. And if so, should we still have a flag to be able to decide whether to compile them or to link with the system (or hackage) libraries?

[adinapoli]

Yes, my preference would be to ship with each platform-specific package the precompiled libs, so that we are resiliant to api changes and everything is self contained.
Furthermore, as you rightly suggested, we should have a flag for them to say "I want to compile hsfml relying on my machine-installed library", something like -funsafe.
This way, the intrigued user can still try to download just sfml-core (a few kb) without any additional deps.

[jeannekamikaze]

Ok. I'll get my hands on it and let you know when I'm done.

[adinapoli]

Great, once you lay out a common pattern I can help with mac os x.

[jeannekamikaze]

I've been working on this and I think I may have something doable for Linux. Only problem is that the CSFML 2.3 precompiled libs are for some reason linked to SFML 2.2 instead of 2.3. Trying to clarify what's going on with SFML devs.

[DavidEichmann]

I've noticed the same issue. Doing an ldd /usr/lib/libcsfml-window.so.2.3 reveals that that CSFML is linked to 2.2!

$ ldd /usr/lib/libcsfml-window.so.2.3
	linux-vdso.so.1 =>  (0x00007ffd4f58a000)
	libsfml-window.so.2.2 => not found
	libsfml-system.so.2.2 => not found
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9d713ef000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9d711d9000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9d70e10000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9d70b06000)
	/lib64/ld-linux-x86-64.so.2 (0x000055555eb1b000)