Edit: If you’re interested in achieving this also, I’ve blogging my results. I have Windows working, and Linux too (with a wrapper to specify where Neko can be found). Thanks @Ibilon for all your help.
Edit #2:Nicolas seems to have accepted this (pull request) into the main Haxe builds. You should get this “for free” on Linux (without an additional wrapper script) in the next Lime version (I’m on 2.1.1, I think that’s the latest.)
Just like this thread about creating a complete standalone package for Windows, is it possible to create a complete stand-alone package for Neko?
I’m trying to create an application in Haxe which uses a plugin architecture. The problem is, it isn’t possible to a) load .n modules at runtime, and b) reflect and find all classes which implement my interface, outside of Neko.
While Haxe does build a complete Neko executable, that still requires end-users to have Neko installed. Is it possible to create a single package through Lime that also includes the Neko runtime?
I came across (and tried to use) the xCross library, but that seems to be a dead end – it doesn’t seem to play well with OpenFL binaries.
Does it work if you put the neko.so/dll in your app directory?
Looking at a generated executable from neko it’s the only dependency to the neko installation.
If that doesn’t work you may want to look into building a statically linked neko.
When you build for Neko, it does not require a Neko installation on your system, per se. The issue is that Neko is bad (in this case) at finding that it has it’s dependency libraries available in the same directory. That is why xcross works (no need to load an NDLL) but this is not good for Lime, since we not only use an NDLL ourselves (by default) but also need to support other NDLLs for extensions. I feel like it might be “close” to working without dependencies, but ultimately am unsure if we should focus on fixing Neko, or using another solution for cross-desktop builds (and fast builds that have a minimum of dependencies)
node.js, node-webkit and Java are three alternatives we could consider (in addition to “fixing Neko” as an option)
I am perplexed. This works with Windows, if I copy a couple of NDLL files over (std, regexp, and zlib). On Linux, I just get an error: error while loading shared libraries: libneko.so: cannot open shared object file: No such file or directory (even when libneko.so is in the executable directory).
Is this something that can/should be fixed? I don’t see much documentation about this.
@singmajesty none of those options have enough reflection support for me to load types from another compiled binary at runtime (see my original post’s link to plugin architecture). So for me, it’s Neko or nothing.
@ibilon it doesn’t work – I don’t know much C++, so I don’t think building a statically-linked DLL isn’t something I would try to do.
Oh, it’s possible that by default linux doesn’t look in the current directory, you’d have to do something like LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH ./myExe or in a bash file (with or without the .sh extension)
Don’t worry about nekotools boot, we already do the same effect when we create a Neko binary for you.
We need one of the following:
A version of Neko that correctly finds libraries located in the same directory
A version of Neko that is statically linked, without complicating the user build or preventing you from also using non-static NDLLs at runtime (ideally)
Xcross uses a version of #2 above, but it is not statically linked without our binary. Theoretically we could be a patch or two away from using Neko like this (its something I’ve wanted to see for a couple years) so could be done, we also have a build server setup, so building a modified version of Neko would not be as big of an issue, and it would not require the system Neko to be the same version if it’s truly standalone
@singmajesty I’m pretty new to this stack, and this is way over my head. If the Haxe contributors eventually submit a “fixed” version of Neko that does this, great. If not, @ibilon 's solution is what I’m going with.
@ashes999 best case scenario we successfully build a neko version that doesn’t require a .so/.dll update the neko version bundled in lime with this new one and next update you’ll just have to recompile.
I just downloaded neko’s source, I’ll see if the makefile can easily modified to do a static linked version.
Right now, I can build a self-standing Windows app, and Linux app (but one that runs with a shell script) with one code base. That’s good enough for me.
One other option is to embed Neko ourselves, I’m not sure if this helps resolve certain issues. Instead of using the precompiled Neko binaries, we actually put Neko inside of the Lime NDLL. Then we could have a small wrapper C++ application that’s precompiled, and just boot up Neko from there (and possibly resolve the paths properly and such)
@singmajesty There’s a discusson on the Haxelang mailing list about this. Andreas suggested using xcross. I tried it, against what comes out of Lime, but I get a runtime error.
Andreas Mokros wrote:
You’re right. I see that as well now. I have a neko executable and it can’t load libneko.so in the same.
In the same directory that should be.
There’s something wrong with loading libneko it seems.
It works for me with xcross. Did you try this? xcross creates excutables with libneko and some ndlls statically linked in.
I don’t know much about the OpenFL pipeline. Is it possible to integrate xcross into the intermediary output from neko? Is this an easier way to solve the problem, or is this already your conclusion (statically link)?
If you append a Neko binary file to the end of the Neko EXE, it will automatically run it. This is the difference between “neko myfile.n” and “myfile.exe”, like what “nekotools boot myfile.n” creates.
The Neko executable looks for a few dynamic libraries, libneko, std, etc. In the code for looking for these references, Neko expects to load them dynamically in another file.
However, in xcross, it’s changed to be built with libneko, std and other dependencies statically linked into the Neko executable. Instead of expecting to load an external file, it expects everything to already be available in the executable.
This is alright for standard Neko stuff, but we need it to load our Lime binary as well. Either it needs to be able to know how to find symbols both in itself (for libneko, etc) as well as externally, or we need to build Lime and link it with Neko ourselves.
Do you mean to say nekotools boot myfile.n creates a file which is your neko binary followed by your .n file?
Now, a different question: How does OpenFL use neko? Some points:
When I installed OpenFL, it put neko in /usr/bin/neko.
Andreas suggests this post holds the solution to fixing neko. It has something to do with the DT_RPATH feature. (I didn’t read or understand the post.)
He provided me with a custom build which includes his changes. (readelf -D neko shows RPATH in the list).
I think it’s the first of the two options you mentioned: A version of Neko that correctly finds libraries located in the same directory
I replaced /usr/bin/neko with his version. (First, I verified that removing it breaks lime; then, I put his over it.)
I deleted export and ran my app with lime test neko -clean. It builds and runs fine.
I ran readelf -D neko against my binary, and RPATH is not present.
Andreas said:
readelf for [your] game executable:
But this is not [correct]. The rpath stuff is missing there. I don’t know how OpenFL creates neko executables. I thought it simply uses nekotools boot. Maybe you can ask on a OpenFL forum.
If, for some reason, you’re interested in seeing his custom neko binary, I’ve uploaded it here. It’ll probably auto-delete after 7 days.
What can you tell us about how OpenFL uses neko to build the final executable? (I tried using Neko and Haxe in a trivial hello-world example (no real code, just a trace statement), and then building the executable with nekotools boot Hello.n, and it does show RPATH.)
Interesting post, it would indeed fix your problem and it’s easy to make such a version of neko (just need to add -Wl,-rpath,\$ORIGIN to the NEKOVM_FLAGS var of neko’s makefile)
/usr/bin/neko is used by haxelib, itself used by the lime/openfl command line tool.
But it uses its bundled version of neko to build executable and not nekoc.
Until @singmajesty update lime with a new linux neko you can replace /usr/lib/haxe/lib/lime/version/templates/neko/bin/neko-linux (or neko-linux64 depending of the arch) with Anderas’ version.
@ibilon it doesn’t seem to work. The new binary isn’t built with his neko executable, the one from the directory you mentioned.
Even if I delete .../templates/neko/bin, I can still build with lime. Only if I delete /usr/bin/neko, does lime test neko stop working – which is why I thought the /usr/bin/neko binary is the one lime used and the one I should replace. Even if lime is using the templates one to build, the build should fail if I delete it. It doesn’t.
At any rate, I replaced both, and I can confirm that the executable doesn’t have RPATH in it. Any thoughts on why?
Andreas mentioned that this works with (and I verified that it works with) nekotools boot blah.n. Is that how Lime builds it? I don’t know if this affects nekoc.
/usr/bin/neko is what “lime test” will use to run the tools
templates/neko/bin is what the Lime tools will use when creating your Neko-based binary. Similar to how “nekotools boot” works, Lime will combine the binary in that template directory with the compiled *.n file, so replacing the files in templates/neko/bin, then running “lime test neko”, accomplishes the same thing as “nekotools boot” would, but using your custom Neko binary
@singmajesty if that’s true, then why don’t I see RPATH when I replace both binaries? It seems like it’s not using /usr/bin/neko.
If templates/neko/bin is used to create my final binary, why do I still get a binary when I delete /templates/neko/bin, rm -rf assets and run lime test neko -clean? Shouldn’t I get an error, or a non-functional build at best?
To clarify again, Andreas’ fix works if you run:
haxe MainClass -neko out.n
nekotools boot out.n
./out
Is Andreas’ neko build useful for you to solve this problem?
