Embed a lot of assets on native

Hey, trying to embed 220 Mb of assets on native, Haxe 3.4.3 crashes out of memory, Haxe 4.0-preview takes forever (30+ min) to compile. On flash the whole compilation takes about a minute (and I thought it was slow!). What can be done?

OpenFL 6.2.2, Lime 5.7.1

Perhaps a faster/better solution would be to look at writing multiple assets to one file (some sort of *.pak or *.zip file) then reading the byte positions out of the larger asset file. Our “asset library” system would definitely be able to support this functionality, this has long been a desired feature to be automated in our tools :slight_smile:

That would be the solution! In the meantime, let’s say I manually zip the assets, and read the archive into the memory on run, what it takes to hook into all Assets.get* calls to read from it? (I use HaxeFlixel)

Extend the AssetLibrary class, and override the various calls for exists and get*

After that’s done, use Assets.registerLibrary

1 Like

@singmajesty

So the approach of loading assets from a zip archive worked, thank you!
But I also want to do it in the background (with lime.app.Future), while showing some stuff, similar to preloader.
What worries me is thread-safety, particularly with code like:

var image = Image.fromBytes(unzip(file));
cachedImages.set(fileName, image);

var audioBuffer = AudioBuffer.fromBytes(unzip(file));
cachedAudioBuffers.set(fileName, audioBuffer);

Font.fromBytes(unzip(file))

where unzip() is

function unzip(file:Entry):ByteArray
{
    return ByteArray.fromBytes(Reader.unzip(file));
}

I got a few rare crashes during the loading, and after that I removed some my suspicious code from the second thread, so now it just loads and unzips data using Haxe’s standard API, but still fills the cache variables of AssetLibrary (the code above), so I’m not worried about overriding get*, and only override exists.

Or is it supposed to be done differently (e.g., only use *.fromBytes inside get* and never from a separate thread) to be thread-safe?

Thank you again!

Maybe you can use the new pak feature of new openfl 7? It should be easier.

Thanks, I have to look into it, but a bit later though!

So the main question persists.

Did you have any information about where the crash occurred?

No, I didn’t save the dumps, unfortunately :frowning:
Maybe it happened in my code now removed, my concern at the moment is a bit theoretical, because I saw in the source code of lime, that, for example, Image.fromBytes() calls Image’s new() which in turn does something with the renderer, or with the handle of the renderer.

Also, not with this issue, but in general, I had problems debugging lime from haxelib as part of my application in Visual Studio, because the debugging information wasn’t available, do I need to rebuild lime myself for debug symbols or need to do something else for it to work?

Yes, we’re at 96 MB for Haxelib releases, I think, and debug binaries would inflate this a lot more. Open to considering including debug binaries for Lime, otherwise you do need to clone from GIT (including submodules) and lime rebuild <target> to compile the binaries :slight_smile:

1 Like