brb
September 30, 2015, 9:15am
#1
Hi all! I’m continuing my openfl 2.2.7 -> 3.3.6 migration and have run into a new memory issue. Shortly after the game starts for the first time, a series of tar files is downloaded, extracted and saved to disk (extraction and disk save are in a seperate thread). At some point during the download/extraction process a malloc error occurs:
CineMagic(36095,0x10f003000) malloc: *** error for object 0x1032451d8: incorrect checksum for freed object - object was probably modified after being freed.
Relevant backtrace:
* thread #15: tid = 0x452909, 0x00007fff8a126bc0 libsystem_malloc.dylib`malloc_error_break, stop reason = breakpoint 1.1
* frame #0: 0x00007fff8a126bc0 libsystem_malloc.dylib`malloc_error_break
frame #1: 0x00007fff8a1205c7 libsystem_malloc.dylib`szone_error + 386
frame #2: 0x00007fff8a124dfb libsystem_malloc.dylib`tiny_malloc_from_free_list + 359
frame #3: 0x00007fff8a1253c3 libsystem_malloc.dylib`szone_malloc_should_clear + 320
frame #4: 0x00007fff8a127868 libsystem_malloc.dylib`malloc_zone_malloc + 71
frame #5: 0x00007fff8a12827c libsystem_malloc.dylib`malloc + 42
frame #6: 0x00007fff8e00c36e libc++abi.dylib`operator new(unsigned long) + 30
frame #7: 0x0000000101033ed3 CineMagic`alloc_root + 115 at CFFI.cpp:844
frame #8: 0x000000010483394d lime.ndll`___lldb_unnamed_function1359$$lime.ndll + 61
frame #9: 0x0000000104814c6f lime.ndll`___lldb_unnamed_function612$$lime.ndll + 63
frame #10: 0x00000001048cbf44 lime.ndll`___lldb_unnamed_function3334$$lime.ndll + 484
frame #11: 0x00000001048d5736 lime.ndll`___lldb_unnamed_function3398$$lime.ndll + 1750
frame #12: 0x00000001048c38f0 lime.ndll`___lldb_unnamed_function3246$$lime.ndll + 2032
frame #13: 0x00000001048c3041 lime.ndll`___lldb_unnamed_function3245$$lime.ndll + 193
frame #14: 0x00000001048a89dc lime.ndll`___lldb_unnamed_function3046$$lime.ndll + 364
frame #15: 0x0000000104815377 lime.ndll`___lldb_unnamed_function627$$lime.ndll + 23
frame #16: 0x0000000101077570 CineMagic`ExternalPrimitive::__run(this=0x00000001036e9a18, a=0x000000010f254a40) + 464 at Lib.cpp:134
frame #17: 0x0000000100002bed CineMagic`Dynamic::operator(this=0x0000000101615f98, inArg0=0x000000010f254a40)(Dynamic const&) + 93 at Dynamic.h:187
frame #18: 0x0000000100452c08 CineMagic`lime::net::curl::CURLEasy_obj::perform(handle=4365368832) + 120 at CURLEasy.cpp:108
frame #19: 0x0000000100073000 CineMagic`openfl::net::URLLoader_obj::requestUrl(this=0x0000000111369660, _=<unavailable>)::_Function_1_1::run(Dynamic) + 224 at URLLoader.cpp:534
I’m currently working on a more specific repro, but have no luck so far. I’m running the mac target on OSX 10.9.5, with openfl 3.3.6, lime 2.6.6. I also experienced this problem on openfl 3.3.5 and lime 2.6.4. Here’s the gist of the backtrace .
Any and all suggestions would be appreciated!
Does you project using -Dlegacy to compile?
Would you also be interested in considering compiling Lime from the source? It is not difficult, but it will build debug versions of the lime.ndll file, so if you have a crash backtrace, it will tell us just what native methods were being called during the crash.
Thanks!
brb
October 1, 2015, 9:44pm
#3
I’m not attempting to use -Dlegacy at present, no. I’ve just rerun with lime compiled from source (with your latest commit to URLLoader
* thread #14: tid = 0x493671, 0x00007fff8df1b866 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGABRT
* frame #0: 0x00007fff8df1b866 libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x00007fff875d635c libsystem_pthread.dylib`pthread_kill + 92
frame #2: 0x00007fff8a90ab1a libsystem_c.dylib`abort + 125
frame #3: 0x00007fff8a120690 libsystem_malloc.dylib`szone_error + 587
frame #4: 0x00007fff8a124dfb libsystem_malloc.dylib`tiny_malloc_from_free_list + 359
frame #5: 0x00007fff8a1253c3 libsystem_malloc.dylib`szone_malloc_should_clear + 320
frame #6: 0x00007fff8a127868 libsystem_malloc.dylib`malloc_zone_malloc + 71
frame #7: 0x00007fff8a12827c libsystem_malloc.dylib`malloc + 42
frame #8: 0x00007fff8e00c36e libc++abi.dylib`operator new(unsigned long) + 30
frame #9: 0x0000000101036cb3 CineMagic`alloc_root + 115 at CFFI.cpp:844
frame #10: 0x000000010484aada lime.ndll`lime::Bytes::Resize(this=0x000000011fcd0330, size=1448) + 90 at Bytes.cpp:163
frame #11: 0x000000010484aa76 lime.ndll`lime::Bytes::Bytes(this=0x000000011fcd0330, size=1448) + 70 at Bytes.cpp:57
frame #12: 0x000000010484adbb lime.ndll`lime::Bytes::Bytes(this=0x000000011fcd0330, size=1448) + 27 at Bytes.cpp:59
frame #13: 0x000000010481b121 lime.ndll`lime::write_callback(ptr=0x00000001044488f0, size=1, nmemb=1448, userp=0x0000000103436dd0) + 97 at CURLBindings.cpp:178
frame #14: 0x000000010494d3d1 lime.ndll`Curl_client_write(conn=0x00000001038a4000, type=1, ptr=0x00000001044488f0, len=1448) + 481 at sendf.c:441
frame #15: 0x000000010495dcd8 lime.ndll`readwrite_data(data=0x0000000104448000, conn=0x00000001038a4000, k=0x0000000104448078, didwhat=0x000000011fcd0540, done=0x000000011fcd068b) + 3112 at transfer.c:731
frame #16: 0x000000010495cc42 lime.ndll`Curl_readwrite(conn=0x00000001038a4000, done=0x000000011fcd068b) + 354 at transfer.c:1050
frame #17: 0x000000010493f163 lime.ndll`multi_runsingle(multi=0x000000010353ba80, now=(tv_sec = 1443734813, tv_usec = 174331), data=0x0000000104448000) + 4435 at multi.c:1487
frame #18: 0x000000010493def3 lime.ndll`curl_multi_perform(multi_handle=0x000000010353ba80, running_handles=0x000000011fcd0798) + 243 at multi.c:1762
frame #19: 0x0000000104910d1b lime.ndll`easy_transfer(multi=0x000000010353ba80) + 395 at easy.c:709
frame #20: 0x000000010490fef6 lime.ndll`easy_perform(data=0x0000000104448000, events=false) + 358 at easy.c:797
frame #21: 0x000000010490fd8a lime.ndll`curl_easy_perform(easy=0x0000000104448000) + 26 at easy.c:816
frame #22: 0x000000010481a188 lime.ndll`lime::lime_curl_easy_perform(easy_handle=4366565376) + 24 at CURLBindings.cpp:138
frame #23: 0x000000010481b9cd lime.ndll`lime_curl_easy_perform__wrap(a0=(mValue = value = 0x0000000111658904)) + 29 at CURLBindings.cpp:597
frame #24: 0x000000010107a350 CineMagic`ExternalPrimitive::__run(this=0x00000001036e9abc, a=0x000000011fcd0a40) + 464 at Lib.cpp:134
frame #25: 0x0000000100002eed CineMagic`Dynamic::operator(this=0x00000001016195a0, inArg0=0x000000011fcd0a40)(Dynamic const&) + 93 at Dynamic.h:187
frame #26: 0x00000001004547f8 CineMagic`lime::net::curl::CURLEasy_obj::perform(handle=4366565376) + 120 at CURLEasy.cpp:108
frame #27: 0x0000000100073380 CineMagic`openfl::net::URLLoader_obj::requestUrl(this=0x0000000110e6877c, _=<unavailable>)::_Function_1_1::run(Dynamic) + 224 at URLLoader.cpp:534
frame #28: 0x0000000100073280 CineMagic`openfl::net::URLLoader_obj::requestUrl(this=0x0000000110e6877c, inArg0=0x000000011fcd0dd8)::_Function_1_1::__run(Dynamic const&) + 64 at URLLoader.cpp:540
Here are the links to 3 backtraces:
bt1 bt2 bt3
For sanity’s sake, I shut off the URLLoader heavy code and no crashes occurred.
As always, please let me know if I can send more info your way!
Do you know of some small test that I could run on my end that would crash as well?
brb
October 1, 2015, 9:52pm
#5
I don’t, unfortunately. I’l do my best to create a simple repro.
We could also try and trace what’s coming into the write callback, and see what values come in when there’s an error. For example, perhaps there’s a case where the length is 0 or less than zero that needs to be ignored
brb
October 1, 2015, 10:05pm
#7
Here are the relevant frame variables:
(lldb) fr s 11
frame #11: 0x000000010481b121 lime.ndll`lime::write_callback(ptr=0x0000000103a8eaf0, size=1, nmemb=1448, userp=0x000000010308a4a0) + 97 at CURLBindings.cpp:178
175
176 }
177
-> 178 Bytes bytes = Bytes (size * nmemb);
179 memcpy (bytes.Data (), ptr, size * nmemb);
180
181 return val_int (val_call3 (callback->get (), bytes.Value (), alloc_int (size), alloc_int (nmemb)));
(lldb) fr v
(void *) ptr = 0x0000000103a8eaf0
(size_t) size = 1
(size_t) nmemb = 1448
(void *) userp = 0x000000010308a4a0
(AutoGCRoot *) callback = 0x000000010308a4a0
(lime::Bytes) bytes = {
_data = 0x0000000000000000
_length = 0
_root = 0x0000000000000000
_value = 0x0000000110333fb4
}
(lldb)
brb
October 2, 2015, 4:21am
#8
I was able to construct a repro. Checkout the following repo:
The HEAD commit repros at a relatively low rate (but does crash if you wanna wait,) and is the simplest code. if you checkout crash:
git checkout crash
This commit crashes reliably (within a minute usually), though is more complicated code.
Happy hunting!
Thanks!
It was running along fine for me, but I noticed the memory use continued to climb. I can’t tell if this is a memory leak internally, or if the application itself is accumulating each download in memory?
It never crashed for me, but I presume that it crashes when it cannot allocate memory anymore?
brb
October 2, 2015, 5:45pm
#10
Hmm, this was with the “crash” commit?
Actually, checking here, I don’t see a “crash” branch. It must just be a local one
Maybe try git push -u origin crash?
brb
October 2, 2015, 6:52pm
#12
Oh sorry, it’s a tag for commit:
d3cf0721cddecf7e7a155236116a877547006fe4
I’ll investigate the memory issue on my side. Would libcurl versions installed on my machine make a difference, or is curl compiled from source self contained?
It should be self-contained. Let me take a look at the commit
EDIT: Just ran again, no crash, but I do see the memory climbing
brb
October 2, 2015, 8:03pm
#14
Well, thanks for looking into it for me. It’s prolly some combination of my system / recursive call / my URLLoader wrapper. I’ll let you know if I find out otherwise.
). The malloc error still occurs though the malloc error can also occur in the main render thread now. I don’t think this was the case previously. Here’s a sinppet from one of the relevant backtraces: