BitmapData.draw(IBitmapDrawable) issues

Hi,

I have a short question. As filters are dog slow in openfl due to software rendering I render a Sprite hierarchy with filters applied into a bitmap data object. Then I add this object to stage hierarchy and unset the filter in the original sprite.

This only seems to work when being in render thread. However all event handler seem to be called from different threads and I use events to issue the rendering into the bitmap data. I tried to inject the rendering into main thread by using haxe.MainLoop.runInMainThread(). But haxe.MainLoop does not seem to be implemented or running when using Neko as target VM. Or its not supported by openfl.

What is a good solution of this problem? Any ideas?

I am looking forward to hearing from you soon.

Best regards
Andreas

There should only be one thread for the main execution of our code, there is code with Java on Android that runs on a different thread, but we should wait to execute our Haxe code in one consistent thread. There is also a BackgroundWorker or a ThreadPool class that can execute in the wrong thread, but handling the onComplete should be executed from the main thread as well.

What callback appears to be a different thread? Thanks :slight_smile:

Hi,

thanx for your answer.
I might be wrong as I am new to flash / openfl world. What I said were assumptions based on observations.
But I will create tomorrow a small test case that hopefully reproduces the error. I will post it tomorrow here in this thread.
Maybe someone finds out then why the rendering does not work.

Best regards
Andreas

We have another a release coming out, and as I was writing the CHANGELOG, I realized that we did fix an issue where progress events from network and file load requests were in fact running on the wrong thread. Do you think that could be related?

Hi,

Hmm. Not sure. I tested several events. And with any event involved the resulting BitmapData was invalid with zero width and height and such.

I e.g. tested Event.ENTER_FRAME, Event.RENDERING. The initial code was issued by mouse events.
In our app are network related events (URLLoader) involved too. But I do not have currently the overview how things are chained exactly. But I guess URLLoader.COMPLETE event can not be related with Event.ENTER_FRAME, Event.RENDERING, right?

BTW: I remember that sometimes drawing the first time did work while the second did not. Hm. Sounds confusing. I am confused myself now. I guess I will just provide an code example tomorrow.

Best regards
Andreas

Oh, are you testing the HTML5 target?

Loading a BitmapData from bytes works on native targets, but does not work immediately on HTML5, due to how image.src works with Base64 strings.

In other words, this will work as expected on native

var bitmapData = BitmapData.fromBytes (bytes);
trace (bitmapData.width);
trace (bitmapData.height);
trace (bitmapData.getPixel (0, 0));

…but on HTML5, a BitmapData created this way will have a width and height of 0, initially, while the data is still being loaded. Use BitmapData.loadFromBytes (or another asynchronous method) to work on HTML5 and all other targets

BitmapData.loadFromBytes (bytes).onComplete (function (bitmapData) {
    trace (bitmapData.width);
    trace (bitmapData.height);
    trace (bitmapData.getPixel (0, 0));
});

You can also use Assets.getBitmapData (we preload automatically to make images immediately available on HTML5) or Loader like a traditional Flash project

With openfl-6,2,1 and lime-5,7,0 our application crashes with the following message:
openfl test neko -debug 2>&1
An exception occured in a neko Thread :
OutsideBounds

Hi,

I created a test application like:
openfl create DisplayingABitmap

And put my test code there. It looks like:

package;

import openfl.display.Bitmap;
import openfl.display.BitmapData;
import openfl.display.DisplayObject;
import openfl.display.DisplayObjectContainer;
import openfl.display.Sprite;
import openfl.events.Event;
import openfl.events.MouseEvent;
import openfl.filters.BitmapFilter;
import openfl.filters.ColorMatrixFilter;
import openfl.geom.ColorTransform;
import openfl.geom.Point;
import openfl.geom.Rectangle;
import openfl.Assets;

class Main extends Sprite {

    private static var FILTER_BLACK_WHITE : ColorMatrixFilter = new ColorMatrixFilter([1 / 3, 1 / 3, 1 / 3, 0, 0, 1 / 3, 1 / 3, 1 / 3, 0, 0, 1 / 3, 1 / 3, 1 / 3, 0, 0, 0, 0, 0, 1, 0]);

	private var bitmap: Bitmap;
	private var cachedBitmapData: BitmapData;
    private var active: Bool;

	public function new () {
		super();
		bitmap = new Bitmap(Assets.getBitmapData("assets/openfl.png"));
		active = false;
		addEventListener(MouseEvent.MOUSE_UP, onMouseUp);
		invalidateCachedBitmap();
	}

    private function invalidateCachedBitmap(): Void {
        removeEventListener(Event.ENTER_FRAME, createCachedBitmap);
        addEventListener(Event.ENTER_FRAME, createCachedBitmap);
    }

    public static function renderToBitmapData(object: DisplayObject, point: Point = null): BitmapData {
        // TODO: *****, avoid using matrix to transform image into bitmap data coordinate space as there is a bug with openfl
        //  no scale yet
        //  no rotation yet
        var _x: Float = object.x;
        var _y: Float = object.y;
        var objectColorTransform: ColorTransform = object.transform.colorTransform;
        var bounds: Rectangle;
        var bitmapData: BitmapData;
        bounds = object.getBounds(object.parent);
        bounds.x = Math.floor(bounds.x);
        bounds.y = Math.floor(bounds.y);
        bounds.height = Math.ceil(bounds.height);
        bounds.width = Math.ceil(bounds.width);
        var realBounds: Rectangle = new Rectangle(0, 0, bounds.width, bounds.height);
        if (object.filters.length > 0) {
            var objectFilters: Array<BitmapFilter> = object.filters;
            var tmpBitmapData: BitmapData;
            var filterRect: Rectangle;
            tmpBitmapData = new BitmapData(Std.int(realBounds.width), Std.int(realBounds.height), false);
            filterRect = tmpBitmapData.generateFilterRect(tmpBitmapData.rect, objectFilters[0]);
            tmpBitmapData.dispose();
            for (i in 1...objectFilters.length){
                tmpBitmapData = new BitmapData(Std.int(filterRect.width), Std.int(filterRect.height), true, 0);
                filterRect = tmpBitmapData.generateFilterRect(tmpBitmapData.rect, objectFilters[i]);
                realBounds = realBounds.union(filterRect);
                tmpBitmapData.dispose();
            }
        }
        realBounds.offset(bounds.x, bounds.y);
        realBounds.width = Math.max(realBounds.width, 1);
        realBounds.height = Math.max(realBounds.height, 1);
        if (Std.is(object, DisplayObjectContainer) == true) {
            var objectDisplayObjectContainer: DisplayObjectContainer = cast(object, DisplayObjectContainer);
            for (i in 0...objectDisplayObjectContainer.numChildren) {
                var childObject: DisplayObject = objectDisplayObjectContainer.getChildAt(i);
                childObject.x-= realBounds.x;
                childObject.y-= realBounds.y;                
            }
        }
        bitmapData = new BitmapData(Std.int(realBounds.width), Std.int(realBounds.height), true, 0);
        bitmapData.draw(object, null, objectColorTransform);
        if (Std.is(object, DisplayObjectContainer) == true) {
            var objectDisplayObjectContainer: DisplayObjectContainer = cast(object, DisplayObjectContainer);
            for (i in 0...objectDisplayObjectContainer.numChildren) {
                var childObject: DisplayObject = objectDisplayObjectContainer.getChildAt(i);
                childObject.x+= realBounds.x;
                childObject.y+= realBounds.y;                
            }
        }
        realBounds.offset(-_x, -_y);
        if (point != null) {
            point.setTo(realBounds.x, realBounds.y);
        }
        return bitmapData;
    }

    private function createCachedBitmap(event: Event): Void {
        trace("createCachedBitmap()");
        while (numChildren > 0) removeChildAt(0);
        if (active == true) {
            bitmap.filters = [];
        } else {
            bitmap.filters = [FILTER_BLACK_WHITE];
        }
        var bitmapPoint : Point = new Point();
        if (cachedBitmapData != null) cachedBitmapData.dispose();
        cachedBitmapData = renderToBitmapData(bitmap, bitmapPoint);
        bitmap.filters = [];
        var sprite: Sprite = new Sprite();
        var button: DisplayObject = new Bitmap(cachedBitmapData);
        sprite.buttonMode = active;
        sprite.x = bitmapPoint.x;
        sprite.y = bitmapPoint.y;
        sprite.addChild(button);
        addChild(sprite);
        removeEventListener(Event.ENTER_FRAME, createCachedBitmap);
    }

    private function onMouseUp(event: MouseEvent) {
        trace("onMouseUp()");
        active = !active;
        invalidateCachedBitmap();
    }

}

Unfortunately I can not test as the program crashes with:
Andreass-Air:bitmapdata-draw-issues andreas$ openfl test neko -debug 2>&1
Called from openfl.filters.ColorMatrixFilter::$statics line 16
Called from openfl.filters._ColorMatrixFilter.ColorMatrixShader::$init line 138
Called from a C function
Called from openfl.filters._ColorMatrixFilter.ColorMatrixShader::new line 177
Called from openfl.display.Shader::get_data line 691
Called from openfl.display.Shader::__init line 259
Called from openfl.display.Shader::__initGL line 284
Called from openfl.display.Shader::__processGLData line 453
Called from EReg::matchedPos line 62
Uncaught exception - Invalid call

Is it possible that this crash is related to latest release?

(I edited the source code to remove previous attached Childs)

Best regards
Andreas

Oh i see, that I forgot to remove the previous attached child. Will fix that. I will see if it helps. But still newest openfl crashes :frowning:

@andreasdr Thanks for the sample code :grinning:

Here’s the fix:

You can also workaround the issue if you create your filter outside of a static declaration:

private static var FILTER_BLACK_WHITE : ColorMatrixFilter;

...
public function new () {
	super();
	if (FILTER_BLACK_WHITE == null) FILTER_BLACK_WHITE = new ColorMatrixFilter([1 / 3, 1 / 3, 1 / 3, 0, 0, 1 / 3, 1 / 3, 1 / 3, 0, 0, 1 / 3, 1 / 3, 1 / 3, 0, 0, 0, 0, 0, 1, 0]);

Hi,

i see shaders in your code for Glowfilter and ColorMatrixFilter. Does this mean that in newest version of OpenFL both filters are not slow anymore due to hardware rendering???

I will try your fix tomorrow. I am in Berlin/Germany. Its midnight here.

Best regards
Andreas

We used shader-based filters in an older version of OpenFL, but had serious performance bottlenecks on mobile. We disabled, improved cacheAsBitmap and implemented software filters. I’d like to see hardware shader filters enabled again, using the new cacheAsBitmap improvements as well as a smarter approach to GL framebuffers in order to improve the performance

1 Like

Hi

as I am new to openfl I like to know if cacheAsBitmap would help with rendering sprite hierarchies with filters applied?

Because If I do normal rendering of buttons with blackwhite filters enabled (even if enabled once and much later disabled) each frame takes like 2 seconds or similar. Caching could help indeed.

Best regards
Andreas

filters implies the use of cacheAsBitmap, so using bitmapData.draw along with bitmap.filters means you are drawing the object twice.

In the meantime until we enable shader-based filters again, you might try this approach:

package;

import openfl.display.Bitmap;
import openfl.display.BitmapData;
import openfl.display.DisplayObject;
import openfl.display.DisplayObjectContainer;
import openfl.display.Shader;
import openfl.display.Sprite;
import openfl.events.Event;
import openfl.events.MouseEvent;
import openfl.filters.BitmapFilter;
import openfl.filters.ColorMatrixFilter;
import openfl.geom.ColorTransform;
import openfl.geom.Point;
import openfl.geom.Rectangle;
import openfl.Assets;

class Main extends Sprite {

    private var bitmapData: BitmapData;
    private var colorMatrixShader: ColorMatrixShader;
    private var active: Bool;

    public function new () {
        super();
        colorMatrixShader = new ColorMatrixShader ();
        colorMatrixShader.init ([1 / 3, 1 / 3, 1 / 3, 0, 0, 1 / 3, 1 / 3, 1 / 3, 0, 0, 1 / 3, 1 / 3, 1 / 3, 0, 0, 0, 0, 0, 1, 0]);
        bitmapData = Assets.getBitmapData("assets/openfl.png");
        active = false;
        addEventListener(MouseEvent.MOUSE_UP, onMouseUp);
        invalidateCachedBitmap();
    }

    private function invalidateCachedBitmap(): Void {
        removeEventListener(Event.ENTER_FRAME, createCachedBitmap);
        addEventListener(Event.ENTER_FRAME, createCachedBitmap);
    }

    private function createCachedBitmap(event: Event): Void {
        trace("createCachedBitmap()");
        while (numChildren > 0) removeChildAt(0);
        var bitmapPoint = new Point();
        var sprite = new Sprite();
        var button = new Bitmap(bitmapData);
        if (active == true) {
            button.shader = null;
        } else {
            button.shader = colorMatrixShader;
        }
        sprite.buttonMode = active;
        sprite.x = bitmapPoint.x;
        sprite.y = bitmapPoint.y;
        sprite.addChild(button);
        addChild(sprite);
        removeEventListener(Event.ENTER_FRAME, createCachedBitmap);
    }

    private function onMouseUp(event: MouseEvent) {
        trace("onMouseUp()");
        active = !active;
        invalidateCachedBitmap();
    }

}


class ColorMatrixShader extends Shader {
    
    
    @:glFragmentSource( 
        
        "varying float vAlpha;
        varying vec2 vTexCoord;
        uniform sampler2D uImage0;
        
        uniform mat4 uMultipliers;
        uniform vec4 uOffsets;
        
        void main(void) {
            
            vec4 color = texture2D (uImage0, vTexCoord);
            
            if (color.a == 0.0) {
                
                gl_FragColor = vec4 (0.0, 0.0, 0.0, 0.0);
                
            } else {
                
                color = vec4 (color.rgb / color.a, color.a);
                color = uOffsets + color * uMultipliers;
                
                gl_FragColor = vec4 (color.rgb * color.a * vAlpha, color.a * vAlpha);
                
            }
            
        }"
        
    )
    
    
    public function new () {
        
        super ();
        
        #if !macro
        data.uMultipliers.value = [ 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1 ];
        data.uOffsets.value = [ 0, 0, 0, 0 ];
        #end
        
    }
    
    
    public function init (matrix:Array<Float>):Void {
        
        var multipliers = data.uMultipliers.value;
        var offsets = data.uOffsets.value;
        
        multipliers[0] = matrix[0];
        multipliers[1] = matrix[1];
        multipliers[2] = matrix[2];
        multipliers[3] = matrix[3];
        multipliers[4] = matrix[5];
        multipliers[5] = matrix[6];
        multipliers[6] = matrix[7];
        multipliers[7] = matrix[8];
        multipliers[8] = matrix[10];
        multipliers[9] = matrix[11];
        multipliers[10] = matrix[12];
        multipliers[11] = matrix[13];
        multipliers[12] = matrix[15];
        multipliers[13] = matrix[16];
        multipliers[14] = matrix[17];
        multipliers[15] = matrix[18];
        
        offsets[0] = matrix[4] / 255.0;
        offsets[1] = matrix[9] / 255.0;
        offsets[2] = matrix[14] / 255.0;
        offsets[3] = matrix[19] / 255.0;
        
    }
    
    
}

Hi,

Nice! I will try this tomorrow. Thanx-

But regarding cacheAsBitmap I would think that if I set it to to true a bitmap of my sprite hierarchy would be generated ONLY if my sprite hierarchy changes. Be it the filter or any other change to a child.

So in my case this would mean that the cache is only generated if I set the filter to BLACKWHITE or NONE.

So this leads to the next question. How does it really work?

Sorry for all the questions.

Best regards
Andreas

Hi,

just tried our project in browser(which is our current target platform). I am impressed. Browser performance is ways better then Neko in MacOSX.
I will stop performance optimizations in our project and continue with completeness.

Do you know when your performance optimizations will be done and maybe released?

Best regards
Andreas

Oh yes, Neko is known to be slow – testing openfl test mac or openfl test html5 is a better indicator of actual performance.

cacheAsBitmap is designed to draw only when the object is changed, but what I meant is that setting bitmap.filters = [ filter ]; implies cacheAsBitmap on the object, so using filters, then using bitmapData.draw (to manually cache) results in caching the object twice, if that makes sense

bitmap.shader is a workaround, currently, for the filter system not using shaders, but I hope that we will be able to introduce a shader-based filter system again in the coming months :slight_smile:

1 Like

Hi,

bitmap.shader is a workaround, currently, for the filter system not using shaders, but I hope that we will be able to Introduce a shader-based filter system again in the coming months

Looking forward to it!!!

Thank you so much for your help. Ill be quite now. :smile:

Best regards
Andreas

1 Like