Tip: Custom Mask workaround

Hey guys,

If you are looking for more robust masking support, the latest release of OpenFL improves on the custom shader system. Among other features, this allows sampling of additional images.

You can only apply this to a flat object (such as a Bitmap, Shape or Tilemap) but until we add framebuffer support internally to flatten more complex objects, this at least provides an option on a GL target.

Old versions of OpenFL did support masks (such as circles) using hardware, but it had crude, rough edges. In the future, I would like to see core support for masks using a system similar to the following, but handled internally:

package;


import openfl.display.BitmapData;
import openfl.display.Bitmap;
import openfl.display.DisplayObject;
import openfl.display.Shader;
import openfl.display.Sprite;
import openfl.filters.ShaderFilter;


class Main extends Sprite {
	
	
	public function new () {
		
		super ();
		
		var bitmapData = new BitmapData (200, 200, false, 0xFFFF0000);
		var bitmap = new Bitmap (bitmapData);
		addChild (bitmap);
		
		var mask = new Sprite ();
		mask.graphics.beginFill (0xFF0000);
		mask.graphics.drawCircle (100, 100, 100);
		
		applyMask (bitmap, mask);
		
	}
	
	
	private function applyMask (bitmap:Bitmap, mask:DisplayObject):Void {
		
		#if flash
		
		bitmap.mask = mask;
		
		#else
		
		var bitmapDataMask = new BitmapData (bitmap.bitmapData.width, bitmap.bitmapData.height, true, 0);
		bitmapDataMask.draw (mask);
		
		var shader = new Shader ();
		shader.glFragmentSource = 
			
			"varying float vAlpha;
			varying vec2 vTexCoord;
			uniform sampler2D uImage0;
			uniform sampler2D uImage1;
			
			void main(void) {
				
				vec4 color = texture2D (uImage0, vTexCoord);
				float mask = texture2D (uImage1, vTexCoord).a;
				
				if (color.a == 0.0 || mask == 0.0) {
					
					gl_FragColor = vec4 (0.0, 0.0, 0.0, 0.0);
					
				} else {
					
					gl_FragColor = vec4 (color.rgb / color.a, mask * color.a * vAlpha);
					
				}
				
			}";
		
		shader.data.uImage1.input = bitmapDataMask;
		
		bitmap.filters = [ new ShaderFilter (shader) ];
		
		#end
		
	}
	
	
}

Hope this helps someone :slight_smile:

3 Likes

Quick question: You don’t pass shader.data.uImage0 - is it passed implicitly? I only have my experience with shaders from ShaderToy, so no experience with passing stuff to GLSL yet.

shader.data is generated automatically based on the GL vertex and fragment sources, and when it is time to render with that shader, it gets populated with the world transform and alpha values. Values (such as our uImage1 texture) are bound automatically

I love you very much, Joshua.

I’m still very new to OpenFL, so please bear with me…

I copied and pasted this code into my project.

  • openfl test flash works fine.
  • however openfl test html5 throws the following error in the browser console:

TypeError: undefined is not an object (evaluating ‘shader.get_data().uImage1.input = bitmapDataMask’)

Any pointers/advice? Is this method even supported for HTML5?

THANKS for any help!

The above code was written for before OpenFL had GL mask support internally, and before we revamped the shader code for the OpenFL 8 release.

If you need only on/off masks, use object.mask like in Flash. If you need to patch in support for alpha-based masking (before we get it off the TODO and implemented), take a look at this sample code, I think it should work:

HAPPY DANCE!!!
THANKS, @singmajesty!!!

This is ultimately what I’m trying to achieve:
image

I’ll let you know how it goes.

THANKS, AGAIN!

1 Like

SUCCESS!!!

package;


import openfl.display.BitmapData;
import openfl.display.Bitmap;
import openfl.display.DisplayObject;
import openfl.display.DisplayObjectShader;
import openfl.display.Shader;
import openfl.display.Sprite;
import openfl.filters.ShaderFilter;

import openfl.filters.BlurFilter;


class Main extends Sprite {
	
	
	public function new () {
		
		super ();
		
		var mc_starspin_Bitmap = new Bitmap(Assets.getBitmapData("images/eg_starspin.png"));
		addChild(mc_starspin_Bitmap);

		mc_starspin_Bitmap.x = -75;
		mc_starspin_Bitmap.y = -90;

		var mask = new Sprite ();
		mask.graphics.beginFill (0xFFFFFF);
		mask.graphics.drawEllipse (150, 15, 150, 400);

		var filter = new BlurFilter(50, 50, 1);
		mask.filters = [filter];

		applyMask (mc_starspin_Bitmap, mask);
		
	}
	
	
	private function applyMask (bitmap:Bitmap, mask:DisplayObject):Void {
		
		#if flash
		
		bitmap.mask = mask;
		
		#else
		
		var bitmapDataMask = new BitmapData (bitmap.bitmapData.width, bitmap.bitmapData.height, true, 0x00FFFFFF);
		bitmapDataMask.draw (mask);
		
		var shader = new BitmapMaskShader ();
		shader.maskImage.input = bitmapDataMask;

		bitmap.shader = shader;
		
		#end
		
	}
	
	
}


class BitmapMaskShader extends DisplayObjectShader {
	
	
	@:glFragmentSource("
		
		#pragma header
		
		uniform sampler2D maskImage;
		
		void main(void) {
			
			#pragma body
			
			float mask = texture2D (maskImage, openfl_TextureCoordv).a;
			gl_FragColor *= mask;
			
		}
		
	")
	
	public function new () {
		
		super ();
		
	}
	
	
}
1 Like

THANKYOU!THANKYOU!THANKYOU! @singmajesty!!!

1 Like

How to create invert mask effect?