Have you tried using the ShaderFilter class? There’s an example class in openfl.filters.BitmapFilterShader that you can use as the basis for writing a custom one. There’s a way to use #pragma head and #pragma body to import the shader’s head and body (if you’re just extending it) or you could write your own completely from scratch.
There’s an example custom shader here:
…but using ShaderFilter, you can do something similar, but it will run as a post-process effect. So we draw the display object like normal, then you can perform an effect on the result. This is often what is desired over displayObject.shader, which will customize the way the object (and each of its children) is drawn at the time they are rendered. A Sprite with 100 children would run a sprite.shader effect 101 times, where sprite.filters = [ myShaderFilter ] would run once on the total result