Lime keyboard event confusion

Taylor_Anderson · February 17, 2017, 3:29pm

So we have a virtual keyboard that we use to take keyboard events (or key onMouseDown events) and convert them into a string that we then send out as a signal, which all our InputFields then listen to.

So the issue I’m having is with the shift key. With OpenFL’s keyboard events, it doesn’t handle shift at all, so initially to get around that we hardcoded our own mapping for shifted values. We realized soon after that this would totally fail with different keyboard layouts, so we had to find a different solution.

On the surface it seems like stage.window.onTextInput and stage.window.onKeyDown are the two solutions to this problem. onTextInput sends out the correct string, even while pressing shift, even on different keyboard layouts.

The problem with this is, this approach doesn’t work completely on Flash. For example: on the Swiss French keyboard, shift-1 is the ‘+’ character, whereas on ENG keyboards that’s ‘!’. On HTML5, when you press shift-1 on a Swiss French keyboard, it will send out the correct signal, the ‘+’ character. On Flash, it will send out ‘!’.

What makes this even weirder is that I got this idea of using the Lime keyboard events from openfl’s TextField class. But somehow, even though we’re using the same events, the TextField class works just fine on Flash, it displays the proper ‘+’ character when you type shift-1 on the Swiss French keyboard.

I was wondering if anyone could shed some light on what TextField does differently to achieve this result (specifically @singmajesty if you have the time )

Thanks,
Taylor

singmajesty · February 17, 2017, 5:14pm

We have tried to unify the window.onKeyDown and window.onKeyUp events to report the same key code for each key, regardless of the platform (please let me know if this is not the case!)

You can see, the key codes don’t change when you use a key modifier (such as shift), though in OpenFL, we do want to emulate Flash, and so we do want to dispatch different values when shift is pressed if possible.

On the other hand, the window.onTextInput event is designed to handle text input, which in the case of Asian languages and combining characters, is not going to always directly map to one-key-one-character.

So let me know if there seem to be any issues with the Lime events.

It sounds as if the real issue at hand is that OpenFL’s conversion of shift-modified events doesn’t translate well to other keyboard layouts:

https://github.com/openfl/openfl/blob/develop/openfl/ui/Keyboard.hx#L375-L469

Perhaps, we can perform some key-code-to-scan-code conversions to determine the keyboard layout. If the scan code “Q” maps to key code “A” (plus some other tells) it’s an AZERTY layout, or similarly detecting if scan code “Y” maps to “Z” for QWERTZ, so on, It’s not perfect, but it seems like the right thing to do.

I’ve created an enhancement here:

Taylor_Anderson · February 17, 2017, 6:00pm

The specific problem I was having was that onTextInput doesn’t always report the correct character on Flash if the keyboard layout is changed. Is that a goal of that function or is that scancode enhancement the only way to do this?

singmajesty · February 17, 2017, 6:11pm

Oh, gotcha. Well maybe we can improve this code

github.com

openfl/lime/blob/develop/lime/_backend/flash/FlashApplication.hx#L213


package lime._backend.flash;




import flash.events.Event;
import flash.ui.MultitouchInputMode;
import flash.ui.Multitouch;
import flash.Lib;
import lime.app.Application;
import lime.app.Config;
import lime.media.AudioManager;
import lime.graphics.Renderer;
import lime.ui.Window;


@:access(lime.app.Application)
@:access(lime.graphics.Renderer)




class FlashApplication {
	
	
	private var cacheTime:Int;
	private var parent:Application;
	
	
	public function new (parent:Application):Void {
		
		this.parent = parent;
		
		Lib.current.stage.frameRate = 60;
		AudioManager.init ();
		
	}
	
	
	public function create (config:Config):Void {
		
		
		
	}
	
	
	public function exec ():Int {
		
		Multitouch.inputMode = MultitouchInputMode.TOUCH_POINT;
		
		cacheTime = Lib.getTimer ();
		handleApplicationEvent (null);
		
		Lib.current.stage.addEventListener (Event.ENTER_FRAME, handleApplicationEvent);
		
		return 0;
		
	}
	
	
	public function exit ():Void {
		
		
		
	}
	
	
	public function getFrameRate ():Float {
		
		return Lib.current.stage.frameRate;
		
	}
	
	
	private function handleApplicationEvent (event:Event):Void {
		
		var currentTime = Lib.getTimer ();
		var deltaTime = currentTime - cacheTime;
		cacheTime = currentTime;
		
		parent.onUpdate.dispatch (deltaTime);
		
		if (parent.renderer != null) {
			
			parent.renderer.onRender.dispatch ();
			parent.renderer.flip ();
			
		}
		
	}
	
	
	public function setFrameRate (value:Float):Float {
		
		if (parent.windows.length > 0) {
			
			for (window in parent.windows) {
				
				window.stage.frameRate = value;
				
			}
			
		} else {
			
			Lib.current.stage.frameRate = value;
			
		}
		
		return value;
		
	}
	
	
}

Taylor_Anderson · February 17, 2017, 6:24pm

It makes sense to me that the Flash platform has some small bug that doesnt report every character correctly, what actually weirded me out the MOST about this issue (and is the reason why I came to you, who knows this stuff much better than I do :P) was that the OpenFL textfield gets the correct characters on HTML5 AND Flash. So it’s gotta be doing something different that sidesteps that issue, I just cant figure out what that might be

singmajesty · February 17, 2017, 6:30pm

Perhaps something like this would help:

stage.addEventListener (KeyboardEvent.KEY_DOWN, function (e) {
	
	trace ("KeyboardEvent -- " + e.charCode + ", " + String.fromCharCode (e.charCode));
	
});

var textField = new TextField ();
textField.border = true;
textField.type = INPUT;
addChild (textField);

textField.addEventListener (Event.CHANGE, function (_) {
	
	trace ("TextField input -- " + textField.text);
	textField.text = "";
	
});

We could run this on Flash, and see what comes through. Perhaps there is a way to improve how we translate KeyboardEvents to mirror how TextField input works on Flash, or perhaps not. If not, perhaps we will have to resort to a hidden TextField on Flash, similar to what we have done on HTML5

Taylor_Anderson · February 17, 2017, 6:49pm

This doesn’t really reflect my issue though, because my issue is the behavior when a key is pressed WHILE the shift key is held down.

I should clarify that I am seeing no issues right now with keyboard event behavior when the shift key is NOT pressed.

singmajesty · February 17, 2017, 6:52pm

Yep, well a test like the above (on Flash Player) should give us a starting point to seeing what Flash Player returns – I believe that Flash is supposed to return different charCodes when pressing shift, but there’s a chance here that’s its not sensitive to the keyboard layout

Taylor_Anderson · February 17, 2017, 8:12pm

Ah okay, I didn’t realize that the charcodes changed when the shift key is pressed. In that case yeah, I have the test open and its returning the same code for both layouts.

Test.hx:29: KeyboardEvent – 0, //shift
Test.hx:29: KeyboardEvent – 33, 1
Test.hx:40: TextField input – !
//changed keyboards
Test.hx:29: KeyboardEvent – 0, //shift
Test.hx:29: KeyboardEvent – 33, 1
Test.hx:40: TextField input – +

Taylor_Anderson · February 21, 2017, 6:47pm

Okay so I’m trying to use a hidden TextField as a fallback for this issue, but my problem now is I need to both take in TextField input AND keyboard events, to handle stuff like arrow keys and backspace. Is there a way to do this? Because right now the textfield will just start taking all the text events for itself.

singmajesty · February 21, 2017, 6:49pm

Does it work if you listen to the TextField for KeyboardEvents? I believe KeyboardEvent is dispatched to stage.focus, so we might need to listen to the field instead of the stage to catch those events

Taylor_Anderson · February 21, 2017, 8:03pm

Okay this works, except the event is firing BEFORE the textfield’s content gets changed, so i’m always one character behind

EDIT: having a separate listener for Event.CHANGE solved it. would rather not have two, but if it works, it works.

Taylor_Anderson · February 23, 2017, 8:01pm

Okay update to this: other Keyboard Events in other components are not firing any more, even when the event listener is added directly to the stage. Commenting out the line stage.focus = t_input (t_input is my textfield) fixes the issues with the other components. Am I doing something wrong? It seems like event listeners added to the stage directly should not be affected by what stage.focus is.

singmajesty · February 23, 2017, 8:05pm

The issue is that when stage.focus changes, it also changes which object dispatches keyboard events. It might be possible to help this by listening to keyboard events on your focus object, but then calling stage.dispatchEvent (event) to rebroadcast it through the stage

Taylor_Anderson · February 23, 2017, 8:28pm

Okay the problem with THIS is (funny story, I made that change like five minutes before this message) is that I’m ALSO listening to Event.CHANGE on textfield to get the content of the textfield (to fix that keyboard layout bug we were talking about further up) so every text input gets dispatched twice.

Taylor_Anderson · February 28, 2017, 2:43pm

The problem with the keycode-scancode check is: at that point, are still hardcoding shift key conversions for every keyboard layout the user might want to use? Is there not any way to get those shift key conversions automatically?

singmajesty · February 28, 2017, 7:47pm

For native platforms, we are using a scan code to key code conversion that should be sensitive to the current user’s keyboard layout.

The issue on Flash is that it appears that the translation between keyCodes and charCodes is set, regardless of the user keyboard layout. It is only when using TextField input that non-QWERTY layouts appear to be respected. I could be wrong?

Taylor_Anderson · February 28, 2017, 8:45pm

Yeah, that’s been my experience so far. I guess I’m just confused about how the keycode-to-scancode conversion works to get this result, but the HTML5 target has the proper behavior for multiple layouts so I know it definitely DOES work.

player_03 · March 2, 2017, 9:39am

Um, maybe I’m not following, but this sounds wrong.

I use the Colemak layout, which means the S and D keys have been moved. When I play a Flash game that offers WASD controls, I can’t place my fingers on the standard “inverted T” shape and expect things to work. Trust me, I’ve tried. (Though at least Colemak is better than Dvorak in this respect, since the keys are all still reachable with one hand. Back when I used Dvorak, I had to revert to QWERTY a lot more often.)

Taylor_Anderson · March 3, 2017, 6:33pm

It seems that the letters are correct on Flash, but when tracing the charcode of shift+[any number], it still gives me the values from the English keyboard.