Why is pynput typing incorrect output, but only on Linux? (lemmy.world)

submitted 1 week ago* (last edited 1 week ago) by zkfcfbzr@lemmy.world to c/python@programming.dev

14 comments fedilink hide all child comments

I have a rather large Python script that I use as basically a replacement for autohotkey. It uses pynput for keyboard and mouse control - and at least on Windows, it works exactly how I expect.

I recently started dual-booting with Linux and have been trying to get the script to work here as well. It does work but with mixed results - in particular, I found that pynput has bizarrely wrong output for special characters, in a way that's both consistent and inconsistent.

The simplest possible case I found that reproduces the error is this script:

import time
from pynput import keyboard

# Sleep statement is just to give time to move the mouse cursor to a text input field
time.sleep(2)

my_kb = keyboard.Controller()

text = '🍆' # Eggplant emoji
my_kb.type(text)

time.sleep(1)

text = '𝕥𝕖𝕤𝕥' # blackboard bold test
my_kb.type(text)

time.sleep(1)

text = '𝐭𝐞𝐬𝐭' # bold test
my_kb.type(text)

When I run that script right now, it produces the output "🍆𝕥𝕥𝕤𝕥𝐭𝐭𝐬𝐭". And if I run it again, it'll produce the same output. And if I change the eggplant emoji to something else, like the regular character 'A', it will still produce the same output (specifically "A𝕥𝕥𝕤𝕥𝐭𝐭𝐬𝐭"). But... If I log out and log back in, then the output changes to something else that's still wrong, but differently. For example, when I changed the eggplant to a regular 'A', then relogged, the output became "A𝕥𝕖𝕖𝕥𝐭𝐞𝐞𝐭". And then that wrong output will keep being the same wrong output until I log out and back in again. If the test strings don't change, then the incorrect outputs don't change on relog - but if they do, then they do.

In the larger script, errors seemed to chain together somehow - like if I produced an eggplant emoji, then tried to write blackboard bold test, I would get "🍆𝕖𝕤🍆". This is despite verifying just before running the pynput.keyboard.Controller.type function that what it was about to type was correct. The issue also happens if I type it character-by-character with press and release functions.

I am very new to Linux. I'm on Linux Mint. I'm running this in a python3 venv that just has pynput and two other external libraries installed. ChatGPT thinks the issue might be related to X11. The issue does not occur at all on Windows, using the exact same code. On Linux there seems to be no issues with typing regular text, just special characters.

you are viewing a single comment's thread
view the rest of the comments

[-] logging_strict@programming.dev 2 points 3 days ago

Took two days to think about your original post. Was thinking, hmmm this package and trouble you are having are both fresh and interesting.

Remote controlling both the mouse and keyboard seems worthy to spend time trying it out.

[-] zkfcfbzr@lemmy.world 2 points 2 days ago* (last edited 2 days ago)

Gonna make some notes since I made some progress tonight (so far).

Within pynput's keyboard's _xorg.py file, in the Controller class, self._keyboard_mapping maps from each key's unique keysym value, which is an integer, to a 2-tuple of integers. The actual keysym for each key in the mapping appears to be correct, but occasionally the 2-tuple duplicates that of another entry in self._keyboard_mapping - and these duplicates correspond precisely to the errors I see in pynput's outputs.

For example, '𝕥' has keysym = 16897381 and '𝕖' has keysym = 16897366, but both 16897381 and 16897366 map, in self._keyboard_mapping, to the 2-tuple (8, 1) - and '𝕖' is indeed printed as '𝕥' by pynput. (𝕥's keysym appears first in self._keyboard_mapping). (The 2-tuple keysyms map to are not unique or consistent, they vary based on the order they were encountered and reset when X resets)

Through testing, I found that this type of error happens precisely every third time a new keysym is added to self._keyboard_mapping, and that every third such mapping always duplicates the 2-tuple of the previous successful mapping.

From that register function I mentioned, the correct 2-tuple should be, I believe, (keycode, index). This is not happening correctly every 3rd registration, but I'm not yet sure why.

However, I did find a bit more than that: register() gets its keycode and index from one of the three functions above it, reuse() borrow() or overwrite(). The problematic keys always get their keycode and index from reuse - and reuse finds the first unused index 0-3 for a given keycode, then returns that. What I found here is that, for the array keycodes, the first element is always duplicated to the third position as well, so indexes 0 and 2 are identical. As an example, here are two values of keycodes from my testing:

keycodes = array('I', [16897386, 0, 16897386, 0, 0, 0, 0, 0, 0, 0])
keycodes = array('I', [16897384, 16897385, 16897384, 0, 0, 0, 0, 0, 0, 0])

With this in mind, I was actually able to fix the bug by changing the line for index in range(4): in reuse() to for index in range(2):. With that change my script no longer produces any incorrect characters.

However, based on the comments in the function, I believe range(4) is the intended behavior and I'm just avoiding the problem instead of fixing it. I have a rather shallow understanding of what these functions or values are trying to accomplish. I don't know why the first element of the array is duplicated to the third element. There's also a different issue I noticed where even when this function returns an index of 3, that index of 3 is never used in self._keyboard_mapping - it uses 1 instead. I'm thinking these may be two separate bugs. Either way, these two behaviors combined explain why it's every third time a new keysym is added to self._keyboard_mapping that the issue happens: While they in theory support an index of 0 1 2 or 3 for each keycode, in practice only indices 0 1 and 3 work since 2 always copies 0 - and whenever 3 is picked it's improperly saved as 1 somewhere.

I may keep investigating the issues in search of a true fix instead of a cover-my-eyes fix.

[-] zkfcfbzr@lemmy.world 1 points 2 days ago

I agree and appreciate it. I've been trying to figure it out myself but feel a bit out of my element.

What I've found is that in pynput's keyboard's _xorg.py file, the Controller class's self._keyboard_mapping seems to map some different keycodes to the same value, and that seems to correlate exactly with the errors I'm seeing. I haven't figured out why yet. I got to thinking it had something to do with the register function in the _resolve_borrowing function but I forget why and I'm too tired to continue for now. I'll continue tomorrow though.