Discussion regarding the project "Using a Game Controller as a Navigation Aid"

[Neetigya Chahar]

Hello everyone! I am a sophomore CSE student and have some experiences in
nodejs, creating and hosting chrome extensions on the store. I have also
done some programming and controlling of IoT devices.

As I was looking to work with open source community through gsoc2020, I
came across the project "Using a Game Controller as a Navigation Aid"

My 30 minutes experience with keyboard navigation:
1. Most of the time, it was tab which I used to navigate around.
2. It was very frustrating to press tab again to reach an element which is
at the centre or near bottom of the page every time new page is loaded.
3. For many websites like stackoverflow it was very touch to spot the
current focus. And for many sites, the focus was instantaneously lost on
next press.
4. Even if focus was clearly visible, for websites like w3schools, first
tab focus traverse through right side menu, which is too long many times.
This again creates need to press tab again and again.
5. Some HTML elements were totally inaccessible for e.g. unable to access
whatsapp web chat options.
6. Canvas elements were not accessible.
7. Although websites like Google search has provided good accessibility
support.
8. Keys other than tab were pretty much usable, but still may have further
improvements.

Both proposed solutions uses chrome extension, as according to me using
extension instead of an electron app is more productive and practical.

What can be done if we don't consider mouse simulation along:
0. Basically we have to map keys on the keyboard to the game pad and read
the game pad output using official Game pad API.
1. Every Page's :focus could be edited to make outline look more clearly.
2. Provide key combo (or press and hold) to automatically press tab
multiple times.
3. We can create a page which tells people how keys of the game pad are
mapped to keyboard keys.
4. We can provide option to edit these mappings and give suggestions.

Using mouse pointer:
Mouse pointer is not part of the web page or the browser.
It is something directly provided by the OS.
There exists no web api to control mouse movement as it is considered a
security vulnerability.
However, the movements can be tracked.

My idea:
I was to thinking of adding a virtual mouse to the sites through extension.
I have tested that.
We can define an element with image of a mouse and the element can be
positioned absolutely.
Now we can set and alter the top and left properties i.e y and x
coordinates of the element. This way we can move the element wherever we
want.
This element can be controlled by joystick, arrow keys in the game pad or
in the keyboard.
Since we know the x and y position of the element, I wrote the following
chuck to perform click action.

    pointer.style.display = 'none'; //Remove our element.
    let test = document.elementFromPoint(pointer.x, pointer.y); //select
the element where our element was.
    test.click(); //click it
    pointer.style.display = 'block'; //Show up again.

The action of removing and showing up again is instantaneous, goes
unnoticed.

What can be done if we consider mouse simulation along:
1. The work of the tab key can be totally shifted to this mouse.
2. Other keys like space, enter, arrow keys, key combos like for closing
and opening tabs will still be usable as earlier.
3. All the problems related to the tab will be solved.
4. We can give a click animation feedback to users as in android, thus
increasing accessibility.
5. Also, when the element is at top of some text (like paragraph), the user
can press a key on game pad and we can read it aloud, bold it, increase
font size, etc.

I hope I didn't annoyed anyone by writing such a text to mailing list. I
tried IRC but there was timezone issue, I will adjust that in future.
Meanwhile I am learning about the community and its projects like infusion.

Feedback awaited!

This is most basic implementation of a virtual mouse.
[image: fluid-project.png]
[image: gsoc20201.png]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20200229/50fe03a7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fluid-project.png
Type: image/png
Size: 23090 bytes
Desc: not available
URL: <http://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20200229/50fe03a7/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gsoc20201.png
Type: image/png
Size: 43059 bytes
Desc: not available
URL: <http://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20200229/50fe03a7/attachment-0001.png>