Support for Canvas/WebGL Interaction Beyond DOM Elements

[epeles]

Is your feature request related to a problem? Please describe.

I'm trying to use Chrome Devtools MCP to test a Three.js-based 3D application, but I've encountered a limitation where the MCP cannot interact with the canvas content itself - only with DOM elements.

Current Behavior

• The <canvas> element itself is in the DOM and can be found
• However, the actual 3D content rendered on the canvas (3D models, objects, interactive elements) cannot be interacted with
• Neither JavaScript evaluation tools nor mouse position tools seem to work for clicking specific coordinates on the canvas

Describe the solution you'd like

It would be helpful to have a way to:

1. Click at specific X/Y coordinates on the canvas (not just DOM elements)
2. Interact with the canvas content through pixel coordinates
3. Potentially use page.mouse.click(x, y) or similar low-level mouse interactions

Describe alternatives you've considered

Is this a known limitation of the MCP architecture, or is there a workaround I'm missing? Can the MCP:

• Use coordinate-based clicking (e.g., browser_mouse_click with pixel coordinates)?
• Execute Chrome Devtools low-level mouse API (e.g., page.mouse.click(x, y))?
• Interact with canvas elements beyond just finding them in the DOM?

Additional context

No response

[OrKoN]

Coordinate based clicking currently is known to not perform so well with the models but I think we can support it as an option.

[natorion]

Yeah, you would need to have a specialized model like computer_use to call our MCP server to make it work reliably.

Generally about interacting with canvas apps, the question is if you could not add a11y information to your three.js game e.g. with https://github.com/pmndrs/react-three-a11y

[BogdanCerovac]

@epeles - by not knowing other facts of your specifics, I guess you do not have control over the canvas generation (otherwise it is quite simple to expose the scene etc to the MCP and very easy to manipulate thereafter).

If coordinate based clicking is not working, then I guess we would need to check the events on document, window, body, canvas, then find out if we could just trigger them via console, as a brute force solution.

I've tested it now and it can be nearly impossible (if for example WebGL uses some advanced methods like state management etc. - then I guess we would need to debug and catch things there via internal debugging, so quite complicated) or it can be quite simple if we have simple code, like in the following example:

Three.js webgl interactive example:

https://threejs.org/examples/webgl_interactive_voxelpainter.html

open console and run <--- this can be done manually for testing, or via script / MCP ofc.

// 1. center of the window (or use your own coords ofc.)
const x = window.innerWidth / 2;
const y = window.innerHeight / 2;

// 2. pointerdown event (as we know that code is using one)
const clickEvent = new PointerEvent('pointerdown', {
    clientX: x,
    clientY: y,
    bubbles: true
});

// 3. dispatch it to the renderer's canvas (as we know that code listens to it)
document.dispatchEvent(clickEvent);

This worked for me, and I could tweak the MCP to run it, but it is ofc. limited to this simple example situation.

So it may perhaps work for your situation if it is not too complex and you can tweak it case-by-case. But not easily scalable for "any" WebGL

Tried it first with bruno-simon.com and quickly got stucked as it's using a state management behind the scenes :)

[epeles]

Thank you @BogdanCerovac for the detailed response and the workaround solution!

The PointerEvent approach you suggested is very helpful. I'll definitely try implementing this via evaluate_script in our MCP integration. It's good to know that for simpler cases, dispatching pointer events directly can work.

For our use case, we're testing a Three.js-based application, and while we don't have full control over the canvas generation, we can potentially expose some APIs to make interaction easier. Your suggestion about exposing the scene to MCP is something we'll explore.

The limitation with complex state management is noted - we'll keep that in mind when designing our test scenarios.

Thanks again for taking the time to test and provide a practical solution! 🚀

[epeles]

Thank you @natorion for the suggestion about using react-three-a11y! That's a great approach for adding accessibility information to Three.js applications. While it might require some changes to the application code, it would definitely make the canvas content more accessible and easier to interact with via MCP. We'll keep this in mind as a potential solution for applications where we can add accessibility attributes.

Thanks for the helpful recommendation! 👍

[epeles]

Thank you @OrKoN for the clarification about coordinate-based clicking! It's good to know that it's a known limitation with the current models, but that it could potentially be supported as an option in the future.

For now, we'll work with the workarounds suggested (PointerEvent dispatching and exposing APIs), but having native coordinate-based clicking support would be a great addition for testing canvas/WebGL applications.

Thanks for the insight! 🚀

[tsenart]

+1 - Ran into this testing uPlot chart annotations. The annotation markers are rendered on canvas and need x/y coordinate clicks to test tooltip pinning behavior. Currently using Playwright's page.mouse.click(x, y) as a workaround.

[matheuscscp]

I'd love this feature so I can finish this game I started and never finished, it's like 90% complete:

https://github.com/matheuscscp/bohnanza