Xterm-like terminals already provide basic facilities for interacting with a mouse, and some provide graphics facilities. It'd thus be quite possible to build or modify a GUI toolkit so that GUI programs just took over the terminal window they were run in, like ncurses programs already do. This'd then automatically work over ssh connections.