While there are already a variety of filesystem designs for different applications, very few support concurrent access from multiple machines: GFS is an example. However, GFS requires some sort of fine-grained IO fencing; this can be done via SCSI or using network communication, but it's non-trivial to set up.
What I'd actually like is a lockless filesystem, where each node is only able to read and write the filesystem, perhaps with some write ordering guarantees. The trick lies in finding ways to do all the normal filesystem operations atomically, such that nodes can all manipulate the filesystem safely without needing to care about coordinating their actions. (I'd imagine that doing space allocation safely would be an interesting challenge, for example.) This isn't a new idea, and work has been done in this area already.
Once you've got that, an interesting exercise would be to extend it to use by multiple cooperating processes on a single machine; this might be of use for RMoX, or for communication between multiple virtual servers on the same hardware.