A more functional mouse
If I were to watch myself using a computer mouse for an hour and transcribe my actions into pseudocode (which I should actually do), it would look like focusWindow(Slack); focusChannel(channel.name); focusWindow(Chrome); goTo(url);. Very imperative code, low abstraction. Low leverage. Just a huge list of functions that sometimes take a parameter. Moreover, you cannot, in general, do anything with the result of your action, or manipulate your action itself as an object, or recurse. You don’t know what’s going to happen until you irreversibly click. You can’t edit your moves for precision. You can’t do for each of a set. You can’t build up little pieces into reusable structures that make it easier to think on a higher level the next time. If I knew Haskell or whatever maybe I’d be able to say something like “a mouse should be a monad!” lol. I so wish I could talk about this formally. But for ten years I’ve been thinking that a good test case would just be: I should be able to select my mouse cursor. If I can figure that out then it at least forces me to confront some crucial questions.
I think I first thought about this in like 2017 in the Kensho 1WTC kitchen after I heard someone talking about “differentiable programs” and didn’t understand what it meant but thought about, well, mouse input is smooth, but the actions we take with it aren’t, how could you smooth all that out and put it some good composable microfoundations…
Experiments:
- Drawing — scratches my longstanding itch for an interface (any interface) that lets you select a mouse cursor
To do:
- Rotation
- Use active everywhere instead of .at(-1)
- Pick color
- Draw
- Controlled cursors
- Stable offset (no jump when you switch cursors)
- Synthetic mouse events
- “Peek” — hold your cursor in place while you look at something else, release and it snaps back
- Delete cursors
- Fix selecting right-to-left or bottom-to-top (should it be an inverted selection then? lol)
- Controlled cursors should be able to control cursors, of course…
Longer term:
- Solid keyboard paradigm for chording — onscreen feedback, capslock to lock any key’s action in place
- Multitouch
- History (select and drag and delete and adjust past mouse actions)
- Endogenize action of creating new cursor scope
- Recursion
- Forces
- Feedforward
- Essay
Little touches:
- Rotate, path-dependency? (Is that desirable?)
- Little animation when selecting a different cursor
- Angle into the screen on press
- Pulse to pop out of the cursor scope
Links:
- https://observablehq.com/d/1daf37b6a44339ca
- https://www.figma.com/design/WbKOOaolk8ANmhhqb4gdeH/Functional-mouse?node-id=0-1&node-type=canvas&t=a36Du4W6Qi3XSUrJ-0
- Notes from I&S unconf session https://www.tldraw.com/r/r971mxT4zJmktIpXkqqst?d=v-2536.-748.9742.5470.page
- https://eschersket.ch/ (via Visnu)
- https://www.todepond.com/lab/tool/
Questions:
- What’s the role of the keyboard? Feels unidiomatic…
- Do I need a virtual mouse coordinate space?
- A button is obviously not robust to translation… is there a smoother differentiable
- How could this possibly work for anything other than drawing?
- For this whole another, do I really wanna be selecting a mouse pointer, or is something more like selecting mouse actions?
- The problem where “to select a mouse pointer you have to be on top of it i.e. synonymous” seems to reveal deeper confusion in my whole mission
- What is a click?
- Has all of this already been tried at the outset of the mouse?
- What’re some other good tests?
- Maybe the mission needs relative and discrete coordinate spaces. Like, if I am selecting something in a list, and I want to repeat it for every item in the list, I do not really care about the height of the rendering of the list item, I care about how many items…
- What’s the eqivalent of a simple hello-world recursive factorial function? Or what’s a good example of functions as first-class objects?
- Can there be negative selections if you drag right to left?
- Is it possible to either do without clicks or ground them on firmer more generalizable microfoundations?
- Did the multitouch smartphone (with no distinction between pointing and pressing, i.e. no hover) set back the development of the mouse?
- What is a “parameter” in the context of a mouse? Is it like Excel with absolute and relative references? You need a parameter to make a function lol. Excel gets away with “abstraction by example” instead of “abtraction by holes”; maybe that’s promising. But also, how would you poke a hole? It’s like, do a mouse move, then look at history, then (instead of editing or fixing the past) delete something to leave the remaining actions underspecified and therefore more functional.
- I move from (0,0) to (10,10) and back to (5,7). If I look at history and drag the second point to (6,6), does the last point end up (1,3) or what?
Tests (use cases):
- Mouse y combinator: achieve recursive mouse actions without names.
- “Spread”; do something for every item in a list
- Umm some precision operation
- Assign a gesture (an action, a stroke) to a spot you can click on, select, enact
- Draw a Sierpinski triangle in one go (and precisely)