Feeling and Seeing: Issues in Force Display

Margaret Minsky, Ming Ouh-young, Oliver Steele, Frederick P. Brooks, Jr., and Max Behensky

Proceedings of the 1990 ACM Symposium on Interactive 3D Graphics (SI3D '90), pp. 235–243

Haptics Force display Control theory HCI Virtual environments

TL;DR. In 1990, the authors built a two-axis joystick that let you feel virtual sandpaper. Part I introduces a trick — compute forces from the slope of a height map — that made it work. Part II asks why it sometimes didn't work, and uses control theory to derive a simple stability rule: stiffness × sampling delay must stay below about twice the viscosity. Along the way: why 100 Hz can feel identical to 1000 Hz, why your arm is stable pushing forward but buzzes pushing sideways, and why none of this was obvious until the hardware started screaming.

Why force display?

In 1990, research in virtual environments was almost entirely about what you could see and hear. This paper makes the case that the next dimension of presence is what you can feel — not just contact events, but the rich signature of a surface's texture, stiffness, and give.

Force display is especially useful for communicating surface texture and bulk properties of objects and environments as well as dynamics of objects. In this way force display technologies augment the strengths of computer-generated graphics and sound in creating convincingly realistic environments. — p. 235

The authors built Sandpaper: an on-screen palette of textured patches you could stroke with a two-axis force-feedback joystick (a retrofitted arcade controller from Atari). The pitch was simple — computer graphics had spent a decade learning to synthesize convincing visual texture; could the same be done for touch?

Part I · The gradient trick

The central idea of Sandpaper is almost embarrassingly simple. Represent the surface as a one-dimensional height map h(x). When the user's joystick is at position x, compute the local slope dh/dx and push back with a horizontal force proportional to it:

Fig. 1 · Interactive gradient technique

Drag on the surface below. The red arrow is the force the motor produces — pointing down the slope, stronger where the bump is steeper. The small trace at the top shows the force history as you stroke, which is exactly what your arm integrates into a perception of "texture."

Bump amplitude 28 Bump spacing 4 Force gain A 90

Recreated after Figure 2 of the paper. The motor never tells your hand "there's a bump here" — it only pushes left or right with the right strength, and your brain synthesizes the bump.

This is the cleanest insight in the paper. You don't need to physically stop the joystick at the peak of a bump; you only need to oppose uphill motion and release downhill motion. A sufficiently fast servo loop does the rest, and the user's sensorimotor system fills in a remarkably convincing mental model of a surface it never actually touched.

Fig. 1b · The 2D gradient field

Lift the trick into two dimensions and it's the same idea: at every point on a virtual surface h(x,y), the motor pushes with force −∇h. Pick a texture, drag the puck, and watch the arrow rotate through the gradient field. Each of these patterns is directly inspired by Sandpaper's texture palette.

Texture Amplitude 20 Spatial frequency 5 Force gain 90

Height is drawn as brightness (bright = peak, dark = valley); the red arrow is the force the motor applies, proportional to the local gradient. Notice that sandpaper's arrow jitters wildly as you move even a pixel — that's the high-frequency content your hand reads as "roughness." The grooves pattern, by contrast, only pushes across the grooves; slide along them and you feel almost nothing.

Pilot studies

Subjects were given several virtual patches, masked from view, and asked to order them by roughness. They could — consistently. Varying amplitude and groove-spacing produced orderings that matched the authors' intuitions about what the parameters "ought" to feel like. Suggestive rather than conclusive, but enough to justify the architecture.

Part II · When illusions break

The unpleasant surprise of building a force display is that it doesn't always behave. Push against a simulated hard wall and the joystick may start to buzz — a high-frequency oscillation that destroys the illusion and, occasionally, your wrist. Part II develops a control-theoretic explanation, and it generalizes to a simple rule.

Fig. 2 · The stability threshold — with sound

Stable operation requires sampling period T < 2B/K. Move the sliders below the threshold and the system buzzes — tap Hear it to synthesize the actual oscillation frequency at your current settings. (Turn your volume down first.)

Stiffness K (N/m) 2773 Viscosity B (N·s/m) 10.0 Sampling rate (Hz) 1000

The paper's exact "hard surface" values: K = 2773, B(radial) = 10, B(tangential) = 1.1. Try both — pushing B from 1.1 to 10 buys you roughly 10× more allowed delay.

Fig. 2b · Stability region and phase portrait

Two views of the same system, driven by the sliders above. Left: the whole (rate, K) plane at your current viscosity, coloured by whether the system is stable. Right: a live simulation of the joystick hitting the wall — the trajectory in (position, velocity) spirals to the origin when stable, and winds outward into a limit cycle when unstable.

Stability region (rate × stiffness)

Green = T < 2B/K (stable); red = buzz. The dot is your current operating point. The three labelled markers reproduce the paper's experimental conditions at K = 2773: 1 kHz (stable), 250 Hz (unstable), and 100 Hz + damping (stable). Drag the B slider — the boundary shifts.

Phase portrait (position × velocity)

A zero-order-hold servo loop at the current rate, with a virtual wall at x = 0. The trajectory fades over time; stable regimes converge to the origin, unstable ones settle into a closed orbit whose size is the perceived buzz amplitude.

Both panels share the K, B, and sampling-rate sliders of Fig. 2. Push the rate below ≈ 2B/(K·T*) and you'll see the dot cross into the red region and the phase trajectory stop spiralling inward.

The counter-intuitive sampling result

Because T* ∝ B/K, you can trade sampling rate for viscosity. The paper reports that 100 Hz with added damping felt indistinguishable from 1000 Hz — while 250 Hz without damping was flatly unstable. Faster is not always better; better-balanced is better.

Fig. 3 · The three experimental conditions

1000 Hz

stable

Baseline: delay is small enough that any reasonable viscosity suffices.

250 Hz

unstable

4× longer delay; the wall buzzes. Users feel — and hear — the oscillation.

100 Hz + viscosity

stable

10× longer delay, damped. Subjects cannot tell it from the 1000 Hz condition.

Why it matters

Sample less. Damp more.

The same perceptual result, ⅒ the update budget.

Radial vs. tangential: why your arm has opinions

The same hardware, the same software, the same wall — and yet every one of the 13 test subjects was stable when pushing the joystick forward and back, and every one was unstable when moving side to side. The reason is mechanical: your arm's viscosity is about 9× higher along its length than across it. The threshold T* = 2B/K cuts differently in each direction.

Fig. 4 · Arm dynamics along two axes

From the paper's Table (p. 242): B(radial) ≈ 10, B(tangential) ≈ 1.1 N·s/m. At K = 2773, the stable period drops from about 7 ms (radial) to about 0.8 ms (tangential). 13 of 13 subjects match the prediction exactly.

Try the task of letting your hand behave like a piece of paper encountering a striking stick. Although one can see the coming stick by its trajectory, and feel its contact with the skin, it is impossible for one to make one's hand act like a piece of paper. — p. 242

Two puzzles about the human arm

The analysis raises two immediate questions. First, if the neural feedback loop through your brain takes ~200 ms, how does your arm ever stabilize contact at all? The answer, from Hogan, is that a relaxed arm behaves as a passive mechanical object with roughly fixed impedance over windows as long as 1.2 seconds — the nervous system is slow, but the muscles-and-tendons are a fast analog low-pass filter that does the stabilizing for free.

Second, if 1000 Hz is well past the bandwidth of kinesthetic perception, why do users feel a difference between 500 Hz and 1000 Hz updates? Because the skin's cutaneous receptors respond well past 400 Hz — and at lower sampling rates the tiny unstable oscillations, even when subthreshold for the arm, can be heard through the fingertip as a buzz.

How this was actually built

Sandpaper was born from an unusual marriage of institutions. Margaret Minsky was a graduate student at the MIT Media Lab working with Brand and Sutherland's Put That There lineage; Ming Ouh-young, Fred Brooks, and others at UNC Chapel Hill had spent years on GROPE, a haptic system for molecular docking that used a multi-thousand-dollar Argonne remote manipulator. The Sandpaper group asked the opposite question: what can you build with something cheap?

1965Sutherland's "The Ultimate Display" proposes that one day we will walk into simulated rooms and touch simulated objects.
1967Project GROPE begins at UNC under Fred Brooks — decades of work on force feedback for chemistry.
1983Atari releases an arcade joystick with two DC motors and position encoders. A few years later, surplus units start reaching research labs.
1989–90The Atari joystick is rewired for servo control; Margaret Minsky's Sandpaper software runs on top. The first subjects sort virtual sandpapers by roughness. The wall-contact condition starts buzzing. The theory section is written to explain why.
1990Paper appears at SI3D '90. Margaret later completes her PhD thesis Computational Haptics (1995) extending the gradient technique to 2D.
1993SensAble Technologies spins out of MIT, commercializing the Phantom — a 3-DOF force-feedback stylus that becomes the default device for the next two decades of haptics research.

What this paper got right

Gradient-based texture — the canonical technique for force display of surfaces, still used in modern haptic APIs.
T ≈ 2B/K — a back-of-envelope stability test you can run on any force-feedback system.
Exploratory motion matters — texture perception is not passive contact but active stroking; interfaces have to allow and reward it.
Multi-sensory reinforcement — the illusion holds even under imperfect force rendering if vision and sound agree.
From parameters to percepts — the call to move from stiffness and bump-height to roughness, softness, and stickiness as first-class descriptors.

We did not use all the theories in designing our first systems. When problems came one by one, we realized that an analysis would be useful. — p. 242

What happened next (and what's still true)

Thirty-five years is a long time in hardware and a short time in perception. Here is what the field did with these ideas:

The Phantom & kinesthetic haptics

SensAble's Phantom stylus (1993–) made 3-DOF force feedback a lab standard. Its rendering loop uses exactly the T < 2B/K envelope; its surface textures are gradient fields.

Game controllers & rumble

The 1997 Rumble Pak and today's DualSense adaptive triggers descend from the same realization — vibrotactile buzz and programmed stiffness shape perception far beyond what the actuators "realistically" simulate.

Apple's Taptic Engine

The iPhone's "click" is an illusion: a linear resonant actuator pulsed in a waveform tuned to the skin's cutaneous bandwidth. Same gradient-trick logic, translated from 2-axis to 1-dimensional time.

Surgical & training sims

Laparoscopic trainers, dental simulators, and needle-insertion systems depend on Part II's stability envelope. Running a virtual wall stiff enough to feel "hard" without buzzing is still the hardest part.

VR controllers

Quest and Vision Pro controllers are kinesthetic-poor (mostly vibrotactile), which is why "picking up" a virtual object still feels wrong. The research problem Sandpaper opened — cheap, convincing force feedback — is still open.

Ultrasonic mid-air haptics

Ultraleap and others project focused ultrasound onto the hand to create touch without contact. The force fields are tiny, which pushes designers straight back to the gradient principle: synthesize percepts, not physics.

Read the paper

Download the PDF

Citation

Minsky, M., Ouh-young, M., Steele, O., Brooks, F. P., & Behensky, M. (1990). Feeling and seeing: issues in force display. In Proceedings of the 1990 Symposium on Interactive 3D Graphics (SI3D '90) (pp. 235–243). ACM.

Key references

Hogan, N. (1989). Controlling Impedance at the Man/Machine Interface. IEEE Int. Conf. on Robotics & Automation. — Source of the 1.2-second arm-impedance timescale.
Lederman, S., & Klatzky, R. (1987). Hand movements: A window into haptic object recognition. Cognitive Psychology, 19(3). — The "lateral motion → texture" exploratory procedure.
Colgate, J. E., & Hogan, N. (1989). An analysis of contact instability in terms of passive physical equivalents. IEEE Int. Conf. on Robotics & Automation.
Ouh-young, M., et al. (1988). Using a manipulator for force display in molecular docking. IEEE Int. Conf. on Robotics & Automation. — The sibling GROPE application at UNC.
Sutherland, I. E. (1965). The ultimate display. Proc. IFIP Congress. — The founding vision this paper is working toward.
Minsky, M. (1995). Computational Haptics: The Sandpaper System for Synthesizing Texture for a Force-Feedback Display. PhD thesis, MIT. — The full-length follow-up.