Feeling and Seeing: Issues in Force Display

Margaret Minsky, Ming Ouh-young, Oliver Steele, Frederick P. Brooks, Jr., and Max Behensky

Proceedings of the 1990 ACM Symposium on Interactive 3D Graphics (SI3D '90), pp. 235–243

Haptics Force display Control theory HCI Virtual environments

TL;DR. In 1990, the authors built a two-axis joystick that let you feel virtual sandpaper. Part I introduces the trick that made it work: compute forces from the slope of a height map. Part II asks why it sometimes didn't work and uses control theory to derive a simple stability rule: stiffness × sampling delay must stay below about twice the viscosity. Along the way: why 100 Hz can feel identical to 1000 Hz, why your arm is stable pushing forward but buzzes pushing sideways, and how audible hardware oscillation exposed the instability.

Touch adds a missing dimension to virtual environments

In 1990, research in virtual environments was almost entirely about what you could see and hear. This paper makes the case that the next dimension of presence is what you can feel. Contact conveys the rich signature of a surface's texture, stiffness, and give.

Force display is especially useful for communicating surface texture and bulk properties of objects and environments as well as dynamics of objects. In this way force display technologies augment the strengths of computer-generated graphics and sound in creating convincingly realistic environments. — p. 235

The authors built Sandpaper: an on-screen palette of textured patches you could stroke with a two-axis force-feedback joystick (a retrofitted arcade controller from Atari). The pitch was simple: computer graphics had spent a decade learning to synthesize convincing visual texture; could the same be done for touch?

Part I · The gradient trick

Sandpaper uses a simple idea. Represent the surface as a one-dimensional height map h(x). When the user's joystick is at position x, compute the local slope dh/dx and push back with a horizontal force proportional to it:

Fig. 1 · Interactive gradient technique

Drag on the surface below. The red arrow is the force the motor produces. It points down the slope, stronger where the bump is steeper. The small trace at the top shows the force history as you stroke, which is exactly what your arm integrates into a perception of "texture."

Bump amplitude 28 Bump spacing 4 Force gain A 90

Recreated after Figure 2 of the paper. The motor only pushes left or right with the right strength; your brain synthesizes the bump.

This is a key insight in the paper. You don't need to physically stop the joystick at the peak of a bump; you only need to oppose uphill motion and release downhill motion. A sufficiently fast servo loop does the rest, and the user's sensorimotor system fills in a convincing mental model of a surface it never actually touched.

Fig. 1b · The 2D gradient field

Lift the trick into two dimensions and it's the same idea: at every point on a virtual surface h(x,y), the motor pushes with force −∇h. Pick a texture, drag the puck, and watch the arrow rotate through the gradient field. Each of these patterns is directly inspired by Sandpaper's texture palette.

Texture Amplitude 20 Spatial frequency 5 Force gain 90

Height is drawn as brightness (bright = peak, dark = valley); the red arrow is the force the motor applies, proportional to the local gradient. Notice that sandpaper's arrow jitters wildly as you move even a pixel. That high-frequency content is what your hand reads as "roughness." The grooves pattern, by contrast, only pushes across the grooves; slide along them and you feel almost nothing.

Pilot studies

Subjects were given several virtual patches, masked from view, and asked to order them by roughness. They could do so consistently. Varying amplitude and groove spacing produced orderings that matched the authors' intuitions about what the parameters "ought" to feel like. The pilot supported continued testing but was not conclusive.

Part II · Sampling delay can destabilize the illusion

The unpleasant surprise of building a force display is that it doesn't always behave. Push against a simulated hard wall and the joystick may start to buzz. This high-frequency oscillation destroys the illusion and can strain the user's wrist. Part II develops a control-theoretic explanation, and it generalizes to a simple rule.

Fig. 2 · The stability threshold — with sound

Stable operation requires sampling period T < 2B/K. Move the sliders below the threshold and the system buzzes. Tap Hear it to synthesize the actual oscillation frequency at your current settings. (Turn your volume down first.)

Stiffness K (N/m) 2773 Viscosity B (N·s/m) 10.0 Sampling rate (Hz) 1000

The paper's exact "hard surface" values: K = 2773, B(radial) = 10, B(tangential) = 1.1. Try both. Raising B from 1.1 to 10 allows roughly 10× more delay.

Fig. 2b · Stability region and phase portrait

Two views of the same system, driven by the sliders above. Left: the whole (rate, K) plane at your current viscosity, colored by whether the system is stable. Right: a live simulation of the joystick hitting the wall. The trajectory in (position, velocity) spirals to the origin when stable, and winds outward into a limit cycle when unstable.

Stability region (rate × stiffness)

Green = T < 2B/K (stable); red = buzz. The dot is your current operating point. The three labeled markers reproduce the paper's experimental conditions at K = 2773: 1 kHz (stable), 250 Hz (unstable), and 100 Hz + damping (stable). Drag the B slider to shift the boundary.

Phase portrait (position × velocity)

A zero-order-hold servo loop at the current rate, with a virtual wall at x = 0. The trajectory fades over time; stable regimes converge to the origin, unstable ones settle into a closed orbit whose size is the perceived buzz amplitude.

Both panels share the K, B, and sampling-rate sliders of Fig. 2. Push the rate below ≈ 2B/(K·T*) and you'll see the dot cross into the red region and the phase trajectory stop spiralling inward.

The counter-intuitive sampling result

Because T* ∝ B/K, you can trade sampling rate for viscosity. The paper reports that 100 Hz with added damping felt indistinguishable from 1000 Hz. In contrast, 250 Hz without damping was flatly unstable. The balance between rate and damping matters more than the rate alone.

Fig. 3 · The three experimental conditions

1000 Hz

stable

Baseline: the short delay remains stable across the viscosity values shown in the experiment.

250 Hz

unstable

4× longer delay; the wall buzzes. Users feel and hear the oscillation.

100 Hz + viscosity

stable

10× longer delay, damped. Subjects cannot tell it from the 1000 Hz condition.

Engineering consequence

Sample less. Damp more.

The same perceptual result, ⅒ the update budget.

Arm viscosity differs by direction

Every one of the 13 test subjects was stable when pushing the same joystick and virtual wall forward and back, and every one was unstable when moving side to side. The reason is mechanical: your arm's viscosity is about 9× higher along its length than across it. The threshold T* = 2B/K cuts differently in each direction.

Fig. 4 · Arm dynamics along two axes

From the paper's Table (p. 242): B(radial) ≈ 10, B(tangential) ≈ 1.1 N·s/m. At K = 2773, the stable period drops from about 7 ms (radial) to about 0.8 ms (tangential). 13 of 13 subjects match the prediction exactly.

Try the task of letting your hand behave like a piece of paper encountering a striking stick. Although one can see the coming stick by its trajectory, and feel its contact with the skin, it is impossible for one to make one's hand act like a piece of paper. — p. 242

Two puzzles about the human arm

The analysis raises two immediate questions. First, if the neural feedback loop through your brain takes ~200 ms, how does your arm ever stabilize contact at all? The answer, from Hogan, is that a relaxed arm behaves as a passive mechanical object with roughly fixed impedance over windows as long as 1.2 seconds. The nervous system is slow, but the muscles and tendons form a fast analog low-pass filter that stabilizes contact passively.

Second, if 1000 Hz is well past the bandwidth of kinesthetic perception, why do users feel a difference between 500 Hz and 1000 Hz updates? Because the skin's cutaneous receptors respond well past 400 Hz. At lower sampling rates, the tiny unstable oscillations, even when subthreshold for the arm, can be heard through the fingertip as a buzz.

Sandpaper grew out of MIT and UNC haptics

Sandpaper was born from an unusual marriage of institutions. Margaret Minsky was a graduate student at the MIT Media Lab working with Brand and Sutherland's Put That There lineage; Ming Ouh-young, Fred Brooks, and others at UNC Chapel Hill had spent years on GROPE, a haptic system for molecular docking that used a multi-thousand-dollar Argonne remote manipulator. The Sandpaper group asked the opposite question: what can you build with lower-cost hardware?

1965Sutherland's "The Ultimate Display" proposes that one day we will walk into simulated rooms and touch simulated objects.
1967Project GROPE begins at UNC under Fred Brooks, starting decades of work on force feedback for chemistry.
1983Atari releases an arcade joystick with two DC motors and position encoders. A few years later, surplus units start reaching research labs.
1989–90The Atari joystick is rewired for servo control; Margaret Minsky's Sandpaper software runs on top. The first subjects sort virtual sandpapers by roughness. The wall-contact condition starts buzzing. The theory section is written to explain why.
1990Paper appears at SI3D '90. Margaret later completes her PhD thesis Computational Haptics (1995) extending the gradient technique to 2D.
1993SensAble Technologies spins out of MIT, commercializing the Phantom, a 3-DOF force-feedback stylus that becomes the default device for the next two decades of haptics research.

The paper's lasting contributions

Gradient-based texture: a technique for force display of surfaces that remains in use in modern haptic APIs.
T ≈ 2B/K: a back-of-envelope stability test for any force-feedback system.
Exploratory motion matters: texture perception depends on active stroking; interfaces have to allow and reward it.
Multisensory reinforcement: the illusion holds under imperfect force rendering if vision and sound agree.
From parameters to percepts: the paper calls for roughness, softness, and stickiness as first-class descriptors in place of stiffness and bump height.

We did not use all the theories in designing our first systems. When problems came one by one, we realized that an analysis would be useful. — p. 242

The same principles still guide haptics

Thirty-five years is a long time in hardware and a short time in perception. Here is what the field did with these ideas:

The Phantom & kinesthetic haptics

SensAble's Phantom stylus (1993–) made 3-DOF force feedback a lab standard. Its rendering loop uses exactly the T < 2B/K envelope; its surface textures are gradient fields.

Game controllers & rumble

The 1997 Rumble Pak and today's DualSense adaptive triggers descend from the same realization that vibrotactile buzz and programmed stiffness shape perception far beyond what the actuators "realistically" simulate.

Apple's Taptic Engine

The iPhone's "click" is an illusion: a linear resonant actuator pulsed in a waveform tuned to the skin's cutaneous bandwidth. Same gradient-trick logic, translated from 2-axis to 1-dimensional time.

Surgical & training sims

Laparoscopic trainers, dental simulators, and needle-insertion systems depend on Part II's stability envelope. Running a virtual wall stiff enough to feel "hard" without buzzing is still the hardest part.

VR controllers

Quest and Vision Pro controllers are kinesthetic-poor (mostly vibrotactile), which is why "picking up" a virtual object still feels wrong. The research problem Sandpaper opened remains unsolved: low-cost force feedback that convincingly reproduces contact.

Ultrasonic mid-air haptics

Ultraleap and others project focused ultrasound onto the hand to create touch without contact. The force fields are tiny, which pushes designers straight back to the gradient principle: synthesize percepts, not physics.

Read the paper

Download the PDF

Citation

Minsky, M., Ouh-young, M., Steele, O., Brooks, F. P., & Behensky, M. (1990). Feeling and seeing: issues in force display. In Proceedings of the 1990 Symposium on Interactive 3D Graphics (SI3D '90) (pp. 235–243). ACM.

Key references

Hogan, N. (1989). Controlling Impedance at the Man/Machine Interface. IEEE Int. Conf. on Robotics & Automation. Source of the 1.2-second arm-impedance timescale.
Lederman, S., & Klatzky, R. (1987). Hand movements: A window into haptic object recognition. Cognitive Psychology, 19(3). Source of the "lateral motion → texture" exploratory procedure.
Colgate, J. E., & Hogan, N. (1989). An analysis of contact instability in terms of passive physical equivalents. IEEE Int. Conf. on Robotics & Automation.
Ouh-young, M., et al. (1988). Using a manipulator for force display in molecular docking. IEEE Int. Conf. on Robotics & Automation. The sibling GROPE application at UNC.
Sutherland, I. E. (1965). The ultimate display. Proc. IFIP Congress. The founding vision this paper is working toward.
Minsky, M. (1995). Computational Haptics: The Sandpaper System for Synthesizing Texture for a Force-Feedback Display. PhD thesis, MIT. The full-length follow-up.