Artificial Life Cinematography
Demetri
Terzopoulos
New York University
Introduction
Computer animation is a fascinating discipline at the intersection of art, science, and technology. It is concerned with the challenge of imparting liveliness to synthetic objects in virtual worlds represented by computer. Audiences everywhere are delighted by state-of-the-art computer animation effects of the sort featured in the modern classic “Jurassic Park” (Universal Pictures, 1993) or in the recent computer animation milestones “Monsters, Inc.” (Walt Disney Productions/Pixar, 2001) and “Final Fantasy: The Spirits Within” (Square USA, 2001). As with traditional cell animation, it is easy to suspend disbelief and imagine that the graphical characters portrayed on screen are more or less alive. However, these characters are by no means living beings. They certainly have no inherent intelligence. In fact, they are hardly autonomous. Rather, these “graphical puppets” must be laboriously hand animated by highly skilled human animators or programmed to mundanely repeat recorded motions deliberately performed by real-life actors under controlled conditions.
In this article, I present a new breed of self-animating graphical characters that largely circumvent the drudgery of manual character animation. Self-animating characters are a form of leading edge, Artificial Life CG technology [1]. “Artificial Life” is an emerging scientific field that is concerned with the computer modeling of phenomena associated with natural, biological life [2]. Our artificial life research has confronted the scientific challenge of developing realistic artificial animals endowed with functional bodies and brains. This research has yielded simulated physical worlds inhabited by sophisticated autonomous agents in the form of graphical characters that are autonomous, intelligent and, at least in some very rudimentary sense, “alive”.
Artificial Fishes
We have devoted significant effort to creating aquatic artificial animals [3][4]. Imagine a virtual marine world inhabited by a variety of self-animating fishes (Figure 1). In the presence of underwater currents, the fishes employ their muscles and fins to swim gracefully around immobile obstacles and among moving aquatic plants and other fishes. They autonomously explore their dynamic world in search of food. Large, hungry predator fishes stalk smaller prey fishes in the deceptively peaceful habitat. Prey fishes swim around contentedly, until the sight of predators compels them to take evasive action. When a predator shark appears in the distance, similar species of prey form schools to improve their chances of survival. As the predator nears a school, the prey fish scatter in terror. A chase ensues in which the predator selects victims and consumes them until satiated. Some species of fishes are untroubled by predators. They find comfortable niches and feed on floating plankton when hungry. Driven by healthy libidos, they perform intricate courtship rituals to attract mates.
|
|
|
|
(a) |
|
|
|
|
|
(b) |
(c) |
|
Figure 1. Artificial fishes in their virtual marine world as it appears to an underwater observer. (a) The three reddish fish are engaged in mating behavior while the other fish are foraging among seaweeds. (b) A school of fish appears in the distance. (c) A predator shark stalking prey. |
Animation as Artificial Life Cinematography
Our artificial life approach to computer animation has led to the production of two computer-animated short subjects, essentially mini-documentaries about the virtual marine world of artificial fishes, that have been screened internationally before large audiences [5][6]. The creative process underlying these animations has the following distinguishing feature: Rather than being a graphical character puppeteer, the computer animator engages in a creative process analogous to that of an underwater nature cinematographer. The animator strategically immerses and positions one or more virtual cameras within the virtual marine world so as to capture interesting “film footage” of the behaviors of artificial fishes. The footage is edited, assembled, and narrated to produce the final documentary. This creative process is fundamentally similar to that associated with the fascinating genre of marine life documentaries for which the Cousteau Society or the National Geographic Society are famous.
In the remainder of the article, I will review the artificial fish models that have made possible our artificial life cinematography approach to computer animation. The comprehensive modeling methodology, in which we model the form, appearance, and basic physics of the animal and its habitat, as well as the animal’s means of locomotion, its perceptual awareness of its world, its behavior, and its ability to learn, has been described in detail elsewhere [3]. Our methodology is generally applicable to the modeling of all sorts of animals, including humans, for use in computer animation.
Functional Modeling of Fish
We have developed a functional model certain species of (teleost) fishes that can automatically animate itself with considerable realism. The artificial fish is an autonomous agent with a realistic deformable body actuated by internal muscles, with eyes, and with a brain that includes motor, perception, behavior, and learning centers. Figure 2 presents a schematic of the functional model and this section reviews its main functional components.
Display Model
First, we want our artificial fishes to capture the form and appearance of a variety of natural fishes with reasonable visual fidelity. To this end, digitized photographs of real fish, such as the images shown in Figure 4(a), are converted into three-dimensional spline surface body models (Figure 4(b)) with the help of interactive image analysis tools, and the image texture is mapped onto the surfaces to produce the final textured geometric display models of the fishes (Figure 4(c)).
|
|
|
|
(a) |
(b) |
|
Figure 3. Biomechanical fish model (a). To emulate a piscine body structure, 23 lumped masses, shown as black dots, are interconnected by 91 viscoelastic elements, shown as lines, each comprising an elastic spring and a viscous damper connected in parallel. Twelve of the elements, depicted as orange lines, serve as contractile muscle actuators. The dotted curves represent functional pectoral fins. Through coordinated muscle and fin actions triggered by the motor center of the artificial fish's brain, this simple physics-based model synthesizes realistic piscine locomotion in virtual fluid. Hydrodynamic locomotion (b). With the tail swinging to the left, the thrust on any point on the body acts opposite to the surface normal at that point. The forward thrust component propels the fish through the simulated water. |
|
Figure 2. Functional artificial fish model. The piscine body harbors a biomechanical
model and a brain with motor, perception, behavior, and learning centers. (The perceptual attention module is marked
“!”.) |
|
(a) (b) (c) Figure 4. From digitized images of fishes (a) to spline surface body models (b) to three-dimensional, textured fish display models (c). |
Biomechanical Model and Locomotion
The artificial fish captures not just the 3D form and appearance of real fishes, but also the basic physics of these animals in their environment. The motor system of the artificial fish (Figure 2) comprises a piscine biomechanical model, including muscle actuators and a set of motor controllers. Figure 3(a) illustrates the mechanical body model, which produces realistic piscine locomotion using only 23 lumped masses and 87 elastic elements. These mechanical components, whose dimensions and physical parameters are modified to model different fishes, are interconnected to maintain the structural integrity of the body as it flexes due to the action of its 12 contractile muscles.
The artificial fish locomotes like real fishes do, by autonomously contracting its muscles. As the body flexes it displaces virtual fluid, which induces local reaction forces normal to the body. These hydrodynamic forces generate thrust, primarily via the caudal fin, which propels the fish forward (Figure 3(b)). The dynamics of the biomechanical model are governed by a system of coupled second-order ordinary differential equations driven by the hydrodynamic forces. A numerical simulator continually integrates these equations of motion forward through time. The biomechanical model achieves a good compromise between realism and computational efficiency, permitting the real-time simulation of fish locomotion.
The motor controllers (Figure 2) coordinate muscle actions to carry out specific motor functions, such as swimming forward, turning left and right, ascending and descending in the water. They translate natural control parameters such as the forward speed or angle of the turn into detailed muscle actions that execute the function. The artificial fish is neutrally buoyant in the virtual water. Its two pectoral fins enable it to navigate freely in its three dimensional world by pitching, rolling, and yawing its body, as well as to stabilize the body during locomotion. Specialized motor controllers coordinate the pectoral fin actions.
Learning
The learning center of its brain (Figure 2) enables the artificial fish to learn how to locomote through practice and sensory reinforcement. Through optimization, the motor learning algorithms discover muscle controllers that produce efficient locomotion. Muscle contractions that produce forward movements are “remembered’’. These partial successes then form the basis for subsequent improvements in swimming technique. Their brain’s learning center also enables the artificial fishes to train themselves to accomplish higher level sensorimotor tasks, such as maneuvering to reach a visible target or learning more complex motor skills (see [7] for the details).
Perception
Artificial fishes are aware of their world through sensory perception. Their perception system relies on a set of on-board virtual sensors to gather sensory information about the dynamic environment. It is necessary to model not only the abilities but also the limitations of animal perception systems in order to achieve natural sensorimotor behaviors. Artificial fishes perceive objects within a limited field view if objects are close enough and not occluded by other opaque objects. The perception center of the artificial fish brain includes a perceptual attention mechanism (indicated by “!” in Figure 2), which enables the artificial fish to sense the world in a task-specific way, hence filtering out sensory information superfluous to its current behavioral needs. For example, while foraging, the artificial fish attends to sensory information about nearby food sources. Reference [8] describes a biomimetic approach to perception.
Behavior
The behavior center of the artificial fish’s brain mediates between its perception system and its motor system (see Figure 2). A set of pre-specified, innate characteristics determine whether the fish is male or female, predator or prey, etc. The behavioral repertoire of the artificial fish comprises a set of behavior routines arranged in a loose hierarchy. These include primitive, reflexive behavior routines, such as obstacle avoidance, as well as more sophisticated motivational behavior routines, such as schooling and mating. An action selection mechanism underlies the goal-directed behavior of the artificial fish in its dynamic world. The action selection mechanism controls the perceptual attention mechanism. Action selection takes into account the innate characteristics of the fish, its mental state as represented by hunger, fear, and libido mental variables, and the incoming stream of sensory information, in order to generate dynamic goals for the artificial fish, such as to avoid an obstacle, to hunt and feed on prey, or to court a potential mate. Exploiting a single-item memory, the action selection mechanism ensures that goals have enough persistence to supports sustained behaviors such as foraging, schooling, and mating.
Conclusion
The science of artificial life can contribute profoundly to the art of computer animation. I described a virtual marine world inhabited by artificial life forms that emulate the appearance, motion, and behavior of marine animals in their natural habitats. Each artificial animal is an autonomous agent in a simulated physical world. It has (i) a three-dimensional body with internal muscle actuators and functional fins that deforms and locomotes in accordance with the principles of biomechanics and hydrodynamics, (ii) sensors, including eyes that can perceive the environment, and (iii) a brain with motor, perception, behavior, and learning centers. Artificial fishes exhibit a repertoire of piscine behaviors that rely on their perceptual awareness of their dynamic habitat. Furthermore, they can learn to locomote through practice and sensory reinforcement.
Our novel approach has enabled us to produce realistic computer animation of natural environments in which the animator plays a role akin to that of a nature cinematographer. In our animated productions, the detailed motions of the artificial fishes emulate the complexity and unpredictability of movement of their natural counterparts, which enhances the visual beauty of the animations.
Our artificial life modeling methodology is broadly applicable to the challenge of realistically modeling animals other than fishes—most interestingly, humans. To this end, cognitive modeling, a logic-based artificial intelligence technique that supports knowledge representation, reasoning, and planning, enables the creation of smart graphical characters that know enough about themselves and their world that they may be directed at an abstract level, more like real human actors [9].
Acknowledgements
I thank my former students Radek Grzeszczuk, Tamer Rabie, and especially Xiaoyuan Tu, a scientist with extraordinary artistic talent who developed the artificial fishes model, for their outstanding contributions to the work reviewed herein.