Shows by the peripatetic Puerto Rican–born artist Rafael Ferrer have been curiously sparse on the exhibition calendar in recent years, so his recent retrospective—or, as its title would have it, "Retro/Active"—at El Museo del Barrio in New York City was a welcome reminder of the powerful, protean oeuvre he has fashioned in more than half a century of artmaking. Forty years ago his work would have been hard to avoid. In 1969 he participated in three of the signal exhibitions of the new wave of conceptual, postminimal, process-oriented and "antiform" art that would dominate the scene for much of the next decade: "Live in Your Head: When Attitudes Become Form" at the Kunsthalle Bern; "Op Losse Schroeven (Square Pegs in Round Holes)" at the Stedelijk Museum, Amsterdam; and "Anti-Illusion: Procedures/Materials" at the Whitney Museum of American Art. The following year Ferrer had a one-man show at New York’s leading gallery, Leo Castelli, and took part in another now-legendary group show, "Information" at the Museum of Modern Art.

Ferrer’s work from those days was not very prominently displayed at El Museo; however, much of it consisted of highly ephemeral installations of unconventional materials—piles of dead leaves and massive blocks of ice were particular favorites. In "Retro/Active" these were represented by means of a wall of small photographs that had been banished to a side room, although it would have been worth the trouble—a lot of trouble, admittedly—to re-create a couple of these ventures into what Carter Ratcliff has called "a medium that was part sculpture, part theater, part guerilla action on the aesthetic front." By contrast, the more conventional paintings and sculptures Ferrer made in the 1950s and early ’60s, when he was still finding his way as an artist, were fully integrated into curator Deborah Cullen’s thematic (rather than chronological) traversal of Ferrer’s career, giving a possibly misleading sense of its overall shape.

Since the ’70s, Ferrer’s work has been strikingly polymorphous, ranging from sculpture and painting to drawing, collage and books. Just as he was associated with process art at the end of the ’60s, around 1980 he started making figurative paintings that seemed to be right in step with the Neo-Expressionism just then on the horizon. Journalists saw this style of painting as a rejection of the avant-garde of the ’70s; but in its crudity, the new painting was yet another attempt to start again from scratch, just as much of the art of the ’70s had been. More than a few of its protagonists, notably Francesco Clemente and David Salle, had, like Ferrer, first essayed the more conceptual aesthetics of the previous decade. I doubt, though, that their paintings would hold up anywhere near as well as Ferrer’s do today. For all the variousness of his efforts, it would be wrong to see Ferrer as an eclectic artist, one whose work lacks coherence or commitment—at least after his years of youthful experimentation, which admittedly lasted longer than is common these days. Born in 1933, he was already in his late 30s by the time he began making the process-oriented installations that can be considered his first mature works.

Along with several new texts, the catalog for "Retro/Active" reprints a long and searching essay by Ratcliff published in the catalog for an exhibition at the Contemporary Arts Center in Cincinnati in 1973. What’s remarkable about "Rafael Ferrer in the Tropical Sublime" is how descriptive it still seems today of the thinking behind Ferrer’s work, even of the work made over the subsequent thirty-seven years, much of which bears very little overt resemblance to anything the artist was doing back then. This is testimony, of course, to a critical perspicuity on Ratcliff’s part that practically amounts to prescience, but also to the fact that the multifariousness of Ferrer’s art nonetheless manifests a dogged insistence. He has simply taken as many approaches as possible to a few recurrent themes—or perhaps it would be better to say a few recurrent obsessions.

Ratcliff shows, too, that Ferrer’s work was always in fundamental tension with the context it appeared to be part of, in the first instance the new forms of art that developed in New York in the wake of minimalism.Think of Robert Morris, with whom Ferrer was closely linked at this time, or Richard Serra, whose works were only the residue of simple physical manipulations of materials as exemplified in his famous "Verb List Compilation": "to roll, to crease, to fold…to bundle, to heap, to gather, to scatter." Ferrer was undoubtedly concerned with the physical character of the materials he was using and with the visible changes one could see them undergo; but at the same time, as Ratcliff insists, he was always a Romantic with an ingrained faith in the power of symbol, myth and metaphor to give meaningful direction to life through art. In Ratcliff’s view, Ferrer’s use of autumn leaves was an invocation of Johnny Mercer’s sentimental standard of that name and Shakespeare’s Sonnet 73—literary and emotional references that were supposed to be taboo for sophisticated artists of the day, who were fundamentally empiricists, intent (as Ratcliff says) on "the construction of non-symbolic objects with no admitted reference to humanity or the natural world." Ferrer used natural materials with blatant symbolic implications, though with a bluntness that headed off any possibility of sentimentality or overly effusive lyricism.

Other early installations by Ferrer juxtaposed similarly transient materials with more solid, evidently sculptural ones, among them steel, tree trunks and neon lights. He also made self-contained sculptures, often in the form of kayaks and other kinds of boats, and began drawing, in crayon, maps of imaginary places. He was summoning the Romantic myth of the journey, the dream of exploration—which, of course, echoed his own real life, moving between Puerto Rico and the American mainland, not to mention Europe, where he had spent an important period in the 1950s, rubbing shoulders with the Surrealists and engaging in deep dialogue with Wifredo Lam, the Cuban-born painter whose work attempted to synthesize a formal syntax derived from Cubism and Surrealism with an iconography reflecting his culture’s African and Indian roots. Ferrer’s use of crayons shows his attraction to the pictorial expressiveness of children, the untutored, people on the margins—another Romantic theme—and he also used them to design masklike forms on paper bags, something he still does today. This multitude of faces constitutes a marvelous vocabulary of legible forms in which observation and invention become indistinguishable.

What separates Ferrer’s paintings of the 1980s and ’90s from the Neo-Expressionism with which they might have been confused (and which they might have influenced) is a predominance of observation. If the art of the 1960s and ’70s, for all the brilliance of its innovations, was rendered too narrow (and, ultimately, academic) by its extreme empiricism, which ruled out of bounds so much of art’s potential material, then the besetting sin of the art that in the 1980s emerged in reaction against it would have to be an excessive subjectivism, an overindulgence in the cultivation of what Harald Szeemann dubbed "individual mythologies" unchecked by any significant external reality. In an interview with Vincent Katz, Ferrer says, "I know that I am attracted to German Expressionism, Neue Sachlichkeit, Dix, Beckmann," which might seem to underline an affinity for the later Neo-Expressionists as well, but like the early twentieth-century Expressionists and unlike the Neo-Expressionists of the 1980s, he was finding his symbols in an encounter with reality, and one with an immediate political dimension at that: the subaltern.

In 1975, after nearly a decade spent mostly in Philadelphia, Ferrer again began spending much of his time in Puerto Rico; in 1985 he gave up his home there for one in the Dominican Republic. It was in Puerto Rico that Ferrer seriously took up painting, after a visit from Alex Katz suddenly convinced him that painting from observation still had a future; and the subject matter of nearly all his best painting has come from his experience of the islands. This has led to misunderstandings. When a critic referred to his style as "faux primitivism," Ferrer objected that the characterization was based on a prejudice about the people he depicted rather than on his way of painting them. "They can call the people in the paintings natives or they can call them inhabitants of this place or the other, but I call them neighbors."

Actually, some of the first paintings Ferrer made after his return to the medium do betray a certain primitivism. I’m thinking of works like El Cuarteto (The Quartet) or Melida la Reina (Melida the Queen), both from 1981, which almost seem like elaborations of his paper-bag mask fantasies. But by mid-decade his style had become distinctly more sophisticated, settling into a sturdy Modernism that would not have looked outrageous to any of Ferrer’s early twentieth-century heroes but with a personal inflection that could never be confused with anyone else’s. Ferrer’s brush is tough, unsentimental; he prefers to show things bluntly rather than suavely coaxing them into visibility. His pictorial space can seem almost hammered into place—as if an imprint of his work as a sculptor. His use of the word "neighbors" to describe his subjects is quite precise. In painting the people who lived near him in the Dominican Republic, he was painting neither familiars—it is telling that although Ferrer has done self-portraits, he has rarely painted his family or close friends—nor complete strangers. Wariness and curiosity register in the faces of many of Ferrer’s subjects, although others appear more ingenuous. There is no false familiarity here, but rather a distance to be negotiated. And it can be negotiated.

A curious light is shed on Ferrer’s art by learning that as a young man he was as deeply involved in music as in painting and sculpture, playing drums professionally in Latin bands until 1966. One of the exhibition’s thematic sections is devoted to representations of music and musicians, making it clear that his fascination with music never abated. The relation between music and the other arts was one of the great themes of Modernism; in the European tradition, where the musical reference was always to the great tradition of classical concert music, this usually pointed toward the possibility of abstraction. Just as a string quartet needs no literary program, neither does a painting. For Ferrer, the implications are different because the music he has in mind comes from a different tradition. "My instrument is the drum," he explained in 1971. "It minimizes all the other assets that music has, like harmony and tone, and concentrates on the fundamental point of time and the ability to split time in ways that are intricate and inventive, and to do that under pressure, which comes from the fact that musical decisions are made at a high rate of speed—they are literally split-second decisions. So they require intuition and the ability to take chances within a structure which has time as a critical element." Ferrer’s understanding of the significance of music is hardly opposed to its potential for abstractness, but neither is that the main point. Instead, Ferrer calls for an aesthetic of spontaneous responsiveness irrespective of subject (or lack thereof)—an aesthetic just as applicable to the figurative painting he would take up a decade later and the seemingly abstract yet symbolically resonant installations he was making at the time. For that matter, his most inventive and intricate elaborations of the structure of time will undoubtedly turn out to have been his paintings. It’s a pleasure to see them again after so long.

Ferrer’s quasi manifesto for music as the fundamental aesthetic is worth keeping in mind when visiting "Christian Marclay: Festival," at the Whitney (through September 26). The Swiss-American artist, born in 1955, might seem to have very little in common with his Puerto Rican elder, yet Marclay is just as insistent as Ferrer that music, as an art of time, can be a model for visual art—if anything, he is even more so. Unlike Ferrer, Marclay has no musical training. But as an art student in Boston in 1979, inspired like many of his generation by punk rock, which "had freed people from the idea that you had to be skilled to play music," he began performing, or as he puts it, "inventing ways to make music when not a musician." Since he couldn’t play an instrument, he began working with musical ready-mades, using vinyl records and turntables as instruments, not unlike the breakbeat DJs who’d started working with the same equipment to revolutionary effect in the Bronx just a few years before, albeit to very different effect. Soon he started treating his records as sculptural material, cutting them up and recombining the pieces to create visual collages that were also bearers of sound collages.

By now, Marclay has become something of a virtuoso of the turntable; the correlate of the punk notion that you don’t need to be skilled to make music is that if you keep making music you will become skilled, though possibly in unusual ways. Increasingly, he has collaborated with trained musicians; and although he still calls himself an artist, he is highly respected as a composer of the avant-garde, even though he does not read or write conventional notation. At the same time, he has continued to use music, records, record covers, musical instruments and images of music and music-making as material for sculpture, collage and video as well as performance.

Not all of this work is at the Whitney; "Festival" is not a full-dress retrospective. Instead, in keeping a fairly tight focus on the notion of the "visual score," curators David Kiehl and Limor Tomer give a concise cross section of his oeuvre that shows it to advantage, despite the absence of some of his best work. Among these I should at least mention two: the astonishing Video Quartet (2002), a four-channel projection that conjures up a new, fourteen-minute-long piece of music out of more than 700 brief clips taken mostly from Hollywood movies—scenes in which people are shown making music or at least making sounds; and Tape Fall (1989), a sculpture consisting of a reel-to-reel tape recorder mounted high up above one’s head, playing a tape of the sound of trickling water that, instead of being taken up by a second reel, falls in elegant coils to form an ever-rising pile on the floor. One misses such pieces but not the relatively large quantity of facile, one-liner-ish works for which Marclay has also been responsible—for instance, assemblages of record covers that create grotesque figures out of, say, a solemnly gesturing orchestra conductor from the waist up matched with a ballerina en pointe from the waist down. In fact, the very notion of the visual score works against the one-liner. It means there always has to be at least a double take—one’s perception of the graphic form should be followed by an effort to understand how it might be understood as a series of cues for making music.

The idea of the visual score—or graphic notation, as it is often called—is hardly a new one, originating with the composers of the New York School of the 1950s: Earle Brown, John Cage, Morton Feldman and Christian Wolff. But Marclay not only pursues the idea to unheard-of extremes; he turns it on its ear. When a composer presents the performer with a graphic score that may allow far more interpretive leeway than a conventional score would, he still has some basic sonic parameters in mind—and in any case, the score was produced with the idea of making music. That’s not always the case with Marclay, for whom the visual takes precedence. And he sometimes asks performers to use as scores things that were never intended to have anything to do with music.

Shuffle (2007) is a deck of seventy-five cards with photographs taken by Marclay. Each shows an image of musical notation found in "real life"—that is to say, on billboards, candy boxes, T-shirts, cars, cakes, umbrellas, teacups—anywhere that a musical note might serve a decorative purpose, but not where it would have been meant to be read as a basis for making music. Yet this is what Marclay proposes: using this deck of cards as a recombinant score. What any given performer might make of, say, a single quaver crossed out, no particular tone specified, or a bunch of notes splashed across a staff with only two rather than five lines is entirely unpredictable; someone who went to hear pianist Anthony Coleman perform Shuffle on August 4 will undoubtedly have heard something with no discernible resemblance to either the version created by the duo of Robin Holcomb and Wayne Horvitz on August 26 or what the great Canadian artist and musician Michael Snow will come up with on September 26. But even for those of us who could never make music out of these paltry and absurd signs, the effort to imagine doing so makes us see the images differently.

Another photographic piece with a somewhat more complicated genesis is Graffiti Composition (1996–2002). In 1996 Marclay had posters of blank staff paper printed and posted all over Berlin. Soon enough, the posters were torn and marked up by passers-by, half covered with other postings, written over—sometimes even with music. The results were photographed, and now these too constitute a score, also being performed several times during the course of the exhibition. From poster installation to photograph to score for an unpredictable sound event: "I like these evolving structures," Marclay says, "where I eventually lose control."

Not only graphic signs have a capacity to score the unforeseeable. Marclay conceives even some of his video works as scores, while Mixed Reviews (1999–2010) is a wall text made from excerpts spliced together from all kinds of music reviews, taking only the parts where the writer was trying to describe a specific sound or sound quality. The performer’s task is to produce this sequence of sounds—a task perhaps made even more challenging by a twist: the piece has been displayed and performed in various countries; each time it is re-presented, it is translated into the language of the host country, and the subsequent version is translated in turn from that, not from the original English. This means that the text shown now at the Whitney is several generations removed from its first version—and presumably the sound sequences it might evoke are different as well.

As much as the sonic matter of music, Marclay takes much of his material from what might be called music’s mythologies—in Roland Barthes’s sense of that word. Unlike Ferrer, who would rather create symbols, Marclay prefers to deconstruct them. He cites conceptual artist Douglas Huebler’s motto, so typical of the 1960s: "The world is full of objects, more or less interesting; I do not wish to add any more." But by reusing existing images, existing records, existing video footage, he comes closer to refraining from adding more meanings than more objects. Yet his work conceals a certain unresolved irony. It is possible to devise out of the banality of our visual environment a score that does not commit the composer—if that is still the right word—to any particular musical statement. Doing so has the desirable effect, at least momentarily, of scrubbing some elements of that environment clean of those heavily stereotyped meanings that constitute its banality, allowing one to see it as also rather wonderful. But the performer who interprets this score, who might also be its composer (Marclay will join singer Shelley Hirsch in musically interpreting the digital slide projection piece Zoom Zoom, 2007–09, on September 22), has no choice but to choose this sound rather than that, this structure rather than that, this myth rather than that. Deconstructing the old mythologies only clears the way for developing new ones. "Absence," as Marclay says, "is a void to be filled with one’s own stories."