Finishing the Hat, the first of two long-awaited volumes of Stephen Sondheim’s lyrics, wit and wisdom, reminds me of another exacting exercise in self re-evaluation: the New York Edition of the fiction of Henry James. In the eighteen prefaces he wrote for the twenty-four-volume set, James revealed the sparks that found ready kindling in his imagination, and shared his many struggles to coax them into stories and novels. James could detect a spark in something as fleeting as casual conversation: “A mere floating particle in the stream of talk.” Because the Broadway musical is a collaborative effort, many of Sondheim’s shows have been set alight by other people’s sparks; but regarding Pacific Overtures (1976), for one, Sondheim shares a truly Jamesian moment of inspiration. He recalls coming upon a Japanese screen at the Metropolitan Museum of Art: “It was like a sudden explosion; it seemed to grow as I looked at it.” The screen, Sondheim suggests, was not just a piece of japonaiserie; rather, it was an aesthetic commandment to honor the principle of “less is more,” not as an abstract idea but as an emotional imperative.

Spanning the first three decades of Sondheim’s career, and revealing more of his creative process than James’s prefaces did of his own, Finishing the Hat includes rejects, rewrites and substitutes of lyrics from thirteen shows that together illuminate the dark, devious road leading to Broadway hits and flops. Like James, Sondheim transformed a popular genre often aimed at the young into an art for, and about, adults. Writing for a smart audience, James and Sondheim have no qualms about appearing smarter-than-thou, but Sondheim is smart about being smarter. Whereas James allowed his followers to call him The Master, Sondheim wrote a tongue-in-cheek song for the revue Sondheim on Sondheim (2010) in which the cast hails him as God. Sondheim, like James, is also ruthless when assessing the talents of famous predecessors. In sidebar “heresies” planted throughout Finishing the Hat, he pillories Ira Gershwin (“His work bespeaks a generous, warm and talented man, but his brother was a genius”), Lorenz Hart (“the laziest of the pre-eminent lyricists”) and Noël Coward (“the Master of Blather”). Sondheim sounds mild compared with James, who practiced summary executions: “Our Mutual Friend…is poor with the poverty not of momentary embarrassment, but of permanent exhaustion.” “Mr. Trollope is simply unable to depict a mind in any liberal sense of the word.” Not even Tolstoy is spared: “From no other great projector of the human image and the human idea is so much truth to be extracted under an equal leakage of its value.”

These judgments would sound coldblooded if they were not part of an encompassing project of self-criticism. Following a common aesthetic commandment—thou shalt not tell, but show—the work of both artists reveals the inner life of characters rather than showcasing the cleverness of its creator. This approach requires scrupulous revision and refinement. It demands technique. Technical acuity, moreover, is not empty virtuosity but a form of knowledge and, eventually, sympathy. Every page of Finishing the Hat is marked by Sondheim’s belief that being an artist requires intellectual vigilance. As James famously advised, “Try to be one of the people on whom nothing is lost!” Sondheim’s sympathies, deep rather than broad, extend mainly to other perspicacious characters; yet his best work strips bare the pitfalls of intelligence, especially when it is allied with the imagination. Intelligence, he shows, is amoral. The painter George in Sunday in the Park With George (1984) and the serial killer in Sweeney Todd (1979) are versions of the same character: brilliant, obsessed and blind to the sources of their own cruelty.

* * *

Art isn’t easy, as Sondheim wrote, especially art based on a highly self-conscious aesthetic; and over the years Sondheim has taken on projects whose difficulties would terrify most composer/lyricists, let alone their financial backers. A Japanese musical (Pacific Overtures)? A show about presidential assassins (Assassins)? Critics—champions of Sondheim as well as skeptics—have often oversimplified the shows on the assumption that Sondheim is a perennially clever kid rather than a stern ethicist. John Lahr, for one, described him as peddling “boulevard nihilism” in Sweeney Todd. It’s not hard to find a character in the shows who sees the world through the eyes of Holden Caulfield—life sucks, everyone’s a phony, we die alone—but that view is part of the story, not an op-ed published under Sondheim’s byline. One could just as easily accuse Sondheim of dabbling in “boulevard idealism” in Merrily We Roll Along (1981), which tracks the relation of innocence and experience in reverse chronology, ending with a bright-eyed affirmation of youthful idealism as corny as Kansas in August.

With an initial run of just sixteen performances, Merrily ranks among Sondheim’s most humiliating flops, even though it boasts no fewer than three of the greatest “showbiz” numbers of all time (“It’s a Hit!” “Franklin Shepard, Inc.” and “Opening Doors”) and one classic ballad (“Not a Day Goes By”). Unfortunately, but perhaps understandably, Finishing the Hat does not shed much light on the show’s failure other than to say that its faults were remedied by casting older actors and the several rounds of rewriting suggested by Sondheim’s current collaborator, James Lapine. Sondheim explains that the show, like several of its predecessors, re-enacted the moral fable of Allegro, the notorious flop by Sondheim’s mentor Oscar Hammerstein (the 17-year-old Sondheim served as Hammerstein’s assistant on the show). In Allegro, a precursor of the “concept musical,” a doctor repeatedly attempts to escape the moral compromises of bourgeois existence, and eventually succeeds. Merrily, based on a 1934 play by George S. Kaufman and Moss Hart, tells the story backward, beginning in 1981 and ending in 1957, and spotlights not a doctor but a songwriter and movie producer, Franklin Shepard, whose moral compromises, never redeemed, leave a trail of broken marriages and soured friendships. The theme of art versus life would return in Sondheim’s work with Sunday in the Park With George, but there, at least, the audience has the assurance of knowing that the George of the first act is the painter Georges Seurat, whose La Grande Jatte attained masterpiece status, thereby, or so many critics assumed, justifying the artist’s thorny life.

Merrily recalls Thomas Mann’s unsettling tales about artists: we don’t know if their art is any good, so we can’t tell if it will atone, even slightly, for the long list of misdeeds committed while making it. For the novelist this ambiguity can be a potent source of irony, but in musical theater irony can be slippery because keeping a show’s point of view clear is difficult. Shepard is rich, but is he a good composer or just a lucky hack? On the original cast recording the overture sounds tinny and tacky. Its tunes could have been fished from the wastebasket of the Broadway songwriter Jule Styne, which would make the overture a bizarre hommage to Sondheim’s collaborator in Gypsy. But perhaps it’s an exposé of Shepard’s mediocrity, assuming that the overture is by Shepard, not Sondheim.

Finishing the Hat allows us to compare the Merrily lyrics from the 1981 disaster production with those from the more successful revised production staged first in 1985 and again a decade later (although without showing us the revised script). The differences contradict the oft-repeated accusation that Sondheim is not interested in his characters. The original production of Merrily opened with a high school anthem, “The Hills of Tomorrow,” composed by the teenage Shepard. It sounds a lot like “Make Our Garden Grow” from Candide, but not as good—which may have been the point. Frank’s first hit song, “Good Thing Going,” which shadows him through the score like a guilty conscience, sounds like “Small World” as rewritten for the Carpenters. It’s hard to say how that irony works. The song also sounds very much like one that could launch a successful showbiz career.

The problem with Merrily, as Sondheim and Lapine came to realize, was a matter of emphasis. The story that needed to be told was not a younger generation’s rejection of their parents’ values, timely as that tale might have been, but rather Shepard’s awareness of the choices in life that had led him into the abyss. It’s a Jamesian angle. The central action of the show, Sondheim realized, took place within Shepard’s mind, and exploring it demanded empathy, not irony. No wonder the show was hard to write, and harder to stage. Rescuing it from disaster required a ruthless reappraisal of the original, a new script, new characters, new songs and no distracting anthem, however pretty. Most of all, it required a sharper definition of the lead characters and their motivations: aren’t both Charley and Mary, the two creative partners Frank betrays on his way to the top, in love with him? I’m not convinced that every problem was solved, or is solvable, but I find the revised version (which can be heard on an imported Jay/TER CD) a far richer backstage drama, dramatically and musically, than even A Chorus Line. Sometimes ambition trumps perfection.

Sondheim, again like James, has always written strong, complicated female characters, from Momma Rose of Gypsy to Fosca of Passion. His songs have inspired great performers: Ethel Merman, Angela Lansbury, Elaine Stritch, Barbara Cook, Bernadette Peters and Donna Murphy. Although many critics have linked the emphasis on the feminine in James’s and Sondheim’s work to homosexuality, a subject James could never explicitly address, and that Sondheim avoided depicting onstage until late in his career (in Bounce, from 2003), it could also be viewed as a reaction to the rise of feminism, a movement James portrayed with a mixture of admiration and satire in The Bostonians. As in James, most men in Sondheim are running scared from women, and yet they are still in charge. Robert, the unmarried protagonist of Company, seems understandably perplexed by the opportunities and traps that surround him. The final song, “Being Alive,” offers him the same advice that James put in the mouth of Lambert Strether, the unmarried protagonist of The Ambassadors: “Live all you can; it’s a mistake not to.”

If art isn’t easy in a feminist age, love is hell. The relationship between the sexes festers like an open wound in many of Sondheim’s shows, and sometimes even his defenders have chosen to ignore the pain. Frank Rich’s rave review in the New York Times of Sunday in the Park With George, which surely helped earn Sondheim a Pulitzer, described only one facet of the show—the George part. Rich termed Sunday “a contemplative modernist musical,” as if it were a Robert Wilson play. The show celebrated Seurat’s “methodical intellectual precision,” which, Rich wrote, Sondheim happened to share. (Sondheim often takes critics to task for identifying him with his characters, but to no avail.) Minimalist cool had moseyed uptown, from SoHo to Times Square, and Rich’s review congratulated all concerned for their impeccable good taste.

The actual show, messier and far more interesting than Rich’s account of it, centers on Seurat’s mistress Dot. In dramatic terms, she functions as an obstacle. George wants to finish La Grande Jatte. Dot wants George’s undivided attention. He obsesses; she pouts. In symbolic terms, though, George and Dot are antitheses: male and female, art and life, reason and passion. This conflict drives the plot and the performance (preserved on DVD), as does the fierce duel between Mandy Patinkin and Bernadette Peters for the audience’s favor. In terms of the plot, George and Dot both win: he finishes the painting, which the world will posthumously call a masterpiece, and she leaves for America carrying his child. Although the second act offers an anodyne, it’s-all-good resolution with the song “Children and Art,” the show offers more questions than answers.

Sondheim’s first romantic period piece, A Little Night Music (1973), set in Sweden at the turn of the twentieth century, may have seemed an anomaly when it appeared on the heels of Company and Follies; but it augured the emergence of Sondheim’s romantic side, a noir version but romantic nevertheless. In his prefaces James called attention to the interplay of romanticism and reality in his fiction, noting that romanticism, manifested in the fairy-tale plots of some of his novels, opens the door to “disconnected and uncontrolled experience—uncontrolled by our general sense of ‘the way things happen.’” Despite critics’ preoccupation with the influence of Allegro on Sondheim, romanticism is his true inheritance from Hammerstein, who got his start working as a lyricist for operettas like The Desert Song and Rose-Marie. Beginning with Show Boat, Hammerstein achieved a synthesis of musical comedy and operetta that lifted the musical into a symbolic realm of representation. The Indian Territory in Oklahoma! and Down East Maine in Carousel are less realistic settings than imaginary locales, geographically distant but emotionally magnified, where Hammerstein could explore contemporary political and psychological issues more powerfully than he did in the apparently realistic setting of Allegro.

Sondheim’s musicals can mostly be divided into two categories: modern-dress musical comedies (Company, Follies, Merrily We Roll Along) and exotic neo-operettas (A Little Night Music, Pacific Overtures, Sweeney Todd, Into the Woods, Passion). While Company established Sondheim as a savvy portraitist of contemporary life, I think operetta, turned inside out and upside down, to be sure, is his true calling. There’s no better proof than a quartet of shows that may become the Sondheim “Ring”: Sweeney Todd, Sunday in the Park With George, Into the Woods and Passion. Even realistic shows like Company and Follies feature songs—“Getting Married Today,” “The Ladies Who Lunch,” “I’m Still Here”—that are romantic in the Jamesian sense, not love songs but moments of such detailed interiority that each one could be a one-act play. Sondheim credits this type of song to Rodgers and Hammerstein, but he has so enhanced the art of the story-song that many of his have taken on a second life as cabaret standards.

With Sweeney Todd Sondheim transcended the conflicting claims of realism and romanticism by placing the entire action within an unreal framework created through staging, writing style and, most important, continuous music that does not interrupt the action but is the action. Sondheim turned the Victorian device of a returning choral ballad into a rhythmic engine that churns throughout the show, powering Sweeney’s unrelenting thirst for revenge. I hope the second volume of Finishing the Hat will illuminate how Sondheim and Lapine further extended this technique in Sunday in the Park With George and Into the Woods. In these shows the romantic realm of song breaks free of its usual boundaries. Into the Woods jump-cuts between the multiple plotlines and chops their respective songs into recurring fragments, creating a show that feels at once fast-paced and monumental. Although I don’t think Sondheim’s shows are—or need to be—operas, anyone writing opera would benefit from studying the speed with which these shows delineate character and plot. Even the most successful recent operas, such as John Adams’s Nixon in China, feel clunky by comparison.

For all its revelations, Finishing the Hat leaves many questions unanswered. Sondheim talks a lot about lyrics, less about music, even though he has said he enjoys composing more than writing. Fortunately Mark Eden Horowitz, senior music specialist at the Library of Congress, which houses Sondheim’s papers, knows the music cold, and in the interviews collected in Sondheim on Music he serves Sondheim as a gracious yet provocative inquisitor, often asking questions about tiny notations in his sketches. Sondheim’s answers reveal much about his working process. Horowitz treats us to Sondheim the composer, who strikes a different figure from the cocky, combative lyricist. Sondheim is as self-conscious about his compositional technique as he is about his lyrical craft, and he clearly learned much from his studies with Milton Babbitt about generating music out of short motifs containing just a few notes, like the subject of a Bach fugue. But there’s also an ad hoc feeling to Sondheim’s musical affinities that comes as a surprise. Regarding Pacific Overtures, for instance, Sondheim talks about the influence of John Cage and the early Broadway composer Jerome Kern, whose hits included “Smoke Gets in Your Eyes” and “The Way You Look Tonight.”

Maybe the second volume of Finishing the Hat will say more about Sondheim the auteur, a question as tricky in the collaborative world of musical theater as it is in the movies (though not, of course, in fiction). Writing about Sondheim is marred by the tendency to blame the collaborators for a show’s every plot misstep or false note, as if Sondheim had phoned in the songs or enjoyed a godlike immunity from criticism; conversely, writers treat every detail of a show as clues about Sondheim’s childhood, love life and politics, as though he had written, produced and directed every note of music, every scrap of lyric and every line of dialogue. Volume one slyly encourages this approach, taking its title from a song in Sunday in the Park With George that celebrates the intense, internal process of artistic creation, which binds the artist to his subject and alienates him from other people. As a credo the song is a cross between Rilke’s “The Panther” and Frank Sinatra’s “My Way.” It is well suited to Sunday’s George, who paints in splendid isolation even when other people are in his atelier, but it distorts the creative process of the Broadway musical. It takes a team to put the hat on the stage.

Sondheim started out working with co-creators like Leonard Bernstein and Jerome Robbins, who were his elders and, at the time, betters. From Company to Merrily We Roll Along he collaborated with a contemporary, Harold Prince: the two reportedly met at a performance of South Pacific when they were both around 20. James Lapine, with whom he has worked since the early 1980s, is almost twenty years Sondheim’s junior. You might expect these relationships, based on the partners’ varying degrees of experience, talent and taste, to be audible in the shows; yet it is Sondheim’s voice that has pervaded their lyrics and music from the beginning of his career. How does collaboration really work? Is it the psychodrama we hear in “Franklin Shepard, Inc.” in Merrily, or the mutual madness of “A Little Priest” in Sweeney Todd? I’m saving a space on my coffee table for answers, and further heresies, in volume two.