What’s the best way to appraise a building: from the inside out or the outside in? You might think the best approach would be from the outside. After all, we always enter buildings that way, with one exception: When we’re born (assuming we were born indoors), we enter the building from within.

Yet perhaps seeing a building like a newborn is best, especially when the building itself is just an infant. The Whitney Museum of American Art has just opened its new home within eyeshot of the Hudson River on Gansevoort Street, in what used to be Manhattan’s meatpacking district. With its ponderous glass-and-metal skin, the new edifice, designed by Renzo Piano, is neither visually dazzling nor particularly elegant; nor is the structure the sort of shape-as-logo design that makes its mark on the city as a graphic silhouette. It is not architecture trying to be sculpture. But I challenge anyone to stroll through its galleries for more than five minutes without taking pleasure in the sheer generosity of its proportions, and how it calmly ushers in light from the river as graciously as it welcomes visitors. In comparison with the Whitney’s old uptown space on Madison Avenue—“harshly handsome” (as Ada Louise Huxtable declared on its debut) and designed by Marcel Breuer as a fortress to keep the urban jungle at bay—the new building is huge, yet its scale is never oppressive, just as the beautiful light spilling across the floors never hardens into a glare. Unlike so many other trophy museums of recent decades, Piano’s does not compete with the art for your attention, but neither does it try, as Yoshio Taniguchi’s 2004 redesign of the Museum of Modern Art unsuccessfully does, to disappear entirely in its favor. The building is quiet and congenial, but no pushover. Without question, the best place to see modern and contemporary art in New York City is now on Gansevoort Street.

Clearly, the building was designed from the inside out—in order to do what a museum is meant to do for the things it houses and the people who use it. But then, the new Whitney has no need to seduce anyone looking at it from Hudson River Park. Sitting at the foot of the High Line, it’s in the thick of one of Manhattan’s newest tourist destinations. More than that, as Michael Kimmelman recently observed in The New York Times, the museum’s location seals a “definitive shift in the city’s social geography”: Downtown is the new uptown, where capital and culture consort under the happily watchful eye of the real-estate industry. As everyone knows by now, gentrification is an art lover, just like you and me. The museum’s nearly anonymous appearance speaks to its merely supporting role in the transformations that the city has been undergoing.

So the new Whitney is a felicitous place for seeing art, but what of the art itself? Right now, through September 27, the museum is presenting a vast exhibition with the coy title “America Is Hard to See,” which features more than a century’s worth of works from its collection—no less than 600 pieces by around 400 artists. The selection is just a smidgen of the 22,000 objects in the Whitney’s holdings, but it’s more than has ever been displayed at once by the museum. The title sounds like an excuse, as if to say, “If we don’t get this right, it’s not our fault. The task is so hard!” A quibble: Is the purpose of American art, or of a museum of American art, only to offer a view of America? Art’s entire history argues against its having a vocation to embrace an essentially national character. I thought of John Lennon’s koanlike lyric “Got to be good-looking ‘cause he’s so hard to see,” and therefore of the song’s title, “Come Together.” On whose terms would this togetherness transpire?

* * *

A small room on the first floor highlights the museum’s origins in the Whitney Studio Club, founded nearly a century ago by Gertrude Vanderbilt Whitney—an artist as well as a collector and patron—as a place for a new generation of artists to meet, work, and exhibit. Her taste was for the painterly realism of the Ashcan School, and in 1916 the group’s leading light, Robert Henri, paid her a fine compliment of a portrait that is still very much our image of the stylish and intelligent young woman who would go on to found the Whitney Museum on West Eighth Street in 1931. In hindsight, the painting inadvertently draws attention to the blind spot in her notionally progressive artistic views, namely her indifference to abstraction: Her reclining posture gives pride of place to the gorgeous gray oval that forms the sofa’s back and also serves as a harbinger of painting’s future direction, albeit one that escaped her attention.

From that first-floor gallery, the traversal of “America Is Hard to See” runs to the top, to the eighth floor, where the museum’s founder might have been surprised to find abstraction and a streamlined machine aesthetic as the main focus of a selection of works from 1910 to 1940. Exiting the elevator, you’ll see several paintings by Marsden Hartley, one of which is Painting, Number 5 (1914–15), as good an illustration as any of why America really is hard to see. Hartley was arguably the first great American modernist painter, but despite this work’s resolutely abstract title, its emotional tenor is anything but cool and removed. Its brooding colors smolder as intensely today as they must have done a century ago. At the composition’s center is the Iron Cross of Prussian militarism; the painting is part of a series of works Hartley made in Berlin in memory of a German officer he’d fallen in love with, and who died in battle at the very outset of World War I. Was the painting on the wrong side of the war that America would later join, and because of a desire that dared not speak its name at the time? Or should the painting’s formal rightness be a reminder that America’s right and wrongs and those of art may not coincide?

On the eighth floor, abstraction retains its connection to the imagery of everyday life—taking its inspiration from nature, from the street, from the quiet and intimacy of still life—but obliquely, and sometimes curiously mixed up, as in I. Rice Pereira’s wonderful Boat Composite (1932), with machine imagery of a sort that Charles Sheeler or Stuart Davis would have appreciated. But rather than using their crisp rectilinear style, Pereira renders the scene in a funkily gnarled manner that recalls Hartley’s or Arthur Dove’s visions of natural forces at work. At times, however, such art is more directly descriptive, and still coming to terms with its European origins; it has never until now occurred to me that Sheeler’s River Rouge Plant (1932) is dominated not by the piled-up blocks of the Ford Motor Company’s great Dearborn, Michigan, factory complex but rather by the dappled waters of the foreground. Here, Sheeler becomes our Canaletto.

On the seventh floor, which features work from 1925 to 1960, the exhibition breaks down into two very distinct and even incompatible sections that might have been more coherently displayed on different floors rather than made to coexist on the same level. Turn to the right as you exit the elevator, and you’ll see a reaction against the tendency toward abstraction that dominates the floor above. There’s the sturdy realism of that most overrated of painters, Edward Hopper, as well as the technical conservatism in which the American Surrealists packaged their imagistic inventions. Maybe it’s because of my broad lack of sympathy with this work that, in this part of the exhibition, the art seemed to have been installed in an inert, somewhat unadventurous way. One experiment the curators have tried throughout the building is to intersperse dense clusters of smaller works—often drawings, prints, and photographs—among the bigger, more eye-catching pieces arrayed one by one. For the most part, this approach isn’t successful. Piano has made all the interior walls of the new Whitney nonstructural, so they can be removed or added at will, and the curators should consider making smaller, more intimate spaces to encourage undistracted viewing and quieter dialogues among the pieces on view.

* * *

But turn to the left as you exit the seventh-floor elevator and you’ll find something else altogether: Abstract Expressionism, which Irving Sandler once dubbed the “triumph of American painting,” and it really looks so here. In the space are all the textbook masterpieces you’d expect—the de Koonings, Pollocks, Rothkos. But there are surprises too: The room is dominated by Lee Krasner’s The Seasons (1957), with its 17 feet of pink and green billowing forms—a powerful declaration of intent that dares anyone to dismiss her as simply Jackson Pollock’s widow. Another striking discovery is Hedda Sterne’s much smaller but equally dynamic New York, N.Y., 1955; its overlapping, sometimes girder-like structures have been delineated with an airbrush. Like Krasner, Sterne was long overshadowed by a better-known husband, Saul Steinberg. Most people only remember her, if at all, as the sole woman in Nina Leen’s famous Life magazine group portrait of “the Irascibles,” a group of New York artists who in 1950 signed a letter protesting the Metropolitan Museum of Art exhibition “American Painting Today”; the sculptors Louise Bourgeois and Mary Callery also signed the letter, but only painters were invited to be photographed. Later, Sterne lamented, “I am known more for that darn photo than for 80 years of work,” and if this one painting is anything to go on, she had a lot to complain about.

Her painting is one of the liveliest in the whole museum, and it’s characteristic of one way in which the new Whitney has made America at least a little easier to see: Without having taken a head count, it appears to me that people with two X chromosomes are a lot more visible as artists here than we are used to seeing in historical surveys of modernism, American or otherwise. And the median skin tone seems to be several shades darker. The artist with the largest number of works on view is Jacob Lawrence, with 10 pieces from his 1947 “War Series.” It’s great to see a painter as good as William H. Johnson get his due; to find the work of an untrained artist like Bill Traylor hanging amid mainstream peers; to notice how a small, brooding abstraction by Norman Lewis can have as much impact as the more extrovert energies of his white Abstract Expressionist colleagues; or to be reminded of how Roy DeCarava transcends the genre of “jazz photography” into which his compellingly subjective portraits of musicians (which are just one part of his oeuvre) are usually slotted.

But black and women artists don’t always slip so easily into a history that has been mostly conceived without considering them. Jumping ahead to the exhibition’s sixth floor, covering the period from 1950 to 1975, a wrong note is sounded when a large-scale collage by Romare Bearden, Eastern Barn (1968), is placed in a room dominated by Pop Art. Bearden’s concern may have been with everyday life, and his use of magazine clippings an acknowledgment of mass culture, but both his subject matter and his formal means are so different from those of artists like Allan D’Arcangelo or James Rosenquist that the juxtaposition seems thoughtless and misleading.

Bearden isn’t the only odd placement in the space devoted to Pop. A small Jasper Johns from 1958, Three Flags, hangs amid much larger works from the 1960s—Andy Warhol’s Green Coca-Cola Bottles (1962), for instance—that reflect his influence. The problem is that Johns is presented here as one example among many, not as the profoundly singular figure he is. And Three Flags is the Johns painting that, when the Whitney acquired it for $1 million in 1980, was the world’s priciest work by a contemporary artist; you’d think the museum would still want to highlight it, but I guess a cool million is chump change these days. A different kind of oddity in this section: hanging Warhol’s Before and After, 4 (1962)—one of his “nose job” paintings—catty-corner to Alex Katz’s The Red Smile (1963). The similarity in scale of the left-turning faces in the two paintings puts a strange and distracting emphasis on the nose in Katz’s portrait of his wife, Ada.

There are other thoughtless juxtapositions on the sixth floor. In a section on abstraction, a painting by the California hard-edge painter John McLaughlin is placed between an Ad Reinhardt black painting and a 1959 painting, mostly white with a green obtuse triangle, by Carmen Herrera—the Cuban-born artist, born in 1915, who labored in near-complete obscurity until her work began to be exhibited widely about a decade ago. In theory, the sequence makes sense: three painters in search of ultimate pictorial refinement. But because McLaughlin’s work contains, between two white borders, a black rectangle on its left and a green one on its right, the result is a crass interior decorator’s effect—the black rectangle on the left of the McLaughlin connecting to the black of the Reinhardt, and the green one on the right connecting to the green in the Herrera, creating a single new composition out of the three paintings, at odds with their makers’ intentions.

It takes time for curators to learn how to use a new space. There’s a learning curve, and the Whitney is just at the beginning of it. Clangers aside, “America Is Hard to See” is a promising beginning. They’ll teach themselves how to place works in order to make subtler, deeper connections.

* * *

The Whitney’s survey of the last century of American art ends on the fifth floor and features work from 1965 to the present. If you’ve seen the Whitney’s recent Biennials, you can probably more or less picture it, and you know whether you’ll enjoy it. For now, I’ll just say that I found a lot to like, and leave it at that. Besides, by this point, after the hundreds of works on view on floors eight through five, if you’re not exhausted, then you haven’t been paying attention. You’ll need to come back for a second look, maybe counterhistorical: Start with the present and work your way back to the past.

Alongside the art that’s conceived from the inside out, “America Is Hard to See” includes a lot that’s been made from the outside in—art that comes from, and sends you back to, the wide world outside the museum’s walls. Maybe because the story of a new museum is necessarily not just an art story but a real-estate story—and because so much of the art on view evokes the cityscape of New York, from Joseph Stella’s painting The Brooklyn Bridge: Variation on an Old Theme (1939) to Zoe Leonard’s color photographs of Lower East Side storefronts circa 2000—by the time I left the building, I had a strong desire to take a new look at my city under the aegis of the art I’d just seen. I found my template in a famous work of conceptual art, Hans Haacke’s Shapolsky et al. Manhattan Real Estate Holdings, a Real-Time Social System, as of May 1, 1971. This compendium of photographs and information is renowned for causing the director of the Guggenheim Museum to cancel an exhibition of Haacke’s work and fire its curator, on the grounds that such material—tracing the intertwined ownership of over 100 slum properties—could not possibly be art.

I couldn’t help noticing that many of the buildings Haacke pictured were in my own neighborhood, the Lower East Side, so I decided to make an expedition around my environs to see what had become of those buildings over the last 44 years. The Lower East Side is a very different place than it was in the 1970s and ’80s. As Luc Sante wrote of his time living in the neighborhood back then, “It was a ruin in the making, and my friends and I were camped out amid its potsherds and tumuli.” In fact, some of the buildings featured in Haacke’s piece no longer exist: There’s a school playground where 216 and 218 Stanton Street must have been; 209 and 211 Eldridge Street were replaced in 1988 by a New York City Housing Authority project called Lower East Side I Infill. At the same time, many of the properties look—at least from the outside—as if they’ve risen in the world, and some of them have upscale bars and cafes on their ground floors. But such appearances can be deceiving: 42 Rivington Street is the site of a charming little French restaurant and wine bar where I’ve whiled away a happy hour or two; but the entrance to the apartments above it is filled with graffiti, and a “no trespassing” sign posted by the Manhattan district attorney’s office suggests that there’s been serious trouble recently. A Google search turns up many recent violations, including reports of no heat or hot water.

It’s hard to know what goes on behind the facade of a building, today or back in 1971 when Haacke was making Shapolsky et al. The piece takes a documentary form, but only to remind you that it is really a formal exercise revealing how little a work of art—or the name of a building’s owners—can reveal. Haacke’s piece exposes a fraudulent real-estate system, according to the Whitney’s accompanying wall text, but not really. Rather, it exposes a system of ownership that lacks transparency, but as to what end that opacity was cultivated we can only speculate. And it tells nothing about the lives of the people who lived in those buildings. How wretched or bearable were the conditions in which they dwelt? Are their lives, or those of their children and grandchildren, any better now that some of the places where they once lived have been gentrified, and they have been priced out as the city’s center of social gravity has shifted downtown? I did manage to find a clue as to what was going on in one of buildings. In Stephen Cohen’s 2007 book The Gay Liberation Youth Movement in New York, you can read an announcement from 1972: “Street Transvestite Action Revolutionaries meet Friday at 6:00 p.m. at Marsha Johnson’s, 211 Eldridge Street, New York, N.Y., apt. 3. For information write: S.T.A.R., c/o Marsha Johnson, at the same address. Power to all the people!”