Here in Cape May, New Jersey, the Magnolia’s white tablecloths and chilled glass salad plates are a throwback to an earlier era. The genteel resort town itself is evocative of New Orleans’ Garden District—but with funnel cake instead of beignets. The area’s oldest operating hotel, the Chalfonte was built in 1876 by Henry Sawyer, a retired Civil War colonel. The 65-room Victorian inn surveys the street from a manicured lawn, a great white birdcage with banks of windows capped in spearmint-striped awnings and double-decker porches iced in lattice.

The scene seems a romanticized version of the South, not South Jersey. But in Cape May, a town with abiding Southern roots, these are not necessarily mutually exclusive. Four generations of African American women with Southern roots have worked at the Chalfonte. And though they have never owned it, they are the restaurant, which has become one of the last existing links to Cape May’s rich and complicated black history—a history that’s been, both forcibly and subliminally, paved over to make way for ice cream stands and carousels.

Philadelphia, 90 miles northwest, claims Cape May as a summertime colony, but before the advent of modern highways it was an equally convenient getaway via steamship from Virginia, Maryland, and other points south. It’s also nearly parallel with Washington, D.C., placing it comfortably below the Mason-Dixon line. Wealthy white Southerners would travel here each summer with their domestic staffs, many members of whom were just a generation removed from slavery.

One of such a staff, Clementine Young worked for the Satterfields of Richmond, Virginia. After the family patriarch, former Confederate general Calvin Satterfield, purchased the Chalfonte in 1911, Young spent 60 years as the hotel’s head chambermaid. “Or as we would say today, housekeeper,” says Lucille Thompson, Young’s 87-year-old granddaughter, sitting on one of the Magnolia’s hairpin-back chairs.

Thompson, who has cooked there for over half-a-century, learned to make fried chicken and rolls from her mother Helen Dickerson, Young’s daughter. Miss Helen, as everyone at the Chalfonte called her, was the head waitress at the Magnolia Room and a talented home cook. In the 1960s, the Satterfields asked her to step in as chef. Thompson and her older sister, Dorothy ‘Dot’ Burton, followed, and officially took over their mother’s stead when Miss Helen retired in 1990. At the end of every season, they’d claim it was their last. And every April, sure as the tides, they’d return.

Until last year, when Burton passed away, and Thompson returned to the kitchen without her. “We worked and lived together,” she says. “It was hard at home without her,” and cooking was a good distraction. Now, Burton’s daughter, Tina Bowser, is by Thompson’s side at the stove, frying chicken breasts in the family’s century-old skillet.

Ghost hunters say Cape May is full of spirits, but the one people rarely talk about is its once-thriving black community.

Its origins lie on the mainland—over marshes and tidal beaches, past Cape Liquors and the Vanilla Bean Creamery with its lavender siding. Here, in the 1830s, in the offshore woods of Lower Township, settlements founded by freed and runaway slaves thrived. Today, it’s the town of Erma, home to my in-laws’ summer place in a campground with bingo nights and Zumba classes.

“By the mid-to-late 19th century, many [African-Americans] moved into the West Cape May area and other places south of what is now the Cape May Canal,” Hope Gaines tells me from her home, a quaint blue cottage on Washington Street. The director of the Center for Community Arts, a foundation that runs black history tours and funds the ongoing restoration of the city’s once segregated elementary school, Gaines explains that the settlers owned farms and worked in tourism. Harriett Tubman was a cook in Cape May in 1852 and, some say, helped establish the city as a stop on the Underground Railroad.

Continuing north on Route 9, modest motels, cheerful farm stands, junkyards, vineyards, Wawa, and Wal-Mart blur by. After crossing into Middle Township, Whitesboro appears. In 1902, George Henry White, a two-term black Republican Congressman from North Carolina, founded a community for blacks fleeing discrimination in Cape May County and Jim Crow in the South. Supported by pillars of church, family and economic sovereignty (White’s bank in Philadelphia helped arrange loans for residents to buy land), Whitesboro flourished.

Over the years, as farming became less viable, residents sought economic opportunity elsewhere. A proud community of about about 2000 remains, but some of the businesses look like they haven’t been touched since I first visited a decade ago. A green cape of algae covers the bottom of the pool at the Hillside Motel. Weeds erupt through the rusted belly of an old oil-drum smoker outside of Tiffany’s Greens Beans ‘n’ Birds. A black mannequin models the latest in 1980s outerwear in an old store window. NOW OPEN, reads its hand-painted sign, but nobody’s been home since shoulder pads went out of style.

In Whitesboro, the shuttered businesses dot Route 9 like gravestones, but in Cape May, where black business really thrived, there are no such markers. The barbershops and bakeries, doctors’ offices and churches, pool halls and beauty shops were razed in the name of urban renewal in the 1960s. Under President Lyndon Johnson’s Great Society movement to eradicate poverty and inequality, cities received federal funds for improvements, some of which were given directly to homeowners. “However, in Cape May, the local government claimed its property under eminent domain and transformed the historically black downtown into a historic tourism area,” says Bernadette Matthews, a member of the Greater Cape May Chamber of Commerce.

At the time, African-Americans comprised about 30 percent of Cape May’s population (less than 10 percent now) and were concentrated downtown—parts of which had fallen into disrepair, compounded by the devastating nor’easter of 1962. “Some [buildings] needed help, some didn’t. My husband’s parents had a home on Jefferson Street that didn’t need any repair,” Matthews says. Still, it was claimed under eminent domain. Matthews’ in-laws were relocated to new housing projects, which were simultaneously erected as the old homes were demolished.

On any given summer night on Cape May’s Washington Street Mall tourists swarm like seagulls. The Mall, a pleasant pedestrian lane that occupies the former epicenter of Cape May’s black community, is lined with restaurants and shops—a tobacconist, a lingerie store, a 5-and-10 with a soda fountain and chili cheese dogs. In the theater-like front window of Fudge Kitchen, confectioners whip shimmering fudge in copper kettles with paddles the size of oars. The Victorian mansions’ spires and widow’s walks watch over visitors while they shop for Christmas ornaments, stand-up paddleboards, and Life Is Good sweatshirts in saltwater taffy shades.

I walk the promenade back to the Magnolia Room for dinner, wondering what it would look like if Johnson’s funds had been passed onto individuals for repairs instead of the government’s invasive intervention. You can’t deny the historic district’s charm—like countless others, my family loves coming here—but the human cost of its creation tempers that idyllic summer sweetness. In enjoying this place, are we complicit?

I’m still sorting through the feeling when a host guides me to a table by the window at the Magnolia Room. She hands me a menu, a culinary legacy of Lucille Thompson’s family: shrimp and grits larded with bacon and cheddar, crab cakes, Cajun-spiced stuffed clams, custardy fried oysters and chicken salad, blackened catfish, ribs with coleslaw, famous fried chicken. As I’m deciding, a server with a basket of plush house-baked dinner rolls passes my table.

Like many of the other dishes Thompson cooks from muscle memory, the rolls are her mother’s recipe. “But I changed ‘em a bit,” she says. Into the bloomed yeast go two scoops of vanilla ice cream, a rapid cooling that she says creates the bread’s supreme texture.

Allocated one per person, the rolls come swaddled in a napkin-lined basket with single-serve butter packets. The server bites her lip when I ask for a refill. “Miss Lucille only makes a certain amount each night,” she explains. An extra costs $1, which I gladly pay.

Miss Lucille surveys the room: “There’s a lot of love here; this is my salvation.” But it’s hard work. Thompson will turn 88 this year, and 88-year-old bodies aren’t built for kitchen labor. She says this season, which ended last month, might be her last.

In other words, she’ll see you in April.