Is billionaire philanthropist Eli Broad’s private art museum in LA a tick in the legacy box or has it secured the city’s contemporary art scene for generations to come?

The huge bunker wrapped in a perforated concrete veil grabs your attention, even with Frank Gehry’s stainless steel Walt Disney Concert Hall shimmering next door.

Step inside the new Broad museum in the heart of Los Angeles’ downtown and a minute-long escalator or ride lifts you slowly from the cave-like foyer through a tunnel to a top floor gallery glowing with soft Californian light and stocked with some of the finest works by many of the biggest names in post-war art.

Start with Jeff Koons’ giant Tulips and turn right for Andy Warhol — a Campbell soup tin, Two Marilyns and Elvis with his pistols. Continue past a Jasper Johns American flag and Robert Rauschenberg toward highlights from the careers of Jean-Michel Basquiat, Damien Hirst, Cindy Sherman, Joseph Beuys and Ed Ruscha. Swoon over the wonderful giant Cy Twomblys and ponder the nine Roy Lichtensteins before you get to more of Koons, taken from the largest collection of his work in the world: a shiny rabbit, a blue balloon dog and Michael Jackson with his chimp Bubbles.

Admire it, then imagine owning all of it. Because the most astounding thing about the eponymous Broad museum is that everything about the place — the remarkable location, the razzle-dazzle of the art on show, the quiet splendour of the Elizabeth Diller building that houses it — is down to one self-made man: an extraordinary, divisive, billionaire philanthropist whom The New Yorker once called “the Lorenzo de’ Medici of Los Angeles”.

Eli Broad’s bestselling memoir begins with a blunt statement: “I am unreasonable”. It argues that his refusal to accept “reasonable” arguments explains why he became the first man to build two Fortune 500 companies in different industries (house building and financial services) and a $6 billion fortune in the process. It also explains how he and his wife Edythe were able to assemble one of the most valuable private art collections in history.

After years of speculation about which institution they might bequeath it to, they have built their own, the Broad, which opened to the public in September. Most of the 2,000-strong collection is on site, stored in a vault in the middle of the building that can be glimpsed as you walk down the stairs.

Broad is sitting in the 30th floor office of his foundation, 10 miles west of his new museum. His silver hair is neatly parted and he is immaculately dressed in a navy suit, white shirt and pale blue patterned tie and pocket square. There are signs age is catching up with the 82-year-old — his voice is soft and hoarse, his hearing poor and he repeats himself occasionally — but there is no doubting his formidable intellect, dry humour or appetite for work.

Swigging from a plastic bottle of water, he says he is still too busy to spend much time with his art. His home is decked with masterpieces but the longest he might spend with one is “a few minutes. Wouldn’t sit there for an hour”. Art was his wife’s passion and came later in life for him. “She says once I got involved the budget went up dramatically.”

As the only child of Jewish Lithuanian immigrants, growing up in the Bronx and then in Detroit, Broad (rhyming with road) had no dreams of great wealth. He collected stamps until girls began to distract him and thought when he started college that he might perhaps “end up with a haberdashery store or something”. It didn’t quite work out that way. He has been a millionaire since he was “26 or 27”. He has pledged to give away 75 per cent of his fortune and has donated more than $4 billion so far. His eyes smile. “We still have a few dollars left.”

From his office Broad has a 180-degree view that begins with 20th Century Fox’s studios below him and sweeps across toward the skyscrapers of downtown and his museum. Framed photographs show Broad with presidents Obama, Clinton, Carter and George W Bush, as well as Kofi Annan and Arnold Schwarzenegger. There is a Jasper Johns and a Julian Lethbridge on his walls while the reception area gets five Ed Ruschas.

I cannot spot it on his desk but his memoir says the “one constant” in his working life has been a paperweight his wife gave him, inscribed with a George Bernard Shaw quote that ends: “All progress depends on the unreasonable man”.

I ask if the LA Times’s description of his wife as the “antidote to his sting” is fair.

“I read that,” he says. “Everyone loves Edy. I don’t think everyone loves me. They respect me.” And he’s fine with that.

Since moving west in 1963, Broad has made himself the dominant force in Los Angeles’s civic and cultural life, with his demanding methods and capacity for putting people’s noses out of joint matched only by his indisputable achievements in bettering the city. In 2003 a Los Angeles magazine cover posed the question: “He has more pull than the mayor, more art than the Getty and more money than God. Does Eli Broad own LA?”

The couple’s biggest charitable investments have been in education and medical research. He has done as much as anyone to breathe life back into the previously rundown Downtown area and has tried and failed to lure an NFL American football franchise to Los Angeles, to buy the Dodgers baseball team and the LA Times newspaper.

“I love this city,” he says. “It has been very good to me. It is a real meritocracy and we have been able to do things we would not have been able to do in other cities. Here it is not a question of what your family background is or what your politics are. If you have good ideas and the energy and resources, then you are accepted.”

Hugh Davies, director of the Museum of Contemporary Art San Diego, cut to the heart of Broad’s complicated legacy when he described him as a “generous and great man” who had jousted with many colleagues but left his adopted city richer for it. “I’m grateful I wasn’t a museum director who had to serve Eli but I salute him from a safe 100-mile-away perspective in San Diego,” he said.

Over the years Broad’s contributions have been vital to the growth of the Museum of Contemporary Art, the Los Angeles County Museum of Art and the Hammer Museum, but he has also resigned from each of their boards. Even his enemies agree that the money for the Disney hall would probably never have been raised without him leading the campaign but typically there were complaints it would never have been such a well-realised building if his attempts to contract out the construction work had succeeded.

The Broads began collecting seriously with a Van Gogh drawing but exchanged that for a Rauschenberg work, setting the tone for a collection heavily weighted toward art of the past 50 years.

“It’s more interesting to get to meet the artists,” says Broad. He believes the best artists he collects are just as bright as the greatest businessmen he has known and often more interesting.

Life would be boring if I spent all my time with other business people, bankers and lawyers

The Broads bought their first Cindy Sherman pieces for a few hundred dollars and he met Basquiat when “he was working in the basement of Annina Nosei’s gallery and living there. I was struck by the work, that it was more than just graffiti. It was very thoughtful. And to think that work which was selling for $5,000 or $7,000 is selling for millions is unbelievable.”

He disapproves of people, often “hedge fund managers”, buying art as an investment and “flipping it”. Instead he cares about bringing his collection to the widest possible audience, which is why entry to the Broad will, unusually for the United States, be free.

The Broad collection has, on average, one work added every week. Broad signs off each one. Does he still get the same buzz he used to? “I do,” he says immediately. “In a way, it is competitive. I’ve got lots of friends who are collectors. Sometimes we get there before others do.”

Sir Nicholas Serota, the director of the Tate, told the Times the museum is a “special experience” and says the Broads’ commitment to buying prime examples of work and buying in depth “allows us to trace the evolution of many significant artists”.

Broad announced 10 years ago that “Los Angeles ought to — I’ll be immodest — become the art capital of the world”.

Today he feels it is, at least, “the contemporary art capital of the world”. Now that’s done and the opening night for 5,000 people is out of the way, does this restless man see another challenge looming over the hill?

“No,” Broad says, breaking into a smile. “I see a holiday.”