A Sexist Joke Cost Ken Fisher $4 Billion in Assets. He Still Runs $121 Billion

An avowed cheapskate who buys shoes at Walmart, Fisher picked up this property years ago at a fire-sale price. It had been a commercial chinchilla farm in the 1940s. Fisher, 69, grew up here in San Mateo County and remembers the freedom he had as a child, hitchhiking in the area or taking the train to nearby San Francisco as a 10-year-old. “It was just a marvelous world that used to exist—just so free and so different,” he says.

Speaking of the business he built, he says, “I had this vision, which was like a sort of preindustrial age family, like a farm.” His wife and children have all worked for the company, Fisher Investments, and the Fishers once lived at this Woodside, Calif., compound, where a statue of a bull fighting a bear adorns a back patio. Some of his former employees—referring to the way many are molded in the founder’s image, or Fisherized—have called his firm the cult on the hill.

Late last year the outside world came crashing into this idyll, after comments Fisher made at a San Francisco conference sparked anger on Twitter and then were reported by Bloomberg News and others. Fisher cautioned against using financial planning—which involves getting people to talk about their money—as a way to sign up new investing clients, comparing that approach to picking up a woman in a bar. But he was crude about it, making a reference to “trying to get into a girl’s pants,” one horrified attendee told Bloomberg. (A recorded excerpt of his remarks broadcast later by CNBC captures Fisher saying you wouldn’t go up to a woman and ask what’s “in your pants.”) In an interview with Bloomberg at the time, Fisher said his comments had been taken out of context.

Large clients including Fidelity Investments and Goldman Sachs Group Inc., as well as some public pension funds, pulled nearly $4 billion from his company in a month. Then and now he says he made similar comments many times, and no one had complained. Still, he apologized, though he also repeatedly said his remarks were misinterpreted. “By saying those inappropriate things, I was demonstrating inappropriateness,” he says, in a two-hour interview, his first extended remarks since the crisis. “I said, ‘It would blow up in your face. You’d come off like a jerk.’ ” In his view, some big investors bolted only because of pressure from the press. “Literally we have people that have told us they wouldn’t fire us if it wasn’t for you people writing about this,” he says, his general counsel and an outside lawyer by his side.

Fisher speaks in a gravelly, stentorian voice, like an anchorman’s. In fact you might have seen him commenting on financial news shows, in his boxy suit and a wide gold bull-and-bear power tie. Or maybe you’ve seen him in one of his ubiquitous ads or learned about him from the bulk-mail pitches he sends out on heavy embossed paper. Perhaps you’ve read one of his many books or his column that ran for years in Forbes. With help from the soaring bull market, his assets under management have more than quadrupled since 2009, to more than $121 billion at the end of December. He’s also gotten rich, with an estimated net worth of about $4 billion.

Fisher seems like a figure from an older Wall Street, before low-cost index funds began to replace investment gurus such as him. He’s still able to charge lucrative fees for actively managing money. He also brings to mind the age before the rise of the corporate HR department and the #MeToo movement. Some former employees describe Fisher Investments’ corporate culture as a “boys’ club,” where the use of crude language had long made some women, and men, uncomfortable. For some institutional investors hearing about Fisher’s comments in October, walking away from the firm was a fairly simple decision. Many of them—especially retirement funds that represent diverse memberships—want to show they support equality for women.

But another part of Fisher’s business is serving about 70,000 individual investors. That kind of money turns out to be a lot stickier—it doesn’t leave easily. In part that may be due to inertia, but it’s also because of the relationship an investment adviser can build with clients. And Fisher has a kind of genius for communicating to small investors that he’s on their side. “He is awesome,” says Kathryn Cardona, a retired teacher and client in rural Oregon, who heard Fisher speak at a luncheon. “He can talk about money and not make it boring. He cracks jokes.” Fisher Investments says it’s brought in hundreds more clients representing billions in new money since early October. Fisher isn’t going anywhere—at least as long as this bull market lasts.
 

Fisher’s father, Philip, was legendary on Wall Street. He was an early practitioner of what’s now known as growth investing and wrote a bestselling 1958 guide to stockpicking that influenced Berkshire Hathaway Inc.’s Warren Buffett. But, whereas Philip offered his investment services to no more than a dozen clients, Ken has a passion for mass marketing. Small investors make up more than two-thirds of Fisher Investments’ business, which Ken founded in 1979 out of his basement.

“Ken himself says, ‘I send junk mail,’ ” says Meir Statman, a finance professor at Santa Clara University, who has collaborated with Fisher on investment research. “There are many professionals who are bashful about marketing. We tend to underestimate the effect of marketing in financial services, as if it were all about alpha,” he says, “alpha” being investor jargon for beating the market.

In 1995, Fisher sent 5,000 letters to high-income prospects—doctors, specifically—at a cost of about $1 a piece. The positive response he got made an impression, and direct mail has captivated Fisher ever since, so much so that he’s used it as a kind of incantation. In the early 2000s he was known to say something that sounded like, “God, I’m someone. God, I’m someone. God, I’m someone,” but it’s really “GDDMSAHNWAM. GDDMSAHNWAM. GDDMSAHNWAM.” It’s an acronym and distillation of his dream: “globally dominant direct marketer separate account high-net-worth asset management.”

In his promotions, Fisher has long stressed his status as a bestselling author, like his father. The younger Fisher had help from his company, which bought copies of his books. Some employees could expense books, which would pile up in warehouses and staffers’ garages, according to people familiar with the arrangement. The company says that it sent books to customers for educational purposes and that they were “a perfectly legitimate business expense.”

Despite being so much in the public eye, Fisher describes himself as an introvert. He can be nerdy, rattling off economic and financial jargon such as “mean variance optimization.” He can also be condescending when challenged. “That’s one of the most basic questions that anyone could possibly ever ask, so let me take you through the basic thinking to see it correctly,” he responds to a question in an interview about his investment approach.

Still he doesn’t come off as a polished finance guy. He drives a 15-year-old Volvo and likes to duct-tape together old shoes. He went to Humboldt State University in rural Arcata, Calif., where for a time he wanted to study forestry. He switched to economics, and after graduation went to work for his dad. While setting up his own business, he used to play slide guitar in dive bars. As it grew, he positioned himself as a market expert and an advocate for the everyday affluent.

Among his books are How to Smell a Rat: The Five Signs of Financial Fraud and Debunkery: Learn It, Do It, and Profit From It—Seeing Through Wall Street’s Money-Killing Myths. He’s very good at showing the many ways brokers, as the joke goes, can make their customers broker. Fisher is best known for his ads that blare, “I HATE Annuities. And you should, too.” He’s even more emphatic in a promotional video, saying, “I would die and go to hell before I would sell an annuity.” He’s picked a fat, largely deserving target. Annuities, which are a kind of investment product issued by insurance companies, are notorious for their punishing fees and confusing rules. And they’re aimed at many of the same people Fisher is selling to—people who may have a half-million or more squirreled away in a 401(k) or an IRA rollover.

Fisher’s ads invite readers to call or click to get a free report. That gets you on the sales-lead list. Some 125 people have filed complaints against Fisher Investments with the Federal Trade Commission since 2016, alleging excessive calls, emails, and mailings, according to records released under the Freedom of Information Act. “CALLS REPEAT EVERY DAY. I SUBMIT A COMPLAINT EVERY DAY. NO CORRECTIVE ACTION TAKEN. WHAT CAN I DO???????????????????,” a resident from Waterford, Pa., wrote in a July 2016 email to the FTC. The company disputes the FTC complaints and says its sales force acts “respectfully and professionally.”

Fisher says the important thing about his approach is that sales and financial advice are kept distinct. Traditional com­mission-based brokers are expected to sell hard—to engage in what Fisher calls “hornswoggling”—at the same time that they’re providing service to customers. “I didn’t like that,” he says. “So I wanted to separate the sales from post-sales service.” Once individual customers sign on with Fisher, his company generally charges 1% to 1.5% of assets per year to run their portfolios. Although that far exceeds the cost of investing in most mutual funds, Fisher says the price includes broader financial advice, as well as extras such as hundreds of private luncheons for clients around the country. “We do things for clients that no one else does,” says Fisher, who calls it “a total offering.”

Bloomberg Businessweek spoke with more than two dozen current and former employees of Fisher Investments. Some describe an always-be-calling operation reminiscent of a David Mamet play. In addition to the Woodside location, Fisher has main offices in the city of San Mateo and Plano, Texas. Several years ago he moved the company’s headquarters to woodsy Camas, Wash., outside of Portland.

Michael Kay, who worked as an account executive from 2017 until June, says he would make at least 250 outbound sales calls a day. “Fisher recruits people out of college,” he says in an interview. “We are poor, smart, and hungry for money, so we can be molded into an efficient closing sales machine.” Kay testified in a 2018 lawsuit in California claiming that Fisher stiffed employees out of overtime—allegations the company denies. A lawyer for the company says Fisher never required so many calls in a day.

When the company loses a client, managers will interrogate investment counselors and rummage through their notes, according to some former employees. Fisher prizes loyalty: He likes to praise those who “bleed Fisher green” and have the initials of Fisher Investments “stamped on their butt,” according to an internal company memo. The company, however, bristles at the cult comparison, saying it prizes independent thought.

Male entry-level account executives sometimes displayed a frat house attitude. “Who do you think is hung like a horse?” one person wrote in a 2013 email. Another named a female co-worker and, in an apparent reference to her genitalia, wrote, “That’s her Laybes though bro.” Fisher provided these emails when Bloomberg asked about the behavior of one of the employees, who’s still with the company. After the email episode he was demoted, was docked three days’ pay, and had his compensation cut to $14 an hour from $24 an hour. Fisher says such episodes inevitably occur in a company with 3,800 employees. “When we find out about things, we deal with them pretty completely,” he says.

Still, some employees complain of repeated comments about women’s clothing and appearance. “I got to hear and witness a lot of things that didn’t sit well with me,” says Joey Hays, who worked in the marketing department from 2013 to 2015 and sat near the sales bullpen. “No sooner would a woman leave the area, and these men would start initiating highly sexualized conversations about these women.” Hays, who is gay, says he also heard employees make homophobic remarks and vulgar, sexual comments about clients. (The company says a female employee once complained to its human resources department about Hays, saying he made an inappropriate comment; Hays says he and others joked about the co-worker, and he later apologized.)

Fisher’s own words were part of the macho atmosphere. “I have a hard-on looking at all of you,” some people remember him saying at a London gathering. Back in 2008, as financial markets were reeling, a woman at a staff meeting asked Fisher if he’d considered shifting into defensive holdings. “Why would I want half a dick?” he replied. Fisher doesn’t dispute that he made comments such as these, though he says he may have been misunderstood.

In a recording from a 2018 investing conference, Fisher can be heard saying: “If I was 30 years old and I had to do it over again, I would have more sex.  … Once you get older, you’re like a Christmas tree. You’re firm once a year, and the balls are just for decoration.” Some in the audience laughed.

A number of women at Fisher Investments say they’ve been fairly treated in what they consider a meritocracy. The company notes that three of five senior executive vice presidents reporting to Chief Executive Officer Damian Ornani are women. One of those leaders, Jill Hitchcock, who oversees the company’s largest business unit and was the former head of human resources, says Fisher’s comments have never offended her because she always considered him to be using analogies to explain complex topics. “If I didn’t believe in what we were doing and the firm and the culture, I wouldn’t have stayed here,” says Hitchcock, a widowed mother of two boys, who’s worked at the company for 20 years. “I’ve also seen how our employees and how our clients react to seeing Ken, which is they love to see Ken speak.”

The company points to an internal survey showing 79% of employees think it’s a great place to work. Jacob Gamble, who worked there from 2002 through 2015, says the company does well by its clients. “It can be hard, and you are held accountable to learn from every single little flaw that comes across in your work,” he says. “But that’s how they’re successful and how they move forward.”

If Fisher’s sales force is aggressive, so is his investment approach. When prospects turn over their nest egg to Fisher, his company often puts the bulk of it in stocks, even for older clients. That’s an outlier in an industry that typically recommends a more conservative approach: All things being equal, younger people should go heavier on stocks, retirees should go in a more fixed-income direction, and so forth. In retirement funds designed for 65-year-olds, Vanguard and Fidelity place just over half of investments in stocks; Charles Schwab and BlackRock, 40%. Fisher has little time for the idea that it’s safer to keep a lot of money in bonds—not when you need to grow enough savings to fund retirement. On a podcast, Fisher once referred to husbands who put their spouse’s money in a mix of stocks and bonds as no better than “wife beaters.” Fisher says, for example, a 65-year-old man may marry a woman 10 years younger who can expect to live many more decades. That means the couple has to plan for a longer future. “If you have a 30-year time horizon, you ought to be starting off with a 100% equity benchmark,” he says.

Besides stocks, another investment he likes is an obscure instrument called an exchange-traded note, or ETN. They’re essentially contracts, issued by a bank, that entitle the owner to a certain payout. The notes’ returns can be linked to the performance of other securities, such as an index of stocks. And they can be designed so they return even more when those stocks rise—and fall even harder when the stocks fall. The risk and costs of ETNs have drawn scrutiny from federal regulators, and most investors shun them. (Although ETNs are backed by major banks, one risk is the possibility that an issuer might not be able to pay.) Fisher Investments is one of the biggest ETN players in the world, with $6 billion of clients’ money invested in them last year, making up a quarter of the market. The company says its ETNs are less volatile than most individual stocks and work in its portfolios to reduce risk relative to return.

The main investment strategy he uses for wealthy clients’ accounts has trailed the benchmark the firm uses over the past decade by about one percentage point annualized. Over the past three and five years, however, it has beaten the same index by a percentage point annually, according to the company. The portfolios he runs for institutions tend to have lower fees, and some have performed better. For Florida’s pension fund, the firm returned, on average, 2 percentage points per year above its benchmark over the past 10 years.

No doubt, leaning into a bull market has been good for Fisher. After all, this one’s been going for more than a decade. Cardona, the Fisher client, says she has confidence in her 100% stock portfolio and also likes the customer service she gets. As for the remarks that got Fisher in trouble, she says people are too politically correct these days. “What he says doesn’t bother me as it doesn’t impact the 401(k),” says Cardona, who is 74.

In a bear market the company’s aggressive stance can backfire. Fisher faced at least a dozen legal claims from customers who lost money in the 2008 financial crisis. Clients alleged that the company advertised a tailored approach while putting everyone in the same stocks, misled them about risks, or failed to build suitable portfolios. Most were resolved in arbitration, where the outcome wasn’t always public. One arbitrator ordered Fisher to pay out $376,000 in damages to a retiree in New Orleans who signed up in September 2007 and lost 57% of her $876,000 nest egg in a little more than a year. But in some other cases, arbitrators sided with Fisher, saying that clients had sought out “growth” portfolios of stocks, understood the risks they were taking, and would have prospered if they had ridden out the downturn as the company advised.

Several former employees say Fisher would identify clients who know prominent people—such as lawyers, politicians, and journalists—flagging them in case they became unhappy with their investments. A 2015 corporate script tells investment counselors to determine whether customers have such high-profile people in their “sphere of influence.” Fisher denies the effort was designed to protect the company. Instead, he says, the script was part of a since-abandoned effort to create a panel of experts the company could harness for research. “This was effectively an innovative attempt that failed,” he says.
 

In response to the negative publicity about Fisher, the company introduced a counteroffensive. On Oct. 29, Hitchcock, the 20-year veteran and senior executive, asked hundreds of women employees to gather for a last-minute meeting at Camas and three other main offices. She asked them to sign releases permitting them to appear in photos for an ad campaign. More than 1,000 employees, or more than a quarter of Fisher’s workforce, are women, and 350 appeared in the ad, the company says. Full-page spreads would ultimately run in the Wall Street Journal, the Dallas Morning News, and newspapers across the country under the headline “You Heard Their Story. Now Hear Ours.” Along with the group photo, it featured seven senior women offering testimonials about the company’s integrity, ethics, and empowerment of women.

Privately some of the women in the ad say they felt pressure to participate. In interviews others say they were delighted to vouch for the company. “People are treated equally here regardless of race, gender, and culture,” says Jessica Smith, a Fisher Investments vice president. “This is a place that really strives to do what’s best for our clients, and that’s much different from an industry that’s just trying to make money.” The woman-focused campaign referred readers to a website that asked for phone numbers, addresses, and emails—making it a source of fresh sales leads.

Back in his Woodside mountain redoubt, Fisher says he’s proud of the culture he’s built, both for men and women. (The company has launched a diversity and inclusion task force.) “Businessweek Best Places to Launch a Career,” reads an award in the office. As he did when evoking the days of the family farm, he can sound like a kind of patriarch as he mourns what he calls “a world that doesn’t exist anymore but existed when I was very young.” Fisher wants young people to join his company out of college and stay until they retire, as they did in the age of the IBM lifer. He revels in noting managers who have been with him for decades. Ever since the backlash against his comments began, Fisher has expressed bewilderment that outsiders would react so strongly. “I realized that there’s no place for that in the work world in any way at all, and I committed myself to never saying words like that again or any things like that again ever,” he says.

During the interview he speaks slowly, as if measuring his words, and pauses frequently. Fisher says his own employees, with whom he’s long worked side by side, never said anything to him when he made an off-color analogy—not once, not ever. “Those people know that I’m not perfect,” he says. “But they also don’t get offended by anything that I do because they know how hard I try for them and have tried for them for a really long time.” —With Janet Lorin
 
Read more: Nobody Makes Money Like Apollo’s Ruthless Founder Leon Black

©2020 Bloomberg L.P.