Amazon’s Most Ambitious Research Project Is a Convenience Store

The Amazon chief executive officer and several assistants pretended to shop, pushing grocery carts down aisles stocked with canned food and plastic fruit and vegetables. There were specialty counters where Amazon employees posing as baristas, butchers, and cheesemongers took orders and added items to Bezos’ imaginary bill.

Afterward, according to a person who was there, Bezos gathered the project executives and told them that while they all had done a fabulous job, the experience felt disjointed. Customers would have to wait for meat, seafood, and fruit to be weighed and added to their bill, which would have been fine except that the major selling point of the store was supposed to be the absence of time-wasting checkout lines. Bezos asked the group to lose the meat and cheese and focus on getting rid of lines and cashiers. “It was one of those Amazon things,” another employee recalls with regret. “We love it—let’s change everything!”

Almost four years later there are 14 Amazon Go stores in Chicago, New York, San Francisco, and Seattle. They’re about a quarter the size of the original mock-up, located in downtown office districts and offering a small selection of sandwiches, meal kits, and convenience store items such as sodas, jams, and potato chips. Just as Bezos had hoped, there are no cash registers. Once customers have scanned a screen from a special app on their phone at the entrance, they just grab their items and walk out the door, while Amazon magically charges their credit card. By all accounts, the company intends to open more of these stores in the months and years ahead.

From a technological perspective, the Go stores are a marvel—a succinct demonstration of Amazon.com Inc.’s capacity to devote vast resources toward applying the state of the art in artificial intelligence to an everyday problem. They also illustrate the company’s tendency to pursue technology for technology’s sake (see: the Fire Phone), resulting in a store that offers all the selection of a 7-Eleven, but with more complexity and cost. Scores of cameras pointed at all angles hang from the ceilings to track shoppers as they wander the aisles, while precise scales embedded in the shelves tabulate products down to the gram to figure out which ones have been picked up. Behind the scenes, sophisticated image recognition algorithms decide who took what—with Amazon workers in offices available to review footage to ensure shoppers are accurately charged. Each store also has a local staff on hand to help people download the Go app, restock shelves, and, in locations with a liquor section, check IDs.

Will all this work be worth it? Some Go stores seem almost deserted except for the lunchtime rush. Employees familiar with Amazon’s internal projections say the outlets in Chicago, in particular, are falling short of expectations, and the company has had to resort to raffles and giveaways of tote bags and other branded goodies. Yet, as the turbulent history of the project suggests, the Go store isn’t so much the culmination of the company’s efforts but something closer to an ongoing experiment. And the potential prize—a big piece of the $12 trillion grocery industry—is one that Amazon, with its limitless resources and appetite for risk, may be in the best position to claim.

Analysts and investors for years asked Bezos whether Amazon might open stores. His answer was usually some variation of “We would love to, but only if we can have a truly differentiated idea,” as he told an interviewer in 2012. “One of the things that we don’t do very well at Amazon is do a me-too product offering.”

It was that summer when he started to think seriously about the opportunity offered by physical retail, which captures 90% of total retail sales in the U.S., according to the Census Bureau. Bezos could see that for a company of Amazon’s size to keep growing, it would have to get into new industries. (The development of the voice-activated Alexa assistant and the creation of the Amazon Studios division, which produces shows such as Bosch and the forthcoming Lord of the Rings prequel, were undertaken around the same time.) To lead the initiative, Bezos tapped Steve Kessel, a senior vice president who’d been in charge of the company’s efforts to develop the Kindle and drag the publishing industry into the age of digital books.

Kessel asked Gianna Puerini, who’d overseen Amazon’s homepage and product recommendations division and who at the time was retired, restoring houses in the Seattle area, to lead the development of the product. Puerini (who retired again earlier this year) set up in a nondescript six-floor building in South Lake Union, a few blocks from Amazon’s headquarters. Because the project was to be secret even from other Amazon employees, one of her first tasks was selecting a code name so boring that no one would pay attention to it, a former colleague says. For the next few years, the team would go by the name IHM, or “inventory health management.”

To oversee engineering, Kessel recruited Dilip Kumar, Bezos’ shadow, or technical adviser—the Amazon employee with the high-profile role of essentially following the CEO around and sitting by his side in meetings for a year. Kumar occasionally dabbled in stand-up comedy at local open-mic nights, but colleagues say at work he was known to be intense and combative.

IHM employees say the early months were filled with open-ended brainstorming and debate. They considered whether they should do Macy’s-style department stores, Walmart-style supercenters, or even electronics stores. One discarded idea involved two-floor stores, with Amazon’s disk-shaped warehouse robots assembling orders on the top floor and then conveyor belts and robots delivering them to customers’ waiting vehicles below.

After a few months, Kumar, Puerini, and their colleagues conceded that most stores in the real world already operate tolerably well, except for one glaring exception: the supermarket, with its irksome checkout lines. Americans shop for groceries an average of two times a week, and the experience of waiting in a checkout line epitomized—to Amazon’s crack team of Type A disrupters, anyway—the spirit-draining unproductivity of offline shopping. “We realized that there’s a lot of good things about shopping in physical stores, but waiting in lines was not one of them,” Kumar says.

Plenty of companies have tried to address this hassle. Apple has employees roving its stores with credit-card-reading tablets, and China’s BingoBox offers self-checkout using RFID chips attached to product packaging. The IHM team wanted to eliminate the bottleneck altogether. In an Amazon tradition meant to ensure teams are working backward from customer needs, they started with a press release, or “PR Faq” in Amazon-speak, announcing the opening of a store without a checkout line. Then they began working on the actual technology to make the store a reality.

It would turn out to be much more difficult, and expensive, than anticipated. To figure out who was buying what in a store without checkout lines, IHM engineers considered using RFID, tracking customers’ cellphones as they walked the aisles, and scanning their face with facial recognition technology. They also discussed asking customers to quickly scan QR codes when they selected items, but even though that would make Amazon’s job easier, it could be weird or unnatural for customers. Finally they settled on using computer vision, a relatively new technology that allows digital cameras and computers to identify items by their visual appearance alone, without any special tracking chips or codes.

Kumar hired computer vision and machine learning scientists, often from other parts of Amazon, without telling them exactly what they’d be working on beforehand. He set deadline after deadline, using upcoming presentations to Bezos or Kessel to motivate them. Engineers put in 70 to 80 hours a week, answering emails and writing Amazon’s classic six-page documents, narrative memos that outline a proposal, in any illusory free time they had on nights and weekends. “We were all living in a cave,” says one.

At first, the IHM team envisioned large-scale stores of about 30,000 square feet, roughly the size of a suburban supermarket. But after a few months, the group decided such a megamarket was overly ambitious and cut the size of the proposed store in half.

Puerini’s group created the first models of stores using kids’ blocks, bookshelves, and other items lying around the office. As the project neared a hoped-for introduction in mid-2015, the company retrofitted the warehouse in south Seattle for a mock-up to show to Bezos. For its first actual store, it also anonymously leased the ground floor of a new luxury apartment building in Seattle’s wealthy Capitol Hill neighborhood. Permits filed with the city included plans for large produce and dairy coolers and an on-site kitchen for preparation of fresh foods.

But then Bezos visited the mock-up with the cheesemongers and put a halt to the progress, betting customers would be drawn to a more streamlined experience—the physical equivalent of the company’s famous one-click ordering—even if it lacked the personal touches of a farmers market or boutique butcher shop.

Kessel convened a team meeting after the Bezos demo and broke the news: They were pivoting to convenience stores. Some engineers were relieved that they were reducing complexity by eliminating items of varying weights, such as produce and meat. Others were crestfallen and left the project, either exhausted from the nonstop pace of work or disappointed by the scaled-down vision. For the next three years, the storefront on Capitol Hill would sit abandoned, in the heart of one of Seattle’s most well-trafficked neighborhoods, its windows mysteriously covered in brown paper.

Bezos and Kessel were growing impatient. So in March 2015, while Puerini and Kumar reworked their concept, they formed a separate group under Kessel to open bookstores. Books were the opposite of food—nonperishable, consistently priced, easy to stock, and, of course, the product category Amazon had the most history with. And because people tend to shop bookstores at a more leisurely pace, it wouldn’t be necessary to try to displace cashiers with technology.

That fall, as the company prepared its first Amazon Books outlet in an upscale mall in Seattle, speculation over how the company would enter physical retail was so feverish that a reporter for GeekWire used a pole with a camera attached to it to peek inside. Around the same time, Bezos sneaked in through a back door to see it for the first time and was delighted. He said he felt as if Amazon’s business was coming full circle.

To longtime members of the IHM project, watching Amazon Books form and take off within the span of a few months was dizzying. They’d been working for three years, and their project didn’t even have a formal name yet. To get one, in early 2016, Puerini’s team came up with the Go brand to convey speed. “Even the word itself is only two characters,” she says. “You can literally grab and go.”

To continue developing the technology, Kumar’s engineers set up a top-secret lab store on the ground floor of the team’s new building, dubbed Otter, on the corner of Fifth Avenue and Bell Street in downtown Seattle. The lab in Otter was accessible only from inside via a pair of locked doors. At first, shelves were packed with fake food fashioned from clay and Styrofoam; shredded green construction paper stood in for lettuce. Employees were frequently asked to visit and to try to fool the technology. They wore heavy coats, walked with crutches, or pushed wheelchairs. They put items back in the wrong place, generating an automatic “untidy item” alert that directed a store clerk to restock the item on the proper shelf. One day, everyone was asked to bring in umbrellas to see if they’d obscure the cameras’ view; on another, employees all wore Seattle Seahawks jerseys to confuse algorithms that distinguished among shoppers based on the color of their clothing.

When the fake food was eventually replaced with real items, employees were asked to shop but under specific scenarios: for example, Puerini recalls, “You’re running to a meeting: Buy a salad and a drink for lunch,” or “You’re in a rush to pick up the kids from day care: Grab milk, strawberries, and cereal for tomorrow morning.” On another day, parents were asked to bring in their young kids, who fidgeted, ran around, and grabbed things, further stress-testing the system. To augment these real-life experiments, the company also developed a digital simulation of the store and populated it with computer-generated shoppers.

Kumar’s engineers were trying to solve one of the trickiest problems in the history of retail: how to figure out what people are grabbing without going through the items one by one at checkout. After years of work, the team had concluded that visual identification of products solely with overhead cameras wasn’t possible. Changes in lighting conditions over the course of a day, the depth of a product’s placement on shelves, hands and bodies that concealed the customized product stickers, or out-of-control toddlers could easily confuse the system.

Eventually they decided to add scales and place more cameras inside shelves. (“Weight provides an additional signal we can use, but most of the heavy lifting is done by the cameras and the vision algorithms,” Kumar says.) Amazon then combined the data to decide who was buying what.

But humans were still needed to supervise these judgments. Separate teams were formed to review footage when the system wasn’t sure about a purchase, a so-called low-confidence event. The creation of the groups led at least some employees to question the entire effort. It “was a tricky thing,” one former participant says. “If we have an army of people looking at footage, is that scaling properly?” (Amazon says human intervention is rare.)

People had another role to play as well: They had to develop meal kit recipes and prepare the daily lunch fare (lamb sandwiches, chicken banh mi, caprese salads). To get ready for the opening of a scaled-down prototype store on Amazon’s new downtown Seattle campus in late 2016, the company hired chefs and staff from chain restaurants. It opened both a kitchen inside the prototype store and a commercial-grade test kitchen near the old warehouse in south Seattle. Uncharacteristically, Amazon splurged. It bought German commercial ovens that cost tens of thousands of dollars each. When something smelled off in the pilot kitchen, Amazon hired a pair of professional smellers to solve the mystery. (The culprit: pickled daikon.)

The kitchens, along with Amazon’s penchant for pursuing rigorous and sometimes inhumane efficiencies in its operations, brought with them another set of unexpected challenges. Because food safety was a top priority, the commercial kitchen was kept bitterly cold, and Amazon initially refused requests from the staff, who had to stand on their feet during shifts, to install mats on the facility’s chilly concrete floor, one employee recalls. After a senior manager from headquarters spent a day observing operations at the cookhouse, the company issued the kitchen staff hoodies and other cold-weather gear. The people involved in the service industry, it turned out, were proving as tricky to manage as Kumar’s algorithms.

The original Go store opened to Amazon employees in December 2016, but a public opening, scheduled for early 2017, was delayed another 12 months. The system tended to freeze when 20 or more shoppers were in the store at the same time. It lost track of products when shoppers picked them up and set them down on a different shelf. The shoppers themselves also got confused. “We noticed lots of customers hesitating at the exit, asking the entry associate if they really could leave,” Puerini says. “In tests we put up a big poster that said, ‘No, really, you can just walk out!’ ” A version of the sign is still there.

Amazon also tinkered with food preparation. It started relying less on its own kitchens and buying more food from outside vendors, including Taylor Farms, which makes salads and sandwiches for Starbucks and 7-Eleven, among others. Those pricey German ovens apparently still sit unused in the original store.

Early on, the Go team envisioned thousands of stores, in every major urban area. “We always wanted to be on every corner,” a former executive says. “We wanted to be as common as Starbucks.” But now, seven years into the project, Amazon is just getting to its 14th store, in San Francisco’s downtown Embarcadero Center. The company has also dramatically slowed the opening of Amazon Books and introduced Amazon 4-star stores, another new format that features a selection of well-reviewed items and Amazon gadgets. These experiments in physical retail hardly affect the company’s financial results in the way Bezos had in mind when he instigated the project in 2012. It’s easy to imagine the CEO cutting bait: He’s talked in the past about trialing concepts for seven years before expecting a financial return.

At the pace it’s going, the Go store will greatly exceed that span before it can hope to pay off the investment. People familiar with the project estimate Amazon has spent hundreds of millions on it, including from $2 million to $3 million on the pilot store alone. One former employee claims it’s one of the most expensive research and development projects in the company’s history, though Kumar disputes that, saying the stores use off-the-shelf hardware and Amazon’s existing cloud computing infrastructure. Still, considering the dense placement of cameras and sensors, and the tech-support crews that are on call at all hours of the week, it’s much more expensive than running, say, a 7-Eleven, which could be staffed by a single cashier and—with the possible exception of the Slurpee machine—have little in the way of bespoke technology.

In customary Amazon fashion, Kumar professes that it’s “still early” for the Go project and notes that “customers love the experience” of walking out without stopping to pay. Analysts (and Yelp reviewers) mostly agree, comparing the experience to the feeling of going through TSA Precheck in the airport: Once you get used to it, you don’t want to go back. That, Kumar says, gives the project “a lot of latitude and degrees of freedom to be able to try other kinds of things.”

It’s those “other things” that beguile investors and observers of the company. Incidents of flops that subsequently lifted Amazon pepper its history, such as the early auctions business, which led to the wildly successful introduction of third-party sellers, and the Fire Phone, many of whose engineers later applied the lessons of failure on Alexa. “Like so many things Amazon does, I’m sure it doesn’t look at it as a convenience store, doesn’t look at it as a bookstore, but looks at it as a data experiment,” says Neil Stern, a senior partner at McMillanDoolittle, a retail consultant. “The stores themselves are not the big idea.”

Kumar himself is cagey about future plans but notes the Go technology could be adapted outside the convenience stores. “If it makes sense for other things, we’ll do it there,” he says.

Meanwhile, Amazon’s commitment to physical retail seems to be expanding. Over the past few years, Kessel has been given oversight of Prime Now and AmazonFresh, the company’s fast-delivery and fresh food operations. When Bezos acquired the Whole Foods Market franchise over the summer of 2017, Kessel was also put in charge of 500 or so Whole Foods stores—and thousands of conventional checkout lanes that require the old-fashioned act of waiting to pay.

And then there’s the midsize grocery store on Capitol Hill that Bezos scotched in the fall of 2015. Earlier this year, Amazon quietly filed new plans with the city of Seattle and, to the relief of neighbors, resumed work on the empty space. Plans for an on-site kitchen were withdrawn, and “optical speed lanes” were added to the blueprints. The store, at more than 10,000 square feet, is significantly larger than the conventional Go format, and the newer concept remains tightly under wraps, with window frosting obscuring the facade that looks out onto East Pike Street.

But if you stand on the sidewalk and squint through a gap in the frosted glass, you can just make out the telltale shelving of what appears to be an Amazon Go store.

©2019 Bloomberg L.P.