You Could Be Competing With Bots to Buy Gifts This Christmas

(Bloomberg Businessweek) -- Jason Kent, a self-described “hacker in residence” at Cequence Security Inc., sat down in front of a pair of computer monitors at his home in Hilliard, Ohio. On them he watched as one of his clients sold off millions of dollars in assets in thousands of transactions conducted across mere minutes. Kent was acting as a digital sentry, guarding not against theft but against arbitrage, in this case of limited-edition sneakers that a sports apparel retailer was releasing for sale on its website: Nike SB Dunks, the latest Yeezys, and some Air Jordans.

Kent was there to make sure ordinary people had an honest shot at buying these sought-after shoes. Standing in his way, on this particular day last March, was an army he describes as “15-year-old kids sitting in a basement somewhere making $200,000 a year reselling sneakers.” To siphon up shoes, they deploy automated shopping software, or “bot” programs, such as Cybersole, GaneshBot, and Kodai, which have roiled the sneaker market and spawned a bot economy that some experts fear could wreak havoc on a number of retail categories during a holiday shopping season that’s already been upended by supply chain drama.

One of Kent’s monitors displayed chatter from Slack channels and Discord messaging groups, where he could communicate with his security team and eavesdrop on “cook groups”—gangs of bot users—as they coordinated campaigns. On the other, he could see detailed data on his client’s inventory. The sale would pit his company’s proprietary software against the bots, which went into action about five minutes before the shoes went on sale. The cook groups deployed scraping software to scour the website for the stock-keeping units, or SKUs, associated with new inventory. As new SKUs came online, the bots would add each item associated with that unique ID number to a shopping cart, and once the sale started they would try to complete the checkout process using preloaded credit card or gift card information. Running this routine many times per second, a single cook could buy hundreds of pairs, which were ostensibly limited to one per customer. “It was painful,” Kent says. Many retailers tend to struggle with web traffic beyond 60,000 page requests per second; coordinated bot campaigns can add tremendous strain. “In this case they tripled that traffic,” he says. “It was just insane.”

For cooks with some software and a little hustle, even a modest haul from the right drop can yield significant profits on StockX and other resale sites. “When the Yeezy ‘Sun’ drop happened, that shoe cost maybe $250 retail, and you saw them on EBay for $600 that same day,” Kent says. In July 2020, Cowen Inc. estimated the sneaker resale market was worth $2 billion in North America alone, with plenty of room for growth. And because Nike Inc. and Adidas AG continue to release popular shoes in relatively limited quantities, retail bots play an increasingly prominent role in determining who gets to buy them and how much they have to pay. They can render price tags all but irrelevant, valuing a pair of sneakers at whatever someone is willing to pay and driving appreciation in the few moments it takes resellers to corner the market.

With supply chains everywhere being throttled, the pain once reserved for sneakerheads appears to be spreading. Bot use has grown and branched out throughout the Covid-19 pandemic, according to Patrick Sullivan, chief technology officer for security strategy at Akamai Technologies Inc., one of the world’s largest providers of security and other network services. “People will try to jump the line and leverage automation to grab anything that has a limited inventory,” he says. “It used to be concert tickets, then purses and tennis shoes, and now it’s vaccine reservations and even more mundane things.”

Big box retailers such as Walmart Inc. are deploying Akamai to stop bot users from gobbling up limited inventories of sought-after items such as Sony Corp.’s PlayStation 5 console, which has been on the market for a year and still sells for more than $300 above retail on StockX. On the resale site’s “collectibles” page, you can find everything from Star Wars Lego sets to Oculus virtual reality headsets to sealed boxes of Pokémon, Yu-Gi-Oh!, and Topps trading cards going for above list price.

Akamai saw a taste in early October of what U.S. consumers could be in for, as bot traffic surged in India during the online shopping rush that typically takes place ahead of the five-day Diwali holiday, which this year begins on Nov. 4. For a two-week period, potentially malicious bot activity targeting companies based in India jumped more than 55% over the previous two weeks. The bots “included web scrapers, automated shopping cart ‘sniper’ bots, and login and checkout abusers,” says Timothy Whitman, a spokesperson for Akamai. “We think this is a sign of things to come for holiday shoppers in the U.S. and Europe.”
 

In 2006, Eric Budish was a Ph.D. candidate at Harvard, carrying out research on the economics of the concert ticket market. The subject had fascinated him since his teen years, when he spent countless hours lined up at New York City Ticketmaster locations for hot concerts, elbow to elbow with old-school scalpers. In one study, Budish explored Ticketmaster Inc.’s attempts at thwarting scalpers, who’d begun to repurpose bots favored by EBay sellers looking to automate some aspects of the nascent online junk sale economy. Compact discs, video games, and DVDs that could sell for $5 or $6 on one site might be available for $2 or $3 on another—low-hanging fruit for a power seller with a rudimentary web-scraping script. The commodities they dealt in were a volume business, but they were cheap enough for buyers that few noticed or cared much about the role bots were playing.

That all changed with the rise of StubHub, the first major resale marketplace for event tickets. Within a few short years of its founding by a pair of investment bankers in 2000, StubHub became vastly more stable and transparent than the informal haggling process that had taken place outside concert venues for decades. But tech-savvy scalpers quickly caught on to how valuable bots could be. From 2002 to 2009, a single company, Wiseguy Tickets, made more than $25 million in bot-fueled profit. And the more tickets its software secured to a particular event, the more it controlled pricing. “For years performers had priced tickets below the market-clearing price, either out of fairness or out of concern for the long-term value of their brand,” says Budish, who’s now a professor of economics at the University of Chicago’s Booth School of Business. “That system kind of worked, but then the internet broke it by giving an economy of scale to the brokers using ticket bots.”

In 2007, EBay Inc. paid $310 million to acquire StubHub, which got a boost that same year when New York state eased regulations meant to curb ticket scalping. Within a few years, Ticketmaster, a longtime leader in the live concert ticketing business, pegged the ticket resale trade at about $15 billion annually and found that bots were siphoning up as many as 60% of the available tickets for some popular events. A lawsuit filed by the company in 2013 accused a single group of scalpers of requesting 200,000 tickets per day using bot software. Finally, in 2016, Congress sought to curb the practice with the Better Online Ticket Sales (or BOTS) Act.

That’s around the time Lucas Titus, the 19-year-old founder and chief executive officer of bot maker Cybersole Ltd., got into the game. One Saturday morning that year, with few commercial retail bots available on the open market, Titus, still in secondary school in London, used a simple open-source script to help buy some Yeezys, which he flipped for a profit. “Once I started to get into reselling, I quickly understood that the only way to make a good profit was to automate the process,” Titus told me in May.

That process used to involve camping out, rather than high-tech arbitrage: For four days in February 2005, almost 100 sneakerheads slept outside New York City’s Reed Space boutique for a chance to buy the Nike SB “Pigeon” Dunk designed by the store’s owner, Jeff Staple. “It was blizzarding,” he later recalled in an interview with blogger Justin Block. “I felt bad. Every night I would buy pizzas for the kids, because they were sleeping outside in a snowstorm for four days.” When the police tried to break up the line, a riot ensued, and a legend was born. “A lot of people call this shoe the shoe that catapulted sneaker culture to the masses,” Staple said.

As sneaker culture increasingly moved online, even giants such as Nike and Adidas struggled with server crashes and fulfillment issues. That started to change in 2011, when New York boutique Kith partnered with e-commerce platform Shopify Inc., which began working out how to manage the flash-sale conditions of sneaker and streetwear drops. But by the time the BOTS Act came around, developers were building workarounds. And because sneakers and other goods aren’t covered by the BOTS Act, Titus was free to build on the legacy of the high-tech ticket scalpers.

Each type of online retail experience became a unique problem to solve. For retail websites that make customers wait in virtual lines, using multiple Google profiles increases the chances of passing through them; for sites using virtual raffles, creating simple scripts allows for more than one entry. The more Titus learned about coding, the more sophisticated his scripts became. Before long, sneaker resellers were reaching out to him on Twitter to ask about buying his software, which in 2018 led him to hire staff and start working on Cybersole full time. He now offers it as a subscription-based service, charging some 5,000 users £300 ($407) for a six-month license.

The bots have become impossible for big shoe companies to ignore. In March, Ann Hebert, Nike’s vice president and general manager for North America, resigned after a Bloomberg Businessweek cover story detailed a lucrative sneaker reselling operation run by her son, Joe. The sneaker news site Complex reported that Nike CEO John Donahoe addressed the departure days later in a virtual meeting with employees, acknowledging that the incident may harm consumer trust surrounding product launches. The company hasn’t publicly addressed the Businessweek story since it was published, and Ann Hebert didn’t respond to requests for comment on that piece or this one.

Nike spokesperson Sandra Carreon-John said in the first story that Hebert had disclosed relevant information about her son’s company to Nike in 2018, and that “There was no violation of company policy, privileged information or conflicts of interest, nor is there any commercial affiliation between WCS LLC and Nike, including the direct buying or selling of Nike products.” Carreon-John declined to respond to inquiries for this piece about why Hebert had resigned but did say, in response to a list of questions about Nike’s approach to bots, “We are constantly looking at the best way to combat bots across our digital ecosystem. Nike is fully committed to making sure that our real, loyal consumers are the ones who get fair access to our products and that we continue to evolve best-in-class solutions in the marketplace.”

Despite Nike’s internal efforts, which Carreon-John declined to describe, bots continue to bring unprecedented volatility to what was once a straightforward retail experience. “People are still asking for transparency, and they are still frustrated and angry,” says Jacques Slade, a sneaker influencer with a popular YouTube channel. “For those who are outside the sneaker bubble, it’s especially confusing. They think, ‘Well, the iPhone is super popular, but Apple will still sell me one, so why can’t I get a pair of Jordans?’ It just doesn’t make sense to them.”

Kent, of Cequence Security, sees signs that Nike is taking the problem seriously, not just for its own direct-to-consumer sneaker drops but in its relationships with retailers. In March he told me he expected the company would eventually threaten to cut allocations of sneakers for retailers that don’t take the problem seriously enough. “We’ve actually had retailers tell us that when they proved that they could get through a drop all right, their Nike reps lit up and started talking about more allocation,” Kent said.

How serious the retailers are is a matter of much online speculation. Titus sees major third-party dealers such as Foot Locker Inc. as ambivalent about stopping bots because, frankly, they’re good for business. “They create a huge amount of demand for sneakers and ensure that they sell out extremely fast,” he told me in May. (Smaller, boutique-style sneaker shops still look to e-commerce partners such as Shopify for their anti-bot solutions, as a recent New York Times article showed.)

Last year, near the end of October, Titus and his team at Cybersole noticed something big: Foot Locker, which had been contracting Akamai to protect its retail websites from near-constant bot attacks, suddenly switched to a different set of tools. “They had been using Akamai for as long as I can remember,” Titus says. “And it’s always been considered one of the most difficult anti-bot solutions to get through, because you need to solve various challenges to beat it, such as proving that you’re typing on your keyboard and using your mouse. Most bots aren’t able to beat it.”

Akamai’s services are ubiquitous, and also among the most expensive on the market. In recent years, as bots and resellers proliferated, more retailers began contracting with Akamai to protect themselves—its clients now include Adidas, Nike, Walmart, and Yeezy Supply. Foot Locker’s sudden reversal surprised Titus: For resellers, the company dropping a gold-standard provider made its online sales more vulnerable to bots. The difference was big enough that Titus and his team saw a dramatic drop in their weekly workload. “I think they’ve given up on stopping bots and are now focused on slowing them down,” he says. If so, he adds, “it would make sense to switch to a cheaper alternative than Akamai.”

Representatives from Foot Locker declined to comment on its decision to part ways with Akamai. Cara Tocci, Foot Locker’s vice president for corporate communications, did say that it was “making significant changes that we are excited to share in the future” but declined to provide details or a timeline for implementation. In the meantime, according to Titus, Foot Locker’s retail sites are vulnerable enough that half a dozen new bots have sprung up to take advantage. “That’s made things quite difficult for us, actually, because we have that much more competition,” he says.
 

Despite Foot Locker’s departure, Akamai’s anti-bot services remain extremely profitable, according to Sullivan, the CTO for security strategy. “We have a line of business built around bots that brings in just under $200 million a year,” he says. “It’s growing at 40% annually, which makes it one of the fastest-growing areas of cybersecurity.” For years, growth was driven by efforts to combat scalping of concert tickets, luxury handbags, and sneakers. But the pandemic proved that bots can be quickly adapted to exploit any market inefficiency. Along with Covid vaccine reservation systems, Sullivan says, such products as N95 masks and hand sanitizer were popular targets of bot attacks when supplies were low.

With each advance, security experts scramble to adapt, then bot software teams respond, and the pattern repeats again, in what Sullivan calls a “very active software development cycle.” The cycle’s speed also arises from the atomized, highly competitive nature of the bot market, where players range from individual resellers writing their own code to flash-in-the-pan bots cooked up by weekend warriors to established subscription services like Kodai, which has dozens of employees. What they’re up against, when it comes to top-flight cybersecurity firms like Akamai, are machine-learning tools that act as a kind of Turing test. “What they are doing to some extent, as a bot, is telling a lie, so the question becomes whether we can spot the lie in what’s happening,” Sullivan says.

Bot developers use their own machine-learning tools to come up with more effective ways of lying. During global product launches, which unfold over many hours as the sun rises across time zones, a bot’s behavior can evolve fast enough that its strategies for gaming a drop will be different from one country to the next. “There’s a lot of intellectual capital being invested on both sides of this operation,” Sullivan says. “It’s very much a cat-and-mouse game.”

Asked how long it might take until bots are so deeply enmeshed in the fabric of retail markets that they can’t be taken out of it, he replies, “We’ve already passed that moment.” If his company were to set up a site for a startup retailer, as a sort of experiment, he figures, “it would be hours, not days, before we were visited by probes from attackers. It’s all automated now, so there’s this constant scanning for vulnerabilities and opportunities.”

Demand for Cybersole, like Kodai and other major retail bot software, has soared high enough during the pandemic that the company has capped subscriptions. Otherwise it would be too hard for Titus and his staff of 12 to keep on top of software updates and customer service issues. It would also risk devaluing the product. “With a limited amount of stock in each store,” Titus says, “our users’ chances of success start to decline if there are too many of them out there.”

And so, naturally, a secondary market for bot software has emerged. Resellers unable to secure a license for Cybersole or Kodai must instead rent access to them from such companies as Tidal Market, which charges $10 per day for access to Cybersole, and Easy Rentals, where one week of access to Kodai costs $150. Sneakerheads who want a single pair of Jordans for themselves, meanwhile, can pay bot-savvy personal shoppers anywhere from $20 to $50 to help them cop it. Some specialize in Nike or Yeezy, while others are tailored to streetwear brands such as Off-White and Supreme.

In the immediate future, with Covid continuing to slow the global supply chain, Budish, the economics professor, expects bots could play a major role in dictating the spread of good cheer in some quarters this holiday season. “Supply chains being screwed up should only exacerbate the usual bot dance,” he says. The scarcity of some in-demand products could introduce “a lot of volatility and unpredictability” to holiday shopping, with retail pricing likely to be even further below what the market will actually pay, for more products, than usual. Foot Locker, Walmart, and their competitors have distribution relationships and brand identities built around selling PlayStation 5s and Air Jordans at the list price or lower, even if they receive less stock than anticipated—a perfect scenario for resellers with the right software, a little hustle, and a grinchy approach to the season.

Titus, for one, agrees. “For bot users targeting retailers,” he says, “I think it will be a very merry Christmas.”
 
Read next: Why Shortages and Delayed Shipping Could Stay Through 2023

©2021 Bloomberg L.P.