$12 Million In Duplicate Charges From Shell Oil Telco Crash

On Saturday (Jan. 29), a telco outage at Shell Oil stations directly caused more than $12 million in duplicate charges for its retail customers. This is yet the latest case of a chain getting burned because the payment industry has no consistent way to deal with in-progress credit and debit charges when systems crash. Does store-and-forward need to be tweaked to be made more crash-proof?

First Data circulated a confidential memo on Monday (Jan. 31) that said it was reversing some 401,120 transactions—totaling $12,135,608.19—of Shell's retail transactions from the weekend. First Data asked issuers to "post the reversals to the cardholder accounts impacted as soon as possible in order to limit negative impact." (Guess "negative impact" means customers screaming at call center reps.)

(Important Story Update: AT&T is now denying that any outage took place on their network and Shell is quickly backtracking. See the latest here.)

The incident happened on Saturday (Jan. 29) when Shell "experienced a system issue that impacted credit and debit card processing," said Shell spokesperson Theodore Rolfvondenbaumen. The credits were mostly made Tuesday (Feb. 1), he said.

Although the details are still being investigated, Rolfvondenbaumen said it was an AT&T "unplanned telecommunications outage" (he conceded that he's never seen a planned telco outage) and that AT&T handles Shell's payment card processing network connections.

"Our IT department is investigating the root cause," he said. "This is the first time it's ever happened in the Shell network."

Among those more than 401,000 transactions were "debit card customers who did experience overdraft fees" along with many credit card customers. Shell is "willing to work directly with debit card customers" to cover the costs of overdraft and related fees. "If they were impacted, if someone has been impacted, we will work to make this right for them," Rolfvondenbaumen said.

This is yet another reminder of the high-cost of retailers accepting debit cards. The interchange fees are lower, but the cards are also much more susceptible to glitching. And when problems happen, the costs are so much higher than for credit cards, as Shell has discovered. Retailers are perhaps too quick to dismiss the retail benefits of zero-liability programs.

The bigger issue, though, is how a telco outage could have caused so many duplicate charges.The bigger issue, though, is how a telco outage could have caused so many duplicate charges. Typically, duplicate charges happen when store employees are dealing with transactions that were in progress at the instant when the outage hit. If the system records are unclear as to whether the charge was fully processed and confirmed or not, associates (and, more likely, the system's automatic process will kick in) will typically take the safe route—in terms of revenue protection—and process the charges again.

Either way, as Shell discovered, this approach can cause a lot of false charges. And it is painful well beyond the accounting cleanup. Beyond the cost of processing the reversals—and hard dollar reimbursement costs, such as for those debit customers who were hit with overdraft fees—there's the hit to the chain's reputation. How will customers who see such double charges feel? Will they assume the chain was trying to rip them off?

This possibility also raises some fun questions. Can Shell legal connect the dots sufficiently to make the case to AT&T? And if so, can they get AT&T to cover all of the losses in any way associated with the outage? Should it be able to?

But let's expand this discussion. Outages that can trigger this store-and-forward headache are many, from power blackouts to telco hiccups (such as what Shell experienced) to—for E-Commerce transactions—a Web site crash or network disconnect or even a backbone provider problem.

Getting back to the AT&T liability issue, should this now be a standard part of performance guarantees? Sort of an SLA (service-level agreement) type of agreement that covers companies of this type?

The short answer is almost certainly not, because the AT&Ts and others are going to legitimately say that the duplicates weren't really caused by the outage but by a problem with the store-and-forward system that they have nothing to do with.

The problem is that store-and-forward systems are supposed to be able to identify those transactions that had been properly processed and those that hadn't. With more than 401,000 dupes coming from as large a chain as Shell, isn't there a fundamental flaw?

This isn't the first time First Data has been involved in a double-charge problem. It experienced one with Hannaford and other chains less than four months ago. In that case, customers were not only double-billed, but the retailers were double-paid. The day before Thanksgiving (Nov. 24, 2010), grocery chain Winn-Dixie—suffering an outage—reported double-charging customers in all of its 485 stores.

There have also been cases of a million double-charged transactions at Starbucks as well as publicized incidents at Macy's plus a big one at Best Buy.