Loading Dock Chaos: CIO Had No Idea What His Passwords Could Do
What happens when the keys to a retailer's supply chain show up on Google? In the case of one multi-billion-dollar regional chain this week, it resulted in the ability of anyone to change the information of all loads expected at the retailer's distribution centers—dates, times, contents of the load, number of pieces, weight, pallets, the product ready date and the vendor call date.
In short, in the hands of an evil-minded competitor (in retail, are there any other kinds?), that Google-provided password could do a huge amount to slow down a rival, in addition to knowing inventory shipment plans so they can be countered. It represents a critical security breach—and one that started with the simple decision to put a confidential manual in a Web site subdirectory. That single password—which was printed in that Google-available PDF—unlocked a third-party's servers and revealed a supply-chain security hole large enough to drive a fleet of Mack trucks through.
What started this week with an analyst's accidental discovery of a retailer's confidential supply-chain manual PDF during a Google search morphed into a series of mega-headaches for the chain's CIO. That now publicly available password opened detailed reports on every single shipment the chain did for as many months as the visitor wanted to see, including details of future shipments and the ability to edit and change those freight schedules. Although the system was supposed to first require shipment details from the user, it actually provided those details to anyone following on-screen guidance.
Due to the ongoing security exposure of this chain—which has yet to change the passwords—StorefrontBacktalk is withholding the retailer's identity from this story. (Update: After this story was published, the password was changed. It was some four days after the chain was alerted to this problem, but it has now been changed.)
When we initially contacted the chain's CIO, he didn't think it was a concern; he believed that the password had extremely limited access, such as for generic routing instructions. Subsequent efforts, though, quickly changed his mind.When we used the password, the screen asked for a wide range of shipment details for verification, which is good, including warehouse, the assigned carrier, the department number and the vendor name. Unfortunately, the screen had a prominent note in red lettering that said a date range was required but that the user could "skip [other fields] to show all loads." We set the date from January 1 through April 30, and it poured out more than 5 MB of ASCII text, detailing the purchase order (P.O.) number, unit number, reference number, vendor name and location, the carrier name and a half-dozen interim ship dates of every shipment within the chain, including future scheduled shipments.
And then using the information from that data dump, other screens allowed extensive shipment editing, including fields to change the chain's P.O. numbers, pick-up name and address, number of pieces, weight, pallets, the product ready date and the vendor call date.
"In our company, there was a misunderstanding of what [the password] gets you," said the chain's CFO, after reviewing the files we accessed from the site. "It's not intended to be very specific. It's not intended to give as much information as it does. This is obviously a poorly controlled program and is extremely poorly done."
On a very lengthy laundry list of problems associated with this type of data leakage is the unnerving possibility that people may have already started using the information against the chain. Done subtly, the type of glitches that could be caused might be dismissed as simple site hiccups. "There will be a data verification where we will be checking the data out there and verifying its accuracy," the CFO said.
The chain put much of the blame on a third-party company that hosted the chain's logistics management, coordinating freight movements from various freight companies. An executive with that vendor said the system can't do anything dangerous and promised to check and get back to us. No one ever called back.
There are four key issues. The first is that the PDF manual should never have had the live password printed in it. The second is that the confidential password-containing PDF manual should not have been discoverable by a search engine. The third is that a single password should never have been used by the tons of people in the retailer's supply chain. And the fourth is that generic passwords should have had much more limited access.
As for the Google spiders finding the document, that could have been made much less likely had the retailer placed a robots.txt file in the Web site's root directory, specifying that this directory shouldn't be indexed, in addition to putting a blank index.htm file in the directory itself, so it couldn't be browsed. Then, unless people knew the exact name of the file they were looking for, it wouldn't show up."I think this demonstrates that security is much more than password protection, PCI compliance and intrusion detection—much more," said Jerry Sheldon, an analyst with the IHL Group. "There is a huge education effort that has to be invoked here. Imagine if someone with nefarious intentions were trying to do harm. I can only imagine the havoc that someone could create in the supply chain after accessing this site: canceling shipments, rerouting stuff, etc."
It appears that the system was not capable of changing or rerouting products, but it could change the information in the supply-chain system, which would change expectations. Consider a truck driving to a distribution center in Austin, Texas, loaded with 20 pallets of air-conditioners that are slated to arrive on Thursday at 4 PM.
A change in this system wouldn't likely reroute that truck. But if the distribution center crew looks up what's coming and someone has changed the expected time of the shipment to Friday at 9 AM, that could be almost as bad. The crew isn't ready for the truck when it arrives, and/or it would waste time waiting for a truck that isn't coming.
Walt Conway, a QSA for 403 Labs and the PCI columnist here at StorefrontBacktalk, said this incident raises a wide range of security issues.
Referring to it as a "case of credentials left in the open for whatever reason that can be used to your detriment by a bad guy, it also reminds me of the Verizon breach report findings that it is the 'unknown unknowns' that hurt you. That is, the unknown places you have confidential [payment card] data that you did not even know existed," Conway said.
The biggest fear, though, is that these types of attacks can happen so quietly, IT might not even be aware of it. "If a bad guy could get this access, what else could he get? That is the big question. It might not be obvious today or even next week. Bad guys are patient. With access to a supply-chain system, the bad guys can capitalize on it to hijack. Think if this were Apple with iPad2s. They could also do any other manner of mayhem a competitor might want to inflict. To me, the idea of a competitor having free access to my IP space is kind of scary," Conway said. "A real bad guy could potentially use this as the start of a probe/penetration test for real and see how far they can get into the internal systems like E-mail (spear phishing), HR (PII), marketing (card data), treasury (banking info and maybe credentials), etc. And the company would be the last to know. Seems a bit disturbing to me."