Surprise Security Testing? Welcome To Worst Practices

The CIO for Tulsa, Okla., was put on administrative leave on October 1, after a security company hired by the city ran an unannounced penetration test, and no one in the IT department realized it was a test. The usual tut-tutting aside ("How could he forget he hired this outfit?"), we're wondering whether it's time to dump the security "best practice" of doing surprise pen tests.

Yes, those tests should be a surprise to the security and ops people. But to the CIO? In today's legal environment, with PCI and personal information on the line? That's crazy. For a retailer, it's even crazier.

What actually happened seems pretty clear at this point: The city's Web sites were taken offline on September 12, after one of the city's servers "was targeted by an unknown source," the city said. Six days later, the city sent out 90,000 letters to people whose personal information was on the server because they had applied for jobs with the city over the past decade. Total cost of the mailing: $20,000. (No, we're not sure how they sent out 90,000 time-critical, individualized letters for 22 cents each.)

Further investigation turned up an E-mail from the pen-testing company, SecurityMetrics, that was sent a few days after the "breach" with results of the test. Oops. That's when CIO Tom Golliver was put on leave. The city also spent another $25,000 on a security postmortem, which exposed problems with the police department's Web site so bad that the Web site is being completely rebuilt.

The standard wisdom on this type of incident is that the breach response checklist should have included checking with the security vendor to make sure it wasn't the source of the attack. One big problem with that: Just because there's a pen test going on doesn't mean you're not also being attacked by cyberthieves.

After all, that was the tactic used in last year's huge attack on Sony's PlayStation Network E-Commerce site: A denial-of-service attack was used as a diversion while thieves targeted information on millions of customers. In that case, both attacks were by the same thieves. But thieves are opportunistic, and they monitor high-profile Web sites constantly. If they spot a site that's under DoS attack—for real or as a test—that's the perfect time to launch their own intrusion.

But that's not the only problem with surprise pen-test attacks, especially for retail chains. There are definitely wrong times for a retailer to get a surprise attack: during a major online sale or promotion, for example, or while newly upgraded equipment is coming online after a planned outage. The point of pen testing is to identify ongoing vulnerabilities—not to interfere with business (or with getting back to business). If the test isn't coordinated with IT executives, there's no way to minimize disruption to the business.The whole idea of surprise pen testing comes from a military mindset: the "surprise" inspection (which usually wasn't that much of a surprise anyway). But no senior officer would launch a surprise inspection if there was any chance that a base would be under actual assault—and E-Commerce sites are always at risk. Surprise inspections were a reasonable model for testing datacenter security in pre-Internet days. Today, that model is all wrong.

That doesn't mean pen testing can't work. But it means the pen test has to be coordinated with someone in the CIO's office who can keep a secret (preferably someone who would like to see Security and Operations people sweating) and can quietly supervise the test—and tell the CIO to call a halt if things get out of hand.

That supervisor would make sure the test window isn't at a bad time for the business, would know in advance the exact time and scope of the test, and would monitor response to make sure that's the only attack going on. And forget about getting an E-mail with results days later—that pen-test supervisor should be on the phone with the security company all through the test.

If a planned DDoS pen test suddenly starts getting access to customer information, then either the tester has gone out of scope or another attacker has jumped in. If the pen test is supposed to attack security for one group of servers and a different group starts throwing up alerts, something is very wrong.

That's exactly the type of thing someone inside the IT organization should be watching for, and it can't happen if the pen test is a true surprise. And if it's a true surprise, you're always at risk for a nightmare scenario: some inexperienced IT operations guy who sees a massive attack in progress decides to play hero, and turns a routine test into a full-scale outage.

Besides, there's a host of other issues that surprise pen testing drags in today—things that weren't an issue even a decade ago. Suppose the pen tester gets access to payment-card information. Does that trigger PCI problems? What if it's personal information of customers? Are there state laws or federal regulations that kick in? Remember, pen testing by its nature mimics what thieves do, and that involves going in unexpected directions—sometimes unexpected by even the testers. You don't want to turn a routine test into an FTC fine or PCI assessment failure, either.

Again, there's nothing wrong with keeping pen tests a secret from your security and ops people. Will that make them a little paranoid? Sure, but they should be a little paranoid when it comes to security. But forget about random timing and scope. There's too much risk for the business.

When the top of the IT food chain doesn't know what's going on, there's no such thing as a nice surprise.