Major Chain Loses PCI Compliance When Data Center Moves

Jeff Hall is a Senior Security Consultant with FishNet Security and has been a QSA since 2007.

One of the nation's 15 largest retail chains had done a tremendous job segmenting its network to reduce the scope of its PCI assessment. All of that was thrown away, though, during a simple data center transition, when Networking made a security change but no one ever bothered to tell senior IT management.

Late last year, the chain decided to move its data center from an in-house facility to a purpose-built data center campus in another part of the United States. The goal was to gain additional raised floor space, energy efficiency and to avoid significant natural disaster risks with the location of the existing data center. In the QSA's review of the new data center, it was seen as a model of energy efficiency and modern design of data centers. So far, so good.

But when the QSA returned for the annual PCI assessment, a review of the core switch and the layer 3 ACLs (Access Control Lists) revealed that all of the switch’s ACLs have been disabled—commented out—for both data centers. The formerly segmented network was totally flat with no segmentation.

The situation had existed since the start of the migration process and would continue for at least another five or six months until the new data center was totally built out with equipment. The reason for this PCI compliance gaff? Networking did it to ensure that production outages did not occur as equipment was migrated from the old data center to the new data center.

The networking group decided to disable the ACLs on the firewalls, routers and switches to accommodate changes to the IP addressing and certain DNS naming conventions that were being implemented with the new data center. From Networking's perspective, production was more sacred than security. And although a few IT people were informed, no one bothered to brief senior IT management.

From a security standpoint, this was a terrible move as it made the cardholder data environment extremely accessible because everything was suddenly in scope. As a result, the external firewall became the only barrier to the CDE (Cardholder Data Environment). The chain should have never disabled the ACLs. Instead, they should have planned better and adjusted the ACLs to work with the new data center.

Networking's thinking was that this was a very temporary move. It was not that the network would run forever in this configuration. It was just to be in place during the transition.

From a PCI compliance standpoint, this temporary change was quite problematic. The timing would make the ACLs disabled well past the end of the annual PCI compliance reporting period. A conference call was quickly scheduled with the merchant’s acquiring bank and key card brands to discuss the PCI compliance reporting options. And, yes, there were some serious differences of opinion between the retailer and the QSA.A conference call was quickly scheduled with the merchant’s acquiring bank and key card brands to discuss the PCI compliance reporting options. And, yes, there were some serious differences of opinion between the retailer and the QSA. Those options were: (1) mark the network flat and not segmented and reference the new data center project as the reason for the situation and non-compliance or, (2) develop a compensating control for the flat network. The reason that the current network configuration results in non-compliance is that not all of the devices on the network are properly configured and controlled to ensure PCI compliance because these devices have been out of scope in past PCI assessments.

If marked non-compliant, this will require that the acquiring bank monitor the data center project through completion and then ensure that the ACLs are placed back in operation to properly segment the network. Although this is the QSA’s preferred option, the chain does not want to be flagged as non-compliant, so they are pushing for a compensating control.

During the acquiring bank conference call, the reporting options are discussed. Although the QSA and internal audit push for the non-compliant Report On Compliance option, the merchant’s management pushes for a compensating control. The QSA discusses the fact that the controls the merchant intends to rely upon do not appear to meet the “above and beyond” requirement.

At the end of the call, the acquiring bank decides they also do not want to deal with a non-compliant ROC, and they tell the QSA to prepare a compensating control that they will evaluate prior to submission of the ROC. A few weeks later, the QSA submits the compensating control for the acquiring bank’s review. Except for a few wording changes, the compensating control is accepted and the merchant’s ROC is completed.

What are the lessons that can be learned from this incident?

  • Acquiring banks have an incentive to ensure that merchants provide them with a compliant ROC.
  • If a retailer can construct a reasonable business case for a compensating control, that will win out over a non-compliant ROC almost every time.
  • PCI compliance is all in the eyes of the beholder, particularly when it comes to the merchant and its acquiring bank and their business relationship. In this case, even though the QSA might have had a strong argument for non-compliance, the relationship between the merchant and acquiring bank trumped that strength.

    Jeff can be reached at [email protected].