PCI's Grading System Is Failing

GuestView Columnist David Taylor is the Founder of the PCI Knowledge Base, Research Director of the PCI Alliance and a former E-Commerce and Security analyst with Gartner.

For months, retailers and Congress have been attacking retail security standards, but few realize that the problem is not in the standard itself. The problem is a grading system that causes most retailers to be out of compliance most of the time because the rules require 100 percent compliance. How often in school did you score 100 percent?

The system is oriented to forcing retailers to fail and it does this by being utterly insensitive to risk, which is surprising because the financial services industry runs on risk management. So if big finance runs on risk management, why are retail payment security rules running away from it?

At the recent Electronic Transactions Association (ETA) conference, Visa and MasterCard executives again defended the standard against industry and government criticisms that the standard is insufficient to prevent breaches. They cited instances where merchants (e.g., Hannaford) and processors (e.g., Heartland) who claimed to be PCI compliant were actually not compliant during the extended period of time during which the breach occurred. That happened because, from a technical perspective, "companies can fall in and out of compliance," and because sometimes "the PCI assessment is not comprehensive enough," according to the executives. From these statements, it sounds like the blame for why a "compliant" company can get breached is due to either technology or the assessors. But I maintain that the culprit in all of this is the grading system used to measure PCI compliance, and it's time for a change. Here's some arguments in favor of this "modest proposal":

  • You May Never be Fully Compliant
    PCI Assessments — whether self-assessments or assessments by QSAs — are generally regarded as being valid only for a point in time. But when is that point in time? Is it the day the ROC or SAQ is signed by the assessor merchant, or the day the assessment has been signed off on by the acquirer, or the day that the ROC or SAQ is reviewed and approved by the card network(s)?

    How does a retailer know when that point in time ends? The technical complexity of the controls is inconsistent with the grading system that requires 100 percent to be compliant. Great standard. Bad grading system.

    Thanks to the grading system, and the fact that many of the PCI controls are "volatile" and can be made ineffective by simple configuration or rule changes, this technically means that an organization may never actually be PCI compliant. That's because, for a typical Level 1 merchant, an assessment will take more than a month, sometimes several months. Thus, it is very possible that between the time the first controls were tested and the time the last controls were tested, changes were made to the first controls such that they are no longer 100 percent compliant.

    Therefore, the merchant is noncompliant even before the ROC or SAQ has been reviewed. Although it is true that some assessments are better than others, the "condition of compliance" is far more subject to the grading system than to the quality of the assessment.

  • When Bad Assessments are Good
    The nature of the grading system, rather than the PCI standard itself, is responsible for most of the dissatisfaction we find in the merchant community. This dissatisfaction has resulted in many merchants seeking out assessors who are "easy graders" or take a fairly lax attitude when filling out the self-assessment. When searching our research database, we see numerous comments about how much merchants like certain assessors because they made the process "painless" or were "easy to work with" or other comments that indicate the merchant was more interested in scoring 100 percent on the test than on getting a thorough evaluation. It's not that these merchants don't want to be secure. It's just that they object to being held to a standard where they have to score 100 percent to pass, and get fined if they don't achieve it. Again: great standard, bad grading system.

  • What if PCI Was Graded on a Curve?
    The PCI SSC recently took a small step toward a risk-based approach to PCI with the spreadsheet-based prioritization tool. Unfortunately, this didn't do anything to change the grading system. But, given that risk has been introduced into the process, I believe that it is possible to modify the grading system to address realistic risk levels for different types of businesses. After all, merchants pay more to process card not present (CNP) transactions because there is more risk, and most of the transaction charges and fees that drive the financial services industry are all risk-based.

    The merchants and financial institutions that make up the PCI SSC and the card networks certainly have enough data to develop risk ratings for each of the mandated PCI controls and conditions of their implementation, so why not grade PCI compliance on a curve?

    The card networks would be happy because they could show increases in their "percent compliant" statistics. The merchants would be happy because they could conduct more realistic self-assessments and the QSAs would be happy because they would have more flexibility to adjust the application of the controls to the particular situation of each of the organizations they are reviewing. This is the easiest way to "bring peace" to the increasingly contentious relationship between the card networks, the banks, the merchants, the assessors and, increasingly, the federal government.

  • The Bottom Line
    I know that none of these arguments are "new" but I do believe that they are especially timely now and more likely to be considered, thanks to increasing government focus on the card networks and their impact on consumers and businesses. We welcome your comments, on either side of the argument. If you want to read some of the comments about the assessment process and the PCI grading system from our research, please visit the PCI Knowledge Base and if you want to discuss this topic, send us an Email at [email protected].