The reason that so many PCI self-assessments are wrong is that they focus on the mainstream business processes of the company. They often ignore a lot of "back-channel" or "just-in-case" practices that result in card data coming into the company not protected by the various PCI and other data security measures to protect more mainstream applications, data repositories and processes.
Here are 3 examples, all of which come from personal experience:
I'm at the Electronic Transactions Association (ETA) and RSA Security conferences this week. When I checked into my hotel, which was paid for in advance via Expedia, the desk clerk said he had to make a photo copy of both my credit card and my driver's license. When I asked him why, he said their auditor requires it, and that the auditor reviews all the photo copies every 30 days.
He also said that if I wanted my photo copy back when I checked out, he would give it to me. From a PCI perspective, it's clear they are shipping thousands of photo copies of card data and drivers license data from each hotel to corporate every month. It's also clear that, since they are returning the photo copies to those guests who request it, that the auditor is reviewing incomplete (and therefore unreliable) data and that the audit likely serves no business purpose, other than being a potential CYA backup, "just in case" something goes wrong.
The company has gathered a lot of confidential data that, I'd be willing to bet money, is not considered in their self-assessment and is, in all likelihood, not reliably purged on a regular basis. This is all risk, with virtually no reward—a classic worst practice.
My wife went to return an item to the local mall, to a major women's clothing chain, but she had forgotten her credit card. "No problem," the clerk told her. All she had to do was give the associate her social security number and they would use the social security number to look up the credit card number.
Considering that my wife never remembered providing her social security number to this store (and given that her husband is a data security geek), she was reluctant, but decided to see of they really had her SSN. They did and the clerk showed her that, yes, she had a lookup feature that allowed her to pull up the full customer record from the SSN. The PAN would also have worked, she was informed.
What I don't know is whether the retailer in question was actually buying customer SSNs from a third party or whether my wife simply forgot she provided this information on, say, a questionnaire at some point. Either way, there is no justification for allowing clerks to have this kind of customer data query functionality or for training clerks to ask for SSNs as a way to look up a PAN. For sure, this is not PCI compliant, but I'd would, again, be willing to bet that this process and this retail clerk access was not included in this retailer's self assessment . Even a QSA might have missed it if the process wasn't documented.
Last fall, I went to a "tent sale" at a midsize sporting goods store, a regional chain. It was held in the parking lot of the store and the store had brought out all the old POS systems they could find in storage to cope with the volume and because their in-store systems weren't portable. As I'm sure you can guess, some of the old systems were neither FACTA nor PCI compliant and showed all 16 digits: the classic "knuckle buster" issue. We've all seen this, but my point is that I think it's very unlikely that the PCI compliance issues associated with continuing to use these systems for special events were either ignored for the purposes of the self-assessment or some note was made about how some "compensating control" was used such as locking up the paper receipts, etc. Again, it's simply a matter of spending the money so that these old systems can be retired (or sold on ebay to another retailer, so it becomes his problem!).
One of the value propositions for Level 2 and 3 merchants initially using a QSA or other PCI consultant to do a gap analysis is that they can catch these "alternate channel" PCI issues, because they have seen them before and know what questions to ask. Even if you don't want to spend money on a QSA or consultant, it is important to ask lots of questions, call some large meetings and solicit input from people involved in all different aspects of the business.
Perhaps the most common problem I've seen in more than 4 years of working in PCI compliance is that months after the PCI assessment is complete, someone will find a new "treasure trove" of card data, or identify a procedure where card data is collected that was missed the first time around. Call it trial and error or whatever you want, my experience has been that most PCI self-assessments will miss the types of card data security issues noted here.
Obviously, we'd really like to hear other examples. All our discussions are 100 percent anonymous, so if you'd like to talk about it, or send an example of your own, visit the PCI Knowledge Base and comment in our discussion forums, or just send us an E-mail at [email protected].