Flaws in the Carbon Layer: Is a Penetration Test Without a Social Engineering Component Really a Penetration Test?

A 403 Labs QSA, PCI columnist Walt Conway has worked in payments and technology for more than 30 years, 10 of them with Visa.

Every QSA gets asked the same question about penetration testing: What is acceptable (translation: what is the least I can do) for PCI compliance? In the current environment of criminal (and state-sponsored) hacking, that is the wrong question. Instead retailers should ask: How do I get the greatest value from the penetration testing I am already required to do? I would like to make the point that at least part of the answer is for every retailer and payment card merchant to include some form of social engineering as a part of their pen testing.

PCI DSS Requirement 11.3 has a lot of detail on when retailers need to conduct pen tests. It recommends, for example, "at least annually and after any significant changes to the environment." In practice, this means retailers need to perform and/or re-perform pen testing after such events as upgrading their operating system, adding a sub-network to the Cardholder Data Environment (CDE), or even adding a Web server to the CDE.

However, the requirement does not specify details on what the pen test should cover other than it should include "network-layer" and "application-layer" testing.

Retailers also have flexibility on their choice of whom to use to conduct the pen test. The only requirements for the actual individual(s) conducting the pen test are that they be "qualified" and have "organizational independence." In practice, this means that the pen tester does not have to be a QSA or even from a QSA company at all. The pen tester can be from an outside security firm. Many merchants with substantial IT and security resources use internal staff to conduct the required pen testing.

Naturally, when the QSA comes in for the annual assessment, she or he will need to confirm the tester’s independence and qualifications. Testing independence means identifying where the pen tester sits in the organization (if an internal resource) or whether the tester has or had any involvement in implementing or maintaining the CDE. Testing qualifications is a bit trickier. The tester should have evidence of training and experience using the myriad pen testing tools.

Your QSA will also need to review the pen test report (yes, the QSA needs to see a pen test report even if internal staff conducts the test) for thoroughness and to confirm that exploitable vulnerabilities that the tester identified were corrected and retested. This last step is accomplished by a second round of testing to confirm the vulnerability is no longer present.

It seems particularly unfortunate to me that the phrase "social engineering" appears nowhere in the PCI DSS. Even more unfortunate, my recent search on the term in the PCI Knowledge Base came up blank as well. This situation is particularly disappointing because social engineering attacks are increasingly frequent and very often effective against retailers.

At its simplest, social engineering involves manipulating individuals to take an action (e.g., click on an attachment containing malware) or divulge confidential information (e.g., a password). In a social engineering attack, the call center or help desk staffer, for example, is manipulated into resetting an account, or the recipient of the phishing email is convinced the attachment is really of interest to their career.

Why should social engineering be a crucial element in every retailer’s pen testing?Why should social engineering be a crucial element in every retailer’s pen testing? Because every retailer on the planet is already being phished, and if you can learn the lessons before the bad guys do, your organization might avoid a very unpleasant experience. According to Verizon’s 2013 Data Breach Investigation Report, social engineering tactics like phishing were four times more common in 2012. The reason for this increase is simple: Phishing succeeds. The reason is "flaws in the carbon layer" (i.e., people). One statistic in the report discussed the inevitability of success by describing how if the bad guys sent three phishing emails, the odds are over 50 percent that one recipient will click on the attachment.

Adding social engineering attacks to a pen test does not have to be expensive or extensive. The difference is that your own pen testers conduct the attacks and analyze the results. Working with the retailer’s training department, these examples provide exquisitely teachable moments to increase awareness and decrease vulnerability with live examples. Like many pen testers, when we pursue social engineering, then testing it often is a combination of email and telephone scenarios. The goal is to identify weaknesses that the client can address.

It is important to add that any social engineering pen test has to be conducted under the principles of no harm, no foul. Specifically, blame cannot be part of the review, and in cases I’ve seen, we don’t even need to identify which person took the bait. The purpose is to learn and reduce the odds of a data breach, not to punish call center, help desk or other internal staff.


I’d like to end this column with a personal note. This is my last column as StorefrontBacktalk’s PCI columnist. Since late 2009, when I was privileged to write my first column, it has been a labor of love. I want to thank all of you who left comments, sent emails with your thoughts, corrected me when I was wrong, argued for or against my position, or just related to me your own personal experiences. I read and treasured each comment.

I know I will continue to follow StorefrontBacktalk.com regularly for its great content and writing. It is just that it is time to pass the PCI columnist torch to others. I’ve shared most of my personal opinions and insights as a QSA working with merchants of all sizes across the country. Thank you for sharing the journey with me.