So Many Logs, So Little Time

A 403 Labs QSA, PCI Columnist Walt Conway has worked in payments and technology for more than 30 years, 10 of them with Visa.

PCI's logging requirements present a particular challenge for retailers, especially those with multiple store locations. How does a retailer with a large number—even thousands—of remote devices efficiently log, harvest those logs and review them daily? Reaching for a vendor suite right away may sound easy, but that is only the beginning of an answer.

Once that suite is in place, retail CIOs should plan for a risk- and security-based assessment of their log management needs and allocate the resources to make the system work. Otherwise, retailers may centralize their logging but find themselves overwhelmed by the white noise of too much data. Or, IT could distribute log reviews down to the store level only to learn that the individual locations cannot achieve separation of duties, back-ups or daily log reviews conducted by someone who knows what he or she is doing.

No simple answers exist. Both centralized and decentralized approaches can be made to work in theory, but the more important factor is taking the time to select and tune your logging system, for whichever approach you follow.

PCI Requirement 10 addresses monitoring of all access to all in-scope system components and cardholder data. It describes what events to log (10.2: for example, all user access to cardholder data, administrator actions, invalid access attempts) and what entries need to be recorded (10.3: for example, user ID, date, time, success or failure).

All QSAs are pained when they hear a client say: "We just bought XYZ log management package, so we're all done with logging for PCI." The reason for this pain is that buying the application is only the first of many steps necessary to achieve an effective logging function. Retailers—especially those with multiple store locations—need to budget time and resources to tune their logging approach so it is effective.

For example, a log management application may be able to monitor failed login attempts in Windows, but this step is just the beginning. The system still needs to be customized for the specific applications, devices (e.g., firewalls, routers, even wireless), antivirus, IDS/IPS and remote access (e.g., VPN) in a retailer's cardholder data environment. That means retailers need to customize their logging so it can monitor the chain's different databases and POS and Web applications.

The next step is to tune the logging system so it provides the alerts you need to see without overwhelming you with an avalanche of false positives or unnecessary alerts. Tuning can be a major effort, and the system must be updated regularly. One key theme is relevance. The other is having the log reviews done by people who know what they are doing.

PCI Requirement 10.6 deals with daily log review. The rationale is that the sooner suspicious activity—such as failed administrator log-in attempts or a successful login to a sensitive system by an authorized administrator but without a reason—is uncovered, the sooner you can act and limit any damage. The result is that tuning is particularly important for retailers with hundreds or thousands of store locations, if they are to comply with this requirement.Let's consider the case of a large retailer with multiple store locations that decides to decentralize daily log review all the way down to the store level. I am not a fan of this approach, but the retailer could put a log management application on each store's server and then customize it to match its systems and to alert management to what is important. If each store has the same applications and systems, the process is potentially feasible.

It is now up to individual store managers (once you train them—another cost) to conduct the daily log review, which presents its own issues. If the system is improperly tuned, the manager will be overwhelmed with alerts. Furthermore, even if the system is properly tuned to reduce unnecessary reporting, it will be extremely difficult for the retailer's IT staff to monitor performance and continue to retune the system so the log alerts remain relevant.

PCI Requirement 10.5 presents another set of challenges with this approach because you would need to apply it to each store location. For example, 10.5.1 says to "limit viewing of audit trails to those with a job-related need." Does the store manager really have a job-related need to view alerts?

Then 10.5.2 requires you to "protect audit trail files from unauthorized modification" and 10.5.5 says to use file-integrity monitoring and change-detection software so no one can change a log file. If a store manager (or a subordinate, if this daily task is delegated) goes rogue or the retailer is subjected to a sophisticated attack, the attackers may attempt to alter the audit trails to cover their tracks.

Lastly, each store needs to back up the logs on a server or other media that is "difficult to alter."

These requirements are hard enough to meet in a centralized IT shop. Talking your QSA into accepting that your chain is meeting them at the store level is likely to take some doing.

An alternative approach is to centralize logging and log reviews. In this case, retailers can install an agent on each store server or explore software as a service (SaaS), which certainly can work well, too. Either option may be expensive, but there are vendor packages available in the market.

Some systems generate mountains of logs, while others generate hardly any at all and still others strike a good balance somewhere in the middle. With a large number of retail locations, the volume of messages will need to be tuned to alert only on significant events. The issues retailers have to deal with are bandwidth for transferring all the data, storing (and backing up) the logs once they are received from the field and then analyzing the actual alerts.

Centralizing daily log review also helps companies meet Requirements 10.5 and 10.6. That approach puts retailers in a better position to demonstrate separation of duties as well as to store and protect the actual log files.

I usually stay away from making too many recommendations. However, from my perspective, the centralized logging and log review approach seems much more sound from both a compliance and a security perspective. There are some good vendor alternatives in the market, but the key to success is tuning the logging system so it matches your environment.

Therefore, writing a check is just a start. Retailers need to budget time to configure, monitor, manage and update their logging strategy. My favorite approach is to assemble a logging team—include both management and security people—to look at your needs from a risk and a security perspective instead of just taking a compliance view. If you are the one heading that team, you have to ask yourself: "Will we know if one of our systems is compromised?" Then keep asking that question until you can answer: "Yes."

Unfortunately, based on the statistics in Verizon's 2010 Data Breach Investigations Report, not everybody is doing a good job meeting Requirement 10.6. Verizon found that in 86 percent of cases, victims had evidence of the breach in their log files but no one was looking. We all have to agree that this situation is not acceptable. What it tells me is that the requirement is sound, but that we need to find a better way to meet it.

What do you think? How do you implement your daily log reviews, especially if you have a lot of stores? What resources are you devoting to tuning your logs so you don't get overwhelmed with the logging equivalent of white noise? I'd like to hear your thoughts. Either leave a comment or E-mail me at [email protected].