The team & I had a few engagements recently where we’ve been asked to conduct penetration testing against a common off the shelf solution (COTS) such as Office 365, Oracle applications and palo alto firewalls. When I questioned organisations why, it becomes apparent that pen-testing is the narrative that is getting pushed throughout the industry. To me, this risks the process becoming new AV; we’re executing a task or activity without thinking through why. Its important we call out that doing a “pen test” on these products is money that could be better spent elsewhere.
why is this counterproductive? Firstly, a COTS platform has (or should have been) been subjected to rigorous security evaluation, weather it be through the myriad of penetration tests prior, a deep dive research activity by a subject matter expert or the ongoing assessment and positive activities undertaken by vendors. Simply throwing a tester at it at for $1200 to $2000 a day for several days is not going to meaningfully contribute to the evaluation much less provide a drastic change in assurance. Unfortunately, unscrupulous salespeople will tell us we need to “pentest X.” The penetration testers, under the sales overlords control, probably wont call this out for fear of losing their jobs or affecting utilisation rates and as a result will remain demoralised with such an uncreative activity. We’ve conditioned ourselves to thinking this is useful both as a result of reports we get but also through the expenditure we have.
I believe there are two more effective alternatives to this approach, namely:
- Design and configuration reviews, which I’ve found myself doing more of, give a more hollistic approach and assurance, however in my experience alot of these have not been informed through “hands on” security controls and will often be subjected to a more conceptual view.
- Total systems approach penetration testing (AKA Red Teaming) should also be considered as a means of factoring in the risk of the system, especially with its interaction against other systems.
In short, spend testing budgets wisely and find alternatives to throwing time and money at things that have already been subjected to testing.
