Vendor Selection: In-house or Outsourced

Occasionally, I encounter companies that want to conduct their own penetration testing. They often claim to have staff members who used to perform penetration testing or individuals who incorporate exploitation as part of their job. While I respect their talent, they frequently lack the necessary skills or, perhaps more importantly, the objectivity to thoroughly test the environments they are responsible for operating. Penetration testing is a skill that demands constant practice, as attack methods evolve over time. The most effective penetration testers refine their abilities by testing a wide range of systems and developing techniques based on extensive experience. This level of expertise is typically attainable only at very large companies or when working for specialized penetration testing firms.

In-house testing can be advantageous, especially in cases where an in-depth understanding of a specific application or system is required. However, your average web app or tech start-up does not necessarily meet these criteria. In such situations, it is highly valuable to engage an objective, skilled external firm for penetration testing. Their expertise and impartiality will yield the best results.

Regardless of whether you in-source or out-source your penetration testing needs, you must have:

Accountability: Designate someone responsible for conducting the test and interpreting the results. Reports that generate no action are not useful.

Executive Leadership Buy-In: Obtain buy-in from executive leadership regarding the level of testing required. The depth and intensity of testing should align with the cybersecurity risk. If necessary, clearly communicate the associated risks to gain leadership support.

Scope Clarity: Understand the scope of what needs testing. Identify what assets you have and precisely define what requires evaluation.

Vendor Vetting

Company Qualifications: Low-cost providers of commodity penetration testing services often utilize offshore resources, which introduces a data access risk. It is crucial to understand who you are potentially allowing into your environment, what data they accessed, and whether all issues were ethically disclosed at the conclusion of testing. To mitigate this risk for US companies, it’s advisable to hire a US-based firm that employs US citizens undergoing annual background checks. Additionally, there’s a tendency to hire hotshot independent contractors, however I have seen mixed results with this approach. While good results are possible, there may be a lack of long-term interest and accountability on the part of the contractor. Ideally, choose a company that is committed to delivering a thorough test and providing support through remediation. Ensure the company has a well-defined process for data handling, confidentiality, and separation of customer data. Standards aligned with ISO 27001, NIST 800-53, and NIST CSF are desirable.

Penetration Tester Certifications: There is not a single go-to certification that penetration testing vendors can acquire to demonstrate competence, although there have been some attempts. CREST certification is an accreditation for penetration testers, aiming to ensure a high-quality and repeatable testing process. While pursuing certifications is commendable, they don’t necessarily replace practical experience. Some hands-on certifications, such as Offensive Security’s OSCP, are valuable. The OSCP’s timed exam, although somewhat dated, serves as a useful screening tool for assessing how well new hires can learn and apply attack methods. However, it is essential to recognize that penetration testing involves a creative element that may not be fully quantifiable through certifications alone.

Requesting a Quote

Timing: The busiest time of year for pen testing vendors is Q4; there is always a mad rush to spend unused budget and meet annual pen test requirements. If you’re looking for the best price and most availability, try testing during Q2.

Scoping: The best way to get a good quote is to be clear about what you want tested. This may sound obvious, but I often don’t get the full details up front. Any request for quote should include these factors that affect effort:

• Internal and External Network Testing: Number of active IP addresses, number and size of networks to be enumerated
• Web Application Testing: Number of apps written in-house and their general functionality. SaaS solutions are generally not tested since the security of those apps is not the customer’s responsibility
• Mobile Application Testing: Number of apps and platforms (IOS/Android)
• API Testing: Number of APIs and endpoints
• Device Testing: Device functional description and physical security expectations when deployed. State if destructive testing is allowed or if only blackbox testing of network communications
• Purple Team and Red Team: Objectives for testing or attack scenarios to verify. Constraints on testing or pertinent rules of engagement