Today is the fourth anniversary of the fourth edition of Anti-Hacker Tool Kit. Technology changes quickly, but many of the underlying principles of security remain the same. Here's an excerpt from the introduction.

Welcome to the fourth edition of the Anti-Hacker Tool Kit. This is a book about the tools that hackers use to attack and defend systems. Knowing how to conduct advanced configuration for an operating system is a step toward being a hacker. Knowing how to infiltrate a system is a step along the same path. Knowing how to monitor an attacker’s activity and defend a system are more points on the path to hacking. In other words, hacking is more about knowledge and creativity than it is about having a collection of tools.

Computer technology solves some problems; it creates others. When it solves a problem, technology may seem wonderful. Yet it doesn’t have to be wondrous in the sense that you have no idea how it works. In fact, this book aims to reveal how easy it is to run the kinds of tools that hackers, security professionals, and hobbyists alike use.

A good magic trick amazes an audience. As the audience, we might guess at whether the magician is performing some sleight of hand or relying on a carefully crafted prop. The magician evokes delight through a combination of skill that appears effortless and misdirection that remains overlooked. A trick works not because the audience lacks knowledge of some secret, but because the magician has presented a sort of story, however brief, with a surprise at the end. Even when an audience knows the mechanics of a trick, a skilled magician may still delight them.

The tools in this book aren’t magical; and simply having them on your laptop won’t make you a hacker. But this book will demystify many aspects of information security. You’ll build a collection of tools by following through each chapter. More importantly, you’ll build the knowledge of how and why these tools work. And that’s the knowledge that lays the foundation for being creative with scripting, for combining attacks in clever ways, and for thinking of yourself as a hacker.

I chose magic as a metaphor for hacking because it resonates with creative thinking and combining mundane elements to achieve extraordinary effects. Hacking (in the sense of information security) involves knowing how protocols and programs are constructed, along with the tools to analyze and attack them. I don't have a precise definition of a hacker because one isn't necessary. Consider it a title to be claimed for yourself or conferred by peers -- your choice.

Another reason the definition is nebulous is that information security spans many topics. You might be an expert in one, or a dabbler in all. In this book you’ll find background information and tools for most of those topics. You can skip around to chapters that interest you.

The Anti- prefix of the title originated from the first edition's bias towards forensics that tended to equate Hacker with Attacker. It didn't make sense to change the title for a book that's made its way into a fourth edition. Plus, I wanted to keep the skull-themed cover.

Regard that prefix as an antidote to the ego-driven, self-proclaimed hacker who thinks knowing how to run canned exploits out of Metasploit makes them an expert. They only know how to repeat a simple trick. Hacking is better thought of as understanding how a trick is constructed or being able to create new ones of your own.

Each chapter sets you up with some of that knowledge. And even if you don't recognize an allusion to Tenar or Gaius Helen Mohiam, there should be plenty of technical content to keep you entertained along the way.

I hope you enjoy the book.

Time, like love, is a universal subject in songs. Time is also a universal theme when discussing vulns; it’s a key component of risk. Equally universal is the heartbreak we feel when finding out about critical vulns or trying to figure out how to fix them.

Identifying vulns is an important part of evaluating an app’s overall risk. Vuln discovery comes from many sources — scanners, crowds, pen tests, red teams, devs, or others. These are also affected by the budget available for vuln discovery.

Fixing vulns is an important part of reducing an app’s overall risk. Whether you primarily find or fix vulns, it’s helpful to shift perspective between the two tasks. Both sides should be working against similar threat models, but the information they have will affect how each interprets, expands, or contracts those models. It’s mutually beneficial when teams can educate each other on how they perceive these models. This is especially important for resolving vulns efficiently and effectively.

My previous post looked at average resolution times across vuln categories. It revealed a stark bias that the category of Redirects and Forwards took by far the greatest amount of days to resolve, with an implication that such vulns likely pose the least risk and therefore require the least investment of DevOps attention.

In continuing the theme of resolution, here’s a graph that shows the relative number of days to resolve vulns of different risk levels. The days are measured against the average resolution time across all vulns. Fewer days represents a faster fix, greater days a longer one.

It's refreshing to see that the most critical vulns are fixed more quickly than average, and unsurprising to see that low-risk vulns take far longer than average. The top and bottom match what we’d expect from DevOps teams prioritizing their efforts based on risk.

But the graph doesn’t form an idealized Z, where the number of days would be least (left-hand side) for the most critical vulns and progressively increase (towards right-hand side) for less critical ones. What’s going on in the middle?

This messy middle provides a seed for many different discussions. As a metric, the number of days until a vuln is resolved is important, but it offers an incomplete story. Some vulns might be easy to find, but complex to fix. Some vulns might have one level risk when reported, but a lower one when reviewed by the DevOps team — not uncommon when the person who discovers the vuln doesn’t have the full context of the vuln’s impact. Others might be resolved by accepting the risk.

(We might also use this to start a discussion on whether risk has been assigned in an effective manner. Calculating and managing risk is something for future posts.)

As in that previous post, I haven’t noted any specific numbers related to the days. We’ll tackle SLAs and concrete timelines in a future post. The focus here is whether the criticality of a vuln influences the time it takes to resolve it. In the ideal world, they should be inversely related (more critical takes less time). But we live in the real world, where DevOps teams must make engineering trade-offs, address underlying problems, revisit assumptions, and review architectures.

Fortunately, it looks like many organizations focus attention on critical vulns. The real world is messy — just listen to all those songs about time and love. Metrics don’t make things any cleaner, but they can give us different ways to evaluate just how messy it is.

Throughout 2017 I explored vuln data to highlight strategies for measuring and maximizing the efficiency of vuln discovery. The primary themes were budget and time — deciding how best to allocate money among different approaches plus evaluating the triggers and frequency of security testing. I was fortunate to present much of this at various conferences, which gave me a chance to collect feedback and engage in interesting discussions about the challenges that DevOps teams face.

In 2018 I’ll have even more data with more dimensions to explore. While it’s important to lay out a strategy for finding vulns in an app. Knowing what risk an app has is only the start. Reducing that risk is the important next step. Strategies for resolving vulns will be a new theme for the new year.

Resolving vulns relates to a trope along the lines of, “Show me your budget and I’ll tell you what you value.” This applies to time as well as money. The vulns you choose to fix and the speed with which you fix them reflect an investment in security. Ideally, it also reflects an understanding of risk.

As a teaser of the content coming next year, here’s a graph that reflects how an organization resolves vulns. It shows the deviation in resolving a specific category of vuln from the average resolution time across all vulns. Fewer days represents a faster fix, greater days a longer one.

In this graph, vulns due to misconfigurations are resolved close to the average, whereas redirects and forwards languish in the DevOps queue for quite a while longer.

We can follow many paths of discussion from a graph like this. Several things influence the speed of fixes, from complexity of code to risk they carry. In the case of redirects and forwards, it seems fair to guess that such vulns represent little or no risk — a hypothesis further reinforced by the category disappearing from the recent OWASP Top 10 update. (We’ll also examine how prevalence of a vuln type impacts its resolution.)

I haven’t even noted what the average number of days is. In a way, it doesn’t matter. The vuln category at the top of the graph clearly receives more attention than the two I’ve labeled. In other words, DevOps teams have a clear priority for such vulns. That’s one indicator of the distinction of BugOps vs. DevOps. Even better is if that category were to completely disappear over time.

It will still be important to discuss threat models and how they help measure risk, especially working with clear, well-considered models instead of falling into counterproductive traps. From there, we’ll take a view of the risk we measure and explore the different ways to reduce it.

This year has only a few days left. A few days left should be the typical response to fixing vulns. Come back next year to find out what new vuln data says about those few days.

The pen test lifecycle is coming to a close. The previous posts have weighed heavily on getting the process started and running smoothly. After all, it’s important to identify vulns within your apps. But most important is fixing them so the app’s users and data can remain well-protected.

Part of keeping a pen test successful is ensuring that the pen testers submit findings with clear risk and reproduction steps. This helps developers understand and address the problem.

A pen test isn’t over once the final report has been delivered. Ideally, the pen testers will remain available to confirm the fixes for each of their findings. Here are a few things to keep in mind when going through the results of a pen test.

1. Resolve Findings Promptly

The time it takes to release fixes is affected both by the complexity of the vuln as well as the maturity of the DevOps team. An efficient team can release a fix within a few days. Unfortunately, sometimes a fix can take a few months.

Part of the process for resolving findings is considering their risk. Critical vulns should be addressed quickly. Other vulns may not need to be addressed at all. These latter types of vulns fall into an “Accepted Risk” category.

Accepted Risk isn’t a loophole to avoid fixing issues. It’s an acknowledgement that some design decisions trade off different levels of security. Some decisions may be choose between two competing approaches to usability without sacrificing security significantly for either one.

2. Address the Underlying Cause

One reason to have pen testers confirm a fix is to ensure the vuln is correctly resolved. In the worst case, a developer may have simply put an attack payload on a deny list instead of applying a proper countermeasure. As an egregious example, this would be like blocking “alert” or “javascript” to prevent a cross-site scripting vuln instead of correctly applying the relevant output encoding for the payload.

3. Look for Vuln Patterns Elsewhere

The nature of a blackbox pen test means that the pen testers don’t have full insight or visibility into the app. As a developer, you do.

Certain types of vulns have patterns that lends themselves to quick searches for similar problems elsewhere in the code. This might be a misused function that lead to an injection attack, a missing CSRF token, or a missing access control check. Take the time to review similar code paths for this kind of vuln.

Conduct a postmortem for high and critical vulns. This exercise brings a team together to discuss what went wrong, whether a proposed fix will be effective, and create action items for additional countermeasures. The goal of a postmortem is to collect and share knowledge about important security vulns. It’s not an exercise in placing blame. While the presence of a vuln implies that someone did make a mistake, it’s important to understand how the mistake was made and how it could be prevented in the future.

4. Create a Regression Test

When possible, create a regression test that can reproduce and catch the vuln in the future. A regression test is also a way to discover other areas where a vuln may be lurking. (This ties into step three above.) Tests should be an integral part of a DevOps pipeline. They ensure code quality as well as build confidence in an app’s stability under constant change.

Always prefer creating a regression test over making a comment in the code. Comments go stale quickly, may be ambiguous, and may often be incorrect. Code that is self-documenting (e.g. informative function and variable names) and whose functions are short enough to fit within a page of text is far easier to maintain.

Going through the effort of creating regression tests also helps reinforce an understanding of the nature of a vuln. By reproducing it, a developer can get a better appreciation for the underlying problem and consider more effective countermeasures. It’s also a way of determining whether your test infrastructure is sufficiently flexible enough to handle certain types of error cases. For example, if you have no way of rendering a web page or inspecting the DOM produced by a resource call, then you may not be able to effectively test for cross-site scripting.

It may not be possible to create a regression test for every vuln, especially complex ones that require multiple steps or exercise a series of flaws.

A pen test should equip you with the knowledge about the current risk exposed by your app and provide recommendations for reducing the risk of the app’s data or users being compromised. The pen test may be a point in time snapshot of the app’s flaws, but how you address the vulns will have a lasting improvement to its security.

Make it harder for vulns to appear. Refactor code so that developers can more easily do the right thing by default. Check out this blog post for additional ideas on making vulns a rare occurrence.

Now that you’ve completed a pen test, it’s time to start planning for the next one. The frequency of testing can align with a calendar, e.g. semi-annual or quarterly, or align with major code releases. In any case, the speed with which you’re able to resolve vulns will also be a signal for how well you’ll be able to manage more testing.

DevOps aims to release code quickly with confidence. Frequent, fast releases aren’t the hard part. The challenge is achieving justifiable confidence that changes won’t break the production environment and, when that inevitably happens, that teams are able to quickly analyze and resolve problems.

This level of maturity requires smart investment in automation, testing, and monitoring. (And people! We’ll dive into that angle in a different post.) Automation increases the pace at which code goes through review, testing, and deployment. Testing (hopefully) detects errors and halts the deployment to prevent bugs from reaching production. Monitoring helps ensure the app remains stable under constant change by providing feedback about its health and activity.

Code isn’t perfect. Perhaps an automated step mishandled an error condition, or testing had a gap in coverage that didn’t exercise functionality affected by a code change. Or maybe the app’s monitoring missed a type of event or omitted useful info. Bugs happen.

Vulnerabilities are bugs that impact the security of the app, its data, or its users. Vuln discovery is important. It’s one of the reasons bug bounties have become so pervasive. We want to know what bugs our apps have, especially if they’ve reached production. And we also have to fix them.

Being able to fix vulns fast is commendable. But a too-narrow focus on speed can turn DevOps into BugOps. BugOps is releasing code quickly to fix vulns without considering their underlying cause. It leads to an endless loop of find-fix-repeat. While it’s important to fix vulns promptly, just adding quick patch after quick patch only makes an app more brittle.

Metrics are a critical tool for decision making. But a shortsighted devotion to metrics aggravates the BugOps mentality. Adhering to SLAs while ignoring root causes creates the illusion of secure code. A quickly-patched app may still rest on a weak architecture.

Another tenet of DevOps is building feedback loops — collecting and responding to actions and events throughout the development pipeline. This should apply equally to vuln discovery.

When vulns appear in production, it’s especially important to analyze how they arrived there and what quality controls they bypassed. They might be due to a mistake, where a coding guideline or established process wasn’t followed. Or they might be due to a misunderstanding, where some flaw in the app’s architecture was exposed or some process didn’t exist.

This analysis can inspire fundamental changes to an app’s design that sweep away whole classes of vulns. Or it may introduce controls that make the exploitation of vulns less impactful and more evident.

Good analysis provides insight into gaps in tools, knowledge, or process. For example, if your testing framework can’t model the types of vulns that are being reported, then you have two problems. One, you won’t be able to create effective regression tests. Two, you’re being underserved by automation.

Good metrics provide insight into how well a DevOps team handles security. Collecting metrics emphasizes what topics are important (hence, worthy of measure). Metrics over time produce trends. Trends provide feedback about the effectiveness of security tactics such as introducing a new tool, adjusting a process, or adopting new programming patterns. Some useful metrics related to vuln discovery are

• Type of discovered vulns. Do certain categories stand out? Do they share similar causes?
• Risk of discovered vulns. Ideally, this would be a common rating based on severity indicated by a CVSS score. No rating is perfect, but CVSS provides a common frame of reference for severity that informs risk.
• Speed of fix. What was the time between discovery of the vuln and the code commit that fixed it? How does this measure against expectations or explicit SLAs?
• Speed of deployment. How long does it take for a commit to reach production? Is there a fast-path for code to address critical issues? Does the app have feature flags that can trivially enable/disable problem areas until a fix is ready?
• Location (e.g. files, objects, functions) of vulns within source code. Review the commit history associated with vulns. Do developers repeatedly address vulns in a particular code path? Is a vulnerable pattern repeated elsewhere, waiting to be reported as vulnerable? Are particular developers responsible for weaker code? Is any automation or tool capable of identifying the vuln?
• Staleness of the location of vulns within source code. In addition to space (i.e. where), capture the time associated with the vuln’s fix. When was the last time the affected code was touched? Is it related to older, legacy code? Is it in newer code? This can help highlight whether the app is on a path of cleanup to improve its overall quality or remains stuck with the same eternal programming mistakes.
• Effort to fix. Related to speed, this is more about the cost associated with fixing vulns. It may be a measure of hours required to analyze the vuln and commit a fix. It could also be the number of people involved in the process. For example, a vuln might require a complex fix or many engineering discussions to weigh trade-offs.

Avoid letting a swarm of vulns chase your team down the BugOps path. As you fix vulns, take the time to figure out how they might have crept into production, adjust tools and processes to catch similar errors that might occur in the future, and track metrics that help show what kind of progress your DevOps team is making to reduce risk.

Keep an eye out for vulns. Keep your vision on the processes that make DevOps successful.