# Engineering Trustworthy Systems  ## Metadata - Author: [[O. Sami Saydjari]] - Full Title: Engineering Trustworthy Systems - Category: #books ## Highlights - Beliefs can be based on strong evidence or on faith. ([Location 24468](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=24468)) - Tags: [[pink]] - One must understand its capability under various conditions of computer and network operations (e.g., traffic load and types) and its vulnerability to attack. For example, if an intrusion detection system is blind to certain classes of attacks [TAN02, IGUR08], it is meaningless and indeed misleading to say that the system has detected no attacks without the disclaimer of the attacks that it cannot possibly detect. ([Location 25779](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=25779)) - Tags: [[pink]] - humorous description of distributed systems is “one in which the failure of a computer you didn’t even know existed can render your own computer unusable” (attributed to Leslie Lamport). A famous example of this is the dependency of modern networked systems on the domain name system (DNS). ([Location 28838](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=28838)) - Tags: [[pink]] - Risk has two important components: likelihood of something bad happening to mission and how bad those somethings are. ([Location 32770](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=32770)) - Tags: [[pink]] - The reality is that cybersecurity is always in a trade-off with mission functionality, which has two important dimensions: performance and functionality. ([Location 32771](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=32771)) - Tags: [[pink]] - Attackers await and leverage faults of all types. ([Location 36264](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=36264)) - Tags: [[pink]] - When thinking about cybersecurity, think immune system, not a single white blood cell. ([Location 36702](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=36702)) - Tags: [[pink]] - Prevention is about stopping attacks through thwarting one or more mechanisms exploited by an attack sequence. For example, limiting the computers that can talk to a system using restrictive firewall rules can substantially narrow the attack surface (vulnerabilities that an attacker can directly reach) available to an attacker. Detection simply determines that an attack step happened and that perhaps a larger attack sequence of steps is in progress. Detection by itself can be valuable if the entire value to the adversary derives from the defender not knowing that the attack happened. For example, if an attacker steals a defender’s war plan for a region, the defender can change the plan, and so the element of surprise is not lost. Reaction is done in connection with detection—it is the action taken to thwart the attack and repair damage to ensure continuity of system function. These classes of cybersecurity instruments are then arranged to play the music of defense. ([Location 36702](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=36702)) - Tags: [[pink]] - Cybersecurity is about effectively reducing risk for reasonable cost and mission impacts. • Cybersecurity requires a trading off mission goals, performance, and functionality. • To understand cybersecurity, one must understand how systems are attacked. • Risk is particularly high at interfaces between systems because of bad assumptions. • Top-down and bottom-up design work together to design effective cybersecurity. • Cybersecurity requires well-informed orchestration of dynamic aspects of mechanisms. ([Location 37138](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=37138)) - Tags: [[pink]] - Even further, an organization should explicitly attempt to engineer itself so that it does not rely so heavily on secrecy. ([Location 41944](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=41944)) - Tags: [[pink]] - Data integrity refers to the confidence that the data in the system has not been maliciously modified. System integrity refers to the ([Location 41947](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=41947)) - Tags: [[pink]] - can use system integrity attacks to mount any other attacks, including data integrity corruption. For this reason, it is the author’s experience that integrity is one of the most undervalued aspects of cybersecurity by most defenders today. 3.6 ([Location 42381](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=42381)) - Tags: [[pink]] - Organizations weigh the needs for confidentiality, integrity, and availability of data. • Cybersecurity protects mission as the body’s immune system protects the body. • Take the attacker’s perspective and seek to minimize the value an attacker will derive. • Avoid concentrating valuable data in one place to avoid creating a high-value target. • Secrets are difficult to protect, tend to proliferate, and can hamper mission. • The mission role of secrecy should be carefully evaluated throughout a system’s life cycle. • Minimize dependency on secrecy and mission loss from the loss of secrecy if it occurs. • Trustworthiness must be proven to stakeholders throughout a system’s life cycle. • Without system integrity, confidentiality and availability are pointless. • Availability depends on systems that are not under the designer’s operational control. • Avoid outsourcing critical functionality and data to potentially untrustworthy entities. ([Location 42817](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=42817)) - Tags: [[pink]] - Focus on strategic risks by focusing on ways that mission elements can fail. • Don’t sweat the small stuff—it’s a distraction that only serves the adversary. • Strategic harm stems from what damages keep the C-suite up at night with worry. • Partition data into tiers of criticality based on the degree to which mission depends on it. • Criticality determines the relative importance of confidentiality, integrity, ([Location 48499](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=48499)) - Tags: [[pink]] - Examine cross-product of the mission and the critical data elements to find strategic harm. • Agreeing on the key strategic harms is enlightening and leverages the “wisdom of crowds.” • Belief attacks are serious, frustrating, and are likely to grow in the future. ([Location 48934](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=48934)) - Tags: [[pink]] - System state can be a difficult concept to fully grasp, partly because it can be at many different levels of abstraction. In the most fundamental form, the state of a memory cell is either one or zero. ([Location 50246](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=50246)) - Tags: [[pink]] - Clearly, knowing, let alone understanding, the full state of a modern system is beyond human faculties at the lowest level of details. That is the purpose of abstraction and the mathematical equivalent of abstraction—modeling (see Section 19.2). Abstraction makes incomprehensible system state intelligible. ([Location 50682](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=50682)) - Tags: [[pink]] - design—the adversary’s model of the defender’s target system is often more accurate than the defenders’ model since the defender often relies on outdated design documents, flawed designs, and buggy implementations for theirs. The adversary has the advantage of looking coldly and objectively from the outside of the system, unbiased by the wishful thinking and delusions of the system designers. ([Location 55926](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=55926)) - Tags: [[pink]] - Attacking views can come in two forms: attack the artifacts (manifestation of models in the form of documents or simulation systems) or attacking the beliefs. The notion of attacking the artifacts themselves is relatively straightforward. One simply attacks the system on which the artifact exists and executes an integrity attack against that artifact. ([Location 56362](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=56362)) - Tags: [[pink]] - Systems are too complex to understand without modeling in some way. • Abstraction helps focus attention and foster better cybersecurity design and operation. • System models include the technology, human behavior, and their palette of actions. ([Location 58112](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=58112)) - Tags: [[pink]] - Models can be inaccurate, incomplete, or obsolete, causing design and operations errors. • Defender models have the advantage of controlling and knowing their system. • Attacker models have the asymmetric advantage of the element of surprise. • Defender and adversary models are themselves valuable targets. • Cybersecurity and reliability are interdependent aspects of trustworthiness. • Assume that an adversary deeply knows the target system’s design and implementation. • More strongly, assume that adversaries are inside the system in the form of malicious code. ([Location 58546](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=58546)) - Tags: [[pink]] - There are three broad categories of intentions: espionage, sabotage, and influence operations. Espionage is about stealing information to gain an advantage; sabotage, the doing of intentional damage to cause disadvantage; influence operations, altering beliefs and decision making, including deceit, to achieve some nefarious goal. ([Location 61169](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=61169)) - Tags: [[pink]] - Adversaries have ready-made cover stories for their attacks by using cut-out organizations (also known as state-tolerated organizations) such as organized crime (e.g., the Russian Business Network [NEWS09]) or hacker clubs (e.g., China’s hundreds of hacker clubs that feed both valuable information and experts to the Chinese military [CHIN15]). 6.1.5 ([Location 62042](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=62042)) - Tags: [[pink]] - As an extension of this idea, programs often have a specific maintenance mode and special access port for developers to gain deep access into the bowels of a program and its data structures. This port is for debugging and patching purposes, after deployment. Such ports and accesses are typically extraordinarily risky and sometimes can be escalated to privilege on other parts of the system. ([Location 65101](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=65101)) - Tags: [[pink]] - Said a different way, a perceptible tick of time to a human is about 1,000 years to a computer. ([Location 65539](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=65539)) - Tags: [[pink]] - Expect increasing underware attacks. Lastly, attacks started in the pure digital world because it was the most ([Location 67721](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=67721)) - Tags: [[pink]] - Some red teams withhold their techniques on the grounds that they are so sensitive that they cannot be shared for fear that the free world will be destroyed. There is, of course, some value in knowing that a system is vulnerable to a Class A (meaning a high-quality and high-skilled red team familiar with the latest and most effective attack techniques) red team, but true value comes from knowing what to do to improve defense. ([Location 69906](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=69906)) - Tags: [[pink]] - Adversaries have a variety of attributes and ranges of values each can take on. • Adversaries are smart, skilled, and knowledgeable despite our wishing otherwise. • Adversaries do not play fair and violate unrealistic or poorly understood assumptions. • Attacker knowledge and capability will continue to improve and become more aggressive. • Attackers will begin targeting cyber-physical systems and underware. • Red teams, as model adversaries, play a critical role in assessing designs and mechanisms. • Cyber exercises are essential in both guiding design and operational preparedness. • Red team work factor is a useful measure of difficulty to compromise system security. ([Location 70780](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=70780)) - Tags: [[pink]] - 8.4.3 Security Kernel and the Reference Monitor Operating systems were essential to system security and particularly to implementing the mandatory policies discussed earlier, operating in the most trusted hardware mode—the supervisor mode. Yet they were too complex to prove correct. The solution was to divide operating systems into three layers: the security kernel, trusted extensions, and untrusted libraries operating in user mode. This layering is shown in Figure ([Location 81703](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=81703)) - Tags: [[pink]] - letter of *-property (no write down) ([Location 82141](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=82141)) - Tags: [[pink]] - Implementations of type enforcement include SELinux [LOSC01] and the Sidewinder firewall [BOEB10]. 8.6 Cybersecurity ([Location 83888](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=83888)) - Tags: [[pink]] - Cybersecurity takes system resources, which necessarily will slow the system down some. Designers should establish design policy and goals regarding limiting allocation of resources to cybersecurity functions. A good rule of thumb is that the totality of security mechanism (not just a single aspect) should collectively take up no more than 10 percent of system resources. ([Location 85199](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=85199)) - Tags: [[pink]] ## New highlights added May 18, 2022 at 10:43 AM - Trustworthiness requires both security-critical functionality and assurance so that functionality operates correctly. The functionality must create a means of separating both computation and memory access. For computation, the processor must separate the instruction set architecture into at least two separate modes: user mode and supervisor mode. The memory must be tagged so that access to that memory can be restricted. Without both of these features, one cannot build a trustworthy system because malicious applications could simply execute instructions to perform arbitrary actions on system resources and could read or corrupt all memory contents, thus making confidentiality, integrity, and availability attacks trivial. These basic functions must be provably correct, tamperproof, and non-bypassable—the three properties of a reference monitor ([Location 90441](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=90441)) - Tags: [[pink]] - This user-supervisor mode separation enables the foundation of trustworthy sharing of system resources. It is necessary, but insufficient. The memory must be labeled and controlled in some fashion. ([Location 90878](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=90878)) - Tags: [[pink]] - Therefore, it is imperative to carefully control access to memory using techniques such as memory mapping, capabilities, and tagging. ([Location 91314](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=91314)) - Tags: [[pink]] - Such low-level software exists to allow vendors to fix bugs and issue updates without having to replace chips, saving millions of dollars and improving the time to market for the benefit of the consumer. At the same time, it is important for cybersecurity engineers to be fully aware of this low-level software and its potential risks [CERT17a]. It represents a soft underbelly that could be attacked and undermine the security of the entire system in very surprising ways. ([Location 92190](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=92190)) - Tags: [[pink]] - • If you cannot trust the hardware, then the work required to trust the software is futile. • Both complex and reduced instruction set architectures support trustworthiness. • Processor uninterruptible privileged mode forms a foundation for trustworthy software. • Memory must be protected using techniques such as memory mapping and capabilities. • Hardware has some software driving its operation, posing a little-known cybersecurity risk. • Inside a computer, devices often communicate freely on a tiny network called a master bus. ([Location 93063](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=93063)) - Tags: [[pink]] - Sometimes, clever mathematicians find ways of determining the key that don’t require full exhaustion of the key space. These are called subexhaustive attacks. ([Location 95685](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=95685)) - Tags: [[pink]] - Random key generation requires random sources, not humans. ([Location 96994](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=96994)) - Tags: [[pink]] - Amateurs attack the algorithm; professionals attack the key management. ([Location 96995](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=96995)) - Tags: [[pink]] - The binding (i.e., association) between a user and their public key is security-critical. If an adversary can subvert a server that makes that association, they can essentially masquerade as a user of their choosing by simply generating a key pair and substituting the public key that they just generated for anyone’s (or everyone’s) public key. This would be a very devastating attack. In addition, a server is not always available, such as in the disconnected devices on separate networks or totally off any network. The solution to preserve the integrity of the binding between a user’s identity and their public key is to cryptographically bind them. This essentially means to digitally sign the two together, combined with some other attributes, such as the period during which it is valid. The data and the digital signature is what is called a public-key certificate. Someone certifies that a particular user identity is indeed associated with a specific public key and implicitly certifies that the private key was somehow securely distributed to the user (and a hope that the user properly protects that private key from disclosure). The someone that makes this certification is called the certificate authority. ([Location 98742](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=98742)) - Tags: [[pink]] - The man-in-the-middle attack is defeated by a digital signature. ([Location 100491](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=100491)) - Tags: [[pink]] - In such cases, a cybersecurity engineer might request that the vendor digitally sign the software before distributing the software or a software patch and ensure that the signature is checked before installation. This prevents an attacker from accomplishing a supply chain attack in the communications portion of the process. Some data could require both protection from disclosure and integrity. Bank ([Location 100926](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=100926)) - Tags: [[pink]] - Cryptographic bypasses are a necessary evil requiring great care. ([Location 101365](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=101365)) - Tags: [[pink]] - Encrypting data can actually make it a high-value target. ([Location 101802](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=101802)) - Tags: [[pink]] - Quantum computing may break existing cryptography. ([Location 101803](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=101803)) - Tags: [[pink]] - is only a matter of time and a handful of required breakthroughs before quantum computing becomes practical [COFF14]. ([Location 102237](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=102237)) - Tags: [[pink]] - Public-key algorithms are based on what are called Non-deterministic Polynomial-time Hard (NP-Hard) problems [AHO74]. Such problems are exponentially hard in practice, but there exists no proof that such algorithms are inherently so. That means that it is theoretically possible for someone to one day discover a polynomial-time (non-exponential) solution to reverse these transformations. Such a discovery would cause a collapse of publickey cryptography based on NP-Hard problems and all data encrypted using public key cryptography. ([Location 102237](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=102237)) - Tags: [[pink]] - Strong cryptography embedded in a weak system is a waste of time. ([Location 102674](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=102674)) - Tags: [[pink]] - right. A long-standing wireless security protocol called WiFi Protected Access (WPA2) was recently found to be vulnerable [GOOD17b], ([Location 102675](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=102675)) - Tags: [[pink]] - Cryptographic design—do not try this at home! ([Location 102676](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=102676)) - Tags: [[pink]] ## New highlights added May 28, 2022 at 6:35 PM - Quantitative risk assessment is essential to cybersecurity. 17.2 Risk as a Primary Metric Some people will incorrectly assert that you can’t measure cybersecurity. On the ([Location 147677](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=147677)) - the attacker’s budget. Push attacker cost and risk beyond their budgets. ([Location 149423](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=149423)) - Risk is a practical and useful metric for cybersecurity engineering in analysis, guiding design, evaluating quality, and predicting performance under a variety of challenging situations and environments. • Measurement is valuable in engineering to characterize systems, evaluate their performance, predict their performance and failure modes, and improve systems. ([Location 156416](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=156416)) - Measurement is key to understanding how to reduce risk and not just move it around. • A key to understanding risk is understanding how attackers gain value from attack. • A defender’s goal is to push the cost and risk of attacks beyond what the adversary can tolerate. • The value of a cybersecurity architecture has to do with how the attacker’s return on investment is decreased, not with how much the defender spent on the cybersecurity system. • Measuring risk of proposed designs and comparing it to an established maximum tolerable risk guides design iteration to a system design with an acceptable risk level. • Risk assessment analysis occurs in three phases: setup of models and requirements, deep analysis on probabilities, and harvesting of information for guiding design. • The collection of cut sets of attack sequences that achieve the attacker’s strategic goal, and their associated probabilities, gives insight on how best to mitigate risk because the source of the risk becomes quite apparent. Questions 1. What is the primary cybersecurity metric and why? 2. What are the four main uses of metrics in engineering? ([Location 156850](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=156850)) - Cybersecurity engineers are in the business of risk mitigation to an acceptable level. • Risk mitigation consists of developing alternative risk mitigation packages or solution sets, assessing the impact of those mitigations, optimizing the investment for various combinations of optional mitigations, deciding on the best investment, and then executing the revised design. • Candidate mitigation packages are guided by risk assessment analysis regarding the primary sources of the baseline risk. • Mitigation costs should be broken down by who pays and in what life-cycle phases the cost occurs so that they can be properly funded, planned, and managed. • Mitigation cost also includes mission impact, which can be monetized by having senior leadership place value on mission impact factors. ([Location 164280](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=164280)) - For each mitigation package, a revised probability of attack success is calculated for every attack leaf node in the attack tree forest. This forms the basis for understanding how effectively each package addresses the representative attacks, and this becomes the basis of their value. • Risk-reduction return on investment is the risk reduction minus the direct cost minus the mission impact, all divided by the direct cost. It gives a sense of the best “bang for the buck.” • Optimization at practical budget levels picks the best combination of risk mitigation packages at various budget levels to help decision makers decide how best to invest and at what budget level. • An optimization investment curve then guides decision makers toward the best investment, given budget constraints, by analyzing characteristics of the curve, particularly at inflection points where slope changes and the incremental return begins to decrease. • The investment decision is important, but it is equally important to follow the decision through during execution. Schedule delays and budget cuts are inevitable. It is important to ensure that the assumptions made during analysis are maintained and that reassessment is triggered when there is significant enough deviation in assumptions made for estimates on probability of attacker success, direct costs, or the indirect costs of mission impact. Questions 1. Summarize the risk mitigation aspect ([Location 164714](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=164714)) - Describe how the keep-it-simple-stupid principle applies to requirements creep. • Summarize how investment in the early phases of system design pays off. • Explain how and under what conditions incremental development improves system success. • Discuss how modularity and abstraction manage complexity. • Compare and contrast layering and modularity and discuss how they work together in design. • Define the concepts of time and space complexity and relate them to system scalability. • Explain why loops and locality of reference are important to performance optimization. • Discuss the role of dividing and conquering in the design process and how it relates to recursion. Cybersecurity architecture is about how to weave together the ([Location 166462](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=166462)) - Good cybersecurity engineers should be students of failure—all kinds of failure. ([Location 167336](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=167336)) - Never blindly trust other systems upon which yours depends. Recover Once a failure is detected, the system must be recovered. Recovery requires 1. Analyzing and assessing the damage done (e.g., corrupted files). 2. Undoing damage through techniques such as database rollback from a stored checkpoint. 3. Analyzing and diagnosing the cause of the failure (e.g., a software bug). 4. Eliminating the cause of the failure (e.g., patching the system to fix a bug). 5. Returning the system to a safe state of operations. 6. Restoring operations with a minimal loss of work. Analyzing and assessing the damage is critical, particularly ([Location 167337](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=167337)) - Failures will occur with 100 percent probability. In cybersecurity, planning for and surviving failure are essential. Like ([Location 167775](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=167775)) - Uncertainty and unpredictability require a margin of safety. ([Location 168211](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=168211)) - Simple tasks are easier to get right than complex ones. ([Location 168649](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=168649)) - Unfortunately, the road to hell is paved with good intentions, and “just one more requirement” turns into 20 or 30 more requirements. This makes a simple system complex and tends to increase budget and delay schedule substantially. It is particularly bad when the requirements are added midstream during development, causing redesign. ([Location 169083](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=169083)) - Beware of requirements creep; it destroys success; resist! ([Location 169084](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=169084)) - Cybersecurity shouldn’t degrade performance more than 10 percent. ([Location 172144](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=172144)) - Murphy’s Law predicts that “anything bad that can happen will happen.” • Prevent failures before they occur. • Detect failures when they occur. • Recover from failures when they are detected. • Tolerate failures until the system recovers. • Unpredictable user behavior, unforeseen circumstances, and uncertainty all require a significant margin of safety. • Risk often moves around and is transformed in surprising ways in response to proposed cybersecurity design changes. • Simple beats complex every time in terms of security, reliability, cost, and schedule. • Systems should be as simple as possible to meet its requirements. • Investments in early design processes tend to pay off over the life cycle of a project because detecting and fixing issues early is significantly cheaper than doing so later in the life cycle. • Incremental development starting with the most basic possible functionality is an effective way to create dependable systems, but this approach must be guided by an overarching architecture, and interfaces must be clearly specified and stable. • Modularity and abstraction are related and powerful approaches to managing complexity. • Decompose complex processes into simple ones, hiding details and data structures. ([Location 173017](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=173017)) - Cybersecurity is not merely a random aggregation of lots of cool security widgets integrated haphazardly into a system. Cybersecurity is about achieving specific system properties that then yield a specified acceptable risk level, as discussed in Section 17.5. In practice, there is always some uncertainty regarding how well that engineering goal is achieved. Although covering the attack space is an essential concept in cybersecurity engineering, it is only through gaining confidence in system properties that one can realistically achieve sound cybersecurity. This is the role of assurance techniques. In the end, good cybersecurity engineers build a strong case for their assertions about the security properties of their design and implementation. This structured argument is sometimes referred to as an assurance case. ([Location 175200](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=175200)) - Standing in counterbalance to the principle of least privilege is the emerging principle of responsibility-to-share. The responsibility-to-share principle acknowledges that excessive sharing restrictions can damage mission as much as sharing information too freely—known as Dunn’s Conundrum [LEE14]. Responsibility-to-share emphasizes customers discovering and pulling the information they need as opposed to owners knowing a priori what clients need and pushing the information to them. Least privilege must be balanced against responsibility-to-share. Need-to-know works well in a very stable and long-standing world of adversaries, ([Location 179134](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=179134)) - damages. Eventually, there may be insurance for such damages, but insurance companies are only now just beginning to figure out how to enter that market and avoid inappropriately covering the risks resulting from sloppy security practices by vendors. Cybersecurity insurance may one day cover residual risk. ([Location 181319](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=181319)) - Yet if the cybersecurity system labels the protected data but then fails to assuredly enforce access based on those labels, then the labels are actually useful to the attacker to indicate exactly where the most valuable data resides. ([Location 183066](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=183066)) - Observe A pilot observes the current situation including their own position and readiness status and that of other friendly forces, enemy forces, and the external environmental factors (e.g., weather). • Orient The pilot uses the observations to orient themselves to understand the implications of the observations in terms of who has the tactical advantage, how an engagement might turn out, and with what likelihoods and consequences. • Decide The pilot would then sift through and evaluate the possibilities generated from the process of orienting themselves and make a decision as to what actions to take in that situation (e.g., engage an enemy in close-range air-to-air combat). • Act Lastly, the pilot would act on plans formulated during the decision phase (e.g., perform a maneuver to get behind the enemy aircraft and fire a weapon). • Iterate The pilot’s action would affect the situation, leading to another cycle around the loop based on the new situation. The pilot observes, orients with respect to that situation, decides on a new set of actions, and then executes those actions. ([Location 189181](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=189181)) - Ensure that the defense action is not worse than the attack. ([Location 190931](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=190931)) - Cyber situation understanding discovers what is happening, while cyber command and control determines what to do about it and does it. • Situation-based decision making lies at the heart of dynamic cybersecurity defense and of various forms of cybersecurity control. • Cyber situation understanding involves grasping the nature of the attack, reasoning about the implications of the attack to the organization’s mission, assessing the actual and potential attack damages, determining the status of the system defenses, and determining the effectiveness of any dynamic defense actions taken to provide feedback to command and control. • Understanding the nature of an attack involves learning what vulnerability it is exploiting, what paths the attack is using, whether those paths are still open, and how to close them. • The four attack paths include the initial infiltration path into the target system, the egress path to exfiltrate protected data, an ingress path for attack control, and a propagation path to spread a self-propagating attack. ([Location 194863](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=194863)) - Mission-mapping, using attack trees to help the defender determine the implications of an attacker successfully achieving one of their attack tree subgoals, can quantify the change in risk and the source of that risk change. • Intelligence regarding what the attacker has done is valuable, but predictive intelligence about what the attacker might do next is even more useful to get ahead of and stop the attacker. • Defenses cannot be optimized for all possible situations, so information about the situation that is unfolding in real time helps to optimize the defense configuration for that situation. • Cyber battle damage assessment determines both the damage that has been done to the system and mission and the potential damage that could be done. This helps determine the right defensive action and whether those actions are being effective. • Knowing the state of system defenses is important to deciding on the proper defensive courses of action. The state relates to all subsystems and mechanisms within those subsystems and includes the health, stress, and duress of those mechanisms; a determination of whether they are operating or not; their current configuration as compared to the specified configuration; and progress on any dynamic defensive actions taken. • To complete the control feedback loop, the cyber situation understanding subsystem must measure and report how effective and dynamic defense action is. ([Location 195298](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=195298)) - Always fight from the high ground. • Attack when the enemy least expects it. • All warfare is based on deception. • The first casualty of any battle is the plan. • There is no situation so bad that panic will not make it worse. • An army marches on its stomach. Many of these stratagems were learned the hard way—through death and calamity. Cybersecurity strategy is slowly being learned in the same way, ([Location 197920](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=197920)) - There are several ways to discover strategic knowledge in cybersecurity to bridge that experience gap: • By analogy • By direct experience • By vicarious experience • Through simulation 23.2.1 Analogy Analogies are one of the key ways ([Location 198356](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=198356)) - realm. A useful interpretation and extension for cybersecurity exists in a book entitled The Art of Information War [NELS95]. Taking each concept from ([Location 198357](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=198357)) - Which routers? Big, centralized routers owned by the major telecommunication companies and Internet service providers. If one can watch the traffic through these routers, one can see a substantial fraction of cyber events in the world. So, height could be interpreted in the hierarchy of routing infrastructure in the network. ([Location 198358](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=198358)) - Gaining control of programmable entities (e.g., firmware) or underlying hardware at the bottom layers could give an attacker complete visibility and complete control of a computer. A computer’s lowest layers are the cybersecurity high ground. ([Location 198359](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=198359)) - Red teams almost always win. ([Location 199670](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=199670)) - Command and control is key to the decision-making cycle. Command develops and decides on a course of action; control monitors the execution and its effectiveness. • The decision-making cycle refers to the iterative process of proceeding through the four phases of the Observe-Orient-Decide-Act (OODA) decision cycle. • Fast decision-making cycles are an advantage in staying ahead of the attacker in dynamic cybersecurity defensive courses of action. • Autonomic control refers to purely automated actions driven by algorithms that are akin to human reflexes and are reserved for situations where reaction time must be faster than humans are capable of. • Autonomic control works in concert with human-in-the-loop command and control to create a hybrid system that has the advantages of both. • Effective control requires knowledge about what to do in various situations. Such knowledge comes from analogy, direct experience, vicarious experience, and simulation. • Analogies are an effective way of jump-starting knowledge from other domains as long as the limitations of the analogy and the consequent limitations on the knowledge are well understood. ([Location 206222](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=206222)) - Direct experience is painful, but valuable as long as the knowledge gained is properly extracted and institutionalized. • Vicarious experience is available from many sources and is not always adequately reviewed, analyzed, and incorporated. • Simulation offers a fast way to gain new knowledge and insights on how to effectively control cybersecurity systems, as long as the underlying models are accurate where it counts. Validation is always an issue. • A playbook is the concept of applying the knowledge acquired about how to control systems. Explicit documentation and updating of playbooks is essential to an organization building its knowledge base on effectively defending itself. • Courses of action must be developed in advance of being in the heat of a cyber battle. Strategic attack trees can be useful in generating possible courses of action. • What constitutes the best possible course of action in a given situation is dependent on a number of factors, including the stakes, the attacker’s capability and goal, uncertainty regarding the attack, the defense posture, the speed of the attack, and the projected effectiveness of the defensive action compared to the consequence of that action to the mission. • Autonomic control is essential to stop or slow down high-speed attacks. • Control theory provides an effective model to move the system toward the defender’s goal state. • Great care must be taken to prevent autonomic action from becoming an attack surface. • Autonomic action should be urgent, reversible, low-impact to mission, easily recoverable, and temporary. • Meta-strategies address how best to apply strategy: they include not overreacting, being unpredictable, and staying ahead of the attackers through interdiction points. ([Location 206657](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=206657)) - Cyberspace acts as an accelerant for normal human propensities. ([Location 209281](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=209281)) - four important things have changed with the increasing reliance on cyberspace: reach, automation, speed, and feedback control. Reach Forty years ago, one would need to own and operate ([Location 212340](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=212340)) - Infosphere pollution will become a critical problem of the future. Trustworthy system considerations extend to societal systems. ([Location 213213](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=213213)) - Cyberwar can be a massively destructive force that could disrupt sovereignty; the world is in an unstable time in history with respect to cyberspace security. • Society depends heavily on critical information infrastructure that is highly interdependent and vulnerable. • A lack of a strategic smoking gun does not mean that cyber weapons are not pointed at critical systems in the fashion of a Mexican standoff. • Society’s dependency on technology continues to grow without bounds; the Internet of Things is likely to increase fragility even faster. • The virtual economy inside the gaming industry and virtual currencies are emerging as a significant part of the real economy and need to be taken more seriously from a cybersecurity perspective. • Fake news became big news in 2016. Social media enables targeted influence on populations to hack the way people think, reducing trust and creating a threat to society and democracy. Questions 1. What are the potential stakes ([Location 213648](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=213648)) - Zero-secrecy operations may become inevitable. ([Location 216270](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=216270)) - Consider coevolution implications of every cybersecurity design. A similar and seemingly obvious corollary for those involved in both cyberattack and cybersecurity business, such as military institutions, is this: do not develop cyberattacks for which you do not have cybersecurity countermeasures designed, developed, and deployed; such cyberattacks will surely come back to bite you in painful places. ([Location 216272](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=216272)) - There are three important events in recent history that bear on this issue. The first is the Stuxnet attack in which nuclear refinement centrifuges were specifically targeted for the purpose of destruction to slow down Iran’s nuclear development program [ZETT14]. ([Location 216708](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=216708)) - The cyberattack on the nation of Georgia is another important example [BUMG09] [HOLL11]. In this case, Russia targeted and attacked Georgian government assets, including websites, as a prelude to a military action against Georgia. ([Location 217143](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=217143)) - In 2007, cyberattacks began on the Estonian banking system [ONEI16] and progressed to governmental systems, libraries, schools, and other infrastructures. ([Location 217144](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=217144)) - By the reasoning of the analogy, we can reasonably hold organizations responsible for protecting themselves against ordinary cyber threats, but we must not make the strategic mistake of placing them at a huge competitive disadvantage by making them financially responsible to bear the cost of defending against nation-state action. ([Location 217582](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=217582)) - Organizations handle ordinary threats; nations, the extraordinary. ([Location 217582](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=217582)) - The notion of a cyber militia is risky by its very nature. The details of such a cyber militia would need to be worked out, including constraints to ensure that cyber vigilantism does not occur and that unilateral action does not unintentionally lead to international incidents that could escalate into physical war. ([Location 218018](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=218018)) - A former director of the Defense Advanced Research Projects Agency, George H. Heilmeier (1975–1977), created a set of questions to guide program managers toward investments that were considered most valuable. These questions are listed and then discussed as follows: • What are you trying to do? Articulate your objectives using absolutely no jargon. • How is it done today, and what are the limits of current practice? • What is new in your approach and why do you think it will be successful? • Who cares? If you are successful, what difference will it make? • What are the risks? • How much will it cost? • How long will it take? • What are the mid-term and final “exams” to check for success? ([Location 220640](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=220640)) - Tags: [[favorite]] - Research culture is a delicate matter. We have discussed the importance of nurturing more fundamental research in the investment mix (Section 25.5.3). Risk is an inherent part of the culture, and so we need to celebrate intelligent failures as well as great successes. Resisting the temptations to reap the near-term rewards of short-term low-risk investment requires a great deal of discipline and courage. A dedication to careful and dependable research results over flawed spectacular-appearing results requires an ethic that must permeate the organization and all of its inhabitants. All of these attributes of research are part of a culture that must be nurtured every day by both talk and action. Allowing one or two leaders to deviate from the culture can have grave consequences. As John Gall points out, “For every human system, there is a type of person adapted to thrive on it” [GALL77]. An important corollary is that a human system can then become immovably adapted to that type of person. Leaders tend to select those who are like ([Location 221952](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=221952)) - them for subordinate leadership positions. Soon, the entire organization is infested by like-minded, wrong-thinkers in research. Trying to clear the pipeline of such a broken culture requires a massive disruption of the organization and a wholesale expulsion of leadership, which will be met with zealous resistance. Such a change, if it can be affected at all, can take years. In the meantime, research can end up in a miserable state of affairs that will be difficult to recover from. In short, research culture is critical and delicate and must be treated as a precious commodity that cannot be compromised for short-term gain. Research culture is delicate and difficult to correct. ([Location 222386](https://readwise.io/to_kindle?action=open&asin=B07F2RV9HN&location=222386))