Software touches practically every facet of industry, academia, and government. It is necessary for many human endeavors, from leisure and entertainment to safety and defense to health and critical infrastructures. Yet, while certain proprietary software manufacturers have become household names, free and open source software (FOSS) is often less known, yet it’s a substantial component of the overall software landscape. The Linux Foundation study, Census II of Free and Open Source Software—Application Libraries, showed that FOSS permeates the software landscape. Moreover, the recent Department of Defense (DoD) memo, Software Development and Open Source Software, underscores not only the importance of FOSS software used by the DoD but also how FOSS has transformed how software is designed, developed, tested, distributed, deployed, operated, and maintained. Importantly, this same memo also cautioned about the increased potential of vulnerabilities and supply chain risks that can accompany the use of all reused software, including FOSS.
The U.S. government and DoD relies on its federally funded research and development centers (FFRDCs), university affiliated research centers (UARCs), and industry to educate about FOSS security and implement practical policies, guidance, processes, and technology to enact the intent of the Software Development and Open Source Software memo. For these reasons, the SEI recently conducted the workshop “Open Source Leadership Jam 2022: A conversation with FFRDCs & UARCs.” The purpose of this workshop was to start the conversation among these entities and begin to coordinate work to elevate the trustworthiness of FOSS and the entire FOSS ecosystem while continuing to enjoy the speed, innovation, transparency, and collaboration it fosters. This SEI blog post highlights the current FOSS landscape, describes the workshop and its ground rules, summarizes the ideas that emerged from this seminal event, and articulates a future vision for ongoing collaboration.
The Free and Open Source Software Landscape in the DoD
The DoD’s 2022 memo defines open source software (OSS) as “software for which the human-readable source code is available for use, study, re-use, modification, enhancement, and redistribution by the users of such software.” For our purposes we will use the terms “free and OSS” (FOSS) as a synonym for OSS. In practice, FOSS is openly developed by collaborative networks of programmers. Creating proposed improvements by anyone is both permitted and encouraged. FOSS projects range in size from a single developer (the median) to many thousands (14 thousand for the Linux kernel). The Linux Foundation study Census II of Free and Open Source Software—Application Libraries, produced in partnership with Harvard Laboratory for Innovation Science and the Open Source Security Foundation (OpenSSF), highlighted the widespread use of FOSS and reported that FOSS is used in an estimated 98 percent of codebases (including proprietary codebases) across a broad spectrum of organizations in the public and private sectors. The DoD echoed this point in its memo on software development and open source software, noting, “There are millions of publicly-available OSS components, libraries, and applications capable of accelerating software modernization activities.” The DoD memo not only underscored the widespread use of FOSS but stressed the importance of open source software to the DoD Modernization Strategy:
The Department’s 2018 Cyber Strategy … directed the Department to increase the use of secure OSS and to use commercial off-the-shelf tools when possible. The Department’s forthcoming Software Modernization Strategy centers on the delivery of resilient software capability at the speed of relevance. OSS forms the bedrock of the software-defined world and is critical in delivering software faster.
Clearly, the widespread use of FOSS and its importance to the DoD’s strategy make ensuring the safety and security of FOSS and the FOSS supply chain essential. As the Linux Foundation study noted, the need to do so came into stark relief when attackers discovered and exploited severe vulnerabilities in widely used FOSS products, such as OpenSSL, log4j, and the Linux kernel. Yet assessing FOSS is different from proprietary software because it requires augmented metrics and indicators of health and stability. What’s more, the DoD articulated two fundamental concerns about using and releasing external software (including as FOSS):
- Using externally maintained code in critical systems potentially creates a path for adversaries to introduce malicious code into DoD systems.
- Imprudent sharing of code developed for DoD systems potentially benefits adversaries by disclosing key innovations.
Due to these concerns, the DoD noted that it must “clearly articulate how, where, and when it participates, contributes, and interacts with the broader OSS community.” To this end, it included guidance on software development and open source software. The following sections present key elements of the DoD’s guidance on FOSS use, FOSS development, and contributing to FOSS projects.
The DoD as a Consumer of FOSS
The DoD espouses Adopt, Buy, Create guidance, in that order, for software acquisitions. Software adoption involves using off-the-shelf (OTS) software, including FOSS and government-off-the-shelf (GOTS) software, and proprietary solutions. DoD programs, however, are often unable to acquire entire solutions OTS, which necessitates custom solutions and consumption that will almost certainly incorporate FOSS.
The landscape of technologies is vast, and programs are largely free to choose for themselves exactly which combination of languages and components they will use to build their solutions. A sampling of the FOSS component landscape reveals the magnitude of possibilities:
- Maven Central (Java): 482K modules
- PyPI (Python): 385K modules
- Nuget (.NET): 313K modules
- Rubygems.org (Ruby): 172K modules
Larger solutions often incorporate software written in more than one language and/or technology stack, so the number of possible configurations is huge. DoD program managers are expected to manage the full lifecycle of FOSS within their programs, which is a hard task given the magnitude of the alternatives.
The DoD as a Producer of FOSS
The memo encourages non-National Security Systems programs to adopt an open-by-default posture when creating custom software. This requires programs to architect their solutions in a way that separates critical and non-critical components. Programs are then encouraged to release non-critical components as open source. Programs are required to balance the needs of program protection with the benefits of releasing non-critical components as open source, such as reducing ongoing development and maintenance costs for themselves and other DoD programs that adopt these components.
The DoD as a Contributor to FOSS
DoD programs are encouraged to actively contribute to FOSS, because reducing the number of customizations a program manages directly improves the maintainability of the software and reduces costs. Regulations allow both government personnel and contractors to directly contribute to FOSS if it is in the interests of the government to do so. As with producing FOSS, contributing to existing FOSS requires the program to balance the need for program protection with the benefits of contributing. In particular, as noted in the DoD memo, “creating a separate, DoD-specific version of any OSS project, for any reason, increases support risk and should be avoided whenever possible.”
Un-Conference: Sparking a Conversation
The “Open Source Leadership Jam workshop took place June 9, 2022 and was led by Aeva Black (Open Source Hacker, Microsoft), Jacob Green (Mosslabs and OSPO++) and David A. Wheeler (Linux Foundation). The meeting was facilitated by my Software Engineering Institute (SEI) colleagues Linda Parker Gates and Aaron Reffett. I also served as a facilitator. We organized the event as an “un-conference” to foster a wide-ranging, free, and open discussion. The un-conference concept allows participants to steer the meeting, relying on their expertise to determine discussion topics and durations. Topics were permitted to evolve during the day, and we put no strict time limit on discussions. Likewise, the law of personal mobility allowed participants to move freely between conversations. We operated under Chatham House Rules:
- Participants are free to use the information received.
- Neither the identity nor the affiliation of the speakers may be attributed (unless specifically authorized).
The workshop leads framed the workshop theme and briefly discussed the architecture of the DoD FOSS memo. After a period of discussion, attendees generated twelve discussion topics. We then used a multi-voting technique to identify the initial topics for discussion for the remainder of the day. We placed the seven remaining topics in a backlog to be addressed if time permitted. The first five topics were selected for discussion, creating an overall structure for the remainder of the workshop that paralleled much of the “consumer” context established by the DoD’s OSS memo.
The ordered list of topics was as follows:
- trusted processes over individual identities
- zero-trust architecture inside the FOSS process
- risk management in the consumption of open source
- supply chain artifact resolution
- institutional structures for community/conversations (OSPO networks)
- international collaboration
- blockers for release of FOSS and how do we resolve them?
- leading versus trailing indicators of software quality
- standards/frameworks
- release without attribution
- actionable experiment generation
- domain composition categorization of priorities
Figure 1: Relationship of Key Topics Identified to DoD Guidance Areas
A summary of the top five topics selected for discussion appears below.
Trusted Processes Over Individual Identities
While much expertise rests on the community of stakeholders, relying on these individuals or individual entities to forge a path for executing the DoD’s guidance on FOSS and software development presents a number of problems. Individual experts come and go, and open source software is often developed either by self-interested volunteers or employees of companies seeking to advance their particular goals within open source project communities. While individual expertise ebbs and flows over time, many open source institutions create a stable and sustainable situation by facilitating the transfer of institutional knowledge to future generations of contributors and maintainers, and establish rigorous development practices to ensure the quality of releases. Consequently, the group agreed that tools and techniques that assess processes and technologies of open source projects are appropriate to measure trustworthiness, rather than methods that focus on individuals, companies, or nations in a blanket manner. They cited Debian, OpenStack, Kubernetes, and Linux Kernel as examples of the gold standard for such processes.
Any process applied to the challenge of FOSS software development in the DoD should be able to address the following questions:
- How do consumers know if a project has a vulnerability embargo process as part of their process?
- What is the best practice for assessing the health and stability of a FOSS project?
- How do you get access to data and information regarding those trusted processes?
- How can you “see” the rigor of FOSS processes (e.g., SLSA)?
- How can you verify human processes (e.g., human review)?
What’s more, the group argued that any process developed to address the challenge of FOSS in the DoD should include the following specifics:
- Development process changes should be reviewed like code (e.g., infrastructure as code [IaaC]).
- Static and dynamic analysis tools are needed to look for malicious FOSS packages (e.g., package-analysis at OpenSSF).
- Verified reproducible builds are needed to counter malicious builds and attributions (e.g., to counter attacks like the one on SolarWinds’ Orion). One participant noted that the tool diffoscope is useful for identifying unexpected differences.
- Multiple tools should be applied in continuous integration/continuous delivery (CI/CD) pipelines to look for vulnerabilities to be addressed.
- The process should rely on mechanisms, such as The Update Framework (TUF) and tools such as in-toto to provide secure updates and evidence of processes performed.
- Privileges granted to packages ought to be reduced.
- Programs should not run at install time (this ability is often used for exfiltration).
- Digital signatures should be used to prevent tampering in transit.
- The process should incorporate independent review (e.g., security audits).
Zero Trust
The zero trust security model has become an important part of the nation’s security posture. In May 2021, President Joseph Biden signed Executive Order 14028, “Improving the Nation’s Cybersecurity,” which explicitly requests agencies to adopt zero trust cybersecurity principles and adjust their network architectures accordingly. The zero trust security model strives to reduce risk inherent in perimeter-based security architectures by removing implied trust and explicitly authenticating and authorizing subjects, assets, and workflows. To support Executive Order 14028, the Cybersecurity and Infrastructure Security Agency (CISA) developed a Zero Trust Maturity Model to help agencies implement zero trust architectures.
In addition to Executive Order 14028, the NIST Special Publication 800-207, Zero Trust Architecture, served as a foundation for the discussion of this key topic. One proposal emerging from the discussion is to incorporate zero trust architecture (ZTA) into FOSS development. It was proposed that zero trust architecture should also be included as an addendum to the next update of the NIST Cybersecurity Framework, and that FOSS and zero trust should be included as a foundation for the NIST Smart Cities and Communities Framework. One of the participants noted that FOSS has already been raised in the context of smart cities, citing the panel “OSS Framework in Smart Cities” at the September 2020 Smart Cities Connect Conference and Expo.
Other ideas put forth included advocating for
- funding research for zero trust
- applying least privileges to specific packages
- creating a recall system that would generate automatic recalls to customers and/or owners
- researching whether zero trust principles can be applied to the CI/CD pipeline
Risk Management in the Consumption of FOSS
Participants offered a number of concrete approaches for helping DoD FOSS consumers navigate risks inherent in its use. For instance, one idea was to provide tools that could help decision makers analyze existing metrics and information on FOSS that indicate risk; for example, OpenSSF Scorecard (which could be invested in and improved), the OpenSSF Best Practices Badge Program, deps.dev (a service that examines sites such as github.com to find up-to-date information about FOSS packages and creates a graph that makes visible any problems), and, generically, repo data (e.g., the number of maintainers, the last commit date, and information about the last commits).
Other ideas included
- investment in tools to detect and prevent malicious packages and automation for such tools
- policy and tools to identify red flag projects
- a CVE for any package and/or branch whose support has ended (e.g., log4j 1.x)
Participants also noted that, for the consumer side, due diligence matters. They suggested the development of policies and/or measures that could address matters such as how efficiently a consumer is at updating FOSS software. The following would be helpful for this purpose:
- use of dependency monitors
- use of package managers
- use of automated test suite
- readiness to update in hours or days
- establishment of a mean time from package update to production release (or similar metric)
- use and tracking of DORA metrics that align with the consumer’s risk tolerance, such as deployment frequency (DF); lead time for changes (LT); mean time to recovery (MTTR); and change failure rate (CFR)
Supply Chain Artifact Resolution
One participant described analyzing the supply chain as “a huge spider tree.” Certainly, this domain is dense, complex, and filled with intersecting paths. The discussion quickly moved to the software bill of materials (SBOM), which the National Telecommunications and Information Administration (NTIA) describes as “a nested inventory for software, a list of ingredients that make up software components.” In the NTIA document Framing Software Component Transparency: Establishing a Common Software Bill of Materials (SBOM), the NTIA further defines the SBOM as
… a formal, machine-readable inventory of software components and dependencies, information about those components, and their hierarchical relationships. These inventories should be comprehensive–or should explicitly state where they could not be. SBOMs may include open source or proprietary software and can be widely available or access-restricted.
Some workshop participants, however, stated that SBOMs they receive are currently focused on a “depth of one” and on managing license risk. They noted that SBOMs need more depth and insight (e.g., dependencies, libraries within packages or containers, to name a few). Moreover, the DoD is more concerned about complete depth for vulnerability analysis rather than license risk. In addition, simply having an SBOM is insufficient. In particular, users must compare the list of components to current lists of known vulnerabilities and then ignore vulnerabilities that cannot be exploited in that context. So, while SBOMs represent one approach to get clarity about the supply chain, by themselves they remain insufficient in meeting the DoD’s requirements for FOSS software development.
Participants also identified another issue related to the supply chain: How can you uniquely identify projects, packages, and software (a point also made by the Census II study noted above)? Any method for doing so would need to be immutable, canonical, and ideally unique. A standards body, such as the IETF, might review and endorse such identification method(s). One proposal seeking to help address inputs is the GitBOM project, which constructs a Merkle Tree using the Git Object ID of all software artifacts in a supply chain and then relies on the tree’s identifier (essentially its git hash) to identify the inputs that created a software package.
Other related concerns include the following
- a verification mechanism is needed for supply chain artifacts
- verification evidence must travel with the artifacts
- multiple language platforms (the method must work across languages, packaging forms, and platforms)
No matter how we proceed in this area, participants recognized the need to seek community involvement and additional DoD stakeholder input.
Institutional Structures
To generate headway on the key topics cited above (and additional shared topics), this discussion focused on concrete ways to organize and collaborate in a structured and sustainable manner to meet the DoD’s guidelines for FOSS software development. Participants agreed on the need to leverage existing institutional structures and create new ones where necessary. Participants also generally agreed on the need to create a mechanism and forum for sustaining conversation and implementation within this community and look at the existing institutional structures of FFRDCs and UARCs paired with the open source organizational structure of Open Source Program Offices (OSPOs), which exist in many private companies in industry. To do so, the following ideas were suggested:
- Leverage FFRDCs, UARCs, and the network between them.
- Create a working group and perform community building with associated mailing lists and meetups for stakeholders by
- avoiding a high barrier to participation
- making it available as a continuing professional development opportunity for participants
- conducting outreach
- engaging high-profile speakers
- Bring new people in by creating a recruiting and outreach network with government, academia, and industry (as these are shared challenges).
- Pick one-to-three work outputs, for instance:
- a DoD FOSS FAQ update developed with this community working group
- a general policy documents update and review by the group
- a training course (e.g., Linux Foundation’s Developing Secure Software)
- a domain-specific collaboration, knowledge exchange, and meeting (for example, NIST and Smart Cities)
- Add MiL-OSS (an existing mailing list/community) as a subgroup.
- Position learning and training offerings as professional development opportunities that are relevant and attractive to software developers. Require training in secure software development for those developing custom software for the government (e.g., Open OSS fundamentals).
- Engage high-profile government speakers, especially from sponsor agencies (e.g., the NIST Smart Cities Conference).
- Coordinate with the U.S. OSS Policy Assembly, as software security is a core public policy concern driving FOSS globally.
- Coordinate with other FOSS security efforts (OpenSSF, OSTIF, etc.).
- Foster international cooperation.
- Leverage cross-agency incentives.
- Fund accredited secure development practices (including FOSS) as part of computer science degrees and related programs (e.g., software engineering). Do not limit FOSS to a security silo.
Synthesizing FOSS Leadership Jam Discussions
Following a productive day of freewheeling discussion of the key topics, participants then began to make associations between the key topics and the focus areas outlined in the DoD’s memo on open source software and software development. The participants from the open source software ecosystems provided recommendations encouraging the DoD consumers to connect with some of the emerging and evolving technology in reproducible builds, SBOM standards, and verifiable supply chain artifact trees.
The participants from the DoD software factories would investigate integrating capabilities into their approval processes and CI/CD pipelines that are amenable to assessing FOSS project health and status (e.g., best practice badges, scorecard, etc.). They would also investigate other technologies for deeper analysis of SBOMs, reproducible builds, and the use of evidence of complete and verifiable artifact trees, all of which would inform cyber risk management activities to better understand their consumption of FOSS. In doing so, the software factories would provide feedback on the use of such capabilities “from the factory floor” directly to the open source communities developing these tools. This would improve the use of such tools by all communities (including the DoD) and would align to the DoD memorandum’s guidance on contributing to FOSS (section 4 of the DoD memo).
The DoD memorandum on open source software represents an excellent balance between the potential and opportunity of open source software and the understandable concerns about security and supply chain dependencies. Participants agreed that the Adopt, Buy, Create guidance and the open by default posture are appropriate, and recognized that security and supply chain issues are best addressed through risk management. Moreover, participants recognized that the SBOM is a key aspect (though not sufficient) for security and supply chain management. The participants also agreed that tools and techniques that assess the trustworthiness of the development processes of open source projects are appropriate, as opposed to methods that focus on tracking open source contributions from individuals, companies, or nations in a blanket manner. The participants identified several specific ideas that could support such an approach. We welcome the opportunity to partner with other federal agencies who could align to efforts in accordance with DoD leadership, particularly those efforts related to the memo that characterizes relevant issues from both the consumer and the producer perspectives.
It is expected that this group will convene again to report on experiences gleaned from employing much of what was discussed during this workshop and to extend these ideas to more DoD stakeholders and other DoD programs that benefit from FOSS.
Additional Resources
The Zero Trust Journey: 4 Phases of Implementation
A Cybersecurity Engineering Strategy for DevSecOps that Integrates with the Software Supply Chain