The Benefits of Vulnerability Discovery and Bug Bounty Programs: Case Studies of Chromium and Firefox

Abstract

Recently, bug-bounty programs have gained popularity and become a significant part of the security culture of many organizations. Bug-bounty programs enable these organizations to enhance their security posture by harnessing the diverse expertise and outside perspective of crowds of external security experts (i.e., bug hunters). However, quantifying the benefits of bug-bounty programs remains elusive, which presents a significant challenge for managing them. Previous studies focused on measuring the value provided by a bug-bounty program in terms of the number of vulnerabilities reported or, in some cases, based on the inherent properties of the reported vulnerabilities, such as severity or exploitability. Importantly, beyond these inherent characteristics of a reported vulnerability, the value of a bug-bounty report also depends on the probability that the reported vulnerability would be discovered by a threat actor before an internal expert could discover and patch it. In this paper, we present a data-driven study of the Chromium and Firefox vulnerability reward programs. First, we estimate the difficulty of discovering a vulnerability using the probability of rediscovery as a novel metric, comparing the rediscovery probabilities of development and stable releases and studying if rediscovery is a memoryless process or if rediscoveries are clustered in time. Our findings show that vulnerability discovery and patching provide clear benefits by making it difficult for threat actors to find vulnerabilities; however, we also identify opportunities for improvement, such as incentivizing bug hunters to focus more on development releases. Second, we compare the types of vulnerabilities that are discovered internally vs. externally and the types of vulnerabilities that are exploited by threat actors. We observe significant differences between vulnerabilities found by external bug hunters, internal security teams, and external threat actors, which indicates that vulnerability reward programs provide an important benefit by complementing the expertise of internal teams, but also that external hunters should be incentivized more to focus on the types of vulnerabilities that are likely to be exploited by threat actors.