Chain of trust: Unraveling the references among Common Criteria certified products (2024)

4.1 Building the vertices and edges

Following the approach of Janovsky et al.[7], we use the public CC artifacts in this study, typically available in the PDF format. These include a variety of documents: a security target (specification of the certified product), web pages with additional metadata and summaries generated by the CC portal, a certification report (a summary of the certification results), a maintenance report (description of minor changes in an already certified product), and a protection profile.

Thesec-certs tool[7] collects certification artifacts and features for all certified products and one can use an API to interact with the tool. Among others, a unique identifier (certificate id) is collected for each certified product. Additionally, the presence of identifiers of different certificates is also collected for each of the certified products with high precision (99%). We build the reference graph from these attributes, i.e., we include the edge $(c_{i},c_{j})$ into the graph iff either the certification report or the security target of $c_{i}$ explicitly mentions²²2The sec-certs tool monitors for such mentions with 25 distinct regular expressions described in the related study[7]. the certificate id of $c_{j}$. To this end, the complete dataset contains 2712 references (edges) in 5394 products (vertices).

4.2 Categorization of reference meanings

To capture the reference context of the edge $(c_{i},c_{j})$, we introduce the edge-labelling function $\ell$, which maps each edge to its categorical code. Before showing how we automatically learn the function $\ell$ on the whole dataset of 2712 edges, we describe how we deduced the different reference contexts.

To determine the initial codes, we exploratively studied the certification artifacts. First, one co-author randomly sampled 100 different edges $(c_{i},c_{j})$ from the established graph and went through the text artifacts of $c_{i}$ to develop the initial codes (i.e., category types) of reference meanings. These codes were then observed by a second co-author who went through a different set of 100 edges. The final categories were then refined in a discussion between these two co-authors, during which the descriptions and examples for codes were synthesized. Overall, we discovered two fundamental reasons (binary codes) that necessitate a reference:

This taxonomy can be further refined to capture more fine-grained context³³3We provide the complete codebook for both the coarse-grained and the fine-grained categories from our replication package[6]., yet at the expense of subtle differences between the codes that are challenging to be automatically extracted from the certification artifacts due to ambiguous natural language used.We provide this finer taxonomy mostly to describe the referencing culture, and we work with the binary codes when presenting results in the next section. The fine categories (denoted later as multiclass codes) are:

Having the codebook available, we sampled 400 edges from the reference graph. Two co-authors then independently reviewed the certification artifacts for these instances and manually annotated the edges with codes. A special code unknown was used when the annotator was unable to decide on the right category.During the annotation task, we noticed 6 ($1.5\%$) cases where the reference was irrelevant to the studied artifacts, e.g., due to a typo in the referenced id or the reference being mistakenly left out in the document. Given the number of such cases, we further disregard it in our work. The annotation agreement for the 5-class taxonomy between the co-authors after the first annotation round was $82\%$. Even better, the agreement on the binary codes was $94\%$. The conflicting and unknown instances were then resolved in a discussion between the annotators. We are now ready to define certificate reach as follows:
For every certificate $c\in\mathcal{C}$, we define its certificate reach as the number of certified products from which a path to $c$ exists in the reference graph, such that all edges on the path are labelled with the component-reuse (C) code.

4.3 Learning the edge-labelling function

The reference context for $(c_{i},c_{j})$ can often be reliably inferred from the sentences surrounding the identifier of the referenced certificate $c_{j}$ in the certification artifacts of $c_{i}$. For instance, the sentence “this is a re-certification of BSI-123-456-CC” clearly signals the re-evaluation category. To learn the function $\ell$, we first extract the sentences surrounding the reference identifiers from the documents. After feature extraction, we train a machine learning classifier to map the features to the final categories. This process is illustrated in Figure1 and we describe it closely in what follows. We split the 400 manually annotated edges in a 50:50 fashion into train:evaluation sets.

5.1 Referencing culture (RQ1)

Out of the total 5394 products, 1659 ($30.76\%$) have at least one reference. However, there is a notable contrast in the referencing habits among smartcards (ICs, Smart Cards and Smart Card-Related Devices and Systems category), smartcard-related devices (Trusted Computing, Products for Digital Signatures, Other products and devices categories), and products from different categories. As illustrated in Figure2(a), majority (1295, $74.08\%$) of smartcards reference some other product. Specifically, 1080 ($61.78\%$) smartcards do reference some other certified product in a component-reuse relation, while 390 ($22.31\%$) smartcards reference some predecessor. The smartcard-related products are much less connected: 138 ($11.92\%$) have a component-reuse reference, while 123 ($10.62\%$) reference their predecessor. The devices unrelated to smartcards very rarely utilize references, with only 90 ($3.62\%$) products referencing some predecessor and 69 ($2.77\%$) products engaging in component-reuse.

To determine whether the average vulnerability impact aggravates in time, we analyzed the reference graph temporal evolution. According to Figure2(b), smartcards reached their highest point with an average of 2.5 transitive component-reuse references in 2017. By November 2023, an average smartcard relies on the security functions of nearly two other certified products. Furthermore, Figure2(c), shows that as of 2023, both smartcards and related devices extend their reach to almost two additional products. It is also apparent that the limited referencing among other categories demonstrates a consistent pattern. In summary, our findings regarding the referencing culture are as follows:
The primary motives for cross-referencing among CC certificates are (i) component reuse among devices and (ii) reference to a predecessor. Smartcard products particularly favour component reuse, with the majority depending on at least one other certified product. The reach of an average smartcard has incrementally increased to approximately 2, around which it currently stabilizes. Although smartcard-related products occasionally have certified dependencies, products from other categories remain largely isolated.

5.2 Certified products with high reach (RQ2)

Within our reference graph induced by component-reuse edges, we examined ten products with the highest reach. The champion, BSI-DSZ-CC-0813-2012 (Infineon smartcard IC M7794 A2 with ROCA-vulnerable RSALib v1.02.013), is transitively referenced from 77 other products. Combined, the top-10 transitively referenced products influence 196 other devices. The share of all active smartcards relying on the top-10 products has varied over time but has consistently exceeded 10% since 2004. By November 2023, the current top-10 products affected 23% of all valid smartcards. With such influence, this makes the high-reach products extremely attractive targets for adversaries. Notably, all of the top-10 devices are, in fact, microcontrollers or some integrated circuits, often with cryptographic libraries. Our analysis revealed that the higher reach of the product is positively associated with a higher Evaluation Assurance Level, and hence the trust in that device. This was confirmed with Spearman’s rank correlation test with the resulting association of $0.23$ (p-value $<2.73\textrm{e}^{-23}$, one-sided alternative).

To better understand the threat that a vulnerability in a high-reach device could have on its surroundings, we measured to how many products it may propagate. In the component-reuse sub-graph, we selected all weakly-connected components with at least 10 certified products (15 different components in total)⁵⁵5The largest component, an outlier with 707 certified products, was infeasible to be analyzed manually and was excluded from the analysis.. In each of the components, we monitored the product with the highest reach and manually annotated all its incoming transitive references with labels from the fine-grained categorization. We then counted the number of component used references in these trees. This is because a critical vulnerability in a node most likely propagates through the component-used edge, as supported by the previous case studies[7]. The high-reach nodes were all microcontrollers or ICs, with the exception of one operating system. We conclude that in the majority of the studied components, a critical flaw might spread dramatically to the derived products. In total, we labelled 170/245 (69%) edges with component-used code. On (macro-)average, 70% of the whole graph component is susceptible to the vulnerability originating from the wide-reach product. All studied components, the labelled edges and a summary table are available from the replication package[6]. This allows us to answer RQ2:
Just a dozen of smartcard devices influence more than 10% of the whole ecosystem at any given time. The typical high-reach device is an integrated circuit at the bottom of the smartcard component stack, implementing critical cryptographic functionality. These devices are generally evaluated to high assurance levels, EAL5+ or more.Our experiment showed that such high-reach devices are indeed used as components in nearly 70% of the products that reference them. As a consequence, critical flaws in these high-reach devices would likely spread to many other products, crippling the broader surrounding.

5.3 Ageing of referenced products (RQ3)

CC certificates have a 5-year validity according to operational guidelines[4], with some schemes (e.g., German and Dutch[1, 14]) requiring a re-evaluation of the products in a composite chain every 18 months to mitigate risks from emerging threats. In this sub-section, we analyze whether archived certificates are being referenced and we check whether the national schemes follow the 18-month policy.

We identified $40$ products that referenced an archived component when they were issued. A manual review showed that, in 17 instances, the certificate archival date was incorrectly marked as its issuance date in the Common Criteria portal, while the dates in the certification reports were correct. We also confirmed this in correspondence with one vendor of such certificates.Furthermore, 6 products do reference components archived 5-12 months earlier, indicating these referenced products were likely active but nearing archival during the evaluation period.

Eight certificates for various versions of the KoCoBox MED+ Netzkonnektoruse the archived STARCOS 3.6 COS C1 smartcard, with the latest versions issued in June 2023, almost three years after thearchival. This raises questions about whether documentation was not updated with newer references or if certain schemes and labs overlook the use of archived products. Additionally, four certificates were collectively archived following a vulnerability discovery, yet their evaluation outcomes were leveraged to quickly issue new certificates referencing those recently archived. Remarkably, the CombICAO Applet v3 ePassport, released after these re-certifications, still cites the compromised components in its certificatebut omits them from the security target. This could indicate documentation discrepancies or even a “race condition” in the certification process, where the vulnerability in the underlying product was disclosed after the assurance evaluation yet before the final certificate was issued.

Additionally, 4 references actually represented a predecessor relationship but were misclassified as component reuse by our model. In another instance, a vendor failed to update a security target with a new reference, leading to a false positive alert, although the corresponding certification report was accurate.

We also explored the rate at which the reach of certified products fades to zero following their archival, when considering the references from active products. We focused on $268$ products that maintained a positive reach at the time of their archival. The findings reveal that the transition to zero reach extends beyond a year on average, as depicted in Figure3(a).

We extended our analysis to still active but ageing products; specifically, we examined the references to certificates older than 18 months to check for adherence to the 18-month validity policy upon re-evaluation. In Figure3(b), we illustrate the Cumulative Distribution Function (CDF) for the age of certificates referenced in smartcard composite evaluations across major schemes: German, French, and Dutch. Across these schemes, NL complies with their policy in nearly all products ($1\%$ violate this rule), while FR and DE lag, with $19\%$ and $23\%$, violating the rule respectively⁶⁶6The average for the whole CC is close to the numbers of DE and FR. The most notable exception is Spain, with only 43% products complying with the 18-month policy. Moreover, recently issued FR products (2020 and newer) improved w.r.t. following this policy, with only 4% violating the rule. DE does not show such a trend.. We do not have a clear explanation for this discrepancy. While our analysis might be slightly affected by the noise from some re-assessments not being published, we believe this analysis shows an interesting trend differentiating the national CC schemes. Overall, we answer RQ3 as follows:
In the CC ecosystem, referencing archived certificates is rare: by November 2023, only 14 different products were found to use components from archived certificates at the time of issuance.The persistence of archived products with positive reach, however, extends well beyond a year.We observed that the Dutch scheme is the strictest towards old components in composite evaluations.In contrast to this, some evaluation bodies certify products that include components nearing the archival or those already archived.

6.0.1 Variability in CC reference patterns.

The use of references in CC varies across product types and certification bodies. Smartcards stand out for their extensive use of references. We argue that such dense reference network enhances security through independent evaluation of sub-components and the ability to track vulnerability spread via explicit, transient dependencies.The frequent use of references in smartcards can be attributed to the nature of these devices, which are built from multiple certified layers to facilitate reuse and reduce costs. This is a marked difference from, e.g., operating systems, which, despite their greater code volume and complexity, seldom rely on certified dependencies.

6.0.2 References represent genuine dependencies.

Programming language package managers typically come with an explicit dependency graph. No such thing is available for the CC, and our investigation aimed to fill this gap. Our research found that vendors are strongly motivated to explicitly reference other certified products, whether as sub-components or predecessors. This approach allows them merely to prove compliant use during evaluation rather than evaluate the security of the referenced products. Consequently, almost all references indicate a strong link between products, with a minimal number of irrelevant instances. In the CC, references can be tracked using identifiers in the certification documents; a method confirmed effective in previous research[7]. Nonetheless, the occasionally ambiguous terminology surrounding the reference identifiers limited our ability to distinguish only between component reuse and predecessor contexts. Despite an approximate 10% error rate of our model (given to a large extent by inherent noise in the certification documents and missing reliable metadata), we believe our analysis reliably identifies overall trends. We also developed a fine-grained taxonomy of reference contexts, though more suited for manual annotation tasks.

6.0.3 Strategic scope reductions.

Recall from Section5.3 that we observed instances where vendors opted to narrow the certification scope following the discovery of vulnerabilities, rather than addressing the issues and pursuing re-certification. While some situations justify this approach – such as components that cannot be updated – in general, such practice raises concerns and merits further investigation in future research. We also noted how the demands of the certification process can influence the shape of the final product. To meet the certification requirements, products may adopt a more strict development approach, incorporating additional safeguards not found in non-certified counterparts. Yet, this adherence can also prompt vendors to deliberately limit the evaluation scope of their product, excluding essential features from the certification scope. Vendors might exclude certain functionalities from their certification claims, despite these features being centric to the nature of the certified product. An illustrative case is the certified network camera (certificate SERTIT-115), which assumes a secure network interface. This anomaly in the certification landscape deserves a broader discussion about the motivation and implications of such practices.

6.0.4 Exploring potential vulnerability spread through references.

The presence of a component-reuse reference is a strong indicator of vulnerability transmission across the graph edge, yet it does not imply it. Accurately assessing this risk requires a deep understanding of the certified configuration scope, including what components are employed and what functionality is not used. The boundaries of the certified products are communicated through complex language in the certification documents, and their automatic identification remains an open problem – we merely point out this issue to be worth considering for future work with a significant expert-driven manual analyses. Our investigation was partly inspired by the notable ROCA vulnerability[13] and its impact on numerous certified products[7]. Our goal was to determine the likelihood of similar incidents occurring, considering their origin in products with a high reach. Also, while direct references in composite products are typically highly relevant, the relevance of indirect references is less certain, as they may utilize less functionality from the products they reference.