Mistrust, Dissonance, And Noise: The Assessment of Technological Activities in Argentina’s Research Career System

Introduction

Over the last decade new literature has contributed substantially to the understanding of university technology transfer, with a particular focus on university–industry collaboration and organizational barriers affecting knowledge commercialization (Pinheiro, Benneworth & Jones, 2012; Arboleda, & Plazas Tenorio 2024; Liaudat, Zukerfeld and Terlizzi 2025). However, the role of academic career evaluation systems, including tenure and promotion criteria, remains largely implicit. Discussions and campaigns in favor of a transformation in research evaluation have multiplied and deepened (Peruginelli and Pölönen 2023; Morgan-Thomas et al. 2024; Rushforth 2025a). In this context, concrete reform proposals have been developed, calling on relevant stakeholders to revise their evaluation mechanisms and criteria. Interventions such as the San Francisco Declaration on Research Assessment (DORA, 2013), the Leiden Manifesto (Hicks et al. 2015), the Metric Tide report (Wilsdon 2016), and the Hong Kong Principles (Moher et al. 2020) are considered landmark declarations that have sought to raise awareness and, in a way, “nudge” (Rushforth 2025a) research actors to reconsider their positions regarding the use of quantitative indicators, open science, and research integrity.

This reform movement, known as Responsible Research Assessment, emphasizes the need to recognize and value a broad spectrum of knowledge, languages, outputs, career trajectories, and institutional profiles in the scientific realm (UNESCO 2021; CLACSO 2022). The basis for this claim is to promote greater inclusion, equity, and fairness in research evaluation processes (Rovelli 2024), but also to signal the types of scientific outcomes and career paths that should be encouraged to achieve the production of socially impactful knowledge (Donovan 2007). Evaluation is viewed not merely as a technical domain—it involves political and strategic considerations about the kind of research and knowledge that should be promoted. Therefore, it is essential to consider different geographic and institutional contexts and to question the idea of a global consensus surrounding the reform agenda (Vasen 2025).

Recognizing the diversity of scientific outputs and career paths implies valuing achievements that go beyond scientific publishing and narrow notions of productivity (Pontika et al. 2022). In particular, technological activities and products (TAPs) form part of the incentive and reward systems for applied research (Molas-Gallart et al. 2002). In our work, we understand TAPs as encompassing both the activities involved in developing technology and transferring it, and the outputs that result from them. These may include intellectual property rights, technological services, advisory work, and similar undertakings. Such activities are typically carried out in collaboration with, or in response to the needs of, non-academic entities. Unlike the notion of social impact, which emphasizes the effects of research on external stakeholders or broader society, TAPs are examined primarily from the perspective of the researcher and the R&D-performing institution, focusing on their role, contributions, and interactions with nonacademic actors. Whether in the context of hiring, tenure, or promotion in academic careers (Genshaft et al. 2016; Alperin et al. 2019; Rice et al. 2020; Bouwma-Gearhart et al. 2021; Melkers, Woolley & Kreth 2023; Pekkola & Siekkinen 2024; Cañibano et al. 2024) or in funding calls for innovation and R&D projects (Ramos-Vielba, Thomas and Aagaard 2022; Norn et al. 2024), TAPs are increasingly part of evaluation processes.

Although the analysis of social impact may offer relevant perspectives, the evaluation of TAPs involves a different set of challenges that must be addressed on their own terms. A number of recent studies have focused on the evaluation of social impact, both at the level of universities and in the context of academic careers and R&D project funding (Ma et al. 2020; Ma and Agnew 2022; Rice et al. 2020; Benneworth and Olmos Peñuela 2019; Urbanc and Jong 2025; Bornmann 2013). A variety of methodologies for measuring ex post social impact have been developed, which are qualitatively different from those used in traditional research evaluation, as they usually involve the development of case studies and interviews or workshops with external stakeholders and aim to assess impact within a broader time frame (Smit and Hessels 2021). By contrast, TAPs may form part of more routine ex post evaluation processes to which academics are subjected. Just as published papers are assessed over a given period, so too are applied and technological activities. However, since the traditional evaluation methodologies used in those contexts, such as peer review, were not designed for this kind of output, they may show significant limitations when applied to TAPs.

This article explores current practices for evaluating TAPs in researchers’ applications for promotion within the Argentine research system. Based on an analysis of external reviewers’ reports on promotion applications, we seek to gather information on how TAP-related content in this context is evaluated and weighted. What do evaluators highlight about TAPs? What difficulties do they perceive? What perspectives do they hold on the relationship between TAPs and scholarly publishing? What criteria do they use to judge the relevance of TAPs?

In the next section, we discuss broader debates around the assessment of social impact and technology transfer activities and then focus on how it has been evaluated specifically within academic careers. We then outline Argentina’s scientific career system—the context for our study—along with the methodology and data sources employed. Following this, two sections present our findings from the quantitative analysis and the qualitative text analysis, respectively. Finally, we highlight the key challenges in evaluating TAPs and explain why this topic warrants separate consideration from social impact assessment, all within the larger goal of fostering a more inclusive understanding of academic contributions.

Literature review

Methods

The analysis of this material followed two different methodological approaches. First, we conducted a textual quantitative analysis using the Alceste method (Analyse des Lexèmes Cooccurrents dans les Énoncés Simples d’un Texte), implemented through the Iramuteq software. This semi-automated method is designed to analyze large corpora by identifying patterns of repetition in simple statements (Reinert 1986, 1990; Romano et al., 2020). It detects dominant “lexical worlds,” defined as groups of words and expressions associated with specific topics or fields, and examines their proximity or distance based on word positioning and repetition patterns. We applied this tool twice to the evaluation reports to assess the extent to which the evaluation of technological activities and products features prominently in the discourse of external reviewers . Using the software’s default parameters (sentence window size = 12), the first corpus comprised 7,127 segments, of which 6,072 were classified (85.20%), while in the second corpus, 638 out of 889 segments were classified (71.77%).

Second, we carried out a qualitative content analysis of section 2 of the expert’s evaluation form (“Technological Activities”) from all 421 reports following an inductive category development procedure (Mayring 2022). Coding was performed by two researchers and an inter-coder check was conducted on a preliminary sample of the texts; discrepancies were discussed in joint sessions. The extracted segments were systematically compared, abstracted, and grouped into higher-order categories. This iterative aggregation process ultimately yielded the five distinct overarching themes that encapsulate the reviewers’ key concerns. Oxley and Gulbrandsen’s (2025) notions of productive and unproductive variability informed the interpretation of inductively emergent themes but did not shape the design of our data collection.

Results

The Lexical Worlds of Evaluation

The Alceste method was applied in two separate runs. In the first, we used the full text of all 421 reports as the corpus. This analysis revealed five distinct lexical clusters, which showed only weak connections with one another (see Figures 1 and 2). By examining the most representative words in each cluster, we can infer the main focus of the reviewers’ comments within each group. Class 2 (grey) centers on applicants’ publication records, with an emphasis on journals and their indexing, reflecting the importance reviewers place on traditional scholarly output. This cluster appears largely isolated from the others. Class 5 (lavender) is associated with the evaluation of applicants’ involvement in training and mentoring activities. Cluster 1 (red) captures reviewers’ assessments of the significance of the applicant’s research results and their future potential as a researcher. Finally, Class 3 and 4 (green and blue) are shaped by discipline-specific language: the green cluster is linked to engineering and computer science, while the blue cluster reflects biological terminology, particularly in the context of veterinary science.

Interestingly, no lexical world emerged related to technological production or knowledge transfer, despite the evaluation form including a dedicated section for this purpose. This absence contrasts with the presence of a distinct lexical world related to training and mentoring graduate students (Class 5, lavender), which reviewers often elaborate on even though it is not explicitly prompted.

Given the lack of specific insights into the evaluation of technological activities and products in the analysis of the full reports, we conducted a second Alceste analysis restricted to Section 2—the section of the form explicitly addressing applied and technological activities (see Figures 3 and 4). Notably, only 274 of the 421 reports (65%) contained processable content in this section; the remaining reports were either left blank or included minimal responses such as “none” or “not applicable.”

This time, the method identified four distinct lexical worlds. Class 1 (red) stands apart from the others and centers on domain-specific references to the biological and veterinary sciences. Class 4 (violet) appears to focus on the perceived social impact of technological developments. Meanwhile, classes 2 (light blue) and 3 (green) are closely interconnected and pertain to the formal aspects of evaluating and certifying technological products and activities.

As mentioned, only 274 of the 421 reports contained evaluable content in this section. Here, we did observe disciplinary differences: in engineering, 77.78% of the reports contained usable information, compared to 58.0% in veterinary sciences and 55.4% in computer science.

Text length of reviewer comments ranged from 11 to 916 words, with an average of 113, a median of 74, indicating that most texts are relatively brief, with only a few outliers. Only 37 reports (13%) exceeded 200 words. Disciplinary differences in this regard were not particularly marked, although the longest texts were found in the engineering field (see Figure 5).

The results of this initial exploratory analysis of the text corpus offer several preliminary insights. First, when considering the full reviewer reports, technological aspects do not appear as a central focus. Instead, reviewers tend to emphasize other dimensions—such as publications, training and mentoring activies, or the development of the trajectory of the applicant’s research agenda. However, when attention is restricted to the specific section dedicated to technological production, certain key terms do emerge. These are primarily related to the formalization of such activities and include references to “STAN,”³ “consulting,” “agreement,” and “patent,” as well as mentions of “SIGEVA,” “documentation,” and “evidence.” To better understand how these terms are used in context and what they reveal about the evaluation process, we conducted a qualitative content analysis of this section. Notably, this section of the expert report seems to be more actively engaged with and valued in engineering, as reflected by a higher proportion of reports containing usable information and a longer average text length compared to the other two disciplines.

Discussion

Our analysis offers several insights into how technological activities and products are evaluated and which difficulties arise in this process. Overall, TAPs emerge as a category that is conceptually close to, yet distinct from, social impact and that sits within existing peer review frameworks for individual careers.

First, most of the literature on the assessment of applied research focuses on the evaluation of social impact, often overlooking TAPs as such. While these two domains may appear similar at first sight, in practice they demand distinct evaluation approaches. Unlike ex ante impact assessments commonly used in grant peer review, ex post assessments of social impact typically require the involvement of external stakeholders and the consideration of medium- to long-term outcomes. These multi-actor approaches are resource-intensive and remain uncommon in individual career assessments, especially in developing countries, where limited budgets and institutional capacity make such procedures difficult to implement.

By contrast, the evaluation of TAPs, also conducted ex post, tends to be incorporated into routine performance reviews (Genshaft et al. 2016). This was clearly the case in our study: external reviewers were expected to assess TAPs alongside more traditional outputs such as publications or grants, often without adequate guidance or a shared understanding of how this should be done. Similar difficulties have been documented by Derrick and Samuel (2016) in the context of the REF, where reviewers expressed discomfort and uncertainty in evaluating societal impact due to the lack of prior experience and benchmarking tools.

Furthermore, while the broader innovation management literature often relies on quantitative indicators of technology transfer to measure technological activities—such as the number of patents, licensing agreements, and spin-offs (Lockett & Wright, 2005)—our study deliberately adopted a qualitative, peer-review-centered lens. Although applicants in the CONICET system can formally log patents, spin-offs, and licenses in their promotion applications, we found that these traditional quantitative metrics have a surprisingly low presence in reviewers’ reports. This absence reflects a broader structural reality of Latin American innovation system: Dutrénit and Arza (2010) note that formal patenting and spin-off creation are among the least important channels for university-industry linkages. Instead, most technological and transfer activities occur through direct engagement with companies via technical services, consulting, or collaborative agreements that do not necessarily generate easily quantifiable intellectual property.

Attention to these informal and service-oriented channels is crucial, as they may among the most frequent modes of university-industry interaction even in highly developed science systems (D’Este & Patel, 2007). Recognizing this reality, the Argentine science system has attempted to implement new mechanisms to legitimize and properly evaluate these non-traditional outputs, most notably through the creation of Technological and Social Development Projects (PDTS). However, despite this institutional push, the uptake of the PDTS framework to reorient research evaluations has been decidedly slow and uneven, constrained by bureaucratic hurdles and entrenched academic evaluation cultures (Naidorf et al 2020).

Second, our analysis identified several procedural issues that interfere with the evaluation of TAPs, particularly regarding the quality and consistency of documentation submitted by applicants. A recurring concern among reviewers is the lack of detailed, verifiable information necessary to support a sound assessment. Although SIGEVA allows for the inclusion of supporting documents—such as agreements, certificates, or technical reports—these materials were often absent or insufficiently detailed.

The contrast with the evaluation of scientific publications is instructive. In the latter case, providing the full text of the article and a DOI is usually sufficient for evaluators to independently verify the work and judge its originality and significance. Many TAPs, in contrast, involve commercial partners or confidential data and are not publicly accessible, making verification more difficult. As a result, reviewers must rely solely on what the applicant submits, which often lacks the rigor or completeness required for robust evaluation.

We also observed frequent dissonance between different sources of information. Technological activities may be described in the CV or in the narrative statement but omitted from the formal SIGEVA sections where evaluators expect to find them. This inconsistency forces reviewers to “dig” through different parts of each file in order to reconstruct the applicant’s trajectory—a process that many find frustrating. One plausible explanation for this dissonance is that many technological activities involve consultancy fees or royalties that may not have been formally reported to the institution. Applicants may therefore be reluctant to include these activities in the official platform and instead mention them only in more informal sections of their application.

These discrepancies introduce “noise” into the evaluation process: instead of a clear and coherent representation of the applicant’s work, reviewers encounter a fragmented and sometimes contradictory account. While such problems can be interpreted as technical flaws in the organization of evaluation (“unproductive variability” in the terms of Oxley and Gulbrandsen (2025), they may also signal deeper systemic issues. In particular, the dissonance between sources can reflect researchers’ need to generate additional income due to inadequate salaries or cumbersome procedures for formally reporting income-generating activities.

Third, reviewers often express uncertainty and mistrust regarding the originality and relevance of TAPs. In several cases, they question whether the activities described genuinely represent innovation or simply routine professional services. In the absence of clear criteria for what counts as an original or impactful TAP, assessments become highly subjective. Some reviewers imply that these activities are motivated more by personal financial interests than by scientific goals, especially when they fall outside the applicant’s main research line. Others, however, recognize their potential importance for the productive sector and socioeconomic development. Reviewers also express frustration when applicants appear not to “play fair” and present low-complexity activities as R&D-related tasks, seemingly to tick another box in the evaluation form (Wróblewska 2021).

Fourth, similar divergences appear in how reviewers approach other fundamental dimensions of evaluation, particularly the temporal horizon and the relationship between technological outputs and publications. With respect to temporality, some evaluators adopt a strict ex post perspective, focusing only on completed TAPs, while others are more flexible and take into account the potential future contributions of ongoing research lines, even when these have not yet produced tangible outputs. In such cases, reviewers may “rescue” a candidate who lacks declared TAPs by emphasizing the expected downstream impact of their work.

A comparable lack of consensus emerges in how reviewers connect TAPs to academic publishing. Some acknowledge that intensive engagement in applied or technological activities may come at the cost of fewer publications, and are willing to treat this as a legitimate trade-off. Others regard this argument with suspicion and maintain that at least part of the work should always be publishable. These disagreements suggest an implicit hierarchy in how most reviewers value outputs: publications remain the core currency of scientific credibility in research careers (Cañibano et al. 2024), while TAPs are seen as secondary—valuable perhaps, but only once the standard benchmarks of productivity have been met.

Fifth, the variability we observed in the evaluation of TAPs can be interpreted through the distinction proposed by Oxley and Gulbrandsen (2025) between productive and unproductive variability in peer review. According to their framework, productive variability arises when differences in evaluation stem from reviewers’ disciplinary expertise or from divergent, yet legitimate, professional perspectives. Unproductive variability, in contrast, is associated with flaws in the design or implementation of evaluation processes—such as unclear guidelines or inadequate documentation—that undermine fairness and consistency.

Our findings suggest the presence of both types. On the one hand, cases of insufficient or inconsistent documentation clearly exemplify unproductive variability. These problems interfere with reviewers’ ability to form sound judgments and could, at least in principle, be mitigated through improved institutional practices, including clearer instructions about what evidence to provide and stricter requirements for coherence across sections of the application. On the other hand, disagreements about what constitutes originality, how to weigh future versus current outputs, or whether publications should take precedence over technological work reflect productive variability. These are not administrative failures but points of interpretative flexibility rooted in different understandings of academic merit. While Oxley and Gulbrandsen emphasize disciplinary expertise as the main source of productive variability, our study suggests that in the Argentine context reviewers also draw heavily on personal normative views about what should count in an academic career.

Finally, despite the growing prominence of research assessment reform agendas—such as CoARA, DORA, and the recent EU high-level conference on reforming research assessment in Copenhagen—which call for the recognition of more diverse contributions, technological and applied products still appear to carry limited weight in practice. Our lexicometric analysis of reviewer reports showed that, even in applied fields, TAPs do not constitute a prominent “lexical world” within the broader evaluation discourse. Other dimensions—particularly publications, training and mentoring activities, and the projected development of the applicant’s research—receive significantly more attention.

Limitations

This study has several limitations that should be acknowledged. First, our analysis is restricted to a single national research career system and focuses specifically on promotion applications in three applied fields. The patterns we identify may therefore not generalize to other disciplines, periods, or institutional settings, nor do they necessarily reflect dynamics in CONICET entry-level evaluations, where criteria and reviewer behavior may differ. Second, we rely exclusively on external reviewers’ written reports and did not have access to the deliberations of panels, boards, or executive bodies, which means we capture only one stage in the evaluative chain. Consequently, we chose not to link these non-binding reports to final promotion outcomes, as subsequent committee mediation makes any direct correlation between reviewer discourse and career consequences unreliable. Third, the bureaucratic and procedural problems we document—such as insufficient documentation and inconsistencies across different sections of SIGEVA—should be interpreted as a snapshot of the system at the time of data collection. Since 2018, CONICET has issued more detailed guidelines and adjusted some procedures for reporting and assessing technological and social development activities, which may alleviate part of these obstacles.

Conclusions

As movements to reform research assessment increasingly promote the recognition of diverse contributions, our study shows that evaluating technological and applied products remains a particularly challenging area. TAPs differ from traditional outputs not only in form but also in the conditions under which they can—or cannot—be evaluated through standard peer review. They often require more intensive engagement from reviewers, as well as context-specific interpretations of application, intent, and outcome. This reinforces the concern that a “one-size-fits-all” evaluation model is likely to overlook crucial dimensions of applied research. Our findings suggest that institutional ambiguity and variability in reviewer criteria complicate the fair evaluation of TAPs. Unlike publications, which can be more readily validated through established norms and infrastructures, TAPs often lack clear documentation standards, stable benchmarks, and shared expectations.

We argue that TAPs should be analyzed and assessed separately from broader concepts such as social impact. While the two are related, they correspond to different stages in the process through which knowledge is appropriated and used, and they call for tailored evaluation approaches. Some of the challenges we identify could, in principle, be addressed by adapting methodologies developed for assessing social impact—such as narrative case studies and multi-actor evaluations—but these approaches are resource-intensive and complex to implement. In developing countries like Argentina, where evaluation systems operate under tight budgetary and administrative constraints, importing such models wholesale is unlikely to be feasible.

In this context, future research should examine whether incremental reforms within existing frameworks might offer a more viable path forward. Yet procedural improvements alone will not resolve the deeper tensions we identify. Beyond such technical refinements, enduring tensions over what counts as academic merit and the marginal status of applied and technological work are rooted in broader institutional cultures. As with any change to research assessment practices, reforms must be discussed, validated, and supported by disciplinary communities, whose interpretive authority and expertise remain central to the process.