Perspectives on relevance – the relevance test in the constructive research approach

Interventionist research (IVR), such as the constructive research approach (CRA), has been suggested as a method to improve the relevance of management accounting (MA) research. Although literature identifies several perspectives on relevance, the current assessment of CRA focuses on practical relevance. Moreover, an overreliance on pragmatism in assessing CRA research in the form of CRA market tests has been criticized. This article analyzes the challenges inherent in conducting and assessing CRA research, both conceptually and with a CRA case example. In order to overcome these possible CRA challenges, we suggest analyzing CRA relevance from multiple perspectives. The perspectives in question are those of practical value relevance, legitimative decision relevance, academic value relevance, and instrumental decision relevance. Further, we suggest that indications of relevance in CRA studies can be analyzed during the research project. In particular, we introduce the relevance test as an explicit part of the CRA research process. We suggest a new tool, the Relevance Diamond would facilitate conducting the relevance test and aid the analysis of CRA relevance from multiple perspectives. Furthermore, we suggest new interpretations of what should constitute a pass in the CRA market tests under special circumstances, thereby contributing to CRA methodology, and especially to the analysis of relevance and ‘battlefields’ of different interests in CRA/IVR projects.


Introduction
Many scholars have called for accounting research that is relevant to practice (Johnson and Kaplan, 1987;Kasanen et al., 1993;Labro and Tuomela, 2003;Malmi and Granlund, 2009;Modell, 2014;Westin and Roberts, 2010).Malmi and Granlund (2009) suggest that a suitable way of creating both practical and theoretical relevance is to solve practical case problems alongside practitioners and theorize the findings using interventionist research (IVR) approaches, such as the constructive research approach (CRA) of Kasanen et al. (1993).In CRA, the researcher actively participates in the innovation of management accounting (MA) constructions (also called constructs), such as new accounting tools.According to the market test idea of Kasanen et al. (1993), the key pragmatic test of constructions is whether they work in practice, as exemplified by whether they are adopted, widely used, or create benefits (see also Jönsson and Lukka, 2005;Labro and Tuomela, 2003).When an organization adopts the construct created in the CRA process, a weak market test has been passed, suggesting that the construct has practical value.Stronger indications of practical usefulness require passing the semi-strong or strong market test, which respectively requires that the construct is used in other organizations, or that financial benefits of the use of the construct apply to multiple businesses.
The success or validity of the CRA method is largely based on the practical usefulness of the construction in an organization or beyond it.However, evaluating the success of CRA projects using the market tests can be challenging: it is not always clear what constitutes 'usefulness' or 'working in practice' (see Labro and Tuomela, 2003;Lukka, 2000;Piirainen and Gonzalez, 2013).Piirainen and Gonzalez (2013) note that the use of a construction may be a problematic measure of success because the adoption of the construction does not actually measure its qualities.For example, sub-optimal constructions may be adopted purely to confer legitimacy.In addition, constructions adopted but soon abandoned may be difficult to assess in terms of relevance, which, conceptually, is not an unambiguous term (see Lukka and Suomala, 2014).
We suggest that CRA constructions might be analyzed from several relevance perspectives, and also based on the potential relevance and relevance over time.IVR projects tend to involve conflicts between various interest groups, such as practitioners and academics, who often have different views and opinions (e.g., Suomala et al., 2014).This suggests that several, even contradictory, perspectives on the relevance of CRA constructs may emerge during the research process.The perspectives on relevance suggested in earlier research include practical and theoretical (or academic) relevance, alongside several other categories, such as value relevance, decision relevance as well as societal, instrumental and 'legitimative' relevance (see Barth et al., 2001;Järvinen, 2006;Lukka and Suomala, 2014;Nicolai and Seidl, 2010).
Although market tests for practical relevance are presented in CRA literature, clear testing rules for other perspectives on relevance, such as the academic relevance of CRA contributions, have not been articulated.This study presents a way to analyze multiple perspectives on relevance in CRA research, called the relevance test (including ex-ante analysis, as visualized in the Relevance Diamond tool, see Appendix).Considering several perspectives on relevance already during the CRA process might increase the theoretical orientation and the publication potential of CRA research (see Lukka and Suomala, 2014).
Explicit analysis of relevance perspectives might also assist the CRA researcher to understand and mitigate the effect of possible conflicts over research direction.
It is challenging to evaluate the validity, impact, or relevance of a CRA construction over a short time span because the wider societal or theoretical impacts are only revealed after a substantial period of time (see Jönsson and Lukka, 2005;Labro and Tuomela, 2003;van der Stede, 2012).Recent value relevance, decision relevance, societal relevance and legitimative relevance discussion establishes that the adoption of a new construction does not necessarily generate much value or increase legitimacy in the eyes of various stakeholder groups (Barth et al., 2001;Lukka and Suomala, 2014;Nicolai and Seidl, 2010;Piirainen and Gonzalez, 2013).In IVR, Jönsson and Lukka (2005) separate the ex-ante consideration (such as how a construct is supposed to work) and the ex-post consideration (the actual outcomes and theoretical research contribution).However, the CRA process steps do not explicitly include the analysis of relevance over time or from multiple perspectives during the CRA process.
Thus, an aim of this article is to contribute to the analysis of the relevance of CRA constructs over time, that is, both ex-ante and ex-post.
Practical and theoretical relevance perspectives in the IVR context can be complementary (Lukka and Suomala, 2014;Suomala et al., 2014).Suomala et al. (2014) noted that compromises are often required during an IVR process to resolve tensions among different interest groups.This suggests that changes in relevance perceptions (and in circumstances) can occur during the research project, and the eventual CRA construction (and its relevance) may be something other than was anticipated.Thus, while acknowledging the value of CRA, we see the potential to amend the current CRA methodology in assessing CRA relevance over time and in developing the market tests.These amendments offer an opportunity to contribute to both accounting theory and practice, especially under circumstances, where conducting the classic CRA market tests can be challenging (e.g., with amidst changing stakeholder views or in the non-profit sector).
This article analyzes CRA methodology and provides tools for the assessment of the CRA contribution and the relevance of CRA.We suggest amendments to the ways in which CRA research is conducted.Accordingly, our research questions are: How can the assessment of CRA research (e.g., the market tests) be developed further?How can the relevance of CRA research be assessed during the CRA process while considering multiple relevance perspectives?
Both conceptual analysis and empirical analysis of the CRA research by Sippola (2008) are used to prompt amendments to the CRA market tests established by Kasanen et al. (1993) and Labro and Tuomela (2003).Analysis of multiple perspectives of relevance and the amended market tests are expected to improve evaluation of CRA projects, and boost the theoretical rigor of the CRA methodology, particularly in special circumstances, such as in fast changing business conditions where the classic (ex-post) CRA market tests may not be fully applicable.
Fulfilling the conditions of the semi-strong and strong market tests (Kasanen et al., 1993) is not necessarily in the interests of the CRA case organization, which might prefer to block the wider use of a construction so as to maintain competitive advantage.Therefore the academic interests of disseminating information and practical efforts of value creation may conflict in a CRA project, and require consideration if CRA research is to be published.Generally, our discussion touches on the methodology, impact, and relevance of MA case research.This article starts with a conceptual analysis of relevance and continues with an illustration of an earlier CRA study.In methodological terms, this resembles the approach of developing a case research process by learning from earlier empirical case projects (Ahrens and Dent, 1998;Labro and Tuomela, 2003;Lukka and Suomala, 2014;Suomala et al., 2014;von Zedtwitz, 2002).This approach may also be referred to as design science, in which a research design is improved with new rules and solutions (van Aken, 2005).Our empirical data are based on the empirical qualitative material (extensive field notes of the interviews and project meetings) collected during a CRA case study (Sippola, 2008), in which a quality cost measurement system was constructed in a software company.The relevance test introduced in this article could be a one-off event, but in longer projects or after significant project changes, the test might be repeated.The relevance test could also facilitate evaluating the relevance of other IVR research types.By noting the potentially conflicting and multilayered nature of the different perspectives on relevance in the case setting, we also contribute to the recent analysis of potential conflicts of interests in IVR research (called battlefields by Suomala et al., 2014).

Relevance in accounting research
Extant research presents several aspects of relevance: value, theoretical, decision, societal and legitimative relevance (Barth et al., 2001;Lukka and Suomala, 2014;Nicolai and Seidl, 2010).In the IVR context, Lukka and Suomala (2014) highlight the importance of practical, theoretical and societal relevance.Lukka and Suomala (2014) define relevance as "something that is of significance for something else" but note that relevance is often connected with values and usefulness.Lukka and Suomala (2014), following Flyvbjerg (2001), suggest that the Aristotelian intellectual virtues correspond to different decision-making and relevance considerations: techne (art/craft) corresponds to practical relevance, episteme (knowledge) corresponds to theoretical relevance and phronesis (practical common knowledge) corresponds to societal relevance.Flyvbjerg (2001, 57) suggests that phronesis is particularly related to ethics and the deliberation of values with reference to praxis.Flyvbjerg (2001, 106-107) points out that society contains several values and sub-groups that are a venue for democratic processes and power games.This suggests that societal relevance is equivocal: there are contending views among interest groups of what is relevant or ethical (Flyvbjerg, 2001, 57;Meyer and Scott, 1983).Thus societal (or social) relevance indicates that a topic is important for some interest groups, although not necessarily for others.Regarding the CRA relevance analysis, we consider societal relevance to relate to several relevance perspectives, but especially to legitimative decision relevance perspective where legitimacy of an issue may differ according to the ethical perceptions and norms of the members of the case-organization, interest groups or society (Flyvbjerg, 2001, 101).Societal relevance also touches upon practical value creation (employment, taxes) as well as academic relevance; where new research insights can resolve societal problems.Lukka and Suomala (2014) infer that in CRA, societal relevance typically follows when the construct passes the market tests.
In accounting research, value relevance means that something has a material impact on the value of an organization.However, proving such an impact is difficult and suggests causality.
For example, an accounting number is relevant if it has a significant association with the equity market value (Amir et al., 1993;Barth et al., 2001).However, in the case of public sector organizations, for example, the value of the organization is not measured by expected discounted future profits or cash flows, but by aspects like the expected value added for society as a whole.The differing perceptions of value or legitimative relevance held by different stakeholder groups can also trigger power games and conflicts in IVR projects (Flyvbjerg, 2001, 3;Tucker and Parker, 2014;Suomala et al., 2014).Thus an assessment of relevance from multiple perspectives might be helpful in balancing stakeholder interests (see Suomala et al., 2014).
Decision relevance, on the other hand, means that information, such as accounting information is relevant if it is capable of making a difference to the decisions of those using it (Barth et al., 2001).Value relevance and decision relevance, however, are connected: in the long run, the value of the firm is a matter of firm performance, something affected by managerial decisions (Barth et al., 2001).Nicolai and Seidl (2010) divided relevance in practical decision-making situations into instrumental and conceptual relevance.The same study also reported that practical relevance considerations include legitimation aspects called legitimative relevance.Instrumental relevance includes forecasts and knowledge of the ways of achieving goals such as improving throughput times.Conceptual relevance refers to the meaning of concepts in a given decision-making situation.Legitimative relevance relates to verifying the credentials of individuals, groups, knowledge, or organizations, and the sources of legitimacy include practical value, expectations, societal values, ideas and prestige1 .
Furthermore, there can be different levels of legitimative relevance, such as case-specific (often related to practical value relevance) and societal legitimacy (related to societal values more generally, see Lukka and Suomala, 2014).
As far as we know, both the ex-ante assessment of relevance and the interplay of the multiple perspectives on relevance have not been addressed together in MA literature.In our view, relevance can include considerations of probable future value or expected legitimacy.
Relevance can also be considered from multiple viewpoints or on different levels, such as those of a case organization, a research organization, or society.Viewing relevance from multiple perspectives also suits CRA projects, which aim to arrive at constructs affecting decision-making in organizations and to make academic contributions.We define relevance in CRA as something that is likely to significantly affect the practical value, academic value, instrumental decisions, or legitimacy of the entity in question, such as a case organization, the researcher, or society.This definition accepts that relevance is not the same for all stakeholders and at all analysis levels.Further, insignificant issues, where neither success nor failure affects profit, legitimacy, academic work, or society, warrant little consideration.Thus deciding whether a CRA construction has relevance may demand explicit analysis regarding multiple relevance perspectives.
Practitioners in the private business context often consider value relevance aspects, such as profitability, even before implementation of a construct (ex-ante), whereas researchers often evaluate academic success and theoretical relevance retrospectively (ex-post) (see Bolton and Stolcis, 2003;Edwards and Emmanuel, 1990;Scapens, 1994;van Aken, 2005).Several practical development needs might emerge during IVR case research, some of which can be far from the initial research topic; creating conflicts over the focus of the research (Suomala et al., 2014).In the IVR context, Jönsson and Lukka (2005) called the practical (case or insider) view the emic perspective on IVR, and the theoretical (academic or outsider) view the etic perspective.In this article, we use the concept of academic relevance to refer to the theoretical (or outsider and learning) perspective.

Perspectives on relevance and the relevance test
Based on the discussion above, for analysis purposes, we next categorize relevance in the CRA context.Following Barth et al. (2001), we first distinguish 1) the decision relevance dimension and 2) the value relevance dimension.These are depicted by the horizontal and vertical dimensions (axes) in Figure A1 in the Appendix.Second, we acknowledge that value may be perceived differently in practice and in theory, and thus differently by practitioners and by academics.This distinction is used to separate the ends of the vertical value relevance dimension into practical value relevance and academic value relevance perspectives.Third, by separating the ends of the horizontal decision relevance dimension, we note that decision relevance includes the instrumental (improves operations in short-term), and legitimative perspectives (see Burns and Scapens, 2000;DiMaggio and Powell, 1983;Meyer and Scott, 1983;Nicolai and Seidl, 2010).
Accordingly, our analytical perspectives on relevance are: 1) practical value relevance (for example long-term financial benefits to the case organization); 2) legitimative decision relevance (stakeholder and societal support for the organization, e.g., funding); 3) academic value relevance (e.g., contributions to science, new knowledge); and 4) instrumental decision relevance (information facilitating short-term improvements, e.g., in throughput times).
Instrumentally relevant information is often a prerequisite to establishing practical value relevance.However, we consider instrumental decision-making relevance to be of a more short-term nature than the (long-term) practical value relevance.For illustrative purposes, these relevance perspectives are portrayed separate in the Appendix, but we acknowledge that the perspectives on relevance can complement each other, be independent, or even conflict.
An example illustrating complementarity would be academic innovations that can have practical value (e.g., activity-based costing, see Kaplan, 1998).Legitimative decision relevance might also co-exist with instrumental decision relevance (see Hyvönen and Järvinen, 2006), as when a firm adopting a new accounting system to enhance legitimacy improves its instrumental performance, and ultimately its value.In contrast, details of corporate scandals have revealed how the pursuit of short-term instrumental benefits, e.g.managerial bonuses, can threaten long-term legitimacy and value.Another example would be tobacco industry, where maximizing value and organizational legitimacy may conflict.
It is important to recognize the difference between the theoretical underpinnings of legitimation and value.Legitimative relevance refers conceptually to socially constructed or subjective perceptions of individuals and groups, which may warrant qualitative research attention, perhaps using institutional theory or actor network ideas (see Latour, 1987;Meyer and Scott, 1983;Oliver, 1992).In contrast, value relevance typically refers to a more functionally understood maximization of profit requiring cost-benefit or cash flow analyses, for example (e.g., Barth et al., 2001), although profit may also be seen as constructed (Hines, 1988).These conceptual differences suggest that qualitative or mixed methods might be useful in CRA studies.
In a case research context featuring conflicting interests among stakeholders, the value and legitimacy perceived by the owners and other stakeholders can differ.Suomala et al. (2014) suggest that CRA projects can become battlefields between different practical and research interests.Nevertheless, even an unsuccessful project that does not produce increased practical value can provide a good theoretical understanding of a case (Granlund, 2001;Jönsson and Lukka, 2005;Malmi, 1997).Practical value relevance resembles the CRA market test idea of Kasanen et al. (1993) in that a good construct adds company value.The strong market test requires, however, benefits to several businesses to be proven while the concept of practical value relevance also refers to the value added for one company (cf.Kasanen et al., 1993).
Understanding the relevance perspectives can help manage a research project so that the research objectives are relevant and acceptable by most project participants, thus improving the likelihood of a CRA project being successful (see also Suomala et al., 2014).The explicit analysis of the four perspectives of relevance (stated above) already during the CRA process is here described as conducting the relevance test.The relevance test is an additional ex-ante analysis that complements the CRA market tests and is based on the potential relevance (or expected relevance) of the CRA construct.Analysis of the relevance perspectives can also be used for assessing the success of CRA research and understanding potential conflicts in IVR studies (see Suomala et al., 2014).We present a tool for relevance testing, the Relevance Diamond (in Figure A1 in the Appendix), which can help visualize and support the assessment of relevance dynamics over time.

Assessment of academic relevance
Relevance is essential in assessing any research but the relevance or impact of a piece of research may also relate to other fields and times (van der Stede, 2012).Scientists tend to follow generally accepted, legitimized, ways of doing research until they are able (or forced) to modify them following the advent of fundamentally influential innovations (Alvesson andSandberg, 2011, 2014;Kuhn, 1970).This suggests that academic research includes both innovation and continuity: novelty, but also reliability or plausibility in qualitative research terms (Golden-Biddle and Locke, 1993).Novelty and plausibility may even be opposed: too much theoretical novelty may undermine plausibility (Nørreklit, 2003).Moreover, there may be different preferences for research practices.Therefore, legitimation and theoretical relevance are not straightforward issues, even among academics in the same field (Hines, 1988;Kuhn, 1970;Latour, 1987;Nørreklit, 2003).
The assessment of the relevance of CRA research has focused on practical relevance, and less consideration has been given to how the academic relevance of the CRA should be assessed.
In CRA research, the division between success and failure is usually determined by the practical implementation or rejection of the construction (Kasanen et al., 1993; also Krumwiede, 1998).This may seem to be a clear-cut division, but the intensity of the use of the construction can vary (Labro and Tuomela, 2003).
The proposed relevance test examines different relevance perspectives, for example by considering the shape of the Relevance Diamond tool (see Appendix).Typically, the relevance perspectives, such as practical and academic relevance, complement each other but they may also be independent.For example, even after successfully completing a practical CRA project, researchers can struggle with the academic publication process (Lukka and Suomala, 2014).On the other hand, publications, new academic knowledge, and theoretical refinements may emerge from failed CRA projects (those where constructions are not adopted) through learning or critique (Alvesson, 2012;Granlund, 2001;Kasurinen, 2002;Labro and Tuomela, 2003;Malmi, 1997).If the construct is rejected, the CRA project might be reported as action research, design science, or another form of IVR (see Darke et al., 1998;Jönsson and Lukka, 2005;Lukka, 2005).A researcher might, for example, identify the reasons behind the non-adoption decision and attribute them to different relevance perspectives, thereby increasing the potential for contribution and the likelihood of publication.
We suggest that assessing academic relevance might include three levels: A) indications of potential academic relevance, some of which can be assessed in the beginning (ex-ante) or during the project, B) publications, and C) institutionalization of the scientific results (e.g., citations and generalizability).First, at level A, a prerequisite of potential academic relevance in CRA is access to data (see also Lukka, 2000Lukka, , 2005)).Indications of potential academic relevance include methodological and theoretical expertise (obtained e.g. by participating in post-graduate courses), the existence of a research gap, the plausibility of the data in both quantitative and qualitative terms (e.g., an interesting case, length of observation period, large number of interviews, or the amount of statistical data), and the plausibility of the method likely to be used in data collection and analysis (Golden-Biddle and Locke, 1993).However, CRA differs from many other MA research forms in that a significant indication of research success is a novel and practically relevant idea, which needs to be developed alongside practitioners in the midst of multiple interests.Thus, in CRA, indications of the potential academic relevance also include the innovation of a novel construction and its adoption.
Second, at level B, the first key question is whether what the researcher learned during the CRA process makes an interesting theoretical contribution to recent MA literature.That contribution may lie, for example, in understanding the cognitive features of case actors related to an MA change process, as opposed to the technical feasibility of the construction.
In addition, there are several types and levels of publication forums, with varying focus areas and academic relevance, as indicated by impact factors, journal rankings, and other qualifications 2 used to differentiate journals.The publishing of an article in a scientific can be interpreted as passing a sort of academic market test 3 for research.At level C, the continued citation of a piece of research indicates strong relevance and institutionalized status in the field, and perhaps even in society.Further, generalizability in academic terms suggests the reliability of the results in question.The presented levels of academic relevance analysis may also apply to other IVR or MA case research types.
3 Relevance in CRA projects 3.1.The constructive research approach (CRA) Kasanen et al. (1993) devised the CRA to encourage MA academics to take a more active role in improving existing practice.According to Lukka (2000Lukka ( , 2005)), the ideal CRA outcome is a solution to the original case problem that works in practice, and has both high practical and theoretical value.Therefore, the value of the construct is an important issue but the guidelines for estimating value relevance in the CRA process might benefit from greater elaboration, for example, in the case of proven financial benefits in one organization.
The CRA process currently includes seven steps (Kasanen et al., 1993, Lukka 2000 4 , 2005;   Labro and Tuomela 2003): 1) to find a practically relevant problem that also has research potential; 2) to examine the potential for long-term research co-operation with the target organization; 3) to obtain a general and comprehensive understanding of the topic; 4) to innovate and construct a theoretically grounded solution idea; 5) to implement the solution and test whether it works in practice; 6) to examine the scope of the solution's applicability; and 7) to show the theoretical connections and the research contribution of the solution.Labro and Tuomela (2003) note that steps 3, 4 and 5 primarily serve to ensure internal validity, while step 6 explicitly deals with external validity.Most CRA steps partly overlap with the previous and following steps of the research process.Step 3 -obtaining an understanding of the topic -continues throughout the entire research process.
In CRA, the research topic should be of direct interest to managers or other decision-makers (Labro and Tuomela, 2003).In theoretical terms, the problem should also have significant potential, and there should not be an obvious solution readily available in the literature (Kasanen et al., 1993;Lukka, 2000).Transferring an existing technique from one field to another is considered novel only if it leads to new knowledge (Kaplan, 1998;Labro and Tuomela, 2003).It is typical for the assessment of practical and academic value relevance to occur in steps 5, 6, and 7. Lukka (2000) outlined two primary ways of contributing to theory in the CRA.First, it is possible that the construct itself is a novel idea that creates a new way to achieve organizational goals.Second, a constructive case study can serve the purpose of developing, illustrating, refining, or testing a theory or theories (Keating, 1995;Granlund, 2001;Lukka 2000Lukka , 2005)).With regard to the CRA step 7, the analysis of the different relevance perspectives influencing the adoption of a CRA construct may help demonstrate the theoretical contribution of CRA studies.Our suggested amendments to CRA methodology (e.g., the relevance test with multiple relevance perspectives, see Appendix) are supported by the recent criticism of an overreliance on pragmatist truth theory (Piirainen and Gonzalez, 2013).
The typical CRA focus on the adoption of the construction is used to establish the usefulness of that construction in the organization, or beyond it.Piirainen and Gonzalez (2013), however, point out that the decision to adopt a construction does not actually reflect its qualities, and offer as examples sub-optimal constructions adopted for legitimation purposes alone, and constructions accepted after their design phase but immediately abandoned.These examples support the argument of analyzing constructions according to the different relevance perspectives over time to determine their contribution.Further, different, changing, and possibly contradictory relevance perspectives and stakeholder opinions can influence the design and use, and the perceived relevance, of constructions (see Barth et al., 2001;Järvinen, 2006;Lukka and Suomala, 2014;Nicolai and Seidl, 2010;Piirainen and Gonzalez, 2013;Suomala et al., 2014).

Relevance in CRA studies
The relevance of a CRA construct is typically assessed through market tests.The weak market test is passed when a manager is willing to apply the construct to an actual decisionmaking problem (Kasanen et al., 1993;Labro and Tuomela, 2003).The semi-strong market test requires proof of the use of the construction beyond the case organization; and the strong market test requires proof of financial benefits from the use of the construction in several businesses (Kasanen et al., 1993;Labro and Tuomela, 2003).In the definition offered by Kasanen et al. (1993), even proven financial benefits in one case organization are not adequate to pass the semi-strong or strong market test.However, in our view, if financial benefits to the case organization are indicated (i.e., there is a significant association between construct and benefits), that is a sufficient condition of value relevance and should thus indicate an alternative way of passing the semi-strong market test.Labro and Tuomela (2003) held that CRA relevance is based on the topic being relevant and the construction accepted, and preferably implemented.Lukka (2000Lukka ( , 2005) ) asserted that the weak market test should refer to the actual implementation of the construct, and not only to the willingness to implement it.Lukka and Kasanen (1995) anticipated CRA researchers always assessing the transferability of their construct, at least to some extent.Labro and Tuomela (2003) offered further guidelines for assessing the relevance of the topic: its strategic importance; its effects on the strategic focus or values of the company; and the optimization of processes (leading to positive developments such as increased profitability).
In order to establish the potential for scientific contribution, the topic should be related to earlier research (Labro and Tuomela, 2003).Labro and Tuomela (2003) also suggest that the use of a construction in the whole organization is more important than its use only in sub-units of the organization.However, the use of a construction in an important department (e.g., on the production line) may well suffice to establish strategic importance.On the other hand, if the topic is of low strategic importance, the potential financial impact of the construction may be insignificant; regardless of whether the construction is used across the whole organization or not.In addition, use of the construction might only be partial, that is, only some parts of the construction are adopted.Partial use of a construction may lead to lower relevance than initially expected.The decision to implement is not always in the hands of one or two managers but may be affected by external factors, such as market fluctuations and decisions made at headquarters.Even obsolete systems can be implemented, and the acceptance of a construction is not necessarily a one-off event but can instead be a long and gradual process of institutionalization (Burns and Scapens, 2000;Lukka, 2007;Järvinen, 2006).Thus market tests might be supplemented with the relevance test suggested in this article in order to assess both ex-ante and ex-post CRA project relevance.
The relevance test allows practitioners and academics to evaluate whether a CRA project has potential relevance.The relevance test may be conducted qualitatively (e.g., with a high/low grading, see the Appendix), or estimated in monetary terms (such as cash flows). 5Mitchell   (2002) notes that a cost-benefit analysis is often needed to justify the adoption of a new solution.The use of multiple perspectives on relevance, however, has the advantage of considering the interplay of different perspectives; for example, the need for legitimacy can sometimes outweigh financial considerations.Thus, we amend the classic market tests of the CRA by adding the analysis of the expected (ex-ante) relevance from multiple relevance perspectives.

The relevance test as part of the CRA
The first step in the CRA is to find a practically relevant problem that also has research potential.Even before the development of the construction some indications of practical relevance could be preliminarily assessed, such as whether the problem area has potential for value related improvements (see Labro and Tuomela, 2003).Indications of academic relevance can also be evaluated, such as the plausibility of data that can be collected and analyzed.In steps two and three (see the list below) the access to the organization is obtained and an understanding of the topic is acquired.
After the preliminary considerations, however, the relevance of the problem and the relevance of the construction devised in step 4 may change, according to the prevalent circumstances, an example would be if funding or technology change drastically.In addition, a construction may address only part of the original problem.Therefore, we suggest that the relevance test is perhaps best conducted during step 4; but it might be conducted after each significant change affecting the stakeholders' perceptions of relevance.In very changeable conditions, conducting relevance testing in a somewhat ongoing (e.g., monthly) basis might be considered.The relevance testing can be done in dialogue with the CRA project stakeholder groups.If the construction is found in step 4 to be of low relevance regarding the various relevance perspectives, it can be discarded without further losses.There might also be time for corrective action, such as changing the focus or collecting more plausible data.As there should be an idea of the construction before it can be evaluated, only the indications of potential relevance can be assessed before step 4. Gradually, during steps 5 to 7, the relevance of the construction can be assessed more rigorously, for example with the classic CRA market tests.
1) To find a practically relevant problem that also has research potential; 2) To examine the potential for long-term research co-operation with the target organization; 3) To obtain a general and comprehensive understanding of the topic; 4) To innovate and construct a theoretically grounded solution idea; * including the relevance test, assessing the potential relevance of the solution idea, (see the four relevance perspectives outlined in this article, and visualized in the Appendix); 5) To implement the solution and test whether it works in practice; 6) To examine the scope of the applicability of the solution; 7) To show the theoretical connections and the research contribution of the solution.
Typically, the strong market test can be conducted and the academic generalizability of the construct discerned only after a substantial time (ex-post).A construction passing the stronger forms of market tests can indicate legitimacy, both in practical and academic terms, gradually increasing from case level considerations to societal level legitimacy.Thus, the shape of the Relevance Diamond tool (visualized in the Appendix) can change according to the increased relevance.

CRA case backgrounds
This study uses a CRA dissertation project (Sippola, 2008) to illustrate the different relevance perspectives and the challenges of CRA market testing under changing conditions.Sippola (2008) presented the idea of measuring quality costs in real-time (referred to as the real-time quality cost model or RQC) for a case company that develops and produces packaged software (SW) used in embedded products such as devices for public transport payment in buses, trains, ferries, and subways.The case managers found it difficult to determine the financial consequences of poor quality.As a result, Sippola was approached by the CFO of the case company during a co-operation meeting between his university and local companies.
Subsequently, a quality cost project was started in the case firm.The aim of the project was to improve the company's earning capacity by measuring, controlling, and cutting quality costs.
Sippola worked with the company for nine months (from October 1, 2001 to June 30, 2002), for one or two days each week.Based on 15 interviews, 7 project group meetings, discussions and company reports, he presented a report to the company highlighting the main themes of the interviews (profitability, project costing, quality, software and hardware).The development ideas under the quality theme included quality system development, improvements in documentation, and quality cost measurement.In March 2002, the testing manager of the company presented a new defect correction and monitoring system to replace the old Excel model used to record SW defects.At this point the researcher realized the possibility for constructing a real-time quality cost system by integrating the data from the company's various databases (on SW debugging, SW inspection, SW testing, and hours worked).Next, the testing manager invested a considerable amount of his working time into programming the construct.Otherwise, the early development work on the RQC was very much dependent on the researcher.The company management strongly supported the idea of the measurement of quality costs in real-time.The managing director was even talking about patenting the RQC construct.Here, we note that investing a considerable managerial effort into creating a construction serves as an indication of the relevance for the organization.

Relevance perspectives in the case context
The first step of CRA is to find a practically relevant problem that also has research potential.
In the case company, the existing quality management system was considered inadequate.
I do not benefit from the [current] quality system … quality documents are difficult to find.(Hardware development engineer) In terms of practical value relevance, the project was expected to reduce the number of SW defects, and reduce quality costs, thereby creating both instrumental improvements and value for the company.
In terms of academic relevance, the researcher considered the potential to base his PhD thesis on the case data: the company had a practical problem concerning quality costs and their analysis with significant potential for process improvements.Further, the case offered good access, interesting data (embedded software business was relatively new and rapidly growing at the time), and strong company support.Moreover, a research gap was soon identified: according to Sippola (2008), the previous literature on real-time quality cost accounting measurement was at best very limited, which opened the way for theoretical contribution arising from a novel construction (see Lukka, 2000).Regarding our earlier relevance discussion (in Section 2.3), several indications of academic relevance potential (level A) were present, although their analysis was partly implicit.We suggest that if academic value relevance is considered explicitly as part of CRA methodology it becomes better incorporated into the CRA process and increases the potential for academic contributions.
The case company decided some information should be considered confidential because it provided a competitive advantage; and accordingly a partial non-disclosure agreement was enforced.The agreement did pose some risk in terms of data availability, and even possible conflict between seeking academic relevance with publications and protecting the company's competitive advantage in creating value.An additional risk was posed by product life cycles changing rapidly in the software business and the case company's profitability not being particularly strong, meaning there were no guarantees the project would be completed.
Accordingly, considering academic value relevance, this CRA case was considered risky because project termination or failure to complete the construction would result in the CRA project being interpreted as a failure (see, Humphrey and Lukka, 2011).
Instrumental decision relevance and long-term value relevance do not necessarily appear at the same time.In the case, reducing SW defects was an instrumental improvement but did not necessarily reduce actual expenses significantly, because many fixed costs, such as salaries, remained the same.Moreover, as the excerpt below demonstrates, some quality costs might be attributable to issues like shortcomings in co-operation.
The co-operation between the sales/marketing and the software department has not worked… the most important thing has been to get more sales.(HR manager) In terms of legitimative decision relevance, the case company marketing department was dedicated to increasing sales and vindicating their own function, with little regard for operational concerns.Consequently, several loss-making projects were accepted that reduced the value of the company and created conflicts in the organization.
The sales department sells everything under the sun…The sales department has promised customers properties that can't be delivered with the resources of the software department, especially within the timetables promised by the sales people.(Software manager) Our biggest project is spoiling our reputation and increasing our costs…No one reads the quality reports; they are of no use in the hardware department.(Project manager 1) This suggests that some projects can even induce what might be termed negative legitimacy for the whole organization, even as they enhance the legitimacy of a sub-unit (e.g., the sales department in the case firm).Institutional practices such as sales bonuses may encourage this kind of sub-optimal behavior.This indicates that differing views regarding the perspectives of relevance may induce conflict, not only among stakeholders but also at different organizational levels and between different functions.Accordingly, the CRA project battlefields between different practical and research interests (see Suomala et al., 2014) can be multilayered: different types of conflict may persist between different societal and organizational battlefield levels, that is, among various professional groups and sub-units that change dynamically over time.For example, the basic aim of increasing turnover per se was initially beneficial for most stakeholders and sub-units, but the sales terms were so disadvantageous to the seller that they later jeopardized the profitability of projects and the credibility of the whole organization.
In our case organization, the views of the departments on how to develop the business differed.Quality measurement was perceived as an activity conducted mainly for the purpose of conferring legitimacy, as when the goal was to retain a quality certificate.
The R&D nerds don't understand anything about the finances of the organization; they have complete blind faith in technology.(CFO) In quality issues a major concern has been that the quality certificate stays on the wall.(SW development engineer) A quality certificate may in fact be essential for closing deals with some clients and therefore also value-relevant, which suggests the relevance perspectives are complementary.Further, the different stakeholder groups of the firm considered the general idea of reducing SW bugs a legitimate goal.Finding this kind of common goal may facilitate balancing the interests among the stakeholder groups (cf.Suomala et al., 2014).

Case developments and lessons for the CRA methodology
Using the RQC model facilitates collection of detailed defect statistics, such as the defect correction hours of the project, and comparing these with total project working hours.This supports the monitoring of the stages of defect fixing.The researcher proposed integrating payroll information into the RQC to quantify the monetary effects of the defect fixing process.A link between the payroll database and the RQC construction seemed feasible, but ultimately that link was not created because sudden external effects and market fluctuations affected the implementation of the construct.In the case company, the whole staff of the company was laid off for three months because the board anticipated losses for the period and decided to reduce fixed costs by EUR 500,000.For this reason, the RQC project was not progressed.The subsequent development of the RQC construct would have required financial investment that was adjudged to be non-essential at that time.This represents a point of discontinuity where a shift in relevance perceptions has occurred, that is, scarce resources demanded new prioritization.In addition, the testing manager of the company, who had been devoting most of his time to the CRA project, was transferred to a time-critical customer project and thus was unable to continue supporting the development of the construction.
Subsequently the testing manager left the company due to the economic uncertainty surrounding it.
The potential future financial benefits of the RQC construction were not explicitly estimated in the case organization.This was because of a lack of resources for development projects but also partly because the original CRA methodology did not incorporate such a feature.In some cases, an estimation of case-specific value relevance might be possible, and serve as an indication of the potential to pass the alternative semi-strong market tests suggested in this paper (see Section 3.2).Working within the time frame of the case company development project 6 meant that it was impossible for the construction to pass the traditional semi-strong or strong market tests, and therefore, the construction was subjected to the detailed weak market test framework of Labro and Tuomela (2003) to analyze its progress in terms of the intensity and extent of use (ranging from ad hoc use by one manager to regular use throughout the entire organization).
When considering the intensity of use, however, even regular use is not necessarily institutionalized use, in which the construction becomes taken-for-granted and accepted by most organizational actors (see Lukka, 2007).Moreover, not all the parts of the planned construction are necessarily used to a similar extent.In the case company, the main parts of the RQC construction, such as the time-recording system and SW defect type and defect correction time, were used in an ad hoc manner in the SW development department of the case company.However, one RQC construction sub-part, the Euro-converter (for evaluating the monetary effects of SW defect correction time), was not regularly used.Therefore, the CRA researcher may need to analyze the utilization rate of the construction, meaning those parts of the construction that are in use, preferably in institutionalized use, that is, coupled with organizational goals and rules or is otherwise likely to benefit the company or enhance its legitimacy (see Burns and Scapens, 2000;Lukka, 2007)  7 .
The case illustrates how significant changes in market conditions or in case circumstances, such as key personnel changes, represent possible points of discontinuity for the CRA project and its success.Such changes may cause the relevance of the construction to be perceived differently than was anticipated at its inception; for example, the perceived short-term instrumental decision relevance of the construction might diminish if the case company needs to reduce costs.Such changes might require the relevance test be repeated (see the Appendix).Thus, under changing conditions, the relevance test might be conducted more than once during a CRA project, perhaps after each significant change in circumstances or in stakeholder views.Such reapplication of the relevance test could help understand stakeholder views (see also Suomala et al., 2014).Sippola (2008) tested the transferability of the solution idea in another company (in the software testing industry).However, this organization did not adopt the construction (Sippola, 2008).7 For example, if the construction has two equally important parts, only one of which is adopted, the utilization rate would be 50 %.
The instrumental decision relevance of the RQC construction stems from benefits like savings in SW defect corrections, and making the software development process more disciplined and manageable.Value relevance was expected to follow from better defect management; however, it was not possible to establish proof of long-term value relevance within the time frame of the project.Academically, the RQC model amended the traditional ex-post calculation of quality costs by adding the real-time perspective.With regard to publication potential, a PhD thesis was published, indicating level B in our academic relevance classification in Section 2.3.In light of case evidence, legitimative decision relevance involves multiple stakeholders with potentially conflicting interests (e.g., a department versus the whole organization).

Discussion and conclusions
This article discusses the relevance of IVR research, particularly the relevance of the CRA.
We suggest amendments to the assessment of the relevance of CRA.As far as we know, the ex-ante assessment of relevance and the interplay of the multiple perspectives on relevance have not been considered together in MA literature.In this article, relevance was first divided into value relevance and decision relevance.More specifically we categorized relevance into: 1) practical value relevance; 2) legitimative decision relevance; 3) academic value relevance; and 4) instrumental decision relevance perspectives (see the Appendix).These perspectives are typically complementary: instrumental decision relevance, for example, can be expected to lead to practical value relevance (e.g., Barth et al., 2001).Further, legitimative decision relevance may co-exist with instrumental decision relevance (Hyvönen and Järvinen, 2006), and academic innovations may have practical value (e.g., Kaplan, 1998).However, the relevance perspectives might also be independent, or, in rare cases, conflicting.Our case illustration provided an example of conflicting relevance perspectives: the legitimation of one department can be pursued at the cost of the long-term value of the whole organization.
Our retrospective analysis of the case data (Sippola, 2008) suggests that significant changes in circumstances, such as market conditions, or key personnel changes represent points of discontinuity where the perceived relevance of the construction changes (cf.Suomala et al., 2014).In order to further enhance the relevance of CRA and to mitigate possible conflicts over relevance or research directions, we argue that the CRA method benefits from explicit analysis of the multiple relevance perspectives already during the project (cf.Kasanen et al., 1993;Labro and Tuomela, 2003;Suomala et al., 2014).
First, this article contributes to CRA methodology by introducing the estimation of potential relevance (the relevance test) as part of the CRA; that is, estimating the potential benefits of the construction during the CRA project.The analysis of relevance perspectives and future relevance may start from indications or qualitative perceptions of relevance (a visual tool, the Relevance Diamond, is presented in the Appendix).In some CRA cases, it could be possible to estimate the process improvements or future savings (or other instrumental benefits), based on the new construct, which could lead to both short-term benefits and long-term value.The relevance test may be part of step 4 of the CRA but in longer projects and after significant changes in circumstances or stakeholder interests (points of discontinuity), it may be conducted more than once, even resembling an ongoing evaluation process, accordingly reflected in the shape of the Relevance Diamond.The adoption of the construction into practice is not always a sufficient measure of the theoretical contribution of the CRA (see Piirainen and Gonzalez, 2013).Conducting relevance analysis from multiple perspectives as suggested in this article supports the analysis of the CRA research contribution.The use of the ideas in this paper, such as the Relevance Diamond tool, may facilitate relevanceenhanced CRA by guiding analyses of the research potential and the effects of MA changes.
Second, we contribute by proposing an alternative interpretation of passing the semi-strong market test (cf.Kasanen et al., 1993), based on the proof of financial benefit to the case organization.While we admit the difficulty of proving financial value, case-specific benefits may be easier to show than benefits to multiple organizations.In addition, managerial perceptions of the financial benefits might serve as an indication of the expected benefits of the CRA construction.
The third contribution of this research lies in amending the Labro and Tuomela (2003) CRA weak market test framework by explicitly considering the utilization rate of the construction.This amendment involves assessing which parts of the construction are in actual use.This is because a construction may include several subparts but not all are necessarily used (e.g., the Euro-converter in the case).Moreover, use can be institutionalized differently among the organization's sub-units, thereby creating varying legitimacy perceptions at sub-unit, case company, stakeholder, or society levels (see Burns and Scapens, 2000;Lukka, 2007).
Nonetheless, even organization-wide use of a construction is not necessarily a clear indication of the magnitude of the benefits delivered by that construction.
Fourth, this study contributes to existing knowledge by expanding the battlefield view presented by Suomala et al. (2014) with the battlefield levels concept.We noted the potentially multilayered nature of the relevance perspectives, both in terms of value and legitimacy, including individual, group, sub-unit, company, field and societal levels (battlefield levels), in which practical, theoretical and societal aspects of relevance may be emphasized differently.While legitimating itself, for example, one department or professional group can bring about negative legitimacy for the whole organization, thus creating conflict between the different levels of the organization.There can be changes in the perceived legitimacy or value relevance of the construction, for example, because of sudden external effects and market fluctuations.In our case, however, the legitimacy of the construction as such did not change but the lack of resources led to laid-offs and the development of the construction ceasing.The case highlights that relevance is dependent on stakeholder views, organizational levels, context, and time.This multilayered view contributes to recent discussions on relevance in IVR research by explicitly adding more reflectivity and level of detail into efforts of balancing the different interests in CRA processes (Lukka and Suomala, 2014;Modell, 2014;Suomala et al., 2014;van der Stede, 2012;Westin and Roberts, 2010).Understanding the dynamics of different relevance perspectives during points of discontinuity may support balancing the views of different subunits and resolving potential conflicts such as those between academic and practical views (cf.Jönsson and Lukka, 2005;Suomala et al., 2014).
With regard to the strong CRA market test, our study illustrates how many instrumental improvements in processes are case and time specific.Thus, the instrumental improvements do not necessarily lead to long-term value, especially in other companies.Furthermore, fulfilling the conditions of the semi-strong and strong market tests by Kasanen et al. (1993) is not necessarily in the interests of the CRA case organization; instead, the case organization may be strongly motivated to prevent the wider use of a construction that is considered to provide it with a competitive advantage (e.g., by using patents and non-disclosure agreements).Therefore, the academic interests of disseminating information and efforts of organizational value creation may conflict during the CRA process.This inherent problem of combining the generalizability (exemplified by publications and industry level benefits) sought in academic research and the case-specific interests (competitive advantage, business secrets) is also a potential source of the conflicts found in IVR studies (cf.Suomala et al., 2014).This problem can partly be avoided by interpreting case-specific value as a proof of passing the semi-strong market test.Moreover, the managerial effort invested in creating the construction offers an indication of its expected usefulness.The motives behind managerial effort and the subsequent decision to adopt the construction (e.g., legitimacy and/or expected value) might be discovered through interviews.In addition to benefiting the CRA, the analysis presented in this article could be useful in other types of IVR or case projects.
During steps 1-3 of the CRA project, it is possible to consider the indications of potential (ex-ante) relevance of the CRA project relating to each relevance perspective of the Relevance Diamond.The relevance test is typically conducted in step 4. Later, it is possible to consider whether there are indications, or even proof, of the CRA construction passing the market tests.Obtaining legitimative relevance can be seen as an institutionalization process of practical or theoretical innovation.The legitimation process of a practical solution or the institutionalization of a research contribution typically requires time because their legitimacy increases from the company level to either the scientific field or business field level and gradually to the societal level, often through a complex interplay of rules and routines involving power games and conflicts (Burns and Scapens, 2000;Flyvbjerg, 2001;Lukka, 2007).
The relevance test may orient the CRA researcher more explicitly towards considerations of theoretical contribution, legitimacy and publishing potential (as called for by Lukka and Suomala, 2014).We suggest that the explicit guidelines for assessing the academic relevance of the CRA presented in this paper, such as analyzing the indications of academic relevance during the early steps of the CRA process, increase the potential for the CRA researcher to make theoretical contributions.Those indications of academic relevance include assessing the quality and amount of data as well as identifying a research gap.The relevance test is expected to offer a way to tie the created construct more closely to theory by explicating the different relevance perspectives influencing the creation and adoption of the construct.
Passing the weak market test indicates relevance to the case organization, at least expected instrumental improvements.Later, the proof of passing the stronger forms of market tests typically indicates value and legitimative relevance also to the organizational field and, ultimately, to society (indicating societal relevance, see Lukka and Suomala, 2014).A1 is decision relevance and the vertical dimension (axis) is value relevance.The decision relevance dimension has two ends, or perspectives: one emphasizing the instrumental decision relevance (e.g., short-term processes) and the other emphasizing the legitimative aspect of decision-making.The vertical long-term value relevance dimension also includes two ends or perspectives: a practical (typically case-specific) and an academic (long-term) value perspective.The four points along the dimensions depict the analysis of these relevance perspectives which may be conducted subjectively, or even qualitatively by using, for example, a scale anchored with 'high' and 'low'.In addition, in the case of academic value relevance, a scale starting from the indications of potential relevance (level A in Section 2.3) and then continuing to publication level (level B) and to institutionalization of the research level (level C).Regarding practical value relevance, there may also be potential relevance and then either indications or proof of passing the weak, semi-strong and strong market tests.These market test levels might be used as a scale of practical relevance.The instrumental decision relevance perspective of the Relevance Diamond might sometimes have a quantitative target score, for instance, throughput time in minutes.Considerations (or the scale) of legitimative decision relevance could start from case managers and increase to field level and eventually to societal level relevance and legitimacy.The solid lines in Figure A1 (connecting the four points) form the Relevance Diamond, the shape of which then visualizes the expected relevance of the CRA construction in terms of these four relevance perspectives.

Appendix. The Relevance Diamond and the relevance perspectives
Expressing relevance dynamics over time and noticing changes in the Relevance Diamond shape can help understand case developments, analyze the potential contribution of CRA/IVR/MA case research, and mitigate conflicts among stakeholders.All relevance perspectives could require reassessment after drastic changes, but depending on the case, it might be cost-effective for the researcher to consider only the most substantial changes in the relevance perspectives.

Figure A1 .
Figure A1.Perspectives on relevance and the Relevance Diamond diagram.

Figure
FigureA1reflects the relevance perspectives and a visual tool for relevance testing -the Relevance Diamond.The horizontal dimension (or axis) in FigureA1is decision relevance and the vertical dimension (axis) is value relevance.The decision relevance dimension has two ends, or perspectives: one emphasizing the instrumental decision relevance (e.g., short-term processes) and the other emphasizing the legitimative aspect of decision-making.The vertical long-term value relevance dimension also includes two ends or perspectives: a practical (typically case-specific) and an academic (long-term) value perspective.The four points along the dimensions depict the analysis of these relevance perspectives which may be conducted subjectively, or even qualitatively by using, for example, a scale anchored with 'high' and 'low'.In addition, in the case of academic value relevance, a scale starting from the indications of potential relevance (level A in Section 2.3) and then continuing to publication level (level B) and to institutionalization of the research level (level C).Regarding practical value relevance, there may also be potential relevance and then either indications or proof of passing the weak, semi-strong and strong market tests.These market test levels might be used as a scale of practical relevance.The instrumental decision relevance perspective of the Relevance Diamond might sometimes have a quantitative target score, for instance, throughput time in minutes.Considerations (or the scale) of legitimative decision relevance could start from case managers and increase to field level and eventually to societal level relevance and legitimacy.The solid lines in FigureA1(connecting the four points) form the Relevance Diamond, the shape of which then visualizes the expected relevance of the CRA construction in terms of these four relevance perspectives.

1.
Practical value relevance (financial long-term relevance): the case perspective 3. Academic value relevance (theoretical value and the outsider & learning) perspective