Evaluation framework for analyzing the applicability of criteria lists for the selection of requirements management tools supporting distributed collaboration and software product line requirements management

Effective requirements management and enabling tools are critical for successfully developing and maintaining services and products. The identification and selection of an appropriate requirements management tool can be a costly, time-consuming, and error-prone undertaking especially in the context of software product line requirements management, requiring the tools to support both product and platform development activities that often involve geographically distributed, collaborating, and competing stakeholders. Criteria lists have been developed to facilitate the selection. This research (1) creates an evaluation framework to review the applicability of the lists for the selection of requirements management tools supporting distributed collaboration and software product line requirements management and (2) leverages the framework to perform the evaluation. The lists are found to provide little support for evaluating the tools from the viewpoints of distributed collaboration and software product line requirements management. Future research is needed to create a more comprehensive list addressing these viewpoints.


Introduction
Requirements engineering (RE) is "the key to improved, on-time, and on-budget delivery of software and systems projects" ( [26], p. xvii).RE is divided into Requirements development (RD) and Requirements management (RM) [39].RD provides a set of reviewed and agreed on requirements.Changes are often unavoidable for reasons such as new business priorities, requirements omissions, or the discovery of new requirements.RM keeps track of changes and ensures requirements are modified in a controlled fashion [24], maintaining the integrity, accuracy and currency of the requirements for the duration of the project [39].
Requirements management becomes increasingly challenging when projects grow in size and complexity [39] and involve software product line (SPL) engineering and distributed collaboration within and across companies [9].SPL engineering leverages domain engineering and application engineering.Domain engineering defines and realizes the common and variable features of the product line by establishing a platform.Application engineering derives applications by exploiting the commonality and binding variable requirements and other artifacts built into the platform.Geographically distributed, collaborating, and competing stakeholders often develop platforms and applications.RM tools are fundamental in such contexts [5,13,24,29,32,34].They [3,7,21,35] range from simple webbased applications to multiuser products with a variety of feature sets and the ability to support large projects [38], making RM more sophisticated and capable [39].
The selection of the most appropriate tools for a specific company or project can be challenging [11].Choosing the wrong tools can hinder the projects [40].Companies should use criteria lists allowing the evaluation of RM tools according to their specific needs.A criteria list allows its user not only to know what types of RM tool functionalities can be expected, but also to improve the selection process by reducing the amount of time required and by making it less error-prone.[11] The construction of a criteria list is a time-consuming and costly activity ( [11], p. 136).Using available lists can save resources.Several lists are available to be used directly or to be taken as references [5,8,16,20,21,31].The lists present different focus areas and levels of detail and are appropriate for diverse situations.Beuche et al. [6] (1) identified the need to incorporate SPL related requirements in current lists to support the selection and appropriation of RM tools during SPL development and (2) recognized the importance of the tool features supporting the collaboration of geographically distributed team members often involved in SPL engineering.
Our preliminary investigation revealed that detailed requirements for the evaluation of SPL RM and distri-buted collaboration features are still missing from the extant criteria lists.Therefore, this paper studies in more depth to what extent the lists support the evaluation of RM tools from the viewpoints of SPL RM and distributed collaboration.The lists are analyzed and compared based on their requirements for the evaluation of the common RM tool features and the features for distributed collaboration and SPL RM.Requirements to evaluate the features are identified.
This research identifies the available lists and analyzes their suitability for companies and teams developing product lines in distributed settings.Two research questions are answered.Do currently available criteria lists for RM tool evaluation allow the assessment of tools supporting distributed collaboration and SPL RM?To what extent do the criteria lists support the evaluation of general RM tool features and SPL RM and distributed collaboration related features?A quantitative analysis of the number of requirements presented by the lists under a unified set of categories is leveraged because it offers better transparency and analysis possibilities than a qualitative analysis of the requirements descriptions.
This research focuses on the lists that have been published entirely and are publically available.It is beyond the scope of this paper to investigate lists containing requirements only for the evaluation of tools supporting requirements engineering activities such as writing, modelling, or eliciting requirements.
This paper proceeds as follows.The research methodology is covered in the next section.The lists are described in Section 3. The framework for evaluating the lists is presented in Section 4. The evaluation is conducted using the framework in Section 5.The paper concludes by stating that future research should create a new comprehensive criteria list to support the evaluation of RM tool features also from the viewpoints of distributed collaboration and SPL RM.

Research methodology
To answer the research questions, a systematic literature review and design science research (DSR) were used.A systematic literature review is ''a means of evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest" ( [23], p. 1).The review compiled evidence, identified gaps in current research, and served as a background for further research (c.f., [23]).The results obtained from the literature review were the input of the DSR."Design science (...) creates and evaluates IT artifacts intended to solve identified organizational problems" ( [19], p. 77).The problem identified in this paper is that the extant lists for RM tool evaluation provide the stakeholders managing SPL requirements and/or leveraging distributed collaboration with limited support.
The literature review is divided into planning the review, conducting the review, and reporting the review [23].During the first phase, the research questions are specified and the review protocol is developed and validated.During the second phase, relevant research is identified, primary studies are selected, their quality is assessed, and data is extracted and synthesized.During the third phase, the review is written and validated.The primary studies of this research deal with the available lists for evaluating RM tools and the features supporting distributed collaboration and SPL RM activities.
The search for the available lists was performed using Google Scholar because the lists should be used not only in academia, but also by the practitioners.The following search expression was used: ("requirements management tools" OR "RM tools") + ("evaluation" OR "selection" OR "assessment" OR "survey") + ("requirements" OR "criteria").
The features supporting distributed collaboration or SPL RM were extracted from peer-reviewed research published in electronic databases ieeeXplore, Science-Direct and SpringerLink.Different combinations of the strings "requirements management", "RM tools", "software product lines", "distributed collaboration", or closely related terms were used.Related articles published before and after the ones obtained from the search were also analyzed [37].
The lists and complementary requirements obtained from the literature review in the fields of SPL RM and distributed collaboration of teams are the basis for creating an evaluation framework and comparing the lists using the framework.
The DSR methodology [28] is followed.It includes six activities: problem identification and motivation, definition of the objectives for a solution, design and development, demonstration, evaluation, and communication.The first two activities were presented in the introduction.The third activity consists of the creation of the artifact, that is, the evaluation framework to compare available lists.The framework is presented in Section 4. The fourth activity demonstrates the use of the artifact by applying it to evaluate and compare lists.The fifth activity consists of observing and measuring how well the artifact solves the stated problem [28].This evaluation is conducted through a controlled experiment [19], testing the usability of the framework and its ability to compare the lists.The results obtained from the application of the framework are presented in Section 5.This paper represents the final activity communicating the resulting knowledge [28].

Criteria lists for evaluating Requirements Management tools
This section overviews the six lists found by the literature review (Table 1).The following six subsections present the lists in their order of creation.

List 1: INCOSE
The INCOSE Tools Database [21] has been continually updated since the late 1990's.It is composed based on a survey of different RM tool vendors.This database allows a direct comparison of tools against the defined set of features covered by the questions of the survey.The survey does not assign priorities to the questions.This survey has been used as a reference for the creation of other lists for RM tool selection [4,18,20,34].

List 2: DSTO
The goal of this list is to help the Australian Defence Organisation (ADO) to assess possible tools to be adopted [16].It identifies most relevant user needs for ADO and provides a hierarchical list of requirements for RM tools derived from the user needs.Requirements are prioritized as follows: critical, important, relevant or useful.There are 11 metacategories in this list (e.g., Requirements data, Management of requirements) divided into 35 categories (e.g., Requirements text, Change control) [16].This list does not present requirements similar to List 1 [21].

List 3: ITEA DESS
The ITEA DESS RM Tool Requirements [8] describes a list of requirements for RM tools and compares four commercial RM tools against 24 criteria.The list includes 78 requirements divided into 18 main categories (e.g., Traceability, Change control) [8].The presentation of the list is not uniform.Some (sub)categories have numbered lists of requirements, while others present bulleted lists, paragraphs, or tables of requirements.The list references List 1 [21] and includes some identical requirements.

List 4: DaimlerChrysler
The list is based on project experiences in automotive, aircraft, and defence system domains [20].
It neither references other lists nor includes requirements identical to the ones included in the five other lists.The list follows a hierarchical organization with three levels: stakeholder, category, and RM tool requirements.The requirements and the categories are prioritized.The list contains 101 requirements divided into 21 categories (e.g., Views, Tool Integration).Categories are grouped under three stakeholders: developers, project administrators, and tool administrators.A tabular evaluation template is available online for the evaluation of RM tools.

List 5: RWTH Aachen
The list of requirements for RM Tools in the context of SPL RM [6] was developed based on List 4 [20] and validated by using it to evaluate four tools for SPL RM.The new requirements were derived from real tool usage.The additions and changes were not extensive [5].Three categories were added, namely Configuration management, Collaborative work and Priorities, containing existing and/or new requirements.The new requirements were prioritized using a scheme of points and priorities, showing the importance of the requirements in relation to SPL RM [5].The list has 110 requirements divided into 24 categories.

List 6: Seilevel
The RM Tools Evaluation Worksheet developed by Seilevel [31] presents the most extensive list of requirements in terms of the number and level of detail of the requirements.The list can be used for evaluating RM tools.It draws upon List 1 and List 2. It contains a prioritized list of 50 use cases of a RM tool, a prioritized list of 233 requirements classified into 23 categories (e.g., Requirements architecture, Baselines), a list of features grouped by the use cases, and a comprehensive list of RM tools.

Other criteria lists
A variety of other lists exists (e.g., [11,18,22]).However, due to their partial availability or different scope, they were excluded from the analysis and comparison.These lists are leveraged in the next section for defining the criteria to compare the content and attributes of the six lists from the viewpoints of geographically distributed collaborative RM [25,29,32] and SPL RM [14,15,27,30,33,36].

Evaluation framework for analyzing and comparing criteria lists
This section presents the evaluation framework to analyze and compare the lists presented in the previous section from two viewpoints: the attributes and the contents of the lists.Two different sets of evaluation criteria were defined.The first subsection presents the attributes for evaluating the organization and presentation of the lists.The second subsection presents (1) the process followed to identify the categories for evaluating the contents of the lists, (2) the categories, and (3) the scoring system for grading the content coverage level of the lists.

Attributes
To analyze and compare the lists, a set of nine distinctive attributes (Table 2) have been identified based on the examination of the six lists covered in the previous section.The goal of this set is to represent the main characteristics of the lists in terms of the presentation, organization, and structuring of the requirements, the system used to identify the requirements, the style used in the requirements description, and the means used to clarify or support the requirements.

Categories
To analyze and compare the content of the lists, a set of requirements categories was created based on the analysis of RM related concepts in the literature, the categories identified in the six lists, and research conducted in the fields of SPL RM and the collaboration of distributed teams.
The category identification process is depicted in Figure 1.Names for each category had to be systematically defined because the lists and/or the literature use different names for equivalent categories.The categories were also grouped into meta-categories, including sets of related categories.
The category identification process produced a set of 36 categories grouped into six meta-categories.The meta-categories were divided into two meta-categories: general content and content for the evaluation of features supporting distributed collaboration and SPL RM.  3-7 present each general content category identified from the currently available lists and the literature.The meta-category "out of scope requirements" covers three categories of features that do not deal with the main or support activities of the RM process or general support features.

Category name Description
Views Criteria evaluating the visualization of the data stored in the tool, the ability of users to adapt these views to their specific needs, and the capability of filtering and sorting the information displayed in the views Navigation Criteria evaluating the user's ability to view and navigate the requirements stored in the tool, modify the position of requirements in the hierarchy in a simple way, perform searches, and add bookmarks Analysis functions Criteria evaluating the ability of the tool in terms of the analysis of the stored data, mainly regarding the status of the project and the consistency of the relations between requirements Import Criteria evaluating the import of requirements into the tool different types of existing documents, as well as the ability to parse and identify requirements from text documents Export Criteria evaluating the export of requirements into different standard formats Specification generation Criteria evaluating the generation of requirement specifications, as well as the ability of the user to define the format and the content of the final generated document Reporting Criteria evaluating the ability of the tool to generate different types of reports based on the information stored in the tool Users and access control Criteria evaluating the administration of users, user groups, user roles, and their permissions to perform different tasks using the tool Offline use Criteria evaluating the ability of the user to view and edit requirements in a disconnected mode, as well as the synchronization of modified data on reconnection Usability Criteria evaluating features that simplify the usage of the tool Table 5 Requirements data management

Requirements architecture
Criteria evaluating the definition of the structure of single requirements and their attributes, the organization of requirements into groups or hierarchies, and the definition of templates and glossaries Requirements capture Criteria evaluating the creation of requirements directly in the RM tool Requirements edition and deletion Criteria evaluating the modification of specific attributes of single requirements or sets of requirements, as well as the removal of requirements from the tool Requirements quality Criteria evaluating the quality (e.g., spelling and grammar) of requirements in the tool as well as the inclusion of other tools to improve the quality of requirements Requirements enrichment Criteria evaluating the formatting of the requirements text, such as the inclusion of symbols, mathematical expressions, or foreign characters; as well as the association of non-text objects, like videos or images, with the requirements descriptions Requirements issue tracking Criteria evaluating the creation of issues related to requirements, the association of the issues with requirements, and the tracking of the issue status.8 and 9 present the categories for the evaluation of features supporting distributed collaboration and SPL RM.
The category "Informal communication" is present in most of the six lists and an important feature to be included in RM tools [32].The category "Awareness" was not included in the lists but RM tools should be able to provide information to maintain the awareness of distributed team members [12,32].The category "Workflow management" was included in two of the lists [20,31].It supports the coordination of work between members of the team and helps to implement RE processes and guide the involved users [17,20].
The categories in Table 9 were not included in the six lists.They were obtained from research describing tool requirements and features supporting SPL RM [5,14,15,27,29,30,33,36].

Scoring system for assessing the category content coverage level.
To rate the level of detail per category quantitatively for each list and to compare the coverage levels of the lists easily, scores are assigned according to the number of requirements the lists contain for each previously identified category.By replacing the number of requirements each list contains per category with a range value, it becomes easier to identify lists including no requirements, some requirements, or numerous requirements for a specific category.
Similar scoring strategies have been applied to evaluate RE and RM tools [1,2,10].However, in these studies, the scores are assigned based on qualitative assessments.Moreover, these studies evaluate tools instead of criteria lists.The scoring scale is presented in Table 10.

SPL requirements management
Criteria evaluating features for (1) the explicit allocation of requirements to the SPL platform or to an individual product, indicating whether the requirement is shared by all members of the SPL or is a special product requirement, as well as (2) the visualization of the SPL information and (3) the possibility to adapt requirements based on user needs Domain requirements representation Criteria evaluating (1) the representation of the defined scope through tables, models, or trees, and (2) the inclusion of mechanisms to control the integrity of domain models by detecting redundancies, anomalies and inconsistencies in the models Product requirements instantiation Criteria evaluating the derivation of requirements for single products from the SPL infrastructure, the analysis of the consistency of the created configurations of product requirements, and the possibility to modify the configurations in the future

Using the evaluation framework to analyze and compare the criteria lists
This section presents the results from the analysis and comparison of the six lists presented in Section 3 based on the framework defined in Section 4.

Comparison of the lists based on presentation and organization attributes
The comparison of the lists based on the defined attributes is summarized in Table 11.Requirements in List 3 are presented using a mixed style, diminishing its clarity and making the document less suitable for direct tool evaluation.This mixed style consists of requirements presented as bulleted lists, numbered lists, a table, and in the form of blocks of text, making the identification of single requirements difficult.In four lists, the requirements have a fixed position in the defined structure, while in the other two the position of the requirements varies from category to category.For example, some requirements of List 2 are at level two under a meta-category, while others are at level four under a subcategory.The requirements in the analyzed lists are described using statements and/or questions.For example, List 2 presents mainly questions.Differences can also be found regarding the level of detail of the requirements in the lists.List 2 and List 6 present requirements with the highest level of detail, specifying the exact RM tool features to be evaluated.In terms of the prioritization of requirements, List 5 inherits the priorities used by List 4 and adds a product line related priority to all requirements.Regarding the levels of priorities used, List 2 applies four levels.List 1 and List 3 assign no priorities.

Analysis and comparison of the lists' contents
This section analyzes and compares the lists in terms of their general content and content related to features supporting distributed collaboration and SPL RM.A series of steps was followed to analyze and compare the lists.The requirements in each list were analyzed, classified, and grouped into the categories identified in Section 4. Some requirements were allocated to more than one category due to their broad scope.Once the requirements of all lists had been classified into categories, a score was assigned per category to each list.

General content.
This section analyzes and compares the lists based on their coverage of the general content meta-categories: "RM main activities and baselining", "Information management", "Requirements data management", "Technical specification, licensing and support" and "Out of scope requirements".
The results of the comparison of the lists based on their coverage of RM main activities and baselining are summarized in Table 12.List 6 is the best because it can be used to evaluate features supporting all the activities in the highest level of detail.
The information management meta-category includes categories related to the input and output of data to and from the tool, different ways to access the data, and the control of users and their rights to access and modify the stored information.The results of the comparison of the lists based on their coverage of these categories of requirements are summarized in Table 13.List 6 is the most comprehensive list, including requirements for all categories.The categories Views and Specification generation have the highest number of requirements, showing the importance of visualizing the data stored in the tool and generating specification documents.
The data management meta-category is used to evaluate features supporting RM.The comparison results are summarized in Table 14.List 6 is the most comprehensive list, including requirements for all six categories.
For the meta-category "Technical specification, licensing and support", five categories of requirements were identified from the analyzed lists.The comparison results are summarized in Table 2. Lists 1 and 6 are the most comprehensive lists, including requirements for all categories.This section compares the lists based on their coverage of the meta-categories "Distributed collaboration" and "SPL requirements".For the meta-category "Distributed collaboration", three categories of requirements were identified mainly based on the literature review.The comparison results are summarized in Table 4. Lists 4, 5 and 6 are the most comprehensive ones, containing requirements for two categories.List 6 presents the highest number of requirements.The SPL requirements meta-category covers categories of requirements allowing the evaluation of features that support SPL RM.The comparison results are summarized in Table 58.List 5 is the most comprehensive, including requirements for all three identified categories.However, the number of included requirements is small.

Summary
In terms of the attributes, most lists present requirements as a table, use a three level structure to organize the requirements, use codes to identify the requirements, formulate requirements as statements using an intermediate level of detail, and prioritize requirements using three levels of priorities.
In terms of the content of the lists, differences prevail regarding the percentage of categories covered by the lists and the level of coverage offered by the lists per category.Figure 2 shows the percentage of categories covered by the lists.To simplify the analysis, the meta-categories "Information management", "Requirements data management", and "Technical specification, licensing and support" are grouped under Support categories.
No list fully covers all the identified category groups.List 4 covers requirements for all in-scope categories, but the coverage is marginal for the "RM activities and baselining" and "Distributed collaboration" meta-categories as well as support categories.List 6 fully covers the "RM activities and baselining" meta-category and the support categories, but offers only a 67% coverage of the "Distributed collaboration" category and no coverage of the "SPL requirements" meta-category.
The number of pluses and minuses assigned to the lists is presented in Table 19.List 6 presents the biggest number of pluses and the smallest number of minuses under general content categories.List 5 presents the best results for distributed collaboration and SPL requirements categories.

Figure 2 Percentage of covered categories per list
Figure 3 depicts the level of coverage provided by the lists for the categories based on the percentage of "+++", "++", "+" and "-" scores assigned to all categories of each list, except the categories considered out of scope.List 6 provides the highest level of detail because it covers the largest number of requirements.It obtained the "+++" score for 42% of the categories and also the highest number of "++" scores.The analysis and comparison of the lists are expected to provide practitioners with useful information about the attributes and content of these lists.The classification of the lists' requirements into a unified set of categories allows the users of the lists to identify the sets of features that can best be evaluated by the different lists.The comparison of the lists has not been available in the industry or academia until now.The comparison results help practitioners to deploy the most appropriate list(s) for selecting RM tools.
The lists originate from different fields and respond to different needs.They exhibit similarities in some areas and differences in others and have their strengths and weaknesses.For the distributed collaboration and SPL RM content categories, a small number of requirements was found.As a result, future research is needed to construct and validate a new list covering these categories.List 6, published by Seilevel [31], will offer the best baseline for creating the new list.The literature reviewed by this paper in the fields of distributed collaboration and SPL requirements will complement the baseline.

Figure 1
Figure 1 Category identification process 4.2.1.General content.Tables3-7present each general content category identified from the currently available lists and the literature.The meta-category "out of scope requirements" covers three categories of features that do not deal with the main or support activities of the RM process or general support features.
tool's ability to version single requirements, sets of requirements, and other objects to allow the comparison of current versions with older versions and the restoration of older versions.Change control Criteria evaluating the management of formal changes to the requirements, formal commenting or discussion capabilities, and the documentation and visualization of changes Status tracking Criteria evaluating the tracking of requirements status as well as the definition of customized status values for specific projects Tracing Criteria evaluating the linking of objects having the same or different type as well as objects stored in the tool or outside the tool Baselining Criteria evaluating the tool's ability to create, compare, and manage baselines No requirements, (+) between 1 and 2 requirements, (++) between 3 and 5 requirements, (+++) 6 or more requirements No requirements, (+) between 1 and 2 requirements, (++) between 3 and 5 requirements, (+++) 6 or more requirements

Figure 3
Figure 3 Percentage of coverage level per list

Table 2
Attributes for analysing and comparing lists

Table 4
Information management

Table 6
Technical specification, licensing and support

Table 7
Out of scope requirements

Table 8
Distributed collaboration

Table 1
Category content coverage scoring values ++ Between three and five requirements identified for the category +++ Six or more requirements identified for the category

Table 12
Comparison of RM main activities and baselining

Table 13
Comparison of information management

Table 14
Comparison of requirements data management

Table 25
No requirements, (+) between 1 and 2 requirements, (++) between 3 and 5 requirements, (+++) 6 or more requirements Requirements for the evaluation of features beyond the RM scope were identified and grouped under the categories Testing, Modelling and Tender evaluation support.The comparison results are summarized in Table36.List 6 includes requirements for all three categories.The requirements for two categories are detailed.

Table 36
Comparison of out of scope requirements 5.2.1.Distributed collaboration and SPL RM content.

Table 19
Number of pluses and minuses assigned under in-scope categories