OPEN EDUCATIONAL RESOURCES REPOSITORIES LITERATURE REVIEW – TOWARDS A COMPREHENSIVE QUALITY APPROACHES FRAMEWORK

Today, Open Educational Resources (OER) are commonly stored, used, adapted, remixed and shared within Learning object repositories (LORs) which have recently started expanding their design to support collaborative teaching and learning. As numbers of OER available freely keep on growing, many LORs struggle to find sustainable business models and get the users’ attention. Previous studies have shown that Quality assurance of the LORs is a significant factor when predicting the success of the repository. Within the study, we analysed technology enhanced learning literature systematically regarding LORs’ quality approaches and specific collaborative instruments. This paper’s theoretical contribution is a comprehensive framework of LOR quality approaches (LORQAF) that demonstrates the wide spectrum of possible approaches taken and classifies them. The purpose of this study is to assist LOR developers in designing sustainable quality assurance approaches utilizing full the potential of collaborative quality assurance tools.


Introduction
For the last two decades, a rapidly growing amount of Open Educational Resources (OER) has become available in Learning objects repositories (LORs) for educators to re-use, republish and share within their communities, supporting collaborative learning (Dimitriadis et al., 2009).Smaller OER repositories are built into federated repositories by being harvested for their metadata to improve access to higher numbers of learning objects (Tzikopoulos et al., 2007).Unfortunately, these repositories are not used up to their full potential (Ochoa & Duval, 2009;Mitchell & Lutters, 2006;Dichev & Dicheva, 2012).Thousands of digital resources are created collaboratively and published online every day, and their quality control, assurance and evaluation are of paramount importance for potential users (Downes, 2007;Palavitsinis et al., 2013).OER enable forms of collaborative learning (Dillenbourg, 1999) and LORs of today can be considered as computer supported collaborative learning (CSCL) environments as they provide users tools for posting knowledge productions into a shared working space and providing tools for progressive discourse interaction between the users (Scardamalia & Bereiter, 1994).Adding social and collaborative features has been a recent trend of LORs to facilitate wider user engagement (Monge et al., 2008;Sánchez-Alonso et al., 2011).
According to previous studies (Barton et al., 2003;Attwell, 2005;Clements & Pawlowski, 2012) quality of OER plays a significant role in the success of the open content repositories (LOR) (Tate & Hosek, 2009;Cechinel et al., 2011).Therefore, it is vital to study LORs quality approaches (Clements et al., 2014) in a systematic way.Previous literature reviews on LOR quality approaches have focused on metadata quality only (Palavitsinis et al., 2013) and in the case of Atenas and Havemann (2014) have defined quality approaches quite simply as any approach which might attract users' to re-use content.However, this is the first systematic LOR quality approaches literature review which looks at quality management as a holistic approach around the repository, not only focusing on the quality instruments but also policies, standardization and pre-publication related quality approaches.This literature review puts emphasis towards collaborative tools such as peer review (Neven & Duval, 2002), which contribute towards the quality assurance of the repository.CSCL is an emerging research field that focuses on how collaborative learning, supported by technology, can enhance peer interaction and work in groups, and how collaboration and technology facilitate sharing and distributing knowledge and expertise among community members (Lipponen et al., 2004).
Learning object repositories quality approaches have previously been classified as (Pawlowski & Clements, 2010): 1.The Generic Approach of Quality standards (e.g.ISO 9000 standards (Stracke, 2009), European Foundation for Quality Management Excellence (EFQM, 2014) 2. Specific Quality Approaches (e.g.Content development criteria or competency requirements) (Leacock & Nesbit, 2007) 3. Specific Quality Instruments (e.g.user generated collaborative quality approaches such as rating (Nesbit et al., 2002), peer review (Neven & Duval, 2002) or recommender systems (Manouselis et al., 2014) In this study, we investigated quality approaches for LORs with a systematic literature review (Kitchenhamn, 2004) in order to understand the holistic phenomenon of quality assurance comprehensively and to form a quality approaches framework which LOR developers can take into account when designing new repositories as well as improving the quality of the existing ones.The classification above was used to guide our review process as the starting theoretical framework.This paper is organized as following: In the second section, we describe the main concepts of educational resources and learning object repositories.In the third chapter we define quality approaches around repositories.Chapter four describes the literature review methodology and systematic mapping of quality approaches.Chapter five presents the analysis of the results and the learning object repositories quality assurance framework (LORQAF).The paper concluded with a summary of results clarifying the contributions of this study for theory and practice.Downes (2007) describes Open Educational Resources (OER) as: "In the system implemented by Creative Commons (widely thought to be representative of an "open" license) authors may stipulate that use requires attribution, that it be non-commercial, or that the product be shared under the same license.According to Wiley & Edwards (2002) a learning object is "any digital resource that can be reused to mediate learning."OECD's (2007) definition was: "Open educational resources are digitized materials offered freely and openly for educators, students and self-learners to use and reuse for teaching, learning and research".Very popular definition of OER is by UNESCO (2002) defining OER as "technology-enabled, open provision of educational resources for consultation, use and adaptation by a community of users for non-commercial purposes".Davis & al. (2010) described educational resources as sets of resources, which have been assembled and described with the intention that they could be picked up and re-used by others.Harmonizing the previous definitions, this study defines OER as "All resources for the purpose of learning, education and training which are freely accessible for the user".In the context of this paper, we recognize that educational resources' synonyms from the technology enhanced learning literature include: 'learning objects, digital resources, digital content, digital resources, reusable learning objects, educational objects, educational resources and educational content'.Digital resources can be shared, re-used and collaboratively created across different countries and cultures (Laurillard, 2008).Open educational resources can support collaborative learning particularly well because they have been designed to be enhanced and repurposed and therefore can support cognitive processes behind collaborative learning (Dimitriadis et al., 2009).OER also provide opportunities for long term collaboration and partnerships beyond people's daily context (Pirkkalainen et al., 2014a).OER's significant milestone in its history was MIT's OpenCourse Ware Initiative (Albelson, 2008) where large amount of courses were made freely available.After MIT's example, many institutions have followed the policy of giving out course materials for free -selling the diplomas or graduation certificates.This way OER can work as a marketing tool for the institute's recruitment.OER certainly have been accepted in the community, but face the common problems of the 21 st century: Information is in such large quantities -how to get the teachers' attention towards these materials?In order for OER to be re-used, they have been most commonly gathered into databases that are linked to a user interface portal.This is called a Learning object repository.

Learning Object Repositories
LOR are multi-functional platforms which are designed to facilitate access to reusable learning objects in a variety of formats, so users can search for, find and make use of this content (Downes, 2001).Learning object repositories can also be defined as digital databases that house learning content, applications and tools such as texts, papers, videos, audio recordings, multimedia applications and social networking tools (McGreal, 2011).The purpose of a repository is not simply safe storage and deliver resources, but allow their administration in terms of updating, identifying, utilizing, sharing and re-using them (Retalis, 2005).OER creation also provides potential for teachers and educators for co-creation and collaboration, which are processes that state-of-the-art LORs try to support through social networking features (Okada et al., 2012).Although such LORs using social software for collaborative learning and teaching raise barriers for users in areas like cultural distance and lack of quality (Pirkkalainen et al, 2014b).Some popular examples of LORs include: Le Mill1 , OER Commons2 and KlasCement3 .

McGreal (2008) classifies learning object repositories into three basic types:
1. Centralized model with content stored on the site 2. Portals that mainly store links and metadata to materials provided by others 3. Repositories with equal role as a content provider and portal McGreal's (2008) study has been widely used as it identified the principal functionalities of LORs as: search/browse OER, view OER, download OER, store OER and download OERs metadata.
Another type of classification is based on the nature of the content and content providers: Learning object repositories might contain resources from a certain topic (thematic repository).Many ministries of education have their own nation-wide portals for all topics (National repository).LORs which harvest metadata from other repositories are called 'Federated repositories'.(Clements et al., 2014) General characteristics of well known LORs were studied by Tzikopoulos, Manouselis, and Vuorikari (2009).Their investigation covered features such as educational subject areas covered, metadata, standard used, LOs availability in different languages, quality control, evaluation mechanisms and intellectual property management.This study provided an overview about LORs' current development status and popular features that they incorporate.Ochoa and Duval (2009) provided a comprehensive quantitative analysis of LORs' growth and usage, in which was discouraging to notice that LORs struggle to keep their users coming back to them, specifically if they were built on project funds -many have trouble extending their community after the initial funding ends.
In a recent study, Zervas et al., (2014) analysed 49 major repositories functionalities, but also published the details of common repositories' user and content amounts.Most project built repositories don't seem to reach the masses of users and their user base remains in a few thousand.In Zervas et al.'s analysis, only two repositories reached over a 100 000 users, Merlot4 (118.874users) and Curriki5 (387.189users).By far the biggest amount of learning objects are within the federated repository system of Ariadne6 (830.297LOs).Information was from February 2014.This study also found out that current LORs' implementation adopts mainly functionalities that are related to the basic functionalities of LORs, whereas functionalities related to the added value services (such as social collaboration tools as well as evaluation tools) component are limited.This provided us with evidence that current LORs are mainly developed for facilitating the storage and retrieval of LOs, whereas functionalities for facilitating collaboration between teachers and learners when using LOs available in LORs are rarely supported, even though repositories have for quite some time already been trying to move towards supporting collaborative learning (Monge et al., 2008).Previous studies have observed that LORs oriented towards the generation of content should also consider quality assessment and not just constrained to content and furthermore, being opened to the entire process of collaborative construction of new knowledge (Pérez-Mateo et al., 2011).

Quality approaches for LORs
Quality can mean different things to different people in different contexts (Clements et al., 2014).We should study quality as a phenomenon, which is part of a given community of practice and a specific product (Ochoa & Duval, 2009).ISO 9000 (2014) standard defines quality as the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.Quality can also be defined as " […] appropriately meeting the stakeholders' objectives and needs which is the result of a transparent, participatory negotiation process within an organization."(Pawlowski, 2007).In the context of OER and LORs quality can for example mean that teacher finds a suitable resource for his/her teaching.
LORs typically have significant problems with the quality of the resources (Pawlowski & Zimmermann, 2007;Pérez-Mateo et al., 2011).Previous studies on LOR have highlighted the issue of quality assurance of repositories, as this is seen as key to provision of quality content to end users (Manouselis & Costopoulou, 2006;Petrides et al., 2008;Atenas & Havemann, 2014).In this study we look at the quality of LORs not only from the perspective of the quality of the OER, but from the perspective of the repository and the services around it.

Generic quality approaches
Generic quality approaches provide quality management or quality assurance procedures independent of the domain.When applied to a certain domain (such as Technology Enhanced Learning or OER), these approaches need to be extended and adapted.Generic approaches contain domain-independent quality approaches and can generally lead to trust in certified organizations.If an organization uses for example the EFQM excellence model, it is assured that all products have been assessed and quality controlled.From a user's point of view, this means that the trust in organizations and thus their products can be increased.While the EFQM-Model is used for self-assessment, the ISO 9000 is used to prove organizations by external assessment to earn a seal of approval (Ehlers & Pawlowski, 2004).

Specific quality approaches
Specific quality approaches provide quality assurance procedures for the domain of Technology Enhanced Learning.Specific approaches aim at achieving high quality of certain products, e.g. at OER and related technologies.By applying certain quality approaches, also here a minimum quality is achieved.Specific quality approaches differ in scope and methodology, ranging from quality marks for education (Pawlowski, 2007) to content development criteria (Leacock & Nesbit, 2007) or competency requirements (Ehlers, 2009).
Specific quality approaches also include outsourcing the evaluation of the OER for external experts (Nielsen, 1994).This might be the most reliable way of judging OER quality, however most LORs do not have a sustainable way of hiring experts, which is why they rely on specific quality instruments which bring in quality assurance through crowd sourcing.

Specific quality instruments
Not all problems of quality can be addressed effectively by machine solutions (Barton et al., 2003).Specific quality instruments (Pawlowski & Clements 2010;Atenas & Havemann, 2014;Hylen, 2006) are commonly known as technological features in the repositories, through which the community of users generate their own quality assurance either directly (rating, reviewing, commenting, flagging etc.) or indirectly (The LOR portal can monitor the users' activities and based on that social data, make automatic promotions of content (recommendation systems).Vargo et al., (2003)

Methodology
This study is motivated by the following objective: To perform a systematic literature review on quality approaches and success measuring of learning object repositories.The goal of the analysis was to answer the following research questions: 1. What kind of approaches & instruments do learning object repositories use for managing their quality?
2. How to classify quality approaches for LORs?
3. Which kinds of characteristics do the approaches have?
The literature review for the quality approaches was conducted using the systematic approach by Fink (2005) as method to describe available knowledge for professional practice.The rigorous approach should be systematic with clear methodology, explicit in the procedures, comprehensive in the analysis and reproducible by others (Fink, 2005).The literature review followed the steps defined by Kitchenham ( 2004) for conducting a rigorous analysis.The steps include: (1) Identify need and define the method, (2) create research question(s), (3) conduct the search for relevant literature, (4) assess the quality and appropriateness of the studies, (5) extract data from the studies, (6) conduct data synthesis and finally (7) interpret the results and write a report.
During the literature analysis, we used synonyms of OER and LORs (defined in chapter 2) in order to identify as many studies in the field as possible.This allowed us to have a better overall scope of the approaches as the varied terminology is often used to express the same phenomenon.For all of the key literature, the main entry points were IEEE Xplore bibliographic database, ACM Digital Library as well as Google scholar.A total of 82 papers from 1994 to 2014 were included in the final analysis for the quality approaches in the field of Technology Enhanced Learning.Most papers focus on the recent 5 years of research.
Currently the oldest learning object repositories are starting to be about 17 years old (Zervas et al., 2014), which makes this the correct period to analyze also the studies on them.
The synthesis part of the literature review takes a constructive approach (Crnkovic, 2010).
Constructive research is suitable for construction of a solution (artifact or a theory) that is based on existing knowledge (Crnkovic, 2010).In our case the approach is to build on existing knowledge on quality approaches and to construct an artifact in form of a framework in order to study the quality approaches for LORs.Therefore, the conceptual framework is aimed towards theory building (Nunamaker, Chen, and Purdin, 1990) by contributing to the body of knowledge with a variety of challenges that require validation in real stakeholder contexts.
The constructive part is combined with the approach of Kitchenham ( 2004) by analyzing, synthesizing and interpreting the literature in order to finalize the data analysis and construct the quality assurance framework.

Quality approaches -a critical analysis of current literature
This section describes how quality approaches have been studied in Technology enhanced learning field.As the main result, our study synthesizes the findings by introducing the Learning object repositories quality assurance framework (LORQAF).To better explain the quality assurance process and actors, we synthesized the data in order to classify the identified quality approaches in the Learning object repositories quality assurance framework LORQAF.
This framework will serve as a holistic approach for understanding the overall picture of LORS quality approaches.LORQAF is presented in Figure 1.For the sake of discussing the variety of quality approaches we will harmonize the approaches to categories according to the quality assurance classification of table 1.For example a technology developer can set automated recommender systems to give out featured resources to the users, but the recommendation algorithms are often based on users' actions in the repository portal, which means that the only through a powerful community can the recommender system approach succeed.Many quality approaches are operationalized before publishing the educational resources, however, quality assurance is an on-going process and most repositories offer both technological as well as user-generated collaborative quality instruments.In the current LOR trend, repositories are moving from pre-publication reviews towards post-publication reviews based on a open community reviewers (Atkins et al., 2007).
The following tables present the dimensions of the framework.The selected characteristics include:  Policy -usually set as an overall approach for the repository by the technology developer  Technological -Automated technology & services provided within the portal  Social -Many quality approaches demand collaborative actions from the community making them quality co-creators The following Tables 2-4 describe quality approaches their characteristic and the key references.

Characteristics References
Use  Many comprehensive reviews of quality approaches (such as Atenas &Havemann., 2014 andPalavitsinis et al., 2013) on LOR quality only include quality instruments, which means that the generic approaches and the policy level of the repositories is considerably less researched than specific approaches and instruments.This might also be due to the complexity of standards like ISO 9000 (ISO, 2014).24 papers from the years of 1998 to 2014 were identified to present generic quality approaches such as quality standards or using standardized metadata as the quality approach.
Table 3 TEL Specific quality approaches 20 papers from 1994 to 2012 were identified to tackle the issue or specific quality approaches.

Discussion
Social interaction is considered to be the dominant factor affecting collaboration in groups (Kreijns et al., 2007).Specific quality instruments such as studied in this paper can increase social interaction and collaboration between users.In fact, the most cited quality approaches from the TEL literature seem to have been specific quality instruments: 'peer reviewing' and 'recommendation systems'.This clearly indicates that the future trend of repositories is not only moving towards facilitating collaborative interaction between users as Chatti et al., 2007 predicted but is doing so through quality assurance related specific instruments.
Unfortunately, the mere existence of these features does not guarantee the repository to be successful.Ratings are an easy feature for repository developers to add, but this study does not go deeper into how much the users are actually rating content or commenting on them.
These kinds of features work in the field of e-commerce where strong communities in web shops like eBay can show user ratings from masses and that actually contributes towards the user's perception of the quality of the object.However, in the field of education, users have different level of motivation in using quality assurance features and such repositories in general.Collaborative instruments alone cannot assure quality if there are no communities to back them up.As quantitative studies of LORs by Ochoa & Duval (2009) and Zervas et al. metadata al., 2013;Kurilovas, 2009;Liddy et al., 2002;Stvilia et al., 2002;Strong et al., 1995; (2014) noticed, there are very little repositories with actual user community strong enough to be working like a wikipedia to assure their quality through collaboration.
Based on this literature review, it is our recommendation that the LOR and CSCL environment developers would take a mixed approach for assuring their quality.Expert review might be not the most economical approach, but it seems to be needed in order to evaluate the substance of the resources in the repository.Once the community is strong enough, the user-generated collaborative quality instruments such as peer reviews, comments and rankings can be trusted more to assure the quality of the LORs and CSCL environments.
LOR and CSCL developers should think of the quality assurance as a holistic approach: 1. What are the quality policies of the repository or environment?
2. Which quality assurance instruments can be automated?
3. Quality assurance before or after the resource is published?
4. Ensuring quality by paying external experts to review resources?
5. Which user-generated collaborative quality instruments we want to include?
6. What will be the most cost-efficient and sustainable quality assurance approach in the future?
It is our expectation that the LORQAF can be a useful tool when choosing the quality assurance approach for a learning repository or updating a previous one.

Summary
As the main contribution of this study, we constructed an LOR quality assurance framework (LORQAF) for LOR developers to take into consideration when building future repositories or updating the existing ones.Within the first part of the study, we analysed LOR quality literature within open and technology enhanced learning domains.Our analysis highlighted the state of the art and compiled a comprehensive overview of the most researched quality approaches, instruments and metrics.The framework is foreseen to be applied in qualitative studies that address LOR Quality approaches.It can also be utilized in quantitative approaches, such as incorporating it for studying LOR success eg.using the DeLone & McLean IS success model (DeLone & McLean, 1992;2003).Our findings indicate that the future of LORs quality assurance relies heavily on collaborative instruments, which encourage users to participate in the co-creation of the CSCL environments.

Figure 1 .
Figure 1.Learning object repositories quality assurance framework (LORQAF) During the data extraction phase (Kitchenham, 2004), we found out that quality assurance strategies often are a combination of many choices made by the LOR developer: Which standards to use, which evaluation criteria to select, which tools to give out to the community.
is presented in table 1.

Table 2 :
Generic quality approaches

Table 4
Specific quality instrumentsTotal of 56 papers were identified to propose specific quality instruments for quality assurance of LORs.The timeline of these studies were published in between 1995 and 2014.Most specific quality instruments have two dimensions: The technological and the social, collaborative characteristic.This means that the LOR developers code technological quality assurance features into the portal interface and then expect the users' to interact with the feature to provide evidence of quality for others in the same community.During this literature review, we identified the total of 15 different quality assurance approaches by specific quality instruments.