Project Document Cover Sheet
Project Acronym: Version: Contact: Date: Project Document Cover Sheet Project Information Project Acronym DIVAS Project Title Developing Intelligent Validation Systems Start Date 12 May 2008 Lead Institution Staffordshire University Project Director Dr Jenny Yorke Project Manager & contact details Sue Lee, LDI, Thompson Library, College Rd, Stoke-on-Trent, ST4 2DE. Partner Institutions n/a Project Web URL http://divasproject.blogspot.com/ Programme Name (and number) (JISC0108) Programme Manager Rob Bristow End Date 30 April 2009 Institutional Responses to Emerging Technologies Document Name Document Title Final Report Reporting Period Author(s) & project role Sue Lee – Project Manager Date July 09 URL if document is posted on project web site Filename Access DIVASFinalReport.docx General dissemination Document History Version 1.0 Date July 2009 Page 1 of 34 Document title: JISC Final Report Last updated: April 2007 Comments Final Version Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 JISC Final Report DIVAS Developing Intelligent Validation Systems Sue Lee, July 09 Page 2 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Table of Contents .................................................................................................................................... 1 JISC Final Report................................................................................................................................... 2 Table of Contents ........................................................................................................................... 3 Executive Summary........................................................................................................................ 4 Background .................................................................................................................................... 4 Aims and Objectives....................................................................................................................... 5 Methodologies ................................................................................................................................ 6 The Community of Practice ............................................................................................................ 6 Describing Validation Documents................................................................................................... 7 Implementation ............................................................................................................................. 14 Implementing a Community of Practice:....................................................................................... 14 Configuring the XCRI-CAP Metadata Schema............................................................................. 14 Outputs and Results ..................................................................................................................... 23 The development and pilot of the Community-of-Practice approach and integration of social networking technologies: .............................................................................................................. 23 The development and pilot of the planned fast-track approach:................................................... 26 Contributions to other JISC projects:............................................................................................ 27 Conclusions .................................................................................................................................. 29 Implications................................................................................................................................... 30 References ................................................................................................................................... 31 Appendix A – Validation Support Network Evaluation Questionnaire .......................................... 32 Acknowledgements This project was conducted under the JISC: JOS e-Administration - Institutional Responses to Emerging Technologies The author would particularly like to acknowledge the contribution, help and support of the following in writing this report; Sue Kneill-Boxley, Ben Scoble, Sam Rowley and Hamza Badenjiki Page 3 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Executive Summary The project had a number of aims: • Speeding up validations by using the output of previous validations. • Using new technologies to link to administrative systems so they can be accessed easily • Reduce the administrative burdens associated with the validation of courses • Make the validation and processes more open • Create a Community of Practice (COP) for validation support We feel we have met the last 2 of the aims fully, and are on the way to meeting the other 3. We have opened up the validation processes in a way they have not been before. The overall approach was to try and get an accurate picture of the processes of validation as possible. We canvassed for experiences, opinions and views from with as many individuals in as many different roles as possible. Though the project was not as successful in meeting our aims as we would have liked we found an enormous amount of qualitative information that we have been able to forward to our colleagues in Quality Services. We have used the expertise gained in running a Community of Practice (COP), to help run a University COP and we now run a series of workshops for academic colleagues on setting up and sustaining a COP. We found that support was needed in different ways to what we envisaged, for example, support was needed for new panel members as well as presenting teams. We found that Staffordshire University Regional Federation (SURF) colleges www.surf.ac.uk had different support issues than the university based people and did not really feel a sense of community with the university – this was outside the remit of this project but useful information. Background Every course validation event held at the university produces a validation report. These reports contain recommendations, conditions and commendations that are fed back to the presenting team so that quality of university awards can be maintained. This documentation is available to university employees though is not easily accessible. This project aims to make these reports easily accessible and allow others about to go through the validation process to learn from their predecessors. The need for university course validations to enable the rapid development of new courses and modes of delivery in response to learner, employer and market demand is becoming a significant issue. In general, validation processes are made sluggish by the enormous administrative burdens they can impose and by the fact that the outputs of each validation are not effectively exploited to inform subsequent validations. Page 4 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 This proposal focuses on addressing the need for institutional administrative systems to facilitate the sustainable take-up of emerging technologies and provide flexible access for their users to valuable functions and resources. It specifically addresses the use of emerging technologies to develop validation processes which are fast and responsive, yet robust, but which are also intelligent, in the sense that they are informed by their own outputs, open and enabling, allowing colleagues across institutions to learn from and exploit the valuable information and decisions they make, to improve agility in course development and the take-up of technology. Aims and Objectives The project aimed to address the range of outcomes sought by the call and develop and deliver: • A 9-month pilot of the proposed Community-of-Practice-based self-service access to validation resources, which will expand on the success and expertise of Staffordshire University’s ‘eLearning Models Project’ in building a vibrant community-of-practice in Technology Supported Learning, to provide The integration of emerging social networking technology – possibly Ning (which the ‘eLearning Models project already uses successfully) – with the University’s learning repository (HIVE) to facilitate access to the resources by the Community-of-Practice. At the beginning of the project there was a change of staffing which led to the start of the project being delayed. Because of this and the fact that the project had a short life span the decision was made that we should not do a full evaluation of the available tools but (as we had already started investigating it) to use NING as the social networking tool. The development and facilitation of this Community-of-Practice, comprising academic practitioners, learning development staff and core quality administration staff A 6-month sub-pilot of the proposed validation ‘fast-track’ process and self-service access to process outputs, i.e. to ‘models of good practice’ distilled from, or informed, by validation outputs; and other important validation decisions. • Expand on the work of the COVARM project to develop and contribute a validation Service Genre encompassing the fast-track and self-service process to the e-Framework for Education and Research. This objective changed once the current validation process at the University had been analysed. The COVARM outputs were compared to current University validation practices and were found to be essentially the same. Little value was to be gained in rehashing the models created by COVARM. For this reason, service-oriented effort of the project focussed on development of a software application to streamline the submission of validation documentation. The application provides an online mechanism for submission of validation documents along with their metadata and uses a Web services API to automatically deposit the documents in our Harvest Road Hive learning content repository. Page 5 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 • Provide a regularly updated blog and/or wiki reporting on project progress and lessons learned. • A report highlighting the challenges, implications and benefits of developing and implementing the validation fast-track and Community-of-Practice -based self-service processes by: Addressing the challenges and ‘how to’s’ of ‘distilling’ validation outputs into accessible ‘models of good practice’ useful to academic practitioners, learning technologists and staff developers; Expanding on the work already published by Staffordshire University’s ‘eLearning Models Project’ by reporting on this unique Community-of-Practice approach and the challenges and benefits of using a social networking tool to facilitate the diverse community highlighted and their self-service access to validation resources; Addressing the issues associated with the ‘structuring’ the validation outputs in the University’s learning repository to ensure their effective discovery, sharing and reuse. Exposing and exploring the issue of the appropriate institutional locus of decisionmaking on the use of emerging technologies to support learning and teaching; including the institutional knowledge and skills, roles and responsibilities required to make effective decisions; Exposing and exploring the issues of cultural change associated with opening-up administrative systems for ‘public’ access Methodologies The Community of Practice To set up the Community of Practice, “The Validation Support Network” (VSN), invitations were sent to faculty Deans, coordinators e.g. Foundation Degrees coordinator, PAMS, Award Leaders, academics and administrative staff. The invitation message was also posted on a shared All-Staff email folder. The message included a brief description of what the Community of Practice is about and an attached document detailing how to get started as a member. Individual Welcome messages were put into the Comment section of each new members personal page and we also had three forums set up for new members to get started right away: Introduce Yourself to the Community; Tell us about your Validation Experiences; What Are Your Expectations of the Community. The approach taken to sustaining the Community of Practice involved organising regular online forums and events and keeping members informed of both these and current member activity through a weekly bulletin. The bulletin went out in the form of an email every Friday lunch time. In addition to going to members it also went to the shared All-Staff email directory to generate interest from University staff not yet joined. Page 6 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 A new forum was opened every fortnight covering topics such as Validation Administration, Writing a Teaching & Learning Strategy and Employer Engagement. Two online events were run – an Ask The Expert event concerning the validation process with a member of the University Quality Improvement Service (QIS), available on line for two hours to answer community members’ queries and Validating Technology Supported Learning event whereby three academic practitioners provided audio and slide/audio presentations regarding their experiences and were available during a fixed two hour time slot to answer questions. Face-to-face sessions were offered to introduce staff to the COP and answer any queries but no members attended. The bulletin was created in email due to restrictions on the number of characters that could be used for a Ning message and also due to limitations concerning design and layout. The Ning ‘Broadcast Message’ feature is more suited to short announcements – it was decided, however, that interim announcements would not be made due to possibility of ‘email overload’ and ‘random’ messages being ignored. Describing Validation Documents The validation documents produced during a Quality Assurance (QA) validation process are in a various formats, primarily word processor (.doc) format, portable document format (.pdf) or in physical paper format and held in the QA office. These documents are stored in a ‘silo’ server or in physical files and folders only accessible by QA staff or on request, within the QA office. The digital formats were organised and stored in a folder structure, with the top-level folder describing the course (completed with the date of the validation event). Within the folder, sub-folders contain pre-validation documents, for example, Module Descriptors or Programme Specifications. Main Folder (Title + Validation Event Date) Sub-folders and documents Page 7 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Individual Documents Image 1: Individual Validation Folder Structure and Documents Moving documents into a repository system Essentially, these current folder structures are ad-hoc, and can contain simply un-aggregated documents with no sub-folder structure or arranged in a more coherent way. In order to make these documents more accessible to users, it was a requirement to move these documents and upload them into a central web-based repository system (HIVE Harvest Road / Giunti Labs). This would allow greater access to the documents and would also allow them to be re-purposed (linked to) for different contexts and systems, for example a social networking site. This approach would therefore require using a structured methodology in order to describe the documents’ role in the validation process and the relationship between them. What solution did you identify? The collection of validation documents are different in each case, however there are several main ‘types’ of documents that are required for the validation process, these can be ‘graded’ into the following 3 categories (see table 1.1) : Validation Process Documents Pre-validation e.g. Validation e.g. Post-validation e.g. Documents that are Documents that are Documents that are presented to the validation produced as a result of the produced as a response to panel. validation and are associated the validation event with the validation event. recommendations. Programme Specification Validation Report Amended Programme Specification Page 8 of 34 Online Guide for staff Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Table 1: Categories of Validation Documents In addition, what is also common to each validation instance is that each validation did have a ‘Programme Specification’ document at the core. This document describes specific aspects of the course, in addition to supporting information about, for example; • Aims • Objectives • Teaching and Learning Outcomes • Assessment Strategies • Programme Structure • QAA Benchmarks • Support Structures • Date Essentially, it was evident that we needed to record specific details about the documents in order for them to be ‘useful’ to a user, insofar as they could be interrogated as part of a resource ‘discovery’ process. This could be achieved through using a metadata schema. Using a Metadata Schema to ‘describe’ documents held in a repository Within HIVE, many different metadata schemas are used to record specific information about an uploaded object (e.g. pdf, or doc). A number of ‘Metadata Schemas’ were investigated — these were already used by the Hive Harvest Road Repository system and are considered international standards: Dublin Core (DC) — Information Resource Metadata Schema (see image 1.5) IMS LOM — Learning Object Metadata Schema (see image 1.6) Page 9 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Image 2: Dublin Core Metadata Schema Elements as viewed in HIVE Page 10 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Image 3: IMS 1.3 LOM Metadata Schema Elements Source = http://upload.wikimedia.org/wikipedia/commons/1/14/LOM_base_schema.png Page 11 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 What issues were there in using already pre-defined metadata schemas? From talking to the QA department, it was clear that there were some key pieces of data that could be recorded by a metadata schema, these are: 1. The ‘Programme Specification’ document elements, as it holds some key information / data that described the course 2. The types of documents used in the validation process, for example, identifying them as being either produced pre-validation, during the validation or as a response to the validation event. There were some specific issues about matching the ‘required’ data held in the ‘Programme Specification’ document and the types of data recorded against each of the Metadata elements. For example, some ‘Programme Specification’ fields matched (explicitly) certain metadata elements very well, where-as some other did not. A ‘work around’ was looked at, but would not be an elegant solution and could cause problems when interrogating the repository. Essentially, the LOM and DC schemas describe ‘Learning Items’, however, we were not asking the system to ‘record’ this type of metadata for our validation document objects. Our validation documents are not traditional learning items and the LOM and DC schemas are not flexible enough to ‘describe’ the types of documents that were produced. Which metadata schema was identified as being the most useful? The decision was made to use XCRI-CAP, as this describes courses (see image 1.8) and not learning objects and most of the ‘Programme Specification’ fields / categories matched XCRI-CAP elements very well. In addition, as we are using the HIVE repository system, we could use the inherent folder (category) structure it uses in order to create folders that organise, or even reflect, the types of documents that are produced during the validation process. Decisions about how the documents are organised and uploaded to HIVE will be addressed during the implementation stage, as we will have to investigate XCRI-Cap and HIVE integration further. In addition, unfortunately HIVE does not have the XCRI-CAP metadata scheme as part of its metadata schema list, so we would need to create the schema and implement a version (an instance) for HIVE. Page 12 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Image 4: Folder / Category structure in Hive Image 5: Diagram of how XCRI-CAP describes courses Page 13 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Implementation The project has a regularly updated blog http://divasproject.blogspot.com/ outlining the project progress. The project also has 2 other blogs. One to detail the technical aspects: http://staffordshireuniversitydivas.blogspot.com/ and http://divascommunityofpractice.blogspot.com/ describing the issues involved in setting up a COP. Implementing a Community of Practice: Contact information, for those likely to have a role in validation, was gathered from various sources, such as email directory and prospectus with some key members of faculty, e.g. PAMs, being contacted directly to obtain lists of Award Leaders. These were then grouped into a Microsoft Outlook group to receive the invitation message to join the Validation Network COP. We also sent out a weekly bulletin by email, reviewing activity that had taken place in the community and also including reports on new membership, forums and planned events. Forums and events were planned through meetings between Chris Gray, Hamza Badenjiki and Sue Kneill-Boxley. Forums were hosted by different members of the team depending on the subject matter, e.g. Validation Administration hosted by Chris Grey of QIS and Innovative Assessment hosted by Hamza Badenjiki. Events involved staff members we knew to have worked in the subject area, e.g. practitioners who had employed Technology Supported Learning through the use of podcasts, and also had staff in attendance to deal with technical problems e.g. I can’t view the presentation, what do I do… There were times when the community went very quiet despite putting up new materials and forums and using the bulletin to enthuse members (& non-members) to become involved. We then employed ‘cold calling’ and speaking to staff who we knew had involvement with e.g. the latest forum subject matter. The response to this was generally very positive and we were able to obtain case studies and reports this way. Configuring the XCRI-CAP Metadata Schema The focus of this implementation stage was to configure the XCRI-CAP metadata schema so that it could be used with the HIVE repository system (see image 7). What challenges were there using XCRI-CAP in HIVE? Unlike Dublin Core (DC) or Learning Object Model (LOM) schemas, HIVE did not already have a Graphical User Interface (GUI) that would allow users to add XCRI-CAP metadata to documents being uploaded to the repository. (See image 6) Page 14 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Image 6: HIVE GUI for Dublin Core Metadata Schema Page 15 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Image 7: XCRI-CAP Metadata Schema Diagram Source: http://www.xcri.org/wiki/images/e/e3/Xcri_cap_1_4_r4.png Page 16 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Essentially, HIVE required a Structure Definition Instance file (SDI) to be written that tells HIVE how to generate a GUI for a particular metadata schema. This user interface is a graphical representation of the metadata schema tree being used, providing the user with the field elements that make up the metadata tree and (associated with each element) the type of data you can record. The SDI file is a recreation of the Extensible Mark-Up Language (XML) instance file and the SDI file contains the following types of information, for example: Titles and descriptions of data items that are displayed on the GUI to the user, The type of field into which the data must be entered, for example, a single-line text entry field or date field, The maximum number of characters that can be entered into a field, Information that specifies whether the field is mandatory or optional, Information that specifies whether multiple versions of the field are allowed, How was the XCRI-CAP SDI file for HIVE created? The HIVE technical team, Sam Rowley and Song Ye, were critical in assisting with the development of the XCRI-CAP version for HIVE. The task relied on creating an XML example of XCRI-CAP and transforming it into an SDI file for HIVE, see image 8. This process was made easier by using XML development software, such as Altova XML Spy. Some compromises had to be made as HIVE could not represent certain attributes of the XCRI-CAP (URL for example), and so an ‘element only’ representation of XCRI-CAP was created. This ‘element only’ solution would then also requires a transformation of data into and out of HIVE in order for the XCRI-CAP XML to be valid and interoperable between systems that want to use the XML output from HIVE. Page 17 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Image 8: Early XML example of the XCRI-CAP schema as viewed in Altova XMLSpy Extracting Metadata from Validation Documents This was a difficult process to resolve and I looked at many ways to make the process of extracting the data from Word into XML for HIVE easier. I tried extracting the data directly from Word (by manually marking-up the document), but found too many ‘bugs’ in the process associated with Microsoft and their writing of the XML when exporting. It simply wasn’t valid for the schema to read Matching Programme Specification Fields with Metadata Elements (a mismatch problem) Page 18 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 We spent time as a project group deciding what type of metadata needed to be recorded, as the type of data had a bearing on what people would be looking for, for example, ‘Faculty’, ‘Subject’ or ‘Type of Delivery’. There was an agreement that the key fields in the ‘Programme Specification’ matched the types of data we wanted, however, we extended the list to include some more data fields. The metadata elements used in XCRI-CAP best fitted the data we wanted to record in the repository, essentially course descriptions. In addition, the HIVE repository allowed us to create collections of documents, keeping the already logical folder structure used by QA for their documents. The remaining issue was ‘marking-up’ what type of document was being submitted / looked at (for example, Student Handbook) Programme Specification Recorded for Each Document in XML The element data type to be recorded Metadata Element to be Recorded = Subject Match in XCRI-CAP = Subject http://www.xcri.org/wiki/index.php/Subject Discussed with Sam about recording subject Multiple instances JACS Coding available in CSV from: http://www.hesa.ac.uk/index.php?option=com_content&tas k=view&id=158&Itemid=233 Metadata Element to be Recorded = Level Match in XCRI-CAP = Level http://www.xcri.org/wiki/index.php/Level Multiple instances recorded Chris to provide a definitive list of levels for Drop-down / Look-up List— e.g. o Advanced University Diploma o BA - Ordinary o BA (Hons) o BA (Hons) Dual Award Metadata Element to be Recorded = Validation Date Match in XCRI-CAP = @generated http://www.xcri.org/wiki/index.php/Generated Date to be recorded = date of validation panel Page 19 of 34 XCRI-CAP Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Metadata Element to be Recorded = Study Mode Match in XCRI-CAP = AttendenceMode http://www.xcri.org/wiki/index.php/AttendanceMode Multiple Instances Drop-down / Look-up List: o Face-2-Face o Distance o Blended Metadata Element to be Recorded = Site of Delivery Match in XCRI-CAP = Venue http://www.xcri.org/wiki/index.php/Venue Multiple instances recorded Am contacting Chris about the addresses — e.g Stoke, Stafford, Litchfield, Shrewsbury Metadata Element to be Recorded = Mode of Delivery Match in XCRI-CAP = StudyMode http://www.xcri.org/wiki/index.php/StudyMode Multiple instances recorded Drop-down / Look-up List: o Full Time o Part Time o Fast Track 2 Year Metadata Element to be Recorded = Faculty/School Match in XCRI-CAP = Faculty ** Created by Sam Rowley Multiple Instances Drop-down / Look-up List: o Faculty of Arts, Media and Design (AMD) o Business School o Faculty of Computing, Engineering and Technology (FCET) o Law School o Faculty of Health o Faculty of Science o Academic Development Institute (ADI) Page 20 of 34 To be created Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Metadata Element to be Recorded = Award Title Match in XCRI-CAP = Title http://www.xcri.org/wiki/index.php/Title Record a String title Multiple instances recorded Metadata Element to be Recorded = Professional Body / Association Match in XCRI-CAP = AccreditedBy http://www.xcri.org/wiki/index.php/AccreditedBy Chris to provide a definitive list of levels for Drop-down / Look-up List— e.g. o Association of Operating Department Practitioners o British Computer Society Image 9: Matching the data to be recorded against the XCRI-CAP elements Amendments to the XCRI-CAP Schema It was decided that there was a requirement to record ‘Faculty’ as a key piece of metadata and so the XCRI-CAP element tree was extended to include this specific element. This ‘Faculty’ element was not in the original XCRI-CAP element tree, but it was felt that users would want to search for this element. What about this self service and ‘findability’ approach for users? In simple terms, in order for the system to be ‘usable’ to the user, we would need to make as easy as possible for the user: the process of uploading documents to HIVE; the entry of metadata (‘marking-up’ of document objects), important for searching and finding these document objects. We sought to address these critical issues by: Allowing users to bulk upload the documents to the repository in a logical way, as collections. Allowing users to add metadata as easily as possible to make documents / collections usable, both in terms of searchable & findable. Allow users to interrogate the repository using a search function, searching Page 21 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Adding metadata to HIVE using an XCRI-CAP GUI During the bulk upload process, the user will be required to enter the metadata for the validation collection. This metadata reflects some of the details entered in the ‘Programme Specifications’ document, see image 2.5. Essentially, the metadata recorded will be ‘common’ and associated with all the documents in the collection, as they are all part of the same validation instance. Individual ‘uniqueness’ of the documents, for example whether it is the Student Handbook, the Programme Specification document or the Mentor Handbook, will be recorded within HIVE (a unique instance of the file in HIVE). Recording Document Types Validation Collection A. Pre-Validation Document Types Validation Support Descriptor Rationale Programme Specification Module Descriptor Handbooks o Award / Student o Module o Placement o Mentor Resources B. Validation Document Types Validation Report C. Post-Validation Document Type Validation Report Response Interrogating the HIVE repository This requires searching against the key elements (as outline by the XCRI-CAP element tree). This approach requires offering a user an interactive interface designed on the principles of the XCRI-CAP elements. To date, we are currently finishing a web application (using web services) that talks to the repository and allows user to bulk upload to the repository. This would essentially be an automated validation Page 22 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 submission process. Once this web app. has been completed, we will move onto developing a web ‘widget’ for the NING community of practice website (using Open Social). This will allow users to interrogate the repository in a third party system. This integration with NING will be an important last step, as this will put the documents into a usable context, for discussion and re-purposing. Outputs and Results The DIVAS project aimed to make validation processes more open and expose the validation outputs making them easily accessible to all practitioners in the university. This should improve and accelerate the validation process practice and allow new staff to learn from others’ experiences of validation through reviewing the previous validation documents through their participation in the community. This report summarizes the activities and outputs of the community of practice for DIVAS project. The main objectives of the evaluation were: • To identify the quantitative measures of membership of the Community-of-Practice and of validation resource access; • To explore the qualitative measures of the use of the social networking tool in facilitating the Community-of-practice and access to resources; • To examine the qualitative measures of the usefulness of the resources to academic practitioners; • To scrutinize the qualitative feedback from validation teams on the effectiveness of the fasttrack approach; • To recognise the value of contributions to other JISC projects. The development and pilot of the Community-of-Practice approach and integration of social networking technologies: The Validation Support Network (VSN) commenced in September 2008, access to VSN community was taken up by 100 members (as at July 2009), 69 academic practitioners, 16 administrative staff and 17 learning development staff from Staffordshire University and SURF Colleges. The data gathered via structured interviews and an online survey revealed that the community partially enhanced the practitioners’ reflective practices and enabled them to refocus their thinking on their existing knowledge of the validation process and generate new knowledge and ideas through the social interaction with other community participants. This was clearly illustrated through the materials generated on Ning social network; in particular those that were deployed by Foundation Degrees group members. Page 23 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 The validation documents did not migrate to HIVE digital repository as intended so the linkage between these materials and the Validation Support Network was not established. Therefore it was not feasible to evaluate the access service to the validations materials. Engaging academic practitioners, key administrative staff and learning development staff, through the planned Community-of-Practice is perceived as a fundamental aspect in order for the community to succeed. VSN provided the capacity to build an interactive, diverse, and sharing professional learning environment through discussions, events and groups. The validation process can seem an overwhelming and lengthy process; therefore it was decided with Quality Improvement Services staff to break the discussion of this process down into 2 logical stages, research and design. This was done to simplify required tasks and documentation. Therefore, biweekly online discussions were agreed to take place along with a weekly email to all members to summarize the activities, events and discussions. This offered the community a sense of movement and liveliness and created a community’s rhythm which flourished the connections between members. Upon launching the online community, emails were sent with a brief introduction of DIVAS project along with the aims and objectives of the community in order to inform members with the potential benefits which could be brought to their roles in the institution. This was also followed with a guide on how to access the community network infrastructure (Ning social network). Furthermore, two icebreaker activities were set up “Introduce yourself to the community” and “What is your expectation of this community of practice?” to collect the stakeholders’ requirements and expectations. Gilly Salmon’s 5 stage model (2006) had been adopted for conducting the asynchronous online discussions. This helped the community e-moderators in facilitating these activities in a well organized structure to enhance the discussion participation rate and the connections between practitioners. The results of the observations and questionnaire demonstrate that members had successfully accessed the online community and were enthusiastic to the notion of the social software as they contributed to the most of discussions. Moreover, a few academics from Health and Computing Faculties had identified opportunities in Ning and integrated it into their teaching delivery; this was supported through the face-to-face training sessions held by the learning development staff in Learning Development and Innovation (LDI) department. Feedback from stakeholders also proves that communication had to a certain extent improved between members across the University through the online discussions. However, the participation rate of the discussions that followed went down, which was mainly due to the project not achieving its whole aims which are describing and uploading validation events outputs to HIVE digital repository and implementing the link to the Validation Support Network. Page 24 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 A wide range of events were organized in the last nine months through the VSN, such as” Ask the Experts: The Validation Process” which was a general synchronous discussion aimed to respond to members various queries about the validation process and capturing their expectations and requirements of the community. This experiment enabled DIVAS project team to get to know the participants more closely and establish good relationships with them. They also encouraged members to find others with similar interests to share ideas online by offering one-to-one help and acknowledgement to ensure positive attitudes towards the start of the experience. This support had both technical and e-moderating natures and was able to minimize the barriers and hurdles which otherwise could hinder staff from participation. “Introduction to the project, NING and how to find your way around...” event explained the useful outcomes of the project, introduced DIVAS community of practice concept and gave academic, admin and support staff the opportunity to familiarize themselves with the VSN (Ning-based network) and raise any relevant questions. “Validating Technology Supported Learning (TSL)” event was aimed to exchange experiences, solve problems and explore new ideas, tools and techniques around TSL. We hoped that by taking part in these events people could tangibly experience being part of the community, (Wenger et al, 2002). In spite of the attendance rate being low on these events, the project team handled this issue competently by undertaking an informal approach to communicate with practitioners, e.g. one-to-one meeting and phone dialogue. These informal, "back channel" discussions were by and large efficient as they encouraged staff to become more open about issues and obstacles they encountered. They allowed us to share their comments and statements in the VSN. This information increased the wealth of the validation resources on the network and enriched its contents. The staff members were encouraged to set up groups in the community to exchange information, ideas and experiences and discuss sections of their own choice of the validation process. (Wenger et al 2002) encourages this approach, he perceives it a key to a good community participation and a healthy degree of movement between different levels of participants (e.g. core, active and peripheral participants). Creating groups for sidelines members increased the chance of having semiprivate interaction instead of forcing them to join the main discussions; this allowed participants at all levels to feel like full members. Access to Validation Documents: The project did finish the development phase which should supposedly allow members to access the validation resources on HIVE; hence the measures of the usefulness of reviewing the validation documents by academic practitioners were not applicable. Nonetheless, regardless of missing a significant part of the project, the community was able to build up a useful resource of validation through the case studies, documents and the communication which took place in the discussions forum. We carried out an online survey (see appendix A) to evaluate the COP “The Validation Support Network”. The feedback received was generally positive and promising as shown in Figure 11. Page 25 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Figure 11 The development and pilot of the planned fast-track approach: One of the project aims was to streamline the validations of Technology Supported Learning (TSL) to further exploit the outputs of successful course validations, to develop and inform ‘models of good practice’ which will then form the basis for the pilot of a ‘fast-track’ approach to course validation, These models should provide a succinct framework to help guide less experienced staff through the process and facilitating reduced administrative burdens. The fast-track approach could not be considered until some time into the project life span as a key member of the original project team left. However, later in the lifetime of the project, a new member joined the project team who conducted this approach through a discussion on Validation Support Network and interviews. A prototype model was produced called “Online Course Planning”, its stages had been specified based on the admin practitioners feedback and document analysis of the validation events outputs. (See Figure 11) Figure 11. Online course Planning Page 26 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 However, the evaluation of this model had not been completed by the project stakeholders due to the lack of the time needed to explore these phases, as a result, the project only outlined the basis for the fast track approach for TSL course validation and this maybe considered later. This had an adverse effect on the administrative staff as their duties had not been entirely reduced as planned. However, having an online community acted as a brief guidance resource for Frequently Asked Questions (FAQ) which slightly reduced the admin burdens in terms of general inquiries. Staff could read the contributions on Validation Support Network to find out whether similar cases and questions had already been raised. Contributions to other JISC projects: Building on the COVARM project was not possible (as explained earlier in the document) though DIVAS did effectively contribute to Enable JISC project. The Validation Support Network acted not only as a vehicle to host Enable documents, discussions and blogs but as a source of information which enlightened Enable team experiences and allowed them to gain a broad vision of validation process and curriculum design in the University. The VSN virtual community was considered as an ideal platform to enhance the collaborative work between SURF colleges and the university teams. The information gathered through the interviews with Enable projects team clearly indicated that the Validation Support Network was recognized as a powerful tool which enabled SURF colleges to engage with the university in a streamlined and agile method. In general, volume of contributions fluctuated with a small but strong core of members engaged. Our research suggested some of the reasons for this were: • Lack of time, practitioners could be pre-occupied with their work schedule and short of time to follow the discussions. • New members may have understood the community to be a source of information and best practice, rather than a participation environment. Perhaps they did not feel confident enough to raise their queries or ask questions fearing that these questions might be naive. • People look at the Validation Support Network as a new initiative and perceived it as another tool they need to learn and use. • People may not want to collaborate and interact using social network software, they prefer other methods of communication i.e. talk, phone and email. • Faculty members already have a strong supporting network, relying on their colleagues in answering their queries or using their personal approach to contact other members in other faculties and schools However, the concept of community of practice to a certain extent added a value to the majority of members and encouraged them to communicate and be confident enough to interact with other Page 27 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 practitioners across the University. The community allowed members to construct informative decisions, ideas and views based on discussions and conversations. Positive feedback was also received from members about the connections they made with previously unknown colleagues. Outcomes • Materials have been created during the process of organising events, e.g. slide presentation concerning the validation of TSL, audio materials on using podcasting. • Many suggestions, findings and ideas have come out of the forums and evaluation survey, o Setting up discussion boards between panel members prior to a validation to coordinate communication effectively and efficiently • o Duplication of awards and issues arising at different validations o Templated supporting documents that focus on Foundation Degrees o Assistance for those involved with developing FD’s. Four case studies, three from Staffordshire University and one external, have been put together in the areas of Market Research, Employer Engagement, eAssessment and preparing for validation. • A very active Foundation Degrees group has been created whose membership largely consists of SURF colleagues. Many of the SURF tutors have produced blog posts concerning the development of their awards, there are scenarios concerning issues for foundation degree participants and provided polls to gauge time taken to produce a new award and support received when putting a new award through validation. Our aim, using social networking tools to provide an architecture of participation to build a crossuniversity Community-of-Practice with flexible and convenient access to user-contributed validation support information for re-use and further exploitation. A Ning social network in place providing simple access to forums, blogs, specialist groups, case studies and validation associated documentation. The network has 99 members from across the University, some have actively contributed helpful and supportive information to forums, e.g. advice for staff new to writing award aims and learning outcomes, written blog posts concerning development of new course and some who have provided useful case studies in areas such as Market Research. We aimed to reduce the administrative burdens associated with the validation of courses. Suggestions have been made, by members of the Community of Practice, regarding both development of the validation process and efficiency of the administration of validations, for example making tracking of amendments clearer and providing a register of experienced panel members and those who are interested in sitting on validation panels. The Community of Practice has been valuable in highlighting issues of concern and areas that staff currently find challenging. It is anticipated that academic and administrative staff, involved with validations, will ultimately benefit from the outcomes of the Community of Practice element of the Page 28 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 project. In the meantime the COP has provided QIS staff with the information required to investigate concerns associated with the validation process. Conclusions The VSN community started well and received reasonable contributions, but this did not last long primarily because the project did not develop the link to the validation outputs on HIVE as intended. Wenger et al (2002) stresses the fact that community’s aliveness can not be contrived, dictated or designed in the traditional sense of specifying a structure or process and then implementing it. Validation Support Network needed the access to validation materials which could make it more alive and generate enough excitement, relevance and value to attract and engage members. Nevertheless project team members were fully engaged in the activities which kept the community alive and reduced the impact of non-development task. It was difficult to keep the level of staff engagement high. With the exception of the Foundation Degrees group (whose staff are required, by contract, to contribute to blogs), members have not created their own groups or started their own forums despite being encouraged through the weekly bulletins. Overall, we were pleased and impressed with suggestions and issues arising from discussions but disappointed with the level of active involvement. The project attempted to do 3 things at once: 1. put docs in hive and describe them, 2. make them available through a COP, 3. Create a COP. These were done in parallel because of time constraints but in retrospect, this would have been better done in a linear way, over 3 periods. However if we had done this linearly within the year allocated to the project because of technical difficulties we would have only done one job to the same extent we have anyway and not touched the other 2 jobs. People liked the idea of the COP though more as support network rather than an access point, this however without a selling point it has been very difficult to get people to use to VSN. But because we didn’t have the access point we are unable to tell at the moment whether they will access it more with the validation outputs in it. Validation is the end point, the project should really be focussed on course development and design, isolating validation as a focus has put people off the project, perhaps had we approached from a course design & development point of view we could have interested them in validation as part of the whole process as the end point. Page 29 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Exposing and exploring the issue of the appropriate institutional locus of decisionmaking on the use of emerging technologies to support learning and teaching; including the institutional knowledge and skills, roles and responsibilities required to make effective decisions; Exposing and exploring the issues of cultural change associated with opening-up administrative systems for ‘public’ access In our research we have found some opposition to this. We have fed this back to QIS and are working with them to address the issues raised. Some of these issues are to do with processes (which we are feeding into the enable project) other are to do with cultural change and we are working with QIS to introduce change effectively based upon the cultural information we have gathered from our research. Validation is just part of the design process and is really just the end point and not the driver. So would it be better to think about the system from a holistic point of view. If we could create a central area that all the team who are designing the award create and store all course design materials that could be used to automatically create the documentation for validation. Validation documents tend to be a culmination of work around the course design. For example the rationale for the course is often put together based around a central idea that the rest of the team feed their opinions via email into? At some point the modules are ordered in a diagram somewhere. The learning outcomes are written down and then cut and pasted into tables. So one big central document – each award team gets an instance created for them and they work on a central document. This would also include administrative staff who could then feed information into it (feeding into time tabling systems?) and from this document the relevant validation documents could be created My thoughts are that staff may not have the time to contribute greatly to the COP. Many may be using the community as a resource – to find information already there rather than engage in discussion. There may be concerns over privacy as all forums are open and comments are viewable by all. There may also be issues concerning confidence to use the system – technical know-how etc. Implications There are some serious implications surrounding the diverse methodologies employed around the institution that have implications for the future of more flexible awards. The success of negotiated pathways may depend upon the processes being standardised. We do believe that as a direct result of the research we have done we can produce a tool to help our disability assessors to coordinate assistance available for new students with disabilities. The research has highlighted the amount of duplication and unnecessary work for everybody involved. We would like and think it should be possible to create a system where validation Page 30 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 documentation is automatically created from course design documents. Award teams should only have produce minimal documentation for a validation panel putting the award in context, market research etc. There are also scalability problems surrounding the issue of bulk uploading documents to HIVE. This would require building a tool that would allow the user to bulk upload a ‘collection’ of validation documents to HIVE. Song Ye and Sam Rowley worked on a JAVA-based interface that would allow a user to ‘drag & drop’ documents to their logical areas, for example pre-validation documents. Details about what type of documents they are will be recorded by HIVE. We are looking at HIVE and how it will create a reference instance file that will record what sub category this file is part of, for example, the file is a pre-validation document with the validation collection. Still need to check with QA about whether this folder relationship is necessary. Where the details about what the nature of the document is, for example a student handbook, are still be worked on. It is most likely that this information will be stored as keywords in the XCRI-CAP schema. This type of metadata will mean that file can be searched for, but as the document will share other (less unique) types of metadata, it will mean that the user will also be aware that it is part of a collection (its context). References Wenger, E., McDermott, R., Snyder, W.,M., (2002), Cultivating Communities of Practice: A Guide to Managing Knowledge - Seven Principles for Cultivating Communities of Practice, Harvard Business School [Online] Available at: http://hbswk.hbs.edu/archive/2855.html [Accessed 05 April 2009] Salmon 2002 Source: http://www2.le.ac.uk/departments/beyond-distance-researchalliance/projects/SMELT Page 31 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Appendix A – Validation Support Network Evaluation Questionnaire Page 32 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Page 33 of 34 Project Acronym: DIVAS Version:1 Contact: Sue Lee Date: July 09 Page 34 of 34
© Copyright 2024