1. No category

Project Acronym:
Version:
Contact:
Date:
Project Document Cover Sheet
Project Information
Project Acronym
DIVAS
Project Title
Developing Intelligent Validation Systems
Start Date
12 May 2008
Lead Institution
Staffordshire University
Project Director
Dr Jenny Yorke
Project Manager &
contact details
Sue Lee, LDI, Thompson Library, College Rd, Stoke-on-Trent, ST4 2DE.
Partner Institutions
n/a
Project Web URL
http://divasproject.blogspot.com/
Programme Name (and
number)
(JISC0108)
Programme Manager
Rob Bristow
End Date
30 April 2009
Institutional Responses to Emerging Technologies
Document Name
Document Title
Final Report
Reporting Period
Author(s) & project role
Sue Lee – Project Manager
Date
July 09
URL
if document is posted on project web site
Filename
Access
DIVASFinalReport.docx
General dissemination
Document History
Version
1.0
Date
July 2009
Page 1 of 34
Document title: JISC Final Report
Last updated: April 2007
Comments
Final Version
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
JISC Final Report
DIVAS
Developing Intelligent Validation Systems
Sue Lee, July 09
Page 2 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Table of Contents
.................................................................................................................................... 1
JISC Final Report................................................................................................................................... 2
Table of Contents ........................................................................................................................... 3
Executive Summary........................................................................................................................ 4
Background .................................................................................................................................... 4
Aims and Objectives....................................................................................................................... 5
Methodologies ................................................................................................................................ 6
The Community of Practice ............................................................................................................ 6
Describing Validation Documents................................................................................................... 7
Implementation ............................................................................................................................. 14
Implementing a Community of Practice:....................................................................................... 14
Configuring the XCRI-CAP Metadata Schema............................................................................. 14
Outputs and Results ..................................................................................................................... 23
The development and pilot of the Community-of-Practice approach and integration of social
networking technologies: .............................................................................................................. 23
The development and pilot of the planned fast-track approach:................................................... 26
Contributions to other JISC projects:............................................................................................ 27
Conclusions .................................................................................................................................. 29
Implications................................................................................................................................... 30
References ................................................................................................................................... 31
Appendix A – Validation Support Network Evaluation Questionnaire .......................................... 32
Acknowledgements
This project was conducted under the JISC: JOS e-Administration - Institutional Responses to
Emerging Technologies
The author would particularly like to acknowledge the contribution, help and support of
the following in writing this report; Sue Kneill-Boxley, Ben Scoble, Sam Rowley and Hamza
Badenjiki
Page 3 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Executive Summary
The project had a number of aims:
•
Speeding up validations by using the output of previous validations.
•
Using new technologies to link to administrative systems so they can be accessed easily
•
Reduce the administrative burdens associated with the validation of courses
•
Make the validation and processes more open
•
Create a Community of Practice (COP) for validation support
We feel we have met the last 2 of the aims fully, and are on the way to meeting the other 3. We have
opened up the validation processes in a way they have not been before. The overall approach was to
try and get an accurate picture of the processes of validation as possible. We canvassed for
experiences, opinions and views from with as many individuals in as many different roles as possible.
Though the project was not as successful in meeting our aims as we would have liked we found an
enormous amount of qualitative information that we have been able to forward to our colleagues in
Quality Services. We have used the expertise gained in running a Community of Practice (COP), to
help run a University COP and we now run a series of workshops for academic colleagues on setting
up and sustaining a COP.
We found that support was needed in different ways to what we envisaged, for example, support was
needed for new panel members as well as presenting teams. We found that Staffordshire University
Regional Federation (SURF) colleges www.surf.ac.uk had different support issues than the university
based people and did not really feel a sense of community with the university – this was outside the
remit of this project but useful information.
Background
Every course validation event held at the university produces a validation report. These reports
contain recommendations, conditions and commendations that are fed back to the presenting team so
that quality of university awards can be maintained. This documentation is available to university
employees though is not easily accessible. This project aims to make these reports easily accessible
and allow others about to go through the validation process to learn from their predecessors.
The need for university course validations to enable the rapid development of new courses and
modes of delivery in response to learner, employer and market demand is becoming a significant
issue. In general, validation processes are made sluggish by the enormous administrative burdens
they can impose and by the fact that the outputs of each validation are not effectively exploited to
inform subsequent validations.
Page 4 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
This proposal focuses on addressing the need for institutional administrative systems to facilitate the
sustainable take-up of emerging technologies and provide flexible access for their users to valuable
functions and resources. It specifically addresses the use of emerging technologies to develop
validation processes which are fast and responsive, yet robust, but which are also intelligent, in the
sense that they are informed by their own outputs, open and enabling, allowing colleagues across
institutions to learn from and exploit the valuable information and decisions they make, to improve
agility in course development and the take-up of technology.
Aims and Objectives
The project aimed to address the range of outcomes sought by the call and develop and deliver:
•
A 9-month pilot of the proposed Community-of-Practice-based self-service access to
validation resources, which will expand on the success and expertise of Staffordshire
University’s ‘eLearning Models Project’ in building a vibrant community-of-practice in
Technology Supported Learning, to provide

The integration of emerging social networking technology – possibly Ning (which the
‘eLearning Models project already uses successfully) – with the University’s learning
repository (HIVE) to facilitate access to the resources by the Community-of-Practice.
At the beginning of the project there was a change of staffing which led to the start of the project being
delayed. Because of this and the fact that the project had a short life span the decision was made that
we should not do a full evaluation of the available tools but (as we had already started investigating it)
to use NING as the social networking tool.

The development and facilitation of this Community-of-Practice, comprising academic
practitioners, learning development staff and core quality administration staff

A 6-month sub-pilot of the proposed validation ‘fast-track’ process and self-service
access to process outputs, i.e. to ‘models of good practice’ distilled from, or informed,
by validation outputs; and other important validation decisions.
•
Expand on the work of the COVARM project to develop and contribute a validation
Service Genre encompassing the fast-track and self-service process to the e-Framework
for Education and Research.
This objective changed once the current validation process at the University had been analysed. The
COVARM outputs were compared to current University validation practices and were found to be
essentially the same. Little value was to be gained in rehashing the models created by COVARM. For
this reason, service-oriented effort of the project focussed on development of a software application to
streamline the submission of validation documentation. The application provides an online mechanism
for submission of validation documents along with their metadata and uses a Web services API to
automatically deposit the documents in our Harvest Road Hive learning content repository.
Page 5 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
•
Provide a regularly updated blog and/or wiki reporting on project progress and lessons
learned.
•
A report highlighting the challenges, implications and benefits of developing and
implementing the validation fast-track and Community-of-Practice -based self-service
processes by:

Addressing the challenges and ‘how to’s’ of ‘distilling’ validation outputs into
accessible ‘models of good practice’ useful to academic practitioners, learning
technologists and staff developers;

Expanding on the work already published by Staffordshire University’s ‘eLearning
Models Project’ by reporting on this unique Community-of-Practice approach and the
challenges and benefits of using a social networking tool to facilitate the diverse
community highlighted and their self-service access to validation resources;

Addressing the issues associated with the ‘structuring’ the validation outputs in the
University’s learning repository to ensure their effective discovery, sharing and reuse.

Exposing and exploring the issue of the appropriate institutional locus of decisionmaking on the use of emerging technologies to support learning and teaching;
including the institutional knowledge and skills, roles and responsibilities required to
make effective decisions;

Exposing and exploring the issues of cultural change associated with opening-up
administrative systems for ‘public’ access
Methodologies
The Community of Practice
To set up the Community of Practice, “The Validation Support Network” (VSN), invitations were sent
to faculty Deans, coordinators e.g. Foundation Degrees coordinator, PAMS, Award Leaders,
academics and administrative staff. The invitation message was also posted on a shared All-Staff
email folder. The message included a brief description of what the Community of Practice is about
and an attached document detailing how to get started as a member.
Individual Welcome messages were put into the Comment section of each new members personal
page and we also had three forums set up for new members to get started right away: Introduce
Yourself to the Community; Tell us about your Validation Experiences; What Are Your Expectations of
the Community.
The approach taken to sustaining the Community of Practice involved organising regular online
forums and events and keeping members informed of both these and current member activity through
a weekly bulletin. The bulletin went out in the form of an email every Friday lunch time. In addition to
going to members it also went to the shared All-Staff email directory to generate interest from
University staff not yet joined.
Page 6 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
A new forum was opened every fortnight covering topics such as Validation Administration, Writing a
Teaching & Learning Strategy and Employer Engagement. Two online events were run – an Ask The
Expert event concerning the validation process with a member of the University Quality Improvement
Service (QIS), available on line for two hours to answer community members’ queries and Validating
Technology Supported Learning event whereby three academic practitioners provided audio and
slide/audio presentations regarding their experiences and were available during a fixed two hour time
slot to answer questions. Face-to-face sessions were offered to introduce staff to the COP and
answer any queries but no members attended.
The bulletin was created in email due to restrictions on the number of characters that could be used
for a Ning message and also due to limitations concerning design and layout. The Ning ‘Broadcast
Message’ feature is more suited to short announcements – it was decided, however, that interim
announcements would not be made due to possibility of ‘email overload’ and ‘random’ messages
being ignored.
Describing Validation Documents
The validation documents produced during a Quality Assurance (QA) validation process are in a
various formats, primarily word processor (.doc) format, portable document format (.pdf) or in physical
paper format and held in the QA office. These documents are stored in a ‘silo’ server or in physical
files and folders only accessible by QA staff or on request, within the QA office. The digital formats
were organised and stored in a folder structure, with the top-level folder describing the course
(completed with the date of the validation event).
Within the folder, sub-folders contain pre-validation documents, for example, Module Descriptors or
Programme Specifications.
Main Folder (Title + Validation Event Date)
Sub-folders and
documents
Page 7 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Individual
Documents
Image 1: Individual Validation Folder Structure and Documents
Moving documents into a repository system
Essentially, these current folder structures are ad-hoc, and can contain simply un-aggregated
documents with no sub-folder structure or arranged in a more coherent way. In order to make these
documents more accessible to users, it was a requirement to move these documents and upload
them into a central web-based repository system (HIVE Harvest Road / Giunti Labs). This would allow
greater access to the documents and would also allow them to be re-purposed (linked to) for different
contexts and systems, for example a social networking site. This approach would therefore require
using a structured methodology in order to describe the documents’ role in the validation process and
the relationship between them.
What solution did you identify?
The collection of validation documents are different in each case, however there are several main
‘types’ of documents that are required for the validation process, these can be ‘graded’ into the
following 3 categories (see table 1.1) :
Validation Process Documents
Pre-validation e.g.
Validation e.g.
Post-validation e.g.
Documents that are
Documents that are
Documents that are
presented to the validation
produced as a result of the
produced as a response to
panel.
validation and are associated
the validation event
with the validation event.
recommendations.

Programme Specification

Validation Report

Amended Programme
Specification

Page 8 of 34
Online Guide for staff
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Table 1: Categories of Validation Documents
In addition, what is also common to each validation instance is that each validation did have a
‘Programme Specification’ document at the core. This document describes specific aspects of the
course, in addition to supporting information about, for example;
•
Aims
•
Objectives
•
Teaching and Learning Outcomes
•
Assessment Strategies
•
Programme Structure
•
QAA Benchmarks
•
Support Structures
•
Date
Essentially, it was evident that we needed to record specific details about the documents in order for
them to be ‘useful’ to a user, insofar as they could be interrogated as part of a resource ‘discovery’
process. This could be achieved through using a metadata schema.
Using a Metadata Schema to ‘describe’ documents held in a repository
Within HIVE, many different metadata schemas are used to record specific information about an
uploaded object (e.g. pdf, or doc). A number of ‘Metadata Schemas’ were investigated — these were
already used by the Hive Harvest Road Repository system and are considered international
standards:

Dublin Core (DC) — Information Resource Metadata Schema (see image 1.5)

IMS LOM — Learning Object Metadata Schema (see image 1.6)
Page 9 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Image 2: Dublin Core Metadata Schema Elements as viewed in HIVE
Page 10 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Image 3: IMS 1.3 LOM Metadata Schema Elements
Source = http://upload.wikimedia.org/wikipedia/commons/1/14/LOM_base_schema.png
Page 11 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
What issues were there in using already pre-defined metadata schemas?
From talking to the QA department, it was clear that there were some key pieces of data that could be
recorded by a metadata schema, these are:
1. The ‘Programme Specification’ document elements, as it holds some key information / data
that described the course
2. The types of documents used in the validation process, for example, identifying them as being
either produced pre-validation, during the validation or as a response to the validation event.
There were some specific issues about matching the ‘required’ data held in the ‘Programme
Specification’ document and the types of data recorded against each of the Metadata elements. For
example, some ‘Programme Specification’ fields matched (explicitly) certain metadata elements very
well, where-as some other did not. A ‘work around’ was looked at, but would not be an elegant
solution and could cause problems when interrogating the repository. Essentially, the LOM and DC
schemas describe ‘Learning Items’, however, we were not asking the system to ‘record’ this type of
metadata for our validation document objects. Our validation documents are not traditional learning
items and the LOM and DC schemas are not flexible enough to ‘describe’ the types of documents that
were produced.
Which metadata schema was identified as being the most useful?
The decision was made to use XCRI-CAP, as this describes courses (see image 1.8) and not learning
objects and most of the ‘Programme Specification’ fields / categories matched XCRI-CAP elements
very well. In addition, as we are using the HIVE repository system, we could use the inherent folder
(category) structure it uses in order to create folders that organise, or even reflect, the types of
documents that are produced during the validation process. Decisions about how the documents are
organised and uploaded to HIVE will be addressed during the implementation stage, as we will have
to investigate XCRI-Cap and HIVE integration further. In addition, unfortunately HIVE does not have
the XCRI-CAP metadata scheme as part of its metadata schema list, so we would need to create the
schema and implement a version (an instance) for HIVE.
Page 12 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Image 4: Folder / Category structure in Hive
Image 5: Diagram of how XCRI-CAP describes courses
Page 13 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Implementation
The project has a regularly updated blog http://divasproject.blogspot.com/
outlining the project progress. The project also has 2 other blogs. One to detail the technical aspects:
http://staffordshireuniversitydivas.blogspot.com/ and http://divascommunityofpractice.blogspot.com/
describing the issues involved in setting up a COP.
Implementing a Community of Practice:
Contact information, for those likely to have a role in validation, was gathered from various sources,
such as email directory and prospectus with some key members of faculty, e.g. PAMs, being
contacted directly to obtain lists of Award Leaders. These were then grouped into a Microsoft Outlook
group to receive the invitation message to join the Validation Network COP.
We also sent out a weekly bulletin by email, reviewing activity that had taken place in the community
and also including reports on new membership, forums and planned events.
Forums and events were planned through meetings between Chris Gray, Hamza Badenjiki and Sue
Kneill-Boxley. Forums were hosted by different members of the team depending on the subject
matter, e.g. Validation Administration hosted by Chris Grey of QIS and Innovative Assessment hosted
by Hamza Badenjiki. Events involved staff members we knew to have worked in the subject area, e.g.
practitioners who had employed Technology Supported Learning through the use of podcasts, and
also had staff in attendance to deal with technical problems e.g. I can’t view the presentation, what do
I do…
There were times when the community went very quiet despite putting up new materials and forums
and using the bulletin to enthuse members (& non-members) to become involved. We then employed
‘cold calling’ and speaking to staff who we knew had involvement with e.g. the latest forum subject
matter. The response to this was generally very positive and we were able to obtain case studies and
reports this way.
Configuring the XCRI-CAP Metadata Schema
The focus of this implementation stage was to configure the XCRI-CAP metadata schema so that it
could be used with the HIVE repository system (see image 7).
What challenges were there using XCRI-CAP in HIVE?
Unlike Dublin Core (DC) or Learning Object Model (LOM) schemas, HIVE did not already have a
Graphical User Interface (GUI) that would allow users to add XCRI-CAP metadata to documents
being uploaded to the repository. (See image 6)
Page 14 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Image 6: HIVE GUI for Dublin Core Metadata Schema
Page 15 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Image 7: XCRI-CAP Metadata Schema Diagram
Source: http://www.xcri.org/wiki/images/e/e3/Xcri_cap_1_4_r4.png
Page 16 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Essentially, HIVE required a Structure Definition Instance file (SDI) to be written that tells HIVE how to
generate a GUI for a particular metadata schema. This user interface is a graphical representation of
the metadata schema tree being used, providing the user with the field elements that make up the
metadata tree and (associated with each element) the type of data you can record.
The SDI file is a recreation of the Extensible Mark-Up Language (XML) instance file and the SDI file
contains the following types of information, for example:

Titles and descriptions of data items that are displayed on the GUI to the user,

The type of field into which the data must be entered, for example, a single-line text entry field or
date field,

The maximum number of characters that can be entered into a field,

Information that specifies whether the field is mandatory or optional,

Information that specifies whether multiple versions of the field are allowed,
How was the XCRI-CAP SDI file for HIVE created?
The HIVE technical team, Sam Rowley and Song Ye, were critical in assisting with the development
of the XCRI-CAP version for HIVE. The task relied on creating an XML example of XCRI-CAP and
transforming it into an SDI file for HIVE, see image 8. This process was made easier by using XML
development software, such as Altova XML Spy. Some compromises had to be made as HIVE could
not represent certain attributes of the XCRI-CAP (URL for example), and so an ‘element only’
representation of XCRI-CAP was created. This ‘element only’ solution would then also requires a
transformation of data into and out of HIVE in order for the XCRI-CAP XML to be valid and
interoperable between systems that want to use the XML output from HIVE.
Page 17 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Image 8: Early XML example of the XCRI-CAP schema as viewed in Altova XMLSpy
Extracting Metadata from Validation Documents
This was a difficult process to resolve and I looked at many ways to make the process of extracting
the data from Word into XML for HIVE easier. I tried extracting the data directly from Word (by
manually marking-up the document), but found too many ‘bugs’ in the process associated with
Microsoft and their writing of the XML when exporting. It simply wasn’t valid for the schema to read
Matching Programme Specification Fields with Metadata Elements (a mismatch problem)
Page 18 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
We spent time as a project group deciding what type of metadata needed to be recorded, as the type
of data had a bearing on what people would be looking for, for example, ‘Faculty’, ‘Subject’ or ‘Type of
Delivery’. There was an agreement that the key fields in the ‘Programme Specification’ matched the
types of data we wanted, however, we extended the list to include some more data fields. The
metadata elements used in XCRI-CAP best fitted the data we wanted to record in the repository,
essentially course descriptions. In addition, the HIVE repository allowed us to create collections of
documents, keeping the already logical folder structure used by QA for their documents. The
remaining issue was ‘marking-up’ what type of document was being submitted / looked at (for
example, Student Handbook)
Programme Specification Recorded for Each Document in
XML
The element data type to be recorded
Metadata Element to be Recorded = Subject
Match in XCRI-CAP = Subject
http://www.xcri.org/wiki/index.php/Subject

Discussed with Sam about recording subject

Multiple instances

JACS Coding available in CSV from:
http://www.hesa.ac.uk/index.php?option=com_content&tas
k=view&id=158&Itemid=233
Metadata Element to be Recorded = Level
Match in XCRI-CAP = Level
http://www.xcri.org/wiki/index.php/Level

Multiple instances recorded

Chris to provide a definitive list of levels for Drop-down
/ Look-up List— e.g.
o
Advanced University Diploma
o
BA - Ordinary
o
BA (Hons)
o
BA (Hons) Dual Award
Metadata Element to be Recorded = Validation Date
Match in XCRI-CAP = @generated
http://www.xcri.org/wiki/index.php/Generated

Date to be recorded = date of validation panel
Page 19 of 34
XCRI-CAP
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Metadata Element to be Recorded = Study Mode
Match in XCRI-CAP = AttendenceMode
http://www.xcri.org/wiki/index.php/AttendanceMode

Multiple Instances

Drop-down / Look-up List:
o
Face-2-Face
o
Distance
o
Blended
Metadata Element to be Recorded = Site of Delivery
Match in XCRI-CAP = Venue
http://www.xcri.org/wiki/index.php/Venue

Multiple instances recorded

Am contacting Chris about the addresses — e.g Stoke,
Stafford, Litchfield, Shrewsbury
Metadata Element to be Recorded = Mode of Delivery
Match in XCRI-CAP = StudyMode
http://www.xcri.org/wiki/index.php/StudyMode

Multiple instances recorded

Drop-down / Look-up List:
o
Full Time
o
Part Time
o
Fast Track 2 Year
Metadata Element to be Recorded = Faculty/School
Match in XCRI-CAP = Faculty ** Created by Sam Rowley

Multiple Instances

Drop-down / Look-up List:
o
Faculty of Arts, Media and Design (AMD)
o
Business School
o
Faculty of Computing, Engineering and
Technology (FCET)
o
Law School
o
Faculty of Health
o
Faculty of Science
o
Academic Development Institute (ADI)
Page 20 of 34
To be created
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Metadata Element to be Recorded = Award Title
Match in XCRI-CAP = Title
http://www.xcri.org/wiki/index.php/Title

Record a String title

Multiple instances recorded
Metadata Element to be Recorded = Professional Body /
Association
Match in XCRI-CAP = AccreditedBy
http://www.xcri.org/wiki/index.php/AccreditedBy

Chris to provide a definitive list of levels for Drop-down
/ Look-up List— e.g.
o
Association of Operating Department
Practitioners
o
British Computer Society
Image 9: Matching the data to be recorded against the XCRI-CAP elements
Amendments to the XCRI-CAP Schema
It was decided that there was a requirement to record ‘Faculty’ as a key piece of metadata and so the
XCRI-CAP element tree was extended to include this specific element. This ‘Faculty’ element was not
in the original XCRI-CAP element tree, but it was felt that users would want to search for this element.
What about this self service and ‘findability’ approach for users?
In simple terms, in order for the system to be ‘usable’ to the user, we would need to make as easy as
possible for the user:

the process of uploading documents to HIVE;

the entry of metadata (‘marking-up’ of document objects), important for searching and finding
these document objects.
We sought to address these critical issues by:

Allowing users to bulk upload the documents to the repository in a logical way, as collections.

Allowing users to add metadata as easily as possible to make documents / collections usable,
both in terms of searchable & findable.

Allow users to interrogate the repository using a search function, searching
Page 21 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Adding metadata to HIVE using an XCRI-CAP GUI
During the bulk upload process, the user will be required to enter the metadata for the validation
collection. This metadata reflects some of the details entered in the ‘Programme Specifications’
document, see image 2.5. Essentially, the metadata recorded will be ‘common’ and associated with all
the documents in the collection, as they are all part of the same validation instance. Individual
‘uniqueness’ of the documents, for example whether it is the Student Handbook, the Programme
Specification document or the Mentor Handbook, will be recorded within HIVE (a unique instance of
the file in HIVE).
Recording Document Types
Validation Collection
A. Pre-Validation Document Types

Validation Support Descriptor

Rationale

Programme Specification

Module Descriptor

Handbooks

o
Award / Student
o
Module
o
Placement
o
Mentor
Resources
B. Validation Document Types

Validation Report
C. Post-Validation Document Type

Validation Report Response
Interrogating the HIVE repository
This requires searching against the key elements (as outline by the XCRI-CAP element tree). This
approach requires offering a user an interactive interface designed on the principles of the XCRI-CAP
elements.
To date, we are currently finishing a web application (using web services) that talks to the repository
and allows user to bulk upload to the repository. This would essentially be an automated validation
Page 22 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
submission process. Once this web app. has been completed, we will move onto developing a web
‘widget’ for the NING community of practice website (using Open Social). This will allow users to
interrogate the repository in a third party system. This integration with NING will be an important last
step, as this will put the documents into a usable context, for discussion and re-purposing.
Outputs and Results
The DIVAS project aimed to make validation processes more open and expose the validation outputs
making them easily accessible to all practitioners in the university. This should improve and
accelerate the validation process practice and allow new staff to learn from others’ experiences of
validation through reviewing the previous validation documents through their participation in the
community.
This report summarizes the activities and outputs of the community of practice for DIVAS project. The
main objectives of the evaluation were:
•
To identify the quantitative measures of membership of the Community-of-Practice and of
validation resource access;
•
To explore the qualitative measures of the use of the social networking tool in facilitating the
Community-of-practice and access to resources;
•
To examine the qualitative measures of the usefulness of the resources to academic
practitioners;
•
To scrutinize the qualitative feedback from validation teams on the effectiveness of the fasttrack approach;
•
To recognise the value of contributions to other JISC projects.
The development and pilot of the Community-of-Practice approach and integration of social
networking technologies:
The Validation Support Network (VSN) commenced in September 2008, access to VSN community
was taken up by 100 members (as at July 2009), 69 academic practitioners, 16 administrative staff
and 17 learning development staff from Staffordshire University and SURF Colleges.
The data gathered via structured interviews and an online survey revealed that the community
partially enhanced the practitioners’ reflective practices and enabled them to refocus their thinking on
their existing knowledge of the validation process and generate new knowledge and ideas through the
social interaction with other community participants. This was clearly illustrated through the materials
generated on Ning social network; in particular those that were deployed by Foundation Degrees
group members.
Page 23 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
The validation documents did not migrate to HIVE digital repository as intended so the linkage
between these materials and the Validation Support Network was not established. Therefore it was
not feasible to evaluate the access service to the validations materials.
Engaging academic practitioners, key administrative staff and learning development staff, through the
planned Community-of-Practice is perceived as a fundamental aspect in order for the community to
succeed. VSN provided the capacity to build an interactive, diverse, and sharing professional learning
environment through discussions, events and groups.
The validation process can seem an overwhelming and lengthy process; therefore it was decided with
Quality Improvement Services staff to break the discussion of this process down into 2 logical stages,
research and design. This was done to simplify required tasks and documentation. Therefore, biweekly online discussions were agreed to take place along with a weekly email to all members to
summarize the activities, events and discussions. This offered the community a sense of movement
and liveliness and created a community’s rhythm which flourished the connections between members.
Upon launching the online community, emails were sent with a brief introduction of DIVAS project
along with the aims and objectives of the community in order to inform members with the potential
benefits which could be brought to their roles in the institution. This was also followed with a guide on
how to access the community network infrastructure (Ning social network). Furthermore, two
icebreaker activities were set up “Introduce yourself to the community” and “What is your expectation
of this community of practice?” to collect the stakeholders’ requirements and expectations. Gilly
Salmon’s 5 stage model (2006) had been adopted for conducting the asynchronous online
discussions. This helped the community e-moderators in facilitating these activities in a well organized
structure to enhance the discussion participation rate and the connections between practitioners.
The results of the observations and questionnaire demonstrate that members had successfully
accessed the online community and were enthusiastic to the notion of the social software as they
contributed to the most of discussions. Moreover, a few academics from Health and Computing
Faculties had identified opportunities in Ning and integrated it into their teaching delivery; this was
supported through the face-to-face training sessions held by the learning development staff in
Learning Development and Innovation (LDI) department.
Feedback from stakeholders also proves that communication had to a certain extent improved
between members across the University through the online discussions. However, the participation
rate of the discussions that followed went down, which was mainly due to the project not achieving its
whole aims which are describing and uploading validation events outputs to HIVE digital repository
and implementing the link to the Validation Support Network.
Page 24 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
A wide range of events were organized in the last nine months through the VSN, such as” Ask the
Experts: The Validation Process” which was a general synchronous discussion aimed to respond to
members various queries about the validation process and capturing their expectations and
requirements of the community. This experiment enabled DIVAS project team to get to know the
participants more closely and establish good relationships with them. They also encouraged members
to find others with similar interests to share ideas online by offering one-to-one help and
acknowledgement to ensure positive attitudes towards the start of the experience. This support had
both technical and e-moderating natures and was able to minimize the barriers and hurdles which
otherwise could hinder staff from participation.
“Introduction to the project, NING and how to find your way around...” event explained the useful
outcomes of the project, introduced DIVAS community of practice concept and gave academic, admin
and support staff the opportunity to familiarize themselves with the VSN (Ning-based network) and
raise any relevant questions. “Validating Technology Supported Learning (TSL)” event was aimed to
exchange experiences, solve problems and explore new ideas, tools and techniques around TSL. We
hoped that by taking part in these events people could tangibly experience being part of the
community, (Wenger et al, 2002). In spite of the attendance rate being low on these events, the
project team handled this issue competently by undertaking an informal approach to communicate
with practitioners, e.g. one-to-one meeting and phone dialogue. These informal, "back channel"
discussions were by and large efficient as they encouraged staff to become more open about issues
and obstacles they encountered. They allowed us to share their comments and statements in the
VSN. This information increased the wealth of the validation resources on the network and enriched
its contents.
The staff members were encouraged to set up groups in the community to exchange information,
ideas and experiences and discuss sections of their own choice of the validation process. (Wenger et
al 2002) encourages this approach, he perceives it a key to a good community participation and a
healthy degree of movement between different levels of participants (e.g. core, active and peripheral
participants). Creating groups for sidelines members increased the chance of having semiprivate
interaction instead of forcing them to join the main discussions; this allowed participants at all levels to
feel like full members.
Access to Validation Documents:
The project did finish the development phase which should supposedly allow members to access the
validation resources on HIVE; hence the measures of the usefulness of reviewing the validation
documents by academic practitioners were not applicable. Nonetheless, regardless of missing a
significant part of the project, the community was able to build up a useful resource of validation
through the case studies, documents and the communication which took place in the discussions
forum. We carried out an online survey (see appendix A) to evaluate the COP “The Validation Support
Network”. The feedback received was generally positive and promising as shown in Figure 11.
Page 25 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Figure 11
The development and pilot of the planned fast-track approach:
One of the project aims was to streamline the validations of Technology Supported Learning (TSL) to
further exploit the outputs of successful course validations, to develop and inform ‘models of good
practice’ which will then form the basis for the pilot of a ‘fast-track’ approach to course validation,
These models should provide a succinct framework to help guide less experienced staff through the
process and facilitating reduced administrative burdens.
The fast-track approach could not be considered until some time into the project life span as a key
member of the original project team left. However, later in the lifetime of the project, a new member
joined the project team who conducted this approach through a discussion on Validation Support
Network and interviews. A prototype model was produced called “Online Course Planning”, its stages
had been specified based on the admin practitioners feedback and document analysis of the
validation events outputs. (See Figure 11)
Figure 11. Online course Planning
Page 26 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
However, the evaluation of this model had not been completed by the project stakeholders due to the
lack of the time needed to explore these phases, as a result, the project only outlined the basis for the
fast track approach for TSL course validation and this maybe considered later. This had an adverse
effect on the administrative staff as their duties had not been entirely reduced as planned. However,
having an online community acted as a brief guidance resource for Frequently Asked Questions
(FAQ) which slightly reduced the admin burdens in terms of general inquiries. Staff could read the
contributions on Validation Support Network to find out whether similar cases and questions had
already been raised.
Contributions to other JISC projects:
Building on the COVARM project was not possible (as explained earlier in the document) though
DIVAS did effectively contribute to Enable JISC project. The Validation Support Network acted not
only as a vehicle to host Enable documents, discussions and blogs but as a source of information
which enlightened Enable team experiences and allowed them to gain a broad vision of validation
process and curriculum design in the University. The VSN virtual community was considered as an
ideal platform to enhance the collaborative work between SURF colleges and the university teams.
The information gathered through the interviews with Enable projects team clearly indicated that the
Validation Support Network was recognized as a powerful tool which enabled SURF colleges to
engage with the university in a streamlined and agile method.
In general, volume of contributions fluctuated with a small but strong core of members engaged. Our
research suggested some of the reasons for this were:
•
Lack of time, practitioners could be pre-occupied with their work schedule and short of time to
follow the discussions.
•
New members may have understood the community to be a source of information and best
practice, rather than a participation environment. Perhaps they did not feel confident enough
to raise their queries or ask questions fearing that these questions might be naive.
•
People look at the Validation Support Network as a new initiative and perceived it as another
tool they need to learn and use.
•
People may not want to collaborate and interact using social network software, they prefer
other methods of communication i.e. talk, phone and email.
•
Faculty members already have a strong supporting network, relying on their colleagues in
answering their queries or using their personal approach to contact other members in other
faculties and schools
However, the concept of community of practice to a certain extent added a value to the majority of
members and encouraged them to communicate and be confident enough to interact with other
Page 27 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
practitioners across the University. The community allowed members to construct informative
decisions, ideas and views based on discussions and conversations. Positive feedback was also
received from members about the connections they made with previously unknown colleagues.
Outcomes
•
Materials have been created during the process of organising events, e.g. slide presentation
concerning the validation of TSL, audio materials on using podcasting.
•
Many suggestions, findings and ideas have come out of the forums and evaluation survey,
o
Setting up discussion boards between panel members prior to a validation to
coordinate communication effectively and efficiently
•
o
Duplication of awards and issues arising at different validations
o
Templated supporting documents that focus on Foundation Degrees
o
Assistance for those involved with developing FD’s.
Four case studies, three from Staffordshire University and one external, have been put
together in the areas of Market Research, Employer Engagement, eAssessment and
preparing for validation.
•
A very active Foundation Degrees group has been created whose membership largely
consists of SURF colleagues. Many of the SURF tutors have produced blog posts concerning
the development of their awards, there are scenarios concerning issues for foundation degree
participants and provided polls to gauge time taken to produce a new award and support
received when putting a new award through validation.
Our aim, using social networking tools to provide an architecture of participation to build a crossuniversity Community-of-Practice with flexible and convenient access to user-contributed validation
support information for re-use and further exploitation. A Ning social network in place providing simple
access to forums, blogs, specialist groups, case studies and validation associated documentation.
The network has 99 members from across the University, some have actively contributed helpful and
supportive information to forums, e.g. advice for staff new to writing award aims and learning
outcomes, written blog posts concerning development of new course and some who have provided
useful case studies in areas such as Market Research.
We aimed to reduce the administrative burdens associated with the validation of courses.
Suggestions have been made, by members of the Community of Practice, regarding both
development of the validation process and efficiency of the administration of validations, for example
making tracking of amendments clearer and providing a register of experienced panel members and
those who are interested in sitting on validation panels.
The Community of Practice has been valuable in highlighting issues of concern and areas that staff
currently find challenging. It is anticipated that academic and administrative staff, involved with
validations, will ultimately benefit from the outcomes of the Community of Practice element of the
Page 28 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
project. In the meantime the COP has provided QIS staff with the information required to investigate
concerns associated with the validation process.
Conclusions
The VSN community started well and received reasonable contributions, but this did not last long
primarily because the project did not develop the link to the validation outputs on HIVE as intended.
Wenger et al (2002) stresses the fact that community’s aliveness can not be contrived, dictated or
designed in the traditional sense of specifying a structure or process and then implementing it.
Validation Support Network needed the access to validation materials which could make it more alive
and generate enough excitement, relevance and value to attract and engage members. Nevertheless
project team members were fully engaged in the activities which kept the community alive and
reduced the impact of non-development task.
It was difficult to keep the level of staff engagement high. With the exception of the Foundation
Degrees group (whose staff are required, by contract, to contribute to blogs), members have not
created their own groups or started their own forums despite being encouraged through the weekly
bulletins. Overall, we were pleased and impressed with suggestions and issues arising from
discussions but disappointed with the level of active involvement.
The project attempted to do 3 things at once:
1. put docs in hive and describe them,
2. make them available through a COP,
3. Create a COP.
These were done in parallel because of time constraints but in retrospect, this would have been better
done in a linear way, over 3 periods. However if we had done this linearly within the year allocated to
the project because of technical difficulties we would have only done one job to the same extent we
have anyway and not touched the other 2 jobs.
People liked the idea of the COP though more as support network rather than an access point, this
however without a selling point it has been very difficult to get people to use to VSN. But because we
didn’t have the access point we are unable to tell at the moment whether they will access it more with
the validation outputs in it.
Validation is the end point, the project should really be focussed on course development and design,
isolating validation as a focus has put people off the project, perhaps had we approached from a
course design & development point of view we could have interested them in validation as part of the
whole process as the end point.
Page 29 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09

Exposing and exploring the issue of the appropriate institutional locus of decisionmaking on the use of emerging technologies to support learning and teaching;
including the institutional knowledge and skills, roles and responsibilities required to
make effective decisions;

Exposing and exploring the issues of cultural change associated with opening-up
administrative systems for ‘public’ access
In our research we have found some opposition to this. We have fed this back to QIS and are working
with them to address the issues raised. Some of these issues are to do with processes (which we are
feeding into the enable project) other are to do with cultural change and we are working with QIS to
introduce change effectively based upon the cultural information we have gathered from our research.
Validation is just part of the design process and is really just the end point and not the driver. So
would it be better to think about the system from a holistic point of view. If we could create a central
area that all the team who are designing the award create and store all course design materials that
could be used to automatically create the documentation for validation. Validation documents tend to
be a culmination of work around the course design. For example the rationale for the course is often
put together based around a central idea that the rest of the team feed their opinions via email into?
At some point the modules are ordered in a diagram somewhere. The learning outcomes are written
down and then cut and pasted into tables.
So one big central document – each award team gets an instance created for them and they work on
a central document. This would also include administrative staff who could then feed information into it
(feeding into time tabling systems?) and from this document the relevant validation documents could
be created
My thoughts are that staff may not have the time to contribute greatly to the COP. Many may be using
the community as a resource – to find information already there rather than engage in discussion.
There may be concerns over privacy as all forums are open and comments are viewable by all. There
may also be issues concerning confidence to use the system – technical know-how etc.
Implications
There are some serious implications surrounding the diverse methodologies employed around the
institution that have implications for the future of more flexible awards. The success of negotiated
pathways may depend upon the processes being standardised. We do believe that as a direct result
of the research we have done we can produce a tool to help our disability assessors to coordinate
assistance available for new students with disabilities.
The research has highlighted the amount of duplication and unnecessary work for everybody
involved. We would like and think it should be possible to create a system where validation
Page 30 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
documentation is automatically created from course design documents. Award teams should only
have produce minimal documentation for a validation panel putting the award in context, market
research etc.
There are also scalability problems surrounding the issue of bulk uploading documents to HIVE. This
would require building a tool that would allow the user to bulk upload a ‘collection’ of validation
documents to HIVE. Song Ye and Sam Rowley worked on a JAVA-based interface that would allow a
user to ‘drag & drop’ documents to their logical areas, for example pre-validation documents. Details
about what type of documents they are will be recorded by HIVE.
We are looking at HIVE and how it will create a reference instance file that will record what sub
category this file is part of, for example, the file is a pre-validation document with the validation
collection. Still need to check with QA about whether this folder relationship is necessary.
Where the details about what the nature of the document is, for example a student handbook, are still
be worked on. It is most likely that this information will be stored as keywords in the XCRI-CAP
schema.
This type of metadata will mean that file can be searched for, but as the document will share other
(less unique) types of metadata, it will mean that the user will also be aware that it is part of a
collection (its context).
References
Wenger, E., McDermott, R., Snyder, W.,M., (2002), Cultivating Communities of Practice: A Guide to
Managing Knowledge - Seven Principles for Cultivating Communities of Practice, Harvard Business
School [Online] Available at: http://hbswk.hbs.edu/archive/2855.html
[Accessed 05 April 2009]
Salmon 2002 Source: http://www2.le.ac.uk/departments/beyond-distance-researchalliance/projects/SMELT
Page 31 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Appendix A – Validation Support Network Evaluation Questionnaire
Page 32 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Page 33 of 34
Project Acronym: DIVAS
Version:1
Contact: Sue Lee
Date: July 09
Page 34 of 34