2+The+validation+process

Home > 2: validation process =2: The validation process=

Participants
The following took part in the validation:
 * Lithuania (Centre of Information Technologies in Education)
 * Austria (Ministry of Education)
 * Spain (CEIP San Felix, Candas, Asturias)
 * Hungary
 * Poland

What is expected?
EUN will ensure that the process is simple and uses as little time as possible, with the emphasis on learning about personalisation and some state of the art developments in Europe and providing feedback to the product developers to ensure that future resources meet teacher and student needs. The work is a useful way of taking part in professional development about personalisation with peers in other countries. Partners working with EUN should: The guidance following has been extracted from a paper by Ben Gurion University, Israel (BGU): **[|Validation_procedure.doc].**
 * 1) Identify schools, teachers and classes prepared to give time between Janaury and June 2008. Questions:
 * 2) How many teachers? We are expected to provide the views of 50 teachers in 20 schools, so let's say up to 20 per country, with a minimum of five. Task 4 - the introduction to iClass - requires a face to face workshop, so it makes sense to work on numbers prepared to take part in this workshop. Teachers can be from the same school; in fact this would be a good way to enable them to compare ideas and work together.
 * 3) Do they have to be ICT experts or ICT teachers? No, a cross section is preferable, with some experts and others who are ICT phobics or beginners. They could be teacher trainers or recently retired or promoted teachers, ie people who work with teachers and understand classroom realities. We do prefer secondary level school teachers rather than primary however because of the nature of the content and user expectations on the system (eg to be a self-directed learner). It would be useful if some teachers could work with their students (aged 14-19 probably) on the validation, but this is a minority.
 * 4) Do participants have to speak English? Not really but the working language of the project is English and the content on the system is in English. However the concepts are not specific to any language.
 * 5) What is the incentive for teachers to take part? A chance to develop professionally (EUN can send a certificate of participation on completion of the work for example) and to learn about virtual learning environments and the pedagogy of personalised learning. All participants will have access to the overall findings and outcomes of the evaluation process. Schools will have the use of the system at no charge, as long as it is still maintained by the project partners.
 * 6) If necessary, translate questionnaires and key materials into the local language. EUN can fund this if necessary.
 * 7) Administer a one hour pre-test questionnaire to teachers and students. It is important to do this before they have any introduction to iClass system and the personalised learning model. Ideally this should be before the end of January 2008, but an extension may be possible. They can do the test without supervision and individually.
 * 8) Provide an introduction to teachers and students about iClass, the system and Self-Regulated Personalised Learning, using the resources in section 3, notably the presentations and the Handbook. The system will be presented either as a mock up or through direct access and use.
 * 9) Can I see the system? Yes: [|__http://demo.iclassproject.com__] . Access is via ID and password. It works in English and French (in parts). There is a prototype of the user interface to the system [|here]. There are no plans to translate the system into other languages. Consider the system a prototype, a proof of concept, to stimulate discussion. No training in its use is necessary.
 * 10) Who provides this introduction? The EUN national contact, with some help from EUN and the iClass project partners.
 * 11) Can schools continue to use the system? Certainly until June 2008 but beyond that (the end of the project) we cannot say because 1) it may remain a prototype and incomplete, and not have end user support, hotlines and other services a school would expect of a learning platform 2) it depends on the outcome of the project and how partners like Siemens and Microsoft want to take the system beyond the end of the project.
 * 12) Administer a one hour post-test questionnaire
 * 13) Conduct focus group interviews with selected teachers and possibly students.
 * 14) Send the results (translating open ended answers into English) to Ben Gurion University (BGU).
 * Details**
 * 1) BGU team has devised a model and strategy to evaluate the progress and effectiveness of iClass.
 * 2) BGU will be in charge of the "micro" level, performing: (1) formative iterative evaluation of iClass; (2) summative evaluation of iClass – validating its impact on its users (teachers and students), to whether it has achieved SRPL (self-regulated personalized learning); (3) formative and summative evaluation of iClass' enveloping methodologies (for example, change management, classroom pedagogy); and (4) comparison analysis between countries using SRPL variables.
 * 3) EUN will be in charge of the "macro" level: (1) evaluating the effect of introducing iClass, as an ICT system, to schools and educational systems; (2) assessing iClass' exploitation prospective; and (3) comparison analysis between countries and ICT systems, using variables regarding exploitation and system introduction effect.
 * 4) EUN and BGU will work in parallel in two levels (micro and macro) in the same schools. Data collected for EUN's research as well as for BGU's summative research will be consolidated into one report for the purpose of iClass evaluation, to be submitted at the final review.


 * Research Design**
 * //Research process//**: Following the release of the current version of iClass system that will take place November 19 2007, users (teachers and students) will be allowed an adaptation period of at least 3 weeks. During this period, users will be encouraged to use iClass in their natural educational surroundings as much as possible. For this end, users will be prompted to use iClass "Handbook", aimed to explain the pedagogical model, describe ways in which iClass system can be integrated in the classroom, define each feature in the system, give a short tutorial, and present examples. Following this period we will distribute the evaluation questionnaires to users (teachers and students), through all participating schools.


 * //Research Sample//**: Research sample includes schools already collaborating with BGU as well as additional schools new to the project. Schools with which we continue our collaborative work include: 4 schools which belong to the European School Network in Belgium; 1 school of the ORT France Network; 2 Israeli open schools – "Masar" and "The open democratic school in Jaffa"; and 1 school situated in the Netherlands - "Aventuryn". Additional schools will be provided by EUN, and will consist of 20 European schools (which will result in 50 teachers).

(1) Participants who use iClass platform (ICT and SRPL prompting); (2) Participants who participate in an SRPL prompting "regular" classroom – SRPL only; (3) Participants who use a ICT which doesn't support SRPL – ICT only; (4) Control group - Participants who do not participate in an SRPL prompting classroom and do not use a ICT. This design aims at controlling as many variables as possible for our assumed effect to be statistically and thus fundamentally reliable.
 * //Research groups//**: In order to achieve reliable results as possible, we have designed an //evaluation design// which includes three experimental groups and one control group, which comprise the four different conditions of using ICT and enhancing SRPL:


 * //Research tools//**: Two primary methods will be used in evaluating the current version of iClass system: Focus group interviews and questionnaires. Both will be used to assess system validity, users' feedback on the system and interface, and users' requirements. Focus group guiding questions will be used in our weekly interviews with the focus group teachers, and will attain information regarding the assimilation of the current version of iClass and the adaptation of users to working with it.

In the process of evaluating the current version of iClass, we will assess relevant parameters designed to address the research objectives (described above). For this aim, questionnaires and focus group interviews will include three main clusters of questions designed to //validate the system//, assess //system usability and// //interface parameters//, and re-assess //users' requirements//. It should be mentioned here again, that as formative evaluation aims to accompany the evolution of the project, and thus its iterative characteristic, some, if not many, parameters assessed in the former evaluation sessions (i.e. base-line and mock-up) will be assessed again here (the following exemplifying questions are presented as questions to students):
 * Scope and variables**

//Transparency//: //Choice//: //Personalization//: //Mindfulness//: //Meaningfulness//: //Exploration:// //Reflection//: //Planning//: //Monitoring//: //Collaboration//:
 * //Validating the System//** – parameters assessed here aim to evaluate whether iClass, in this current and early version, is heading the right way to achieve self-regulated personalized learning in users, as it is defined in iClass pedagogical model:

//Usability// //Visualization// //Logicality//
 * //User Interface related variables//** – Interface evaluation is designed to provide the system designers with users' feedback on platform usability, friendliness, complexity, visual parameters, etc.:


 * //Enveloping methodologies related variables//** – parameters assessed here will address classroom pedagogy methodology and some aspects of change management methodology. These methodologies are designed to address the know-how of the way in which iClass should be used in the classroom according to the pedagogical model, and the level of acceptance of and commitments to iClass and its values in schools. For example, parameters will assess users' understanding of the Handbook designed to give a tutorial of how to use iClass in a mindful and meaningful way, to reach personalization of the learning process. Users will also be asked how they would change the handbook.


 * //Users' requirements (new information)//** – Elicit participants to suggest new ideas that will improve iClass as supporting and enhancing SRPL.


 * //Specific tools//** – Participants will be elicited to react on specific tools and features: Planner, Personal Space, Pool of interests, etc.


 * //Writing an evaluation report//**: BGU will prepare and submit an evaluation report concerning this evaluation stage. This report will include findings and conclusions reached by the evaluation team regarding teachers' and students' feedback on the current version of iClass. Conclusions will result in recommendations for future development of iClass and its enveloping methodologies.

Formative evaluation will be carried out according to plan: we will continue formative evaluation of iClass and re-assess parameters related to validating the system, system usability, user interface parameters and parameters addressing iClass enveloping methodologies, as described above. As this milestone will be assessed as part of the formative evaluation stage, we will continue with the formative research design, research groups and research tools described above. This stage will result in an evaluation report, portraying the evolutionary development of iClass.
 * iClass milestone, April 1**

This will be the last and conclusive stage of the evaluation process of iClass. Here we aim to assess the third evaluative goal - //impact// of iClass on its users: to whether iClass facilitates SRPL in users.
 * Release III, Summative evaluation, June 2008**

(1) Cognitive and psychological processes which are SRPL related based on iClass pedagogical model, and which were pre-assessed in the base-line stage: SRL, personal intrinsic motivation (for teaching and learning), sense of autonomy, sense of acceptance, etc (based on Ryan & Deci's SRL Questionnaires; and The Intrinsic Motivation Inventory (IMI)). Here these parameters will be re-assessed and compared to their assessed values in the base-line stage. (2) Parameters assessed here aim to evaluate whether iClass has achieved self-regulated personalized learning in users, the way it is defined in iClass pedagogical model. These parameters were assessed and re-assessed in previous formative stages, and thus will be evaluated for cumulative changes.
 * //Variables//**: variables that will be assessed here will be impact related and thus will aim to assess SRPL related parameters, in two clusters of questions:


 * //Research tools//**: Two primary methods will be used in this stage as well: Focus group interviews and questionnaires.


 * //Writing an evaluation report//**: Results and conclusion will be consolidated into a summative evaluation report. This final report will be collaboratively devised with EUN team (as described above).

Flagg, B. N. (1990). //Formative Evaluation for Educational Technologies//. NJ: Lawrence Erlbaum.
 * References**:

Ryan & Deci's SRL Questionnaires, retrieved from: http://www.psych.rochester.educ/SDT/index.html

The Intrinsic Motivation Inventory (IMI), retrieved from: http://www.psych.rochester.educ/SDT/index.html