Principles of good assessment and feedback: theory and practice – D. Nicol (2007)

Nicol added to the seven principles, formulated with his colleague MacFarlane-Dick  in an earlier paper, to ten principles – which include:

1. Clarify what good performance is about (goals, criteria, standards)

2. Encourage time and effort on challenging learning tasks

3. Give high quality information on students’ learning

4. Encourage positive motivation and self esteem

5. Encourage interaction and dialogue between peer and teacher-student

6. Facilitate self-assessment and reflection

7. Provides option of choice in assessment to learners

8. Involve students in decision-making about assessment policy and practice

9. Support developing learner communities

10. Assist teachers to adapt teaching to students’ needs

I think Nicol’s ideas and framework are applicable to teaching and learning in general, as very good practice.

Nicol applies two dimensions when implementing the above principles, which are:

–       engagement-empowerment – the extent to which the student regulate and tsake responsibility for their learning

–       academic-social – the extent to which the academic and social experiences support each other in the learning and development

Figure 2 in the paper represents the ten principles in relation with the two dimensions. According to Nicol the use of the framework, has proved helpful but these principles will have to be used/applied relevant to the specific context.  

Two positive comments and two criticisms about Nicol’s framework.

 Positives:

Principle 1 – important to state right at the beginning and remind students from time to time: Clarify what good performance is (goals, criteria and standards). 

 Principle 10 – Help teachers adapt teaching to student needs. Supporting students on their level is important however, teachers should be careful in handling this especially if he/she is inexperienced, with learners who may take advantage of the situation

 

Negatives:

Principles 2-5 are about empowering learners to take responsibility for their own learning, which is good but can be difficult to implement with less motivated learners

Principle 8 – Involve students in decision-making about assessment policy and practice. I think I will inform the students and ask for their feedback but will limit the choice. Although I don’t think it is that easy to change assessment policies???

Activity4: Reflecting on feedback

In using the technology, assessment and feedback practices have a lot of potential to achieve the aims of assessment, however it is still under-utilised. The use of for example, multiple-choice tests (if used in a relevant situation), has several benefits in enhancing learning.

From the JISC report (2010), technology-enhanced assessment and feedback refers to practices that provide some, or all, of the following benefits:

• Greater variety and authenticity in the design of assessments
• Improved learner engagement e.g. through interactive formative assessments with adaptive feedback
• Choice in the timing and location of assessments
• Wider skills and attributes not easily assessed by other means, are captured, e.g. through simulations, e-portfolios and interactive games
• Efficient submission, marking, moderation and data storage processes
• Consistent, accurate results with opportunities to combine human and computer marking
• Immediate feedback
• Increased opportunities for learners to act on feedback, e.g. reflecting in e-portfolios
• Innovative approaches based around use of creative media and online peer and self-assessment
• Accurate, timely and accessible evidence on the effectiveness of curriculum design and delivery

I found the following research interesting (Mason,?), in that it depends on a students’ ability – the expertise level in the content area – how effective they could utilize different kinds of feedback. Students with low achievement levels find more benefit from more immediate feedback compared to high achieving students who benefit more from delayed feedback (Gaynor and Roper, cited by Mason).

The explanation may be that higher achieving students, rethink the incorrect information and “go back” to their previous information base where the delayed feedback then adds to the time for them to actively process the information. The immediate feedback given to the low achieving student, possibly with a less accurate understanding of the basic information, provides them the opportunity to correct their mistakes.

However, in some cases, challenges accompany the benefits, e.g. Nicols’ paper (2006) on Formative assessment and self-regulated learning, shows learners can control their own learning and become self-regulated learners. Learners assess their own work, build their own feedback – thus their proactive activities impact on the assessment and learning which the teacher/tutor provides, which again impacts on the feedback given to the student aimed to improve the learners’ performance and learning.  I think this can be a challenge to both the learner and the tutor as feedback should be appropriate and as Ball (2013) mentions in her blog: “In a way it can make or break the activity” – where “giving feedback can be a crucial part of any activity”.

The authors (Nicol and MacFarlane) mention more problems when applying the transmission view (where formative assessment and feedback is still seen and controlled by teachers and feedback conceptualised as a transmission process). Summarized, they consist of: formative assessment in the hands of the teacher only, will limit students to develop the self‐regulation skills needed in preparation for lifelong learning; before students are able to “use” the feedback information, they need to understand them as they are complex and students require clarity; the beliefs of students can regulate the effects of feedback messages  (external feedback has been shown to influence how students feel about themselves (positively or negatively), and what and how they learn); the workload of teachers increases as student numbers become larger.

References:

JISC (2010).Effective assessment in a digital age [online] Available online: http://www.jisc.ac.uk/media/documents/programmes/elearning/digiassass_eada.pdf (accessed 5 July 2013)

Ball, A. (2013) reflections on h817 reflecting on feedback 1 July 2013  http://alisonball.edublogs.org/2013/07/01/reflecting-on-feedback/ (accessed 4 July 2013)

Mason, B.J., Bruning, R. (?) Providing Feedback in Computer-based Instruction: What the Research Tells Us [online] Available at: http://dwb.unl.edu/Edit/MB/MasonBruning.html (accessed 5 July 2013)

Nicol, D.J., MacFarlane-Dick, D. (2006) Formative assessment and self-regulated learning [online] Available at: http://www.tandfonline.com/doi/full/10.1080/03075070600572090#.UdiGu6xv5AI (accessed 5 July 2013)

Activity 3 – Automatic feedback generated from electronic assessments

The use of interactive on-line formative quizzes in mathematics – J Ekins (2007)

The UK Open University offers two mathematics programmes via distance learning. Both have tutor-marked assignments and computer-marked assignments (which consist of multiple choice tests).

I think the feedback is useful especially in this subject as it provides immediate feedback which fosters interaction between the learner and the study material – as tailored feedback to the answers are provided. The learner can respond immediately after each attempt (multiple attempts are allowed). A “hint” option assists those who need extra help. The broader range (design) of question types assesses more skills as well as, makes it more interesting for the learners.

The feedback provided after successive attempts, gives more detailed hints and appropriate study advice, it also provides practice for summative assessment.

Feedback from actual users (used to evaluate the quizzes) “thinking aloud” in video format, showed the quizzes stimulated learning. The learners enjoyed, followed-up on references and read the feedback. I think there is value in receiving feedback and to which you can do something about.

I think the benefits lie in the “shorter quizzes with detailed tailored feedback” – as mentioned but also in the immediate feedback provided.

A disadvantage is that the authoring of the quizzes is more time-consuming.

Improving the feedback?

I would invite options/feedback/suggestions from the learners to discuss their ideas.

Teachers could “revise” the feedback/programme to identify where the weak areas exist and improve accordingly.

Ekins (2007) mentions more need to be analysed e.g. “how different types of question and feedback stimulate learning”.

Week 21 – Activity 1 Assessment for learning: Beyond the Black Box – Assessment Reform Group, 1999

Previously the focus was on testing students, assessing them in increasing the amount of testing – yet no evidence exist that this will enhance learning.

This paper focuses on assessment (shown in new research) used on a regular basis in the classroom, is key to better learning. Learners should have ownership in their own learning – understanding the learning goals to which they are working towards which importantly include, motivation and skills in achieving success and understanding – which will lead to successful lifelong learning.

Assessment for learning links to learning – enhancing the effectiveness of how assessment is performed to promote learning, which can improve the performance of learners, even more so for lower-achieving learners.

Five key factors determine the learning through assessment:

·         Give effective feedback to pupils

·         Involve pupils actively in own learning

·         take results of assessment into account and adjust teaching

·          recognition of influence of assessment on the motivation and self-esteem of pupils

·         allowing learners to assess themselves and understand improving themselves

A few inhibiting factors are mentioned e.g. a tendency for teachers to assess quantities and presentation, attention to marking and grading, comparing pupils with other pupils, feedback of teachers used to serve social and managerial purposes, teachers unaware of the needs of pupils for learning.

To progress in their (learners) further learning “Assessment for learning is appropriate in all situations and helps to identify the next steps to build on success and strengths as well as to correct weaknesses” (p. 7)

Characteristics of assessment that promote learning are:

• it is embedded in a view of teaching and learning of which it is an essential part
• it involves sharing learning goals with pupils
• it aims to help pupils know and to recognise the standards that they are aiming for
• it involves pupils self-assessment
• it provides feedback which leads to pupils recognising their next steps and how to take them
• it is underpinned by confidence that every student can improve
• it involves both teacher and pupils reviewing and reflecting on assessment data

In the context of lifelong learning, it is important that learners should be aware and able to direct their learning for themselves.

Teachers should assist the learners with assessment in learning using the following methods:

• ·         observation which includes listening
• ·         questioning on an open basis
• ·         setting tasks requiring certain skills and ideas
• ·         invite communication from learners
• ·         discussing words and the use of it

H817 – Activity 21 – Conduct an evaluation and respond to evaluation of your project

 

What have you learned from the evaluators’ comments on your project?

I’ve learned that the interactivity component plays a major role to the user. In our project (Group A’s) site: Exploring a local history site using mobile technology and social web tools, we’ve focused specifically on a Treasure Hunt, using an App. It is important to keep in mind that the user desires the effective performance of an application – which implies that users, and in our case, the visitors’ needs should be anticipated in detail. Unless the information, tools, functions are not available the visitor becomes frustrated, therefore land marks and a HELP function with steps of guidance concentrating on the user who may experience problems, are essential. The visitor should be able to, without leaving a specific screen, be able to invoke help from any other screen. 

What have you learned from evaluating other projects?

Whilst evaluating the project of Group B: Digital diaries, I found the user interface design usability as being very important, as the user, in this case the student nurses’ attitudes and in the end, their learning, is the main issue to be considered in the designing process. People/nurses are not necessarily experts and can get easily frustrated with complex and/or no visible navigation e.g. “How to create a Google account” or the Blogger screenshots. These nurses are usually quite busy with a lot on their mind and it will be unfair expecting them to know how to navigate in an environment which they are not familiar with – they need to be comfortable with the minimum navigation with the maximum ease of use. 

What did you change and what more would you change in your project, in light of these new insights?

I would like to make the following changes:

·         to provide more detailed instructions especially in the initial stages

·         include a larger map of castle

·         needs to add to the Prototype e.g. questions can vary based on age, language and difficulty

·         show smart phone platform in prototype. Staff at castle should be able to assist visitors if problems arise

·         flexibility of use of App needs more clarification in prototype

·         flexibility should be available to switch to other elements of the App while doing the Treasure Hunt

·         questions should be selected carefully so that the site becomes relevant to the visitors, should be phrased with regards to recognition rather than recall. Example questions should be included to indicate to recognition rather than recall

·         change the prototype to include in Activities (general page) mentioning the certificate and picture prizes  as motivation to complete the task – stimulating active learning. This should be indicated earlier on in the App. The same applies to the social media – to be mentioned earlier.

·         relevance of project site for visitors needs to be highlighted

·         Learning theory – the learner should be able to add to his/her exploration of the castle. Goals, objectives and outcomes need to be indicated clearly in prototype

 

What are the advantages and limitations of this evaluation process?

The evaluation process which is a quick and cheap method gives valuable ideas to the designers how to improve the user interface and it can be done before the actual users use the system. As the usability problems identified usually relate to aspects of the interface, it can be shown quick and easily e.g. the layout of the information or use of colours.

In applying the heuristics effectively, an expertise is required from the evaluators especially in a specific domain. In cases where the evaluators don’t form part of the team of developers, problems may hinder communication e.g. where they are not aware of why specific decisions were made. The heuristic process focus more on problems than on providing solutions

What does the heuristic evaluation process neglect?

The heuristic process provides recommendations to improve the design after identifying usability problems. However, usually the interface functionality and design issues are emphasized and not the interaction which takes place between the user and the product – thus, heuristic evaluation should enhance the users’ experience. Usually the designers receive feedback on the problematic aspects of the design and not regarding the more positive aspects.

Reflection – Block 3

The process for our project is shaping up but it is an issue in our team to get together at the same time.
In the past week we’ve concentrated mainly on the activity in creating our storyboards. I’ve created my storyboard in Lino and added it to the project site – a screenshot and a link. I’ve enjoyed this activity. It reminded me of Yishay Mor’s video where he explained linking the personas and targets, linking them to illustrate connections. Yet in the storyboard it is more about how to organize potential activities for learners/visitors and get feedback in the design.  As there is only one other storyboard added to the project site, I’ve commented on it. Initially I’ve also started to contribute to the collaborative lino started by one of our group members but then rather contributed to the Google template which is now added to the project site – to which we’ve contributed collaboratively. Currently we are contributing to the Features and notes site but I find the OpenDesignStudio not always easy to use – a bit frustrating, e.g. I tried to add in the “Features and notes” are of the site ut didn’t manage – find it not user friendly.
Another issue is the roles for team members were identified but I think we have different understandings of what it implies. I have the role of an information manager – to me all the templates are already in place where we add and contribute our information – thus my role in this capacity is less active. At my workplace I work with information in databases, I don’t create databases.

H17 – Week 17 – Reflection

In the past week we’ve added to our factors and concerns and force map. We are also in the process of narrowing down on what the challenge should be – trying to be more specific. The indication is that we will rather focus on the virtual aspects – Learning history through the site visit and beyond by using mobile technology and social media to week-end tourists visiting the Bolsover Castle. Although we want to focus more on a digital experience for the visitors, I think the creation of a community feeling will bring more direct contact with our intended audience.

An invitation to a webinair on creating digital experiences, made me realized that we should increase on our engagement and inversion in designing /delivering digital experiences in our project which should add value in reaching the learning objective. This would be relevant to our project especially as we will focus on the virtual experience of the visitors. Our interactions in designing this learning activity should be coordinated and all the pages/sites should perform as though they are one. This will add to the visitors’ digital experience in visiting the castle and contribute as an inspiration to interact in the digital activities offered on the site and also inspires others.

In reading the pedagogical theories and technology-related theoretical frameworks, I’ve thought that we should focus more on the use of relevant research in designing learning activities. A lot of research is produced and disseminated but is there support for the use thereof? The authors Campbell and Levin (2013) states research use requires action informed by evidence and requires a shift from the general to understanding research implications applied to practices at individual, organizational and system levels.

Reference:

Campbell,C, and Levin, B. 2013. ”Building the capacity to use research in education requires a sustained strategic and systemic effort”, Impact of Social Sciences [online] Available from:   http://blogs.lse.ac.uk/impactofsocialsciences/2013/05/29/building-capacity-to-mobilise-knowledge-and-use-research-to-improve-education/  (accessed: 30 May 2013)     

 

STARR: a techno pedagogical case Authors: Hazari, S.,

STARR: a techno pedagogical case

Authors: Hazari, S., North, A., Moreland, D. (2009) “Investigating pedagogical value of wiki technology”. Journal of Information Systems Education, V. 20, no. 2 [online] Available at: http://piedmonteddathenscohort.pbworks.com/f/Investigating+Pedagogical+Value+of+Wiki+Technology.pdf (accessed 25 May 2013)

 Situation

This study investigated the pedagogical value of wiki technology (Web 2.0 tool) as a tool for teaching and learning. The relationship with factors were identified which may potentially improve outcomes of the learners. The four factors, used as an instrument include: learning/pedagogy, motivation, group interaction and technology. Reliability and validity were used to assess the perceptions of the learners of the value of wiki technology – using four hypotheses. The theoretical framework is based on constructivism and the engagement theory. A parallel with our project is in using blogs (Web 2.0 technology) to reflect and support our learning, as well as wikis. Another parallel: we engage in the collective sharing of items thereby creating a virtual online community – actively collaborating.

Divergence: The four hypotheses, namely: previously web development experience, previous work experience, gender and age were used with a PVW score (pedagogical value of wiki) to test the hypotheses – not used in our project

Task

The designers tried to encourage instructors to explore the potential benefits in using wikis – to promote collaboration in group assignments and make students comfortable in using the Web 2.0 tools. Parallels to our project: the design of an effective environment promoting collaborative learning, the creation/changing and tracking of collaborative content – easy and quick start and expanding of sites, easy to see how much each individual in a group has contributed, the development of skills associated with teamwork and sharing ideas. With regards to performance expectations – both the process and the product used by team members, are considered in developing the final outcome of the project. Also, the use of a constructivist approach which incorporated pedagogical tools which includes e.g. multiple perspectives, encourage ownership in the learning process, and the engagement theory – encourages collaboration and student engagement via the use of technology tools and systems

Actions and results

A review of literature on educational uses of wikis was performed.

Items were developed for each dimension – listed and categorized.

A EFA (explanatory factor analysis) to hypothesize the four factors was used.

Questionnaire items were developed. Questionnaire included the four factors: overall learning/pedagogy, motivation, group interaction and technology

The tool: Wikispaces was used – orientation given to students and a video developed by the instructor. A wiki assignment was chosen with a collaborative aspect and group interaction, to critique journal articles (a common article and a management consultant case report).

 An online survey was used.

The survey was pilot-tested with a small group.

There were 70 respondents to the questionnaire – include open-ended comments

The open-ended comments by the students on the wikis indicated they didn’t find the wiki space difficult to use.

Limitations prevented the use of confirmatory factor analysis in confirming or refuting the solution of the four factors. Inter correlations in factors were high, probably because the sample size was not large enough. In addressing the limitation, the scale would be considered unidimensional until more work is completed validating individual constructs. The study offers a basis for further research. The study also provides insights to educational staff who want to explore wikis in a collaborative environment in teaching and learning.

Reflections

The insights of the determinative indicators provided by the study will be useful to educators measuring the pedagogical value of wiki technology. As Web 2.0 tools are widely used by learners, further research of the assessment of these tools with the scale developed in this study, should be encouraged e.g. measuring the pedagogical value available in the emerging technologies. In our project, we use blogs to reflect, as well as wikis – encouraging social collaboration.

A gap exists to develop new standards in accommodating interoperability between Web 2.0 tools and different course management systems and also, in sharing learning objects and social pedagogy tools in the different environments. In using wikis – and similarly all Web 2.0 tools – students build on the knowledge of peer students and become “participatory communities” (Hazari,2009)

The need exists to move from tutor-delivered teaching to student-facilitated learning.

 

 

Activity 12 – ReflectionAfter completing Activity 12: review

Activity 12 – Reflection

After completing Activity 12: review a case study or theoretical framework, I see a clearer picture. I reviewed: Investigating pedagogical value of wiki technology. As wikis are a component of the Web 2.0 technology tools, which is a tool for teaching and learning – it provides collaborative features and learning opportunities which we also apply currently in the forum and in our team website doing our project where we are expected to use social media tools and mobile devices.

We are contributing in a collaborative environment where the sites were easy and quick to create and expand. Thanks to MC and everybody else! It is also easy to see each individual’s contribution in the group. However, we developed new skills associated with team work and share ideas. Looking at the performance expectation – both the process and product are considered in developing the final outcome of the project.

This study also provides insights to educational staff to explore wikis in a collaborative environment in teaching and learning. There is a shift in moving from tutor-delivered teaching to student-facilitated learning.

Activity 6: Create and share personasWhat are the

Activity 6: Create and share personas

What are the similarities and differences between the personas you proposed and those that your team mates have contributed

My two personas: Mary Dean and John Wilkens

-The similarities:

Persona Mary is 61 years old. She shares similarities with a few other characters as there are 5 more personas within her age group – including 45, 52, 65 and 75. Mary is comfortable in using a computer as well as the Web 2.0 tools but impatient with her phone technology, yet she uses her mobile phone.

Persona John Wilkens aged 52 years, also has technical computer skills, uses fb and LinkedIn and is comfortable using his mobile. Although some of the other personas in this age group use the technology, they also experience difficulties with the technology but they have access to a device.

Mary is an energetic person and has good relationships with people. John also has excellent customer service skills – he is good with people. Some of the other personas are specifically interested in history but about all the personas in this age group want to experience more adventure and see new places – to enjoy their lives.  

-Differences:

In contrast to the middle age and retired age group, there are more, younger personas which include: 2 nine year old girls in primary school, 4 teenagers (3 aged 16, 1 aged 17) and 1 persona aged 22 years old. Most of them are more familiar with the technology tools e.g. they own a tablet, iPhone or smart phone and they enjoy the Web 2.0 tools e.g. fb, twitter, a blog.

Not all the characters are living near the historic site: Bolsover Castle – some live near the Castle but some personas will have to travel quite a distance to visit the Castle. With regards to the language – 2 personas are second language learners.

Activity 6 asks – What have you learned from this exercise about your design space and about the design process?

I think the design of 2 personas by each group member presented the opportunity to discuss and comment on the different perspectives of the personas which deepens learning –involving reflection. I think it challenges my learning in the sense of improving knowledge and a better understanding and to be able to choose appropriate technology tools and social media in accommodating the group which we are going to select.

I enjoyed this activity – creating personas.