Yongmei Bentley & Habte Selassie, University of Bedfordshire

Introduction

E-learning continues to develop rapidly supported by increasing sophistication of information technology and by better understanding of how to make content and delivery more effective. Moreover, new forms of e-learning support system are being introduced to higher education institutions in an effort to meet the student-centred learning paradigms recommended by UNESCO (UNESCO, 1998). The creation and implementation of effective quality assurance for such learning processes has been identified as one of the most challenging tasks. Jara & Mellar (2010) and Martínez-Argüelles et al. (2010) point out that the collection of student feedback should be a central part of strategies to monitor the quality and standards of teaching and learning in higher education institutions for both conventional learning and e-learning. Jara & Mellar (2010) note also that while research into e-learning abounds, studies that focus on the effectiveness of the provision of e-learning are limited, and that this is a gap to be filled.

This article reports on the evaluation of, and the consequent changes to, the global e-learning MBA programme from the University of Bedfordshire. The research was conducted by three members of the MBA team - two Senior Lecturers and the E-learning Development Manager. The aim was to investigate the learning experiences and perceptions of the students and to evaluate the effectiveness of the e-learning support system.

Literature Review

The rapid growth of online academic course provision worldwide has changed the learning environment for both students and teachers (Landry et al., 2008; Lapointe & Reisetter, 2008; Williams & Williams, 2010). In terms of quality assurance, Zygouris-Coe, et al. (2009) note that instituting a well-structured quality assurance process can be expensive and time consuming, but can be worth the effort. For example, the study undertaken by Kidney et al. (2007) supports this. They state that the merit, quality and success of the e-learning programme they investigated were mainly due to the proper application of the quality assurance strategies. Moreover, Rajasingham (2009) notes that new educational paradigms and models that challenge conventional assumptions and indicators of quality assurance are becoming possible with the help of the increasing sophistication in information technology.

Deepwell (2007) makes a distinction between quality assurance and evaluation, and views evaluation as an instrument of quality enhancement rather than quality assurance. While the measurement of student feedback is recognised as an important component of quality assurance, there have been mixed reports as to its effectiveness. For example, Gurău & Drillon (2009) state that analysing users' perceptions regarding an e-learning system can provide valuable data to evaluate and improve its functioning and performance. On the other hand, Jara & Mellar (2010) report from their research findings that student feedback was not always fully adequate to support quality enhancement. Researchers are cautioned that they will, therefore, need to make judgements in this area. Finally, Lapointe & Reisetter (2008) suggest that the new reality of online learning demands a reassessment of our understanding of what makes for the most productive student engagement. The findings below are intended to help move towards an answer to this question.

Overview of the global MBA Programme

The MBA programme investigated here is a two-year part-time MBA delivered in the UK, and at partner overseas universities in Oman, India, Germany, Poland, South Africa and Switzerland. The programme is mainly delivered online via the UK University's e-learning learning support system, but also includes periods of face-to-face teaching. For the overseas sites such teaching is delivered by the lecturers from the UK University who make regular visits to the overseas sites, and by lecturers of the counterpart local universities in partnership with the University's course leaders.

Typically the blended learning for this programme consists of four weekends of face-to-face lectures and workshops at the local Business School and ten voice-over-the-Internet live classroom sessions in each semester. This is supported by the provision of extensive online learning materials, easy access online discussion boards for collaborative learning, an e-library and many other resources. The revised course structure of this programme, based in part on the findings from the first and second rounds of the questionnaire survey, consists of five modules in the first year and five in the second:

Year 1: Leading and Managing People; Accounting for Leaders; Marketing Products and Services in a Dynamic Environment; Mobilising Creativity and Innovation; and a Leadership project.

Year 2: Strategy; Operations and Project Management; Entrepreneurship and Small Business Management; Thriving in a Competitive Global Context, and an Integrated Management project.

Research Method

The primary research approach for this study was longitudinal survey (Dillon et al., 1987), using semi-structured questionnaires to collect data from respondents. Three rounds of survey were conducted across selected course sites at different points in time, thus enabling examination of the changes that occurred in the attitudes and satisfaction levels of the students.

Survey forms were either delivered online or handed out in class. Table 1 gives an overview of the three rounds of questionnaire survey conducted over a period of three years. There were 149 valid responses out of 290 students taking the course when these surveys were carried out, representing a 56% response rate.

Table 1: Overview of the questionnaire survey (N=149)

Discussion of key Issues

For evaluating this blended-learning MBA programme, the evaluation process was designed to measure the quality and effectiveness of technology-enhanced teaching, and the learning experience of the students on the course. The investigation focussed in particular on the areas of: course management, learning and teaching, online learning content, assessment, the learning support systems, and students' overall experience of taking this course (see Table 2 for specific areas surveyed).

The first survey round

As Table 1 shows, in the first survey round 63 completed questionnaires were received from four MBA centres - Oman, India, UK and Poland. Survey results are presented in Table 2.

Overall, the results showed a fairly high level of satisfaction with the programme, with an average of 62% of the respondents perceiving the average of 29 aspects of the course being measured as either 'excellent' or 'good', versus 25% perceiving this 'average', and 13% as 'poor'. In terms of service quality, 22 out of the 29 aspects surveyed showed a combined percentage of 'excellent' and 'good' to be greater than the combined percentage of 'average' and 'poor'. Six results had these percentages about the same, and one (item 17, on Voice Café) had this percentage significantly reversed.

Table 2: Results from the first survey round (% response by category)

Q	Survey Items	Excellent	Good	Average	Poor
1	Registration process/ administrative support	31	55	11	3
2	Teaching at Induction/assignment support	20	60	16	4
3	Teaching at local institution	18	58	24	0
4	Module Handbooks	22	47	27	4
5	Local tutor support	15	45	33	7
6	Module information from local institution	9	67	22	2
7	Local support facilities	9	51	26	14
8	Textbook availability	18	33	22	27
9	Usefulness of CD ROMS	20	44	22	14
10	CD ROM materials/ weekly online material	20	44	22	14
11	Reading materials on BREO	26	38	27	9
12	Learning resources	22	45	22	11
13	UK tutor support	11	51	33	5
14	Slides on BREO	25	44	25	6
15	Audio/video clips	3	45	21	31
16	Relevant website links	15	43	29	13
17	Voice cafe/WIMBA	11	26	25	38
18	Course/module response	2	47	42	9
19	IT training and support/ online guides	11	35	42	12
20	Assignment instructions	16	53	27	4
21	Assignment submission procedures	16	55	13	16
22	Assignment feedback	15	36	29	20
23	Referral procedure	13	57	23	7
24	Failure procedures	18	39	17	26
25	Academic offence procedure	16	44	16	24
26	Social networking opportunities	9	38	33	20
27	Overall experience with the tutors	24	51	23	2
28	Overall experience of online support	8	49	36	7
29	Overall experience of undertaking MBA	20	46	29	5
	Average of responses shown:	16	46	25	13

As this was a semi-structured questionnaire survey, the respondents were given the opportunity to offer comments where appropriate in order to provide more detailed information on the topic areas being investigated, and to encourage suggestions for improving the course content, delivery approach, and support systems. Overall, the comments indicated a need for more support for the students' independent learning process. Suggestions generated by this first round of survey, combined with feedback from UK and local staff, were then considered. A detailed discussion of the changes that resulted is excluded here due to restriction on article length, but is given in Bentley et al. (2010).

The second survey round

Subsequent to the course changes a second survey round was conducted in 2009 covering Oman and India, with 37 valid responses received. This second survey had 19 questions the same as in the first round. The primary purpose was to measure any improvement (or otherwise) resulting from the course changes, and to uncover additional issues. A comparison between all three survey rounds is given in Table 3. For consistency, question numbers are those of the first round.

An examination of the results from the first two survey rounds showed that 14 out of the 19 topic areas investigated in common across the surveys had improvements in the second survey in the 'positive' response category (% of 'excellent' plus 'good'), with nine areas showing a gain of 10% or more. Significant improvements included: teaching at induction by the UK lecturers; the quality of module handbooks; UK tutor support; audio/video clips; ICT training and support; social networking opportunities; overall experience of online support; and the overall experience of undertaking the MBA. However, it was surprising to see that despite effort for improvement after the first survey some areas did worse, including teaching by local institutions, and the referral assessment and the failure procedures. This indicated that further effort was needed to improve the students' experience, though in part it might be that later students were more demanding, especially for services related to certain university procedures. Overall, the average percentage of respondents who rated the course as 'excellent' and 'good' on the topics included in this comparison improved from 63% in the first survey to 74% in the second. This was seen as an encouraging finding, and justified the extensive work by all the parties involved to improve the design and provision of the course.

 

Table 3: Comparison of the results of equivalent questions from all three rounds (% response by category)

The third survey round

The third survey round in 2010 covered Oman, Germany and UK, with 49 valid responses obtained. As can be seen in Table 3, there were 11 questions which covered the same areas as in the first and second surveys.

The programme had been further changed following the second survey, but not radically in most areas, so for many of the questions that were the same it was not surprising that the results were similar. For example, on the key question that asked about the students' overall experience of the MBA, there was a marginal improvement (from 79% to 81%), but unlikely to be significant. Some other findings were disappointing. For example, evaluation of the teaching at the local partner universities was again rated lower than previously. This fall in rating was partly counterbalanced, however, by a large jump in the rating of quality of local tutor support, reflecting the significant effort that had been put into improving this area.

Respondents were again encouraged to make comments where they felt this could improve future course provision, and all replies were useful. Most were fairly straightforward, and not surprisingly some respondents asked for additional resources (for example, more tutor time or faster response on assignment marking) that were unlikely to be met without a change in level of staff provision.

A further issue related to the perennial question of assigning the proportion of group work to Individual work. Nevertheless, overall, the third round of the survey (and the two preceding ones) gave a generally positive picture of how the students felt about the course, with some highly complementary remarks being made when students were asked to summarise their general satisfaction with the course. Additional analyses of the survey findings are on-going which will help further improvements to be made.

Reflections on the research

One of the main limitations was the change in sites surveyed at each round. Pragmatic considerations had led to these changes, including timing of when courses were offered, and occasions on which surveys could be carried out without interrupting teaching schedules. For this reason the comparability of responses between survey rounds needs to be handled with some caution. But set against this, in general the course was being taught to students of a rather similar level across all sites and with similar expectations such that it is felt that on balance useful conclusions could be drawn. Changes were also made to the question sets over time. This is not thought a major impediment to the overall value of the surveys, as most such changes were determined by straightforward factors, such as questions being no longer relevant, or where new questions were designed to measure course changes following a previous survey round.

Conclusions

The article reports on the evaluation of a blended-learning MBA programme provided by the University of Bedfordshire both in the UK and at a number of overseas partner-institution sites. The evaluation was a longitudinal study, involving a sequence of surveys of students' perceptions of the course. Findings from these surveys, plus other review procedures, were used to make changes in both course content and course delivery.

The research is contrasted with much of the research into the evaluation of e-learning courses which relies on data gathered just once, and where cross-sectional designs have been applied. By contrast, the longitudinal approach employed here enabled the charting of changes over time, thus enriching the process of course design, and monitoring the changing satisfaction of students and other stakeholders.

Overall, the survey findings indicated a high level of satisfaction with the MBA programme, and this satisfaction increased following the changes made. In particular, the findings supported the detailed choices by the providing university and the partnering institutions in the structure and content of the blended-learning approach adopted.

The findings give credence to the view that an e-delivery approach is well suited to work-based part-time MBA students who are 'relatively mature, already business-aware, conversant with information technology, and have access to the e-learning facilities and resources required' (Priestman, 2010). Furthermore, the findings supported the claim of Gurău & Drillon (2009) that student feedback can provide valuable data to evaluate and improve the functioning and performance of an e-learning system. By identifying areas for course improvement, this research has attempted to implement the concept of 'evaluation as an instrument of quality enhancement, rather than just quality assurance', as suggested by Deepwell (2007).

It is hoped that the outcomes of the evaluation of e-learning in this research have improved not only the quality and effectiveness of this particular programme, but may also help improve the quality and effectiveness of the teaching and learning processes of global blended-learning courses offered by other institutions across the world.

Acknowledgements

The authors thank the University's MBA team for their support and the students for taking the survey.

References

• Bentley, Y., Shegunshi, A. & Scannell, M. (2010) 'Evaluating the Impact of Distance Learning Support Systems on the Learning Experience of MBA Students in a Global Context'. Electronic Journal of eLearning, Vol. 8, No. 2, pp. 51-62.
• Deepwell, F. (2007) 'Embedding quality in e-learning implementation through evaluation'. Journal of Educational Technology & Society, Vo. 10, No. 2, pp. 34-43.
• Dillon, W. R., Madden. T.J. & Firtle, N. H. (1994). Marketing Research in a Marketing Management. 3^rd. Edition. Burr Ridge: Richard Irwin Inc.
• Gurău, C. & Drillon, D. (2009). 'Evaluating the effectiveness of an international e-learning system: The case of Montpellier Business School'. Proceedings of the International Conference on e-Learning, 2009, pp. 174-181.
• Jara, M., & Mellar, H. (2010) 'Quality enhancement for e-learning courses: The role of student feedback'. Computers and Education, Vol 54, No. 3, pp. 709-714.
• Kidney, G., Cummings, L. & Boehm, A. (2007) 'Toward a quality assurance approach to e-learning courses'. International Journal on E-Learning, Vol. 6, No. 1, pp. 17-30.
• Landry, B. J., Payne, D. & Koger, M. S. (2008) 'From 'chalk and talk' to online offerings: keeping pace with technology in education'. International Journal of Management Education, Vol. 2, No. 3, pp. 300-317.
• Lapointe, L. & Reisetter, M. (2008). 'Belonging online: Students' perceptions of the value and efficacy of an online learning community'. International Journal on E-Learning, Vol. 7, No. 4, pp. 641-65.
• Martínez-Argüelles, M., Castán, J. & Juan, A. (2010). 'How do students measure service quality in e-learning? A case study regarding an internet-based university'. Electronic Journal of e-Learning, Vol. 8, No. 2, pp. 151-59.
• Priestman, T. (2010) MBA Approval Document, University of Bedfordshire, 26 May 2010.
• Rajasingham, L. (2009). 'Breaking boundaries: Quality e-learning for global knowledge society'. International Journal of Emerging Technologies in Learning, Vol. 4, No. !, pp. 58-65.
• UNESCO (1998). 'World declaration on Higher Education for the twenty-first century: vision and action'. Proceedings of the World Conference on Higher Education, Paris, 5-9 October, 1998.
• Williams, M., & Williams, J. (2010) ' Evaluating a model of business school students' acceptance of web-based course management systems'. The International Journal of Management Education, Vol. 8, No.3, pp. 59-70.
• Zygouris-Coe, V., Swan, B., & Ireland, J. (2009). 'Online learning and quality assurance'. International Journal on E-Learning, Vol. 8, No.1, pp. 127-146.