Julian Rennie, Unitec Institute of Technology, New Zealand

Abstract

The Design studio learning system within New Zealand Tertiary Design Schools has a unique critique method, (often called 'The Crit'); The Crit event itself is rather a 'veiled' process and has been analyzed and written about extensively. There has also been some negative feedback from students that this form of critiquing process is not necessarily a good type of feedback process. Is there a method that protects the student's privacy related to his or her own design work and at the same time maintains the Design School's integrity of supplying reasoned and fair assessment within the wider Profession? A field trial scenario was designed and arranged with a group of volunteer design students, so each in turn, could sit-in and witness their own assessment / feedback session. This paper reports on this field trial, (timed to occur after the critique). This paper analyses this experiment, exploring the field trial responses, looking for links within a wider Educational literature base to the ground this 'Fly on the Wall' scenario within known pedagogies. NB. This scenario is not proposing to supplant 'The Crit,' rather the intention being in addition to it.

Keywords: student feedback, student assessment, learning transparency

Introduction

In response to the nineteenth century Industrial Revolution, the Ecole Des Beaux Arts, (School of Fine Arts founded in 1819), in Paris, set up an architectural educational system where the 'learning by doing,'(Anthony, 1991, p.9), replaced the existing apprenticeship system. Under this new system students were put into ateliers or studios, which were led by patrons or Masters. And the evaluation of the student work was done via a 'behind-closed-doors jury' system. The students got their marked work back with little or no comment from the assessment jury.

Today, studios within Design Schools around the world have evolved into being places which incorporate rigorous iterations of drawing and model making; in a process which Schön likens the design student to a 'reflective practitioner,' (1983). The review or critique of such studio generated work, (often called 'The Crit'), is the student pin-up of all their work, and then each student in turn stands up, and presents their work to both peers and the 'Jury,' (which often contains outside practising designers as well as the studio tutors). The assessment is done after the 'Crit,' in private, usually by the studio tutors only. The marks are then released publicly so the students can see how they have performed relative to their cohort. At Unitec, Auckland, New Zealand the studio tutors also write up a comments sheet for each student, this forms the formative feedback aspect of the process.

There has been a lot of discussion about the problematic nature of the 'The Crit' process from the student point of view, (refer to the poignant cartoons in Parnell et al., 2007), although it remains the cornerstone of the review system in Design schools. There has been little analysis of the associated assessment process, and its lack of transparency following on from 'The Crit'. A field trial was designed and the data obtained from this trial was analysed to look for linkages within a range of educational literature to evaluate this experiment.

The Research Project

The findings of this paper are based on a field trial set within Unitec, Auckland, New Zealand, which I piloted in 2007. The proposed field trial method was put to Unitec's Ethics Committee, upon receiving approval; a call for volunteers was made to a group of second year Bachelor of Landscape Architecture students. This field trial took place three days after a final studio Crit, which was the culmination of five weeks work designing and drawing up construction details for a proposed intervention on an urban site.

Each volunteer was asked to read and sign off an 'Agreement of Participation,' before the commencement of the experiment. Then, one at a time, each student entered the assessment room and sat behind the two tutors. Each student was allowed to watch and listen to the tutors, as they discussed that particular author's work. The volunteer was not allowed to speak during the assessment process.

Following the event, and related only to this author's field trial, each student was asked to verbally comment on something which was 'good' about the process they had just witnessed, and something which was 'hard' to hear about their work / performance, and any other learning's. This verbal data was triangulated via a written questionnaire, which was filled out anonymously, and in private, in a separate room. The questionnaire focussed on: how useful was this 'fly on the wall' event to them as a learner? How useful was it in developmental terms to them as a budding designer? The volunteer was then asked to compare this technique with other feedback types they had previously experienced. And, in addition, each student was also asked to rate this event on a scale of 1 to 7, (with 1 being the least useful and 7 being the most useful).

In line with Denscombe's concept of 'Content analysis,' which relates to 'hidden aspects,' (2010, p.282), within the text are sought out and analysed, the received feedback from the questionnaire seem to evoke four educational themes:

• Direct Learning
• Life-skill Learning
• Positivity Accentuated
• Deep Learning

Research Analysis

A lot of educational literature has been written around the area of traditional exams and assignment type assessment and the associated pitfalls of these methods as a way of testing learning. However not much has been said about proposing new ways of increasing the transparency of the current summative and formative assessment methods in the hope of enhancing student learning. This field trial is one such attempt to experiment in a real world learning situation and eliciting student feedback comments both good and bad. The student comments received from the field trial then became the data, which was analyzed and reflected upon whilst researching within a wide research literature base to try and ground the common threads and themes found within.

Direct Learning

The time taken to look at, consider and grade each students work (comprising four A1 sized sheets of detailed drawings) generally took about 10-15 minutes for each student. The volunteer could see and hear 'first hand' the tutors as they worked out the individual feedback comments and the associated grade. The assessment criteria, used to critique the learning outcomes were released at the start of the studio programme and were therefore known to each student prior to pinning their work up for the final Crit. Likewise, the grade descriptors were released within the School's Handbook at the beginning of the year, and were widely known throughout the cohort. By allowing the student to be in the marking room, in effect like 'a fly on the wall,' enhances the transparency of the assessment process for the author of the work.

The tutors discussed the work presented and its author candidly. The tutors spoke with care and respect, more so than normal due to the fact that the author was in indeed present, 'but it took surprisingly little time to forget that the student was indeed [overhearing the discussion],' (Francis, 2007). This aligns with Higgins, Hartley & Skelton who cite students asking that their feedback comments be 'more personal and direct, then it would be more helpful,' (Higgins et al., 2002, p.56).

During the process it was also apparent that the two tutors' own behaviour was modified, (compared to current 'behind closed doors' versions of this process, for example: no swear words were used, and 'gossip type chat' was not aired). The tutors discussion seemed to remain at all times animated yet professional, focussed on the actual work, establishing suitable feedback comments, and a matching grade. Sometimes, the tutors disagreed with what grade / feedback comments to append to the work, yet the transparency afforded the 'eavesdropper' to see and hear first hand how the tutors argued the various aspects back and forth till a decision was reached. These characteristics fall into line with some of Boud's thoughts about offering good feedback, namely: '[the tutor's did not use] fancy words or abstract language, [and they were] consciously non-judgmental,' (1986, p.31).

Care and consideration were shown in relation to both the feedback wording and the decided upon grade. 'It is up to the…[student]…to accept or reject them,' (Boud, 1986, p.31). There were no 'double word meaning,' or 'lost in translation' or other 'out of context' type issues, (as can occur via written language feedback techniques), this direct method shows the tutor's 'warts and all,' and with the procedure's transparency allowing the student to witness it all in real-time. As a couple of trial participants noted within the returned questionnaires:

Q5. It's more informative, you seeing the marking, you can't get better feedback.

Q5. Far more thorough and [in] depth…[than]…when you just receive a paper slip and can find it hard to understand where the marker is coming from.

This 'fly on the wall' type session could be seen as a type of 'Experiential Learning' (Kolb, 1984), a learning process that revolves around observing a concrete experience, followed by some reflection, leading to the formation of abstract concepts by the learner and perhaps later testing these concepts in new situations.

In addition, it is apparent that this technique can expose the so-called 'hidden criteria,' aspects that tutors may have. For example: sometimes a tutor did not 'visually like' the actual work, these type of personal opinions were aired openly in the discussion with the other tutor, who in turn, made sure that this did not cloud a fair and reasoned assessment.

By the time students get to tertiary level education they have been subjected to various assessment methods, perhaps the transparency afforded by this 'fly on the wall' technique could go some way to dispel some of the myths they have about how their grades and feedback are established; for further discussion about 'myths related to learning,' refer to Zmuda (2010).

Life-Skill Learning

One field trial participant noted:

Q.4 More information about what its like in the professional realm.

This comment seems to acknowledge that the student knows that learning the goes beyond the mere studio setting. To get a glimpse, as it were, into the landscape architect's office: hearing the tutors discuss wider issues provoked by their work, perhaps seeing them disagree over issues and hear their resolution, all these experiences would seem to be valuable insights for a budding designer.

Another participant noted, how pleased that her personality and involvement in the studio were noted and taken into account during the assessment — this was a surprise to her, as she herself says:

Q.1 It was a unique opportunity to learn more about things discussed in marking an assignment other than the technical requirements of the brief – i.e.

• Attitude
• Scale
• Commitment
• Interest in Landscape Arch., etc.

For the tutors it seemed natural to think about this student not only as a potential professional designer but also as an individual, and to pass comment on her other attributes, as a human being seemed normal and valid. 'Assessment is not just about measuring knowledge or skills, (summative), nor about correcting and directing learning (formative),' (Havnes & McDowell, 2008, p.210). That the personality and character of a person could come into play within the assessment process was a refreshing revelation for the above student, (in fact this does currently happen, albeit unknown, in the behind closed door version), but the transparency afforded by this 'fly on the wall' technique allows the student to witness its use, which in turn may benefit their self-esteem.

In addition, Kerka, describes how the job market has changed markedly, that the days of a 'job for life' are long gone, rather what is required is 'individuals should consider themselves a collection of attitudes and skills,' (1997, p.1). Given this fast paced and ever-changing world Barnett also promotes a learning style within such an unknown future might be deemed 'to encapsulate [the] right relationships between persons and the changing world in which they [end up being] placed,' (2004, p.259). The transparency afforded by this eavesdropping technique in hearing first hand those skill qualities may well be a step in that direction for that listening student to hear something about their own attitudes and reflect upon how such knowledge might be used in modifying their own behaviour.

Positivity Accentuated

A field trial participant noted:

Q.7. It's nice to hear positive feedback and good to know that most of the issues that arise with the work have been considered.

As tutors, we always try to assess from a positive angle, we are always trying to find something worthwhile about each and every student's design work, and there always is something to praise, (even if it is the quality of line-work within the drawings). When there are problems or things haven't been quite resolved, the tutors tended to wonder aloud: 'had the author thought about such and such?' Rather than saying taking the negative approach: 'that looks weak' or that: 'wouldn't work.'

Industry based 'Strength Based Learning' or 'Life Based Learning,' (Sharon et al., 2006), involves a similar method to this 'fly on the wall' model. These strength based techniques can be best described as '[an] emerging paradigm for organisational change is based on asset or strength based approaches for individual and organisational growth and change,' (Sharon et al., 2006, p.4).

'Strengths-Based Development' also has similarities to this 'fly on the wall' concept. As a method it can be described as 'rather than spending time helping their associates become 'well rounded,' many…managers have instead invested time in learning about individual talents of each of their associates, and managing with those unique talents in mind. This concept not only applies to managers, but to educators, [and] students…' (Hodges & Clifton, 2004, p.256).

Broadfoot transcribes similar thoughts about the power of positivity, which she coins as 'learning power.' To her, learning power means: 'having positive views about one's own capacity to learn and a degree of confidence and resilience which enables one to work through the challenges and setbacks that genuine learning inevitably presents,' (2008, p.210). Overhearing positive things about oneself is always welcome regardless of where one is at in life, 'a positive reinforcer often is seen as something pleasant, desirable, or valuable that a person will try to get' (Miltenberger, 2012, p.66).

Deep Learning

From this field trial the idea that 'deep learning,' (as opposed to 'panning' or memorising information), is in fact sought by the student learners, was elicited by following feedback comment:

Q.7. I think it will help to make one's work evolve or develop more.

According to Havnes & McDowell, 'deep learning' 'means taking an active approach to learning; trying to 'make sense'; using a variety of ideas and approaches; and being able to reflect on learning and act on one's reflections,' (2008, p. 210). Similarly this concept was evoked by the following participant responses:

Q.1. It was helpful to watch someone try to navigate between drawings.

Q.5. It proves the readability of your work.

Design, as a field of endeavour, is not right or wrong type subject, it is subjective and the learning required to grasp its multiple faceted nature is part of a long progression. For example: one finds out about the history of design and its social impacts, the techniques and methods required to 'put something together,' its contractual and legal aspects — these are just a few examples of some of the issues that need to be assimilated over a lifetime by the designer.

Within tertiary education, replicating the professional realm remains fraught with difficulties, as Teymur quips: 'the Design studio is about 'learning' and the Office about 'earning,' '(1992, p.36). 'The raison d'être of a higher education is that it provides a foundation on which a lifetime of learning in work and other settings can be built,' (Boud & Falchikov, 2006, p.399). By allowing the student some transparency to such 'professional type' of activities, (albeit watching and listening), then perhaps students can be exposed to a greater depth of experiences.

Another feedback comment was:

Q.5 Compared to Crits, I [found] it more useful as it [was] more private.

From this, it would seem to reinforce of the Design School's ongoing and abiding intention to protect each and every student's private realm in terms of their assessment. Within this 'fly on the wall' scenario, the tutors ensured that each student's work was not compared or 'benchmarked' with other work from within the cohort. Also, it would appear that the listener was not too concerned about hearing the feedback comments alone, (knowing that no other peers where present). Compare this with 'The Crit' situation which is more public, and often tutors 'dumb down' the feedback, so as not to put down that student in front of his / her peers. This eavesdropping type session was an intimate event, with close scrutiny of the individual work, a full and frank discussion took place in a wholly transparent manner by the tutors, who focused on the various strengths, (and any areas that need strengthening), related to the work and its author. As Knight & Yorke, state: 'Understanding, (as a term, [is], preferred to 'knowledge' because of its implication of depth), is the key outcome of higher education,' (2003, p.9). The design process has to be engaged with, it has to be encountered, felt out by trial and error, it is simply not just knowing in the sense of: 'who was the first man to land on the moon?' And, as Sullivan, adds: 'Research has indicated that deep learning is linked to providing a stimulus in a way that leads students to focus more clearly on their particular topic, and then giving them the opportunity to reflect on and respond creatively to their chosen topic so that they can claim ownership of it,' (2002, p.127). This idea of 'ownership' or preference of a way of working around and through a design problem is important at this time for a student, because what the tutors are really trying to do is help students to find their own individual 'voice.' As Schools of Design and the wider Profession are always trying to encourage individual responses to new and evolving social issues of the day.

What Was Not Helpful About This 'Eavesdropping' Experiment?

The received comments from the participating student volunteers centred on logistical matters: time waiting, timetabling, and time taken, these were totally valid and are also vitally important when considering the expansion such an assessment / feedback event to encompass an entire studio class.

Discussions and Suggestions For Practice

Rather surprisingly the field trial feedback responses and grades were overwhelmingly positive:

Q.6 An in depth analysis on paper and verbally of how the work was marked.

Q.6. Do this, because it keeps you in touch with the reality of your work, instead of kind of forgetting about it once it's handed in.

Q.7. I think it is better than handing it in, waiting 2 weeks, then receive a grade, because this time gap separates you from your work and your grade. The direct marking of your work is of greater benefit.

Associated with all the comments received from the students, the overall support rating from the survey was 5.91, (out of a possible maximum of 7). My fellow staff member also backed up the student claims; he too thought it was a worthwhile experiment and said, 'it was easy to adapt into the role,' (Francis, 2007).

In an effort to make it a 'win-win' type scenario: a way of further refining this technique could be to use a pre-written feedback comments sheet and during the tutor discussions, those comments that don't apply are struck out, (with a large felt-tip), and any additional comments are written on the sheet by hand. The tutors then sign it, a scan is then made of the final comment sheet version, (this scanned version then becomes a copy for the School's records), and the original sheet is given to the student as he / she leaves the room. Assessment, mark allocation, (un-moderated mark only), and feedback all done in one neat package.

Limitations of This Study

The findings of this field trial should be treated tentatively. This was a small sample of people, (6 out of a possible 22). The end-of-year timeslot used to trial this experiment may also contribute to such a low turn out, (students probably just wanted to get away to their summer recess). In addition, the types of students who volunteered to partake in the field trial were the 'keen' students of the class.

It could be argued that we should have had a control group to compare this experimental trial, (however this would have had ethical implications, for example, we would have ended up with two parts of the one class being assessed via different methods, and how does one equate the two assessment methods across that group of students?). However, this was marking in 'real time' and we still had to assess the other 16 students. Which we did, behind closed doors, and it should be noted: that we as tutors slipped into our old habits of gossiping, 'slating' student work and other bad habits whilst we graded and worked out feedback for each student.

Another limiting aspect would be: Does this scenario work with the assessment / feedback of a student who is about to fail the course? (Or, does not get as good a grade as they thought they would get?). How would the comments / grade be received in such an intimate environment? Would that student be able to remain silent, (or possibly burst into tears)?What pressures would be put on the tutors in handling such delicate situations in front of the student? These scenarios remain untested and leave the way open for more and larger trials.

Having said all of the above, I feel the warm support shown by the volunteer participants and associated staff, (together with the high student ratings supporting the concept), makes me confident that this 'fly on the wall' technique has some potential.

Conclusions

'Students can, with difficulty, escape from the effects of poor teaching, they cannot, (by definition, if they want to graduate), escape the effects of poor assessment,' (Boud, 1995, p.35). Until Tertiary Education facilities switch to formative assessment only, this 'fly on the wall' field trial shows the positive affects that transparency can bring during the assessment process. It not only affects tutor behaviour for the better, it allows the student private yet direct access to positive feedback comments about their work and themselves. These witnessed comments could in turn activate deep learning and foster critical thinking within the student.

However, this was a small sample of respondents, so the paper's intention at this time is to expose this idea to the wider educational profession and apply for funding within the Faculty to do similar field trials with larger sample pools, in the hope of building upon this experiment in student assessment and learning transparency within Design Schools.

As interviewed and quoted by Anthony, 'Architect: Charles Moore says: 'One of the legacies of the Beaux-Arts that we still have with us is that secrecy, the business of retiring into a room where nobody could see what you were doing…To keep people from copying each other is presumably why this secrecy was set up. Yet so much of [design] practice is indeed copying each other, building on each other's ideas, and keeping other people interested in what's going on,' ' (1991, p.204). This paper attempts to show how a student learner can be a 'fly on the wall' for a time, inside that room, and potentially glean something meaningful about themselves, their work, their potential, and something more about the Design process by making their own assessment and feedback event more transparent.

Acknowledgements

I would like to thank Wendy Baker for her help in mentoring me through this field trial.

References

• Anthony, K. (1991). Design Juries on Trial. The Renaissance of the Design Studio. New York: Van Nostrand Reinhold.
• Barnett, R. (2004) Learning for an unknown future. Higher Education Research & Development, 23 (3), pp. 247-260.
• Boud, D. (Ed.). (1986) Implementing Student Self-Assessment (HERDSA green guide ; no. 5).Adelaide: Higher Education Research and Development Society of Australasia Inc.
• Boud, D. (1995) Assessment and Learning: Contradictory or Complementary? In P. Knight (Ed.), Assessment for Learning in Higher Education. London: Kogan Page Limited.
• Boud, D., & Falchikov, N. (2006) Aligning assessment with long-term learning. Assessment & Evaluation in Higher Education, 31 (4), pp. 399 — 413.
• Broadfoot, P. (2008) 'Assessment for learners. Assessment literacy and the development of learner power'. In A. Havnes & L. McDowell (Eds.), Balancing Dilemmas in Assessment and Learning in Contemporary Education. New York: Routledge.
• Denscombe, M. (2010) The Good Research Guide for small-scale social research Projects (Fourth ed.). Maidenhead: Open University Press.
• Francis, K. (2007) Personal discussion related to field trial: 'Fly on the Wall.' Auckland: Unpublished.
• Havnes, A., & McDowell, L. (Eds.).(2008) Balancing Dilemmas in Assessment and Learning in Contemporary Education. New York: Routledge.
• Higgins, R., Hartley, P., & Skelton, A. (2002) 'The Conscientious Consumer: reconsidering the role of assessment feedback in student learning'. Studies in Higher Education. 27 (1), pp. 53-64.
• Hodges, T. D., & Clifton, D. O. (2004) 'Strengths-Based Development in Practice'. In P. A. Linley & S. Joseph (Eds.), Positive Psychology in Practice. Hoboken, New Jersey: John Wiley & Sons.
• Kerka, S. (1997) 'Will we all be portfolio workers?' In O. S. University (Ed.), Trends and Issues Alerts. Columbus, Ohio: ERIC Clearinghouse on Adult, Career and Vocational Education.
• Knight, P. T., & Yorke, M. (2003) Assessment, Learning and Employability. Maidenhead: Society for Research into Higher Education & Open University Press.
• Kolb, D. A. (1984) Experiential Learning. Englewood Cliffs, New Jersey: Prentice Hall.
• Miltenberger, R. G. (2012) Behavior Modification Principles and Procedures (Fifth ed.). Belmont, California: Wadsworth Cengage Learning.
• Parnell, R., Sara, R., Doidge, C., & Parsons, M. (Eds.) (2007) The Crit. An Architecture Student's Handbook (Second ed.). Oxford: Architectural Press.
• Sharon, M., Jasinski, M., & Weatherley, R. (2006) Life Base Learning A strength based approach for capability development in vocational and technical education. Sydney: TAFE NSW International Centre for Vocational and Educational Learning.
• Schön, D. (1983) The Reflective Practitioner. How Professionals Think in Action. New York: Basic Books.
• Sullivan, K. (2002) 'Read, Think and Be Merry for in Two Weeks Your Assignment is Due'. In P. Schwartz & G. Webb (Eds.), Assessment. Case Studies, Experience and Practice from Higher Education. London: Kogan Page Ltd.
• Teymur, N. (1992) Architectural Education. Issues in educational practice and policy. London: ?uestion Press.
• Zmuda, A. (2010) Breaking Free from Myths About Teaching and Learning: Innovation as an Engine for Student Success. www.scribd.com/doc/57602559/Zmuda-Breaking-Free-From-Myths-About-Teaching-and-Learning-Innovation-as-an-Engine-for-Student-Success-Libro