On 17 October, Dr Brigid Rooney (Associate Dean Teaching and Learning, Arts) and Dr Susan Thomas (Teaching Development Coordinator for Arts, Humanities and Social Sciences, ITL), interviewed three Arts students on their experiences of the USE feedback process. Stanley (Stan) Green is a first-year, mature-age student majoring in History; Briannon McLoughlin is a second-year student majoring in Indonesian and Philosophy, and Rebecca (Bec) Santos is a third-year English major. Drs Rooney and Thomas were awarded a Large TIES grant to investigate current processes of collecting, analysing and disseminating USE feedback and to make recommendations for streamlining the process and closing the feedback loop. The following is an excerpt from the 17 October discussion.
Left to right: Stanley Green, Rebecca Santos, Briannon McLoughlin
BRIGID: What have you experienced when filling out a Unit of Study Evaluation (USE) - what was your perception of the purpose of the exercise-and how did you find the experience?
STAN: Yes, well I did that at the end of the first semester. And I understood that it was intended for the department concerned to see that they were providing the right sort of material, the right sort of information, to see if there were any criticisms and basically to improve the Unit of Study (UoS). It would be nice to see the results, transmitted into something tangible.
BRIANNON: This is my fourth semester at the university, so I suppose so far that I have done about a dozen USEs, and I have had a lot of positive and negative experiences with it. I think that sometimes in subjects where I have felt it was a really great subject, I've used it as a way to compliment the things that I thought were really well done. And then in other subjects I've had lecturers or tutors who've made USE a very rushed experience, and have sort of given the impression that it's less than important. I know that I filled one out last week - we just finished our subject in week 10 - and the lecturer said "we've got 10 minutes left, I've got some surveys here, and if you want to fill them out you can. And if not, you can go". I was watching to see how many people left. And at least a quarter of the people packed their bags and went out, and a couple went and did the ticks in the number boxes, and then there were maybe a dozen of us left, actually writing in the spaces to extend our response. So um, yeah I think I've had a bit of both worlds. But on the flipside of that, I've had heaps of lecturers who have been really positive and even funny about it. They've said "I'm going to leave the room... don't feel pressured... you can tell the truth." I think in smaller classes it's a much more expressive form of response. In the large classes, your response is seen as less important I think.
BEC: Yeah, I'm always one of the people who stay towards the end. It feels really awful to say, but if ever throughout a semester a tutor or lecturer transgressed in my opinion, I'd think, "oh yeah, I'm saving that up for the unit of study evaluation." But by the time I'd get there, I'd forget all the discrepancies I'd had with my tutor or lecturer. It's difficult for me to treat the unit of study evaluations with any degree of seriousness, which they obviously deserve, because it's never explicitly outlined how your answers will be translated into new policy or new ways of going about the subject, or even really where the unit of study evaluations go once you put them in the little envelope. If they're ever going to be used for something more substantial than vitriolic outpourings from unsatisfied students or laudatory comments from satisfied students, I think students need to know how their comments will be translated as some sort of pragmatic outcome. The explanation process has been lacking.
BRIGID: Do all lecturers explain it the same way in your experience? Or have you got that similar diversity of experience that Briannon has talked about?
BEC: It's not really explained in my experience. By that time of semester, people are generally exhausted, and there's a big drop off towards the end of lectures, because there's no more content to be covered, just concluding comments.
BRIANNON: I think that's very true.
SUSAN: Would it make a difference in the way in which you filled out the form, or the way in which you approached the exercise, if someone told you how they were going to use the data?
BEC: Yes.
BRIANNON: I absolutely agree.
BEC: It's hard to engage with a process when you don't actually understand what the process is for, and also it seems really vague and a bit fake as well. Inevitably it won't influence my experience of the UoS. So, while I think it's good to get overall feedback, I'm not comfortable with the idea that feedback is only solicited in this hyper formalised way, at the end of semester. I think it should be continual.
STAN: I have little or no faith in that format of evaluation, either pre-uni or currently. I think that if one wants to be relevant and helpful, then you can do it in a small group, like we are here. Lecturers and tutors have to be responsible for acknowledging and obtaining feedback, and implementing the results, rather than just being responsible for some paperwork at the end of semester.
BEC: I don't necessarily agree that you can replace USE surveys with a focus group, or that focus groups would achieve the same sort of aims, because the survey at the end is good for getting a large sample, and also being able to extrapolate from that, broader trends that might happen across the discipline, or happen across the faculty. But I think what you're saying was really useful because I don't think that it needs to be as formalised as focus groups, but I think that opportunities during tutorials for example, maybe midway through semester, there could be a 10 minute chat about how people think the UoS is going. And I think that would help institute a culture of open communication within the particular unit of study, because it's not this little process at the end where you fill out boxes but don't know where the paper goes.
STAN: But it could be a mixture of both, because with both, you might get some sort of correlation, which gives you more concrete direction.
BRIGID: I'm getting the strong impression from listening to you, that you've never experienced any other form of feedback in your units of study, other than this formal survey instrument.
ALL: That is correct.
BRIANNON: That brings us back to Susan's question before about if we knew more about where the process was going, it would make it a better process from the students' point of view.
STAN: I'm very impressed with the Academic Writing (ENGL1000) system of postings - the discussion board. I think that that's absolutely perfect. It's wonderful to read what students write to one another, because not everyone feels comfortable communicating in tutorials.
SUSAN: Would you feel differently toward the USE, or would you be more motivated to take it seriously if it were only one component of a feedback structure that you saw in the UoS from the very beginning?
STAN: Anything more than the end of semester pink sheets of paper has got to be an improvement.
BEC: I really agree. The USE forms occupy a very privileged space, because they are purely about feedback, as opposed to merely anecdotal 'how do you think you're going' corridor chats. So I would say that if it was part of a broader strategy, then that's an intrinsically good thing, but I still think USE should remain as your final conclusion to the UoS. I would always advocate for more informal chats, and perhaps even for the lecturer or tutor to establish some sort of discussion at the beginning that asks students: How do you want to give me feedback? How do you prefer to give feedback? Because of issues of anonymity and equity, not everyone is going to feel comfortable putting a post which is quite explicit with their name on it, or walking up to a lecturer after class, and saying that you're doing a really bad job in X, Y and Z.
BRIANNON: I wouldn't want anyone to walk away from this discussion thinking that no one takes the USE seriously. It varies so greatly. Anonymity is a huge part of it. In a small class, with say a dozen students, they might feel that it's easier to respond to their lecturer, because they've developed a personal relationship with them. They might feel that writing something down is going to be read by someone. Whereas in a class of 600 there is a big mind set of "what difference is it going to make, it's not really that important" and therefore if we had more opportunities to talk, as Bec was saying, it might encourage people to see the structure of evaluation throughout the course.
BRIGID: The question of anonymity comes up here and it's an interesting issue in terms of how it might direct or not direct your own responses to these surveys. How do you feel when you're writing your answers to the surveys? Do you feel like you can be very frank? Or do you feel like you want to pull your punches a little bit? Or do you feel inspired to write very positive things or constructive criticism?
BEC: I think that it allows the person to be frank, if they choose to be so. Personally, I try to frame things in a constructive way, so if I've had a problem with the way things have gone, I've said this is what the issue is; this is why I think that it's a problem; and, these are a couple of options as to how I think it could be improved. But, it's difficult to ever say that to anyone's face. And I've never gone up to a lecturer and said, "I kind of think you're doing a bad job. Here are some options I have on the table. I think that we should just hash it out personally over coffee". That's never going to happen. So anonymity is useful to the process because it lets you be honest.
BRIANNON: I agree with that. I feel more comfortable writing what I truly think, and letting it all out - all the issues that I've thought about all semester. I think [USE] is a great opportunity to do that without worrying what anybody else will think about what I have done or how it might be interpreted by someone who is going to be marking my exam. But by the same measure, if you've had a really great experience, sometimes it's nice to be able to tell someone to their face how great they've done. But I've watched other people filling out USE forms and seen some students write in large capital letters one rather awful word, describing what they thought about the lecturer. And I've thought "well that's not particularly constructive," and because it's anonymous, they don't worry about it-no one is going to come and say you didn't do that properly. So I think that it has pros and cons.
STAN: I agree with my learned colleagues here. I think that for the totality of the university, anonymity is the right way to go, because then people can express freely what they think. And if some of them want to be ratbags and write rotten things, well then that's their problem. But from my point of view I don't need to be anonymous. I feel quite comfortable not being anonymous. But that's got to do with age if you like, or experience. I don't feel intimidated by lecturers or tutors.
BRIANNON: I think that anonymity makes it formal as well. Having it on paper like that and knowing that it goes somewhere special in those special envelopes...
SUSAN: Just for the record, the UoS coordinator sees every word you write, so your comments are not disappearing into a vacuum.
STAN: But we need another level, another tier to back it up.
BRIGID: So what kind of information are you given when the surveys are handed out? What do lecturers typically say to you before handing out the survey? What do you know already about the process?
BRIANNON: I think they have something to read out. I'm recalling someone holding a piece of paper, reading out all the things that need to be done.
SUSAN: The procedure...
BRIANNON: The procedure paper, that's it. And I know I've heard that a few times, but I think it's something a lot of people tune out because it's not particularly interesting.
BRIGID: Like flight information.... You know, buckle up, and put your head in the brace position...
STAN: There is no consistency. Some lecturers leave the room, and some don't. Some explain it, and some just say here's a bit of paper, fill it in. There is no "this is the way that it should be done. Please don't deviate from it".
SUSAN: The issue of student fatigue is discussed quite often in relation to USE. Are you too tired to fill out these forms or a bit bewildered by the lack of transparency surrounding the process? Is it worth making a distinction between these two things?
BEC: I think that student fatigue is twofold: Firstly, it's that time of semester when people just stop turning up to class because it's not content-based anymore, so for the really important survey procedure to happen at that particular time, it's fatigue in a very real sense in that everyone's too tired and bored at this time of semester. But it is also student fatigue in the sense that it's just another form that you have to fill out. It's the same kind of format that you get for market research and people stopping you in a shopping centre, and people on Eastern Avenue trying to get you to sign petitions. The USE form doesn't demarcate itself effectively from all the other kind of crap - can I say "crap"? - that you have to do on a daily basis.
BRIGID: I'm wondering if you've ever been in a UoS where someone has actually explicitly said to you - a teacher, a lecturer, a co-ordinator - how past evaluations have helped them to think about the unit and design it differently or change a few things?
BEC: No
BRIANNON: Yes, I have actually.
STAN: Twice actually.
BRIANNON: I've been studying Indonesian since I first started at the university, and in my opinion, compared to all the other subjects that I've done, in all the other departments, the Indonesian department has a particularly good feedback structure, and I've always felt like I could give feedback directly during class and converse about it outside class. Half way through each semester, we are given an informal written evaluation form to fill out, a few questions, and it's very casual. We talk about it, we fill it out. And our lecturer will come in the next week and say: "I noticed you wrote this", and we talk about how we can improve what we're doing, and what we liked and what we didn't like. And maybe that's enabled by the small class size. But also it's just that constant, and we expect that soon we're going to be able to tell them what we think. And at the beginning of semester, we talk about what happened at the end of last semester with our USEs. Our lecturer would come and say "I noticed that some of you didn't really like this, so let's think if there's a better way to do that". So I know that I always feel particularly good about that subject.
BRIGID: Do you think that that's had any effect on how you have approached the USE forms? Or have you noticed how other students are approaching it?
BRIANNON: Given that it's a small class, I really do think that that has a lot to do with it - there are only 10 of us studying it this semester. As a whole, I think that we all think that if we all write something, something will happen. Because we can discuss and say "oh yeah, I agree" or "Oh! I forgot about that" and we really expand on our answers. And I noticed that in some of my larger classes, people don't expand on their answers at all - they write one or two lines, and leave it at that. Whereas we see it as an opportunity to really let them know everything that we're thinking. Because we can see that they want to know more.
STAN: Well I haven't had as detailed an experience as that. But there are at least two occasions when we were given the impression by the head of the unit that they were very enthusiastic about knowing how they were going and how their structure was, and how their lectures were going. And in other cases, it never really came up at all. So I think that you definitely get a very good impression, and you get encouraged, if you see your unit head making enquiries. So far for me, it's really limited.
BEC: Even though it's quite obviously not explicitly discussed, do lecturers actually make adjustments to their own teaching styles according to the comments made by students?
BRIGID: Before I respond directly to that, what I want to notice there is that you're seeing the USE as being something that would affect someone's teaching style. And it's interesting, because it's also about the UoS itself. It's design and delivery. I'm wondering when you fill it in, whether you're thinking more about individual teachers or more about the UoS design overall.
BEC: There has only been one UoS where I've loved the lecturer, but disliked the content. So that's the only time when I've shifted my mind frame from individual lecturers and their teaching styles to something that's purely focused on the UoS make up. Particularly in English, you choose a UoS because you love the content, so you know that the unit of study is going to be more or less good. The only variable in all of that is how well the UoS is packaged, or how well it's given to you and that's why that sort of thing sticks out to be commented on, versus other possible things.
BRIANNON: I agree. I think that we know that feeling of not being happy with what we've been taught, but having really enjoyed being there all the same. And I'm interested in what you said about looking at it as an evaluation of the UoS. I imagine that most people would see it more of an analysis of what they've done - so the experience that they've had with their lecturer and their tutor. And that would differ greatly...
BRIGID: Of the person rather than the whole package?
BRIANNON: Yeah, of the individual. I'm not sure I've ever seen it as a package deal so much. Because that would take a much more prepared thought process. I'd have to think about that before I wrote. And given that it's a quick, fill out this form sort of process, I don't really have a lot of time to sit back and think "What do I think about this UoS", in an articulate, holistic way.
BRIGID: Do you ever find the questions themselves frustrating to answer? Do you wish that you were asked different sorts of questions to the ones that are posed to you?
BRIANNON: They're so generic.
STAN: Can I just touch on one thing: I don't think that there are such things as bad subjects, or bad lecturers. I think that if you get a good lecturer, an enthusiastic lecturer, or an enthusiastic tutor, it doesn't matter. They can inspire people to participate and they can bring the best out of what the material is.
BRIGID: Would the rest of you agree with that?
BRIANNON: Yeah, I think that Stan articulated it better than I did.
SUSAN: What about team-taught UoS, where you may like one of the lecturers very much, and their teaching seems relevant. But then you've got others who aren't as effective-or you may feel that the material is out of sequence. On one hand, it is an issue about the lecturer, but on the other, it's about something the coordinator could do to improve the delivery of the UoS.
BRIANNON: In one of my first year units, they have a situation where they try and touch on a whole lot of different areas, under six different brackets, and they break it up into the two courses. In the first semester you do three different areas and you have three different lecturers doing three different things. And I would certainly say that it's a situation where I would critique the structure of the UoS rather than the individuals themselves.
SUSAN: So that's a UoS issue, not a teaching performance issue.
BRIANNON: It is. And I hadn't really thought about it before, and you're quite right.
STAN: This illustrates why there needs to be instructions by the lecturer, or tutor, before pen goes to paper, so that everyone is clear on the purpose of USE.
BRIGID: But could too many instructions skew the responses?
BRIANNON: Do you mean that perhaps students wouldn't want their lecturer to get into trouble?
BRIGID: That kind of thing. There might be a loyalty factor... I'm just thinking it through a little bit.
BRIANNON: It's important. I'm not sure what to think.
BEC: I don't feel equipped to give a sociological analysis that those kinds of questions necessitate, but I think at the end of the day all you really need to know is what it is, what it's used for, where it's going, and how students will be able to check up on the results. That kind of discussion, if pitched in that kind of way, hopefully wouldn't be polarizing or make people take it less or more seriously. It's just to explain what it is and the original problem is that no one really knows what it is.
STAN: I'm going to bring a donkey's tail to the next filling out of forms, and stick it on.