The Value of Probing Assumptions

I was on a consultation call a few weeks ago about an ethics application.  The project was seeking feedback from participants about access to specific mental health information, and in my feedback to their application, I noted that their demographic question concerning the gender of the participant was probably too narrow.  The applicant asked for some advice how to address the comment.

On the one hand, they considered dropping the question as it a.) didn’t obviously connect to their research question, and b.) the literature supporting this branch of mental health was pretty well-studied in terms of incidence rates for the condition along the sex dimension, so they might not learn anything new by asking for the participant’s gender or sex.  On the other hand, if they left it in, they had to contend with whether they should use sex or gender as the focus of the question.  Since the mental health topic they were researching was a medical condition, it seemed like (biological) sex was the more salient feature, whereas my feedback suggested that if they chose gender, they would need to ensure it was inclusive.

While discussing the implications on the phone, I tried to tease out what the purpose of the study was.  Their study was collecting qualitative information about how people access information.  In the context of the demographic information, they weren’t seeking to know how a person’s sex/gender relates to the condition itself.  But, I wonder aloud, it seemed the purpose of the study was to understand how people seek information, which could arguably be influenced by one’s culture, behaviour, socialization, and experience of how the world treats them.  In that way, you would want to focus less on a person’s physiology and instead you might discover interesting differences in how a person seeks information based on their life-experiences.

The applicant noted that they started the phone call intending to drop the question from the survey, and through my line of questions and probes, was convinced to keep the question and modify it to be more inclusive.

I am not telling this story as a normative push on how we should conduct inquiry (though by reading through the lines, you should get a sense of how I feel about the topic).  Instead, I share this story as an example of why posing good questions is important to remove ambiguity and clarify thought.  One of the goals of our ethics board when we review applications is to make implied premises explicit so that we can be sure of what we take as a given when we set out to study a research question.  We often default to accepted practice and proceed with common tools, but sometimes we don’t think carefully through the implications of what using those tools means.  By leveraging my outsider status, I have an opportunity to get the applicants to explain concepts and lines of reason without assuming I share the same understanding of the material that they do.  This helps to spot those areas where the project is weakened by unsupported claims and assumptions.

Stay Awesome,

Ryan

Advertisements

Beware the Salesman

Podcast ads.

Hacks and routines.

Blogs that promote and sponsored content.

Look to the incentives and see if they align with your own.

What’s in their interest might not be in mine.

This might change over time.  Sources that don’t sell might eventually sell.  There is a future where I might try to sell you something or join my mailing list.

Beware the one who sells you solutions for problems they don’t have.

Beware the sponsored content coming from authorities you like.

Beware the salesman who makes universal the particular.  The one who puts everyone into a clearly defined box.  The one who makes money on your problems.  The one who charges to join their community.

I’m not saying these are all inherently bad things; merely that they merit many second thoughts.

Secret shortcuts don’t exist.  If they were valuable, they wouldn’t be secret and they wouldn’t be a shortcut – they would be the norm.

Everyone has to make a living.  Everyone has something to sell.  But not everyone has your best interest in mind.

Caveat emptor.

Stay Awesome,

Ryan

Book Ideas – Types of Feedback

I’ve been reading Scott Young’s recently released book, Ultralearning, and I think it’s a pretty good summary of how one can take on an intense learning project for personal and professional development.  It functions like an autodidact’s road map with plenty of good tips, insights, and stories to round things out.  Elements of the animated bibliography are present, but I don’t find it contrived in its execution.  The stories help frame the chapter and serve as an introduction to the core material.

It’s funny how last week I was talking about mnemonic devices, because after drafting that post I ended up reading about the concept in Chapter 10 of the book as it dealt with ways of supporting retention of material you learn.

In chapter 9 of the book, Young talks about ways of providing feedback in the learning process, whether the feedback is provided from others or feedback you can use in your own learning process.  He parses out three kinds of feedback that I found interesting, not only for my own personal use in learning, but also as something I should keep in mind as a teacher.

The three kinds of feedback he outlines are outcome feedback, informational feedback, and corrective feedback.  Each type of feedback serves a specific purpose, and you should be mindful of the context the feedback is given, as the wrong type of feedback can set you back in your learning.

Outcome feedback – provides information on whether you are getting answers right or if you are meeting a pre-identified set of learning objectives.  It tells you that you are right but doesn’t give any indication of why (or why you are wrong).

Informational feedback – provides further information to explain the underlying reason why something is right or wrong.  It can be informative to re-affirm what you have learned, and can identify key areas of strength or weakness, however it does not create a concrete process forward.

Corrective feedback – provides, as the name indicates, a path forward for the learner in terms of how to overcome deficiencies.  It details not only how one is right/wrong, why they are right/wrong, but how to address or avoid being wrong.  This type of feedback not only requires a level of comprehension of the material, but requires sufficient understanding to teach the underlying processes to the learner through explanation, demonstration, suggestion, etc.

As a teacher, it’s important to know what kind of feedback is warranted and under what circumstances.  Most of us tend to focus just on outcomes, but students often don’t learn from pure outcome assessment.  Rather, you need to take the further steps to go beyond an evaluation and ensure you are addressing the underlying deficiency present in the student’s performance.  Outcome assessment is awesome because it’s quick and definitive, but it’s also lazy if your goal is to improve your students.  On the other hand, corrective feedback is desirable but it’s labour-intensive and must be done carefully so as not to remove critical thinking from your student – you don’t want them to merely follow your instructions but instead you want to promote their thinking and reasoning through problems without your guidance.

Stay Awesome,

Ryan

Memorization and my Campus Response Experience

In January of 2008, I was walking through my university campus’s student centre and passed by a table for the UW Campus Response Team, whom were recruiting volunteers for the new semester.  I doubled back, chatted with the team members, and signed-up to participate in their interview process.  I had taken first aid courses periodically during my cub scout and army cadet days, plus I had ran some basic first aid courses while abroad, so it felt like a good fit.

In retrospect, my “experience” was quite paltry, but I had shown the team managers that I had enough of the “right stuff” that they invited me to join the team and participate in the weekend training course they put on for new recruits.  It’s an intense crash course in first aid skills that were well beyond my experiences and the training spanned several hours Friday night and all days Saturday and Sunday, before you perform your final scenario test to qualify as a secondary responder.

The material covered was largely derived from emergency first responder courses, along with some material covered for pre-hospital trauma professions (e.g. fire fighters and paramedics).  The training was designed to create heuristics in the responder’s mind to quickly flow through critical details while gathering as much information as possible and start treatment momentum.  The last thing you want is for a responder to have to intentionally think through what steps they should follow, because it shunts cognitive capacity away from situational awareness and into operational procedures.

In an effort to automate one’s thinking, you end up doing a lot of mock scenarios and skill drills.  As a responder, you end up creating a script in your mind to follow.  The script is based on a common set of things to attend to, which you follow according to handy mnemonics and other memory aids.

Despite the mnemonics functioning to provide mental triggers for actions, you still need to learn the process to go along with the mnemonics, and from the start of training weekend, you only have precious few hours after training concludes for the day to encode the information out of your working memory and into longer term storage.

I needed a way to quickly drill myself and aid in recall.  The system I settled on was to get some window writable markers and write out my mnemonic devices on the bathroom mirror.  Every time I used or walked passed the washroom, I would attempt to fill in as many of the mnemonics as I could remember, and note where I made mistakes.  Through constant repetition, I was able to turn:

E
M
C
A
P
I
E

A
V
P
U

P
E
A
R
L

A
B
C
D

S
A
M
P
L
E

O
P
Q
R
S
T

Into

Environment check
Mechanism of injury?
Count the casualties
Allied agencies?
Personal protection
Introduce yourself
Events leading?

Alert
Verbal
Pain
Unresponsive

Pupils are
Equal
And
Reactive to
Light

Airway
Breathing
Circulation
Deadly bleeds

Signs and Symptoms
Allergies
Medications
Past medical history
Last meal/beverage intake
Events leading

Onset
Provocation
Quality
Radiation
Severity
Time

It was a quick and dirty way to give myself quick feedback on these concepts that I could readily apply to my first aid treatment during training and eventually on shift.  Any time I lost momentum or felt nervous about the judges evaluating me, I would mentally go back to my bathroom mirror and fill in the blanks.  I haven’t been on the first aid team in almost a decade but these concepts easily come back to me, even during my crazy nights at the bar.  It’s a testament to the stickiness of the ideas and the effectiveness of the drills.

Stay Awesome,

Ryan

Knowing When To Quit

A recent article talking about knowing when to quit/retire from teaching had me reflecting on my own experiences with quitting.  Truthfully, I can’t recall many instances where I quit something.  Often, I will drag out experiences long after they have been useful, and instead of quitting as an active decision, I’m more likely to let things fall away through neglect.  Perhaps there isn’t a strong difference between the two since my history is littered with things that I eventually stopped doing.  I suppose in my mind, the difference comes down to whether I made a decision to stop – whether I took ownership over the act.

The strongest instance where I actively made a decision was when I stopped hosting at the local karaoke bar.  I was three or four years into my tenure as a host, and for the most part I enjoyed the experience.  I had a regular crew of friends who would come in and make the night interesting.  However, towards the end, I grew to resent patrons coming in who weren’t my friends.  I worked the slowest night, so if things were quiet, we’d shut down early.  But if patrons filtered in and kept purchasing stuff, we’d stay open.  Catering to the average customer felt like a chore, rather than chumming with friends with our own song preferences and inside jokes.

I started to dislike going into work, and even to this day I don’t sing much like I did while I was a host.  I’ll grab the mic from time to time, but I don’t go out to enjoy karaoke anymore.  I still work security at the bar, but I stopped hosting all together.

I made the decision to stop hosting because a small part of me knew it was time to move on.  I learned what I could from the experience, cherished the memories it gave me, but I recognized that I no longer wanted to spend time doing it.  I think that’s the critical part in the art of quitting.  It’s not about actually quitting or the how.  Instead, it’s about recognizing when the time has come and why.

Sometimes we have to slog it out in things we hate.  We don’t quit those things because we assign value to the activity (or someone else has assigned value and we are dragged along for the ride).  But quitting is more than stopping a thing you don’t like.  It’s about recognizing when the thing is no longer of value to you; that it won’t take you where you need it to go.  It is the recognition that your time is better suited elsewhere.  The art of quitting ultimately comes down to taking an active role in how you choose to spend your time.

Stay Awesome,

Ryan

Masterclass in Political Oration – Jon Stewart

There are some people that when they speak, I will stop to listen.  We have many examples of people who are gifted public speakers, but to me few are more powerful than Jon Stewart, former host of the Daily Show.  He spoke at a House sub-committee hearing last week and so thoroughly presented his case, the bill passed unanimously.  I hope the initiative continues as smoothly through the House proper and the Senate, and is eventually passed into law, because the hypocrisy and virtue-signalling is appalling.  At the centre of Stewart’s argument is the notion that the sacrifice and bravery of the responders during 9/11 should be honoured by taking care of those who are suffering because of their service that day.

Public speaking as a skill is hard, but there is more than just vocalizing the words.  Stewart’s presentation, his ethos (he has earned the right to speak through his work), his pathos (the passion he speaks from on behalf of those he’s fighting for) and the pure logos (no one can form a devastating argument from observations the way a comedian can) all come together to give us a masterclass in political oration.

Give it a watch.  It made me feel choked up.

 

Stay Awesome,

Ryan

Grading Drift

I’ve been reflecting on the concept of grade drift recently. My brain usually turns to grade drifting at the end of each semester as I evaluate how objectively I have scored the students. If you’ve never taught before, what I call “grade drift” is the tendency for an instructor to allow their grading standards to drift as they grade an assignment. I’m not sure how prevalent this is for assessments that are concerned with exact answers, such as math, science, engineering, programming, etc. But in assessments that deal with qualitative or creative responses, it’s common for you as an instructor to change your evaluation thresholds as you make your way through the pile of papers.

There are a few reasons why this happens. In my first courses, my grades would drift if I didn’t have a sufficiently robust marking guide to help me narrow down the kinds of answers I was looking for in a student’s response. Now that I’ve taught the course a few times, I know generally what I’m looking for and can go through the assignments with a checklist of items. This form of grade drift is usually related to lacking experience in teaching and grading, especially if your course delivery isn’t tied to your course learning outcomes.

However, there is another reason why your grades might drift. On the one hand, you want to approach your evaluations as if they are objective – that there are clear right and wrong answers. After all, your grading should be defensible, and one way to defend the grade you grant is by pointing to an objective standard. However, on the other hand, you want your evaluations of your student’s work to allow for imagination, creativity, and novel connections between ideas. You also want your evaluations to acknowledge that your classes have unique compositions of people, with their own experiences and their own progressions through your course. Just because you teach the same material each semester doesn’t mean your students progress through it at the same rate. Different students will take the material differently, which means the average response you read through will be different for each cohort of students.

This means that some groups will be “better” than others in how they perform. You can choose to penalize the students for not meeting an abstract standard that you’ve set, or you can meet the students where they are and make it your goal in the course to improve their performance as you go.

I know that some instructors will take the first approach. They will believe their teaching, evaluations, and courses are reflective of an external, eternal, objective standard; they don’t think they are being arbitrary. I’m willing to bet that these are also instructors who believe that a 70% in their course is “like a 50% in other courses” (a literal thing I’ve heard said by a peer). I think this approach is wrong and it reflects a misunderstanding of what it means to pass a course. When a student passes a course, it means they have met the minimum requirements of the course learning objectives. Anything above that should reflect varying degrees of competence and mastery. If your course is designed so that a student is only “just” meeting the standards with anything above a 70%, you haven’t calibrated the course or your expectations very well. At an undergraduate level, your job is to elevate your students, and improve their abilities while meeting the program learning outcome objectives.

I find myself on the other side of this issue. I understand what the goals are of the course I teach, and my aim is less concerned with ensuring my students can adequately explain all the theory they are exposed to in the course. Instead, my goals is to make my students better thinkers and writers from where they enter my course to their exit. If they can explain Aristotle’s ideas of tragedy by the end, great! But I won’t lose sleep if they can’t. My course is a general elective, and that means I’m supposed to round them out as students and people. I care less about them absorbing my esoteric knowledge and instead I care more about them learning how to think, reason, and communicate their ideas clearly.

And that’s why sometimes my grades drift as I score the student’s rubrics. I start off with some ideas of where they could be, then I calibrate my grading to meet them at their level. It’s a little more work for me, but I think it better captures the student’s performance.

As indicated above, there are a few strategies for overcoming this, such as having a clear rubric and clear notes on what you are looking for.  Also, marking all the students on one question, one at a time, rather than marking an entire submission will make it easy to compare student fairly.

While this is an unfortunate thing (the students like to think they are being objectively evaluated) this will happen as you come to understand the natural curve of your class.  You can either grade your students to one standard, or you can reflexively respond to your class’s own knowledge and aptitudes.  If your course objectives are clear, then you can feel free to adapt your grading to match the student’s progress.

Stay Awesome,

Ryan