Sunday, February 28, 2016
Interesting Reads
This page is updated as I find interesting reads. There is no particular topic of these articles. Just things I find interesting. I may comment or not depending on my mood 😃
Wednesday, February 24, 2016
Candace's notes from Learning Analytics: Risks, Benefits, and EthicalIssues
Learning Analytics: Risks, Benefits, and Ethical Issues
Sharon Slade Open University senior lecturer
Development of ethical policies for student data
Mitchell Stevens assoc of Ed at Stanford, director of data policy,
George Siemens prof at UT at Arlington, press of the society of learning analytics
Gene Gloechner School of Ed CSU IRB
What is the purpose of LA:
Sharon:
Use data to influence student outcomes - learning Analytics
Individual targeted interventions to have a positive effect
Predictive modeling
George - practices for a range of areas, not comfortable with business Analytics
LA - any activity involving the learner as the center
LMS, social media, wearables,
Mitchell - learning sciences developed in last 20 yrs. funded by NSF, largely focused in STEM k-12
2012 universities created MOOCS mania. Computer sciences really jumped into MOOCS
Digital mediated instruction - DMI
Gene - reverse of standard research models, analysis data to get research questions . What are the ethical questions involved. How is information going to be used? How do we info the students about the use of data?
Positive usages of LA, how LA can positively effect the interaction of students & faculty and student outcomes:
George: greater sense of autonomy for the student using neuroscience ( not behaviorism 2.0) student should see all data in an accessible and understandable format, what do we value?
Sharon: we don't really understand what the use of LA is in the university. Student vs institutional goals.
Mitchell: Belmont report
Managerial approach (efficiencies)to start with but now moving towards improving instructional interactions
Gene: mood meter app as an example for the use of analytics.
Tom Siller, how do we help students use the LA?
George: How do we get students to next step decisions, complexity since view, don't drop all data to the students, start small and then move forward. Use data to form next steps...
Sean Burns, LA CSU, what kinds of interventions do you suggest, where can we go?
Sharon: how confident are we in the predictive data? Context with explanation, relate to info about the student.
Mitchell: there's a lot of low hanging fruit with which we can use to help with the needs of undergrads. Operations need to be reorganized. How college works? Academically adrift
What are the potential risks in using LA with student/faculty interactions?
Sharon: make sure to keep in mind the student is a person, information gathered isn't the whole picture. Predictive modeling, doesn't necessarily means the next student with the same data is going to behave in the same way. An individual is more than their data.
George: concern that are we going to liberate the learner or dumb down the material?
Mitchell: how hard should college be? What kind of environment do we what college to be? Dangerous to be a "click through" experience
Faculty usage, don't we already have enough?
Mitchell: instant feedback, get more info on a large scale to see what concepts/tasks are being learned or not.
Potential risks of using student data?
Mitchell: people react to situations differently. "Click Through U" data becomes perspective and/or prospective.
George: numbers are precise but learning is a sloppy, social experience.
Student affairs - advisors support structures, what is their roll?
Mitchell: it's a widely distributed enterprise, division of labor
Ethical hazards/issues of learning or academic achievement
George: data is transactional, like FB, who owns the data? Non use, overlook the use of the data to assist students.
Gene: who owns the data?
Mitchell: what should the first principles be compliance? Data are joint ventures, data exists at the intersection of several parties.
Sharon: it's not ethical to ignore the information we have. There's just too much data so we need to pair down which data to use to help students.
Ethical issues regarding privacy issues, consent, etc.
Gene: the goal us to help the students.
The ethical use of student data at open university
Sharon: opt out option, informed consent, prior to registration, transparency
As Big-Data Companies Come to Teaching, a Pioneer Issues a Warning
FEBRUARY 23, 2016
Candace M. Thille helped kick-start the move to bring big data to college teaching.
She has founded the Open Learning Initiative at Carnegie Mellon University, won millions of dollars in grants, and been a fixture on the lecture circuit about the power of so-called adaptive learning, where data-powered algorithms serve up content keyed to what a student is ready to learn next. Publishers, venture-capital investors, and foundations have followed her lead. They’ve poured hundreds of millions into new companies and new products vying to score big contracts with colleges, sometimes promising to be the "robot tutor" for struggling students.
It seems like a classic business success story.
The Chronicle of Higher Education’s Re:Learning project provides stories and analysis about this change moment for learning.
- Sign up for our weekly newsletter
- Join the discussion on Facebook
- Listen to the podcast (coming soon)
She still believes that adaptive learning will become an increasingly important tool in teaching. But she fears that rapid commercialization is exactly the wrong way to foster innovation at this early stage. What’s more, she thinks professors and higher-education leaders are making a dangerous mistake by letting companies take the lead in shaping the learning-analytics market.
Ms. Thille, who moved to Stanford University in the fall of 2013, has only recently begun to go public with that critique, voicing it to a few small audiences. But as she shared during extensive interviews with The Chronicle over the past few weeks, it’s a message she hopes college leaders and professors will heed, if only because she’s a messenger who understands, despite all the hype, both how "crude" and simplistic many of the products are today, and how educationally valuable they could one day become.
Her concerns boil down to these:
- Colleges should have more control over this field. Like it or not, she argues, using data to predict student needs and deliver the right material at the right time will become essential. "And a core tenet of any business is that you don’t outsource your core business process," she notes.
- Companies aren’t as well equipped to develop and test new teaching algorithms as colleges are. She argues that colleges are the ideal living laboratories for any teaching system because they are home both to the research on learning and the actual teaching. As she puts it, "You have a very quick feedback loop, where the research informs the practice and the practice informs the research."
- When companies lead the development of learning software, the decisions those systems make are hidden from professors and colleges. Ms. Thille says companies that won’t share their processes are essentially saying, “Just trust the black box.” To most academics, she says, “That's alchemy, that's not science."
The proprietary “black boxes” are the algorithms that might automatically serve up, say, an extra lesson on quadratic equations when a student's responses to a quiz indicate that she didn't quite grasp the concept. Every algorithm, and every decision about what data it will weigh, is also ultimately a pedagogical judgment call. For the companies selling adaptive software, "that's where the gold is,” says Ms. Thille.
She contends that the demands of the market — with venture capitalists expecting returns 30 times what they’ve invested and companies facing pressure to deliver products priced so they don’t scare away customers — will inhibit innovation rather than foster it, even if the companies have the best intentions.
Ms. Thille (it's pronounced "Till"), who is in her late 50s and spent 18 years in the private sector before joining academe, is a person who chooses her words carefully. She recognizes that her critique of the burgeoning learning-analytics marketplace is also fundamentally a critique of the commercialization model of the investor-fueled ed-tech sector, and in some sense, of her own Stanford community.
But she is also convinced that colleges need to find ways to raise the money for research and development of learning software so that companies don’t end up owning the classroom delivery system of the future. Colleges could come together to build such systems, or perhaps the federal government could step in, as it did with the Darpa research that led to the Internet. Society has stepped up before for matters of importance, notes Ms. Thille. "I would claim that this is one of them."
Big Claims
Ms. Thille’s argument could easily be dismissed as naïve, or even self-interested — she does, after all, head up a research lab that lives by grants. But it meshes with a broader national conversation now surfacing among academics and other experts over the growing role that data and algorithms play in higher education.
Data-driven technologies already touch many corners of the student experience. "Student success" tools send text messages reminding students to see an adviser. Course-suggestion engines at some colleges can recommend that a student switch majors, from pre-med to history, if he did poorly in Biology 101. But educators are especially focused on the ways data engines are used in teaching.
Companies such as Knewton, whose chief executive boasted to an NPR reporter that the software was like "a robot tutor in the sky that can semi-read your mind," epitomize the problem, says George Siemens, an internationally known theorist on digital technologyand the professor who co-taught the very first MOOC, in 2008.
"They make very bold claims, but they aren’t involved in the research community at all. That means we can’t validate their algorithms. We can’t validate the results that they say they’re getting," he says. "That’s a system that doesn’t serve the future of education well at all."
“They make very bold claims, but they aren't involved in the research community at all. That means we can't validate their algorithms.”
He compares the state of learning platforms to the earliest days of networked computing, when it wasn’t clear whether the open Internet or a paid service like America Online would become dominant. If the learning-analytics model goes the way AOL hoped to go, says Mr. Siemens, "we don’t have the ability to create a rich ecosystem on top of that platform."The chief executive of Knewton, Jose Ferreira, declined to comment for this article.
A small community of academic researchers and others are beginning to press on this issue. Mr. Siemens hopes others will too. "Students should care an awful lot because they’re the ones being sorted algorithmically," he adds. Among those concerned is Ms. Thille’s successor at Carnegie Mellon, Norman Bier. Without greater transparency for the way tools direct students and dashboards signal their progress to professors, "we face a real danger of hurting our students," he says. The systems need to be more open, he says, so professors can understand and trust them without having to get an advanced degree in computer science.
But the trend is going in the other direction. More and more colleges are turning to the commercial market for their adaptive-learning products, often in response to vigorous marketing by companies. That includes Acrobatiq, a company that was spun off by Carnegie Mellon after Ms. Thille left. The university still owns a stake in it.
Acrobatiq advertises that its products are "powered by Carnegie Mellon" and "based on cognitive science and educational theory" of the Open Learning Initiative, even though none of the company’s products use OLI-developed software.
Eric Frank, Acrobatiq’s CEO, counters that companies can innovate, and notes that his company focuses on the important role of making sure its products work for a broad range of professors. "We spend less time refining our predictive models and algorithms and more time trying to help faculty to use them," he says.
As more and more colleges work with more and more companies, he acknowledges, researchers will lose out on opportunities to collect large sets of data on how students learn and use them to advance the field because the data will be "bifurcated into these little fiefdoms." Acrobatiq’s contracts, he notes, do ensure that colleges own the data. But he notes that the learning-analytics industry still lacks the kinds of standards and protocols that would make it easy to extract, organize, and share such data among institutions for research purposes.
Even some foundations and education associations can operate in ways that undermine the momentum for open learning analytics. For example, in inviting universities to take part in a new $4.6-million Bill & Melinda Gates Foundation grant for the members of a new Personalized Learning Consortium, the Association of Public and Land-Grant Universities specified that applicants could use only 19 specific products approved by the foundation. All of them are owned by companies. One of them is Acrobatiq; the Gates foundation is also an investor in that company.
The foundation specified those "approved suppliers" for the quality of their products and because they "appear likely to be robust and sustainable for the future" as businesses, according to the request for proposals for the grant.
Travis Reindl, a spokesman for the Gates foundation, said last week that the list was not "the be-all and end-all of the possible providers" for the grantees and that other products could be added even after the winning institutions are selected. at the end of May. Asked specifically about whether open courses from OLI or those Ms. Thille is now creating at Stanford would be allowed, he said, "APLU will provide the updated provider list to the selected institutions. The list is still being developed."
As for Acrobatiq, he said the foundation was "not in any circumstances" trying to favor that company. "There is no thumb on the scale" for Acrobatiq, he said.
Deeper Learning
Ms. Thille, who now co-directs a team of 15 master’s and Ph.D. students in the sleek, high-ceilinged Lytics Lab, on prime real estate on the Stanford campus, can’t pinpoint an "aha!" moment when she began to feel a sense of urgency for her concerns. While she maintains many cordial relationships with the ed-tech crowd in her Silicon Valley neighborhood and around the country, she was influenced in part by seeing the flood of investments and acquisitions in the market: Knewton alone has received more than $157 million, including $10 million this month, according to CrunchBase.
One of her Ph.D. students was another influence. Last spring he came up with a wholly new approach to creating predictive-algorithmic models for deciding what content a student should see next in an adaptive course. She thinks his "deep learning" approach runs circles around Bayesian Knowledge Tracing, the approach used by OLI, her Stanford lab, and many of the commercial learning-analytics products.
"As a researcher I was thrilled," says Ms. Thille, beaming at the thought of a graduate student who could "beat my stuff" and make it better. "But you can imagine if you were a company whose whole proprietary value proposition was built on, and built around" the existing model. Companies, she says, would find it pretty difficult to say, "OK, that’s not good anymore, now I’ll switch over to a completely different method."That deep-learning method may prove not to be ideal either, says Ms. Thille, but it illustrates just how early-stage the field of learning analytics really is. For that reason, she says, colleges should be cautious in their dealings with companies. "Go ahead, experiment with some of these systems," she advises. "Just don’t get locked in for a long-term contract and develop your whole approach around a particular system, because it’s going to change, or it should change."
Back when she was at Carnegie Mellon, Ms. Thille initially preached the value of adaptive courses not so much as a tool for students but as one for professors, a way for them to gain greater insights into how their students were making their way through the material. Later, studies found at least some of the methods were also helping students complete courses more quickly and successfully than standard courses were.
The courses developed at OLI were important, she says, because they were created collectively by professors at all kinds of institutions with all kinds of students. "We need the robustness of that perspective and that context to design an environment that can meet the needs of the diversity of students."
But until she joined the faculty at Stanford, Ms. Thille had never actually taught a course — which only fueled some of the skepticism of her work from professors around the country who saw (and may still see) adaptive courses as an attempt to replace live professors.
Ms. Thille used to say publicly that any professor who could be replaced by a course deserved to be. She insists she always meant that as a compliment to the insights and skills that professors bring to teaching that an adaptive course could never replace. But she no longer uses that phrase because too many people took it the wrong way. She says she still believes it, though.
Ms. Thille received her doctorate from the University of Pennsylvania in 2013. And today she does regularly teach courses — four a year. That’s only reinforced her faith in the value of adaptive courses to instructors.
In her "Introduction to Data Analysis and Interpretation" course, where she uses OLI statistics courseware, she sees exactly what the students understand, and "based on that, I decide what we’re going to do in class the next day."
In another course, "Engineering Education and Online Learning," where she doesn’t use OLI courseware, "I’ll tell you, it is frustrating. I feel like I’m flying blind," she says. "I have no idea what meaning they’re making of it."
As part of her latest work, she’s also creating a way for professors to see why an algorithm does what it does, and to select different "skill-modeling" approaches if they prefer them. The professor wouldn’t have to be "some big data geek" to see what’s going on, she says. And even if professors aren’t interested in digging into that level of detail, they could have the confidence of knowing that the system had been "peer-reviewed in an academic way."
Commercial products, of course, offer some insights to professors, too, but not in the same detail. And some professors who use them say their opacity can be frustrating.
“We need to understand how students learn, and we're not getting that.”
Irene R. Foster, an assistant professor of economics at George Washington University who says she loves technology and new products, uses a McGraw-Hill product called Aleks to help her students brush up on the algebra they need for the course."We’re better off with it than without it," she says, but she finds its analytics engine a bit crude. "I don’t really know how smart it is," she says. When students miss problems, it just generates more questions on the same topic. "I’m not sure if it can fully capture why they are missing the problems." And in some cases, she says, the software doesn’t distinguish whether the student doesn’t understand the mathematics concept or simply messed up the underlying arithmetic. "We need to understand how students learn, and we’re not getting that."
R.G. Wilmot Lampros, chief product officer for Aleks, says the ideas that undergird the product, referred to as Knowledge Space Theory, were developed by professors at the University of California at Irvine and are in the public domain. It’s "there for anybody to vet," he says. But McGraw-Hill has no more plans to make the product's analytics algorithms public than Google would for its latest search algorithm.
"I know that there are a few results that our customers have found counterintuitive," he says, but the company’s own analyses of its algebra products have found they are 97 percent accurate in predicting when a student is ready to learn the next topic.
As for Ms. Thille’s broader critique, Mr. Lampros is not persuaded. "It’s a complaint about capitalism," he says. The original theoretical work behind Aleks was financed by the National Science Foundation, but after that ended, he says, "it would have been dead without business revenues."
On the Horizon
Ms. Thille stops short of decrying capitalism. But she does say that letting the market alone shape the future of learning analytics would be a mistake.
Colleges should be investing in learning analytics in the same way that they invest in maintaining their buildings, she says.
Some moves on the horizon could, in fact, give a boost to the movement to open up learning algorithms.
In June, Carnegie Mellon announced that it would openly license the OLI software, along with several other key pieces of software that are collectively based on more than $30 million in research. The university said it hoped that the technologies could serve as the foundation for a large-scale, open-source collaboration with other universities interested in forming a community of users and researchers in the field of learning analytics. Mr. Bier says they should be available by the summer.
Mr. Siemens says he and several other experts in learning analytics plan to seek NSF funding for a major project in open-analytics research.
Ms. Thille, meanwhile, is taking a page from the collaborative approaches she’s used in creating OLI courses to develop some new financing models for open-analytics research.
One idea, she says, could be a new technology platform to allow crowdsourced contributions from universities, enabling even "micro but meaningful participation" by professors who may lack the expertise or the time to play a major role but still want to contribute to the effort.
With the vocabulary of a data scientist, not a PR pro, Ms. Thille paints the choices starkly. "To commodify it at this point in time, when we’re still doing very active research in it, almost assures that we’ll get less innovative, suboptimal products," she argues. But if colleges themselves took up the learning-analytics challenge to "co-construct" learning activities, institutions could both "support teaching and learning and help us understand better how learning happens."
Goldie Blumenstyk writes about the intersection of business and higher education. Check out www.goldieblumenstyk.com for information on her new book about the higher-education crisis; follow her on Twitter @GoldieStandard; or email her at goldie@chronicle.com.
Join the conversation about this article on the Re:Learning Facebook page.
This article is part of:
Principle 2: What Students Already Know Affects Their Learning
20 psychological principles for teachers #2 Prior knowledge
http://www.learningspy.co.uk/psychology/top-20-psychological-principles-for-teachers-2-what-students-already-know-affects-their-learning/
http://www.learningspy.co.uk/psychology/top-20-psychological-principles-for-teachers-2-what-students-already-know-affects-their-learning/
Why Don't Students Like School: Chapter 4
“Why Is It So Hard for Students to Understand Abstract Ideas?”
Excerpt From: Daniel T. Willingham. “Why Don't Students Like School?.” iBooks. https://itun.es/us/SexLw.l
“The cognitive principle that guides this chapter is:
We understand new things in the context of things we already know, and most of what we know is concrete.”
Understanding Is Remembering in Disguise
“What do cognitive scientists know about how students understand things? The answer is that they understand new ideas (things they don’t know) by relating them to old ideas (things they do know).”
“One principle is the usefulness of analogies; they help us understand something new by relating it to something we already know about. ”
i.e storytelling
“Another consequence of our dependence on prior knowledge is our need for concrete examples.”
“It’s not the concreteness, it’s the familiarity that’s important; but most of what students are familiar with is concrete, because abstract ideas are so hard to understand.”
This is exactly why I advocate using Undergrad Teaching Assistants in my class. They let me know if the examples used in the lectures are "too out there" for the students in the class. Granted most Graduate students aren't too much older than the Undergrads, but they have a different perspective and they are taking more theoretical classes which can give them examples to use in class but may not be concrete enough for their students.
“So, understanding new ideas is mostly a matter of getting the right old ideas into working memory and then rearranging them—making comparisons we hadn’t made before, or thinking about a feature we had previously ignored.”
“No one can pour new ideas into a student’s head directly. Every new idea must build on ideas that the student already knows. ”
“To dig deeper into what helps students understand, we need to address these two issues. First, even when students “understand,” there are really degrees of comprehension. One student’s understanding can be shallow while another’s is deep. Second, even if students understand in the classroom, this knowledge may not transfer well to the world outside the classroom.”
This is where I struggle the most. We teach how to do a screen print in Week 2, explain what it does, and then by Week 5 when the students have to do it for a project, they "forget" or can't transfer what we did in the classroom to their homework assignment. I'm working on this by starting each project in class in a flipped classroom model. IMHO flipping the classroom helps with getting things out of long term memory into working memory.
Why Is Knowledge Shallow?
“Rote knowledge might lead to giving the right response, but it doesn’t mean the student is thinking."
BUYA! This is my quote of the day! Students may be comfortable with technology but are they actually thinking about technology and how to use it effectively.
“Rote knowledge (as I’m using the term) means you have no understanding of the material.”
“Much more common than rote knowledge is what I call shallow knowledge, meaning that students have some understanding of the material but their understanding is limited.”
“A student with deep knowledge knows more about the subject, and the pieces of knowledge are more richly interconnected. The student understands not just the parts but also the whole. This understanding allows the student to apply the knowledge in many different contexts, to talk about it in different ways, to imagine how the system as a whole would change if one part of it changed, and so forth. ”
“In addition, the student would be able to consider what if questions...”
“They can think through this sort of question because the pieces of their knowledge are so densely interconnected.”
“Thus deep knowledge means understanding everything—both the abstraction and the examples, and how they fit together. ”
Why Doesn’t Knowledge Transfer?
“If someone understands an abstract principle, we expect they will show transfer. When knowledge transfers, that means they have successfully applied old knowledge to a new problem."
“When we read or when we listen to someone talking, we are interpreting what is written or said in light of what we already know about similar topics.”
This is what makes teaching at a University so difficult. Students coming from all over the world,all with different educational experiences and interpreting based on what they already know. I suppose this could be an argument for a "common core curriculum" (for lack of a better term) across the world, BUT there's that pesky "interpretation" piece. No matter if there was a worldwide "common core curriculum" each and every student is going the make his/her own interpretations.
“The surface structures of the solved textbook problem and the new problem are different—one is about a hardware store’s inventory and the other is about cell phone plans—but the student knows he should disregard the surface structure and focus on the deep structure."
This is a question for the Math people... Are too many word problems the problem in getting the structural information or formulas into long term memory? Are students too focused on the semantics that they aren't understanding the structure of the math problem. Just food for thought :)
“In other words, it’s a mistake to think of our old knowledge transferring to a new problem only when the source of that background knowledge is obvious to us.”
Implications for the Classroom
To Help Student Comprehension, Provide Examples and Ask Students to Compare Them
Make Deep Knowledge the Spoken and Unspoken Emphasis
Make Your Expectations for Deep Knowledge Realistic
Excerpt From: Daniel T. Willingham. “Why Don't Students Like School?.” iBooks. https://itun.es/us/SexLw.l
“The cognitive principle that guides this chapter is:
We understand new things in the context of things we already know, and most of what we know is concrete.”
Understanding Is Remembering in Disguise
“What do cognitive scientists know about how students understand things? The answer is that they understand new ideas (things they don’t know) by relating them to old ideas (things they do know).”
“One principle is the usefulness of analogies; they help us understand something new by relating it to something we already know about. ”
i.e storytelling
“Another consequence of our dependence on prior knowledge is our need for concrete examples.”
“It’s not the concreteness, it’s the familiarity that’s important; but most of what students are familiar with is concrete, because abstract ideas are so hard to understand.”
This is exactly why I advocate using Undergrad Teaching Assistants in my class. They let me know if the examples used in the lectures are "too out there" for the students in the class. Granted most Graduate students aren't too much older than the Undergrads, but they have a different perspective and they are taking more theoretical classes which can give them examples to use in class but may not be concrete enough for their students.
“So, understanding new ideas is mostly a matter of getting the right old ideas into working memory and then rearranging them—making comparisons we hadn’t made before, or thinking about a feature we had previously ignored.”
“No one can pour new ideas into a student’s head directly. Every new idea must build on ideas that the student already knows. ”
“To dig deeper into what helps students understand, we need to address these two issues. First, even when students “understand,” there are really degrees of comprehension. One student’s understanding can be shallow while another’s is deep. Second, even if students understand in the classroom, this knowledge may not transfer well to the world outside the classroom.”
This is where I struggle the most. We teach how to do a screen print in Week 2, explain what it does, and then by Week 5 when the students have to do it for a project, they "forget" or can't transfer what we did in the classroom to their homework assignment. I'm working on this by starting each project in class in a flipped classroom model. IMHO flipping the classroom helps with getting things out of long term memory into working memory.
Why Is Knowledge Shallow?
“Rote knowledge might lead to giving the right response, but it doesn’t mean the student is thinking."
BUYA! This is my quote of the day! Students may be comfortable with technology but are they actually thinking about technology and how to use it effectively.
“Rote knowledge (as I’m using the term) means you have no understanding of the material.”
“Much more common than rote knowledge is what I call shallow knowledge, meaning that students have some understanding of the material but their understanding is limited.”
“A student with deep knowledge knows more about the subject, and the pieces of knowledge are more richly interconnected. The student understands not just the parts but also the whole. This understanding allows the student to apply the knowledge in many different contexts, to talk about it in different ways, to imagine how the system as a whole would change if one part of it changed, and so forth. ”
“In addition, the student would be able to consider what if questions...”
“They can think through this sort of question because the pieces of their knowledge are so densely interconnected.”
“Thus deep knowledge means understanding everything—both the abstraction and the examples, and how they fit together. ”
Why Doesn’t Knowledge Transfer?
“If someone understands an abstract principle, we expect they will show transfer. When knowledge transfers, that means they have successfully applied old knowledge to a new problem."
“When we read or when we listen to someone talking, we are interpreting what is written or said in light of what we already know about similar topics.”
This is what makes teaching at a University so difficult. Students coming from all over the world,all with different educational experiences and interpreting based on what they already know. I suppose this could be an argument for a "common core curriculum" (for lack of a better term) across the world, BUT there's that pesky "interpretation" piece. No matter if there was a worldwide "common core curriculum" each and every student is going the make his/her own interpretations.
“The surface structures of the solved textbook problem and the new problem are different—one is about a hardware store’s inventory and the other is about cell phone plans—but the student knows he should disregard the surface structure and focus on the deep structure."
This is a question for the Math people... Are too many word problems the problem in getting the structural information or formulas into long term memory? Are students too focused on the semantics that they aren't understanding the structure of the math problem. Just food for thought :)
“In other words, it’s a mistake to think of our old knowledge transferring to a new problem only when the source of that background knowledge is obvious to us.”
Implications for the Classroom
To Help Student Comprehension, Provide Examples and Ask Students to Compare Them
Make Deep Knowledge the Spoken and Unspoken Emphasis
Make Your Expectations for Deep Knowledge Realistic
Thursday, February 18, 2016
Flipped classroom video
Hi All!
I asked my office assistant, Madi, to find a video explaining the Flipped Classroom. She found a good one. I'm adding this to my class in the Fall.
Flipped Classroom Explained
I asked my office assistant, Madi, to find a video explaining the Flipped Classroom. She found a good one. I'm adding this to my class in the Fall.
Flipped Classroom Explained
Wednesday, February 17, 2016
General Technology links
Top 100 Tools for Learning 2015
http://c4lpt.co.uk/directory/top-100-tools/
Anki App
https://www.ankiapp.com/index.html#download
TheBrain http://www.thebrain.com/products/thebrain/download/
VoiceThread
https://voicethread.com/
Anki App
https://www.ankiapp.com/index.html#download
TheBrain http://www.thebrain.com/products/thebrain/download/
VoiceThread
https://voicethread.com/
TOP 20 PRINCIPLES FROM
PSYCHOLOGY FOR PreK–12
TEACHING AND LEARNING
Coalition for Psychology in Schools and Education
http://www.apa.org/ed/schools/cpse/top-twenty-principles.pdf
Coalition for Psychology in Schools and Education
http://www.apa.org/ed/schools/cpse/top-twenty-principles.pdf
The Case Against Mandating Math for Students
FEBRUARY 26, 2016
Why Don't Students Like School: Chapter 3
“Why Do Students Remember Everything That’s on Television and Forget Everything I Say?”
Excerpt From: Daniel T. Willingham. “Why Don't Students Like School?.” iBooks. https://itun.es/us/SexLw.l
“Your memory system lays its bets this way: if you think about something carefully, you’ll probably have to think about it again, so it should be stored. Thus your memory is not a product of what you want to remember or what you try to remember; it’s a product of what you think about.”
“Whatever students think about is what they will remember. The cognitive principle that guides this chapter is: Memory is the residue of thought.
To teach well, you should pay careful attention to what an assignment will actually make students think about (not what you hope they will think about), because that is what they will remember.”
Question for myself, based on the Underground railroad example, if my students struggle with the technology in my course, then they are "thinking" about their struggle with the technology and not the course content.
“things can’t get into long-term memory unless they have first been in working memory. So this is a somewhat complex way of explaining the familiar phenomenon: If you don’t pay attention to something, you can’t learn it!”
“Does the professor seem like a nice person, and is the class well organized? ”
“The emotional bond between students and teacher—for better or worse—accounts for whether students learn.”
The Power of Stories
“The human mind seems exquisitely tuned to understand and remember stories—so much so that psychologists sometimes refer to stories as “psychologically privileged,” meaning that they are treated differently in memory than other types of material. ”
the four Cs:
“causality, which means that events are causally related to one another.”
"conflict - a story has a main character pursuing a goal, but he or she is unable to reach that goal.”
"complications - subproblems that arise from the main goal.”
"character - a good story is built around strong, interesting characters, and the key to those qualities is action. A skillful storyteller shows rather than tells the audience what a character is like.”
“using a story structure brings several important advantages”
1) “First, stories are easy to comprehend, because the audience knows the structure, which helps to interpret the action.”
2) “Second, stories are interesting. ”
3) “Third, stories are easy to remember. Because comprehending stories requires lots of medium-difficulty inferences”
Putting Story Structure to Work
“The story structure applies to the way you organize the material that you encourage your students to think about, not to the methods you use to teach the material.”
“When it comes to teaching, I think of it this way: The material I want students to learn is actually the answer to a question. On its own, the answer is almost never interesting. But if you know the question, the answer may be quite interesting. That’s why making the question clear is so important. But I sometimes feel that we, as teachers, are so focused on getting to the answer, we spend insufficient time making sure that students understand the question and appreciate its significance."
But What If There Is No Meaning?
“Memorizing meaningless material is commonly called rote memorization.”
“There are times when a teacher may deem it important for a student to have such knowledge ready in long-term memory as a stepping-stone to understanding something deeper. ”
Implications for the Classroom
Review Each Lesson Plan in Terms of What the Student Is Likely to Think About
Think Carefully About Attention Grabbers
Use Discovery Learning with Care - “Discovery learning is probably most useful when the environment gives prompt feedback about whether the student is thinking about a problem in the right way. ”
Design Assignments So That Students Will Unavoidably Think About Meaning
Don’t Be Afraid to Use Mnemonics
Try Organizing a Lesson Plan Around the Conflict - “If I’m continually trying to build bridges between students’ daily lives and their school subjects, the students may get the message that school is always about them, whereas I think there is value, interest, and beauty in learning about things that don’t have much to do with me.” Is this the culture currently being created in K-12? Is this why I'm getting so much push back from students about my class?
Excerpt From: Daniel T. Willingham. “Why Don't Students Like School?.” iBooks. https://itun.es/us/SexLw.l
“Your memory system lays its bets this way: if you think about something carefully, you’ll probably have to think about it again, so it should be stored. Thus your memory is not a product of what you want to remember or what you try to remember; it’s a product of what you think about.”
“Whatever students think about is what they will remember. The cognitive principle that guides this chapter is: Memory is the residue of thought.
To teach well, you should pay careful attention to what an assignment will actually make students think about (not what you hope they will think about), because that is what they will remember.”
Question for myself, based on the Underground railroad example, if my students struggle with the technology in my course, then they are "thinking" about their struggle with the technology and not the course content.
“things can’t get into long-term memory unless they have first been in working memory. So this is a somewhat complex way of explaining the familiar phenomenon: If you don’t pay attention to something, you can’t learn it!”
https://belmontteach.files.wordpress.com/2014/06/slide1.jpg
“What’s more mysterious is why, when they are paying attention, they sometimes learn and sometimes don’t. What else is needed besides attention? A reasonable guess is that we remember things that bring about some emotional reaction. ”
“Things that create an emotional reaction will be better remembered, but emotion is not necessary for learning.”
“Repetition is another obvious candidate for what makes learning work.”
“One of the remarkable findings was that knowing about the future test didn’t improve subjects’ memories. ”
“In this case it matters because judging pleasantness makes you think about what the word means and about other words that are related to that meaning. ”
“Whatever you think about, that’s what you remember. Memory is the residue of thought.”
So if students are repeatedly told they are "digital natives", they assume this to be true because they repeatedly use smartphones thus reinforcing the notion they are digitally savvy. When they take my class, they are dismayed that they can't perform tasks using a desktop computer. The hypothesis I have come to since starting this class (EDUC792) is that the skills needed to use apps on mobile technologies are not the same as the skills needed to use applications on desktop computers. This is a problem because in the workforce for the foreseeable future, say 5 to 10 years when they will be entering the workforce, students will not be using their mobile devices solely to perform their work activities.
I've also being confronted several times this semester by students asking, not politely, why they have to take my class, whether is be because they are a Mac user or whether they believe they won't use the knowledge and skills in the course in their particular major. They can't see past their immediate situation or contemplate what may be needed in the future. They are in higher education to learn, but their resistance to learn new knowledge and skills baffles me.
The answer to my thoughts... “If you do think about something, then it’s likely that you’ll want to think about it in the same way in the future.”
“Therefore, a teacher’s goal should almost always be to get students to think about meaning.”
“For material to be learned (that is, to end up in long-term memory), it must reside for some period in working memory—that is, a student must pay attention to it. Further, how the student thinks of the experience completely determines what will end up in long-term memory. The obvious implication for teachers is that they must design lessons that will ensure that students are thinking about the meaning of the material.”
What Good Teachers Have in Common
“Trying to make the material relevant to students’ interests doesn’t work. ”
“Students often refer to good teachers as those who “make the stuff interesting.” It’s not that the teacher relates the material to students’ interests—rather, the teacher has a way of interacting with students that they find engaging.”
“Teachers need to be both well organized and approachable.”
“The emotional bond between students and teacher—for better or worse—accounts for whether students learn.”
The Power of Stories
“The human mind seems exquisitely tuned to understand and remember stories—so much so that psychologists sometimes refer to stories as “psychologically privileged,” meaning that they are treated differently in memory than other types of material. ”
the four Cs:
“causality, which means that events are causally related to one another.”
"conflict - a story has a main character pursuing a goal, but he or she is unable to reach that goal.”
"complications - subproblems that arise from the main goal.”
"character - a good story is built around strong, interesting characters, and the key to those qualities is action. A skillful storyteller shows rather than tells the audience what a character is like.”
“using a story structure brings several important advantages”
1) “First, stories are easy to comprehend, because the audience knows the structure, which helps to interpret the action.”
2) “Second, stories are interesting. ”
3) “Third, stories are easy to remember. Because comprehending stories requires lots of medium-difficulty inferences”
Putting Story Structure to Work
“The story structure applies to the way you organize the material that you encourage your students to think about, not to the methods you use to teach the material.”
“When it comes to teaching, I think of it this way: The material I want students to learn is actually the answer to a question. On its own, the answer is almost never interesting. But if you know the question, the answer may be quite interesting. That’s why making the question clear is so important. But I sometimes feel that we, as teachers, are so focused on getting to the answer, we spend insufficient time making sure that students understand the question and appreciate its significance."
But What If There Is No Meaning?
“Memorizing meaningless material is commonly called rote memorization.”
“There are times when a teacher may deem it important for a student to have such knowledge ready in long-term memory as a stepping-stone to understanding something deeper. ”
Implications for the Classroom
Review Each Lesson Plan in Terms of What the Student Is Likely to Think About
Think Carefully About Attention Grabbers
Use Discovery Learning with Care - “Discovery learning is probably most useful when the environment gives prompt feedback about whether the student is thinking about a problem in the right way. ”
Design Assignments So That Students Will Unavoidably Think About Meaning
Don’t Be Afraid to Use Mnemonics
Try Organizing a Lesson Plan Around the Conflict - “If I’m continually trying to build bridges between students’ daily lives and their school subjects, the students may get the message that school is always about them, whereas I think there is value, interest, and beauty in learning about things that don’t have much to do with me.” Is this the culture currently being created in K-12? Is this why I'm getting so much push back from students about my class?
Monday, February 8, 2016
Why Don't Students Like School: Chapter 2
“How Can I Teach Students the Skills They Need When Standardized Tests Require Only Facts?”
Excerpt From: Daniel T. Willingham. “Why Don't Students Like School?.” iBooks. https://itun.es/us/SexLw.l
“Research from cognitive science has shown that the sorts of skills that teachers want for students —such as the ability to analyze and to think critically—require extensive factual knowledge. The cognitive principle that guides this chapter is:
Factual knowledge must precede skill.”
“In this chapter I show that this argument is false. Data from the last thirty years lead to a conclusion that is not scientifically challengeable: thinking well requires knowing facts, and that’s true not simply because you need something to think about. The very processes that teachers care about most—critical thinking processes such as reasoning and problem solving—are intimately intertwined with factual knowledge that is stored in long-term memory (not just found in the environment).”
Not having studied cognitive science before, I'm finding this book very interesting and thought provoking. The interconnection of long-term memory and working memory explains a lot of what I'm seeing in my students behaviors. They are great at using their phones and apps but not so much when it comes to desktop applications. Their confidence in using their phones leads them into believing they are great at using computers (society and the media tell them this too). Apps and applications don't necessarily work the same and therefore there's no long-term memory from which to pull from when they need it.
Additionally, they tend to use the "save credentials" options so they aren't required to enter usernames and passwords. Granted it saves time, but then the usernames and passwords aren't placed into long-term memory and able to be recalled when needed. It would be interesting to see how many times students reset their eID passwords with say a 12-month period of time due to their lack of remembering the password...
I'm sure there will be more on this topic in the spacing article which I haven't read yet.
“The phenomenon of tying together separate pieces of information from the environment is called chunking. The advantage is obvious: you can keep more stuff in working memory if it can be chunked. The trick, however, is that chunking works only when you have applicable factual knowledge in long-term memory. ”
“Thus, background knowledge allows chunking, which makes more room in working memory, which makes it easier to relate ideas, and therefore to comprehend.”
“This experiment indicates that we don’t take in new information in a vacuum. We interpret new things we read in light of other information we already have on the topic.”
“I’ve listed four ways that background knowledge is important to reading comprehension: (1) it provides vocabulary; (2) it allows you to bridge logical gaps that writers leave; (3) it allows chunking, which increases room in working memory and thereby makes it easier to tie ideas together; and (4) it guides the interpretation of ambiguous sentences.”
Background Knowledge Is Necessary for Cognitive Skills
“Generalizations that we can offer to students about how to think and reason successfully in the field may look like they don’t require background knowledge, but when you consider how to apply them, they actually do.”
Factual Knowledge Improves Your Memory
“The researchers had people learn either a lot or just a little about subjects that were new to them (for example, Broadway musicals). Then they had them read other, new facts about the subject, and they found that the “experts” (those who had earlier learned a lot of facts about the subject) learned new facts more quickly and easily than the “novices” (who had earlier learned just a few facts about the subject).”
“We remember much better if something has meaning.”
“Television, video games, and the sorts of Internet content that students lean toward (for example, social networking sites, music sites, and the like) are for the most part unhelpful. Researchers have painstakingly analyzed the contents of the many ways that students can spend their leisure time. Books, newspapers, and magazines are singularly helpful in introducing new ideas and new vocabulary to students.”
SNARK ALERT: OK that explains things... Social media and iTunes are destroying our kid's long-term memory.
“Knowledge is more important, because it’s a prerequisite for imagination, or at least for the sort of imagination that leads to problem solving, decision making, and creativity.”
“As I’ve shown in this chapter, the cognitive processes that are most esteemed—logical thinking, problem solving, and the like—are intertwined with knowledge.”
Implications for the Classroom
How to Evaluate Which Knowledge to Instill
“The question, What should students be taught? is equivalent not to What knowledge is important? but rather to What knowledge yields the greatest cognitive benefit?”
“Cognitive science leads to the rather obvious conclusion that students must learn the concepts that come up again and again—the unifying ideas of each discipline. ”
Be Sure That the Knowledge Base Is Mostly in Place When You Require Critical Thinking
“Critical thinking is not a set of procedures that can be practiced and perfected while divorced from background knowledge.”
Shallow Knowledge Is Better Than No Knowledge
Do Whatever You Can to Get Kids to Read
Knowledge Acquisition Can Be Incidental
Start Early
Knowledge Must Be Meaningful
Excerpt From: Daniel T. Willingham. “Why Don't Students Like School?.” iBooks. https://itun.es/us/SexLw.l
“Research from cognitive science has shown that the sorts of skills that teachers want for students —such as the ability to analyze and to think critically—require extensive factual knowledge. The cognitive principle that guides this chapter is:
Factual knowledge must precede skill.”
“In this chapter I show that this argument is false. Data from the last thirty years lead to a conclusion that is not scientifically challengeable: thinking well requires knowing facts, and that’s true not simply because you need something to think about. The very processes that teachers care about most—critical thinking processes such as reasoning and problem solving—are intimately intertwined with factual knowledge that is stored in long-term memory (not just found in the environment).”
Not having studied cognitive science before, I'm finding this book very interesting and thought provoking. The interconnection of long-term memory and working memory explains a lot of what I'm seeing in my students behaviors. They are great at using their phones and apps but not so much when it comes to desktop applications. Their confidence in using their phones leads them into believing they are great at using computers (society and the media tell them this too). Apps and applications don't necessarily work the same and therefore there's no long-term memory from which to pull from when they need it.
Additionally, they tend to use the "save credentials" options so they aren't required to enter usernames and passwords. Granted it saves time, but then the usernames and passwords aren't placed into long-term memory and able to be recalled when needed. It would be interesting to see how many times students reset their eID passwords with say a 12-month period of time due to their lack of remembering the password...
I'm sure there will be more on this topic in the spacing article which I haven't read yet.
“The phenomenon of tying together separate pieces of information from the environment is called chunking. The advantage is obvious: you can keep more stuff in working memory if it can be chunked. The trick, however, is that chunking works only when you have applicable factual knowledge in long-term memory. ”
“Thus, background knowledge allows chunking, which makes more room in working memory, which makes it easier to relate ideas, and therefore to comprehend.”
“This experiment indicates that we don’t take in new information in a vacuum. We interpret new things we read in light of other information we already have on the topic.”
“I’ve listed four ways that background knowledge is important to reading comprehension: (1) it provides vocabulary; (2) it allows you to bridge logical gaps that writers leave; (3) it allows chunking, which increases room in working memory and thereby makes it easier to tie ideas together; and (4) it guides the interpretation of ambiguous sentences.”
Background Knowledge Is Necessary for Cognitive Skills
“Generalizations that we can offer to students about how to think and reason successfully in the field may look like they don’t require background knowledge, but when you consider how to apply them, they actually do.”
Factual Knowledge Improves Your Memory
“The researchers had people learn either a lot or just a little about subjects that were new to them (for example, Broadway musicals). Then they had them read other, new facts about the subject, and they found that the “experts” (those who had earlier learned a lot of facts about the subject) learned new facts more quickly and easily than the “novices” (who had earlier learned just a few facts about the subject).”
“We remember much better if something has meaning.”
“Television, video games, and the sorts of Internet content that students lean toward (for example, social networking sites, music sites, and the like) are for the most part unhelpful. Researchers have painstakingly analyzed the contents of the many ways that students can spend their leisure time. Books, newspapers, and magazines are singularly helpful in introducing new ideas and new vocabulary to students.”
SNARK ALERT: OK that explains things... Social media and iTunes are destroying our kid's long-term memory.
“Knowledge is more important, because it’s a prerequisite for imagination, or at least for the sort of imagination that leads to problem solving, decision making, and creativity.”
“As I’ve shown in this chapter, the cognitive processes that are most esteemed—logical thinking, problem solving, and the like—are intertwined with knowledge.”
Implications for the Classroom
How to Evaluate Which Knowledge to Instill
“The question, What should students be taught? is equivalent not to What knowledge is important? but rather to What knowledge yields the greatest cognitive benefit?”
“Cognitive science leads to the rather obvious conclusion that students must learn the concepts that come up again and again—the unifying ideas of each discipline. ”
Be Sure That the Knowledge Base Is Mostly in Place When You Require Critical Thinking
“Critical thinking is not a set of procedures that can be practiced and perfected while divorced from background knowledge.”
Shallow Knowledge Is Better Than No Knowledge
Do Whatever You Can to Get Kids to Read
Knowledge Acquisition Can Be Incidental
Start Early
Knowledge Must Be Meaningful
Thursday, February 4, 2016
Tuesday, February 2, 2016
Why Don't Students Like School: Chapter 1
“People like to solve problems, but not to work on unsolvable problems. If schoolwork is always just a bit too difficult for a student, it should be no surprise that she doesn’t like school much.”
Excerpt From: Daniel T. Willingham. “Why Don't Students Like School?.” iBooks. https://itun.es/us/SexLw.l
Curious that he uses the pronoun "she" in this sentence when it's generally thought men/males are the problem solvers or "fixers".
“The cognitive principle that guides this chapter is:
People are naturally curious, but we are not naturally good thinkers; unless the cognitive conditions are right, we will avoid thinking.”
Interesting... I seem to think, come up with new ideas ALL THE TIME. It's actually quite frustrating because unless I write them down immediately they're gone or forgotten. So I usually text my office assistant or email myself the idea.
“The implication of this principle is that teachers should reconsider how they encourage their students to think, in order to maximize the likelihood that students will get the pleasurable rush that comes from successful thought.”
The Mind Is Not Designed for Thinking
“In an empty room are a candle, some matches, and a box of tacks. The goal is to have the lit candle about five feet off the ground. You’ve tried melting some of the wax on the bottom of the candle and sticking it to the wall, but that wasn’t effective. How can you get the lit candle five feet off the ground without having to hold it there?”
I know the answer to this one!!!
Excerpt From: Daniel T. Willingham. “Why Don't Students Like School?.” iBooks. https://itun.es/us/SexLw.l
Curious that he uses the pronoun "she" in this sentence when it's generally thought men/males are the problem solvers or "fixers".
“The cognitive principle that guides this chapter is:
People are naturally curious, but we are not naturally good thinkers; unless the cognitive conditions are right, we will avoid thinking.”
Interesting... I seem to think, come up with new ideas ALL THE TIME. It's actually quite frustrating because unless I write them down immediately they're gone or forgotten. So I usually text my office assistant or email myself the idea.
“The implication of this principle is that teachers should reconsider how they encourage their students to think, in order to maximize the likelihood that students will get the pleasurable rush that comes from successful thought.”
The Mind Is Not Designed for Thinking
“In an empty room are a candle, some matches, and a box of tacks. The goal is to have the lit candle about five feet off the ground. You’ve tried melting some of the wax on the bottom of the candle and sticking it to the wall, but that wasn’t effective. How can you get the lit candle five feet off the ground without having to hold it there?”
I know the answer to this one!!!
If you haven't seen Daniel Pink's TED talk it's a good one:
Dan Pink: The puzzle of motivation
“This problem illustrates three properties of thinking. First, thinking is slow. ”
“Second, thinking is effortful; you don’t have to try to see, but thinking takes concentration.”
“Finally, thinking is uncertain. ”
“How does a teacher make the hundreds of decisions necessary to get through her day? The answer is that when we can get away with it, we don’t think. Instead we rely on memory. ”
“—but your memory system is much more reliable than your thinking system, and it provides answers quickly and with little effort.”
My memory is better than I thought...
“...two ways in which your brain is set up to save you from having to think. First, some of the most important functions (for example, vision and movement) don’t require thought”
“Second, you are biased to use memory to guide your actions rather than to think. ”
So if students are Googling things all the time and not using their memory, how much is the use of technology impairing memory?
People Are Naturally Curious, but Curiosity Is Fragile
“In the last ten years neuroscientists have discovered that there is overlap between the brain areas and chemicals that are important in learning and those that are important in the brain’s natural reward system. ”
This would seem to support the use of electronic learning systems where students get immediate feedback.
“So, if content is not enough to keep your attention, when does curiosity have staying power? The answer may lie in the difficulty of the problem. If we get a little burst of pleasure from solving a problem, then there’s no point in working on a problem that is too easy—there’ll be no pleasure when it’s solved because it didn’t feel like much of a problem in the first place. Then too, when you size up a problem as very difficult, you are judging that you’re unlikely to solve it, and are therefore unlikely to get the satisfaction that comes with the solution.”
Hmmm... something to think about when I design lessons. If the content is too easy for most students, will they loose interest at the beginning and not follow through when the content of the course gets harder or more unfamiliar?
“To summarize, I’ve said that thinking is slow, effortful, and uncertain. Nevertheless, people like to think—or more properly, we like to think if we judge that the mental work will pay off with the pleasurable feeling we get when we solve a problem. So there is no inconsistency in claiming that people avoid thought and in claiming that people are naturally curious—curiosity prompts people to explore new ideas and problems, but when we do, we quickly evaluate how much mental work it will take to solve the problem. If it’s too much or too little, we stop working on the problem if we can.”
How Thinking Works
“Long-term memory is the vast storehouse in which you maintain your factual knowledge of the world”
“Just about the simplest model of the mind possible.”
“Thinking occurs when you combine information (from the environment and long-term memory) in new ways.”
Snarky comment... 18 X 7 I did it as described in the book. How does "new math" get to the answer to this problem and does it involve procedural knowledge?
“...procedural knowledge, which is your knowledge of the mental procedures necessary to execute tasks. If thinking is combining information in working memory, then procedural knowledge is a list of what to combine and when—it’s like a recipe to accomplish a particular type of thought. ”
“In sum, successful thinking relies on four factors: information from the environment, facts in long-term memory, procedures in long-term memory, and the amount of space in working memory. If any one of these factors is inadequate, thinking will likely fail.”
Implications for the Classroom
Be Sure That There Are Problems to Be Solved
Respect Students’ Cognitive Limits
- “The solution to working memory overloads is straightforward: slow the pace, and use memory aids such as writing on the blackboard that save students from keeping too much information in working memory.”
Reconsider When to Puzzle Students
- “by conducting a demonstration or presenting a fact that we think students will find surprising”
Change the Pace
Keep a Diary
Subscribe to:
Posts (Atom)