Classroom Metrics – Real World Case Study for Games in Schools – Extra Credits
- Articles, Blog

Classroom Metrics – Real World Case Study for Games in Schools – Extra Credits


It would be immensely helpful if we could
measure the impact of games in the classroom to assess their effectiveness as teaching tools. Fortunately, somebody went and did that already. ♪ [Extra Credits theme] ♪ Over the course of his work with Games For Good, James has often spoken with
the folks at Filament Games, a company that not only creates educational games, but actually works on the much harder task of
making games that are classroom ready. Now, here on Extra Credits, we often talk about what good
games could do in the classroom, but they’re actually making games that live up
to the almost impossible standard of fitting the curriculum currently mandated for schools, so that a teacher could use them
in the classroom right now, today, without worrying that it might
not meet all the required standards. Last time James talked to them, he spoke about how important
getting real, concrete data on the efficacy of games
in the classroom was to his work, and, to his surprise, they told him that they had
recently finished just such a study. So it is with their help and thanks to their hard work, that we are able to talk about it on the show today – to spread the word and, hopefully,
to encourage others to pursue similar studies. In a previous episode about
games’ literacy in the classroom, we discussed the barriers that exist
to widespread classroom adoption of games, focusing on the importance of promoting literacy and game mechanics and terminology to educators. Today, I want to focus on what happens
when we move past those barriers, and explain why games are so important
for education in the process. The first and most obvious answer to the question of: “Why games for education?” is that the current educational environment
practically begs for games content. Trends like the mandate for 21st century skills, one-to-one programs, the maker movement, project-based learning, and personalized learning are all concepts that are complex enough
to deserve their own episodes, but, in short, each of these trends is
oriented towards preparing students for a working world that demands teamwork, an understanding of technology, and self motivation. And there’s tons of research
to show that learning games work, but games are only a part of the equation. Teachers are the other core piece. There was a great article on this in the Journal of Educational Psychology, but, short version: When we think about games in the classroom,
we *have* to think about teachers, because teachers are designing
the experiences their students have every day. It’s up to the teacher to create or
select instructional tools and decide how best to
incorporate them into their lessons. These decisions are as important,
maybe more important, than the qualities of the particular tools involved. Using games can improve learning
under a variety of circumstances, but the greatest games have been achieved by teachers who surround students game experiences with additional support and instruction. So, for the folks at Filament, curricula was the first and most obvious feature to wrap around the games, making it easier for the teacher
to bring games into the classroom by providing teachers with an obvious place
to insert games into their current curriculum. Then, to couple that with the other
advantage games have for teachers, their next priority feature was metrics. They created a way for teachers to get a more in depth look into how each student was learning. After all, normally a teacher only gets to see the end result of the student’s homework. But games can record everything along the way, providing a lot more information about
how and where a student may struggle. Then, with all of this in place, they reached out
to Sun Prairie School District in Wisconsin, which deployed the Filament Learning Games library across their entire district. They worked with Mary Headington, a fantastic teacher who was eager to help them study how one game from the Filament Learning Library made a difference in her classroom. Because, truth is, even though we all believe – and I would even say know –
that games have incredible potential as learning tools – I mean, James has seen it, Filament has seen it – without hard data about they can be applied *right now* in classrooms, it’s really hard to get major institutions to adopt them. Besides, without this data,
as much as we know that games can help, we don’t know exactly how to refine our games, where our assumptions may be wrong, or exactly how we can
better integrate games into the classroom. So getting that data is what Filament set out to do, which brings us back to the study with Mary Headington. The game she used is called Planet Mechanic – a sandbox light simulation
of the Sun, Earth, Moon system, in which students can manipulate planetary bodies to see the impact those manipulations have on the ground level planetary conditions. Students can adjust features like
atmosphere, tilt, rotation and lunar cycles to change temperature, atmospheric
pressure, and time on the planet. These are concepts that are pretty
challenging to grasp without a tangible representation, and the game makes them more
digestible and accessible to students. For her study, Mary had
all of her students take a pre-assessment to test their current knowledge
of the content in the game. She then split her students into three sections: a section where students would only play the game, a section where students would only
receive traditional instruction from her, and a section where students would play both the game and supplement that
gameplay with traditional instruction. Each section used the same amount of class time, and the sections that
received direct instruction from Mary were tasked with a variety of supplemental activities. Students viewed
interactive visualizations, drew diagrams, and used hands-on models of the Earth and Sun to reinforce their learning. And afterwards, all three sections
took another post-assessment. The results might surprise you. The scores of students who only played the game were almost flat. The average grades for this group went up by .1%. Students who only received traditional
instruction fared considerably better, scores went up by an average of 6%, showing strong growth between the pre- and post-test, thanks to Mary’s skilled guidance. But the students who played the game
*and* received traditional instruction? For those students, the average scores
went up by a whopping ten percent – a full letter grade. So what does this indicate about
the place of games in the classroom? These results verify that the teacher’s role
in guiding the classroom is as crucial as ever, with the games acting as a
force multiplier for her instruction. This is how we need to view games for learning: a pedagogical force multiplier. I guess I should explain that. A force multiplier refers to some factor
that increases or multiplies effectiveness. In the context of the classroom, a learning
game takes the content we deliver to students and amplifies their engagement,
investment, and retention. The results of the study are clear: without the educator’s critical role in the classroom,
the game as a stand alone tool doesn’t suffice. But, with an educator’s guidance,
the game helps the educator to exceed the results that they
would get in the absence of that game. And Mary had her own thoughts to share
on what games accomplished for her students. She said, “Games are never going to replace the teacher. But games *are* changing how the teacher teaches. Traditional instruction focuses on providing students with the information we want them to know and then delivering it in a controlled manner. But in this study, we witnessed students struggling with the concept and then determine what they needed to know. They then used this information
to direct their own personalized learning.” This looks different for each student. Some students chose to conference with the teacher. Some looked to classroom resources and Internet links, while others went out on their own
in search of the information. These skills teach students *how* to learn,
as opposed to being told *what* to learn. If we provide situations for students
to continually practice these skills, especially in a game environment, we will find that their motivation and thirst for knowledge increases. Teachers are finding that education is becoming
more of a partnership with the student. and games foster that dynamic. As we move more and more rapidly toward a future that suffuses technology into every aspect of our lives, the idea that video games can serve as a
foundational tool for instruction will become increasingly mainstream. Filament is trying to do their part
to lay the groundwork for an era of classroom content that includes video games by default, but they need help from the communities
that serve as stakeholders for school districts. If you want to see your local school district
adopt a game based learning program, get involved and advocate for more forward facing programs in your community. Schools really need that community support
to justify the hardware and software purchases that equip their students for the future. If they’re past the hardware adoption
point, and are ready to look at content, point them to the
Filament Games District Implementation Guide. We’ve put a link down below. It’s a free resource that helps
administrators wrap their heads around adding game based learning to their curricula offerings. It’s a pretty easy argument to make. [Outro music starts playing]
Our success as a country depends on a well functioning and effective educational system. It’s on us to fill that system with forward facing programs like game based learning that help to maximize our students’ acheivement and to pave their way to success in the modern world. Thank you for watching, and we’ll see you next week. ♪ [Outro music] ♪

About James Carlton

Read All Posts By James Carlton

100 thoughts on “Classroom Metrics – Real World Case Study for Games in Schools – Extra Credits

  1. Good idea…but one limited study on one subject with a subsection of a group leaves alot lacking. I do strongly believe the findings…but we need more hard facts to back it up especially for those who doubt it.

  2. I feel like leaving students with traditional resources is just as effective as leaving them with new ones. Some people might look at this study and say " well up by 6% is pretty good, so we should just keep doing what we're doing then." But this isn't what it's saying. From what I understand, the study showed (intentionally or not) that tools alone, be they sticks or the holodeck, can not teach a student. A teacher must be working with a student to help them learn. If the student is only presented with the knowledge it won't sink in very well. Most teachers only present this information and it works about as well as if you just left them to read the textbook…or play the game. The group that was taught traditionally in this study was actively working with the teacher, using tools other than games. We do this in schools today already. This is what lab projects and field trips are. They're just a different kind of tool. A hammer is no more important than a saw, they just have different uses. The point is just that video games are a very effective tool for both teacher and student. To keep up with this tools analogy: it's like a screwdriver verses a drill. Both accomplish the same thing, but one does it much more effectively. Although both fail to do anything if there is not someone properly skilled using them.

  3. A study about gamified learning initiated by a company that makes their money by gamified learning comes to the conclusion that gamified learning is a good idea. Yeah, sure, big whoop. Next time I consult the Monsanto-funded studies to asses if GMO's are a good idea.

    The results may be valid, but sometimes the messenger is a valid point to dismiss the result until replicated by a non-involved party or put through rigorous peer review. And shame on EC for uncritically hooray this without any shred of critical distance – at best overly enthusiastic, but potentially misleading edutainment, at worst undisclosed advertisement. Your pick is as good as mine.

  4. I am a professional tutor, I love games, play games, and have made a few amature games. Is there some way I can get involved with Games for Good? testing materials, or volunteering my knowledge and skills in some way?

  5. I wish I'd had games to encourage learning how to learn when I was growing up. Great work, EC and Filament Games!

  6. As expected, the hybrid system works best.
    Here's my issue- all the gamers are up in arms like 'yay, games can help education, woo!'
    But what about people who don't have an interest in videogames? They'll just end up being left behind if they aren't the type to quickly grasp game mechanics or concepts.

    My other concern is eyestrain, as someone who has been staring at screens for multiple years(am 18 now, and my perscriptiom keeps going up) I don't really feel comfortable with bright screens becoming a standard in classrooms. I like reading on paper or a kindle paperwhite more, for example.

  7. The methods described for the study were flawed. There was no control group that received no instruction whatsoever. Taking a pre-assessment can just make you better at the final assessment all on its own, depending on the questions, content, and analysis given. Also, were the students randomly assigned?

    Also, you cannot take these results as having any statistical significance without some analysis. If it were only 30 students with 10 in each group, you'd be very hard pressed to show this was more than just a change by chance. These are promising results, but it doesn't actually mean much.

  8. Do you think comparisons between this and the effect of the Hamilton Musical? As an engaging medium that makes the student want to search out their own information.

  9. This is one of many pieces to the puzzle of how to get our children to learn how to learn when young then to apply it when older
    At this time common core almost mandates 1 teaching style to be used and never went away from effectively tell them to learn in this one way or fail
    This video is a peice to use to get that part of common core out of the way to help students learn quicker and retain knowledge better

  10. Hey Extra Credits, could you guys go over how you make a Power Curve for a game? You talked about it back in your"Delta of Randomness" episodes and it would be nice to know how to approach it as a designer.

  11. What if the game had its own teacher (AI)? Or the game had an uplink to the teacher?

    What if the game adopted a mobile model with challenges that give rewards to the player?

  12. That classroom study had no control group of students who were taught using conventional methods, so the game results are essentially pointless. If you're going to bring games into the classroom you're going to have to show they are more capable than conventional methods when it comes to teaching kids.

  13. At the beginning I want to say that I am a big, BIG fan of ExtraCredits. I am also very, very inde developer. We have in front of us the first attempt to create a role-playing game from scratch. Unfortunately, knowledge about the structure of branched dialogues systems is very … general. Does anyone know where I can find more detailed information on the construction of such systems? I would be grateful.

  14. Your argument seems convincing on the average student, but I do wonder where the 4 procentile difference comes from. Are the top of the students super motivated and are they increasing the average by a lot (those who were already motivated but needed extra stimula in any way shape of form) and the lower rank students dont increase? Or is every student's performance increased equally?

    Was the study done in a classroom with all levels mixed and showed classrooms with seperated levels of the students all the same consistent result?

  15. I'm a 17 year old aspiring to become a research psychologist in the future, and an area I'm interested in is how psychology can inform education. I'm also an avid gamer, who wants to see games used for good. So this video was perfect for me, I'm going to look into the Filament company more! Perhaps in the future I'll be able to contribute research looking at how video games can be involved in education.

  16. I'm actually reading a book on teaching, and the very first chapter focuses on how a teacher is someone who is supposed to cause the student to learn, rather than simply giving the student the information. It seems as though games will essentially be a really effective tool to further this core of a teacher. Really cool to see

  17. My science teacher uses this and it actually really helps. If you want to try this, try the gizmo website. It has tons of games that can really help.

  18. Oh wow… I just realized that the question is not putting games "in" to education but rather putting games BACK into education. Before the advent of computers you had to do a lot more analysis and problem solving for even the simplest tasks.

    For example – Looking up how to spell a word in a physical dictionary means having to do logical deduction on the probable spelling of that word. Using that physical dictionary also means being exposed to NEW words every time you turn a page.

    The Dewey Decimal system shelves books by IDEAS, not just author last name. So the book that is useful for your project is probably sitting next to several other useful books. And every research project I had needed to use at least 3 or 4 different "idea sections" in the library. And every search of the library walks you past ideas you've never seen before. Modern wiki searches are much more specific then that. So younger students have lost that element of getting "lost" in new knowledge. (Getting sucked into TV tropes is not the same)

    And what about the traditional essay tests that are now being replaced by standardized multiple choice questions? Or the even more outdated ORAL exams? There is an element of deduction, creativity and logical thinking you don't need for a scan-tron test.

    How many other elements of modern gaming can you think of that exist in old-fashioned schooling methods?

  19. I hereby call for all paradox grand strat games be mandatory in history classes. It's all I ever did in history class anyway so it would be an improvement for me no matter what.

  20. This study sounds kinda fishy… "company that makes educational game did a study to find out how efficient educational games are." Is the way its coming across as to me.

  21. Love the idea that exploration, direct advisement, and archival access based learning is being looked at in all of this. The less we shove kids into a corner with how they approach learning, the more they'll remain interested. That 4% bonus to scores when games are used in tandem with curriculum is a fantastic finding.

  22. Game based education will probably be easier adopted by the society if it gets another (more conservative?) name.
    What this "official" name could be? How about "Interactive education"?

  23. "It's a pedagogical force multiplier. I guess I should explain that. A force multiplier is…"
    That's not the part of that sentence that confused me, you jerk.

  24. I hope she's got a second test in line to evaluate retention of the knowledge…. I'm curious to see if that changes anything.

  25. The problem is getting the students interest in the subject material which the video games industry has mastered to the point of it being labeled as an addiction yet the main mechanics that cause this are often discarded with the main focus of providing a bare bones gimmick that has exactly the content for it to be "educational material".

    The best educational video game simply puts the player in an environment which is heavily influenced by the subject being studied and gives the player goals which require them to understand the material to complete. This way the game gives the player information that they are searching for rather than force feeding them then punishing them if they lose.

  26. dude this is awesome but I am a new designer and most of these videos having no l relevance to what to do

  27. Please talk about evoland (1 and 2), those games are so great for this chanel because these games use a lot of diferent mechanics in the same game, if James work (and because of that the extra credits chapters) are about of that this is a great game to be an example, the first use zelda, final fantasy and even diablo mechanics (i´m not an expert so i only use the more know game with that mechanic), the second uses as many as i know and more, so it is a very good game to see for future developers (i´m only a gamer, not a developer)

  28. i don't think the instructor was necessary i think the choice of game was simply bad.. a sandbox game has no guidance the kids did whatever was coolest… and thus didn't learn a more directed game with a clear goal could accomplish the role the teacher played in the study. a decent example of how a fun game could teach without an instructor would be KSP. if it was equipped with a standard educational information section similar to a text book but phrased more like a walk through or player handbook a student could learn the rigor.. i suppose another study is needed filament

  29. I discovered you guys only yesterday, but I've rarely encountered channels as educative as yours, which is quite a feat. Keep up the great work!

  30. A few problems. There is a conflict of interest when the company selling the games also runs the study for how good the games are. Next the study is too small and the increase could be a statistical anomaly. Finally its only a 4% increase over the teacher, likely not significant enough to justify spending a large amount of money on it. There are already proven things that can be done to improve student scores, like smaller classroom sizes, and giving more incentives(prizes which almost always come out of the teachers money). I am disappointed that Extra Credits would be so excited about a single isolated, possibly biased study with only a 4% increase. I might be a little jaded as a teacher that has seen school districts waste large amounts of money on what ever new fad in education was going around with no change in student achievement. I know studies can be rigged and I feel more research through independent groups are needed before spending lots of money to push this out on a wide scale.

  31. Of course, the fact that the game was a sandbox might have had some effects. I imagine a more intentionally instructive game might have fared slightly better on its own, but that's neither here nor there.

  32. The Four C's of 21st Century Skills? I had never heard of that…and I don't think any of that was attempted to be taught in any of the schools I was at. Possibly "Collaborator" due to there being lots of group projects and "Communicator" in that regurgitation of information involves effective communication, but that's about it.

    Then again, the school district I was in was desperate to try not to look like a failing one, and so it was centered around standardized tests. It's easy to show off to the rest of the world good test scores…though it failed even in that. Anyone remember the California High School Exit Exam? It had a 50% pass rate when I graduated and only asked the test-takers to understand the questions being asked. Then again, the students were kind of dysfunctional, and at my high school, a huge amount (perhaps a good 25% of them) purposefully failed them by turning in blank answer sheets.

    Still, I can say that my schools never prepared me for the working world. For college, maybe, but there was zero interest in what happens afterward. Not when there are test scores to improve.

  33. Quantitatively, I don't think that 6% to 10% increase to be that much significant. How many times was the study replicated? Sadly, this might as well be purely coincidental for what I know. Don't misunderstand me, I love the idea of teaching with games but studies as such should be carried away more rigorously. Otherwise they're just hurting the credibility of the cause.

  34. Now they just use god awful online math made by some one on the other side of the united states. And its not the material the teacher uses so what the hell are they there for. I'm actually for this but I think it'll take the entire corrupt system crashing down to change anything.

  35. I am pursuing a degree in K-12 Music Educator, and every education episode of yours I watch is fantastic, but always leaves me with a question: Can games be implemented into a musical classroom environment? If so, how? Is it as simple as the music simulators like Rock Band? What effects would games have at introducing students to new styles of music? What would the implication of studying video game music in conjunction with older styles have in broadening student's musical tastes? And other such questions

  36. I made a game on scratch for a presentation in my history class and although they probably did teach them a lot (13 year old, not an industry veteran so the games implemented would be infinitely better) it definitely got a lot of people really interested to learn about it and pay more attention to our teacher.

  37. 6:30 No. The students that played only the game didn't play the game at all. They weren't introduced to the topic so they thought it would be boring and never played it.

    And even if they opened it, the game is just a simulation. You have to play it for a pretty long while to come up with the concepts in the classroom. If the game came with instructions as well, it would perform much better…

  38. In an economy class my class played a game in which we were in charge of equal sized companies. We could spend money on R&D, advertising, number of products to produce and some other things that I forget.

    The point is that it gave me some hands on experience with supply and demand. We sold 2800 units two months ago 2900 units last month and we just doubled our spending on marketing so I bet we should produce 3200 products and hopefully we don't run out of products by the end of the month.

  39. I work for a company where we're taking this up a step. we teach kids k-8 programming, and have them program their own games. A lot of schools in the area do as well.

  40. I have a problem with your conclusion. After the test the scores went up by .1%, 6% and 10% respectively. And you conclde that therefore games are only viable with the guidance of a teacher who supplements the game or lets the game supplement the class.

    But the game used in that class was supposed to be a supplement tool and not designed to teach a student everything so you can't draw that conclusion in general.
    There might be games (or at least the possibility of such) that can teach a student everything he needs to know on a specific topic.
    I too think that a learning game is better with a teacher supporting it but you can't draw this conclusion out of the example you provided. And the test group was also not sufficiently large enough to compensate for random results so a study is needed with more test subjects.

  41. The 10% increase group where prob honor role students. What other group would have been tasked with doing twice as much work?

  42. There are multiple skills that games arent going to help , such as reading , hand eye coordination, critical thinking, and besides people often forget that students are one of the main factors in their learning, many of them are simply lazy. studing is hardwork and takes a lot of time and thinking on how to better learn. Until people start demanding more from kids education wont improve.

  43. While I do agree video-games would be an excellent tool (and studies prove that), the problem is the education system that is currently in place enforces the mentality of memorization as opposed to actually trying to learn.

  44. Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo Labo

  45. i love how smart you guys are ! in this video you're clearly supporting something that would benefit you as i know james is involved a lot in this whole "game for education" thing, but you're not saying : "hey they are not doing it now and they dont really want to do it, how lame !". instead you're pushing for the right thing : actually studying the subject beforehand. And this is because you do BELIEVE it is a great idea that will benefit others (which if we want to circle back, is probably one of the reason james got involved) and studying that will just prove it. that's clearly a WIN-WIN and i wish you the best in these regards. being born in 95 i think i'm of the first generation for which it is common to have played educational games at home when i was little and i have very fond memories of them ! (READER RABBIT FTW) great video once again

  46. my primary school was private and had a big budget caus they had massive fee's and they had this game downloaded in the computer lab and it was a touchtyping game where you had to progress through levels by touchtyping with a cover over your keyboard. it was a massive help and now touchtyping is something that comes like secound nature to me and i see a lot of kids my age struggling with typing fast which is weird for me.

  47. i took a look at their website and they have amazing games i would love to play but i doubt my school will ever pick it up after they failed implementing minecraft into the curiculum

  48. I would agree but when they have these games they heavily reduce the fun and fun leading to more learning and actually getting motivation

Leave a Reply

Your email address will not be published. Required fields are marked *