Taxonomy of Problems (Part 2): Ways and what to assess

In my last post, I tossed out a loose taxonomy to name four different types of problems:

  • Content Learning Problems
  • Exploratory Problems
  • Conceptual Understanding Problems
  • Assessment Problems

I felt it necessary for myself. Up until now, I’d been labeling all problem equally: they’re problems! They’re tasks that are supposed to get students to learn stuff! But that implies a one-size-fits-all-ness that I don’t think is practical. The planning, time frame, facilitation, scaffolding, and – for our purposes in this post – assessment and wrap-up all look different, even if the task itself doesn’t look that different (after all, ideally we’re all using nonroutine problems with a low bar and a high ceiling regardless of whether it’s being used for formatively assessing student understanding or creating new knowledge).

It’s tough to throw out exact examples for assessment since we’re all working from different standards and tools. So I’m going to restrict it to the following universe of things to assess problems on: New Tech Network’s (where I work) most common Schoolwide Learning Outcomes (SWLOs) and the Common Core Standards of Mathematical Practice.

things to assess

Now, different teachers and different schools I’ve worked with utilize these different halmarks differently. In fact, many schools have difficulty even defining many of these indicators of student learning, let alone assessing. But nevertheless, we’re trying to get a general look and feel to what a problem rubric would look like, depending on what you’re actually trying to accomplish from said problem. We’re talking broad-brush here.

Content Learning Problems

Things to assess: Oral Communication, Professionalism/Work Ethic, Make sense of problems and persevere in solving them, Look for and make use of structure, Look for and express regularity in repeated reasoning

This might just be personal preference, but I’d be wary of assessing content knowledge in a learning opportunity for a student. If we are distinguishing between learning and confirmation problems, we might want to more rigorously assess content on the latter. Another one of my favorite wrap-up activities is this quick check-up as an exit ticket.

Exploratory Problems

Things to assess: Critical Thinking, Oral Communication, Collaboration, Model with Mathematics, Construct viable arguments and critique the reasoning of others, Use appropriate tools strategically

Assuming that the time-frame is a bit longer for an exploratory problem, and that the solutions and solution routes are varying, the wrap-up could consist of a formal presentation, followed by panel-style questioning.

Conceptual Understanding Problems

Things to assess: Critical Thinking, Collaboration, Written Communication, Reason abstractly and quantitatively, Construct viable arguments and critique the reasoning of others, Look for and make use of structure, Look for and express regularity in repeated reasoning

Here, I think it makes sense to have students reflect on and communicate what they’ve learned.

Assessment Problems

Things to assess: Critical Thinking, Written Communication, Reason abstractly and quantitatively, Use appropriate tools strategically, Attend to precision

In this case, one can easily envision a rubric that assesses the items above. Assuming these tasks are a bit more individualized, a written piece – almost like the free response section of an AP exam – might make sense. I’ll leave it up to the reader’s discretion whether or not to allot numerical point values.


With these self-recommendations in hand, we can more easily (hopefully!) pick and chose what would go in a rubric and where, if a rubric is one of the tools in your toolbox.

Again, the idea is to make things easier, not more complex. And to better target outcomes for each and every problem. From these recommendations we might be able to construct a loose, lean problem planning template that is directly tied to the indicators you’re trying to peg with a particular problem. Maybe even some planned facilitation and scaffolding moves as well.

Developing a Taxonomy of Problems: Not all problems are implemented equally

Why would we design all problems and facilitation in a similar way without having the type of problem identified?

It’s possible I’ve been a bit too broad-brush when describing Problem Based Learning (PrBL) in terms of task design and facilitation. I’m beginning to wonder if we need a taxonomy of problems. After all, every problem implemented in a classroom will have different intended outcomes which can affect the design and facilitation of the task. After a while, you start to notice the similarities and patterns that like problems attend to. You also notices the differences. What I have not encountered, thus far, is a way of classifying the problems, which I think would make it easier to design, facilitate and assess. Once you have the problem type identified, that may allow you to use different tools, templates, rubrics, etc. around which the task may be designed.

Maybe it would be better if I just got to it. Consider this an attempt at Problem Taxonomy.

The four types of problems:


Problems for Learning/Constructing New Knowledge

These are problems that foster new knowledge within students.

Content Learning Problems

These are problems that have a predetermined, content-oriented outcome. Most of the time, this is what is often meant by Problem Based Learning. Or at least, it’s what I’ve basically meant in the past. Content Learning Problems are directly tied to a specific standard or standards. Scaffolding is often planned by the teacher ahead of time. Students work collaboratively and plan, strategize, struggle, and are coached toward a solution with the aid of the teacher. These may take 1-3 days.

Exploratory Learning Problems

These are problems that may foster new knowledge within students, but there is not a specific content standard tied to it. Although, Mathematical Practice standards, such as those defined in the Common Core, or Bryan’s Habits of a Mathematician truly shine in this type of problem. The solution and solution route may be unknown by the teacher. Related, the teacher may not have the scaffolding planned or predetermined until the need is made manifest. There is no prescribed method toward a solution and collaborative groups may have differing solutions and solution routes. These may take 1-5 days. Some may even call these “projects”.

Problems for Confirmation

These are problems intended to stand by themselves with minimal assistance or facilitation by the teacher. Students are to demonstrate the knowledge they have gained through Problems for Learning. I should note that, despite the naming convention, these problems don’t necessarily preclude learning opportunities.

Conceptual Understanding Problems

These are where a student puts the pieces together and begins to speak fluently about the content. If there was any confusion about the mathematical concept before, it’ll get crushed here. The scaffolding is quite intentionally student-centered with the specific intention of getting students to discuss the mathematics. Possibly the problem itself is more purely mathematical, or at least along the Skynet Line. Technology such as Geogebra investigations may be involved in order to solidify reasoning. These may take roughly a day or two.

Assessment Problems

Don’t tell the school district I worked for, but I once gave a single problem for my final exam of the year. It was a problem adapted from one of the Dana Center’s Assessments. I said, basically, “here’s your problem, you have two hours to show me what you’ve got. Now go!” The subtext of which was, “according to district rules, this single problem will count for 25% of your grade for the semester.” That’s basically what this kind of problem entails. Possibly solved individually, these problems are tied directly to content, require some decoding, and offer a chance for all to excel. I’d doubt there would be a presentation involved. Ideally (unlike in the scenario I described above), there would be some formative feedback or revision process before a numerical grade is attached. The point is, these are problems where students should know the content involved and be able to explain it with great fluidity.


So what do you think? Does this taxonomy work for you? Obviously the fine Art of Teaching necessitates that many of these types of problems overlap and intermingle. But in the design of a task, it’s important that you determine exactly what the outcomes should be. Are you constructing new mathematical knowledge within your students? Are you offering a place of creativity and non-linear thinking? Are you solidifying knowledge (Jo Boaler refers to this as “compression”)? Are you assessing understanding? Until you answer these questions, I’d suggest you can’t really fully develop the task.

In the next post, I’ll talk about what and ways to assess each of these types of problems. 

“Isn’t Problem Based Learning easier than Project Based Learning?” and 10 other myths about PrBL. (“Real or not real”)

About a year ago, I started advocating and pushing towards a Problem Based approach in mathematics, as opposed to a solely Project Based approach, which many/most of my peers currently employ. But before we go any further, let’s better parse the differences between Project- (PBL) and Problem-Based Learning (PrBL). I realize that different people define and implement Problem and Project Based learning wildly differently. Some things I would define as problems, others would define as projects, and vice versa. So here’s what I mean by Problem and Project Based Learning:

Pick your favorite diagram.

Figure 1.

Figure 2.

Figure 3 (sorry for the word- and acronym-slaw in this one).

The primary differences are size, scope, and end product. And then also a particular mathematical concept may require several problem scenarios instead on just one project scenario. Here are some untruths and half-truths I’ve heard about moving toward a Problem Based approach. I’d say the two hallmarks of PrBL (in my head at least) are as follows: 1) Each problem is about 1-4 days long and 2) the problem scenario comes first, the instruction and scaffolding succeeds the scenario.

Here are some misconceptions about Problem Based Learning. Let’s do this “Real or not real” style, since the Hunger Games is so popular with you kids.

“Problem Based Learning represents a step back in giving students authentic mathematical experiences.”

Not real. Maybe we need to define what is an “authentic mathematical experience.” A mathematical experience to me is something that promotes mathematical habits of mind. I’m totally going to steal from Bryan here (@doingmath) and point you towards his thinking through mathematical habits of mind. I’d argue that a problem can better facilitate the many of the mathematical habits of mind.

“Problem Based Learning is easier to facilitate than PBL.”

Not real. Due to the tighter time frame, a PrBL Math teacher always has to be formatively assessing students, differentiating instruction and generating good problems. You don’t have weeks to create a workshop, you have a day or so. So you either have to be well prepared for any and all student knowledge gaps and/or quick to respond to Need-to-Knows from students. To me, the work of PrBL is more challenging.

“Problem Based Learning allows teachers to be more ‘traditional’ in their facilitation.”

Not real. Like in PBL, the teacher acts more as a coach or facilitator, rather than a primary source/gatekeeper of knowledge. Students are still given a fair amount of autonomy in their problem solving strategies. Also, lectures still suck.

“Problem Based Learning is more about the math content, whereas PBL is more about 21st Century Skills.”

Sort of real. While I do believe PrBL can mesh nicely with 21st Century Skills, I am acknowledging of the fact that it does promote math content knowledge more than, say, writing a position article for a local newspaper. The fact is that by having the end product be a Socratic dialog among students rather than, say, a powerpoint presentation, may promote math content more than 21C Skills.

“Problem Based Learning is better for students who require more math remediation.”

Not real. In fact, I’d suggest that PrBL is exceptionally appropriate for advanced math courses such as Calculus. PrBL does tend to strip away some of the “psuedo-context” that often makes the math hidden within the weeds of a scenario.

“Problem Based Learning is better for novice teachers, or teachers new to an inquiry-based approach.”

Not real. As I mentioned earlier, PrBL is more work, more difficult, and requires higher order mathematical thinking and content knowledge and skills. It also requires teachers to always be on the lookout for a good math problem. It enhances your mathematical “spidey-sense”.

“If I’m teaching a mathematical concept, I have to make the choice to use either PBL or PrBL; it’s one or the other.”

Not real. There’s no reason a mixture of problems and projects, where each are appropriate, may be used. I would just caution to really think about your Projects: is the time spent in product refinement really enhancing mathematical understanding? Or is it just time spend doing “cool stuff?”

“Problem Based Learning doesn’t allow for authentic real world connections.”

Not real. The main differences are in the size and scope of the undertaking. Here are some or my problem ideas that take 1-4 days that have real world applications:

For more, I’d highly recommend Brian Marks’ awesome YummyMath.

However, I will concede that PrBL does allow for the use of non-real-world connections, and promote pure mathematical conceptual understanding, whereas a Project really sort of needs to be couched in some real-world product. My follow-up question: is it really such a bad thing if the math has nowhere to hide in a problem?

“In Problem Based Learning, students are just given a problem and expected to work through it on their own.'”

Not real. This might be a misconception about PBL in general: that students are just supposed to “figure it out” or that the teacher isn’t expected to “teach” as much. This is flat out incorrect or improper implementation of PBL and PrBL. Every problem ought to have some form of scaffolding along with it. Maybe it’s a lecture, maybe it’s one of these tasks, maybe it’s students sharing their solution route ideas and then going forward as a class. While students are expected to begin to attempt a solution without initial handholding, the facilitator is still expected to address Need-to-Knows with some sort of scaffolding. And the teacher is also responsible for getting students off on the right foot by facilitating some sort of strategizing and brainstorming process.

“You don’t really assess problems in the same way as you do projects.”

Sort of real. I suppose this is more personal preference, but there’s no reason you can’t assess the a problem in the same way you can a project. The main difference is that it might be impractical to develop, say, a full, zillion rowed and columned rubric every couple days. However, the practice of using a rubric to assess across multiple skills and proficiencies is still an excellent practice (despite what others have suggested) . So, a suggestion:

  • Develop one rubric per unit, or batch of problems. Focus on a couple or one particular key learning outcome aside from Math content. For example, written communication: have students have some sort of writing component in every problem for a couple weeks and use the same rubric, or the same written communication component of that rubric.

One other note about assessment for now: I’d suggest it’s easier to assess something like Critical Thinking through the use of a Problem rather than a Project. In a Project, you summatively assess the final, polished product. In a problem, all that scratchwork, brainstorming, and multiple solution attempts are right there on the paper. I’m saying it’s any easier to assign a numerical score to Critical Thinking, just that it can be more evident in Problem Based Learning.

“Non-math teachers can’t help me refine my problems.”

Not real. If anything, I’d suggest non-math teachers are particularly adept at helping you refine your problems. Often, math teachers become so insular and like-minded we can see what the problem is trying to get at. We’re particularly adept at sifting through “psuedo-context” to find out what problems are really asking, while other teachers (and, ultimately, our students) may not be. So by enlisting their support, either by Critical Friend-ing or by actually facilitating the Need-to-Know process, you’ll probably get a better idea of what the problem will look like in an actual real-live class with real-live students instead of a bunch of math teachers.


What misconceptions have you head about Problem Based Learning that I/we can address in the comments?

Problem Based Learning, start-to-finish, in Ten Minutes

Want to learn more about Problem Based Learning but don’t have time to read several posts with graphics? Want to see what a student-centered math unit looks like from start to finish, but would prefer to see it visually and hear it in a nasally voice? Well, look no further, my friends! I recorded a little video in which I discuss five-ish steps to a problem, start-to-finish. It’s about 10 minutes and you can hear me doing terrible impressions of students all while I had a cold. Also, it was made with a crappy movie making program (I won’t reveal what program it was, but let’s just say it comes free with Windows 7 and it rhymes with “Shmoovie Shmaker”). So apologies for that. Hopefully what it lacks in design it makes up for in usefulness.

Here are five stages to Problem implementation discussed in the video, start-to-finish.

  1. Posing of the Problem
  2. Work on the Problem
  3. Intervention as questions arise
  4. Students apply scaffolded instruction
  5. A solution is reached

Although, as I look at it now, it probably shouldn’t appear so linear and step-by-step. It should probably be something like this:

1. The Problem is Posed

2-???. Work ↔ Intervention ↔ Apply scaffolding

???+1. A solution is reached.

Anyway, for more on Problem (or Inquiry) Based Learning and more in-depth discussions, here is some linkage.

Also, I zipped up all the files used for the above presentation, including slides, audio, and the transcript. So if you want to, like, re-record it in your own less annoying/nasally voice, have at it. Or I supposed you could isolate the audio and listen to it as a podcast for some ungodly reason. Anyway, here are the files (note: the slides are in PNG images format and not in proper order; anything else?).

Standard disclaimer: I would also like to formally declare that I don’t have all the answers. Frankly, I’m not sure I have very many answers at all. I do have a lot of questions though.

Inquiry-based mathematics: the posing of a problem is only the beginning of the problem-posing process.

We’ve been exploring some of the steps to an inquiry-based lesson in mathematics recently. In the last post, I tossed out a few .png images and laid out a few general steps in preparation for actually getting into the meat of inquiry-based mathematics instruction. Which we’ll do so starting today.

Step 1: Posing a problem

To oversimplify (more-so than I already do), the primary difference between an inquiry-based classroom and a traditional classroom is the placement of the problem statement: it comes before the instruction, thereby giving rise to the need for instruction. The problem is front-loaded, not the instruction.

For a run-down of what makes a quality problem, there are lots of places to go. In short, they A) are interesting to students (either in the scenario itself or in the manner in which it’s posed), and B) have multiple entry points. It’s also nice if there are multiple pathways to a solution (or even has multiple solutions) but that’s not a dealbreaker.

Now that you’ve got the problem, it has to be posed to the students. This can be done several different ways. Maybe it’s a video, a letter, a demonstration or a student simulation, but something delivers the problem to the student. New Tech calls this an “entry event”. Dan Meyer calls it “Act 1.” Whatever it’s called, it is intended to ignite student curiosity about the problem.

However, the problem posing does not end after the problem is posed. The posing of the problem is only the beginning of the problem-posing process. For most classrooms, along with an “entry event” there needs to be a strategizing process in place for students.

One way of doing this is the “Know/Need-to-Know/Next-Steps” process. Here, students begin brainstorming everything they know about the problem and everything they need to know about the problem in order to solve it. This can be done as a class or in groups, jigsaw style. Google docs is a nice tool. As is scrumblr and wall wisher. Of course, a white board works just fine.

But even this neat sounding process is fraught with peril. Here’s an example of Knows/Need-to-Knows I saw in a class the other day, captured in a google doc.

I love this list. Not because these are good knows/need-to-knows/next-steps but because of precisely the opposite: it encapsulates what can go wrong with the K/NTK/NS process.

As you can see, the knows, need-to-knows, and next-steps are not very mathematically rich. And frankly, if your need-to-knows simply restate the problem or are focused on things like “when’s it due?” then there’s really not much of a point to it.

So how do we get from moribund knows/need-to-knows to mathematically rich (and useful) ones? It’s a difficult quandary. How do we get students to say things like “how do I find the zeroes on a graph?” when they haven’t been instructed on finding the zeroes on a graph? It’s a tough quandary, and frankly, you may not get students reciting the exact content lesson you desire. But here’s one way to prepare for this.

Develop a list of anticipated knows/need-to-knows/next-steps and a list of desired K/NTK/NS. Then start filling in the questions you could ask to get from one to the other. Like this.

Try to anticipate the strategies students will come up with and generate questions that will steer them into a more mathematically and problem-solving-esque mode. Or better yet: try running the K/NTK/NS process with non-math teachers at your school, see what they come up with: I’m betting mathematically it’ll serve as a pretty good approximation of of student brainstorming.

Perhaps the most critical act that can sink an inquiry based lesson is when a problem is posed and students are simply set loose without any guidance into their work time. I’ve seen it (and been a part of it many times): teacher poses a problem, students brainstorm, teacher sets students loose, students are unable to engage in the problem as the brainstorming process yielded very little in terms of actionable items. In general, I wouldn’t set the students off to work until they have a list of two or three potential next-steps along with two or three good need-to-knows. But then, that may be just me.

Often teachers ask “yeah, but when do I actually teach?” when transitioning to a inquiry-based environment. The answer is: the nano-second after the problem is posed. Your instruction begins with helping students craft strategies and access prior knowledge that pertains to the problem. That’s as much teaching as anything else you could provide for students.

An oversimplified model of an inquiry-based lesson, with visual aids

Last week, I mentioned that, having begun to attempt to slay one of the two giants of inquiry-based math instruction, I’d be steering into a potentially trickier aspect of inquiry based instruction: namely that of instruction and facilitation.

Most of us learned math like this.

We have decades of evidence suggesting that this method of instruction is not only ineffective, but damaging – both to students’ confidence and love of the subject (see Jo Boaler’s awesome “What’s Math Got To Do With It” for more). But honestly, I think that battle has essentially been won. Most of us (I think) are in agreement that this isn’t the ideal way to teach math, or any other subject. But the question is “how?”

Even in more-or-less traditional high schools nowadays, you’ll see something more like this.

Students need to be actively and collaboratively involved in the problem solving process, but what does that really look like? How do we give students both the freedom of solving a problem collaboratively and in novel ways, while still providing enough support to help students along the way?

That’s where the real art of teaching lies. One has to be nimble to adjust instruction based on student need, but prepared enough to be able to anticipate and address the need.

So let’s start with the “ideal” inquiry based lesson, start-to-finish, then in future posts we’ll go back and analyze the process further.

A model of inquiry-based instruction.

Step 1: The problem is posed.

Usually the problem is introduced along with some sort of class or group discussion facilitated by the teacher where students identify key components of the problem and begin strategizing.

Step 2: Students begin work on the problem.

As the students work together toward a solution, the teacher checks in with each group and each student, probing for understanding and answering any clarifying questions.

Step 3: Questions begin popping up from the students.

As the students are working through the problem, questions related to the intended content begin to crop up. As students begin to struggle with the problem a “critical mass” (or “tipping point) of perplexity occurs.

Step 4: Appropriate scaffolding and/or instruction is provided by the teacher.

Based on the students questions, the teacher provides instruction in some format. Maybe it’s a lecture, maybe it’s groups sharing out, maybe it’s analyzing samples of student work, maybe it’s a research resource, maybe it’s an investigation, activity or lab.

Step 5: Students work on the problem some more, after being provided instruction.

Having acquired the requisite content knowledge from the instruction, students proceed to work on the problem.

Step 6: Students solve the problem.

Students finalize their solutions. Usually some sort of informal sharing out or presentation is accompanying. The teacher asks probing questions to get students to make generalizations about their work and promote sense-making.


Now, this is clearly an over-simplified model of what a classroom actually looks like. Every step along the way is fraught with different challenges and obstacles to understanding which need to be addressed. I hope that the simplicity of this model does not imply that inquiry-based instruction is simple: far from it! For example, in our fantastical little classroom above students appeared to be all having the exact same question at the exact same time. Obviously that doesn’t ever happen in classrooms. (also, it looks like we lost a couple students from the first image of this post to the next)

I would also like to formally declare that I don’t have all the answers. Frankly, I’m not sure I have very many answers at all. I do have a lot of questions though.

In order to address the monumental challenges, we’ll be looking at each step in depth over the next few weeks, discuss particular challenges, differentiation strategies, etc. My preference would be to get your input and suggestions, since I’m far from an expert.

But before I do, what do we think of this little utopian situation? Did I miss anything? Would you swap out one of the steps for something else? In my desire for simplicity I may have glossed over something or left something out entirely. Please chime in in the comments.

Inquiry-Based instruction, in a PNG-nutshell

In talking to math teachers about an Inquiry- , Project- , or Problem-Based approach, these are the following questions that come up most often.

1) How am I supposed to cover all the standards using this approach?


2) “So, when do I actually teach?”

An attempt at the first question is reflected in the Great Inquiry-Based Curriculum Mapping Project, from a couple weeks ago.

The second question can be a bit loaded, especially when you move the emphasis from word to word (as I did, by emphasizing “actually“).

We can discuss the second question a bit more in-depth going forward, but I’d like to attempt to simplify the nebulousness of “inquiry-based instruction.” When teachers ask “when do I actually teach?” I think they’re asking when do they stand up in front of the class and demonstrate examples and processes? And is such a time ever appropriate for an inquiry-based classroom environment?

But before we get into the weeds of such a rich topic of discussion, let me posit this to you: I would suggest that the change from a “traditional” approach to an “inquiry-based” approach may be as simple as moving from this

to this:

(apologies for the computer science jargon)

Now, obviously there’s a lot more to it than just a couple diagrams, but the point is this: instruction still happens, but it simply happens after students have attempted a problem and within the context of a problem. Instead of saying “Today class, we’re learning about slope, here’s a lecture,” followed by a lecture, followed by a problem set, the practice is in some sense, simply reversed: “Today class, here’s a problem,” followed by instruction about,say, slope.

So yes: instruction is still useful and necessary for an inquiry-based environment. And I would also say yes: lecture or direct instruction is often a appropriate tool to transmit mathematical knowledge in an inquiry-based environment. (Although, I would warn against it’s overuse, lest it become the default mode of instruction.)

The deeper questions of when and how do I instruct is a bit more of a dance that I hope to at least partially address in the coming posts. But in the meantime, let me hazard a broad-brush answer at these.

When: after students have had a goodly amount of time to discuss the problem with each other, and at least begin to attempt a solution. Maybe at least 30 minutes?

How: it depends in part on the number of students struggling with the content. If every group is having difficulty even starting the problem, then a whole-class lecture may be appropriate. If half the class is struggling, maybe some share-out, gallery-walk, and/or group-student-exchanges may be appropriate (or better yet: Kate’s “Speed Dating” activity). If only a few students are unable to jump into the problem, a small workshop may be necessary, while groups discuss and assess their solutions.

But these are broad-brush, haphazard solutions to potentially a much bigger question. I’d love to begin aggregating and categorizing math scaffolding activities and to have a discussion about when they may be appropriate.