Deconstructing the US Math Education “Crisis,” Part I

Actual public school middle school math / US Dept. of Education, CC BY 2.0

75% of our nation's high school seniors probably aren’t mathematically incompetent, despite a recent recurrence of the nearly annual hysteria to the contrary.

Associated Press, May 7, 2014: US 12th graders make an "ABYSMAL showing" in math and reading according to the 2013 NAEP results.  "Only about one-quarter are performing proficiently or better in math and just 4 in 10 in reading." "America's high school seniors lack critical math and reading skills for an increasingly competitive global economy." (emphases added) 

PANIC!  CRISIS!  Blood in the streets! Dogs and cats getting married ...

BUT ... WAIT a minute.  Is this really a crisis?  Are 75% of our 12^th graders honestly mathematical nitwits, as reporting on the NAEP results seems to imply?  I wanted to know, so I mucked around the NAEP website. 

Based on what I gleaned from the NAEP Test Framework, the skills that the "Basic up to almost Proficient" students generally lack (compared to the "Proficient and Above" students) are unlikely to negatively impact their general readiness for college or work.  These skills are necessary in specific STEM-related positions where advanced mathematical concepts are utilized on a daily - or even monthly - basis.  "STEM-related" jobs are anticipated to number just 5.6% of all jobs in 2022.  

All hand-wringing aside, this is not a crisis. 

Q1: How is "Proficient" defined?  How does it fit into the rubric of the performance levels that the National Center for Ed Statistics uses?

A1: The 2013 NAEP Math Assessment Framework defines the levels as follows:

“Basic denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade.

Proficient represents solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and appropriate analytical skills.

Advanced represents superior performance.

These levels are intended to provide descriptions of what students should know and be able to do in mathematics.” (Emphasis in original)

As to the last point, it is not clear which of the first two describes the level at which all or even most students *should* minimally be performing.  The definition of "Basic" is defined in reference to “Proficient,” which leaves something to be desired.  Similarly Proficient’s “solid academic performance for [the] grade” doesn’t resonate with meaning for me, either.  Maybe it’s a term of art well understood among educational assessment and statistical professionals, but not by the likes of me.

Q2: What exactly does one need to do to demonstrate knowledge and ability at a "Proficient" level for math?

A2: I’m wading through the 88 page Math Assessment Framework, but that’s a post all its own.  In brief, the 12^th graders’ assessments cover what they “should know.”  The assessment’s drafters note that “what they should know” includes concepts usually introduced in courses beyond the standard, three year high school math sequence of two years of algebra, one year of geometry (Algebra I, Geometry, Advanced Algebra & Trig.)  In going through the framework, one can see a significant number of objectives which are asterisked to indicate that these are generally covered in pre-calculus advanced math (standard-track 4^th year high school math), calculus (advanced placement senior math), or beyond.  

Also worth noting: the test was administered part-way through the academic year, between January and March 2013.  The students were tested on a number of concepts that most, even those pursuing a standard four-year math track, hadn't even encountered yet.

The vast majority of students would do well to master basic concepts through Algebra I and some geometry for work and for life outside of it.  An understanding of statistical methods allows students to make some sense of the many statistics thrown around - and often misinterpreted - on a daily basis, and I'm heartened to learn that K-12 students are now learning more about statistical methods than we did 20-some years ago.  A perhaps substantial minority would also benefit from a good understanding of some basic calculus concepts (which is generally taught in high school honors classes in the senior year anyway, and may or may not be included in the NAEP.)  But trig?  Not so much. (*N.2.)

Q3:  How did 12^th graders perform overall? What percentages performed at the various thresholds measured?

A3:

"Below Basic" level: 36% of student test-takers

"At or Above Basic": 64%

"At or Above Proficient": 25%, and

"At Advanced": just 2%. 

What to take away from these data points depends on what it means for a 12^th grader to perform in the range between “Basic” and “Proficient.”  But I suspect that rather than panicking about our "At or Above Proficient" showing, it would make more sense to a bit more calmly, and more deliberately, concern ourselves about the 36% who are "Below Basic."  This figure may still be cause for concern (I emphasize, MAY), but it’s certainly not the CRISIS implied by focusing on the fact that 75% of students didn't perform at a level meeting or exceeding “Proficient.”

Q4. Does the difference between the skills and knowledge generally exhibited by "Proficient and Above" students versus the "Basic up to almost Proficient" students actually MATTER in terms of their readiness for college or work, outside of specific STEM-related fields where advanced mathematical concepts are utilized on a daily - or even monthly - basis? 

If not, then the percentage of students scoring "Proficient or Above" does NOT actually tell us anything meaningful about the GENERAL college and workplace "readiness" of America's 12th graders.  All it tells us is what percentage are prepared to pursue further education and training in math-intensive fields.  And if that's the case, 25% "Proficient or Above" is probably just fine, or even well AHEAD of where we need to be. 

Why? Well, the U.S. Bureau of Labor Statistics projects that "Employment in occupations related to STEM—science, technology, engineering, and mathematics—is projected to grow [from 8 million] to more than 9 million between 2012 and 2022."  The BLS further projects that total jobs will number around 160.9 million in 2022.  In other words, STEM jobs are anticipated make up just a bit more than FIVE PERCENT of all jobs in 2022 (5.6%, to be precise.) 

But isn't the demand/need for STEM workers growing terribly rapidly?  No.  STEM jobs are expected to comprise only a slightly higher portion of new U.S. jobs in 2012-2022: 1 million out of 15.6 million new jobs, or 6.4% of them. (*N.1.)  

A4: Based on what I've gleaned from the Test Framework thus far, I suspect the answer is actually ... NO. Not really.  In 2022, about 94% of workers will be employed in jobs that are NOT STEM-related.  Thus, for the overwhelming majority of US workers, the ability to solve quadratic equations, tell the difference between a sine wave and a cosine wave, or use ANY of the knowledge one is supposed to acquire from a high school trigonometry course (usually taken in junior year) is unlikely to have ANY practical use in their work life.  

Q5. What proportion of student test-takers who'd been enrolled in honors-level math performed at the "Proficient" level?  How about at the "Advanced" level? 

If "Advanced" is actually meant to encompass the understanding necessary to pursue further education and a career in a math-intensive field, which is what it seems like it should mean, then we would expect the overwhelming majority of honors math students – particularly those who are studying calculus – to perform at the "Advanced" level on the NAEP.  If the actual results are substantially lower, then either "Advanced" doesn’t mean what it should, or there are fundamental problems with the testing methodology (the way it's written and/or scored.)

A5: I wasn't able to find a direct answer to this question.  But I was able to ascertain that a significantly greater share of high school seniors had studied calculus, the typical 12th grade honors-track math course, than performed at the "Advanced" level on the NAEP test.

In 2009, the most recent year for which data is available, 16% of students graduating from high school had already taken a calculus course.  Granted, not all 12^th graders actually graduate in a given year, and the NAEP results are supposed to be generalizable to the population of all 12^th graders, not just those who graduate.  

Using 12^th grade enrollment and graduation figures for public schools, and assuming that private school seniors aren’t more likely than their public school peers to fail to graduate (which seems a reasonable assumption), we can reach an estimate of the percentage of all seniors who'd studied calculus.

2008-2009 figures here and here allow us to estimate that at least 9.5% of all 12^th graders that year were taking calculus or had previously.  That’s almost five times the 2% rate at which 12^th grade NAEP test-takers scored at the “Advanced” level in 2013.  Which leaves me thinking that “Advanced” is likely a much loftier level of achievement than the ordinary reader (this one included) would expect.

Amanda Paulson at the Christian Science Monitor reported that 50% of all students scoring in the top quartile had taken calculus, and an additional 34% had taken pre-calculus.  Since only 25% scored "Proficient or better," this tells us that 84% of those scoring at "Proficient" or better had taken (or were in the midst of taking) the fourth-year standard-track or honors-track math class.

... These are just some of the reasons to be skeptical about the 12th grader readiness "crisis" allegedly indicated by the NAEP math test results.  The sheer number of the advanced mathematical concepts listed in the Testing Framework, to be discussed later, provides further, substantial support for such skepticism.  

  

*N.1. The on-going frenzy over STEM jobs and STEM-preparedness begs further exploration, and seems to be a topic worthy of its own series of posts in the future.

*N.2. High school trig teachers: please tell me if I'm missing something here.  I actually did use this knowledge in studying economics in grad school, but after recently reviewing a comprehensive trig outline, I am hard pressed to identify any practical uses for this material outside of "STEM fields."