During the August 18 public meeting of the DC state board of education, that body’s president, Ward 5 rep Zachary Parker, appeared to have mixed up two recent DC education studies. (See it here, starting at about 1:29:20.)

The study Parker appeared to think one of the public meeting witnesses was discussing was recently released by researchers at American University, who had examined IMPACT, the DCPS teacher evaluation system. Their study found that IMPACT was biased. (Shocking, I know—no one ever saw this coming, except when they did.)

Instead of that study, however, public witness Duncan Chaplin testified to the DC state board about a study he is a co-author of, recently released by the private research firm Mathematica.

Chaplin’s study examined NAEP (National Assessment of Educational Progress) and SAT data for DC’s publicly funded schools to conclude that education reform and mayoral control in DC are a success.

If who commissioned the study is publicly unclear, who funded it is not:

Chaplin’s Mathematica study was funded entirely by Arnold Ventures, which funds a variety of private research projects. Arnold Ventures is led by its founder and funder, John Arnold, whose resume reads like a right-wing ed reform dream: former Enron analyst and billionaire investor; co-founder of KIPP; board member of the City Fund, which supports school privatizing around the country, including in DC; and funder of a variety of anti-public pension efforts.

Perhaps more to the DC point:

John Arnold has donated money to DC politicians—specifically, Adrian Fenty, our mayor’s mentor and the person who instituted mayoral control of DC schools, and Brandon Todd, the mayor’s political ally and Ward 4 council successor.

As if a study concluding that DC mayoral control of schools is successful that is also funded by a group headed by a man who happened to give more than $4000 to Mayor Bowser’s political friends (one of whom instituted mayoral control) isn’t a little, uh, conflicted, the study itself appears to have been commissioned by the mayor and circulated to select council members months ago.

Indeed, because of the study’s private funding, the DC contracts and procurement database has no information that would allow the public to understand exactly who asked for this and what the terms were.

(A FOIA about this study filed with the deputy mayor for education’s office has not yet been fulfilled, despite a long-past statutory deadline.)

Playing With Numbers

To be sure, Arnold Ventures’ Mathematica study is only the latest in a long line of ed reform-funded studies that illustrate how DC education data can be used to further political narratives. (See here for a few other examples.)

In this case, the study uses a counterfactual construction that is curiously speculative: estimating student outcomes as if mayoral control had never existed. Notwithstanding that it’s impossible to know exactly what would have happened, some data show growing student achievement in DC before mayoral control.

Thus, if one is speculating counterfactually, it seems one could just as fruitfully argue that ed reform has stunted student growth rather than the opposite. That is, by extending that pre-mayoral-control growth and presuming it would have kept going absent mayoral control and ed reform, one could say that we ought to be doing much, much better than we currently are!

(Fun, isn’t it, to see how this works?)

The authors also claim that the impact of ed reform in DC schools was less at its beginning than of late—which may be true, except that the study doesn’t take into account other factors over that time, including increased per pupil and teacher spending. The researchers appear to presume that the increase of test scores over time happened in a well-insulated place as a direct result of ed reform and mayoral control.

Perhaps the study’s biggest red flag (well, besides its funding) was that it used NAEP data for its analysis, with the presumption that NAEP data are an accurate measure of achievement in all directions. But because NAEP data do not use individual student data, in a dynamic and very mixed student population like DC’s (hello mobility! hello socioeconomic inequality!) a lot may be missed.

(Among the folks who have explicated exactly how much can be missed with NAEP data is the DC auditor (see p. 95 here (or p. 99 of the PDF)). The auditor has also pointed out (i.e., see p. 12ff here) that unlike what this Mathematica study shows, NAEP data show relatively poor results for DC and a growing achievement gap.)

The Once and Future RPP

In January, the mayor announced the selection of the Urban Institute as the partner with DC education agencies on an education data research practice partnership (RPP).

The idea of the RPP was posited as a counterweight to the political influence of the mayor and her deputies on DC education data under mayoral control and how that data is used (and abused) and reported (or not).

The fact that Urban had, years before, been meeting secretly with DC education leaders to formulate a plan in which Urban would be the head of such a partnership went unmentioned in the mayor’s triumphant January announcement.

Also unmentioned was a minority report from a member of the panel that selected Urban, detailing a number of reasons why she could not vote for Urban. The list included Urban’s funding, the potential for conflict of interest, and research that does not “meet the accepted academic standards of high-quality RPP research.”

But no one can say DC politics lets such obstacles remain hurdles for long—which may help explain how Duncan Chaplin, the recent Mathematica’s study co-author, ended up testifying before the state board about that study without any clear explication that

–Chaplin works for Mathematica now, but has also worked for the Urban Institute and

—Mathematica is one of the partners on the RPP and

–Mathematica was responsible for errors in IMPACT that resulted in at least one teacher losing their job erroneously.

(Hey, we’ve come full circle! Pace Zachary Parker!)

At the end of June, Urban announced that the RPP had an executive director, Josephine Bias Robinson. According to Ms. Bias Robinson, the advisory committee that the RPP legislation provided for is being formed now, and although planned, public outreach is still TBD.

The DC Auditor, The Mayor, And School Data

The RPP’s work will inevitably come in the context of a scathing report on DC education data released in March by the DC auditor.

That report outlines not only how our state superintendent of education (OSSE) failed to collect and warehouse actionable education data, but also how OSSE failed to act on the education data it already had (despite receiving millions from DC taxpayers and the federal government for that purpose). The upshot is a black box of outcomes and annual investments of nearly $2 billion for DC’s publicly funded schools–with victims only occasionally in the news.

Since its March release, the auditor’s report on DC education data has reverberated in many directions.

For instance, on March 19 the DC council held a hearing on the auditor’s report in which government witnesses from various DC education agencies (i.e., all the mayor’s people) disputed the report’s findings.

Then, in April, in addendum written testimony to the DC council for that March 19 hearing, the auditor again made clear that on a number of metrics, DC is not doing better under ed reform and mayoral control. Specifically, school choice, charters, and endless school seats are NOT guaranteeing quality in terms of test scores–despite constant DC ed reform rhetoric otherwise.

Also in that written testimony, the public learned about yet another data confrontation, this time between the auditor and the deputy mayor for education (DME). That confrontation took place in February, before the March audit was released.

Specifically, the auditor referenced a June 2019 document, signed by the DME, to provide for the March audit data from school years 2013-14 through 2018-19.

But neither the DME nor OSSE (which is under the oversight of the DME) provided the SY18-19 data for the audit—even though it was available in fall 2020, months before the audit was released. Without the SY18-19 data, the auditor could not verify graduation rates because there were only 3 school years of data.

(Makes you wonder what is so damning about that SY18-19 data—after all, it’s not like DC hasn’t seen graduation rates manipulated again and again. Or maybe that was the point of omitting it?)

Fascinatingly, the DME responded to the auditor that the data at the time of the request wasn’t available and that providing it now, in 2021, would be outside the scope of the original work because, well, it’s 2021! The DME also alleged that the auditor was out of compliance with FERPA (the family educational rights and privacy act), which guarantees student privacy.

[Confidential to the DME: While I understand that no one outside your office found any wrongdoing by the auditor, I can attest that this June, I received material via FOIA from OSSE that revealed the exact numbers of students enrolled at every DC publicly funded school in identifiable categories. As far as I know, this violates DC’s own FERPA rule of not revealing data for less than 10 students in identifiable categories. I am sure as DME you will be taking this up with the newly appointed (yet still unconfirmed!) acting head of OSSE who, like you, has had some experience with expanding charter outreach in Philadelphia.]

The auditor’s work in correcting DC’s education data record in 2021 didn’t stop with that March education data audit.

For instance, in the wake of the June 8 budget hearing for DC education agencies, the auditor sent a letter to the DC council to clarify and correct statements made by a variety of the mayor’s education leaders at the hearing. In the letter, the auditor reiterated huge DC education data lacunae, including

–how OSSE does not require DC charter schools to use early childhood education quality assessments or collect individual educator salary data, despite the large footprint in DC for early childhood education in charters; and
–how OSSE does not connect career and technical education (CTE) data to enrollment or collecting course data to help illuminate CTE effectiveness; and
–how DC lacks an early warning system for students; and
–how DC lacks data on vaccinations across the city’s school children.

Then, on June 29, the auditor sent another letter to the DC council, clarifying how OSSE obfuscated to the council regarding a statewide longitudinal data system (SLDS) and noting that OSSE fails to collect information about student credits and courses and their entry and exit. The auditor also noted that OSSE’s student information system needs to have data from all DC LEAs, not just 17 (!). And the auditor noted that OSSE must collect courses and have an early warning system.

By August 10, when the DC council passed the budget support act for FY22, it contained the following provisions:

“The subtitle requires the OSSE to develop a plan for: creating a standardized course coding system for the courses offered by the District’s Local Education Agencies (LEAs) and include credit hours, student enrollment, and grades; building and implementing an early warning system to flag students at risk for disengaging from school; and making improvements to the District’s Statewide Longitudinal Data System that align with the National forum of Education Statistics. “

Sounds good—until you get to the kicker:

“OSSE must complete and submit the plan to the Council by March 14, 2022.”

So, the auditor theoretically got what she wanted—except not really, because there’s no date for instituting any of this. There’s, literally, only a date for a plan.

(See here for an excellent summary of why this is important.)

To be fair, a plan is more that the state board itself got when it waded into legislative matters regarding OSSE collecting and disseminating data on teachers and teacher retention in DC’s publicly funded schools. A bill introduced in 2019 for that purpose died quietly, but the state board re-introduced it in July. Let us hope it fares better than its predecessor.

Still, all of this begs one question:

Outside The DC Auditor’s Office, Why Is Robust Education Data Routinely Ignored In DC?

In a word: money.

DC is a wealthy city whose publicly funded schools receive annually more than $2 BILLION in public money, which no one in the city has ever tracked fulsomely–much less determined if using that money to pay many, highly remunerated executives for 60-some privately run LEAs is really a wise use of our resources.

So who are you going to complain to about this state of affairs: the mayor, who ensures it is this way? The DC council, which no longer even has dedicated education committee staff? The state board, which has no power except that of persuasion?

As it is, that annual outlay of $2 BILLION of DC taxpayer money gets around to more than just DC students and those handsomely remunerated charter executives. It also goes to elected leaders–and their privately sourced initiatives as well as their causes and supporters–by way of assets accumulated via banking those public funds.

In DC, that amounts to an annual march of tens of millions from public coffers to private hands under mayoral control.

Take so-called high-dosage/high-intensity tutoring. Recall that was being pushed by all the mayor’s education leaders from 2020 on. We first heard about learning loss during virtual instruction, and then about the need for tutoring of the sort that not only would address it, but could be best provided by–wait for it–private groups and un-unionized teaching staff.

In all fairness, that may be the case.

It also may not be.

Simply put, without robust education data, you cannot evaluate that effort or even focus on the students who may be best served by it–even as the DC council has just ensured tens of millions will go toward such tutoring in the next few years (see p. 45 here).

Instead, as with so many ed-reform-driven ideas like high-dosage tutoring, the public resources invested in such efforts become one big slushy fund to help one big slushy machine in service of one big slushy idea: that publicly run anything is inferior to privately run anything. (And what are a few profits between friends on the way to that illustrious goal?)

Worse, the exigency of a public health emergency like covid encourages authoritarian disruption of the sort that mayoral control was designed to promulgate. As testimony before the state board last week revealed, the good people of DC may soon find out how that authoritarian disruption works vis a vis a plan promoted by the mayor that brooks almost no virtual instruction in DCPS (remember that learning loss!). As if to underscore that leader-centric approach, OSSE has even included in its opt-in student testing form a clause waiving liability against the city for covid.

The good people of New Orleans know this emergency-induced disruption well, having watched their public schools disappear by fiat in the wake of a hurricane, only to reappear as charter schools.

Robust data prevents such easy assumptions of power and the big slushy machines behind it–including the research that money funds.

Can that happen with DC’s education data? That’s the $2 billion per year question.