Sunday, May 30, 2021

Remember Benghazi?

Think back with me to those days of yore when conservatives blew 89 bowels about the Benghazi attack. Remember all those investigations that found nothing? They were so angry about an attack on a makeshift consulate/CIA base that they were beside themselves. So you would think they would want an investigation into what happened on January 6th, right?


Take your hypocrisy and shove it up your ass. We are coming for you. And we won't rest until you are in prison. 

Friday, May 28, 2021

Is Test Prep for Standardized Testing Cheating?

There's a great example of disingenuity in today's Washington Post, penned by arch-conservative George F. Will:

Thomas Jefferson High School (TJ), a selective STEM magnet school with a national reputation for excellence, has what the school board in suburban Fairfax County, Va., considers a problem: Too many Asian American students excel on the admission test. The current TJ student body is 73 percent Asian American, 17.7 percent White, 3.3 percent Hispanic or Latino, 1 percent Black and 6 percent other. So, the board has decided to eliminate the test. Admissions will be based on a “holistic” assessment of applicants, meaning whatever admissions officials want it to mean.
Will, long a champion of white privilege, wants to roll back any form of "diversity" in admissions. In the guise of protesting anti-Asian discrimination he blasts people who question the validity of standardized testing. And he can't resist large numbers of "scare quotes" to make his point:
The TJ parents’ complaint notes that in 2018, a retired county middle school teacher ominously told Virginia’s General Assembly that Asian American parents are “ravenous” for opportunities for their children. In 2020, a member of the state legislature spoke of, but did not specify, “unethical ways” Asian American parents “push their kids into [TJ].” Presumably, they push their children to do well on standardized tests. At a 2020 town hall meeting, Fairfax County’s schools superintendent stigmatized TJ’s student body majority by complaining that Asian American parents spend “thousands upon thousands” on test preparation. Virginia’s secretary of education later denounced such studying as comparable to “performance enhancement drugs” in sports: cheating. 

The fact is, George, spending thousands of dollars to push your kids to do well on standardized tests is the very essence of cheating.

Here's how it works: parents hire tutors who have taken the test in previous years. Each year these tutors quiz test takers to regurgitate the questions that appeared on the test. They prepare "study guides" (note the scare quotes!) that have the questions and the answers. Students memorize the answers, not necessarily understanding the questions.

Now, creating a good standardized test is an expensive proposition. I used to work at a company that did computerized standardized testing worldwide. Writing a good test isn't just a matter of jotting down random questions. It involves creating a large bank of items (questions), then having a large number of people answer those items, including people who know the subject well. The results are then analyzed and statistics are generated. Bad items (ones that don't predict true ability) are tossed.
Items are added and removed each year, but banks of standardized items generally remain stable over time because they're expensive to create.
This is why many people consider study guides and tutors to be unethical. They literally give the kids the answers to the test. Yeah, they don't know exactly which questions will be on each year's test, and they can't be sure they've got the official answer, but they make the kids memorize everything.

The problem is, memorizing answers to canned questions is not the same as acquiring critical problem solving abilities. But it does give the advantage to children of parents who can afford to throw money at it.

When I worked for that testing company we delivered the written certification tests for pilots on computers, getting rid of the paper-and-pencil tests, which took forever to grade. Pilot trainees at flight schools wanted instant results so they could get into the cockpit that day and do their practical exams. 
Over time we computerized airline mechanic testing as well. One of the airlines wanted to move its service operations to China (this was in the Nineties, back when companies started selling out American workers big time). So the FAA had us set up a testing center in China to accommodate the airline.

The FAA's written mechanics test consisted of, as I recall, 25 or 50 multiple-choice questions. The paper version of the test had several uniquely-numbered standard "forms," each drawing the prescribed number of random items from the larger item bank.

I was at work one day when the sales guy called me from China. "The tests are wrong," he said.
 "What? I don't think so. They're just like the paper ones. They're very simple, straightforward tests. Nothing fancy."

He paused. "But they're in the wrong order."

"Huh? There is no 'order.' We randomized the item delivery order at the FAA's request."

"Well, we have a problem. These guys don't speak English."

"But..." And then it hit me. The Chinese mechanics had memorized the order of the items and the answers for every numbered form of the test. But when we computerized the test the order was determined randomly on delivery, making it impossible for the Chinese-speaking mechanics to cheat.
When it came to the written test, the question with pilots and mechanics was always, Does this really matter? The test is just a formality, everyone said. The practical exam will sort at the bad ones. 

But in this case, it really would matter. All the manuals for these planes were written in English. All the markings on the engines and wings were in English. If the mechanics can't read English, they can't read the manuals. And they can't properly service the planes.

Now, I'm not saying that all Chinese people cheat on tests. Clearly, Americans cheat too (I mean, how else did Donald Trump get through high school and college?). 

But there are definitely cultural attitudes about cheating: in some places, like the Soviet Union, cheating was a way of life. In college in the late Seventies I had friends from Russia and Ukraine, smart guys, who would openly cheat on every test even though they did not need to, just because they could. I didn't know any Americans who cheated because they wouldn't even talk about it, must less admit to it.

The testing company I worked for regularly ran statistics on pass rates for students at all our testing centers to look for cheating. Many of the centers were at schools that provided training for the subject material (we did a lot of certification for computer network technicians and the like).

Some centers, in China and India in particular, had much higher pass rates than others. So we sent in "secret shoppers" and discovered that some of these testing centers offered an extra service: they would stand behind you and tell you the answers to the questions.

Protecting the integrity of the item banks is one of the biggest problems in testing. Some of these testing centers took photos of the questions as they appeared on the screen so they could develop "study guides." 
That was almost 30 years ago -- it's worse now with a cell phone in every pocket and even glasses that have built-in cameras. There's a long history of schemes to steal questions for tests like the SAT (like this and this and this).

I'm sure that Will and many Asian families do not consider spending thousands of dollars on tutors and study guides on standardized test prep to be cheating. But it is because of the ubiquity of item theft.

But there is a larger point here. We require that doctors and nurses and lawyers pass rigorous written examinations before they practice medicine and law. Pilots have to pass tests before they can fly. Real estate agents have to be certified.

That makes sense. People should be qualified before they can do work that can affect people's lives.

But should that rationale apply to kids getting into a high school? How are they ever going to learn this stuff if they're never even given a chance? I mean, these kids aren't doing brain surgery there. They're going there to learn basic science and math.

The admissions process for a school like Thomas Jefferson should determine whether kids can benefit from going to that school -- can they absorb the material and earn a passing grade?
By their very nature standardized multiple choice exams do not test a student's ability to learn and solve real problems -- they test the ability to regurgitate memorized data. Learning and memorization are not the same thing.
Schools like this historically only want the "best" students. Why? Why shouldn't every student have an equal shot at going to a good school?

I suspect that the real purpose of standardized tests is to puff up the egos of school administrators and keep their pass rates high with a minimum of effort. And, sadly, to keep out poor kids, who have historically been ethnic minorities. 
But that's being turned on it's head with the success of many Asian Americans. Will does have a point that white parents are being hypocritical complaining about Asian Americans using the same scam white folks came up with to keep themselves on top.
To a large degree the ability to learn something depends more on motivation and persistence than previous accomplishments. 
Why should a student motivated by a fear of disappointing their wealthy parents get into a good school while a student motivated by inherent curiosity and an actual love of science and technology is rejected because their poor parents can't afford to buy study guides and pay tutors?

Sunday, May 09, 2021