Seeking (Economic) Enlightenment

I think I'm about as susceptible as any other human to having my opinions formed by someone else. All I have to do for this to happen is read someone else's opinion about something, before seeing that something for myself.

So when I found out about Economic Enlightenment in Relation to College-going, Ideology, and Other Variables: A Zogby Survey of Americans, by Daniel B. Klein of George Mason University and Zeljka Buturovic of Zogby International, I decided to read it for myself, properly.

The paper, you see, concludes that politically left-wing people know a lot less about economics than do politically right-wing people.

This has caused something of a stir.

I deliberately avoided reading any other analysis of the paper before I read it myself, and then wrote most of this interminable post. (Had I more time, I would have made this shorter.)

Then I put a bit on the end that links to other discussions of the paper, and summarises the stuff that I missed.

I did all this instead of writing about Lego printers or something because I've been thinking, recently, about scientific papers and their interpretation and reporting. Mass-media science reporting has, I think, never been lousier. If you really pay attention to mass-media science reports, you'll hardly have time to worry about GMOs giving you CJD and UFOs landing at HAARP, because you'll be too busy clearing out your pantry, because whatever stuff cured cancer last week has now been conclusively shown to cause it.

You can usually still get reasonable interpretations of new findings from something like Scientific American, but normal everyday news sources are worse than useless, spraying anti-facts all over the place daily.

If you want to know what some particular piece of research really means, you therefore have to go to the source yourself. This is an important skill for modern humans.

It's tempting to not read the actual paper at all - or just scan the abstract - and then read what some blogger you like said about it. But you really should dig into papers properly, at least occasionally. Now that it's so often possible to have the whole thing in front of you, for free, in a matter of seconds, there's no excuse for just reading what some newspaper journalist mistakenly thinks he read.

I suspected that Economic Enlightenment in Relation to Blah Blah Blah was just a barrow-pushing junk survey, because that's what a lot of political polling is. Complete-garbage surveys are all that various interest groups need to move their barrows along, after all. It only took Sir Humphrey a minute to persuade Bernard that he simultaneously supported and opposed the reintroduction of conscription, so why try harder? Why cover your ideological nakedness with a real fig leaf, when a scrap of paper's cheaper?

Whether or not this particular study was junk, I knew I could easily find some journalist telling me that it was. Or that it wasn't. So I downloaded it myself (PDF), and read it. Feel free to do so yourself, before reading on to have my own opinions stamped on your brain.

The Buturovic/Klein Economic Enlightenment survey has a pretty clear conclusion. To the great delight of the Objectivist playpen in the Wall Street Journal's op-ed pages, this survey found that "the left" in the USA "flunks Econ 101".

The United States has, of course, no actual left wing that any of us foreigners can identify. When US "liberals" agree with policies that were too authoritarian for Richard Nixon, then they're only "liberal" in a relative sense. (That's right - US political blocs are defined by relativism! Or relativity, or something! My god - it'll be social justice next!)

The elephant-in-the-room problem with the Buturovic/Klein paper is that although it was conducted by Zogby International, a respected and above-board organisation, the actual respondents were from an "Online Panel", not a proper random sample. Zogby invited 64,000 people to take part in the survey, and those 64,000 were of course already biased in favour of people who can access the Internet and care to be involved in surveys.

(The Zogby Online Panel appears to be something you can sign up for. Surely you don't have to actively sign up to be surveyable in this way... but if you don't, how can they contact you without spamming? If the Online Panel is actually the same near-meaningless fluff as TV ratings, then the whole project is in dire danger at the outset.)

Anyway, only 4,835 of the people invited to participate responded. That's a 7.6% response rate, which is about par for the course for entertainment-value-only Internet polls. It's a serious, serious problem for any survey that's meant to have some scientific rigour.

You can try to balance things out by weighting responses so that the responders' demographics match those of the whole population; Zogby usually seem to do that with their own Internet surveys. But the authors of this paper didn't do it, and may not actually have been able to do it, if subsets-of-a-subset problems would have left them giving large weight multiples to very, very small slices of the respondent base, giving rise to error bars taller than the whole chart.

Here, though, is one of several points where this paper doesn't follow the standard Crap-Survey script. The authors didn't weight the data, but they make it all available online, so you can see what they were actually working from, and massage it yourself if you like.

(Here's the "survey instrument" in Word DOC format; here's the results in Excel XLS format.)

You may be wondering how the "Economic Enlightenment" survey defines "Economic Enlightenment". And, indeed, how it defines "the left" and "the right".

Well, Economic Enlightenment - "EE" from now on - is meant to be your ability to understand economic reality. Like, I suppose if you can understand that if you don't have much money then it's probably better to rent accommodation than to take out a large zero-deposit mortgage, then that's an economically-enlightened decision.

I think it's uncontroversial that most people in the modern Western world don't have a lot of economic sense. The credit-card companies wouldn't be sitting on such a wonderful green gusher of cash if people-in-general realised that holding your damn horses until you can actually afford something, rather than borrowing at 20%-plus to buy it, will let you own a lot more stuff. People keep borrowing big to buy a brand new car, too; I'd put that decision on my definitely-not-EE list.

(Actually, I think there's a bit of a no-real-left-wing sort of situation in the EE world, too. Look at all of the people in the affluent West who consider it completely normal to be deep, deep in entirely optional debt for your whole adult life. In comparison, anybody with the vaguest semblance of actual money-sense looks like some sort of Oracle of Infallible Wisdom. I dunno what Warren Buffett would count as on this scale; perhaps he'd be a strongly-superhuman Banksian economic Mind.)

The EE survey admits on the first page that their "designation of enlightened answers" may be a "controversial interpretive issue", and that they specifically went out huntin' for "leftist mentalities", without asking questions slanted the other way.

This is another big and significant problem.

Their page-3 example of a survey question, for instance, is "Restrictions on housing development make housing less affordable", with the usual multiple-choice answers from "Strongly Agree" to "Strongly Disagree" and "Not Sure".

The authors use this question as an example of how they "Gauge Economic Enlightenment", because a question apparently has to have at least this definite an "enlightened" answer to be worthy of contributing to an EE score.

But they admit that there are still confounding factors, because different people will have different opinions about what the question's really asking.

What sort of "restrictions", for instance, might we be talking about? Does "affordable" relate to initial purchase price alone, or purchase price plus maintenance and making-good of a shoddily-built house, treatment for the lung disease you got from un-"restricted" asbestos insulation batts shedding fibres into the HVAC ducts, et cetera? What does it "cost" if an electrical fault burns the house down, and you die? What if an un-"restricted" housing industry forms a cartel that builds houses out of damp cardboard and forces poor people to live in them - for a price that's exactly as un-"affordable" as makes the cartel the most money - or live in the park?

The paper doesn't really go into that much detail in its brief discussion of confounding effects, which given its respondents, all living in the troubled-but-not-total-chaos current mainstream US economy, is probably fair enough. There are infinite possible wiggy reasons why someone might mean something strange by their answer to what you thought was a clear question, and if your sample's big enough and random enough (which, once again, is a problem for this paper...) you can iron most of that out.

But I think there's one confounder that should have been mentioned specifically:

Deliberate lies.

Someone who's sympathetic to the current US "radical conservative" movement may personally believe that Sarah Palin is an idiot, but tell a pollster that she's a genius, just to Fight the Good Fight. Similarly, someone who wasn't paying attention during the most recent interminable US Presidential campaigns and so was under the impression that Obama had promised to immediately end both wars and nationalise Halliburton, may tell a pollster that he's 100% happy with the President even though he's actually very disappointed.

(For the same reason, I find it difficult to believe any survey about the sexual activities of teenagers. Religious loonies often seem to get very excited when a survey comes back saying that 90% of 14-year-old boys have had sex a thousand or more times. I don't know whether I'd rather those loonies are so upset because they actually believe the survey, or if they know that it's BS and are feigning belief to advance their own agenda.)

There's definitely some slanted question-selection going on in the EE survey; they admit as much. They had 16 multiple-choice questions in the actual survey, but they chose only eight of those questions to make up the final EE score for respondents. They say they eliminated the questions that were "too vague or too narrowly factual, or because the enlightened answer is too uncertain or arguable" - but I'd say the "narrowly factual" part shouldn't be a problem at all. Ask someone what an "interest rate" or "inflation" is; if they don't know, their EE score drops. An awful lot of people don't seem to understand income-tax brackets; there's another great question for a more factual EE test.

(Perhaps such questions would measure mere economic "literacy", not "enlightenment", though.)

Several of the dropped questions also seem to me to be more likely to get "leftist" answers, since they include, for instance, "Business contracts benefit all parties" and "In the USA, more often than not, rich people were born rich".

But here, again, is evidence that this isn't a pure obviously-fake barrow-pushing trash-poll. The paper tells you they dropped the questions, and why (rightly or not), and the authors also tell you what the dropped questions were. Leaving that last detail out is exactly the sort of thing that trash-pollsters do, because that way you can avoid disclosing that they, for instance, asked 50 questions and published only the ones whose aggregate answers happened to support their thesis.

(The downloadable full results also include responses to all of the dropped questions. I'd do some analysis of that data, if this post wasn't already the size of a holiday novel.)

On page 4 of the paper, there's a list of the eight questions they used, and the answers they deemed "Unenlightened":

1. Restrictions on housing development make housing less affordable.
Unenlightened: Disagree
2. Mandatory licensing of professional services increases the prices of those
Unenlightened: Disagree
3. Overall, the standard of living is higher today than it was 30 years ago.
Unenlightened: Disagree
4. Rent control leads to housing shortages.
Unenlightened: Disagree
5. A company with the largest market share is a monopoly.
Unenlightened: Agree
6. Third-world workers working for American companies overseas are being
Unenlightened: Agree
7. Free trade leads to unemployment.
Unenlightened: Agree
8.Minimum wage laws raise unemployment.
Unenlightened: Disagree

To keep this post under the 50,000-word mark, I leave determination of all possible well-reasoned but "unenlightened" answers to these questions as an exercise for the reader. Questions 1, 2 and 8 seem pretty straightforward, if simplistic, to me.

I also at first thought it was pretty hard to reason your way to the "wrong" answer for question 3, as well - but then I realised how many ways there are to measure "standard of living", besides "number of features in your car" and "number of televisions in your house". Is "standard of living" the same as "mean household income"? If not, how not? Answers on a postcard, please.

And the rest of the questions, not to put too fine a point on it, seem to me to be wide friggin' open. Not least because of the lack of any clear definition of terms.

I know, for instance, that there are people in the USA who seriously advance the idea that no Third-World workers for US companies are being exploited in any way, because apparently being paid a quarter of a living wage toward the large debt you incurred when you started work in the crap-for-fat-Westerners factory isn't exploitation. But if you take the completely crazy position that maybe some people in the Third World are being exploited by US companies, and therefore disagree with assertion 6, you're officially Un-Enlightened.

The very next paragraph of the paper, entertainingly, says that any objection such as this is "tendentious and churlish".

I may now find myself required to challenge the pinguid, sesquipedaliaphiliac diplozoon responsible for this paper to a duel.

Let us now move beyond the lack of imagination of the authors as regards valid objections to their definition of enlightenment, and their questionable... question... selection, and move on to the demographic differences between the people who scored high, and low, in EE (whatever, if anything, the EE score actually measures).

The paper's headline "discovery" is that going to college didn't give respondents a statistically-significant higher EE score.

I imagine that tertiary education specifically involving economics - or just a weekend personal-finance-management seminar, for that matter - would have an effect on the EE score - at least, for the four questions that really do seem to have pretty clear objectively-correct answers. But since most university students wisely avoid the dismal science with the same zeal with which sane people avoid the Continental postmodernists, it's hardly surprising that just passing through a university does not cause one to pick up knowledge of economics by osmosis, without studying it.

(Colleges won't make you study it, either. Later in the paper, it points out that of "50 leading universities" surveyed by these people, exactly none had compulsory economics courses.)

It's practically a truism that knowing a great deal about one subject has little to no effect on your knowledge of other subjects. Actually, people who're very knowledgeable about one thing often incorrectly assume that they've got the right end of the stick about some completely different subject. Scam artists love Ph.Ds. (See also, "Engineers' Disease".)

Research that confirms the "obvious" is still valuable, even if it's routinely reported in News Of The Weird stories and derided by politicians who're trying to reduce "wasteful" government spending. (By, of course, taking funding away from anything that they reckon sounds a bit silly, and giving it to the people who've given them a rent-free flat.)

Still, though, EE and college education being uncorrelated doesn't look like a big discovery to me. (If the EE testing method itself is fatally flawed, of course, then no correlation, or lack thereof, means anything.)

What else you got, guys?

Well, there's the left/right thing.

One very simple way of pigeonholing people as "left" or "right" in political ideology is to just to ask them, which this survey did. Once again, the lack of a real left wing in the US political dialogue means that a reborn Dwight D. Eisenhower would now be categorised as a State-trampling tax-and-spend socialist enemy-emboldener - but never mind that for now. The survey asked respondents to categorise themselves as "Progressive/very liberal", "Liberal", "Moderate", "Conservative", "Very conservative", "Libertarian", "Not sure" or "Refuse to answer".

And lo, those who admitted that they were infected with the terrifying disease of "progressivism" scored the very worst on the EE scale, with a neat diminishing-wrongness progression as you proceed toward "Very Conservative". And then a score a little better again - though not statistically-significantly so - for the brave and hardy "Libertarians"!

Once again, this was presented according to proper scientific standards, with a full breakdown and confidence interval listed. The error bars are wide enough to, as I said, mean the Very Conservatives might actually have beaten the Libertarians, but the overall order is clear. And the authors mention, again, that questions specifically aimed at "typical conservative or libertarian policy positions" might have changed the results. (Like, I dunno, maybe "Illegal immigrants are a major drain on the American taxpayer.")

But they, again, conclude, "Naaah." (I paraphrase.)

Next we get a bunch of little tables demonstrating that people who voted for Obama have miserable EE (but people who voted for Nader or the Green Party's Cynthia McKinney score even worse - though the error bars are of course really large for these unpopular "wasted vote" candidates).

Who else scored badly? Oh, just black people, Hispanics, citydwellers, Jews and Muslims, union members, and people with no direct or familial connection with the armed forces.

Who else scored well? "Atheist/realist/humanists", people who did not consider themselves to be "a born-again, evangelical, or fundamentalist Christian", and people who go to church "rarely" or "never". I'm not sure what this is supposed to mean, but it's amusing.

Oh, and "Married" people beat every kind of single person, and beat by a wider margin people in a "civil union/domestic partnership". "Asian/Pacific" respondents scored even better than white folk. NASCAR fans scored better than others, too. (Somewhere in America there must be a married Japanese-American NASCAR-loving atheist Republican who has a perfect model of the entire world economy turning and twinkling in his mind's eye.)

And Nader and McKinney voters may have scored miserably, but people who voted for the Libertarian candidate Bob Barr got an average score even better than those wily McCain voters, though again with a big enough error bar that they might not really have scored higher in a bigger survey.

Registered Libertarian voters scored better than people affiliated with different parties, and in response to "Do you consider yourself to be mostly a resident of: your city or town, America, or planet earth", Planet Earthers scored worst, followed by "not sure/refused", then "my town", then "America". (Presence or absence of a subsequent "Fuck Yeah!" was not recorded.)

All this isn't quite the results that a modern US "radical conservative" would really want to see, but that's just because religious beliefs and a good EE score appear to be incompatible (though "Other/no affiliation" for the religion question scored even worse than those silly Muslims!). Apart from that, the results are driving straight down the radical-conservative road. In brief, the authors' thesis that conservatives and Libertarian-ish people have higher Economic Enlightenment than members of the Pinko-Green Communist Alliance was solidly supported across the board of their questions.

There were several really nice line-fits, too. I mean, check this out:

Income versus 'economic enlightenment'

The more you make, the more economically enlightened you are! Makes sense, doesn't it, kids?

But wait a minute. Look at that tiny little error bar for those brilliant high-EE "$100K+" respondents. A smaller error bar means a larger sample. Did they really have more respondents making $100,000 or more than any of the other income brackets?

According to the raw data - yes, they did! The breakdown for the 4,835 respondents was:

No answer: 593 (12% of respondents)
Below $25K: 277 (6%)
$25-$35K: 337 (7%)
$35-$50K: 541 (11%)
$50-$75K: 941 (19%)
$75-$100K: 757 (16%)
$100K+: 1389 (29%)

Now, this was total household income, not the personal income of the person answering the survey. But the median and mean household incomes for the USA in 2004 were $44,389 and $60,528, respectively. I doubt that either figure has shot up past $100,000 in the last six years.

When 29% of respondents are making around twice as much as the average income - the number of $100K-plus responses is only marginally smaller than the $35-$50K and $50-$75K respondents put together - serious "skewed sample" alarm bells should start ringing.

The paper does have quite a bit of discussion of the problems with their testing technique, but of course concludes that none of them invalidate the study. Or mean that they should have applied weighting to try to un-skew their strange self-selected sample.

(There's also, once again, the possibility of deliberate deception. Respondents might seek to give their ideology-driven answers to other questions more weight by claiming household income much higher than what they actually make.)

On finishing reading the paper - which, gentle reader, means that the end of this epic saga of a blog post is also in sight - I figured that the main problems with it were this obviously unbalanced self-selected sample, the lack of any weighting to attempt to compensate for the sample bias, and the selection of the questions used to construct the EE score.

Apart from that, I reckoned this was a decent paper. It's rather sad that the best you can say about so many media-touted studies is that they conform to the minimum standards for an academic paper - presenting methodology and results, and not blatantly lying. But still, it's better than nothing.

("Yeah, that car he sold me WAS full of rust, but at least it really was a car, not just a couple of bikes covered with tape.")

Perhaps I'm so easily impressed by any paper that achieves the basic benchmarks for publication in a peer-reviewed journal because I'm so used to examining the rather different evidentiary paperwork of true out-there crackpots. Those guys often insist that their magic potion or antigravity machine has been tested by some prestigious institution or corporation - UCLA, Bristol-Myers Squibb, the US military, and of course poor old NASA. But when you ask who actually did the test, and when, and whether it was published anywhere... well, you may end up with a photocopy of a photocopy of a photocopy of something that might originally have been on university letterhead. Or test results from special secret scientists or car-gizmo testers who always seem to find things that nobody else can. But you'll probably only receive abuse.

Compared with that, this paper is a magnificent solid-gold triumph of the scientific method.

What, I now wondered, do people who do not bear the mental scars of numerous encounters with extremely independent thinkers make of the Buturovic/Klein study?

I returned to the page that, seemingly years ago, alerted me to the study's existence in the first place: This question on Ask MetaFilter.

Commenters there linked to this FiveThirtyEight piece by Nate Silver - who has an economics degree.

Silver has previously written that Zogby's "regular polls" were acceptably accurate in the last US Presidential election. But "Zogby Interactive", the "Internet Panel", has consistently been appallingly inaccurate. Because, yes, you really do get on the Internet Panel by just signing up at the Web site!

Knowing this, I feel I now have no option but to class any actual academic researcher who uses the Zogby Internet Panel, but doesn't weight the results and stretch the error bars accordingly, as being deliberately deceptive. There is no excuse for pretending that the Internet Panel is directly representative of anything but itself, even if you take care to ask an unbiased series of questions, which Buturovic and Klein clearly did not.

In this particular case, Silver once again points out the lousiness of the Zogby Internet Panel, and the questionability of the "Economic Enlightenment" questions. He also mentions that some of the questions do not have a clear answer even according to professional economists, " Klein should know, since he's commissioned several surveys of them."

This does further damage to the headline "college education doesn't teach economics" finding; actually, the more you learned at university about economics, the more likely you appear to be to give the "wrong" answer for one of the EE questions. This turns that finding into a tautology.

Nate Silver's conclusion from this is that the study is "junk science". If Silver's post had been the only thing I read about this study then I'd agree with him; having actually read the study, I still agree with him, because what he noticed lines up with what I noticed.

Another MetaFilter commenter pointed out that the questions asked will allow anybody who sticks to mainstream US "conservative" viewpoints to, regardless of their actual level of comprehension of what they're saying, get an excellent EE score.

Commenters also came up with a number of theories about why the paper is the way that it is, for instance perhaps because of a conscious or, just barely possibly, unconscious desire to contribute to the US radical-conservative echo chamber about universities being hotbeds of crazy left-wing brainwashing.

(It's true, you know. By and large, the more education someone's received, the more likely they are to hold "leftist" political views. Clearly, brainwashing is the only possible explanation for this.)

And then there's the issue of mining for correlations. If you measure a lot of things and then shuffle the data around until you find something that correlates with something else, you may have discovered a real relationship. But as a dataset increases in size, the chance of finding a statistically-significant but entirely spurious correlation in there somewhere approaches one. Hunt through the data until you find similar-looking graphs and you may indeed have discovered that G causes R, or that R causes G, or that both R and G have a common cause that you didn't measure. But G and R may also appear connected by a pure fluke.

The Buturovic/Klein poll contains a sort of back-door correlation-mining; the question selection seems to have guaranteed the overall "conservatives smart, liberals dumb" conclusions.

(Another commenter was surprised that there wasn't any Laffer Curve BS in the survey. And yet another commenter cunningly attempted to lengthen this post by mentioning a two-question ideology-versus-science test involving "deadweight loss".)

Someone also said that Zogby "is a bit of a joke among other pollsters". But I find it hard to dislike John Zogby himself:

Honestly, I could have better used the hours I spent poring over this study. You could probably have better used the time you spent reading this page.

But there is, at long last, a point to this beyond just debunking that one fatally-flawed study. It is:

The next time you see a reference to a scientific paper on a subject that interests you, if it's possible to dig up the paper without having to trek to the nearest university library or something, do so, and read it for yourself.

(If you've got a standard worse-than-useless newspaper science article in front of you and you're trying to figure out who the "scientists" are who've allegedly discovered the cure for oh-god-not-again, Google Scholar is a good place to start. Note, however, that the modern mass-media science story is based on press releases from university and corporate PR bodies, who are famous for sending puffed-up announcements about studies that haven't actually quite been published yet. If it ain't been published, you ain't gonna find it in Google Scholar, PubMed or anywhere else.)

You need advanced education to understand some scientific papers. You're probably not going to get a lot from a paper about, for instance, cryptography or particle physics, unless you're already quite knowledgeable in those fields.

But a lot of papers, definitely including many of the psychological, sociological and medical/epidemiological papers that are so popular with the newspapers, can be comprehended with nothing more than a bit of light Wikipedia use and basic knowledge about statistics and probability. That latter knowledge is, of course, useful in all sorts of other situations too.

(You can get a basic tutorial in stats and probability from Wikipedia too, or in a more structured and entertaining form from the classic How to Lie With Statistics, and/or Joel Best's much more recent Damned Lies and Statistics. John Allen Paulos' Innumeracy is also excellent.)

At the very least, it's a salutary mental exercise to understand what a good study's saying, or to figure out what's wrong with a bad one. And it can also tip you off about the reliability of different sources of information about scientific discoveries. Who knows - you may find that your local newspaper has a science reporter who's actually good!

The developed world is entirely built upon a foundation of science, and the basic interchangeable unit of scientific research is not, as one might suppose, the undergraduate lab assistant, but the published paper. To float along on the surface of the world's science and technology without ever looking at the papers from which it is all built is like eating meat daily without taking any interest in what happens in a slaughterhouse.

I've been to a slaughterhouse.

I find reading scientific papers somewhat less unpleasant.

14 Responses to “Seeking (Economic) Enlightenment”

  1. Alex Whiteside Says:

    I really wanted to read the whole thing before moving on to your analysis, but I hit page 178 and couldn't take any more. It's like taking your car to an incompetent mechanic and watching as he systematically destroys each component while stating in a soothing voice, over and over, that he knows what he's doing.

  2. corinoco Says:

    Wow. I'm dizzy now, just from reading Dan's article; or it might be the flu.

    I know that you can prove pretty much anything with statistics; the 8 questions they asked read more like the sentence construction used by Astrologers. Certainly they were pretty bloody simplistic. Particularly Question 3 - What "standard of living"? There isn't a bloody ISO standard for that is there? Hmmph.

    I partly-remember a quote from somewhere: "We always put the three 'joke' stories at the end of the news - The Weather, The Economic Report, and a story about puppies" Or something along those lines. It was either 'Broadcast News' or Douglas Adams. I forget.

    I'm pretty sure you could very easily statistically prove that ALL of the ultra-wealthy who have made money through stocks, trading, etc. have done so with what amounts to massive insider trading. You just wouldn't live very long until you had a very improbable but nasty accident.

    I've just gone and read some of the links and comments too. I think I've just solved Fermi's Paradox: any sufficiently advanced civilization is pretty much wiped out 150-200 earth-years after 'discovering' economics. At the very least they lose any ability to attain orbit or even interplanetary creatured-flight as such tasks are 'leftist' and 'economically unenlightened'

    Next we'll hear it is best to make only shoes, and the wheel is economically unviable.

    Someone please shoot this planet, it is fail.

  3. bfabry Says:

    *Before reading your comments*

    I was pretty upset at the idea of having to wade through a 170 page paper just to feel justified in my opinion that zogby is a shill polling company (a seed planted by and so you can't imagine how happy I was when I got 4 pages in and was able to heave a sigh of relief at the questions.

    I don't have to read any further, they're bad enough to make all the rest of the poll pointless :-)

    EDIT: *dammit* dan, web page design 101, elements stick out, text doesn't. If I see a "Name" and an "Email" field I'm going to assume that I don't need to log in before I bother to read any text. *grumble*

  4. trouserlord Says:

    Based on the questions used in this study, it would appear that the "Economically Enlightened" person could be more simply and clearly defined as "Selfish Arsehole". In light of this I find the conclusions of this study not the least bit controversial.

  5. dr_w00t Says:

    Hey Dan, thanks for writing... always enjoy your longer blog posts. They are lively, fun and entertaining, and I always feel 0.005% (compound) smarter for having read them.

    With everything going on in the world, it seems like we are watching the decline of the Roman Empire or something, before our very eyes.

    Governments more and more self-serving, less and less useful to those they were actually elected to serve... scams and alternative mumbo-jumbo at every turn, critical thinking actively suppressed by various elements.

    It seems a bit like the turn of the millennium heralded the proliferation of ignorance to me... The path of the rational man is beset on all sides by the iniquities of the selfish and the tyranny of ignorant men. (It would be way cooler if a black dude with an Afro said it).

    Maybe I'm just getting older and the cynicism is becoming more deeply ingrained, and "when I was a boy!" and all that. I know, for example, my generation consumes less alcohol than my parents', for all the MSM ballyhoo about the binge drinking culture of the youngins and whatnot.

    Anyhow I hope your posting frequency picks up again. Even if (actually especially if) you post about your life and thoughts rather than the 1/10th robotic T-Rex mounted Gatling laser you built.

  6. Mohonri Says:

    I'm no statistician, but I fail to see the importance of weighting in the context of this story. When you're doing an election-year poll, you adjust the weighting based on the political affiliation of the respondents, so that the stats line up with the actual political alignment of the populace. For example, consider a Presidential Approval Rating poll. If 40% of respondents are democrats and 30% are republicans (obviously a US poll), but the actual population is 38%/32%, then you adjust the weight of the two samples in order to get the aggregate number closer to reality.

    In this case, since the different groups (let's take the income categories as an example) are not aggregated, I think the results are still pretty valid. This assumes, of course, that the respondents gave correct information on their background. Had the responses been compiled into a "36% of Americans scored less than 3/8 on an economics quiz," then weighting would have been a factor.

    There are a number of issues with this poll, but I don't think weighting is one of them, since we're not seeing "aggregate" statistics.


    Wait, you built a T-Rex with a Gatling laser cannon on it? And you haven't posted a thorough review yet!? Dan, you've been holding out on us!

  7. Daniel Rutter Says:

    The reason why weighting would help - if only a little - in this case is because of the miserably non-random sample. Weighting wouldn't solve the problem - it wouldn't come close to doing that - but it'd at least show willing.

    The problem with a self-selected sample is that you cannot tell how representative the respondents are of the population at large. Even assuming they all tell the truth, the statistical relationship between your white-female-suburbanites-with-household-income-X-and-so-forth and the complete set of white-female-suburbanites-with-household-income-X-and-so-forth is unknown. You can fudge the numbers until the distribution of qualities in the sample roughly lines up with the distribution of those qualities in the actual population, but you can't do anything about whatever biases self-selection created.

    The classic problem with self-selected samples is that you end up mainly polling people who have strong opinions on the subject and know about your poll. If your attitudes-on-sex-education poll receives a bunch of bussed-in bloc votes from fundamentalist churches, or your attitudes-on-drug-prohibition poll gets zillions of voters coming from the front page of Reddit, your sample will be so skewed as to have no real scientific value at all. You may be able to dig some tiny little nugget of truth out of it, but it's not likely to be worth the effort.

    It's like those broken medical trials which claim to demonstrate that Potion X cured Condition Y, but because of self-selected samples, undefined endpoints, no blinding, blah blah blah, actually only demonstrate that for the people you studied in the month you studied them, Potion X did not prevent Condition Y from clearing up by itself.

  8. Coderer Says:

    Notes as I read:
    * Maybe like you say it's open to interpretation, but I consider it pretty clear cut that "housing development" != "housing construction" (maybe that's a US-English thing?). "Development" would specifically be more like zoning issues, e.g. who can build what where. I'm sure what they were specifically fishing for was e.g. wetlands protections, etc. There's no doubt that *development* restrictions drive up housing costs due to simple supply and demand.

    * On "enlightenment" versus "literacy": I'd say the "narrowly factual" questions are like polling people about sports trivia, versus understanding of the rules of the game. It's one thing to know Cal Ripken's batting average from a given year, and quite another to know how many balls before you walk -- a test that measures how well you understand baseball would benefit far more from the latter than the former.

    * Do you actually drop the "H" sound in "holiday"? I thought that was strictly Cockney -- it's how Dick Van Dyke says it to Mary Poppins.

    [No, that was just an error :-). It was originally "the size of an {something starting with a vowel}", but I changed that, without changing the "an". Fixed now! -Dan]

    * Failure to teach even basic economics in our mandatory schooling years has always been a sore point for me -- seriously, I need 13 years of 3-times-a-week Phys Ed, but you couldn't spare a single semester for Econ 101?

    Short closing thought: my wife has an unfortunate habit of roping me in to controversial viewpoints on the Internet, the details of which I won't bore you with here. I usually tell her what you're telling us -- go hunt down the source, read the original yourself, make up your mind. But she makes a valid point, which you (kind of) come to yourself at the end: she doesn't have time to spend hours reading every source document, so the best she can do is find somebody she *trusts* to make a fair conclusion and see what they think. This is harder for me, maybe because my male/engineering/analytic mind wants to work through from first principles rather than go the "trust" route that fits with her female/empathetic/social mental wiring, but I can see where she's coming from.

    The simplest approach, of course, is to just agree with the findings that suit your existing world view :D

  9. Anne Says:

    @Coderer: it's not the maleness of your mind that makes you want direct evidence. But it might be your training. In science the ultimate arbiter of truth is agreement with experiment, and it's often possible to determine the truth of ideas with a minimum of trust in individual people. But there are perfectly-functional disciplines where the determination of truth is based on other standards. The most straightforward example is possibly history. If you want to know, say, how Stalin came to power, you can't do an experiment to test the truth of your ideas. You have to trust sources. But this isn't a blind thing; the focus of a professional historian's training is precisely in how to evaluate sources and sift their statements for biases and omissions in order to come to the best possible picture of the truth. Of course there are questions where the answer is debated or simply impossible to determine, but that's true in science as well.

    In short, it's perfectly possible to do critical thinking based on evaluating sources and their biases and trustworthiness. It's just a different way of thinking than scientists are trained in. (Engineers I can't say; there's a perception among scientists that engineers just trust the first statement they find in a textbook, but I think this is pure snobbishness.)

  10. Alan Says:

    It's odd, but this reminds me of the Census taken every few years by the Australian Government. Filling in the form is compulsory, which is itself a contentious issue.
    As a result, last time the Census was being issued, the was a suggestion that people list their religion as "Jedi Knight". The implication being that established religions must be listed on subsequent forms.

    You could call such a response a deliberate lie, or a rejection of current mainstream religions. I suppose I should look up how many people responded - it should be available from the Australian Bureau of Statistics.

    (Hmmm. Could I use my religion as an excuse to get a high power laser? Every jedi needs a light sabre.)

  11. loseweightslow Says:

    Is the police force considered a licenced professional service?
    Because when it becomes unlicenced the price for policing goes through the roof.

  12. pjcamp Says:

    Zogby Interactive is not that respected. For that matter, neither is Zogby Phone. See stats whiz Nate Silver:

    where Zogby scores the absolute highest pollster-introduced error. He also has no listed commitment to transparency and disclosure in his results.

  13. felixfelix Says:

    Ah, there is a special dismalness about economics, and surveys, and especially economics surveys. But aside from the demographics being undercooked even the raw data is a bit rubbish.

    Take for instance the truthiness of the link between minimum wage and higher unemployment. At the moment Wikipedia has it covered nicely: weak evidence of the link in the literature, and evidence of large scale publication bias, And amusingly, this places Paul Krugman and Joe Stiglitz as unenlightened on this question.

    EE isn't a measure of economic literacy let alone "enlightenment"; I'd argue it's more a proxy for whether you subscribe to right wing talking points.

  14. Jonadab Says:

    Taken at face value, I would have considered questions 3, 4, 5, 7, and 8 to be beyond (coherent rational) dispute (by any informed person). I have never seen *any* clear definition for "standard of living" that could be used to challenge 3; 4 and 8 are completely straightforward paraphrases of standard econ-101 information about price ceilings and floors, respectively; the only way to miss 5 is to fail to realize that some markets are divided among many small competitors, which would imply you know basically nothing about macroeconomics; and the only way I can come up with off the top of my head to miss #7 is to get confused about the distinction between currency and wealth (which, granted, is a very common mistake -- but not one that an informed person ought to make).

    But yeah, some of the others are open to fiddly interpretation. 1 and 2 are basically true as generalizations, but the wording could be interpreted as absolute, and there are interesting exceptions (some of which you point out). Number six, similarly, needs a word or two of clarification. Does the question mean to ask whether *all* such workers are necessarily being exploited by virtue of the stated situation, or only whether there are *any* such cases that involve exploitation? The latter question is clearly a straw man (*obviously* there exist at least some cases of exploitation; nobody would seriously argue the contrary), whereas the former is a subject of actual debate in some circles, but a person could theoretically be informed about economics without following politics closely enough to know that, or, alternately, the reader could be sufficiently skeptical about the poll to expect straw-man questions. Also, the word "exploited" has an obscure technical definition under which *all* workers are exploited by their employers without any implication of wrongdoing. It *should* be clear in context this isn't what's meant, but someone who is highly skeptical of the poll could convince himself otherwise.

    I *suspect* that these failures are a result of inadvertent failure to construct the questions with sufficient care. I've seen enough tests in my life to know that most education professionals (much less people in other fields) are pretty bad at consistently writing good questions, so I see no reason to suppose malice. The problems can all be easily explained by an unremarkable level of incompetence. Actually, the existence of five questions that I deem good in a set of ten puts the thing well above average, in my experience -- not that that makes its conclusions necessarily fully sound. The sampling problems you point out are, of course, legitimate concerns.

    I will also take a moment to point out that there are *certain* economically non-viable ideas that are more commonly embraced by those on the left side of the US political spectrum. For example, central planning and price controls are *significantly* more often promoted by the left. (This does not, however, imply that there aren't *other* economically non-viable notions more often promoted from the other side of the fence.)

    > I imagine that tertiary education specifically involving
    > economics - or just a weekend personal-finance-management
    > seminar, for that matter - would have an effect on the EE score

    No doubt. We know that the quality of economics education in the US (in most countries, for that matter) leaves something to be desired.

    > Colleges won't make you study it, either.

    Well, some do. My alma mater, for instance, does require at least one semester of economics for all four-year degrees. However, relatively small schools (the enrollment was under 700 while I was there) frequently have stiffer requirements than large universities, and a sample of one may not be entirely statistically rigorous in any event.

    > One very simple way of pigeonholing people as "left" or "right"
    > in political ideology is to just to ask them, which this survey did.

    In the US, you can pretty much just ask them their zipcode. You'd have to throw out the people who live in the "swing" districts, but that's a fairly small percentage of the population. Even in reliable-predictor swing states like Ohio, most of the population live in strongly-slanted geographic areas. (For example, Westerville leans to the right; East Cleveland goes the other way.) Even throwing out the swing districts you'd get 5-10% error, from the few people who sit on the other side of the fence from almost all of their neighbors, but it's probably more reliable than asking people whether they're right or left (which tends to get slanted by local perception -- "very conservative" in southern California is about the same as "off-the-charts liberal" in the Midwest).

    A more reliable way to peg US votes is to ask them how they feel about various specific US political figures past and present. Reagan is consistently loved by conservatives; liberals universally love FDR and hate Quayle and Palin; and so on. If you include enough data points, you can identify most swing voters (who generally like everyone popular and frown on everyone unpopular) and also throw out anomalies, e.g., anyone who dislikes Lincoln, or anyone who likes Quayle and dislikes Reagan, is statistical noise and can be safely ignored.

Leave a Reply