Unimaginable catastrophe

Oh shit. You may have come across this already -- it seems the kinks in the riser pipe of the Deepwater Horizon well are the only thing preventing an uncontrolled gusher, and the pipe is steadily being eroded by sand carried in the escaping crude oil. If the wellhead blows wide open, according to one expert, the hole could gush 150,000 barrels of oil a day. That would be the equivalent of the Exxon Valdez spill every two days, and there is no way they are going to be able to stop this for weeks, if not months. And it seems pretty clear to me from reading the account that this possibility is not remote. In fact, it sounds as though it's almost inevitable.

If that happens. . . but let's not go there yet. Let's just talk about something I think about quite a bit, which is how us foolish humans ought to think about events that have immense consequences, but which we believe to be of low probability. While there are actually a few reasons why I have been against offshore oil exploration for quite a while now, having to do with taking a long view of a finite petroleum resource and a finite atmosphere, most people who have opposed offshore drilling were mostly worried about an event such as this -- an environmental catastrophe. And of course the experts all said pish tosh, we know what we're doing, it's perfectly safe.

Politicians don't have any expertise about these matters, but they do know that the world's largest multinational corporations, with tens of billions of dollars to spend on PR, advertising, and campaign contributions, are in favor of drilling; and so are a lot of people in coastal states who want jobs, and legislators who want tax revenue with low political cost. Opponents don't have anything to offer but basically unsupportable fears. Except it turns out they were right.

There might be a few other questions we should rethink.

Now pay attention!

Because you're going to have to keep a couple of ideas straight -- concepts that are kind of similar but aren't the same. This part tends to confuse people so you've been warned.

So I'll start with what I hope is a fairly simple idea that will be a half step toward the slightly harder one. If we don't know anything about a population, and then we take a sample from it, the sample is our best estimate of what the population is like. We know the mean and standard deviation of the sample, but we don't know for sure how similar that is to the population. But suppose we do know the distribution of a population, and we want to know if a sample somebody gave us is likely to have come from that population or not.

Suppose we had the medical records of of a whole lot of children, 1,795 of whom had asthma, and .55 were male and .45 were female. We happen to know that .51 of all children are male, and .49 of all children are female. (It’s true! There are slightly more boys born than girls. However, the men start to die off so eventually there are more women than men.) So the question is, does this mean that boys are more likely to have asthma than girls? (These numbers are just made up by the way. I probably shouldn't have used a real disease.)

What we need to do in this case is compute a number called "Z". I don't know why they call it Z, they just do. This is the distance of an observation from the mean, measured in standard deviations.

In this case, the standard error for a sample of 1,795 children of whom .51 are male is approximately .012. (Actually it's 0.011799154872714672808505450202002, but close enough.) How do I know that? Because the standard deviation of a binomial distribution, as you may recall from last time, is the square root of P*Q/n. P is .51, and Q is .49. n is 1,795. Get out your calculator and try it! The difference between .55 and .51 is .04, which is more than 3 times .012. In other words, the observed outcome is more than 3 standard deviations from the expected outcome. The probability of this happening is so small – less than .001 – that it is extremely unlikely that boys have the same risk of asthma as girls. We say in this case that Z= 3.3333 etc. Z is also called the standard normal deviate.

Yup, it's normal and it's standard and it's deviate, all at the same time. What's the world coming to?

But notice something else, which is also very important - the difference is not very large. It's highly unlikely to arise just by chance, because we have a large sample. With big samples, you can detect small differences. But that doesn't mean they necessarily matter very much.

Anyway, hang onto that idea -- how many standard deviations are you from the mean? -- because it is very useful.

If fail to plan . . .

I don't know why I don't write about these issues more -- I actually have a master's degree in urban and environmental policy, I am involved in a study of near highway pollution, and I am an activist on the issue of healthy communities and environmental justice. Most of the e-mails I get from publicists go straight to the trash, but for some reason a flack sent me a link to this article by Jonathan Lerner which I recommend. (I don't see who's making a profit from this so that's why it's a bit unusual.)

The fact is our civilization is not sustainable on the premise that we need to surround ourselves with two tons of steel and propel it down a highway at 70 miles per hour using fossil fuel for two hours a day to get to work and back, and many more hours each week to buy groceries, take Buffy to her flute lessons, and oh yeah, get the car repaired. We can't keep turning farmland and woodland into suburban sprawl to give more and more people a chance to live that way. This entire way of life is doomed -- and maybe a lot sooner than most people think. (The preceding link is from the Apocalypse Now party, but if Richard Branson is right and peak oil is now, well ... tell me why they are wrong. And the well blowout in the Gulf of Mexico is going to throw a big spanner into the plans to stave it off by drilling in the ocean. Just sayin'.)

Most people are just going to have to live in urban or urban-style communities that are dense enough to support walkable shopping and have mass transit connections to the rest of the world. Those communities can work -- I live in Jamaica Plain which is jes' fine, let me tell you, with two of the world's greatest urban parks, great shops and restaurants, safe streets and a strong, welcoming and diverse community. But plenty of our urban areas are social and physical wastelands where it is not good for people to live. Transportation is the foundation on which successful 21st century communities can be built. Not just the Supertrains! between cities that Atrios is so enthusiastic about -- although those are good -- but well designed, accessible local webs of light rail and LNG-powered buses to fill in the places between, along with sidewalks and bike paths.

Tie the people to great grocery stores, farmers' markets, jobs, educational opportunities, recreational opportunities, and get them out of their cars, and you can have a healthy community anywhere. If we're going to have a future, that's it.

Don Berwick

It has received very little attention in the mass media, but a lot of attention in the circles where I travel, that the president has nominated Donald M. Berwick to direct the Centers for Medicare and Medicaid Services. Apparently Dr. Berwick engaged in lengthy discussions, possibly negotiations, with the administration before agreeing to be nominated. I was initially surprised because I didn't think the CMS director had much of a policy portfolio, that it was essentially an administrative job. That Berwick accepted is a signal that we will see substantial innovation and experimentation in health care financing and organization in Medicare, which the federal government controls directly. CMS can encourage and support the states in redesigning Medicaid, though it has less direct influence.

Berwick, who now runs the Institute for Healthcare Improvement, located in the People's Republic of Cambridge, Massachusetts, is well known as a health care (I still prefer it as two words) visionary and advocate for patients. You can learn his basic schtick here, it's called The Triple Aim. If you've been reading Stayin' Alive, you already know that the goal of universal coverage is inseparable from cost containment. But rather than defining universal coverage per se as part of the triple aim, Berwick allows it to fall out as a consequent, if that. The triple aim is better experience of health care, improving the health of populations, and reducing per capita costs. If universal coverage serves those ends, so be it, but he doesn't try to prove that theorem.

He does argue, however, that the triple aim requires eliminating the fragmentation of our present system, and the misalignment between the incentives of individual institutions and his social goals. This means large scale integration through some sort of authority or another -- a national health service or single payer works, but he's willing to tolerate smaller scale systems -- that is dedicated to these goals, rather than, say, the enrichment of executives or the prestige and market share of an academic medical center. He pretty much concludes that global budgets are necessary, within which resources can be allocated to optimize the first two goals; and that the goal of population health requires redirecting funds away from medical services to more primary public health.

All of this is true, but it means goring many a large and powerful ox. While I believe it is very much pro-liberty, it doesn't sound very libertarian. In fact, some will detect a pinkish hue over the whole thing. I am curious as to how the Confederate Party will approach his confirmation hearing. I hope we'll find out soon.

Isn't a "central limit" a contradiction in terms?

Maybe so but the Central Limit Theorem is the key to Gaussian statistics. The theorem states:

The distribution of sample means from any population, regardless of the population distribution, will be normal, with a mean equal to the population mean.

By a "sample," we mean cases from the population, be it individual people or whatever else the population consists of, chosen at random, or at least in a way that is unrelated to the probability of having the characteristic we are interested in. That is what we call an "unbiased" sample. Bias in this context doesn't have anything to do with prejudice, it's just question of whether our probabilities are somehow distorted.

For this to be true, the sample also has to be large enough. It doesn't really start to work until you have samples of at least 30 and you really need bigger samples than that to do anything useful.

So I supposed you would like me to explain the theorem. Well okay then. One way of looking at it is to go back to our coin flips. The underlying distribution of coin flips in the world is not a normal distribution. It looks like this:

For obvious reasons, we call that a "rectangular" distribution. Half the flips are heads, and half the flips are tails. What our normal curve was actually showing us was the distribution of sample means from many samples of arbitrarily large size. The most likely result is half tails and half heads, with results farther and farther from the population mean increasingly unlikely.

In my magic bag, I have an infinite number of green and blue marbles; 80% are green. Obviously, this is not a normal distribution, but, if I take a million samples . . .















I get this distribution. Yep, it's a normal distribution with a mean of .8, the population mean.

The standard deviation of this curve is called the “standard error of the proportion.” The bigger the sample, the smaller it is. We could figure it out by using the binomial formula, computing the proportion of all possible values of the sample mean, and plugging them into the formula for the standard deviation . . . But there is a much simpler way. The formula for the Standard Error of the Proportion is:



As you may recall, p is the probability of one outcome, q the probability of the other, and what you don't know yet is that n is the sample size. So, bigger samples mean smaller standard errors, which is another way of saying that you can have more confidence that your sample mean is close to the population mean. Big samples are good but they cost more. So they're a luxury we don't always have.

Soon, I will march on with this. It will get more interesting, I hope, when we actually put it to use for a good cause.

Denialism isn't just annoying

or stupid, or perverse. It's downright ugly and evil. I am proximally inspired to these sentiments by this article into today's All the News that Fits, about the efforts of the South African government to ramp up HIV prevention and treatment. The best effort is that 365,000 South Africans died unnecessarily while former president Thabo Mbeki was under the thrall of AIDS denialist Peter Duesberg and other cranks. The harm done to innocent children and families by the deluded cult that blames vaccines for autism belongs to the scientific fraudster Andrew Wakefield and narcissistic upper class twit Robert Kennedy Jr. The catastrophe that looms for the entire planet from anthropogenic climate change might have been averted but for Rush Limbaugh and another crank scientist, Dixie Lee Ray. (Most people don't know about Vulgar Pigboy's seminal role in AGW denialism -- it's well worth publicizing.)

What RFK Jr. and Rush Limbaugh have in common is that they both saw a chance to promote themselves, build a following, and ultimately make money by pretending to be lone crusaders against a deceitful scientific establishment. These "science as conspiracy" theories are nothing but repulsive slurs against people who work hard, generally for quite modest incomes, and who share a culture of dedication to truth. Really. The whole stance is nothing but a scam designed to separate people from their senses as well as their money. Unfortunately, while Kennedy and Limbaugh are just charlatans, they have managed to catch a lot of innocent victims in their net of true belief, and we all have to suffer the consequences. Meanwhile, they go on being rich and influential.

Feh.

Gaussing onward . . .

Ana writes, "One of the BIG problems of statistics is that analysts use and abuse measures and stats developed for the ‘normal’ curve, applying them when not appropriate." I'm going to get to critical perspectives on the use and abuse of statistics, which I hope will be more interesting, but I hope people will be patient while I continue my march through the basics.

We derived the normal curve from a binomial distribution, but of course we aren't only interested in binomials -- variables that can take only two values, such as coin flips or whether or not somebody tests positive for a disease, for example. We're also interested in the distribution of variables that can have many values, or even continuous values such as height, weight and blood pressure. Our binomial distribution started out as coin flips, but what we actually wound up charting was a continuous variable -- the proportion of heads in an arbitrarily large series of coin flips.

In the Real World, it's quite rare to find a distribution that is perfectly normal like our coin flip example. Some real world distributions look sorta kinda normal, and many do not. (The reason that the distribution of IQs appears very close to normal is because they intentionally design and score the test so that it will be, BTW. Many of the normal looking curves you will see are really artificial.) Nevertheless, the normal distribution can be useful in thinking about real world populations that aren't normally distributed. We're heading toward the explanation for that but first we need a few more basic ideas.

The Standard Deviation is a measure of dispersion – the tendency of values to cluster near the mean, or be more spread out. You can calculate it regardless of whether a distribution is normal, but for really weirdly skewed distributions it might be misleading, or at least not very informative. Here's how you compute it:

• Calculate the mean: Add up all of the values in your population or sample, and then divide by the number of cases. The "mean" is a synonym for what we ordinarily call the average. We can abbreviate it "M". (Sometimes the Greek letter μ (mu) is used, but why be pretentious?)



• For each case in your sample, subtract M from its value -- X. The result could be a negative or a positive number, obviously, depending on whether X is larger or smaller than M. Then square the result, i.e. calculate (X-M) * (X-M). This way, you'll convert all the numbers into positive numbers.



• Add up all the resulting numbers, then divide by the total number of cases, N. You have now computed a new kind of average, the average of the squared distances from the mean of all your cases.



• This is called the variance. Usually, people then take the square root of the whole thing, to make up for the fact that the original values were all squared. The result is called the Standard Deviation, abbreviated sd

We can write the formula for this easily enough. If formulas make you sick, just avert your eyes.



The thing that looks like a big "E" is actually the Greek letter sigma, which is like our "S". It stands for sum. That means you add up the values of each individual case, just like I said. The rest of it is also just like I said -- what you are adding up is X-M squared, then dividing by N, then taking the square root of the whole thing. Tah Dah! Now you can read a fancy mathematical formula.

MAJOR IMPORTANT FACT: In a normal distribution, about two thirds (.6826) of all values are within one sd of the mean; about 95% (.9545) are within two sd of the mean; more than 99% (.9973) are within three sd of the mean. This is true of all normal distributions.

Now we are getting close to the magical key. Next time!

More virtuosic politics from the Democratic Party

As I assume you have noticed, a major bloc of opposition to the recent health care reform legislation consists of Medicare beneficiaries who have been convinced by Faux News and Rush Limbaugh that it will cut their benefits and shove them out to die on an ice floe. Of course, as always when referring to Faux News and the Vulgar Pigboy, the opposite is true.

Right now, Medicare beneficiaries get a single "welcome" checkup when they first sign up. After that, Medicare pays for "treatment of disease" plus limited, specific preventive services such as mammograms. But under the reform legislation, they'll now get an annual visit, absolutely free, including 100% of the screening tests, vaccinations and every other service recommended by the U.S. Preventive Services Task Force. They'll also get a personal prevention plan, and a personal health risk assessment. Once the seniors realize the legislation didn't cut their benefits, but actually increased them, and in a really great way, they'll come back to the Democratic Party in droves and the threat of losing control of the House in November will be over.

Oh wait. The new benefits take effect on January 1, 2011. #%$!#%%^ brilliant, Harry.

So what exactly is normal?

If you're a curve, that is. The world famous normal curve is:

• A continuous, symmetrical curve w/ both tails extending to infinity.


• The mean, median and mode -- which I will now explain are the same.


• Described by two parameters (i.e., numbers that can vary): arithmetic mean, and standard deviation -- which I will also explain.

As you may recall, we got our normal curve from a binomial distribution -- heads and tails. So what does it mean, to talk about the mean? (Or to miss New Orleans?) Easy. Just assign a numerical value to heads and tails. Usually in the case of a binomial people use 1 and 0. So the value of the most common result -- half heads and half tails -- is going to be .5 = 1/2. It should be easy to see why. Let's say we flipped our coin 100 times. The most frequent result, which is called the mode or modal value, is going to be 50 heads and 50 tails. 50 * 1 = 50, and 50 * 0 = 0, and 50 = 0 = 50. So the total numerical score for that result is 50. The arithmetic mean, or average, is the total numerical value divided by the number of cases, i.e. coin flips. 50/100 = .5. You can see that all the values on each side of the curve have to average out to .5 as well because they cancel each other out. 49 heads and 51 tails cancels out 51 heads and 49 tails, and so on. Therefore .5 is both the mean and the mode of our curve.

The median is the number in the middle: half the values are lower, and half the values are higher. Basically, you can see that the median is also .5 for the same reason the mean is .5: the curve is symmetrical. Half the values are to the left, and lower than .5; and half the values are to the right, and higher than .5. So there you go. To review:

• The mean is the sum of all values, divided by N (N is the size of the population).


• The median is that value which divides the population in half – 50% are greater, 50% are less.


• The mode is the most common value.

A normal curve doesn't have to describe a binomial distribution. It can also describe a distribution of some continuous value with a real numerical meaning, such as people's height or weight. Such real world values wouldn't often be exactly normal but let's pretend they are for the sake of argument. (We'll get to why this is worth doing later.) Then the mean, median and mode, while still the same as each other, would be whatever value happened to pertain in reality, such as 5'7" and 165 pounds.

Enough of this for now. Any questions will gladly be accepted.

it's a big planet . . .

but subjectively shrinking. Many people have wondered if all the recent news about earthquakes, and now a volcano, means that something unusual is happening to the thin skin of solid rock we all float around on. (I have read that if the earth were shrunk to the size of a basketball, the solid crust would be the thickness of a postage stamp.) According to the U.S. Geological Survey, the frequency of strong earthquakes around the world has not been increasing.

However, the human population has gotten a lot bigger, and a lot more urban, in more places, in a very short time. Remember, earthquakes don't kill people: buildings do, when they shake and fall down. The result is that a lot of cities are now more or less in the situation of Port au Prince, including Tehran, as Robin Pomeroy reports for Reuters. You may have read about the Friday prayer leader last week who blamed earthquakes on scantily clad women, but Mahmoud Ahmedinejad, whatever craziness may pertain to him, knows better. Quoting Pomeroy's story,

Like the people of San Francisco, Tehranis know their sprawling metropolis is due for a massive earthquake. In Iran, where building standards have not advanced as quickly as the population, some estimate millions could be killed or maimed. In an Islamic society where disasters are often seen as acts of God, Ahmadinejad told housing officials they could no longer rely on the power of prayer to save Tehran from annihilation.

"Tehran has 13 million inhabitants. If an incident happens, how can we manage it? Therefore, Tehran should be evacuated," said Ahmadinejad, a former mayor of the city, announcing financial aid for people who move to towns with a population of less than 25,000. "At least 5 million people should leave Tehran," he said.

Obviously this is not going to happen, but if it were somehow enforced, it would be an economic, social and ecological catastrophe in itself. The fact of the matter is that we humans have gotten ourselves stuck, badly jammed up in a lot of ways all at once. There are just too damned many of us and we can't make it work.

That's a cold fact.

Oh yeah, there's also this: The combined global land and ocean surface temperature anomaly for March 2010 was 0.77°C (1.39°F) above the 20th century average, resulting in the warmest March since records began in 1880. Of course, this is all a fraud -- it's part of Obama's secret plot for Communist world domination.

At least something is normal around here

Sometimes people do this bit using a magic bag containing an infinite number of red and green marbles, but I'm just going to flip some coins. We'll let a magic guy do it to make it interesting anyway.


Heads are H, and tails are T. That's easy. And we know that if you flip a coin once, the probability is 50% heads, and 50% tails, so P(H)=.5 and P(T)=.5

So what if you flip a coin twice? What's the probability of getting two heads? (Jeopardy! music plays for 15 seconds.)

Okay, as you remember from last time, the two coin flips are independent events -- the first one has no influence over the second one. (Believing otherwise is called the "gambler's fallacy" -- you start to think your number is "due" because it hasn't hit for a while.) So you get the probability of two independent events both occurring by multiplying the individual probabilities together. The probability of heads the first time is .5, and the probability of heads the second time is .5:

P(H H)= .5 * .5 = .25

It will happen 1/4, 25%., .25 of the time. Two tails in a row will also happen 1/4 of the time. As you may also remember, the probability of either one of two mutually exclusive events happening is the sum of the probability of each.

P (H H) + P (T T) = .25 + .25 = .5

As we also know, something has to happen, which means that all the probabilities have to add up to one, which means that what is left -- 1 heads and 1 tails -- must have a probability of .5. That makes it, obviously, twice as likely as TT or HH. Why is that? Because there are two ways it can happen: tails then heads; and heads then tails.

P (T H) = .25 and P (H T) = .25, so the probability of either one happening is .25 + .25 = .5. I'll even show you a graphical representation:





Okay, let's keep going. Let's do four coin flips.



You should be able to figure out for yourself how to compute the probability of each possible combination. There is a general formula for this, which is called the Binomial Distribution -- binomial meaning "two names," or "two things." I will show it to you just for the halibut, but it isn't necessary and you don't need to know it. (I can't show it in this post because I haven't been able to get the terms in the formula to line up correctly. Blogger is not very math friendly.) What you do need to know is the limit of the binomial distribution, i.e., what it starts to look like as the number of coin flips gets very large. Here it is:



I put some real data under the curve just to show you that in the real world, it never comes out exactly right -- that's the way it is with probability. But do you recognize that curve? Yes, it's the world famous Normal Curve, the Bell Curve. It is the basis for an entire approach to statistics. It was discovered by a guy named Gauss, so when we do statistics based on the normal curve, we call it "Gaussian statistics." That is the adventure on which we will now embark. Meanwhile, here are some equations for you to ignore.

This is the shortest way to represent the computations concerning 3 coin flips. "P" is the probability of heads, and "Q" is the probability of tails. They happen to be the same in this case.

3T=TTT=.5*.5*.5=Q^3=.5^3=.125
2T, 1H=HTT, THT, TTH=3pq^2=3(.5)^3=.375
1T, 2H=HHT, HTH, THH=3p^2q=3(.5)^3=.375
3H=HHH=P^3=.5^3=.125
.125+.375+.375+.125=1.

I honestly thought this was satire . . .

Apparently not. Sample:

I don’t know if I should Buenos Aires or Bonjour, or... this is such a melting pot. This is so beautiful. I love this diversity. Yeah. There were a whole bunch of guys named Tony in the photo line, I know that. And in the introduction too, in the instructions to you all, I got a kick out of the instruction “No heckling.” I am so used to the heckling, it’s okay! We’re used to it. They just hit you into the boards and maybe get called for a penalty or whatever, but we can handle that too. . . .
I’m wanting to, though, kind of shift away from the political. I’m just getting off the trough from doing a lot of Tea Parties across the US, man those are a blast. [applause] They’re rowdy and they’re wild and it’s just another melting pot, there’s just diversity there and all walks of life and all forms of partisanship and non partisanship just wanting good things to happen in this part of the world. It’s been a blast. The shift from the political, so now that I have that shift from the political but still have that desire to talk about the economy and talk about energy and resources and national security and all those things. I was telling Todd, okay, this is like [inaudible] on the vice presidential campaign trail, where you never really knew what you were getting into when you get into that line before you were interviewed. Obviously, sometimes I never knew what I was getting into in an interview. Obviously!

And on and on and on it goes unto ever more profound depths of stupidity until the brain turns to lemon jello. Who needs The Onion any more?

Okay, let's gamble

I was considering another diversion into philosophyland, with a post yammering about the limitations of data and reification and all that jive, but I just know y'all are out there salivating to get to the damn probability and statistics already. So okay.

As I have already said, the very idea of probability is somewhat philosophically intractable but we just have to live with that. In public health, we live in a probabilistic world as a function of the limitations of our knowledge. So there you are.

Probabilities are closely related to the idea of rates. For example, the probability of death (in a given population) during a given year is the same as the death rate (i.e., incidence of death). Of course we didn't know the death rate until after the year was over; but we often use the idea of probability in connection with a prediction about the future. That means we have to estimate the probability of dying based on prior experience, if we want to look forward. We'll be talking a lot about estimates and errors as we go on.

Probabilities can be expressed as fractions where the denominator is the total population, and the numerator the number of events. That's just a way of repeating myself -- they are similar to incidence rates. However, because incidence rates can include events that happen two or more times to the same person, the probability of the event befalling a single person can be somewhat less than the incidence rate for the event. With death, obviously, this doesn’t matter, but it could with say, ear infections. We'll see how it gets tricky sometimes, in this and other ways.

Probabilities can range from zero (no way, never happen, not a chance) to 1 (happens every time, mortal lock, death and taxes). For you gamblers out there, keep in mind -- and this is important -- that probabilities expressed this way are different from odds. A probability of .75 -- which you can also call 75% -- represents odds of 3 to 1. That's easy to figure out. Let's say there's a 25% chance of winning on a scratch ticket. That means there's a 75% chance of losing, because the percentages have to add up to 100. Something has to happen! 75/25=3. You're three times as likely to lose as you are to win, that's 3 to 1. A probability of .5, obviously, represents 1 to 1 odds. And so on. We won't go back to odds until much later, because for most of what we're going to do, it's much easier to work with probabilities.

You can do arithmetic with probabilities to figure out more complicated situations. Suppose you have a situation in which there are 3 possible outcomes, and they are mutually exclusive. That means only one of them can happen. For example, you apply to college. You might get admitted with a scholarship, you might get admitted without a scholarship, or you might get rejected. If you want to know your chance of getting admitted, with or without a scholarship, just add together the probabilities of each outcome. Again, here is a simple formula:

P(X) + P(Y)=P(X or Y)

If X means getting a scholarship, and Y means getting in without a scholarship, there you have it. As you can also figure out, if Z means not getting in at all, P(Z)=1 - P(X or Y), because one of those three outcomes must happen and the sum of all three probabilities must be 1. But remember, this only works if the events are mutually exclusive!

What if they aren't? What if we're worried about two things happening, such as you get into Harvard and you get into Yale? (No endorsement implied.) In this situation, you have to assume that the events are independent, which means the admissions offices aren't talking to each other and saying we'll take Joe and you can have Sally, which would totally mess it up. In this case, you multiply the two probabilities together:

P(X) * P(Y)= P(X andY)

We like to use "*" to mean multiplication, instead of "X" as you might have learned in school, because, obviously, we also use "X" a lot to mean "whatever," so we try to minimize confusion. We don't always succeed, as you will see. Unfortunately there are symbols that get used for two or more purposes, but I'll try to keep it to a minimum.

So, if your chance of getting into Harvard is 20%, and your chance of getting into Yale is 50% because your father went there, your chance of getting into both is

.2 * .5 = .1

That's 10%. If you want to make sure to get into Yale, ask your father to give them a million bucks.

Next time, I'll get to more complicated situations. If anything is unclear, or you have suggestions for making it more clear, please let me know.

Okay, get out your notebooks

First we need a bit of vocabulary -- what we call some of the kinds of rates or proportions that we commonly talk about in public health. Last time, I said that "rate" and "proportion" basically mean the same thing, and they do; but we tend to use them in slightly different contexts. We can say that:

A “rate” means the proportion of a population which either 1) has some characteristic at a given time, or 2) to which some event occurs during a given period.

In public health, a “population” normally means some defined group of humans – but in statistics, a “population” can consist of anything, and sometimes our units aren’t people – they might be hospitals, or procedures, or laboratory rats, for example.

To compute a proportion, divide the number of “cases” (people with the characteristic) by the total population of interest. If you have 18 people, and 3 of them have nose rings, the nose ring proportion is 3/18=.167=16.7%. As I said last time, all three ways of writing it mean exactly the same thing.*

There are two major kinds of rates:

PREVALENCE

and

INCIDENCE

Prevalence means the proportion of people who have the characteristic at a specific point in time. So, in our group of 18 people, the prevalence of nose rings is .167, or one out of six if you prefer.

The incidence is the proportion of people who acquired the characteristic within a specified period of time. Usually, it's calculated for a calendar year, but it certainly doesn't have to be! If it turns out that 2 of our 18 people already had nose rings as of January 1, 2009; and 1 of them got a nose ring for the first time during 2009, the incidence of nose rings for 2009 is 1/18=.055555555555555555555 and on to infinity which we would probably round off as .056 or 5.6%. Next time, I'll use 20 people with a prevalence of 5 nose rings and an incidence of 2 so you'll get nice simple fractions and snappy finite decimals. But the idea is the same. And now I'm going to write a dreaded formula!

# of people who have get the thing / total number of people in the group = incidence

When we write formulas we like to save space by using short symbols in place of long phrases so we might write this as:

R/N=I

where R=number of people with who get nose rings, N = the total population we're studying, and I=incidence. There you go. Easy as pie, I hope. (Thanks to anonymous for pointing out my careless blunder.)




*3/18 happens to be the same as 1/6, and one sixth can't actually be written as a decimal because it would be infinite, so 1.67 or 67% is only a close approximation. Not worth worrying about.

Probability and Statistics 101: Pre-lesson lesson

This isn't even the first installment in the curriculum. It's something I'm going to hand out to read before the first class. Many of you will find it completely unnecessary. Some may feel insulted by it, but please don't be: lots of people out there actually, truly need to read this. If you feel you don't, read it anyway as a lesson in what lots of people actually need to read. If you are matho-phobic or hate math or think you just can't possibly understand it, this will help. I hope.

In public health science, and for that matter in most fields of scientific inquiry, we frequently talk about proportions, rates or percentages.

These all mean exactly the same thing!

When people speak of proportions, there is a tendency to represent them as decimals, like this: .9 (pronounced "point nine.") When people speak of rates, they are probably a bit more likely to represent them as fractions, like this: 9/10 (pronounced "nine tenths," or "nine out of ten.") To get the percentage, you just multiply by 100: 90% (pronounced "ninety percent.")

They all mean exactly the same thing! There is no difference at all. I would not be doing any violence to the English language or to mathematics to speak of a proportion of .9, a proportion of 9/10, or a proportion of 90%. There is no difference at all, not even a little tiny difference. None.

For some reason, when I show people a percentage -- 90%, 87%, whatever -- they usually say that's fine, they get it. But when I show them .9, or .87, the very same people often say they don't get it, they can't possibly understand it, it's math and it's just too confusing. That is like saying you understand me perfectly well when I say a Ford is a car, but it's impossible to understand or follow what I'm saying if I say a Ford is an automobile.

There's nothing to understand. There's nothing to "get." There is no deep meaning or special attribute of the symbol .9 that makes it any different from the symbol 90%. If you are at all confused, just keep in mind that the space just to the right of the decimal point is 10ths. .9 = 9 tenths = 9/10 = 90/100 = 90%. The next space is hundredths. .97 = 9 tenths + 7 100ths = 90 100ths + 7 100ths = 97 100ths = 97%. That's all there is to it. If you still think you don't get it, it's because you are convinced there must be more to it than that, that there must be something I'm not telling you, some deeper secret. There isn't.

Probabilities: Probabilities are a lot like proportions. If 90% or .9 or 9/10ths of the people in the room are right handed, and I pick somebody at random, the probability that person will be right handed is 90%, .9, or 9/10. We will often say "nine out of ten" for a probability but it means exactly the same thing to say .9 or 9/10 or 90%. It obviously saves space to write .9 and it makes calculations easier, so that's what I will do most of the time. If this bothers you, you can think "ninety percent" every time you see .9, and it will be just fine.

Formulas and equations: Mathematical formulas and equations give lots of people the heebie jeebies. As soon as they see one, they stop reading and they stop thinking. One reason for this, I think, is the convention, when writing about mathematical subjects, of first writing down the equation, and then telling the reader what the symbols mean. The result is that you'll see a bit fat equation sitting in the middle of the page, and you have no idea what it means, so you feel stupid. But remember, at this point, even Einstein has no idea what it means. You have to read the next sentence, where the person tells you what it means, and look up at the equation and back and forth between the equation and the sentence underneath it to get what all the symbols mean. For example, suppose I write this:

P(RH) = .9

You're baffled and you're mad at me because you don't know what that means. But all you had to do was wait for me to tell you. "P" means the probability of whatever term is enclosed in the parentheses right after it. "RH" means being right-handed. So P(RH)=.9 means "The probability of being right-handed is point nine" (or ninety percent if you prefer).

Why didn't I just say so in the first place? Because I want to use this idea more generally: I want to use this structure in arguments and equations where I'm not just talking about being right-handed and .9, but whatever phenomenon and whatever probability happens to apply. The equation is a handy structure that doesn't take a lot of space to write down and captures the ideas I want to talk about, not just the specific case.

So what I am going to do is first, discuss concepts qualitatively, in prose, with as little use of mathematical symbols as I can get away with. I'll only use mathematical symbols when I'm pretty sure it will make it easier, but we're going to be talking about mathematics so it will be impossible to avoid referring to mathematical concepts such as one thing being equal to another.

Then I'll use one or more specific examples and maybe do some arithmetic with them to show you how the idea works.

Then, and only then, I will present general formulas that embody the idea. If you don't like that part, you can skip it. It doesn't need to turn you off to the rest of it.

Any questions? Are we ready to begin?

Deep Thought

No, the title does not signal an inane quip, I really do want to start my discussion of probability and statistics with some musings on the philosophy of science. I'll try to keep it all as clear and bullshit free as possible. Let me know if I fail.

For a long time, as people did what we today recognize as science or something like it, philosophers didn't fire up any neurons worrying about what they were doing. Archimedes, Eratosthenes, Galileo, Newton, Herschel, Darwin, pretty much just did their thing, while philosophers worried about the moral virtues and the noumenon and what not. By the 20th Century, however, distinguishing between scientific and unscientific beliefs and methods of inquiry had obviously gotten to be important. Philosophers came up with an idea called positivism, which in a nutshell held that a statement is meaningful only if it can be verified, and that it's meaning is equivalent to the means by which it can be verified.

This would seem to imply that there is some stuff that we can totally know -- assertions that can be fully established as true -- and that any other kind of assertion is just gobbledygook of no interest to those who wish to live by the light of reason. However, it soon became clear that this doesn't really work. The kinds of general statements about the world that constitute interesting science are difficult or maybe even impossible to prove in a formal sense. Newton's theory of gravitation, as a matter of fact, was not exactly true. It was true as far as anyone could tell with the accuracy of measurement and the conditions available to Newton's scrutiny but then along came Einstein and with better instruments we found that his theory is more accurate.

But does that mean that Einstein's theory is proved, or that even in principle we will ever have the means to demonstrate as a matter of formal certainty that it isn't wrong somehow, somewhere? Not at all. In fact physicists have been working very hard for a long time to do Einstein one better, and develop a theory that unifies gravity with the other cosmic forces. So they certainly don't think Einstein's theory is a settled matter. But does that mean it is not a scientific theory? Obviously not.

So Karl Popper proposed that at the very least, even if we couldn't really prove anything, a scientific proposition had to be falsifiable. There had to be tests available to show that it was not so. But that turned out to be a bit of an oversimplification as well. One spanner was thrown into the works by the Austrian mathematician Kurt Goedel, who proved that in any logical system at least as complicated as number theory -- basically everything you can say about numbers using arithmetic -- there are true propositions that can never be proved, which implies that their negations -- false propositions -- can never be falsified. To be sure, he's talking about abstract systems of deduction, not empirical investigation per se, but scientists use mathematical reasoning all the time to describe the world, plugging empirical entities into formal systems of logic. So Goedel's theorem (called the Incompleteness of First Order Arithmetic) was a bit disturbing.

Another problem was that physicists discovered that the world at very small scales -- where the electrons and protons and photons disport themselves -- is not deterministic, but can only be described in terms of probabilities. By the time you pile up enough atoms to make an object apprehensible to the senses, those probabilities average out so that it is extremely unlikely your coffee cup will suddenly jump off the table. But it is only extremely unlikely, it is not officially impossible.

Since the very tiny quantum world and the larger scale world are linked, this means that the universe as a whole does not have the deterministic quality that animated positivism and related philosophical schools. People used to think that if you could know the position and momentum of every particle in the universe, and you had a sufficiently powerful computer, you could know the future. But it turns out that you can't simultaneously know the position and momentum of even a single particle, nor can you know when it might decay or change its energy state. You can only make a probabilistic statement about such events. If you wanted to assure the unpredictability of the future, you could rig up a detector that would respond to a radioactive decay event by tripping a switch that would do something really, really big, like blow up a dam; and you could have it wait for an event with a 50% probability of happening at any time within 100 years. Hmmm.

Absent such exotic experiments, the macro world is not so uncertain. If you knew all of the forces acting on a roulette wheel and the ball -- the precise coefficient of friction of the bearings, if you could map out the turbulence of the air around it and precisely how it affects the motion of the ball and the slowing of the wheel, the coefficient of restitution of the ball vis a vis whatever parts of the wheel it might touch, and so on, and the exact vector of the ball and speed of the wheel at the beginning of the spin, in principle you could predict what number will come up. But you can't possibly know all that. The tiniest variation in initial conditions, below what you can possibly measure, will produce a completely different result. Like the weather, it is a so-called chaotic system.

What does it mean when the guy on TV with the perfect teeth and the hairpiece molded from a single piece of plastic says there is a 50% chance of rain tomorrow? How can we verify his statement? Either it rains tomorrow or it doesn't; either way, he was right. So is the statement meaningless? No. What it means is that if he makes that prediction 100 times, we should expect it to rain about 50 times. It might not rain exactly 50 times -- maybe it rains 52 times, or 47 times, and we'll let him get away with it. But if it rains 80 times, we'll decide he's a lousy forecaster.

Similarly, assuming there are 38 slots on the roulette wheel, we'll call it honest if, in a thousand spins, each number comes up about 1/38 of the time. Unfortunately, 1,000 is not divisible by 38, so each number cannot possibly come up the same number of times. We'd like it if most hit 26 times and some hit 27, but that's actually very unlikely to happen. You'll most likely have some 29s or 30s, and some 23s and 24s in there, and maybe even bigger deviations. That's why once in a while, somebody walks away from the table a winner.

It turns out that science depends very heavily on probabilities. When we observe variations in populations -- whether they be of people or of stars -- we can only estimate the probability that the same patterns pertain to the people or stars we haven't observed. We can't really prove anything, or necessarily know how to falsify most of the interesting scientific theories we might come up with. That doesn't mean we might not end up falsifying any of them eventually, it just means we don't have to know how to do it when we first assert them in order for the assertion to be meaningful.

That's more than enough for now.

My mission . . .

should I decide to accept it -- actually I have decided to accept it, but I don't yet know if it's Mission Impossible -- is to develop a curriculum in probability and statistics, not for dummies, but for real people who are not math majors, or not necessarily any kind of majors, including people who have not attended college.

Naturally, that means I'll be trying out some stuff here. So you'll have to get used to it. I've thrown up posts from time to time on the subject, but now I'll have to be more systematic and, I also realize, more philosophical. Probabilistic reasoning is pervasive in science. The essentially probabilistic nature of most scientific propositions has essentially overturned the positivist theories of knowledge which once were considered the basic philosophy of science, and has also made the falsification criterion for scientific propositions obsolete, at least in its simple form.

Oh wait, that must seem like word salad to most people. I have to explain really basic stuff. Why do we think we know what we think we know? Why is it not a contradiction to say that science is a path to truth but most scientific findings are false? True fact! But not a defense of Paul Feyerabend, or post-modernism, or creationism or pseudoscience. What is taught in science classes, and the big important stuff that scientists believe, is true, in the sense of being as true as we can make it although probably not exactly true; but new scientific findings are out on the edge of what we don't know, not in the comfortable middle of what we do know. And in order to understand that, you have to understand probability and statistics, including that pesky Bayes fellow.

So I'll do my best.

The Chair recognizes the Senator from Wellpoint

Since I've already set you up with one downhead riff for today, here's another from three public health lawyers in the new NEJM. You probably haven't thought much about that pesky Citizens United Supreme Court ruling since it came down in January but that's because you're living in a fool's paradise. That's the one about how corporations are people and therefore have the same First Amendment rights as you do to spend a billion dollars on an election campaign. Hey, what are you complaining about? You can spend a billion dollars if you want to, kwitcherbitchin.

Anyhow, they point out that insurance companies can now spend unlimited bucks to run attack ads against candidates they don't like, in the month before the 2010 election. Of course they would never do such a thing, just to protect their pecuniary interests. I mean, come on, corporations have souls, they have moral scruples, they care what happens to you.

If you think about it, drug companies have certain ideas about the FDA, beverage companies have certain ideas about sugar taxes and what's in school cafeterias, Monsanto has certain ideas about what can and cannot be dumped into the river . . . and just like you, they can now spend billions of dollars to elect the candidates they like. Viz the previous post, consider the poor, beleaguered coal companies.

It's gonna be a wonderful second decade of the 21st Century, lemme tell yuh.

Shape of the earth . . .

Do views no longer differ? Here's a straight-up story from CBS News. March was the warmest on record, and "climate researchers have been reporting rising global temperatures for several years as a result of what is called the Greenhouse Effect, in which rising levels of carbon dioxide and others gases in the atmosphere trap heat instead of allowing it to escape out into space." No qualifiers, no obligatory second opinion from denialists, no bullshit.

On the other hand, it's a pretty low-key report. This is in fact the biggest story of, well, the millennium, and it only gets major attention from the corporate media when the well-funded denialist publicity machine dumps a new load of BS. So please, bring yourself down with this. Do it for the children.

Rationing

KG Smolderen and a multitude of the doctorly host report in the new JAMA that people who lack health insurance, or who are worried about out of pocket costs, delay longer before they show up at the hospital after a heart attack. Actually it's worse than that because they could only study the people who eventually made it there alive.

I would like to say, "Well duhhhh," but keep in mind that a favorite line of people who opposed the reform legislation was that we already do have universal health care when we really need it. Indeed, the greatest president of the 21st Century, the visionary whose portrait will one day be on the $3 bill, said "I mean, people have access to health care in America. After all, you just go to an emergency room."

Maybe so, but if they know it's going to cost them thousands, or tens of thousands of dollars, they are not inclined to do so. They'll hope it's just a bad case of indigestion. When people get there late, they are more likely to be rehospitalized, so it just ends up costing more in the end -- except of course for the people who have the good grace to just drop dead. Of course, the uninsured people are more likely to have a heart attack in the first place because their blood pressure and cholesterol are uncontrolled.

But we can't provide universal coverage because then we'll have rationing. Dumbasses.