Sins of Analytical Chemistry

  • : Assigning the return value of new by reference is deprecated in /home/pachydom/public_html/modules/modules/vbcore/vbcore.module on line 430.
  • strict warning: Only variables should be assigned by reference in /home/pachydom/public_html/modules/modules/vbcore/vbcore.module on line 213.
  • strict warning: Only variables should be assigned by reference in /home/pachydom/public_html/modules/modules/vbcore/vbcore.module on line 213.
  • strict warning: Only variables should be assigned by reference in /home/pachydom/public_html/modules/modules/vbcore/vbcore.module on line 213.
  • strict warning: Only variables should be assigned by reference in /home/pachydom/public_html/modules/modules/vbcore/vbcore.module on line 213.
  • strict warning: Only variables should be assigned by reference in /home/pachydom/public_html/modules/modules/vbcore/vbcore.module on line 213.
  • strict warning: Only variables should be assigned by reference in /home/pachydom/public_html/modules/modules/vbcore/vbcore.module on line 213.
  • strict warning: Only variables should be assigned by reference in /home/pachydom/public_html/modules/modules/vbcore/vbcore.module on line 213.
  • strict warning: Only variables should be assigned by reference in /home/pachydom/public_html/modules/modules/vbcore/vbcore.module on line 213.

Science, they say, is the truly noble pursuit. Men and women from all national, racial and cultural backgrounds come together to work tirelessly, dedicating their intellects and their training to the improvement of the human condition and the advancement of our knowledge. Regardless of the nature of scientists as individuals, science as a whole can only march towards greater truth and the betterment of the entire world.

Well… not quite.

Science, as an abstract concept, as a raw collection of knowledge and a drive to further understanding, is this noble pursuit described above. But a lot of science is focused on practical considerations, rather than abstract theorising. The less “pure” the science is, the more subject it is to the whims of politics. And the more I learn about analytical chemistry, the more disillusioned I become with it

Analytical chemistry is the science of investigating samples to determine what is present, and in what quantity. As such it is widely used in and out of the lab, from testing the purity of newly synthesised compounds, to testing rivers for pollution, to testing clothes samples for blood samples. That’s right, all that forensics, CSI stuff fits squarely within the framework of analytical chemistry. And it is nothing like the TV shows.

There are laboratories specifically dedicated towards analysis of samples in every industrialised nation, each with its own national board overseeing these facilities. In Australia, RACI (Royal Australian Chemical Institute) handles this function and more – as well as accrediting professional chemists, it promotes chemistry in the wider society. There are analogous national and international bodies elsewhere.

Together, they have developed an accreditation system for analytical laboratories – any lab that passes their tests is recognised worldwide as a quality facility. Makes sense, after all, since often analytical labs need to submit evidence into courts of law. Having national and international standards is a good way to ensure the data and conclusions being presented can be trusted.

Except for one little thing: accreditation is, in practical terms, meaningless.

Every now and then laboratories are tested. The largest scale test is the International Measurement Evaluation Program, whose purpose is to compare the results of labs from different countries using different equipment and techniques to quantify an analyte. Some lab somewhere carefully creates a sample; let’s just say for arguments sake it is water containing 40 mg/L of copper ions, with some iron and table salt thrown in. This sample is then duplicated and sent to a number of different labs, which are told they need to find the amount of copper present. A simple, yet fair, test of their abilities.

No lab will get 40 mg/L, simply because that is not how measurements are made. A lab might measure the sample and say the copper concentration is 39.8 ± 0.1 mg/L, which means they are saying they are 95% sure the answer is between 39.7 and 39.9 mg/L. That is a good result for this example. But some might get 35 ± 15 mg/L, which is technically true but the range of values is enormous. And some might get 120 ± 5 mg/L, which makes you wonder what they were drinking the night before the test

(By the way, yes, I am making these numbers up, but this example is not unheard of. In fact, if, say, a dozen labs measure the same sample, these results are fairly typical. Just to be clear, I am not exaggerating for effect here).

So some labs do pretty well, some labs get the right answer but with enormous uncertainty in the actual value, and some are way off. Well, we can’t all be winners. But there is a consistent trend as to which labs perform well in these tests.

It is not the accredited labs – some perform well and some perform badly, just like the non-accredited labs. It is not the labs with the more expensive equipment – I always enjoy seeing a lab with 5 year old, $100,000 machines outperforming 1 year old, $2m machines. No nation or continent does better than any other.

The answer seems to be this: the more qualified your staff are, and the more effort you put into training them, the better your analytical results. That’s it, nothing special. Big, expensive machines help, they help a lot, but the money is better spent on people. And accreditation, while establishing your reputation worldwide, means little in terms of the accuracy or precision of any results obtained.

So mistakes are made, even by the larger laboratories. And there are many, many smaller ones. What does this mean?

*Health care tests. Things like testing for viruses or cholesterol are not 100% accurate. It has been estimated that within the US, one-quarter of these tests performed are repeated measurements due to problems. These repeats cost roughly $36 billion all up.

*In the UK, a survey performed by the Laboratory of the Government Chemist found that 29% of analytical results were so incorrect, they cost the customer money. Imagine having to take extra steps to filter your water because a lab detected arsenic that simply wasn’t there.

*The Azaria Chamberlain case, aka “The Dingo Stole My Baby” incident. Stains found in the car belonging to the Chamberlains were reported by an analytical lab to be the blood of a human infant. It was later revealed that the analyst, working at an accredited laboratory, only tested for the presence of iron. Having found it, he concluded it was blood.

The stain was from paint thinner.

But maybe I’m being mean. They make mistakes, but who doesn’t? They are still useful, still performing a valued service, still wielding science in an attempt to better society.

Well, not quite.

Most of the analytical laboratories work for their respective governments, those that don’t still work closely with them. After all, they need to maintain their national accreditation. What this means is simple – if any government needs an excuse to suddenly and without warning deny a sudden importation of food, all they need is for one of their labs to “find” a problem. Mercury in shrimp. Arsenic in grapes. If you are looking for a certain contaminant and are willing to fudge a little, you can find anything. Independent verification of the food’s suitability will come, but it will come too late – a shipload of goods sitting offshore for weeks, maybe months, will render the shipment worthless.

Works better than tariffs at protecting a government’s own agriculture industry.

But there is an even more fundamental issue at stake here. One proven way of developing wealth is through trade, and one of the easiest things for most nations to produce are agricultural products. Ideally, a 3rd world country could start to pull itself out of poverty by growing food and selling off the excess. But no other nation will buy from them, not without a national body of internationally accredited analytical laboratories to test the food for contaminants. So the idea is this: a poor nation, which can’t even manage to scrape together the resources to set up factories, is expected to assemble their own world-class laboratory.

There are international agencies dedicated to the evaluation of analytical laboratories. But is there a Chemists Without Borders? Not that I know of. Could they set up some sort of deal where the richer nations purchase the food, as well as test it? Well, maybe, but that has all sorts of problems – the exporter is now at the mercy of the importer. Mild quality issues could be invented to drive down the price. How about allowing the food to be exported, just labelled with a “buyer beware” logo? The poorer nations selling, with the poorer nations their consumers might just work.

But that right there is the problem. I find myself thinking about the political biases of scientific investigations, something which should never happen. There will always be politics behind science, as whoever controls the flow of grant money controls what is and isn’t researched, but with analytical chemistry we have something different – the labs are funded by a government, but a government is also a customer in all this. When they have a desire for a test to go a certain way, the results will be skewed, science and fairness be damned.

Analytical chemistry is a highly practical science. The results it gives are used by other sectors of society, including those that fund the science. It can be used by federal governments to shape international agreements. It is a tool, not just for scientists but for politicians and lawyers, and that is where its sins lie. Real world applications draw the interest of those who see it as a means to an end. The beauty of it is, while something like technology is universally verifiable and requires correct science, analytical results are different from case to case. The results might be wrong, by design or accident, but until others verify the investigation no one will know.

Science is only a noble pursuit when no one cares about the results. Once people see a chance to make money, or score a point, sins will begin to creep in. Sinning will send you to Hell, and Hell is other people, so the only solution is to keep scientists away from everyone else. But if we want science to mean anything we have to accept that, try as we might, it will be infected with the evils of humanity.

Comments

Wow, what a fantastic article, AW.

Is there any remotely realistic hypothetical where you could guard against the sort of political biases you wrote about?

Quote Originally posted by Zuul View post
Is there any remotely realistic hypothetical where you could guard against the sort of political biases you wrote about?
The only way to remove bias, in theory, would be to have international traded goods tested and regulated by some international body. This would reduce bias, but since the wealthy nations would be funding the majority of this body, there would still be a bias. But it would be close.

Really, it depends on how much you trust the impartiality of the UN. If you trust it a lot, then this idea could work. If not, then not.

But this idea would be unpopular - nations would not pay to have power taken out of their own hands.

Excellent article, as always, AW. I hadn't thought about the implications for trade in terms of laboratory error. Or even directed results.

The Sins of Blog Writing

In dredging up the Azaria Chamberlain case to make a point, you have committed some of the very sins to which you are referring. The information presented was badly researched (or not researched at all) and in the "I heard it from a bloke who heard it from a bloke"-vein. The irony of this in an article knocking lack of scientific rigour leaves me dumfounded.

* The forensic scientist did not test only for the presence of iron. Where on Earth did you get that idea? What the forensic lab used was an antiserum against haemoglobin F, which was supposed to have been a starting point for narrowing down the investigation, but was instead used as immediate evidence for blood. And then, you leave a dramatic pause to say

"It was paint thinner."

It was not. Did you just make that up? It was a sound-deadener sprayed on the wheel arch of the car, which made its way to the underside of the car's dash. How would your supposed "test for iron" have reacted to paint thinner, anyway?

This small section of your post was extraordinarily sloppy, and negates the credibility of the entire article. This type of writing would be better suited to tabloid journalism than in a scientific environment.

Quote Originally posted by Unregistered View post
In dredging up the Azaria Chamberlain case to make a point, you have committed some of the very sins to which you are referring. The information presented was badly researched (or not researched at all) and in the "I heard it from a bloke who heard it from a bloke"-vein. The irony of this in an article knocking lack of scientific rigour leaves me dumfounded.
As is often the case, irony runs much thicker than you could possibly know.

* The forensic scientist did not test only for the presence of iron. Where on Earth did you get that idea? What the forensic lab used was an antiserum against haemoglobin F, which was supposed to have been a starting point for narrowing down the investigation, but was instead used as immediate evidence for blood. And then, you leave a dramatic pause to say

"It was paint thinner."

It was not. Did you just make that up? It was a sound-deadener sprayed on the wheel arch of the car, which made its way to the underside of the car's dash. How would your supposed "test for iron" have reacted to paint thinner, anyway?
Firstly, in your rush to cal me unscientific you forgot to throw in citations. But for fun I am going to assume you are completely right and I am completely wrong about the case.

Secondly, I didn't just make it up. My source for this was a series of lectures delivered by a university professor who works in analytical chemistry, in particular its relation to the law. If you are interested, I can email you the powerpoint of the presentation in question.

So perhaps the fault is mine for assuming that a professor talking about his area of study would get the details correct. But here's the beauty - if he got these details wrong, that goes some way towards confirming the central thesis of this article - that scientists, as individuals, can make mistakes.

Thirdly, you have a problem with the idea of finding traces of metal in a liquid found inside a car? How interesting.

This small section of your post was extraordinarily sloppy, and negates the credibility of the entire article. This type of writing would be better suited to tabloid journalism than in a scientific environment.
Really? Thanks!

Sorry, but I don't know how to interpret this as an insult. In the hierarchy of scientific truth, at the top is peer reviewed studies, near the bottom is tabloid journalism and at the very bottom is blog writing. In short, you have elevated me to the same level as paid professionals, sloppy as they may be.

Finally, I will say this - that detail, in no way contradicts the central thesis of the article. It is exactly that, a minor detail, a case study to highlight a general trend. There are plenty of cases like it out there, any of why can be substituted in its place without altering the conclusions of the article. Meanwhile, if that is the only error I have made in all the articles and posts relating to science that I have written then I am way ahead of the average.

This was a fascinating article, I have never really even thought about how we know how accurate lab results are -- much less why people might not want the results to be correct. Thanks, AllWalker.

Whoops, double post.

Whoops! Wrong thread.