Could This Be The Worst AI Paper Published?
A recently published paper in Nature claims that the carbon emissions of writing are lower for AI than it is for humans. It arrives to this conclusion using ridiculous methodology.
A recently published paper in Nature made the following claim:
Now… in addition to being a medical doctor, I happen to also be a bit of a writer.
You guys might not know this, but I was editor of my school’s magazine. I held editorial positions at various publications while studying medicine at Cambridge. Founded two online publications at Cambridge. I write these blog posts myself — as well as the scripts for the science videos I publish on YouTube.
I’m sure it’s obvious from my style of writing, but I don’t use AI in my writing.
This isn’t to say that I’m fundamentally against the use of artificial intelligence.
I use various AI tools to do other stuff. Sometimes I’ll give ChatGPT a video script I wrote and ask it to critically appraise it, pointing out how it can be improved.
Anyway, you can imagine how curious I was to find out more about this paper.
The Paper’s Findings
According to the study, when writing a one page piece of text, AI has a carbon footprint that is up to 1500 times less than that of a human author.
The authors of the study use CO2e as a proxy for carbon footprint. CO2e is carbon dioxide equivelent, so in the context of Figure 1, it represents the grams of carbon dioxide emissions produced by AI and humans when writing one page of text.
Humans in the US produce 1400g of CO2e emissions when writing a single page of text — compared to ChatGPT’s 2.2g of carbon dioxide equivalent emissions.
Except… this definitely feels too good to be true.
Spoiler Alert: It was.
Ridiculous Methodology
I began wondering how the paper’s authors arrived at the carbon dioxide emissions produced by human writers (1400g in the US and 180g in India).
Writing isn’t exactly a polluting exercise, and by the author’s own admission, they separated the emissions produced by the computers used in writing.
So, here’s how they did it:
In summary:
They took the total annual emissions produced by an average person in both the United States and India, which were 15 and 1.9 metric tons, respectively.
They found out how long (on average) it took a person to write one page of text, which (for some reason) was defined as just 250 words in this paper.
The figure they calculated was 0.8 hours (roughly 48 minutes).
Finally, they did some funny math where they calculated the emissions produced in writing a page of text from the total annual emissions, casually “assuming that a person’s emissions while writing are consistent with their overall annual impact.” I’m sorry, but what the actual fuck is this?
Have the authors of this paper gone insane?
You can’t use a person’s total annual emissions like that to calculate the emissions produced when writing… because writing isn’t a resource intensive activity. Writing isn’t like driving your car to and from work. It’s not like buying stuff online that comes in a million different packages. Putting words on a page only requires that you be alive, literate, and have a means of recording your thoughts and ideas. Pretending like it’s equivalent to fast fashion is weird af.
The only way we can make this “logic” work is if the authors of the study somehow believe that if humans stopped writing (delegating the task to AI), they’d cease to exist (and consume) for the duration of the time saved.
Other Problems
The paper also has some other issues, specifically in how it obtains the extremely low emissions produced by ChatGPT writing 250 words (2.2g of CO2e).
The “scientists” get that number after a series of calculations (or, should I say, manipulations?). The only problem with that? OpenAI doesn’t really release a lot of data to go by. At best, all you’ll get is a very incomplete picture.
So, the authors of the study did the only reasonable thing and got their data from a Medium article by a man called Chris Pointon — who appears to be a serial tech entrepreneur and holds an electronic and computer engineering degree, according to his LinkedIn account.
Now, I actually think Mr Pointon did a good job in his blog post.
At the end of his post, he does point out some limitations in his analysis.
This is directly from Mr Pointon’s article:
What’s missing from this quick analysis:
The actual number of queries per day that OpenAI users are generating
Emissions from training the model. In an article about the CO₂ footprint of a single ChatGPT instance, Kasper Groes Albin Ludvigsen lists this at 522 tCO2e. These emissions are amortised over the lifetime of the model
CO₂ emissions of the end-user equipment accessing ChatGPT . This includes power consumption and a share of the emissions from producing and disposing of the device. It’s probably the largest component of its footprint and impossible to calculate without knowing what devices are accessing ChatGPT and from where. OpenAI could provide some idea of this if they add something similar to the Website Carbon Calculator to the service.
Non-GPU emissions like networking, RAM and SSDs
The embodied carbon of datacenter
Mr Pointon would most likely make for a better scientist than any of the “researchers” who published this ridiculous paper. Am I being too harsh?
Either way, I have a new cool video out on YouTube about why AI can’t match human intelligence (and the brain) in efficiency. Watch below:
Video References
Attwell D, Laughlin SB. An energy budget for signaling in the grey matter of the brain. J Cereb Blood Flow Metab. 2001 Oct;21(10):1133-45. doi: 10.1097/00004647-200110000-00001. PMID: 11598490.
Balasubramanian V. Brain power. Proc Natl Acad Sci U S A. 2021 Aug 10;118(32):e2107022118. doi: 10.1073/pnas.2107022118. PMID: 34341108; PMCID: PMC8364152.
Baumann, O. (2023, April 27). How much energy do we expend using our brains? Retrieved March 13, 2024, from https://bond.edu.au/news/how-much-energy-do-we-expend-using-our-brains
DXOMARK (2023, December 8). Apple iPhone 15 Pro Max Battery test. Retrieved March 13, 2024, from https://www.dxomark.com/apple-iphone-15-pro-max-battery-test/
Howarth C, Gleeson P, Attwell D. Updated energy budgets for neural computation in the neocortex and cerebellum. J Cereb Blood Flow Metab. 2012 Jul;32(7):1222-32. doi: 10.1038/jcbfm.2012.35. Epub 2012 Mar 21. PMID: 22434069; PMCID: PMC3390818
Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., & Wei, F. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. ArXiv. /abs/2402.17764
Madrigal, A. C. (2014, October 17). The Energy In Things. Retrieved March 13, 2024, from https://www.theatlantic.com/technology/archive/2014/10/the-energy-in-things/381557/
Tomlinson, B., Black, R.W., Patterson, D.J. et al. The carbon emissions of writing and illustrating are lower for AI than for humans. Sci Rep 14, 3732 (2024). https://doi.org/10.1038/s41598-024-54271-x
Top500 (2023, November 1). Green 500 List November 2023 Release. Retrieved March 13, 2024, from https://www.top500.org/lists/green500/2023/11/