The Laws of Thought

Tom Griffiths

“Backpropagation was not used to adapt a supervised learning system; it was to translate Freud’s ideas into mathematics, to implement a flow of what Freud called ‘psychic energy’ through the system.”

This book is supposed to cover the history of how people have tried to rationalize thought. It has some interesting tidbits, like the quote above, or the fact that Hinton is a direct descendant of Boole. It is way too computation-pilled for my tastes, and I would have liked some philosophizing as well. I do not know who the audience for this book is. The already sparse theoretical examples were more typoset than typeset. Almost every subchapter starts with a Wikipedia dump of the people involved that I cannot tell you who’s who and which random American university they are from. I do not know if there’s a better way of presenting all this information, all I know is that I do not like it.

Expanding on the technical details, the typesetting of the production rules are completely off. The replacement rules often have something that should be on the left column instead show up on the right column, e.g. a NounPhrase is split across a column as Noun and Phrase, and a Sentence $\to$ Apple Sentence Banana is instead a Sentence $\to$ Apple and a Sentence $\to$ Banana. I blame the book editors for this one, and surprising that this wasn’t caught before it went out to print. There’s weirdly a wrong quotation mark for the word green, which is usually a TeX error. I wonder if the book was written in TeX first. There is also the typo about a card with 7 on one side and 3 on the other. These are minor nitpicks that can be fixed in a second print.

Peirce taught logic at Johns Hopkins University for five years but was fired after beginning a relationship with a much younger woman before he was formally divorced from his first wife (with some similarities to the fate of John Watson forty years later).

Referring to being fired because of sleeping with a much younger woman as someone’s “fate” rubs me the wrong way, there’s a bit of inevitability to it. Sounds like he had no other option but to do it, this is how it would turn out and there’s no other possibility. Seems to remove a bit of agency from John Watson and Charles Sander Peirce’s actions.

The explanation of how backpropagation succeeded, which I would be very interested in knowing, was a bit of a damp squib. It basically goes: Rumelhart derived the math, and coded it up. It worked, but was very slow. He sat on it and didn’t do anything about it (oooooo suspense, we’re gonna figure out what made it work later). Well, one day Hinton says he coded it up and it works fine. End of story. Is it that it still took thousands of iterations but hardware improvements made it feasible? Why did it not get stuck in local minima? Did it get stuck in local minima and this was ok? Deeply unsatisfying.

The example about triangle inequality is interesting, about asteroid and buckle being close to belt, but being far away from each other. I wonder what the correct model for this is. Also, for a similarity function $s$ on pairs of words, finding three words $x, y, z$ with $s(x, y)$ and $s(y, z)$ small, with large $s(x, z)$ seems like exactly the kind of clues that would make for a fun crossword puzzle. I asked an LLM for some more examples and also vibecoded a script to find more of these, and somehow the examples that the LLM gave were so much better than the code. Here’s a few more: (river, bank, mortgage), (tree, bark, dog), (computer, mouse, cheese), (traffic, jam, strawberry).

Something that was new to me and interesting was the Bayesian explanation of why a single example suffices in some settings. First time I’ve gotten something out of a Bayesian, all I usually get is people making up prior probabilities and arguing from there (not that that doesn’t exist here). I still need to think about hypotheses falling off as an exponential function of distance, but I’ll probably not.

For myself later:

Asymmetry: hairdresser:comb::pitcher:baseball is good, while pitcher:baseball::hairdresser:comb is not. Triangle inequality: nurse:patient::mother:baby is good, mother:baby::frog:tadpole is good, but nurse:patient::frog:tadpole is not.