Summary

todo

Raw notes

2024-12 genius makers book

P33 Hinton not interested in mathematics, including in linear algebra. He would just get help from others to help solving the differential equations

First chapter of the book is about how connectionism was apparently disproven by Marvin Minsky. But Hinton was nit convinced.

Hearing initial issue was, they only tried singlw layer networks. Huh why.

P37, back prop. Rumelhart

P39 Boltzmann Machine

P45 oh wow , Pomerleu, ALVINN, first truck , Chevy, self driving. Using Backprop, from Hinton published paper around 1985-1986. Truck drove 60mph, from Pittsburgh to Eerie.

P50 Hinton, and his need to understand the brain, and Yann LeCunn , electrical engineer, interested in the hardware of computing, mathematics of neural nets, and broad AI. P51 Known for his convolutional neural network. Hinton as a fox, LeCunn as hedgehog. Fox knows many little things, hedgehog 🦔 knows one big thing.

Hmm but those CNNs didn’t work on cats, dogs. Only digits. Early on at least.

AT&T split up again and Yann left Bell Labs.

P56 John Brockett linguist panic attack, thought it was a heart attack, after 6 years, building rules, and seeing the early statisticaj translation, around 2003, at Microsoft, Seattle office.

P58 also people were allergic to the word neural from the word neural network, one example is that Yann LeCun removed the neural and convolutional neural network and advertised this as the convolutional network instead.

P59 , wow this is pretty hilarious so Jürgen Schmidhuber, who is known for building the LSTM originally worked with an Italian liquor manufacturer named Angelo Dalle Molle.

And Jeffrey Hinton would sometimes joke that LSTM stud stands for it looks silly to me

P68 cool, so the discussion is that there was a speech recognition paper by Jeffrey Hinton and another researcher from Vancouver. Li Deng, said that this paper just wasn’t done right. Didnt clearly show the work perhaps.

P72,73, about haha how Hinton at building 99, Microsofft was trying to convince Deng, go for he $10,000 GPU, but Alex Acero, who apparently would oversee Siri later on, said nahh we must spare this expense haha.

And its Peter Lee from Carnegie Mellon, looking over expenses, saying if he came earlier, in retrospect, he would have also tried really hard to put a stop to this connectionist neural nuts work that he felt was funny and did not work

P74. Wow. So working with Deng, Hinton, also with two of his grad students, including Dahl, who knew GPUs really well, they built q system thatoutperformed what Microsoft had been hand tuning over last decade. Thus was 2005.

This speech recognition win, was seen as an inflection point for neural networks .

P84, how Andrew Ng ran into Jeff Dean in a Google kitchen!

P85, Jeff Dean, sounding, built out distributed systems that page rank ran on.

P87, Dean using his 20% time help, Ng, big distributed NN.

P88, The Cat Paper. Ng , Dean and Corrado. After, Ng left for starting Coursera but recommended Hinton as a replacement.

P90. Hinton saying of Dean, no ego, only collaboration. And resourceful, “he created a kind of canopy where the brain team could operate. And we didnt have to worry about anything else. If you needed something, you asked Jeff and he got it.” This was in reference to say, some max compute they reached and Jeff got some whatevee additional $2million additional.

Hinton compared Dean to Isaac Newton except Newton was an asshole. Newton like most smart people held grudges, but Hinton saying, “Jeff Dean doesnt seem to have that element to his personality”

P91, Cat paper, learning without labels. Raw unlabeled. Probably masking? But hmm, turned out, labeled, had higher performance. Hmm but yea semisupervised, would end up being the future anyways right?

P96-97 Alex net winning ImageNet competition in around 2012, in Florence Italy, with Alex Krizhevsky, on Hinton and Sutskevar team. People criticized the imagenet database kind of not realistic enough, so not generalizable . But still Yann Lecunn was in the audience and he felt vindicated. It was proof it worked, because they used CNN’s, his algo from 1980s. But this time with GPUs.

Oh wow, AlexNet paper, one of most cited computer science papers.

Then December 2012, the created and auctioned DNNresearch for $44million. But i dont get it, the paper was public. So whats the company have ?

2025-01-09 Just read, P100 - P116, from 11:12 to 12:03, 50 minutes. Only added red dots, not notes though. 17/50 ‎ = 0.34 pages per minute.

Book is 310 pages . How much longer? Approximately, minutes_remaining = (310 - 116)/0.34 ‎ = 570.588 minutes_remaining /60 ‎ = 9.51 Wow, 10 more hours.

200/17‎ = 11.765 hour long sessions, yea.

Makes sense that 300/17‎ = 17.647 bouts, hour each, I can finish a book of this style in a month.

Chapter 6. About the founders of Deep Mind and about them trying to make a game to fund their “how does the brain work” research. Google , with Hinton, Dean, meet at Deep Mind UK office and Dean after they introduce their research, asks to look at their code. They say okay, and Dean comes out kf that like, yea they hace what they are doing in there. They agree to Google Acquisition.

The Deep Mind founders, Demis Hassabis, game champion and game maker, Shane Legg, New Zealander computer science researcher that Hassabis met at Gatsby Unit at University College London, they both had super intelligence as a goal while others “eye rolled” at this. They met Mustafa Suleyman, their financial mind.

P108,109 Their appeal to Peter Thiel. Thiel also brought ij funding from Elon Musk too.

Hassabis had founded earlier, Elixir a game company.

The DeepMind research was Reinforcement Learning, solving the Breakout arcade game and others. Google didnt use RL yet so they were interested.

Google rented a gulfstream jet so Hinton could join! They got stuck in elevator at Deep mind. Also David Silver.

They thought games are a way to show clear objective ML progress. Points. And in ML, you need good objective function after all. But also games make for good demos!

chapter 7 This was about Facebook and their lab. And about how Microsoft and their Windows, was holding them back. And how the Google acquisition of Hinton’s DNN Research set the tone for how much AI researchers were the next NFL quarterbacks salary wise. And sounds like Yan Lecunn and Hinton set the tone for keeping his NYU professor position, working for tech and university split schedule. Hinton had paid his lawyer $400,000 I think around keeping his Toronto professorship also. Also Yan Lecunn negotiated that research should be open when joining Facebook.

There was an early moment when Mark Zuckerberg asked, “What’s NIPS?” He was building a vision of what AI could be at Facebook. In those early days, 2013, Facebook was very much traditional software and then AI came in as “identify the object or person in this photo,.” And I know from the interview between Ryan Peterman and Evan King (https://youtu.be/yZ4J98u7GLo, ) that Facebook is very much using AI to remove or throttle dangerous content images and words . Interview with Zuckerberg and Rogan the other day also Zuckerberg was explaining to Rogan about Precision Recall curves basically and that they are making a conscious business decision to reduce false positives of flagging peoples accounts as suspicious but that it has to come at the cost or reduced recall.

Oh yea and for Zuck it was the “AI is the Next Big Thing” moment and they need to be on top of this. Oh yea and initially Yan Lecunn was hesitant about Facebook also because of how the growth obsessed culture and short term financial gains would be contrary to long term AI vision. But Zuckerberg first AI hire, he sat next to him, because this was his strategy with new initiatives. And apparently other people felt a certain way about a “long term” researcher sitting so close to Zuck. This topic, like yesterday I was listening to Huberman and Jordan Peterson interview conversation that Huberman saying yea research prepares you for super long multi year horizons where you may not get a reward for a long time !

Chapter 8 Alan Eustace and his scuba approach to sky diving. He says hes not a daredevil, because derisks .

Initially, Google not interested in GPUs, because they pioneered distributed dumb machines.

Hmm they were trying to get GPUs for Alex Krizhevsky, but by then he already moved to the self driving car stuff.

Cool story because he and Anelia Angelova, both at Google Brain, were kind of teaming up to work on self driving in spare time, over a christmas break. Because currently the self driving team were using non deep learning computer vision strategies.

After the break, they shared their work, for identifying pedestrians, and they were invited to join full time. Eventually Waymo, spun off.

And even Google Search, Amit Singhal, also got convinced. Rank Brain, 2015, more accurate.

Oh cool and Deep mind, Demis Hassabis, reduced, power consumption, Google data centers. Paid for the cost of acquiring DeepMind!

Updates to talking digital assistants. Andrew Ng, at Baidu.

Hinton said Alex Krizhevsky , P142, was less Trump than others , less cared about credit, more about just progress. “It is just mathematics.” Nonlinear regression. Technology been around for decades, just now finally enough data. But technology itself not intelligent.

Hmm initially Ilya Sutskever, thinking AGI, too out there, but when later at Google through Brain, working with London, believing more in the big data thing.

Sutskever helped shape sequence to sequence translation task. Embeddings, P144. Train English, French data, together. Probably sentence similarity task. He showed at NIPS 2014, “minimum innovation, for maximum results”.

TPUs! Jeff Dean, noticing economics of services, translation etc, not scaling. His idea, yea, new chip, in house . Integer. For inference. Wow, from 10 second translation standard hardware, to milliseconds. Before Baidu. TPUs!

Chapter 9

Open AI , formation, Sam Altman bringing people together, in guise that Nick Bostrom Superintelligence fear assuaged, AGI. They dont buy the “you can shut it down like your car” argument.

Future of Life Institute, Puerto Rico, 2014, people pledging , inclufing Elon Musk.

Greg Brockman, putting the team together, for Altman, from Stripe. Napa Valley.

Sutskefer, on the fence, even though damn, 2 million from Google, still chose Open AI and Yan Lecunn saying it won’t work , not enough money.

Chapter 12 P199, interesting theme is what people like doing. Joushua Bengio, was not interested joinijg Microsoft because he likes Montreal, French, and advising startups, including Maluuba, ok. Similarly, Yan Lecuun, was after the long research, and did similarly point out to Sutskevar, big tech like Google will not have it.