Intelligent Technology (2024 Fall): News on AI

1. Nobel Prize.
   • The Nobel Prize in Physics 2024. For foundational discoveries and inventions that enable machine learning with artificial neural networks.
   • John J. Hopfield and Geoffrey Hinton. They trained artificial neural networks using physics.
     • Would there be a physical (equi. mathematical) model for a newtork of bi-state ({0,1} or {-1,+1}) neurons to memorize binary patterns? Answer: Ising model.
       • John J. Hopfield proposed a deterministic model, later coined the Hopfield network. If the size of each pattern is N, the size of the Hopfield network is N. That is to say, it consists of N bi-state neurons. The goal of a Hopfield network is to model the process of pattern reconstruction. If a noisy pattern is presented to the neurons, the Hopfield network should be able to reconstruct a pattern close to its noise-free counterpart.
       • Geoffrey Hinton proposed a stochastic model called Boltzmann machine. This machine consists of N+M bi-state neurons which are grouped in two layers, namely visible and hidden. The N neurons in the visible layer mimic the sensory neurons receiving environmental signals. The M neurons in the hidden layer mimic the neurons in our brain responsible to encode sensory signals. The goal of a Boltzmann machine is again to model the process of pattern reconstruction. Once a noisy pattern is presented to the neurons in the visible layer, the machine is able to reconstruct the noise-free pattern at the visible layer. The mechanism is repeating (1) visible-to-hidden processing and (2) hidden-to-visible processing. Iterating (1) and (2) a number of times, it is hoped that the neurons at the visible layer will give a pattern close to its noise-free counterpart.
     • What should be the learning rules for the models?
       • The 'learning rule' for a Hopfield newtork to memorize binary patterns is based on the idea of association. Learning by the principle of association is also called Hebbian learning, due to a postulate from Donald Hebb presented in his book The Organization of Behavior in 1949.
       • The 'learning rule' for a Boltzmann machine to memorize binary patterns is based on the idea of maximum likelihood, which is a parametric estimation method in statistics. Later, this learning is also called Boltzmann learning.
     • Given a noisy pattern, how does the network reconstruct the corresponding noise-free pattern?
       • Either Hopfield network or Boltzmann machine, the pattern reconstruction process is associated with an (nonlinear) energy function. Different patterns have different values of energy. The energy corresponding to the reconstructed pattern is essentially a local minima of the energy function.
       • The dynamics of pattern reconstruction process is equivalent to the dynamics of 'gradient descent'-move to a local minimum of an energy function.
     • Note that the process of pattern reconstruction in a Hopfield network (resp. Boltzmann machine) is equivalent to searching for pattern its energy is a local minima.
       • The process is essentially the same as a searching for a minima of a cost function in an optimization problem. With proper design on the model parameters, a Hopfield network (resp. Boltzmann machine) could thus be applied in solving an optimization problem.
       • Along this line, many Hopfield network-based algorithms have been proposed for solving various optimization problems in the subsequent decades. Many algorithms and their analog circuit implemnetaions have been proposed for solving various linear programming problems. It should be highlighted that analog computing has long been investigated in parallel to digital computing.
       • By the same principle, many Boltzmann machine-based algorithms have been proposed for solving combinatorial optimizataion problems in the subsequent decades.
       • An idea called simulated annealing has been applied in developing Boltzmann machine-based algorithms for solving combinatorial optimization problems.
       • Nowadays, many so-called AI solutions for industrial process design (resp. optimization) are developed along this direction. Strictly speaking, these solutions are not intelligence.
     • What should be the 'leraning rule' for a feed-forward neural newtork? Answer: Gradient descent.
       • Geoffrey Hinton, together with David E. Rumelhart and Ronald William, introduced the so-called back-propagation (equi. error back-propagation) learning rule for multi-layer Perceptron in 1986.
       • Back-propagation learning rule has been critized for decades. One critice is about its biological plausibility - how error information could propagate backward in a multi-layer neural network.
     • Other contributions of Geoffrey Hinton in machine learning (resp. statistical learning).
       • Blotzmann machine-based autoencoder.
       • Restricted Boltzmann machine (RBM) and RBM-based autoencoder.
       • Belief network and deep belif network.
       • Deep Boltzmann machine and deep RBM.
       • Layer-wise unsupervised learning together with supervised fine-tuned. To me, it could be treated as a pratical solution for deep neural network learning. Not until a deep model has been trained, it is unable to predict the optimality (resp. performance) of the model.
       • AlexNet, a so-called deep convolution neural network, for object recognition in 2012. GPU is applied in the training of the AlexNet. One point should be noted that the structure of AlexNet is inspired by at least two earlier models, namely Neoconitron introduced by Kunihiko Fukushima in 1979 and the Lenet introduced by Yann LeCun in 1989. Both Neoconitron and Lenet were inspired by the works of David Marr in the 1970s. The achivement of AlexNet has influenced the development of many auto-driving systems and traffic control systems.
     • The earlier contributions of John J. Hopfield and Geoffrey Hinton, to me, are belong to the Type I intelligent technology. They proposed models to interprete the behaviors of some propeties of possible neuronal networks in a brain (resp. neural information processing in a brain). Their contributions have not much inspired the development of the large language models (LLMs) which relate to language processing.
   • Nobel Prize laurerates who have contributions in the "neuroscience" and the "theory of mind".
     • Charles Scott Sherrington, Edgar Douglas Adrian (1932 Nobel Prize in Physiology or Medicine).
       • Their all-or-none principle influenced Warren MuCulloch and Walter Pitts proposing a threshold logic neuron (TLN) model in 1943. With TLN, a neuronal network can perform logical operations as in a Turing computing machine.
       • In the 1940s, MuCulloch-Pitts neuron model is not the only model for the property of a neuron. Threshold linear model (the so-called rectified liner moddel (ReLU)) was another model proposed in that period of time.
     • Alan Hodgkin, Andrew Huxley (1963 Nobel Prize in Physiology or Medicine).
       • A biological neuronal model nowadys called the Hodgin-Huxley neuron model.
       • It influenced additional models proposed for the physical behavior of a neuron. Those models are now collectively called spike neuron models.
       • These models have now been the major models for simulataing and analog harware realisations of a biological plausible neuronal network. These hardware realizations are under the area of neuromorphic technology.
     • Leon Cooper (1972 Nobel Prize in Physics).
       • Bienenstock, Elie L.; Leon Cooper; Paul Munro (January 1982). "Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex", Journal of Neuroscience. 2 (1): 32–48.
     • Gerald Maurice Edelman (1972 Nobel Prize in Physiology or Medicine).
       • Neural Darwinism: The Theory of Neuronal Group Selection (Basic Books, 1987).
       • Bright Air, Brilliant Fire: On the Matter of the Mind (Basic Books, 1992).
     • Friedrich August von Hayek (1974 Nobel Memorial Prize in Economic Sciences).
       • The Sensory Order: An Inquiry Into the Foundations of Theoretical Psychology, 1952.
     • Herbert Alexander Simon (1978 Nobel Memorial Prize in Economic Sciences).
       • Models of Man: Social and Rational- Mathematical Essays on Rational Human Behavior in a Social Setting, 1957.
       • Human Problem-Soving, (with Allen Newell), 1972.
     • Eric R. Kandel (2000 Nobel Prize in Physiology or Medicine).
     • John J. Hopfield, Geoffrey Hinton (2024 Nobel Prize in Physics).
2. Biological plausible neuronal model restricts the development of application-oriented algorithms.
   • From application point of view, a neural information processing system with artificial neuronal model might give better performance as compared with the biological plausible neuronal model counterpart.
   • Language processing could be modelled as a neural information processing system with artificial neuronal model.
3. Voice assistant.
   • Offline voice assistants: Voice assistants without Internet connection. It is an example of a real edge AI.
4. Image processing and object recognition.
   • LENET: An optical character recognizer.
   • ImageNet.
   • 3D-Net: Monocular 3D object recognition for traffic monitoring.
   • How Does Tesla's Autopilot Work?
   • Stability.AI.
   • From mid-2010s to late-2010s: Object recognition (auto tagging, imge-to-text) => Image captioning (image-to-text)
   • From late-2010s to present: Text-to-Image.
5. Fake video. (Video + Image => Video)
   • Barack Obama speech (2017).
   • It's Getting Harder to Spot a Deep Fake Video (2019).
   • Demo on Deep Live Cam (2024). Note: GPU required.
6. Large language models.
   • Chinese.
     • Baidu Ernie (文心一言).
     • Tencent Hunyuan (腾讯混元).
     • BAAI WuDao (悟道).
     • General Language Model (GLM) developed by Tsinghua University.
     • Huawei PanGu (盘古大模型).
   • English.
     • OpenAI ChatGPT.
     • Google Gemini.
   • Google Gemini Live Demo.
   • Demo of GPT-4o, with OpenAI-o1 model which has reasoning capability.
7. Use of AI tools for the work of a youtuber.
   • Create an Internet celebrity using free AI tools, like ChatGPT, Google Translate, Midjourney and DeepFake.
   • YouTuber experience sharing: Towards a profitable YouTuber with the use of AI tools.
8. Firms/Labs/Conferences/Talks.
   • Tesla AI & Robotic Research.
   • Generative AI.
   • Annual Conference on Neural Information Processing Systems (NeurIPS).
   • World Robot Conference 2024.
   • Interview Fei-Fei Li, December 11, 2023.
   • Fei-Fei Li, With Spatial Intelligence, AI Will Understand the Real World, TED, May 2024.
     • Image/Vision => Spatial intelligence (Video-to-Text).
     • Robotic vision => Text (Spatial intelligence) => Action.
     • Text description for the robotic vision, i.e. spatial intelligence.
     • Human instruction + Text description => Actions.
9. Robot Soccer Competition.
   • Robocup, the organizer of the robot soccer competition since 1997.
   • RoboCup 2011, finals: Nao-Devils vs. B-Human.
   • RoboCup 2018 MSL Final - Tech United vs CAMBADA.
   • RoboCup 2023 - Bordeaux - Tech United vs Falcons.
   • RoboCup 2024: Team RoMeLa Finals, Humanoid AdultSize LeagueRoboCup 2024 Finals.
10. Radio Broadcast.
    • Radio spectrum.
    • Radio Taiwan, AM radio.
    • Hit Radio, FM radio.
11. Societal Issues.
    • 發現沒，AI正在讓20到35歲的人變窮, October 2024.