Stochastic Gradient Descent Made Fun

The expression for the slope of a function is:

What does it mean when the slope is zero?

What is the expression for the x-coordinate of the vertex of a parabola?

If you connect two dots that are equidistant from the vertex with a line, what is the slope of that line?

For the upward-opening parabola y = x², how does the slope change from the left to right side of the vertex?

Using this quartic equation as an example: y = x⁴ + 2x³ − 3x² + x + 1 We have something that looks like a vertex, but since it's a higher order polynomial, it's a local minimum: there's no formula for this! It's at (-2.22, -13.60).

In real life problem solving, why might finding a vertex or local min/max be useful?

If f(x) represents the amount of water pollution, and x is the amount of a resource used in an engineering project, would it be better to find the local minimum or local maximum of this function? Explain your reasoning.

But the world is a lot more complicated than one number in and one number out! We can actually extend this concept to 3D! Don't worry about all the underlying math, I want to focus on intuition. 3D functions are cool; they are normally not taught until Calculus III. Below is the 3D equivalent of a parabola called a paraboloid:

Why do you think the real world doesn't normally use 2D functions for science and engineering?

Let's say you could walk inside that paraboloid above, but you must be blindfolded. If you didn't exactly know where the bottom is, how would you figure out where the lowest point is? What concept from the start of the lesson helps us solve this question? Hint: There is no equation for this problem.

Computers can crunch a bunch of math insanely fast. We can also use context clues to help computers make a lot of rapid guesses back-to-back, so they get closer to the output we want.

Remember the concept of local minima/maxima and how we talked about slope? Those concepts all exist in 3D too.

So how do you think a computer can find the local minimum of a function if all we know is the input, output, and slope of where we are "standing"? Hint: Imagine you are walking in the paraboloid from the last page trying to find the very bottom.

So instead of walking inside the paraboloid, let's imagine we have a super weird bowl with marbles inside. We drop marbles somewhere randomly in this bowl, and wherever the marbles end up landing decides how much money is saved. The deeper inside the bowl they end up, the better.

We have two holes. Think of the deepest hole as the global minimum and the other hole is a local minimum.

Let's say you could control the bowl with your hands and you want as many marbles as possible to end up in the deeper hole. You are blindfolded once again, so you can't tilt the bowl in the direction you know has the deeper hole.

What could you do to try to get the most marbles in the deepest hole after the marbles are dropped? Could anything be done?

What in the world does this have to do with ChatGPT?

AI like ChatGPT stores how words relate to each other as a whole bunch of numbers organized in a specific way. Check this out:

In this first example, ChatGPT stores the concept of “Royalty” on the x-axis and “Gender Identity” on the y-axis.

In this second example, ChatGPT stores the concept of “Activity Intensity” on the x-axis and “Temperature” on the y-axis.

Here, we can see how we can perform math operations between word concepts in the way ChatGPT thinks. If we go by the coordinate pairs, we can calculate: Monarch = Queen - Woman.

Before we jump into four dimensions, it helps to think about Flatland, written by Edwin Abbott Abbott: a world where the inhabitants can only see two dimensions. Its class depictions are also a satire of the rigidity of Victorian-era social hierarchy. A 3D object passing through their world would look like a changing 2D slice or shadow, because they can only observe its projection into their space. Vector projection is the same core idea in math: we take something with more information and ask, "How much of it points in this direction?" A word vector can be projected onto axes like royalty, temperature, or activity intensity, and each projection gives us one meaningful part of the full concept.

Now let’s add one more dimension. Below, the words sit in 3D concept space, but they only "exist" when the slider matches their hidden 4th dimension. You can choose different "lenses" to see how AI organizes concepts like Flavor, Vibe, or Arcane Power.
So ChatGPT does all of this, but instead of pairs of numbers, it is able to perform these operations on lists of thousands of numbers at a time. This is because there are so many more relationships between words outside of “Royalty” and “Gender Identity” or “Activity Intensity” and “Temperature”.

So instead of 2D space (with parabolas) or even 3D space (with paraboloids)...

We are working with functions in THOUSAND-PLUS DIMENSIONAL SPACE!!!!

What does it mean to "train AI"? We are using two analogies. One analogy is words coming into place. Another analogy is a bowl being shaped.

Both visuals are simplified ways to imagine training: the model is gradually learning patterns from data.

What do you think happens when we raise or lower the temperature of the output of ChatGPT? Yes, this is shaking a bowl in a thousand-plus dimensional space. (Hint: "Shaping" the bowl is different from "shaking" it.)

To conclude, we started by talking about how to calculate the slope of a function, and we ended with the insanity of dropping marbles in strange, thousand-plus-dimensional bowls to explain how ChatGPT works. What you are learning in class now aligns with the foundation of not only AI, but how the world works in SO many ways. Below, please connect ONE topic you learned in class to ONE new idea you learned today.