Stochastic Gradient Descent

The expression for the slope of a function is:

What does it mean when the slope is zero?

What is the expression for the x-coordinate of the vertex of a parabola?

If you connect two dots that are equidistant from the vertex with a line, what is the slope of that line?

How does the slope change from the left to right side of the vertex of any parabola?

Using this quartic equation as an example: y = x⁴ + 2x³ − 3x² + x + 1 We have something that looks like a vertex, but since it's a higher order polynomial, it's a local minimum: there's no formula for this! It's at (-2.22, -13.60).

In real life problem solving, why might finding a vertex or local min/max be useful?

If f(x) represents the amount of water pollution, and x is the amount of a resource used in an engineering project, would it be better to find the local minimum or local maximum of this function? Explain your reasoning.

But the world is a lot more complicated than one number in and one number out! We can actually extend this concept to 3D! Don't worry about all the underlying math, I want to focus on intuition. 3D functions are cool; they are normally not taught until Calculus III. Below is the 3D equivalent of a parabola called a paraboloid:

Why do you think the real world doesn't normally use 2D functions for science and engineering?

Let's say you could walk inside that paraboloid above, but you must be blindfolded. If you didn't exactly know where the bottom is, how would you figure out where the lowest point is? What concept from the start of the lesson helps us solve this question? Hint: There is no equation for this problem.

Computers can crunch a bunch of math insanely fast. We can also use context clues to help computers make a lot of rapid guesses back-to-back, so they get closer to the output we want.

Remember the concept of local minima/maxima and how we talked about slope? Those concepts all exist in 3D too.

So how do you think a computer can find the local minimum of a function if all we know is the input, output, and slope of where we are "standing"? Hint: Imagine you are walking in the paraboloid from the last page trying to find the very bottom.

So instead of walking inside the paraboloid, let's imagine we have a super weird bowl with marbles inside. We drop marbles somewhere randomly in this bowl, and wherever the marbles end up landing decides how much money is saved. The deeper inside the bowl they end up, the better.

We have two holes. Think of the deepest hole as the global minimum and the other hole is a local minimum.

Let's say you could control the bowl with your hands and you want as many marbles as possible to end up in the deeper hole. You are blindfolded once again, so you can't tilt the bowl in the direction you know has the deeper hole.

What could you do to try to get the most marbles in the deepest hole after the marbles are dropped? Could anything be done?

What in the world does this have to do with ChatGPT?

AI like ChatGPT stores how words relate to each other as a whole bunch of numbers organized in a specific way. Check this out:

In this first example, ChatGPT stores the concept of “Royalty” on the x-axis and “Gender Identity” on the y-axis.

In this second example, ChatGPT stores the concept of “Activity Intensity” on the x-axis and “Temperature” on the y-axis.

Here, we can see how we can perform math operations between word concepts in the way ChatGPT thinks. If we go by the coordinate pairs, we can calculate: Monarch = Queen - Woman.

So ChatGPT does all of this, but instead of pairs of numbers, it is able to perform these operations on lists of MILLIONS of numbers at a time. This is because there are so many more relationships between words outside of “Royalty” and “Gender Identity” or “Activity Intensity” and “Temperature”.

So instead of 2D space (with parabolas) or even 3D space (with paraboloids)...

We are working with functions in MILLION DIMENSIONAL SPACE!!!!

Remember when we talked about dropping marbles in bowls and shaking a little bit? ChatGPT does this while working in a dimensional space WE CAN’T EVEN COMPREHEND!! Long story short, ChatGPT’s outputs are based on sets of predictions that use information from relationships between words. Accuracy, comprehension, and output from these AI models can be thought of as a more complicated version of finding a local minimum, like the vertex of a parabola.

Reminder on our new terms: stochastic is a fancy word for random, and this concept is also referred to as “noise” or “temperature”. Temperature is more widely used in the context of language models like ChatGPT.

What do you think happens when we raise or lower the temperature of the output of ChatGPT? Yes, this is shaking a bowl in a million dimension space.

To conclude, we started by talking about how to calculate the slope of a function, and we ended with the insanity of dropping marbles in strange, million-dimension bowls to explain how ChatGPT works. What you are learning in class now aligns with the foundation of not only AI, but how the world works in SO many ways. Below, please connect ONE topic you learned in class to ONE new idea you learned today.