Pythagorean Theorem/Miracle
I want to share an interesting puzzle that I encountered and finally was able to figure it out. Here is how it goes. Having the gradient of a two-variable function f(x, y) at point (x0, y0) to be equal (2, 1); What is the best direction to go in order to increase the function considering you have a certain step budget denoted by g? This is where the contention begins.
I think that it is intuitively better to only walk in the direction of x since there you would get twice as much as the progress if you would walk in the y-direction. And spending any fraction of your step g in the direction of y seems wasteful. Hence, I would only walk along the x-axis to increase my function.
However, mathematicians say another thing. Resorting to the definition of the gradient which is “the direction at which maximum increase can be attained”, one should move in the direction of the gradient which is (2, 1). This seems so counterintuitive! Why should I care about the less promising axis which in this case is y? I do not like the less fruitful direction.
After a great deal of research and adventure, finally, I found what is wrong with my intuition. I call it the “Miracle of the Pythagorean Theorem.” Imagine your step budget is g=13. If you only move in the direction of x which my intuition dictates, you would get a 26=13*2 points increase in your function. But we know that 13² = 12² + 5² and 13² = 13² + 0². This means that in Pythagorean Theorem you can trade 1 portion of your step in the direction of x and get 5 portions of step in the direction of y. Accordingly, your increase in the function would be 29=12*2 + 5*1 > 26=13*2.
This improvement was achieved because we walked in the direction of (12, 5) as compared to (13, 0). The hidden aspect that I was missing in my intuition is that the Pythagorean Theorem allows you to trade 1 portion of step in the x-axis and get 5 portions in the y-axis.
To be more clear, when you spend your step budget g=13 in the direction of (12, 5) as opposed to only walking in the direction of (13, 0), it is as if your step size was extended to 17 from 13. It is as if you moved 5 portions of step in the direction of y (5 points increase) and 12 portions of step in the direction of x (24 points increase). And that is the foundation of this counterintuitive observation.
This is the fault of a one-dimensionality mindset which presumes that it is better to spend all your step budget in the best (one) dimension and ignore other (seemingly unattractive) dimensions. I am thinking perhaps everything else such as life, career, development, and research, follow the same rule. If this is true, we better not reject minor improvements in other dimensions. Nevertheless, we should draw a line between exploration and distraction.
