Data Science and Machine Learning (Theory and Projects) A to Z - Deep Neural Networks and Deep Learning Basics: Weight I

The video tutorial explains gradient descent, focusing on convex loss functions and their global minima. It highlights challenges in neural networks, such as non-convex loss functions leading to local minima. The vanishing and exploding gradient problems are discussed, emphasizing the importance of proper weight initialization and activation function choices. Strategies for weight initialization, including using normal distributions, are covered to improve learning efficiency. The tutorial concludes with a preview of future topics like learning rates.

What is the primary goal of gradient descent in a convex loss function?

• To find the local maximum

• To find the local minimum

• To find the global minimum

• To find the global maximum

Why is weight initialization important in non-convex loss functions?

• It ensures the function remains convex

• It prevents overfitting

• It affects the convergence to a local minimum

• It determines the learning rate

What problem arises when weights are initialized to zero with sigmoid activation functions?

• The learning rate becomes too high

• The network overfits the data

• The network becomes too complex

• The activations and gradients become zero

What is one of the main reasons for choosing ReLU over sigmoid activation functions?

• To decrease the number of parameters

• To ensure weights are always positive

• To avoid the vanishing gradient problem

• To increase the learning rate

How does the variance of the normal distribution for weight initialization depend on the layer size?

• It is inversely proportional to the learning rate

• It increases with more units in the layer

• It decreases with more units in the layer

• It is constant regardless of layer size

What is the primary benefit of using normal random variables for weight initialization?

• It speeds up the convergence to a local minimum

• It reduces the number of parameters

• It guarantees finding the global minimum

• It ensures weights are always positive

What is the ultimate goal when dealing with local minima in neural networks?

• To find the global minimum

• To find a feasible minimum

• To ensure all weights are zero

• To avoid any minima