Advances in Deep Learning Lab(AIML308P)19/May/25

What is a key benefit of utilizing the RMSprop optimizer in training neural networks?

• It maintains a constant learning rate across all parameters.

• It is less efficient in terms of memory usage compared to other optimizers.

• It adjusts the learning rate based on the average of recent gradients, which helps in dealing with non-stationary objectives.

• It does not allow for learning rate adjustments during training.

How does the Adagrad optimizer adapt the learning rate for each parameter during training?

• Adagrad maintains a constant learning rate throughout the training process.

• Adagrad adjusts the learning rate based on the historical gradients for each parameter.

• Adagrad reduces the learning rate exponentially over time.

• Adagrad modifies the learning rate by accumulating the squared gradients to scale the learning rate for each parameter.

What does NG acceleration stand for in optimization?

• Next Generation acceleration

• Nonlinear Gradient acceleration

• Newton's Gradient method

• Newton's Gradient acceleration

Describe the concept of gradient descent in machine learning.

• Gradient descent is a technique for data visualization in machine learning.

• Gradient descent is a method for increasing the loss function.

• Gradient descent is a type of neural network architecture.

• Gradient descent is an optimization algorithm that minimizes the loss function by iteratively adjusting model parameters in the direction of the negative gradient.

How does adaptive gradient differ from standard gradient descent?

• Adaptive gradient adjusts learning rates per parameter based on historical gradients, while standard gradient descent uses a fixed learning rate.

• Adaptive gradient and standard gradient descent are identical in their approach.

• Adaptive gradient uses a fixed learning rate for all parameters.

• Standard gradient descent adjusts learning rates based on historical gradients.

What are the key parameters of the ADAM optimizer?

• momentum, decay rate, batch size

• learning rate, alpha, gamma

• learning rate, beta1, beta2, epsilon, weight decay

• step size, beta1, beta3

When would it be more advantageous to use ADAM instead of RMSPROP?

• Use ADAM for problems with large datasets and complex models.

• Choose ADAM when computational resources are limited.

• ADAM is preferable for simpler problems with small datasets.

• Use ADAM only when you have no other optimization options.

What is the role of momentum in gradient descent optimization?

• Momentum slows down the optimization process.

• Momentum only affects the learning rate.

• Momentum helps accelerate gradient descent and reduces oscillations, leading to faster convergence.

• Momentum has no effect on convergence speed.

How does the learning rate affect the convergence of gradient descent?

• A higher learning rate always guarantees faster convergence.

• The learning rate affects convergence by controlling the step size; too small slows down convergence, while too large can cause divergence.

• The learning rate has no effect on convergence.

• The learning rate only affects the accuracy of the final result.

What are the potential drawbacks of using a fixed learning rate?

• Potential drawbacks of using a fixed learning rate include overshooting the optimal solution, slow convergence, inability to adapt to loss function changes, and suboptimal performance for different datasets.

• Using a fixed learning rate improves model accuracy for all datasets.

• Fixed learning rates always guarantee optimal solutions.

• A fixed learning rate adapts automatically to changes in the loss function.