Fundamentals of Neural Networks - Activation Function

The video tutorial covers the basics of neural networks, starting with linear and logistic regression as motivations. It delves into the design of neural networks, including forward and backward propagation. The focus then shifts to activation functions, explaining their importance and how they can be linear or nonlinear. The tutorial discusses common activation functions like sigmoid, tanh, ReLU, and Leaky ReLU, highlighting their characteristics and applications. The video emphasizes the importance of choosing the right activation function based on the data and provides insights into how experienced data scientists make these decisions.

What is the primary motivation behind the design of neural networks?

• To replace traditional programming

• To mimic human brain functionality

• To enhance data storage capabilities

• To improve linear regression models

Which of the following is true about activation functions in neural networks?

• They are always linear

• They are only used in output layers

• They are chosen based on the data

• They are fixed for all models

What is the main reason for using activation functions in neural networks?

• To ensure outputs are binary

• To increase training speed

• To reduce model complexity

• To introduce non-linearity

What is the mathematical formulation of the sigmoid activation function?

• z^2 + 1

• max(0, z)

• e^(z) - e^(-z)

• 1 / (1 + e^(-z))

What is the range of the sigmoid activation function?

• -1 to 1

• 0 to 1

• -∞ to ∞

• 0 to ∞

Which activation function is most suitable for binary classification tasks?

• Sigmoid

• ReLU

• Tanh

• Linear

How does the range of the tanh activation function differ from the sigmoid function?

• It is narrower, from 0 to 0.5

• It is inverted, from 1 to -1

• It is the same, from 0 to 1

• It is wider, from -1 to 1

What is a key advantage of using the tanh activation function?

• It is computationally simpler

• It has a wider range than sigmoid

• It is always positive

• It is the default choice for all models

What is the primary characteristic of the ReLU activation function?

• It is a linear function

• It outputs only negative values

• It outputs values between -1 and 1

• It outputs the input directly if positive, otherwise zero

Why might one choose a leaky ReLU over a standard ReLU?

• To ensure all outputs are positive

• To allow some negative values

• To increase computational efficiency

• To simplify the model