Transformers Quiz

What does autoregressive mean?

• It refers to a model that predicts future values based on past values.

• It refers to a model that uses external variables to make predictions.

• It refers to a model that only considers the most recent data point.

• It refers to a model that doesn't rely on historical data.

What is attention in the context of Machine Learning?

• A mechanism that allows the model to generate shorter output sequences

• A feature of the model that allows it to train faster

• The encoder of a model

• A mechanism that allows the model to focus on specific parts of the input

What is masking?

• To increase the dimensionality of the input data

• Hiding certain parts of the input sequence

• Hiding some of the model's parameters

• Hindering the model's ability to learn effectively

What does an embedding dimension/layer?

• Transform discrete encoded tokens into continuous vector representations

• A specific attention mechanism for transformers

• The sequence length of input text

• Increase the dimensionality of the input sequence

What is self-attention?

• When every token attends to itself in the input sequence

• When every token attends to every other token in the same input sequence

• When every token attends to every other token in different input sequence

• When every token attends only to itself

What is cross attention

• When every token attends to itself in the input sequence

• When every token attends to every other token in the same input sequence

• When every token attends to every other token in different input sequence

• When every token attends only to itself

What is the point of the skip connections found in the transformer architecture?

• They reduce the number of computations required, speeding up training

• They improve the interpretability of the model's predictions

• They provide shortcuts for gradients to flow directly through the network, addressing the vanishing gradient problem

• They increase the depth of the model, allowing it to learn more complex features

What is Multi-head attention?

• It is a type of attention mechanism that only focuses on one part of the input at a time

• It refers to the model's ability to process multiple sequences of data in parallel

• It is a mechanism that allows the model to pay attention to multiple parts of the input simultaneously

• It is a technique used to speed up the training process of the Transformer