SummerSchool-Quiz8

Transformer models do not have recurrent units but can still perform sequence modeling.

• True

• False

As the number of training examples goes to infinity, your model will have:

• Low bias

• High Bias

• Same Bias

• Depends on the model’s variance

Compared to the encoder-decoder model which does not use an attention mechanism, we expect the attention model to have the greatest advantage when:

• The input sequence length is large.

• The input sequence length is small.

• The vocabulary size is large.

• The vocabulary size is small.

You have a friend whose mood is heavily dependent on the current and past few days’ weather. You’ve collected data for the past 365 days on the weather, which you represent as a sequence as x<1>, …, x<365>. You’ve also collected data on your friend’s mood, which you represent as y<1>, …, y<365>. You’d like to build a model to map from x→y. Should you use a Unidirectional RNN or Bidirectional RNN for this problem?

• Bidirectional RNN, because this allows the prediction of mood on day t to take into account more information

• Bidirectional RNN, because this allows backpropagation to compute more accurate gradients

• Unidirectional RNN, because the value of y<t> depends only on x<1>,…,x<t>, but not on x<t+1>,…,x<365>

• Unidirectional RNN, because the value of y depends only on x<t> , and not other days

In beam search, if you increase the beam width, which of the following would you expect to be true?

• Beam search will run more slowly

• Beam search will use up more memory

• Beam search will generally find better solutions

• Beam search will converge after fewer steps

• Beam search will run much faster as more options can be considered

How does decoder module of the transformer model avoid seeing the tokens that do not appear yet in output sequence?

• Multi-head attention

• Positional encoding

• Self attention

• Masking future positions

Which concept in transformer allows for inducing the sequence information in input tokens:

• Multi-head attention

• Positional encoding

• Self attention

• Masking future positions before the softmax step

Which of the following is a symptom of overfitting?

• Large estimated weights

• Good generalization to previously unseen data

• Simple decision boundary

• Complex decision boundary

Teacher forcing uses the actual output from the training dataset at time step t as input in the next time step (t+1), instead of the output generated by your model.

• True

• False