ClsPooler(self, embedding_dim:int=None, cls_is_last_token:bool=False)

Just takes the first vector from a list of vectors (which in a transformer is typically the [CLS] token) and returns it. For BERT, it's recommended to use BertPooler instead.

Registered as a Seq2VecEncoder with name "cls_pooler".


  • embedding_dim: int, optional This isn't needed for any computation that we do, but we sometimes rely on get_input_dim and get_output_dim to check parameter settings, or to instantiate final linear layers. In order to give the right values there, we need to know the embedding dimension. If you're using this with a transformer from the transformers library, this can often be found with model.config.hidden_size, if you're not sure.
  • cls_is_last_token: bool, optional The [CLS] token is the first token for most of the pretrained transformer models. For some models such as XLNet, however, it is the last token, and we therefore need to select at the end.


ClsPooler.forward(self, tokens:torch.Tensor, mask:torch.BoolTensor=None)

Defines the computation performed at every call.

Should be overridden by all subclasses.

.. note:: Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.


ClsPooler.get_input_dim(self) -> int

Returns the dimension of the vector input for each element in the sequence input to a Seq2VecEncoder. This is not the shape of the input tensor, but the last element of that shape.


ClsPooler.get_output_dim(self) -> int

Returns the dimension of the final vector output by this Seq2VecEncoder. This is not the shape of the returned tensor, but the last element of that shape.