We present new results to model and understand the role of encoder-decoder design in machine learning (ML) from an information-theoretic angle. We use two main information concepts, …

In the attention mechanism, as in the vanilla encoder-decoder model, the vector c is a single vector that is a function of the hidden states of the encoder. instead of being taken from the …

The simplest such score, called dot-product attention, implements relevance as similarity: measuring how similar the decoder hidden state is to an encoder hidden state, by computing …

Neural model with a sequence of discrete symbols as an input that generates another sequence of discrete symbols as an output. What is it good for? Neural decoder is a conditional …

The Encoder-Decoder architecture combines sequence-to-one and a one-to-sequence models: The encoder is “just” an ordinary RNN, producing a context as its result; The decoder is a …

Encoder-Decoder: (Neural) model that takes in a sequence of discrete symbols and generates another sequence.

Encoder-Decoder Architectures, Attention & Transformers Zachary Lipton & Henry Chai 10701 — November 15th