yokome.models.language_model

class yokome.models.language_model.LanguageModel(model_dir=None, params=None, seed=None, warm_start_from=None, production_mode=False, *, save_summary_steps=100, keep_checkpoint_max=5, language=None, vocabulary=None)

A neural language model that estimates the probability of a sentence.

Parameters

  • model_dir (str) – Where to store all relevant model data. If None, a generic location based on the current date and time will be used.

  • params (dict) – The model parameters.

  • seed (int) – The seed to use for the underlying Tensorflow graph.

  • warm_start_from (str) – A directory containing model parameter values for initialization.

  • production_mode (bool) – Whether to use the production or the training model.

  • save_summary_steps (int) – The periodicity at which to save summaries.

  • keep_checkpoint_max (int) – The maximum number of recent checkpoint files to keep.

  • language (yokome.language.Language) – The language of the language model.

  • vocabulary – A mapping from input units to integer values, or a sequence of input units. This is used to encode the incoming data numerically. If None, a pickled mapping is expected to be found in the model directory, named encoder.pickle. Every input unit that is not found in this vocabulary is considered to be an out-of-vocabulary unit.

estimate_probability(sentences, batch_size, sample_size=0)

Estimate the probability of the specified sentences.

Parameters

  • sentences – A sequence of sentences. Each sentence will be tokenized using the language object provided at language model creation.

  • batch_size (int) – The number of sentences to estimate the probability for in parallel.

  • sample_size (int) – The number of sentence alternatives to sample. If this is greater than zero, for every list of token candidates after tokenization, one token candidate is chosen for each sample. If it is 0, all sentence alternatives (i.e. all combinations of token candidates) are enumerated and used for probability estimation.

Returns

An iterable over one dictionary per sentence, each of the form

{
  'log2_word_probs': <word log-probabilities per alternative>,
  'log2_sentence_probs': <sentence log-probability per alternative>
}

production_dir()

Get the directory where the model for production is stored.

train(trn_set, evl_set, max_epochs=1, batch_size=1, max_generalization_loss=None, shuffle=False, random_state=None, verbose=False)

Train the model.

Parameters

  • trn_set – A sequence of sentences, a training set. Each sentence will be tokenized using the language object provided at language model creation.

  • trn_set – A sequence of sentences, an evaluation set. Each sentence will be tokenized using the language object provided at language model creation.

  • max_epochs (int) – The maximum number of epochs to train for. The actual number of epochs may be less if the training process stops early.

  • batch_size (int) – The number of sentences to estimate the probability for in parallel.

  • max_generalization_loss (float) – The maximum generalization loss at which the training process is still continued.

  • shuffle (bool) – Whether to shuffle the samples for each epoch.

  • random_state – The random state used for shuffling. May be a numpy.RandomState instance, an int seed, or None. If None, an unseeded pseudo-random number generator will be used.

  • verbose (bool) – Whether to show progress indicators.

training_dir()

Get the directory where the model for training is stored.

validate(vld_set, batch_size=1)

Evaluate the model performance on a validation set.

Parameters

  • vld_set – A sequence of sentences, a validation set. Each sentence will be tokenized using the language object provided at language model creation.

  • batch_size (int) – The number of sentences to estimate the probability for in parallel.

Returns

A dictionary containing the metrics evaluated on the validation set. Contains an entry 'loss' for the loss and an entry 'global_step' for the global step for which this validation was performed.

class yokome.models.language_model.SaverHook(model_dir)

A helper class that allows to save the model to one directory during training and to provide the best model from a different directory for production mode.

end(session)

Called at the end of session.

The session argument can be used in case the hook wants to run final ops, such as saving a last checkpoint.

If session.run() raises exception other than OutOfRangeError or StopIteration then end() is not called. Note the difference between end() and after_run() behavior when session.run() raises OutOfRangeError or StopIteration. In that case end() is called but after_run() is not called.

Args:

session: A TensorFlow Session that will be soon closed.