PPMI Filtering: The Mathematics of Domain Relevance

What is PPMI?

Positive Pointwise Mutual Information (PPMI) measures the statistical association strength between a token $w$ and a target enterprise domain $D$:

PPMI(w, D) = max( 0, log2( P(w, D) / ( P(w) * P(D) ) ) )

Where:
  P(w, D) = Probability of token w occurring within the domain dataset D
  P(w)    = Marginal probability of token w in a standard general English corpus
  P(D)    = Prior probability of the domain subset
    

Intuition: General vs Domain Specific Vocabulary

Tokens that occur uniformly across all natural language receive a PPMI close to zero, whereas terminology that clusters specifically inside your enterprise dataset receives a high score:

Python Reference Implementation

Below is the algorithmic core utilized inside moro.data.compiler to score token distributions:

import numpy as np
from collections import Counter

def calculate_ppmi(
    domain_tokens: Counter,
    general_tokens: Counter,
    total_domain: int,
    total_general: int,
) -> dict[str, float]:
    """Calculate Positive Pointwise Mutual Information for domain vocabulary."""
    ppmi_scores = 

    for token, domain_count in domain_tokens.items():
        # Probability of token within enterprise domain:
        p_w_domain = domain_count / total_domain

        # Background probability across general corpus:
        general_count = general_tokens.get(token, 1)
        p_w_general = general_count / total_general

        # Calculate Pointwise Mutual Information:
        pmi = np.log2(p_w_domain / p_w_general)
        ppmi_scores[token] = float(max(0.0, pmi))

    return ppmi_scores
  

Tuning PPMI Thresholds in moro.yaml

Customize the sensitivity of the MI Guard inside your configuration:

dataset:
  source: ./data/raw/contracts.jsonl
  # Minimum PPMI for a sample to be marked domain-relevant:
  min_ppmi: 1.0

  # PPMI threshold required to trigger an MI Guard override on rare clusters:
  mi_guard_ppmi_threshold: 2.0