Off-Policy Evaluation via Doubly Robust Estimators Using Historical Organic Landing Page Logs

Introduction: The Offline Evaluation Problem in Search Landing Pages

Deploying interactive layout configurations, dynamic component ordering, and personalized modular blocks on high-intent organic landing pages presents a fundamental optimization challenge: live A/B testing on search traffic risks exposing high-value search visits to inferior page variations. In organic acquisition, SERP visibility is hard-won, and algorithmic shifts punish drops in behavioral engagement metrics (such as dwell time, return-to-SERP rate, and conversion velocity). To sustainably increase organic revenue without degrading conversion rates or search engine ranking signals through unconstrained online exploration, machine learning systems must evaluate new policy candidates before routing production traffic to them.

Off-Policy Evaluation (OPE) provides the mathematical framework for assessing a new target policy $pi$ using observational interaction logs generated by a historical logging policy $mu$. However, naive offline metrics (such as historical average conversion rate per layout) suffer from catastrophic selection bias because past page allocations were non-random—they were dictated by previous heuristics, contextual rules, or earlier bandit iterations. This guide details the derivation, algorithmic implementation, and production edge cases of Doubly Robust (DR) estimation on historical organic search landing page logs to evaluate layout optimization models prior to live online execution.

Mathematical Formulation of Off-Policy Evaluation in Organic Search Contexts

Consider an organic landing page ecosystem where a session begins with an organic entry. The environment generates a multi-dimensional context vector $X in mathcal{X}$, drawn from an unknown distribution $P(X)$. The context vector encapsulates features observable at request time:

  • Query Intent & Semantics: Dense embeddings of the referring search query (when available via search referrer headers or Google Search Console landing-page-to-query attribution matrices), token counts, and classified intent categories (e.g., transactional, commercial investigation).
  • Client Environment: Device class, viewport dimensions, operating system, network latency profile, and geolocation.
  • Site Topology & Crawl State: Current page category depth, canonical tag status, crawl frequency cluster, and cached inventory levels of linked product catalogs.

Conditioned on context $X$, a historical logging policy $mu(a|X) = P(A = a | X)$ selects an action $A in mathcal{A}$ from a discrete set of structural page configurations (e.g., alternating between a product-grid-first layout, a testimonial-rich educational layout, or a dynamic filter-first module). The environment then yields an observed scalar reward $Y in mathbb{R}$, typically operationalized as Revenue Per Session (RPS) or profit margin per session. We record a historical trajectory log $mathcal{D} = {(X_i, A_i, P_i, Y_i)}_{i=1}^n$, where $P_i equiv mu(A_i|X_i)$ denotes the logged propensity score.

Our objective is to estimate the expected value (policy value) $V(pi)$ of an unreleased target policy $pi(a|X)$ designed to increase organic revenue:

$$V(pi) = mathbb{E}_{X sim P(X), A sim pi(cdot|X)} [Y]$$

The Failure of the Direct Method (DM)

The Direct Method trains a regression model $hat{q}(x, a) approx mathbb{E}[Y | X = x, A = a]$ on $mathcal{D}$ using loss functions such as mean squared error or Huber loss. The policy value is estimated as:

$$hat{V}_{DM}(pi) = frac{1}{n} sum_{i=1}^n sum_{a in mathcal{A}} pi(a | X_i) hat{q}(X_i, a)$$

While $hat{V}_{DM}$ exhibits low variance because it does not depend on inverse probabilities, it introduces immense statistical bias if the function class of $hat{q}$ fails to capture the true underlying data-generating process $mathbb{E}[Y|X, A]$—a condition almost guaranteed in high-dimensional search environments due to unobserved searcher intent variables.

The Variance Pathology of Inverse Propensity Scoring (IPS)

Conversely, Inverse Propensity Scoring (IPS) corrects for selection bias by re-weighting observed rewards by the Radon-Nikodym derivative of the target policy with respect to the logging policy:

$$hat{V}_{IPS}(pi) = frac{1}{n} sum_{i=1}^n frac{pi(A_i | X_i)}{mu(A_i | X_i)} Y_i$$

IPS is mathematically unbiased under the strict assumption of common support ($mu(a|x) > 0 implies pi(a|x) > 0$ for all $a, x$). However, in organic search routing, propensities $mu(A_i | X_i)$ for non-dominant layouts can be very small (e.g., $mu approx 0.001$). This causes the importance weights $w_i = frac{pi(A_i|X_i)}{mu(A_i|X_i)}$ to explode, leading to catastrophic estimator variance: $mathbb{V}[hat{V}_{IPS}(pi)] to infty$.

The Doubly Robust (DR) Estimator: Derivation and Properties

The Doubly Robust (DR) estimator integrates the Direct Method and Inverse Propensity Scoring by utilizing $hat{q}(X, A)$ as a baseline control variate to minimize the variance of IPS, while retaining counterfactual unbiasedness even when the reward model $hat{q}$ is misspecified.

Formally, the DR estimator for a stochastic target policy $pi$ is defined as:

$$hat{V}_{DR}(pi) = frac{1}{n} sum_{i=1}^n left( sum_{a in mathcal{A}} pi(a | X_i) hat{q}(X_i, a) + frac{pi(A_i | X_i)}{mu(A_i | X_i)} left( Y_i – hat{q}(X_i, A_i) right) right)$$

For a deterministic target policy where $pi(X_i) = a^*$, this reduces to:

$$hat{V}_{DR}(pi) = frac{1}{n} sum_{i=1}^n left( hat{q}(X_i, pi(X_i)) + frac{mathbb{I}(A_i = pi(X_i))}{mu(A_i | X_i)} left( Y_i – hat{q}(X_i, A_i) right) right)$$

Proof Sketch of Double Robustness

To verify the double robustness property, consider the conditional expectation of an individual DR observation score $Gamma_i$, conditioned on $X_i$:

$$mathbb{E}[Gamma_i | X_i] = sum_{a in mathcal{A}} pi(a | X_i) hat{q}(X_i, a) + mathbb{E}_{A_i sim mu, Y_i sim P(Y|X, A)} left[ frac{pi(A_i | X_i)}{mu(A_i | X_i)} (Y_i – hat{q}(X_i, A_i)) ,middle|, X_i right]$$

Expanding the second term via the law of iterated expectations:

$$sum_{a in mathcal{A}} mu(a | X_i) left[ frac{pi(a | X_i)}{mu(a | X_i)} (q^*(X_i, a) – hat{q}(X_i, a)) right] = sum_{a in mathcal{A}} pi(a | X_i) (q^*(X_i, a) – hat{q}(X_i, a))$$

Where $q^*(X_i, a) = mathbb{E}[Y_i | X_i, A_i = a]$. Combining both terms:

$$mathbb{E}[Gamma_i | X_i] = sum_{a in mathcal{A}} pi(a | X_i) hat{q}(X_i, a) + sum_{a in mathcal{A}} pi(a | X_i) q^*(X_i, a) – sum_{a in mathcal{A}} pi(a | X_i) hat{q}(X_i, a) = sum_{a in mathcal{A}} pi(a | X_i) q^*(X_i, a)$$

Notice that:

  • Case 1: The outcome model is correct ($hat{q} = q^*$). The residuals $(Y_i – hat{q}(X_i, A_i))$ have mean zero. The second term vanishes in expectation regardless of errors in the logging policy estimation $hat{mu}$, ensuring unbiasedness.
  • Case 2: The logging policy is correct ($mu(A_i|X_i)$ is exact). The importance weight cancels the expectation denominator $mu(a|X_i)$, causing the model terms $sum pi hat{q}$ to subtract out cleanly, leaving only the ground-truth value $sum pi q^*$, ensuring unbiasedness even if $hat{q}$ is wildly incorrect.

Addressing Search-Specific Observational Biases and Practical Edge Cases

Applying the DR estimator to organic landing page datasets violates several naive laboratory assumptions. Search traffic logs exhibit structural anomalies that require domain-specific adjustments.

1. Positivity Violations and Propensity Clipping

Deterministic routing rules historically embedded in ecommerce platforms (e.g., “always show desktop users in geo-US layout A”) completely violate the common support assumption ($mu(A_i|X_i) = 0$). To prevent the variance from dominating the Mean Squared Error (MSE) of the estimator, apply Weight-Trimmed Doubly Robust (DR-w) or Clipped Doubly Robust estimation:

$$w_i^{clip} = minleft( frac{pi(A_i | X_i)}{mu(A_i | X_i)}, M right)$$

The threshold $M$ introduces a controlled amount of bias in exchange for a massive reduction in sample variance, bounded by optimizing the bias-variance trade-off over Pareto frontiers:

$$M^* = argmin_M left( widehat{text{Bias}}(hat{V}_{DR}(M))^2 + widehat{mathbb{V}}(hat{V}_{DR}(M)) right)$$

2. Filtering Non-Human Search Traffic

Search engine web crawlers (Googlebot, Bingbot) and third-party scraping bots evaluate static markup and render client-side DOMs, generating false session logs with zero revenue ($Y_i = 0$). If included in $mathcal{D}$, these non-transacting sessions corrupt $hat{q}(X, A)$ by inflating contextual zero-inflation boundaries. Logs must undergo deterministic verification (reverse DNS lookup validation of IP blocks for Googlebot, signature-based crawler heuristics) to cleanse the training corpus down to genuine human organic traffic.

3. Extreme Non-Stationarity in Query Clusters

Organic query volume fluctuates with seasonal consumer interest and SERP feature churn (e.g., changes to Google AI Overviews or sponsored ad expansions). As a result, $P(X)$ shifts across time. To handle covariate shift without invalidating OPE:

  • Segment off-policy evaluation across rolling historical windows (e.g., 14-day sliding evaluations).
  • Implement density ratio reweighting: compute $w_{text{shift}}(X) = frac{P_{text{eval}}(X)}{P_{text{train}}(X)}$ using an auxiliary classifier to downweight obsolete contextual distributions.

Production Implementation: Cross-Fitting and High-Dimensional Vectorization

A frequent error in off-policy landing page modeling is training $hat{q}$ and evaluating $hat{V}_{DR}$ on the same dataset. This induces an in-sample overfitting bias that overestimates the expected lift of candidate layouts. To guarantee valid asymptotic properties, production pipelines must execute $K$-fold cross-fitting (DML style):

  1. Partition log dataset $mathcal{D}$ into $K$ disjoint folds ${I_k}_{k=1}^K$.
  2. For each fold $k$, train the outcome regression model $hat{q}^{(-k)}$ on $mathcal{D} setminus I_k$.
  3. Evaluate the doubly robust scores on the held-out partition $I_k$ using $hat{q}^{(-k)}$.
  4. Aggregate cross-fitted scores across all partitions to produce the final estimate.
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import KFold

class DoublyRobustLandingPageEvaluator:
    def __init__(self, clip_max=30.0, n_splits=5):
        self.clip_max = clip_max
        self.n_splits = n_splits

    def evaluate(self, X, A, P, Y, target_policy_probs):
        """
        X: np.ndarray, shape (N, D) - Context features (query, geo, device)
        A: np.ndarray, shape (N,) - Categorical action indices taken by mu
        P: np.ndarray, shape (N,) - Propensities mu(A_i | X_i)
        Y: np.ndarray, shape (N,) - Session revenue (RPS)
        target_policy_probs: np.ndarray, shape (N, K) - Probability distribution over actions under pi
        """
        N, num_actions = target_policy_probs.shape
        q_hat_actions = np.zeros((N, num_actions))
        
        kf = KFold(n_splits=self.n_splits, shuffle=True, random_state=42)
        
        # Cross-fitting loop to avoid in-sample bias
        for train_idx, test_idx in kf.split(X):
            X_train, X_test = X[train_idx], X[test_idx]
            A_train, A_test = A[train_idx], A[test_idx]
            Y_train = Y[train_idx]
            
            for a in range(num_actions):
                mask = (A_train == a)
                if np.sum(mask) > 10:
                    model = GradientBoostingRegressor(n_estimators=100, max_depth=4)
                    model.fit(X_train[mask], Y_train[mask])
                    q_hat_actions[test_idx, a] = model.predict(X_test)
                else:
                    # Fallback for sparse action contexts
                    q_hat_actions[test_idx, a] = np.mean(Y_train) if len(Y_train) > 0 else 0.0

        # Calculate importance weights with clipping
        pi_observed = target_policy_probs[np.arange(N), A]
        importance_weights = pi_observed / np.clip(P, 1e-6, 1.0)
        clipped_weights = np.minimum(importance_weights, self.clip_max)
        
        # Expected reward under target policy according to outcome model
        v_model = np.sum(target_policy_probs * q_hat_actions, axis=1)
        
        # Residual error on observed historical action
        observed_q = q_hat_actions[np.arange(N), A]
        residuals = Y - observed_q
        
        # Doubly robust per-sample score
        dr_scores = v_model + clipped_weights * residuals
        
        v_dr = np.mean(dr_scores)
        variance_dr = np.var(dr_scores, ddof=1) / N
        ci_95 = 1.96 * np.sqrt(variance_dr)
        
        return {
            "policy_value": float(v_dr),
            "standard_error": float(np.sqrt(variance_dr)),
            "ci_lower": float(v_dr - ci_95),
            "ci_upper": float(v_dr + ci_95)
        }

Deployment Strategy: Gating Live Bandit Execution

Off-policy evaluation should not replace live experimentation; it acts as a risk-mitigation filter. Candidate policies trained to personalize layouts must clear quantitative DR-value thresholds offline before exposure to live user sessions.

Once a candidate policy proves counterfactually superior—demonstrating a statistically significant improvement in offline revenue per session over the active logging baseline—it can safely be promoted to online contextual exploration. This transition is managed by multi-armed bandit systems designed to dynamically route search sessions while safeguarding organic rank metrics, as detailed in our guide on Multi-Armed Bandit Optimization of Dynamic Intent-Matched Landing Page Modules for RPS Growth.

By enforcing this offline verification pipeline, search engineering teams prevent flawed algorithmic layout policies from degrading the user experience on key landing pages. Instead, logging, evaluating via doubly robust estimation, and deploying through bounded online bandits forms a closed-loop system designed to safely and systematically increase organic revenue.

Related Guides in This Series

Tagged