Knoebels Wait Time Model 2026-05-29

Tags: #MachineLearning #LinearAlgebra #DataScience #Python #XGBoost #FeatureEngineering

1. Pipeline Architecture: The `.pipe()` Method

Instead of messy nested functions or temporary variables, professional Data Engineering uses pandas .pipe(). It processes data like a conveyor belt, maintaining pure readability.

# The Master Pipeline
df_model_ready = (df
    .pipe(create_dummies_for_existing)
    .pipe(fill_height_nulls)
    .pipe(add_time_features)
    .pipe(add_promo_features)
    .pipe(finalize_df)
)

Rule of Thumb: Use .assign() inside these functions to create new variables without overwriting the original dataframe.

2. Feature Engineering & The Rules of Linear Algebra

Linear Regression algorithms are just arithmetic engines solving $(A^T A)^{-1} A^T y$ . They require strict matrix hygiene.

The Dummy Variable Trap (Perfect Multicollinearity)

If you one-hot encode a category (like day_of_week), you must drop one column to act as the baseline (e.g., Sunday).

The Math: If you don’t drop a column, the sum of the dummy columns perfectly equals the Intercept column (a column of 1s). Your columns are no longer mathematically independent.
The Result: The matrix loses its Rank, a Null Space is created, $(A^T A)$ becomes singular (non-invertible), and the Python code crashes.

Interaction Terms (The “AND” Gate)

Linear regression only knows how to add constants. It cannot naturally calculate an AND condition (e.g., “Is it Friday AND a Thrill Ride?”).

The Fix: You must do the non-linear multiplication for the algorithm.
Example: interaction = bargain_night * is_thrill_ride. This allows the model to assign a specific $\beta$ coefficient to the unique synergy of those two events (e.g., proving coaster lines shrink on wristband nights).

Domain-Logic Imputation

Never blindly fill NaN values with averages (e.g., filling a missing maximum height with the park average).

Use domain knowledge: If minimum_height is Null, it physically means 0 inches (anyone can ride). Fill with 0.

3. Classic Linear Regression (The Inference Model)

Use statsmodels (not scikit-learn) for Inference. It cracks open the black box and shows you the universal laws of the dataset.

Interpreting the OLS Summary

Intercept ( $\beta_0$ ): The baseline wait time when all other dummy variables are 0. Pro-tip: Intentionally drop a highly stable, average ride (like the Grand Carousel) so your Intercept becomes a relatable baseline.
Coefficients (coef): The exact number of minutes a feature adds or subtracts from the baseline.
P-value (P>|t|): If $< 0.05$ , the math is highly confident this isn’t random noise.
R-squared: The percentage of human behavior/variance your model successfully explains (e.g., 0.34 = 34%).

Diagnostics & The “Silent Killer”

Skew / Kurtosis: Tests if your errors form a perfect Bell Curve. (Human behavior rarely does, hence long tails/outliers).
Condition Number: The ratio of the largest SVD singular value to the smallest. If > 1,000, your matrix is “ill-conditioned.”
Variance Inflation Factor (VIF): A score that detects highly correlated (but not perfectly identical) columns.
- If VIF > 10, the columns are fighting each other for the same $\beta$ weight. Drop one.
- Note: Mixing large scales (inches: 0-120) with small scales (booleans: 0-1) can artificially inflate the Condition Number even if VIF is healthy.

4. Tree-Based Models (The Prediction Model)

Transitioning from ISLP Chapter 3 to Chapter 8.

XGBoost vs. Linear Regression

Math difference: XGBoost doesn’t invert matrices. It uses logical If/Then splits.
Robustness: XGBoost does not care about the Dummy Variable Trap, collinearity, or NaN values. It just ignores redundant data and splits on the rest.

The Bias-Variance Tradeoff (Hyperparameter Tuning)

If a tree grows too deep, it memorizes the training data (High Variance / Overfitting) and fails in the real world. We tune knobs to find the sweet spot:

max_depth: How many “AND” conditions the tree can chain together.
n_estimators: The number of sequential trees built to correct the errors of the previous ones.
learning_rate ( $\eta$ ): Forces the model to take tiny, cautious steps so it doesn’t over-correct.

Bayesian Optimization (`BayesSearchCV`)

Grid Search is exhaustive and blind. Bayesian Optimization builds a lightweight probability map (Gaussian Process) of the error landscape.

How it works: It balances Exploration (testing blank areas of the map) with Exploitation (drilling down into known good areas).
The Result: It finds the optimal XGBoost hyperparameters in ~30 tries instead of 300+.

# Standard Bayesian Optimization Implementation
from skopt import BayesSearchCV
from skopt.space import Real, Integer
from xgboost import XGBRegressor

search_spaces = {
    'max_depth': Integer(3, 12),
    'n_estimators': Integer(100, 1000),
    'learning_rate': Real(0.01, 0.3, 'log-uniform')
}

bayes_search = BayesSearchCV(
    estimator=XGBRegressor(random_state=42),
    search_spaces=search_spaces,
    n_iter=30, # Finds the best settings in just 30 combinations
    cv=3,
    scoring='neg_mean_absolute_error'
)