2025-06-01 - Dev Log

I’m so far out into the weeds on this

So I’m doing this to get a feel for MLOps work, and instead I’m larping as a data scientist because I want this model to work well so I can actually make a neat little viz out of my bandwidth meter…

Oh well, I can’t stop now. Also, these are particularly engaging concepts, and I enjoy learning for its own sake.

I think when I get home today I’ll install Pop!_OS on one of my extra hard drives in my desktop so that I can leverage the RTX 2080 I have for training XGBoost models.

Fail

Pop!_OS ended up being a waste of my time. It would boot and GDM would hang. I could get to a TTY with ctrl + alt + F2 and I spent too much time trying to troubleshoot this. I suppose I might have gotten my environment set up and running like that, but, if I’m just going to have an OS with no working desktop environment installed I’ll just use Arch btw…

XGBoost

Initially I wanted to get Linux installed on one of my spare drives so that I could set up the environment from my server the exact same way on my desktop, which would let me utilize my RTX 2080 for training the XGBoost models. However, I noticed that of the two models I was training, the Random Forest model was taking up 90% of the execution time. That fact coupled with the frustrations with Pop_OS led me to just focus on XGBoost for now and how I might improve that a little bit before moving on to other topics.

Feature engineering continued

I was doing some reading and decided to try a few things:

• Expanding Window Statistics - Just a mean or stdev that begins in row 1 and grows over time including all prior rows. I thought this sounded interesting but it did not end up being useful in my case
• Rate of Change - This is like t_rate_lag_1h but instead of the previous value it’s the difference of the current value and the lagged value. This one (the 1 hour specifically) did have a large impact on the error rate. This kind of surprised me given how tightly coupled (it should be) to the existing 1h lagged feature.
• Fourier Transforms - I didn’t end up trying this, sounds interesting though. I think you would need a more cyclical dataset for this to really be useful

Ultimately I was able to save another 100kb off the error rates:

XGBoost Mean Absolute Error (MAE):      163,722
XGBoost Root Mean Squared Error (RMSE): 336,143

XGBoost Feature importance (gain):
t_rate_lag_1h:            3,209,611,968,512
t_rate_lag_2h:            2,004,976,795,648
t_rate_diff_1h_lag_1h:       76,138,577,920
t_rate_diff_2h:              39,309,844,480
t_rate_roll_mean_24h:        29,012,504,576
t_rate_diff_1h:              20,849,551,360
hour_sin:                     3,578,800,896
hour_cos:                     2,848,401,152
day_of_week_sin:              2,526,856,704
hour:                         2,481,849,344
month:                        2,194,845,952

Hyperparameter tuning enhancements

Now using Optuna for parameter tuning now:

def objective(trial):
    param = {
            'verbosity': 0,
            'objective': 'reg:squarederror',
            'tree_method': 'hist',  # CPU-friendly, use 'gpu_hist' if you have GPU
            'n_estimators': trial.suggest_int('n_estimators', 100, 800),
            'max_depth': trial.suggest_int('max_depth', 3, 10),
            'learning_rate': trial.suggest_float('learning_rate', 0.005, 0.2, log=True),
            'subsample': trial.suggest_float('subsample', 0.5, 1.0),
            'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
            'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 1.0, log=True),
            'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 1.0, log=True),
            'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
            'gamma': trial.suggest_float('gamma', 0, 0.5),
    }

    tscv = TimeSeriesSplit(n_splits=5)
    maes = []

    for train_idx, val_idx in tscv.split(X_train):
        # Use the stored indices to slice the NumPy arrays
        X_tr, X_val = X_train[train_idx], X_train[val_idx]
        y_tr, y_val = y_train[train_idx], y_train[val_idx]

        model = xgb.XGBRegressor(**param)
        model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False) 
        preds = model.predict(X_val)
        maes.append(mean_absolute_error(y_val, preds))

    return np.mean(maes)

# 3. Run Optuna Optimization
study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=40)  # Adjust n_trials based on your resources

print("Best trial:")
print(study.best_trial.params)

# 4. Train the Final Model with the Best Hyperparameters
best_params = study.best_trial.params
final_model = xgb.XGBRegressor(**best_params, objective='reg:squarederror', tree_method='hist')
final_model.fit(X_train, y_train)

Anyway lets actually use the model

Recursive forecasting w/ lag features

Given that the features assigned the highest importance values are lagged features, I don’t think we can get away with simply using the last known value if we want to generate worthwhile predictions beyond a couple of days so…

Basically we’re going to slap the logic from our feature engineering step into a for loop so we can use the prediction output as we go as part of the lagged/windowed features.

The initial results were somewhat disappointing:

It seems to me like the model is oscillating between a handful of values. Given that the model is dominated by lag features I think it’s having trouble generalizing when used in a recursive manner for generating predictions. Incidentally, looking at what the total consumed bandwidth would be on June 30th @ 23:00 according to the model: 4333.77 GB This is an absolutely plausible monthly total for me; so I guess there’s that.

Next Steps

In any case I’m going to proceed with implementing nightly re-training, since new actual data is coming in every hour. I’m also going to work on deploying this model in docker with some endpoints exposed like /get7dayprediction. The API will handle querying postgres for the required historical data, feature engineering, and return the predictions.

Direct Multi-step Forecasting - Instead of recursive prediction, training the model to predict all 30 future steps at once (multi-output regression). This sounds interesting.

Try Prophet/SARIMA models