2026-03-08 - Dev Log

Bandwidth Prediction — Gemini-CLI Iteration Review

Revisited the bandwidth prediction project. This iteration was started with Gemini CLI as a standalone exploration in c0smere_devops/bandwidth-prediction/, separate from the earlier ZenML pipeline work. Data source is the same (secv_scrape on stygies), but this version explores a wider variety of models.

Models Explored

Model	Type	MAE (GB/hr)	R²	Notes
Tuned XGBoost (grid search)	Regression	4.09	0.088	Best performer. `lr=0.01, depth=3, subsample=0.8`
Highres XGBoost	Regression	4.40	0.163	Cyclical time features + rolling stats
XGBoost (simple)	Regression	4.32	0.061	Basic feature set for comparison
Random Forest	Regression	4.79-5.02	~-0.02	Weaker than XGBoost across the board
Prophet	Forecasting	5.18	-0.02	Pure seasonality, no lag features
Probabilistic (quantile GBR)	Uncertainty	—	—	74.8% of actuals within 10th-90th bounds
Classification (XGBoost)	3-class	—	53% acc	Low/Medium/High usage categories
HMM (4-state Gaussian)	Unsupervised	—	—	Idle/Light/Heavy/Massive state discovery

Bugs Found & Fixed (Claude Review)

Data leakage — model.py, classification_model.py, daily_model.py all used train_test_split() which randomly shuffles time series data, letting future data leak into training. Replaced with temporal splits everywhere.
Counter reset handling — eda.py and trends.py used the full cumulative counter value on SNMP counter resets, injecting bogus multi-GB spikes as single-hour “usage”. Now correctly marked as NaN.
Prophet eval on training data — was measuring fit quality, not prediction quality. Now uses 90/10 held-out split.
Grid search used test data — tuned_xgboost.py ran CV on all data including the test set. Now restricted to training data only.
Broken plot — model.py had plot lines commented out, saved blank figures.
Hardcoded DB password — moved to env var (SECV_DB_PASSWORD).
Negative predictions — added np.clip(..., 0, None) everywhere since bandwidth can’t be negative.
Integer month encoding — replaced with cyclical sin/cos in tuned_xgboost.py (Dec→Jan is distance 1, not 11).
Memory leaks — added plt.close() after every savefig() across all 12 scripts.

Key Observations

R² is low across the board — this is expected with honest temporal splits. The high variance in the data (std ~8.4 GB/hr vs mean ~5.9 GB/hr) makes point prediction inherently hard.
Lag features dominate importance — lag_1h is consistently the #1 feature. The models are mostly capturing short-term autocorrelation. This matches the earlier ZenML iteration’s finding that recursive forecasting beyond 2-3 hours causes oscillation.
Probabilistic approach most practical — the quantile regression model’s 74.8% coverage is arguably more useful than any single-point prediction.
HMM state discovery is interesting — found 4 clean states: Idle (1.1 GB/hr, 16%), Light (3.3 GB/hr, 48%), Heavy (7.2 GB/hr, 31%), Massive (28.1 GB/hr, 6%).

Potential Next Steps

Log-transform target — the data is heavily right-skewed; log-scaling could help
Train only on recent data — 2024-2026 usage patterns differ significantly from 2015-2023 (visible in trend analysis). Older data may be noise.
Data cap feature — historical cap utilization influenced behavior, could have predictive value (as noted in June 2025 dev log)
Multi-output regression — predict upstream + downstream separately instead of just total
Deploy as API — similar to the ZenML iteration’s FastAPI container approach

Project Structure

All files in c0smere_devops/bandwidth-prediction/:

fetch_data.py → eda.py → model scripts (data pipeline)
5 trained models in models/, best is tuned_xgb_bandwidth_model.pkl (232KB)
24+ plots in plots/, interactive dashboard in dashboard/index.html