Home Internet Bandwidth Analysis 💻

Project Idea 1: Bandwidth Usage Forecasting System 📈

• 2025-05-30 - Dev Log 1
• 2025-05-30 - Dev Log 2
• 2025-05-31 - Dev Log
• 2025-06-01 - Dev Log
• 2025-06-05 - Dev Log
• 2026-03-08 - Dev Log

• Goal: Predict your future bandwidth consumption (e.g., daily, weekly, or even hourly if your data granularity supports it).
• ML Problem: Time Series Forecasting.
• Potential Models:
  • Classical: ARIMA, SARIMA, Exponential Smoothing.
  • Machine Learning: Random Forest, Gradient Boosting (with lagged features, time-based features).
  • Deep Learning: LSTMs, GRUs (if you want to explore this).
  • Libraries: Facebook Prophet, Sktime.
• MLOps Angle & Demo Points (with ZenML):
  1. Data Ingestion Step: Connects to your PostgreSQL DB, queries data, outputs as DataFrame/CSV.
     • Homelab Consideration: ZenML pipeline directly accesses local PostgreSQL.
  2. Data Validation Step: Use Great Expectations (via ZenML integration) to validate incoming data (missing timestamps, non-negative usage, expected ranges).
  3. Preprocessing & Feature Engineering Step:
     • Handle timestamps (extract hour, day of week, month, year, is_weekend).
     • Create lagged features (usage from T-1, T-2).
     • Create rolling window statistics (e.g., 7-day rolling average).
  4. Model Training Step: Train chosen forecasting model. ZenML tracks model, hyperparameters, and metrics (e.g., MAE, RMSE, MAPE) using experiment tracking (e.g., local MLflow).
  5. Model Evaluation Step: Evaluate on a hold-out test set using time-series-aware cross-validation.
  6. (Optional) Model Deployment Step:
     • Deploy as a simple local REST API (FastAPI/Flask, Dockerized) returning forecasts. ZenML custom deployment step.
     • Or, a “batch inference pipeline” running on a schedule (ZenML schedulers) to generate and store forecasts.
  7. Visualization: Simple web app (Streamlit/Dash) to show:
     • Historical usage.
     • Model’s forecasts.
     • Could be a ZenML pipeline step generating an HTML report or a separate app.
  8. Retraining Pipeline: Demonstrate a ZenML pipeline for retraining on new data (manual or scheduled).

Project Idea 2: Anomaly Detection in Bandwidth Usage ⚠️

• Goal: Identify unusual patterns in bandwidth usage (network issues, unexpected device activity).
• ML Problem: Time Series Anomaly Detection.
• Potential Models/Techniques:
  • Statistical methods (e.g., 3-sigma rule on rolling mean/std).
  • Isolation Forest.
  • One-Class SVM.
  • Autoencoders (reconstruction error as anomaly score).
• MLOps Angle & Demo Points (with ZenML):
  1. Data Ingestion & Validation: Similar to the forecasting project.
  2. Preprocessing & Feature Engineering: Similar, but features might focus on deviations.
  3. Model Training Step: Train anomaly detection model. For unsupervised, “training” might be fitting to “normal” data.
  4. Threshold Tuning/Evaluation Step:
     • If labeled anomalies exist (even manual), evaluate with precision/recall.
     • Otherwise, set threshold for anomaly scores and visually inspect.
  5. Inference Pipeline/Deployment Step:
     • ZenML pipeline runs daily, fetches latest data, flags anomalies.
     • Could deploy a service to check recent usage.
     • Could trigger notifications (email, log) on anomaly detection.
  6. Visualization: Dashboard showing:
     • Historical usage with anomalies highlighted.
     • Anomaly scores over time.

Why This is a Great Portfolio Dataset ✨

• Real-World Relevance: Everyone understands internet usage.
• Time-Series Complexity: Showcases skills in handling time-dependent data.
• Full MLOps Lifecycle: You can implement nearly the entire MLOps lifecycle.
• Personal & Relatable: Your data makes it engaging; you can tell stories around patterns.
• Homelab Friendly: Doable on your server without initial cloud costs, extendable later.

Initial Steps to Get Started 🚀

1. EDA (Exploratory Data Analysis):
   • Connect to Postgres (e.g., psycopg2, sqlalchemy).
   • Load into Pandas DataFrame.
   • Determine granularity (hourly, daily?).
   • Plot the time series: look for trends, seasonality, outliers.
   • Check for missing data and plan handling.
2. Define a Specific Goal: Choose one project (forecasting is often a good start).
3. Set up ZenML: Install, initialize a local repository (zenml init).
4. Start Simple:
   • Create your first ZenML step for data ingestion.
   • Gradually add more steps to your pipeline.