AgriYield MLOps
A production-grade ML-powered crop yield prediction platform for UK agricultural fields. Combines live weather data, soil attributes, and an XGBoost regression model to predict yield (kg/ha) per field, served via a real-time FastAPI backend and an interactive Leaflet map dashboard.
Features
Interactive UK field map: 113 fields coloured by predicted yield (red → green)
XGBoost yield model: trained on real UK yield data (CYCleSS, 934 fields) or synthetic fallback, baked into the Docker image at CI time
Live weather integration: Open-Meteo API fetches real-time temperature, precipitation, and solar radiation per field
Confidence intervals: every prediction includes an 80% CI band
Drift detection: PSI-based feature drift monitoring with per-field warning badges
CI/CD pipeline: GitHub Actions trains the model, builds the Docker image, pushes to GHCR, and deploys to Render on every push to main
Automated tests: pytest runs in CI before build; /health verified post-deploy
Smart caching: Redis (with in-memory fallback) caches weather API responses for 1 hour
Prometheus metrics: /metrics endpoint for Grafana integration
Architecture
GitHub Actions CI trains and packages the model into a Docker image, which is deployed on Render. The FastAPI backend serves predictions using live weather data and cached responses.
ML Model
XGBoost Regressor trained on real UK yield data (CYCleSS, 934 field-year records). Features include lat, lon, area, crop type, soil type, region, weather vars, ET₀, soil moisture, NDVI, and week of year. RMSE ~1759 kg/ha on holdout.
Live Weather Ingestion
Open-Meteo API fetches temperature, precipitation, and solar radiation per field at inference time. Fallback chain: Redis cache → Live API → In-memory → UK seasonal defaults.
FastAPI Serving
GET /fields predicts all 113 fields with confidence intervals. GET /metrics exposes Prometheus endpoints. Static HTML/JS dashboard served via Leaflet.js map.
CI/CD Pipeline
GitHub Actions generates training data, trains XGBoost, bakes model.pkl into Docker image, pushes to GHCR, and triggers Render deploy, all on every push to main.
Monitoring & Drift
PSI (Population Stability Index) computed per feature on every /fields call. Fields with PSI > 0.2 display a DRIFT warning badge. Prometheus metrics available at /metrics.
Smart Caching
Redis caches weather API responses for 1 hour with in-memory fallback. Ensures the free-tier Render instance stays responsive under load without hammering the weather API.
ML Model
XGBoost Regressor trained end-to-end in CI: no stale artefacts, no manual uploads.
Algorithm: XGBoost Regressor
Target: yield_kg_per_ha
Features: lat, lon, area, crop type, soil type, region, temperature, precipitation, solar radiation, ET₀, soil moisture, NDVI, week of year
Train/test split: 80/20
RMSE: ~1759 kg/ha (real CYCleSS data); ~1724 kg/ha (synthetic fallback)
Confidence interval: ±15% of prediction (configurable via CI_WIDTH env var)
The model is retrained from scratch on every CI run. The trained model.pkl is baked directly into the production Docker image: no stale artefacts, no manual uploads.
Live Weather Integration
Weather data is fetched from the Open-Meteo API (free, no API key required) for each field's coordinates.
Features used at inference time:
- Temperature (°C): temperature_2m_max today
- Precipitation (mm): precipitation_sum today
- Solar radiation (MJ/m²): shortwave_radiation_sum today
Fallback chain: Redis cache → Live API → In-memory cache → UK seasonal defaults. The STALE DATA badge appears when defaults are used.
Project Structure
agri-yield/
├── .github/workflows/deploy.yml # Train → Build → Push → Deploy
├── data/seed/uk_fields.csv # 113 UK farm fields with metadata
├── ingestion/openmeteo_live.py # Live weather fetcher with caching
├── serving/
│ ├── app.py # FastAPI app + /fields endpoint
│ ├── model.py # Model loader + predict()
│ └── static/ # Dashboard HTML/CSS/JS
├── training/
│ ├── train_and_export.py # XGBoost training (real-data first, fallback to synthetic)
│ ├── prepare_real_data.py # CYCleSS real data ingestion
│ └── utils/features.py # Canonical FEATURE_COLS
├── monitoring/psi_detector.py # PSI drift detector
├── training/prepare_real_data.py # Real UK yield data pipeline (CYCleSS)
├── generate_data.py # Synthetic data generator (fallback)
├── Dockerfile.prod # Production image (bakes model.pkl)
└── pyproject.toml
CI/CD Pipeline
Every push to main triggers a fully automated pipeline.
push to main
│
▼
[1] Generate training data (real CYCleSS or synthetic fallback)
│
▼
[2] Train XGBoost → save model.pkl
│
▼
[3] Run pytest (feature cols, temporal split, metrics)
│
▼
[4] docker build -f Dockerfile.prod (bakes model.pkl + git SHA version)
│
▼
[5] docker push ghcr.io/hulashc/agri-yield:latest
│
▼
[6] curl RENDER_DEPLOY_HOOK → Render redeploys
│
▼
[7] Poll /health until 200 → verify deploy succeeded
Three Decisions That Matter
These aren't implementation details; they're what separates this project from a Jupyter notebook demo.
Temporal train/test splitting
Any shuffle-based split in this pipeline is a bug. Yield data has strong temporal autocorrelation: shuffling leaks future harvest seasons into training, inflating every metric while the deployed model silently fails. Uses temporal 80/20 split sorted by week: no shuffle, no future leakage.
Model baked at CI time, not at serving time
The model.pkl is trained and baked into the Docker image during CI, not downloaded at container start. This eliminates runtime dependency on model registries, reduces cold start time, and guarantees every deploy has a matching, immutable model artefact.
Multi-layer weather fallback chain
Weather API failures don't crash the service. A four-level fallback chain (Redis → Live API → In-memory → UK seasonal defaults) ensures predictions always return, with a STALE DATA badge when defaults are used. This is production-grade defensive design.
Is This Free to Run?
Yes, the entire project runs on free tiers. No paid API keys required.
Fully free
- →XGBoost, FastAPI, Uvicorn: all open source
- →Open-Meteo API: free, no API key required
- →GitHub Actions: free tier for public repos
- →GitHub Container Registry (GHCR): free for public images
- →Render free tier: hosts the live demo (expect ~50s cold start)
- →Redis (in-memory fallback): no separate Redis instance needed
If scaling up
- →Paid Render instance: removes cold start, ~$7/month
- →Dedicated Redis: for persistent caching across restarts
- →Custom domain: if you want to brand the deployment
What This Project Proves
Every component demonstrates a distinct engineering capability for data engineering and MLOps roles.
End-to-End MLOps
From synthetic data generation to deployed, monitored inference, the full lifecycle is automated in CI/CD. No manual steps, no Jupyter notebooks in production.
API Integration & Resilience
Live weather API integration with a four-level fallback chain demonstrates robust external dependency management, a core skill for production data systems.
Docker & Deployment
Multi-stage Docker build with model baked in at CI time, pushed to GHCR, and deployed to Render. Demonstrates containerization and deployment pipeline skills.
CI/CD Pipeline Engineering
GitHub Actions pipeline that trains, builds, pushes, and deploys on every push. Infrastructure-as-code for the entire ML lifecycle.
Monitoring & Observability
PSI-based drift detection with per-field warning badges, Prometheus metrics endpoint, and Grafana-ready monitoring. Production-grade observability on a free tier.
Interactive Visualization
Leaflet.js map dashboard with 113 colour-coded fields shows the ability to build user-facing data products, not just backend APIs.
Try the live demo or explore the full source code:
Visit Live Demo