A production-grade ML-powered crop yield prediction platform for UK agricultural fields. Combines live weather data, soil attributes, and an XGBoost regression model to predict yield (kg/ha) per field, served via a real-time FastAPI backend and an interactive Leaflet map dashboard.
Interactive UK field map: 113 fields coloured by predicted yield (red → green)
XGBoost yield model: trained on real UK yield data (CYCleSS, 934 fields) or synthetic fallback, baked into the Docker image at CI time
Live weather integration: Open-Meteo API fetches real-time temperature, precipitation, and solar radiation per field
Confidence intervals: every prediction includes an 80% CI band
Drift detection: PSI-based feature drift monitoring with per-field warning badges
CI/CD pipeline: GitHub Actions trains the model, builds the Docker image, pushes to GHCR, and deploys to Render on every push to main
Automated tests: pytest runs in CI before build; /health verified post-deploy
Smart caching: Redis (with in-memory fallback) caches weather API responses for 1 hour
Prometheus metrics: /metrics endpoint for Grafana integration
GitHub Actions CI trains and packages the model into a Docker image, which is deployed on Render. The FastAPI backend serves predictions using live weather data and cached responses.
XGBoost Regressor trained on real UK yield data (CYCleSS, 934 field-year records). Features include lat, lon, area, crop type, soil type, region, weather vars, ET₀, soil moisture, NDVI, and week of year. RMSE ~1759 kg/ha on holdout.
Open-Meteo API fetches temperature, precipitation, and solar radiation per field at inference time. Fallback chain: Redis cache → Live API → In-memory → UK seasonal defaults.
GET /fields predicts all 113 fields with confidence intervals. GET /metrics exposes Prometheus endpoints. Static HTML/JS dashboard served via Leaflet.js map.
GitHub Actions generates training data, trains XGBoost, bakes model.pkl into Docker image, pushes to GHCR, and triggers Render deploy, all on every push to main.
PSI (Population Stability Index) computed per feature on every /fields call. Fields with PSI > 0.2 display a DRIFT warning badge. Prometheus metrics available at /metrics.
Redis caches weather API responses for 1 hour with in-memory fallback. Ensures the free-tier Render instance stays responsive under load without hammering the weather API.
XGBoost Regressor trained end-to-end in CI: no stale artefacts, no manual uploads.
Algorithm: XGBoost Regressor
Target: yield_kg_per_ha
Features: lat, lon, area, crop type, soil type, region, temperature, precipitation, solar radiation, ET₀, soil moisture, NDVI, week of year
Train/test split: 80/20
RMSE: ~1759 kg/ha (real CYCleSS data); ~1724 kg/ha (synthetic fallback)
Confidence interval: ±15% of prediction (configurable via CI_WIDTH env var)
The model is retrained from scratch on every CI run. The trained model.pkl is baked directly into the production Docker image: no stale artefacts, no manual uploads.
Weather data is fetched from the Open-Meteo API (free, no API key required) for each field's coordinates.
Features used at inference time:
- Temperature (°C): temperature_2m_max today
- Precipitation (mm): precipitation_sum today
- Solar radiation (MJ/m²): shortwave_radiation_sum today
Fallback chain: Redis cache → Live API → In-memory cache → UK seasonal defaults. The STALE DATA badge appears when defaults are used.
agri-yield/
├── .github/workflows/deploy.yml # Train → Build → Push → Deploy
├── data/seed/uk_fields.csv # 113 UK farm fields with metadata
├── ingestion/openmeteo_live.py # Live weather fetcher with caching
├── serving/
│ ├── app.py # FastAPI app + /fields endpoint
│ ├── model.py # Model loader + predict()
│ └── static/ # Dashboard HTML/CSS/JS
├── training/
│ ├── train_and_export.py # XGBoost training (real-data first, fallback to synthetic)
│ ├── prepare_real_data.py # CYCleSS real data ingestion
│ └── utils/features.py # Canonical FEATURE_COLS
├── monitoring/psi_detector.py # PSI drift detector
├── training/prepare_real_data.py # Real UK yield data pipeline (CYCleSS)
├── generate_data.py # Synthetic data generator (fallback)
├── Dockerfile.prod # Production image (bakes model.pkl)
└── pyproject.toml
Every push to main triggers a fully automated pipeline.
push to main
│
▼
[1] Generate training data (real CYCleSS or synthetic fallback)
│
▼
[2] Train XGBoost → save model.pkl
│
▼
[3] Run pytest (feature cols, temporal split, metrics)
│
▼
[4] docker build -f Dockerfile.prod (bakes model.pkl + git SHA version)
│
▼
[5] docker push ghcr.io/hulashc/agri-yield:latest
│
▼
[6] curl RENDER_DEPLOY_HOOK → Render redeploys
│
▼
[7] Poll /health until 200 → verify deploy succeeded
These aren't implementation details; they're what separates this project from a Jupyter notebook demo.
Any shuffle-based split in this pipeline is a bug. Yield data has strong temporal autocorrelation: shuffling leaks future harvest seasons into training, inflating every metric while the deployed model silently fails. Uses temporal 80/20 split sorted by week: no shuffle, no future leakage.
The model.pkl is trained and baked into the Docker image during CI, not downloaded at container start. This eliminates runtime dependency on model registries, reduces cold start time, and guarantees every deploy has a matching, immutable model artefact.
Weather API failures don't crash the service. A four-level fallback chain (Redis → Live API → In-memory → UK seasonal defaults) ensures predictions always return, with a STALE DATA badge when defaults are used. This is production-grade defensive design.
Yes, the entire project runs on free tiers. No paid API keys required.
Every component demonstrates a distinct engineering capability for data engineering and MLOps roles.
From synthetic data generation to deployed, monitored inference, the full lifecycle is automated in CI/CD. No manual steps, no Jupyter notebooks in production.
Live weather API integration with a four-level fallback chain demonstrates robust external dependency management, a core skill for production data systems.
Multi-stage Docker build with model baked in at CI time, pushed to GHCR, and deployed to Render. Demonstrates containerization and deployment pipeline skills.
GitHub Actions pipeline that trains, builds, pushes, and deploys on every push. Infrastructure-as-code for the entire ML lifecycle.
PSI-based drift detection with per-field warning badges, Prometheus metrics endpoint, and Grafana-ready monitoring. Production-grade observability on a free tier.
Leaflet.js map dashboard with 113 colour-coded fields shows the ability to build user-facing data products, not just backend APIs.
Try the live demo or explore the full source code:
↗ Visit Live Demo