From 85d2f11ed63b5e686e07ed488ccea1c4766cd710 Mon Sep 17 00:00:00 2001 From: Timon Date: Tue, 3 Feb 2026 17:31:02 +0100 Subject: [PATCH] issue 113 done, added propper yaml to all files --- r_app/10_create_per_field_tiffs.R | 76 ++++--- r_app/20_ci_extraction.R | 71 +++++-- r_app/21_convert_ci_rds_to_csv.R | 57 +++++- r_app/30_interpolate_growth_model.R | 59 +++++- r_app/40_mosaic_creation.R | 291 +++++++++++++++++++++++++++ r_app/40_mosaic_creation_per_field.R | 61 ++++-- r_app/80_calculate_kpis.R | 73 +++++-- r_app/_SCRIPT_HEADER_TEMPLATE.R | 54 +++++ 8 files changed, 634 insertions(+), 108 deletions(-) create mode 100644 r_app/40_mosaic_creation.R create mode 100644 r_app/_SCRIPT_HEADER_TEMPLATE.R diff --git a/r_app/10_create_per_field_tiffs.R b/r_app/10_create_per_field_tiffs.R index fe8b989..7ee2ee2 100644 --- a/r_app/10_create_per_field_tiffs.R +++ b/r_app/10_create_per_field_tiffs.R @@ -1,42 +1,54 @@ -# ============================================================================== -# SmartCane Script 10: Create Per-Field TIFFs -# ============================================================================== -# +# ============================================================================ +# SCRIPT 10: Create Per-Field TIFFs (Data Organization & Splitting) +# ============================================================================ # PURPOSE: -# Split full-farm satellite TIFFs into per-field file structure across TWO phases: +# Split full-farm satellite TIFFs into per-field file structure. Supports +# two phases: legacy data migration and ongoing new downloads. Transforms +# single large-file architecture into per-field directory structure for +# clean aggregation in downstream scripts (Script 20, 40, 80/90). # -# PHASE 1 - MIGRATION (Legacy Data): -# Input: merged_final_tif/{DATE}.tif (5-band: R,G,B,NIR,CI - with CI calculated) -# Output: field_tiles_CI/{FIELD}/{DATE}.tif -# Status: One-time reorganization of existing data; will be removed after 2-3 weeks +# INPUT DATA: +# - Source: laravel_app/storage/app/{project}/merged_tif/ or merged_final_tif/ +# - Format: GeoTIFF (4-band RGB+NIR or 5-band with CI) +# - Naming: {YYYY-MM-DD}.tif (full farm mosaic) # -# PHASE 2 - PROCESSING (New Downloads): -# Input: merged_tif/{DATE}.tif (4-band: R,G,B,NIR - raw from Planet API) -# Output: field_tiles/{FIELD}/{DATE}.tif -# Status: Ongoing for all new downloads; always runs (not conditional) +# OUTPUT DATA: +# - Destination: laravel_app/storage/app/{project}/field_tiles/ +# - Format: GeoTIFF (4-band RGB+NIR, same as input) +# - Structure: field_tiles/{FIELD}/{YYYY-MM-DD}.tif +# - Naming: Per-field GeoTIFFs organized by field and date # -# INTEGRATION WITH DOWNSTREAM SCRIPTS: -# - Script 20 (CI Extraction): -# Reads from field_tiles/{FIELD}/{DATE}.tif -# Adds CI calculation → outputs to field_tiles_CI/{FIELD}/{DATE}.tif (5-band) -# - Script 40 (Mosaic Creation): -# Reads from field_tiles_CI/{FIELD}/{DATE}.tif (via per-field weekly aggregation) -# Creates weekly_mosaic/{FIELD}/week_{WW}.tif +# USAGE: +# Rscript 10_create_per_field_tiffs.R [project] # -# ARCHITECTURE: -# This script uses field/date folder organization: -# field_tiles/ -# ├── field_1/ -# │ ├── 2024-01-15.tif -# │ └── 2024-01-16.tif -# └── field_2/ -# ├── 2024-01-15.tif -# └── 2024-01-16.tif +# Example (Windows PowerShell): +# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/10_create_per_field_tiffs.R angata # -# Benefits: Upstream scripts iterate per-field → per-date, enabling clean -# aggregation for mosaics (Script 40) and KPIs (Script 80/90). +# PARAMETERS: +# - project: Project name (character) - angata, chemba, xinavane, esa, simba # -# ============================================================================== +# CLIENT TYPES: +# - cane_supply (ANGATA): Yes - primary data organization script +# - agronomic_support (AURA): Yes - supports field-level analysis +# +# DEPENDENCIES: +# - Packages: terra, sf, tidyverse +# - Utils files: parameters_project.R, 00_common_utils.R, 10_create_per_field_tiffs_utils.R +# - External data: Field boundaries (pivot.geojson) +# - Data directories: merged_tif/, field_tiles/ (created if missing) +# +# NOTES: +# - Supports two-phase migration: legacy (merged_final_tif) and ongoing (merged_tif) +# - Automatically detects and handles field boundaries from pivot.geojson +# - Geometry validation and repair applied via st_make_valid() +# - Critical for downstream Scripts 20, 40, and KPI calculations +# - Creates per-field structure that enables efficient per-field processing +# +# RELATED ISSUES: +# SC-111: Script 10 refactoring and geometry repair +# SC-112: Utilities restructuring (uses 00_common_utils.R) +# +# ============================================================================ library(terra) diff --git a/r_app/20_ci_extraction.R b/r_app/20_ci_extraction.R index cedb9c6..d5acc79 100644 --- a/r_app/20_ci_extraction.R +++ b/r_app/20_ci_extraction.R @@ -1,25 +1,58 @@ -# CI_EXTRACTION.R -# ============== -# This script processes satellite imagery to extract Canopy Index (CI) values for agricultural fields. -# It handles image processing, masking, and extraction of statistics by field/sub-field. -# Supports both 4-band and 8-band PlanetScope data with automatic band detection and cloud masking. +# ============================================================================ +# SCRIPT 20: Canopy Index (CI) Extraction from Satellite Imagery +# ============================================================================ +# PURPOSE: +# Extract Canopy Index (CI) from 4-band or 8-band satellite imagery and +# mask by field boundaries. Supports automatic band detection, cloud masking +# with UDM2 (8-band), and per-field CI value extraction. Produces both +# per-field TIFFs and consolidated CI statistics for growth model input. # -# Usage: Rscript 02_ci_extraction.R [end_date] [offset] [project_dir] [data_source] -# - end_date: End date for processing (YYYY-MM-DD format) -# - offset: Number of days to look back from end_date -# - project_dir: Project directory name (e.g., "angata", "aura", "chemba") -# - data_source: Data source directory - "merged_tif_8b" (default) or "merged_tif" (4-band) or "merged_final_tif" -# If tiles exist (daily_tiles_split/), they are used automatically +# INPUT DATA: +# - Source: laravel_app/storage/app/{project}/field_tiles/{FIELD}/{DATE}.tif +# - Format: GeoTIFF (4-band RGB+NIR from Planet API, or 8-band with UDM2) +# - Requirement: Field boundaries (pivot.geojson) for masking # -# Examples: -# # Angata 8-band data (with UDM cloud masking) -# & 'C:\Program Files\R\R-4.4.3\bin\x64\Rscript' r_app/20_ci_extraction.R 2026-01-02 7 angata merged_tif_8b -# -# # Aura 4-band data -# Rscript 20_ci_extraction.R 2025-11-26 7 aura merged_tif +# OUTPUT DATA: +# - Destination: laravel_app/storage/app/{project}/field_tiles_CI/{FIELD}/{DATE}.tif +# - Format: GeoTIFF (5-band: R,G,B,NIR,CI as float32) +# - Also exports: combined_CI/combined_CI_data.rds (wide format: fields × dates) # -# # Auto-detects and uses tiles if available: -# Rscript 20_ci_extraction.R 2026-01-02 7 angata (uses tiles if daily_tiles_split/ exists) +# USAGE: +# Rscript 20_ci_extraction.R [end_date] [offset] [project] [data_source] +# +# Example (Windows PowerShell): +# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/20_ci_extraction.R 2026-01-02 7 angata merged_tif +# +# PARAMETERS: +# - end_date: End date for processing (character, YYYY-MM-DD format) +# - offset: Days to look back from end_date (numeric, default 7) +# - project: Project name (character) - angata, chemba, xinavane, esa, simba +# - data_source: Data source directory (character, optional) - "merged_tif" (default), "merged_tif_8b", "merged_final_tif" +# +# CLIENT TYPES: +# - cane_supply (ANGATA): Yes - core data processing +# - agronomic_support (AURA): Yes - supports field health monitoring +# +# DEPENDENCIES: +# - Packages: terra, sf, tidyverse, lubridate, readxl, furrr, future +# - Utils files: parameters_project.R, 00_common_utils.R, 20_ci_extraction_utils.R +# - External data: Field boundaries (pivot.geojson), harvest data (harvest.xlsx) +# - Data directories: field_tiles/, field_tiles_CI/, combined_CI/ +# +# NOTES: +# - CI formula: (NIR - Red) / (NIR + Red); normalized to 0-5 range +# - 8-band data automatically cloud-masked using UDM2 (band 7-8) +# - 4-band data assumes clear-sky Planet PSScene imagery +# - Parallel processing via furrr for speed optimization +# - Output RDS uses wide format (fields as rows, dates as columns) for growth model +# - Critical dependency for Script 30 (growth model) and Script 80 (KPIs) +# +# RELATED ISSUES: +# SC-112: Utilities restructuring +# SC-108: Core pipeline improvements +# +# ============================================================================ + # 1. Load required packages # ----------------------- diff --git a/r_app/21_convert_ci_rds_to_csv.R b/r_app/21_convert_ci_rds_to_csv.R index be458b6..7c8e740 100644 --- a/r_app/21_convert_ci_rds_to_csv.R +++ b/r_app/21_convert_ci_rds_to_csv.R @@ -1,16 +1,53 @@ -# 02b_CONVERT_CI_RDS_TO_CSV.R -# ============================ -# Convert combined_CI_data.rds (output of script 02) to CSV format for Python -# This script runs AFTER script 02 (CI extraction) and creates a CSV that Python -# can use for harvest date detection WITHOUT requiring the 'model' column (which -# comes from script 03 after interpolation and harvest dates are known). +# ============================================================================ +# SCRIPT 21: Convert CI RDS to CSV (Python Compatibility Format) +# ============================================================================ +# PURPOSE: +# Convert consolidated CI data from R's wide format (RDS) to Python-compatible +# long format (CSV). Prepares per-field CI time series for harvest detection +# models and Python ML workflows without requiring interpolated/modeled values. # -# Usage: Rscript 02b_convert_ci_rds_to_csv.R [project_dir] -# - project_dir: Project directory name (e.g., "esa", "chemba", "angata") +# INPUT DATA: +# - Source: laravel_app/storage/app/{project}/combined_CI/combined_CI_data.rds +# - Format: RDS (wide format: fields × dates with CI values) +# - Requirement: Script 20 must have completed CI extraction # -# Output: CSV file at laravel_app/storage/app/{project_dir}/Data/extracted_ci/cumulative_vals/ci_data_for_python.csv -# Columns: field, sub_field, Date, FitData, DOY, value (alias for FitData) +# OUTPUT DATA: +# - Destination: laravel_app/storage/app/{project}/Data/extracted_ci/cumulative_vals/ +# - Format: CSV (long format) +# - Columns: field, sub_field, Date, FitData, DOY, value # +# USAGE: +# Rscript 21_convert_ci_rds_to_csv.R [project] +# +# Example (Windows PowerShell): +# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/21_convert_ci_rds_to_csv.R angata +# +# PARAMETERS: +# - project: Project name (character) - angata, chemba, xinavane, esa, simba +# +# CLIENT TYPES: +# - cane_supply (ANGATA): Yes - data export +# - agronomic_support (AURA): Yes - Python ML integration +# +# DEPENDENCIES: +# - Packages: tidyverse, lubridate, zoo +# - Utils files: parameters_project.R, 00_common_utils.R +# - Input data: combined_CI_data.rds from Script 20 +# - Data directories: extracted_ci/cumulative_vals/ +# +# NOTES: +# - Transformation: Wide format (fields as rows, dates as columns) → Long format +# - Time series: Preserves all CI values without interpolation +# - DOY (Day of Year): Calculated from date for seasonal analysis +# - Python integration: CSV format compatible with pandas/scikit-learn workflows +# - Used by: Python harvest detection models (harvest_date_prediction.py) +# - Optional: Run only when exporting to Python for ML model training +# +# RELATED ISSUES: +# SC-112: Utilities restructuring +# SC-108: Core pipeline improvements +# +# ============================================================================ suppressPackageStartupMessages({ library(tidyverse) diff --git a/r_app/30_interpolate_growth_model.R b/r_app/30_interpolate_growth_model.R index 633617a..2b4bcc7 100644 --- a/r_app/30_interpolate_growth_model.R +++ b/r_app/30_interpolate_growth_model.R @@ -1,15 +1,54 @@ -# filepath: c:\Users\timon\Resilience BV\4020 SCane ESA DEMO - Documenten\General\4020 SCDEMO Team\4020 TechnicalData\WP3\smartcane\r_app\interpolate_growth_model.R +# ============================================================================ +# SCRIPT 30: Growth Model Interpolation (CI Time Series) +# ============================================================================ +# PURPOSE: +# Interpolate Canopy Index (CI) values across time to create continuous +# growth curves. Fills gaps in measurement dates, applies smoothing via +# LOESS, and generates daily CI estimates and cumulative statistics for +# each field. Enables downstream yield prediction and trend analysis. # -# INTERPOLATE_GROWTH_MODEL.R -# ========================= -# This script interpolates CI (Chlorophyll Index) values between measurement dates -# to create a continuous growth model. It generates daily values and cumulative -# CI statistics for each field. +# INPUT DATA: +# - Source: laravel_app/storage/app/{project}/combined_CI/combined_CI_data.rds +# - Format: RDS (wide format: fields × dates with CI values) +# - Requirement: Field boundaries (pivot.geojson) and harvest data (harvest.xlsx) # -# Usage: Rscript interpolate_growth_model.R [project_dir] [data_source] -# - project_dir: Project directory name (e.g., "chemba") -# - data_source: (Optional) Data source directory - "merged_tif" (default), "merged_tif_8b" -# & 'C:\Program Files\R\R-4.4.3\bin\x64\Rscript' r_app/30_interpolate_growth_model.R angata merged_tif +# OUTPUT DATA: +# - Destination: laravel_app/storage/app/{project}/interpolated_ci/ +# - Format: RDS files per field (daily CI estimates) +# - Also exports: Growth model curves as RDS (cumulative CI, daily values) +# +# USAGE: +# Rscript 30_interpolate_growth_model.R [project] +# +# Example (Windows PowerShell): +# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/30_interpolate_growth_model.R angata +# +# PARAMETERS: +# - project: Project name (character) - angata, chemba, xinavane, esa, simba +# +# CLIENT TYPES: +# - cane_supply (ANGATA): Yes - core growth monitoring +# - agronomic_support (AURA): Yes - field health trend analysis +# +# DEPENDENCIES: +# - Packages: tidyverse, lubridate +# - Utils files: parameters_project.R, 00_common_utils.R, 30_growth_model_utils.R +# - External data: Field boundaries (pivot.geojson), harvest data (harvest.xlsx) +# - Input data: combined_CI_data.rds from Script 20 +# - Data directories: interpolated_ci/ (created if missing) +# +# NOTES: +# - Interpolation method: LOESS smoothing with span = 0.3 (sensitive to local trends) +# - Gap-filling: Assumes continuous growth between measurements; skips clouds +# - Cumulative CI: Sum of daily interpolated values from planting to current date +# - Used by: Script 80 (KPI trends) and Script 12 (yield forecasting) +# - Critical for 8-week CV trend calculation and 4-week growth categorization +# +# RELATED ISSUES: +# SC-112: Utilities restructuring +# SC-108: Core pipeline improvements +# +# ============================================================================ # 1. Load required packages # ----------------------- diff --git a/r_app/40_mosaic_creation.R b/r_app/40_mosaic_creation.R new file mode 100644 index 0000000..669277c --- /dev/null +++ b/r_app/40_mosaic_creation.R @@ -0,0 +1,291 @@ +# ============================================================================ +# SCRIPT 40: Weekly Mosaic Creation (CI Band Aggregation) +# ============================================================================ +# PURPOSE: +# Create weekly 5-band (R, G, B, NIR, CI) mosaics from daily satellite +# imagery. Aggregates multi-day CI data into single weekly composite raster +# for field-level analysis. Supports per-field or single-file architectures. +# +# INPUT DATA: +# - Daily per-field TIFFs: laravel_app/storage/app/{project}/daily_tiles/{YYYY-MM-DD}/*.tif +# (or single-file mosaics: merged_tif/{YYYY-MM-DD}.tif + pivot.geojson masking) +# - CI data (RDS): laravel_app/storage/app/{project}/combined_CI/combined_CI_data.rds +# - Field boundaries: laravel_app/storage/app/{project}/pivot.geojson +# +# OUTPUT DATA: +# - Destination: laravel_app/storage/app/{project}/weekly_mosaic/ +# - Format: 5-band GeoTIFF (uint16) +# - Naming: week_{WW}.tif (week number + year, e.g., week_35_2025.tif) +# - Spatial: Raster aligned to field boundaries; CRS preserved +# +# USAGE: +# Rscript 40_mosaic_creation.R [end_date] [offset] [project] [file_name] [data_source] +# +# Example (Windows PowerShell): +# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation.R 2026-01-12 7 aura +# +# PARAMETERS: +# - end_date: End date (YYYY-MM-DD format); required for weekly aggregation +# - offset: Days to look back (typically 7 for one week) +# - project: Project name (aura, angata, chemba, xinavane, esa, simba) +# - file_name: Custom output filename (optional; default: week_{WW}_{YYYY}.tif) +# - data_source: Data folder (optional; auto-detects merged_tif or merged_tif_8b) +# +# CLIENT TYPES: +# - cane_supply (ANGATA): Yes - harvest readiness timeline depends on weekly mosaic +# - agronomic_support (AURA): Yes - KPI calculation requires weekly CI bands +# +# DEPENDENCIES: +# - Packages: sf, terra, tidyverse, lubridate, here +# - Utils files: parameters_project.R, 00_common_utils.R, 40_mosaic_creation_utils.R +# - Input data: Daily per-field TIFFs (Script 10) + CI extraction (Script 20) +# - Data: field boundaries (pivot.geojson), harvest dates (if available) +# +# NOTES: +# - Weekly aggregation: Combines 7 days of daily data into single composite +# - 5-band output: R, G, B, NIR, and Canopy Index (CI) derived from NDVI +# - Tiling support: Handles per-field TIFF architecture; auto-mosaics if needed +# - Data source auto-detection: Searches merged_tif/ or merged_tif_8b/ folders +# - Command-line driven: Designed for batch scheduling (cron/Task Scheduler) +# - Downstream: Script 80 (KPI calculation) depends on weekly_mosaic/ output +# - Performance: Multi-file mosaicing (~25 fields) takes 5-10 minutes per week +# +# RELATED ISSUES: +# SC-113: Script header standardization +# SC-112: Utilities restructuring +# SC-111: Script 10 geometry validation +# +# ============================================================================ + +# 1. Load required packages +# ----------------------- +suppressPackageStartupMessages({ + library(sf) + library(terra) + library(tidyverse) + library(lubridate) + library(here) +}) + +# 2. Process command line arguments and run mosaic creation +# ------------------------------------------------------ +main <- function() { + # Capture command line arguments + args <- commandArgs(trailingOnly = TRUE) + + # Process project_dir argument with default + if (length(args) >= 3 && !is.na(args[3])) { + project_dir <- as.character(args[3]) + } else if (exists("project_dir", envir = .GlobalEnv)) { + project_dir <- get("project_dir", envir = .GlobalEnv) + } else { + # Default project directory + project_dir <- "angata" + message("No project_dir provided. Using default:", project_dir) + } + + # Make project_dir available globally so parameters_project.R can use it + assign("project_dir", project_dir, envir = .GlobalEnv) + + # Process end_date argument with default + if (length(args) >= 1 && !is.na(args[1])) { + # Parse date explicitly in YYYY-MM-DD format from command line + end_date <- as.Date(args[1], format = "%Y-%m-%d") + if (is.na(end_date)) { + message("Invalid end_date provided. Using current date.") + end_date <- Sys.Date() + } + } else if (exists("end_date_str", envir = .GlobalEnv)) { + end_date <- as.Date(get("end_date_str", envir = .GlobalEnv)) + } else { + # Default to current date if no argument is provided + end_date <- Sys.Date() + message("No end_date provided. Using current date: ", format(end_date)) + } + + # Process offset argument with default + if (length(args) >= 2 && !is.na(args[2])) { + offset <- as.numeric(args[2]) + if (is.na(offset) || offset <= 0) { + message("Invalid offset provided. Using default (7 days).") + offset <- 7 + } + } else { + # Default to 7 days if no argument is provided + offset <- 7 + message("No offset provided. Using default:", offset, "days") + } + + # Process data_source argument (optional, passed from pipeline) + # If provided, use it; otherwise auto-detect + data_source_from_args <- NULL + if (length(args) >= 5 && !is.na(args[5]) && nchar(args[5]) > 0) { + data_source_from_args <- as.character(args[5]) + message("Data source explicitly provided via arguments: ", data_source_from_args) + } + + # 3. Initialize project configuration + # -------------------------------- + + # Detect which data source directory exists (merged_tif or merged_tif_8b) + # IMPORTANT: Only consider a folder as valid if it contains actual files + laravel_storage <- here::here("laravel_app/storage/app", project_dir) + + # Load centralized path structure + tryCatch({ + source("r_app/parameters_project.R") + paths <- setup_project_directories(project_dir) + }, error = function(e) { + message("Note: Could not open files from r_app directory") + message("Attempting to source from default directory instead...") + tryCatch({ + source("parameters_project.R") + paths <- setup_project_directories(project_dir) + message("✓ Successfully sourced files from default directory") + }, error = function(e) { + stop("Failed to source required files from both 'r_app' and default directories.") + }) + }) + data_source <- if (has_8b_data) { + message("Auto-detected data source: merged_tif_8b (8-band optimized) - contains files") + "merged_tif_8b" + } else if (has_legacy_data) { + message("Auto-detected data source: merged_tif (legacy 4-band) - contains files") + "merged_tif" + } else { + message("Warning: No valid data source found (both folders empty or missing). Using default: merged_tif") + "merged_tif" + } + } + + # Set global data_source for parameters_project.R + assign("data_source", data_source, envir = .GlobalEnv) + + tryCatch({ + source("r_app/parameters_project.R") + source("r_app/00_common_utils.R") + source("r_app/40_mosaic_creation_utils.R") + safe_log(paste("Successfully sourced files from 'r_app' directory.")) + }, error = function(e) { + message("Note: Could not open files from r_app directory") + message("Attempting to source from default directory instead...") + tryCatch({ + source("parameters_project.R") + paths <- setup_project_directories(project_dir) + message("✓ Successfully sourced files from default directory") + }, error = function(e) { + stop("Failed to source required files from both 'r_app' and default directories.") + }) + }) + + # Use centralized paths (no need to manually construct or create dirs) + merged_final <- paths$growth_model_interpolated_dir # or merged_final_tif if needed + daily_vrt <- paths$vrt_dir + + safe_log(paste("Using growth model/mosaic directory:", merged_final)) + safe_log(paste("Using daily VRT directory:", daily_vrt)) + + # 4. Generate date range for processing + # --------------------------------- + dates <- date_list(end_date, offset) + safe_log(paste("Processing data for week", dates$week, "of", dates$year)) + + # Create output filename + # Only use custom filename if explicitly provided (not empty string) + file_name_tif <- if (length(args) >= 4 && !is.na(args[4]) && nchar(args[4]) > 0) { + as.character(args[4]) + } else { + paste0("week_", sprintf("%02d", dates$week), "_", dates$year, ".tif") + } + + safe_log(paste("Output will be saved as:", file_name_tif)) + + # 5. Create weekly mosaics - route based on project tile detection + # --------------------------------------------------------------- + # The use_tile_mosaic flag is auto-detected by parameters_project.R + # based on whether tiles exist in merged_final_tif/ + + if (!exists("use_tile_mosaic")) { + # Fallback detection if flag not set (shouldn't happen) + merged_final_dir <- file.path(laravel_storage, "merged_final_tif") + tile_detection <- detect_tile_structure_from_merged_final(merged_final_dir) + use_tile_mosaic <- tile_detection$has_tiles + } + + if (use_tile_mosaic) { + # TILE-BASED APPROACH: Create per-tile weekly MAX mosaics + # This is used for projects like Angata with large ROIs requiring spatial partitioning + # Input data comes from merged_final_tif/{grid_size}/{DATE}/{DATE}_XX.tif (5-band tiles from script 20) + tryCatch({ + safe_log("Starting per-tile mosaic creation (tile-based approach)...") + + # Detect grid size from merged_final_tif folder structure + # Expected: merged_final_tif/5x5/ or merged_final_tif/10x10/ etc. + merged_final_base <- file.path(laravel_storage, "merged_final_tif") + grid_subfolders <- list.dirs(merged_final_base, full.names = FALSE, recursive = FALSE) + # Look for grid size patterns like "5x5", "10x10", "20x20" + grid_patterns <- grep("^\\d+x\\d+$", grid_subfolders, value = TRUE) + + if (length(grid_patterns) == 0) { + stop("No grid size subfolder found in merged_final_tif/ (expected: 5x5, 10x10, etc.)") + } + + grid_size <- grid_patterns[1] # Use first grid size found + safe_log(paste("Detected grid size:", grid_size)) + + # Point to the grid-specific merged_final_tif directory + merged_final_with_grid <- file.path(merged_final_base, grid_size) + + # Set output directory for per-tile mosaics, organized by grid size (from centralized paths) + # Output: weekly_tile_max/{grid_size}/week_WW_YYYY_TT.tif + tile_output_base <- file.path(paths$weekly_tile_max_dir, grid_size) + # Note: no dir.create needed - paths$weekly_tile_max_dir already created by setup_project_directories() + dir.create(tile_output_base, recursive = TRUE, showWarnings = FALSE) # Create grid-size subfolder + + created_tile_files <- create_weekly_mosaic_from_tiles( + dates = dates, + merged_final_dir = merged_final_with_grid, + tile_output_dir = tile_output_base, + field_boundaries = field_boundaries + ) + + safe_log(paste("✓ Per-tile mosaic creation completed - created", + length(created_tile_files), "tile files")) + }, error = function(e) { + safe_log(paste("ERROR in tile-based mosaic creation:", e$message), "ERROR") + traceback() + stop("Mosaic creation failed") + }) + + } else { + # SINGLE-FILE APPROACH: Create single weekly mosaic file + # This is used for legacy projects (ESA, Chemba, Aura) expecting single-file output + tryCatch({ + safe_log("Starting single-file mosaic creation (backward-compatible approach)...") + + # Set output directory for single-file mosaics (from centralized paths) + single_file_output_dir <- paths$weekly_mosaic_dir + + created_file <- create_weekly_mosaic( + dates = dates, + field_boundaries = field_boundaries, + daily_vrt_dir = daily_vrt, + merged_final_dir = merged_final, + output_dir = single_file_output_dir, + file_name_tif = file_name_tif, + create_plots = FALSE + ) + + safe_log(paste("✓ Single-file mosaic creation completed:", created_file)) + }, error = function(e) { + safe_log(paste("ERROR in single-file mosaic creation:", e$message), "ERROR") + traceback() + stop("Mosaic creation failed") + }) + } +} + +if (sys.nframe() == 0) { + main() +} + \ No newline at end of file diff --git a/r_app/40_mosaic_creation_per_field.R b/r_app/40_mosaic_creation_per_field.R index e7bb27d..e42909f 100644 --- a/r_app/40_mosaic_creation_per_field.R +++ b/r_app/40_mosaic_creation_per_field.R @@ -1,25 +1,54 @@ -# 40_MOSAIC_CREATION_PER_FIELD.R -# =============================== -# Per-Field Weekly Mosaic Creation +# ============================================================================ +# SCRIPT 40: Weekly Mosaic Creation (Per-Field CI Aggregation) +# ============================================================================ +# PURPOSE: +# Aggregate daily per-field CI TIFFs into weekly mosaics. Handles multi-date +# merging, cloud masking, and produces 5-band weekly output for reporting +# and KPI calculations. Supports both per-field and grid-based tile architecture. # -# Creates weekly mosaics FROM per-field daily CI TIFFs (output from Script 20) -# TO per-field weekly CI TIFFs (input for Scripts 90/91 reporting). +# INPUT DATA: +# - Source: laravel_app/storage/app/{project}/field_tiles_CI/{FIELD}/{DATE}.tif +# - Format: GeoTIFF (5-band: R,G,B,NIR,CI as float32) +# - Dates: All available dates within week range # -# ARCHITECTURE: -# Input: field_tiles_CI/{FIELD}/{DATE}.tif (5-band daily, per-field from Script 20) -# Output: weekly_mosaic/{FIELD}/week_WW_YYYY.tif (5-band weekly, per-field) +# OUTPUT DATA: +# - Destination: laravel_app/storage/app/{project}/weekly_mosaic/{FIELD}/ +# - Format: GeoTIFF (5-band: R,G,B,NIR,CI) +# - Naming: week_WW_YYYY.tif (WW = ISO week, YYYY = ISO year) # # USAGE: -# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R [end_date] [offset] [project_dir] +# Rscript 40_mosaic_creation_per_field.R [end_date] [offset] [project] # -# ARGUMENTS: -# end_date: End date for processing (YYYY-MM-DD format, default: today) -# offset: Days to look back from end_date (typically 7 for one week, default: 7) -# project_dir: Project directory (e.g., "aura", "angata", "chemba", "esa", default: "angata") +# Example (Windows PowerShell): +# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R 2026-01-12 7 angata # -# EXAMPLES: -# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R 2026-01-12 7 aura -# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R 2025-12-31 7 angata +# PARAMETERS: +# - end_date: End date for processing (character, YYYY-MM-DD format, default today) +# - offset: Days to look back (numeric, default 7 for one week) +# - project: Project name (character) - angata, chemba, xinavane, esa, simba +# +# CLIENT TYPES: +# - cane_supply (ANGATA): Yes - weekly monitoring +# - agronomic_support (AURA): Yes - field health reporting +# +# DEPENDENCIES: +# - Packages: terra, sf, tidyverse, lubridate +# - Utils files: parameters_project.R, 00_common_utils.R, 40_mosaic_creation_per_field_utils.R +# - Input data: Daily per-field CI TIFFs from Script 20 +# - Data directories: field_tiles_CI/, weekly_mosaic/ +# +# NOTES: +# - Aggregation method: Maximum CI value per pixel across week (handles clouds) +# - ISO week-year used for consistent date handling across year boundaries +# - Supports both single-file and tiled mosaic architectures +# - Output feeds Scripts 80 (KPI calculations) and 90/91 (reporting) +# - Critical for trend analysis: week-over-week CI comparison +# +# RELATED ISSUES: +# SC-112: Script 40 cleanup (deleted legacy mosaic utils files) +# SC-108: Core pipeline improvements +# +# ============================================================================ # 1. Load required packages # ----------------------- diff --git a/r_app/80_calculate_kpis.R b/r_app/80_calculate_kpis.R index 3bea267..7c96f9e 100644 --- a/r_app/80_calculate_kpis.R +++ b/r_app/80_calculate_kpis.R @@ -1,28 +1,59 @@ -# 80_CALCULATE_KPIS.R (CONSOLIDATED KPI CALCULATION) # ============================================================================ -# UNIFIED KPI CALCULATION SCRIPT -# -# This script combines: -# 1. Per-field weekly analysis (from 09c: field-level trends, phases, statuses) -# 2. Farm-level KPI metrics (from old 09: 6 high-level indicators) +# SCRIPT 80: Key Performance Indicator (KPI) Calculation +# ============================================================================ +# PURPOSE: +# Calculate per-field and farm-level KPIs from weekly CI mosaics. Computes +# field uniformity, growth trends (4-week and 8-week), area change detection, +# TCH forecasts, stress identification, and weed presence. Generates +# comprehensive Excel/CSV/RDS exports for dashboards and stakeholder reporting. # -# FEATURES: -# - Per-field analysis with SC-64 enhancements (4-week trends, CI percentiles, etc.) -# - Farm-level KPI calculation (6 metrics for executive overview) -# - Parallel processing (tile-aware, 1000+ fields supported) -# - Comprehensive Excel + RDS + CSV exports (21 columns per spec) -# - Test mode for development - -# CRITICAL INTEGRATIONS: +# INPUT DATA: +# - Source 1: laravel_app/storage/app/{project}/weekly_mosaic/{FIELD}/week_*.tif +# - Source 2: Field boundaries (pivot.geojson) and harvest data (harvest.xlsx) +# - Source 3: Historical field stats (RDS from previous weeks) # -# 1. IMMINENT_PROB FROM HARVEST MODEL (MODEL_307) -# [✓] Load script 31 output: {project}_week_{WW}_{YYYY}.csv -# Columns: field, imminent_prob, detected_prob, week, year -# [✓] LEFT JOIN to field_analysis_df by field -# [✓] Use actual harvest probability data instead of placeholder +# OUTPUT DATA: +# - Destination: laravel_app/storage/app/{project}/output/ +# - Format: Excel (.xlsx), CSV (.csv), RDS (.rds) +# - Files: {project}_field_analysis_week{WW}_{YYYY}.xlsx + metadata # -# 2. AGE FROM HARVEST.XLSX (SCRIPTS 22 & 23) -# [✓] Load harvest.xlsx with planting_date (season_start) +# USAGE: +# Rscript 80_calculate_kpis.R [project] [week] [year] +# +# Example (Windows PowerShell): +# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/80_calculate_kpis.R angata 5 2026 +# +# PARAMETERS: +# - project: Project name (character) - angata, chemba, xinavane, esa, simba +# - week: ISO week number (numeric, 1-53, default current week) +# - year: ISO year (numeric, default current year) +# +# CLIENT TYPES: +# - cane_supply (ANGATA): Yes - uses 80_utils_cane_supply.R (placeholder) +# - agronomic_support (AURA): Yes - uses 80_utils_agronomic_support.R (6 KPI funcs) +# +# DEPENDENCIES: +# - Packages: terra, sf, tidyverse, lubridate, writexl, spdep +# - Utils files: parameters_project.R, 00_common_utils.R, 80_utils_agronomic_support.R, 80_utils_cane_supply.R +# - External data: Field boundaries (pivot.geojson), harvest data (harvest.xlsx) +# - Input data: Weekly mosaic TIFFs (Script 40 output) +# - Data directories: weekly_mosaic/, output/ (created if missing) +# +# NOTES: +# - KPIs calculated: Field Uniformity (CV), Area Change (pixel %), TCH Forecast, +# Growth/Decline Trend, Weed Presence (spatial autocorrelation), Gap Filling % +# - Client-aware: Conditional sourcing based on client_config$script_90_compatible +# - Exports: 21-column output with field-level metrics, phase, status, alerts +# - Supports 4-week and 8-week trend analysis from historical RDS cache +# - Critical dependency for Scripts 90/91 (reporting/dashboards) +# - Uses Moran's I for spatial clustering detection (weed/stress patterns) +# +# RELATED ISSUES: +# SC-112: Script 80 utilities restructuring (common + client-aware modules) +# SC-108: Core pipeline improvements +# SC-100: KPI definition and formula documentation +# +# ============================================================================ # [✓] Extract planting dates per field # [✓] Calculate Age_week = difftime(report_date, planting_date, units="weeks") # diff --git a/r_app/_SCRIPT_HEADER_TEMPLATE.R b/r_app/_SCRIPT_HEADER_TEMPLATE.R new file mode 100644 index 0000000..83dac6f --- /dev/null +++ b/r_app/_SCRIPT_HEADER_TEMPLATE.R @@ -0,0 +1,54 @@ +# ============================================================================ +# SCRIPT XX: [DESCRIPTIVE TITLE] +# ============================================================================ +# PURPOSE: +# [What this script does in 2-3 sentences. Example: Downloads satellite +# imagery from Planet API, processes 4-band RGB+NIR data, and saves +# merged GeoTIFFs for use by downstream analysis stages.] +# +# INPUT DATA: +# - Source: [Which directory/files it reads. Example: Planet API, hardcoded bbox] +# - Format: [TIFF, RDS, GeoJSON, etc. Example: 4-band uint16 GeoTIFF] +# - Location: [Full path example] +# +# OUTPUT DATA: +# - Destination: [Which directory/files it creates. Example: laravel_app/storage/app/{project}/merged_tif/] +# - Format: [TIFF, RDS, CSV, etc. Example: Single-band float32 GeoTIFF] +# - Naming convention: [Example: {YYYY-MM-DD}.tif] +# +# USAGE: +# Rscript XX_script_name.R [arg1] [arg2] [arg3] +# +# Example (Windows PowerShell): +# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/XX_script_name.R angata 2026-01-15 +# +# PARAMETERS: +# - arg1: [Description, type, and valid values. Example: project (character) - angata, chemba, xinavane, esa, simba] +# - arg2: [Description, type, and valid values. Example: date (character, optional) - ISO format YYYY-MM-DD; default today] +# +# CLIENT TYPES: +# - cane_supply (ANGATA): [Yes/No - if Yes, briefly explain any differences] +# - agronomic_support (AURA): [Yes/No - if Yes, briefly explain any differences] +# +# DEPENDENCIES: +# - Packages: [dplyr, tidyr, terra, sf, readr, writexl, etc.] +# - Utils files: [parameters_project.R, 00_common_utils.R, XX_utils.R, etc.] +# - External data: [harvest.xlsx, pivot.geojson, etc.] +# - Data directories: [laravel_app/storage/app/{project}/] +# +# NOTES: +# [Any special considerations, assumptions, or future improvements. +# Example: Cloud filtering uses CI >= 0.5 threshold. Multi-field support +# implemented via field geometry masking from pivot.geojson.] +# +# RELATED ISSUES: +# SC-XXX: [Brief description of related work] +# +# HISTORY: +# 2026-02-03: [Description of change, if refactored or enhanced] +# ============================================================================ + +# NOTE: This is a TEMPLATE file for documentation purposes only. +# When creating a new script or updating an existing one, copy this template +# and fill in all sections with accurate information about your specific script. +# Then delete this comment block.