issue 113 done, added propper yaml to all files

This commit is contained in:
Timon 2026-02-03 17:31:02 +01:00
parent 8d560ff155
commit 85d2f11ed6
8 changed files with 634 additions and 108 deletions

View file

@ -1,42 +1,54 @@
# ============================================================================== # ============================================================================
# SmartCane Script 10: Create Per-Field TIFFs # SCRIPT 10: Create Per-Field TIFFs (Data Organization & Splitting)
# ============================================================================== # ============================================================================
#
# PURPOSE: # PURPOSE:
# Split full-farm satellite TIFFs into per-field file structure across TWO phases: # Split full-farm satellite TIFFs into per-field file structure. Supports
# two phases: legacy data migration and ongoing new downloads. Transforms
# single large-file architecture into per-field directory structure for
# clean aggregation in downstream scripts (Script 20, 40, 80/90).
# #
# PHASE 1 - MIGRATION (Legacy Data): # INPUT DATA:
# Input: merged_final_tif/{DATE}.tif (5-band: R,G,B,NIR,CI - with CI calculated) # - Source: laravel_app/storage/app/{project}/merged_tif/ or merged_final_tif/
# Output: field_tiles_CI/{FIELD}/{DATE}.tif # - Format: GeoTIFF (4-band RGB+NIR or 5-band with CI)
# Status: One-time reorganization of existing data; will be removed after 2-3 weeks # - Naming: {YYYY-MM-DD}.tif (full farm mosaic)
# #
# PHASE 2 - PROCESSING (New Downloads): # OUTPUT DATA:
# Input: merged_tif/{DATE}.tif (4-band: R,G,B,NIR - raw from Planet API) # - Destination: laravel_app/storage/app/{project}/field_tiles/
# Output: field_tiles/{FIELD}/{DATE}.tif # - Format: GeoTIFF (4-band RGB+NIR, same as input)
# Status: Ongoing for all new downloads; always runs (not conditional) # - Structure: field_tiles/{FIELD}/{YYYY-MM-DD}.tif
# - Naming: Per-field GeoTIFFs organized by field and date
# #
# INTEGRATION WITH DOWNSTREAM SCRIPTS: # USAGE:
# - Script 20 (CI Extraction): # Rscript 10_create_per_field_tiffs.R [project]
# Reads from field_tiles/{FIELD}/{DATE}.tif
# Adds CI calculation → outputs to field_tiles_CI/{FIELD}/{DATE}.tif (5-band)
# - Script 40 (Mosaic Creation):
# Reads from field_tiles_CI/{FIELD}/{DATE}.tif (via per-field weekly aggregation)
# Creates weekly_mosaic/{FIELD}/week_{WW}.tif
# #
# ARCHITECTURE: # Example (Windows PowerShell):
# This script uses field/date folder organization: # & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/10_create_per_field_tiffs.R angata
# field_tiles/
# ├── field_1/
# │ ├── 2024-01-15.tif
# │ └── 2024-01-16.tif
# └── field_2/
# ├── 2024-01-15.tif
# └── 2024-01-16.tif
# #
# Benefits: Upstream scripts iterate per-field → per-date, enabling clean # PARAMETERS:
# aggregation for mosaics (Script 40) and KPIs (Script 80/90). # - project: Project name (character) - angata, chemba, xinavane, esa, simba
# #
# ============================================================================== # CLIENT TYPES:
# - cane_supply (ANGATA): Yes - primary data organization script
# - agronomic_support (AURA): Yes - supports field-level analysis
#
# DEPENDENCIES:
# - Packages: terra, sf, tidyverse
# - Utils files: parameters_project.R, 00_common_utils.R, 10_create_per_field_tiffs_utils.R
# - External data: Field boundaries (pivot.geojson)
# - Data directories: merged_tif/, field_tiles/ (created if missing)
#
# NOTES:
# - Supports two-phase migration: legacy (merged_final_tif) and ongoing (merged_tif)
# - Automatically detects and handles field boundaries from pivot.geojson
# - Geometry validation and repair applied via st_make_valid()
# - Critical for downstream Scripts 20, 40, and KPI calculations
# - Creates per-field structure that enables efficient per-field processing
#
# RELATED ISSUES:
# SC-111: Script 10 refactoring and geometry repair
# SC-112: Utilities restructuring (uses 00_common_utils.R)
#
# ============================================================================
library(terra) library(terra)

View file

@ -1,25 +1,58 @@
# CI_EXTRACTION.R # ============================================================================
# ============== # SCRIPT 20: Canopy Index (CI) Extraction from Satellite Imagery
# This script processes satellite imagery to extract Canopy Index (CI) values for agricultural fields. # ============================================================================
# It handles image processing, masking, and extraction of statistics by field/sub-field. # PURPOSE:
# Supports both 4-band and 8-band PlanetScope data with automatic band detection and cloud masking. # Extract Canopy Index (CI) from 4-band or 8-band satellite imagery and
# mask by field boundaries. Supports automatic band detection, cloud masking
# with UDM2 (8-band), and per-field CI value extraction. Produces both
# per-field TIFFs and consolidated CI statistics for growth model input.
# #
# Usage: Rscript 02_ci_extraction.R [end_date] [offset] [project_dir] [data_source] # INPUT DATA:
# - end_date: End date for processing (YYYY-MM-DD format) # - Source: laravel_app/storage/app/{project}/field_tiles/{FIELD}/{DATE}.tif
# - offset: Number of days to look back from end_date # - Format: GeoTIFF (4-band RGB+NIR from Planet API, or 8-band with UDM2)
# - project_dir: Project directory name (e.g., "angata", "aura", "chemba") # - Requirement: Field boundaries (pivot.geojson) for masking
# - data_source: Data source directory - "merged_tif_8b" (default) or "merged_tif" (4-band) or "merged_final_tif"
# If tiles exist (daily_tiles_split/), they are used automatically
# #
# Examples: # OUTPUT DATA:
# # Angata 8-band data (with UDM cloud masking) # - Destination: laravel_app/storage/app/{project}/field_tiles_CI/{FIELD}/{DATE}.tif
# & 'C:\Program Files\R\R-4.4.3\bin\x64\Rscript' r_app/20_ci_extraction.R 2026-01-02 7 angata merged_tif_8b # - Format: GeoTIFF (5-band: R,G,B,NIR,CI as float32)
# # - Also exports: combined_CI/combined_CI_data.rds (wide format: fields × dates)
# # Aura 4-band data
# Rscript 20_ci_extraction.R 2025-11-26 7 aura merged_tif
# #
# # Auto-detects and uses tiles if available: # USAGE:
# Rscript 20_ci_extraction.R 2026-01-02 7 angata (uses tiles if daily_tiles_split/ exists) # Rscript 20_ci_extraction.R [end_date] [offset] [project] [data_source]
#
# Example (Windows PowerShell):
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/20_ci_extraction.R 2026-01-02 7 angata merged_tif
#
# PARAMETERS:
# - end_date: End date for processing (character, YYYY-MM-DD format)
# - offset: Days to look back from end_date (numeric, default 7)
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
# - data_source: Data source directory (character, optional) - "merged_tif" (default), "merged_tif_8b", "merged_final_tif"
#
# CLIENT TYPES:
# - cane_supply (ANGATA): Yes - core data processing
# - agronomic_support (AURA): Yes - supports field health monitoring
#
# DEPENDENCIES:
# - Packages: terra, sf, tidyverse, lubridate, readxl, furrr, future
# - Utils files: parameters_project.R, 00_common_utils.R, 20_ci_extraction_utils.R
# - External data: Field boundaries (pivot.geojson), harvest data (harvest.xlsx)
# - Data directories: field_tiles/, field_tiles_CI/, combined_CI/
#
# NOTES:
# - CI formula: (NIR - Red) / (NIR + Red); normalized to 0-5 range
# - 8-band data automatically cloud-masked using UDM2 (band 7-8)
# - 4-band data assumes clear-sky Planet PSScene imagery
# - Parallel processing via furrr for speed optimization
# - Output RDS uses wide format (fields as rows, dates as columns) for growth model
# - Critical dependency for Script 30 (growth model) and Script 80 (KPIs)
#
# RELATED ISSUES:
# SC-112: Utilities restructuring
# SC-108: Core pipeline improvements
#
# ============================================================================
# 1. Load required packages # 1. Load required packages
# ----------------------- # -----------------------

View file

@ -1,16 +1,53 @@
# 02b_CONVERT_CI_RDS_TO_CSV.R # ============================================================================
# ============================ # SCRIPT 21: Convert CI RDS to CSV (Python Compatibility Format)
# Convert combined_CI_data.rds (output of script 02) to CSV format for Python # ============================================================================
# This script runs AFTER script 02 (CI extraction) and creates a CSV that Python # PURPOSE:
# can use for harvest date detection WITHOUT requiring the 'model' column (which # Convert consolidated CI data from R's wide format (RDS) to Python-compatible
# comes from script 03 after interpolation and harvest dates are known). # long format (CSV). Prepares per-field CI time series for harvest detection
# models and Python ML workflows without requiring interpolated/modeled values.
# #
# Usage: Rscript 02b_convert_ci_rds_to_csv.R [project_dir] # INPUT DATA:
# - project_dir: Project directory name (e.g., "esa", "chemba", "angata") # - Source: laravel_app/storage/app/{project}/combined_CI/combined_CI_data.rds
# - Format: RDS (wide format: fields × dates with CI values)
# - Requirement: Script 20 must have completed CI extraction
# #
# Output: CSV file at laravel_app/storage/app/{project_dir}/Data/extracted_ci/cumulative_vals/ci_data_for_python.csv # OUTPUT DATA:
# Columns: field, sub_field, Date, FitData, DOY, value (alias for FitData) # - Destination: laravel_app/storage/app/{project}/Data/extracted_ci/cumulative_vals/
# - Format: CSV (long format)
# - Columns: field, sub_field, Date, FitData, DOY, value
# #
# USAGE:
# Rscript 21_convert_ci_rds_to_csv.R [project]
#
# Example (Windows PowerShell):
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/21_convert_ci_rds_to_csv.R angata
#
# PARAMETERS:
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
#
# CLIENT TYPES:
# - cane_supply (ANGATA): Yes - data export
# - agronomic_support (AURA): Yes - Python ML integration
#
# DEPENDENCIES:
# - Packages: tidyverse, lubridate, zoo
# - Utils files: parameters_project.R, 00_common_utils.R
# - Input data: combined_CI_data.rds from Script 20
# - Data directories: extracted_ci/cumulative_vals/
#
# NOTES:
# - Transformation: Wide format (fields as rows, dates as columns) → Long format
# - Time series: Preserves all CI values without interpolation
# - DOY (Day of Year): Calculated from date for seasonal analysis
# - Python integration: CSV format compatible with pandas/scikit-learn workflows
# - Used by: Python harvest detection models (harvest_date_prediction.py)
# - Optional: Run only when exporting to Python for ML model training
#
# RELATED ISSUES:
# SC-112: Utilities restructuring
# SC-108: Core pipeline improvements
#
# ============================================================================
suppressPackageStartupMessages({ suppressPackageStartupMessages({
library(tidyverse) library(tidyverse)

View file

@ -1,15 +1,54 @@
# filepath: c:\Users\timon\Resilience BV\4020 SCane ESA DEMO - Documenten\General\4020 SCDEMO Team\4020 TechnicalData\WP3\smartcane\r_app\interpolate_growth_model.R # ============================================================================
# SCRIPT 30: Growth Model Interpolation (CI Time Series)
# ============================================================================
# PURPOSE:
# Interpolate Canopy Index (CI) values across time to create continuous
# growth curves. Fills gaps in measurement dates, applies smoothing via
# LOESS, and generates daily CI estimates and cumulative statistics for
# each field. Enables downstream yield prediction and trend analysis.
# #
# INTERPOLATE_GROWTH_MODEL.R # INPUT DATA:
# ========================= # - Source: laravel_app/storage/app/{project}/combined_CI/combined_CI_data.rds
# This script interpolates CI (Chlorophyll Index) values between measurement dates # - Format: RDS (wide format: fields × dates with CI values)
# to create a continuous growth model. It generates daily values and cumulative # - Requirement: Field boundaries (pivot.geojson) and harvest data (harvest.xlsx)
# CI statistics for each field.
# #
# Usage: Rscript interpolate_growth_model.R [project_dir] [data_source] # OUTPUT DATA:
# - project_dir: Project directory name (e.g., "chemba") # - Destination: laravel_app/storage/app/{project}/interpolated_ci/
# - data_source: (Optional) Data source directory - "merged_tif" (default), "merged_tif_8b" # - Format: RDS files per field (daily CI estimates)
# & 'C:\Program Files\R\R-4.4.3\bin\x64\Rscript' r_app/30_interpolate_growth_model.R angata merged_tif # - Also exports: Growth model curves as RDS (cumulative CI, daily values)
#
# USAGE:
# Rscript 30_interpolate_growth_model.R [project]
#
# Example (Windows PowerShell):
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/30_interpolate_growth_model.R angata
#
# PARAMETERS:
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
#
# CLIENT TYPES:
# - cane_supply (ANGATA): Yes - core growth monitoring
# - agronomic_support (AURA): Yes - field health trend analysis
#
# DEPENDENCIES:
# - Packages: tidyverse, lubridate
# - Utils files: parameters_project.R, 00_common_utils.R, 30_growth_model_utils.R
# - External data: Field boundaries (pivot.geojson), harvest data (harvest.xlsx)
# - Input data: combined_CI_data.rds from Script 20
# - Data directories: interpolated_ci/ (created if missing)
#
# NOTES:
# - Interpolation method: LOESS smoothing with span = 0.3 (sensitive to local trends)
# - Gap-filling: Assumes continuous growth between measurements; skips clouds
# - Cumulative CI: Sum of daily interpolated values from planting to current date
# - Used by: Script 80 (KPI trends) and Script 12 (yield forecasting)
# - Critical for 8-week CV trend calculation and 4-week growth categorization
#
# RELATED ISSUES:
# SC-112: Utilities restructuring
# SC-108: Core pipeline improvements
#
# ============================================================================
# 1. Load required packages # 1. Load required packages
# ----------------------- # -----------------------

291
r_app/40_mosaic_creation.R Normal file
View file

@ -0,0 +1,291 @@
# ============================================================================
# SCRIPT 40: Weekly Mosaic Creation (CI Band Aggregation)
# ============================================================================
# PURPOSE:
# Create weekly 5-band (R, G, B, NIR, CI) mosaics from daily satellite
# imagery. Aggregates multi-day CI data into single weekly composite raster
# for field-level analysis. Supports per-field or single-file architectures.
#
# INPUT DATA:
# - Daily per-field TIFFs: laravel_app/storage/app/{project}/daily_tiles/{YYYY-MM-DD}/*.tif
# (or single-file mosaics: merged_tif/{YYYY-MM-DD}.tif + pivot.geojson masking)
# - CI data (RDS): laravel_app/storage/app/{project}/combined_CI/combined_CI_data.rds
# - Field boundaries: laravel_app/storage/app/{project}/pivot.geojson
#
# OUTPUT DATA:
# - Destination: laravel_app/storage/app/{project}/weekly_mosaic/
# - Format: 5-band GeoTIFF (uint16)
# - Naming: week_{WW}.tif (week number + year, e.g., week_35_2025.tif)
# - Spatial: Raster aligned to field boundaries; CRS preserved
#
# USAGE:
# Rscript 40_mosaic_creation.R [end_date] [offset] [project] [file_name] [data_source]
#
# Example (Windows PowerShell):
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation.R 2026-01-12 7 aura
#
# PARAMETERS:
# - end_date: End date (YYYY-MM-DD format); required for weekly aggregation
# - offset: Days to look back (typically 7 for one week)
# - project: Project name (aura, angata, chemba, xinavane, esa, simba)
# - file_name: Custom output filename (optional; default: week_{WW}_{YYYY}.tif)
# - data_source: Data folder (optional; auto-detects merged_tif or merged_tif_8b)
#
# CLIENT TYPES:
# - cane_supply (ANGATA): Yes - harvest readiness timeline depends on weekly mosaic
# - agronomic_support (AURA): Yes - KPI calculation requires weekly CI bands
#
# DEPENDENCIES:
# - Packages: sf, terra, tidyverse, lubridate, here
# - Utils files: parameters_project.R, 00_common_utils.R, 40_mosaic_creation_utils.R
# - Input data: Daily per-field TIFFs (Script 10) + CI extraction (Script 20)
# - Data: field boundaries (pivot.geojson), harvest dates (if available)
#
# NOTES:
# - Weekly aggregation: Combines 7 days of daily data into single composite
# - 5-band output: R, G, B, NIR, and Canopy Index (CI) derived from NDVI
# - Tiling support: Handles per-field TIFF architecture; auto-mosaics if needed
# - Data source auto-detection: Searches merged_tif/ or merged_tif_8b/ folders
# - Command-line driven: Designed for batch scheduling (cron/Task Scheduler)
# - Downstream: Script 80 (KPI calculation) depends on weekly_mosaic/ output
# - Performance: Multi-file mosaicing (~25 fields) takes 5-10 minutes per week
#
# RELATED ISSUES:
# SC-113: Script header standardization
# SC-112: Utilities restructuring
# SC-111: Script 10 geometry validation
#
# ============================================================================
# 1. Load required packages
# -----------------------
suppressPackageStartupMessages({
library(sf)
library(terra)
library(tidyverse)
library(lubridate)
library(here)
})
# 2. Process command line arguments and run mosaic creation
# ------------------------------------------------------
main <- function() {
# Capture command line arguments
args <- commandArgs(trailingOnly = TRUE)
# Process project_dir argument with default
if (length(args) >= 3 && !is.na(args[3])) {
project_dir <- as.character(args[3])
} else if (exists("project_dir", envir = .GlobalEnv)) {
project_dir <- get("project_dir", envir = .GlobalEnv)
} else {
# Default project directory
project_dir <- "angata"
message("No project_dir provided. Using default:", project_dir)
}
# Make project_dir available globally so parameters_project.R can use it
assign("project_dir", project_dir, envir = .GlobalEnv)
# Process end_date argument with default
if (length(args) >= 1 && !is.na(args[1])) {
# Parse date explicitly in YYYY-MM-DD format from command line
end_date <- as.Date(args[1], format = "%Y-%m-%d")
if (is.na(end_date)) {
message("Invalid end_date provided. Using current date.")
end_date <- Sys.Date()
}
} else if (exists("end_date_str", envir = .GlobalEnv)) {
end_date <- as.Date(get("end_date_str", envir = .GlobalEnv))
} else {
# Default to current date if no argument is provided
end_date <- Sys.Date()
message("No end_date provided. Using current date: ", format(end_date))
}
# Process offset argument with default
if (length(args) >= 2 && !is.na(args[2])) {
offset <- as.numeric(args[2])
if (is.na(offset) || offset <= 0) {
message("Invalid offset provided. Using default (7 days).")
offset <- 7
}
} else {
# Default to 7 days if no argument is provided
offset <- 7
message("No offset provided. Using default:", offset, "days")
}
# Process data_source argument (optional, passed from pipeline)
# If provided, use it; otherwise auto-detect
data_source_from_args <- NULL
if (length(args) >= 5 && !is.na(args[5]) && nchar(args[5]) > 0) {
data_source_from_args <- as.character(args[5])
message("Data source explicitly provided via arguments: ", data_source_from_args)
}
# 3. Initialize project configuration
# --------------------------------
# Detect which data source directory exists (merged_tif or merged_tif_8b)
# IMPORTANT: Only consider a folder as valid if it contains actual files
laravel_storage <- here::here("laravel_app/storage/app", project_dir)
# Load centralized path structure
tryCatch({
source("r_app/parameters_project.R")
paths <- setup_project_directories(project_dir)
}, error = function(e) {
message("Note: Could not open files from r_app directory")
message("Attempting to source from default directory instead...")
tryCatch({
source("parameters_project.R")
paths <- setup_project_directories(project_dir)
message("✓ Successfully sourced files from default directory")
}, error = function(e) {
stop("Failed to source required files from both 'r_app' and default directories.")
})
})
data_source <- if (has_8b_data) {
message("Auto-detected data source: merged_tif_8b (8-band optimized) - contains files")
"merged_tif_8b"
} else if (has_legacy_data) {
message("Auto-detected data source: merged_tif (legacy 4-band) - contains files")
"merged_tif"
} else {
message("Warning: No valid data source found (both folders empty or missing). Using default: merged_tif")
"merged_tif"
}
}
# Set global data_source for parameters_project.R
assign("data_source", data_source, envir = .GlobalEnv)
tryCatch({
source("r_app/parameters_project.R")
source("r_app/00_common_utils.R")
source("r_app/40_mosaic_creation_utils.R")
safe_log(paste("Successfully sourced files from 'r_app' directory."))
}, error = function(e) {
message("Note: Could not open files from r_app directory")
message("Attempting to source from default directory instead...")
tryCatch({
source("parameters_project.R")
paths <- setup_project_directories(project_dir)
message("✓ Successfully sourced files from default directory")
}, error = function(e) {
stop("Failed to source required files from both 'r_app' and default directories.")
})
})
# Use centralized paths (no need to manually construct or create dirs)
merged_final <- paths$growth_model_interpolated_dir # or merged_final_tif if needed
daily_vrt <- paths$vrt_dir
safe_log(paste("Using growth model/mosaic directory:", merged_final))
safe_log(paste("Using daily VRT directory:", daily_vrt))
# 4. Generate date range for processing
# ---------------------------------
dates <- date_list(end_date, offset)
safe_log(paste("Processing data for week", dates$week, "of", dates$year))
# Create output filename
# Only use custom filename if explicitly provided (not empty string)
file_name_tif <- if (length(args) >= 4 && !is.na(args[4]) && nchar(args[4]) > 0) {
as.character(args[4])
} else {
paste0("week_", sprintf("%02d", dates$week), "_", dates$year, ".tif")
}
safe_log(paste("Output will be saved as:", file_name_tif))
# 5. Create weekly mosaics - route based on project tile detection
# ---------------------------------------------------------------
# The use_tile_mosaic flag is auto-detected by parameters_project.R
# based on whether tiles exist in merged_final_tif/
if (!exists("use_tile_mosaic")) {
# Fallback detection if flag not set (shouldn't happen)
merged_final_dir <- file.path(laravel_storage, "merged_final_tif")
tile_detection <- detect_tile_structure_from_merged_final(merged_final_dir)
use_tile_mosaic <- tile_detection$has_tiles
}
if (use_tile_mosaic) {
# TILE-BASED APPROACH: Create per-tile weekly MAX mosaics
# This is used for projects like Angata with large ROIs requiring spatial partitioning
# Input data comes from merged_final_tif/{grid_size}/{DATE}/{DATE}_XX.tif (5-band tiles from script 20)
tryCatch({
safe_log("Starting per-tile mosaic creation (tile-based approach)...")
# Detect grid size from merged_final_tif folder structure
# Expected: merged_final_tif/5x5/ or merged_final_tif/10x10/ etc.
merged_final_base <- file.path(laravel_storage, "merged_final_tif")
grid_subfolders <- list.dirs(merged_final_base, full.names = FALSE, recursive = FALSE)
# Look for grid size patterns like "5x5", "10x10", "20x20"
grid_patterns <- grep("^\\d+x\\d+$", grid_subfolders, value = TRUE)
if (length(grid_patterns) == 0) {
stop("No grid size subfolder found in merged_final_tif/ (expected: 5x5, 10x10, etc.)")
}
grid_size <- grid_patterns[1] # Use first grid size found
safe_log(paste("Detected grid size:", grid_size))
# Point to the grid-specific merged_final_tif directory
merged_final_with_grid <- file.path(merged_final_base, grid_size)
# Set output directory for per-tile mosaics, organized by grid size (from centralized paths)
# Output: weekly_tile_max/{grid_size}/week_WW_YYYY_TT.tif
tile_output_base <- file.path(paths$weekly_tile_max_dir, grid_size)
# Note: no dir.create needed - paths$weekly_tile_max_dir already created by setup_project_directories()
dir.create(tile_output_base, recursive = TRUE, showWarnings = FALSE) # Create grid-size subfolder
created_tile_files <- create_weekly_mosaic_from_tiles(
dates = dates,
merged_final_dir = merged_final_with_grid,
tile_output_dir = tile_output_base,
field_boundaries = field_boundaries
)
safe_log(paste("✓ Per-tile mosaic creation completed - created",
length(created_tile_files), "tile files"))
}, error = function(e) {
safe_log(paste("ERROR in tile-based mosaic creation:", e$message), "ERROR")
traceback()
stop("Mosaic creation failed")
})
} else {
# SINGLE-FILE APPROACH: Create single weekly mosaic file
# This is used for legacy projects (ESA, Chemba, Aura) expecting single-file output
tryCatch({
safe_log("Starting single-file mosaic creation (backward-compatible approach)...")
# Set output directory for single-file mosaics (from centralized paths)
single_file_output_dir <- paths$weekly_mosaic_dir
created_file <- create_weekly_mosaic(
dates = dates,
field_boundaries = field_boundaries,
daily_vrt_dir = daily_vrt,
merged_final_dir = merged_final,
output_dir = single_file_output_dir,
file_name_tif = file_name_tif,
create_plots = FALSE
)
safe_log(paste("✓ Single-file mosaic creation completed:", created_file))
}, error = function(e) {
safe_log(paste("ERROR in single-file mosaic creation:", e$message), "ERROR")
traceback()
stop("Mosaic creation failed")
})
}
}
if (sys.nframe() == 0) {
main()
}

View file

@ -1,25 +1,54 @@
# 40_MOSAIC_CREATION_PER_FIELD.R # ============================================================================
# =============================== # SCRIPT 40: Weekly Mosaic Creation (Per-Field CI Aggregation)
# Per-Field Weekly Mosaic Creation # ============================================================================
# PURPOSE:
# Aggregate daily per-field CI TIFFs into weekly mosaics. Handles multi-date
# merging, cloud masking, and produces 5-band weekly output for reporting
# and KPI calculations. Supports both per-field and grid-based tile architecture.
# #
# Creates weekly mosaics FROM per-field daily CI TIFFs (output from Script 20) # INPUT DATA:
# TO per-field weekly CI TIFFs (input for Scripts 90/91 reporting). # - Source: laravel_app/storage/app/{project}/field_tiles_CI/{FIELD}/{DATE}.tif
# - Format: GeoTIFF (5-band: R,G,B,NIR,CI as float32)
# - Dates: All available dates within week range
# #
# ARCHITECTURE: # OUTPUT DATA:
# Input: field_tiles_CI/{FIELD}/{DATE}.tif (5-band daily, per-field from Script 20) # - Destination: laravel_app/storage/app/{project}/weekly_mosaic/{FIELD}/
# Output: weekly_mosaic/{FIELD}/week_WW_YYYY.tif (5-band weekly, per-field) # - Format: GeoTIFF (5-band: R,G,B,NIR,CI)
# - Naming: week_WW_YYYY.tif (WW = ISO week, YYYY = ISO year)
# #
# USAGE: # USAGE:
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R [end_date] [offset] [project_dir] # Rscript 40_mosaic_creation_per_field.R [end_date] [offset] [project]
# #
# ARGUMENTS: # Example (Windows PowerShell):
# end_date: End date for processing (YYYY-MM-DD format, default: today) # & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R 2026-01-12 7 angata
# offset: Days to look back from end_date (typically 7 for one week, default: 7)
# project_dir: Project directory (e.g., "aura", "angata", "chemba", "esa", default: "angata")
# #
# EXAMPLES: # PARAMETERS:
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R 2026-01-12 7 aura # - end_date: End date for processing (character, YYYY-MM-DD format, default today)
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R 2025-12-31 7 angata # - offset: Days to look back (numeric, default 7 for one week)
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
#
# CLIENT TYPES:
# - cane_supply (ANGATA): Yes - weekly monitoring
# - agronomic_support (AURA): Yes - field health reporting
#
# DEPENDENCIES:
# - Packages: terra, sf, tidyverse, lubridate
# - Utils files: parameters_project.R, 00_common_utils.R, 40_mosaic_creation_per_field_utils.R
# - Input data: Daily per-field CI TIFFs from Script 20
# - Data directories: field_tiles_CI/, weekly_mosaic/
#
# NOTES:
# - Aggregation method: Maximum CI value per pixel across week (handles clouds)
# - ISO week-year used for consistent date handling across year boundaries
# - Supports both single-file and tiled mosaic architectures
# - Output feeds Scripts 80 (KPI calculations) and 90/91 (reporting)
# - Critical for trend analysis: week-over-week CI comparison
#
# RELATED ISSUES:
# SC-112: Script 40 cleanup (deleted legacy mosaic utils files)
# SC-108: Core pipeline improvements
#
# ============================================================================
# 1. Load required packages # 1. Load required packages
# ----------------------- # -----------------------

View file

@ -1,28 +1,59 @@
# 80_CALCULATE_KPIS.R (CONSOLIDATED KPI CALCULATION)
# ============================================================================ # ============================================================================
# UNIFIED KPI CALCULATION SCRIPT # SCRIPT 80: Key Performance Indicator (KPI) Calculation
# # ============================================================================
# This script combines: # PURPOSE:
# 1. Per-field weekly analysis (from 09c: field-level trends, phases, statuses) # Calculate per-field and farm-level KPIs from weekly CI mosaics. Computes
# 2. Farm-level KPI metrics (from old 09: 6 high-level indicators) # field uniformity, growth trends (4-week and 8-week), area change detection,
# TCH forecasts, stress identification, and weed presence. Generates
# comprehensive Excel/CSV/RDS exports for dashboards and stakeholder reporting.
# #
# FEATURES: # INPUT DATA:
# - Per-field analysis with SC-64 enhancements (4-week trends, CI percentiles, etc.) # - Source 1: laravel_app/storage/app/{project}/weekly_mosaic/{FIELD}/week_*.tif
# - Farm-level KPI calculation (6 metrics for executive overview) # - Source 2: Field boundaries (pivot.geojson) and harvest data (harvest.xlsx)
# - Parallel processing (tile-aware, 1000+ fields supported) # - Source 3: Historical field stats (RDS from previous weeks)
# - Comprehensive Excel + RDS + CSV exports (21 columns per spec)
# - Test mode for development
# CRITICAL INTEGRATIONS:
# #
# 1. IMMINENT_PROB FROM HARVEST MODEL (MODEL_307) # OUTPUT DATA:
# [✓] Load script 31 output: {project}_week_{WW}_{YYYY}.csv # - Destination: laravel_app/storage/app/{project}/output/
# Columns: field, imminent_prob, detected_prob, week, year # - Format: Excel (.xlsx), CSV (.csv), RDS (.rds)
# [✓] LEFT JOIN to field_analysis_df by field # - Files: {project}_field_analysis_week{WW}_{YYYY}.xlsx + metadata
# [✓] Use actual harvest probability data instead of placeholder
# #
# 2. AGE FROM HARVEST.XLSX (SCRIPTS 22 & 23) # USAGE:
# [✓] Load harvest.xlsx with planting_date (season_start) # Rscript 80_calculate_kpis.R [project] [week] [year]
#
# Example (Windows PowerShell):
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/80_calculate_kpis.R angata 5 2026
#
# PARAMETERS:
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
# - week: ISO week number (numeric, 1-53, default current week)
# - year: ISO year (numeric, default current year)
#
# CLIENT TYPES:
# - cane_supply (ANGATA): Yes - uses 80_utils_cane_supply.R (placeholder)
# - agronomic_support (AURA): Yes - uses 80_utils_agronomic_support.R (6 KPI funcs)
#
# DEPENDENCIES:
# - Packages: terra, sf, tidyverse, lubridate, writexl, spdep
# - Utils files: parameters_project.R, 00_common_utils.R, 80_utils_agronomic_support.R, 80_utils_cane_supply.R
# - External data: Field boundaries (pivot.geojson), harvest data (harvest.xlsx)
# - Input data: Weekly mosaic TIFFs (Script 40 output)
# - Data directories: weekly_mosaic/, output/ (created if missing)
#
# NOTES:
# - KPIs calculated: Field Uniformity (CV), Area Change (pixel %), TCH Forecast,
# Growth/Decline Trend, Weed Presence (spatial autocorrelation), Gap Filling %
# - Client-aware: Conditional sourcing based on client_config$script_90_compatible
# - Exports: 21-column output with field-level metrics, phase, status, alerts
# - Supports 4-week and 8-week trend analysis from historical RDS cache
# - Critical dependency for Scripts 90/91 (reporting/dashboards)
# - Uses Moran's I for spatial clustering detection (weed/stress patterns)
#
# RELATED ISSUES:
# SC-112: Script 80 utilities restructuring (common + client-aware modules)
# SC-108: Core pipeline improvements
# SC-100: KPI definition and formula documentation
#
# ============================================================================
# [✓] Extract planting dates per field # [✓] Extract planting dates per field
# [✓] Calculate Age_week = difftime(report_date, planting_date, units="weeks") # [✓] Calculate Age_week = difftime(report_date, planting_date, units="weeks")
# #

View file

@ -0,0 +1,54 @@
# ============================================================================
# SCRIPT XX: [DESCRIPTIVE TITLE]
# ============================================================================
# PURPOSE:
# [What this script does in 2-3 sentences. Example: Downloads satellite
# imagery from Planet API, processes 4-band RGB+NIR data, and saves
# merged GeoTIFFs for use by downstream analysis stages.]
#
# INPUT DATA:
# - Source: [Which directory/files it reads. Example: Planet API, hardcoded bbox]
# - Format: [TIFF, RDS, GeoJSON, etc. Example: 4-band uint16 GeoTIFF]
# - Location: [Full path example]
#
# OUTPUT DATA:
# - Destination: [Which directory/files it creates. Example: laravel_app/storage/app/{project}/merged_tif/]
# - Format: [TIFF, RDS, CSV, etc. Example: Single-band float32 GeoTIFF]
# - Naming convention: [Example: {YYYY-MM-DD}.tif]
#
# USAGE:
# Rscript XX_script_name.R [arg1] [arg2] [arg3]
#
# Example (Windows PowerShell):
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/XX_script_name.R angata 2026-01-15
#
# PARAMETERS:
# - arg1: [Description, type, and valid values. Example: project (character) - angata, chemba, xinavane, esa, simba]
# - arg2: [Description, type, and valid values. Example: date (character, optional) - ISO format YYYY-MM-DD; default today]
#
# CLIENT TYPES:
# - cane_supply (ANGATA): [Yes/No - if Yes, briefly explain any differences]
# - agronomic_support (AURA): [Yes/No - if Yes, briefly explain any differences]
#
# DEPENDENCIES:
# - Packages: [dplyr, tidyr, terra, sf, readr, writexl, etc.]
# - Utils files: [parameters_project.R, 00_common_utils.R, XX_utils.R, etc.]
# - External data: [harvest.xlsx, pivot.geojson, etc.]
# - Data directories: [laravel_app/storage/app/{project}/]
#
# NOTES:
# [Any special considerations, assumptions, or future improvements.
# Example: Cloud filtering uses CI >= 0.5 threshold. Multi-field support
# implemented via field geometry masking from pivot.geojson.]
#
# RELATED ISSUES:
# SC-XXX: [Brief description of related work]
#
# HISTORY:
# 2026-02-03: [Description of change, if refactored or enhanced]
# ============================================================================
# NOTE: This is a TEMPLATE file for documentation purposes only.
# When creating a new script or updating an existing one, copy this template
# and fill in all sections with accurate information about your specific script.
# Then delete this comment block.