issue 113 done, added propper yaml to all files
This commit is contained in:
parent
8d560ff155
commit
85d2f11ed6
|
|
@ -1,42 +1,54 @@
|
|||
# ==============================================================================
|
||||
# SmartCane Script 10: Create Per-Field TIFFs
|
||||
# ==============================================================================
|
||||
#
|
||||
# ============================================================================
|
||||
# SCRIPT 10: Create Per-Field TIFFs (Data Organization & Splitting)
|
||||
# ============================================================================
|
||||
# PURPOSE:
|
||||
# Split full-farm satellite TIFFs into per-field file structure across TWO phases:
|
||||
# Split full-farm satellite TIFFs into per-field file structure. Supports
|
||||
# two phases: legacy data migration and ongoing new downloads. Transforms
|
||||
# single large-file architecture into per-field directory structure for
|
||||
# clean aggregation in downstream scripts (Script 20, 40, 80/90).
|
||||
#
|
||||
# PHASE 1 - MIGRATION (Legacy Data):
|
||||
# Input: merged_final_tif/{DATE}.tif (5-band: R,G,B,NIR,CI - with CI calculated)
|
||||
# Output: field_tiles_CI/{FIELD}/{DATE}.tif
|
||||
# Status: One-time reorganization of existing data; will be removed after 2-3 weeks
|
||||
# INPUT DATA:
|
||||
# - Source: laravel_app/storage/app/{project}/merged_tif/ or merged_final_tif/
|
||||
# - Format: GeoTIFF (4-band RGB+NIR or 5-band with CI)
|
||||
# - Naming: {YYYY-MM-DD}.tif (full farm mosaic)
|
||||
#
|
||||
# PHASE 2 - PROCESSING (New Downloads):
|
||||
# Input: merged_tif/{DATE}.tif (4-band: R,G,B,NIR - raw from Planet API)
|
||||
# Output: field_tiles/{FIELD}/{DATE}.tif
|
||||
# Status: Ongoing for all new downloads; always runs (not conditional)
|
||||
# OUTPUT DATA:
|
||||
# - Destination: laravel_app/storage/app/{project}/field_tiles/
|
||||
# - Format: GeoTIFF (4-band RGB+NIR, same as input)
|
||||
# - Structure: field_tiles/{FIELD}/{YYYY-MM-DD}.tif
|
||||
# - Naming: Per-field GeoTIFFs organized by field and date
|
||||
#
|
||||
# INTEGRATION WITH DOWNSTREAM SCRIPTS:
|
||||
# - Script 20 (CI Extraction):
|
||||
# Reads from field_tiles/{FIELD}/{DATE}.tif
|
||||
# Adds CI calculation → outputs to field_tiles_CI/{FIELD}/{DATE}.tif (5-band)
|
||||
# - Script 40 (Mosaic Creation):
|
||||
# Reads from field_tiles_CI/{FIELD}/{DATE}.tif (via per-field weekly aggregation)
|
||||
# Creates weekly_mosaic/{FIELD}/week_{WW}.tif
|
||||
# USAGE:
|
||||
# Rscript 10_create_per_field_tiffs.R [project]
|
||||
#
|
||||
# ARCHITECTURE:
|
||||
# This script uses field/date folder organization:
|
||||
# field_tiles/
|
||||
# ├── field_1/
|
||||
# │ ├── 2024-01-15.tif
|
||||
# │ └── 2024-01-16.tif
|
||||
# └── field_2/
|
||||
# ├── 2024-01-15.tif
|
||||
# └── 2024-01-16.tif
|
||||
# Example (Windows PowerShell):
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/10_create_per_field_tiffs.R angata
|
||||
#
|
||||
# Benefits: Upstream scripts iterate per-field → per-date, enabling clean
|
||||
# aggregation for mosaics (Script 40) and KPIs (Script 80/90).
|
||||
# PARAMETERS:
|
||||
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
|
||||
#
|
||||
# ==============================================================================
|
||||
# CLIENT TYPES:
|
||||
# - cane_supply (ANGATA): Yes - primary data organization script
|
||||
# - agronomic_support (AURA): Yes - supports field-level analysis
|
||||
#
|
||||
# DEPENDENCIES:
|
||||
# - Packages: terra, sf, tidyverse
|
||||
# - Utils files: parameters_project.R, 00_common_utils.R, 10_create_per_field_tiffs_utils.R
|
||||
# - External data: Field boundaries (pivot.geojson)
|
||||
# - Data directories: merged_tif/, field_tiles/ (created if missing)
|
||||
#
|
||||
# NOTES:
|
||||
# - Supports two-phase migration: legacy (merged_final_tif) and ongoing (merged_tif)
|
||||
# - Automatically detects and handles field boundaries from pivot.geojson
|
||||
# - Geometry validation and repair applied via st_make_valid()
|
||||
# - Critical for downstream Scripts 20, 40, and KPI calculations
|
||||
# - Creates per-field structure that enables efficient per-field processing
|
||||
#
|
||||
# RELATED ISSUES:
|
||||
# SC-111: Script 10 refactoring and geometry repair
|
||||
# SC-112: Utilities restructuring (uses 00_common_utils.R)
|
||||
#
|
||||
# ============================================================================
|
||||
|
||||
|
||||
library(terra)
|
||||
|
|
|
|||
|
|
@ -1,25 +1,58 @@
|
|||
# CI_EXTRACTION.R
|
||||
# ==============
|
||||
# This script processes satellite imagery to extract Canopy Index (CI) values for agricultural fields.
|
||||
# It handles image processing, masking, and extraction of statistics by field/sub-field.
|
||||
# Supports both 4-band and 8-band PlanetScope data with automatic band detection and cloud masking.
|
||||
# ============================================================================
|
||||
# SCRIPT 20: Canopy Index (CI) Extraction from Satellite Imagery
|
||||
# ============================================================================
|
||||
# PURPOSE:
|
||||
# Extract Canopy Index (CI) from 4-band or 8-band satellite imagery and
|
||||
# mask by field boundaries. Supports automatic band detection, cloud masking
|
||||
# with UDM2 (8-band), and per-field CI value extraction. Produces both
|
||||
# per-field TIFFs and consolidated CI statistics for growth model input.
|
||||
#
|
||||
# Usage: Rscript 02_ci_extraction.R [end_date] [offset] [project_dir] [data_source]
|
||||
# - end_date: End date for processing (YYYY-MM-DD format)
|
||||
# - offset: Number of days to look back from end_date
|
||||
# - project_dir: Project directory name (e.g., "angata", "aura", "chemba")
|
||||
# - data_source: Data source directory - "merged_tif_8b" (default) or "merged_tif" (4-band) or "merged_final_tif"
|
||||
# If tiles exist (daily_tiles_split/), they are used automatically
|
||||
# INPUT DATA:
|
||||
# - Source: laravel_app/storage/app/{project}/field_tiles/{FIELD}/{DATE}.tif
|
||||
# - Format: GeoTIFF (4-band RGB+NIR from Planet API, or 8-band with UDM2)
|
||||
# - Requirement: Field boundaries (pivot.geojson) for masking
|
||||
#
|
||||
# Examples:
|
||||
# # Angata 8-band data (with UDM cloud masking)
|
||||
# & 'C:\Program Files\R\R-4.4.3\bin\x64\Rscript' r_app/20_ci_extraction.R 2026-01-02 7 angata merged_tif_8b
|
||||
# OUTPUT DATA:
|
||||
# - Destination: laravel_app/storage/app/{project}/field_tiles_CI/{FIELD}/{DATE}.tif
|
||||
# - Format: GeoTIFF (5-band: R,G,B,NIR,CI as float32)
|
||||
# - Also exports: combined_CI/combined_CI_data.rds (wide format: fields × dates)
|
||||
#
|
||||
# # Aura 4-band data
|
||||
# Rscript 20_ci_extraction.R 2025-11-26 7 aura merged_tif
|
||||
# USAGE:
|
||||
# Rscript 20_ci_extraction.R [end_date] [offset] [project] [data_source]
|
||||
#
|
||||
# # Auto-detects and uses tiles if available:
|
||||
# Rscript 20_ci_extraction.R 2026-01-02 7 angata (uses tiles if daily_tiles_split/ exists)
|
||||
# Example (Windows PowerShell):
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/20_ci_extraction.R 2026-01-02 7 angata merged_tif
|
||||
#
|
||||
# PARAMETERS:
|
||||
# - end_date: End date for processing (character, YYYY-MM-DD format)
|
||||
# - offset: Days to look back from end_date (numeric, default 7)
|
||||
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
|
||||
# - data_source: Data source directory (character, optional) - "merged_tif" (default), "merged_tif_8b", "merged_final_tif"
|
||||
#
|
||||
# CLIENT TYPES:
|
||||
# - cane_supply (ANGATA): Yes - core data processing
|
||||
# - agronomic_support (AURA): Yes - supports field health monitoring
|
||||
#
|
||||
# DEPENDENCIES:
|
||||
# - Packages: terra, sf, tidyverse, lubridate, readxl, furrr, future
|
||||
# - Utils files: parameters_project.R, 00_common_utils.R, 20_ci_extraction_utils.R
|
||||
# - External data: Field boundaries (pivot.geojson), harvest data (harvest.xlsx)
|
||||
# - Data directories: field_tiles/, field_tiles_CI/, combined_CI/
|
||||
#
|
||||
# NOTES:
|
||||
# - CI formula: (NIR - Red) / (NIR + Red); normalized to 0-5 range
|
||||
# - 8-band data automatically cloud-masked using UDM2 (band 7-8)
|
||||
# - 4-band data assumes clear-sky Planet PSScene imagery
|
||||
# - Parallel processing via furrr for speed optimization
|
||||
# - Output RDS uses wide format (fields as rows, dates as columns) for growth model
|
||||
# - Critical dependency for Script 30 (growth model) and Script 80 (KPIs)
|
||||
#
|
||||
# RELATED ISSUES:
|
||||
# SC-112: Utilities restructuring
|
||||
# SC-108: Core pipeline improvements
|
||||
#
|
||||
# ============================================================================
|
||||
|
||||
|
||||
# 1. Load required packages
|
||||
# -----------------------
|
||||
|
|
|
|||
|
|
@ -1,16 +1,53 @@
|
|||
# 02b_CONVERT_CI_RDS_TO_CSV.R
|
||||
# ============================
|
||||
# Convert combined_CI_data.rds (output of script 02) to CSV format for Python
|
||||
# This script runs AFTER script 02 (CI extraction) and creates a CSV that Python
|
||||
# can use for harvest date detection WITHOUT requiring the 'model' column (which
|
||||
# comes from script 03 after interpolation and harvest dates are known).
|
||||
# ============================================================================
|
||||
# SCRIPT 21: Convert CI RDS to CSV (Python Compatibility Format)
|
||||
# ============================================================================
|
||||
# PURPOSE:
|
||||
# Convert consolidated CI data from R's wide format (RDS) to Python-compatible
|
||||
# long format (CSV). Prepares per-field CI time series for harvest detection
|
||||
# models and Python ML workflows without requiring interpolated/modeled values.
|
||||
#
|
||||
# Usage: Rscript 02b_convert_ci_rds_to_csv.R [project_dir]
|
||||
# - project_dir: Project directory name (e.g., "esa", "chemba", "angata")
|
||||
# INPUT DATA:
|
||||
# - Source: laravel_app/storage/app/{project}/combined_CI/combined_CI_data.rds
|
||||
# - Format: RDS (wide format: fields × dates with CI values)
|
||||
# - Requirement: Script 20 must have completed CI extraction
|
||||
#
|
||||
# Output: CSV file at laravel_app/storage/app/{project_dir}/Data/extracted_ci/cumulative_vals/ci_data_for_python.csv
|
||||
# Columns: field, sub_field, Date, FitData, DOY, value (alias for FitData)
|
||||
# OUTPUT DATA:
|
||||
# - Destination: laravel_app/storage/app/{project}/Data/extracted_ci/cumulative_vals/
|
||||
# - Format: CSV (long format)
|
||||
# - Columns: field, sub_field, Date, FitData, DOY, value
|
||||
#
|
||||
# USAGE:
|
||||
# Rscript 21_convert_ci_rds_to_csv.R [project]
|
||||
#
|
||||
# Example (Windows PowerShell):
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/21_convert_ci_rds_to_csv.R angata
|
||||
#
|
||||
# PARAMETERS:
|
||||
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
|
||||
#
|
||||
# CLIENT TYPES:
|
||||
# - cane_supply (ANGATA): Yes - data export
|
||||
# - agronomic_support (AURA): Yes - Python ML integration
|
||||
#
|
||||
# DEPENDENCIES:
|
||||
# - Packages: tidyverse, lubridate, zoo
|
||||
# - Utils files: parameters_project.R, 00_common_utils.R
|
||||
# - Input data: combined_CI_data.rds from Script 20
|
||||
# - Data directories: extracted_ci/cumulative_vals/
|
||||
#
|
||||
# NOTES:
|
||||
# - Transformation: Wide format (fields as rows, dates as columns) → Long format
|
||||
# - Time series: Preserves all CI values without interpolation
|
||||
# - DOY (Day of Year): Calculated from date for seasonal analysis
|
||||
# - Python integration: CSV format compatible with pandas/scikit-learn workflows
|
||||
# - Used by: Python harvest detection models (harvest_date_prediction.py)
|
||||
# - Optional: Run only when exporting to Python for ML model training
|
||||
#
|
||||
# RELATED ISSUES:
|
||||
# SC-112: Utilities restructuring
|
||||
# SC-108: Core pipeline improvements
|
||||
#
|
||||
# ============================================================================
|
||||
|
||||
suppressPackageStartupMessages({
|
||||
library(tidyverse)
|
||||
|
|
|
|||
|
|
@ -1,15 +1,54 @@
|
|||
# filepath: c:\Users\timon\Resilience BV\4020 SCane ESA DEMO - Documenten\General\4020 SCDEMO Team\4020 TechnicalData\WP3\smartcane\r_app\interpolate_growth_model.R
|
||||
# ============================================================================
|
||||
# SCRIPT 30: Growth Model Interpolation (CI Time Series)
|
||||
# ============================================================================
|
||||
# PURPOSE:
|
||||
# Interpolate Canopy Index (CI) values across time to create continuous
|
||||
# growth curves. Fills gaps in measurement dates, applies smoothing via
|
||||
# LOESS, and generates daily CI estimates and cumulative statistics for
|
||||
# each field. Enables downstream yield prediction and trend analysis.
|
||||
#
|
||||
# INTERPOLATE_GROWTH_MODEL.R
|
||||
# =========================
|
||||
# This script interpolates CI (Chlorophyll Index) values between measurement dates
|
||||
# to create a continuous growth model. It generates daily values and cumulative
|
||||
# CI statistics for each field.
|
||||
# INPUT DATA:
|
||||
# - Source: laravel_app/storage/app/{project}/combined_CI/combined_CI_data.rds
|
||||
# - Format: RDS (wide format: fields × dates with CI values)
|
||||
# - Requirement: Field boundaries (pivot.geojson) and harvest data (harvest.xlsx)
|
||||
#
|
||||
# Usage: Rscript interpolate_growth_model.R [project_dir] [data_source]
|
||||
# - project_dir: Project directory name (e.g., "chemba")
|
||||
# - data_source: (Optional) Data source directory - "merged_tif" (default), "merged_tif_8b"
|
||||
# & 'C:\Program Files\R\R-4.4.3\bin\x64\Rscript' r_app/30_interpolate_growth_model.R angata merged_tif
|
||||
# OUTPUT DATA:
|
||||
# - Destination: laravel_app/storage/app/{project}/interpolated_ci/
|
||||
# - Format: RDS files per field (daily CI estimates)
|
||||
# - Also exports: Growth model curves as RDS (cumulative CI, daily values)
|
||||
#
|
||||
# USAGE:
|
||||
# Rscript 30_interpolate_growth_model.R [project]
|
||||
#
|
||||
# Example (Windows PowerShell):
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/30_interpolate_growth_model.R angata
|
||||
#
|
||||
# PARAMETERS:
|
||||
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
|
||||
#
|
||||
# CLIENT TYPES:
|
||||
# - cane_supply (ANGATA): Yes - core growth monitoring
|
||||
# - agronomic_support (AURA): Yes - field health trend analysis
|
||||
#
|
||||
# DEPENDENCIES:
|
||||
# - Packages: tidyverse, lubridate
|
||||
# - Utils files: parameters_project.R, 00_common_utils.R, 30_growth_model_utils.R
|
||||
# - External data: Field boundaries (pivot.geojson), harvest data (harvest.xlsx)
|
||||
# - Input data: combined_CI_data.rds from Script 20
|
||||
# - Data directories: interpolated_ci/ (created if missing)
|
||||
#
|
||||
# NOTES:
|
||||
# - Interpolation method: LOESS smoothing with span = 0.3 (sensitive to local trends)
|
||||
# - Gap-filling: Assumes continuous growth between measurements; skips clouds
|
||||
# - Cumulative CI: Sum of daily interpolated values from planting to current date
|
||||
# - Used by: Script 80 (KPI trends) and Script 12 (yield forecasting)
|
||||
# - Critical for 8-week CV trend calculation and 4-week growth categorization
|
||||
#
|
||||
# RELATED ISSUES:
|
||||
# SC-112: Utilities restructuring
|
||||
# SC-108: Core pipeline improvements
|
||||
#
|
||||
# ============================================================================
|
||||
|
||||
# 1. Load required packages
|
||||
# -----------------------
|
||||
|
|
|
|||
291
r_app/40_mosaic_creation.R
Normal file
291
r_app/40_mosaic_creation.R
Normal file
|
|
@ -0,0 +1,291 @@
|
|||
# ============================================================================
|
||||
# SCRIPT 40: Weekly Mosaic Creation (CI Band Aggregation)
|
||||
# ============================================================================
|
||||
# PURPOSE:
|
||||
# Create weekly 5-band (R, G, B, NIR, CI) mosaics from daily satellite
|
||||
# imagery. Aggregates multi-day CI data into single weekly composite raster
|
||||
# for field-level analysis. Supports per-field or single-file architectures.
|
||||
#
|
||||
# INPUT DATA:
|
||||
# - Daily per-field TIFFs: laravel_app/storage/app/{project}/daily_tiles/{YYYY-MM-DD}/*.tif
|
||||
# (or single-file mosaics: merged_tif/{YYYY-MM-DD}.tif + pivot.geojson masking)
|
||||
# - CI data (RDS): laravel_app/storage/app/{project}/combined_CI/combined_CI_data.rds
|
||||
# - Field boundaries: laravel_app/storage/app/{project}/pivot.geojson
|
||||
#
|
||||
# OUTPUT DATA:
|
||||
# - Destination: laravel_app/storage/app/{project}/weekly_mosaic/
|
||||
# - Format: 5-band GeoTIFF (uint16)
|
||||
# - Naming: week_{WW}.tif (week number + year, e.g., week_35_2025.tif)
|
||||
# - Spatial: Raster aligned to field boundaries; CRS preserved
|
||||
#
|
||||
# USAGE:
|
||||
# Rscript 40_mosaic_creation.R [end_date] [offset] [project] [file_name] [data_source]
|
||||
#
|
||||
# Example (Windows PowerShell):
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation.R 2026-01-12 7 aura
|
||||
#
|
||||
# PARAMETERS:
|
||||
# - end_date: End date (YYYY-MM-DD format); required for weekly aggregation
|
||||
# - offset: Days to look back (typically 7 for one week)
|
||||
# - project: Project name (aura, angata, chemba, xinavane, esa, simba)
|
||||
# - file_name: Custom output filename (optional; default: week_{WW}_{YYYY}.tif)
|
||||
# - data_source: Data folder (optional; auto-detects merged_tif or merged_tif_8b)
|
||||
#
|
||||
# CLIENT TYPES:
|
||||
# - cane_supply (ANGATA): Yes - harvest readiness timeline depends on weekly mosaic
|
||||
# - agronomic_support (AURA): Yes - KPI calculation requires weekly CI bands
|
||||
#
|
||||
# DEPENDENCIES:
|
||||
# - Packages: sf, terra, tidyverse, lubridate, here
|
||||
# - Utils files: parameters_project.R, 00_common_utils.R, 40_mosaic_creation_utils.R
|
||||
# - Input data: Daily per-field TIFFs (Script 10) + CI extraction (Script 20)
|
||||
# - Data: field boundaries (pivot.geojson), harvest dates (if available)
|
||||
#
|
||||
# NOTES:
|
||||
# - Weekly aggregation: Combines 7 days of daily data into single composite
|
||||
# - 5-band output: R, G, B, NIR, and Canopy Index (CI) derived from NDVI
|
||||
# - Tiling support: Handles per-field TIFF architecture; auto-mosaics if needed
|
||||
# - Data source auto-detection: Searches merged_tif/ or merged_tif_8b/ folders
|
||||
# - Command-line driven: Designed for batch scheduling (cron/Task Scheduler)
|
||||
# - Downstream: Script 80 (KPI calculation) depends on weekly_mosaic/ output
|
||||
# - Performance: Multi-file mosaicing (~25 fields) takes 5-10 minutes per week
|
||||
#
|
||||
# RELATED ISSUES:
|
||||
# SC-113: Script header standardization
|
||||
# SC-112: Utilities restructuring
|
||||
# SC-111: Script 10 geometry validation
|
||||
#
|
||||
# ============================================================================
|
||||
|
||||
# 1. Load required packages
|
||||
# -----------------------
|
||||
suppressPackageStartupMessages({
|
||||
library(sf)
|
||||
library(terra)
|
||||
library(tidyverse)
|
||||
library(lubridate)
|
||||
library(here)
|
||||
})
|
||||
|
||||
# 2. Process command line arguments and run mosaic creation
|
||||
# ------------------------------------------------------
|
||||
main <- function() {
|
||||
# Capture command line arguments
|
||||
args <- commandArgs(trailingOnly = TRUE)
|
||||
|
||||
# Process project_dir argument with default
|
||||
if (length(args) >= 3 && !is.na(args[3])) {
|
||||
project_dir <- as.character(args[3])
|
||||
} else if (exists("project_dir", envir = .GlobalEnv)) {
|
||||
project_dir <- get("project_dir", envir = .GlobalEnv)
|
||||
} else {
|
||||
# Default project directory
|
||||
project_dir <- "angata"
|
||||
message("No project_dir provided. Using default:", project_dir)
|
||||
}
|
||||
|
||||
# Make project_dir available globally so parameters_project.R can use it
|
||||
assign("project_dir", project_dir, envir = .GlobalEnv)
|
||||
|
||||
# Process end_date argument with default
|
||||
if (length(args) >= 1 && !is.na(args[1])) {
|
||||
# Parse date explicitly in YYYY-MM-DD format from command line
|
||||
end_date <- as.Date(args[1], format = "%Y-%m-%d")
|
||||
if (is.na(end_date)) {
|
||||
message("Invalid end_date provided. Using current date.")
|
||||
end_date <- Sys.Date()
|
||||
}
|
||||
} else if (exists("end_date_str", envir = .GlobalEnv)) {
|
||||
end_date <- as.Date(get("end_date_str", envir = .GlobalEnv))
|
||||
} else {
|
||||
# Default to current date if no argument is provided
|
||||
end_date <- Sys.Date()
|
||||
message("No end_date provided. Using current date: ", format(end_date))
|
||||
}
|
||||
|
||||
# Process offset argument with default
|
||||
if (length(args) >= 2 && !is.na(args[2])) {
|
||||
offset <- as.numeric(args[2])
|
||||
if (is.na(offset) || offset <= 0) {
|
||||
message("Invalid offset provided. Using default (7 days).")
|
||||
offset <- 7
|
||||
}
|
||||
} else {
|
||||
# Default to 7 days if no argument is provided
|
||||
offset <- 7
|
||||
message("No offset provided. Using default:", offset, "days")
|
||||
}
|
||||
|
||||
# Process data_source argument (optional, passed from pipeline)
|
||||
# If provided, use it; otherwise auto-detect
|
||||
data_source_from_args <- NULL
|
||||
if (length(args) >= 5 && !is.na(args[5]) && nchar(args[5]) > 0) {
|
||||
data_source_from_args <- as.character(args[5])
|
||||
message("Data source explicitly provided via arguments: ", data_source_from_args)
|
||||
}
|
||||
|
||||
# 3. Initialize project configuration
|
||||
# --------------------------------
|
||||
|
||||
# Detect which data source directory exists (merged_tif or merged_tif_8b)
|
||||
# IMPORTANT: Only consider a folder as valid if it contains actual files
|
||||
laravel_storage <- here::here("laravel_app/storage/app", project_dir)
|
||||
|
||||
# Load centralized path structure
|
||||
tryCatch({
|
||||
source("r_app/parameters_project.R")
|
||||
paths <- setup_project_directories(project_dir)
|
||||
}, error = function(e) {
|
||||
message("Note: Could not open files from r_app directory")
|
||||
message("Attempting to source from default directory instead...")
|
||||
tryCatch({
|
||||
source("parameters_project.R")
|
||||
paths <- setup_project_directories(project_dir)
|
||||
message("✓ Successfully sourced files from default directory")
|
||||
}, error = function(e) {
|
||||
stop("Failed to source required files from both 'r_app' and default directories.")
|
||||
})
|
||||
})
|
||||
data_source <- if (has_8b_data) {
|
||||
message("Auto-detected data source: merged_tif_8b (8-band optimized) - contains files")
|
||||
"merged_tif_8b"
|
||||
} else if (has_legacy_data) {
|
||||
message("Auto-detected data source: merged_tif (legacy 4-band) - contains files")
|
||||
"merged_tif"
|
||||
} else {
|
||||
message("Warning: No valid data source found (both folders empty or missing). Using default: merged_tif")
|
||||
"merged_tif"
|
||||
}
|
||||
}
|
||||
|
||||
# Set global data_source for parameters_project.R
|
||||
assign("data_source", data_source, envir = .GlobalEnv)
|
||||
|
||||
tryCatch({
|
||||
source("r_app/parameters_project.R")
|
||||
source("r_app/00_common_utils.R")
|
||||
source("r_app/40_mosaic_creation_utils.R")
|
||||
safe_log(paste("Successfully sourced files from 'r_app' directory."))
|
||||
}, error = function(e) {
|
||||
message("Note: Could not open files from r_app directory")
|
||||
message("Attempting to source from default directory instead...")
|
||||
tryCatch({
|
||||
source("parameters_project.R")
|
||||
paths <- setup_project_directories(project_dir)
|
||||
message("✓ Successfully sourced files from default directory")
|
||||
}, error = function(e) {
|
||||
stop("Failed to source required files from both 'r_app' and default directories.")
|
||||
})
|
||||
})
|
||||
|
||||
# Use centralized paths (no need to manually construct or create dirs)
|
||||
merged_final <- paths$growth_model_interpolated_dir # or merged_final_tif if needed
|
||||
daily_vrt <- paths$vrt_dir
|
||||
|
||||
safe_log(paste("Using growth model/mosaic directory:", merged_final))
|
||||
safe_log(paste("Using daily VRT directory:", daily_vrt))
|
||||
|
||||
# 4. Generate date range for processing
|
||||
# ---------------------------------
|
||||
dates <- date_list(end_date, offset)
|
||||
safe_log(paste("Processing data for week", dates$week, "of", dates$year))
|
||||
|
||||
# Create output filename
|
||||
# Only use custom filename if explicitly provided (not empty string)
|
||||
file_name_tif <- if (length(args) >= 4 && !is.na(args[4]) && nchar(args[4]) > 0) {
|
||||
as.character(args[4])
|
||||
} else {
|
||||
paste0("week_", sprintf("%02d", dates$week), "_", dates$year, ".tif")
|
||||
}
|
||||
|
||||
safe_log(paste("Output will be saved as:", file_name_tif))
|
||||
|
||||
# 5. Create weekly mosaics - route based on project tile detection
|
||||
# ---------------------------------------------------------------
|
||||
# The use_tile_mosaic flag is auto-detected by parameters_project.R
|
||||
# based on whether tiles exist in merged_final_tif/
|
||||
|
||||
if (!exists("use_tile_mosaic")) {
|
||||
# Fallback detection if flag not set (shouldn't happen)
|
||||
merged_final_dir <- file.path(laravel_storage, "merged_final_tif")
|
||||
tile_detection <- detect_tile_structure_from_merged_final(merged_final_dir)
|
||||
use_tile_mosaic <- tile_detection$has_tiles
|
||||
}
|
||||
|
||||
if (use_tile_mosaic) {
|
||||
# TILE-BASED APPROACH: Create per-tile weekly MAX mosaics
|
||||
# This is used for projects like Angata with large ROIs requiring spatial partitioning
|
||||
# Input data comes from merged_final_tif/{grid_size}/{DATE}/{DATE}_XX.tif (5-band tiles from script 20)
|
||||
tryCatch({
|
||||
safe_log("Starting per-tile mosaic creation (tile-based approach)...")
|
||||
|
||||
# Detect grid size from merged_final_tif folder structure
|
||||
# Expected: merged_final_tif/5x5/ or merged_final_tif/10x10/ etc.
|
||||
merged_final_base <- file.path(laravel_storage, "merged_final_tif")
|
||||
grid_subfolders <- list.dirs(merged_final_base, full.names = FALSE, recursive = FALSE)
|
||||
# Look for grid size patterns like "5x5", "10x10", "20x20"
|
||||
grid_patterns <- grep("^\\d+x\\d+$", grid_subfolders, value = TRUE)
|
||||
|
||||
if (length(grid_patterns) == 0) {
|
||||
stop("No grid size subfolder found in merged_final_tif/ (expected: 5x5, 10x10, etc.)")
|
||||
}
|
||||
|
||||
grid_size <- grid_patterns[1] # Use first grid size found
|
||||
safe_log(paste("Detected grid size:", grid_size))
|
||||
|
||||
# Point to the grid-specific merged_final_tif directory
|
||||
merged_final_with_grid <- file.path(merged_final_base, grid_size)
|
||||
|
||||
# Set output directory for per-tile mosaics, organized by grid size (from centralized paths)
|
||||
# Output: weekly_tile_max/{grid_size}/week_WW_YYYY_TT.tif
|
||||
tile_output_base <- file.path(paths$weekly_tile_max_dir, grid_size)
|
||||
# Note: no dir.create needed - paths$weekly_tile_max_dir already created by setup_project_directories()
|
||||
dir.create(tile_output_base, recursive = TRUE, showWarnings = FALSE) # Create grid-size subfolder
|
||||
|
||||
created_tile_files <- create_weekly_mosaic_from_tiles(
|
||||
dates = dates,
|
||||
merged_final_dir = merged_final_with_grid,
|
||||
tile_output_dir = tile_output_base,
|
||||
field_boundaries = field_boundaries
|
||||
)
|
||||
|
||||
safe_log(paste("✓ Per-tile mosaic creation completed - created",
|
||||
length(created_tile_files), "tile files"))
|
||||
}, error = function(e) {
|
||||
safe_log(paste("ERROR in tile-based mosaic creation:", e$message), "ERROR")
|
||||
traceback()
|
||||
stop("Mosaic creation failed")
|
||||
})
|
||||
|
||||
} else {
|
||||
# SINGLE-FILE APPROACH: Create single weekly mosaic file
|
||||
# This is used for legacy projects (ESA, Chemba, Aura) expecting single-file output
|
||||
tryCatch({
|
||||
safe_log("Starting single-file mosaic creation (backward-compatible approach)...")
|
||||
|
||||
# Set output directory for single-file mosaics (from centralized paths)
|
||||
single_file_output_dir <- paths$weekly_mosaic_dir
|
||||
|
||||
created_file <- create_weekly_mosaic(
|
||||
dates = dates,
|
||||
field_boundaries = field_boundaries,
|
||||
daily_vrt_dir = daily_vrt,
|
||||
merged_final_dir = merged_final,
|
||||
output_dir = single_file_output_dir,
|
||||
file_name_tif = file_name_tif,
|
||||
create_plots = FALSE
|
||||
)
|
||||
|
||||
safe_log(paste("✓ Single-file mosaic creation completed:", created_file))
|
||||
}, error = function(e) {
|
||||
safe_log(paste("ERROR in single-file mosaic creation:", e$message), "ERROR")
|
||||
traceback()
|
||||
stop("Mosaic creation failed")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
if (sys.nframe() == 0) {
|
||||
main()
|
||||
}
|
||||
|
||||
|
|
@ -1,25 +1,54 @@
|
|||
# 40_MOSAIC_CREATION_PER_FIELD.R
|
||||
# ===============================
|
||||
# Per-Field Weekly Mosaic Creation
|
||||
# ============================================================================
|
||||
# SCRIPT 40: Weekly Mosaic Creation (Per-Field CI Aggregation)
|
||||
# ============================================================================
|
||||
# PURPOSE:
|
||||
# Aggregate daily per-field CI TIFFs into weekly mosaics. Handles multi-date
|
||||
# merging, cloud masking, and produces 5-band weekly output for reporting
|
||||
# and KPI calculations. Supports both per-field and grid-based tile architecture.
|
||||
#
|
||||
# Creates weekly mosaics FROM per-field daily CI TIFFs (output from Script 20)
|
||||
# TO per-field weekly CI TIFFs (input for Scripts 90/91 reporting).
|
||||
# INPUT DATA:
|
||||
# - Source: laravel_app/storage/app/{project}/field_tiles_CI/{FIELD}/{DATE}.tif
|
||||
# - Format: GeoTIFF (5-band: R,G,B,NIR,CI as float32)
|
||||
# - Dates: All available dates within week range
|
||||
#
|
||||
# ARCHITECTURE:
|
||||
# Input: field_tiles_CI/{FIELD}/{DATE}.tif (5-band daily, per-field from Script 20)
|
||||
# Output: weekly_mosaic/{FIELD}/week_WW_YYYY.tif (5-band weekly, per-field)
|
||||
# OUTPUT DATA:
|
||||
# - Destination: laravel_app/storage/app/{project}/weekly_mosaic/{FIELD}/
|
||||
# - Format: GeoTIFF (5-band: R,G,B,NIR,CI)
|
||||
# - Naming: week_WW_YYYY.tif (WW = ISO week, YYYY = ISO year)
|
||||
#
|
||||
# USAGE:
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R [end_date] [offset] [project_dir]
|
||||
# Rscript 40_mosaic_creation_per_field.R [end_date] [offset] [project]
|
||||
#
|
||||
# ARGUMENTS:
|
||||
# end_date: End date for processing (YYYY-MM-DD format, default: today)
|
||||
# offset: Days to look back from end_date (typically 7 for one week, default: 7)
|
||||
# project_dir: Project directory (e.g., "aura", "angata", "chemba", "esa", default: "angata")
|
||||
# Example (Windows PowerShell):
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R 2026-01-12 7 angata
|
||||
#
|
||||
# EXAMPLES:
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R 2026-01-12 7 aura
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/40_mosaic_creation_per_field.R 2025-12-31 7 angata
|
||||
# PARAMETERS:
|
||||
# - end_date: End date for processing (character, YYYY-MM-DD format, default today)
|
||||
# - offset: Days to look back (numeric, default 7 for one week)
|
||||
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
|
||||
#
|
||||
# CLIENT TYPES:
|
||||
# - cane_supply (ANGATA): Yes - weekly monitoring
|
||||
# - agronomic_support (AURA): Yes - field health reporting
|
||||
#
|
||||
# DEPENDENCIES:
|
||||
# - Packages: terra, sf, tidyverse, lubridate
|
||||
# - Utils files: parameters_project.R, 00_common_utils.R, 40_mosaic_creation_per_field_utils.R
|
||||
# - Input data: Daily per-field CI TIFFs from Script 20
|
||||
# - Data directories: field_tiles_CI/, weekly_mosaic/
|
||||
#
|
||||
# NOTES:
|
||||
# - Aggregation method: Maximum CI value per pixel across week (handles clouds)
|
||||
# - ISO week-year used for consistent date handling across year boundaries
|
||||
# - Supports both single-file and tiled mosaic architectures
|
||||
# - Output feeds Scripts 80 (KPI calculations) and 90/91 (reporting)
|
||||
# - Critical for trend analysis: week-over-week CI comparison
|
||||
#
|
||||
# RELATED ISSUES:
|
||||
# SC-112: Script 40 cleanup (deleted legacy mosaic utils files)
|
||||
# SC-108: Core pipeline improvements
|
||||
#
|
||||
# ============================================================================
|
||||
|
||||
# 1. Load required packages
|
||||
# -----------------------
|
||||
|
|
|
|||
|
|
@ -1,28 +1,59 @@
|
|||
# 80_CALCULATE_KPIS.R (CONSOLIDATED KPI CALCULATION)
|
||||
# ============================================================================
|
||||
# UNIFIED KPI CALCULATION SCRIPT
|
||||
# SCRIPT 80: Key Performance Indicator (KPI) Calculation
|
||||
# ============================================================================
|
||||
# PURPOSE:
|
||||
# Calculate per-field and farm-level KPIs from weekly CI mosaics. Computes
|
||||
# field uniformity, growth trends (4-week and 8-week), area change detection,
|
||||
# TCH forecasts, stress identification, and weed presence. Generates
|
||||
# comprehensive Excel/CSV/RDS exports for dashboards and stakeholder reporting.
|
||||
#
|
||||
# This script combines:
|
||||
# 1. Per-field weekly analysis (from 09c: field-level trends, phases, statuses)
|
||||
# 2. Farm-level KPI metrics (from old 09: 6 high-level indicators)
|
||||
# INPUT DATA:
|
||||
# - Source 1: laravel_app/storage/app/{project}/weekly_mosaic/{FIELD}/week_*.tif
|
||||
# - Source 2: Field boundaries (pivot.geojson) and harvest data (harvest.xlsx)
|
||||
# - Source 3: Historical field stats (RDS from previous weeks)
|
||||
#
|
||||
# FEATURES:
|
||||
# - Per-field analysis with SC-64 enhancements (4-week trends, CI percentiles, etc.)
|
||||
# - Farm-level KPI calculation (6 metrics for executive overview)
|
||||
# - Parallel processing (tile-aware, 1000+ fields supported)
|
||||
# - Comprehensive Excel + RDS + CSV exports (21 columns per spec)
|
||||
# - Test mode for development
|
||||
|
||||
# CRITICAL INTEGRATIONS:
|
||||
# OUTPUT DATA:
|
||||
# - Destination: laravel_app/storage/app/{project}/output/
|
||||
# - Format: Excel (.xlsx), CSV (.csv), RDS (.rds)
|
||||
# - Files: {project}_field_analysis_week{WW}_{YYYY}.xlsx + metadata
|
||||
#
|
||||
# 1. IMMINENT_PROB FROM HARVEST MODEL (MODEL_307)
|
||||
# [✓] Load script 31 output: {project}_week_{WW}_{YYYY}.csv
|
||||
# Columns: field, imminent_prob, detected_prob, week, year
|
||||
# [✓] LEFT JOIN to field_analysis_df by field
|
||||
# [✓] Use actual harvest probability data instead of placeholder
|
||||
# USAGE:
|
||||
# Rscript 80_calculate_kpis.R [project] [week] [year]
|
||||
#
|
||||
# 2. AGE FROM HARVEST.XLSX (SCRIPTS 22 & 23)
|
||||
# [✓] Load harvest.xlsx with planting_date (season_start)
|
||||
# Example (Windows PowerShell):
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/80_calculate_kpis.R angata 5 2026
|
||||
#
|
||||
# PARAMETERS:
|
||||
# - project: Project name (character) - angata, chemba, xinavane, esa, simba
|
||||
# - week: ISO week number (numeric, 1-53, default current week)
|
||||
# - year: ISO year (numeric, default current year)
|
||||
#
|
||||
# CLIENT TYPES:
|
||||
# - cane_supply (ANGATA): Yes - uses 80_utils_cane_supply.R (placeholder)
|
||||
# - agronomic_support (AURA): Yes - uses 80_utils_agronomic_support.R (6 KPI funcs)
|
||||
#
|
||||
# DEPENDENCIES:
|
||||
# - Packages: terra, sf, tidyverse, lubridate, writexl, spdep
|
||||
# - Utils files: parameters_project.R, 00_common_utils.R, 80_utils_agronomic_support.R, 80_utils_cane_supply.R
|
||||
# - External data: Field boundaries (pivot.geojson), harvest data (harvest.xlsx)
|
||||
# - Input data: Weekly mosaic TIFFs (Script 40 output)
|
||||
# - Data directories: weekly_mosaic/, output/ (created if missing)
|
||||
#
|
||||
# NOTES:
|
||||
# - KPIs calculated: Field Uniformity (CV), Area Change (pixel %), TCH Forecast,
|
||||
# Growth/Decline Trend, Weed Presence (spatial autocorrelation), Gap Filling %
|
||||
# - Client-aware: Conditional sourcing based on client_config$script_90_compatible
|
||||
# - Exports: 21-column output with field-level metrics, phase, status, alerts
|
||||
# - Supports 4-week and 8-week trend analysis from historical RDS cache
|
||||
# - Critical dependency for Scripts 90/91 (reporting/dashboards)
|
||||
# - Uses Moran's I for spatial clustering detection (weed/stress patterns)
|
||||
#
|
||||
# RELATED ISSUES:
|
||||
# SC-112: Script 80 utilities restructuring (common + client-aware modules)
|
||||
# SC-108: Core pipeline improvements
|
||||
# SC-100: KPI definition and formula documentation
|
||||
#
|
||||
# ============================================================================
|
||||
# [✓] Extract planting dates per field
|
||||
# [✓] Calculate Age_week = difftime(report_date, planting_date, units="weeks")
|
||||
#
|
||||
|
|
|
|||
54
r_app/_SCRIPT_HEADER_TEMPLATE.R
Normal file
54
r_app/_SCRIPT_HEADER_TEMPLATE.R
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# ============================================================================
|
||||
# SCRIPT XX: [DESCRIPTIVE TITLE]
|
||||
# ============================================================================
|
||||
# PURPOSE:
|
||||
# [What this script does in 2-3 sentences. Example: Downloads satellite
|
||||
# imagery from Planet API, processes 4-band RGB+NIR data, and saves
|
||||
# merged GeoTIFFs for use by downstream analysis stages.]
|
||||
#
|
||||
# INPUT DATA:
|
||||
# - Source: [Which directory/files it reads. Example: Planet API, hardcoded bbox]
|
||||
# - Format: [TIFF, RDS, GeoJSON, etc. Example: 4-band uint16 GeoTIFF]
|
||||
# - Location: [Full path example]
|
||||
#
|
||||
# OUTPUT DATA:
|
||||
# - Destination: [Which directory/files it creates. Example: laravel_app/storage/app/{project}/merged_tif/]
|
||||
# - Format: [TIFF, RDS, CSV, etc. Example: Single-band float32 GeoTIFF]
|
||||
# - Naming convention: [Example: {YYYY-MM-DD}.tif]
|
||||
#
|
||||
# USAGE:
|
||||
# Rscript XX_script_name.R [arg1] [arg2] [arg3]
|
||||
#
|
||||
# Example (Windows PowerShell):
|
||||
# & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/XX_script_name.R angata 2026-01-15
|
||||
#
|
||||
# PARAMETERS:
|
||||
# - arg1: [Description, type, and valid values. Example: project (character) - angata, chemba, xinavane, esa, simba]
|
||||
# - arg2: [Description, type, and valid values. Example: date (character, optional) - ISO format YYYY-MM-DD; default today]
|
||||
#
|
||||
# CLIENT TYPES:
|
||||
# - cane_supply (ANGATA): [Yes/No - if Yes, briefly explain any differences]
|
||||
# - agronomic_support (AURA): [Yes/No - if Yes, briefly explain any differences]
|
||||
#
|
||||
# DEPENDENCIES:
|
||||
# - Packages: [dplyr, tidyr, terra, sf, readr, writexl, etc.]
|
||||
# - Utils files: [parameters_project.R, 00_common_utils.R, XX_utils.R, etc.]
|
||||
# - External data: [harvest.xlsx, pivot.geojson, etc.]
|
||||
# - Data directories: [laravel_app/storage/app/{project}/]
|
||||
#
|
||||
# NOTES:
|
||||
# [Any special considerations, assumptions, or future improvements.
|
||||
# Example: Cloud filtering uses CI >= 0.5 threshold. Multi-field support
|
||||
# implemented via field geometry masking from pivot.geojson.]
|
||||
#
|
||||
# RELATED ISSUES:
|
||||
# SC-XXX: [Brief description of related work]
|
||||
#
|
||||
# HISTORY:
|
||||
# 2026-02-03: [Description of change, if refactored or enhanced]
|
||||
# ============================================================================
|
||||
|
||||
# NOTE: This is a TEMPLATE file for documentation purposes only.
|
||||
# When creating a new script or updating an existing one, copy this template
|
||||
# and fill in all sections with accurate information about your specific script.
|
||||
# Then delete this comment block.
|
||||
Loading…
Reference in a new issue