- Improved safe_log function to include timestamps and conditional logging
- Added diagnostic messages for field visualization processing
- Updated CI map rendering parameters for consistency
- Refined raster mapping functions in report_utils for clarity
- Added .png files to .gitignore
- Implemented `run_spectral_extraction.py` to batch extract spectral indices (NDVI, BSI, NDWI) from 4-band TIFF files, saving results in a structured CSV format.
- Created `spectral_features.py` for calculating various spectral indices, including NDVI, BSI, NDWI, CI_green, CI_red, GNDVI, SAVI, and EVI2.
- Added Jupyter notebook `02_season_normalization_analysis.ipynb` for analyzing season-length normalization, comparing ESA and Angata spectral indices, and visualizing growth patterns.
- Included functions for computing season age, plotting trajectories, and performing peak and amplitude analysis.
- Updated `create_CI_map` and `create_CI_diff_map` functions to enforce a 1:1 aspect ratio for consistent map sizing.
- Modified `ci_plot` function to adjust widths of arranged maps for better layout.
- Changed raster merging method in `aggregate_per_field_mosaics_to_farm_level` from `mosaic` to `merge` for improved handling of field data.
- Introduced `test_kpi_validation.R` script to validate the structure of KPI RDS files, ensuring expected KPIs are present.
- Added `test_overview_maps_aggregation.R` script to test the aggregation pipeline for overview maps, including loading field mosaics, creating a farm-level mosaic, and generating visualizations.
- Consolidate 80_kpi_utils.R, 80_weekly_stats_utils.R, 80_report_building_utils.R into three client-aware files:
- 80_utils_common.R (50+ functions): Shared utilities, constants, and helpers
- 80_utils_agronomic_support.R: AURA-specific KPI calculations (6 KPIs)
- 80_utils_cane_supply.R: ANGATA placeholder for future expansion
- Move all internal helpers (calculate_cv, calculate_change_percentages, calculate_spatial_autocorrelation, extract_ci_values, etc.) to common
- Add MORAN_THRESHOLD_* constants to common
- Fix parameters_project.R field boundaries path (removed extra 'Data' directory)
- Update 80_calculate_kpis.R with conditional client-type sourcing logic
- Validate both ANGATA (cane_supply) and AURA (agronomic_support) workflows with comprehensive testing
- All 96+ functions accounted for; ready for production use
- Changed gap calculation from Q25 to 2σ below median method (kpi_utils.R)
- Integrated gap filling into script 80 with tile-based processing
- Added Gap_score column to field analysis output (Excel/CSV/RDS)
- Fixed memory issues by processing 25 tiles individually instead of merging
- Fixed Field_id matching to ensure gap scores populate correctly
Gap scores now calculate for all 1185 fields with range 0-11.25%
Works with tile-based mosaics (Angata 5x5 grid) without memory errors
- Script 80: Add fallback logic to detect and handle single-file mosaics
- Weekly stats utils: Support both tile-based and single-file mosaic detection
- Pipeline runner: Auto-detect mosaic mode (tiled vs single-file)
- Flexible grid size detection for tile-based projects (5x5, 10x10, etc)
Fixes:
- Script 80 now checks weekly_tile_max/{grid_size} first, falls back to weekly_mosaic
- calculate_field_statistics handles both tile patterns and single-file patterns
- run_full_pipeline detects project mode automatically
- All verification checks are now flexible and don't assume fixed paths
Projects like 'aura' (small ROI < 10km) will use single-file approach automatically
Projects like 'angata' (large ROI >= 10km) will use tile-based approach automatically
- Updated `detect_mosaic_mode` function to check for grid-size subdirectories in addition to tile-named files.
- Added comprehensive tests for DOY reset logic in `test_doy_logic.py`.
- Implemented feature extraction tests in `test_feature_extraction.py`.
- Created tests for growing window method in `test_growing_window_only.py`.
- Developed a complete model inference test in `test_model_inference.py`.
- Added a debug script for testing two-step refinement logic in `test_script22_debug.py`.
- Added a progress bar to the tile download process for better user feedback.
- Simplified the tile overlap checking logic in the R script to improve performance and readability.
Major Changes:
- NEW: Scripts 09 & 10 for KPI calculation and enhanced reporting
- NEW: Shell script wrappers (01-10) for easier execution
- NEW: R packages flextable and officer for enhanced Word reports
- NEW: DEPLOYMENT_README.md with complete deployment guide
- RENAMED: Numbered R scripts (02, 03, 04) for clarity
- REMOVED: Old package management scripts (using renv only)
- UPDATED: Workflow now uses scripts 09->10 instead of 05
Files Changed: 90+ files
New Packages: flextable, officer
New Scripts: 09_run_calculate_kpis.sh, 10_run_kpi_report.sh
Documentation: DEPLOYMENT_README.md, EMAIL_TO_ADMIN.txt
See DEPLOYMENT_README.md for full deployment instructions.
Timon
DimitraVeropoulou