Timon/SmartCane
1
1
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
475 commits 1 branch 0 tags 252 MiB
Commit graph

247 commits

Author SHA1 Message Date
Timon 4d6bba828f refactored 10 20 2026-01-29 21:10:24 +01:00
Timon d1f352f21c updated sc-91 2026-01-29 17:26:03 +01:00
Timon 4445f72e6f works for angata and aura 2026-01-29 16:04:04 +01:00
Timon dd915f9b9e adjusting scripts to work for all projects 2026-01-29 09:04:46 +01:00
Timon b2de819fc4 Fix backward compatibility: Support non-tiled (single-file) mosaic projects 
- Script 80: Add fallback logic to detect and handle single-file mosaics
- Weekly stats utils: Support both tile-based and single-file mosaic detection
- Pipeline runner: Auto-detect mosaic mode (tiled vs single-file)
- Flexible grid size detection for tile-based projects (5x5, 10x10, etc)

Fixes:
- Script 80 now checks weekly_tile_max/{grid_size} first, falls back to weekly_mosaic
- calculate_field_statistics handles both tile patterns and single-file patterns
- run_full_pipeline detects project mode automatically
- All verification checks are now flexible and don't assume fixed paths

Projects like 'aura' (small ROI < 10km) will use single-file approach automatically
Projects like 'angata' (large ROI >= 10km) will use tile-based approach automatically
 2026-01-28 11:19:07 +01:00
Timon 9a55d2fcf8 Refactor full pipeline script to include intelligent checking of existing outputs and dynamic execution of scripts. Added new Python scripts for RGB validation and evaluation template creation. Enhanced error handling and logging throughout the pipeline. 2026-01-27 08:58:06 +01:00
Timon 4143bdf4d7 updateing csv file 2026-01-20 12:06:45 +01:00
Timon c11b10a73f working on visualisation, not working too well... 2026-01-18 11:25:31 +01:00
Timon 9f312131d7 integration lstm 31 and 80 complete - angata kpi file now dynamic 2026-01-18 10:14:08 +01:00
Timon 1f5add7485 refactored script 80 2026-01-18 09:27:16 +01:00
Timon 7975f8ad06 8 week data working now 2026-01-18 07:39:30 +01:00
Timon 3e4430b3be added last columns of trends, only harvest imminent and age left to do 2026-01-16 09:10:39 +01:00
Timon 8d84c8cab5 phase 2 cv trend implemented 2026-01-16 08:30:47 +01:00
Timon 6e88acef25 added some more stats and functions calculate and export csv and rds file for future ref 2026-01-16 08:04:45 +01:00
Timon 4e94a9a78b expanding csv table 2026-01-15 15:35:16 +01:00
Timon fabbf3214d Enhance harvest detection logic and testing framework 
- Updated `detect_mosaic_mode` function to check for grid-size subdirectories in addition to tile-named files.
- Added comprehensive tests for DOY reset logic in `test_doy_logic.py`.
- Implemented feature extraction tests in `test_feature_extraction.py`.
- Created tests for growing window method in `test_growing_window_only.py`.
- Developed a complete model inference test in `test_model_inference.py`.
- Added a debug script for testing two-step refinement logic in `test_script22_debug.py`.
 2026-01-15 14:30:54 +01:00
Timon d6448b8703 Refactor: Update system architecture documentation to reflect Python integration and add complete pipeline overview 2026-01-15 10:25:38 +01:00
Timon 458b8247be Cleanup: Fix CI formula, reorganize shell scripts and test files 
- Fixed CI calculation: changed from NDVI (NIR-Red)/(NIR+Red) to correct NIR/Green-1 formula in:
  * process_single_tile() function
  * create_ci_band() utility function
  * Updated create_mask_and_crop() documentation

- Renamed numbered shell scripts for clarity (matching R script numbering):
  * 01_run_planet_download -> 10_planet_download.sh
  * 02_run_ci_extraction -> 20_ci_extraction.sh
  * 03_run_growth_model -> 30_growth_model.sh
  * 04_run_mosaic_creation -> 40_mosaic_creation.sh
  * 09_run_calculate_kpis -> 80_calculate_kpis.sh
  * 10_run_kpi_report -> 90_kpi_report.sh

- Archived obsolete shell scripts to old_sh/:
  * build_mosaic.sh, build_report.sh, interpolate_growth_model.sh
  * 05_run_dashboard_report.sh, 06_run_crop_messaging.sh
  * 11_run_yield_prediction.sh/ps1
  * runcane.sh, runpython.sh, smartcane.sh, update_RDS.sh

- Deleted test/debug files and temporary outputs:
  * analyze_*.R, benchmark_gpu_vs_cpu.py, convert_angata_harvest.py
  * debug_mosaic.R, examine_kpi_results.R, generate_sar_report.R
  * inspect_8band_structure.R, inspect_tif_bands.R
  * old_working_utils.R, predict_harvest_operational.R
  * run_kpi_calculation.R, run_report.R, simple_sar_test.R
  * data_validation_tool/, harvest_ci_pattern_analysis.png, kpi_debug.out

- Enhanced harvest prediction: Added threshold tuning (0.40-0.45) and field type handling

- Enhanced mosaic creation: Improved tile detection and routing logic
 2026-01-14 16:58:51 +01:00
Timon d365b5838b Move: archive old KPI scripts (09, 09b, 09c, 20) to old_scripts/ for testing 2026-01-14 11:37:13 +01:00
Timon 04e1641a18 Create 80_calculate_kpis.R: unified KPI calculation (per-field SC-64 + farm-level metrics) 2026-01-14 11:36:06 +01:00
Timon 62106eeb3b Update: source() calls to reference renamed util files (20_, 30_, 40_) 2026-01-14 11:24:27 +01:00
Timon b7ef583efa Rename: number utils to match their main scripts 2026-01-14 11:16:23 +01:00
Timon 5aa456fbf5 Rename: 01_ to 10_ for stage consistency 2026-01-14 11:12:11 +01:00
Timon e69a7c5465 Rename: align stage numbering with Bitbucket structure 2026-01-14 11:10:28 +01:00
Timon 252b4049b3 Rename: align stage numbering with Bitbucket structure 2026-01-14 11:08:33 +01:00
Timon ef29cb5c2c Move experimental files to r_app/experiments 2026-01-14 11:07:40 +01:00
Timon 6d5509dd1b Remove: cleanup test/draft files not in Bitbucket 2026-01-14 11:05:57 +01:00
Timon dfa0aa900d updated website and small things 2026-01-14 11:02:45 +01:00
Timon fc7e5f1ee0 Enhance download progress feedback and optimize tile overlap checks 
- Added a progress bar to the tile download process for better user feedback.
- Simplified the tile overlap checking logic in the R script to improve performance and readability.
 2026-01-13 11:30:38 +01:00
Timon 03f269302a SC-58: Implement grid filtering and per-date tiling 
- Filter master grid to only overlapping tiles based on GeoJSON field boundaries
- Implement per-date tiling for resumable processing (can stop/restart anytime)
- Add grid-size versioning (subfolder structure: daily_tiles_split/5x5/, etc)
- Skip already-processed dates for idempotent execution
- Upfront extent checking (one-time), per-date processing (resumable)
- Reduce storage by excluding empty tiles outside field boundaries
 2026-01-13 09:35:51 +01:00
Timon cb63cf00b8 updated code to fit in workflow better 2026-01-12 16:33:23 +01:00
Timon 182d13d593 wip 2026-01-12 14:55:25 +01:00
Timon dc7c346854 updated whole workflow and architercutre md 2026-01-12 09:34:13 +01:00
Timon d22dc2f96e commit all stuff 2026-01-06 14:17:37 +01:00
Timon d5fd4bb463 Add KPI reporting system and deployment documentation 
Major Changes:
- NEW: Scripts 09 & 10 for KPI calculation and enhanced reporting
- NEW: Shell script wrappers (01-10) for easier execution
- NEW: R packages flextable and officer for enhanced Word reports
- NEW: DEPLOYMENT_README.md with complete deployment guide
- RENAMED: Numbered R scripts (02, 03, 04) for clarity
- REMOVED: Old package management scripts (using renv only)
- UPDATED: Workflow now uses scripts 09->10 instead of 05

Files Changed: 90+ files
New Packages: flextable, officer
New Scripts: 09_run_calculate_kpis.sh, 10_run_kpi_report.sh
Documentation: DEPLOYMENT_README.md, EMAIL_TO_ADMIN.txt

See DEPLOYMENT_README.md for full deployment instructions.
 2025-10-14 11:49:30 +02:00
Martin Folkerts d258a2bf2e cleaned the folders, made some parameters 2025-09-05 15:23:41 +02:00
Martin Folkerts 77d67c36ca Merge branch 'code-improvements' 2025-06-25 11:03:33 +02:00
Timon a6aa002d33 minor visualistion chagnes 2025-06-25 11:01:37 +02:00
Martin Folkerts 0e7f5cf11d Merge branch 'code-improvements' 2025-06-25 10:47:11 +02:00
Timon ea6842af48 minor visualistion chagnes 2025-06-25 10:39:26 +02:00
Martin Folkerts cc6b9392d6 Merge branch 'code-improvements' 2025-06-25 09:35:19 +02:00
Timon b2fcd66d2b klaar voor testen, wat plaatjes e.d. aangepast 2025-06-24 21:02:54 +02:00
Timon 4d6439d5e0 Add reproducible R package management system 
- Add renv.lock with exact package versions for reproducibility
- Add package_manager.R for automated package installation/updates
- Add extract_current_versions.R for package discovery
- Configure renv environment for team collaboration

All team members can now run source('r_app/package_manager.R') to get identical package environment
 2025-06-24 14:51:22 +02:00
Martin Folkerts a316b30afc Merge branch 'code-improvements' 2025-06-24 09:58:56 +02:00
Timon 6efcc8cfec Fix CI report pipeline: update tmap v4 syntax, add continuous color scales, fix formatting 
- Updated all CI maps to use tm_scale_continuous() for proper tmap v4 compatibility
- Added fixed color scale limits (1-8 for CI, -3 to +3 for differences) for consistent field comparison
- Fixed YAML header formatting issues in CI_report_dashboard_planet.Rmd
- Positioned RGB map before CI overview map as requested
- Removed all obsolete use_breaks parameter references
- Enhanced error handling and logging throughout the pipeline
- Added new experimental analysis scripts and improvements to mosaic creation
 2025-06-19 20:37:20 +02:00
Timon bb2a599075 Enhanced SmartCane executive summary report with explanatory text and fixed priority map coloring 
Added explanatory text for all visualizations
Fixed priority map color scheme (red=high priority, green=low priority)
Improved error handling in farm health data calculations
Added fallback mechanisms for missing data
 2025-04-23 09:47:19 +02:00
Timon 2bed5949fa Refactor CI plotting functions to use proper parameterization 
- Refactored ci_plot and cum_ci_plot functions to accept explicit parameters
- Added default values for backward compatibility
- Updated CI_report_dashboard_planet.Rmd to pass explicit parameters
- Added input validation for better error handling
- Added parameter documentation in function headers
- Removed reliance on global variables for better maintainability
 2025-04-22 21:41:18 +02:00
Timon 07aee7bed1 –[200~Improve CI report visualization and migrate to terra package 
- Replace raster package with terra throughout the codebase
- Update map visualizations with better layout and legends
- Add descriptive headers to report sections
- Improve map legend positioning and sizing
- Enhance error handling for missing data
- Remove redundant legends in field-specific visualizations
- Optimize figure dimensions to prevent page overflow
- Expand documentation of CI index and report components
- Update package dependencies in packages.
 2025-04-22 20:55:02 +02:00
Timon fbc8f5e11b wip 2025-04-22 17:43:11 +02:00
Martin Folkerts eb9803fbf1 Removed yieldprediction from CI_report_dashboard_planet.Rmd edited online with Bitbucket 2025-04-02 12:53:29 +00:00
Martin Folkerts ddad4240d5 added conditional to not run prediction if requirements are not met 2025-03-25 16:09:31 +01:00
Timon 29e2867b41 adding docx 2025-03-25 10:42:04 +01:00
Martin Folkerts dd1ee2b1a0 adjusted dates to iso dates 2025-01-07 14:24:38 +01:00
Timon d0ff25acf9 timely save 2024-12-16 16:26:34 +01:00
Martin Folkerts 9b29dc4d57 updated signature of cum ci plot 2024-12-03 09:32:04 +01:00
Martin Folkerts 3bfa7e1750 some update to get mkulazi to work 2024-11-18 13:41:58 +01:00
Martin Folkerts 576250982c fixed plot colouring and made ci_extraction resilient against none file; 2024-11-11 20:55:59 +01:00
Martin Folkerts c8ef406545 added Catalog package with config and changed iteration to use available_slots 2024-11-11 10:26:02 +01:00
Martin Folkerts db68fd8e83 removed column for plot colors 2024-11-07 11:43:09 +01:00
Martin Folkerts 3093c86e3a okay 2024-10-31 22:35:59 +01:00
Timon c1a32bf6ad Refactor cum_ci_plot function for enhanced plotting options; added parameters for plot type and season filtering; improved x-axis labeling and updated ggplot aesthetics. Also, continued work on yield prediction in R. 2024-10-31 11:21:39 +01:00
Martin Folkerts ddb4fdbc02 updated date format for excel 2024-10-28 15:57:23 +01:00
Martin Folkerts f08da04c3b added DownloadCreditsCommand some updates 2024-10-23 09:34:52 +02:00
Martin Folkerts 056b8dd67b okay 2024-09-17 14:56:33 +02:00
Martin Folkerts 4beca30d2e okay 2024-08-30 16:27:32 +02:00
Martin Folkerts 7bf4f0930d added log as project file; 2024-08-30 16:16:26 +02:00
Martin Folkerts 499744db6d moved logging arroudn 2024-08-30 12:09:13 +02:00
Martin Folkerts 7d671ba69c moved end_date and added a is.na wrapping to set when running in R_Studio 2024-08-30 10:37:02 +02:00
Timon 4dd84b10a5 source(utils) aangepast voor online versie 2024-08-30 09:52:02 +02:00
Timon bd9b7d4ab0 adjusted all functions and cleaned scripts incl rmd file 2024-08-29 16:48:24 +02:00
Timon 611bc17017 adjusted all functions and cleaned scripts 2024-08-29 16:35:59 +02:00
Martin Folkerts ad2f41482f updated create_report.sh to have borders option 2024-08-28 14:52:42 +02:00
Martin Folkerts 2598306643 removed chemba as a option from parameters so it is treathed as a regular project 2024-08-28 10:18:53 +02:00
Martin Folkerts 23ba088434 added logging to getTifsAsZip 2024-08-27 11:46:34 +02:00
Martin Folkerts 509f924c58 me 2024-08-26 16:23:52 +02:00
Martin Folkerts c5f9c8e3ca okay 2024-08-26 16:19:22 +02:00
Martin Folkerts dce7344947 updated project 2024-08-26 14:01:57 +02:00
Martin Folkerts 64a576d638 okay 2024-08-20 17:01:58 +02:00
Martin Folkerts fcd900075b toch niet 2024-07-19 15:59:28 +02:00
Martin Folkerts 3e74c72da0 okay 2024-07-19 15:54:00 +02:00
Martin Folkerts 3a081bf72a wip; 2024-07-19 15:08:14 +02:00
Martin Folkerts 7fc9f552ab okay 2024-07-19 15:04:23 +02:00
Martin Folkerts 3813c40ac6 okay 2024-07-19 14:49:05 +02:00
Martin Folkerts 86532128aa okay 2024-07-19 14:45:41 +02:00
Martin Folkerts 14dee6def4 olay 2024-07-19 14:42:03 +02:00
Martin Folkerts 861244fec1 okay 2024-07-19 14:38:50 +02:00
Martin Folkerts 0109a57b04 run with one day of 2024-07-19 14:10:49 +02:00
Martin Folkerts 591a30d2a8 wip 2024-07-19 14:02:37 +02:00
Martin Folkerts 19e3f79371 wip 2024-07-19 13:28:14 +02:00
Martin Folkerts 8bac58b121 okay 2024-07-19 10:57:50 +02:00
Martin Folkerts 24cbea7b0b wip 2024-07-18 15:57:13 +02:00
Martin Folkerts 5645ad83e5 changes 2024-07-17 14:53:46 +02:00
Martin Folkerts 0b1826dccb wip 2024-07-15 11:45:21 +02:00
Martin Folkerts 2988516db2 added conditional to skip field/subfield when no downloads available in current rds 2024-07-15 10:41:13 +02:00
Martin Folkerts 3ea7eec68e some more work 2024-07-09 15:47:53 +02:00
Martin Folkerts 1de6dd4edf wip 2024-07-04 21:16:12 +02:00
Martin Folkerts d9fafcea43 wip 2024-07-02 16:51:55 +02:00
Martin Folkerts 1f194c5670 wip 2024-07-01 13:39:21 +02:00
guillaume91 3f7043199b okay 2024-06-26 13:00:22 +02:00
guillaume91 ea38c952ba Split 2_CI_data_prep.R into 2 files (+3 utils files) 2024-06-26 12:07:21 +02:00
Martin Folkerts af1135e52a wip 2024-06-25 13:37:45 +02:00
Martin Folkerts 607a221028 changed mosaic r script to work with end_date and offset instead of week 2024-06-24 11:39:47 +02:00
Martin Folkerts 1e841a0aa6 wip 2024-06-18 09:17:57 +02:00
Martin Folkerts e2138a378a some updates 2024-05-13 19:35:17 +02:00
Martin Folkerts 95633a553c fixed whitespace issue in subchunkify 2024-05-06 11:46:51 +02:00
Timon b6c576ca57 working rmd file, removed renv because it messed up my R and could not get it to work 2024-04-18 16:19:17 +02:00
Martin Folkerts 5886f59bd8 wip 2024-04-18 13:43:14 +02:00
Martin Folkerts 4051b802e2 wip 2024-04-18 11:11:24 +02:00
Martin Folkerts d2fa24fc6c added some changes to timons merged branch 2024-03-18 11:55:35 +01:00
Timon 88d871d865 updated harvest datata stadnadr 2024-03-14 13:01:19 +01:00
Timon 4053de50df updated harvest datata stadnadr 2024-03-14 13:00:02 +01:00
Timon 66d6f80532 adjusted scripts to be more generalised 2024-03-13 10:27:10 +01:00
Timon 238520ff89 parameterised harvestdata etc 2024-03-12 22:18:57 +01:00
Martin Folkerts 62b9942a53 wip 2024-03-12 12:09:29 +01:00
Martin Folkerts 25715ddef6 wip 2024-03-11 10:36:47 +01:00
Martin Folkerts f7d660356c okay; 2024-03-10 21:53:20 +01:00
Martin Folkerts e996a274db merged testing 2024-03-05 09:40:46 +01:00
Martin Folkerts c222314950 wip 2024-03-05 09:15:46 +01:00
Timon 952700f2c4 forgot to save last R file 2024-03-04 17:22:08 +01:00
Timon 6c00c57b37 changed CI extract to extract all sat images if parameter new_project_question is set to TRUE, to add when new project start 2024-03-04 17:12:20 +01:00
Timon 86f4a49d30 minor changes in script to get it running on my device 2024-03-04 12:15:23 +01:00
Timon 07a54eab25 minor changes 2024-03-01 16:32:53 +01:00
Timon a976350ce5 first commit test 2024-03-01 14:41:55 +01:00
Martin Folkerts 645929ae4e fixed build-report refactored enum status and HasStatusTrait 2024-02-14 21:01:15 +01:00
Martin Folkerts a7a333fb0f worked with timon to harmonize report output 2024-02-08 17:06:35 +01:00
Martin Folkerts 3a18e3b8a7 okeay 2024-01-31 11:58:01 +01:00
Martin Folkerts 0cfd8ad8bf fix 2024-01-31 11:54:02 +01:00
Martin Folkerts 9f60d3db8a added leading zero to weeknumbers so mosaics van be found for week numbers smaller then 9 2024-01-31 11:47:56 +01:00
Martin Folkerts a19e7667e6 wip 2024-01-30 15:17:26 +01:00
Martin Folkerts e224080232 added leading zero to weeknumber in file name format 2024-01-30 15:14:28 +01:00
Martin Folkerts 33cb0dd7da wip 2024-01-24 21:46:08 +01:00
Martin Folkerts d3c2c18918 revised report 2024-01-16 14:43:22 +01:00
Martin Folkerts 3eaf471f05 wip 2024-01-10 21:42:32 +01:00
Martin Folkerts da9d94880f wip 2024-01-09 10:49:25 +01:00
Martin Folkerts 4c22ab6758 wip 2024-01-03 22:06:03 +01:00
Martin Folkerts 3fb55ddcce added rule for missing mergedTifs; 2023-12-24 18:02:17 +01:00
Martin Folkerts bd4ac224ad added CI graphs to pdf 2023-12-12 16:37:47 +01:00
Martin Folkerts 36d74ed2af working version of all python scripts and r-scripts 2023-12-12 14:58:44 +01:00
Martin Folkerts c40ff0c79f updated scripts 2023-12-12 11:20:05 +01:00
Martin Folkerts 3fe61f8ab8 wip 2023-11-06 12:14:13 +01:00
Martin Folkerts dbc8a13600 wip\ 2023-10-31 15:03:32 +01:00
Martin Folkerts 0b7294340c updated r_app/.gitignore to reflect install strategy. 
removed renv and .Rprofile from project so every instance should run its own enviroment based on installPackages.R.
 2023-10-31 14:12:12 +01:00
Martin Folkerts 2a872cf8a3 wip 2023-10-17 11:08:44 +02:00
Martin Folkerts 3ce436b6a0 added renv::restore() and renv::activate() 2023-10-16 17:34:11 +02:00
Martin Folkerts dda9ad8871 build version of r.lock 2023-10-16 17:20:46 +02:00
Martin Folkerts 568565bc76 wip 2023-10-16 15:45:52 +02:00
Martin Folkerts 78e6567182 wip 2023-10-10 17:02:35 +02:00