Timon/SmartCane
1
1
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
465 commits 1 branch 0 tags 252 MiB
Commit graph

238 commits

Author SHA1 Message Date
Timon 1f5add7485 refactored script 80 2026-01-18 09:27:16 +01:00
Timon 7975f8ad06 8 week data working now 2026-01-18 07:39:30 +01:00
Timon 3e4430b3be added last columns of trends, only harvest imminent and age left to do 2026-01-16 09:10:39 +01:00
Timon 8d84c8cab5 phase 2 cv trend implemented 2026-01-16 08:30:47 +01:00
Timon 6e88acef25 added some more stats and functions calculate and export csv and rds file for future ref 2026-01-16 08:04:45 +01:00
Timon 4e94a9a78b expanding csv table 2026-01-15 15:35:16 +01:00
Timon fabbf3214d Enhance harvest detection logic and testing framework 
- Updated `detect_mosaic_mode` function to check for grid-size subdirectories in addition to tile-named files.
- Added comprehensive tests for DOY reset logic in `test_doy_logic.py`.
- Implemented feature extraction tests in `test_feature_extraction.py`.
- Created tests for growing window method in `test_growing_window_only.py`.
- Developed a complete model inference test in `test_model_inference.py`.
- Added a debug script for testing two-step refinement logic in `test_script22_debug.py`.
 2026-01-15 14:30:54 +01:00
Timon d6448b8703 Refactor: Update system architecture documentation to reflect Python integration and add complete pipeline overview 2026-01-15 10:25:38 +01:00
Timon 458b8247be Cleanup: Fix CI formula, reorganize shell scripts and test files 
- Fixed CI calculation: changed from NDVI (NIR-Red)/(NIR+Red) to correct NIR/Green-1 formula in:
  * process_single_tile() function
  * create_ci_band() utility function
  * Updated create_mask_and_crop() documentation

- Renamed numbered shell scripts for clarity (matching R script numbering):
  * 01_run_planet_download -> 10_planet_download.sh
  * 02_run_ci_extraction -> 20_ci_extraction.sh
  * 03_run_growth_model -> 30_growth_model.sh
  * 04_run_mosaic_creation -> 40_mosaic_creation.sh
  * 09_run_calculate_kpis -> 80_calculate_kpis.sh
  * 10_run_kpi_report -> 90_kpi_report.sh

- Archived obsolete shell scripts to old_sh/:
  * build_mosaic.sh, build_report.sh, interpolate_growth_model.sh
  * 05_run_dashboard_report.sh, 06_run_crop_messaging.sh
  * 11_run_yield_prediction.sh/ps1
  * runcane.sh, runpython.sh, smartcane.sh, update_RDS.sh

- Deleted test/debug files and temporary outputs:
  * analyze_*.R, benchmark_gpu_vs_cpu.py, convert_angata_harvest.py
  * debug_mosaic.R, examine_kpi_results.R, generate_sar_report.R
  * inspect_8band_structure.R, inspect_tif_bands.R
  * old_working_utils.R, predict_harvest_operational.R
  * run_kpi_calculation.R, run_report.R, simple_sar_test.R
  * data_validation_tool/, harvest_ci_pattern_analysis.png, kpi_debug.out

- Enhanced harvest prediction: Added threshold tuning (0.40-0.45) and field type handling

- Enhanced mosaic creation: Improved tile detection and routing logic
 2026-01-14 16:58:51 +01:00
Timon d365b5838b Move: archive old KPI scripts (09, 09b, 09c, 20) to old_scripts/ for testing 2026-01-14 11:37:13 +01:00
Timon 04e1641a18 Create 80_calculate_kpis.R: unified KPI calculation (per-field SC-64 + farm-level metrics) 2026-01-14 11:36:06 +01:00
Timon 62106eeb3b Update: source() calls to reference renamed util files (20_, 30_, 40_) 2026-01-14 11:24:27 +01:00
Timon b7ef583efa Rename: number utils to match their main scripts 2026-01-14 11:16:23 +01:00
Timon 5aa456fbf5 Rename: 01_ to 10_ for stage consistency 2026-01-14 11:12:11 +01:00
Timon e69a7c5465 Rename: align stage numbering with Bitbucket structure 2026-01-14 11:10:28 +01:00
Timon 252b4049b3 Rename: align stage numbering with Bitbucket structure 2026-01-14 11:08:33 +01:00
Timon ef29cb5c2c Move experimental files to r_app/experiments 2026-01-14 11:07:40 +01:00
Timon 6d5509dd1b Remove: cleanup test/draft files not in Bitbucket 2026-01-14 11:05:57 +01:00
Timon dfa0aa900d updated website and small things 2026-01-14 11:02:45 +01:00
Timon fc7e5f1ee0 Enhance download progress feedback and optimize tile overlap checks 
- Added a progress bar to the tile download process for better user feedback.
- Simplified the tile overlap checking logic in the R script to improve performance and readability.
 2026-01-13 11:30:38 +01:00
Timon 03f269302a SC-58: Implement grid filtering and per-date tiling 
- Filter master grid to only overlapping tiles based on GeoJSON field boundaries
- Implement per-date tiling for resumable processing (can stop/restart anytime)
- Add grid-size versioning (subfolder structure: daily_tiles_split/5x5/, etc)
- Skip already-processed dates for idempotent execution
- Upfront extent checking (one-time), per-date processing (resumable)
- Reduce storage by excluding empty tiles outside field boundaries
 2026-01-13 09:35:51 +01:00
Timon cb63cf00b8 updated code to fit in workflow better 2026-01-12 16:33:23 +01:00
Timon 182d13d593 wip 2026-01-12 14:55:25 +01:00
Timon dc7c346854 updated whole workflow and architercutre md 2026-01-12 09:34:13 +01:00
Timon d22dc2f96e commit all stuff 2026-01-06 14:17:37 +01:00
Timon d5fd4bb463 Add KPI reporting system and deployment documentation 
Major Changes:
- NEW: Scripts 09 & 10 for KPI calculation and enhanced reporting
- NEW: Shell script wrappers (01-10) for easier execution
- NEW: R packages flextable and officer for enhanced Word reports
- NEW: DEPLOYMENT_README.md with complete deployment guide
- RENAMED: Numbered R scripts (02, 03, 04) for clarity
- REMOVED: Old package management scripts (using renv only)
- UPDATED: Workflow now uses scripts 09->10 instead of 05

Files Changed: 90+ files
New Packages: flextable, officer
New Scripts: 09_run_calculate_kpis.sh, 10_run_kpi_report.sh
Documentation: DEPLOYMENT_README.md, EMAIL_TO_ADMIN.txt

See DEPLOYMENT_README.md for full deployment instructions.
 2025-10-14 11:49:30 +02:00
Martin Folkerts d258a2bf2e cleaned the folders, made some parameters 2025-09-05 15:23:41 +02:00
Martin Folkerts 77d67c36ca Merge branch 'code-improvements' 2025-06-25 11:03:33 +02:00
Timon a6aa002d33 minor visualistion chagnes 2025-06-25 11:01:37 +02:00
Martin Folkerts 0e7f5cf11d Merge branch 'code-improvements' 2025-06-25 10:47:11 +02:00
Timon ea6842af48 minor visualistion chagnes 2025-06-25 10:39:26 +02:00
Martin Folkerts cc6b9392d6 Merge branch 'code-improvements' 2025-06-25 09:35:19 +02:00
Timon b2fcd66d2b klaar voor testen, wat plaatjes e.d. aangepast 2025-06-24 21:02:54 +02:00
Timon 4d6439d5e0 Add reproducible R package management system 
- Add renv.lock with exact package versions for reproducibility
- Add package_manager.R for automated package installation/updates
- Add extract_current_versions.R for package discovery
- Configure renv environment for team collaboration

All team members can now run source('r_app/package_manager.R') to get identical package environment
 2025-06-24 14:51:22 +02:00
Martin Folkerts a316b30afc Merge branch 'code-improvements' 2025-06-24 09:58:56 +02:00
Timon 6efcc8cfec Fix CI report pipeline: update tmap v4 syntax, add continuous color scales, fix formatting 
- Updated all CI maps to use tm_scale_continuous() for proper tmap v4 compatibility
- Added fixed color scale limits (1-8 for CI, -3 to +3 for differences) for consistent field comparison
- Fixed YAML header formatting issues in CI_report_dashboard_planet.Rmd
- Positioned RGB map before CI overview map as requested
- Removed all obsolete use_breaks parameter references
- Enhanced error handling and logging throughout the pipeline
- Added new experimental analysis scripts and improvements to mosaic creation
 2025-06-19 20:37:20 +02:00
Timon bb2a599075 Enhanced SmartCane executive summary report with explanatory text and fixed priority map coloring 
Added explanatory text for all visualizations
Fixed priority map color scheme (red=high priority, green=low priority)
Improved error handling in farm health data calculations
Added fallback mechanisms for missing data
 2025-04-23 09:47:19 +02:00
Timon 2bed5949fa Refactor CI plotting functions to use proper parameterization 
- Refactored ci_plot and cum_ci_plot functions to accept explicit parameters
- Added default values for backward compatibility
- Updated CI_report_dashboard_planet.Rmd to pass explicit parameters
- Added input validation for better error handling
- Added parameter documentation in function headers
- Removed reliance on global variables for better maintainability
 2025-04-22 21:41:18 +02:00
Timon 07aee7bed1 –[200~Improve CI report visualization and migrate to terra package 
- Replace raster package with terra throughout the codebase
- Update map visualizations with better layout and legends
- Add descriptive headers to report sections
- Improve map legend positioning and sizing
- Enhance error handling for missing data
- Remove redundant legends in field-specific visualizations
- Optimize figure dimensions to prevent page overflow
- Expand documentation of CI index and report components
- Update package dependencies in packages.
 2025-04-22 20:55:02 +02:00
Timon fbc8f5e11b wip 2025-04-22 17:43:11 +02:00
Martin Folkerts eb9803fbf1 Removed yieldprediction from CI_report_dashboard_planet.Rmd edited online with Bitbucket 2025-04-02 12:53:29 +00:00
Martin Folkerts ddad4240d5 added conditional to not run prediction if requirements are not met 2025-03-25 16:09:31 +01:00
Timon 29e2867b41 adding docx 2025-03-25 10:42:04 +01:00
Martin Folkerts dd1ee2b1a0 adjusted dates to iso dates 2025-01-07 14:24:38 +01:00
Timon d0ff25acf9 timely save 2024-12-16 16:26:34 +01:00
Martin Folkerts 9b29dc4d57 updated signature of cum ci plot 2024-12-03 09:32:04 +01:00
Martin Folkerts 3bfa7e1750 some update to get mkulazi to work 2024-11-18 13:41:58 +01:00
Martin Folkerts 576250982c fixed plot colouring and made ci_extraction resilient against none file; 2024-11-11 20:55:59 +01:00
Martin Folkerts c8ef406545 added Catalog package with config and changed iteration to use available_slots 2024-11-11 10:26:02 +01:00
Martin Folkerts db68fd8e83 removed column for plot colors 2024-11-07 11:43:09 +01:00
Martin Folkerts 3093c86e3a okay 2024-10-31 22:35:59 +01:00
Timon c1a32bf6ad Refactor cum_ci_plot function for enhanced plotting options; added parameters for plot type and season filtering; improved x-axis labeling and updated ggplot aesthetics. Also, continued work on yield prediction in R. 2024-10-31 11:21:39 +01:00
Martin Folkerts ddb4fdbc02 updated date format for excel 2024-10-28 15:57:23 +01:00
Martin Folkerts f08da04c3b added DownloadCreditsCommand some updates 2024-10-23 09:34:52 +02:00
Martin Folkerts 056b8dd67b okay 2024-09-17 14:56:33 +02:00
Martin Folkerts 4beca30d2e okay 2024-08-30 16:27:32 +02:00
Martin Folkerts 7bf4f0930d added log as project file; 2024-08-30 16:16:26 +02:00
Martin Folkerts 499744db6d moved logging arroudn 2024-08-30 12:09:13 +02:00
Martin Folkerts 7d671ba69c moved end_date and added a is.na wrapping to set when running in R_Studio 2024-08-30 10:37:02 +02:00
Timon 4dd84b10a5 source(utils) aangepast voor online versie 2024-08-30 09:52:02 +02:00
Timon bd9b7d4ab0 adjusted all functions and cleaned scripts incl rmd file 2024-08-29 16:48:24 +02:00
Timon 611bc17017 adjusted all functions and cleaned scripts 2024-08-29 16:35:59 +02:00
Martin Folkerts ad2f41482f updated create_report.sh to have borders option 2024-08-28 14:52:42 +02:00
Martin Folkerts 2598306643 removed chemba as a option from parameters so it is treathed as a regular project 2024-08-28 10:18:53 +02:00
Martin Folkerts 23ba088434 added logging to getTifsAsZip 2024-08-27 11:46:34 +02:00
Martin Folkerts 509f924c58 me 2024-08-26 16:23:52 +02:00
Martin Folkerts c5f9c8e3ca okay 2024-08-26 16:19:22 +02:00
Martin Folkerts dce7344947 updated project 2024-08-26 14:01:57 +02:00
Martin Folkerts 64a576d638 okay 2024-08-20 17:01:58 +02:00
Martin Folkerts fcd900075b toch niet 2024-07-19 15:59:28 +02:00
Martin Folkerts 3e74c72da0 okay 2024-07-19 15:54:00 +02:00
Martin Folkerts 3a081bf72a wip; 2024-07-19 15:08:14 +02:00
Martin Folkerts 7fc9f552ab okay 2024-07-19 15:04:23 +02:00
Martin Folkerts 3813c40ac6 okay 2024-07-19 14:49:05 +02:00
Martin Folkerts 86532128aa okay 2024-07-19 14:45:41 +02:00
Martin Folkerts 14dee6def4 olay 2024-07-19 14:42:03 +02:00
Martin Folkerts 861244fec1 okay 2024-07-19 14:38:50 +02:00
Martin Folkerts 0109a57b04 run with one day of 2024-07-19 14:10:49 +02:00
Martin Folkerts 591a30d2a8 wip 2024-07-19 14:02:37 +02:00
Martin Folkerts 19e3f79371 wip 2024-07-19 13:28:14 +02:00
Martin Folkerts 8bac58b121 okay 2024-07-19 10:57:50 +02:00
Martin Folkerts 24cbea7b0b wip 2024-07-18 15:57:13 +02:00
Martin Folkerts 5645ad83e5 changes 2024-07-17 14:53:46 +02:00
Martin Folkerts 0b1826dccb wip 2024-07-15 11:45:21 +02:00
Martin Folkerts 2988516db2 added conditional to skip field/subfield when no downloads available in current rds 2024-07-15 10:41:13 +02:00
Martin Folkerts 3ea7eec68e some more work 2024-07-09 15:47:53 +02:00
Martin Folkerts 1de6dd4edf wip 2024-07-04 21:16:12 +02:00
Martin Folkerts d9fafcea43 wip 2024-07-02 16:51:55 +02:00
Martin Folkerts 1f194c5670 wip 2024-07-01 13:39:21 +02:00
guillaume91 3f7043199b okay 2024-06-26 13:00:22 +02:00
guillaume91 ea38c952ba Split 2_CI_data_prep.R into 2 files (+3 utils files) 2024-06-26 12:07:21 +02:00
Martin Folkerts af1135e52a wip 2024-06-25 13:37:45 +02:00
Martin Folkerts 607a221028 changed mosaic r script to work with end_date and offset instead of week 2024-06-24 11:39:47 +02:00
Martin Folkerts 1e841a0aa6 wip 2024-06-18 09:17:57 +02:00
Martin Folkerts e2138a378a some updates 2024-05-13 19:35:17 +02:00
Martin Folkerts 95633a553c fixed whitespace issue in subchunkify 2024-05-06 11:46:51 +02:00
Timon b6c576ca57 working rmd file, removed renv because it messed up my R and could not get it to work 2024-04-18 16:19:17 +02:00
Martin Folkerts 5886f59bd8 wip 2024-04-18 13:43:14 +02:00
Martin Folkerts 4051b802e2 wip 2024-04-18 11:11:24 +02:00
Martin Folkerts d2fa24fc6c added some changes to timons merged branch 2024-03-18 11:55:35 +01:00
Timon 88d871d865 updated harvest datata stadnadr 2024-03-14 13:01:19 +01:00
Timon 4053de50df updated harvest datata stadnadr 2024-03-14 13:00:02 +01:00
Timon 66d6f80532 adjusted scripts to be more generalised 2024-03-13 10:27:10 +01:00
Timon 238520ff89 parameterised harvestdata etc 2024-03-12 22:18:57 +01:00
Martin Folkerts 62b9942a53 wip 2024-03-12 12:09:29 +01:00
Martin Folkerts 25715ddef6 wip 2024-03-11 10:36:47 +01:00
Martin Folkerts f7d660356c okay; 2024-03-10 21:53:20 +01:00
Martin Folkerts e996a274db merged testing 2024-03-05 09:40:46 +01:00
Martin Folkerts c222314950 wip 2024-03-05 09:15:46 +01:00
Timon 952700f2c4 forgot to save last R file 2024-03-04 17:22:08 +01:00
Timon 6c00c57b37 changed CI extract to extract all sat images if parameter new_project_question is set to TRUE, to add when new project start 2024-03-04 17:12:20 +01:00
Timon 86f4a49d30 minor changes in script to get it running on my device 2024-03-04 12:15:23 +01:00
Timon 07a54eab25 minor changes 2024-03-01 16:32:53 +01:00
Timon a976350ce5 first commit test 2024-03-01 14:41:55 +01:00
Martin Folkerts 645929ae4e fixed build-report refactored enum status and HasStatusTrait 2024-02-14 21:01:15 +01:00
Martin Folkerts a7a333fb0f worked with timon to harmonize report output 2024-02-08 17:06:35 +01:00
Martin Folkerts 3a18e3b8a7 okeay 2024-01-31 11:58:01 +01:00
Martin Folkerts 0cfd8ad8bf fix 2024-01-31 11:54:02 +01:00
Martin Folkerts 9f60d3db8a added leading zero to weeknumbers so mosaics van be found for week numbers smaller then 9 2024-01-31 11:47:56 +01:00
Martin Folkerts a19e7667e6 wip 2024-01-30 15:17:26 +01:00
Martin Folkerts e224080232 added leading zero to weeknumber in file name format 2024-01-30 15:14:28 +01:00
Martin Folkerts 33cb0dd7da wip 2024-01-24 21:46:08 +01:00
Martin Folkerts d3c2c18918 revised report 2024-01-16 14:43:22 +01:00
Martin Folkerts 3eaf471f05 wip 2024-01-10 21:42:32 +01:00
Martin Folkerts da9d94880f wip 2024-01-09 10:49:25 +01:00
Martin Folkerts 4c22ab6758 wip 2024-01-03 22:06:03 +01:00
Martin Folkerts 3fb55ddcce added rule for missing mergedTifs; 2023-12-24 18:02:17 +01:00
Martin Folkerts bd4ac224ad added CI graphs to pdf 2023-12-12 16:37:47 +01:00
Martin Folkerts 36d74ed2af working version of all python scripts and r-scripts 2023-12-12 14:58:44 +01:00
Martin Folkerts c40ff0c79f updated scripts 2023-12-12 11:20:05 +01:00
Martin Folkerts 3fe61f8ab8 wip 2023-11-06 12:14:13 +01:00
Martin Folkerts dbc8a13600 wip\ 2023-10-31 15:03:32 +01:00
Martin Folkerts 0b7294340c updated r_app/.gitignore to reflect install strategy. 
removed renv and .Rprofile from project so every instance should run its own enviroment based on installPackages.R.
 2023-10-31 14:12:12 +01:00
Martin Folkerts 2a872cf8a3 wip 2023-10-17 11:08:44 +02:00
Martin Folkerts 3ce436b6a0 added renv::restore() and renv::activate() 2023-10-16 17:34:11 +02:00
Martin Folkerts dda9ad8871 build version of r.lock 2023-10-16 17:20:46 +02:00
Martin Folkerts 568565bc76 wip 2023-10-16 15:45:52 +02:00
Martin Folkerts 78e6567182 wip 2023-10-10 17:02:35 +02:00