Enhance download progress feedback and optimize tile overlap checks
- Added a progress bar to the tile download process for better user feedback. - Simplified the tile overlap checking logic in the R script to improve performance and readability.
This commit is contained in:
parent
03f269302a
commit
fc7e5f1ee0
432
.github/copilot-instructions.md
vendored
432
.github/copilot-instructions.md
vendored
|
|
@ -1,138 +1,322 @@
|
|||
# Copilot Instruct- **Crop Analysis & Messaging:**
|
||||
- Main script: `r_app/06_crop_messaging.R`
|
||||
- Usage: `Rscript 06_crop_messaging.R [current_week] [previous_week] [estate_name]`
|
||||
- Two-dimensional alerting: Time (week-over-week changes) + Space (field uniformity/patches)
|
||||
- Handles missing weeks due to clouds (CI band = 0)
|
||||
- Output: WhatsApp-ready text (.txt) and Word reports (.docx) with farm-wide summary, missing data notes, areas in hectares and acres, and interpretation guides for columns
|
||||
- Filenames include estate name (e.g., `crop_messaging_simba.txt`, `crop_messaging_simba.docx`)or SmartCane Codebase
|
||||
# SmartCane Copilot Instructions for AI Coding Agents
|
||||
|
||||
## Big Picture Architecture
|
||||
- **Three main components:**
|
||||
- `r_app/`: R scripts for crop analysis, package management, and reporting
|
||||
- `r_app/experiments/sar_dashboard/`: Production SAR dashboard system (Python + R)
|
||||
- `python_app/` & `python_scripts/`: Python notebooks and scripts for satellite data download and preprocessing
|
||||
- `laravel_app/`: Laravel PHP web application for user-facing features
|
||||
- **Data Flow:**
|
||||
- Satellite data is downloaded/preprocessed in Python, stored in `python_scripts/data/` or `r_app/experiments/sar_dashboard/data/`
|
||||
- R scripts in `r_app/` analyze, visualize, and report on this data
|
||||
- SAR dashboard combines Python download + R analysis + Word report generation
|
||||
- Reports and outputs are saved in `output/` or `r_app/experiments/sar_dashboard/`
|
||||
- Laravel app may consume outputs for web display (integration is project-specific)
|
||||
## Architecture Overview
|
||||
|
||||
**SmartCane** is a multi-stage agricultural intelligence platform that processes satellite imagery into crop health analysis and harvest predictions. The system spans **Python, R, and PHP**, with a file-based processing architecture.
|
||||
|
||||
### Core Data Pipeline
|
||||
```
|
||||
[Python] Download 4-band imagery
|
||||
→ [R Stage 01] Create tile grid & split into 25 tiles/day
|
||||
→ [R Stage 02] Extract CI (Canopy Index) per field
|
||||
→ [R Stage 03] Growth model interpolation (smooth time series)
|
||||
→ [R Stage 04] Create weekly 5-band mosaics
|
||||
→ [R Stage 05] Field-level KPI calculation & alerting
|
||||
→ [R Stage 06] Generate Word/HTML reports
|
||||
```
|
||||
|
||||
**Key Data Flow**:
|
||||
- Raw 4-band GeoTIFFs (RGB+NIR, uint16) downloaded from Planet API via Python
|
||||
- Stored in: `laravel_app/storage/app/{project}/merged_tif/`
|
||||
- Field boundaries (`pivot.geojson`) used across all stages for masking/analysis
|
||||
- Outputs: RDS (intermediate data), TIF (rasters), Excel/Word (reports)
|
||||
|
||||
### Main Components
|
||||
| Component | Purpose | Key Files |
|
||||
|-----------|---------|-----------|
|
||||
| `r_app/` | Core 6-stage R pipeline | `01-10_*.R`, `*_utils.R`, `parameters_project.R` |
|
||||
| `python_app/` | Satellite download & ML harvest prediction | `00_download_8band_pu_optimized.py`, `01-02_harvest_*.py` |
|
||||
| `r_app/experiments/sar_dashboard/` | SAR (Sentinel-1 radar) analysis | `download_s1_*.py`, `generate_sar_report.R` |
|
||||
| `laravel_app/` | Web dashboard (optional integration) | Standard Laravel structure |
|
||||
|
||||
## Critical Developer Workflows
|
||||
- **R Package Management:**
|
||||
- Always run `r_app/package_manager.R` after pulling changes or before analysis
|
||||
- Commit `renv.lock` but NOT the `renv/` folder
|
||||
- Use `source("r_app/package_manager.R")` in RStudio or `Rscript r_app/package_manager.R` in terminal
|
||||
- **Crop Analysis & Messaging:**
|
||||
- Main script: `r_app/06_crop_messaging.R`
|
||||
- Usage: `Rscript 06_crop_messaging.R [current_week] [previous_week] [estate_name]`
|
||||
- Two-dimensional alerting: Time (change trends) + Space (field uniformity/patches)
|
||||
- Handles missing weeks due to clouds (CI band = 0)
|
||||
- Output: WhatsApp-ready text, CSV data, .docx reports, and Markdown tables
|
||||
- **SAR Analysis & Reporting:**
|
||||
- **SAR Dashboard:** Production-ready Word reports for SAR data analysis
|
||||
- **Main folder:** `r_app/experiments/sar_dashboard/`
|
||||
- **Download script:** `r_app/experiments/sar_dashboard/download_s1_simba.py` (for Simba) or `download_s1_[client].py`
|
||||
- **Report generation:** `Rscript r_app/experiments/sar_dashboard/generate_sar_report.R [client_name]`
|
||||
- **Test script:** `Rscript r_app/experiments/sar_dashboard/test_sar_dashboard.R`
|
||||
- **Data source:** `r_app/experiments/sar_dashboard/data/[client]/weekly_SAR_mosaic/`
|
||||
- **Features:** RGB visualization (each band = different week), SAR indices (RVI, cross-pol ratio), harvest detection, field uniformity analysis, time series plots
|
||||
- **Output:** Word document (.docx) with comprehensive SAR analysis and visualizations
|
||||
- **Field boundaries:** Uses `r_app/experiments/pivot.geojson` for field polygons
|
||||
- **Python Data Download:**
|
||||
- Notebooks/scripts in `python_app/` and `python_scripts/` handle satellite data acquisition
|
||||
- Check `requirements_*.txt` for dependencies
|
||||
- **Laravel App:**
|
||||
- Standard Laravel conventions (see `laravel_app/README.md`)
|
||||
- Use `artisan` for migrations, tests, etc.
|
||||
|
||||
### 1. R Package Setup (DO FIRST)
|
||||
```powershell
|
||||
Rscript r_app/package_manager.R
|
||||
# Manages renv.lock; commit this but NOT renv/ folder
|
||||
```
|
||||
|
||||
### 2. Full Pipeline (Typical Weekly Run)
|
||||
```powershell
|
||||
# 2a. Download satellite data
|
||||
cd python_app
|
||||
python 00_download_8band_pu_optimized.py angata # or chemba, xinavane, etc.
|
||||
|
||||
# 2b. Run full R pipeline
|
||||
cd ../r_app
|
||||
Rscript 01_create_master_grid_and_split_tiffs.R
|
||||
Rscript 02_ci_extraction.R
|
||||
Rscript 03_interpolate_growth_model.R
|
||||
Rscript 04_mosaic_creation.R
|
||||
Rscript 09_field_analysis_weekly.R
|
||||
Rscript 10_CI_report_with_kpis_simple.Rmd
|
||||
```
|
||||
|
||||
### 3. Python Data Download
|
||||
- **Script**: `00_download_8band_pu_optimized.py`
|
||||
- **Usage**: `python 00_download_8band_pu_optimized.py [PROJECT] [--options]`
|
||||
- **Key Options**: `--date`, `--resolution`, `--clear-all`
|
||||
- **Batch Mode**: `python download_planet_missing_dates.py --start 2025-11-01 --end 2025-12-24 --project angata`
|
||||
- **Output**: `laravel_app/storage/app/{project}/merged_tif/{YYYY-MM-DD}.tif` (4-band uint16)
|
||||
- **Cost**: ~1,500-2,000 PU/date (optimized for cloud masking & bbox reduction)
|
||||
|
||||
### 4. Stage-Specific Execution
|
||||
```powershell
|
||||
# Only KPI + reporting (if earlier stages done):
|
||||
Rscript 09_field_analysis_weekly.R
|
||||
Rscript 10_CI_report_with_kpis_simple.Rmd
|
||||
|
||||
# Crop messaging/alerts (WhatsApp-ready output):
|
||||
Rscript 06_crop_messaging.R [week] [prev_week] [estate_name]
|
||||
|
||||
# Experimental harvest prediction:
|
||||
Rscript 11_yield_prediction_comparison.R
|
||||
Rscript 12_temporal_yield_forecasting.R
|
||||
|
||||
# SAR (radar) analysis:
|
||||
cd r_app/experiments/sar_dashboard
|
||||
python download_s1_simba.py && Rscript generate_sar_report.R simba
|
||||
```
|
||||
|
||||
## Project-Specific Conventions
|
||||
- **Field Uniformity & Alerting:**
|
||||
- **Two-dimensional analysis**: Time (week-over-week changes) + Space (field homogeneity)
|
||||
- **Message categories**: 🚨 URGENT, ⚠️ ALERT, ✅ POSITIVE, 💡 OPPORTUNITY
|
||||
- **Uniformity thresholds**: CV < 0.15 (good), CV < 0.08 (excellent), CV > 0.25 (poor)
|
||||
- **Change detection**: Increase > 0.5, Decrease < -0.5 (configurable thresholds)
|
||||
- **Spatial patterns**: Moran's I analysis for clustering detection
|
||||
- **Missing data handling**: Clouds (CI=0) trigger spatial-only analysis
|
||||
- **Output Formatting:**
|
||||
- Word reports (.docx) include split tables for wide data, with column widths set for readability
|
||||
- Interpretation guides provided under each table explaining columns like 'acceptable %' and 'change' thresholds
|
||||
- Areas reported in both hectares and acres
|
||||
- **Package Management:**
|
||||
- Minimum versions enforced for critical R packages (see `PACKAGE_MANAGEMENT.md`)
|
||||
- All package changes go through `package_manager.R`
|
||||
- **SAR-Specific Analysis:**
|
||||
- **Data characteristics:** SAR (radar) penetrates clouds, all-weather capability, measures backscatter intensity
|
||||
- **Bands:** VV (vertical-vertical), VH (vertical-horizontal), dB scaled for analysis
|
||||
- **Indices:** RVI (Radar Vegetation Index), cross-polarization ratio, crop structure index
|
||||
- **Harvest detection:** Identifies completely bare fields by backscatter threshold and temporal change
|
||||
- **RGB visualization:** Each band represents different week for change detection
|
||||
- **Data availability:** Sentinel-1 provides ~6-day revisit, weekly composites recommended
|
||||
- **Field boundaries:** Critical for SAR analysis - ensure `pivot.geojson` is current and accurate
|
||||
|
||||
### Configuration & Parameterization
|
||||
- **Central config**: `r_app/parameters_project.R` — sets PROJECT, paths, field boundaries, data_dir
|
||||
- **Project names**: angata, chemba, xinavane, esa, simba (affects file paths, field boundaries, thresholds)
|
||||
- **Hard-coded dependencies are problematic** (see SC-50, SC-60) — use parameterized paths instead of `"pivot.geojson"` literals
|
||||
- All scripts source `parameters_project.R` at start to get global config
|
||||
|
||||
### Data Storage & Naming
|
||||
- **Daily TIFFs**: `merged_tif/{YYYY-MM-DD}.tif` (4 bands: R,G,B,NIR)
|
||||
- **Tiles**: `daily_tiles_split/{YYYY-MM-DD}/{YYYY-MM-DD}_{TILE_ID}.tif` (25 tiles per day)
|
||||
- **CI data (RDS)**: `combined_CI/combined_CI_data.rds` (cumulative, wide format: fields × dates)
|
||||
- **Weekly mosaic**: `weekly_mosaic/week_{WW}.tif` (5 bands: R,G,B,NIR,CI)
|
||||
- **Outputs**: `output/` for reports; Excel/Word saved with date/week naming
|
||||
|
||||
### Field Uniformity & Alerting
|
||||
- **Two-dimensional analysis**: Time (week-over-week trends) + Space (within-field homogeneity)
|
||||
- **Uniformity metric**: Coefficient of Variation (CV) — CV < 0.15 (good), < 0.08 (excellent), > 0.25 (poor)
|
||||
- **Change thresholds**: +0.5 (increase alert), -0.5 (decrease alert) — tunable in code
|
||||
- **Alert categories**: 🚨 URGENT, ⚠️ ALERT, ✅ POSITIVE, 💡 OPPORTUNITY
|
||||
- **Cloud handling**: When CI=0 (no data), skip temporal analysis, spatial-only assessment
|
||||
|
||||
### RDS File Conventions
|
||||
- **Wide format**: Rows = fields, Columns = dates (CI values)
|
||||
- **Key files**:
|
||||
- `combined_CI_data.rds` — all fields, all dates, cumulative
|
||||
- `All_pivots_Cumulative_CI_quadrant_year_v2.rds` — growth model output with quadrant-level analysis
|
||||
- `{project}_kpi_summary_tables_week{WW}.rds` — weekly KPI results
|
||||
- **RDS I/O**: Managed by utility functions (`ci_extraction_utils.R`, `growth_model_utils.R`)
|
||||
|
||||
### Word Report Output
|
||||
- Templates in `r_app/` (e.g., `10_CI_report_with_kpis_simple.Rmd`)
|
||||
- Use flextable for split-column tables (wide data rendered as multiple tables)
|
||||
- Include interpretation guides (thresholds, units: hectares + acres)
|
||||
- Filenames: `SmartCane_Report_week{WW}_{YYYY}.docx`
|
||||
|
||||
## Integration Points & Dependencies
|
||||
- **R ↔ Python:**
|
||||
- R scripts expect preprocessed data from Python scripts
|
||||
- Data location conventions: `python_scripts/data/`, `pivot.geojson`
|
||||
- **R ↔ Laravel:**
|
||||
- Laravel may read outputs from R analysis (integration is custom)
|
||||
- **External:**
|
||||
- Sentinel-1 SAR data (via SentinelHub API), Planet optical data, field boundaries (GeoJSON), R/Python packages
|
||||
|
||||
## Examples
|
||||
- To run a full crop analysis workflow:
|
||||
```powershell
|
||||
Rscript r_app/package_manager.R ; Rscript r_app/06_crop_messaging.R 32 31 simba
|
||||
```
|
||||
- To run crop messaging with cloud handling:
|
||||
```powershell
|
||||
Rscript r_app/06_crop_messaging.R 30 29 chemba # Only spatial analysis if week 29 has clouds
|
||||
```
|
||||
- To generate SAR dashboard report:
|
||||
```powershell
|
||||
cd r_app/experiments/sar_dashboard
|
||||
python download_s1_simba.py # Download SAR data for Simba (last 8 weeks)
|
||||
Rscript generate_sar_report.R simba # Generate Word report
|
||||
```
|
||||
- To test SAR dashboard setup:
|
||||
```powershell
|
||||
cd r_app/experiments/sar_dashboard
|
||||
Rscript test_sar_dashboard.R
|
||||
```
|
||||
### R ↔ Python
|
||||
- Python downloads → R expects `merged_tif/` filled with 4-band TIFFs
|
||||
- R harvest prediction (`11_*.R`, `12_*.R`) uses ML models from Python (`model_307.pt`, `model_config.json`, scalers)
|
||||
|
||||
## Key Files & Directories
|
||||
- `r_app/package_manager.R`, `PACKAGE_MANAGEMENT.md`: Package logic & workflow
|
||||
- `r_app/06_crop_messaging.R`, `r_app/crop_messaging_utils.R`: Crop analysis & messaging logic
|
||||
- `r_app/experiments/crop_messaging/crop_analysis_messaging.R`: Experimental messaging script
|
||||
- `r_app/experiments/sar_dashboard/`: Complete SAR dashboard system
|
||||
- `download_s1_simba.py`: SAR data download for Simba fields
|
||||
- `generate_sar_report.R`: Generate Word document SAR reports
|
||||
- `test_sar_dashboard.R`: Test SAR dashboard components
|
||||
- `SAR_Dashboard_Report.Rmd`: RMarkdown template for Word reports
|
||||
- `sar_dashboard_utils.R`: SAR analysis utility functions
|
||||
- `data/[client]/weekly_SAR_mosaic/`: Downloaded SAR data organized by week
|
||||
- `python_scripts/`, `python_app/`: Data download/preprocessing
|
||||
- `output/`: All generated reports
|
||||
- `laravel_app/`: Web application
|
||||
### R ↔ GeoJSON (Field Boundaries)
|
||||
- **File**: `laravel_app/storage/app/{project}/pivot.geojson`
|
||||
- **Used in**: All stages (download, CI extraction, masking, reporting)
|
||||
- **Critical**: Ensure geometry is current; affects all per-field statistics
|
||||
|
||||
## Environment Notes
|
||||
- On Windows, R can be found at: `C:\Program Files\R\R-4.4.3\bin\x64\R.exe`
|
||||
### R ↔ Harvest Data
|
||||
- **File**: `laravel_app/storage/app/{project}/harvest.xlsx`
|
||||
- **Used in**: Growth model (Stage 03), field analysis (Stage 05), reporting
|
||||
- **Format**: Date columns, field identifiers, harvest event flags
|
||||
|
||||
## Documentation & File Creation Policy
|
||||
**IMPORTANT: Minimize markdown file creation to reduce repo clutter**
|
||||
### SAR (Separate Ecosystem, experimental)
|
||||
- Independent Python download + R reporting pipeline
|
||||
- Data stored: `r_app/experiments/sar_dashboard/data/{client}/weekly_SAR_mosaic/`
|
||||
- Outputs: Word reports with VV/VH backscatter, RVI index, harvest detection
|
||||
- Field boundaries: `r_app/experiments/pivot.geojson` (project-specific)
|
||||
|
||||
- **Do NOT create** README.md, START_HERE.md, QUICK_START.md, INDEX.md or any other .mD files automatically
|
||||
- **Only create .md files when:**
|
||||
- User explicitly requests it
|
||||
- A single index/guide for an entire folder (ONE per folder max)
|
||||
- Critical architecture/setup documentation that doesn't exist
|
||||
- **Instead:**
|
||||
- Add comments directly in scripts explaining purpose & usage
|
||||
- Use inline documentation (docstrings, comments)
|
||||
- Reference existing docs rather than creating duplicates
|
||||
- **Experiments folders:** Keep clean - code + minimal comments, no separate guides per experiment
|
||||
- **When in doubt:** Ask the user if they want documentation before creating files
|
||||
## Common Patterns & Gotchas
|
||||
|
||||
### Pattern: Utility Functions for Reusable Logic
|
||||
- `ci_extraction_utils.R` — tile detection, RDS I/O, CI calculation variants
|
||||
- `growth_model_utils.R` — interpolation, smoothing, gap-filling
|
||||
- `kpi_utils.R` — threshold-based alerting, uniformity metrics
|
||||
- **Why**: Keeps main scripts readable, enables testing individual logic
|
||||
|
||||
### Pattern: Source Config Once
|
||||
```r
|
||||
source("parameters_project.R") # Sets PROJECT, data_dir, field_boundaries_path, etc.
|
||||
# All downstream code uses these globals
|
||||
```
|
||||
|
||||
### Gotcha: File Path Dependencies
|
||||
- **Problem**: Hard-coded paths like `file.path("laravel_app/storage/app", PROJECT, "pivot.geojson")`
|
||||
- **Better**: Pass as parameters or configure in `parameters_project.R`
|
||||
- **Benefit**: Code reusable across projects; easier testing
|
||||
|
||||
### Gotcha: Cloud Masking in CI Extraction
|
||||
- If `CI == 0` globally for a date → entire date flagged as cloudy
|
||||
- Growth model drops entire date rather than interpolating; impacts trend analysis
|
||||
|
||||
## AI Agent Behavior & Approach
|
||||
|
||||
### Terminal Commands
|
||||
- **R execution on Windows**: Use PowerShell `&` operator with full R path
|
||||
- Syntax: `& "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" script.R`
|
||||
- Example: `& "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" 01_create_master_grid_and_split_tiffs.R`
|
||||
- The `&` operator invokes commands with spaces in the path
|
||||
- Verify R path is correct for the Windows installation before running
|
||||
|
||||
### Testing & Temporary Code
|
||||
- **Test file naming**: Always include `test` in filename to signal removal (e.g., `test_ci_extraction.R`, `test_download.py`)
|
||||
- **Test locations**:
|
||||
- R tests: `r_app/test/test_*.R`
|
||||
- Python tests: `python_app/test/test_*.py`
|
||||
- **Default test project**: Use `angata` for development/testing unless specified otherwise
|
||||
- **Cleanup**: Test files are temporary—learnings are copied to production scripts, then test files are deleted
|
||||
- **Do NOT commit** test files to main branch
|
||||
|
||||
### Output & Documentation
|
||||
- **NO auto-generated summary files**: Do NOT create README.md, SUMMARY.txt, or any markdown/text summaries unless explicitly requested
|
||||
- **Chat-based summaries only**: Briefly summarize results and findings directly in chat conversation
|
||||
- This keeps the repo clean and avoids clutter; user will ask for docs if needed
|
||||
|
||||
### Critical Thinking & Partnership
|
||||
- **Ask clarifying questions** before implementing:
|
||||
- "Why do you want to modify this stage? What problem are we solving?"
|
||||
- "Have you considered [alternative approach]? It might be simpler/faster/cleaner"
|
||||
- "Is this for a specific project (angata/esa/chemba)? That affects which configs to change"
|
||||
- **Suggest alternatives**:
|
||||
- If a request seems inefficient, propose better options
|
||||
- Point out if changes might affect other stages or projects
|
||||
- Highlight potential issues (e.g., "This change requires updating parameters_project.R for each project")
|
||||
- **Challenge assumptions**:
|
||||
- "Do we need to modify Stage 02, or would a configuration change in parameters_project.R work?"
|
||||
- "Is this a one-time fix or a pattern we should refactor?"
|
||||
- "Will this break existing reports or KPI calculations?"
|
||||
- **Be a thinking partner, not an order-taker**: Help you make better decisions, not just execute requests
|
||||
|
||||
## File Structure (Key Locations)
|
||||
|
||||
```
|
||||
r_app/
|
||||
├── 01_create_master_grid_and_split_tiffs.R
|
||||
├── 02_ci_extraction.R
|
||||
├── 03_interpolate_growth_model.R
|
||||
├── 04_mosaic_creation.R
|
||||
├── 06_crop_messaging.R
|
||||
├── 09_field_analysis_weekly.R
|
||||
├── 10_CI_report_with_kpis_simple.Rmd
|
||||
├── 11_yield_prediction_comparison.R
|
||||
├── 12_temporal_yield_forecasting.R
|
||||
├── *_utils.R (ci_extraction, growth_model, kpi, report, crop_messaging)
|
||||
├── parameters_project.R
|
||||
├── package_manager.R
|
||||
├── renv.lock
|
||||
├── system_architecture/system_architecture.md (full architecture doc)
|
||||
└── experiments/sar_dashboard/
|
||||
├── download_s1_*.py
|
||||
├── generate_sar_report.R
|
||||
└── sar_dashboard_utils.R
|
||||
|
||||
python_app/
|
||||
├── 00_download_8band_pu_optimized.py
|
||||
├── download_planet_missing_dates.py
|
||||
├── 01_harvest_baseline_prediction.py
|
||||
├── 02_harvest_imminent_weekly.py
|
||||
├── model_307.pt
|
||||
└── requirements_*.txt
|
||||
|
||||
laravel_app/
|
||||
├── storage/app/{project}/
|
||||
│ ├── merged_tif/
|
||||
│ ├── daily_tiles_split/
|
||||
│ ├── combined_CI/
|
||||
│ └── weekly_mosaic/
|
||||
└── ... (standard Laravel)
|
||||
|
||||
output/
|
||||
└── (all generated reports, Excel, Word, HTML)
|
||||
```
|
||||
|
||||
### Critical Thinking & Partnership
|
||||
- **Ask clarifying questions** before implementing:
|
||||
- "Why do you want to modify this stage? What problem are we solving?"
|
||||
- "Have you considered [alternative approach]? It might be simpler/faster/cleaner"
|
||||
- "Is this for a specific project (angata/esa/chemba)? That affects which configs to change"
|
||||
- **Suggest alternatives**:
|
||||
- If a request seems inefficient, propose better options
|
||||
- Point out if changes might affect other stages or projects
|
||||
- Highlight potential issues (e.g., "This change requires updating parameters_project.R for each project")
|
||||
- **Challenge assumptions**:
|
||||
- "Do we need to modify Stage 02, or would a configuration change in parameters_project.R work?"
|
||||
- "Is this a one-time fix or a pattern we should refactor?"
|
||||
- "Will this break existing reports or KPI calculations?"
|
||||
- **Be a thinking partner, not an order-taker**: Help you make better decisions, not just execute requests
|
||||
|
||||
## Linear Issue Integration
|
||||
|
||||
### Creating Issues
|
||||
To create a Linear issue, simply ask in the chat:
|
||||
|
||||
```
|
||||
"Create a Linear issue: Fix hard-coded paths in Stage 02 CI extraction"
|
||||
```
|
||||
|
||||
Provide context like:
|
||||
- **Title**: Clear, action-oriented (what needs doing)
|
||||
- **Description**: Problem statement, impact, acceptance criteria
|
||||
- **Project**: Which project (Inception Phase Angata, General backlog, etc.)
|
||||
- **Priority**: High, Medium, Low
|
||||
- **Assignee**: Who should work on it (e.g., yourself, Dimitra)
|
||||
- **Related issues**: Reference other issues if it blocks/relates to them
|
||||
|
||||
Example:
|
||||
```
|
||||
Create Linear issue:
|
||||
Title: Only create tiles that overlap with GeoJSON boundaries
|
||||
Description: Master grid creates all 25 tiles even when empty. Filter by field geometry to save storage.
|
||||
Project: Inception Phase Angata
|
||||
Priority: Medium
|
||||
Related: SC-50 (parameterization work)
|
||||
```
|
||||
|
||||
### Referencing Issues
|
||||
In chat, reference issues by their ID to include full context:
|
||||
|
||||
```
|
||||
"Work on SC-59 - update the system architecture documentation"
|
||||
```
|
||||
|
||||
This pulls the issue details into the conversation so I understand the full scope.
|
||||
|
||||
### Workflow
|
||||
1. **Create issue** → Clear task definition
|
||||
2. **Reference issue in chat** → I fetch details automatically
|
||||
3. **Ask for implementation** → I work toward issue's acceptance criteria
|
||||
4. **Close issue** → Mark as done in Linear, summarize in chat
|
||||
|
||||
|
||||
|
||||
## Debugging & Troubleshooting
|
||||
|
||||
### Data Validation Checkpoints
|
||||
After each major stage, verify:
|
||||
- **Post-Download**: merged_tif/ contains expected date ranges; file sizes ~150-300MB each
|
||||
- **Post-CI Extraction**: combined_CI_data.rds dimensions match (# fields × # dates); no all-NA columns
|
||||
- **Post-Growth Model**: Interpolated values are within expected CI range; no unexpected gaps
|
||||
- **Pre-Reporting**: Weekly mosaic TIF has 5 bands; field analysis RDS has KPI columns present
|
||||
|
||||
## Next Steps for AI Agents
|
||||
|
||||
1. **Understanding a Script**: Check `parameters_project.R` first for config, then trace utility functions
|
||||
2. **Adding Features**: Determine which stage (01-06 or experimental) it belongs to; follow existing pattern in that stage
|
||||
3. **Testing**: Use standalone test data in `r_app/experiments/` or small date ranges with `--start/--end` flags
|
||||
4. **Documentation**: Update `r_app/system_architecture/system_architecture.md` when architecture changes
|
||||
5. **Refactoring**: Avoid hard-coded paths; parameterize and test across multiple projects (angata, esa, etc.)
|
||||
|
||||
---
|
||||
_If any section is unclear or missing, please provide feedback for further refinement._
|
||||
|
||||
_For detailed system architecture, see `r_app/system_architecture/system_architecture.md`. For related Linear issues (code quality, architecture docs), see SC-59, SC-60, SC-61._
|
||||
|
|
|
|||
|
|
@ -327,7 +327,6 @@ def download_tile(
|
|||
client = SentinelHubDownloadClient(config=config)
|
||||
client.download(download_list, max_threads=1) # Sequential to track PU
|
||||
|
||||
print(f" ✓ Downloaded tile")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
|
|
@ -365,14 +364,20 @@ def download_date(
|
|||
|
||||
successful = 0
|
||||
for idx, bbox in enumerate(bbox_list, 1):
|
||||
print(f" [{idx}/{len(bbox_list)}]", end=" ")
|
||||
if download_tile(date_str, bbox, output_dir, config, byoc, resolution):
|
||||
successful += 1
|
||||
|
||||
percentage = (idx / len(bbox_list)) * 100
|
||||
bar_length = 40
|
||||
filled = int(bar_length * idx / len(bbox_list))
|
||||
bar = '█' * filled + '░' * (bar_length - filled)
|
||||
print(f"\r {percentage:3.0f}% |{bar}| {idx}/{len(bbox_list)}", end='', flush=True)
|
||||
|
||||
# Delay to avoid rate limiting (0.002s between requests - can be aggressive with small tiles)
|
||||
time.sleep(0.002)
|
||||
|
||||
print(f"\n Result: {successful}/{len(bbox_list)} tiles downloaded")
|
||||
print() # Newline after progress bar
|
||||
print(f" Result: {successful}/{len(bbox_list)} tiles downloaded")
|
||||
return successful
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -66,35 +66,33 @@ if ("name" %in% names(field_boundaries_sf)) {
|
|||
|
||||
cat(" Fields: ", paste(field_names, collapse = ", "), "\n", sep = "")
|
||||
|
||||
# Helper function: Check if a tile overlaps with any field
|
||||
# Helper function: Check if a tile overlaps with any field (simple bbox overlap)
|
||||
tile_overlaps_fields <- function(tile_extent, field_geoms) {
|
||||
tryCatch({
|
||||
# Create a tile polygon from extent
|
||||
tile_poly <- st_as_sfc(st_bbox(c(
|
||||
xmin = tile_extent$xmin,
|
||||
xmax = tile_extent$xmax,
|
||||
ymin = tile_extent$ymin,
|
||||
ymax = tile_extent$ymax
|
||||
), crs = 4326))
|
||||
# Simple bounding box overlap test - no complex geometry operations
|
||||
# Two boxes overlap if: NOT (box1.xmax < box2.xmin OR box1.xmin > box2.xmax OR
|
||||
# box1.ymax < box2.ymin OR box1.ymin > box2.ymax)
|
||||
|
||||
# Convert tile_poly to sf object for comparison
|
||||
tile_sf <- st_sf(geometry = tile_poly)
|
||||
|
||||
# Filter out empty geometries from field_geoms
|
||||
field_valid <- field_geoms[!st_is_empty(field_geoms)]
|
||||
|
||||
if (length(field_valid) == 0) {
|
||||
return(FALSE) # No valid fields
|
||||
# For each field geometry, check if it overlaps with tile bbox
|
||||
for (i in seq_len(length(field_geoms))) {
|
||||
# Skip empty geometries
|
||||
if (st_is_empty(field_geoms[i])) {
|
||||
next
|
||||
}
|
||||
|
||||
# Get field bbox
|
||||
field_bbox <- st_bbox(field_geoms[i])
|
||||
|
||||
# Check bbox overlap (simple coordinate comparison)
|
||||
x_overlap <- !(tile_extent$xmax < field_bbox$xmin || tile_extent$xmin > field_bbox$xmax)
|
||||
y_overlap <- !(tile_extent$ymax < field_bbox$ymin || tile_extent$ymin > field_bbox$ymax)
|
||||
|
||||
if (x_overlap && y_overlap) {
|
||||
return(TRUE) # Found overlap!
|
||||
}
|
||||
}
|
||||
|
||||
# Create sf from valid fields for intersection check
|
||||
field_sf <- st_sf(geometry = field_valid)
|
||||
|
||||
# Check for ANY intersection
|
||||
result <- st_intersects(tile_sf, field_sf, sparse = FALSE)
|
||||
|
||||
# Return TRUE if any intersection found
|
||||
return(any(result, na.rm = TRUE))
|
||||
return(FALSE) # No overlap found
|
||||
|
||||
}, error = function(e) {
|
||||
cat(" ⚠️ Error checking overlap: ", e$message, "\n", sep = "")
|
||||
|
|
|
|||
Loading…
Reference in a new issue