SmartCane/r_app/system_architecture/system_architecture.md

<!-- filepath: c:\Users\timon\Resilience BV\4020 SCane ESA DEMO - Documenten\General\4020 SCDEMO Team\4020 TechnicalData\WP3\smartcane\r_app\system_architecture.md -->
# SmartCane System Architecture

## Overview

The SmartCane system is a comprehensive agricultural intelligence platform that processes satellite imagery and farm data to provide agronomic insights for sugarcane farmers. The system architecture follows a modular, layered approach with clear separation of concerns between data acquisition, processing, and presentation.

## Architectural Layers

The SmartCane system follows a layered architecture pattern, which is a standard approach in software engineering for organizing complex systems. This architecture divides the system into distinct functional layers, each with specific responsibilities. While these layers aren't explicitly shown as separate visual elements in the diagrams, they help conceptualize how components are organized by their function:



### 1. Data Acquisition Layer
- **Role**: Responsible for fetching raw data from external sources and user inputs
- **Components**: Manual Sentinel Hub Requests, Python API Downloader, User Input Interface
- **Functions**: Manual request setup on Sentinel Hub Requests Builder for specific client fields, connects to satellite data providers, downloads imagery, manages API credentials, performs preliminary data validation

### 2. Processing Layer (SmartCane Engine)
- **Role**: Core analytical engine that transforms raw data into actionable insights
- **Components**: Python API Downloader (pre-processing), R Processing Engine (analytics)
- **Functions**: Image processing, cloud masking, crop index calculation, field boundary processing, statistical analysis, report generation

### 3. Presentation Layer
- **Role**: Delivers insights to end users in accessible formats
- **Components**: Laravel Web App, Email Delivery System
- **Functions**: Interactive dashboards, visualization, report delivery, user management, project scheduling

### 4. Data Storage Layer
- **Role**: Persists system data across processing cycles
- **Components**: File System, Database
- **Functions**: Stores raw imagery, processed rasters, analytical results, user data, configuration

## Key Subsystems

### 1. Python API Downloader
- **Role**: Acquires and pre-processes satellite imagery
- **Inputs**: API credentials, field boundaries, date parameters, evaluation scripts
- **Outputs**: Raw satellite images, merged GeoTIFFs, virtual rasters
- **Interfaces**: External satellite APIs (Planet via Sentinel Hub), file system
- **Orchestration**: Triggered by shell scripts from the Laravel application

### 2. R Processing Engine
- **Role**: Performs advanced analytics and generates insights
- **Inputs**: Processed satellite imagery, field boundaries, harvest data, project parameters
- **Outputs**: Crop indices, mosaics, RDS data files, agronomic reports
- **Interfaces**: File system, report templates
- **Orchestration**: Triggered by shell scripts from the Laravel application

### 3. Laravel Web Application
- **Role**: Provides operator interface and orchestrates the overall system
- **Inputs**: User data, configuration settings
- **Outputs**: Web interface, scheduling, report delivery
- **Interfaces**: Users, database, file system
- **Orchestration**: Controls execution of the SmartCane Engine via shell scripts

### 4. Shell Script Orchestration
- **Role**: Bridges between web application and processing components
- **Functions**: Triggers processing workflows, manages execution environment, handles errors
- **Examples**: runcane.sh, runpython.sh, build_mosaic.sh, build_report.sh

## Data Flow

### Overview
The SmartCane system processes data through a series of well-defined stages, each building upon the previous stage's outputs. Data flows from raw satellite imagery through multiple transformation and analysis steps before becoming actionable insights.

### Detailed Data Flow by Stage

#### 1. **Input Stage** - Data Acquisition Setup
   - **Actors**: System operators (internal team)
   - **Actions**:
     - Manually prepare and submit requests on Sentinel Hub Requests Builder for specific client fields
     - Upload farm data (field boundaries in GeoJSON, harvest data in Excel) via Laravel Web App
     - Configure project parameters (date ranges, project directory, analysis thresholds)
   - **Outputs**:
     - API credentials stored in environment variables
     - Field boundaries: `laravel_app/storage/app/{project}/Data/pivot.geojson` (or `pivot_2.geojson` for ESA)
     - Harvest data: `laravel_app/storage/app/{project}/Data/harvest.xlsx`
     - Project metadata stored in database

#### 2. **Acquisition Stage** - Raw Satellite Data Download
   - **Trigger**: Laravel scheduler or manual execution via shell scripts (`runpython.sh`, `01_run_planet_download.sh`)
   - **Script**: `python_app/01_planet_download.py` (or `.ipynb`)
   - **Inputs**:
     - API credentials (`SH_CLIENT_ID`, `SH_CLIENT_SECRET`)
     - Collection ID for BYOC (Bring Your Own Collection)
     - Field boundaries GeoJSON
     - Date range parameters (`DATE`, `DAYS` environment variables)
     - Evalscript for band selection and cloud masking
   - **Process**:
     1. Parse date range into daily slots
     2. Load field boundaries and split into bounding boxes (5x5 grid)
     3. Check image availability via Sentinel Hub Catalog API
     4. Download images tile by tile for each date and bbox
     5. Merge tiles into daily mosaics using GDAL
     6. Clean up intermediate files
   - **Intermediate Data**:
     - Raw tiles: `single_images/{date}/response.tiff`
     - Virtual rasters: `merged_virtual/merged{date}.vrt`
   - **Outputs**:
     - Daily merged GeoTIFFs: `merged_tif/{date}.tif` (4 bands: Red, Green, Blue, NIR)
     - File naming convention: `YYYY-MM-DD.tif`
   - **Parameters Used**:
     - `resolution = 3` (meters per pixel)
     - `max_threads = 15` for download
     - Grid split: `(5, 5)` bounding boxes

#### 3. **Processing Stage - Step 1: CI Extraction**
   - **Trigger**: Shell script `02_run_ci_extraction.sh`
   - **Script**: `r_app/02_ci_extraction.R` + `ci_extraction_utils.R`
   - **Inputs**:
     - Daily merged GeoTIFFs from acquisition stage
     - Field boundaries (loaded via `parameters_project.R`)
     - Command-line args: `end_date`, `offset` (days lookback), `project_dir`
   - **Process**:
     1. Load configuration via `parameters_project.R`
     2. Generate date list for processing week
     3. For each daily image:
        - Calculate Canopy Index: `CI = NIR / Green - 1`
        - Crop to field boundaries
        - Mask invalid pixels (0 values → NA)
        - Create VRT for fast access
     4. Extract CI statistics per field using `exactextractr`
     5. Save daily extractions as RDS files
     6. Combine daily extractions into cumulative dataset
   - **Intermediate Data**:
     - Processed daily rasters: `merged_final_tif/{date}.tif` (5 bands: R, G, B, NIR, CI)
     - Daily VRTs: `Data/vrt/{date}.vrt`
     - Daily extractions: `Data/extracted_ci/daily_vals/extracted_{date}_whole_field.rds`
   - **Outputs**:
     - Cumulative CI dataset: `Data/extracted_ci/cumulative_vals/combined_CI_data.rds`
     - Structure: `field`, `sub_field`, `{date1}`, `{date2}`, ... (wide format)
   - **Calculations**:
     - Canopy Index (CI) formula: `(NIR / Green) - 1`
     - Statistics per field: `mean`, `count`, `notNA`

#### 4. **Processing Stage - Step 2: Growth Model Interpolation**
   - **Trigger**: Shell script `03_run_growth_model.sh`
   - **Script**: `r_app/03_interpolate_growth_model.R` + `growth_model_utils.R`
   - **Inputs**:
     - Combined CI data (from Step 1)
     - Harvest data with season dates
     - Command-line arg: `project_dir`
   - **Process**:
     1. Load combined CI data and pivot to long format
     2. For each field and season:
        - Filter CI measurements within season dates
        - Apply linear interpolation to fill daily gaps: `approxfun()`
        - Calculate daily CI and cumulative CI
     3. Combine all fields and seasons
   - **Outputs**:
     - Interpolated growth model: `Data/extracted_ci/cumulative_vals/All_pivots_Cumulative_CI_quadrant_year_v2.rds`
     - Structure: `Date`, `DOY`, `FitData`, `value`, `CI_per_day`, `cumulative_CI`, `model`, `season`, `subField`, `field`
   - **Calculations**:
     - `CI_per_day` = today's CI - yesterday's CI
     - `cumulative_CI` = cumulative sum of daily CI values
     - `DOY` (Day of Year) = sequential days from season start

#### 5. **Processing Stage - Step 3: Weekly Mosaic Creation**
   - **Trigger**: Shell script `04_run_mosaic_creation.sh`
   - **Script**: `r_app/04_mosaic_creation.R` + `mosaic_creation_utils.R`
   - **Inputs**:
     - Daily VRT files
     - Field boundaries
     - Command-line args: `end_date`, `offset`, `project_dir`, `file_name` (optional)
   - **Process**:
     1. Find VRT files matching date range
     2. Assess cloud coverage for each image:
        - Calculate % missing pixels per image
        - Classify as: <5% cloud (good), <45% cloud (acceptable), >45% (poor)
     3. Select best images based on cloud coverage
     4. Create composite using `max` function across selected images
     5. Crop to field boundaries
   - **Outputs**:
     - Weekly mosaic: `weekly_mosaic/week_{WW}_{YYYY}.tif` (5 bands: R, G, B, NIR, CI)
     - ISO week numbering used (e.g., `week_41_2025.tif`)
   - **Parameters**:
     - `CLOUD_THRESHOLD_STRICT = 5%` - Preferred images
     - `CLOUD_THRESHOLD_RELAXED = 45%` - Acceptable images
     - Mosaic function: `max` (takes highest CI value across images)

#### 6. **Processing Stage - Step 4: KPI Calculation**
   - **Trigger**: Shell script `09_run_calculate_kpis.sh`
   - **Script**: `r_app/09_calculate_kpis.R` + `kpi_utils.R`
   - **Inputs**:
     - Current week mosaic
     - Previous week mosaic
     - Field boundaries
     - Harvest data (for tonnage_ha)
     - Cumulative CI data (for yield prediction)
     - Command-line args: `end_date`, `offset`, `project_dir`
   - **Process** (6 KPIs calculated):
     1. **Field Uniformity**: Calculate CV (coefficient of variation) per field
     2. **Area Change**: Compare current vs previous week CI, classify as improving/stable/declining
     3. **TCH Forecasted**: Train Random Forest model on historic yield data, predict for mature fields (≥240 days)
     4. **Growth Decline**: Detect declining fields using CI change + spatial autocorrelation (Moran's I)
     5. **Weed Presence**: Detect rapid CI increase (>2.0 units/week) in young fields (<240 days)
     6. **Gap Filling**: Identify areas with low CI (lowest 25th percentile)
   - **Outputs**:
     - KPI results: `reports/kpis/kpi_results_week{WW}.rds`
     - Summary tables: `reports/kpis/{project}_kpi_summary_tables_week{WW}.rds`
     - Field details: `reports/kpis/{project}_field_details_week{WW}.rds`
     - CSV exports: `reports/kpis/csv/*.csv`
   - **Calculations & Thresholds**:
     - CV thresholds: <0.15 (Excellent), <0.25 (Good), <0.35 (Moderate), ≥0.35 (Poor)
     - Change threshold: ±0.5 CI units
     - Weed threshold: >2.0 CI units/week increase + <10% area (Low), <25% (Moderate), ≥25% (High)
     - Decline risk: Combines CI decrease severity with spatial heterogeneity
     - Canopy closure age: 240 days (8 months)

#### 7. **Processing Stage - Step 5: Report Generation**
   - **Trigger**: Shell script `10_run_kpi_report.sh`
   - **Script**: `r_app/10_CI_report_with_kpis_simple.Rmd` (R Markdown)
   - **Inputs**:
     - Weekly mosaics (current and previous)
     - KPI results from Step 4
     - Field boundaries
     - Project parameters
   - **Process**:
     1. Load weekly mosaics and KPI data
     2. Generate visualizations:
        - RGB field maps
        - CI heatmaps
        - Change detection maps
        - KPI summary charts
     3. Create field-by-field analysis pages
     4. Compile into Word document
   - **Outputs**:
     - Executive report: `reports/SmartCane_Report_week{WW}_{YYYY}.docx`
     - HTML version (optional): `reports/SmartCane_Report_week{WW}_{YYYY}.html`

#### 8. **Output Stage** - Insight Delivery
   - **Actors**: Laravel Web App, Email system
   - **Actions**:
     - Laravel accesses generated reports from file system
     - Reports are attached to emails
     - Emails sent to configured recipients
     - (Future) Reports displayed in web dashboard
   - **Outputs**:
     - Email reports delivered to farm managers
     - PDF/DOCX attachments with weekly analysis
     - (Future) Interactive web dashboard

### Data Persistence and Storage

#### File System Structure
```
laravel_app/storage/app/{project}/
├── Data/
│   ├── pivot.geojson (or pivot_2.geojson)           # Field boundaries
│   ├── harvest.xlsx                                  # Historic yield data
│   ├── vrt/                                          # Virtual rasters (daily)
│   └── extracted_ci/
│       ├── daily_vals/                               # Daily CI extractions (RDS)
│       └── cumulative_vals/
│           ├── combined_CI_data.rds                  # Cumulative wide-format CI
│           └── All_pivots_Cumulative_CI_quadrant_year_v2.rds  # Interpolated growth model
├── merged_tif/                                       # Raw daily downloads (4 bands)
├── merged_final_tif/                                 # Processed daily CI rasters (5 bands)
├── weekly_mosaic/                                    # Weekly composite mosaics
│   └── week_{WW}_{YYYY}.tif
├── reports/
│   ├── kpis/                                         # KPI calculation results
│   │   ├── kpi_results_week{WW}.rds
│   │   ├── csv/                                      # CSV exports
│   │   └── field_level/                              # Per-field KPI data
│   ├── SmartCane_Report_week{WW}_{YYYY}.docx        # Final reports
│   └── SmartCane_Report_week{WW}_{YYYY}.html
└── logs/
    └── {YYYYMMDD}.log                                # Execution logs
```

#### Database Storage
- **Projects table**: Project metadata, client info, scheduling
- **Users table**: Operator and client accounts
- **Execution logs**: Script run history, success/failure tracking
- **Email queue**: Pending report deliveries

### Key Parameters by Stage

| Stage | Parameter | Source | Value/Default |
|-------|-----------|--------|---------------|
| **Download** | Resolution | Hardcoded | 3 meters |
| | Days lookback | Env var `DAYS` | 7-28 |
| | Bbox split | Hardcoded | (5, 5) |
| | Max threads | Hardcoded | 15 |
| **CI Extraction** | Offset | Command-line | 7 days |
| | Min valid pixels | Hardcoded | 100 |
| | CI formula | Hardcoded | (NIR/Green) - 1 |
| **Mosaic** | Cloud threshold strict | Hardcoded | 5% |
| | Cloud threshold relaxed | Hardcoded | 45% |
| | Composite function | Hardcoded | max |
| | Week system | Hardcoded | ISO 8601 |
| **KPI** | CV Excellent | Hardcoded | <0.15 |
| | CV Good | Hardcoded | <0.25 |
| | Weed threshold | Hardcoded | 2.0 CI/week |
| | Canopy closure | Hardcoded | 240 days |
| | Change threshold | Hardcoded | ±0.5 CI |

### Intermediate Data Transformations

1. **4-band to 5-band raster**: Raw download (R,G,B,NIR) → Processed (R,G,B,NIR,CI)
2. **Wide to long CI data**: Combined CI data (wide by date) → Long format (Date, field, value)
3. **Point measurements to continuous**: Sparse CI observations → Daily interpolated values
4. **Daily to weekly**: Multiple daily images → Single weekly composite
5. **Raster to statistics**: Spatial CI values → Per-field summary metrics
6. **Field-level to farm-level**: Individual field KPIs → Aggregated farm summaries

## System Integration Points

- **Python-R Integration**: Data handover via file system (GeoTIFF, virtual rasters)
- **Engine-Laravel Integration**: Orchestration via shell scripts, data exchange via file system and database
- **User-System Integration**: Web interface, file uploads, email notifications

## Developed/Customized Elements

- **Custom Cloud Masking Algorithm**: Specialized for agricultural applications in tropical regions
- **Crop Index Extraction Pipeline**: Tailored to sugarcane spectral characteristics
- **Reporting Templates**: Designed for agronomic decision support
- **Shell Script Orchestration**: Custom workflow management for the system's components

## Strategic Role of Satellite Data

Satellite data is central to the SmartCane system, providing:
- Regular, non-invasive field monitoring
- Detection of spatial patterns not visible from ground level
- Historical analysis of crop performance
- Early warning of crop stress or disease
- Quantification of field variability for precision agriculture

## Pilot Utilization Sites

The SmartCane system is currently operational in Mozambique, Kenya, and Tanzania. Future pilot deployments and expansions are planned for Uganda, Colombia, Mexico, Guatemala, South Africa, and Zambia.

---

## System Architecture Diagrams

Below are diagrams illustrating the system architecture from different perspectives.

### Detailed Data Flow and Processing Pipeline

This comprehensive diagram shows the complete data processing pipeline from raw satellite data acquisition through to final report generation. It illustrates all major processing steps, intermediate data products, key parameters, and decision points in the SmartCane system.

```mermaid
graph TD
    %% ===== INPUTS =====
    subgraph INPUTS["📥 INPUTS"]
        API["fa:fa-key API Credentials<br/>(SH_CLIENT_ID, SH_CLIENT_SECRET)"]
        FieldBoundaries["fa:fa-map Field Boundaries<br/>pivot.geojson / pivot_2.geojson"]
        HarvestData["fa:fa-table Harvest Data<br/>harvest.xlsx<br/>(season dates, tonnage_ha)"]
        DateParams["fa:fa-calendar Date Parameters<br/>(end_date, offset, days)"]
        ProjectConfig["fa:fa-cog Project Config<br/>(project_dir, resolution=3m)"]
    end

    %% ===== STAGE 1: DOWNLOAD =====
    subgraph DOWNLOAD["🛰️ STAGE 1: SATELLITE DATA DOWNLOAD"]
        Download["fa:fa-download 01_planet_download.py<br/>───────────────<br/>• Parse date range into slots<br/>• Split field into 5x5 bboxes<br/>• Check image availability<br/>• Download tiles (4 bands)<br/>• Merge with GDAL"]
        
        DownloadParams["📊 Parameters:<br/>• resolution = 3m<br/>• max_threads = 15<br/>• bbox_split = (5,5)<br/>• bands = [R,G,B,NIR]"]
        
        RawTiles["💾 Intermediate:<br/>single_images/{date}/<br/>response.tiff"]
        
        DailyMosaics["📦 OUTPUT:<br/>merged_tif/{date}.tif<br/>───────────────<br/>4 bands: Red, Green,<br/>Blue, NIR<br/>3m resolution"]
    end

    %% ===== STAGE 2: CI EXTRACTION =====
    subgraph CI_EXTRACTION["🔬 STAGE 2: CI EXTRACTION"]
        CIScript["fa:fa-calculator 02_ci_extraction.R<br/>───────────────<br/>• Calculate CI = NIR/Green - 1<br/>• Crop to field boundaries<br/>• Mask invalid pixels<br/>• Extract stats per field"]
        
        CIParams["📊 Parameters:<br/>• offset = 7 days<br/>• min_valid_pixels = 100<br/>• mask zeros as NA"]
        
        ProcessedRasters["💾 Intermediate:<br/>merged_final_tif/{date}.tif<br/>5 bands + VRT files"]
        
        DailyExtract["💾 Intermediate:<br/>extracted_ci/daily_vals/<br/>extracted_{date}.rds<br/>(field, sub_field, mean_CI)"]
        
        CombinedCI["📦 OUTPUT:<br/>combined_CI_data.rds<br/>───────────────<br/>Wide format:<br/>(field, date1, date2, ...)"]
    end

    %% ===== STAGE 3: GROWTH MODEL =====
    subgraph GROWTH_MODEL["📈 STAGE 3: GROWTH MODEL INTERPOLATION"]
        GrowthScript["fa:fa-chart-line 03_interpolate_growth_model.R<br/>───────────────<br/>• Pivot CI to long format<br/>• Filter by season dates<br/>• Linear interpolation<br/>• Calculate daily & cumulative CI"]
        
        GrowthParams["📊 Calculations:<br/>• approxfun() interpolation<br/>• CI_per_day = diff(CI)<br/>• cumulative_CI = cumsum(CI)<br/>• DOY = days from season start"]
        
        InterpolatedModel["📦 OUTPUT:<br/>All_pivots_Cumulative_CI<br/>_quadrant_year_v2.rds<br/>───────────────<br/>(Date, DOY, FitData,<br/>CI_per_day, cumulative_CI,<br/>field, season)"]
    end

    %% ===== STAGE 4: WEEKLY MOSAIC =====
    subgraph WEEKLY_MOSAIC["🗺️ STAGE 4: WEEKLY MOSAIC CREATION"]
        MosaicScript["fa:fa-images 04_mosaic_creation.R<br/>───────────────<br/>• Find VRTs for date range<br/>• Assess cloud coverage<br/>• Select best images<br/>• Composite with max()"]
        
        CloudAssess["🔍 Cloud Assessment:<br/>• <5% cloud → excellent<br/>• <45% cloud → acceptable<br/>• >45% cloud → poor"]
        
        MosaicParams["📊 Parameters:<br/>• ISO week numbering<br/>• composite = max<br/>• crop to boundaries"]
        
        WeeklyMosaic["📦 OUTPUT:<br/>weekly_mosaic/<br/>week_{WW}_{YYYY}.tif<br/>───────────────<br/>5 bands: R,G,B,NIR,CI<br/>Cloud-free composite"]
    end

    %% ===== STAGE 5: KPI CALCULATION =====
    subgraph KPI_CALC["📊 STAGE 5: KPI CALCULATION (6 KPIs)"]
        KPIScript["fa:fa-calculator-alt 09_calculate_kpis.R<br/>───────────────<br/>Calculates 6 KPIs from<br/>current + previous week"]
        
        KPI1["1️⃣ Field Uniformity<br/>• CV = SD / mean<br/>• <0.15 = Excellent<br/>• <0.25 = Good"]
        
        KPI2["2️⃣ Area Change<br/>• Compare week over week<br/>• Classify improving/<br/>  stable/declining areas"]
        
        KPI3["3️⃣ TCH Forecast<br/>• Random Forest model<br/>• Predict yield for fields<br/>  ≥240 days old"]
        
        KPI4["4️⃣ Growth Decline<br/>• CI decrease + spatial<br/>  autocorrelation (Moran's I)<br/>• Risk scoring"]
        
        KPI5["5️⃣ Weed Presence<br/>• Rapid CI increase<br/>  (>2.0 units/week)<br/>• Only fields <240 days"]
        
        KPI6["6️⃣ Gap Filling<br/>• Identify lowest 25%<br/>  CI areas<br/>• Gap severity score"]
        
        KPIParams["📊 Thresholds:<br/>• CV: 0.15, 0.25, 0.35<br/>• Change: ±0.5 CI<br/>• Weed: 2.0 CI/week<br/>• Canopy: 240 days<br/>• Moran's I: 0.7-0.95"]
        
        KPIResults["📦 OUTPUT:<br/>kpi_results_week{WW}.rds<br/>───────────────<br/>• Summary tables<br/>• Field-level details<br/>• CSV exports"]
    end

    %% ===== STAGE 6: REPORTING =====
    subgraph REPORTING["📄 STAGE 6: REPORT GENERATION"]
        ReportScript["fa:fa-file-word 10_CI_report_with_kpis_simple.Rmd<br/>───────────────<br/>• Load mosaics & KPIs<br/>• Generate visualizations<br/>• Field-by-field analysis<br/>• Compile to Word/HTML"]
        
        Visualizations["🎨 Visualizations:<br/>• RGB field maps<br/>• CI heatmaps<br/>• Change detection<br/>• KPI charts"]
        
        FinalReport["📦 OUTPUT:<br/>SmartCane_Report<br/>_week{WW}_{YYYY}.docx<br/>───────────────<br/>• Executive summary<br/>• Farm-wide KPIs<br/>• Field analysis pages<br/>• Recommendations"]
    end

    %% ===== OUTPUTS =====
    subgraph OUTPUTS["📤 OUTPUTS & DELIVERY"]
        EmailDelivery["fa:fa-envelope Email Delivery<br/>───────────────<br/>Reports sent to<br/>farm managers"]
        
        WebDashboard["fa:fa-desktop Web Dashboard<br/>(Future)<br/>───────────────<br/>Interactive field<br/>monitoring"]
    end

    %% ===== ORCHESTRATION =====
    Laravel["fa:fa-server Laravel Web App<br/>───────────────<br/>Schedules & triggers<br/>processing pipeline"]
    
    ShellScripts["fa:fa-terminal Shell Scripts<br/>───────────────<br/>01-10_run_*.sh<br/>Execute each stage"]

    %% ===== DATA FLOW CONNECTIONS =====
    API --> Download
    FieldBoundaries --> Download
    DateParams --> Download
    ProjectConfig --> Download
    
    Download --> DownloadParams
    DownloadParams --> RawTiles
    RawTiles --> DailyMosaics
    
    DailyMosaics --> CIScript
    FieldBoundaries --> CIScript
    CIScript --> CIParams
    CIParams --> ProcessedRasters
    ProcessedRasters --> DailyExtract
    DailyExtract --> CombinedCI
    
    CombinedCI --> GrowthScript
    HarvestData --> GrowthScript
    GrowthScript --> GrowthParams
    GrowthParams --> InterpolatedModel
    
    ProcessedRasters --> MosaicScript
    FieldBoundaries --> MosaicScript
    MosaicScript --> CloudAssess
    CloudAssess --> MosaicParams
    MosaicParams --> WeeklyMosaic
    
    WeeklyMosaic --> KPIScript
    FieldBoundaries --> KPIScript
    HarvestData --> KPIScript
    InterpolatedModel --> KPIScript
    KPIScript --> KPI1
    KPIScript --> KPI2
    KPIScript --> KPI3
    KPIScript --> KPI4
    KPIScript --> KPI5
    KPIScript --> KPI6
    KPI1 & KPI2 & KPI3 & KPI4 & KPI5 & KPI6 --> KPIParams
    KPIParams --> KPIResults
    
    WeeklyMosaic --> ReportScript
    KPIResults --> ReportScript
    FieldBoundaries --> ReportScript
    ReportScript --> Visualizations
    Visualizations --> FinalReport
    
    FinalReport --> EmailDelivery
    FinalReport --> WebDashboard
    
    Laravel --> ShellScripts
    ShellScripts -.->|Triggers| Download
    ShellScripts -.->|Triggers| CIScript
    ShellScripts -.->|Triggers| GrowthScript
    ShellScripts -.->|Triggers| MosaicScript
    ShellScripts -.->|Triggers| KPIScript
    ShellScripts -.->|Triggers| ReportScript

    %% ===== STYLING =====
    style INPUTS fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    style DOWNLOAD fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    style CI_EXTRACTION fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    style GROWTH_MODEL fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    style WEEKLY_MOSAIC fill:#fce4ec,stroke:#c2185b,stroke-width:2px
    style KPI_CALC fill:#e0f2f1,stroke:#00796b,stroke-width:2px
    style REPORTING fill:#fff9c4,stroke:#f9a825,stroke-width:2px
    style OUTPUTS fill:#ffebee,stroke:#c62828,stroke-width:2px
    
    style DailyMosaics fill:#ffccbc,stroke:#333,stroke-width:1px
    style CombinedCI fill:#ffccbc,stroke:#333,stroke-width:1px
    style InterpolatedModel fill:#ffccbc,stroke:#333,stroke-width:1px
    style WeeklyMosaic fill:#ffccbc,stroke:#333,stroke-width:1px
    style KPIResults fill:#ffccbc,stroke:#333,stroke-width:1px
    style FinalReport fill:#ffccbc,stroke:#333,stroke-width:1px
```

### Overall System Architecture

This diagram provides a high-level overview of the complete SmartCane system, showing how major components interact. It focuses on the system boundaries and main data flows between the Python API Downloader, R Processing Engine, Laravel Web App, and data storage components. This view helps understand how the system works as a whole.

```mermaid
graph TD
    A["fa:fa-satellite External Satellite Data Providers API"] --> PyDL["fa:fa-download Python API Downloader"];
    C["fa:fa-users Users: Farm Data Input e.g., GeoJSON, Excel"] --> D{"fa:fa-laptop-code Laravel Web App"};

    subgraph SmartCane System
        PyDL --> G["fa:fa-folder-open File System: Raw Satellite Imagery, Rasters, RDS, Reports, Boundaries"];
        E["fa:fa-cogs R Processing Engine"] -- Reads --> G;
        E -- Writes --> G;
        
        D -- Manages/Triggers --> F["fa:fa-terminal Shell Script Orchestration"];
        F -- Executes --> PyDL;
        F -- Executes --> E;
        
        D -- Manages/Accesses --> G;
        D -- Reads/Writes --> H["fa:fa-database Database: Project Metadata, Users, Schedules"];

        E -- Generates --> I["fa:fa-file-alt Agronomic Reports: DOCX, HTML"];
        D -- Accesses/Delivers --> I;
    end

    D --> J["fa:fa-desktop Users: Web Interface (future)"];
    I -- Via Email (SMTP) --> K["fa:fa-envelope Users: Email Reports"];

    style E fill:#f9f,stroke:#333,stroke-width:2px
    style D fill:#bbf,stroke:#333,stroke-width:2px
    style PyDL fill:#ffdd57,stroke:#333,stroke-width:2px
```

### R Processing Engine Detail

This diagram zooms in on the R Processing Engine subsystem, detailing the internal components and data flow. It shows how raw satellite imagery and field data progress through various R scripts to produce crop indices and reports. The diagram highlights the data transformation pipeline within this analytical core of the SmartCane system.

```mermaid
graph TD
    subgraph R Processing Engine

        direction TB

        subgraph Inputs
            SatelliteImages["fa:fa-image Raw Satellite Imagery"]
            FieldBoundaries["fa:fa-map-marker-alt Field Boundaries .geojson"]
            HarvestData["fa:fa-file-excel Harvest Data .xlsx"]
            ProjectParams["fa:fa-file-code Project Parameters .R"]
        end

        subgraph Core R Scripts & Processes
            ParamConfig("fa:fa-cogs parameters_project.R")
            MosaicScript("fa:fa-images mosaic_creation.R")
            CIExtractionScript("fa:fa-microscope ci_extraction.R")
            ReportUtils("fa:fa-tools executive_report_utils.R")
            DashboardRmd("fa:fa-tachometer-alt CI_report_dashboard_planet_enhanced.Rmd")
            SummaryRmd("fa:fa-list-alt CI_report_executive_summary.Rmd")
        end

        subgraph Outputs
            WeeklyMosaics["fa:fa-file-image Weekly Mosaics .tif"]
            CIDataRDS["fa:fa-database CI Data .rds"]
            CIRasters["fa:fa-layer-group CI Rasters .tif"]
            DashboardReport["fa:fa-chart-bar Dashboard Report .docx/.html"]
            SummaryReport["fa:fa-file-invoice Executive Summary .docx/.html"]
        end

        %% Data Flow
        ProjectParams --> ParamConfig;

        SatelliteImages --> MosaicScript;
        FieldBoundaries --> MosaicScript;
        ParamConfig --> MosaicScript;
        MosaicScript --> WeeklyMosaics;

        WeeklyMosaics --> CIExtractionScript;
        FieldBoundaries --> CIExtractionScript;
        ParamConfig --> CIExtractionScript;
        CIExtractionScript --> CIDataRDS;
        CIExtractionScript --> CIRasters;

        CIDataRDS --> ReportUtils;
        CIRasters --> ReportUtils;
        HarvestData --> ReportUtils;
        ParamConfig --> ReportUtils;

        ReportUtils --> DashboardRmd;
        ReportUtils --> SummaryRmd;
        ParamConfig --> DashboardRmd;
        ParamConfig --> SummaryRmd;

        DashboardRmd --> DashboardReport;
        SummaryRmd --> SummaryReport;

    end

    ShellOrchestration["fa:fa-terminal Shell Scripts e.g., build_mosaic.sh, build_report.sh"] -->|Triggers| R_Processing_Engine["fa:fa-cogs R Processing Engine"]

    style R_Processing_Engine fill:#f9f,stroke:#333,stroke-width:2px
    style Inputs fill:#ccf,stroke:#333,stroke-width:1px
    style Outputs fill:#cfc,stroke:#333,stroke-width:1px
    style Core_R_Scripts_Processes fill:#ffc,stroke:#333,stroke-width:1px
```

### Python API Downloader Detail

This diagram focuses on the Python API Downloader subsystem, showing its internal components and workflow. It illustrates how API credentials, field boundaries, and other inputs are processed through various Python functions to download, process, and prepare satellite imagery. This view reveals the technical implementation details of the data acquisition layer.

```mermaid
graph TD
    subgraph Python API Downloader

        direction TB

        subgraph Inputs_Py [Inputs]
            APICreds["fa:fa-key API Credentials (SH_CLIENT_ID, SH_CLIENT_SECRET)"]
            DateRangeParams["fa:fa-calendar-alt Date Range Parameters (days_needed, specific_date)"]
            GeoJSONInput["fa:fa-map-marker-alt Field Boundaries (pivot.geojson)"]
            ProjectConfig["fa:fa-cogs Project Configuration (project_name, paths)"]
            EvalScripts["fa:fa-file-code Evalscripts (JS for cloud masking & band selection)"]
        end

        subgraph Core_Python_Logic_Py [Core Python Logic & Libraries]
            SetupConfig["fa:fa-cog SentinelHubConfig & BYOC Definition"]
            DateSlotGen["fa:fa-calendar-check Date Slot Generation (slots)"]
            GeoProcessing["fa:fa-map GeoJSON Parsing & BBox Splitting (geopandas, BBoxSplitter)"]
            AvailabilityCheck["fa:fa-search-location Image Availability Check (SentinelHubCatalog)"]
            RequestHandler["fa:fa-paper-plane Request Generation (SentinelHubRequest, get_true_color_request_day)"]
            DownloadClient["fa:fa-cloud-download-alt Image Download (SentinelHubDownloadClient, download_function)"]
            MergeUtility["fa:fa-object-group Tile Merging (gdal.BuildVRT, gdal.Translate, merge_files)"]
            CleanupUtility["fa:fa-trash-alt Intermediate File Cleanup (empty_folders)"]
        end

        subgraph Outputs_Py [Outputs]
            RawSatImages["fa:fa-file-image Raw Downloaded Satellite Imagery Tiles (response.tiff in dated subfolders)"]
            MergedTifs["fa:fa-images Merged TIFs (merged_tif/{slot}.tif)"]
            VirtualRasters["fa:fa-layer-group Virtual Rasters (merged_virtual/merged{slot}.vrt)"]
            DownloadLogs["fa:fa-file-alt Console Output Logs (print statements)"]
        end
        
        ExternalSatAPI["fa:fa-satellite External Satellite Data Providers API (Planet via Sentinel Hub)"]

        %% Data Flow for Python Downloader
        APICreds --> SetupConfig;
        DateRangeParams --> DateSlotGen;
        GeoJSONInput --> GeoProcessing;
        ProjectConfig --> SetupConfig;
        ProjectConfig --> GeoProcessing;
        ProjectConfig --> MergeUtility;
        ProjectConfig --> CleanupUtility;
        EvalScripts --> RequestHandler;

        DateSlotGen -- Available Slots --> AvailabilityCheck;
        GeoProcessing -- BBox List --> AvailabilityCheck;
        SetupConfig --> AvailabilityCheck;
        AvailabilityCheck -- Filtered Slots & BBoxes --> RequestHandler;
        
        RequestHandler -- Download Requests --> DownloadClient;
        SetupConfig --> DownloadClient;
        DownloadClient -- Downloads Data From --> ExternalSatAPI;
        ExternalSatAPI -- Returns Image Data --> DownloadClient;
        DownloadClient -- Writes --> RawSatImages;
        DownloadClient -- Generates --> DownloadLogs;
        
        RawSatImages --> MergeUtility;
        MergeUtility -- Writes --> MergedTifs;
        MergeUtility -- Writes --> VirtualRasters;
        
    end

    ShellOrchestratorPy["fa:fa-terminal Shell Scripts (e.g., runpython.sh triggering planet_download.ipynb)"] -->|Triggers| Python_API_Downloader["fa:fa-download Python API Downloader"];
    
    style Python_API_Downloader fill:#ffdd57,stroke:#333,stroke-width:2px
    style Inputs_Py fill:#cdeeff,stroke:#333,stroke-width:1px
    style Outputs_Py fill:#d4efdf,stroke:#333,stroke-width:1px
    style Core_Python_Logic_Py fill:#fff5cc,stroke:#333,stroke-width:1px
    style ExternalSatAPI fill:#f5b7b1,stroke:#333,stroke-width:2px
```

### SmartCane Engine Integration Diagram

This diagram illustrates the integration of Python and R components within the SmartCane Engine. Unlike the first diagram that shows the overall system, this one specifically focuses on how the two processing components interact with each other and the rest of the system. It emphasizes the orchestration layer and data flows between the core processing components and external systems.

```mermaid
graph TD
    %% External Systems & Users
    Users_DataInput["fa:fa-user Users: Farm Data Input (GeoJSON, Excel, etc.)"] --> Laravel_WebApp;
    ExternalSatAPI["fa:fa-satellite External Satellite Data Providers API"];

    %% Main Application Components
    Laravel_WebApp["fa:fa-globe Laravel Web App (Frontend & Control Plane)"];
    Shell_Orchestration["fa:fa-terminal Shell Script Orchestration (e.g., runcane.sh, runpython.sh, build_mosaic.sh)"];      subgraph SmartCane_Engine ["SmartCane Engine (Data Processing Core)"]
        direction TB
        Python_Downloader["fa:fa-download Python API Downloader"];
        R_Engine["fa:fa-chart-line R Processing Engine"];
    end
    %% Data Storage
    FileSystem["fa:fa-folder File System (Raw Imagery, Rasters, RDS, Reports, Boundaries)"];
    Database["fa:fa-database Database (Project Metadata, Users, Schedules)"];

    %% User Outputs
    Users_WebView["fa:fa-desktop Users: Web Interface (future)"];
    Users_EmailReports["fa:fa-envelope Users: Email Reports (Agronomic Reports)"];
    AgronomicReports["fa:fa-file-alt Agronomic Reports (DOCX, HTML)"];

    %% --- Data Flows & Interactions ---

    %% Laravel to Orchestration & Engine
    Laravel_WebApp -- Manages/Triggers --> Shell_Orchestration;
    Shell_Orchestration -- Executes --> Python_Downloader;
    Shell_Orchestration -- Executes --> R_Engine;

    %% Python Downloader within Engine
    ExternalSatAPI -- Satellite Data --> Python_Downloader;
    Python_Downloader -- Writes Raw Data --> FileSystem;
    %% Inputs to Python (simplified for this view - details in Python-specific diagram)
    %% Laravel_WebApp -- Provides Config/Boundaries --> Python_Downloader; 


    %% R Engine within Engine
    %% Inputs to R (simplified - details in R-specific diagram)
    %% Laravel_WebApp -- Provides Config/Boundaries --> R_Engine;
    R_Engine -- Reads Processed Data/Imagery --> FileSystem;
    R_Engine -- Writes Derived Products --> FileSystem;
    R_Engine -- Generates --> AgronomicReports;

    %% Laravel interaction with Data Storage
    Laravel_WebApp -- Manages/Accesses --> FileSystem;
    Laravel_WebApp -- Reads/Writes --> Database;
    
    %% Output Delivery
    Laravel_WebApp --> Users_WebView;
    AgronomicReports --> Users_EmailReports;
    %% Assuming a mechanism like SMTP, potentially triggered by Laravel or R-Engine completion
    Laravel_WebApp -- Delivers/Displays --> AgronomicReports;


    %% Styling
    style SmartCane_Engine fill:#e6ffe6,stroke:#333,stroke-width:2px
    style Python_Downloader fill:#ffdd57,stroke:#333,stroke-width:2px
    style R_Engine fill:#f9f,stroke:#333,stroke-width:2px
    style Laravel_WebApp fill:#bbf,stroke:#333,stroke-width:2px
    style Shell_Orchestration fill:#f0ad4e,stroke:#333,stroke-width:2px
    style FileSystem fill:#d1e0e0,stroke:#333,stroke-width:1px
    style Database fill:#d1e0e0,stroke:#333,stroke-width:1px
    style ExternalSatAPI fill:#f5b7b1,stroke:#333,stroke-width:2px
    style AgronomicReports fill:#d4efdf,stroke:#333,stroke-width:1px
```

## Future Directions

The SmartCane platform is poised for significant evolution, with several key enhancements and new capabilities planned to further empower users and expand its utility:

-   **Advanced Management Dashboard**: Development of a more comprehensive and interactive management dashboard to provide users with deeper insights and greater control over their operations.
-   **Enhanced Yield Prediction Models**: Improving the accuracy and granularity of yield predictions by incorporating more variables and advanced machine learning techniques.
-   **Integrated Weather and Irrigation Advice**: Leveraging weather forecast data and soil moisture information (potentially from new data sources) to provide precise irrigation scheduling and weather-related agronomic advice.
-   **AI-Guided Agronomic Advice**: Implementing sophisticated AI algorithms to analyze integrated data (satellite, weather, soil, farm practices) and offer tailored, actionable agronomic recommendations.
-   **Automated Advice Generation**: Developing capabilities for the system to automatically generate and disseminate critical advice and alerts to users based on real-time data analysis.
-   **Expanded Data Source Integration**:
    -   **Radar Data**: Incorporating radar satellite imagery (e.g., Sentinel-1) for all-weather monitoring capabilities, particularly useful during cloudy seasons for assessing crop structure, soil moisture, and biomass.
    -   **IoT and Ground Sensors**: Integrating data from in-field IoT devices and soil sensors for highly localized and continuous monitoring of environmental and soil conditions.
-   **Client-Facing Portal**: Exploration and potential development of a client-facing portal to allow end-users direct access to their data, dashboards, and reports, complementing the current internal management interface.

These future developments aim to transform SmartCane into an even more powerful decision support system, fostering sustainable and efficient agricultural practices.

## Conclusion and Integration Summary

The SmartCane system architecture demonstrates a well-integrated solution that combines different technologies and subsystems to solve complex agricultural challenges. Here is a summary of how the key subsystems work together:

### Subsystem Integration

1. **Data Flow Sequence**
   - The Laravel Web App initiates the workflow and manages user interactions
   - Shell scripts orchestrate the execution sequence of the processing subsystems
   - The Python API Downloader acquires raw data from external sources
   - The R Processing Engine transforms this data into actionable insights
   - Results flow back to users through the web interface and email reports

2. **Technology Integration**
   - **Python + R**: Different programming languages are leveraged for their respective strengths—Python for API communication and data acquisition, R for statistical analysis and report generation
   - **Laravel + Processing Engine**: Clear separation between web presentation layer and computational backend
   - **File System + Database**: Hybrid data storage approach with file system for imagery and reports, database for metadata and user information

3. **Key Integration Mechanisms**
   - **File System Bridge**: The different subsystems primarily communicate through standardized file formats (GeoTIFF, GeoJSON, RDS, DOCX)
   - **Shell Script Orchestration**: Acts as the "glue" between subsystems, ensuring proper execution sequence and environment setup
   - **Standardized Data Formats**: Use of widely-accepted geospatial and data formats enables interoperability

4. **Extensibility and Scalability**
   - The modular architecture allows for replacement or enhancement of individual components
   - The clear subsystem boundaries enable parallel development and testing
   - Standard interfaces simplify integration of new data sources, algorithms, or output methods

The SmartCane architecture balances complexity with maintainability by using well-established technologies and clear boundaries between subsystems. The separation of concerns between data acquisition, processing, and presentation layers ensures that changes in one area minimally impact others, while the consistent data flow pathways ensure that information moves smoothly through the system.