SmartCane/webapps/docs/DEV_LAPTOP_EXECUTION.md

Developer Laptop Manual Execution

This document explains how to manually run the SmartCane pipeline on a Windows developer machine, without the SOBIT Laravel job queue. This is the primary workflow for development and testing.

Overview: Manual Execution Architecture

Instead of web UI buttons and job queues, developers execute R and Python scripts directly in PowerShell, controlling each stage manually.

%% Manual Execution Architecture
flowchart TD
    A["Developer<br/>PowerShell Terminal"] -->|Edit params| B["parameters_project.R<br/>Set PROJECT, dates, paths"]
    
    B -->|Run Stage 00| C["python 00_download_8band_pu_optimized.py<br/>Stage 00"]
    
    C -->|Run Stage 10| D["Rscript 10_create_per_field_tiffs.R<br/>Stage 10"]
    
    D -->|Run Stage 20| E["Rscript 20_ci_extraction_per_field.R<br/>Stage 20"]
    
    E -->|Run Stage 30| F["Rscript 30_interpolate_growth_model.R<br/>Stage 30"]
    
    F -->|Run Stage 40| G["Rscript 40_mosaic_creation_per_field.R<br/>Stage 40"]
    
    G -->|Run Stage 80| H["Rscript 80_calculate_kpis.R<br/>Stage 80"]
    
    H -->|Run Stage 90 OR 91| I{"Client Type?"}
    
    I -->|agronomic_support| J["rmarkdown::render<br/>90_CI_report_*.Rmd"]
    I -->|cane_supply| K["rmarkdown::render<br/>91_CI_report_*.Rmd"]
    
    J -->|Output| L["Word Report<br/>Excel KPI Tables<br/>GeoTIFFs"]
    K -->|Output| L

---

Prerequisites & Environment Setup

System Requirements

• OS: Windows 10+
• R: Version 4.4.0+ (from https://cran.r-project.org/)
• Python: Version 3.9+ (from https://www.python.org/)
• RStudio: Optional but recommended (for debugging)

One-Time Configuration

Step 1: Install R Packages

cd c:\Users\{YOUR_USERNAME}\Documents\SmartCane_code

# Run package manager to install/update all dependencies
& "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app\package_manager.R

# This reads renv.lock and installs exact versions into renv/ folder

What happens:

• package_manager.R uses renv::restore() to install packages from renv.lock
• All packages isolated to project (not system-wide)
• Ensures reproducibility across team members

Step 2: Verify R Installation

# Check R installation path
& "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" --version

# Should output: R version 4.4.3 (or similar)

Step 3: Verify Python Installation

# Check Python
python --version

# Should output: Python 3.9.x or higher

# Create/activate virtual environment (optional but recommended)
python -m venv venv_smartcane
.\venv_smartcane\Scripts\Activate.ps1

# Install Python dependencies
pip install -r python_app\requirements_linux.txt

Step 4: Set Environment Variables (Optional)

# Define R executable path as variable (for easier copy-paste)
$R_EXE = "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe"
$PROJECT = "angata"  # or "chemba", "aura", etc.
$END_DATE = "2026-02-19"
$OFFSET = 7

# Or add to PowerShell $PROFILE for persistence
# Add these lines to C:\Users\{YOUR_USERNAME}\Documents\PowerShell\profile.ps1

---

Stage-by-Stage Execution

Stage 00 (Optional): Download Satellite Imagery

Purpose: Fetch 4-band GeoTIFFs from Planet API

When to run:

• When you need fresh data (weekly or on-demand)
• Not needed if satellite TIFFs already in merged_tif/ directory

Command:

cd python_app

# Download for a specific date
python 00_download_8band_pu_optimized.py angata --date 2026-02-19

# Or use batch download for multiple dates
python download_planet_missing_dates.py --start 2025-12-24 --end 2026-02-19 --project angata

cd ..

Expected Output:

laravel_app/storage/app/angata/merged_tif/2026-02-19.tif   (~200 MB)

Troubleshooting:

• Auth error: Check Planet API credentials in environment
• Date missing: Download script skips dates already saved
• Cloud cover: Script applies UDM1 cloud mask; may skip high-cloud days

---

Stage 10: Create Per-Field Tiles

Purpose: Split merged farm TIFF into individual field files

Prerequisite:

• merged_tif/2026-02-19.tif must exist
• Data/pivot.geojson must exist and be valid

Command:

$R_EXE = "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe"
$PROJECT = "angata"
$END_DATE = "2026-02-19"
$OFFSET = 7

& $R_EXE r_app/10_create_per_field_tiffs.R $PROJECT $END_DATE $OFFSET

Parameters:

• PROJECT: Project name (angata, chemba, aura, etc.)
• END_DATE: Date in YYYY-MM-DD format
• OFFSET: Days to look back (7 = last week)

Expected Output:

laravel_app/storage/app/angata/field_tiles/
├── Field_001/
│   ├── 2026-02-12.tif  (4-band)
│   ├── 2026-02-13.tif
│   └── 2026-02-19.tif
├── Field_002/
│   └── ...
└── ...

Console Output:

[1] "Loading parameters..."
[1] "Processing dates 2026-02-12 to 2026-02-19"
[1] "Field_001: splitting tile..."
[1] "Field_002: splitting tile..."
[1] "Stage 10 completed successfully"

---

Stage 20: Extract Canopy Index (CI)

Purpose: Calculate CI per field and per pixel from 4-band TIFFs

Prerequisite:

• Stage 10 completed (field_tiles/ populated)
• Data/pivot.geojson exists

Command:

& $R_EXE r_app/20_ci_extraction_per_field.R $PROJECT $END_DATE $OFFSET

Expected Output:

laravel_app/storage/app/angata/field_tiles_CI/
├── Field_001/
│   ├── 2026-02-12.tif  (5-band: R,G,B,NIR,CI)
│   └── 2026-02-19.tif

Data/extracted_ci/
├── daily_vals/
│   └── Field_001/
│       ├── 2026-02-12.rds
│       └── 2026-02-19.rds
└── cumulative_vals/
    └── combined_CI_data.rds   (WIDE format: fields × dates)

Console Output:

[1] "Computing CI index..."
[1] "Field_001: CI = 1.23 (mean), 0.45 (sd)"
[1] "Field_002: CI = 1.19 (mean), 0.38 (sd)"
[1] "Saving combined_CI_data.rds..."
[1] "Stage 20 completed successfully"

---

Stage 30: Interpolate Growth Model

Purpose: Smooth CI time series and fill gaps (handles clouds)

Prerequisite:

• Stage 20 completed (combined_CI_data.rds exists)
• Data/harvest.xlsx recommended (required for cane_supply projects)

Command:

# No date/offset parameters for Stage 30 — it processes all available CI data
& $R_EXE r_app/30_interpolate_growth_model.R $PROJECT

Expected Output:

Data/extracted_ci/cumulative_vals/
└── All_pivots_Cumulative_CI_quadrant_year_v2.rds
    # (long format: field × date × interpolated_ci × daily_change × cumulative_ci)

Console Output:

[1] "Loading combined CI data..."
[1] "Applying LOESS interpolation (span=0.3)..."
[1] "Season 2025-10 → 2026-03: Field_001 interpolated 42 dates, filled 3 gaps"
[1] "Saving interpolated growth model..."
[1] "Stage 30 completed successfully"

---

Stage 40: Create Weekly Mosaics

Purpose: Aggregate daily per-field TIFFs into weekly MAX-composites

Prerequisite:

• Stage 20 completed (field_tiles_CI/ populated)

Command:

# Process mosaics for END_DATE week, looking back OFFSET days
& $R_EXE r_app/40_mosaic_creation_per_field.R $END_DATE $OFFSET $PROJECT

Expected Output:

laravel_app/storage/app/angata/weekly_mosaic/
├── Field_001/
│   ├── week_07_2026.tif  (5-band, MAX-aggregated for ISO week 7)
│   ├── week_06_2026.tif
│   └── ...
├── Field_002/
│   └── ...

Console Output:

[1] "Computing weekly mosaics for week 07 (2026-02-16 to 2026-02-22)..."
[1] "Field_001: aggregating 7 daily TIFFs..."
[1] "Field_002: aggregating 7 daily TIFFs..."
[1] "Saving weekly_mosaic/Field_001/week_07_2026.tif..."
[1] "Stage 40 completed successfully"

---

Stage 80: Calculate KPIs

Purpose: Compute field-level KPIs from weekly mosaics (client-type dependent)

Prerequisite:

• Stage 40 completed (weekly_mosaic/ populated)
• Stage 30 completed (growth model data for trends)
• Data/pivot.geojson exists
• Data/harvest.xlsx exists (required for cane_supply)

Command:

# KPI calculation (client type determined from PROJECT name in parameters_project.R)
& $R_EXE r_app/80_calculate_kpis.R $END_DATE $PROJECT $OFFSET

Expected Output:

laravel_app/storage/app/angata/reports/
├── angata_field_analysis_week07_2026.xlsx  (21-column spreadsheet)
└── kpis/
    └── angata_kpi_summary_tables_week07.rds

Console Output (agronomic_support type):

[1] "Client type: agronomic_support"
[1] "Loading weekly mosaic data..."
[1] "Computing uniformity KPI (CV)..."
[1] "Computing area change KPI..."
[1] "Computing TCH forecast..."
[1] "Computing growth decline..."
[1] "Computing weed presence (Moran's I)..."
[1] "Computing gap fill quality..."
[1] "Saving kpi_summary_tables_week07.rds..."
[1] "Stage 80 completed successfully"

---

Stages 90/91: Generate Word Reports

Purpose: Render RMarkdown to Microsoft Word (client-type specific)

Prerequisite:

• Stage 80 completed (KPI summary RDS + Excel exist)
• Data/pivot.geojson exists
• Data/extracted_ci/cumulative_vals/combined_CI_data.rds exists

Command for Agronomic Support (Aura, Chemba, etc.):

& $R_EXE -e `
  "rmarkdown::render('r_app/90_CI_report_with_kpis_agronomic_support.Rmd', `
   params=list(data_dir='$PROJECT', report_date=as.Date('$END_DATE')), `
   output_file='SmartCane_Report_agronomic_support_${PROJECT}_week07_2026.docx', `
   output_dir='laravel_app/storage/app/$PROJECT/reports')"

Command for Cane Supply (Angata):

& $R_EXE -e `
  "rmarkdown::render('r_app/91_CI_report_with_kpis_cane_supply.Rmd', `
   params=list(data_dir='$PROJECT', report_date=as.Date('$END_DATE')), `
   output_file='SmartCane_Report_cane_supply_${PROJECT}_week07_2026.docx', `
   output_dir='laravel_app/storage/app/$PROJECT/reports')"

Expected Output:

laravel_app/storage/app/angata/reports/
└── SmartCane_Report_cane_supply_angata_week07_2026.docx  (Word file with tables, charts, maps)

Console Output:

[1] "Rendering RMarkdown..."
[1] "Loading KPI summary data..."
[1] "Loading weekly mosaics..."
[1] "Creating plots..."
[1] "Rendering Word document..."
[1] "Output: laravel_app/storage/app/angata/reports/SmartCane_Report_*.docx"

---

Complete Pipeline: Single Command Sequence

One-Liner Scripts (PowerShell)

Setup Variables (run once per session):

$R_EXE = "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe"
$PROJECT = "angata"
$END_DATE = "2026-02-19"
$OFFSET = 7

Full Pipeline (if all data already downloaded):

Write-Host "Starting SmartCane pipeline for $PROJECT on $END_DATE..."

# Stage 10
Write-Host "[Stage 10] Creating field tiles..."
& $R_EXE r_app/10_create_per_field_tiffs.R $PROJECT $END_DATE $OFFSET

# Stage 20
Write-Host "[Stage 20] Extracting CI..."
& $R_EXE r_app/20_ci_extraction_per_field.R $PROJECT $END_DATE $OFFSET

# Stage 30
Write-Host "[Stage 30] Interpolating growth model..."
& $R_EXE r_app/30_interpolate_growth_model.R $PROJECT

# Stage 40
Write-Host "[Stage 40] Creating weekly mosaics..."
& $R_EXE r_app/40_mosaic_creation_per_field.R $END_DATE $OFFSET $PROJECT

# Stage 80
Write-Host "[Stage 80] Calculating KPIs..."
& $R_EXE r_app/80_calculate_kpis.R $END_DATE $PROJECT $OFFSET

# Stage 90/91 (client-type dependent)
Write-Host "[Stage 90/91] Rendering report..."
$CLIENT_TYPE = "cane_supply"  # Determine from parameters_project.R
if ($CLIENT_TYPE -eq "agronomic_support") {
    $TEMPLATE = "r_app/90_CI_report_with_kpis_agronomic_support.Rmd"
} else {
    $TEMPLATE = "r_app/91_CI_report_with_kpis_cane_supply.Rmd"
}

& $R_EXE -e `
  "rmarkdown::render('$TEMPLATE', `
   params=list(data_dir='$PROJECT', report_date=as.Date('$END_DATE')), `
   output_file='SmartCane_Report_${PROJECT}_week07_2026.docx', `
   output_dir='laravel_app/storage/app/$PROJECT/reports')"

Write-Host "Pipeline completed! Report: laravel_app/storage/app/$PROJECT/reports/"

Batch Processing Multiple Weeks

Use Batch Runner (R script that loops weeks):

# Aura batch processing (weeks 49–8, Dec 3 2025 - Feb 4 2026)
& $R_EXE -e "source('r_app/batch_pipeline_aura.R')"

# Manually loop custom date range
$startDate = [DateTime]::ParseExact("2026-01-28", "yyyy-MM-dd", $null)
$endDate = [DateTime]::ParseExact("2026-02-19", "yyyy-MM-dd", $null)
$current = $startDate

while ($current -le $endDate) {
    $dateStr = $current.ToString("yyyy-MM-dd")
    Write-Host "Processing week of $dateStr..."
    
    & $R_EXE r_app/40_mosaic_creation_per_field.R $dateStr 7 "angata"
    & $R_EXE r_app/80_calculate_kpis.R $dateStr "angata" 7
    
    $current = $current.AddDays(7)
}

---

Configuration: parameters_project.R

Location: r_app/parameters_project.R

This file defines global settings used by all stages.

# ============================================================================
# SmartCane Project Configuration
# ============================================================================

# Project settings
PROJECT <- Sys.getenv("PROJECT")  # Set by calling script or manually
if (PROJECT == "") {
    PROJECT <- "angata"  # Default project
}

# Client type mapping
CLIENT_TYPE_MAP <- list(
  "angata"     = "cane_supply",
  "chemba"     = "agronomic_support",
  "xinavane"   = "agronomic_support",
  "esa"        = "agronomic_support",
  "simba"      = "agronomic_support",
  "aura"       = "agronomic_support"
)

CLIENT_TYPE <- CLIENT_TYPE_MAP[[PROJECT]]

# Data directory (Laravel storage)
data_dir <- file.path(
    dirname(getwd()),  # Up one level from r_app
    "laravel_app/storage/app",
    PROJECT
)

# Key file paths
pivot_path <- file.path(data_dir, "Data", "pivot.geojson")
harvest_path <- file.path(data_dir, "Data", "harvest.xlsx")
merged_tif_dir <- file.path(data_dir, "merged_tif")
field_tiles_dir <- file.path(data_dir, "field_tiles")
field_tiles_ci_dir <- file.path(data_dir, "field_tiles_CI")
weekly_mosaic_dir <- file.path(data_dir, "weekly_mosaic")

# KPI thresholds (customizable)
CI_THRESHOLD <- 1.0
CV_GOOD <- 0.15
CV_EXCELLENT <- 0.08
CV_POOR <- 0.25

# Print configuration summary
cat("\n=== SmartCane Configuration ===\n")
cat("Project:", PROJECT, "\n")
cat("Client Type:", CLIENT_TYPE, "\n")
cat("Data Directory:", data_dir, "\n\n")

How to Use:

1. All scripts start with source("parameters_project.R")
2. Use global variables: PROJECT, CLIENT_TYPE, data_dir, etc.
3. To change project: Edit this file OR set PROJECT env var before running

---

Troubleshooting Common Issues

Issue: "File not found: combined_CI_data.rds"

Cause: Stage 20 not completed.

Solution:

# Run Stage 20 again with correct date range
& $R_EXE r_app/20_ci_extraction_per_field.R angata 2026-02-19 7

Issue: "Error in rmarkdown::render()"

Cause: RMarkdown template not found or missing dependencies.

Solution:

# Check template file exists
Test-Path "r_app/90_CI_report_with_kpis_agronomic_support.Rmd"

# Reinstall R packages
& $R_EXE r_app/package_manager.R

Issue: "GDAL error: Cannot open file"

Cause: Incorrect pivot.geojson path or file doesn't exist.

Solution:

# Check pivot.geojson exists
Test-Path "laravel_app/storage/app/angata/Data/pivot.geojson"

# Verify path in parameters_project.R

Issue: Python download fails with "Cloud cover too high"

Cause: Planet API filtering out days with >90% clouds.

Solution:

# Check available dates in merged_tif directory
Get-ChildItem laravel_app/storage/app/angata/merged_tif/

# Or edit Python script to use permissive cloud threshold
# Line: cloud_cover_threshold = 0.95  # 95% clouds allowed

---

Development Workflow Best Practices

1. Testing Single Stage in Isolation

# Test Stage 20 without running full pipeline
$PROJECT = "angata"
$END_DATE = "2026-02-19"
$OFFSET = 7

# Prerequisite: Stage 10 must be done, or manually create field_tiles/
& $R_EXE r_app/20_ci_extraction_per_field.R $PROJECT $END_DATE $OFFSET

# Review output
Get-ChildItem laravel_app/storage/app/$PROJECT/field_tiles_CI/

# View CI values
$data <- readRDS("laravel_app/storage/app/$PROJECT/Data/extracted_ci/cumulative_vals/combined_CI_data.rds")
head(data)

2. Debugging RMarkdown

# Render with verbose output
& $R_EXE -e `
  "rmarkdown::render('r_app/90_CI_report_with_kpis_agronomic_support.Rmd', `
   params=list(data_dir='aura', report_date=as.Date('2026-02-19')), `
   knit_root_dir=getwd(), clean=FALSE)"

# Check intermediate files
Get-ChildItem r_app/ -Filter "*_files" -Directory

3. Using RStudio for Interactive Development

# Open project in RStudio
# File > Open Project > r_app/r_app.Rproj

# Then in RStudio console:
# - source("parameters_project.R")
# - source("20_ci_extraction_per_field.R")  # Run script line by line
# - debug(extract_ci_per_field)  # Set breakpoints

---

Next Steps

• See ARCHITECTURE_DATA_FLOW.md for understanding pipeline flow
• See CLIENT_TYPE_ARCHITECTURE.md for client-specific KPI differences
• See SOBIT_DEPLOYMENT.md for production server alternative
• See ARCHITECTURE_INTEGRATION_GUIDE.md for choosing execution model