# SOBIT Linux Server Deployment Architecture

This document explains how SmartCane runs on SOBIT (the production Linux server) through Laravel web interface, contrasting with manual dev laptop execution.

## High-Level SOBIT Architecture

SOBIT hosts a Laravel-based web application that orchestrates Python and R pipeline execution through a **job queue system**. When a user clicks a button in the web UI, a job is created, pushed to a queue, and workers execute the actual Python/R scripts.

```mermaid
%% SOBIT Deployment Architecture
flowchart TD
    A["Web UI<br/>Laravel Dashboard<br/>Project Controller"] 
    
    B["User Action<br/>e.g., Download Data,<br/>Generate Report"]
    
    C["Job Dispatch<br/>ProjectDownloadTiffJob,<br/>ProjectMosaicJob, etc."]
    
    D["Queue Worker<br/>Executes Job Handler"]
    
    E["Shell Script Wrapper<br/>runpython.sh<br/>20_ci_extraction.sh<br/>90_kpi_report.sh"]
    
    F["Python/R Executable<br/>00_download_8band_pu_optimized.py<br/>20_ci_extraction_per_field.R<br/>rmarkdown::render"]
    
    G["Output Files<br/>laravel_app/storage/app/{PROJECT}/"]
    
    H["Next Job Dispatch<br/>job chaining"]
    
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H -.-> D
```

---

## Laravel Job Classes & Queue Flow

SmartCane uses Laravel's **queue system** to manage asynchronous task execution. Each stage has a corresponding Job class.

### Job Class Hierarchy

| Job Class | Shell Script | R/Python Script | Input File | Output File | Next Stage |
|-----------|-------------|-----------------|------------|------------|-----------|
| **ProjectDownloadTiffJob** | `runpython.sh` | `00_download_8band_pu_optimized.py` | date param | `merged_tif/{DATE}.tif` | ProjectCreateFieldTilesJob |
| **ProjectCreateFieldTilesJob** | (direct Rscript) | `10_create_per_field_tiffs.R` | `merged_tif/{DATE}.tif` | `field_tiles/{FIELD}/{DATE}.tif` | ProjectCIExtractionJob |
| **ProjectCIExtractionJob** | `20_ci_extraction.sh` | `20_ci_extraction_per_field.R` | `field_tiles/...` | `combined_CI_data.rds` | ProjectGrowthModelJob |
| **ProjectGrowthModelJob** | `30_growth_model.sh` | `30_interpolate_growth_model.R` | `combined_CI_data.rds` | `All_pivots_...rds` | ProjectMosaicGeneratorJob |
| **ProjectMosaicGeneratorJob** | `40_mosaic_creation.sh` | `40_mosaic_creation_per_field.R` | `field_tiles_CI/...` | `weekly_mosaic/{FIELD}/week_*.tif` | ProjectKPICalculationJob |
| **ProjectKPICalculationJob** | `80_calculate_kpis.sh` | `80_calculate_kpis.R` | `weekly_mosaic/...` | `field_analysis_*.xlsx` + RDS | ProjectReportGeneratorJob |
| **ProjectReportGeneratorJob** | `90_kpi_report.sh` | `90_*.Rmd` or `91_*.Rmd` render | Excel + RDS | `SmartCane_Report_*.docx` | ✅ Complete |

---

## Execution Flow: From Web UI to Output

### Scenario: User Clicks "Download Latest Data" for Project "Angata"

#### Step 1: Web UI Dispatch

**File**: `laravel_app/app/Http/Controllers/ProjectController.php`

```php
public function downloadData(Request $request, $projectName)
{
    $project = Project::where('name', $projectName)->firstOrFail();
    
    // Dispatch job to queue
    dispatch(new ProjectDownloadTiffJob($project, $request->date));
    
    return response()->json(['status' => 'Job queued']);
}
```

#### Step 2: Job Queue Entry

**File**: `laravel_app/app/Jobs/ProjectDownloadTiffJob.php`

```php
class ProjectDownloadTiffJob implements ShouldQueue
{
    public $project;
    public $date;
    
    public function __construct(Project $project, $date = null)
    {
        $this->project = $project;
        $this->date = $date ?? Carbon::now()->toDateString();
    }
    
    public function handle()
    {
        // Execute shell script wrapper
        $command = "bash " . base_path() . "/runpython.sh " .
                   "--date=" . $this->date . " " .
                   "--project_dir=" . $this->project->directory;
        
        $process = new Process(explode(' ', $command));
        $process->run();
        
        if ($process->isSuccessful()) {
            // Dispatch next job
            dispatch(new ProjectCreateFieldTilesJob($this->project, $this->date));
        } else {
            // Log error and notify user
            Log::error("Download failed for {$this->project->name}: " . $process->getErrorOutput());
        }
    }
}
```

#### Step 3: Shell Script Execution (runpython.sh)

**File**: `laravel_app/../runpython.sh`

```bash
#!/bin/bash
# Wrapper for Python download script

DATE=${1#--date=}
PROJECT_DIR=${2#--project_dir=}

cd /path/to/SmartCane_code/python_app

# Run Python with project-specific environment
python 00_download_8band_pu_optimized.py $PROJECT_DIR --date $DATE

if [ $? -eq 0 ]; then
    echo "Download successful for $PROJECT_DIR on $DATE"
    exit 0
else
    echo "Download failed for $PROJECT_DIR on $DATE"
    exit 1
fi
```

#### Step 4: Python Download (Stage 00)

**File**: `python_app/00_download_8band_pu_optimized.py`

```python
import sys
from datetime import datetime

project_dir = sys.argv[1]  # "angata"
date_str = sys.argv[2] if len(sys.argv) > 2 else datetime.now().strftime('%Y-%m-%d')

# Authenticate with Planet API
auth = get_planet_credentials()
bbox = get_project_bbox(project_dir)  # From database or config

# Download 4-band TIFF
tiff_path = f"laravel_app/storage/app/{project_dir}/merged_tif/{date_str}.tif"
download_and_save_tiff(auth, bbox, date_str, tiff_path)

print(f"Downloaded to {tiff_path}")
```

#### Step 5: Job Chaining (Automatic)

**Back in ProjectDownloadTiffJob.php**:

```php
if ($process->isSuccessful()) {
    // Dispatch next job in pipeline
    dispatch(new ProjectCreateFieldTilesJob($this->project, $this->date));
}
```

This triggers `ProjectCreateFieldTilesJob`, which calls the next shell script, and so on.

---

## Shell Script Wrappers: Design & Responsibility

SOBIT wrapper scripts ensure Python and R scripts run with correct environment variables and working directory.

### Root-Level Shell Scripts

**Location**: `c:\Users\timon\Documents\SmartCane_code\` (root)

| Script | Purpose | Calls | Environment Setup |
|--------|---------|-------|-------------------|
| `10_planet_download.sh` | Stage 10 wrapper | `Rscript 10_create_per_field_tiffs.R` | Sets R_LIBS, PYTHONPATH |
| `20_ci_extraction.sh` | Stage 20 wrapper | `Rscript 20_ci_extraction_per_field.R` | R environment + data paths |
| `30_growth_model.sh` | Stage 30 wrapper | `Rscript 30_interpolate_growth_model.R` | Growth model data path |
| `40_mosaic_creation.sh` | Stage 40 wrapper | `Rscript 40_mosaic_creation_per_field.R` | Sentinel config for mosaics |
| `80_calculate_kpis.sh` | Stage 80 wrapper | `Rscript 80_calculate_kpis.R` | KPI utility loading |
| `90_kpi_report.sh` | Stage 90/91 wrapper | `rmarkdown::render()` via Rscript | RMarkdown dependencies |
| `runpython.sh` | Python wrapper | `python 00_download_8band_pu_optimized.py` | Python venv activation |

### Example: `20_ci_extraction.sh`

```bash
#!/bin/bash
# CI Extraction wrapper for SOBIT

set -e  # Exit on error

# Load environment
export R_LIBS="/opt/R/library"
export PYTHONPATH="/opt/python/lib/python3.9/site-packages:$PYTHONPATH"
PROJECT_DIR=$1
END_DATE=$2
OFFSET=${3:-7}

# Change to code directory
cd /var/www/smartcane/code

# Execute R script
Rscript r_app/20_ci_extraction_per_field.R "$PROJECT_DIR" "$END_DATE" "$OFFSET"

if [ $? -eq 0 ]; then
    echo "[$(date)] CI extraction completed for $PROJECT_DIR"
    # Log success to Laravel job tracking
    echo "Success" > "laravel_app/storage/logs/${PROJECT_DIR}_stage20_${END_DATE}.log"
else
    echo "[$(date)] CI extraction FAILED for $PROJECT_DIR"
    exit 1
fi
```

### Example: `90_kpi_report.sh`

```bash
#!/bin/bash
# RMarkdown report rendering wrapper

set -e

PROJECT_DIR=$1
REPORT_DATE=$2
CLIENT_TYPE=$3  # "agronomic_support" or "cane_supply"

cd /var/www/smartcane/code

# Determine which RMarkdown template to use
if [ "$CLIENT_TYPE" = "agronomic_support" ]; then
    REPORT_TEMPLATE="r_app/90_CI_report_with_kpis_agronomic_support.Rmd"
else
    REPORT_TEMPLATE="r_app/91_CI_report_with_kpis_cane_supply.Rmd"
fi

# Render RMarkdown to Word
Rscript -e "
  rmarkdown::render(
    '$REPORT_TEMPLATE',
    params = list(
      data_dir = '$PROJECT_DIR',
      report_date = as.Date('$REPORT_DATE')
    ),
    output_file = 'SmartCane_Report_${CLIENT_TYPE}_${PROJECT_DIR}_${REPORT_DATE}.docx',
    output_dir = 'laravel_app/storage/app/$PROJECT_DIR/reports'
  )
"

echo "[$(date)] Report generation completed for $PROJECT_DIR"
```

---

## Data Storage on SOBIT

All data is stored in Laravel's standard storage directory with project-based subdirectories.

### Storage Structure

```
laravel_app/storage/app/
├── angata/                          # Cane Supply project
│   ├── merged_tif/                  # Stage 00 Python output
│   │   ├── 2026-02-12.tif
│   │   └── 2026-02-19.tif
│   ├── field_tiles/                 # Stage 10 output
│   ├── field_tiles_CI/              # Stage 20 output
│   ├── Data/
│   │   ├── pivot.geojson            # Field boundaries (input)
│   │   ├── harvest.xlsx             # Harvest dates (input)
│   │   ├── extracted_ci/            # Stage 20 CI data
│   │   └── growth_model_interpolated/  # Stage 30 data
│   ├── weekly_mosaic/               # Stage 40 output
│   └── reports/                     # Stages 80/90/91 output
│       ├── SmartCane_Report_*.docx
│       ├── angata_field_analysis_week*.xlsx
│       └── kpis/
│           └── *.rds
│
├── aura/                            # Agronomic Support project
│   └── (same structure as angata)
│
├── chemba/
├── xinavane/
├── esa/
└── simba/
```

### Permissions Model

- **Web server user** (www-data on Linux): Can read/write to all storage subdirectories
- **Laravel artisan commands**: Have full access via `Storage::disk('local')`
- **Job queue workers**: Execute as www-data, access via storage symlink

---

## Job Queue Configuration

**File**: `laravel_app/config/queue.php`

```php
'default' => env('QUEUE_CONNECTION', 'database'),

'connections' => [
    'database' => [
        'driver' => 'database',
        'table' => 'jobs',
        'queue' => 'default',
        'retry_after' => 1800,  // 30 min timeout per job
        'after_commit' => false,
    ],
    
    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => env('REDIS_QUEUE', 'default'),
        'after_commit' => false,
    ],
],
```

### Starting Queue Worker

On SOBIT, the queue worker runs in the background:

```bash
# Manual start (for debugging)
php artisan queue:work --queue=default --timeout=1800

# Production (supervisor-managed)
# Supervisor config: /etc/supervisor/conf.d/smartcane-worker.conf
[program:smartcane-queue-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/smartcane/artisan queue:work --queue=default --timeout=1800
autostart=true
autorestart=true
numprocs=2
redirect_stderr=true
stdout_logfile=/var/log/smartcane-queue.log
```

---

## Error Handling & Retries

### Job Failure Scenario

If a shell script returns non-zero exit code:

```php
// In job handle() method
if (!$process->isSuccessful()) {
    Log::error("Job failed: " . $process->getErrorOutput());
    
    // Laravel automatically retries if $tries > 1
    // Configurable in job class:
    // public $tries = 3;
    // public $retryAfter = 300;  // 5 minutes between retries
}
```

### Monitoring & Alerts

- **Failed Jobs Table**: `jobs_failed` table in Laravel database
- **Logs**: `storage/logs/laravel.log` + individual stage logs
- **User Notification**: Job status visible in web UI; email alerts can be configured

---

## Deployment vs Dev Laptop: Key Differences

| Aspect | **SOBIT (Production)** | **Dev Laptop** |
|--------|----------------------|----------------|
| **Execution Model** | Async job queue | Synchronous PowerShell |
| **User Interaction** | Web UI clicks → jobs | Manual script calls |
| **Data Location** | `laravel_app/storage/app/{PROJECT}/` | Same (shared Laravel directory) |
| **Error Handling** | Job retries, logs in database | Terminal output only |
| **Parallelization** | Multiple queue workers | Single sequential execution |
| **Monitoring** | Web dashboard + Laravel logs | Console output only |
| **Environment Setup** | Bash scripts set env vars | Manual R/Python environment |
| **Scheduling** | Can use Laravel scheduler for automated runs | Manual cron or batch scripts |

---

## Running Full Pipeline on SOBIT via Web UI

### User Workflow

1. **Navigate** to ProjectController dashboard
   ```
   http://sobit-server/projects/angata
   ```

2. **Click** "Download Latest Data" button
   - ProjectDownloadTiffJob queued with current date
   - Web UI shows "Job submitted"

3. **Queue Worker** (background process) executes jobs in sequence
   - Downloads TIFF (Stage 00)
   - Dispatches Stage 10 job
   - Creates field tiles
   - Dispatches Stage 20 job
   - (etc. through Stage 91)

4. **Monitor** progress via Dashboard
   - Job history tab shows completed jobs
   - Report links appear when Stage 91 completes
   - Download Word/Excel from reports section

### Command-Line Submission (Alternative)

Developer can manually trigger jobs via Laravel artisan:

```bash
# SSH into SOBIT
ssh user@sobit-server

# Manually dispatch job
php artisan smartcane:process-pipeline angata --date=2026-02-19 --async

# Or using job dispatch directly
php artisan queue:work --queue=default
```

---

## Troubleshooting SOBIT Deployment

### Issue: Job Stuck in Queue

```bash
# Check job queue depth
SELECT COUNT(*) FROM jobs WHERE queue = 'default';

# Retry failed jobs
php artisan queue:retry --all

# Clear old jobs
php artisan queue:clear
```

### Issue: Shell Script Can't Find R/Python

**Cause**: Environment variables not set in shell wrapper.

**Fix**: Add to shell script:
```bash
export PATH="/opt/R/bin:/opt/python/bin:$PATH"
export R_HOME="/opt/R"
source /opt/python/venv/bin/activate
```

### Issue: Permission Denied on Storage Files

**Cause**: Files created by web server, permission mismatch.

**Fix**:
```bash
sudo chown -R www-data:www-data laravel_app/storage/app/*
sudo chmod -R 755 laravel_app/storage/app/
```

---

## Next Steps

- See [ARCHITECTURE_DATA_FLOW.md](ARCHITECTURE_DATA_FLOW.md) for stage details
- See [CLIENT_TYPE_ARCHITECTURE.md](CLIENT_TYPE_ARCHITECTURE.md) for how jobs route based on project type
- See [DEV_LAPTOP_EXECUTION.md](DEV_LAPTOP_EXECUTION.md) for alternative manual execution model
- See [ARCHITECTURE_INTEGRATION_GUIDE.md](ARCHITECTURE_INTEGRATION_GUIDE.md) for choosing SOBIT vs dev laptop