Timon/SmartCane
1
1
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
SmartCane/webapps/docs/SOBIT_DEPLOYMENT.md

14 KiB
Raw Blame History

SOBIT Linux Server Deployment Architecture

This document explains how SmartCane runs on SOBIT (the production Linux server) through Laravel web interface, contrasting with manual dev laptop execution.

High-Level SOBIT Architecture

SOBIT hosts a Laravel-based web application that orchestrates Python and R pipeline execution through a job queue system. When a user clicks a button in the web UI, a job is created, pushed to a queue, and workers execute the actual Python/R scripts.

%% SOBIT Deployment Architecture
flowchart TD
    A["Web UI<br/>Laravel Dashboard<br/>Project Controller"] 
    
    B["User Action<br/>e.g., Download Data,<br/>Generate Report"]
    
    C["Job Dispatch<br/>ProjectDownloadTiffJob,<br/>ProjectMosaicJob, etc."]
    
    D["Queue Worker<br/>Executes Job Handler"]
    
    E["Shell Script Wrapper<br/>runpython.sh<br/>20_ci_extraction.sh<br/>90_kpi_report.sh"]
    
    F["Python/R Executable<br/>00_download_8band_pu_optimized.py<br/>20_ci_extraction_per_field.R<br/>rmarkdown::render"]
    
    G["Output Files<br/>laravel_app/storage/app/{PROJECT}/"]
    
    H["Next Job Dispatch<br/>job chaining"]
    
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H -.-> D

Laravel Job Classes & Queue Flow

SmartCane uses Laravel's queue system to manage asynchronous task execution. Each stage has a corresponding Job class.

Job Class Hierarchy

Job Class Shell Script R/Python Script Input File Output File Next Stage
ProjectDownloadTiffJob runpython.sh 00_download_8band_pu_optimized.py date param merged_tif/{DATE}.tif ProjectCreateFieldTilesJob
ProjectCreateFieldTilesJob (direct Rscript) 10_create_per_field_tiffs.R merged_tif/{DATE}.tif field_tiles/{FIELD}/{DATE}.tif ProjectCIExtractionJob
ProjectCIExtractionJob 20_ci_extraction.sh 20_ci_extraction_per_field.R field_tiles/... combined_CI_data.rds ProjectGrowthModelJob
ProjectGrowthModelJob 30_growth_model.sh 30_interpolate_growth_model.R combined_CI_data.rds All_pivots_...rds ProjectMosaicGeneratorJob
ProjectMosaicGeneratorJob 40_mosaic_creation.sh 40_mosaic_creation_per_field.R field_tiles_CI/... weekly_mosaic/{FIELD}/week_*.tif ProjectKPICalculationJob
ProjectKPICalculationJob 80_calculate_kpis.sh 80_calculate_kpis.R weekly_mosaic/... field_analysis_*.xlsx + RDS ProjectReportGeneratorJob
ProjectReportGeneratorJob 90_kpi_report.sh 90_*.Rmd or 91_*.Rmd render Excel + RDS SmartCane_Report_*.docx ✅ Complete

Execution Flow: From Web UI to Output

Scenario: User Clicks "Download Latest Data" for Project "Angata"

Step 1: Web UI Dispatch

File: laravel_app/app/Http/Controllers/ProjectController.php

public function downloadData(Request $request, $projectName)
{
    $project = Project::where('name', $projectName)->firstOrFail();
    
    // Dispatch job to queue
    dispatch(new ProjectDownloadTiffJob($project, $request->date));
    
    return response()->json(['status' => 'Job queued']);
}

Step 2: Job Queue Entry

File: laravel_app/app/Jobs/ProjectDownloadTiffJob.php

class ProjectDownloadTiffJob implements ShouldQueue
{
    public $project;
    public $date;
    
    public function __construct(Project $project, $date = null)
    {
        $this->project = $project;
        $this->date = $date ?? Carbon::now()->toDateString();
    }
    
    public function handle()
    {
        // Execute shell script wrapper
        $command = "bash " . base_path() . "/runpython.sh " .
                   "--date=" . $this->date . " " .
                   "--project_dir=" . $this->project->directory;
        
        $process = new Process(explode(' ', $command));
        $process->run();
        
        if ($process->isSuccessful()) {
            // Dispatch next job
            dispatch(new ProjectCreateFieldTilesJob($this->project, $this->date));
        } else {
            // Log error and notify user
            Log::error("Download failed for {$this->project->name}: " . $process->getErrorOutput());
        }
    }
}

Step 3: Shell Script Execution (runpython.sh)

File: laravel_app/../runpython.sh

#!/bin/bash
# Wrapper for Python download script

DATE=${1#--date=}
PROJECT_DIR=${2#--project_dir=}

cd /path/to/SmartCane_code/python_app

# Run Python with project-specific environment
python 00_download_8band_pu_optimized.py $PROJECT_DIR --date $DATE

if [ $? -eq 0 ]; then
    echo "Download successful for $PROJECT_DIR on $DATE"
    exit 0
else
    echo "Download failed for $PROJECT_DIR on $DATE"
    exit 1
fi

Step 4: Python Download (Stage 00)

File: python_app/00_download_8band_pu_optimized.py

import sys
from datetime import datetime

project_dir = sys.argv[1]  # "angata"
date_str = sys.argv[2] if len(sys.argv) > 2 else datetime.now().strftime('%Y-%m-%d')

# Authenticate with Planet API
auth = get_planet_credentials()
bbox = get_project_bbox(project_dir)  # From database or config

# Download 4-band TIFF
tiff_path = f"laravel_app/storage/app/{project_dir}/merged_tif/{date_str}.tif"
download_and_save_tiff(auth, bbox, date_str, tiff_path)

print(f"Downloaded to {tiff_path}")

Step 5: Job Chaining (Automatic)

Back in ProjectDownloadTiffJob.php:

if ($process->isSuccessful()) {
    // Dispatch next job in pipeline
    dispatch(new ProjectCreateFieldTilesJob($this->project, $this->date));
}

This triggers ProjectCreateFieldTilesJob, which calls the next shell script, and so on.

Shell Script Wrappers: Design & Responsibility

SOBIT wrapper scripts ensure Python and R scripts run with correct environment variables and working directory.

Root-Level Shell Scripts

Location: c:\Users\timon\Documents\SmartCane_code\ (root)

Script Purpose Calls Environment Setup
10_planet_download.sh Stage 10 wrapper Rscript 10_create_per_field_tiffs.R Sets R_LIBS, PYTHONPATH
20_ci_extraction.sh Stage 20 wrapper Rscript 20_ci_extraction_per_field.R R environment + data paths
30_growth_model.sh Stage 30 wrapper Rscript 30_interpolate_growth_model.R Growth model data path
40_mosaic_creation.sh Stage 40 wrapper Rscript 40_mosaic_creation_per_field.R Sentinel config for mosaics
80_calculate_kpis.sh Stage 80 wrapper Rscript 80_calculate_kpis.R KPI utility loading
90_kpi_report.sh Stage 90/91 wrapper rmarkdown::render() via Rscript RMarkdown dependencies
runpython.sh Python wrapper python 00_download_8band_pu_optimized.py Python venv activation

Example: 20_ci_extraction.sh

#!/bin/bash
# CI Extraction wrapper for SOBIT

set -e  # Exit on error

# Load environment
export R_LIBS="/opt/R/library"
export PYTHONPATH="/opt/python/lib/python3.9/site-packages:$PYTHONPATH"
PROJECT_DIR=$1
END_DATE=$2
OFFSET=${3:-7}

# Change to code directory
cd /var/www/smartcane/code

# Execute R script
Rscript r_app/20_ci_extraction_per_field.R "$PROJECT_DIR" "$END_DATE" "$OFFSET"

if [ $? -eq 0 ]; then
    echo "[$(date)] CI extraction completed for $PROJECT_DIR"
    # Log success to Laravel job tracking
    echo "Success" > "laravel_app/storage/logs/${PROJECT_DIR}_stage20_${END_DATE}.log"
else
    echo "[$(date)] CI extraction FAILED for $PROJECT_DIR"
    exit 1
fi

Example: 90_kpi_report.sh

#!/bin/bash
# RMarkdown report rendering wrapper

set -e

PROJECT_DIR=$1
REPORT_DATE=$2
CLIENT_TYPE=$3  # "agronomic_support" or "cane_supply"

cd /var/www/smartcane/code

# Determine which RMarkdown template to use
if [ "$CLIENT_TYPE" = "agronomic_support" ]; then
    REPORT_TEMPLATE="r_app/90_CI_report_with_kpis_agronomic_support.Rmd"
else
    REPORT_TEMPLATE="r_app/91_CI_report_with_kpis_cane_supply.Rmd"
fi

# Render RMarkdown to Word
Rscript -e "
  rmarkdown::render(
    '$REPORT_TEMPLATE',
    params = list(
      data_dir = '$PROJECT_DIR',
      report_date = as.Date('$REPORT_DATE')
    ),
    output_file = 'SmartCane_Report_${CLIENT_TYPE}_${PROJECT_DIR}_${REPORT_DATE}.docx',
    output_dir = 'laravel_app/storage/app/$PROJECT_DIR/reports'
  )
"

echo "[$(date)] Report generation completed for $PROJECT_DIR"

Data Storage on SOBIT

All data is stored in Laravel's standard storage directory with project-based subdirectories.

Storage Structure

laravel_app/storage/app/
├── angata/                          # Cane Supply project
│   ├── merged_tif/                  # Stage 00 Python output
│   │   ├── 2026-02-12.tif
│   │   └── 2026-02-19.tif
│   ├── field_tiles/                 # Stage 10 output
│   ├── field_tiles_CI/              # Stage 20 output
│   ├── Data/
│   │   ├── pivot.geojson            # Field boundaries (input)
│   │   ├── harvest.xlsx             # Harvest dates (input)
│   │   ├── extracted_ci/            # Stage 20 CI data
│   │   └── growth_model_interpolated/  # Stage 30 data
│   ├── weekly_mosaic/               # Stage 40 output
│   └── reports/                     # Stages 80/90/91 output
│       ├── SmartCane_Report_*.docx
│       ├── angata_field_analysis_week*.xlsx
│       └── kpis/
│           └── *.rds
│
├── aura/                            # Agronomic Support project
│   └── (same structure as angata)
│
├── chemba/
├── xinavane/
├── esa/
└── simba/

Permissions Model

  • Web server user (www-data on Linux): Can read/write to all storage subdirectories
  • Laravel artisan commands: Have full access via Storage::disk('local')
  • Job queue workers: Execute as www-data, access via storage symlink

Job Queue Configuration

File: laravel_app/config/queue.php

'default' => env('QUEUE_CONNECTION', 'database'),

'connections' => [
    'database' => [
        'driver' => 'database',
        'table' => 'jobs',
        'queue' => 'default',
        'retry_after' => 1800,  // 30 min timeout per job
        'after_commit' => false,
    ],
    
    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => env('REDIS_QUEUE', 'default'),
        'after_commit' => false,
    ],
],

Starting Queue Worker

On SOBIT, the queue worker runs in the background:

# Manual start (for debugging)
php artisan queue:work --queue=default --timeout=1800

# Production (supervisor-managed)
# Supervisor config: /etc/supervisor/conf.d/smartcane-worker.conf
[program:smartcane-queue-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/smartcane/artisan queue:work --queue=default --timeout=1800
autostart=true
autorestart=true
numprocs=2
redirect_stderr=true
stdout_logfile=/var/log/smartcane-queue.log

Error Handling & Retries

Job Failure Scenario

If a shell script returns non-zero exit code:

// In job handle() method
if (!$process->isSuccessful()) {
    Log::error("Job failed: " . $process->getErrorOutput());
    
    // Laravel automatically retries if $tries > 1
    // Configurable in job class:
    // public $tries = 3;
    // public $retryAfter = 300;  // 5 minutes between retries
}

Monitoring & Alerts

  • Failed Jobs Table: jobs_failed table in Laravel database
  • Logs: storage/logs/laravel.log + individual stage logs
  • User Notification: Job status visible in web UI; email alerts can be configured

Deployment vs Dev Laptop: Key Differences

Aspect SOBIT (Production) Dev Laptop
Execution Model Async job queue Synchronous PowerShell
User Interaction Web UI clicks → jobs Manual script calls
Data Location laravel_app/storage/app/{PROJECT}/ Same (shared Laravel directory)
Error Handling Job retries, logs in database Terminal output only
Parallelization Multiple queue workers Single sequential execution
Monitoring Web dashboard + Laravel logs Console output only
Environment Setup Bash scripts set env vars Manual R/Python environment
Scheduling Can use Laravel scheduler for automated runs Manual cron or batch scripts

Running Full Pipeline on SOBIT via Web UI

User Workflow

  1. Navigate to ProjectController dashboard

    http://sobit-server/projects/angata

  2. Click "Download Latest Data" button

    • ProjectDownloadTiffJob queued with current date
    • Web UI shows "Job submitted"

  3. Queue Worker (background process) executes jobs in sequence

    • Downloads TIFF (Stage 00)
    • Dispatches Stage 10 job
    • Creates field tiles
    • Dispatches Stage 20 job
    • (etc. through Stage 91)

  4. Monitor progress via Dashboard

    • Job history tab shows completed jobs
    • Report links appear when Stage 91 completes
    • Download Word/Excel from reports section

Command-Line Submission (Alternative)

Developer can manually trigger jobs via Laravel artisan:

# SSH into SOBIT
ssh user@sobit-server

# Manually dispatch job
php artisan smartcane:process-pipeline angata --date=2026-02-19 --async

# Or using job dispatch directly
php artisan queue:work --queue=default

Troubleshooting SOBIT Deployment

Issue: Job Stuck in Queue

# Check job queue depth
SELECT COUNT(*) FROM jobs WHERE queue = 'default';

# Retry failed jobs
php artisan queue:retry --all

# Clear old jobs
php artisan queue:clear

Issue: Shell Script Can't Find R/Python

Cause: Environment variables not set in shell wrapper.

Fix: Add to shell script:

export PATH="/opt/R/bin:/opt/python/bin:$PATH"
export R_HOME="/opt/R"
source /opt/python/venv/bin/activate

Issue: Permission Denied on Storage Files

Cause: Files created by web server, permission mismatch.

Fix:

sudo chown -R www-data:www-data laravel_app/storage/app/*
sudo chmod -R 755 laravel_app/storage/app/

Next Steps