# SmartCane Code-Improvements Merge Guide (Complete) **Target Repo**: https://bitbucket.org/sobitnl/smartcane/src/master/ **Source Branch**: `code-improvements` (Timon's experimental area) **Status**: Ready for merge (all changes, including optional enhancements) --- ## 📋 SCOPE: CORE + OPTIONAL FEATURES ### 🔴 CORE (Required for merge) - Database migration (client_type column) - File path changes (merged_final_tif → field_tiles_CI) - Laravel form/job/mailing updates - Shell script wrappers (5 files) - Python package files (2 files) ### 🟡 OPTIONAL (Post-merge enhancements) - Country-based project organization (MZ/UG/TZ folders) - Download scheduling (staggered 00:01 per-project) - Project search feature - Harvest prediction setup docs --- ## 🚀 FULL IMPLEMENTATION CHECKLIST ``` ═══════════════════════════════════════════════════════════════ PHASE 1: CORE MERGE (Required) ═══════════════════════════════════════════════════════════════ [1.1] Database Migration (10 min) ✓ Create: database/migrations/YYYY_MM_DD_add_client_type_to_projects_table.php ✓ Run: php artisan migrate [1.2] Laravel Model Changes (20 min) ✓ Edit: laravel_app/app/Models/Project.php - Add 'client_type' to $fillable - Update getMergedTiffList() — path change - Update startDownload() — path change - Update getTifsAsZip() — path change - Add getLatestKpiFile() method [1.3] Laravel Job Changes (15 min) ✓ Edit: laravel_app/app/Jobs/ProjectDownloadTiffJob.php - Change path in handleForDate() ✓ Edit: laravel_app/app/Jobs/ProjectMosiacGeneratorJob.php - Replace command array - Improve error handling [1.4] Laravel Forms (15 min) ✓ Edit: laravel_app/app/Livewire/Projects/ProjectManager.php - Add client_type to form ✓ Edit: Project form Blade template - Add @error('formData.client_type') {{ $message }} @enderror ``` --- ### STEP 7: Edit `laravel_app/app/Livewire/Forms/MailingForm.php` **In `saveAndSendMailing()` static method:** ```php public static function saveAndSendMailing($report, $subject, $message, $recipients) { if ($report->documentExists()) { $mailing = $report->project->mailings()->create([ 'subject' => $subject, 'message' => $message, 'report_id' => $report->id, ]); // Attach main report $mailing->attachments()->create([ 'name' => $report->name, 'path' => $report->path, ]); // Auto-attach KPI Excel for cane_supply projects if ($report->project->client_type === 'cane_supply') { $kpiFile = $report->project->getLatestKpiFile(); if ($kpiFile) { $mailing->attachments()->create([ 'name' => 'KPI Summary', 'path' => $kpiFile, ]); } } $mailing->recipients()->createMany($recipients); Mail::to($mailing->recipients()->pluck('email')->toArray()) ->send(new \App\Mail\ReportMailer($mailing, $report)); } else { self::sendReportNotFoundNotificationToAdmin($report); } } ``` --- ### STEP 8: CREATE 5 Shell Script Wrappers **File: `10_create_per_field_tiffs.sh`** ```bash #!/bin/bash # Wrapper for R script 10: Create per-field TIFFs # Usage: ./10_create_per_field_tiffs.sh --project=angata set -e PROJECT="" while [[ $# -gt 0 ]]; do case $1 in --project=*) PROJECT="${1#*=}" ;; --*) ;; # Ignore other args esac shift done [ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; } cd "$(dirname "$0")/r_app" Rscript -e "PROJECT='$PROJECT'; source('parameters_project.R'); source('10_create_per_field_tiffs.R')" ``` **File: `21_convert_ci_rds_to_csv.sh`** ```bash #!/bin/bash # Wrapper for R script 21: Convert CI RDS to CSV # Usage: ./21_convert_ci_rds_to_csv.sh --project=angata set -e PROJECT="" while [[ $# -gt 0 ]]; do case $1 in --project=*) PROJECT="${1#*=}" ;; --*) ;; esac shift done [ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; } cd "$(dirname "$0")/r_app" Rscript -e "PROJECT='$PROJECT'; source('parameters_project.R'); source('21_convert_ci_rds_to_csv.R')" ``` **File: `22_harvest_baseline_prediction.sh`** ```bash #!/bin/bash # Wrapper for Python script 22: Harvest baseline prediction # Usage: ./22_harvest_baseline_prediction.sh --project=angata set -e PROJECT="" while [[ $# -gt 0 ]]; do case $1 in --project=*) PROJECT="${1#*=}" ;; --*) ;; esac shift done [ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; } cd "$(dirname "$0")/python_app" if command -v conda &> /dev/null; then conda run -n pytorch_gpu python 22_harvest_baseline_prediction.py "$PROJECT" 2>&1 || \ conda run -n pytorch_cpu python 22_harvest_baseline_prediction.py "$PROJECT" 2>&1 else python 22_harvest_baseline_prediction.py "$PROJECT" fi ``` **File: `23_convert_harvest_format.sh`** ```bash #!/bin/bash # Wrapper for Python script 23: Convert harvest format # Usage: ./23_convert_harvest_format.sh --project=angata set -e PROJECT="" while [[ $# -gt 0 ]]; do case $1 in --project=*) PROJECT="${1#*=}" ;; --*) ;; esac shift done [ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; } cd "$(dirname "$0")/python_app" if command -v conda &> /dev/null; then conda run -n pytorch_gpu python 23_convert_harvest_format.py "$PROJECT" 2>&1 || \ conda run -n pytorch_cpu python 23_convert_harvest_format.py "$PROJECT" 2>&1 else python 23_convert_harvest_format.py "$PROJECT" fi ``` **File: `31_harvest_imminent_weekly.sh`** ```bash #!/bin/bash # Wrapper for Python script 31: Harvest imminent weekly # Usage: ./31_harvest_imminent_weekly.sh --project=angata set -e PROJECT="" while [[ $# -gt 0 ]]; do case $1 in --project=*) PROJECT="${1#*=}" ;; --*) ;; esac shift done [ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; } cd "$(dirname "$0")/python_app" if command -v conda &> /dev/null; then conda run -n pytorch_gpu python 31_harvest_imminent_weekly.py "$PROJECT" 2>&1 || \ conda run -n pytorch_cpu python 31_harvest_imminent_weekly.py "$PROJECT" 2>&1 else python 31_harvest_imminent_weekly.py "$PROJECT" fi ``` --- ### STEP 9: CREATE 2 Python Package Files **File: `python_app/requirements_harvest.txt`** ``` torch>=2.0.0 pandas>=1.5.0 numpy>=1.23.0 scikit-learn>=1.3.0 GDAL>=3.7.0 sentinelhub>=3.9.0 shapely>=2.0.0 pyproj>=3.4.0 ``` **File: `python_app/environment_pytorch.yml`** ```yaml name: pytorch_gpu channels: - pytorch - nvidia - conda-forge dependencies: - python=3.10 - pytorch::pytorch - pytorch::torchvision - pytorch::torchaudio - pytorch::pytorch-cuda=11.8 - gdal>=3.7.0 - pip - pip: - sentinelhub>=3.9.0 - shapely>=2.0.0 - pyproj>=3.4.0 ``` --- ### STEP 10: CORE TESTING CHECKLIST ```bash # 1. Migration php artisan migrate # ✅ Expected: No errors, client_type column added # 2. Download test # Go to Laravel UI → Create project with client_type=agronomic_support # → Download Manager → Add image → Download # Expected: File in laravel_app/storage/app/{project}/field_tiles_CI/ # 3. Mosaic test # Go to Mosaic Manager → Create mosaic # Check logs: grep "Unknown option" laravel.log # Expected: No --data_dir errors, mosaic created # 4. Mail test # Create project with client_type=cane_supply # Generate & send report # Expected: Email has 2 attachments (report + KPI Excel) # 5. Shell wrapper test ./10_create_per_field_tiffs.sh --project=angata # Expected: R script executes without error ``` ✅ **CORE MERGE COMPLETE** --- ## 🟡 PHASE 2: ENHANCEMENTS (Post-Merge) ### OPTIONAL 1: Country-Based Organization **Why**: Organize projects by geographic location (MZ/UG/TZ folders) **Create Migration**: `database/migrations/YYYY_MM_DD_add_country_to_projects_table.php` ```php string('country')->default('Mozambique')->after('name'); $table->string('country_code', 2)->default('MZ')->after('country'); }); // Update existing projects DB::table('projects')->where('name', 'angata')->update(['country' => 'Mozambique', 'country_code' => 'MZ']); DB::table('projects')->where('name', 'aura')->update(['country' => 'Mozambique', 'country_code' => 'MZ']); DB::table('projects')->where('name', 'chemba')->update(['country' => 'Mozambique', 'country_code' => 'MZ']); DB::table('projects')->where('name', 'xinavane')->update(['country' => 'Tanzania', 'country_code' => 'TZ']); DB::table('projects')->where('name', 'esa')->update(['country' => 'Kenya', 'country_code' => 'KQ']); DB::table('projects')->where('name', 'simba')->update(['country' => 'Uganda', 'country_code' => 'UG']); DB::table('projects')->where('name', 'john')->update(['country' => 'Uganda', 'country_code' => 'UG']); DB::table('projects')->where('name', 'huss')->update(['country' => 'Tanzania', 'country_code' => 'TZ']); } public function down(): void { Schema::table('projects', function (Blueprint $table) { $table->dropColumn(['country', 'country_code']); }); } }; ``` **Update Project.php `$fillable`:** ```php protected $fillable = [ 'name', 'country', 'country_code', 'download_path', 'client_type', // ... rest ]; ``` **Update ProjectManager.php:** ```php public array $countries = [ 'MZ' => 'Mozambique', 'TZ' => 'Tanzania', 'UG' => 'Uganda', 'KQ' => 'Kenya', 'SA' => 'South Africa', 'ZW' => 'Zimbabwe', 'BR' => 'Brazil', 'MX' => 'Mexico', 'IN' => 'India', ]; public function createProject() { $projectIdentifier = $this->formData['id'] ?? null; Validator::make( ['name' => $this->formData['name']], ['name' => ['required', Rule::unique('projects')->ignore($projectIdentifier)]] )->validate(); $projectPath = $this->formData['country_code'] . '/' . $this->formData['name']; Storage::makeDirectory($projectPath, recursive: true); $project = Project::create([ 'name' => $this->formData['name'], 'country' => $this->formData['country'], 'country_code' => $this->formData['country_code'], 'download_path' => $projectPath, 'client_type' => $this->formData['client_type'] ?? 'agronomic_support', ]); return redirect()->route('project.show', [$project->name, 'settings']); } ``` **Add to Blade template:** ```blade
``` --- ### OPTIONAL 2: Download Scheduling **Why**: Avoid API rate limits by staggering downloads per project at 00:01 #### Option A: Linux Cron (If server is Linux) **Add to `/etc/cron.d/smartcane_downloads`:** ```bash # Stagger downloads by 10 minutes per project 1 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py angata 2>&1 | logger 15 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py chemba 2>&1 | logger 25 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py xinavane 2>&1 | logger 35 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py esa 2>&1 | logger 45 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py simba 2>&1 | logger 0 1 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py aura 2>&1 | logger ``` #### Option B: Windows Task Scheduler (If server is Windows) ```powershell # Create task for each project $taskName = "SmartCane-Download-angata" $action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-NoProfile -WindowStyle Hidden -Command python C:\smartcane\python_app\00_download_8band_pu_optimized.py angata" $trigger = New-ScheduledTaskTrigger -Daily -At 00:01 Register-ScheduledTask -TaskName $taskName -Action $action -Trigger $trigger -RunLevel Highest ``` #### Option C: Laravel Task Scheduler **Add to `laravel_app/app/Console/Kernel.php`:** ```php protected function schedule(Schedule $schedule) { $schedule->exec('python python_app/00_download_8band_pu_optimized.py angata') ->dailyAt('00:01'); $schedule->exec('python python_app/00_download_8band_pu_optimized.py chemba') ->dailyAt('00:15'); $schedule->exec('python python_app/00_download_8band_pu_optimized.py xinavane') ->dailyAt('00:25'); $schedule->exec('python python_app/00_download_8band_pu_optimized.py esa') ->dailyAt('00:35'); $schedule->exec('python python_app/00_download_8band_pu_optimized.py simba') ->dailyAt('00:45'); $schedule->exec('python python_app/00_download_8band_pu_optimized.py aura') ->dailyAt('01:00'); } ``` --- ### OPTIONAL 3: Project Search Feature **Why**: Find projects quickly if there are many **Add to `laravel_app/app/Livewire/Projects/ProjectList.php`:** ```php public string $searchQuery = ''; public function getProjectsProperty() { $query = Project::query(); if (!empty($this->searchQuery)) { $query->where('name', 'like', '%' . $this->searchQuery . '%') ->orWhere('download_path', 'like', '%' . $this->searchQuery . '%'); } return $query->orderBy('name')->paginate(15); } ``` **Add to Blade template:** ```blade
@forelse($this->projects as $project)

{{ $project->name }}

{{ $project->download_path }}

{{ $project->client_type }}
@empty

No projects found

@endforelse
{{ $this->projects->links() }} ``` --- ### OPTIONAL 4: Harvest Date Prediction Setup **Why**: Enable harvest Date forecasting for cane_supply projects **Create conda environment:** ```bash conda env create -f python_app/environment_pytorch.yml # Activate conda activate pytorch_gpu # Or CPU-only if no GPU conda create -n pytorch_cpu python=3.10 pytorch::pytorch torchvision torchaudio -c pytorch conda activate pytorch_cpu ``` **Run baseline prediction (once):** ```bash python python_app/22_harvest_baseline_prediction.py angata python python_app/23_convert_harvest_format.py angata ``` **Schedule weekly prediction:** ```bash # Add to cron (Linux) 0 23 * * 0 conda run -n pytorch_gpu python /home/user/smartcane/python_app/31_harvest_imminent_weekly.py angata 2>&1 | logger # Or Task Scheduler (Windows) # Similar to download scheduling above, but Sunday 23:00 ``` --- ### OPTIONAL 5: Area Unit Selection UI (Hectares vs Acres) **Why**: Allow projects to choose their preferred area unit (hectares or acres) in reports and dashboards **Status**: R/Python business logic complete. Database schema + UI implementation pending. **What's already done** (in this codebase): - ✅ Unified area calculation function in `80_utils_common.R`: `calculate_area_from_geometry()` - ✅ Area unit preference in `parameters_project.R`: `AREA_UNIT_PREFERENCE` (default: "hectare") - ✅ Helper function: `get_area_unit_label()` for dynamic "ha" or "ac" display - ✅ Refactored scripts 80/90/91 to use unified function and support user's area preference - ✅ Area now included in KPI outputs (CSV/RDS/Excel) from script 80 - ✅ Scripts 90/91 read area from KPI files instead of recalculating **What needs implementation (for your Laravel colleague)**: **Step 1: Create database migration** ```php enum('preferred_area_unit', ['hectare', 'acre']) ->default('hectare') ->after('client_type'); }); } public function down(): void { Schema::table('projects', function (Blueprint $table) { $table->dropColumn('preferred_area_unit'); }); } }; ``` **Step 2: Update Project model** (`laravel_app/app/Models/Project.php`) ```php protected $fillable = [ // ... existing fields ... 'preferred_area_unit', // ADD THIS ]; ``` **Step 3: Add form UI** (`laravel_app/app/Livewire/Projects/ProjectManager.php` or Blade template) ```blade
@error('formData.preferred_area_unit') {{ $message }} @enderror
``` **Step 4: Pass preference to R scripts** (in job/shell wrapper) When launching R scripts, read `project->preferred_area_unit` and either: - **Option A**: Write to `parameters_project.R` dynamically before script execution - **Option B**: Pass as environment variable to scripts (R scripts read `Sys.getenv("AREA_UNIT")`) Example (PowerShell wrapper): ```powershell $areaUnit = $project->preferred_area_unit # From database $env:AREA_UNIT = $areaUnit # Set environment variable & "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/80_calculate_kpis.R $project ``` **Testing checklist**: - [ ] Database migration runs successfully - [ ] Project form shows area unit radio buttons/dropdown - [ ] Can select and save area unit preference - [ ] Area unit persists in database - [ ] Run script 80 with one project set to "hectare", observe KPI output - [ ] Run script 80 with another project set to "acre", compare outputs - [ ] Reports (scripts 90/91) display area in user's chosen unit **Notes**: - Default preference: "hectare" (metric standard) - Conversion factor used: 0.404686 (1 hectare = 0.404686 acres) - All area calculations use EPSG:6933 (equal-area projection) for accuracy - Area column in KPI outputs named dynamically: "Area_ha" or "Area_ac" --- ## 📊 Summary: What Gets Changed | Category | Files Modified | Changes Required | |----------|---|---| | Database | migrations/ | 1 file: add client_type column | | Models | Project.php | 5 edits: fillable, 3 methods, 1 new method | | Jobs | 2 files | ProjectDownloadTiffJob (1 line), ProjectMosiacGeneratorJob (full array) | | Forms | 3 files | ProjectManager.php, Blade template, MailingForm.php | | Scripts | 5 files created | Shell wrappers (R/Python) | | Python | 2 files created | requirements_harvest.txt, environment_pytorch.yml | **TOTAL**: 6 files created + 6 files modified + 1 template modified = **13 changes** --- ## ✅ FINAL VERIFICATION After ALL changes (core + optionals), test: ```bash # 1. Migration worked php artisan migrate # 2. Download saves to correct path # → Download image → check laravel_app/storage/app/{project}/field_tiles_CI/ # 3. Mosaic runs without errors # → Create mosaic → check logs for no --data_dir errors # 4. Mail has 2 attachments for cane_supply # → Send report for cane_supply project → verify report + KPI Excel # 5. Shell wrappers work ./10_create_per_field_tiffs.sh --project=angata # → Should execute R script successfully # 6. Search works (if implemented) # → Search for project by name on Projects page # 7. Country filter works (if implemented) # → Filter projects by country code ``` --- ## 🌍 POST-MERGE: Data Recreation Strategy After merge is live, existing projects need new directory structure. ### Option A: Delete & Redownload (Small projects) **Projects**: aura, chemba, xinavane, esa, simba ``` 1. Backup project folder (optional) 2. Delete project from Laravel UI 3. Recreate with new client_type selector 4. Redownload 2-3 years of data (~50-150 GB per project) 5. Run pipeline normally ``` ### Option B: Preserve merged_tif (Large projects) **Projects**: angata ``` 1. Backup merged_tif/ folder externally 2. Delete all other folders in project 3. Keep only: merged_tif/ 4. Run Scripts 10-80 on existing data → Regenerates field_tiles_CI/, reports/, etc. 5. No need to redownload ``` --- ## 🔴 CORE vs OPTIONAL Quick List **MUST DO** (for merge): - ✅ Database migration - ✅ Project.php edits (4 path changes + 1 new method) - ✅ Job edits (ProjectDownloadTiffJob, ProjectMosiacGeneratorJob) - ✅ Form edits (ProjectManager, MailingForm, Blade template) - ✅ 5 shell wrappers - ✅ 2 Python files **NICE-TO-HAVE** (post-merge): - 🟡 Country organization (adds ~45 min) - 🟡 Download scheduling (adds ~30 min) - 🟡 Project search (adds ~30 min) - 🟡 Harvest prediction setup (adds ~20 min) --- **Ready to implement everything?** All code is copy-paste ready above.