feat: Add implementation guide for SmartCane code improvements

- Introduced core and optional features for the code improvements merge.
- Documented detailed implementation checklist for core features including database migration, model updates, job changes, form modifications, shell script wrappers, and Python package files.
- Added optional enhancements such as country-based project organization, download scheduling, project search feature, and harvest prediction setup documentation.
- Included testing checklist and post-merge data recreation strategy.
This commit is contained in:
Timon 2026-02-19 15:03:14 +01:00
parent 4953f01c96
commit 6bdbea12cc
5 changed files with 992 additions and 113 deletions

967
IMPLEMENTATION_GUIDE.md Normal file
View file

@ -0,0 +1,967 @@
# SmartCane Code-Improvements Merge Guide (Complete)
**Target Repo**: https://bitbucket.org/sobitnl/smartcane/src/master/
**Source Branch**: `code-improvements` (Timon's experimental area)
**Status**: Ready for merge (all changes, including optional enhancements)
---
## 📋 SCOPE: CORE + OPTIONAL FEATURES
### 🔴 CORE (Required for merge)
- Database migration (client_type column)
- File path changes (merged_final_tif → field_tiles_CI)
- Laravel form/job/mailing updates
- Shell script wrappers (5 files)
- Python package files (2 files)
### 🟡 OPTIONAL (Post-merge enhancements)
- Country-based project organization (MZ/UG/TZ folders)
- Download scheduling (staggered 00:01 per-project)
- Project search feature
- Harvest prediction setup docs
---
## 🚀 FULL IMPLEMENTATION CHECKLIST
```
═══════════════════════════════════════════════════════════════
PHASE 1: CORE MERGE (Required)
═══════════════════════════════════════════════════════════════
[1.1] Database Migration (10 min)
✓ Create: database/migrations/YYYY_MM_DD_add_client_type_to_projects_table.php
✓ Run: php artisan migrate
[1.2] Laravel Model Changes (20 min)
✓ Edit: laravel_app/app/Models/Project.php
- Add 'client_type' to $fillable
- Update getMergedTiffList() — path change
- Update startDownload() — path change
- Update getTifsAsZip() — path change
- Add getLatestKpiFile() method
[1.3] Laravel Job Changes (15 min)
✓ Edit: laravel_app/app/Jobs/ProjectDownloadTiffJob.php
- Change path in handleForDate()
✓ Edit: laravel_app/app/Jobs/ProjectMosiacGeneratorJob.php
- Replace command array
- Improve error handling
[1.4] Laravel Forms (15 min)
✓ Edit: laravel_app/app/Livewire/Projects/ProjectManager.php
- Add client_type to form
✓ Edit: Project form Blade template
- Add <select> for client_type
✓ Edit: laravel_app/app/Livewire/Forms/MailingForm.php
- Auto-attach KPI Excel for cane_supply
[1.5] Shell Script Wrappers (15 min)
✓ Create: 10_create_per_field_tiffs.sh
✓ Create: 21_convert_ci_rds_to_csv.sh
✓ Create: 22_harvest_baseline_prediction.sh
✓ Create: 23_convert_harvest_format.sh
✓ Create: 31_harvest_imminent_weekly.sh
[1.6] Python Package Files (5 min)
✓ Create: python_app/requirements_harvest.txt
✓ Create: python_app/environment_pytorch.yml
[1.7] Testing Core (20 min)
✓ Run migration ✓ Download test ✓ Mosaic test ✓ Mail test
═══════════════════════════════════════════════════════════════
PHASE 2: ENHANCEMENTS
═══════════════════════════════════════════════════════════════
[2.1] Country Organization (45 min)
- Add DB migration (country, country_code columns)
- Update Project.php fillable
- Add ProjectManager form fields
- Add Blade country selector + auto-populate
- Add country filtering to ProjectList
- Create "Add New Country" feature
[2.2] Download Scheduling (30 min)
- Option A: Windows Task Scheduler
- Option B: Linux cron
- Option C: Laravel Task Scheduler
[2.3] Project Search (30 min)
- Add search/filter to ProjectList.php
- Add Blade input fields
[2.4] Harvest Prediction Setup (20 min)
- Document conda env setup
- Document script execution
```
---
## 🔴 PHASE 1: CORE MERGE
### STEP 1: Create & Run Database Migration
**File**: `database/migrations/2024_02_19_000000_add_client_type_to_projects_table.php`
```php
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
public function up(): void
{
Schema::table('projects', function (Blueprint $table) {
$table->enum('client_type', ['agronomic_support', 'cane_supply'])
->default('agronomic_support')
->after('download_path');
});
}
public function down(): void
{
Schema::table('projects', function (Blueprint $table) {
$table->dropColumn('client_type');
});
}
};
```
**Run migration:**
```bash
php artisan migrate
```
---
### STEP 2: Edit `laravel_app/app/Models/Project.php`
**Edit 2.1: Add to `$fillable` array**
Find and add `'client_type'` after `'download_path'`:
```php
protected $fillable = [
'name',
'download_path',
'client_type', // ADD THIS
'mail_template',
'mail_subject',
'mail_frequency',
'mail_day',
'mail_scheduled',
'pivot_json_path',
'span_json_path',
'harvest_json_path',
'min_harvest_date',
'borders',
];
```
**Edit 2.2: Update `getMergedTiffList()` method (around line 232)**
```php
public function getMergedTiffList()
{
return collect(Storage::files($this->download_path.'/field_tiles_CI')) // CHANGED
->filter(fn($file) => Str::endsWith($file, '.tif'))
->sortByDesc(function ($file) {
$parts = explode('_', str_replace('.tif', '', $file));
$date = $parts[1];
return $date;
})
->values();
}
```
**Edit 2.3: Update `startDownload()` method (around line 265)**
```php
public function startDownload(Carbon $date)
{
$downloadRequest = $this->downloads()->updateOrCreate(
[
'project_id' => $this->id,
'name' => sprintf('%s.tif', $date->format('Y-m-d')),
],
[
'path' => sprintf('%s/%s/%s.tif', $this->download_path, 'field_tiles_CI', $date->format('Y-m-d')), // CHANGED
]
);
ProjectDownloadTiffJob::dispatch($downloadRequest, $date);
}
```
**Edit 2.4: Update `getTifsAsZip()` method (around line 489)**
```php
public function getTifsAsZip()
{
return collect(Storage::files($this->download_path . '/field_tiles_CI')) // CHANGED
->filter(fn($file) => Str::endsWith($file, '.tif'))
->values();
}
```
**Edit 2.5: Add new method `getLatestKpiFile()`**
Add this method at the end of the Project class:
```php
public function getLatestKpiFile(): ?string
{
$kpiPath = $this->download_path . '/reports/kpis/';
try {
$files = Storage::files($kpiPath);
} catch (\Exception $e) {
return null;
}
return collect($files)
->filter(fn($f) => Str::endsWith($f, '.xlsx'))
->sortByDesc(fn($f) => Storage::lastModified($f))
->first();
}
```
---
### STEP 3: Edit `laravel_app/app/Jobs/ProjectDownloadTiffJob.php`
**Around line 73 in `handleForDate()` method:**
Change:
```php
$path = $project->download_path . '/merged_final_tif/' . $filename;
```
To:
```php
$path = $project->download_path . '/field_tiles_CI/' . $filename;
```
---
### STEP 4: Edit `laravel_app/app/Jobs/ProjectMosiacGeneratorJob.php`
**Lines 50-70, replace the `$command` array initialization:**
OLD:
```php
$command = [
sprintf('%sbuild_mosaic.sh', $projectFolder),
sprintf('--end_date=%s', $this->mosaic->end_date->format('Y-m-d')),
sprintf('--offset=%s', $this->mosaic->offset),
sprintf('--data_dir=%s', $this->mosaic->project->download_path),
sprintf('--file_name_tif=%s', basename($this->mosaic->path)),
];
```
NEW:
```php
$command = [
sprintf('%s40_mosaic_creation_per_field.sh', $projectFolder),
sprintf('--project=%s', $project->name),
sprintf('--end_date=%s', $this->mosaic->end_date->format('Y-m-d')),
sprintf('--offset=%s', $this->mosaic->offset),
];
```
**Also improve exception handling (around line 65-75):**
```php
try {
$process = ProcessNew::timeout(300)
->env(['PATH' => $currentPath.':/usr/local/Cellar/pandoc/3.1.8/bin/pandoc'])
->start($command, function (string $type, string $output) use ($project) {
ProjectLogger::log($project, $output);
$this->throwIfOutputContainsError($output);
});
$results = $process->wait();
if ($results->successful()) {
$this->mosaic->setStatusSuccess();
}
} catch (\RuntimeException|ProcessTimedOutException|ProcessFailedException $e) {
ProjectLogger::log($project, "MOSAIC JOB ERROR: " . $e->getMessage());
$this->mosaic->setStatusFailed();
throw $e;
}
```
---
### STEP 5: Edit `laravel_app/app/Livewire/Projects/ProjectManager.php`
**In `createProject()` method:**
```php
public function createProject()
{
$projectIdentifier = $this->formData['id'] ?? null;
Validator::make(
['name' => $this->formData['name']],
['name' => ['required', Rule::unique('projects')->ignore($projectIdentifier), 'string', 'max:255']]
)->validate();
$project = Project::create([
'name' => $this->formData['name'],
'download_path' => $this->makeValidDirectoryName($this->formData['name']),
'client_type' => $this->formData['client_type'] ?? 'agronomic_support', // ADD THIS
]);
return redirect()->route('project.show', [$project->name, 'settings']);
}
```
**In `resetFormData()` method:**
```php
private function resetFormData()
{
$this->formData = [
'name' => '',
'client_type' => 'agronomic_support', // ADD THIS
'mail_template' => '',
'mail_subject' => '',
'mail_frequency' => '',
'mail_day' => '',
// ... rest of fields
];
}
```
---
### STEP 6: Edit Project Form Blade Template
**Find the project create/edit form and add this field:**
```blade
<div class="form-group mb-3">
<label for="client_type" class="form-label">Client Type <span class="text-danger">*</span></label>
<select wire:model="formData.client_type" id="client_type" class="form-control" required>
<option value="">-- Select Client Type --</option>
<option value="agronomic_support">Agronomic Support</option>
<option value="cane_supply">Cane Supply</option>
</select>
@error('formData.client_type')
<span class="text-danger small">{{ $message }}</span>
@enderror
</div>
```
---
### STEP 7: Edit `laravel_app/app/Livewire/Forms/MailingForm.php`
**In `saveAndSendMailing()` static method:**
```php
public static function saveAndSendMailing($report, $subject, $message, $recipients) {
if ($report->documentExists()) {
$mailing = $report->project->mailings()->create([
'subject' => $subject,
'message' => $message,
'report_id' => $report->id,
]);
// Attach main report
$mailing->attachments()->create([
'name' => $report->name,
'path' => $report->path,
]);
// Auto-attach KPI Excel for cane_supply projects
if ($report->project->client_type === 'cane_supply') {
$kpiFile = $report->project->getLatestKpiFile();
if ($kpiFile) {
$mailing->attachments()->create([
'name' => 'KPI Summary',
'path' => $kpiFile,
]);
}
}
$mailing->recipients()->createMany($recipients);
Mail::to($mailing->recipients()->pluck('email')->toArray())
->send(new \App\Mail\ReportMailer($mailing, $report));
} else {
self::sendReportNotFoundNotificationToAdmin($report);
}
}
```
---
### STEP 8: CREATE 5 Shell Script Wrappers
**File: `10_create_per_field_tiffs.sh`**
```bash
#!/bin/bash
# Wrapper for R script 10: Create per-field TIFFs
# Usage: ./10_create_per_field_tiffs.sh --project=angata
set -e
PROJECT=""
while [[ $# -gt 0 ]]; do
case $1 in
--project=*) PROJECT="${1#*=}" ;;
--*) ;; # Ignore other args
esac
shift
done
[ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; }
cd "$(dirname "$0")/r_app"
Rscript -e "PROJECT='$PROJECT'; source('parameters_project.R'); source('10_create_per_field_tiffs.R')"
```
**File: `21_convert_ci_rds_to_csv.sh`**
```bash
#!/bin/bash
# Wrapper for R script 21: Convert CI RDS to CSV
# Usage: ./21_convert_ci_rds_to_csv.sh --project=angata
set -e
PROJECT=""
while [[ $# -gt 0 ]]; do
case $1 in
--project=*) PROJECT="${1#*=}" ;;
--*) ;;
esac
shift
done
[ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; }
cd "$(dirname "$0")/r_app"
Rscript -e "PROJECT='$PROJECT'; source('parameters_project.R'); source('21_convert_ci_rds_to_csv.R')"
```
**File: `22_harvest_baseline_prediction.sh`**
```bash
#!/bin/bash
# Wrapper for Python script 22: Harvest baseline prediction
# Usage: ./22_harvest_baseline_prediction.sh --project=angata
set -e
PROJECT=""
while [[ $# -gt 0 ]]; do
case $1 in
--project=*) PROJECT="${1#*=}" ;;
--*) ;;
esac
shift
done
[ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; }
cd "$(dirname "$0")/python_app"
if command -v conda &> /dev/null; then
conda run -n pytorch_gpu python 22_harvest_baseline_prediction.py "$PROJECT" 2>&1 || \
conda run -n pytorch_cpu python 22_harvest_baseline_prediction.py "$PROJECT" 2>&1
else
python 22_harvest_baseline_prediction.py "$PROJECT"
fi
```
**File: `23_convert_harvest_format.sh`**
```bash
#!/bin/bash
# Wrapper for Python script 23: Convert harvest format
# Usage: ./23_convert_harvest_format.sh --project=angata
set -e
PROJECT=""
while [[ $# -gt 0 ]]; do
case $1 in
--project=*) PROJECT="${1#*=}" ;;
--*) ;;
esac
shift
done
[ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; }
cd "$(dirname "$0")/python_app"
if command -v conda &> /dev/null; then
conda run -n pytorch_gpu python 23_convert_harvest_format.py "$PROJECT" 2>&1 || \
conda run -n pytorch_cpu python 23_convert_harvest_format.py "$PROJECT" 2>&1
else
python 23_convert_harvest_format.py "$PROJECT"
fi
```
**File: `31_harvest_imminent_weekly.sh`**
```bash
#!/bin/bash
# Wrapper for Python script 31: Harvest imminent weekly
# Usage: ./31_harvest_imminent_weekly.sh --project=angata
set -e
PROJECT=""
while [[ $# -gt 0 ]]; do
case $1 in
--project=*) PROJECT="${1#*=}" ;;
--*) ;;
esac
shift
done
[ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; }
cd "$(dirname "$0")/python_app"
if command -v conda &> /dev/null; then
conda run -n pytorch_gpu python 31_harvest_imminent_weekly.py "$PROJECT" 2>&1 || \
conda run -n pytorch_cpu python 31_harvest_imminent_weekly.py "$PROJECT" 2>&1
else
python 31_harvest_imminent_weekly.py "$PROJECT"
fi
```
---
### STEP 9: CREATE 2 Python Package Files
**File: `python_app/requirements_harvest.txt`**
```
torch>=2.0.0
pandas>=1.5.0
numpy>=1.23.0
scikit-learn>=1.3.0
GDAL>=3.7.0
sentinelhub>=3.9.0
shapely>=2.0.0
pyproj>=3.4.0
```
**File: `python_app/environment_pytorch.yml`**
```yaml
name: pytorch_gpu
channels:
- pytorch
- nvidia
- conda-forge
dependencies:
- python=3.10
- pytorch::pytorch
- pytorch::torchvision
- pytorch::torchaudio
- pytorch::pytorch-cuda=11.8
- gdal>=3.7.0
- pip
- pip:
- sentinelhub>=3.9.0
- shapely>=2.0.0
- pyproj>=3.4.0
```
---
### STEP 10: CORE TESTING CHECKLIST
```bash
# 1. Migration
php artisan migrate
# ✅ Expected: No errors, client_type column added
# 2. Download test
# Go to Laravel UI → Create project with client_type=agronomic_support
# → Download Manager → Add image → Download
# Expected: File in laravel_app/storage/app/{project}/field_tiles_CI/
# 3. Mosaic test
# Go to Mosaic Manager → Create mosaic
# Check logs: grep "Unknown option" laravel.log
# Expected: No --data_dir errors, mosaic created
# 4. Mail test
# Create project with client_type=cane_supply
# Generate & send report
# Expected: Email has 2 attachments (report + KPI Excel)
# 5. Shell wrapper test
./10_create_per_field_tiffs.sh --project=angata
# Expected: R script executes without error
```
✅ **CORE MERGE COMPLETE**
---
## 🟡 PHASE 2: ENHANCEMENTS (Post-Merge)
### OPTIONAL 1: Country-Based Organization
**Why**: Organize projects by geographic location (MZ/UG/TZ folders)
**Create Migration**: `database/migrations/YYYY_MM_DD_add_country_to_projects_table.php`
```php
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
use Illuminate\Support\Facades\DB;
return new class extends Migration
{
public function up(): void
{
Schema::table('projects', function (Blueprint $table) {
$table->string('country')->default('Mozambique')->after('name');
$table->string('country_code', 2)->default('MZ')->after('country');
});
// Update existing projects
DB::table('projects')->where('name', 'angata')->update(['country' => 'Mozambique', 'country_code' => 'MZ']);
DB::table('projects')->where('name', 'aura')->update(['country' => 'Mozambique', 'country_code' => 'MZ']);
DB::table('projects')->where('name', 'chemba')->update(['country' => 'Mozambique', 'country_code' => 'MZ']);
DB::table('projects')->where('name', 'xinavane')->update(['country' => 'Tanzania', 'country_code' => 'TZ']);
DB::table('projects')->where('name', 'esa')->update(['country' => 'Kenya', 'country_code' => 'KQ']);
DB::table('projects')->where('name', 'simba')->update(['country' => 'Uganda', 'country_code' => 'UG']);
DB::table('projects')->where('name', 'john')->update(['country' => 'Uganda', 'country_code' => 'UG']);
DB::table('projects')->where('name', 'huss')->update(['country' => 'Tanzania', 'country_code' => 'TZ']);
}
public function down(): void
{
Schema::table('projects', function (Blueprint $table) {
$table->dropColumn(['country', 'country_code']);
});
}
};
```
**Update Project.php `$fillable`:**
```php
protected $fillable = [
'name',
'country',
'country_code',
'download_path',
'client_type',
// ... rest
];
```
**Update ProjectManager.php:**
```php
public array $countries = [
'MZ' => 'Mozambique',
'TZ' => 'Tanzania',
'UG' => 'Uganda',
'KQ' => 'Kenya',
'SA' => 'South Africa',
'ZW' => 'Zimbabwe',
'BR' => 'Brazil',
'MX' => 'Mexico',
'IN' => 'India',
];
public function createProject()
{
$projectIdentifier = $this->formData['id'] ?? null;
Validator::make(
['name' => $this->formData['name']],
['name' => ['required', Rule::unique('projects')->ignore($projectIdentifier)]]
)->validate();
$projectPath = $this->formData['country_code'] . '/' . $this->formData['name'];
Storage::makeDirectory($projectPath, recursive: true);
$project = Project::create([
'name' => $this->formData['name'],
'country' => $this->formData['country'],
'country_code' => $this->formData['country_code'],
'download_path' => $projectPath,
'client_type' => $this->formData['client_type'] ?? 'agronomic_support',
]);
return redirect()->route('project.show', [$project->name, 'settings']);
}
```
**Add to Blade template:**
```blade
<div class="form-group mb-3">
<label for="country" class="form-label">Country</label>
<select wire:model="formData.country" id="country" class="form-control" required>
@foreach($countries as $code => $name)
<option value="{{ $name }}" @selected($formData['country'] === $name)>
{{ $name }} ({{ $code }})
</option>
@endforeach
</select>
</div>
<div class="form-group mb-3">
<label for="country_code" class="form-label">Country Code (Auto-populated)</label>
<input type="text" wire:model="formData.country_code" id="country_code" readonly class="form-control" />
</div>
```
---
### OPTIONAL 2: Download Scheduling
**Why**: Avoid API rate limits by staggering downloads per project at 00:01
#### Option A: Linux Cron (If server is Linux)
**Add to `/etc/cron.d/smartcane_downloads`:**
```bash
# Stagger downloads by 10 minutes per project
1 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py angata 2>&1 | logger
15 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py chemba 2>&1 | logger
25 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py xinavane 2>&1 | logger
35 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py esa 2>&1 | logger
45 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py simba 2>&1 | logger
0 1 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py aura 2>&1 | logger
```
#### Option B: Windows Task Scheduler (If server is Windows)
```powershell
# Create task for each project
$taskName = "SmartCane-Download-angata"
$action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-NoProfile -WindowStyle Hidden -Command python C:\smartcane\python_app\00_download_8band_pu_optimized.py angata"
$trigger = New-ScheduledTaskTrigger -Daily -At 00:01
Register-ScheduledTask -TaskName $taskName -Action $action -Trigger $trigger -RunLevel Highest
```
#### Option C: Laravel Task Scheduler
**Add to `laravel_app/app/Console/Kernel.php`:**
```php
protected function schedule(Schedule $schedule)
{
$schedule->exec('python python_app/00_download_8band_pu_optimized.py angata')
->dailyAt('00:01');
$schedule->exec('python python_app/00_download_8band_pu_optimized.py chemba')
->dailyAt('00:15');
$schedule->exec('python python_app/00_download_8band_pu_optimized.py xinavane')
->dailyAt('00:25');
$schedule->exec('python python_app/00_download_8band_pu_optimized.py esa')
->dailyAt('00:35');
$schedule->exec('python python_app/00_download_8band_pu_optimized.py simba')
->dailyAt('00:45');
$schedule->exec('python python_app/00_download_8band_pu_optimized.py aura')
->dailyAt('01:00');
}
```
---
### OPTIONAL 3: Project Search Feature
**Why**: Find projects quickly if there are many
**Add to `laravel_app/app/Livewire/Projects/ProjectList.php`:**
```php
public string $searchQuery = '';
public function getProjectsProperty()
{
$query = Project::query();
if (!empty($this->searchQuery)) {
$query->where('name', 'like', '%' . $this->searchQuery . '%')
->orWhere('download_path', 'like', '%' . $this->searchQuery . '%');
}
return $query->orderBy('name')->paginate(15);
}
```
**Add to Blade template:**
```blade
<div class="search-bar mb-4">
<input type="text"
wire:model.live="searchQuery"
placeholder="Search projects..."
class="form-control" />
</div>
<div class="projects-list">
@forelse($this->projects as $project)
<div class="project-card">
<h3>{{ $project->name }}</h3>
<p>{{ $project->download_path }}</p>
<span class="badge badge-info">{{ $project->client_type }}</span>
</div>
@empty
<p>No projects found</p>
@endforelse
</div>
{{ $this->projects->links() }}
```
---
### OPTIONAL 4: Harvest Date Prediction Setup
**Why**: Enable harvest Date forecasting for cane_supply projects
**Create conda environment:**
```bash
conda env create -f python_app/environment_pytorch.yml
# Activate
conda activate pytorch_gpu
# Or CPU-only if no GPU
conda create -n pytorch_cpu python=3.10 pytorch::pytorch torchvision torchaudio -c pytorch
conda activate pytorch_cpu
```
**Run baseline prediction (once):**
```bash
python python_app/22_harvest_baseline_prediction.py angata
python python_app/23_convert_harvest_format.py angata
```
**Schedule weekly prediction:**
```bash
# Add to cron (Linux)
0 23 * * 0 conda run -n pytorch_gpu python /home/user/smartcane/python_app/31_harvest_imminent_weekly.py angata 2>&1 | logger
# Or Task Scheduler (Windows)
# Similar to download scheduling above, but Sunday 23:00
```
---
## 📊 Summary: What Gets Changed
| Category | Files Modified | Changes Required |
|----------|---|---|
| Database | migrations/ | 1 file: add client_type column |
| Models | Project.php | 5 edits: fillable, 3 methods, 1 new method |
| Jobs | 2 files | ProjectDownloadTiffJob (1 line), ProjectMosiacGeneratorJob (full array) |
| Forms | 3 files | ProjectManager.php, Blade template, MailingForm.php |
| Scripts | 5 files created | Shell wrappers (R/Python) |
| Python | 2 files created | requirements_harvest.txt, environment_pytorch.yml |
**TOTAL**: 6 files created + 6 files modified + 1 template modified = **13 changes**
---
## ✅ FINAL VERIFICATION
After ALL changes (core + optionals), test:
```bash
# 1. Migration worked
php artisan migrate
# 2. Download saves to correct path
# → Download image → check laravel_app/storage/app/{project}/field_tiles_CI/
# 3. Mosaic runs without errors
# → Create mosaic → check logs for no --data_dir errors
# 4. Mail has 2 attachments for cane_supply
# → Send report for cane_supply project → verify report + KPI Excel
# 5. Shell wrappers work
./10_create_per_field_tiffs.sh --project=angata
# → Should execute R script successfully
# 6. Search works (if implemented)
# → Search for project by name on Projects page
# 7. Country filter works (if implemented)
# → Filter projects by country code
```
---
## 🌍 POST-MERGE: Data Recreation Strategy
After merge is live, existing projects need new directory structure.
### Option A: Delete & Redownload (Small projects)
**Projects**: aura, chemba, xinavane, esa, simba
```
1. Backup project folder (optional)
2. Delete project from Laravel UI
3. Recreate with new client_type selector
4. Redownload 2-3 years of data (~50-150 GB per project)
5. Run pipeline normally
```
### Option B: Preserve merged_tif (Large projects)
**Projects**: angata
```
1. Backup merged_tif/ folder externally
2. Delete all other folders in project
3. Keep only: merged_tif/
4. Run Scripts 10-80 on existing data
→ Regenerates field_tiles_CI/, reports/, etc.
5. No need to redownload
```
---
## 🔴 CORE vs OPTIONAL Quick List
**MUST DO** (for merge):
- ✅ Database migration
- ✅ Project.php edits (4 path changes + 1 new method)
- ✅ Job edits (ProjectDownloadTiffJob, ProjectMosiacGeneratorJob)
- ✅ Form edits (ProjectManager, MailingForm, Blade template)
- ✅ 5 shell wrappers
- ✅ 2 Python files
**NICE-TO-HAVE** (post-merge):
- 🟡 Country organization (adds ~45 min)
- 🟡 Download scheduling (adds ~30 min)
- 🟡 Project search (adds ~30 min)
- 🟡 Harvest prediction setup (adds ~20 min)
---
**Ready to implement everything?** All code is copy-paste ready above.

View file

@ -1,2 +0,0 @@
install.packages(c("googledrive", "here", "tidyverse", "lubridate", "readxl", "googlesheets4", "here", "sf", "tidyverse", "lubridate", "terra", "exactextractr")
install.packages("packages/CIprep_0.1.4.tar.gz",repos=NULL, type="source")

View file

@ -48,11 +48,23 @@ public static function saveAndSendMailing($report, $subject, $message, $recipien
'report_id' => $report->id, 'report_id' => $report->id,
]); ]);
// Attach main report
$mailing->attachments()->create([ $mailing->attachments()->create([
'name' => $report->name, 'name' => $report->name,
'path' => $report->path, 'path' => $report->path,
]); ]);
// For cane_supply projects, also attach latest KPI Excel file
if ($report->project->client_type === 'cane_supply') {
$kpiFile = $report->project->getLatestKpiFile();
if ($kpiFile) {
$mailing->attachments()->create([
'name' => 'KPI Data - ' . basename($kpiFile),
'path' => $kpiFile,
]);
}
}
$mailing->recipients()->createMany($recipients); $mailing->recipients()->createMany($recipients);
Mail::to($mailing->recipients()->pluck('email')->toArray()) Mail::to($mailing->recipients()->pluck('email')->toArray())
->send(new \App\Mail\ReportMailer($mailing, $report)); ->send(new \App\Mail\ReportMailer($mailing, $report));

View file

@ -251,6 +251,19 @@ class="w-5 h-5 text-red-400 dark:text-red-200">
class="relative w-11 h-6 bg-gray-200 peer-focus:outline-none peer-focus:ring-4 peer-focus:ring-indigo-300 dark:peer-focus:ring-indigo-800 rounded-full peer dark:bg-gray-700 peer-checked:after:translate-x-full rtl:peer-checked:after:-translate-x-full peer-checked:after:border-white after:content-[''] after:absolute after:top-[2px] after:start-[2px] after:bg-white after:border-gray-300 after:border after:rounded-full after:h-5 after:w-5 after:transition-all dark:border-gray-600 peer-checked:bg-indigo-600"></div> class="relative w-11 h-6 bg-gray-200 peer-focus:outline-none peer-focus:ring-4 peer-focus:ring-indigo-300 dark:peer-focus:ring-indigo-800 rounded-full peer dark:bg-gray-700 peer-checked:after:translate-x-full rtl:peer-checked:after:-translate-x-full peer-checked:after:border-white after:content-[''] after:absolute after:top-[2px] after:start-[2px] after:bg-white after:border-gray-300 after:border after:rounded-full after:h-5 after:w-5 after:transition-all dark:border-gray-600 peer-checked:bg-indigo-600"></div>
</label> </label>
</div> </div>
<div class="mb-2">
<x-label for="client_type" value="{{ __('Client Type') }}"/>
<select id="client_type" class="border-gray-300 dark:border-gray-700 dark:bg-gray-900 dark:text-gray-300 focus:border-indigo-500 dark:focus:border-indigo-600 focus:ring-indigo-500 dark:focus:ring-indigo-600 rounded-md shadow-sm mt-1 block w-full"
wire:model="formData.client_type"
@if($this->formData['id'] ?? false) disabled @endif>
<option value="agronomic_support">{{ __('Agronomic Support') }}</option>
<option value="cane_supply">{{ __('Cane Supply') }}</option>
</select>
@if($this->formData['id'] ?? false)
<p class="text-sm text-gray-500 mt-1">{{ __('Client type cannot be changed after project creation') }}</p>
@endif
<x-input-error for="client_type" class="mt-2"/>
</div>
</form> </form>
</div> </div>
</div> </div>

View file

@ -1,111 +0,0 @@
[INFO] 2025-06-24 14:49:29 - SmartCane Project - Package Manager Started
[INFO] 2025-06-24 14:49:29 - Working directory: C:/Users/timon/Resilience BV/4020 SCane ESA DEMO - Documenten/General/4020 SCDEMO Team/4020 TechnicalData/WP3/smartcane
[INFO] 2025-06-24 14:49:29 - Checking renv initialization...
[INFO] 2025-06-24 14:49:29 - ✓ renv already initialized
[INFO] 2025-06-24 14:49:29 - ✓ renv already active
[INFO] 2025-06-24 14:49:29 -
=== INITIAL STATE ===
[INFO] 2025-06-24 14:49:29 - === PACKAGE REPORT ===
[INFO] 2025-06-24 14:49:29 - dplyr | Required: >= 1.1.4 | Installed: 1.1.4 | ✅ OK
[INFO] 2025-06-24 14:49:29 - here | Required: >= 1.0.1 | Installed: 1.0.1 | ✅ OK
[INFO] 2025-06-24 14:49:29 - lubridate | Required: >= 1.9.4 | Installed: 1.9.4 | ✅ OK
[INFO] 2025-06-24 14:49:29 - readr | Required: >= 2.1.5 | Installed: 2.1.5 | ✅ OK
[INFO] 2025-06-24 14:49:29 - readxl | Required: >= 1.4.5 | Installed: 1.4.5 | ✅ OK
[INFO] 2025-06-24 14:49:29 - stringr | Required: >= 1.5.1 | Installed: 1.5.1 | ✅ OK
[INFO] 2025-06-24 14:49:29 - tidyr | Required: >= 1.3.1 | Installed: 1.3.1 | ✅ OK
[INFO] 2025-06-24 14:49:29 - purrr | Required: >= 1.0.2 | Installed: 1.0.2 | ✅ OK
[INFO] 2025-06-24 14:49:29 - magrittr | Required: >= 2.0.0 | Installed: 2.0.3 | ✅ OK
[INFO] 2025-06-24 14:49:29 - exactextractr | Required: >= 0.10.0 | Installed: 0.10.0 | ✅ OK
[INFO] 2025-06-24 14:49:29 - raster | Required: >= 3.6.32 | Installed: 3.6.32 | ✅ OK
[INFO] 2025-06-24 14:49:29 - sf | Required: >= 1.0.19 | Installed: 1.0.19 | ✅ OK
[INFO] 2025-06-24 14:49:29 - terra | Required: >= 1.8.43 | Installed: 1.8.43 | ✅ OK
[INFO] 2025-06-24 14:49:29 - ggplot2 | Required: >= 3.5.1 | Installed: 3.5.1 | ✅ OK
[INFO] 2025-06-24 14:49:29 - tmap | Required: >= 4.0 | Installed: 4.0 | ✅ OK
[INFO] 2025-06-24 14:49:29 - gridExtra | Required: >= 2.3 | Installed: 2.3 | ✅ OK
[INFO] 2025-06-24 14:49:29 - knitr | Required: >= 1.50 | Installed: 1.50 | ✅ OK
[INFO] 2025-06-24 14:49:29 - rmarkdown | Required: >= 2.21.0 | Installed: 2.29 | ✅ OK
[INFO] 2025-06-24 14:49:29 - tidyverse | Required: >= 2.0.0 | Installed: 2.0.0 | ✅ OK
[INFO] 2025-06-24 14:49:29 - caret | Required: >= 7.0.1 | Installed: 7.0.1 | ✅ OK
[INFO] 2025-06-24 14:49:29 - CAST | Required: >= 1.0.3 | Installed: 1.0.3 | ✅ OK
[INFO] 2025-06-24 14:49:29 - randomForest | Required: >= 4.7.1.2 | Installed: 4.7.1.2 | ✅ OK
[INFO] 2025-06-24 14:49:29 - rsample | Required: >= 1.3.0 | Installed: 1.3.0 | ✅ OK
[INFO] 2025-06-24 14:49:29 - furrr | Required: >= 0.3.1 | Installed: 0.3.1 | ✅ OK
[INFO] 2025-06-24 14:49:29 - future | Required: >= 1.40.0 | Installed: 1.40.0 | ✅ OK
[INFO] 2025-06-24 14:49:29 - progressr | Required: >= 0.15.1 | Installed: 0.15.1 | ✅ OK
[INFO] 2025-06-24 14:49:29 - reshape2 | Required: >= 1.4.4 | Installed: 1.4.4 | ✅ OK
[INFO] 2025-06-24 14:49:29 - zoo | Required: >= 1.8.13 | Installed: 1.8.13 | ✅ OK
[INFO] 2025-06-24 14:49:29 - === END PACKAGE REPORT ===
[INFO] 2025-06-24 14:49:29 -
=== PACKAGE INSTALLATION/UPDATES ===
[INFO] 2025-06-24 14:49:29 - === PACKAGE MANAGEMENT STARTED ===
[INFO] 2025-06-24 14:49:29 - R version: R version 4.4.2 (2024-10-31 ucrt)
[INFO] 2025-06-24 14:49:29 - ✓ dplyr version 1.1.4 meets requirement (>= 1.1.4)
[INFO] 2025-06-24 14:49:29 - ✓ here version 1.0.1 meets requirement (>= 1.0.1)
[INFO] 2025-06-24 14:49:29 - ✓ lubridate version 1.9.4 meets requirement (>= 1.9.4)
[INFO] 2025-06-24 14:49:29 - ✓ readr version 2.1.5 meets requirement (>= 2.1.5)
[INFO] 2025-06-24 14:49:29 - ✓ readxl version 1.4.5 meets requirement (>= 1.4.5)
[INFO] 2025-06-24 14:49:29 - ✓ stringr version 1.5.1 meets requirement (>= 1.5.1)
[INFO] 2025-06-24 14:49:29 - ✓ tidyr version 1.3.1 meets requirement (>= 1.3.1)
[INFO] 2025-06-24 14:49:29 - ✓ purrr version 1.0.2 meets requirement (>= 1.0.2)
[INFO] 2025-06-24 14:49:29 - ✓ magrittr version 2.0.3 meets requirement (>= 2.0.0)
[INFO] 2025-06-24 14:49:29 - ✓ exactextractr version 0.10.0 meets requirement (>= 0.10.0)
[INFO] 2025-06-24 14:49:29 - ✓ raster version 3.6.32 meets requirement (>= 3.6.32)
[INFO] 2025-06-24 14:49:29 - ✓ sf version 1.0.19 meets requirement (>= 1.0.19)
[INFO] 2025-06-24 14:49:29 - ✓ terra version 1.8.43 meets requirement (>= 1.8.43)
[INFO] 2025-06-24 14:49:29 - ✓ ggplot2 version 3.5.1 meets requirement (>= 3.5.1)
[INFO] 2025-06-24 14:49:29 - ✓ tmap version 4.0 meets requirement (>= 4.0)
[INFO] 2025-06-24 14:49:29 - ✓ gridExtra version 2.3 meets requirement (>= 2.3)
[INFO] 2025-06-24 14:49:29 - ✓ knitr version 1.50 meets requirement (>= 1.50)
[INFO] 2025-06-24 14:49:29 - ✓ rmarkdown version 2.29 meets requirement (>= 2.21.0)
[INFO] 2025-06-24 14:49:29 - ✓ tidyverse version 2.0.0 meets requirement (>= 2.0.0)
[INFO] 2025-06-24 14:49:29 - ✓ caret version 7.0.1 meets requirement (>= 7.0.1)
[INFO] 2025-06-24 14:49:29 - ✓ CAST version 1.0.3 meets requirement (>= 1.0.3)
[INFO] 2025-06-24 14:49:29 - ✓ randomForest version 4.7.1.2 meets requirement (>= 4.7.1.2)
[INFO] 2025-06-24 14:49:29 - ✓ rsample version 1.3.0 meets requirement (>= 1.3.0)
[INFO] 2025-06-24 14:49:29 - ✓ furrr version 0.3.1 meets requirement (>= 0.3.1)
[INFO] 2025-06-24 14:49:29 - ✓ future version 1.40.0 meets requirement (>= 1.40.0)
[INFO] 2025-06-24 14:49:29 - ✓ progressr version 0.15.1 meets requirement (>= 0.15.1)
[INFO] 2025-06-24 14:49:29 - ✓ reshape2 version 1.4.4 meets requirement (>= 1.4.4)
[INFO] 2025-06-24 14:49:29 - ✓ zoo version 1.8.13 meets requirement (>= 1.8.13)
[INFO] 2025-06-24 14:49:29 - Package management complete: 28 success, 0 failures
[INFO] 2025-06-24 14:49:29 - Updating renv lockfile...
[ERROR] 2025-06-24 14:49:33 - ✗ Failed to update lockfile: aborting snapshot due to pre-flight validation failure
[INFO] 2025-06-24 14:49:33 -
=== FINAL STATE ===
[INFO] 2025-06-24 14:49:33 - === PACKAGE REPORT ===
[INFO] 2025-06-24 14:49:33 - dplyr | Required: >= 1.1.4 | Installed: 1.1.4 | ✅ OK
[INFO] 2025-06-24 14:49:33 - here | Required: >= 1.0.1 | Installed: 1.0.1 | ✅ OK
[INFO] 2025-06-24 14:49:33 - lubridate | Required: >= 1.9.4 | Installed: 1.9.4 | ✅ OK
[INFO] 2025-06-24 14:49:33 - readr | Required: >= 2.1.5 | Installed: 2.1.5 | ✅ OK
[INFO] 2025-06-24 14:49:33 - readxl | Required: >= 1.4.5 | Installed: 1.4.5 | ✅ OK
[INFO] 2025-06-24 14:49:33 - stringr | Required: >= 1.5.1 | Installed: 1.5.1 | ✅ OK
[INFO] 2025-06-24 14:49:33 - tidyr | Required: >= 1.3.1 | Installed: 1.3.1 | ✅ OK
[INFO] 2025-06-24 14:49:33 - purrr | Required: >= 1.0.2 | Installed: 1.0.2 | ✅ OK
[INFO] 2025-06-24 14:49:33 - magrittr | Required: >= 2.0.0 | Installed: 2.0.3 | ✅ OK
[INFO] 2025-06-24 14:49:33 - exactextractr | Required: >= 0.10.0 | Installed: 0.10.0 | ✅ OK
[INFO] 2025-06-24 14:49:33 - raster | Required: >= 3.6.32 | Installed: 3.6.32 | ✅ OK
[INFO] 2025-06-24 14:49:33 - sf | Required: >= 1.0.19 | Installed: 1.0.19 | ✅ OK
[INFO] 2025-06-24 14:49:33 - terra | Required: >= 1.8.43 | Installed: 1.8.43 | ✅ OK
[INFO] 2025-06-24 14:49:33 - ggplot2 | Required: >= 3.5.1 | Installed: 3.5.1 | ✅ OK
[INFO] 2025-06-24 14:49:33 - tmap | Required: >= 4.0 | Installed: 4.0 | ✅ OK
[INFO] 2025-06-24 14:49:33 - gridExtra | Required: >= 2.3 | Installed: 2.3 | ✅ OK
[INFO] 2025-06-24 14:49:33 - knitr | Required: >= 1.50 | Installed: 1.50 | ✅ OK
[INFO] 2025-06-24 14:49:33 - rmarkdown | Required: >= 2.21.0 | Installed: 2.29 | ✅ OK
[INFO] 2025-06-24 14:49:33 - tidyverse | Required: >= 2.0.0 | Installed: 2.0.0 | ✅ OK
[INFO] 2025-06-24 14:49:33 - caret | Required: >= 7.0.1 | Installed: 7.0.1 | ✅ OK
[INFO] 2025-06-24 14:49:33 - CAST | Required: >= 1.0.3 | Installed: 1.0.3 | ✅ OK
[INFO] 2025-06-24 14:49:33 - randomForest | Required: >= 4.7.1.2 | Installed: 4.7.1.2 | ✅ OK
[INFO] 2025-06-24 14:49:33 - rsample | Required: >= 1.3.0 | Installed: 1.3.0 | ✅ OK
[INFO] 2025-06-24 14:49:33 - furrr | Required: >= 0.3.1 | Installed: 0.3.1 | ✅ OK
[INFO] 2025-06-24 14:49:33 - future | Required: >= 1.40.0 | Installed: 1.40.0 | ✅ OK
[INFO] 2025-06-24 14:49:33 - progressr | Required: >= 0.15.1 | Installed: 0.15.1 | ✅ OK
[INFO] 2025-06-24 14:49:33 - reshape2 | Required: >= 1.4.4 | Installed: 1.4.4 | ✅ OK
[INFO] 2025-06-24 14:49:33 - zoo | Required: >= 1.8.13 | Installed: 1.8.13 | ✅ OK
[INFO] 2025-06-24 14:49:33 - === END PACKAGE REPORT ===
[INFO] 2025-06-24 14:49:33 - Package management completed in 7.72 seconds
[INFO] 2025-06-24 14:49:33 - Log saved to: C:/Users/timon/Resilience BV/4020 SCane ESA DEMO - Documenten/General/4020 SCDEMO Team/4020 TechnicalData/WP3/smartcane/package_manager.log
[SUCCESS] 2025-06-24 14:49:33 - 🎉 All packages successfully managed!
[INFO] 2025-06-24 14:49:33 - 📋 Next steps:
[INFO] 2025-06-24 14:49:33 - 1. Test your R scripts to ensure everything works
[INFO] 2025-06-24 14:49:33 - 2. Commit renv.lock to version control
[INFO] 2025-06-24 14:49:33 - 3. Share this script with your team