Timon
b487cc983f
- Renamed translation function from `t` to `tr_key` for clarity and consistency. - Updated all instances of translation calls in `90_CI_report_with_kpis_agronomic_support.Rmd` and `91_CI_report_with_kpis_cane_supply.Rmd` to use `tr_key`. - Introduced a new helper function `get_area_unit_label` to manage area unit preferences across the project. - Modified area calculations in `91_CI_report_with_kpis_cane_supply.Rmd` to utilize area from analysis data instead of recalculating. - Added area unit preference setting in `parameters_project.R` to allow for flexible reporting in either hectares or acres. - Updated `MANUAL_PIPELINE_RUNNER.R` to include language parameter for report generation. - Adjusted translations in the `translations.xlsx` file to reflect changes in the report structure.
1074 lines
30 KiB
Markdown
1074 lines
30 KiB
Markdown
# SmartCane Code-Improvements Merge Guide (Complete)
|
|
|
|
**Target Repo**: https://bitbucket.org/sobitnl/smartcane/src/master/
|
|
**Source Branch**: `code-improvements` (Timon's experimental area)
|
|
**Status**: Ready for merge (all changes, including optional enhancements)
|
|
|
|
---
|
|
|
|
## 📋 SCOPE: CORE + OPTIONAL FEATURES
|
|
|
|
### 🔴 CORE (Required for merge)
|
|
- Database migration (client_type column)
|
|
- File path changes (merged_final_tif → field_tiles_CI)
|
|
- Laravel form/job/mailing updates
|
|
- Shell script wrappers (5 files)
|
|
- Python package files (2 files)
|
|
|
|
### 🟡 OPTIONAL (Post-merge enhancements)
|
|
- Country-based project organization (MZ/UG/TZ folders)
|
|
- Download scheduling (staggered 00:01 per-project)
|
|
- Project search feature
|
|
- Harvest prediction setup docs
|
|
|
|
---
|
|
|
|
## 🚀 FULL IMPLEMENTATION CHECKLIST
|
|
|
|
```
|
|
═══════════════════════════════════════════════════════════════
|
|
PHASE 1: CORE MERGE (Required)
|
|
═══════════════════════════════════════════════════════════════
|
|
|
|
[1.1] Database Migration (10 min)
|
|
✓ Create: database/migrations/YYYY_MM_DD_add_client_type_to_projects_table.php
|
|
✓ Run: php artisan migrate
|
|
|
|
[1.2] Laravel Model Changes (20 min)
|
|
✓ Edit: laravel_app/app/Models/Project.php
|
|
- Add 'client_type' to $fillable
|
|
- Update getMergedTiffList() — path change
|
|
- Update startDownload() — path change
|
|
- Update getTifsAsZip() — path change
|
|
- Add getLatestKpiFile() method
|
|
|
|
[1.3] Laravel Job Changes (15 min)
|
|
✓ Edit: laravel_app/app/Jobs/ProjectDownloadTiffJob.php
|
|
- Change path in handleForDate()
|
|
✓ Edit: laravel_app/app/Jobs/ProjectMosiacGeneratorJob.php
|
|
- Replace command array
|
|
- Improve error handling
|
|
|
|
[1.4] Laravel Forms (15 min)
|
|
✓ Edit: laravel_app/app/Livewire/Projects/ProjectManager.php
|
|
- Add client_type to form
|
|
✓ Edit: Project form Blade template
|
|
- Add <select> for client_type
|
|
✓ Edit: laravel_app/app/Livewire/Forms/MailingForm.php
|
|
- Auto-attach KPI Excel for cane_supply
|
|
|
|
[1.5] Shell Script Wrappers (15 min)
|
|
✓ Create: 10_create_per_field_tiffs.sh
|
|
✓ Create: 21_convert_ci_rds_to_csv.sh
|
|
✓ Create: 22_harvest_baseline_prediction.sh
|
|
✓ Create: 23_convert_harvest_format.sh
|
|
✓ Create: 31_harvest_imminent_weekly.sh
|
|
|
|
[1.6] Python Package Files (5 min)
|
|
✓ Create: python_app/requirements_harvest.txt
|
|
✓ Create: python_app/environment_pytorch.yml
|
|
|
|
[1.7] Testing Core (20 min)
|
|
✓ Run migration ✓ Download test ✓ Mosaic test ✓ Mail test
|
|
|
|
═══════════════════════════════════════════════════════════════
|
|
PHASE 2: ENHANCEMENTS
|
|
═══════════════════════════════════════════════════════════════
|
|
|
|
[2.1] Country Organization (45 min)
|
|
- Add DB migration (country, country_code columns)
|
|
- Update Project.php fillable
|
|
- Add ProjectManager form fields
|
|
- Add Blade country selector + auto-populate
|
|
- Add country filtering to ProjectList
|
|
- Create "Add New Country" feature
|
|
|
|
[2.2] Download Scheduling (30 min)
|
|
- Option A: Windows Task Scheduler
|
|
- Option B: Linux cron
|
|
- Option C: Laravel Task Scheduler
|
|
|
|
[2.3] Project Search (30 min)
|
|
- Add search/filter to ProjectList.php
|
|
- Add Blade input fields
|
|
|
|
[2.4] Harvest Prediction Setup (20 min)
|
|
- Document conda env setup
|
|
- Document script execution
|
|
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 🔴 PHASE 1: CORE MERGE
|
|
|
|
### STEP 1: Create & Run Database Migration
|
|
|
|
**File**: `database/migrations/2024_02_19_000000_add_client_type_to_projects_table.php`
|
|
|
|
```php
|
|
<?php
|
|
|
|
use Illuminate\Database\Migrations\Migration;
|
|
use Illuminate\Database\Schema\Blueprint;
|
|
use Illuminate\Support\Facades\Schema;
|
|
|
|
return new class extends Migration
|
|
{
|
|
public function up(): void
|
|
{
|
|
Schema::table('projects', function (Blueprint $table) {
|
|
$table->enum('client_type', ['agronomic_support', 'cane_supply'])
|
|
->default('agronomic_support')
|
|
->after('download_path');
|
|
});
|
|
}
|
|
|
|
public function down(): void
|
|
{
|
|
Schema::table('projects', function (Blueprint $table) {
|
|
$table->dropColumn('client_type');
|
|
});
|
|
}
|
|
};
|
|
```
|
|
|
|
**Run migration:**
|
|
```bash
|
|
php artisan migrate
|
|
```
|
|
|
|
---
|
|
|
|
### STEP 2: Edit `laravel_app/app/Models/Project.php`
|
|
|
|
**Edit 2.1: Add to `$fillable` array**
|
|
|
|
Find and add `'client_type'` after `'download_path'`:
|
|
|
|
```php
|
|
protected $fillable = [
|
|
'name',
|
|
'download_path',
|
|
'client_type', // ADD THIS
|
|
'mail_template',
|
|
'mail_subject',
|
|
'mail_frequency',
|
|
'mail_day',
|
|
'mail_scheduled',
|
|
'pivot_json_path',
|
|
'span_json_path',
|
|
'harvest_json_path',
|
|
'min_harvest_date',
|
|
'borders',
|
|
];
|
|
```
|
|
|
|
**Edit 2.2: Update `getMergedTiffList()` method (around line 232)**
|
|
|
|
```php
|
|
public function getMergedTiffList()
|
|
{
|
|
return collect(Storage::files($this->download_path.'/field_tiles_CI')) // CHANGED
|
|
->filter(fn($file) => Str::endsWith($file, '.tif'))
|
|
->sortByDesc(function ($file) {
|
|
$parts = explode('_', str_replace('.tif', '', $file));
|
|
$date = $parts[1];
|
|
return $date;
|
|
})
|
|
->values();
|
|
}
|
|
```
|
|
|
|
**Edit 2.3: Update `startDownload()` method (around line 265)**
|
|
|
|
```php
|
|
public function startDownload(Carbon $date)
|
|
{
|
|
$downloadRequest = $this->downloads()->updateOrCreate(
|
|
[
|
|
'project_id' => $this->id,
|
|
'name' => sprintf('%s.tif', $date->format('Y-m-d')),
|
|
],
|
|
[
|
|
'path' => sprintf('%s/%s/%s.tif', $this->download_path, 'field_tiles_CI', $date->format('Y-m-d')), // CHANGED
|
|
]
|
|
);
|
|
ProjectDownloadTiffJob::dispatch($downloadRequest, $date);
|
|
}
|
|
```
|
|
|
|
**Edit 2.4: Update `getTifsAsZip()` method (around line 489)**
|
|
|
|
```php
|
|
public function getTifsAsZip()
|
|
{
|
|
return collect(Storage::files($this->download_path . '/field_tiles_CI')) // CHANGED
|
|
->filter(fn($file) => Str::endsWith($file, '.tif'))
|
|
->values();
|
|
}
|
|
```
|
|
|
|
**Edit 2.5: Add new method `getLatestKpiFile()`**
|
|
|
|
Add this method at the end of the Project class:
|
|
|
|
```php
|
|
public function getLatestKpiFile(): ?string
|
|
{
|
|
$kpiPath = $this->download_path . '/reports/kpis/';
|
|
|
|
try {
|
|
$files = Storage::files($kpiPath);
|
|
} catch (\Exception $e) {
|
|
return null;
|
|
}
|
|
|
|
return collect($files)
|
|
->filter(fn($f) => Str::endsWith($f, '.xlsx'))
|
|
->sortByDesc(fn($f) => Storage::lastModified($f))
|
|
->first();
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### STEP 3: Edit `laravel_app/app/Jobs/ProjectDownloadTiffJob.php`
|
|
|
|
**Around line 73 in `handleForDate()` method:**
|
|
|
|
Change:
|
|
```php
|
|
$path = $project->download_path . '/merged_final_tif/' . $filename;
|
|
```
|
|
|
|
To:
|
|
```php
|
|
$path = $project->download_path . '/field_tiles_CI/' . $filename;
|
|
```
|
|
|
|
---
|
|
|
|
### STEP 4: Edit `laravel_app/app/Jobs/ProjectMosiacGeneratorJob.php`
|
|
|
|
**Lines 50-70, replace the `$command` array initialization:**
|
|
|
|
OLD:
|
|
```php
|
|
$command = [
|
|
sprintf('%sbuild_mosaic.sh', $projectFolder),
|
|
sprintf('--end_date=%s', $this->mosaic->end_date->format('Y-m-d')),
|
|
sprintf('--offset=%s', $this->mosaic->offset),
|
|
sprintf('--data_dir=%s', $this->mosaic->project->download_path),
|
|
sprintf('--file_name_tif=%s', basename($this->mosaic->path)),
|
|
];
|
|
```
|
|
|
|
NEW:
|
|
```php
|
|
$command = [
|
|
sprintf('%s40_mosaic_creation_per_field.sh', $projectFolder),
|
|
sprintf('--project=%s', $project->name),
|
|
sprintf('--end_date=%s', $this->mosaic->end_date->format('Y-m-d')),
|
|
sprintf('--offset=%s', $this->mosaic->offset),
|
|
];
|
|
```
|
|
|
|
**Also improve exception handling (around line 65-75):**
|
|
|
|
```php
|
|
try {
|
|
$process = ProcessNew::timeout(300)
|
|
->env(['PATH' => $currentPath.':/usr/local/Cellar/pandoc/3.1.8/bin/pandoc'])
|
|
->start($command, function (string $type, string $output) use ($project) {
|
|
ProjectLogger::log($project, $output);
|
|
$this->throwIfOutputContainsError($output);
|
|
});
|
|
$results = $process->wait();
|
|
if ($results->successful()) {
|
|
$this->mosaic->setStatusSuccess();
|
|
}
|
|
} catch (\RuntimeException|ProcessTimedOutException|ProcessFailedException $e) {
|
|
ProjectLogger::log($project, "MOSAIC JOB ERROR: " . $e->getMessage());
|
|
$this->mosaic->setStatusFailed();
|
|
throw $e;
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### STEP 5: Edit `laravel_app/app/Livewire/Projects/ProjectManager.php`
|
|
|
|
**In `createProject()` method:**
|
|
|
|
```php
|
|
public function createProject()
|
|
{
|
|
$projectIdentifier = $this->formData['id'] ?? null;
|
|
Validator::make(
|
|
['name' => $this->formData['name']],
|
|
['name' => ['required', Rule::unique('projects')->ignore($projectIdentifier), 'string', 'max:255']]
|
|
)->validate();
|
|
|
|
$project = Project::create([
|
|
'name' => $this->formData['name'],
|
|
'download_path' => $this->makeValidDirectoryName($this->formData['name']),
|
|
'client_type' => $this->formData['client_type'] ?? 'agronomic_support', // ADD THIS
|
|
]);
|
|
return redirect()->route('project.show', [$project->name, 'settings']);
|
|
}
|
|
```
|
|
|
|
**In `resetFormData()` method:**
|
|
|
|
```php
|
|
private function resetFormData()
|
|
{
|
|
$this->formData = [
|
|
'name' => '',
|
|
'client_type' => 'agronomic_support', // ADD THIS
|
|
'mail_template' => '',
|
|
'mail_subject' => '',
|
|
'mail_frequency' => '',
|
|
'mail_day' => '',
|
|
// ... rest of fields
|
|
];
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### STEP 6: Edit Project Form Blade Template
|
|
|
|
**Find the project create/edit form and add this field:**
|
|
|
|
```blade
|
|
<div class="form-group mb-3">
|
|
<label for="client_type" class="form-label">Client Type <span class="text-danger">*</span></label>
|
|
<select wire:model="formData.client_type" id="client_type" class="form-control" required>
|
|
<option value="">-- Select Client Type --</option>
|
|
<option value="agronomic_support">Agronomic Support</option>
|
|
<option value="cane_supply">Cane Supply</option>
|
|
</select>
|
|
@error('formData.client_type')
|
|
<span class="text-danger small">{{ $message }}</span>
|
|
@enderror
|
|
</div>
|
|
```
|
|
|
|
---
|
|
|
|
### STEP 7: Edit `laravel_app/app/Livewire/Forms/MailingForm.php`
|
|
|
|
**In `saveAndSendMailing()` static method:**
|
|
|
|
```php
|
|
public static function saveAndSendMailing($report, $subject, $message, $recipients) {
|
|
if ($report->documentExists()) {
|
|
$mailing = $report->project->mailings()->create([
|
|
'subject' => $subject,
|
|
'message' => $message,
|
|
'report_id' => $report->id,
|
|
]);
|
|
|
|
// Attach main report
|
|
$mailing->attachments()->create([
|
|
'name' => $report->name,
|
|
'path' => $report->path,
|
|
]);
|
|
|
|
// Auto-attach KPI Excel for cane_supply projects
|
|
if ($report->project->client_type === 'cane_supply') {
|
|
$kpiFile = $report->project->getLatestKpiFile();
|
|
if ($kpiFile) {
|
|
$mailing->attachments()->create([
|
|
'name' => 'KPI Summary',
|
|
'path' => $kpiFile,
|
|
]);
|
|
}
|
|
}
|
|
|
|
$mailing->recipients()->createMany($recipients);
|
|
Mail::to($mailing->recipients()->pluck('email')->toArray())
|
|
->send(new \App\Mail\ReportMailer($mailing, $report));
|
|
} else {
|
|
self::sendReportNotFoundNotificationToAdmin($report);
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### STEP 8: CREATE 5 Shell Script Wrappers
|
|
|
|
**File: `10_create_per_field_tiffs.sh`**
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# Wrapper for R script 10: Create per-field TIFFs
|
|
# Usage: ./10_create_per_field_tiffs.sh --project=angata
|
|
|
|
set -e
|
|
|
|
PROJECT=""
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case $1 in
|
|
--project=*) PROJECT="${1#*=}" ;;
|
|
--*) ;; # Ignore other args
|
|
esac
|
|
shift
|
|
done
|
|
|
|
[ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; }
|
|
|
|
cd "$(dirname "$0")/r_app"
|
|
Rscript -e "PROJECT='$PROJECT'; source('parameters_project.R'); source('10_create_per_field_tiffs.R')"
|
|
```
|
|
|
|
**File: `21_convert_ci_rds_to_csv.sh`**
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# Wrapper for R script 21: Convert CI RDS to CSV
|
|
# Usage: ./21_convert_ci_rds_to_csv.sh --project=angata
|
|
|
|
set -e
|
|
|
|
PROJECT=""
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case $1 in
|
|
--project=*) PROJECT="${1#*=}" ;;
|
|
--*) ;;
|
|
esac
|
|
shift
|
|
done
|
|
|
|
[ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; }
|
|
|
|
cd "$(dirname "$0")/r_app"
|
|
Rscript -e "PROJECT='$PROJECT'; source('parameters_project.R'); source('21_convert_ci_rds_to_csv.R')"
|
|
```
|
|
|
|
**File: `22_harvest_baseline_prediction.sh`**
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# Wrapper for Python script 22: Harvest baseline prediction
|
|
# Usage: ./22_harvest_baseline_prediction.sh --project=angata
|
|
|
|
set -e
|
|
|
|
PROJECT=""
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case $1 in
|
|
--project=*) PROJECT="${1#*=}" ;;
|
|
--*) ;;
|
|
esac
|
|
shift
|
|
done
|
|
|
|
[ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; }
|
|
|
|
cd "$(dirname "$0")/python_app"
|
|
|
|
if command -v conda &> /dev/null; then
|
|
conda run -n pytorch_gpu python 22_harvest_baseline_prediction.py "$PROJECT" 2>&1 || \
|
|
conda run -n pytorch_cpu python 22_harvest_baseline_prediction.py "$PROJECT" 2>&1
|
|
else
|
|
python 22_harvest_baseline_prediction.py "$PROJECT"
|
|
fi
|
|
```
|
|
|
|
**File: `23_convert_harvest_format.sh`**
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# Wrapper for Python script 23: Convert harvest format
|
|
# Usage: ./23_convert_harvest_format.sh --project=angata
|
|
|
|
set -e
|
|
|
|
PROJECT=""
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case $1 in
|
|
--project=*) PROJECT="${1#*=}" ;;
|
|
--*) ;;
|
|
esac
|
|
shift
|
|
done
|
|
|
|
[ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; }
|
|
|
|
cd "$(dirname "$0")/python_app"
|
|
|
|
if command -v conda &> /dev/null; then
|
|
conda run -n pytorch_gpu python 23_convert_harvest_format.py "$PROJECT" 2>&1 || \
|
|
conda run -n pytorch_cpu python 23_convert_harvest_format.py "$PROJECT" 2>&1
|
|
else
|
|
python 23_convert_harvest_format.py "$PROJECT"
|
|
fi
|
|
```
|
|
|
|
**File: `31_harvest_imminent_weekly.sh`**
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# Wrapper for Python script 31: Harvest imminent weekly
|
|
# Usage: ./31_harvest_imminent_weekly.sh --project=angata
|
|
|
|
set -e
|
|
|
|
PROJECT=""
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case $1 in
|
|
--project=*) PROJECT="${1#*=}" ;;
|
|
--*) ;;
|
|
esac
|
|
shift
|
|
done
|
|
|
|
[ -z "$PROJECT" ] && { echo "ERROR: --project required"; exit 1; }
|
|
|
|
cd "$(dirname "$0")/python_app"
|
|
|
|
if command -v conda &> /dev/null; then
|
|
conda run -n pytorch_gpu python 31_harvest_imminent_weekly.py "$PROJECT" 2>&1 || \
|
|
conda run -n pytorch_cpu python 31_harvest_imminent_weekly.py "$PROJECT" 2>&1
|
|
else
|
|
python 31_harvest_imminent_weekly.py "$PROJECT"
|
|
fi
|
|
```
|
|
|
|
---
|
|
|
|
### STEP 9: CREATE 2 Python Package Files
|
|
|
|
**File: `python_app/requirements_harvest.txt`**
|
|
|
|
```
|
|
torch>=2.0.0
|
|
pandas>=1.5.0
|
|
numpy>=1.23.0
|
|
scikit-learn>=1.3.0
|
|
GDAL>=3.7.0
|
|
sentinelhub>=3.9.0
|
|
shapely>=2.0.0
|
|
pyproj>=3.4.0
|
|
```
|
|
|
|
**File: `python_app/environment_pytorch.yml`**
|
|
|
|
```yaml
|
|
name: pytorch_gpu
|
|
channels:
|
|
- pytorch
|
|
- nvidia
|
|
- conda-forge
|
|
dependencies:
|
|
- python=3.10
|
|
- pytorch::pytorch
|
|
- pytorch::torchvision
|
|
- pytorch::torchaudio
|
|
- pytorch::pytorch-cuda=11.8
|
|
- gdal>=3.7.0
|
|
- pip
|
|
- pip:
|
|
- sentinelhub>=3.9.0
|
|
- shapely>=2.0.0
|
|
- pyproj>=3.4.0
|
|
```
|
|
|
|
---
|
|
|
|
### STEP 10: CORE TESTING CHECKLIST
|
|
|
|
```bash
|
|
# 1. Migration
|
|
php artisan migrate
|
|
# ✅ Expected: No errors, client_type column added
|
|
|
|
# 2. Download test
|
|
# Go to Laravel UI → Create project with client_type=agronomic_support
|
|
# → Download Manager → Add image → Download
|
|
# Expected: File in laravel_app/storage/app/{project}/field_tiles_CI/
|
|
|
|
# 3. Mosaic test
|
|
# Go to Mosaic Manager → Create mosaic
|
|
# Check logs: grep "Unknown option" laravel.log
|
|
# Expected: No --data_dir errors, mosaic created
|
|
|
|
# 4. Mail test
|
|
# Create project with client_type=cane_supply
|
|
# Generate & send report
|
|
# Expected: Email has 2 attachments (report + KPI Excel)
|
|
|
|
# 5. Shell wrapper test
|
|
./10_create_per_field_tiffs.sh --project=angata
|
|
# Expected: R script executes without error
|
|
```
|
|
|
|
✅ **CORE MERGE COMPLETE**
|
|
|
|
---
|
|
|
|
## 🟡 PHASE 2: ENHANCEMENTS (Post-Merge)
|
|
|
|
### OPTIONAL 1: Country-Based Organization
|
|
|
|
**Why**: Organize projects by geographic location (MZ/UG/TZ folders)
|
|
|
|
**Create Migration**: `database/migrations/YYYY_MM_DD_add_country_to_projects_table.php`
|
|
|
|
```php
|
|
<?php
|
|
|
|
use Illuminate\Database\Migrations\Migration;
|
|
use Illuminate\Database\Schema\Blueprint;
|
|
use Illuminate\Support\Facades\Schema;
|
|
use Illuminate\Support\Facades\DB;
|
|
|
|
return new class extends Migration
|
|
{
|
|
public function up(): void
|
|
{
|
|
Schema::table('projects', function (Blueprint $table) {
|
|
$table->string('country')->default('Mozambique')->after('name');
|
|
$table->string('country_code', 2)->default('MZ')->after('country');
|
|
});
|
|
|
|
// Update existing projects
|
|
DB::table('projects')->where('name', 'angata')->update(['country' => 'Mozambique', 'country_code' => 'MZ']);
|
|
DB::table('projects')->where('name', 'aura')->update(['country' => 'Mozambique', 'country_code' => 'MZ']);
|
|
DB::table('projects')->where('name', 'chemba')->update(['country' => 'Mozambique', 'country_code' => 'MZ']);
|
|
DB::table('projects')->where('name', 'xinavane')->update(['country' => 'Tanzania', 'country_code' => 'TZ']);
|
|
DB::table('projects')->where('name', 'esa')->update(['country' => 'Kenya', 'country_code' => 'KQ']);
|
|
DB::table('projects')->where('name', 'simba')->update(['country' => 'Uganda', 'country_code' => 'UG']);
|
|
DB::table('projects')->where('name', 'john')->update(['country' => 'Uganda', 'country_code' => 'UG']);
|
|
DB::table('projects')->where('name', 'huss')->update(['country' => 'Tanzania', 'country_code' => 'TZ']);
|
|
}
|
|
|
|
public function down(): void
|
|
{
|
|
Schema::table('projects', function (Blueprint $table) {
|
|
$table->dropColumn(['country', 'country_code']);
|
|
});
|
|
}
|
|
};
|
|
```
|
|
|
|
**Update Project.php `$fillable`:**
|
|
|
|
```php
|
|
protected $fillable = [
|
|
'name',
|
|
'country',
|
|
'country_code',
|
|
'download_path',
|
|
'client_type',
|
|
// ... rest
|
|
];
|
|
```
|
|
|
|
**Update ProjectManager.php:**
|
|
|
|
```php
|
|
public array $countries = [
|
|
'MZ' => 'Mozambique',
|
|
'TZ' => 'Tanzania',
|
|
'UG' => 'Uganda',
|
|
'KQ' => 'Kenya',
|
|
'SA' => 'South Africa',
|
|
'ZW' => 'Zimbabwe',
|
|
'BR' => 'Brazil',
|
|
'MX' => 'Mexico',
|
|
'IN' => 'India',
|
|
];
|
|
|
|
public function createProject()
|
|
{
|
|
$projectIdentifier = $this->formData['id'] ?? null;
|
|
Validator::make(
|
|
['name' => $this->formData['name']],
|
|
['name' => ['required', Rule::unique('projects')->ignore($projectIdentifier)]]
|
|
)->validate();
|
|
|
|
$projectPath = $this->formData['country_code'] . '/' . $this->formData['name'];
|
|
Storage::makeDirectory($projectPath, recursive: true);
|
|
|
|
$project = Project::create([
|
|
'name' => $this->formData['name'],
|
|
'country' => $this->formData['country'],
|
|
'country_code' => $this->formData['country_code'],
|
|
'download_path' => $projectPath,
|
|
'client_type' => $this->formData['client_type'] ?? 'agronomic_support',
|
|
]);
|
|
return redirect()->route('project.show', [$project->name, 'settings']);
|
|
}
|
|
```
|
|
|
|
**Add to Blade template:**
|
|
|
|
```blade
|
|
<div class="form-group mb-3">
|
|
<label for="country" class="form-label">Country</label>
|
|
<select wire:model="formData.country" id="country" class="form-control" required>
|
|
@foreach($countries as $code => $name)
|
|
<option value="{{ $name }}" @selected($formData['country'] === $name)>
|
|
{{ $name }} ({{ $code }})
|
|
</option>
|
|
@endforeach
|
|
</select>
|
|
</div>
|
|
|
|
<div class="form-group mb-3">
|
|
<label for="country_code" class="form-label">Country Code (Auto-populated)</label>
|
|
<input type="text" wire:model="formData.country_code" id="country_code" readonly class="form-control" />
|
|
</div>
|
|
```
|
|
|
|
---
|
|
|
|
### OPTIONAL 2: Download Scheduling
|
|
|
|
**Why**: Avoid API rate limits by staggering downloads per project at 00:01
|
|
|
|
#### Option A: Linux Cron (If server is Linux)
|
|
|
|
**Add to `/etc/cron.d/smartcane_downloads`:**
|
|
|
|
```bash
|
|
# Stagger downloads by 10 minutes per project
|
|
1 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py angata 2>&1 | logger
|
|
15 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py chemba 2>&1 | logger
|
|
25 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py xinavane 2>&1 | logger
|
|
35 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py esa 2>&1 | logger
|
|
45 0 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py simba 2>&1 | logger
|
|
0 1 * * * /usr/bin/python /home/user/smartcane/python_app/00_download_8band_pu_optimized.py aura 2>&1 | logger
|
|
```
|
|
|
|
#### Option B: Windows Task Scheduler (If server is Windows)
|
|
|
|
```powershell
|
|
# Create task for each project
|
|
$taskName = "SmartCane-Download-angata"
|
|
$action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-NoProfile -WindowStyle Hidden -Command python C:\smartcane\python_app\00_download_8band_pu_optimized.py angata"
|
|
$trigger = New-ScheduledTaskTrigger -Daily -At 00:01
|
|
Register-ScheduledTask -TaskName $taskName -Action $action -Trigger $trigger -RunLevel Highest
|
|
```
|
|
|
|
#### Option C: Laravel Task Scheduler
|
|
|
|
**Add to `laravel_app/app/Console/Kernel.php`:**
|
|
|
|
```php
|
|
protected function schedule(Schedule $schedule)
|
|
{
|
|
$schedule->exec('python python_app/00_download_8band_pu_optimized.py angata')
|
|
->dailyAt('00:01');
|
|
$schedule->exec('python python_app/00_download_8band_pu_optimized.py chemba')
|
|
->dailyAt('00:15');
|
|
$schedule->exec('python python_app/00_download_8band_pu_optimized.py xinavane')
|
|
->dailyAt('00:25');
|
|
$schedule->exec('python python_app/00_download_8band_pu_optimized.py esa')
|
|
->dailyAt('00:35');
|
|
$schedule->exec('python python_app/00_download_8band_pu_optimized.py simba')
|
|
->dailyAt('00:45');
|
|
$schedule->exec('python python_app/00_download_8band_pu_optimized.py aura')
|
|
->dailyAt('01:00');
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### OPTIONAL 3: Project Search Feature
|
|
|
|
**Why**: Find projects quickly if there are many
|
|
|
|
**Add to `laravel_app/app/Livewire/Projects/ProjectList.php`:**
|
|
|
|
```php
|
|
public string $searchQuery = '';
|
|
|
|
public function getProjectsProperty()
|
|
{
|
|
$query = Project::query();
|
|
|
|
if (!empty($this->searchQuery)) {
|
|
$query->where('name', 'like', '%' . $this->searchQuery . '%')
|
|
->orWhere('download_path', 'like', '%' . $this->searchQuery . '%');
|
|
}
|
|
|
|
return $query->orderBy('name')->paginate(15);
|
|
}
|
|
```
|
|
|
|
**Add to Blade template:**
|
|
|
|
```blade
|
|
<div class="search-bar mb-4">
|
|
<input type="text"
|
|
wire:model.live="searchQuery"
|
|
placeholder="Search projects..."
|
|
class="form-control" />
|
|
</div>
|
|
|
|
<div class="projects-list">
|
|
@forelse($this->projects as $project)
|
|
<div class="project-card">
|
|
<h3>{{ $project->name }}</h3>
|
|
<p>{{ $project->download_path }}</p>
|
|
<span class="badge badge-info">{{ $project->client_type }}</span>
|
|
</div>
|
|
@empty
|
|
<p>No projects found</p>
|
|
@endforelse
|
|
</div>
|
|
|
|
{{ $this->projects->links() }}
|
|
```
|
|
|
|
---
|
|
|
|
### OPTIONAL 4: Harvest Date Prediction Setup
|
|
|
|
**Why**: Enable harvest Date forecasting for cane_supply projects
|
|
|
|
**Create conda environment:**
|
|
|
|
```bash
|
|
conda env create -f python_app/environment_pytorch.yml
|
|
|
|
# Activate
|
|
conda activate pytorch_gpu
|
|
|
|
# Or CPU-only if no GPU
|
|
conda create -n pytorch_cpu python=3.10 pytorch::pytorch torchvision torchaudio -c pytorch
|
|
conda activate pytorch_cpu
|
|
```
|
|
|
|
**Run baseline prediction (once):**
|
|
|
|
```bash
|
|
python python_app/22_harvest_baseline_prediction.py angata
|
|
python python_app/23_convert_harvest_format.py angata
|
|
```
|
|
|
|
**Schedule weekly prediction:**
|
|
|
|
```bash
|
|
# Add to cron (Linux)
|
|
0 23 * * 0 conda run -n pytorch_gpu python /home/user/smartcane/python_app/31_harvest_imminent_weekly.py angata 2>&1 | logger
|
|
|
|
# Or Task Scheduler (Windows)
|
|
# Similar to download scheduling above, but Sunday 23:00
|
|
```
|
|
|
|
---
|
|
|
|
### OPTIONAL 5: Area Unit Selection UI (Hectares vs Acres)
|
|
|
|
**Why**: Allow projects to choose their preferred area unit (hectares or acres) in reports and dashboards
|
|
|
|
**Status**: R/Python business logic complete. Database schema + UI implementation pending.
|
|
|
|
**What's already done** (in this codebase):
|
|
- ✅ Unified area calculation function in `80_utils_common.R`: `calculate_area_from_geometry()`
|
|
- ✅ Area unit preference in `parameters_project.R`: `AREA_UNIT_PREFERENCE` (default: "hectare")
|
|
- ✅ Helper function: `get_area_unit_label()` for dynamic "ha" or "ac" display
|
|
- ✅ Refactored scripts 80/90/91 to use unified function and support user's area preference
|
|
- ✅ Area now included in KPI outputs (CSV/RDS/Excel) from script 80
|
|
- ✅ Scripts 90/91 read area from KPI files instead of recalculating
|
|
|
|
**What needs implementation (for your Laravel colleague)**:
|
|
|
|
**Step 1: Create database migration**
|
|
|
|
```php
|
|
<?php
|
|
|
|
use Illuminate\Database\Migrations\Migration;
|
|
use Illuminate\Database\Schema\Blueprint;
|
|
use Illuminate\Support\Facades\Schema;
|
|
|
|
return new class extends Migration
|
|
{
|
|
public function up(): void
|
|
{
|
|
Schema::table('projects', function (Blueprint $table) {
|
|
$table->enum('preferred_area_unit', ['hectare', 'acre'])
|
|
->default('hectare')
|
|
->after('client_type');
|
|
});
|
|
}
|
|
|
|
public function down(): void
|
|
{
|
|
Schema::table('projects', function (Blueprint $table) {
|
|
$table->dropColumn('preferred_area_unit');
|
|
});
|
|
}
|
|
};
|
|
```
|
|
|
|
**Step 2: Update Project model** (`laravel_app/app/Models/Project.php`)
|
|
|
|
```php
|
|
protected $fillable = [
|
|
// ... existing fields ...
|
|
'preferred_area_unit', // ADD THIS
|
|
];
|
|
```
|
|
|
|
**Step 3: Add form UI** (`laravel_app/app/Livewire/Projects/ProjectManager.php` or Blade template)
|
|
|
|
```blade
|
|
<div class="form-group mb-3">
|
|
<label for="preferred_area_unit" class="form-label">Area Unit Preference <span class="text-danger">*</span></label>
|
|
<div class="btn-group btn-group-sm" role="group">
|
|
<input type="radio" class="btn-check" name="preferred_area_unit"
|
|
id="unit_hectare" value="hectare"
|
|
wire:model="formData.preferred_area_unit" />
|
|
<label class="btn btn-outline-primary" for="unit_hectare">Hectares (ha)</label>
|
|
|
|
<input type="radio" class="btn-check" name="preferred_area_unit"
|
|
id="unit_acre" value="acre"
|
|
wire:model="formData.preferred_area_unit" />
|
|
<label class="btn btn-outline-primary" for="unit_acre">Acres (ac)</label>
|
|
</div>
|
|
@error('formData.preferred_area_unit')
|
|
<span class="text-danger small">{{ $message }}</span>
|
|
@enderror
|
|
</div>
|
|
```
|
|
|
|
**Step 4: Pass preference to R scripts** (in job/shell wrapper)
|
|
|
|
When launching R scripts, read `project->preferred_area_unit` and either:
|
|
- **Option A**: Write to `parameters_project.R` dynamically before script execution
|
|
- **Option B**: Pass as environment variable to scripts (R scripts read `Sys.getenv("AREA_UNIT")`)
|
|
|
|
Example (PowerShell wrapper):
|
|
```powershell
|
|
$areaUnit = $project->preferred_area_unit # From database
|
|
$env:AREA_UNIT = $areaUnit # Set environment variable
|
|
& "C:\Program Files\R\R-4.4.3\bin\x64\Rscript.exe" r_app/80_calculate_kpis.R $project
|
|
```
|
|
|
|
**Testing checklist**:
|
|
- [ ] Database migration runs successfully
|
|
- [ ] Project form shows area unit radio buttons/dropdown
|
|
- [ ] Can select and save area unit preference
|
|
- [ ] Area unit persists in database
|
|
- [ ] Run script 80 with one project set to "hectare", observe KPI output
|
|
- [ ] Run script 80 with another project set to "acre", compare outputs
|
|
- [ ] Reports (scripts 90/91) display area in user's chosen unit
|
|
|
|
**Notes**:
|
|
- Default preference: "hectare" (metric standard)
|
|
- Conversion factor used: 0.404686 (1 hectare = 0.404686 acres)
|
|
- All area calculations use EPSG:6933 (equal-area projection) for accuracy
|
|
- Area column in KPI outputs named dynamically: "Area_ha" or "Area_ac"
|
|
|
|
---
|
|
|
|
## 📊 Summary: What Gets Changed
|
|
|
|
| Category | Files Modified | Changes Required |
|
|
|----------|---|---|
|
|
| Database | migrations/ | 1 file: add client_type column |
|
|
| Models | Project.php | 5 edits: fillable, 3 methods, 1 new method |
|
|
| Jobs | 2 files | ProjectDownloadTiffJob (1 line), ProjectMosiacGeneratorJob (full array) |
|
|
| Forms | 3 files | ProjectManager.php, Blade template, MailingForm.php |
|
|
| Scripts | 5 files created | Shell wrappers (R/Python) |
|
|
| Python | 2 files created | requirements_harvest.txt, environment_pytorch.yml |
|
|
|
|
**TOTAL**: 6 files created + 6 files modified + 1 template modified = **13 changes**
|
|
|
|
---
|
|
|
|
## ✅ FINAL VERIFICATION
|
|
|
|
After ALL changes (core + optionals), test:
|
|
|
|
```bash
|
|
# 1. Migration worked
|
|
php artisan migrate
|
|
|
|
# 2. Download saves to correct path
|
|
# → Download image → check laravel_app/storage/app/{project}/field_tiles_CI/
|
|
|
|
# 3. Mosaic runs without errors
|
|
# → Create mosaic → check logs for no --data_dir errors
|
|
|
|
# 4. Mail has 2 attachments for cane_supply
|
|
# → Send report for cane_supply project → verify report + KPI Excel
|
|
|
|
# 5. Shell wrappers work
|
|
./10_create_per_field_tiffs.sh --project=angata
|
|
# → Should execute R script successfully
|
|
|
|
# 6. Search works (if implemented)
|
|
# → Search for project by name on Projects page
|
|
|
|
# 7. Country filter works (if implemented)
|
|
# → Filter projects by country code
|
|
```
|
|
|
|
---
|
|
|
|
## 🌍 POST-MERGE: Data Recreation Strategy
|
|
|
|
After merge is live, existing projects need new directory structure.
|
|
|
|
### Option A: Delete & Redownload (Small projects)
|
|
**Projects**: aura, chemba, xinavane, esa, simba
|
|
|
|
```
|
|
1. Backup project folder (optional)
|
|
2. Delete project from Laravel UI
|
|
3. Recreate with new client_type selector
|
|
4. Redownload 2-3 years of data (~50-150 GB per project)
|
|
5. Run pipeline normally
|
|
```
|
|
|
|
### Option B: Preserve merged_tif (Large projects)
|
|
**Projects**: angata
|
|
|
|
```
|
|
1. Backup merged_tif/ folder externally
|
|
2. Delete all other folders in project
|
|
3. Keep only: merged_tif/
|
|
4. Run Scripts 10-80 on existing data
|
|
→ Regenerates field_tiles_CI/, reports/, etc.
|
|
5. No need to redownload
|
|
```
|
|
|
|
---
|
|
|
|
## 🔴 CORE vs OPTIONAL Quick List
|
|
|
|
**MUST DO** (for merge):
|
|
- ✅ Database migration
|
|
- ✅ Project.php edits (4 path changes + 1 new method)
|
|
- ✅ Job edits (ProjectDownloadTiffJob, ProjectMosiacGeneratorJob)
|
|
- ✅ Form edits (ProjectManager, MailingForm, Blade template)
|
|
- ✅ 5 shell wrappers
|
|
- ✅ 2 Python files
|
|
|
|
**NICE-TO-HAVE** (post-merge):
|
|
- 🟡 Country organization (adds ~45 min)
|
|
- 🟡 Download scheduling (adds ~30 min)
|
|
- 🟡 Project search (adds ~30 min)
|
|
- 🟡 Harvest prediction setup (adds ~20 min)
|
|
|
|
---
|
|
|
|
**Ready to implement everything?** All code is copy-paste ready above.
|
|
|