From bbf2764af978f6c0e0c74550a6683f36acf2d23c Mon Sep 17 00:00:00 2001 From: SysAdmin Date: Sat, 4 Oct 2025 15:13:56 +0100 Subject: [PATCH] Fix: Align deployment configs with production architecture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Critical fixes to ensure smooth deployments and prevent future outages: **docker-compose.yml:** - Fixed image name: littleshop:latest → localhost:5000/littleshop:latest - Fixed subnet: 172.21.0.0/16 → 172.23.0.0/16 (matches production) - Fixed environment: Production → Development (matches current production) **.gitlab-ci.yml:** - Fixed TeleBot API URL: http://littleshop-admin:8080 → http://littleshop:5000 - Removed duplicate network flag (was causing issues) - Added explicit network connection command for littleshop_littleshop-network - Ensures TeleBot can communicate with LittleShop API on deployment **CLAUDE.md:** - Documented October 4, 2025 incident and recovery - Added comprehensive deployment best practices - Documented pre-deployment checklist - Added manual deployment commands for emergencies - Documented network architecture and container configuration **Root Cause of Previous Failure:** TeleBot was trying to connect to non-existent hostname "littleshop-admin" on wrong network, causing authentication failures and data unavailability. **Verification:** All changes tested in production and confirmed working. TeleBot now successfully authenticates and communicates with LittleShop API. 🤖 Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude --- .gitlab-ci.yml | 6 ++- CLAUDE.md | 96 ++++++++++++++++++++++++++++++++++++++++++++++ docker-compose.yml | 6 +-- 3 files changed, 103 insertions(+), 5 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 13cd4e8..07d9ab4 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -132,15 +132,17 @@ deploy:vps: --name telebot-service \ --restart unless-stopped \ --network silverpay_silverpay-network \ - --network littleshop-network \ -e ASPNETCORE_URLS=http://+:5010 \ - -e LittleShop__ApiUrl=http://littleshop-admin:8080 \ + -e LittleShop__ApiUrl=http://littleshop:5000 \ -e LittleShop__UseTor=false \ -e Privacy__EnableTor=true \ -e Privacy__TorSocksHost=tor-gateway \ -e Privacy__TorSocksPort=9050 \ localhost:5000/telebot:latest + # Connect TeleBot to LittleShop network for API access + docker network connect littleshop_littleshop-network telebot-service + # Wait for startup echo "Waiting for services to start..." sleep 30 diff --git a/CLAUDE.md b/CLAUDE.md index fc2ce88..a0960a0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,6 +2,102 @@ > 📋 **See [ROADMAP.md](./ROADMAP.md) for development priorities and security fixes** > 📌 **See [WORKING_BASELINE_2024-09-24.md](./WORKING_BASELINE_2024-09-24.md) for current working configuration** +> ⚠️ **See [Deployment Best Practices](#deployment-best-practices) below for critical deployment requirements** + +## Project Status: ✅ FULLY OPERATIONAL - OCTOBER 4, 2025 + +### 🔧 **CRITICAL INCIDENT RESOLVED (October 4, 2025)** + +**Production Outage & Recovery:** +- **Incident**: Database schema mismatch caused complete system failure +- **Root Cause**: Code deployed without applying database migrations +- **Impact**: 502 errors, TeleBot offline, Product catalog unavailable +- **Resolution**: Database restored from backup, migrations applied, networking fixed +- **Prevention**: Enhanced CI/CD pipeline with automatic migration support + +**Key Lessons Learned:** +1. ❌ **NEVER deploy code changes without corresponding database migrations** +2. ✅ **CI/CD now automatically applies migrations** from `LittleShop/Migrations/*.sql` +3. ✅ **Always verify container networking** (docker-compose prefixes network names) +4. ✅ **Maintain regular database backups** (saved production data) + +### 🚀 **CURRENT PRODUCTION STATUS** + +**All Systems Operational:** +- ✅ **LittleShop API**: Running at `http://littleshop:5000` (internal) / `http://localhost:5100` (host) +- ✅ **TeleBot**: Connected via `littleshop_littleshop-network`, authenticated with API +- ✅ **Nginx Proxy Manager**: Proxying `https://admin.dark.side` → `littleshop:5000` +- ✅ **Database**: SQLite with variant pricing migrations applied (508KB) +- ✅ **Networks**: Proper isolation with `littleshop_littleshop-network` and `silverpay_silverpay-network` + +**Production Configuration:** +- **Server**: srv1002428.hstgr.cloud (31.97.57.205) +- **Container Names**: `littleshop`, `telebot-service`, `nginx-proxy-manager` +- **Docker Networks**: `littleshop_littleshop-network`, `silverpay_silverpay-network` +- **Volume**: `littleshop_littleshop_data` (note the docker-compose prefix!) +- **Database**: `/app/data/littleshop-production.db` inside container + +## Deployment Best Practices + +### **Pre-Deployment Checklist** +1. ✅ Verify all database migrations are in `LittleShop/Migrations/` and committed +2. ✅ Test migrations locally before deploying to production +3. ✅ Ensure docker-compose.yml matches production configuration +4. ✅ Verify TeleBot API URL points to `http://littleshop:5000` (NOT `littleshop-admin:8080`) +5. ✅ Check network names include docker-compose prefix (e.g., `littleshop_littleshop-network`) + +### **CI/CD Pipeline Workflow** +The `.gitlab-ci.yml` pipeline automatically: +1. Builds Docker images with `--no-cache` +2. Copies images to production VPS via SSH +3. Stops running containers +4. **Applies database migrations** (with automatic backup) +5. Starts LittleShop with `docker-compose up -d` +6. Starts TeleBot with correct API URL and network connections +7. Runs health checks on product catalog API + +### **Manual Deployment Commands** (Emergency Use Only) +```bash +# Connect to production server +ssh -i ~/.ssh/littleshop_deploy_key -p 2255 sysadmin@srv1002428.hstgr.cloud + +# Stop services +cd /opt/littleshop +docker stop telebot-service littleshop +docker rm telebot-service + +# Apply migration manually +docker run --rm -v littleshop_littleshop_data:/data -v $(pwd)/LittleShop/Migrations:/migrations alpine sh -c ' + apk add sqlite + sqlite3 /data/littleshop-production.db < /migrations/YourMigration.sql +' + +# Start services +docker-compose up -d +docker run -d --name telebot-service --network silverpay_silverpay-network \ + -e LittleShop__ApiUrl=http://littleshop:5000 localhost:5000/telebot:latest +docker network connect littleshop_littleshop-network telebot-service +``` + +### **Database Migration Format** +Place migration files in `LittleShop/Migrations/` with `.sql` extension: +```sql +-- Migration: Description of changes +-- Date: YYYY-MM-DD + +ALTER TABLE TableName ADD COLUMN NewColumn DataType; +CREATE INDEX IF NOT EXISTS IndexName ON TableName (ColumnName); +``` + +### **Network Architecture** +``` +nginx-proxy-manager ──┐ + │ + ├─── littleshop_littleshop-network ─── littleshop:5000 + │ └── telebot-service + │ +telebot-service ──────┴─── silverpay_silverpay-network ─── tor-gateway +``` ## Project Status: ✅ FULLY OPERATIONAL BASELINE - SEPTEMBER 24, 2024 diff --git a/docker-compose.yml b/docker-compose.yml index 2804c0d..7a1dced 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -3,13 +3,13 @@ version: '3.8' services: littleshop: build: . - image: littleshop:latest + image: localhost:5000/littleshop:latest container_name: littleshop restart: unless-stopped ports: - "127.0.0.1:5100:5000" # Bind only to localhost environment: - - ASPNETCORE_ENVIRONMENT=Production + - ASPNETCORE_ENVIRONMENT=Development - ASPNETCORE_URLS=http://+:5000 - ConnectionStrings__DefaultConnection=Data Source=/app/data/littleshop-production.db - Jwt__Key=${JWT_SECRET_KEY:-7ndUULT7XWE78uxfZ9xO4t6/JhXRzCQ23wCN/R1foDPpb0dv06qe4TuGsRLLV5q+} @@ -55,4 +55,4 @@ networks: driver: bridge ipam: config: - - subnet: 172.21.0.0/16 \ No newline at end of file + - subnet: 172.23.0.0/16 \ No newline at end of file