littleshop/Hostinger/memoires.txt

================================================================================
                        BTCPAY SERVER DEPLOYMENT MEMOIRES
================================================================================
Project: LittleShop Multi-Cryptocurrency Payment System
Deployment Date: September 11-12, 2025
Target: Hostinger VPS (srv1002428.hstgr.cloud / thebankofdebbie.giize.com)
Status: LEARNING EXPERIENCE - COMPLEX SYSTEM WITH FUNDAMENTAL ISSUES

================================================================================
                           DEPLOYMENT TIMELINE
================================================================================

📅 September 11, 2025:
- Initial BTCPay Server installation attempted on Hostinger VPS
- Discovered Bitcoin daemon restarting due to pruning configuration issues
- Multiple cryptocurrency setup attempted (BTC, DOGE, XMR, DASH, LTC)

📅 September 12, 2025:
- Major disk space crisis discovered (129GB consumed by non-pruned blockchains)
- Extensive troubleshooting of Bitcoin pruning configuration
- Documentation and cleanup of lessons learned

================================================================================
                        CRITICAL DISCOVERIES
================================================================================

🚨 **MAJOR ISSUE: BTCPAY DOCKER COMPOSE CONFIGURATION SYSTEM IS BROKEN**

Root Problem: BTCPay's docker-compose generator creates corrupted YAML that prevents 
environment variables from being properly passed to cryptocurrency containers.

Evidence:
- BITCOIN_EXTRA_ARGS appears correctly in docker-compose.yml
- Environment variable is EMPTY when checked inside Bitcoin container
- Multiple YAML format attempts all failed (|-, |, >, single-line escaped)
- Manual bitcoin.conf modifications get overwritten by entrypoint script

Technical Details:
- Bitcoin container uses /entrypoint.sh that overwrites bitcoin.conf from environment
- Environment variable parsing in BTCPay template system is unreliable
- Configuration hierarchy: .env → docker-compose.yml → container (breaks at last step)

================================================================================
                         ATTEMPTED SOLUTIONS
================================================================================

❌ **APPROACH 1: Manual bitcoin.conf Editing**
   Method: Directly add prune=10000 to bitcoin.conf in Docker volume
   Result: FAILED - Container entrypoint overwrites config file on startup
   Lesson: Bitcoin container completely regenerates config from environment variables

❌ **APPROACH 2: Docker Compose YAML Direct Editing** 
   Method: Modify BITCOIN_EXTRA_ARGS in generated docker-compose.yml
   Result: FAILED - YAML formatting corruption prevents variable parsing
   Lesson: BTCPay's multiline YAML generation is fragile and unreliable

❌ **APPROACH 3: Environment File Override**
   Method: Add BITCOIN_EXTRA_ARGS directly to /opt/.env file
   Result: FAILED - Environment variables not inherited properly
   Lesson: BTCPay doesn't use .env file for Docker Compose environment variables

❌ **APPROACH 4: YAML Format Variations**
   Method: Tried |- (literal), | (literal), > (folded), single-line escaped
   Result: ALL FAILED - Environment variable still empty in container
   Lesson: The issue is not YAML syntax but fundamental parsing/generation bug

❌ **APPROACH 5: Docker Compose Override File**
   Method: Create docker-compose.override.yml to override Bitcoin configuration
   Result: PARTIAL SUCCESS - Pruning config read but RPC authentication broken
   Status: Closest to working solution, needs refinement

❌ **APPROACH 6: Clean Bitcoin Core from Scratch**
   Method: Build standard Bitcoin Core container bypassing BTCPay entirely
   Result: MOUNT ISSUES - Docker volume configuration problems
   Status: Interrupted due to complexity

================================================================================
                           SPACE MANAGEMENT CRISIS
================================================================================

🚨 **DISK SPACE EMERGENCY (September 12, 2025)**

Crisis Discovery:
- Litecoin daemon: 78GB (no pruning configured)
- Dogecoin daemon: 51GB (no pruning configured)  
- Monero daemon: 6.5GB
- Total impact: 135GB consumed (34% of 394GB disk)

Resolution:
- Emergency stop of all cryptocurrency daemons
- Manual deletion of blockchain data: sudo rm -rf /var/lib/docker/volumes/*/data/*
- Space recovered: 129GB freed
- Final usage: 63GB used / 316GB available (safe)

Lesson Learned:
ALL cryptocurrency daemons need explicit pruning configuration, not just Bitcoin.
Default behavior downloads full blockchains (50-80GB each).

================================================================================
                       CRYPTOCURRENCY INTEGRATION STATUS
================================================================================

✅ **WORKING SERVICES:**
   - BTCPay Web Interface: Operational (https://thebankofdebbie.giize.com)
   - Database: PostgreSQL running and accessible
   - SSL/TLS: nginx reverse proxy with Let's Encrypt working
   - Tor Network: Hidden services configured and operational

⚠️ **CRYPTOCURRENCY STATUS:**
   Bitcoin (BTC): 
   - Container runs but pruning config not applied
   - Shows height 0 in BTCPay interface
   - RPC connectivity issues with NBXplorer
   
   Dogecoin (DOGE):
   - Container runs and loads block index
   - Shows height 0 in BTCPay interface  
   - RPC not ready during startup phase
   
   Monero (XMR):
   - Daemon container operational
   - Wallet container restarting (configuration issues)
   - Missing from BTCPay interface (NBXplorer not configured)
   
   Ethereum (ETH):
   - Configured in BTCPAY_CRYPTOS environment
   - NO CONTAINERS CREATED (possibly unsupported in this BTCPay version)
   
   Zcash (ZEC):
   - Only wallet container present, main daemon missing
   - Not appearing in BTCPay interface

❌ **CORE PROBLEM:**
   NBXplorer (blockchain explorer) only configured for "btc,doge" instead of full 
   cryptocurrency set. This explains why other cryptocurrencies don't appear in 
   BTCPay interface even when containers are running.

================================================================================
                          TECHNICAL ARCHITECTURE ANALYSIS
================================================================================

**BTCPay Server Components:**
1. **BTCPay Application**: Web interface, store management, payment processing
2. **NBXplorer**: Blockchain explorer that connects BTCPay to cryptocurrency daemons
3. **Cryptocurrency Daemons**: Bitcoin Core, Dogecoin Core, Monero, etc.
4. **Database**: PostgreSQL for BTCPay data storage
5. **Proxy**: nginx with SSL termination and Tor integration

**Configuration Flow:**
   .env file → BTCPay setup script → docker-compose generation → container environment → config files

**Failure Points Identified:**
- Step 3→4: docker-compose to container environment (YAML parsing broken)
- Step 4→5: Container environment to config files (entrypoint script issues)

**Working Components:**
- BTCPay web interface and database
- SSL/nginx proxy infrastructure  
- Tor network integration
- Basic container orchestration

**Broken Components:**
- Cryptocurrency daemon configuration management
- Bitcoin pruning configuration persistence
- Multi-cryptocurrency NBXplorer integration

================================================================================
                            LESSONS LEARNED
================================================================================

🔧 **Docker & Configuration Management:**
1. **BTCPay Complexity**: BTCPay Server's Docker setup is overly complex with multiple 
   layers of configuration that can break independently
   
2. **Environment Variable Reliability**: Docker Compose multiline YAML strings are 
   fragile and prone to parsing failures in BTCPay's template system
   
3. **Container Entrypoint Behavior**: Cryptocurrency containers completely regenerate 
   config files from environment variables, ignoring manual modifications
   
4. **Override File Limitations**: docker-compose.override.yml works for passing 
   variables but doesn't guarantee proper parsing by container entrypoints

🪙 **Cryptocurrency Management:**
1. **Pruning is Critical**: Without explicit pruning, cryptocurrency daemons will 
   consume 50-80GB each, quickly filling disk space
   
2. **Sync Time Reality**: Tor-only networking significantly slows blockchain sync 
   (12-24 hours for Bitcoin vs 2-4 hours clearnet)
   
3. **RPC Dependency**: BTCPay requires cryptocurrency RPC to be fully operational 
   before showing proper status (height 0 = RPC not ready)
   
4. **NBXplorer Central Role**: All cryptocurrencies must be configured in NBXplorer 
   to appear in BTCPay interface, regardless of daemon status

📊 **Resource Planning:**
1. **Storage Requirements**: Even pruned Bitcoin (10GB) + multiple altcoins can 
   consume 50+ GB during sync before pruning kicks in
   
2. **Memory Usage**: Multiple cryptocurrency daemons running simultaneously 
   requires careful memory allocation
   
3. **Network Bandwidth**: Initial blockchain download over Tor is bandwidth intensive
   
4. **Monitoring Necessity**: Real-time disk space monitoring essential during setup

================================================================================
                           SUCCESSFUL APPROACHES
================================================================================

✅ **What Actually Worked:**

1. **Manual Command Line Parameters**: 
   Direct Bitcoin Core with command line pruning parameters worked perfectly
   Evidence: "Prune configured to target 10000 MiB on disk for block and undo files."

2. **Docker Volume Management**:
   Manual deletion of blockchain data effective for space recovery
   Command: sudo rm -rf /var/lib/docker/volumes/*/data/*

3. **Service Isolation**:
   Individual container management more reliable than BTCPay's orchestration
   Docker individual start/stop commands work better than btcpay-restart.sh

4. **Configuration Verification**:
   Direct log analysis most reliable method for confirming configuration application
   grep -E '(prune|Prune)' provides definitive confirmation

================================================================================
                            RECOMMENDATIONS
================================================================================

🎯 **For Future Cryptocurrency Payment Systems:**

**SIMPLE APPROACH (Recommended):**
1. Use standard Bitcoin Core Docker image with direct configuration
2. Mount proper bitcoin.conf file with known working settings
3. Create simple payment processing API that connects to Bitcoin RPC
4. Avoid complex orchestration systems like BTCPay for basic needs

**BTCPAY APPROACH (If Required):**
1. Start with single cryptocurrency (Bitcoin only)
2. Use docker-compose.override.yml for configuration overrides
3. Expect configuration issues and plan for extensive troubleshooting
4. Monitor disk space continuously during setup
5. Test in regtest mode first to verify connectivity

**INFRASTRUCTURE REQUIREMENTS:**
- Minimum 1TB storage for multiple cryptocurrencies
- Real-time disk monitoring and alerts
- Automated backup of cryptocurrency wallet data
- Network redundancy for Tor connectivity

================================================================================
                             CURRENT STATE
================================================================================

**System Status (September 12, 2025):**
- Host: Hostinger VPS (394GB storage, 316GB available)
- BTCPay Web Interface: Operational
- Bitcoin Daemon: Stopped (pruning configuration failed)
- Dogecoin Daemon: Running but not syncing properly  
- Other Cryptocurrencies: Partially configured, not operational
- Disk Space: Safe (crisis resolved through manual cleanup)

**Working Components:**
- SSL certificates and nginx proxy
- Tor network integration
- BTCPay application framework
- Database and core infrastructure

**Unresolved Issues:**
- Bitcoin pruning configuration persistence
- Multi-cryptocurrency NBXplorer integration
- Height 0 display in BTCPay interface (RPC connectivity)
- Missing Ethereum and Zcash main daemons

**Documentation Status:**
- Technical discoveries recorded in CLAUDE.md
- Infrastructure details updated in Infrastructure.txt
- Complete troubleshooting history preserved

================================================================================
                               FINAL ASSESSMENT
================================================================================

**Time Investment:** 6+ hours of intensive troubleshooting
**Success Rate:** Partial (infrastructure working, cryptocurrencies problematic)
**Learning Value:** High (discovered fundamental BTCPay limitations)
**Production Readiness:** Low (requires significant additional work)

**Recommendation:** 
For production cryptocurrency payment processing, consider simpler alternatives 
to BTCPay Server. The complexity-to-reliability ratio is unfavorable for 
straightforward payment processing needs.

A simple Bitcoin Core node + custom payment API would be more reliable and 
maintainable than BTCPay's complex Docker orchestration system.

================================================================================
                          END OF DEPLOYMENT MEMOIRES
================================================================================
Total Configuration Attempts: 15+
Working Solutions Found: 1 (partial - docker-compose.override.yml)
Time to Working System: 6+ hours (still incomplete)
Complexity Rating: EXCESSIVE for basic cryptocurrency payment processing

Conclusion: BTCPay Server is a powerful but overly complex system that requires 
extensive expertise to configure properly. For basic needs, simpler solutions 
are more appropriate.