================================================================================ BTCPAY SERVER DEPLOYMENT MEMOIRES ================================================================================ Project: LittleShop Multi-Cryptocurrency Payment System Deployment Date: September 11-12, 2025 Target: Hostinger VPS (srv1002428.hstgr.cloud / thebankofdebbie.giize.com) Status: LEARNING EXPERIENCE - COMPLEX SYSTEM WITH FUNDAMENTAL ISSUES ================================================================================ DEPLOYMENT TIMELINE ================================================================================ 📅 September 11, 2025: - Initial BTCPay Server installation attempted on Hostinger VPS - Discovered Bitcoin daemon restarting due to pruning configuration issues - Multiple cryptocurrency setup attempted (BTC, DOGE, XMR, DASH, LTC) 📅 September 12, 2025: - Major disk space crisis discovered (129GB consumed by non-pruned blockchains) - Extensive troubleshooting of Bitcoin pruning configuration - Documentation and cleanup of lessons learned ================================================================================ CRITICAL DISCOVERIES ================================================================================ 🚨 **MAJOR ISSUE: BTCPAY DOCKER COMPOSE CONFIGURATION SYSTEM IS BROKEN** Root Problem: BTCPay's docker-compose generator creates corrupted YAML that prevents environment variables from being properly passed to cryptocurrency containers. Evidence: - BITCOIN_EXTRA_ARGS appears correctly in docker-compose.yml - Environment variable is EMPTY when checked inside Bitcoin container - Multiple YAML format attempts all failed (|-, |, >, single-line escaped) - Manual bitcoin.conf modifications get overwritten by entrypoint script Technical Details: - Bitcoin container uses /entrypoint.sh that overwrites bitcoin.conf from environment - Environment variable parsing in BTCPay template system is unreliable - Configuration hierarchy: .env → docker-compose.yml → container (breaks at last step) ================================================================================ ATTEMPTED SOLUTIONS ================================================================================ ❌ **APPROACH 1: Manual bitcoin.conf Editing** Method: Directly add prune=10000 to bitcoin.conf in Docker volume Result: FAILED - Container entrypoint overwrites config file on startup Lesson: Bitcoin container completely regenerates config from environment variables ❌ **APPROACH 2: Docker Compose YAML Direct Editing** Method: Modify BITCOIN_EXTRA_ARGS in generated docker-compose.yml Result: FAILED - YAML formatting corruption prevents variable parsing Lesson: BTCPay's multiline YAML generation is fragile and unreliable ❌ **APPROACH 3: Environment File Override** Method: Add BITCOIN_EXTRA_ARGS directly to /opt/.env file Result: FAILED - Environment variables not inherited properly Lesson: BTCPay doesn't use .env file for Docker Compose environment variables ❌ **APPROACH 4: YAML Format Variations** Method: Tried |- (literal), | (literal), > (folded), single-line escaped Result: ALL FAILED - Environment variable still empty in container Lesson: The issue is not YAML syntax but fundamental parsing/generation bug ❌ **APPROACH 5: Docker Compose Override File** Method: Create docker-compose.override.yml to override Bitcoin configuration Result: PARTIAL SUCCESS - Pruning config read but RPC authentication broken Status: Closest to working solution, needs refinement ❌ **APPROACH 6: Clean Bitcoin Core from Scratch** Method: Build standard Bitcoin Core container bypassing BTCPay entirely Result: MOUNT ISSUES - Docker volume configuration problems Status: Interrupted due to complexity ================================================================================ SPACE MANAGEMENT CRISIS ================================================================================ 🚨 **DISK SPACE EMERGENCY (September 12, 2025)** Crisis Discovery: - Litecoin daemon: 78GB (no pruning configured) - Dogecoin daemon: 51GB (no pruning configured) - Monero daemon: 6.5GB - Total impact: 135GB consumed (34% of 394GB disk) Resolution: - Emergency stop of all cryptocurrency daemons - Manual deletion of blockchain data: sudo rm -rf /var/lib/docker/volumes/*/data/* - Space recovered: 129GB freed - Final usage: 63GB used / 316GB available (safe) Lesson Learned: ALL cryptocurrency daemons need explicit pruning configuration, not just Bitcoin. Default behavior downloads full blockchains (50-80GB each). ================================================================================ CRYPTOCURRENCY INTEGRATION STATUS ================================================================================ ✅ **WORKING SERVICES:** - BTCPay Web Interface: Operational (https://thebankofdebbie.giize.com) - Database: PostgreSQL running and accessible - SSL/TLS: nginx reverse proxy with Let's Encrypt working - Tor Network: Hidden services configured and operational ⚠️ **CRYPTOCURRENCY STATUS:** Bitcoin (BTC): - Container runs but pruning config not applied - Shows height 0 in BTCPay interface - RPC connectivity issues with NBXplorer Dogecoin (DOGE): - Container runs and loads block index - Shows height 0 in BTCPay interface - RPC not ready during startup phase Monero (XMR): - Daemon container operational - Wallet container restarting (configuration issues) - Missing from BTCPay interface (NBXplorer not configured) Ethereum (ETH): - Configured in BTCPAY_CRYPTOS environment - NO CONTAINERS CREATED (possibly unsupported in this BTCPay version) Zcash (ZEC): - Only wallet container present, main daemon missing - Not appearing in BTCPay interface ❌ **CORE PROBLEM:** NBXplorer (blockchain explorer) only configured for "btc,doge" instead of full cryptocurrency set. This explains why other cryptocurrencies don't appear in BTCPay interface even when containers are running. ================================================================================ TECHNICAL ARCHITECTURE ANALYSIS ================================================================================ **BTCPay Server Components:** 1. **BTCPay Application**: Web interface, store management, payment processing 2. **NBXplorer**: Blockchain explorer that connects BTCPay to cryptocurrency daemons 3. **Cryptocurrency Daemons**: Bitcoin Core, Dogecoin Core, Monero, etc. 4. **Database**: PostgreSQL for BTCPay data storage 5. **Proxy**: nginx with SSL termination and Tor integration **Configuration Flow:** .env file → BTCPay setup script → docker-compose generation → container environment → config files **Failure Points Identified:** - Step 3→4: docker-compose to container environment (YAML parsing broken) - Step 4→5: Container environment to config files (entrypoint script issues) **Working Components:** - BTCPay web interface and database - SSL/nginx proxy infrastructure - Tor network integration - Basic container orchestration **Broken Components:** - Cryptocurrency daemon configuration management - Bitcoin pruning configuration persistence - Multi-cryptocurrency NBXplorer integration ================================================================================ LESSONS LEARNED ================================================================================ 🔧 **Docker & Configuration Management:** 1. **BTCPay Complexity**: BTCPay Server's Docker setup is overly complex with multiple layers of configuration that can break independently 2. **Environment Variable Reliability**: Docker Compose multiline YAML strings are fragile and prone to parsing failures in BTCPay's template system 3. **Container Entrypoint Behavior**: Cryptocurrency containers completely regenerate config files from environment variables, ignoring manual modifications 4. **Override File Limitations**: docker-compose.override.yml works for passing variables but doesn't guarantee proper parsing by container entrypoints 🪙 **Cryptocurrency Management:** 1. **Pruning is Critical**: Without explicit pruning, cryptocurrency daemons will consume 50-80GB each, quickly filling disk space 2. **Sync Time Reality**: Tor-only networking significantly slows blockchain sync (12-24 hours for Bitcoin vs 2-4 hours clearnet) 3. **RPC Dependency**: BTCPay requires cryptocurrency RPC to be fully operational before showing proper status (height 0 = RPC not ready) 4. **NBXplorer Central Role**: All cryptocurrencies must be configured in NBXplorer to appear in BTCPay interface, regardless of daemon status 📊 **Resource Planning:** 1. **Storage Requirements**: Even pruned Bitcoin (10GB) + multiple altcoins can consume 50+ GB during sync before pruning kicks in 2. **Memory Usage**: Multiple cryptocurrency daemons running simultaneously requires careful memory allocation 3. **Network Bandwidth**: Initial blockchain download over Tor is bandwidth intensive 4. **Monitoring Necessity**: Real-time disk space monitoring essential during setup ================================================================================ SUCCESSFUL APPROACHES ================================================================================ ✅ **What Actually Worked:** 1. **Manual Command Line Parameters**: Direct Bitcoin Core with command line pruning parameters worked perfectly Evidence: "Prune configured to target 10000 MiB on disk for block and undo files." 2. **Docker Volume Management**: Manual deletion of blockchain data effective for space recovery Command: sudo rm -rf /var/lib/docker/volumes/*/data/* 3. **Service Isolation**: Individual container management more reliable than BTCPay's orchestration Docker individual start/stop commands work better than btcpay-restart.sh 4. **Configuration Verification**: Direct log analysis most reliable method for confirming configuration application grep -E '(prune|Prune)' provides definitive confirmation ================================================================================ RECOMMENDATIONS ================================================================================ 🎯 **For Future Cryptocurrency Payment Systems:** **SIMPLE APPROACH (Recommended):** 1. Use standard Bitcoin Core Docker image with direct configuration 2. Mount proper bitcoin.conf file with known working settings 3. Create simple payment processing API that connects to Bitcoin RPC 4. Avoid complex orchestration systems like BTCPay for basic needs **BTCPAY APPROACH (If Required):** 1. Start with single cryptocurrency (Bitcoin only) 2. Use docker-compose.override.yml for configuration overrides 3. Expect configuration issues and plan for extensive troubleshooting 4. Monitor disk space continuously during setup 5. Test in regtest mode first to verify connectivity **INFRASTRUCTURE REQUIREMENTS:** - Minimum 1TB storage for multiple cryptocurrencies - Real-time disk monitoring and alerts - Automated backup of cryptocurrency wallet data - Network redundancy for Tor connectivity ================================================================================ CURRENT STATE ================================================================================ **System Status (September 12, 2025):** - Host: Hostinger VPS (394GB storage, 316GB available) - BTCPay Web Interface: Operational - Bitcoin Daemon: Stopped (pruning configuration failed) - Dogecoin Daemon: Running but not syncing properly - Other Cryptocurrencies: Partially configured, not operational - Disk Space: Safe (crisis resolved through manual cleanup) **Working Components:** - SSL certificates and nginx proxy - Tor network integration - BTCPay application framework - Database and core infrastructure **Unresolved Issues:** - Bitcoin pruning configuration persistence - Multi-cryptocurrency NBXplorer integration - Height 0 display in BTCPay interface (RPC connectivity) - Missing Ethereum and Zcash main daemons **Documentation Status:** - Technical discoveries recorded in CLAUDE.md - Infrastructure details updated in Infrastructure.txt - Complete troubleshooting history preserved ================================================================================ FINAL ASSESSMENT ================================================================================ **Time Investment:** 6+ hours of intensive troubleshooting **Success Rate:** Partial (infrastructure working, cryptocurrencies problematic) **Learning Value:** High (discovered fundamental BTCPay limitations) **Production Readiness:** Low (requires significant additional work) **Recommendation:** For production cryptocurrency payment processing, consider simpler alternatives to BTCPay Server. The complexity-to-reliability ratio is unfavorable for straightforward payment processing needs. A simple Bitcoin Core node + custom payment API would be more reliable and maintainable than BTCPay's complex Docker orchestration system. ================================================================================ END OF DEPLOYMENT MEMOIRES ================================================================================ Total Configuration Attempts: 15+ Working Solutions Found: 1 (partial - docker-compose.override.yml) Time to Working System: 6+ hours (still incomplete) Complexity Rating: EXCESSIVE for basic cryptocurrency payment processing Conclusion: BTCPay Server is a powerful but overly complex system that requires extensive expertise to configure properly. For basic needs, simpler solutions are more appropriate.