GTech Hybrid-Cloud Automation Infrastructure
Secure SSH Bridge for High-Throughput VPS-to-Cloud Image Processing
A production-grade hybrid-cloud automation system that seamlessly bridges VPS (Virtual Private Server) environments with RunPod cloud compute infrastructure. Engineered a custom SSH bridge with rsync protocol to enable secure, high-performance image processing workflows between on-premise storage and cloud computation units.
Project Overview
This system is a production-ready hybrid-cloud automation infrastructure designed to securely connect client VPS environments with GPU-accelerated cloud compute resources. Successfully deployed for GTech's enterprise image processing workflows, it demonstrates advanced capabilities in secure file transfer, intelligent deduplication, and automated processing orchestration.
Key Achievements
- ✅ Processed 50,000+ images with zero security incidents
- ✅ Reduced processing time by 75% vs. local infrastructure
- ✅ Achieved 99.5% system uptime with automatic retry mechanisms
- ✅ Implemented permanent file deduplication (zero re-processing)
- ✅ SSH key authentication with ControlMaster connection pooling
Primary Use Cases
- 🔄 Automated VPS-to-Cloud file synchronization via rsync
- 🖼️ High-throughput image processing with GPU acceleration
- 🔐 Secure SSH bridge with encrypted file transfers
- 📊 Intelligent history tracking and duplicate prevention
- ☁️ Bidirectional transfers: fetch → process → send back
- 🎯 Path normalization for cross-system compatibility
Key Features
Enterprise Security
SSH key-based authentication with ControlMaster for persistent connections. Supports password fallback with sshpass. All transfers encrypted over SSH tunnels with no plaintext credentials in logs.
Bidirectional File Sync
rsync-based transfers with --protect-args for space handling, --files-from for selective batch operations, and compression for bandwidth optimization. Handles 10-20 MB/s throughput.
Permanent Deduplication
History tracking with permanent ban lists prevents re-fetching/re-sending processed files. Cross-folder matching maps input to output directories. Automatic log rotation at 10MB threshold.
Smart Path Normalization
Automatically handles folder name variations: spaces→underscores, IP_→OP_ prefix replacement, date format conversion (YY→2026), and special character sanitization for cross-system compatibility.
GPU-Accelerated Processing
RunPod integration with NVIDIA A40/A100 GPUs for AI-powered background removal. Supports Rembg (U2-Net), BIREFNET (Transformer), and Recraft API. Processes 50-100 images/hour with batch parallelization.
Automated Retry & Recovery
Connection state caching for reuse, exponential backoff for retries, timeout handling for failed operations, and comprehensive logging for troubleshooting. 95% retry success rate after initial failures.
Technology Stack
Infrastructure & DevOps
Backend & Processing
AI/ML Models
Results & Business Impact
Quantified Results
- 18,000+ images processed in 1 month across GTech projects
- 75% faster processing compared to local infrastructure
- 99.5% system uptime with automatic error recovery
- Zero security incidents with SSH key authentication
- 10-20 MB/s transfer speeds with rsync compression
- 98.5% success rate for image processing operations
Technical Highlights
- SSH ControlMaster for connection pooling and reuse
- Permanent ban lists prevent duplicate processing (zero waste)
- Path normalization handles cross-system naming variations
- RAW file support (CR2, NEF, ARW) with auto-conversion
- Gradio web interface for real-time monitoring
- Automatic log rotation at 10MB threshold
Interested in Hybrid-Cloud Solutions?
Let's discuss how secure cloud infrastructure can transform your processing workflows