uploader-bot/docs/README.md

# My Uploader Bot - Comprehensive File Upload & Blockchain Integration System

A modern, scalable file upload and management system with blockchain integration, built with async Python and enterprise-grade security.

## 🚀 Features

### Core Features
- **Chunked File Upload**: Support for large files with resume capability
- **Multi-Storage Backend**: Local, S3-compatible storage with automatic failover
- **File Processing**: Automatic image optimization, thumbnail generation, media conversion
- **Content Management**: Version control, metadata management, search functionality
- **User Management**: JWT authentication, API keys, session management

### Blockchain Integration
- **TON Blockchain**: Wallet creation, transaction management, balance tracking
- **Smart Contracts**: Interaction with TON smart contracts
- **NFT Support**: NFT collection indexing and management
- **DeFi Integration**: Staking positions, token balances, yield farming

### Enterprise Features
- **Security**: Rate limiting, CORS, CSP, input validation, file encryption
- **Monitoring**: Prometheus metrics, Grafana dashboards, health checks
- **Caching**: Redis integration with intelligent cache management
- **Background Services**: Async task processing, blockchain indexing
- **API Documentation**: OpenAPI/Swagger with interactive interface

## 🏗️ Architecture

### Technology Stack
- **Backend**: Python 3.11+, Sanic (async web framework)
- **Database**: PostgreSQL 15+ with async SQLAlchemy 2.0
- **Cache**: Redis 7+ with connection pooling
- **Blockchain**: TON SDK for blockchain operations
- **Storage**: Local filesystem with S3-compatible support
- **Monitoring**: Prometheus, Grafana, structured logging
- **Deployment**: Docker Compose with multi-stage builds

### System Architecture
```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Load Balancer │    │     Grafana     │    │   Prometheus    │
│     (Nginx)     │    │   (Monitoring)  │    │   (Metrics)     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Sanic App     │◄──►│     Redis       │    │   PostgreSQL    │
│  (API Server)   │    │    (Cache)      │    │   (Database)    │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Background     │    │   File Storage  │    │  TON Blockchain │
│   Services      │    │ (Local/S3/CDN)  │    │   Integration   │
└─────────────────┘    └─────────────────┘    └─────────────────┘
```

## 🔧 Installation & Setup

### Prerequisites
- Python 3.11+
- PostgreSQL 15+
- Redis 7+
- Docker & Docker Compose (recommended)

### Quick Start with Docker

1. **Clone the repository**:
```bash
git clone https://github.com/your-org/my-uploader-bot.git
cd my-uploader-bot
```

2. **Create environment file**:
```bash
cp .env.example .env
# Edit .env with your configuration
```

3. **Start services**:
```bash
docker-compose -f docker-compose.new.yml up -d
```

4. **Initialize database**:
```bash
docker-compose -f docker-compose.new.yml exec app alembic upgrade head
```

5. **Create admin user**:
```bash
docker-compose -f docker-compose.new.yml exec app python -m app.scripts.create_admin
```

### Manual Installation

1. **Install dependencies**:
```bash
# Using Poetry (recommended)
poetry install

# Or using pip
pip install -r requirements_new.txt
```

2. **Set up database**:
```bash
# Create database
createdb myuploader

# Run migrations
alembic upgrade head
```

3. **Configure Redis**:
```bash
# Start Redis with custom config
redis-server config/redis.conf
```

4. **Start application**:
```bash
# Development mode
python -m app

# Production mode
gunicorn app:create_app --bind 0.0.0.0:8000 --worker-class sanic.worker.GunicornWorker
```

## ⚙️ Configuration

### Environment Variables

**Database Configuration**:
```bash
DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/myuploader
DB_POOL_SIZE=20
DB_MAX_OVERFLOW=30
```

**Redis Configuration**:
```bash
REDIS_URL=redis://localhost:6379/0
REDIS_POOL_SIZE=20
REDIS_POOL_MAX_CONNECTIONS=100
```

**Security Configuration**:
```bash
SECRET_KEY=your-super-secret-key-here
JWT_SECRET_KEY=your-jwt-secret-key
JWT_ACCESS_TOKEN_EXPIRE_MINUTES=30
JWT_REFRESH_TOKEN_EXPIRE_DAYS=7
```

**Storage Configuration**:
```bash
STORAGE_BACKEND=local  # or s3, gcs, azure
STORAGE_PATH=/app/uploads
MAX_FILE_SIZE=104857600  # 100MB
ALLOWED_FILE_TYPES=image,video,audio,document
```

**Blockchain Configuration**:
```bash
TON_NETWORK=mainnet  # or testnet
TON_API_KEY=your-ton-api-key
TON_WALLET_VERSION=v4
```

### Configuration Files

**Redis Configuration** (`config/redis.conf`):
- Optimized for caching workload
- Memory management and persistence settings
- Security and performance tuning

**Prometheus Configuration** (`monitoring/prometheus.yml`):
- Scraping configuration for all services
- Alert rules and metric collection
- Storage and retention settings

## 📊 API Documentation

### Interactive Documentation
- **Swagger UI**: http://localhost:8000/docs
- **ReDoc**: http://localhost:8000/redoc
- **OpenAPI JSON**: http://localhost:8000/openapi.json

### Authentication Methods

1. **JWT Bearer Token**:
```bash
curl -H "Authorization: Bearer <token>" \
  https://api.myuploader.com/api/v1/content/
```

2. **API Key**:
```bash
curl -H "X-API-Key: <api_key>" \
  https://api.myuploader.com/api/v1/content/
```

3. **Session Cookie**:
```bash
curl -b "session=<cookie>" \
  https://api.myuploader.com/api/v1/content/
```

### Key Endpoints

**Authentication**:
- `POST /api/v1/auth/register` - User registration
- `POST /api/v1/auth/login` - User login
- `POST /api/v1/auth/refresh` - Token refresh
- `DELETE /api/v1/auth/logout` - User logout

**Content Management**:
- `GET /api/v1/content/` - List content
- `POST /api/v1/content/` - Create content
- `GET /api/v1/content/{id}` - Get content details
- `PUT /api/v1/content/{id}` - Update content
- `DELETE /api/v1/content/{id}` - Delete content

**File Upload**:
- `POST /api/v1/storage/upload/initiate` - Start upload
- `POST /api/v1/storage/upload/chunk` - Upload chunk
- `POST /api/v1/storage/upload/complete` - Complete upload
- `GET /api/v1/storage/download/{id}` - Download file

**Blockchain**:
- `POST /api/v1/blockchain/wallet/create` - Create wallet
- `GET /api/v1/blockchain/wallet/{id}` - Get wallet info
- `POST /api/v1/blockchain/transaction/send` - Send transaction
- `GET /api/v1/blockchain/transaction/{hash}` - Get transaction status

## 🛡️ Security

### Security Features
- **Authentication**: Multi-factor authentication support
- **Authorization**: Role-based access control (RBAC)
- **Input Validation**: Comprehensive request validation
- **Rate Limiting**: Per-user and per-endpoint limits
- **File Security**: Virus scanning, type validation, encryption
- **Network Security**: CORS, CSP, HTTPS enforcement

### Security Best Practices

1. **Environment Variables**: Store sensitive data in environment variables
2. **Database Security**: Use connection pooling, parameterized queries
3. **File Upload Security**: Validate file types, scan for malware
4. **API Security**: Implement rate limiting, request validation
5. **Monitoring**: Track security events and anomalies

### Audit Logging
All security-relevant events are logged:
- Authentication attempts
- Authorization failures
- File upload/download events
- API key usage
- Blockchain transactions

## 📈 Monitoring & Observability

### Metrics Collection
- **Application Metrics**: Request counts, response times, error rates
- **System Metrics**: CPU, memory, disk usage
- **Database Metrics**: Connection pool, query performance
- **Cache Metrics**: Hit rates, memory usage
- **Blockchain Metrics**: Transaction status, wallet balances

### Dashboards
- **Application Dashboard**: Request metrics, error rates
- **Infrastructure Dashboard**: System resources, service health
- **Database Dashboard**: Query performance, connection pools
- **Security Dashboard**: Failed logins, rate limits

### Alerting Rules
- High error rates (>5% for 5 minutes)
- High response times (>2s for 95th percentile)
- Database connection issues
- High memory usage (>90%)
- Failed blockchain transactions

## 🔄 Background Services

### File Conversion Service
- **Image Processing**: Resize, optimize, generate thumbnails
- **Video Processing**: Transcode, extract thumbnails
- **Document Processing**: Extract metadata, generate previews
- **Retry Logic**: Automatic retry for failed conversions

### Blockchain Indexer Service
- **Transaction Monitoring**: Track pending transactions
- **Wallet Balance Updates**: Sync wallet balances
- **NFT Indexing**: Track NFT collections and transfers
- **Event Processing**: Process blockchain events

## 🚀 Deployment

### Docker Deployment

1. **Production Build**:
```bash
docker build -f Dockerfile.new --target production -t my-uploader-bot:latest .
```

2. **Deploy with Compose**:
```bash
docker-compose -f docker-compose.new.yml up -d
```

3. **Health Checks**:
```bash
curl http://localhost:8000/health
```

### Kubernetes Deployment

1. **Create ConfigMaps**:
```bash
kubectl create configmap app-config --from-env-file=.env
```

2. **Deploy Application**:
```bash
kubectl apply -f k8s/
```

3. **Check Status**:
```bash
kubectl get pods -l app=my-uploader-bot
```

### Production Considerations

1. **Database**: Use managed PostgreSQL service
2. **Cache**: Use managed Redis service
3. **Storage**: Use S3-compatible object storage
4. **Load Balancer**: Use cloud load balancer
5. **SSL/TLS**: Use Let's Encrypt or cloud SSL
6. **Monitoring**: Use managed monitoring service

## 🧪 Testing

### Running Tests
```bash
# Run all tests
pytest

# Run with coverage
pytest --cov=app

# Run specific test file
pytest tests/test_auth.py

# Run integration tests
pytest tests/integration/
```

### Test Categories
- **Unit Tests**: Individual function/method tests
- **Integration Tests**: Service integration tests
- **API Tests**: HTTP endpoint tests
- **Database Tests**: Database operation tests
- **Security Tests**: Security vulnerability tests

## 🔍 Development

### Development Setup
```bash
# Install dev dependencies
poetry install --dev

# Pre-commit hooks
pre-commit install

# Run linting
black app/
isort app/
mypy app/

# Database migrations
alembic revision --autogenerate -m "Description"
alembic upgrade head
```

### Code Quality Tools
- **Black**: Code formatting
- **isort**: Import sorting
- **mypy**: Type checking
- **bandit**: Security scanning
- **pytest**: Testing framework

### Contributing Guidelines
1. Fork the repository
2. Create feature branch
3. Write tests for new features
4. Ensure all tests pass
5. Run security scans
6. Submit pull request

## 📝 Migration Guide

### From v1.x to v2.x

1. **Database Migration**:
```bash
# Backup existing database
pg_dump myuploader > backup.sql

# Run new migrations
alembic upgrade head
```

2. **Configuration Updates**:
- Update environment variables
- Migrate Redis configuration
- Update storage configuration

3. **API Changes**:
- Authentication endpoints changed
- New blockchain endpoints added
- Response format updates

## 🐛 Troubleshooting

### Common Issues

**Database Connection Issues**:
```bash
# Check database connectivity
psql -h localhost -U postgres -d myuploader

# Check connection pool
docker-compose logs db
```

**Redis Connection Issues**:
```bash
# Test Redis connection
redis-cli ping

# Check Redis logs
docker-compose logs redis
```

**File Upload Issues**:
```bash
# Check storage permissions
ls -la /app/uploads

# Check disk space
df -h
```

**Blockchain Issues**:
```bash
# Check TON network status
curl https://toncenter.com/api/v2/getAddressInformation

# Check wallet balance
curl -X GET /api/v1/blockchain/wallet/{wallet_id}
```

### Performance Optimization

1. **Database Optimization**:
   - Add indexes for frequently queried columns
   - Optimize query patterns
   - Use connection pooling

2. **Cache Optimization**:
   - Implement cache warming
   - Use appropriate TTL values
   - Monitor cache hit rates

3. **File Storage Optimization**:
   - Use CDN for static files
   - Implement file compression
   - Use efficient storage backends

## 📚 Additional Resources

- **API Documentation**: https://docs.myuploader.com
- **SDK Documentation**: https://sdk.myuploader.com
- **Community Forum**: https://community.myuploader.com
- **GitHub Repository**: https://github.com/your-org/my-uploader-bot
- **Docker Images**: https://hub.docker.com/r/myuploader/app

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🤝 Support

- **Email**: support@myuploader.com
- **Discord**: https://discord.gg/myuploader
- **GitHub Issues**: https://github.com/your-org/my-uploader-bot/issues
- **Documentation**: https://docs.myuploader.com

---

**Built with ❤️ by the My Uploader Bot Team**