14 KiB

Raw Blame History

My Uploader Bot - Comprehensive File Upload & Blockchain Integration System

A modern, scalable file upload and management system with blockchain integration, built with async Python and enterprise-grade security.

🚀 Features

Core Features

Chunked File Upload: Support for large files with resume capability
Multi-Storage Backend: Local, S3-compatible storage with automatic failover
File Processing: Automatic image optimization, thumbnail generation, media conversion
Content Management: Version control, metadata management, search functionality
User Management: JWT authentication, API keys, session management

Blockchain Integration

TON Blockchain: Wallet creation, transaction management, balance tracking
Smart Contracts: Interaction with TON smart contracts
NFT Support: NFT collection indexing and management
DeFi Integration: Staking positions, token balances, yield farming

Enterprise Features

Security: Rate limiting, CORS, CSP, input validation, file encryption
Monitoring: Prometheus metrics, Grafana dashboards, health checks
Caching: Redis integration with intelligent cache management
Background Services: Async task processing, blockchain indexing
API Documentation: OpenAPI/Swagger with interactive interface

🏗️ Architecture

Technology Stack

Backend: Python 3.11+, Sanic (async web framework)
Database: PostgreSQL 15+ with async SQLAlchemy 2.0
Cache: Redis 7+ with connection pooling
Blockchain: TON SDK for blockchain operations
Storage: Local filesystem with S3-compatible support
Monitoring: Prometheus, Grafana, structured logging
Deployment: Docker Compose with multi-stage builds

System Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Load Balancer │    │     Grafana     │    │   Prometheus    │
│     (Nginx)     │    │   (Monitoring)  │    │   (Metrics)     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Sanic App     │◄──►│     Redis       │    │   PostgreSQL    │
│  (API Server)   │    │    (Cache)      │    │   (Database)    │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Background     │    │   File Storage  │    │  TON Blockchain │
│   Services      │    │ (Local/S3/CDN)  │    │   Integration   │
└─────────────────┘    └─────────────────┘    └─────────────────┘

🔧 Installation & Setup

Prerequisites

Python 3.11+
PostgreSQL 15+
Redis 7+
Docker & Docker Compose (recommended)

Quick Start with Docker

Clone the repository:

git clone https://github.com/your-org/my-uploader-bot.git
cd my-uploader-bot

Create environment file:

cp .env.example .env
# Edit .env with your configuration

Start services:

docker-compose -f docker-compose.new.yml up -d

Initialize database:

docker-compose -f docker-compose.new.yml exec app alembic upgrade head

Create admin user:

docker-compose -f docker-compose.new.yml exec app python -m app.scripts.create_admin

Manual Installation

Install dependencies:

# Using Poetry (recommended)
poetry install

# Or using pip
pip install -r requirements_new.txt

Set up database:

# Create database
createdb myuploader

# Run migrations
alembic upgrade head

Configure Redis:

# Start Redis with custom config
redis-server config/redis.conf

Start application:

# Development mode
python -m app

# Production mode
gunicorn app:create_app --bind 0.0.0.0:8000 --worker-class sanic.worker.GunicornWorker

⚙️ Configuration

Environment Variables

Database Configuration:

DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/myuploader
DB_POOL_SIZE=20
DB_MAX_OVERFLOW=30

Redis Configuration:

REDIS_URL=redis://localhost:6379/0
REDIS_POOL_SIZE=20
REDIS_POOL_MAX_CONNECTIONS=100

Security Configuration:

SECRET_KEY=your-super-secret-key-here
JWT_SECRET_KEY=your-jwt-secret-key
JWT_ACCESS_TOKEN_EXPIRE_MINUTES=30
JWT_REFRESH_TOKEN_EXPIRE_DAYS=7

Storage Configuration:

STORAGE_BACKEND=local  # or s3, gcs, azure
STORAGE_PATH=/app/uploads
MAX_FILE_SIZE=104857600  # 100MB
ALLOWED_FILE_TYPES=image,video,audio,document

Blockchain Configuration:

TON_NETWORK=mainnet  # or testnet
TON_API_KEY=your-ton-api-key
TON_WALLET_VERSION=v4

Configuration Files

Redis Configuration (config/redis.conf):

Optimized for caching workload
Memory management and persistence settings
Security and performance tuning

Prometheus Configuration (monitoring/prometheus.yml):

Scraping configuration for all services
Alert rules and metric collection
Storage and retention settings

📊 API Documentation

Interactive Documentation

Swagger UI: http://localhost:8000/docs
ReDoc: http://localhost:8000/redoc
OpenAPI JSON: http://localhost:8000/openapi.json

Authentication Methods

JWT Bearer Token:

curl -H "Authorization: Bearer <token>" \
  https://api.myuploader.com/api/v1/content/

API Key:

curl -H "X-API-Key: <api_key>" \
  https://api.myuploader.com/api/v1/content/

Session Cookie:

curl -b "session=<cookie>" \
  https://api.myuploader.com/api/v1/content/

Key Endpoints

Authentication:

POST /api/v1/auth/register - User registration
POST /api/v1/auth/login - User login
POST /api/v1/auth/refresh - Token refresh
DELETE /api/v1/auth/logout - User logout

Content Management:

GET /api/v1/content/ - List content
POST /api/v1/content/ - Create content
GET /api/v1/content/{id} - Get content details
PUT /api/v1/content/{id} - Update content
DELETE /api/v1/content/{id} - Delete content

File Upload:

POST /api/v1/storage/upload/initiate - Start upload
POST /api/v1/storage/upload/chunk - Upload chunk
POST /api/v1/storage/upload/complete - Complete upload
GET /api/v1/storage/download/{id} - Download file

Blockchain:

POST /api/v1/blockchain/wallet/create - Create wallet
GET /api/v1/blockchain/wallet/{id} - Get wallet info
POST /api/v1/blockchain/transaction/send - Send transaction
GET /api/v1/blockchain/transaction/{hash} - Get transaction status

🛡️ Security

Security Features

Authentication: Multi-factor authentication support
Authorization: Role-based access control (RBAC)
Input Validation: Comprehensive request validation
Rate Limiting: Per-user and per-endpoint limits
File Security: Virus scanning, type validation, encryption
Network Security: CORS, CSP, HTTPS enforcement

Security Best Practices

Environment Variables: Store sensitive data in environment variables
Database Security: Use connection pooling, parameterized queries
File Upload Security: Validate file types, scan for malware
API Security: Implement rate limiting, request validation
Monitoring: Track security events and anomalies

Audit Logging

All security-relevant events are logged:

Authentication attempts
Authorization failures
File upload/download events
API key usage
Blockchain transactions

📈 Monitoring & Observability

Metrics Collection

Application Metrics: Request counts, response times, error rates
System Metrics: CPU, memory, disk usage
Database Metrics: Connection pool, query performance
Cache Metrics: Hit rates, memory usage
Blockchain Metrics: Transaction status, wallet balances

Dashboards

Application Dashboard: Request metrics, error rates
Infrastructure Dashboard: System resources, service health
Database Dashboard: Query performance, connection pools
Security Dashboard: Failed logins, rate limits

Alerting Rules

High error rates (>5% for 5 minutes)
High response times (>2s for 95th percentile)
Database connection issues
High memory usage (>90%)
Failed blockchain transactions

🔄 Background Services

File Conversion Service

Image Processing: Resize, optimize, generate thumbnails
Video Processing: Transcode, extract thumbnails
Document Processing: Extract metadata, generate previews
Retry Logic: Automatic retry for failed conversions

Blockchain Indexer Service

Transaction Monitoring: Track pending transactions
Wallet Balance Updates: Sync wallet balances
NFT Indexing: Track NFT collections and transfers
Event Processing: Process blockchain events

🚀 Deployment

Docker Deployment

Production Build:

docker build -f Dockerfile.new --target production -t my-uploader-bot:latest .

Deploy with Compose:

docker-compose -f docker-compose.new.yml up -d

Health Checks:

curl http://localhost:8000/health

Kubernetes Deployment

Create ConfigMaps:

kubectl create configmap app-config --from-env-file=.env

Deploy Application:

kubectl apply -f k8s/

Check Status:

kubectl get pods -l app=my-uploader-bot

Production Considerations

Database: Use managed PostgreSQL service
Cache: Use managed Redis service
Storage: Use S3-compatible object storage
Load Balancer: Use cloud load balancer
SSL/TLS: Use Let's Encrypt or cloud SSL
Monitoring: Use managed monitoring service

🧪 Testing

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=app

# Run specific test file
pytest tests/test_auth.py

# Run integration tests
pytest tests/integration/

Test Categories

Unit Tests: Individual function/method tests
Integration Tests: Service integration tests
API Tests: HTTP endpoint tests
Database Tests: Database operation tests
Security Tests: Security vulnerability tests

🔍 Development

Development Setup

# Install dev dependencies
poetry install --dev

# Pre-commit hooks
pre-commit install

# Run linting
black app/
isort app/
mypy app/

# Database migrations
alembic revision --autogenerate -m "Description"
alembic upgrade head

Code Quality Tools

Black: Code formatting
isort: Import sorting
mypy: Type checking
bandit: Security scanning
pytest: Testing framework

Contributing Guidelines

Fork the repository
Create feature branch
Write tests for new features
Ensure all tests pass
Run security scans
Submit pull request

📝 Migration Guide

From v1.x to v2.x

Database Migration:

# Backup existing database
pg_dump myuploader > backup.sql

# Run new migrations
alembic upgrade head

Configuration Updates:

Update environment variables
Migrate Redis configuration
Update storage configuration

API Changes:

Authentication endpoints changed
New blockchain endpoints added
Response format updates

🐛 Troubleshooting

Common Issues

Database Connection Issues:

# Check database connectivity
psql -h localhost -U postgres -d myuploader

# Check connection pool
docker-compose logs db

Redis Connection Issues:

# Test Redis connection
redis-cli ping

# Check Redis logs
docker-compose logs redis

File Upload Issues:

# Check storage permissions
ls -la /app/uploads

# Check disk space
df -h

Blockchain Issues:

# Check TON network status
curl https://toncenter.com/api/v2/getAddressInformation

# Check wallet balance
curl -X GET /api/v1/blockchain/wallet/{wallet_id}

Performance Optimization

Database Optimization:
- Add indexes for frequently queried columns
- Optimize query patterns
- Use connection pooling
Cache Optimization:
- Implement cache warming
- Use appropriate TTL values
- Monitor cache hit rates
File Storage Optimization:
- Use CDN for static files
- Implement file compression
- Use efficient storage backends

📚 Additional Resources

API Documentation: https://docs.myuploader.com
SDK Documentation: https://sdk.myuploader.com
Community Forum: https://community.myuploader.com
GitHub Repository: https://github.com/your-org/my-uploader-bot
Docker Images: https://hub.docker.com/r/myuploader/app

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Support

Email: support@myuploader.com
Discord: https://discord.gg/myuploader
GitHub Issues: https://github.com/your-org/my-uploader-bot/issues
Documentation: https://docs.myuploader.com

Built with ❤️ by the My Uploader Bot Team

14 KiB Raw Blame History