first commit
This commit is contained in:
273
README.md
Normal file
273
README.md
Normal file
@@ -0,0 +1,273 @@
|
||||
# TGLoader
|
||||
|
||||
A powerful Telegram bot and web interface for downloading media from various platforms including YouTube, Instagram, and direct links. Built with Python, Pyrogram, FastAPI, and SQLAlchemy.
|
||||
|
||||
## Features
|
||||
|
||||
- **Multi-platform Support**: Download videos and media from YouTube, Instagram, and direct URLs
|
||||
- **Telegram Bot Interface**: Easy-to-use bot commands for downloading media
|
||||
- **Web Admin Panel**: Full-featured web dashboard for task management and user administration
|
||||
- **Task Queue System**: Efficient background task processing with progress tracking
|
||||
- **File Management**: Automatic file splitting for large files (>2GB), thumbnail generation, and cleanup
|
||||
- **Database Migrations**: Automatic database schema migrations using Alembic
|
||||
- **Session Management**: Support for both in-memory and Redis-based sessions
|
||||
- **Access Control**: Configurable user authorization, admin roles, and private mode
|
||||
- **Cookie Support**: Instagram authentication via cookies file
|
||||
- **Rate Limiting**: Configurable download limits (file size, duration, concurrent tasks)
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.11+
|
||||
- PostgreSQL (recommended) or SQLite
|
||||
- Redis (optional, for session storage)
|
||||
- FFmpeg (for video thumbnail generation)
|
||||
- Telegram Bot Token and API credentials
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Clone the repository
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd tgloader
|
||||
```
|
||||
|
||||
### 2. Install dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 3. Install FFmpeg
|
||||
|
||||
**Linux:**
|
||||
```bash
|
||||
sudo apt-get install ffmpeg
|
||||
```
|
||||
|
||||
**macOS:**
|
||||
```bash
|
||||
brew install ffmpeg
|
||||
```
|
||||
|
||||
**Windows:**
|
||||
Download from [ffmpeg.org](https://ffmpeg.org/download.html)
|
||||
|
||||
### 4. Configure environment
|
||||
|
||||
Create a `.env.local` file (for local development) or `.env` file (for Docker):
|
||||
|
||||
```env
|
||||
# Telegram Bot
|
||||
BOT_TOKEN=your_bot_token
|
||||
TELEGRAM_API_ID=your_api_id
|
||||
TELEGRAM_API_HASH=your_api_hash
|
||||
OWNER_ID=your_telegram_user_id
|
||||
|
||||
# Authorization
|
||||
AUTHORIZED_USERS=user_id1,user_id2
|
||||
ADMIN_IDS=admin_id1,admin_id2
|
||||
BLOCKED_USERS=
|
||||
PRIVATE_MODE=false
|
||||
|
||||
# Database
|
||||
DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/tgloader
|
||||
# Or for SQLite: sqlite+aiosqlite:///./data/bot.db
|
||||
|
||||
# Redis (optional)
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB=0
|
||||
USE_REDIS_SESSIONS=false
|
||||
|
||||
# Web
|
||||
WEB_HOST=0.0.0.0
|
||||
WEB_PORT=5000
|
||||
WEB_SECRET_KEY=your_secret_key_here
|
||||
|
||||
# Logging
|
||||
LOG_LEVEL=INFO
|
||||
LOG_FILE=logs/bot.log
|
||||
|
||||
# Media Download
|
||||
COOKIES_FILE=instagram.cookies # Path to cookies file for Instagram (Netscape format)
|
||||
|
||||
# Download Limits
|
||||
MAX_FILE_SIZE= # Maximum file size in bytes (empty = no limit)
|
||||
MAX_DURATION_MINUTES= # Maximum duration in minutes (empty = no limit)
|
||||
MAX_CONCURRENT_TASKS=5 # Maximum concurrent tasks per user
|
||||
```
|
||||
|
||||
### 5. Set up Instagram cookies (optional)
|
||||
|
||||
For Instagram downloads, create a `instagram.cookies` file in Netscape format. You can export cookies from your browser using extensions like "Get cookies.txt LOCALLY" or "cookies.txt".
|
||||
|
||||
### 6. Initialize database
|
||||
|
||||
The database will be automatically initialized and migrated on first run using Alembic.
|
||||
|
||||
## Usage
|
||||
|
||||
### Running Locally
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
```
|
||||
|
||||
The bot and web interface will start together. Access the web admin panel at `http://localhost:5000`.
|
||||
|
||||
### Running with Docker
|
||||
|
||||
1. Build and start services:
|
||||
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
2. View logs:
|
||||
|
||||
```bash
|
||||
docker-compose logs -f app
|
||||
```
|
||||
|
||||
3. Stop services:
|
||||
|
||||
```bash
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
## Telegram Bot Commands
|
||||
|
||||
- `/start` - Start the bot and get help
|
||||
- `/help` - Show available commands
|
||||
- `/download <url>` - Download media from a URL
|
||||
- `/status` - Check your active tasks
|
||||
- `/cancel <task_id>` - Cancel a task
|
||||
|
||||
Simply send a URL to the bot to start downloading.
|
||||
|
||||
## Web Admin Panel
|
||||
|
||||
Access the admin panel at `http://localhost:5000` (or your configured host/port).
|
||||
|
||||
### Features
|
||||
|
||||
- **Dashboard**: Overview of tasks, users, and system status
|
||||
- **Task Management**: View, create, and manage download tasks
|
||||
- **User Management**: View and manage authorized users
|
||||
- **OTP Authentication**: Secure login with one-time passwords
|
||||
|
||||
### First Login
|
||||
|
||||
1. Enter your Telegram User ID
|
||||
2. Request an OTP code
|
||||
3. Check your Telegram messages for the OTP
|
||||
4. Enter the OTP to log in
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
tgloader/
|
||||
├── alembic/ # Database migrations
|
||||
│ ├── versions/ # Migration files
|
||||
│ └── env.py # Alembic environment configuration
|
||||
├── bot/ # Telegram bot module
|
||||
│ ├── modules/
|
||||
│ │ ├── access_control/ # User authorization and permissions
|
||||
│ │ ├── media_loader/ # Media download handlers
|
||||
│ │ ├── message_handler/ # Bot command and callback handlers
|
||||
│ │ └── task_scheduler/ # Task queue and executor
|
||||
│ └── utils/ # Utility functions
|
||||
├── shared/ # Shared code between bot and web
|
||||
│ ├── config.py # Application settings
|
||||
│ └── database/ # Database models and session management
|
||||
├── web/ # Web application
|
||||
│ ├── admin/ # Admin panel routes and templates
|
||||
│ └── utils/ # Web utilities (auth, CSRF, etc.)
|
||||
├── main.py # Application entry point
|
||||
├── requirements.txt # Python dependencies
|
||||
├── docker-compose.yml # Docker Compose configuration
|
||||
└── Dockerfile # Docker image definition
|
||||
```
|
||||
|
||||
## Database Migrations
|
||||
|
||||
The project uses Alembic for database migrations. Migrations are automatically applied on startup.
|
||||
|
||||
### Manual Migration Commands
|
||||
|
||||
```bash
|
||||
# Create a new migration
|
||||
alembic revision --autogenerate -m "Description"
|
||||
|
||||
# Apply migrations
|
||||
alembic upgrade head
|
||||
|
||||
# Rollback one migration
|
||||
alembic downgrade -1
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
All configuration is done through environment variables. See the `.env.local` example above for all available options.
|
||||
|
||||
### Key Settings
|
||||
|
||||
- **PRIVATE_MODE**: If `true`, only users in `AUTHORIZED_USERS` or database can use the bot
|
||||
- **USE_REDIS_SESSIONS**: Use Redis for session storage instead of in-memory (recommended for production)
|
||||
- **MAX_CONCURRENT_TASKS**: Limit the number of simultaneous downloads per user
|
||||
- **COOKIES_FILE**: Path to cookies file for Instagram authentication
|
||||
|
||||
## Development
|
||||
|
||||
### Code Style
|
||||
|
||||
- Follow PEP 8 style guidelines
|
||||
- Use type hints for all function parameters and return values
|
||||
- Write docstrings for all classes and functions
|
||||
- Use English for all code, comments, and documentation
|
||||
|
||||
### Testing
|
||||
|
||||
Run the application locally and test with various URLs:
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
```
|
||||
|
||||
### Logging
|
||||
|
||||
Logs are written to `logs/bot.log` by default. Log level can be configured via `LOG_LEVEL` environment variable.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Instagram Downloads Fail
|
||||
|
||||
- Ensure `COOKIES_FILE` is set and the cookies file exists
|
||||
- Cookies file must be in Netscape format
|
||||
- Verify cookies are not expired
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
- Check `DATABASE_URL` is correct
|
||||
- Ensure PostgreSQL is running (if using PostgreSQL)
|
||||
- Verify database credentials
|
||||
|
||||
### Large File Downloads
|
||||
|
||||
Files larger than 2GB are automatically split into parts. Each part is sent separately to Telegram.
|
||||
|
||||
## License
|
||||
|
||||
[Add your license here]
|
||||
|
||||
## Contributing
|
||||
|
||||
[Add contribution guidelines here]
|
||||
|
||||
## Support
|
||||
|
||||
[Add support information here]
|
||||
|
||||
Reference in New Issue
Block a user