Skip to main content

Introduction

Author: Leon Schmid

The OpenEU Backend is built using FastAPI and deployed on Render, leveraging a modern Python architecture with comprehensive data collection, AI-powered processing, and robust authentication. The system integrates multiple European parliamentary data sources through automated scrapers and provides intelligent analysis through LLM services.

🛠 Core Framework & Runtime​

  • FastAPI – Modern, fast web framework for building APIs with automatic OpenAPI documentation.
  • Python 3.13 – Latest Python runtime with enhanced performance and type safety.
  • Uvicorn – Lightning-fast ASGI server implementation.
  • Poetry – Modern dependency management and packaging tool.

🗄️ Database & Authentication​

  • Supabase – PostgreSQL database with real-time subscriptions, authentication, and API generation.
  • PostgreSQL – Robust relational database with advanced querying capabilities.
  • JWT Authentication – Secure token-based authentication with Supabase integration.

🤖 AI & Machine Learning​

  • OpenAI – GPT models for intelligent text processing and analysis.
  • Cohere – Enterprise-grade language models for text understanding.
  • LangChain – Framework for developing applications with language models.

🕷️ Data Collection & Web Scraping​

  • Scrapy – Professional web scraping framework for large-scale data extraction.
  • Playwright – Browser automation for dynamic content scraping.
  • crawl4ai – AI-powered web crawling and content extraction.
  • BeautifulSoup) – HTML/XML parsing library.
  • Requests – HTTP library for API integrations.

📧 Communication & Translation​

  • Brevo – Transactional email service for notifications and alerts.
  • OpenAI Translation – Custom LLM-powered multilingual translation to English with custom prompt.
  • Jinja2 – Modern templating engine for email and document generation.

⚡ Performance & Caching​

  • FastAPI-Cache2 – High-performance caching with in-memory backend.
  • rapidfuzz – Fast string matching and fuzzy search.

🔄 Background Jobs & Scheduling​

  • schedule – Python job scheduling library for automated tasks.
  • Custom JobScheduler – Enhanced scheduling wrapper with timeout handling and error tracking.
  • Background Processing – Process-based and thread-based execution for different job types.
  • For detailed information, see our Scheduling & Jobs documentation.

🧪 Code Quality & Linting​

  • Ruff – Extremely fast Python linter and formatter (replaces Black, isort, flake8).
  • MyPy – Static type checker for Python code.
  • pre-commit – Git hooks for automated code quality checks.
  • Type Hints – Comprehensive type annotations for better IDE support and error detection.

🐳 Deployment & Infrastructure​

  • Render – Cloud deployment platform with automatic builds and scaling.
  • Docker – Containerization for consistent development and deployment environments.
  • GitHub Actions – Automated CI/CD pipeline for testing and deployment.
  • Supabase Branching – Database preview deployments for feature development.

🔐 Security Features​

  • JWT Token Validation – Secure authentication with Supabase
  • API Key Protection – Background job endpoints secured with tokens
  • CORS Policy – Restricted to approved domains (localhost, netlify.app, openeu.csee.tech)
  • Environment Isolation – Development vs. production configuration separation