Apache Airflow: Airflow 3.1.0

We are thrilled to announce the release of **Apache Airflow 3.1.0**, an update that puts humans at the center of data workflows. Read more about what 3.1.0 brings in https://airflow.apache.org/blog/airflow-3.1.0/ 📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/ \ 📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/ \ 📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/ \ 🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html \ 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0 Apache Airflow 3.1.0 represents an extraordinary community effort, showcasing the vibrant ecosystem that drives this project forward with **163 contributors** making this release possible across **1,400+ commits**. ## Screenshots    <img width="1600" height="775" alt="favorite" src="https://github.com/user-attachments/assets/0dec07d2-56b6-4b02-adcc-6431d89fa52c" /> <img width="1871" height="1118" alt="gantt" src="https://github.com/user-attachments/assets/3a53106f-352c-477c-9398-f2f7c5ae8b16" />  ## New Features - Add Calendar and Gantt chart views to modern React UI with enhanced filtering (#54252, #51667) - Add Python 3.13 support for Airflow runtime and dependencies (#46891) - Add ``SQLAlchemy 2.0`` support with various compatibility fixes for ``Python 3.13`` (#52233, #52518, #54940) - Add support for the ``psycopg3`` postgres driver (#52976) - Add ability to track & display user who triggers DAG runs (#51738, #53510, #54164, #55112) - Add toggle for log grouping in task log viewer for better organization (#51146) - Add tag filtering improvements with Any/All selection options (#51162) - Add comprehensive filtering for DAG runs, task instances, and audit logs (#53652, #54210, #55082) - Add ``XCom`` browsing with filtering and improved navigation (#54049) - Add bulk task instance actions and deletion endpoints (#50443, #50165, #50235) - Add DAG run deletion functionality through UI (#50368) - Add test connection button for connection validation (#51055) - Add hyperlink support for URLs in XCom values (#54288) - Add pool column to task instances list and improve pool integration (#51185, #51031) - Add drag-and-drop log grouping and improved log visualization (#51146) - Add color support for XCom JSON display (#51323) - Add configuration column to DAG runs page (#51270) - Add enhanced note visibility and management in task headers (#51764, #54163) - Introduce React plugin system (AIP-68) for modern UI extensions (#52255) - Add support for external view plugins via iframe integration (#51003, #51889) - Add dashboard integration capabilities for custom React apps (#54131, #54144) - Add comprehensive plugin development tools and documentation (#53643) - Implement complete HITL operator suite (``HITLOperator``, ``ApprovalOperator``, ``HITLEntryOperator``) for human decision workflows (#52868) - Add HITL UI integration with role-based access and form handling (#53035) - Add HITL API endpoints with filtering and query support (#53376, #53923) - Add HITL utility functions for generating URLs to required actions page (#54827) - Improve HITL user experience with bug fixes, UI enhancements, and data model consistency (#55463, #55539, #55575, #55546, #55543, #55536, #55535) - Add ordering and filtering support for HITL details endpoints (#55217) - Add "No Response Received" required action state (#55149) - Add operator filter for HITL task instances (#54773) - Implement deadline alert system for proactive DAG monitoring (AIP-86) (#53951, #53903, #53201, #55086) - Add configurable reference points and notification callbacks (#50677, #50093) - Add deadline calculation and tracking in DAG execution lifecycle (#51638, #50925) - Add comprehensive UI translation support for 16 languages (#51266, #51038, #51219, #50929, #50981, #51793 and more) - Add right-to-left (RTL) layout support for Arabic and Hebrew (#51376) - Add language selection interface and browser preference detection (#51369) - Add translation completeness validation and automated checks (#51166, #51131) - Add calendar data API endpoints for DAG execution visualization (#52748) - Add endpoint to watch DAG runs until completion (#51920, #53346) - Add DAG run ID pattern search functionality (#52437) - Add multi-sorting capabilities for improved data navigation (#53408) - Add bulk connection deletion API and UI (#51201) - Add task group detail pages across DAG runs (#50412, #50309) - Add asset event tracking with last event timestamps (#50060, #50279) - Add ``has_import_errors`` filter to Core API GET ``/dags`` endpoint (#54563) - Add dag_version filter to get_dag_runs endpoint (#54882) - Add pattern search for event log endpoint (#55114) - Add dry_run support with consistent audit log handling (#55116) - Add utility functions for generic filter counting (#54817) - Add keyboard navigation for Grid view interface (#51784) - Add improved error handling for plugin import failures (#49643) - Add plugin validation in ``/plugins`` API with warnings for invalid plugins (#55673) - Improve accessibility for screen readers and assistive technologies with proper language detection (#55839) - Add enhanced variable management with upsert operations (#48547) - Add favorites/pinning support for DAG dashboard organization (#51264) - Add system theme support with automatic OS preference detection (#52649) - Add hotkey shortcut to toggle between Grid and Graph views (#54667) - Add queued DAGs filter button to DAGs page (#55052) - Add DAG parsing duration visibility in UI (#54752) - Add owner links support in DAG Header UI for better navigation (#50627) - Add ``dag_display_name`` aliases for improved API consistency (#50332, #50065, #50014, #49933, #49641) - Add enhanced search capabilities with SearchParamsKeys constants (#55218) - Add ALL_DONE_MIN_ONE_SUCCESS trigger rule for flexible task dependencies (#53959) - Add fail_when_dag_is_paused parameter to TriggerDagRunOperator for better control (#48214) - Add ``XCom`` validation to prevent empty keys in ``XCom.set()`` and ``XCom.get()`` operations (#46929) - Add collapsible plugin menu when multiple plugins are present (#55265) - Add external view plugin categories (admin, browse, docs, user) (#52737) - Add iframe plugins integration to DAG pages (#52795) - Add plugin error display in UI with comprehensive error handling (#49643, #49436) - Add collapsible failed task logs to prevent React error overflow (#54377) - Add dynamic legend system for calendar view (#55155) - Add React UI for Edge functionality (#53563) - Add pending actions display to DAG UI (#55041) - Add description field for filter parameters (#54903) - Add Catalan language support to Airflow UI (#55013) - Add Hungarian language support to Airflow UI (#54716) - Add map_index validation in categorize_task_instances (#54791) - Add Grid view UX improvements (#54846) - Add HITL UX improvements for better user experience (#54990) - Add async support for Notifiers (AIP-86) (#53831) - Add filtering capabilities for tasks view (#54484) - Add asset-based filtering support to DAG API endpoint (#54263) - Add iframe plugins to navigation (#51706) - Add RTL (right-to-left) layout support for Arabic and Hebrew (#51376) - Add test connection button to UI (#51055) - Add task instance bulk actions endpoint (#50443) - Add connection bulk deletion functionality (#51201) - Add pool column to task instances list (#51185) - Add ``iframe_views`` to backend plugin support (#51003) - Add keyboard shortcuts to clear and mark state for task instances and DAG runs (#50885) - Add deadline relationship to DAG runs and deadline model (#50925, #50093) - Add DAG run deletion UI (#50368) - Add task instance deletion UI and endpoint (#50235, #50165) - Switch all airflow logging to structlog (#52651, #55434, #55431, #55638) - Add Filter Bar to Audit Log (#55487) - Add Filters UI for Asset View (#54640) - Update color palette and leverage Chakra semantic tokens (#53981, #55739) - Improve calendar view UI with enhanced tooltips and visual fixes (#55476) ## Bug Fixes - Fix DAG list filtering to include ``QUEUED`` runs with null ``start_date`` (#52668) - Fix XCom deletion failure for mapped task instances through bulk deletion API (#51850) - Fix XCom deletion failure for mapped task instances (#54954) - Fix task timeout handling within task SDK (#54089) - Fix task instance tries API duplicate entries (#50597) - Fix connection validation and type checking during construction (#54759) - Fix mapped task instance index display in Task Instances tab (#55363) - Fix Gantt chart state mismatch with Grid view (#55300) - Fix Gantt chart status color display issues (#55296) - Fix XCom mapping for dynamically-mapped task groups (#51556) - Fix missing ``ti_successes`` and related metrics in Airflow 3.0 Task SDK (#55322) - Fix bulk operation permissions for connection, pool and variable (#55278) - Fix ``clearTaskInstances`` API: Restore ``include_past``/``future`` support on UI (#54416) - Fix migration when XCom has NaN values (#53812) - Fix HITL related UI schema generated by prek hooks (#55204) - Fix consistent no-log handling for tasks with try_number=0 in API and UI (#55035) - Fix timezone conversion in datetime trigger parameters (#54593) - Fix audit log payload for DAG pause/unpause actions (#55091) - Fix pushing None as an XCom value (#55080) - Fix scheduler processing of cleared running tasks stuck in RESTARTING state (#55084) - Fix XCom deletion failure for mapped task instances (#54954) - Fix outgoing graph edges should exit opposite of incoming edges (#54789) - Fix external links in Navigation buttons (#52220) - Fix Error when viewing DAG details of a no longer configured bundle (#52086) - Fix compatibility with new numpy and pandas versions (#52071) - Fix connection recovery from URI when host has protocol (#51953) - Fix last DAG run not showing on DAG listing (#51115) - Fix task instance tries API returning duplicate entries (#50597) - Fix Graph view vanishing and loading issues (#53886, #54756) - Fix rendered template display formatting for better readability (#53657) - Fix Grid view expand/collapse button functionality (#54257) - Fix tooltip visibility and positioning issues (#53913) - Fix grid keyboard navigation focus management (#54271) - Fix plugin registration for invalid objects and middleware registration (#55264, #55399) - Fix external links for plugins with undefined URL routes (#55221) - Fix language display consistency and flag representation (#51560, #51177) - Fix RTL layout rendering for Arabic and Hebrew interfaces (#51853) - Fix graph export cropping when view is partial (#55012) - Fix log viewer "Toggle Source" to hide only source fields, not all structured log fields (#55474) - Output on stdout/stderr from within tasks is now filterable in the Sources list in the UI log view (#55508) - Redact JWT tokens in task logs (#55499) - Fix grid view to handle long task name (#55332) - Allow slash characters in Variable keys similar to Airflow 2.x (#55324) - Fix Grid cache invalidation for multi-run task operations (#55504) - Fix Gantt chart rendering issues (#55554) - Fix ``XCom`` access in DAG processor callbacks for notifiers (#55542) - Fix alignment of arrows in RTL mode for right-to-left languages (#55619) - Fix connection form extras not inferring correct type in UI (#55492) - Fix incorrect log timestamps in UI when ``default_timezone`` is not UTC (#54431) - Fix handling of priority_weight for DAG processor callbacks (#55436) - Fix pointless requests from Gantt view when there is no Run ID (#55668) - Ensure filename and ``lineno`` of logger calls are present in Task Logs (#55581) - Fix DAG disappearing after callback execution in stale detection (#55698) - Fix DB downgrade to Airflow 2 when fab tables exists (#55738) - Fix UI stats endpoint causing dashboard loading issues (#55733) - Fix unintended console output when DAG not found in ``serialized_dag`` table (#54972) - Fix scheduler handling of orphaned tasks from Airflow 2 during upgrade (#55848) - Fix logging format to respect existing configuration during upgrade to prevent unexpected log format changes (#55824) - Fix Grid view crashes when DAG version information is missing (#55771) - Fix compatibility for custom triggers migrating from Airflow 2.x that use synchronous connection calls (#55799) - Fix DAG runs triggered from UI incorrectly marked as REST API triggers instead of UI triggers (#54650) - Fix XCom API responses failing when encountering non-serializable objects by falling back to string representation (#55880) - Fix asset queue display in UI showing incorrect timestamps for deleted queue events (#54652) - Fix SQLite database migrations failing due to foreign key constraint handling (#55883) - Fix DAG deserialization failure when using non-default weight_rule values like 'absolute' (#55906) - Fix async connection retrieval in triggerer context preventing event loop blocking (#55812) - Fix Airflow downgrade compatibility by handling serialized DAG format conversion from v3 to v2 (#55975) - Fix 'All Log Levels' filter not working in task log viewer (#55851) - Fix Grid view scrollbar overlapping issues on Firefox browser (#55960) - Fix Gantt chart misalignment with Grid view layout (#55995) - Fix Grid view task names being extremely collapsed and unreadable when displaying many DAG runs (#55997) - Fix ``LocalExecutor`` race condition where tasks could start before database state was committed (#56010) - Use SecretCache for connection and variable access in task sdk (#55972) ## Miscellaneous - Move secrets masker to shared distribution for better modularity (#54449) - Move email notifications from scheduler to DAG processor for better architecture (#55238) - Add graph UI load optimization with latest run info endpoint (#53429) - Optimize UI bundle size by moving translations to dynamic loading (#51735) - Relocate Task SDK components for improved separation (#55174, #54795) - Refactor trigger rule utilities and weight rule consolidation (#54797, #53393) - Remove deprecated Airflow 2.x modules and legacy imports (#50482) - Clean up unused code and improve module organization (#52176, #52173, #53031) - Add SQLAlchemy 2.0 CI support for future compatibility (#52233) - Improve test fixtures and SDK communication testing (#54795, #50603) - Add translation completeness linting and validation tools (#51166) - Upgrade to latest versions of important dependencies (#55350) - Move webserver configuration options to API section (#50693, #50656) - Improve DAG bundle handling and versioning support (#47592) - Add database management CLI tools for external database operations (#50657) - Add comprehensive HITL operator documentation and examples (#54618) - Add guards for registering middlewares from plugins (#55399) - Optimize Gantt group expansion with de-bouncing and deferred rendering (#55334) - Differentiate between triggers and watchers currently running for better visibility (#55376) - Removed unused config: ``dag_stale_not_seen_duration`` (#55601, #55684) - Update UI's query client strategy for improved performance (#55528) - Unify datetime format across the UI for consistency (#55572) - Mark React Apps as Experimental for Airflow 3.1 release (#55478) - Improve OOM error messaging for clearer task failure diagnosis (#55602) - Display responder username for better audit trail in HITL workflows (#55509) - The constraint file do not contain developer dependencies anymore (#53631) - Add hyperlinks to ``dag_id`` column in DAG Runs and Task Instances pages for better navigation (#55648) - Add responsive web design (RWD) support to Grid view (#55745) ## Doc Only Changes - Add comprehensive Human-in-the-Loop operator tutorial and examples (#54618) - Add deadline alerts configuration and usage documentation (#53727) - Make term Dag consistent in docs task-sdk (#55100) - Add migration guide for upgrading from legacy SLA functionality to deadline alerts (#55743) - Add DAG bundles triggerer limitation documentation (#55232) - Add deadline alerts usage guides and best practices (#53727) - Remove ``--preview`` flag from ``ruff check`` instructions for Airflow 3 upgrade path (#55516) - Add documentation for context parameter (#55377)