Apache Airflow 2.3.0

# New Features - Add dynamic task mapping (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-42+milestone%3A%22Airflow+2.3.0%22) - New Grid View replaces Tree View (#18675) - Templated ``requirements.txt`` in Python Operators (#17349) - Allow reuse of decorated tasks (#22941) - Move the database configuration to a new section (#22284) - Add ``SmoothOperator`` (#22813) - Make operator's ``execution_timeout`` configurable (#22389) - Events Timetable (#22332) - Support dag serialization with custom ``ti_deps`` rules (#22698) - Support log download in task log view (#22804) - support for continue backfill on failures (#22697) - Add ``dag-processor`` cli command (#22305) - Add possibility to create users in LDAP mode (#22619) - Add ``ignore_first_depends_on_past`` for scheduled jobs (#22491) - Update base sensor operator to support XCOM return value (#20656) - Add an option for run id in the ui trigger screen (#21851) - Enable JSON serialization for connections (#19857) - Add REST API endpoint for bulk update of DAGs (#19758) - Add queue button to click-on-DagRun interface. (#21555) - Add ``list-import-errors`` to ``airflow dags`` command (#22084) - Store callbacks in database if ``standalone_dag_processor`` config is True. (#21731) - Add LocalKubernetesExecutor (#19729) - Add ``celery.task_timeout_error`` metric (#21602) - Airflow ``db downgrade`` cli command (#21596) - Add ``ALL_SKIPPED`` trigger rule (#21662) - Add ``db clean`` CLI command for purging old data (#20838) - Add ``celery_logging_level`` (#21506) - Support different timeout value for dag file parsing (#21501) - Support generating SQL script for upgrades (#20962) - Add option to compress Serialized dag data (#21332) - Branch python operator decorator (#20860) - Add Audit Log View to Dag View (#20733) - Add missing StatsD metric for failing SLA Callback notification (#20924) - Add ``ShortCircuitOperator`` configurability for respecting downstream trigger rules (#20044) - Allow using Markup in page title in Webserver (#20888) - Add Listener Plugin API that tracks TaskInstance state changes (#20443) - Add context var hook to inject more env vars (#20361) - Add a button to set all tasks to skipped (#20455) - Cleanup pending pods (#20438) - Add config to warn public deployment exposure in UI (#18557) - Log filename template records (#20165) - Added windows extensions (#16110) - Showing approximate time until next dag_run in Airflow (#20273) - Extend config window on UI (#20052) - Add show dag dependencies feature to CLI (#19985) - Add cli command for 'airflow dags reserialize` (#19471) - Add missing description field to Pool schema(REST API) (#19841) - Introduce DagRun action to change state to queued. (#19353) - Add DAG run details page (#19705) - Add role export/import to cli tools (#18916) - Adding ``dag_id_pattern`` parameter to the ``/dags`` endpoint (#18924) # Improvements - Show schedule_interval/timetable description in UI (#16931) - Added column duration to DAG runs view (#19482) - Enable use of custom conn extra fields without prefix (#22607) - Initialize finished counter at zero (#23080) - Improve logging of optional provider features messages (#23037) - Meaningful error message in resolve_template_files (#23027) - Update ImportError items instead of deleting and recreating them (#22928) - Add option ``--skip-init`` to db reset command (#22989) - Support importing connections from files with ".yml" extension (#22872) - Support glob syntax in ``.airflowignore`` files (#21392) (#22051) - Hide pagination when data is a single page (#22963) - Support for sorting DAGs in the web UI (#22671) - Speed up ``has_access`` decorator by ~200ms (#22858) - Add XComArg to lazy-imported list of Airflow module (#22862) - Add more fields to REST API dags/dag_id/details endpoint (#22756) - Don't show irrelevant/duplicated/"internal" Task attrs in UI (#22812) - No need to load whole ti in current_state (#22764) - Pickle dag exception string fix (#22760) - Better verification of Localexecutor's parallelism option (#22711) - log backfill exceptions to sentry (#22704) - retry commit on MySQL deadlocks during backfill (#22696) - Add more fields to REST API get DAG(dags/dag_id) endpoint (#22637) - Use timetable to generate planned days for current year (#22055) - Disable connection pool for celery worker (#22493) - Make date picker label visible in trigger dag view (#22379) - Expose ``try_number`` in airflow vars (#22297) - Add generic connection type (#22310) - Add a few more fields to the taskinstance finished log message (#22262) - Pause auto-refresh if scheduler isn't running (#22151) - Show DagModel details. (#21868) - Add pip_install_options to PythonVirtualenvOperator (#22158) - Show import error for ``airflow dags list`` CLI command (#21991) - Pause auto-refresh when page is hidden (#21904) - Default args type check (#21809) - Enhance magic methods on XComArg for UX (#21882) - py files don't have to be checked ``is_zipfiles`` in refresh_dag (#21926) - Fix TaskDecorator type hints (#21881) - Add 'Show record' option for variables (#21342) - Use DB where possible for quicker ``airflow dag`` subcommands (#21793) - REST API: add rendered fields in task instance. (#21741) - Change the default auth backend to session (#21640) - Don't check if ``py`` DAG files are zipped during parsing (#21538) - Switch XCom implementation to use ``run_id`` (#20975) - Action log on Browse Views (#21569) - Implement multiple API auth backends (#21472) - Change logging level details of connection info in ``get_connection()`` (#21162) - Support mssql in airflow db shell (#21511) - Support config ``worker_enable_remote_control`` for celery (#21507) - Log memory usage in ``CgroupTaskRunner`` (#21481) - Modernize DAG-related URL routes and rename "tree" to "grid" (#20730) - Move Zombie detection to ``SchedulerJob`` (#21181) - Improve speed to run ``airflow`` by 6x (#21438) - Add more SQL template fields renderers (#21237) - Simplify fab has access lookup (#19294) - Log context only for default method (#21244) - Log trigger status only if at least one is running (#21191) - Add optional features in providers. (#21074) - Better multiple_outputs inferral for @task.python (#20800) - Improve handling of string type and non-attribute ``template_fields`` (#21054) - Remove un-needed deps/version requirements (#20979) - Correctly specify overloads for TaskFlow API for type-hinting (#20933) - Introduce notification_sent to SlaMiss view (#20923) - Rewrite the task decorator as a composition (#20868) - Add "Greater/Smaller than or Equal" to filters in the browse views (#20602) (#20798) - Rewrite DAG run retrieval in task command (#20737) - Speed up creation of DagRun for large DAGs (5k+ tasks) by 25-130% (#20722) - Make native environment Airflow-flavored like sandbox (#20704) - Better error when param value has unexpected type (#20648) - Add filter by state in DagRun REST API (List Dag Runs) (#20485) - Prevent exponential memory growth in Tasks with custom logging handler (#20541) - Set default logger in logging Mixin (#20355) - Reduce deprecation warnings from www (#20378) - Add hour and minute to time format on x-axis of all charts using nvd3.lineChart (#20002) - Add specific warning when Task asks for more slots than pool defined with (#20178) - UI: Update duration column for better human readability (#20112) - Use Viewer role as example public role (#19215) - Properly implement DAG param dict copying (#20216) - ``ShortCircuitOperator`` push XCom by returning python_callable result (#20071) - Add clear logging to tasks killed due to a Dagrun timeout (#19950) - Change log level for Zombie detection messages (#20204) - Better confirmation prompts (#20183) - Only execute TIs of running DagRuns (#20182) - Check and run migration in commands if necessary (#18439) - Log only when Zombies exists (#20118) - Increase length of the email and username (#19932) - Add more filtering options for TI's in the UI (#19910) - Dynamically enable "Test Connection" button by connection type (#19792) - Avoid littering postgres server logs with "could not obtain lock" with HA schedulers (#19842) - Renamed ``Connection.get_hook`` parameter to make it the same as in ``SqlSensor`` and ``SqlOperator``. (#19849) - Add hook_params in SqlSensor using the latest changes from PR #18718. (#18431) - Speed up webserver boot time by delaying provider initialization (#19709) - Configurable logging of ``XCOM`` value in PythonOperator (#19378) - Minimize production js files (#19658) - Add ``hook_params`` in ``BaseSqlOperator`` (#18718) - Add missing "end_date" to hash components (#19281) - More friendly output of the airflow plugins command + add timetables (#19298) - Add sensor default timeout config (#19119) - Update ``taskinstance`` REST API schema to include dag_run_id field (#19105) - Adding feature in bash operator to append the user defined env variable to system env variable (#18944) - Duplicate Connection: Added logic to query if a connection id exists before creating one (#18161) # Bug Fixes - Use inherited 'trigger_tasks' method (#23016) - In DAG dependency detector, use class type instead of class name (#21706) - Fix tasks being wrongly skipped by schedule_after_task_execution (#23181) - Fix X-Frame enabled behaviour (#23222) - Allow ``extra`` to be nullable in connection payload as per schema(REST API). (#23183) - Fix ``dag_id`` extraction for dag level access checks in web ui (#23015) - Fix timezone display for logs on UI (#23075) - Include message in graph errors (#23021) - Change trigger dropdown left position (#23013) - Don't add planned tasks for legacy DAG runs (#23007) - Add dangling rows check for TaskInstance references (#22924) - Validate the input params in connection ``CLI`` command (#22688) - Fix trigger event payload is not persisted in db (#22944) - Drop "airflow moved" tables in command ``db reset`` (#22990) - Add max width to task group tooltips (#22978) - Add template support for ``external_task_ids``. (#22809) - Allow ``DagParam`` to hold falsy values (#22964) - Fix regression in pool metrics (#22939) - Priority order tasks even when using pools (#22483) - Do not clear XCom when resuming from deferral (#22932) - Handle invalid JSON metadata in ``get_logs_with_metadata endpoint``. (#22898) - Fix pre-upgrade check for rows dangling w.r.t. dag_run (#22850) - Fixed backfill interference with scheduler (#22701) - Support conf param override for backfill runs (#22837) - Correctly interpolate pool name in ``PoolSlotsAvailableDep`` statues (#22807) - Fix ``email_on_failure`` with ``render_template_as_native_obj`` (#22770) - Fix processor cleanup on ``DagFileProcessorManager`` (#22685) - Prevent meta name clash for task instances (#22783) - remove json parse for gantt chart (#22780) - Check for missing dagrun should know version (#22752) - Fixes ``ScheduleInterval`` spec (#22635) - Fixing task status for non-running and non-committed tasks (#22410) - Do not log the hook connection details even at DEBUG level (#22627) - Stop crashing when empty logs are received from kubernetes client (#22566) - Fix bugs about timezone change (#22525) - Fix entire DAG stops when one task has end_date (#20920) - Use logger to print message during task execution. (#22488) - Make sure finalizers are not skipped during exception handling (#22475) - update smart sensor docs and minor fix on ``is_smart_sensor_compatible()`` (#22386) - Fix ``run_id`` k8s and elasticsearch compatibility with Airflow 2.1 (#22385) - Allow to ``except_skip`` None on ``BranchPythonOperator`` (#20411) - Fix incorrect datetime details (DagRun views) (#21357) - Remove incorrect deprecation warning in secrets backend (#22326) - Remove ``RefreshConfiguration`` workaround for K8s token refreshing (#20759) - Masking extras in GET ``/connections/<connection>`` endpoint (#22227) - Set ``queued_dttm`` when submitting task to directly to executor (#22259) - Addressed some issues in the tutorial mentioned in discussion #22233 (#22236) - Change default python executable to python3 for docker decorator (#21973) - Don't validate that Params are JSON when NOTSET (#22000) - Add per-DAG delete permissions (#21938) - Fix handling some None parameters in kubernetes 23 libs. (#21905) - Fix handling of empty (None) tags in ``bulk_write_to_db`` (#21757) - Fix DAG date range bug (#20507) - Removed ``request.referrer`` from views.py (#21751) - Make ``DbApiHook`` use ``get_uri`` from Connection (#21764) - Fix some migrations (#21670) - [de]serialize resources on task correctly (#21445) - Add params ``dag_id``, ``task_id`` etc to ``XCom.serialize_value`` (#19505) - Update test connection functionality to use custom form fields (#21330) - fix all "high" npm vulnerabilities (#21526) - Fix bug incorrectly removing action from role, rather than permission. (#21483) - Fix relationship join bug in FAB/SecurityManager with SQLA 1.4 (#21296) - Use Identity instead of Sequence in SQLAlchemy 1.4 for MSSQL (#21238) - Ensure ``on_task_instance_running`` listener can get at task (#21157) - Return to the same place when triggering a DAG (#20955) - Fix task ID deduplication in ``@task_group`` (#20870) - Add downgrade to some FAB migrations (#20874) - Only validate Params when DAG is triggered (#20802) - Fix ``airflow trigger`` cli (#20781) - Fix task instances iteration in a pool to prevent blocking (#20816) - Allow depending to a ``@task_group`` as a whole (#20671) - Use original task's ``start_date`` if a task continues after deferral (#20062) - Disabled edit button in task instances list view page (#20659) - Fix a package name import error (#20519) (#20519) - Remove ``execution_date`` label when get cleanup pods list (#20417) - Remove unneeded FAB REST API endpoints (#20487) - Fix parsing of Cloudwatch log group arn containing slashes (#14667) (#19700) - Sanity check for MySQL's TIMESTAMP column (#19821) - Allow using default celery command group with executors subclassed from Celery-based executors. (#18189) - Move ``class_permission_name`` to mixin so it applies to all classes (#18749) - Adjust trimmed_pod_id and replace '.' with '-' (#19036) - Pass custom_headers to send_email and send_email_smtp (#19009) - Ensure ``catchup=False`` is used in example dags (#19396) - Edit permalinks in OpenApi description file (#19244) - Navigate directly to DAG when selecting from search typeahead list (#18991) - [Minor] Fix padding on home page (#19025) # Doc only changes - Update doc for DAG file processing (#23209) - Replace changelog/updating with release notes and ``towncrier`` now (#22003) - Fix wrong reference in tracking-user-activity.rst (#22745) - Remove references to ``rbac = True`` from docs (#22725) - Doc: Update description for executor-bound dependencies (#22601) - Update check-health.rst (#22372) - Stronger language about Docker Compose customizability (#22304) - Update logging-tasks.rst (#22116) - Add example config of ``sql_alchemy_connect_args`` (#22045) - Update best-practices.rst (#22053) - Add information on DAG pausing/deactivation/deletion (#22025) - Add brief examples of integration test dags you might want (#22009) - Run inclusive language check on CHANGELOG (#21980) - Add detailed email docs for Sendgrid (#21958) - Add docs for ``db upgrade`` / ``db downgrade`` (#21879) - Update modules_management.rst (#21889) - Fix UPDATING section on SqlAlchemy 1.4 scheme changes (#21887) - Update TaskFlow tutorial doc to show how to pass "operator-level" args. (#21446) - Fix doc - replace decreasing by increasing (#21805) - Add another way to dynamically generate DAGs to docs (#21297) - Add extra information about time synchronization needed (#21685) - Update debug.rst docs (#21246) - Replaces the usage of ``postgres://`` with ``postgresql://`` (#21205) - Fix task execution process in ``CeleryExecutor`` docs (#20783) # Misc/Internal - Bring back deprecated security manager functions (#23243) - Replace usage of ``DummyOperator`` with ``EmptyOperator`` (#22974) - Deprecate ``DummyOperator`` in favor of ``EmptyOperator`` (#22832) - Remove unnecessary python 3.6 conditionals (#20549) - Bump ``moment`` from 2.29.1 to 2.29.2 in /airflow/www (#22873) - Bump ``prismjs`` from 1.26.0 to 1.27.0 in /airflow/www (#22823) - Bump ``nanoid`` from 3.1.23 to 3.3.2 in /airflow/www (#22803) - Bump ``minimist`` from 1.2.5 to 1.2.6 in /airflow/www (#22798) - Remove dag parsing from db init command (#22531) - Update our approach for executor-bound dependencies (#22573) - Use ``Airflow.Base.metadata`` in FAB models (#22353) - Limit docutils to make our documentation pretty again (#22420) - Add Python 3.10 support (#22050) - [FEATURE] add 1.22 1.23 K8S support (#21902) - Remove pandas upper limit now that SQLA is 1.4+ (#22162) - Patch ``sql_alchemy_conn`` if old postgres scheme used (#22333) - Protect against accidental misuse of XCom.get_value() (#22244) - Order filenames for migrations (#22168) - Don't try to auto generate migrations for Celery tables (#22120) - Require SQLAlchemy 1.4 (#22114) - bump sphinx-jinja (#22101) - Add compat shim for SQLAlchemy to avoid warnings (#21959) - Rename ``xcom.dagrun_id`` to ``xcom.dag_run_id`` (#21806) - Deprecate non-JSON ``conn.extra`` (#21816) - Bump upper bound version of ``jsonschema`` to 5.0 (#21712) - Deprecate helper utility ``days_ago`` (#21653) - Remove ```:type``` lines now ``sphinx-autoapi`` supports type hints (#20951) - Silence deprecation warning in tests (#20900) - Use ``DagRun.run_id`` instead of ``execution_date`` when updating state of TIs (UI & REST API) (#18724) - Add Context stub to Airflow packages (#20817) - Update Kubernetes library version (#18797) - Rename ``PodLauncher`` to ``PodManager`` (#20576) - Removes Python 3.6 support (#20467) - Add deprecation warning for non-json-serializable params (#20174) - Rename TaskMixin to DependencyMixin (#20297) - Deprecate passing execution_date to XCom methods (#19825) - Remove ``get_readable_dags`` and ``get_editable_dags``, and ``get_accessible_dags``. (#19961) - Remove postgres 9.6 support (#19987) - Removed hardcoded connection types. Check if hook is instance of DbApiHook. (#19639) - add kubernetes 1.21 support (#19557) - Add FAB base class and set import_name explicitly. (#19667) - Removes unused state transitions to handle auto-changing view permissions. (#19153) - Chore: Use enum for ``__var`` and ``__type`` members (#19303) - Use fab models (#19121) - Consolidate method names between Airflow Security Manager and FAB default (#18726) - Remove distutils usages for Python 3.10 (#19064) - Removing redundant ``max_tis_per_query`` initialisation on SchedulerJob (#19020) - Remove deprecated usage of ``init_role()`` from API (#18820) - Remove duplicate code on dbapi hook (#18821)