Apache Airflow 2.2.0

### New Features - AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030, #16352, #17030, #17122, #17414, #17552, #17755, #17989, #18084, #18088, #18244, #18266, #18420, #18434, #18421, #18475, #18499, #18573, #18522, #18729, #18706, #18742, #18786, #18804) - AIP-40: Add Deferrable "Async" Tasks (#15389, #17564, #17565, #17601, #17745, #17747, #17748, #17875, #17876, #18129, #18210, #18214, #18552, #18728, #18414) - Add a Docker Taskflow decorator (#15330, #18739) - Add Airflow Standalone command (#15826) - Display alert messages on dashboard from local settings (#18284) - Advanced Params using json-schema (#17100) - Ability to test connections from UI or API (#15795, #18750) - Add Next Run to UI (#17732) - Add default weight rule configuration option (#18627) - Add a calendar field to choose the execution date of the DAG when triggering it (#16141) - Allow setting specific ``cwd`` for BashOperator (#17751) - Show import errors in DAG views (#17818) - Add pre/post execution hooks [Experimental] (#17576) - Added table to view providers in Airflow ui under admin tab (#15385) - Adds secrets backend/logging/auth information to provider yaml (#17625) - Add date format filters to Jinja environment (#17451) - Introduce ``RESTARTING`` state (#16681) - Webserver: Unpause DAG on manual trigger (#16569) - API endpoint to create new user (#16609) - Add ``insert_args`` for support transfer replace (#15825) - Add recursive flag to glob in filesystem sensor (#16894) - Add conn to jinja template context (#16686) - Add ``default_args`` for ``TaskGroup`` (#16557) - Allow adding duplicate connections from UI (#15574) - Allow specifying multiple URLs via the CORS config option (#17941) - Implement API endpoint for DAG deletion (#17980) - Add DAG run endpoint for marking a dagrun success or failed(#17839) - Add support for ``kinit`` options ``[-f|-F]`` and ``[-a|-A]`` (#17816) - Queue support for ``DaskExecutor`` using Dask Worker Resources (#16829, #18720) - Make auto refresh interval configurable (#18107) ### Improvements - Small improvements for Airflow UI (#18715, #18795) - Rename ``processor_poll_interval`` to ``scheduler_idle_sleep_time`` (#18704) - Check the allowed values for the logging level (#18651) - Fix error on triggering a dag that doesn't exist using ``dagrun_conf`` (#18655) - Add muldelete action to ``TaskInstanceModelView`` (#18438) - Avoid importing DAGs during clean DB installation (#18450) - Require can_edit on DAG privileges to modify TaskInstances and DagRuns (#16634) - Make Kubernetes job description fit on one log line (#18377) - Always draw borders if task instance state is null or undefined (#18033) - Inclusive Language (#18349) - Improved log handling for zombie tasks (#18277) - Adding ``Variable.update`` method and improving detection of variable key collisions (#18159) - Add note about params on trigger DAG page (#18166) - Change ``TaskInstance`` and ``TaskReschedule`` PK from ``execution_date`` to ``run_id`` (#17719) - Adding ``TaskGroup`` support in ``BaseOperator.chain()`` (#17456) - Advises the kernel to not cache log files generated by Airflow (#18054) - Allow filtering DAGS by tags in the REST API (#18090) - Optimize imports of Providers Manager (#18052) - Adds capability of Warnings for incompatible community providers (#18020) - Serialize the ``template_ext`` attribute to show it in UI (#17985) - Add ``robots.txt`` and ``X-Robots-Tag`` header (#17946) - Refactor ``BranchDayOfWeekOperator``, ``DayOfWeekSensor`` (#17940) - Update error message to guide the user into self-help mostly (#17929) - Update to Celery 5 (#17397) - Avoid redirect loop for users with no permissions (#17838) - Improve graph view load time for dags with open groups (#17821) - Add links to provider's documentation (#17736) - Remove Marshmallow schema warnings (#17753) - Rename ``none_failed_or_skipped`` by ``none_failed_min_one_success`` trigger rule (#17683) - Remove ``[core] store_dag_code`` & use DB to get Dag Code (#16342) - Rename ``task_concurrency`` to ``max_active_tis_per_dag`` (#17708) - Import Hooks lazily individually in providers manager (#17682) - Adding support for multiple task-ids in the external task sensor (#17339) - Replace ``execution_date`` with ``run_id`` in airflow tasks run command (#16666) - Avoid endless redirect loop when user has no roles (#17613) - Make output from users cli command more consistent (#17642) - Open relative extra links in place (#17477) - Move ``worker_log_server_port`` option to the logging section (#17621) - Use gunicorn to serve logs generated by worker (#17591) - Improve validation of Group id (#17578) - Simplify 404 page (#17501) - Add XCom.clear so it's hookable in custom XCom backend (#17405) - Add deprecation notice for ``SubDagOperator`` (#17488) - Support DAGS folder being in different location on scheduler and runners (#16860) - Remove /dagrun/create and disable edit form generated by F.A.B (#17376) - Enable specifying dictionary paths in ``template_fields_renderers`` (#17321) - error early if virtualenv is missing (#15788) - Handle connection parameters added to Extra and custom fields (#17269) - Fix ``airflow celery stop`` to accept the pid file. (#17278) - Remove DAG refresh buttons (#17263) - Deprecate dummy trigger rule in favor of always (#17144) - Be verbose about failure to import ``airflow_local_settings`` (#17195) - Include exit code in ``AirflowException`` str when ``BashOperator`` fails. (#17151) - Adding EdgeModifier support for chain() (#17099) - Only allows supported field types to be used in custom connections (#17194) - Secrets backend failover (#16404) - Warn on Webserver when using ``SQLite`` or ``SequentialExecutor`` (#17133) - Extend ``init_containers`` defined in ``pod_override`` (#17537) - Client-side filter dag dependencies (#16253) - Improve executor validation in CLI (#17071) - Prevent running ``airflow db init/upgrade`` migrations and setup in parallel. (#17078) - Update ``chain()`` and ``cross_downstream()`` to support ``XComArgs`` (#16732) - Improve graph view refresh (#16696) - When a task instance fails with exception, log it (#16805) - Set process title for ``serve-logs`` and ``LocalExecutor`` (#16644) - Rename ``test_cycle`` to ``check_cycle`` (#16617) - Set process title for Worker when using ``LocalExecutor`` (#16623) - Add schema as ``DbApiHook`` instance attribute (#16521, #17423) - Improve compatibility with MSSQL (#9973) - Add transparency for unsupported connection type (#16220) - Call resource based fab methods (#16190) - Format more dates with timezone (#16129) - Replace deprecated ``dag.sub_dag`` with ``dag.partial_subset`` (#16179) - Treat ``AirflowSensorTimeout`` as immediate failure without retrying (#12058) - Marking success/failed automatically clears failed downstream tasks (#13037) - Add close/open indicator for import dag errors (#16073) - Add collapsible import errors (#16072) - Always return a response in TI's ``action_clear`` view (#15980) - Add cli command to delete user by email (#15873) - Use resource and action names for FAB permissions (#16410) - Rename DAG concurrency (``[core] dag_concurrency``) settings for easier understanding (#16267, #18730) - Calendar UI improvements (#16226) - Refactor: ``SKIPPED`` should not be logged again as ``SUCCESS`` (#14822) - Remove version limits for dnspython (#18046, #18162) - Accept custom run ID in TriggerDagRunOperator (#18788) ### Bug Fixes - Make REST API patch user endpoint work the same way as the UI (#18757) - Properly set ``start_date`` for cleared tasks (#18708) - Ensure task_instance exists before running update on its state(REST API) (#18642) - Make ``AirflowDateTimePickerWidget`` a required field (#18602) - Retry deadlocked transactions on deleting old rendered task fields (#18616) - Fix `retry_exponential_backoff` divide by zero error when retry delay is zero (#17003) - Improve how UI handles datetimes (#18611, #18700) - Bugfix: dag_bag.get_dag should return None, not raise exception (#18554) - Only show the task modal if it is a valid instance (#18570) - Fix accessing rendered ``{{ task.x }}`` attributes from within templates (#18516) - Add missing email type of connection (#18502) - Don't use flash for "same-page" UI messages. (#18462) - Fix task group tooltip (#18406) - Properly fix dagrun update state endpoint (#18370) - Properly handle ti state difference between executor and scheduler (#17819) - Fix stuck "queued" tasks in KubernetesExecutor (#18152) - Don't permanently add zip DAGs to ``sys.path`` (#18384) - Fix random deadlocks in MSSQL database (#18362) - Deactivating DAGs which have been removed from files (#17121) - When syncing dags to db remove ``dag_tag`` rows that are now unused (#8231) - Graceful scheduler shutdown on error (#18092) - Fix mini scheduler not respecting ``wait_for_downstream`` dep (#18338) - Pass exception to ``run_finished_callback`` for Debug Executor (#17983) - Make ``XCom.get_one`` return full, not abbreviated values (#18274) - Use try/except when closing temporary file in task_runner (#18269) - show next run if not none (#18273) - Fix DB session handling in ``XCom.set`` (#18240) - Sort adopted tasks in ``_check_for_stalled_adopted_tasks`` method (#18208) - Fix external_executor_id not being set for manually run jobs (#17207) - Fix deleting of zipped Dags in Serialized Dag Table (#18243) - Return explicit error on user-add for duplicated email (#18224) - Remove loading dots even when last run data is empty (#18230) - Swap dag import error dropdown icons (#18207) - Automatically create section when migrating config (#16814) - Set encoding to utf-8 by default while reading task logs (#17965) - Set task state to failed when pod is DELETED while running (#18095) - Apply parent dag permissions to subdags (#18160) - Change id collation for MySQL to case-sensitive (#18072) - Logs task launch exception in ``StandardTaskRunner`` (#17967) - Applied permissions to ``self._error_file`` (#15947) - Hide variable import form if user lacks permission (#18000) - Fix blank dag dependencies view (#17990) - Add missing menu access for dag dependencies and configurations pages (#17450) - Fix passing Jinja templates in ``DateTimeSensor`` (#17959) - Fixing bug which restricted the visibility of ImportErrors (#17924) - Fix grammar in ``traceback.html`` (#17942) - Fix ``DagRunState`` enum query for ``MySQLdb`` driver (#17886) - Fixed button size in "Actions" group. (#17902) - Fix log links on graph TI modal (#17862) - Only show import errors for DAGs a user can access (#17835) - Show all import_errors from zip files (#17759) - fix EXTRA_LOGGER_NAMES param and related docs (#17808) - Use one interpreter for Airflow and gunicorn (#17805) - Fix: Mysql 5.7 id utf8mb3 (#14535) - Fix dag_processing.last_duration metric random holes (#17769) - Automatically use ``utf8mb3_general_ci`` collation for MySQL (#17729) - Fix wrong query on running tis (#17631) - Add root to tree refresh url (#17633) - Do not delete running DAG from the UI (#17630) - Rescue if a DagRun's DAG was removed from db (#17544) - fix: filter condition of ``TaskInstance`` does not work #17535 (#17548) - Regression on pid reset to allow task start after heartbeat (#17333) - Dont use TaskInstance in CeleryExecutor.trigger_tasks (#16248) - Remove locks for upgrades in MSSQL (#17213) - Create virtualenv via python call (#17156) - Ensure a DAG is acyclic when running ``DAG.cli()`` (#17105) - Translate non-ascii characters (#17057) - Change the logic of ``None`` comparison in ``model_list`` template (#16893) - Have UI and POST /task_instances_state API endpoint have same behaviour (#16539) - ensure task is skipped if missing sla (#16719) - Fix direct use of ``cached_property`` module (#16710) - Fix TI success confirm page (#16650) - Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173) - Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989) - Modify return value check in python virtualenv jinja template (#16049) - Fix dag dependency search (#15924) - Make custom JSON encoder support ``Decimal`` (#16383) - Bugfix: Allow clearing tasks with just ``dag_id`` and empty ``subdir`` (#16513) - Convert port value to a number before calling test connection (#16497) - Handle missing/null serialized DAG dependencies (#16393) - Correctly set ``dag.fileloc`` when using the ``@dag`` decorator (#16384) - Fix TI success/failure links (#16233) - Correctly implement autocomplete early return in ``airflow/www/views.py`` (#15940) - Backport fix to allow pickling of Loggers to Python 3.6 (#18798) - Fix bug that Backfill job fail to run when there are tasks run into ``reschedule`` state (#17305, #18806) ### Doc only changes - Update ``dagbag_size`` documentation (#18824) - Update documentation about bundle extras (#18828) - Fix wrong Postgres ``search_path`` set up instructions (#17600) - Remove ``AIRFLOW_GID`` from Docker images (#18747) - Improve error message for BranchPythonOperator when no task_id to follow (#18471) - Improve guidance to users telling them what to do on import timeout (#18478) - Explain scheduler fine-tuning better (#18356) - Added example JSON for airflow pools import (#18376) - Add ``sla_miss_callback`` section to the documentation (#18305) - Explain sentry default environment variable for subprocess hook (#18346) - Refactor installation pages (#18282) - Improves installing from sources pages for all components (#18251) - Doc: Use ``closer.lua`` script for downloading sources (#18179) - Improves quick-start docker-compose warnings and documentation (#18164) - Production-level support for MSSQL (#18382) - Update non-working example in documentation (#18067) - Improve the description of how to handle dynamic task generation (#17963) - Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs (#16866) - Update max_tis_per_query to better render on the webpage (#17971) - Adds Github Oauth example with team based authorization (#17896) - Update docker.rst (#17882) - Example xcom update (#17749) - Add doc warning about connections added via envvars (#17915) - fix wrong documents around upgrade-check.rst (#17903) - Add Brent to Committers list (#17873) - Improves documentation about modules management (#17757) - Remove deprecated metrics from metrics.rst (#17772) - Make sure "production-readiness" of docker-compose is well explained (#17731) - Doc: Update Upgrade to v2 docs with Airflow 1.10.x EOL dates (#17710) - Doc: Replace deprecated param from docstrings (#17709) - docs(dagowner): describe dag owner more carefully (#17699) - docs(impersonation): update note so avoid misintrepretation (#17701) - Docs: Make ``DAG.is_active`` read-only in API (#17667) - Improve cross-links to operators and hooks references (#17622) - Update documentation regarding Python 3.9 support (#17611) - Fix MySQL database character set instruction (#17603) - Document overriding ``XCom.clear`` for data lifecycle management (#17589) - Path correction in docs for airflow core (#17567) - docs(celery): reworded, add actual multiple queues example (#17541) - Doc: Add FAQ to speed up parsing with tons of dag files (#17519) - Improve image building documentation for new users (#17409) - Doc: Strip unnecessary arguments from MariaDB JIRA URL (#17296) - Update warning about MariaDB and multiple schedulers (#17287) - Doc: Recommend using same configs on all Airflow components (#17146) - Move docs about masking to a new page (#17007) - Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc (#16926) - Docs: Better description for ``pod_template_file`` (#16861) - Add Aneesh Joseph as Airflow Committer (#16835) - Docs: Added new pipeline example for the tutorial docs (#16548) - Remove upstart from docs (#16672) - Add new committers: ``Jed`` and ``TP`` (#16671) - Docs: Fix ``flask-ouathlib`` to ``flask-oauthlib`` in Upgrading docs (#16320) - Docs: Fix creating a connection docs (#16312) - Docs: Fix url for ``Elasticsearch`` (#16275) - Small improvements for README.md files (#16244) - Fix docs for ``dag_concurrency`` (#16177) - Check syntatic correctness for code-snippets (#16005) - Improvements for Docker Image docs (#14843) - Add proper link for wheel packages in docs. (#15999) - Add Docs for ``default_pool`` slots (#15997) - Add memory usage warning in quick-start documentation (#15967) - Update example ``KubernetesExecutor`` ``git-sync`` pod template file (#15904) - Docs: Fix Taskflow API docs (#16574) - Added new pipeline example for the tutorial docs (#16084) - Updating the DAG docstring to include ``render_template_as_native_obj`` (#16534) - Update docs on setting up SMTP (#16523) - Docs: Fix API verb from ``POST`` to ``PATCH`` (#16511) ### Misc/Internal - Renaming variables to be consistent with code logic (#18685) - Simplify strings previously split across lines (#18679) - fix exception string of ``BranchPythonOperator`` (#18623) - Add multiple roles when creating users (#18617) - Move FABs base Security Manager into Airflow. (#16647) - Remove unnecessary css state colors (#18461) - Update ``boto3`` to ``<1.19`` (#18389) - Improve coverage for ``airflow.security.kerberos module`` (#18258) - Fix kinesis test (#18337) - Fix provider test acessing importlib-resources (#18228) - Silence warnings in tests from using SubDagOperator (#18275) - Fix usage of ``range(len())`` to ``enumerate`` (#18174) - Test coverage on the autocomplete view (#15943) - Update version added fields in airflow/config_templates/config.yml (#18128) - Add "packaging" to core requirements (#18122) - Adds LoggingMixins to BaseTrigger (#18106) - Limit colorlog version (6.x is incompatible) (#18099) - Fix building docs in ``main`` builds (#18035) - Remove upper-limit on ``tenacity`` (#17593) - Remove redundant ``numpy`` dependency (#17594) - Bump ``mysql-connector-python`` to latest version (#17596) - Make ``pandas`` an optional core dependency (#17575) - Add more typing to airflow.utils.helpers (#15582) - Chore: Some code cleanup in ``airflow/utils/db.py`` (#17090) - Refactor: Remove processor_factory from DAG processing (#16659) - Remove AbstractDagFileProcessorProcess from dag processing (#16816) - Update TaskGroup typing (#16811) - Update ``click`` to 8.x (#16779) - Remove remaining Pylint disables (#16760) - Remove duplicated try, there is already a try in create_session (#16701) - Removes pylint from our toolchain (#16682) - Refactor usage of unneeded function call (#16653) - Add type annotations to setup.py (#16658) - Remove SQLAlchemy <1.4 constraint (#16630) (Note: our dependencies still have a requirement on <1.4) - Refactor ``dag.clear`` method (#16086) - Use ``DAG_ACTIONS`` constant (#16232) - Use updated ``_get_all_non_dag_permissions`` method (#16317) - Add updated-name wrappers for built-in FAB methods (#16077) - Pins ``docutils`` to <0.17 until breaking behaviour is fixed (#16133) - Remove ``TaskInstance.log_filepath`` attribute (#15217) - Removes unnecessary function call in ``airflow/www/app.py`` (#15956) - Move ``plyvel`` to google provider extra (#15812) - Update permission migrations to use new naming scheme (#16400) - Use resource and action names for FAB (#16380) - Swap out calls to ``find_permission_view_menu`` for ``get_permission`` wrapper (#16377) - Fix deprecated default for ``fab_logging_level`` to ``WARNING`` (#18783) - Allow running tasks from UI when using ``CeleryKubernetesExecutor`` (#18441)