Apache Airflow: Airflow 1.10.7, 2019-12-24

### New Features - [AIRFLOW-5088] [AIP-24] Persisting serialized DAG in DB for webserver scalability (#5992) - [AIRFLOW-6083] Adding ability to pass custom configuration to AWS Lambda client. (#6678) - [AIRFLOW-5117] Automatically refresh EKS API tokens when needed (#5731) - [AIRFLOW-5118] Add ability to specify optional components in DataprocClusterCreateOperator (#5821) - [AIRFLOW-5681] Allow specification of a tag or hash for the git_sync init container (#6350) - [AIRFLOW-6025] Add label to uniquely identify creator of Pod (#6621) - [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) (#5489) - [AIRFLOW-5751] add get_uri method to Connection (#6426) - [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id (#6655) - [AIRFLOW-2694] Declare permissions in DAG definition (#4642) - [AIRFLOW-4940] Add DynamoDB to S3 operator (#5663) - [AIRFLOW-4161] BigQuery to Mysql Operator (#5711) - [AIRFLOW-6041] Add user agent to the Discovery API client (#6636) - [AIRFLOW-6089] Reorder setup.py dependencies and add ci (#6681) - [AIRFLOW-5921] Add bulk_load_custom to MySqlHook (#6575) - [AIRFLOW-5854] Add support for `tty` parameter in Docker related operators (#6542) - [AIRFLOW-4758] Add GcsToGDriveOperator operator (#5822) ### Improvements - [AIRFLOW-3656] Show doc link for the current installed version (#6690) - [AIRFLOW-5665] Add path_exists method to SFTPHook (#6344) - [AIRFLOW-5729] Make InputDataConfig optional in Sagemaker's training config (#6398) - [AIRFLOW-5045] Add ability to create Google Dataproc cluster with custom image from a different project (#5752) - [AIRFLOW-6132] Allow to pass in tags for the AzureContainerInstancesOperator (#6694) - [AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization (#6715) - [AIRFLOW-5947] Make the json backend pluggable for DAG Serialization (#6630) - [AIRFLOW-6239] Filter dags return by last_dagruns (to only select visible dags, not all dags) (#6804) - [AIRFLOW-6095] Filter dags returned by task_stats (to only select visible dags, not all dags) (#6684) - [AIRFLOW-4482] Add execution_date to "trigger DagRun" API response (#5260) - [AIRFLOW-1076] Add get method for template variable accessor (#6793) - [AIRFLOW-5194] Add error handler to action log (#5883) - [AIRFLOW-5936] Allow explicit get_pty in SSHOperator (#6586) - [AIRFLOW-5474] Add Basic auth to Druid hook (#6095) - [AIRFLOW-5726] Allow custom filename in RedshiftToS3Transfer (#6396) - [AIRFLOW-5834] Option to skip serve_logs process with ``airflow worker`` (#6709) - [AIRFLOW-5583] Extend the 'DAG Details' page to display the start_date / end_date (#6235) - [AIRFLOW-6250] Ensure on_failure_callback always has a populated context (#6812) - [AIRFLOW-6222] http hook logs response body for any failure (#6779) - [AIRFLOW-6260] Drive _cmd config option by env var (``AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD`` for example) (#6801) - [AIRFLOW-6168] Allow proxy_fix middleware of webserver to be configurable (#6723) - [AIRFLOW-5931] Use os.fork when appropriate to speed up task execution. (#6627) - [AIRFLOW-4145] Allow RBAC roles permissions, ViewMenu to be over-ridable (#4960) - [AIRFLOW-5928] Hive hooks load_file short circuit (#6582) - [AIRFLOW-5313] Add params support for awsbatch_operator (#5900) - [AIRFLOW-2227] Add delete method to Variable class (#4963) - [AIRFLOW-5082] Add subject in AwsSnsHook (#5694) - [AIRFLOW-5715] Make email, owner context available (#6385) - [AIRFLOW-5345] Allow SqlSensor's hook to be customized by subclasses (#5946) - [AIRFLOW-5417] Fix DB disconnects during webserver startup (#6023) - [AIRFLOW-5730] Enable get_pandas_df on PinotDbApiHook (#6399) - [AIRFLOW-3235] Add list function in AzureDataLakeHook (#4070) - [AIRFLOW-5442] implementing get_pandas_df method for druid broker hook (#6057) - [AIRFLOW-5883] Improve count() queries in a few places (#6532) - [AIRFLOW-5811] Add metric for externally killed task count (#6466) - [AIRFLOW-5758] Support the custom cursor classes for the PostgreSQL hook (#6432) - [AIRFLOW-5766] Use httpbin.org in http_default (#6438) - [AIRFLOW-5798] Set default ExternalTaskSensor.external_task_id (#6431) - [AIRFLOW-5643] Reduce duplicated logic in S3Hook (#6313) - [AIRFLOW-5562] Skip grant single DAG permissions for Admin role. (#6199) - [AIRFLOW-6192] Stop creating Hook from SFTPSensor.__init__ (#6748) - [AIRFLOW-5749][AIRFLOW-4162] Support the "blocks" component for the Slack operators (#6418) - [AIRFLOW-5693] Support the "blocks" component for the Slack messages (#6364) - [AIRFLOW-5714] Collect SLA miss emails only from tasks missed SLA (#6384) - [AIRFLOW-5049] Add validation for src_fmt_configs in bigquery hook (#5671) - [AIRFLOW-6177] Log DAG processors timeout event at error level, not info (#6731) - [AIRFLOW-6180] Improve kerberos init in pytest conftest (#6735) - [AIRFLOW-6159] Change logging level of the heartbeat message to DEBUG (#6716) - [AIRFLOW-6144] Improve the log message of airflow scheduler (#6710) - [AIRFLOW-6099] Add host name to task runner log (#6688) - [AIRFLOW-6045] Error on failed execution of compile_assets (#6640) - [AIRFLOW-5144] Add confirmation on delete button click (#6745) - [AIRFLOW-6099] Add host name to task runner log (#6688) - [AIRFLOW-5915] Add support for the new documentation theme (#6563) - [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550) - [AIRFLOW-5888] Use psycopg2-binary for postgres operations (#6533) - [AIRFLOW-5870] Allow -1 for pool size and optimise pool query (#6520) ### Bug Fixes - [AIRFLOW-XXX] Bump Jira version to fix issue with async - [AIRFLOW-XXX] Add encoding to fix Cyrillic output when reading back task logs (#6631) - [AIRFLOW-5304] Fix extra links in BigQueryOperator with multiple queries (#5906) - [AIRFLOW-6268] Prevent (expensive) ajax calls on home page when no dags visible (#6839) - [AIRFLOW-6259] Reset page to 1 with each new search for dags (#6828) - [AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic schema (#6754) - [AIRFLOW-3632] Only replace microseconds if execution_date is None in trigger_dag REST API (#6380) - [AIRFLOW-5458] Bump Flask-AppBuilder to 2.2.0 (for Python >= 3.6) (#6607) - [AIRFLOW-5072] gcs_hook should download files once (#5685) - [AIRFLOW-5744] Environment variables not correctly set in Spark submit operator (#6796) - [AIRFLOW-3189] Remove schema from DbHook.get_uri response if None (#6833) - [AIRFLOW-6195] Fixed TaskInstance attrs not correct on UI (#6758) - [AIRFLOW-5889] Make polling for AWS Batch job status more resillient (#6765) - [AIRFLOW-6043] Fix bug in UI when "filtering by root" to display section of dag (#6638) - [AIRFLOW-6033] Fix UI Crash at "Landing Times" when task_id is changed (#6635) - [AIRFLOW-3745] Fix viewer not able to view dag details (#4569) - [AIRFLOW-6175] Fixes bug when tasks get stuck in "scheduled" state (#6732) - [AIRFLOW-5463] Make Variable.set when replacing an atomic operation (#6807) - [AIRFLOW-5582] Add get_autocommit to JdbcHook (#6232) - [AIRFLOW-5867] Fix webserver unit_test_mode data type (#6517) - [AIRFLOW-5819] Update AWSBatchOperator default value (#6473) - [AIRFLOW-5709] Fix regression in setting custom operator resources. (#6331) - [AIRFLOW-5658] Fix broken navigation links (#6374) - [AIRFLOW-5727] SqoopHook: Build --connect parameter only if port/schema are defined (#6397) - [AIRFLOW-5695] use RUNNING_DEPS to check run from UI (#6367) - [AIRFLOW-6254] obscure conn extra in logs (#6817) - [AIRFLOW-4824] Add charset handling for SqlAlchemy engine for MySqlHook (#6816) - [AIRFLOW-6091] Add flushing in execute method for BigQueryCursor (#6683) - [AIRFLOW-6256] Ensure Jobs table is cleared when resetting DB (#6818) - [AIRFLOW-5224] Add encoding parameter to GoogleCloudStorageToBigQuery (#6297) - [AIRFLOW-5179] Remove top level __init__.py (#5818) - [AIRFLOW-5660] Attempt to find the task in DB from Kubernetes pod labels (#6340) - [AIRFLOW-6241] Fix typo in airflow/gcp/operator/dataflow.py (#6806) - [AIRFLOW-6171] Apply .airflowignore to correct subdirectories (#6784) - [AIRFLOW-6018] Display task instance in table during backfilling (#6612) - [AIRFLOW-6189] Reduce the maximum test duration to 8 minutes (#6744) - [AIRFLOW-6141] Remove ReadyToRescheduleDep if sensor mode == poke (#6704) - [AIRFLOW-6054] Add a command that starts the database consoles (#6653) - [AIRFLOW-6047] Simplify the logging configuration template (#6644) - [AIRFLOW-6017] Exclude PULL_REQUEST_TEMPLATE.md from RAT check (#6611) - [AIRFLOW-4560] Fix Tez queue parameter name in mapred_queue (#5315) - [AIRFLOW-2143] Fix TaskTries graph counts off-by-1 (#6526) - [AIRFLOW-5873] KubernetesPodOperator fixes and test (#6523) - [AIRFLOW-5869] BugFix: Some Deserialized tasks have no start_date (#6519) - [AIRFLOW-4020] Remove DAG edit permissions from Viewer role (#4845) - [AIRFLOW-6263] Fix broken WinRM integration (#6832) - [AIRFLOW-5836] Pin azure-storage-blob version to <12 (#6486) - [AIRFLOW-4488] Fix typo for non-RBAC UI in max_active_runs_per_dag (#6778) - [AIRFLOW-5942] Pin PyMSSQL to <3.0 (#6592) - [AIRFLOW-5451] SparkSubmitHook don't set default namespace (#6072) - [AIRFLOW-6271] Printing log files read during load_test_config (#6842) - [AIRFLOW-6308] Unpin Kombu for Python 3 ### Misc/Internal - [AIRFLOW-6009] Switch off travis_wait for regular tests (#6600) - [AIRFLOW-6226] Always reset warnings in tests - [AIRFLOW-XXX] Remove cyclic imports and pylint hacks in Serialization (#6601) - [AIRFLOW-XXX] Bump npm from 6.4.1 to 6.13.4 in /airflow/www (#6815) - [AIRFLOW-XXX] Remove executable permission from file - [AIRFLOW-XXX] Group AWS & Azure dependencies (old ``[emr]`` etc. extra still work) - [AIRFLOW-5487] Fix unused warning var (#6111) - [AIRFLOW-5925] Relax funcsigs and psutil version requirements (#6580) - [AIRFLOW-5740] Fix Transient failure in Slack test (#6407) - [AIRFLOW-6058] Running tests with pytest (#6472) - [AIRFLOW-6066] Added pre-commit checks for accidental debug stmts (#6662) - [AIRFLOW-6060] Improve conf_vars context manager (#6658) - [AIRFLOW-6044] Standardize the Code Structure in kube_pod_operator.py (#6639) - [AIRFLOW-4940] Simplify tests of DynamoDBToS3Operator (#6836) - [AIRFLOW-XXX] Update airflow-jira release management script (#6772) - [AIRFLOW-XXX] Add simple guidelines to unit test writing (#6846) - [AIRFLOW-6309] Fix stable build on Travis ### Doc only changes - [AIRFLOW-6211] Doc how to use conda for local virtualenv (#6766) - [AIRFLOW-5855] Fix broken reference in custom operator doc (#6508) - [AIRFLOW-5875] Fix typo in example_qubole_operator.py (#6525) - [AIRFLOW-5702] Fix common docstring issues (#6372) - [AIRFLOW-5640] Document and test ``email`` parameters of BaseOperator (#6315) - [AIRFLOW-XXX] Improve description OpenFaaS Hook (#6187) - [AIRFLOW-XXX] GSoD: How to make DAGs production ready (#6515) - [AIRFLOW-XXX] Use full command in examples (#5973) - [AIRFLOW-XXX] Update docs to accurately describe the precedence of remote and local logs (#5607) - [AIRFLOW-XXX] Fix example "extras" field in mysql connect doc (#5285) - [AIRFLOW-XXX] Fix wrong inline code highlighting in docs (#5309) - [AIRFLOW-XXX] Group executors in one section (#5834) - [AIRFLOW-XXX] Add task lifecycle diagram to documentation (#6762) - [AIRFLOW-XXX] Highlight code blocks (#6243) - [AIRFLOW-XXX] Documents about task_concurrency and pool (#5262) - [AIRFLOW-XXX] Fix incorrect docstring parameter (#6649) - [AIRFLOW-XXX] Add link to XCom section in concepts.rst (#6791) - [AIRFLOW-XXX] Update kubernetes doc with correct path (#6774) - [AIRFLOW-XXX] Add information how to configure pytest runner (#6736) - [AIRFLOW-XXX] more GSOD improvements (#6585) - [AIRFLOW-XXX] Clarified a grammatically incorrect sentence (#6667) - [AIRFLOW-XXX] Add notice for Mesos Executor deprecation in docs (#6712) - [AIRFLOW-XXX] Update list of pre-commits (#6603) - [AIRFLOW-XXX] Updates to Breeze documentation from GSOD (#6285) - [AIRFLOW-XXX] Clarify daylight savings time behavior (#6324) - [AIRFLOW-XXX] GSoD: Adding 'Create a custom operator' doc (#6348) - [AIRFLOW-XXX] Add resources & links to CONTRIBUTING.rst (#6405) - [AIRFLOW-XXX] Update chat channel details from gitter to slack (#4149) - [AIRFLOW-XXX] Add logo info to readme (#6349) - [AIRFLOW-XXX] Fixed case problem with CONTRIBUTING.rst (#6329) - [AIRFLOW-XXX] Google Season of Docs updates to CONTRIBUTING doc (#6283)