Apache Airflow 1.7.0 — Release Notes & Engineering Impact

Raw changelog for the record, generated with: `~/node_modules/github-changes/bin/index.js -o airbnb -r airflow --only-pulls --use-commit-body --between-tags 1.6.2...1.7.0rc1 --token {your_gh_token_here}` ### 1.7.0 - [#1147](https://github.com/airbnb/airflow/pull/1147) Enhance CLI Test command to accept a JSON-formatted dictionary of par… (@r39132) - [#975](https://github.com/airbnb/airflow/pull/975) statuses column on /admin shows only active or most recent dag_runs (@jtschoonhoven) - [#1127](https://github.com/airbnb/airflow/pull/1127) FAQ entry about start_date (@mistercrunch) - [#1129](https://github.com/airbnb/airflow/pull/1129) Showing active dag runs as in (3/16) in tooltip (@mistercrunch) - [#1144](https://github.com/airbnb/airflow/pull/1144) More explicit zindex on modals (@mistercrunch) - [#1143](https://github.com/airbnb/airflow/pull/1143) small fixes to previous bq project inclusion pr. (@mtagle) - [#1139](https://github.com/airbnb/airflow/pull/1139) Allow specificiation of project in BigQuery Hook methods (@mtagle) - [#1110](https://github.com/airbnb/airflow/pull/1110) Add date support to MySQL to GCS operator (@criccomini) - [#1135](https://github.com/airbnb/airflow/pull/1135) Added start_date initialization for DagRun creation within schedule_dag(self, dag_id) (@RvN76) - [#1140](https://github.com/airbnb/airflow/pull/1140) License check (@bolkedebruin) - [#1138](https://github.com/airbnb/airflow/pull/1138) Add support for BigQuery User Defined Functions in BigQuery operator (@LeBlanc) - [#1132](https://github.com/airbnb/airflow/pull/1132) Add custom email backends. (@jmcarp) - [#1119](https://github.com/airbnb/airflow/pull/1119) Add gcloud-based GCSHook (@jlowin) - [#1134](https://github.com/airbnb/airflow/pull/1134) Parameterizing DagBag import timeouts (@mistercrunch) - [#1045](https://github.com/airbnb/airflow/pull/1045) Add startup scripts for upstart based systems (@d-lee) - [#1131](https://github.com/airbnb/airflow/pull/1131) Adding ssh connection type to webform (@hyperborea) - [#792](https://github.com/airbnb/airflow/pull/792) add error handling for slack api (@abridgett) - [#1115](https://github.com/airbnb/airflow/pull/1115) This patch adds license checking for Airflow. (@bolkedebruin) - [#1116](https://github.com/airbnb/airflow/pull/1116) Add WePay and committer list to README.md (@criccomini) - [#1118](https://github.com/airbnb/airflow/pull/1118) Don't force installation of GCP API client dependencies (@jlowin) - [#1090](https://github.com/airbnb/airflow/pull/1090) Adding template support in qbol operator (@msumit) - [#1108](https://github.com/airbnb/airflow/pull/1108) new company + link to pitfalls (@kretes) - [#1104](https://github.com/airbnb/airflow/pull/1104) Add INT24 (MEDIUMINT) support to MySQL to Google cloud storage operator (@criccomini) - [#1076](https://github.com/airbnb/airflow/pull/1076) SubDAG docs and examples (@nicktrav) - [#1101](https://github.com/airbnb/airflow/pull/1101) modify datastore hook so that authorization is maintained for the lif… (@mtagle) - [#1099](https://github.com/airbnb/airflow/pull/1099) Adding @mention in Airbnb and links to airbnb open source projects (@artwr) - [#1096](https://github.com/airbnb/airflow/pull/1096) Make SqlAlchemy pool_recycle and pool_size configurable (@amread) - [#1070](https://github.com/airbnb/airflow/pull/1070) Proper sqlalchemy syntax for desc (@mistercrunch) - [#1079](https://github.com/airbnb/airflow/pull/1079) SID Oracle DB connection support (@biln) - [#1094](https://github.com/airbnb/airflow/pull/1094) Adding to inits (@artwr) - [#1093](https://github.com/airbnb/airflow/pull/1093) Add MySQL to BQ support for TINYINT (@criccomini) - [#1088](https://github.com/airbnb/airflow/pull/1088) Fix MySQL to Google cloud storage scoping. (@criccomini) - [#1075](https://github.com/airbnb/airflow/pull/1075) Pig hook and operator stub (@artwr) - [#1084](https://github.com/airbnb/airflow/pull/1084) Replace deprecated flask.ext.\* with flask_\* (@jeffwidman) - [#1082](https://github.com/airbnb/airflow/pull/1082) Cleanup Contributing.md (@jeffwidman) - [#1086](https://github.com/airbnb/airflow/pull/1086) Default to 0 if no rows loaded in GCS to BQ operator. (@criccomini) - [#1080](https://github.com/airbnb/airflow/pull/1080) Add MySQL->GCS, GCS->BQ operators (@criccomini) - [#1073](https://github.com/airbnb/airflow/pull/1073) Fixing small issue with qbol operator and hook (@msumit) - [#1068](https://github.com/airbnb/airflow/pull/1068) Add a new hook for google datastore (@mtagle) - [#1065](https://github.com/airbnb/airflow/pull/1065) dag pausing should pause queued tasks as well (@mistercrunch) - [#1067](https://github.com/airbnb/airflow/pull/1067) Add two methods to BigQueryBaseCursor: (@mtagle) - [#1064](https://github.com/airbnb/airflow/pull/1064) Add Thumbtack to organizations using Airflow (@natekupp) - [#1063](https://github.com/airbnb/airflow/pull/1063) Documenting task details doc_ feature (@mistercrunch) - [#1049](https://github.com/airbnb/airflow/pull/1049) Allow the use of the autoconfig client (@bolkedebruin) - [#1056](https://github.com/airbnb/airflow/pull/1056) Add output encoding option to BashOperator (@Attsun1031) - [#996](https://github.com/airbnb/airflow/pull/996) Added the dst filename to template_fields GoogleCloudStorageDownloadOperator (@0xR) - [#1051](https://github.com/airbnb/airflow/pull/1051) Running unit tests with local executor (@mistercrunch) - [#1008](https://github.com/airbnb/airflow/pull/1008) Add FTPSHook class. (@geeknam) - [#1055](https://github.com/airbnb/airflow/pull/1055) Improve [Hide/Show all series] perforamce (@lumengxi) - [#1054](https://github.com/airbnb/airflow/pull/1054) CLI's trigger_dag now accepts --conf as json (@mistercrunch) - [#1053](https://github.com/airbnb/airflow/pull/1053) Add Kogan.com to the list of users (@geeknam) - [#1046](https://github.com/airbnb/airflow/pull/1046) Documenting the cluster policy feature (@mistercrunch) - [#1044](https://github.com/airbnb/airflow/pull/1044) Updating the Readme with a link to the TriggerDagRunOperator post (@r39132) - [#1043](https://github.com/airbnb/airflow/pull/1043) Adding an example to illustrate the TriggerDagRunOperator (@r39132) - [#1039](https://github.com/airbnb/airflow/pull/1039) Only set headers and delimiters for CSV exports in Google BigQuery hook (@criccomini) - [#1026](https://github.com/airbnb/airflow/pull/1026) Minor documentation tweaks to the FAQ under the fernet key section (@r39132) - [#1025](https://github.com/airbnb/airflow/pull/1025) Add notes on connection password encryption (@d-lee) - [#1018](https://github.com/airbnb/airflow/pull/1018) add SSL support for SMTP (@bbrumi) - [#1020](https://github.com/airbnb/airflow/pull/1020) Add BigQuery copy operator. (@criccomini) - [#964](https://github.com/airbnb/airflow/pull/964) Clean test database out in between unit test runs (@nicktrav) - [#1019](https://github.com/airbnb/airflow/pull/1019) Add direct dependencies for Google cloud contribs (@criccomini) - [#985](https://github.com/airbnb/airflow/pull/985) [airflow][presto] Gracefully handle 503 errors and avoid eval() (@airbnb) - [#1013](https://github.com/airbnb/airflow/pull/1013) Fixed issue 1012: pool not used with celery executor (@rdavison) - [#1011](https://github.com/airbnb/airflow/pull/1011) drop the tmp table after ingestion (@dayzzz) - [#995](https://github.com/airbnb/airflow/pull/995) Fix LDAP error messages when login fails. (@criccomini) - [#1006](https://github.com/airbnb/airflow/pull/1006) Fix forgetting to expose yesterday_ds_nodash and tomorrow_ds_nodash (@0xR) - [#1003](https://github.com/airbnb/airflow/pull/1003) Update to the README such as adding Hootsuite (@r39132) - [#998](https://github.com/airbnb/airflow/pull/998) Pedantic documentation tweaks (@joshmarlow) - [#940](https://github.com/airbnb/airflow/pull/940) Refresh border coloring in graph view without having to refresh the entire page (@DinoCow) - [#987](https://github.com/airbnb/airflow/pull/987) Add logout button to Airflow (@criccomini) - [#930](https://github.com/airbnb/airflow/pull/930) When either data_profiler_filter or superuser_filter aren't defined,… (@mtagle) - [#994](https://github.com/airbnb/airflow/pull/994) there could be dot in the key string, which is illegal in the hive ta… (@dayzzz) - [#993](https://github.com/airbnb/airflow/pull/993) Added yesterday_ds_nodash and tommorow_ds_nodash (@0xR) - [#992](https://github.com/airbnb/airflow/pull/992) Rename BashOperator.xcom_push member to avoid collision with parent class (@seregasheypak) - [#978](https://github.com/airbnb/airflow/pull/978) Fixing conflicting params in default_args (@mistercrunch, @nicktrav) - [#984](https://github.com/airbnb/airflow/pull/984) Add tests for params handling in Dag construction (@nicktrav) - [#986](https://github.com/airbnb/airflow/pull/986) [documentation] document that max_active_runs can prevent a DAG from running (@aoen) - [#983](https://github.com/airbnb/airflow/pull/983) Added destination_dataset_table to template_fieds of bigquery_operator (@0xR) - [#981](https://github.com/airbnb/airflow/pull/981) Add BigQuery PEP 249 support (@criccomini) - [#980](https://github.com/airbnb/airflow/pull/980) Support user-defined macros and params in dry-run backfills with task… (@r39132) - [#976](https://github.com/airbnb/airflow/pull/976) More verbose logging for are_dependencies_met when called from run (@mistercrunch) - [#944](https://github.com/airbnb/airflow/pull/944) Fix statuses in dag list for dags with dag_ids prefixed by a number (@wil5for) - [#942](https://github.com/airbnb/airflow/pull/942) Set effective user of (web)hdfs hooks using connection config, optionally overridden by constructor (@xadhoom) - [#965](https://github.com/airbnb/airflow/pull/965) Adding mock lib to devel extras_require (@mistercrunch) - [#963](https://github.com/airbnb/airflow/pull/963) Only schedule DagRuns between start and end dates (@nicktrav) - [#948](https://github.com/airbnb/airflow/pull/948) Fix encryption alert msg for env var, configuration->conf refactor (@mistercrunch) - [#960](https://github.com/airbnb/airflow/pull/960) Fix parsing file that contains multi byte char (@Attsun1031) - [#949](https://github.com/airbnb/airflow/pull/949) Allow for domain-wide delegation in google cloud apps (@mtagle) - [#958](https://github.com/airbnb/airflow/pull/958) Fix bug in reporting of attempt number when queuing tasks in a pool (@r39132) - [#946](https://github.com/airbnb/airflow/pull/946) Add dag_state to cli (@wil5for) - [#956](https://github.com/airbnb/airflow/pull/956) Support new configuration dags_are_paused_at_creation that, when True… (@r39132) - [#945](https://github.com/airbnb/airflow/pull/945) Add documentation to gcs_download_operator (@criccomini) - [#947](https://github.com/airbnb/airflow/pull/947) Encrypt Variables if Fernet key provided (@r39132) - [#943](https://github.com/airbnb/airflow/pull/943) Fix running custom pool task with mark-success is not marked success (@Attsun1031) - [#816](https://github.com/airbnb/airflow/pull/816) Implement SSH Execute Operator (@KeenS) - [#941](https://github.com/airbnb/airflow/pull/941) Fix process_subdir bug (@Attsun1031) - [#939](https://github.com/airbnb/airflow/pull/939) Show rendered templates from the CLI (@mistercrunch) - [#224](https://github.com/airbnb/airflow/pull/224) PrestoToMySqlTransfer (@mistercrunch) - [#801](https://github.com/airbnb/airflow/pull/801) allow slack attachments to be templated (@abridgett) - [#938](https://github.com/airbnb/airflow/pull/938) [hotfix] dag missing from dagbag (@mistercrunch) - [#862](https://github.com/airbnb/airflow/pull/862) handle default option for extra_options argument in HttpHook.run method (@JoergRittinger) - [#935](https://github.com/airbnb/airflow/pull/935) Fix and refactor the httphook (@mistercrunch) - [#919](https://github.com/airbnb/airflow/pull/919) Add support for three-legged OAuth for Google connections. Useful for… (@criccomini) - [#937](https://github.com/airbnb/airflow/pull/937) Statsd abstraction (@mistercrunch) - [#743](https://github.com/airbnb/airflow/pull/743) Get connection string containing username and password during runtime (@praveev) - [#926](https://github.com/airbnb/airflow/pull/926) Normalize plugin paths that include both slashes and dots (@jasonjho) - [#934](https://github.com/airbnb/airflow/pull/934) Support for encrypting the connection extra field (@r39132) - [#928](https://github.com/airbnb/airflow/pull/928) fix function name error: active_tasks (@flying5) - [#882](https://github.com/airbnb/airflow/pull/882) Docker operator (@asnir) - [#925](https://github.com/airbnb/airflow/pull/925) Small change to fix a crash when reading log files from remote workers (@kmevissen) - [#868](https://github.com/airbnb/airflow/pull/868) Encrypt logs (@vansivallab) - [#927](https://github.com/airbnb/airflow/pull/927) [fix] disregarding adhoc tasks when closing dag runs (@mistercrunch) - [#918](https://github.com/airbnb/airflow/pull/918) Reverting production issues from 876 and undead (@mistercrunch) - [#922](https://github.com/airbnb/airflow/pull/922) Add link to SmartNews in README (@takus) - [#875](https://github.com/airbnb/airflow/pull/875) SLA Miss Alert Callbacks : Allow DAGs to specify a callback function for SLA miss handling (@r39132) - [#917](https://github.com/airbnb/airflow/pull/917) Allowing for relative path and dot notation for -sd (@mistercrunch) - [#901](https://github.com/airbnb/airflow/pull/901) Make Google Cloud Storage download operator use a filename, not a fil… (@criccomini) - [#912](https://github.com/airbnb/airflow/pull/912) Add Clover Health to Airflow users (@vansivallab) - [#813](https://github.com/airbnb/airflow/pull/813) Add "search_scope" as a configuration variable for LDAP (@NeilHanlon) - [#910](https://github.com/airbnb/airflow/pull/910) Add similarweb airflow puppet module to the readme (@danielbenzvi) - [#909](https://github.com/airbnb/airflow/pull/909) Fixes typo (@rosner) - [#907](https://github.com/airbnb/airflow/pull/907) Call `Session.remove` after each run, to survive DB restart (@KMK-ONLINE) - [#906](https://github.com/airbnb/airflow/pull/906) Update some datetime column default args for consistent treatment (@t1m0thy) - [#904](https://github.com/airbnb/airflow/pull/904) Bugix for TriggerDagRunOperator (@t1m0thy) - [#905](https://github.com/airbnb/airflow/pull/905) Add SimilarWeb to the who uses airflow section (@danielbenzvi) - [#903](https://github.com/airbnb/airflow/pull/903) Fixing the tutorial. Removing an unnecessary import of MySqlOperator … (@cesararevalo) - [#797](https://github.com/airbnb/airflow/pull/797) Add super user and profiler to ldap (@criccomini) - [#898](https://github.com/airbnb/airflow/pull/898) Making force a task instance member, so it becomes available for (@iddoav) - [#896](https://github.com/airbnb/airflow/pull/896) Cherry picked commits out of recent rollback (@mistercrunch) - [#894](https://github.com/airbnb/airflow/pull/894) [hotfix] fixing infinite retries in prod (@mistercrunch) - [#893](https://github.com/airbnb/airflow/pull/893) Fix ISSUE 798: password printed to STDOUT when running initdb, resetdb, upgradedb (@DinoCow) - [#889](https://github.com/airbnb/airflow/pull/889) Merging airbnb_prod hotfixes into master (@mistercrunch) - [#887](https://github.com/airbnb/airflow/pull/887) Killing tasks that aren't in a running state (@mistercrunch) - [#888](https://github.com/airbnb/airflow/pull/888) Airbnb prod (@mistercrunch) - [#808](https://github.com/airbnb/airflow/pull/808) Adding support for QDS (Qubole Date Services) (@msumit) - [#876](https://github.com/airbnb/airflow/pull/876) Airbnb prod (@iivvaall) - [#884](https://github.com/airbnb/airflow/pull/884) Fixing issue where try_number isn't incremented (@mistercrunch) - [#878](https://github.com/airbnb/airflow/pull/878) state wasn't being saved (@abridgett) - [#879](https://github.com/airbnb/airflow/pull/879) Readme update (@thibault-ketterer) - [#835](https://github.com/airbnb/airflow/pull/835) Issue 832: creating run_id if not given while executing trigger_dag (@msumit) - [#867](https://github.com/airbnb/airflow/pull/867) Airbnb prod (@mistercrunch) - [#866](https://github.com/airbnb/airflow/pull/866) [hotfix] fixing subdag not refreshing properly (@mistercrunch) - [#865](https://github.com/airbnb/airflow/pull/865) Bigquery and Google Cloud Storage operators (@criccomini) - [#861](https://github.com/airbnb/airflow/pull/861) Bigquery operator (@criccomini) - [#779](https://github.com/airbnb/airflow/pull/779) Adding base date and run number form to Task Duration and Landing Times views (@MaximeKestemont) - [#854](https://github.com/airbnb/airflow/pull/854) [hotfix] subdag not showing up (@mistercrunch) - [#853](https://github.com/airbnb/airflow/pull/853) Fixing a bug where some dags can't be retrived from DagBag.get_dag (@mistercrunch) - [#848](https://github.com/airbnb/airflow/pull/848) Delete dags from dagbag (@mistercrunch) - [#851](https://github.com/airbnb/airflow/pull/851) add a parameter for number of shard in batch ingestion (@dayzzz) - [#850](https://github.com/airbnb/airflow/pull/850) Update models.py to increase password field length (@rahul342) - [#763](https://github.com/airbnb/airflow/pull/763) Added title attribute for logs (@JordyMoos) - [#817](https://github.com/airbnb/airflow/pull/817) Fix ISSUE-812 (@bolkedebruin) - [#840](https://github.com/airbnb/airflow/pull/840) Fixing SLA handling related bug (@mistercrunch) - [#841](https://github.com/airbnb/airflow/pull/841) Add a Gitter chat badge to README.md (@gitter-badger) - [#839](https://github.com/airbnb/airflow/pull/839) Adding visibility as to which dag is pickleable (@mistercrunch) - [#826](https://github.com/airbnb/airflow/pull/826) Feature/cli fixes (@abridgett) - [#830](https://github.com/airbnb/airflow/pull/830) Add Sidecar Interactive to list of companies using Airflow (@robottokauf3) - [#825](https://github.com/airbnb/airflow/pull/825) color alternate rows so it's easier to use (@abridgett) - [#802](https://github.com/airbnb/airflow/pull/802) Implemented GitHub Enterprise Authentication (@mtp401) - [#831](https://github.com/airbnb/airflow/pull/831) order dag run drop down in graph view (@abridgett) - [#828](https://github.com/airbnb/airflow/pull/828) Fix invalid syntax in SSHHook (@robottokauf3) - [#822](https://github.com/airbnb/airflow/pull/822) Merging prod hotfixes into master (@mistercrunch) - [#824](https://github.com/airbnb/airflow/pull/824) Upgrade flask admin (@mistercrunch) - [#823](https://github.com/airbnb/airflow/pull/823) Fixing bad ONE_FAILED in recent PR (@mistercrunch)