dbt Core: dbt 0.15.0

## dbt 0.15.0 (November 25, 2019) ### Breaking changes - Support for Python 2.x has been dropped [as it will no longer be supported on January 1, 2020](https://www.python.org/dev/peps/pep-0373/) - Compilation errors in .yml files are now treated as errors instead of warnings ([#1493](https://github.com/fishtown-analytics/dbt/issues/1493), [#1751](https://github.com/fishtown-analytics/dbt/pull/1751)) - The 'table_name' field field has been removed from Relations - The existing `compile` and `execute` rpc tasks have been renamed to `compile_sql` and `execute_sql` ([#1779](https://github.com/fishtown-analytics/dbt/issues/1779), [#1798](https://github.com/fishtown-analytics/dbt/pull/1798)) ([docs](https://docs.getdbt.com/v0.15/docs/rpc)) - Custom materializations must now manage dbt's Relation cache ([docs](https://docs.getdbt.com/v0.15/docs/creating-new-materializations#section-6-update-the-relation-cache)) ### Installation notes: dbt v0.15.0 uses the `psycopg2-binary` dependency (instead of `psycopg2`) to simplify installation on platforms that do not have a compiler toolchain installed. pip users should run `pip uninstall psycopg2` before upgrading their version of dbt to 0.15.0. If after upgrading you experience segmentation faults, crashes, or installation errors, you can set the `DBT_PSYCOPG2_NAME` environment variable to `psycopg2` to change the dependency that dbt installs. This may require a compiler toolchain and development libraries. ```bash $ DBT_PSYCOPG2_NAME=psycopg2 pip install dbt ``` You may also install specific dbt plugins directly by name. This has the advantage of only installing the Python requirements needed for your particular database: ```bash # Installing dbt-snowflake and dbt-bigquery directly bypasses the psycopg2 dependency $ pip install dbt-snowflake $ pip install dbt-bigquery # Installing dbt-postgres and dbt-redshift are also supported $ pip install dbt-postgres $ pip install dbt-redshift ``` ## Changes ### Core #### Features - Add a JSON logger ([#1237](https://github.com/fishtown-analytics/dbt/issues/1237), [#1791](https://github.com/fishtown-analytics/dbt/pull/1791)) ([docs](https://docs.getdbt.com/v0.15/docs/global-cli-flags#section-log-formatting)) - Add structured logging to dbt ([#1704](https://github.com/fishtown-analytics/dbt/issues/1704), [#1799](https://github.com/fishtown-analytics/dbt/issues/1799), [#1715](https://github.com/fishtown-analytics/dbt/pull/1715), [#1806](https://github.com/fishtown-analytics/dbt/pull/1806)) - Add partial parsing option to the profiles.yml file ([#1835](https://github.com/fishtown-analytics/dbt/issues/1835), [#1836](https://github.com/fishtown-analytics/dbt/pull/1836), [#1487](https://github.com/fishtown-analytics/dbt/issues/1487)) ([docs](https://docs.getdbt.com/v0.15/docs/configure-your-profile#section-partial-parsing)) - Support configurable query comments in SQL queries ([#1643](https://github.com/fishtown-analytics/dbt/issues/1643), [#1864](https://github.com/fishtown-analytics/dbt/pull/1864)) ([docs](https://docs.getdbt.com/v0.15/docs/configuring-query-comments)) - Support atomic full-refreshes for incremental models ([#525](https://github.com/fishtown-analytics/dbt/issues/525), [#1682](https://github.com/fishtown-analytics/dbt/pull/1682)) - Support snapshot configs in dbt_project.yml ([#1613](https://github.com/fishtown-analytics/dbt/issues/1613), [#1759](https://github.com/fishtown-analytics/dbt/pull/1759)) ([docs](https://docs.getdbt.com/v0.15/docs/snapshots#section-configuring-snapshots-in-dbt_project-yml)) - Support cache modifications in materializations ([#1683](https://github.com/fishtown-analytics/dbt/issues/1683), [#1770](https://github.com/fishtown-analytics/dbt/pull/1770)) ([docs](https://docs.getdbt.com/v0.15/docs/creating-new-materializations#section-6-update-the-relation-cache)) - Support `quote` parameter to Accepted Values schema tests ([#1873](https://github.com/fishtown-analytics/dbt/issues/1873), [#1876](https://github.com/fishtown-analytics/dbt/pull/1876)) ([docs](https://docs.getdbt.com/v0.15/docs/testing#section-accepted-values)) - Support Python 3.8 ([#1886](https://github.com/fishtown-analytics/dbt/pull/1886)) - Support filters in sources for `dbt source snapshot-freshness` invocation ([#1495](https://github.com/fishtown-analytics/dbt/issues/1495), [#1776](https://github.com/fishtown-analytics/dbt/pull/1776)) ([docs](https://docs.getdbt.com/v0.15/docs/using-sources#section-filtering-sources)) - Support external table configuration in yml source specifications ([#1784](https://github.com/fishtown-analytics/dbt/pull/1784)) - Improve CLI output when running snapshots ([#1768](https://github.com/fishtown-analytics/dbt/issues/1768), [#1769](https://github.com/fishtown-analytics/dbt/pull/1769)) #### Fixes - Fix for unhelpful error message for malformed source/ref inputs ([#1660](https://github.com/fishtown-analytics/dbt/issues/1660), [#1809](https://github.com/fishtown-analytics/dbt/pull/1809)) - Fix for lingering backup tables when incremental models are full-refreshed ([#1933](https://github.com/fishtown-analytics/dbt/issues/1933), [#1931](https://github.com/fishtown-analytics/dbt/pull/1931)) - Fix for confusing error message when errors are encountered during compilation ([#1807](https://github.com/fishtown-analytics/dbt/issues/1807), [#1839](https://github.com/fishtown-analytics/dbt/pull/1839)) - Fix for logic error affecting the two-argument flavor of the `ref` function ([#1504](https://github.com/fishtown-analytics/dbt/issues/1504), [#1515](https://github.com/fishtown-analytics/dbt/pull/1515)) - Fix for invalid reference to dbt.exceptions ([#1569](https://github.com/fishtown-analytics/dbt/issues/1569), [#1609](https://github.com/fishtown-analytics/dbt/pull/1609)) - Fix for "cannot run empty query" error when pre/post-hooks are empty ([#1108](https://github.com/fishtown-analytics/dbt/issues/1108), [#1719](https://github.com/fishtown-analytics/dbt/pull/1719)) - Fix for confusing error when project names shadow context attributes ([#1696](https://github.com/fishtown-analytics/dbt/issues/1696), [#1748](https://github.com/fishtown-analytics/dbt/pull/1748)) - Fix for incorrect database logic in docs generation which resulted in columns being "merged" together across tables ([#1708](https://github.com/fishtown-analytics/dbt/issues/1708), [#1774](https://github.com/fishtown-analytics/dbt/pull/1774)) - Fix for seed errors located in dependency packages ([#1723](https://github.com/fishtown-analytics/dbt/issues/1723), [#1723](https://github.com/fishtown-analytics/dbt/issues/1723)) - Fix for confusing error when schema tests return unexpected results ([#1808](https://github.com/fishtown-analytics/dbt/issues/1808), [#1903](https://github.com/fishtown-analytics/dbt/pull/1903)) - Fix for twice-compiled `statement` block contents ([#1717](https://github.com/fishtown-analytics/dbt/issues/1717), [#1719](https://github.com/fishtown-analytics/dbt/pull/1719)) - Fix for inaccurate output in `dbt run-operation --help` ([#1767](https://github.com/fishtown-analytics/dbt/issues/1767), [#1777](https://github.com/fishtown-analytics/dbt/pull/1777)) - Fix for file rotation issues concerning the `logs/dbt.log` file ([#1863](https://github.com/fishtown-analytics/dbt/issues/1863), [#1865](https://github.com/fishtown-analytics/dbt/issues/1865), [#1871](https://github.com/fishtown-analytics/dbt/pull/1871)) - Fix for missing quotes in incremental model build queries ([#1847](https://github.com/fishtown-analytics/dbt/issues/1847), [#1888](https://github.com/fishtown-analytics/dbt/pull/1888)) - Fix for incorrect log level in `printer.print_run_result_error` ([#1818](https://github.com/fishtown-analytics/dbt/issues/1818), [#1823](https://github.com/fishtown-analytics/dbt/pull/1823)) ### Docs - Show seeds and snapshots in the Project and Database views ([docs#37](https://github.com/fishtown-analytics/dbt-docs/issues/37), [docs#25](https://github.com/fishtown-analytics/dbt-docs/issues/25), [docs#52](https://github.com/fishtown-analytics/dbt-docs/pull/52)) - Show sources in the Database tree view ([docs#20](https://github.com/fishtown-analytics/dbt-docs/issues/20), [docs#52](https://github.com/fishtown-analytics/dbt-docs/pull/52)) - Show edges in the DAG between models and seeds ([docs#15](https://github.com/fishtown-analytics/dbt-docs/issues/15), [docs#52](https://github.com/fishtown-analytics/dbt-docs/pull/52)) - Show Accepted Values tests and custom schema tests in the column list for models ([docs#52](https://github.com/fishtown-analytics/dbt-docs/pull/52)) - Fix links for "Refocus on node" and "View documentation" in DAG context menu for seeds ([docs#52](https://github.com/fishtown-analytics/dbt-docs/pull/52)) ### Server - Support docs generation ([#1781](https://github.com/fishtown-analytics/dbt/issues/1781), [#1801](https://github.com/fishtown-analytics/dbt/pull/1801)) - Support custom tags ([#1822](https://github.com/fishtown-analytics/dbt/issues/1822), [#1828](https://github.com/fishtown-analytics/dbt/pull/1828)) - Support invoking `deps` on the rpc server ([#1834](https://github.com/fishtown-analytics/dbt/issues/1834), [#1837](https://github.com/fishtown-analytics/dbt/pull/1837)) - Support invoking `run-operation` and `snapshot` on the rpc server ([#1875](https://github.com/fishtown-analytics/dbt/issues/1875), [#1878](https://github.com/fishtown-analytics/dbt/pull/1878)) - Suppport `--threads` argument to `cli_args` method ([#1897](https://github.com/fishtown-analytics/dbt/issues/1897), [#1909](https://github.com/fishtown-analytics/dbt/pull/1909)) - Support reloading the manifest when a SIGHUP signal is received ([#1684](https://github.com/fishtown-analytics/dbt/issues/1684), [#1699](https://github.com/fishtown-analytics/dbt/pull/1699)) - Support invoking `compile`, `run`, `test`, and `seed` on the rpc server ([#1488](https://github.com/fishtown-analytics/dbt/issues/1488), [#1652](https://github.com/fishtown-analytics/dbt/pull/1652)) - Support returning compilation logs from the last compile in the `status` method ([#1703](https://github.com/fishtown-analytics/dbt/issues/1703), [#1775](https://github.com/fishtown-analytics/dbt/pull/1715)) - Support asyncronous `compile_sql` and `run_sql` methods ([#1706](https://github.com/fishtown-analytics/dbt/issues/1706), [#1735](https://github.com/fishtown-analytics/dbt/pull/1735)) - Improve re-compilation performance ([#1824](https://github.com/fishtown-analytics/dbt/issues/1824), [#1830](https://github.com/fishtown-analytics/dbt/pull/1830)) ### Postgres / Redshift - Support running dbt against schemas which contain materialized views on Postgres ([#1698](https://github.com/fishtown-analytics/dbt/issues/1698), [#1833](https://github.com/fishtown-analytics/dbt/pull/1833)) - Support distyle AUTO in Redshift model configs ([#1882](https://github.com/fishtown-analytics/dbt/issues/1882), [#1885](https://github.com/fishtown-analytics/dbt/pull/1885)) ([docs](https://docs.getdbt.com/v0.15/docs/redshift-configs#section-using-sortkey-and-distkey)) - Fix for internal errors when run against mixed-case logical databases ([#1800](https://github.com/fishtown-analytics/dbt/issues/1800), [#1936](https://github.com/fishtown-analytics/dbt/pull/1936)) ### Snowflake - Support `copy grants` option in Snowflake model configs ([#1744](https://github.com/fishtown-analytics/dbt/issues/1744), [#1747](https://github.com/fishtown-analytics/dbt/pull/1747)) ([docs](https://docs.getdbt.com/v0.15/docs/snowflake-configs#section-copying-grants)) - Support warehouse configuration in Snowflake model configs ([#1358](https://github.com/fishtown-analytics/dbt/issues/1358), [#1899](https://github.com/fishtown-analytics/dbt/issues/1899), [#1788](https://github.com/fishtown-analytics/dbt/pull/1788), [#1901](https://github.com/fishtown-analytics/dbt/pull/1901)) ([docs](https://docs.getdbt.com/v0.15/docs/snowflake-configs#section-configuring-virtual-warehouses)) - Support secure views in Snowflake model configs ([#1730](https://github.com/fishtown-analytics/dbt/issues/1730), [#1743](https://github.com/fishtown-analytics/dbt/pull/1743)) ([docs](https://docs.getdbt.com/v0.15/docs/snowflake-configs#section-secure-views)) - Fix for unclosed connections preventing dbt from exiting when Snowflake is used with client_session_keep_alive ([#1271](https://github.com/fishtown-analytics/dbt/issues/1271), [#1749](https://github.com/fishtown-analytics/dbt/pull/1749)) - Fix for errors on Snowflake when dbt schemas contain `LOCAL TEMPORARY` tables ([#1869](https://github.com/fishtown-analytics/dbt/issues/1869), [#1872](https://github.com/fishtown-analytics/dbt/pull/1872)) ### BigQuery - Support KMS Encryption in BigQuery model configs ([#1829](https://github.com/fishtown-analytics/dbt/issues/1829), [#1851](https://github.com/fishtown-analytics/dbt/issues/1829)) ([docs](https://docs.getdbt.com/v0.15/docs/bigquery-configs#section-managing-kms-encryption)) - Improve docs generation speed by leveraging the information schema ([#1576](https://github.com/fishtown-analytics/dbt/issues/1576), [#1795](https://github.com/fishtown-analytics/dbt/pull/1795)) - Fix for cache errors on BigQuery when dataset names are capitalized ([#1810](https://github.com/fishtown-analytics/dbt/issues/1810), [#1881](https://github.com/fishtown-analytics/dbt/pull/1881)) - Fix for invalid query generation when multiple `options` are provided to a `create table|view` query ([#1786](https://github.com/fishtown-analytics/dbt/issues/1786), [#1787](https://github.com/fishtown-analytics/dbt/pull/1787)) - Use `client.delete_dataset` to drop BigQuery datasets atomically ([#1887](https://github.com/fishtown-analytics/dbt/issues/1887), [#1881](https://github.com/fishtown-analytics/dbt/pull/1881)) ### Under the Hood #### Dependencies - Drop support for `networkx 1.x` ([#1577](https://github.com/fishtown-analytics/dbt/issues/1577), [#1814](https://github.com/fishtown-analytics/dbt/pull/1814)) - Upgrade `werkzeug` to 0.15.6 ([#1697](https://github.com/fishtown-analytics/dbt/issues/1697), [#1814](https://github.com/fishtown-analytics/dbt/pull/1814)) - Pin `psycopg2` dependency to 2.8.x to prevent segfaults ([#1221](https://github.com/fishtown-analytics/dbt/issues/1221), [#1898](https://github.com/fishtown-analytics/dbt/pull/1898)) - Set a strict upper bound for `jsonschema` dependency ([#1817](https://github.com/fishtown-analytics/dbt/issues/1817), [#1821](https://github.com/fishtown-analytics/dbt/pull/1821), [#1932](https://github.com/fishtown-analytics/dbt/pull/1932)) #### Everything else - Provide test names and kwargs in the manifest ([#1154](https://github.com/fishtown-analytics/dbt/issues/1154), [#1816](https://github.com/fishtown-analytics/dbt/pull/1816)) - Replace JSON Schemas with data classes ([#1447](https://github.com/fishtown-analytics/dbt/issues/1447), [#1589](https://github.com/fishtown-analytics/dbt/pull/1589)) - Include test name and kwargs in test nodes in the manifest ([#1154](https://github.com/fishtown-analytics/dbt/issues/1154), [#1816](https://github.com/fishtown-analytics/dbt/pull/1816)) - Remove logic around handling `archive` blocks in the `dbt_project.yml` file ([#1580](https://github.com/fishtown-analytics/dbt/issues/1580), [#1581](https://github.com/fishtown-analytics/dbt/pull/1581)) - Remove the APIObject class ([#1762](https://github.com/fishtown-analytics/dbt/issues/1762), [#1780](https://github.com/fishtown-analytics/dbt/pull/1780)) ## Contributors Thanks all for your contributions to dbt! :tada: - [@captainEli](https://github.com/captainEli) ([#1809](https://github.com/fishtown-analytics/dbt/pull/1809)) - [@clausherther](https://github.com/clausherther) ([#1876](https://github.com/fishtown-analytics/dbt/pull/1876)) - [@jtcohen6](https://github.com/jtcohen6) ([#1784](https://github.com/fishtown-analytics/dbt/pull/1784)) - [@tbescherer](https://github.com/tbescherer) ([#1515](https://github.com/fishtown-analytics/dbt/pull/1515)) - [@aminamos](https://github.com/aminamos) ([#1609](https://github.com/fishtown-analytics/dbt/pull/1609)) - [@JusLarsen](https://github.com/JusLarsen) ([#1903](https://github.com/fishtown-analytics/dbt/pull/1903)) - [@heisencoder](https://github.com/heisencoder) ([#1823](https://github.com/fishtown-analytics/dbt/pull/1823)) - [@tjengel](https://github.com/tjengel) ([#1885](https://github.com/fishtown-analytics/dbt/pull/1885)) - [@Carolus-Holman](https://github.com/tjengel) ([#1747](https://github.com/fishtown-analytics/dbt/pull/1747), [#1743](https://github.com/fishtown-analytics/dbt/pull/1743)) - [@kconvey](https://github.com/tjengel) ([#1851](https://github.com/fishtown-analytics/dbt/pull/1851)) - [@darrenhaken](https://github.com/darrenhaken) ([#1787](https://github.com/fishtown-analytics/dbt/pull/1787)) ------------------------- # Louisa May Alcott Louisa May Alcott was born in 1832 in Philadelphia, Pennsylvania. Alcott's early education included lessons from noted transcendentalists Henry David Thoreau, Ralph Waldo Emerson, Nathaniel Hawthorne, Margaret Fuller, and Julia Ward Howe, all of whom were family friends. In 1868, Alcott wrote _Little Women_, a children's novel set in the Alcott family home some miles due north of Philadelphia. The novel is loosely based on Alcott's childhood experiences with her three sisters. The novel was well-received at the time and is still popular today. As an adult, Alcott was an abolitionist and a feminist. In 1847, she and her family served as station masters on the Underground Railroad. During this time, they housed a fugitive slave and had discussions with Frederick Douglass, a national leader of the abolitionist movement in America. Alcott read and admired the _[Declaration of Sentiments](https://en.wikipedia.org/wiki/Declaration_of_Sentiments)_, published by the Seneca Falls Convention on women's rights, advocating for women's suffrage. To learn more about Louisa May Alcott's life, consider visiting her Boston home (featured on the Boston Women's Heritage Trail) or her childhood home, Orchard House, which is now a museum that pays homage to Louisa May Alcott and her family with a focus on education.