Skip to main content

Move from dbt Core to dbt Cloud: Get started

Updated Total estimated time: 3-4 hours
Migration
dbt Core
dbt Cloud
Intermediate
Menu

    Introduction​

    Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service.

    dbt Cloud is the fastest and most reliable way to deploy dbt. It enables you to develop, test, deploy, and explore data products using a single, fully managed service. It also supports:

    Learn more about dbt Cloud features.

    dbt Core is an open-source tool that enables data teams to transform data following analytics engineering best practices using a command line interface. It must be self-hosted and maintained.

    What you'll learn​

    This guide outlines the steps you need to take to move from dbt Core to dbt Cloud and highlights the necessary technical changes:

    • Account setup: Learn how to create a dbt Cloud account, invite team members, and configure it for your team.
    • Data platform setup: Find out about connecting your data platform to dbt Cloud.
    • Git setup: Learn to link your dbt project's Git repository with dbt Cloud.
    • Developer setup: Understand the setup needed for developing in dbt Cloud.
    • Environment variables: Discover how to manage environment variables in dbt Cloud, including their priority.
    • Orchestration setup: Learn how to prepare your dbt Cloud environment and jobs for orchestration.
    • Models configuration: Get insights on validating and running your models in dbt Cloud, using either the dbt Cloud IDE or dbt Cloud CLI.
    • What's next?: Summarizes key takeaways and introduces what to expect in the following guides.

    Prerequisites​

    Account setup​

    This section outlines the steps to set up your dbt Cloud account and configure it for your team.

    1. Create your dbt Cloud account.

    2. Provide user access and invite users to your dbt Cloud account and project.

    3. Configure Single Sign-On (SSO) or Role-based access control (RBAC) for easy and secure access. enterprise

      • This removes the need to save passwords and secret environment variables locally.

    4. In Account settings, enable partial parsing to only reparse changed files, saving time.

    5. In Account settings, enable Git repo caching for job reliability & third-party outage protection. enterprise

    Data platform setup​

    This section outlines the considerations and methods to connect your data platform to dbt Cloud.

    1. In dbt Cloud, set up your data platform connections and environment variables. dbt Cloud can connect with a variety of data platform providers including:

    2. You can verify your data platform connections by clicking the Test connection button in your deployment and development credentials settings.

    Additional configuration​

    Explore these additional configurations to optimize your data platform setup further:

    1. Use OAuth connections, which enables secure authentication using your data platform’s SSO. enterprise

    Git setup​

    Your existing dbt project source code should live in a Git repository. In this section, you will connect your existing dbt project source code from Git to dbt Cloud.

    1. Ensure your dbt project is in a Git repository.

    2. In Account settings, select Integrations, and connect and configure Git in dbt Cloud to connect your Git repository:

    Additional configuration​

    Explore these additional configurations to optimize your Git setup further:

    1. Log into dbt Cloud using OAuth connections to integrate with your source code platform. It automatically links to the repository using one of the native integrations set at the account level. enterprise

      Set up groups for dbt project access with those configured for repository access to streamline permissions.

    Developer setup​

    This section highlights the development configurations you’ll need for your dbt Cloud project. The following categories are covered in this section:

    dbt Cloud environments​

    The concept of an environment in dbt Cloud is the same as a target in dbt Core.

    The primary difference between a dbt Cloud environment and a target in dbt Core is that you can make these configurations through the dbt Cloud UI instead of within the profiles.yml file.

    This difference streamlines the process of switching between development, staging, and production contexts, removing the need to manually edit the profiles.yml file. dbt Cloud environments also integrate with additional features such as job scheduling, version control, and more — making it easier to manage the full lifecycle of your dbt projects within a single platform. You can set up or customize target names in dbt Cloud.

    Initial setup steps​

    1. Set up development environment — Set up your development environment and development credentials. You’ll need this to access your dbt project and start developing.

    2. dbt Core version — In your dbt Cloud environment and credentials, use the same dbt Core version you use locally. You can run dbt --version in the command line to find out which version of dbt Core you’re using.

      • Once your full migration is complete, consider upgrading your environments to Keep on latest version to always get the latest features and more.

    3. Connect to your data platform — When using dbt Cloud, you can connect to your data platform directly in the UI.

      • Each environment is roughly equivalent to an entry in your profiles.yml file. This means you don't need a profiles.yml file in your project.

    4. Development tools — Set up your development workspace with the dbt Cloud CLI or dbt Cloud IDE to edit and develop your dbt code in your tool of choice.

    Additional configuration​

    Explore these additional configurations to optimize your developer setup further:

    1. Custom target names — If you’re using a custom target.name in your project, we recommend you set them using environment variables. Alternatively, you can update it at the developer credentials level.

    dbt Cloud commands​

    1. Review the dbt commands supported for dbt Cloud development. For example, dbt debug isn’t needed in dbt Cloud since the UI displays logs for your viewing.

    Environment variables​

    This section will help you understand how to set up and manage dbt Cloud environment variables for your project. The following categories are covered:

    In dbt Core, environment variables, or the env_var function, are defined in the profiles.yml file. In dbt Cloud, you can set environment variables in the dbt Cloud user interface (UI). Read Set up environment variables for more info.

    Environment variables in dbt Cloud​

    • dbt Cloud environment variables must be prefixed with DBT_ (including DBT_ENV_CUSTOM_ENV_ or DBT_ENV_SECRET_).
    • If your dbt Core environment variables don’t follow this naming convention, perform a “find and replace” in your project to make sure all references to these environment variables contain the proper naming conventions.
    • dbt Cloud secures environment variables, offering additional measures for sensitive values, such as prefixing keys with DBT_ENV_SECRET_ to obscure them in logs and the UI.
    Setting project level and environment level valuesSetting project level and environment level values

    dbt Cloud environment variables order of precedence​

    Environment variables in dbt Cloud are managed with a clear order of precedence, allowing users to define values at four levels (highest to lowest order of precedence):

    • The job level (job override) or in the IDE for an individual developer (personal override). Highest precedence
    • The environment level, which can be overridden by the job level or personal override.
    • A project-wide default value, which can be overridden by the environment level, job level, or personal override.
    • The optional default argument supplied to the env_var Jinja function in the code. Lowest precedence
    Environment variables order of precedenceEnvironment variables order of precedence

    Set environment variables in dbt Cloud​

    • To set these variables for an entire project or specific environments, navigate to Deploy > Environments > Environment variables tab.
    • To set these variables at the job level, navigate to Deploy > Jobs > Select your job > Settings > Advanced settings.
    • To set these variables at the personal override level, navigate to Profile Settings > Credentials > Select your project > Environment variables.

    Orchestration setup​

    This section outlines the considerations and methods to set up your dbt Cloud environments and jobs for orchestration. The following categories are covered in this section:

    dbt Cloud environments​

    To use the dbt Cloud's job scheduler, set up one environment as the production environment. This is the deployment environment. You can set up multiple environments for different stages of your deployment pipeline, such as development, staging/QA, and production.

    Initial setup steps​

    1. dbt Core version — In your environment settings, configure dbt Cloud with the same dbt Core version.

      • Once your full migration is complete, we recommend upgrading your environments to Keep on latest version to always get the latest features and more. You only need to do this once.

    2. Configure your jobs — Create jobs for automated or event-driven dbt jobs. You can use cron execution, manual, pull requests, or API triggers.

    Additional configuration​

    Explore these additional configurations to optimize your dbt Cloud orchestration setup further:

    1. Custom target names — Use environment variables to set a custom target.name for every corresponding dbt Cloud job at the environment level.

    2. dbt commands — Add any relevant dbt commands to execute your dbt Cloud jobs runs.

    3. Notifications — Set up notifications by configuring email and Slack alerts to monitor your jobs.

    4. Monitoring tools — Use monitoring tools like run history, job retries, job chaining, dashboard status tiles, and more for a seamless orchestration experience.

    5. API access — Create API auth tokens and access to dbt Cloud APIs as needed. teamenterprise

    6. dbt Explorer — If you use dbt Explorer and run production jobs with an external orchestrator, ensure your production jobs run dbt run or dbt build to update and view models and their metadata in dbt Explorer. Running dbt compile alone will not update model metadata. In addition, features like column-level lineage also requires catalog metadata produced through running dbt docs generate. teamenterprise

    CI/CD setup​

    Building a custom solution to efficiently check code upon pull requests is complicated. With dbt Cloud, you can enable continuous integration / continuous deployment (CI/CD) and configure dbt Cloud to run your dbt projects in a temporary schema when new commits are pushed to open pull requests.

    This build-on-PR functionality is a great way to catch bugs before deploying to production, and an essential tool for data practitioners.

    1. Set up an integration with a native Git application (such as Azure DevOps, GitHub, GitLab) and a CI environment in dbt Cloud.
    2. Create a CI/CD job to optimize workflows.
    3. Run your jobs in a production environment to fully implement CI/CD. Future pull requests will also leverage the last production runs to compare against.

    Models configuration​

    In this section, you’ll be able to validate whether your models run or compile correctly in your development tool of choice: The dbt Cloud IDE or dbt Cloud CLI.

    You’ll want to make sure you set up your development environment and credentials.

    1. In your development tool of choice, you can review your dbt project, ensure it's set up correctly, and that you can run dbt commands. For example:

      • Run dbt compile to make sure your project compiles correctly.
      • Run a few models in the dbt Cloud IDE or dbt Cloud CLI to ensure you’re experiencing accurate results in development.

    2. Once your first job has successfully run in your production environment, use dbt Explorer to view your project's resources (such as models, tests, and metrics) and their data lineage  to gain a better understanding of its latest production state. teamenterprise

    What’s next?​

    Congratulations on completing the first part of your move to dbt Cloud 🎉!

    You have learned:

    • How to set up your dbt Cloud account
    • How to connect your data platform and Git repository
    • How to configure your development, orchestration, and CI/CD environments
    • How to set up environment variables and validate your models.

    For next steps, we'll soon share other guides on how to manage your move and tips/faqs. Stay tuned!

    0