Skip to main content
Version: devel

HubSpot

Need help deploying these sources or figuring out how to run them in your data stack?
Join our Slack community or Book a call with a dltHub Solutions Engineer.

HubSpot is a customer relationship management (CRM) software and inbound marketing platform that helps businesses attract visitors, engage customers, and close leads.

This HubSpot dlt verified source and pipeline example loads data using the “HubSpot API” to the destination of your choice.

NameDescription
contactsvisitors, potential customers, leads
companiesinformation about organizations
dealsdeal records, deal tracking
ticketsrequests for help from customers or users
productspricing information of a product
quotesprice proposals that salespeople can create and send to their contacts
hubspot_events_for_objectsweb analytics events for a given object type and object ids

To get details about endpoints that can be loaded, see hubspot/settings.py.

Setup guide​

Grab credentials​

Note: As of November 30, 2022, HubSpot API Keys are being deprecated and are no longer supported. Instead, we recommend authenticating using a private app access token or OAuth access token.

Create a private app and get an authentication token before running the pipeline example. Follow these steps:

  1. In HubSpot, click the ⚙️ icon to access settings.

  2. Under "Account Setup" in the left sidebar, choose "Integrations" > "Private Apps".

  3. Select “Create a private app”.

  4. In the “Basic Info” tab, provide a name and description.

  5. In the “Scopes” tab, grant:

    • Read scopes for CMS, CRM, and Settings.
    • Permissions for:
      business-intelligence, actions, crm.export, e-commerce, oauth, tickets

  6. Click "Create app" > "Continue Creating".

  7. Click "Show token" and store it for ".dlt/secrets.toml".

Note: The HubSpot UI, which is described here, might change. The full guide is available at this link.

Initialize the verified source​

To get started with your data pipeline, follow these steps:

  1. Enter the following command:

    dlt init hubspot duckdb

    This command will initialize the pipeline example with HubSpot as the source and duckdb as the destination.

  2. If you'd like to use a different destination, simply replace duckdb with the name of your preferred destination.

  3. After running this command, a new directory will be created with the necessary files and configuration settings to get started.

For more information, read the guide on how to add a verified source.

Add credentials​

  1. Inside the .dlt folder, you'll find a file called secrets.toml, which is where you can securely store your access tokens and other sensitive information. It's important to handle this file with care and keep it safe. Here's what the file looks like:

    # put your secret values and credentials here
    # do not share this file and do not push it to GitHub
    [sources.hubspot]
     api_key = "api_key" # please set me up!

  2. Replace the access_token value with the previously copied one to ensure secure access to your Hubspot resources.

  3. Enter credentials for your chosen destination as per the docs.

For more information, read the General Usage: Credentials.

Run the pipeline​

  1. Before running the pipeline, ensure that you have installed all the necessary dependencies by running the command:
    pip install -r requirements.txt
  2. You're now ready to run the pipeline! To get started, run the following command:
    python hubspot_pipeline.py
  3. Once the pipeline has finished running, you can verify that everything loaded correctly by using the following command:
    dlt pipeline <pipeline_name> show
    For example, the pipeline_name for the above pipeline example is hubspot_pipeline, you may also use any custom name instead.

For more information, read the guide on how to run a pipeline.

Sources and resources​

dlt works on the principle of sources and resources.

Default endpoints​

You can write your own pipelines to load data to a destination using this verified source. However, it is important to note the complete list of the default endpoints given in hubspot/settings.py.

Source hubspot​

This function returns a list of resources to load companies, contacts, deals, tickets, products, and web analytics events data into the destination.

@dlt.source(name="hubspot")
def hubspot(
    api_key: str = dlt.secrets.value,
    include_history: bool = False,
) -> Sequence[DltResource]:
   ...

api_key: The key used to authenticate with the HubSpot API. Configured in "secrets.toml".

include_history: This parameter, when set to "True", loads the history of property changes for the specified entities.

Resource companies​

This resource function fetches data from the "companies" endpoint and loads it to the destination, replacing any existing data.

@dlt.resource(name="companies", write_disposition="replace")
def companies(
   api_key: str = api_key,
   include_history: bool = include_history,
   props: Sequence[str] = DEFAULT_COMPANY_PROPS,
   include_custom_props: bool = True,
) -> Iterator[TDataItems]:
   """Hubspot companies resource"""
   yield from crm_objects(
      "company",
      api_key,
      include_history=include_history,
      props=props,
      include_custom_props=include_custom_props,
   )

This resource function takes the same arguments, api_key and include_history as the "hubspot" source described above, but also supports two additional parameters. include_custom_props - indicates if all the properties of CRM objects, except Hubspot driven (prefixed with hs_), are to be extracted. props - the list of properties to extract in addition to the custom properties. Similar to this, resource functions "contacts", "deals", "tickets", "products", and "quotes" retrieve data from the Hubspot API.

Resource hubspot_events_for_objects​

This function loads web analytics events for specific objects from the Hubspot API into the destination.

@dlt.resource
def hubspot_events_for_objects(
     object_type: THubspotObjectType,
     object_ids: List[str],
     api_key: str = dlt.secrets.value,
     start_date: pendulum.DateTime = STARTDATE,
) -> DltResource:
   ...

object_type: One of the Hubspot object types as defined in hubspot/settings.py..

object_ids: List of object IDs to track events.

api_key: The key used to authenticate with the HubSpot API. Configured in "secrets.toml".

start_date: The initial date time from which to start getting events, default to "01-01-2000", configured in hubspot/settings.py..

Customization​

Create your own pipeline​

If you wish to create your own pipelines, you can leverage source and resource methods from this verified source.

  1. Configure the pipeline by specifying the pipeline name, destination, and dataset as follows:

    pipeline = dlt.pipeline(
        pipeline_name="hubspot",  # Use a custom name if desired
        destination="duckdb",  # Choose the appropriate destination (e.g., duckdb, redshift, post)
        dataset_name="hubspot_data"  # Use a custom name if desired
    )

    To read more about pipeline configuration, please refer to our documentation.

  2. To load all the data from contacts, companies, deals, products, tickets, and quotes into the destination.

    load_data = hubspot()
    load_info = pipeline.run(load_data)
    print(load_info)

  3. To load data from contacts and companies, with time history using the "with_resources" method.

    load_data = hubspot(include_history=True).with_resources("companies","contacts")
    load_info = pipeline.run(load_data)
    print(load_info)
    1. include_history loads property change history and entities as separate tables. By default, it is set as False.

  4. By default, all the custom properties of a CRM object are extracted. If you want only particular fields, set the flag include_custom_props=False and add a list of properties with the props arg.

    load_data = hubspot()
    load_data.contacts.bind(props=["date_of_birth", "degree"], include_custom_props=False)
    load_info = pipeline.run(load_data.with_resources("contacts"))

  5. If you want to read all the custom properties of CRM objects and some additional (e.g., Hubspot driven) properties.

    load_data = hubspot()
    load_data.contacts.bind(props=["hs_content_membership_email", "hs_content_membership_email_confirmed"])
    load_info = pipeline.run(load_data.with_resources("contacts"))

  1. To load the web analytics events of a given object type.

    resource = hubspot_events_for_objects("company", ["7086461639", "7086464459"])
    # Here, object type: company, and object IDs: 7086461639 and 7086464459
    load_info = pipeline.run([resource])
    print(load_info)

    1. This function uses "object_type" and "object_id" as arguments.

    2. This function loads data incrementally and tracks the occurred_at.last_value parameter from the previous pipeline run. Refer to our official documentation for more information on incremental loading.

Additional info​

If you encounter the following error while processing your request:

ERROR

Your request to HubSpot is too long to process. The maximum allowed query length is 2000 symbols, while your list is 2125 symbols long.

Please note that by default, HubSpot requests all default properties and all custom properties (which are user-created properties in HubSpot). Therefore, you need to request specific properties for each entity (contacts, companies, tickets, etc.).

Default properties are defined in settings.py, and you can change them.

The custom properties could cause the error as there might be too many of them available in your HubSpot. To change this, you can pass include_custom_props=False when initializing the source:

info = p.run(hubspot(include_custom_props=False))

Or, if you wish to include them, you can modify settings.py.

Additional Setup guides​

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.