> ## Documentation Index
> Fetch the complete documentation index at: https://wisdomai-add-trigger-agent-api.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect WisdomAI to Databricks

WisdomAI connects to Databricks using a **Client ID** and **Client Secret** for authentication. This method ensures secure and granular control over the data WisdomAI can access.

## Prerequisites

Before you begin, make sure you have the following:

* An active Databricks workspace.
* Permissions to create and manage **Service Principals** within your Databricks account or workspace.
* A Databricks **SQL Warehouse** set up and running.

## 1. Obtain Databricks Connection Details

You will need the following four pieces of information from your Databricks environment:

1. **Host**: The URL of your Databricks workspace, which typically looks like `https://<region>.cloud.databricks.com/`. You can find this in your SQL Warehouse connection details.
2. **HTTP Path**: The specific path to your SQL Warehouse, usually starting with `/sql/1.0/warehouses/`. You can find this in the "Connection Details" tab of your SQL Warehouse.
3. **Client ID**: The Application ID of your Databricks Service Principal.
4. **Client Secret**: The secret key generated for your Databricks Service Principal.

### How to Find Your Connection Details

Follow these steps to find the necessary credentials.

#### Host and HTTP Path

1. Navigate to your Databricks Workspace.
2. Go to \*\*SQL Warehouses \*\*(or SQL Endpoints).
3. Select the warehouse you want to connect to.
4. Click on the **Connection Details** tab. Here you will find the **Server Hostname** (your Host) and the **HTTP Path**.

#### Client ID and Client Secret (using a Databricks-managed Service Principal)

You'll first need to create a Service Principal in Databricks.

**Step 1: Create a Service Principal**

1. Log in to your Databricks environment.
   * **A) account-level permissions:** Log in to your Databricks **account console**.
   * **B) administrator-level permissions:** Log in to your Databricks **workspace** as an administrator.
2. Navigate to the Service Principals section.
   * **A) From Account Console:** Navigate to **User management > Service principals**.
   * **B) From Workspace:** Navigate to **Admin Settings** (click your username in the top right, then Admin Settings) > **Identity and access > Service principals**.
3. Click **Add service principal**.
4. Select **Databricks managed**.
5. Provide a descriptive **Name** for the service principal (e.g., WisdomAI-ServicePrincipal).
6. Click **Add service principal**.
7. Once created, you will see the **Application ID** for this service principal. This is your **Client ID**. Copy this value.

**Step 2: Generate a Client Secret for the Service Principal**

1. From the service principal's page, go to the **Secrets** tab.
2. Under OAuth secrets, click **Generate secret**.
3. Set a lifetime for the secret (up to 730 days) and click **Generate**.

   <Note>
     Choose an expiration that balances security and operational convenience.
   </Note>
4. Immediately copy the displayed secret. This is your **Client Secret**.

<Warning>
  The Client Secret is only shown once and cannot be retrieved later. Please store it in a secure location immediately.
</Warning>

## 2. Assign Permissions in Databricks

The Service Principal needs specific permissions to access the required data that WisdomAI needs.

* **SQL Warehouse/Cluster Permissions**: Grant **CAN USE** permission to the Service Principal on the SQL Warehouse you are connecting to. You can do this from the **Permissions** tab of the SQL Warehouse.
* **Data Privileges**: The service principal requires the following privileges in Unity Catalog (or Hive Metastore):
  * `USE CATALOG` on the target catalog.
  * `USE SCHEMA` on the target schema.
  * `SELECT` on the tables and views you want WisdomAI to access.

You can grant these permissions using SQL commands. Here are some examples:

```sql
-- Grant permission to use a specific catalog
GRANT USAGE ON CATALOG <your_catalog_name> TO `<service_principal_id>`;

-- Grant permission to use a specific schema
GRANT USAGE ON SCHEMA <your_catalog_name>.<your_schema_name> TO `<service_principal_id>`;

-- Grant select permission on table
GRANT SELECT ON TABLE <your_catalog_name>.<your_schema_name>.<your_table_name> TO `<service_principal_id>`;

-- Grant select permission on all tables in a schema
GRANT SELECT ON ALL TABLES IN SCHEMA <your_catalog_name>.<your_schema_name> TO `<service_principal_id>`;
```

Replace `<service_principal_id>` with the Application ID of your service principal, and `<your_catalog_name>`,`<your_schema_name>`, and `<your_table_name>` with your specific values.

## 3. Connect WisdomAl to Databricks

Once you have gathered the required information, you can configure the connection in WisdomAl:

1. In WisdomAl, navigate to the **Connections** section and click **Add Connection**.
2. Select **Databricks** as the data source type.
3. Fill in the connection details in the "Databricks connection details" section.
   * **Connection Name**: Choose a descriptive name for your connection (e.g., "Databricks Prod").
   * **Host**: Paste the Server Hostname you obtained from Databricks.
   * **HTTP Path**: Paste the HTTP Path for your SQL Warehouse.
   * **Client ID**: Enter your Client ID (Application ID). This field is mandatory.
   * **Client Secret**: Enter your Client Secret (the value copied immediately after creation). This field is mandatory.
   * **Catalog Filters (Optional)**: Specify any Catalog Filters if you want to further restrict the catalogs/schemas WisdomAl crawls.
4. Click **Save and Sync metadata**. WisdomAl will use the provided credentials to connect to Databricks and scan the metadata of the specified catalogs and tables.

<img src="https://mintcdn.com/wisdomai-add-trigger-agent-api/24Y8uyueYkarqKQ-/images/databricks.png?fit=max&auto=format&n=24Y8uyueYkarqKQ-&q=85&s=6b669903a10a5f98cae6d314ed8b7ede" alt="image.png" width="3002" height="1720" data-path="images/databricks.png" />

## Security Considerations and Best Practices

* **Least Privilege**: Always follow the principle of least privilege. Only grant the necessary permissions to the Service Principal.
* **Secure Key Management**: Treat your Client ID and Client Secret like passwords. Avoid sharing them via insecure channels, such as unencrypted email. Use secure methods, such as LastPass or other secure file-sharing services.
* **Credential Rotation**: Regularly rotate your Client Secrets to enhance security, especially before their expiration.

## Troubleshooting Common Issues

Having trouble? Here are solutions to some frequently encountered problems.

### Authentication Failed or Invalid Credentials

* Ensure the Host, HTTP Path, Client ID, and Client Secret are complete and correct. No extra characters, spaces, or missing lines.
* Verify that the Client Secret has not expired or been revoked in Databricks.
* Confirm that the Service Principal exists and is enabled in Databricks.

### Permission Denied Errors

* Double-check that the Service Principal has the necessary `CAN USE` permission on the SQL Warehouse.
* Verify that the Service Principal has the correct `USE CATALOG`, `USE SCHEMA`, and `SELECT` privileges on the Databricks catalogs, schemas, and tables you are trying to access.
* Confirm that the SQL Warehouse is running and accessible.

### Catalog Not Found or Table Not Found

* Verify the spelling of the catalog or table names.
* If using catalog filters in WisdomAl, ensure the catalog is included in the filter.
* Confirm that the Service Principal has permissions on the specific catalog you are trying to access.

### Connection Timeout or Network Errors

* Verify that WisdomAl's CIDR block (3`5.238.115.103/32` or `34.82.248.105/32`) is allowlisted in your Snowflake network policies or any corporate firewalls.