Skip to main content

I am trying to understand the driver behavior when connecting to an Azure DataBricks workspace using AzureServicePrincipal. Given the Azure Service principal and secret it seems to first obtain the Azure AD token and then it tries to call DataBricks REST API using the Azure AD token to obtain a Personal Access Token for the service principal. 

This is a problem because we do not allow personal access tokens in our environment. Is there a reason why it cannot use the Azure AD token directly to obtain access? Appreciate any insights or help.

Hi @ntropic,

Your observation appears to be correct. Indeed it does seem like the current authorization process for the Azure Service Principal is as follows:

  1. The driver retrieves an AzureAD token from Microsoft Azure based on the Azure properties provided in the connection configuration.
  2. Once the AzureAD token is retrieved, the driver makes use of the token and creates a Personal Access Token (PAT) from Databricks.
  3. This newly retrieved PAT is used to authenticate the REST API for Databricks.

We are not entirely sure about the reason behind this initial implementation in the Azure Databricks driver however looking at the Microsoft documentation it does look like it is possible to access the REST API without the need of a PAT from Databricks.
 https://learn.microsoft.com/en-us/azure/databricks/dev-tools/service-prin-aad-token 

That said, we have a ticket in our system about this and the developers are currently looking into making this change. This means once the change is implemented the driver will only use the token generated from Microsoft Azure, eliminating the need to create a Databricks PAT. We will be sure to update you once the change is made.

Thank you very much for the response

Reply


Virtual Event: Data Virtualization, Reimagined Ma”arch 14, 2024 style=
Terms and ConditionsCookie settings