Support for more authentication mechanisms

[ghjklw]

Describe the feature

Thanks to the work done in #827, dbt-databricks systematically uses Databricks' databricks.sdk.core.Config to configure authentication mechanisms.

As of now, only a limited subset (PAT, OAuth U2M, Oauth M2M and Azure client/secret) of the authentication mechanisms are exposed. However the new implementation makes it really easy to extend it further based on all the options offered by Databricks' SDK for Python which follows itself Databricks unified authentication.

I am very interested in implementing it for Azure CLI and Azure Managed Identity and would love to submit a PR.

Describe alternatives you've considered

• The first alternative would be to follow the current pattern in https://github.com/databricks/dbt-databricks/blob/1.10.latest/dbt/adapters/databricks/credentials.py while trying to stick as closely as possible to Databricks unified authentication behavior/variable names.
• The second alternative would be the one I mentioned in an earlier comment to pass transparently the options to Databricks' SDK for Python in order to maximize the compatibility with all the options availables and be able to quickly deliver newer options as they're implemented. I would then make sure to not break the existing functionality by keeping the existing logic and only passing the options through to Databricks SDK if none of the existing options matched the configuration.

Who will this benefit?

Any user who wants to leverage more authentication options, and potentially increase security.
For the project I am currently working on:¨

• dbt-databricks is the last component not leveraging Azure CLI authentication in the development environment.
• It is also the only component preventing us from deactivating Databricks PAT
• In Azure, we already rely extensively on managed identity, but have to fallback to client id/secret for dbt-databricks

Such a change would therefore be a significant improvement both in terms of convenience and security.

Are you interested in contributing this feature?

Happy to implement it 😃

[benc-db]

@ghjklw This would be a great feature to add, just so long as we can ensure that it doesn't disrupt existing auth mechanisms (as you've called out :) ). I think progressing toward support unified auth is the right long term move, but even if you just add the mechanisms that you need, I would be happy to review and consider for inclusion.

[Rhiyo]

One thing I like to make use of else where with the Config class is if you don't give it any parameters, i.e just call it with Config(), it will automatically try and get variables with specific env vars and use those to connect. We do this with OAUTH - have our OAUTH credentials mapped to environment variables at a container level and don't bother passing them in to Config.

It'd be great if you could not specify any auth type or auth credentials in the dbt profile config, and if that's the case it'd just use an empty Config() and try and workout the connection using available environment variables.

[benc-db]

@Rhiyo one of the goals of moving towards universal client auth is getting to a place where that use case works as expected, and works the same across all Databricks apps. We'll get there eventually :)

[Rhiyo]

This is something that is happening for the soda core library as well. Trying to workout the best way to authenticate. A yet to be merged PR I made on that library.

It's trying to solve a very similar problem (although it does come with it's own set of baggage when it mixes iin with the other connection types). One thing that I do here is leverage the "configuration" key available in the yaml credentials - this can take abritrary key value pairs which I sent across as kwargs directly into the Config class.

(I just noticed it's the same OP who of this post that sparked the conversation in the soda library as well.)