Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE] 'dict' object is not callable when m2m authentication #776

Open
HugoDife opened this issue Sep 27, 2024 · 2 comments
Open

[ISSUE] 'dict' object is not callable when m2m authentication #776

HugoDife opened this issue Sep 27, 2024 · 2 comments

Comments

@HugoDife
Copy link

Description
I´m trying to connect m2m to databricks following the instructions in Databricks but when running the code I am getting the following error:

INFO:databricks.sql.thrift_backend:Error during request to server: {"method": "OpenSession", "session-id": null, "query-id": null, "http-code": null, "error-message": "", "original-exception": "'dict' object is not callable", "no-retry-reason": "non-retryable error", "bounded-retry-delay": null, "attempt": "1/4", "elapsed-seconds": "0.00045418739318847656/900.0"}
Traceback (most recent call last):
  File "/Users/test/Projects/databricks-test/dabricks_test.py", line 134, in <module>
    measures3 = measure_databricks_sql_performance(dbsql_config_m2m_pro, "SELECT * FROM catalog.schema.table LIMIT 2")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test/Projects/databricks-test/dabricks_test.py", line 99, in measure_databricks_sql_performance
    with sql.connect(**dbsql_config, _retry_stop_after_attempts_count=4, _tls_trusted_ca_file="/Users/test/Desktop/certificate/zscaler.pem") as connection:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test/miniconda3/envs/databricks/lib/python3.11/site-packages/databricks/sql/__init__.py", line 90, in connect
    return Connection(server_hostname, http_path, access_token, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test/miniconda3/envs/databricks/lib/python3.11/site-packages/databricks/sql/client.py", line 247, in __init__
    self._open_session_resp = self.thrift_backend.open_session(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test/miniconda3/envs/databricks/lib/python3.11/site-packages/databricks/sql/thrift_backend.py", line 549, in open_session
    response = self.make_request(self._client.OpenSession, open_session_req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test/miniconda3/envs/databricks/lib/python3.11/site-packages/databricks/sql/thrift_backend.py", line 478, in make_request
    self._handle_request_error(error_info, attempt, elapsed)
  File "/Users/test/miniconda3/envs/databricks/lib/python3.11/site-packages/databricks/sql/thrift_backend.py", line 308, in _handle_request_error
    raise network_request_error

From what I could see the class OAuthCredentialsProvider extends from CredentialsProvider, however the class CredentialsProvider in the databricks-sql-connector dont match the output, one directly return dict while the other return Callable[[], Dict[str, str]].

Is there something I am missing? Shouldnt these two classes be compatible?

Reproduction
Available in Databricks

Expected behavior
Everything work and return the result query.

Is it a regression?
Did this work in a previous version of the SDK? If so, which versions did you try?
Didn´t test

Other Information

  • OS: macOS
  • Version: 0.32.3
@shivonchain
Copy link

Getting the same error. Repro + stacktrace/debug logs below.

import logging
import os
import sys

from databricks import sql
from databricks.sdk.core import Config, oauth_service_principal

logging.basicConfig(
    stream=sys.stderr,
    level=logging.INFO,
    format="%(asctime)s [%(name)s][%(levelname)s] %(message)s",
)
logging.getLogger("databricks.sdk").setLevel(logging.DEBUG)
logging.getLogger("databricks.sql").setLevel(logging.DEBUG)
host = os.getenv("DATABRICKS_HOST")
http_path = os.getenv("DATABRICKS_HTTP_PATH")
client_id = os.getenv("DATABRICKS_OAUTH_CLIENT_ID")
client_secret = os.getenv("DATABRICKS_OAUTH_CLIENT_SECRET")
conn = sql.connect(
    server_hostname=host,
    http_path=http_path,
    credentials_provider=oauth_service_principal(
        Config(
            host=f"https://{host}",
            client_id=client_id,
            client_secret=client_secret,
        )
    ),
)
2024-10-23 12:28:02,003 [databricks.sdk][DEBUG] Attempting to configure auth: pat
2024-10-23 12:28:02,003 [databricks.sdk][DEBUG] Attempting to configure auth: basic
2024-10-23 12:28:02,003 [databricks.sdk][DEBUG] Attempting to configure auth: metadata-service
2024-10-23 12:28:02,004 [databricks.sdk][DEBUG] Attempting to configure auth: oauth-m2m
2024-10-23 12:28:03,414 [databricks.sdk.oauth][DEBUG] Retrieving token for <concealed by 1Password>
2024-10-23 12:28:04,137 [databricks.sql.thrift_backend][DEBUG] retry parameter: _retry_delay_min given_or_default 1.0
2024-10-23 12:28:04,137 [databricks.sql.thrift_backend][DEBUG] retry parameter: _retry_delay_max given_or_default 60.0
2024-10-23 12:28:04,137 [databricks.sql.thrift_backend][DEBUG] retry parameter: _retry_stop_after_attempts_count given_or_default 30
2024-10-23 12:28:04,137 [databricks.sql.thrift_backend][DEBUG] retry parameter: _retry_stop_after_attempts_duration given_or_default 900.0
2024-10-23 12:28:04,137 [databricks.sql.thrift_backend][DEBUG] retry parameter: _retry_delay_default given_or_default 5.0
2024-10-23 12:28:04,149 [databricks.sql.thrift_backend][DEBUG] Sending request: OpenSession(<REDACTED>)
2024-10-23 12:28:04,149 [databricks.sql.thrift_backend][INFO] Error during request to server: {"method": "OpenSession", "session-id": null, "query-id": null, "http-code": null, "error-message": "", "original-exception": "'dict' object is not callable", "no-retry-reason": "non-retryable error", "bounded-retry-delay": null, "attempt": "1/30", "elapsed-seconds": "0.0002009868621826172/900.0"}
Traceback (most recent call last):
  File "/Users/shivgupta/projects/biztech-dea/btda_dagster/file.py", line 22, in <module>
    conn = sql.connect(
           ^^^^^^^^^^^^
  File "/Users/shivgupta/projects/biztech-dea/venv/lib/python3.12/site-packages/databricks/sql/__init__.py", line 90, in connect
    return Connection(server_hostname, http_path, access_token, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shivgupta/projects/biztech-dea/venv/lib/python3.12/site-packages/databricks/sql/client.py", line 233, in __init__
    self._open_session_resp = self.thrift_backend.open_session(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shivgupta/projects/biztech-dea/venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 578, in open_session
    response = self.make_request(self._client.OpenSession, open_session_req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shivgupta/projects/biztech-dea/venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 507, in make_request
    self._handle_request_error(error_info, attempt, elapsed)
  File "/Users/shivgupta/projects/biztech-dea/venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 337, in _handle_request_error
    raise network_request_error
databricks.sql.exc.RequestError: Error during request to server

@shivonchain
Copy link

This may be related to the discussion here: databricks/databricks-sql-python#423

Fixed using @kravets-levko's example below

from databricks.sdk.core import Config, oauth_service_principal
from databricks import sql
import os

server_hostname = os.getenv("DATABRICKS_SERVER_HOSTNAME")

def credential_provider():
  config = Config(
    host          = f"https://{server_hostname}",
    client_id     = os.getenv("DATABRICKS_CLIENT_ID"),
    client_secret = os.getenv("DATABRICKS_CLIENT_SECRET")
  )

  def get_headers():
    return oauth_service_principal(config)               

  return get_headers


with sql.connect(server_hostname      = server_hostname,
                 http_path            = os.getenv("DATABRICKS_HTTP_PATH"),
                 credentials_provider = credential_provider) as connection:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants