Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql_alchemy_conn is sqlite instead of postgres #398

Open
mcornwell16 opened this issue Oct 4, 2024 · 0 comments
Open

sql_alchemy_conn is sqlite instead of postgres #398

mcornwell16 opened this issue Oct 4, 2024 · 0 comments

Comments

@mcornwell16
Copy link

I built the image with all default settings from the repo and noticed that the sql_alchemy_con is a sqlite URL after exec'ing into the local-runner container and running the command airflow config list.

I was investigating because I noticed running airflow variables list after exec'ing into the container returned errors saying the table could not be found.

Here is a snippet from airflow config retrieved from within the container:

alembic_ini_file_path = alembic.ini
sql_alchemy_conn = sqlite:////usr/local/airflow/airflow.db
# sql_alchemy_engine_args = 
sql_engine_encoding = utf-8
# sql_engine_collation_for_ids = 
sql_alchemy_pool_enabled = False
sql_alchemy_pool_size = 5
sql_alchemy_max_overflow = 10
sql_alchemy_pool_recycle = 1800
sql_alchemy_pool_pre_ping = True
sql_alchemy_schema = 
# sql_alchemy_connect_args = 
# sql_alchemy_session_maker = 
load_default_connections = True
max_db_retries = 3
check_migrations = False

Here is the error output when running airflow variables list:

  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: variable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/__main__.py", line 62, in main
    args.func(args)
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/cli.py", line 379, in _wrapper
    f(*args, **kwargs)
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/variable_command.py", line 42, in variables_list
    variables = session.scalars(select(Variable)).all()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1778, in scalars
    return self.execute(
           ^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: variable
[SQL: SELECT variable.val, variable.id, variable."key", variable.description, variable.is_encrypted 
FROM variable]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

On an side - if I set AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow in either the docker-compose or .env.localrunner - all works fine. The sql_alchemy_conn gets properly set within the airflow config and running airflow variables list runs without error.

@mcornwell16 mcornwell16 changed the title sql_alchemy_conn is sqlit instead of postgres sql_alchemy_conn is sqlite instead of postgres Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant