Adding a DATABASES["default"]["TEST"]["NAME"] test database creation during pytest testing?

[HilbertSpecs]

Hello All,

I have an existing non cookiecutter Django project where in settings.py: definition of default database I was able to create a separate test database that was created and used every time pytest was being run and when completed the test database was destroyed after. It was clearly stated in the console output every time pytest was run.

# non cookiecutter project
#settings.py
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': "dbname",
        'TEST': {
            'NAME': "testdbname",
        },
        'USER': "user",
        'PASSWORD': "password",
        'HOST': "host",
        'PORT': "port",
    }
}

I have started a new cookiecutter project where in config/settings/base.py the default database is defined as:

# cookiecutter project
#config/settings/base.py
DATABASES = {"default": env.db("DATABASE_URL")}

Now when I run pytest tests the tests are running against the DATABASES["default"] database.

There is no test database being created/used/destroyed output in the console. I am trying to achieve the same functionality I had in non cookiecutter django project where every time pytest is run pytest is creating a test database running tests on it and then destroying the testdb after.

I have tried adding the following line to:

# cookiecutter project
#config/settings/base.py
DATABASES["default"]["TEST"]["NAME"]="testdbname"

But I keep getting a KEY_ERROR "TEST".
I am not allowed to define a test database like that.

QUESTION :

Does anyone know how to create the equivalent functionality in cookiecutter project?

[browniebroke]

Like this?

DATABASES["default"]["TEST"] = {"NAME": "testdbname"}

Also, you should probably set this in config/settings/test.py

[HilbertSpecs]

@browniebroke Thanks for your response. I appreciate your help. If you don't mind helping further-can you help me with further understanding?

Yes that way of defining the Test Database New Name solved the KEY_ERROR "TEST" error that was being thrown also thank you for pointing out the extra advice that I should move it to config/settings/test.py I put it there and the newly named "testdbname" was created locally.

Although during this process I realized that in previous runs pytest had already created the default test database locally named "test_dbname" which is default test database name for postgres database according to documentation.

I didn't bother using new "testdbname" database once I realized one was already created I just went back to using the original "test_dbname" database that was previously created in initial pytest run.

Upon further investigation I realized that some of my unit tests were missing the pytest marker:

@pytest.mark.django_db

So I believe that when pytest was running running all tests:

1. The unmarked django_db unit tests were hitting the default database.
2. While the marked django_db unit tests were hitting the "test_dbname" default test database.

Please correct me if I'm wrong.

ONLY Once the mark was added to unit tests: they started to use the "test_dbname" database.

@pytest.mark.django_db

Running in PyCharm test runner: the following output would be displayed

The last line clearly stating that existing "test_dbname" was being used to run the tests on.

Operations to perform:
  Synchronize unmigrated apps: 
  Apply all migrations:
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
  No migrations to apply.
Using existing test database for alias 'default' ('test_dbname')...

Although when running: 'pytest' command in terminal running app both locally and in docker-compose

% pytest

I get no clear indication that tests are being run on the test database "test_dbname".
They are all being successfully run and they all pass BUT:

The following line is missing from pytest output:

'Using existing test database for alias 'default' ('test_dbname')...'

Previously in non cookiecutter project when I would run 'pytest' command in terminal it would always clearly state a similar output of creating/using test database like the above line BUT NOW in cookiecutter project that line IS MISSING from the terminal output but if the tests are run in PyCharm Run/Debug test runner the line IS INCLUDED in console output.

Question:

How can I be sure that when running 'pytest' command it is running tests against "test_dbname" database and not default database? Am I missing a pytest.ini file configuration setting somewhere?

[browniebroke]

I'm no pytest expert but AFAIK, the @pytest.mark.django_db is required for any tests that use the database. Without it, pytest doesn't silently use the default DB, it raises and exception: RuntimeError: Database access not allowed, use the "django_db" mark, or the "db" or "transactional_db" fixtures to enable it..

I'm not sure why your tests are using the default DB, I never seen that before... That's more likely an issue with pytest-django than cookiecutter-django. You may want to bring this topic up to their repo, or to a discussion forum with more folks, like pytest/Django forum/discords or Stackoverflow.

[HilbertSpecs]

Okay thanks for your response I appreciate it. I'll ask on pytest-django discussion. That's what I thought I was supposed to happen based on my understanding of what I was reading in the documentation. That Runtime Error is never being thrown: all of my unit tests are marked with the @pytest.mark.django_db and all my tests are passing locally and in docker compose so I'm assuming the default database is not being hit by the test. PgAdmin Dashboard is just making it look like default database is being hit so I'm not sure what is causing that.

[HilbertSpecs]

For some reason when Querying database using DjangoORM: (i.e.) Model.objects.all()
I'm still getting the RuntimeError:

Database access not allowed, use the "django_db" mark, or the "db" or "transactional_db" fixtures to enable it.

Pytest is not able to find or connect to my test database (it's not properly configured) and the tests can not be collected before even running.

Although if I manually connect to test database using psycopg in code that is being tested using:

conn = psycopg.connect(os.environ.get("TEST_DATABASE_URL"))

All tests pass test database is found connected and properly used.

Say I have a .env file in project root directory with the following two Database Urls: .env

# both of these connection strings work when connecting with psycopg and running tests.
# local dbs (default/test) configured without password
DATABASE_URL=postgres://user:@host:port/dbname
TEST_DATABASE_URL=postgres://user:@host:port/test_dbname

Now in cookiecutter project: config/settings/base.py

DATABASES = {"default": env.db("DATABASE_URL")}

And in cookiecutter project: config/settings/test.py

DATABASES["default"]["TEST"] = {"NAME": "test_dbname"}

I want to configure the test database to be found using similar functionality as in base.py using DATABASE_URL I want it to look something like this in config/settings/test.py:

# did not work
DATABASES["default"]["TEST"] = env.db("TEST_DATABASE_URL")

So that the pytest can properly find the test database when running locally.
Again if I connect to test database manually using psycopg connection and query database using Raw SQL the database is found and tests pass but for some reason DjangoORM is not able to find the test database connection on its own.

I have also tried this in config/settings/test.py:

# did not work
# matches connection string parameters from earlier
DATABASES["default"]["TEST"] = {
    'NAME': 'test_dbname',
    'USER': 'postgres',
    'HOST': '127.0.0.1',
    'PORT': '5432',
}

Questions:
1. Does anyone know how to properly configure a test database locally like this using a TEST_DATABASE_URL?
2. Or what could be causing pytest-django to not be able to find test database when using DjangoORM?
3. Do I need to explicitly connect to test database when using DjangoORM?
(I thought it was supposed to connect to default database configured in base.py unless a test database was configured properly at which point when running pytest the default connection would be to the test database?)

Summary:
DjangoORM not finding test database when running tests in pytest-django without explicitly connecting to test database with psycopg conn string.
Psycopg connection string TEST_DATABASE_URL can be explictly connected to first using:
conn = psycopg.connect(os.environ.get("TEST_DATABASE_URL"))
and then successfully queried using RawSQL: (only after explicitly connecting with psycopg conn string)

I don't want to use RawSQL I want to use the DjangoORM am I missing something here?

All of my tests are properly marked with the pytest fixture: @pytest.mark.django_db(transaction=True)

The test database is running locally next to default database in Postgres app

Any advice helps thanks.

[HilbertSpecs]

To anyone that might have a similar issue:

# did work 
DATABASES["default"]["TEST"] = env.db("TEST_DATABASE_URL")

The problem was occurring when I was unit testing functions that were hitting the database.

# originally in my pytest test.py file I was importing the function like so:
from parent_directory.python_file import function_hitting_database

# accessing the function being tested in the pytest unit test like so:
function_hitting_database()

# When I changed the import statement in pytest file to this:
import parent_directory

# And then accessed the function being tested in the pytest unit test like so:
parent_directory.python_file.function_hitting_database()

The pytest collection error went away and all tests were able to collect and pass successfully.

Apparently when using DjangoORM to connect to the Database this pytest django_db mark safety feature works well it did not allow me to import functions hitting the database directly in pytest file I had to import the parent directory and access the function being tested using the '.' operator in the unit test otherwise the Runtime Error django_db mark was being thrown during collection.

But when connecting to Database using RawSQL in the function being tested the import of the function directly was allowed which I assume was trying to hit the default database at collection time as a consequence of the function being imported directly shortly after as the test suite ran properly it was hitting the test database during the actual unit test which was marked by django_db mark as it was supposed to be.

Initially using RawSQL it was able to bypass this pytest safety feature this could explain why my default database was showing transactions in PgAdmin every time I was running tests as well as the Test Database being hit during the actual test suite being run.

After switching to DjangoORM queries pytest disallowed this import of functions hitting the default database directly adjusting the imports statements fixed the error now pytest collection works and all tests pass successfully.