Misc: E2E test, migration, upgrade script, testcase setup UI, custom …

…problem-class (#94)

* fix: missing third-party js script

* chore: add redis-db and service port install option

* refactor: we don't need multi-threading to do cleanup work

* refactor: there should be no distinction between kernels and users
Follow TOJ Spec

* feat(UI): feat: add password eye
Copy From TNFSH-Scoreboard

* feat: add db and redis migration

* feat: add simple upgrade script

* feat: problem add allow submit option

* feat: user add custom motto

* perf: incremental refresh challenge_state
The challenge_state is used to cache all challenge results.
When refreshing the challenge_state, the database will recalculate all data about the challenges, but many challenges are unnecessarily updated.
So the incremental refresh only updates challenges that have changed.

* refactor: remove unused problem expire field

* refactor: move get problem state from `list_pro()` to `map_acct_rate()`

* feat: redirect to sign-in page when user is a guest

* test: add e2e test

* feat: prevent unauthorized user from updating password

* test: add user motto test

* test: add allow_submit option test

* fix: change progress bar text

* fix: pdf file is displayed as garbled text

* feat: add a hash to check file integrity during upload

* fix: escape characters when content contain code block

* fix: do not push empty url to history

* feat: add testcase setup ui
In this PR, we add the full file management UI. Now we can use the web UI to manage our test case files, attachments, and checkers. We no longer need to upload a problem package to overwrite files to achieve this goal.
We also introduce multiple language limit settings, allowing us to set specific languages. This is important for Python 3 or Java because they are usually slower than compiled languages like C or C++.
We will provide multiple test case file operations, which can reduce many single and duplicate operations for users.

* feat: add a simple log viewer for log params

* feat: new problem class system

In this PR, we added a customizable proclass for users, which they can
set to be publicly shared for others to use.
We improved the original proclass selection menu and added a new
interface for this purpose.
We also introduced a feature to collect proclass, allowing users to
gather their favorite proclass.

* fix: follow the target behavior when a link has a target attribute

* refactor: move `self.acct` to the default namespace to reduce argument passing

* perf: use batch inserts to reduce SQL execution

* fix: corrected wrong acct_id in a multi-threading environment

* perf: use SQL to calculate user rank instead of Python
In TOJ, response time decreased from 2000ms to about 300ms

* fix: only update the code hash when the code is actually submitted

* ci: bypass installation restrictions of the package manager

* ci: remove unnecessary zip compression, as actions/upload-artifact will handle this

* fix: module not found error

* refactor: make ruff and pyright happy

* Refactor: Improve migration logic and error handling in main function

* fix: correct wrong log message grammar

* chore: remove `f.close()`, as the with statement handles this

---------

Co-authored-by: lifeadventurer <[email protected]>

.github/workflows/tests.yml

@@ -0,0 +1,69 @@
+name: tests
+on:
+  push:
+    branches:
+      - '**'
+
+permissions:
+  contents: read
+
+jobs:
+  e2etest:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+
+      - name: Install Poetry
+        run: |
+          curl -sSL https://install.python-poetry.org | python3 -
+
+      - name: Install PostgreSQL, Redis, Dos2Unix
+        run: |
+          sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor | sudo tee /usr/share/keyrings/postgresql.gpg
+          echo deb [arch=amd64 signed-by=/usr/share/keyrings/postgresql.gpg] http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main | sudo tee /etc/apt/sources.list.d/postgresql.list
+          sudo apt update -y
+          sudo apt install -f -y postgresql-16 postgresql-client-16 redis dos2unix
+          sudo sed -i 's/peer/trust/' /etc/postgresql/16/main/pg_hba.conf
+          sudo service postgresql start
+          sudo service redis-server start
+
+      - name: Install Coverage
+        run: |
+          $HOME/.local/bin/poetry add --dev coverage
+
+      - name: Install Project dependencies
+        run: |
+          sed -i '/^mkdocs-material/d' pyproject.toml # test don't need mkdocs-material
+          rm poetry.lock # we need to remove it because we change the pyproject file
+          $HOME/.local/bin/poetry install
+          $HOME/.local/bin/poetry add beautifulsoup4
+          $HOME/.local/bin/poetry add requests
+
+      - name: Deploy NTOJ-Judge
+        run: |
+          cd $HOME
+          git clone https://github.com/tobiichi3227/NTOJ-Judge
+          cd NTOJ-Judge/src
+          sudo pip3 install tornado cffi --break-system-packages
+          chmod +x ./runserver.sh
+          sudo ./runserver.sh > output.log 2>&1 &
+
+      - name: Run e2e test
+        run: |
+          cd src
+          chmod +x runtests.sh
+          ./runtests.sh
+
+      - name: Output judge log
+        run: |
+          cd $HOME/NTOJ-Judge/src
+          cat output.log
+
+      - name: Upload Coverage Report
+        uses: actions/upload-artifact@v3
+        with:
+          name: coverage-report
+          path: src/htmlcov

migration/20240920182633_add_challenge_contest_id_index.py

@@ -0,0 +1,2 @@
+async def dochange(db_conn, rs_conn):
+    await db_conn.execute('CREATE INDEX challenge_idx_contest_id ON public.challenge USING btree (contest_id)')

migration/20240925015623_add_problem_allow_submit.py

@@ -0,0 +1,2 @@
+async def dochange(db, rs):
+    await db.execute('ALTER TABLE problem ADD allow_submit boolean DEFAULT true')

migration/20240925163422_user_add_motto.py

@@ -0,0 +1,14 @@
+async def dochange(db, rs):
+    await db.execute(
+        '''
+        ALTER TABLE account ADD motto character varying DEFAULT ''::character varying
+        '''
+    )
+
+    result = await db.fetch("SELECT last_value FROM account_acct_id_seq;")
+    cur_acct_id = int(result[0]["last_value"])
+
+    for acct_id in range(1, cur_acct_id + 1):
+        await rs.delete(f"account@{acct_id}")
+
+    await rs.delete("acctlist")

migration/20240925173122_add_incremental_challenge_state.py

@@ -0,0 +1,108 @@
+async def dochange(db, rs):
+    await db.execute('DROP MATERIALIZED VIEW challenge_state;')
+
+    await db.execute(
+    '''
+        CREATE TABLE challenge_state (
+            chal_id integer NOT NULL,
+            state integer,
+            runtime bigint DEFAULT 0,
+            memory bigint DEFAULT 0,
+            rate integer DEFAULT 0
+        );
+    ''')
+    await db.execute(
+    '''
+        ALTER TABLE ONLY public.challenge_state
+            ADD CONSTRAINT challenge_state_forkey_chal_id FOREIGN KEY (chal_id) REFERENCES public.challenge(chal_id) ON DELETE CASCADE;
+    ''')
+
+    await db.execute("ALTER TABLE challenge_state ADD CONSTRAINT challenge_state_unique_chal_id UNIQUE(chal_id);")
+
+
+    await db.execute(
+    '''
+        CREATE TABLE last_update_time (
+             view_name TEXT PRIMARY KEY,
+             last_update TIMESTAMP WITH TIME ZONE
+        );
+    '''
+    )
+
+    await db.execute("INSERT INTO last_update_time (view_name, last_update) VALUES ('challenge_state', NOW());")
+
+    await db.execute("ALTER TABLE test ADD COLUMN last_modified TIMESTAMP WITH TIME ZONE DEFAULT NOW();")
+
+    await db.execute("CREATE INDEX idx_test_last_modified ON test (last_modified);")
+    await db.execute("CREATE UNIQUE INDEX ON test_valid_rate (pro_id, test_idx);")
+
+    await db.execute(
+    '''
+        CREATE OR REPLACE FUNCTION update_test_last_modified()
+        RETURNS TRIGGER AS $$
+        BEGIN
+            NEW.last_modified = NOW();
+            RETURN NEW;
+        END;
+        $$ LANGUAGE plpgsql;
+    ''')
+
+    await db.execute(
+    '''
+        CREATE TRIGGER test_last_modified_trigger
+        BEFORE UPDATE ON test
+        FOR EACH ROW EXECUTE FUNCTION update_test_last_modified();
+    ''')
+
+    await db.execute(
+    '''
+        CREATE OR REPLACE FUNCTION refresh_challenge_state_incremental()
+        RETURNS VOID AS $$
+        DECLARE
+            last_update_time TIMESTAMP WITH TIME ZONE;
+        BEGIN
+            SELECT last_update INTO last_update_time
+            FROM last_update_time
+            WHERE view_name = 'challenge_state';
+
+            WITH challenge_summary AS (
+                SELECT
+                    t.chal_id,
+                    MAX(t.state) AS max_state,
+                    SUM(t.runtime) AS total_runtime,
+                    SUM(t.memory) AS total_memory,
+                    SUM(CASE WHEN t.state = 1 THEN tvr.rate ELSE 0 END) AS total_rate
+                FROM test t
+                LEFT JOIN test_valid_rate tvr ON t.pro_id = tvr.pro_id AND t.test_idx = tvr.test_idx
+                WHERE t.last_modified > last_update_time
+                GROUP BY t.chal_id
+            ),
+            upsert_result AS (
+                INSERT INTO challenge_state (chal_id, state, runtime, memory, rate)
+                SELECT
+                    chal_id,
+                    max_state,
+                    total_runtime,
+                    total_memory,
+                    total_rate
+                FROM challenge_summary
+                ON CONFLICT (chal_id) DO UPDATE
+                SET
+                    state = EXCLUDED.state,
+                    runtime = EXCLUDED.runtime,
+                    memory = EXCLUDED.memory,
+                    rate = EXCLUDED.rate
+                WHERE
+                    challenge_state.state != EXCLUDED.state OR
+                    challenge_state.runtime != EXCLUDED.runtime OR
+                    challenge_state.memory != EXCLUDED.memory OR
+                    challenge_state.rate != EXCLUDED.rate
+            )
+
+            UPDATE last_update_time
+            SET last_update = NOW()
+            WHERE view_name = 'challenge_state';
+        END;
+        $$ LANGUAGE plpgsql;
+    ''')
+    await db.execute('SELECT refresh_challenge_state_incremental();')

migration/20240926201600_remove_problem_expire_field.py

@@ -0,0 +1,19 @@
+async def dochange(db, rs):
+    await db.execute('DROP MATERIALIZED VIEW test_valid_rate;')
+    await db.execute('ALTER TABLE problem DROP COLUMN expire;')
+    await db.execute(
+        '''
+            CREATE MATERIALIZED VIEW public.test_valid_rate AS
+            SELECT test_config.pro_id,
+               test_config.test_idx,
+               count(DISTINCT account.acct_id) AS count,
+               test_config.weight AS rate
+              FROM (((public.test
+                JOIN public.account ON ((test.acct_id = account.acct_id)))
+                JOIN public.problem ON (((((test.pro_id = problem.pro_id)) AND (test.state = 1)))))
+                RIGHT JOIN public.test_config ON (((test.pro_id = test_config.pro_id) AND (test.test_idx = test_config.test_idx))))
+            GROUP BY test_config.pro_id, test_config.test_idx, test_config.weight
+            WITH NO DATA;
+        ''')
+    await db.execute('REFRESH MATERIALIZED VIEW test_valid_rate;')
+    await rs.delete('prolist')

migration/20240930220712_add_testcase.py

@@ -0,0 +1,12 @@
+import json
+
+async def dochange(db, rs):
+    test_configs = await db.fetch('SELECT pro_id, test_idx, metadata FROM test_config;')
+
+    for pro_id, test_group_idx, metadata in test_configs:
+        metadata = json.loads(metadata)
+        for i in range(len(metadata["data"])):
+            metadata["data"][i] = str(metadata["data"][i])
+
+        await db.execute('UPDATE test_config SET metadata = $1 WHERE pro_id = $2 AND test_idx = $3',
+                   json.dumps(metadata), pro_id, test_group_idx)

migration/20241001153200_move_test_fields_to_problem.py

@@ -0,0 +1,49 @@
+import json
+
+async def dochange(db, rs):
+    CHECK_TYPES = {
+        "diff": 0,
+        "diff-strict": 1,
+        "diff-float": 2,
+        "ioredir": 3,
+        "cms": 4
+    }
+
+    await db.execute("ALTER TABLE problem ADD check_type integer DEFAULT 0")
+    await db.execute("ALTER TABLE problem ADD is_makefile boolean DEFAULT false")
+    await db.execute("""
+        ALTER TABLE problem ADD "limit" jsonb DEFAULT '{"default": {"timelimit": 0, "memlimit":0}}'::jsonb
+    """)
+    await db.execute("ALTER TABLE problem ADD chalmeta jsonb DEFAULT '{}'::jsonb")
+
+    res = await db.fetch("SELECT pro_id FROM problem;")
+    for pro in res:
+        pro_id = pro['pro_id']
+        limit = {
+            'default': {
+                'timelimit': 0,
+                'memlimit': 0,
+            }
+        }
+        f_check_type = 0
+        f_is_makefile = False
+        f_chalmeta = {}
+
+        res = await db.fetch('SELECT check_type, compile_type, chalmeta, timelimit, memlimit FROM test_config WHERE pro_id = $1', pro_id)
+        for check_type, compile_type, chalmeta, timelimit, memlimit in res:
+            f_check_type = CHECK_TYPES[check_type]
+            f_is_makefile = compile_type == 'makefile'
+            f_chalmeta = json.loads(chalmeta)
+            limit['default']['timelimit'] = timelimit
+            limit['default']['memlimit'] = memlimit
+
+        await db.execute("UPDATE problem SET check_type = $1, is_makefile = $2, \"limit\" = $3, chalmeta = $4 WHERE pro_id = $5",
+                         f_check_type, f_is_makefile, json.dumps(limit), json.dumps(f_chalmeta), pro_id)
+
+
+    await db.execute('ALTER TABLE test_config DROP COLUMN check_type;')
+    await db.execute('ALTER TABLE test_config DROP COLUMN score_type;')
+    await db.execute('ALTER TABLE test_config DROP COLUMN compile_type;')
+    await db.execute('ALTER TABLE test_config DROP COLUMN chalmeta;')
+    await db.execute('ALTER TABLE test_config DROP COLUMN timelimit;')
+    await db.execute('ALTER TABLE test_config DROP COLUMN memlimit;')

migration/20241005212800_add_trigger_for_test_deleted.py

@@ -0,0 +1,19 @@
+async def dochange(db, rs):
+    await db.execute(
+    '''
+        CREATE OR REPLACE FUNCTION delete_challenge_state()
+        RETURNS TRIGGER AS $$
+        BEGIN
+            DELETE FROM challenge_state WHERE chal_id = OLD.chal_id;
+            RETURN OLD;
+        END;
+        $$ LANGUAGE plpgsql;
+    ''')
+
+    await db.execute(
+    '''
+        CREATE TRIGGER trigger_delete_challenge_state
+        AFTER DELETE ON test
+        FOR EACH ROW
+        EXECUTE FUNCTION delete_challenge_state();
+    ''')

migration/20241006130410_change_log_param_to_jsonb.py

@@ -0,0 +1,7 @@
+async def dochange(db, rs):
+    await db.execute(
+        "ALTER TABLE log ALTER COLUMN params TYPE jsonb USING params::jsonb"
+    )
+    await db.execute(
+        "ALTER TABLE log ALTER COLUMN params SET DEFAULT '{}'::jsonb"
+    )

migration/20241006151800_problem_class.py

@@ -0,0 +1,22 @@
+class ProClassConst:
+    OFFICIAL_PUBLIC = 0
+    OFFICIAL_HIDDEN = 1
+    USER_PUBLIC = 2
+    USER_HIDDEN = 3
+
+
+async def dochange(db, rs):
+    # NOTE: rename
+    await db.execute("ALTER TABLE pubclass RENAME TO proclass")
+    await db.execute("ALTER SEQUENCE pubclass_pubclass_id_seq RENAME TO proclass_proclass_id_seq")
+    await db.execute("ALTER TABLE proclass RENAME COLUMN pubclass_id TO proclass_id")
+    await db.execute("ALTER TABLE proclass RENAME CONSTRAINT pubclass_pkey TO proclass_pkey")
+
+    await db.execute('''ALTER TABLE proclass ADD "desc" text DEFAULT \'\'''')
+    await db.execute("ALTER TABLE proclass ADD acct_id integer")
+    await db.execute('ALTER TABLE proclass ADD "type" integer')
+    await db.execute(
+        "ALTER TABLE proclass ADD CONSTRAINT proclass_forkey_acct_id FOREIGN KEY (acct_id) REFERENCES account(acct_id) ON DELETE CASCADE"
+    )
+    await db.execute('UPDATE proclass SET "type" = $1', ProClassConst.OFFICIAL_PUBLIC)
+    await db.execute('ALTER TABLE proclass ALTER COLUMN "type" SET NOT NULL')

migration/20241007235900_add_user_proclass_collection.py

@@ -0,0 +1,9 @@
+async def dochange(db, rs):
+    await db.execute("ALTER TABLE account ADD proclass_collection integer[] NOT NULL DEFAULT '{}'::integer[]")
+    result = await db.fetch("SELECT last_value FROM account_acct_id_seq;")
+    cur_acct_id = int(result[0]['last_value'])
+
+    for acct_id in range(1, cur_acct_id + 1):
+        await rs.delete(f"account@{acct_id}")
+
+    await rs.delete('acctlist')