Merge pull request #695 from BC-SECURITY/release/5.6.3

v5.6.3 into main

.github/CONTRIBUTING.md

@@ -45,11 +45,11 @@ The `main` branch in `BC-SECURITY/Empire` automatically syncs.
 
 ### Code Formatting and Linting
 
-* As of Empire 4.4, we are using [psf/black](https://github.com/psf/black) for code formatting.
+* We are using [psf/black](https://github.com/psf/black) for code formatting.
     * Black is a Python code formatter that helps to keep the codebase uniform and easy to read
-* As of Empire 4.4, we are using [PyCQA/isort](https://github.com/PyCQA/isort)
+* We are using [PyCQA/isort](https://github.com/PyCQA/isort)
     * Isort is a Python utility that sorts and formats imports.
-* As of Empire 5.0.1, we are using [charliermarsh/ruff](https://github.com/charliermarsh/ruff) for linting.
+* We are using [charliermarsh/ruff](https://github.com/charliermarsh/ruff) for linting.
     * Ruff is a python linter that helps identify common bugs and style issues.
 * After implementing your changes:
     1. run `black .` (or `poetry run black .`).
@@ -66,5 +66,4 @@ Please write tests for your code! We use [pytest](https://docs.pytest.org/en/lat
 For tests that take >20-30 seconds, please add the `@pytest.mark.slow` decorator to the test function. This will allow us to skip the slow tests when running the tests, unless we explicitly want to run them with `pytest --runslow`.
 
 ## Upgrading dependencies
-
 Dependencies can be upgraded using [poetry-plugin-up](https://github.com/MousaZeidBaker/poetry-plugin-up).

.github/workflows/lint-and-test.yml

@@ -2,8 +2,6 @@ name: Lint and Test
 
 on:
   pull_request:
-    paths-ignore:
-      - '**.md'
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
@@ -14,11 +12,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
-      - uses: psf/black@23.1.0
+      - uses: psf/black@23.7.0
       - uses: isort/isort-action@master
+        with:
+          isort-version: 5.12.0
       - name: Run ruff
         run: |
-          pip install ruff
+          pip install ruff==0.0.283
           ruff .
   matrix-prep-config:
     runs-on: ubuntu-latest

.gitmodules

@@ -49,3 +49,6 @@
 [submodule "empire/server/csharp/Covenant/Data/ReferenceSourceLibraries/RunOF"]
 	path = empire/server/csharp/Covenant/Data/ReferenceSourceLibraries/RunOF
 	url = https://github.com/BC-SECURITY/RunOF.git
+[submodule "empire/server/plugins/Report-Generation-Plugin"]
+	path = empire/server/plugins/Report-Generation-Plugin
+	url = https://github.com/BC-SECURITY/Report-Generation-Plugin.git

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
 -   repo: https://github.com/psf/black
-    rev: 23.1.0
+    rev: 23.7.0
     hooks:
     - id: black
       language_version: python3.9
@@ -11,7 +11,8 @@ repos:
       - id: isort
         name: isort (python)
 
-- repo: https://github.com/charliermarsh/ruff-pre-commit
-  rev: 'v0.0.236'
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  rev: v0.0.283
   hooks:
     - id: ruff
+      args: [--fix, --exit-non-zero-on-fix]

CHANGELOG.md

@@ -7,7 +7,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
--  Update the github issue templates to use forms (@Vinnybod)
+## [5.6.3] - 2023-08-27
+
+-   Updated Starkiller to v2.5.3
+-   Added Advanced Reporting Plugin and dependencies (@Cx01N)
+-   Pin linters in the workflow
+-   Catch error when starting up database that was seeded by an older version of Empire (@Vinnybod)
+-   Updated Windows BAT launcher to use Base64 for all payloads (@Cx01N)
+
+## [5.6.2] - 2023-08-09
+
+-   Update the github issue templates to use forms (@Vinnybod)
+-   Fix issue with option validator throwing error for strict non-required options (@Vinnybod)
+-   Allow Starkiller to load even if the git pull fails if the dir exists (@Vinnybod)
+-   Update listener descriptions to not specify languages since Empire supports more languages now
+
+## [5.6.1] - 2023-08-02
+
+## [5.6.0] - 2023-07-25
+
+-   Upgrade dependencies
+-   Upgrade Dockerfile to bullseye and 3.11.4
+-   Allow download_service to accept a pathlib.Path object to create a download (@Vinnybod)
+-   Fix file option for listeners, stagers, plugins (@Vinnybod)
+-   Add tags to Listeners, Agents, Agent Tasks, Plugin Tasks, Credentials, and Downloads (@Vinnybod)
+    -   Add endpoints to add, edit, and delete tags for each resource type
+    -   Add tag list endpoint
+    -   Add tag filters to Agent Tasks, Plugin Tasks, and Downloads
+    -   Add events for new and updated tags
+-   Fix user filters for tasks to include tasks without any users (@Vinnybod)
+-   Refactor stager and listener tests to work better in parallel (@Vinnybod)
+-   Add a Invoke-PhishingLNK Module (@0xFFaraday)
+-   Fix changelog link in README (@theguly)
 
 ## [5.5.4] - 2023-07-20
 -   Updated Starkiller to v2.4.3
@@ -546,7 +577,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 -   Updated shellcoderdi to newest version (@Cx01N)
 -   Added a Nim launcher (@Hubbl3)
 
-[Unreleased]: https://github.com/BC-SECURITY/Empire-Sponsors/compare/v5.5.4...HEAD
+[Unreleased]: https://github.com/BC-SECURITY/Empire-Sponsors/compare/v5.6.3...HEAD
+
+[5.6.3]: https://github.com/BC-SECURITY/Empire-Sponsors/compare/v5.6.2...v5.6.3
+
+[5.6.2]: https://github.com/BC-SECURITY/Empire-Sponsors/compare/v5.6.1...v5.6.2
+
+[5.6.1]: https://github.com/BC-SECURITY/Empire-Sponsors/compare/v5.6.0...v5.6.1
+
+[5.6.0]: https://github.com/BC-SECURITY/Empire-Sponsors/compare/v5.5.4...v5.6.0
 
 [5.5.4]: https://github.com/BC-SECURITY/Empire-Sponsors/compare/v5.5.3...v5.5.4
 

Dockerfile

@@ -13,7 +13,7 @@
 # -----BUILD ENTRY-----
 
 # image base
-FROM python:3.11.3-buster
+FROM python:3.11.4-bullseye
 
 # extra metadata
 LABEL maintainer="bc-security"

empire/server/api/app.py

@@ -3,6 +3,7 @@
 import os
 from datetime import datetime
 from json import JSONEncoder
+from pathlib import Path
 
 import socketio
 import uvicorn
@@ -42,14 +43,27 @@ def default(self, o):
         return JSONEncoder.default(self, o)
 
 
-def load_starkiller(v2App):
-    sync_starkiller(empire_config.dict())
+def load_starkiller(v2App, ip, port):
+    try:
+        sync_starkiller(empire_config.dict())
+    except Exception as e:
+        log.warning("Failed to load Starkiller: %s", e, exc_info=True)
+        log.warning(
+            "If you are trying to pull Starkiller from a private repository ("
+            "such as Starkiller-Sponsors), make sure you have the proper ssh "
+            "credentials set in your Empire config. See "
+            "https://docs.github.com/en/github/authenticating-to-github"
+            "/connecting-to-github-with-ssh"
+        )
 
-    v2App.mount(
-        "/",
-        StaticFiles(directory=f"{empire_config.starkiller.directory}/dist"),
-        name="static",
-    )
+    if (Path(empire_config.starkiller.directory) / "dist").exists():
+        v2App.mount(
+            "/",
+            StaticFiles(directory=f"{empire_config.starkiller.directory}/dist"),
+            name="static",
+        )
+        log.info("Starkiller served at the same ip and port as Empire Server")
+        log.info(f"Starkiller served at http://localhost:{port}/index.html")
 
 
 def initialize(secure: bool = False, ip: str = "0.0.0.0", port: int = 1337):
@@ -66,6 +80,7 @@ def initialize(secure: bool = False, ip: str = "0.0.0.0", port: int = 1337):
     from empire.server.api.v2.plugin import plugin_api, plugin_task_api
     from empire.server.api.v2.profile import profile_api
     from empire.server.api.v2.stager import stager_api, stager_template_api
+    from empire.server.api.v2.tag import tag_api
     from empire.server.api.v2.user import user_api
     from empire.server.server import main
 
@@ -97,6 +112,7 @@ def shutdown_event():
     v2App.include_router(meta_api.router)
     v2App.include_router(plugin_task_api.router)
     v2App.include_router(plugin_api.router)
+    v2App.include_router(tag_api.router)
 
     v2App.add_middleware(
         EmpireCORSMiddleware,
@@ -131,18 +147,7 @@ def shutdown_event():
 
     setup_socket_events(sio, main)
 
-    try:
-        load_starkiller(v2App)
-        log.info(f"Starkiller served at http://{ip}:{port}/index.html")
-    except Exception as e:
-        log.warning("Failed to load Starkiller: %s", e, exc_info=True)
-        log.warning(
-            "If you are trying to pull Starkiller from a private repository ("
-            "such as Starkiller-Sponsors), make sure you have the proper ssh "
-            "credentials set in your Empire config. See "
-            "https://docs.github.com/en/github/authenticating-to-github"
-            "/connecting-to-github-with-ssh"
-        )
+    load_starkiller(v2App, ip, port)
 
     cert_path = os.path.abspath("./empire/server/data/")
 

empire/server/api/v2/agent/agent_api.py

@@ -24,6 +24,7 @@
     NotFoundResponse,
     OrderDirection,
 )
+from empire.server.api.v2.tag import tag_api
 from empire.server.core.config import empire_config
 from empire.server.core.db import models
 from empire.server.server import main
@@ -50,6 +51,9 @@ async def get_agent(uid: str, db: Session = Depends(get_db)):
     raise HTTPException(404, f"Agent not found for id {uid}")
 
 
+tag_api.add_endpoints_to_taggable(router, "/{uid}/tags", get_agent)
+
+
 @router.get("/checkins", response_model=AgentCheckIns)
 def read_agent_checkins_all(
     db: Session = Depends(get_db),

empire/server/api/v2/agent/agent_dto.py

@@ -5,6 +5,7 @@
 from pydantic import BaseModel
 
 from empire.server.api.v2.shared_dto import PROXY_ID
+from empire.server.api.v2.tag.tag_dto import Tag, domain_to_dto_tag
 from empire.server.core.db import models
 
 
@@ -48,6 +49,7 @@ def domain_to_dto_agent(agent: models.Agent):
         archived=agent.archived,
         # Could make this a typed class later to match the schema
         proxies=to_proxy_dto(agent.proxies),
+        tags=list(map(lambda x: domain_to_dto_tag(x), agent.tags)),
     )
 
 
@@ -111,6 +113,7 @@ class Agent(BaseModel):
     archived: bool
     stale: bool
     proxies: Optional[Dict]
+    tags: List[Tag]
 
 
 class Agents(BaseModel):

empire/server/api/v2/agent/agent_task_api.py

@@ -38,6 +38,8 @@
     NotFoundResponse,
     OrderDirection,
 )
+from empire.server.api.v2.tag import tag_api
+from empire.server.api.v2.tag.tag_dto import TagStr
 from empire.server.core.agent_service import AgentService
 from empire.server.core.agent_task_service import AgentTaskService
 from empire.server.core.db import models
@@ -83,6 +85,9 @@ async def get_task(
     )
 
 
+tag_api.add_endpoints_to_taggable(router, "/{agent_id}/tasks/{uid}/tags", get_task)
+
+
 @router.get("/tasks", response_model=AgentTasks)
 async def read_tasks_all_agents(
     limit: int = -1,
@@ -96,13 +101,15 @@ async def read_tasks_all_agents(
     status: Optional[AgentTaskStatus] = None,
     agents: Optional[List[str]] = Query(None),
     users: Optional[List[int]] = Query(None),
+    tags: Optional[List[TagStr]] = Query(None),
     query: Optional[str] = None,
     db: Session = Depends(get_db),
 ):
     tasks, total = agent_task_service.get_tasks(
         db,
         agents=agents,
         users=users,
+        tags=tags,
         limit=limit,
         offset=(page - 1) * limit,
         include_full_input=include_full_input,
@@ -145,6 +152,7 @@ async def read_tasks(
     order_direction: OrderDirection = OrderDirection.desc,
     status: Optional[AgentTaskStatus] = None,
     users: Optional[List[int]] = Query(None),
+    tags: Optional[List[TagStr]] = Query(None),
     db: Session = Depends(get_db),
     db_agent: models.Agent = Depends(get_agent),
     query: Optional[str] = None,
@@ -153,6 +161,7 @@ async def read_tasks(
         db,
         agents=[db_agent.session_id],
         users=users,
+        tags=tags,
         limit=limit,
         offset=(page - 1) * limit,
         include_full_input=include_full_input,

empire/server/api/v2/agent/agent_task_dto.py

@@ -8,6 +8,7 @@
     DownloadDescription,
     domain_to_dto_download_description,
 )
+from empire.server.api.v2.tag.tag_dto import Tag, domain_to_dto_tag
 from empire.server.core.db import models
 
 
@@ -41,6 +42,7 @@ def domain_to_dto_task(
         status=task.status,
         created_at=task.created_at,
         updated_at=task.updated_at,
+        tags=list(map(lambda x: domain_to_dto_tag(x), task.tags)),
     )
 
 
@@ -59,6 +61,7 @@ class AgentTask(BaseModel):
     status: models.AgentTaskStatus
     created_at: datetime
     updated_at: datetime
+    tags: List[Tag]
 
 
 class AgentTasks(BaseModel):

empire/server/api/v2/credential/credential_api.py

@@ -16,6 +16,7 @@
 )
 from empire.server.api.v2.shared_dependencies import get_db
 from empire.server.api.v2.shared_dto import BadRequestResponse, NotFoundResponse
+from empire.server.api.v2.tag import tag_api
 from empire.server.core.db import models
 from empire.server.server import main
 
@@ -41,6 +42,9 @@ async def get_credential(uid: int, db: Session = Depends(get_db)):
     raise HTTPException(404, f"Credential not found for id {uid}")
 
 
+tag_api.add_endpoints_to_taggable(router, "/{uid}/tags", get_credential)
+
+
 @router.get("/{uid}", response_model=Credential)
 async def read_credential(
     uid: int, db_credential: models.Credential = Depends(get_credential)

empire/server/api/v2/credential/credential_dto.py

@@ -3,6 +3,8 @@
 
 from pydantic import BaseModel
 
+from empire.server.api.v2.tag.tag_dto import Tag, domain_to_dto_tag
+
 
 def domain_to_dto_credential(credential):
     return Credential(
@@ -17,6 +19,7 @@ def domain_to_dto_credential(credential):
         notes=credential.notes,
         created_at=credential.created_at,
         updated_at=credential.updated_at,
+        tags=list(map(lambda x: domain_to_dto_tag(x), credential.tags)),
     )
 
 
@@ -32,6 +35,7 @@ class Credential(BaseModel):
     notes: Optional[str]
     created_at: datetime
     updated_at: datetime
+    tags: List[Tag]
 
 
 class Credentials(BaseModel):