Added Suricata Docker deployment, enabled flow marking, some usability and documentation improvements

.env.example

@@ -13,5 +13,11 @@ TICK_START="2018-06-27T13:00+02:00"
 TICK_LENGTH=180000
 
 #PCAP_OVER_IP="host.docker.internal:1337"
-#For multiple PCAP_OVER_IP you can comma separate
-#PCAP_OVER_IP="host.docker.internal:1337,otherhost.com:5050"
+# For multiple PCAP_OVER_IP you can comma separate
+#PCAP_OVER_IP="host.docker.internal:1337,otherhost.com:5050"
+# Set BPF filter expression (see https://www.tcpdump.org/manpages/pcap-filter.7.html)
+#BPF="port 8080"
+
+# Directory for Suricata files (see suricata/etc, suricata/lib/rules, suricata/logs)
+# (they should be generated on first run)
+#SURICATA_DIR_HOST="./suricata"

.gitignore

@@ -132,3 +132,4 @@ workspace.xml
 .idea
 
 /traffic
+suricata

README.md

@@ -13,15 +13,20 @@ Tulip was developed by Team Europe for use in the first International Cyber Secu
 * Synchronized with Suricata.
 * Flow diffing
 * Time and size-based plots for correlation.
-* Linking HTTP sessions together based on cookies (Experimental, disabled by default)
+* Linking HTTP sessions together based on cookies (Experimental*, disabled by default)
+* PCAP-over-IP with BPF filtering support**
+
+\* - to enable, add `-experimental` after `./assembler` in `docker-compose.yml`
+
+\*\* - to enable, configure PCAP-over-IP server (e.g. [pcap-broker](https://github.com/fox-it/pcap-broker) as suggested in [PR 24](https://github.com/OpenAttackDefenseTools/tulip/pull/24)) and set `PCAP_OVER_IP` (and `BPF` if necessary) in `.env`
 
 ## Screenshots
 ![](./demo_images/demo1.png)
 ![](./demo_images/demo2.png)
 ![](./demo_images/demo3.png)
 
 ## Configuration
-Before starting the stack, edit `services/configurations.py`:
+Before starting the stack, edit `services/api/configurations.py`:
 
 ```
 vm_ip = "10.60.4.1"
@@ -45,21 +50,54 @@ docker-compose up -d --build
 ```
 To ingest traffic, it is recommended to create a shared bind mount with the docker-compose. One convenient way to set this up is as follows:
 1. On the vulnbox, start a rotating packet sniffer (e.g. tcpdump, suricata, ...)
-1. Using rsync, copy complete captures to the machine running tulip (e.g. to /traffic)
-1. Add a bind to the assembler service so it can read /traffic
+```bash
+tcpdump -i eth0 -G 180 -w "traffic_%H:%M:%S.pcap" port 8080
+```
+2. Using rsync, copy complete captures to the machine running tulip (e.g. to /traffic)
+```bash
+rsync -avz -e ssh --progress [email protected]:/pcaps ./pcaps
+```
+3. Add a bind to the assembler service so it can read /traffic
+   > (Just change `TRAFFIC_DIR_HOST` in `.env`)
 
 The ingestor will use inotify to watch for new pcap's and suricata logs. No need to set a chron job.
 
 
 ## Suricata synchronization
 
+### Run in Docker
+
+Configure `SURICATA_DIR_HOST` in `.env`.
+
+Create some rules (404 for testing):
+```bash
+. .env
+mkdir -p ${SURICATA_DIR_HOST}/{etc,lib/rules,log}
+echo 'alert tcp any any -> any any (msg: "404 Not Found"; http.stat_code; content:"404"; metadata: tag notfound; sid:4; rev: 1;)' >> ${SURICATA_DIR_HOST}/lib/rules/suricata.rules
+```
+
+After that run (default config for `eve.json` logging was good enough):
+
+```bash
+docker compose -f docker-compose-suricata.yml up -d --build
+```
+
 ### Metadata
 Tags are read from the metadata field of a rule. For example, here's a simple rule to detect a path traversal:
 ```
 alert tcp any any -> any any (msg: "Path Traversal-../"; flow:to_server; content: "../"; metadata: tag path_traversal; sid:1; rev: 1;)
 ```
 Once this rule is seen in traffic, the `path_traversal` tag will automatically be added to the filters in Tulip.
 
+> [!NOTE]
+>
+> After editing Suricata rules (renaming or id change) please:
+>
+> Remove old logs: `rm ${SURICATA_DIR_HOST}/log/*` (otherwise old signatures will be repopulated).
+>
+> Restart Docker containers.
+>
+> If database was only restarted (not dropped), try cleaning tags/signatures with `python wipe_tags.py`.
 
 ### eve.json
 Suricata alerts are read directly from the `eve.json` file. Because this file can get quite verbose when all extensions are enabled, it is recommended to strip the config down a fair bit. For example:
@@ -88,7 +126,7 @@ Your Tulip instance will probably contain sensitive CTF information, like flags
 
 # Contributing
 If you have an idea for a new feature, bug fixes, UX improvements, or other contributions, feel free to open a pull request or create an issue!      
-When opening a pull request, please target the `devel` branch.
+When opening a pull request, please target the latest development branch.
 
 # Credits
 Tulip was written by [@RickdeJager](https://github.com/rickdejager) and [@Bazumo](https://github.com/bazumo), with additional help from [@Sijisu](https://github.com/sijisu). Thanks to our fellow Team Europe players and coaches for testing, feedback and suggestions. Finally, thanks to the team behind [flower](https://github.com/secgroup/flower) for opensourcing their tooling.

docker-compose-suricata.yml

@@ -0,0 +1,96 @@
+version: "3.2"
+services:
+  mongo:
+    image: mongo:5
+    networks:
+      - internal
+    restart: always
+    ports:
+      - "127.0.0.1:27017:27017"
+
+  frontend:
+    build:
+      context: frontend
+      dockerfile: Dockerfile-frontend
+    image: tulip-frontend:latest
+    restart: unless-stopped
+    ports:
+      - "3000:3000"
+    depends_on:
+      - mongo
+    networks:
+      - internal
+    environment:
+      API_SERVER_ENDPOINT: http://api:5000/
+
+  api:
+    build:
+      context: services/api
+      dockerfile: Dockerfile-api
+    image: tulip-api:latest
+    restart: unless-stopped
+    depends_on:
+      - mongo
+    networks:
+      - internal
+    volumes:
+      - ${TRAFFIC_DIR_HOST}:${TRAFFIC_DIR_DOCKER}:ro
+    environment:
+      TULIP_MONGO: ${TULIP_MONGO}
+      TULIP_TRAFFIC_DIR: ${TRAFFIC_DIR_DOCKER}
+      FLAG_REGEX: ${FLAG_REGEX}
+      TICK_START: ${TICK_START}
+      TICK_LENGTH: ${TICK_LENGTH}
+
+  assembler:
+    build:
+      context: services/go-importer
+      dockerfile: Dockerfile-assembler
+    image: tulip-assembler:latest
+    restart: unless-stopped
+    depends_on:
+      - mongo
+    networks:
+      - internal
+    volumes:
+      - ${TRAFFIC_DIR_HOST}:${TRAFFIC_DIR_DOCKER}:ro
+    command: "./assembler -dir ${TRAFFIC_DIR_DOCKER}"
+    environment:
+      TULIP_MONGO: ${TULIP_MONGO}
+      FLAG_REGEX: ${FLAG_REGEX}
+      PCAP_OVER_IP: ${PCAP_OVER_IP}
+      BPF: ${BPF}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  enricher:
+    build:
+      context: services/go-importer
+      dockerfile: Dockerfile-enricher
+    image: tulip-enricher:latest
+    restart: unless-stopped
+    depends_on:
+      - mongo
+      - suricata
+    networks:
+      - internal
+    volumes:
+      - ${SURICATA_DIR_HOST}/log:/suricata
+      - ${TRAFFIC_DIR_HOST}:${TRAFFIC_DIR_DOCKER}:ro
+    command: "./enricher -eve /suricata/eve.json"
+    environment:
+      TULIP_MONGO: ${TULIP_MONGO}
+
+  suricata:
+    image: jasonish/suricata:7.0
+    restart: unless-stopped
+    volumes:
+      - ${TRAFFIC_DIR_HOST}:${TRAFFIC_DIR_DOCKER}:ro
+      - ${SURICATA_DIR_HOST}/log:/var/log/suricata
+      - ${SURICATA_DIR_HOST}/etc:/etc/suricata
+      - ${SURICATA_DIR_HOST}/lib:/var/lib/suricata
+    environment:
+      SURICATA_OPTIONS: "-l /var/log/suricata -v -r ${TRAFFIC_DIR_DOCKER} --pcap-file-continuous"
+
+networks:
+  internal:

docker-compose.yml

@@ -36,7 +36,7 @@ services:
     volumes:
       - ${TRAFFIC_DIR_HOST}:${TRAFFIC_DIR_DOCKER}:ro
     environment:
-      TULIP_MONGO: mongo:27017
+      TULIP_MONGO: ${TULIP_MONGO}
       TULIP_TRAFFIC_DIR: ${TRAFFIC_DIR_DOCKER}
       FLAG_REGEX: ${FLAG_REGEX}
       TICK_START: ${TICK_START}
@@ -59,10 +59,10 @@ services:
       TULIP_MONGO: ${TULIP_MONGO}
       FLAG_REGEX: ${FLAG_REGEX}
       PCAP_OVER_IP: ${PCAP_OVER_IP}
+      BPF: ${BPF}
     extra_hosts:
       - "host.docker.internal:host-gateway"
 
-
   enricher:
     build:
       context: services/go-importer

frontend/src/api.ts

@@ -77,28 +77,28 @@ export const tulipApi = createApi({
       query: ({ id, star }) => `/star/${id}/${star ? "1" : "0"}`,
       // TODO: optimistic cache update
 
-      // async onQueryStarted({ id, star }, { dispatch, queryFulfilled }) {
-      //   // `updateQueryData` requires the endpoint name and cache key arguments,
-      //   // so it knows which piece of cache state to update
-      //   const patchResult = dispatch(
-      //     tulipApi.util.updateQueryData("getFlows", undefined, (flows) => {
-      //       // The `flows` is Immer-wrapped and can be "mutated" like in createSlice
-      //       const flow = flows.find((flow) => flow._id.$oid === id);
-      //       if (flow) {
-      //         if (star) {
-      //           flow.tags.push("starred");
-      //         } else {
-      //           flow.tags = flow.tags.filter((tag) => tag != "starred");
-      //         }
-      //       }
-      //     })
-      //   );
-      //   try {
-      //     await queryFulfilled;
-      //   } catch {
-      //     patchResult.undo();
-      //   }
-      // },
+      async onQueryStarted({ id, star }, { dispatch, queryFulfilled }) {
+        // `updateQueryData` requires the endpoint name and cache key arguments,
+        // so it knows which piece of cache state to update
+        const patchResult = dispatch(
+          tulipApi.util.updateQueryData("getFlows", {service: "undefined", includeTags: [], excludeTags:[]}, (flows) => {
+            // The `flows` is Immer-wrapped and can be "mutated" like in createSlice
+            const flow = flows.find((flow) => flow._id.$oid === id);
+            if (flow) {
+              if (star) {
+                flow.tags.push("starred");
+              } else {
+                flow.tags = flow.tags.filter((tag) => tag != "starred");
+              }
+            }
+          })
+        );
+        try {
+          await queryFulfilled;
+        } catch {
+          patchResult.undo();
+        }
+      },
     }),
   }),
 });

frontend/src/components/Corrie.tsx

@@ -114,6 +114,7 @@ export const Corrie = () => {
           >
             volume
           </button>
+          <p className="text-left px-2 py-2">After clicking on a flow, press 'w' to scroll to it in flow list</p>
         </div>
       </div>
       <div className="flex-1 w-full overflow-hidden p-4">

frontend/src/components/FlowList.tsx

@@ -13,6 +13,7 @@ import {
   START_FILTER_KEY,
   END_FILTER_KEY,
   FLOW_LIST_REFETCH_INTERVAL_MS,
+  FORCE_REFETCH_ON_STAR,
 } from "../const";
 import { useAppSelector, useAppDispatch } from "../store";
 import { toggleFilterTag } from "../store/filter";
@@ -91,6 +92,7 @@ export function FlowList() {
 
   const onHeartHandler = async (flow: Flow) => {
     await starFlow({ id: flow._id.$oid, star: !flow.tags.includes("starred") });
+    if(FORCE_REFETCH_ON_STAR) refetch();
   };
 
   const navigate = useNavigate();
@@ -137,7 +139,30 @@ export function FlowList() {
     [transformedFlowData]
   )
 
+  useHotkeys('x', async () => {
+    if(transformedFlowData) {
+      let flow = transformedFlowData[flowIndex ?? 0]
+      await onHeartHandler(flow);
+    }
+  })
+
   useHotkeys('j', () => setFlowIndex(fi => Math.min((transformedFlowData?.length ?? 1)-1, fi + 1)), [transformedFlowData?.length]);
+  useHotkeys('w', () => {
+    if(transformedFlowData) {
+      let idAtIndex = transformedFlowData[flowIndex ?? 0]._id.$oid;
+      if (idAtIndex != openedFlowID) {
+        let flowids = flowData?.map((flow, idx) => ([flow._id.$oid, idx]))
+        if (flowids) {
+          let found = flowids.filter((el)=>(el[0] == openedFlowID))
+          if (found.length > 0) {
+            let n = Number(found[0][1])
+            setFlowIndex(n)
+          }
+        }
+      }
+    }
+  }
+  );
   useHotkeys('k', () => setFlowIndex(fi => Math.max(0, fi - 1)));
   useHotkeys('i', () => {
     setShowFilters(true)
@@ -151,6 +176,12 @@ export function FlowList() {
       dispatch(toggleFilterTag("flag-out"))
     }
   }, [availableTags]);
+  useHotkeys('t', () => {
+    setShowFilters(true)
+    if ((availableTags ?? []).includes("starred")) {
+      dispatch(toggleFilterTag("starred"))
+    }
+  }, [availableTags]);
   useHotkeys('r', () => refetch());
 
   const [showFilters, setShowFilters] = useState(false);

frontend/src/components/Header.tsx

@@ -269,7 +269,7 @@ function SecondDiff() {
     setSearchParams(searchParams);
   }
 
-  useHotkeys("g", () => {
+  useHotkeys("e", () => {
     setSecondDiffFlow();
   });
 
@@ -322,6 +322,9 @@ export function Header() {
   let [searchParams] = useSearchParams();
   const { setToLastnTicks, currentTick, setTimeParam } = useMessyTimeStuff();
 
+  let navigate = useNavigate();
+
+  useHotkeys('g', () => navigate(`/corrie?${searchParams}`, { replace: true }))
   useHotkeys('a', () => setToLastnTicks(5));
   useHotkeys('c', () => {
     (document.getElementById("startdateselection") as HTMLInputElement).value = "";

frontend/src/const.ts

@@ -12,3 +12,5 @@ export const SERVICE_REFETCH_INTERVAL_MS = 15000;
 export const TICK_REFETCH_INTERVAL_MS = 10000;
 export const FLOW_LIST_REFETCH_INTERVAL_MS = 30000;
 export const MAX_LENGTH_FOR_HIGHLIGHT = 400000;
+
+export const FORCE_REFETCH_ON_STAR = true;