Skip to content

Commit

Permalink
Merge pull request #40 from GACWR/main_dev_branch
Browse files Browse the repository at this point in the history
Main dev branch
  • Loading branch information
Jovonni authored May 25, 2020
2 parents 0d7a82a + dc08216 commit ae56fe2
Show file tree
Hide file tree
Showing 77 changed files with 6,659 additions and 3,681 deletions.
10 changes: 10 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,17 @@ core/.mypy_cache/*
*.pyc
__pycache__/*
.mypy_cache/*
.mypy_cache
*.mypy_cache
/core/__pycache__/*
/core/.mypy_cache/*
core/__pycache__/*
core/.mypy_cache/*
interface/dist/
/interface/dist/
*.DS_Store
/**/.DS_Store
/**/__pycache__
/**/.mypy_cache
/**/.pyc
/**/*.mypy_cache
Empty file removed DockerfileUI
Empty file.
8 changes: 6 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,16 @@ check:
--ignore-missing-imports ;
profile_model:
cd core/ ; python3.7 core.py profile_model ${model_name};
uis: #ui server
cd interface/ ; node server.js
rd: # react development server
cd interface/ ; npm run start
rb: # react build
cd interface/ ; npm run build
electron: # launch electron
cd interface/ ; npm run start-electron
electron_static: # launch electron static react
cd interface/ ; npm run start-electron-static
package: #package react
cd interface/ ; npm run package;
save_dev:
git add * -v ; git commit -am ${M}-v ; git push origin master:main_dev_branch -v;
test:
Expand Down
52 changes: 36 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
# OpenUBA (Open User Behavior Analytics)
<img src="images/logo.png" width="35%" style="float-left" />

# Open User Behavior Analytics
A robust, and flexible open source User & Entity Behavior Analytics (UEBA) framework used for Security Analytics. Developed with luv by Data Scientists & Security Analysts from the Cyber Security Industry.

### White Paper
[source](https://github.com/GACWR/ouba-paper)

*This project is a work in progress and in a pre-alpha state; input and contributions are warmly welcome*

| Status Type | Status |
Expand All @@ -10,10 +15,6 @@ A robust, and flexible open source User & Entity Behavior Analytics (UEBA) frame
| `Issues` | [![Issues](https://img.shields.io/github/issues/GACWR/OpenUBA.svg)](https://github.com/GACWR/OpenUBA/issues) |
| `Closed Issues` | [![GitHub issues-closed](https://img.shields.io/github/issues-closed/GACWR/OpenUBA.svg)](https://GitHub.com/GACWR/OpenUBA/issues?q=is%3Aissue+is%3Aclosed) |
| `Last Commit` | [![Last commit](https://img.shields.io/github/last-commit/GACWR/OpenUBA.svg)](https://github.com/GACWR/OpenUBA/commits/master) |
| `UI Docker Stars` | [![Docker Stars](https://img.shields.io/docker/stars/gacwr/openuba-ui.svg)](https://hub.docker.com/r/gacwr/openuba-ui) |
| `UI Docker Pulls` | [![Docker Pulls](https://img.shields.io/docker/pulls/gacwr/openuba-ui.svg)](https://hub.docker.com/r/gacwr/openuba-ui) |
| `UI Docker Automated` | [![Docker Automated](https://img.shields.io/docker/cloud/automated/gacwr/openuba-ui.svg)](https://hub.docker.com/r/gacwr/openuba-ui) |
| `UI Docker Build` | [![Docker Build](https://img.shields.io/docker/cloud/build/gacwr/openuba-ui.svg)](https://hub.docker.com/r/gacwr/openuba-ui) |
| `Server Docker Stars` | [![Docker Stars](https://img.shields.io/docker/stars/gacwr/openuba-server.svg)](https://hub.docker.com/r/gacwr/openuba-server) |
| `Server Docker Pulls` | [![Docker Pulls](https://img.shields.io/docker/pulls/gacwr/openuba-server.svg)](https://hub.docker.com/r/gacwr/openuba-server) |
| `Server Docker Automated` | [![Docker Automated](https://img.shields.io/docker/cloud/automated/gacwr/openuba-server.svg)](https://hub.docker.com/r/gacwr/openuba-server) |
Expand All @@ -25,31 +26,46 @@ A robust, and flexible open source User & Entity Behavior Analytics (UEBA) frame
| `Code Size` | [![Code size in bytes](https://img.shields.io/github/languages/code-size/GACWR/OpenUBA.svg)](https://github.com/GACWR/OpenUBA) |
| `Chat` | ![Discord](https://img.shields.io/discord/683561405928177737) |

## Problem
Many UBA platforms typically use a "black box" approach to data science practices, which may work best for security analysts who are not interested in the nuts and bolts of the underlying models being used to generate anomalies, baselines, and cases. These platforms view their models as IP.

# Solution
OUBA takes an "open-model" approach, and is designed for the small subset of security analysts who have authentic curiosity about what models are doing, and how they work under the hood. We believe in the scientific computing community, and its contributions over the years (libraries, toolkits, etc). In security, rule/model transparency is key, for compliance, response/investigation, and decision making.

To take it a step further, OUBA also makes use of a community driven marketplace for models, similar to a plugin-store, where plugins are security models. This marketplace is where users of OUBA can install security models for their own use cases. Model developers can also upload their models, enabling other OUBA users to reuse them, whether for free, or compensation -- the choice is up to the model developer to make.

## Architecture
<img src="images/framework.jpg" width="750px" />
<img src="images/framework.jpg" width="600px" />


## Goals
To Build a lightweight, SIEM Agnostic, UEBA Framework focused on providing:
- Modeling
- Model Management
- Community-driven Model Library
- Model Library (both community/internally driven)
- Model Version Control
- Ready-to-use model modules
- Feedback Loop for continuous model training
- "Shadow Mode" for model and risk score experimentation
- Simple model configuration workflow
- Model groups
- Single-fire & Sequential models
- "White-box" model standard
- Rule Engine
- Single-fire & deviation-based rules
- Dashboard
- Modern stack
- Modular components
- Live updating
- Global state, and component state
- Features
- Rule Storage/Management
- Case Management
- Peer-oriented/community intel
- Lightweight, SIEM-agnostic architecture
- Flexible/open dataset support
- Alerting/Ticketing system
- Alerting
- Browser & desktop applications

## Stack
- Client Dashboard
Expand Down Expand Up @@ -83,26 +99,30 @@ To Build a lightweight, SIEM Agnostic, UEBA Framework focused on providing:
- Flat File


# User Interface (placeholder, UI being updated now)
<img src="images/ui.png" width="750px" />

The interface is meant to observe system events, and anomalies

# Interface (under active development)
<img src="images/ui.png" width="600px" />

## Views
- Dashboard (index)
- Home
- Models
- Anomalies
- Cases
- Modeling
- Settings

# Model Library
OpenUBA implements a model library purposed with hosting "ready-to-use" models, both developed by us, and the community. For starters, we host the default model repository, similar to any popular package manager (npm, cargo, etc). However, developers can host their own model repository for use in their own instance of OpenUBA.

## Model Library Workflow:
<img src="images/ouba-model-library.jpg" width="600px" />

## Installation/Usage
Go to [INSTALL.md](https://github.com/GACWR/OpenUBA/blob/master/docs/INSTALL.md)


## Get the updated code & documentation on XS code [here](https://cp.xscode.com/GACWR/OpenUBA)
Our main development, and documentation branches are first pushed to our sponsorship repository, and then eventually pushed to our public free repository. To obtain the most updated code, and documentation for OpenUBA, subscribe to our XS Code repository.

## Twitter
http://twitter.com/OpenUBA

## Discord (Main Server, and Dev Chat)
Discord Server: https://discord.gg/Ps9p9Wy
Expand Down
Binary file modified core/.DS_Store
Binary file not shown.
2 changes: 1 addition & 1 deletion core/.mypy_cache/3.7/api.meta.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"data_mtime": 1587529195, "dep_lines": [16, 17, 18, 19, 20, 21, 22, 23, 1, 1, 1, 1, 1, 1], "dep_prios": [10, 5, 5, 5, 5, 5, 10, 10, 5, 30, 30, 30, 30, 30], "dependencies": ["logging", "entity", "user", "enum", "display", "typing", "requests", "json", "builtins", "abc", "database", "json.encoder", "requests.models", "types"], "hash": "f96afb2a96e224f2b66fae496eec40da", "id": "api", "ignore_all": false, "interface_hash": "46a418b7569afd0dec40bea940888dff", "mtime": 1586530946, "options": {"allow_redefinition": false, "allow_untyped_globals": false, "always_false": [], "always_true": [], "bazel": false, "check_untyped_defs": false, "disallow_any_decorated": false, "disallow_any_explicit": false, "disallow_any_expr": false, "disallow_any_generics": false, "disallow_any_unimported": false, "disallow_incomplete_defs": false, "disallow_subclassing_any": false, "disallow_untyped_calls": false, "disallow_untyped_decorators": false, "disallow_untyped_defs": false, "follow_imports": "normal", "follow_imports_for_stubs": false, "ignore_errors": false, "ignore_missing_imports": true, "implicit_reexport": true, "local_partial_types": false, "mypyc": false, "no_implicit_optional": false, "platform": "darwin", "plugins": [], "show_none_errors": true, "strict_equality": false, "strict_optional": true, "strict_optional_whitelist": null, "warn_no_return": true, "warn_return_any": false, "warn_unreachable": false, "warn_unused_ignores": false}, "path": "api.py", "plugin_data": null, "size": 3260, "suppressed": [], "version_id": "0.761"}
{"data_mtime": 1590221816, "dep_lines": [16, 17, 18, 19, 20, 21, 22, 23, 1, 1, 1, 1, 1], "dep_prios": [10, 5, 5, 5, 5, 5, 10, 10, 5, 30, 30, 30, 30], "dependencies": ["logging", "entity", "user", "enum", "display", "typing", "requests", "json", "builtins", "abc", "database", "json.encoder", "requests.models"], "hash": "098e9deec1c9ac21b880dfc19daebc92", "id": "api", "ignore_all": false, "interface_hash": "46a418b7569afd0dec40bea940888dff", "mtime": 1588890541, "options": {"allow_redefinition": false, "allow_untyped_globals": false, "always_false": [], "always_true": [], "bazel": false, "check_untyped_defs": false, "disallow_any_decorated": false, "disallow_any_explicit": false, "disallow_any_expr": false, "disallow_any_generics": false, "disallow_any_unimported": false, "disallow_incomplete_defs": false, "disallow_subclassing_any": false, "disallow_untyped_calls": false, "disallow_untyped_decorators": false, "disallow_untyped_defs": false, "follow_imports": "normal", "follow_imports_for_stubs": false, "ignore_errors": false, "ignore_missing_imports": true, "implicit_reexport": true, "local_partial_types": false, "mypyc": false, "no_implicit_optional": false, "platform": "darwin", "plugins": [], "show_none_errors": true, "strict_equality": false, "strict_optional": true, "strict_optional_whitelist": null, "warn_no_return": true, "warn_return_any": false, "warn_unreachable": false, "warn_unused_ignores": false}, "path": "api.py", "plugin_data": null, "size": 3311, "suppressed": [], "version_id": "0.761"}
2 changes: 1 addition & 1 deletion core/.mypy_cache/3.7/core.data.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion core/.mypy_cache/3.7/core.meta.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"data_mtime": 1587530483, "dep_lines": [21, 23, 24, 25, 26, 27, 28, 29, 30, 32, 33, 33, 1, 1, 1, 1, 1, 1, 1, 22, 34], "dep_prios": [5, 10, 10, 10, 5, 5, 5, 5, 5, 10, 10, 10, 5, 30, 30, 30, 30, 30, 30, 5, 10], "dependencies": ["flask", "logging", "threading", "time", "model", "test", "process", "api", "display", "unittest", "trace", "sys", "builtins", "abc", "flask.app", "flask.helpers", "flask.json", "types", "typing"], "hash": "871acc3803dc2b0457d8d2c9ef043b2b", "id": "core", "ignore_all": false, "interface_hash": "7faa962fd68d1c95d1608c6eac535168", "mtime": 1587141805, "options": {"allow_redefinition": false, "allow_untyped_globals": false, "always_false": [], "always_true": [], "bazel": false, "check_untyped_defs": false, "disallow_any_decorated": false, "disallow_any_explicit": false, "disallow_any_expr": false, "disallow_any_generics": false, "disallow_any_unimported": false, "disallow_incomplete_defs": false, "disallow_subclassing_any": false, "disallow_untyped_calls": false, "disallow_untyped_decorators": false, "disallow_untyped_defs": false, "follow_imports": "normal", "follow_imports_for_stubs": false, "ignore_errors": false, "ignore_missing_imports": true, "implicit_reexport": true, "local_partial_types": false, "mypyc": false, "no_implicit_optional": false, "platform": "darwin", "plugins": [], "show_none_errors": true, "strict_equality": false, "strict_optional": true, "strict_optional_whitelist": null, "warn_no_return": true, "warn_return_any": false, "warn_unreachable": false, "warn_unused_ignores": false}, "path": "core.py", "plugin_data": null, "size": 4651, "suppressed": ["flask_cors", "coloredlogs"], "version_id": "0.761"}
{"data_mtime": 1590221817, "dep_lines": [21, 23, 24, 25, 26, 27, 28, 29, 30, 32, 33, 33, 1, 1, 1, 1, 1, 1, 1, 1, 1, 22, 34], "dep_prios": [5, 10, 10, 10, 5, 5, 5, 5, 5, 10, 10, 10, 5, 20, 20, 30, 30, 30, 30, 30, 30, 5, 10], "dependencies": ["flask", "logging", "threading", "time", "model", "test", "process", "api", "display", "unittest", "trace", "sys", "builtins", "json", "requests", "abc", "flask.app", "flask.helpers", "flask.json", "types", "typing"], "hash": "e9bc7f8df654ea5ae5fbadf7697938c9", "id": "core", "ignore_all": false, "interface_hash": "ad2ac45ed9e95279f9cce0492273b787", "mtime": 1590218114, "options": {"allow_redefinition": false, "allow_untyped_globals": false, "always_false": [], "always_true": [], "bazel": false, "check_untyped_defs": false, "disallow_any_decorated": false, "disallow_any_explicit": false, "disallow_any_expr": false, "disallow_any_generics": false, "disallow_any_unimported": false, "disallow_incomplete_defs": false, "disallow_subclassing_any": false, "disallow_untyped_calls": false, "disallow_untyped_decorators": false, "disallow_untyped_defs": false, "follow_imports": "normal", "follow_imports_for_stubs": false, "ignore_errors": false, "ignore_missing_imports": true, "implicit_reexport": true, "local_partial_types": false, "mypyc": false, "no_implicit_optional": false, "platform": "darwin", "plugins": [], "show_none_errors": true, "strict_equality": false, "strict_optional": true, "strict_optional_whitelist": null, "warn_no_return": true, "warn_return_any": false, "warn_unreachable": false, "warn_unused_ignores": false}, "path": "core.py", "plugin_data": null, "size": 5556, "suppressed": ["flask_cors", "coloredlogs"], "version_id": "0.761"}
2 changes: 1 addition & 1 deletion core/.mypy_cache/3.7/dataset.data.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion core/.mypy_cache/3.7/dataset.meta.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"data_mtime": 1587530566, "dep_lines": [21, 24, 25, 1, 1, 1, 22, 23], "dep_prios": [10, 5, 5, 5, 30, 30, 10, 10], "dependencies": ["logging", "typing", "enum", "builtins", "abc", "types"], "hash": "f946d34cac2783afec7ecd31afac01f8", "id": "dataset", "ignore_all": false, "interface_hash": "abca2bf43c06b930952ba29a520a71a8", "mtime": 1587530563, "options": {"allow_redefinition": false, "allow_untyped_globals": false, "always_false": [], "always_true": [], "bazel": false, "check_untyped_defs": false, "disallow_any_decorated": false, "disallow_any_explicit": false, "disallow_any_expr": false, "disallow_any_generics": false, "disallow_any_unimported": false, "disallow_incomplete_defs": false, "disallow_subclassing_any": false, "disallow_untyped_calls": false, "disallow_untyped_decorators": false, "disallow_untyped_defs": false, "follow_imports": "normal", "follow_imports_for_stubs": false, "ignore_errors": false, "ignore_missing_imports": true, "implicit_reexport": true, "local_partial_types": false, "mypyc": false, "no_implicit_optional": false, "platform": "darwin", "plugins": [], "show_none_errors": true, "strict_equality": false, "strict_optional": true, "strict_optional_whitelist": null, "warn_no_return": true, "warn_return_any": false, "warn_unreachable": false, "warn_unused_ignores": false}, "path": "dataset.py", "plugin_data": null, "size": 7111, "suppressed": ["pandas", "numpy"], "version_id": "0.761"}
{"data_mtime": 1590221815, "dep_lines": [21, 24, 25, 26, 27, 1, 1, 1, 1, 1, 22, 23], "dep_prios": [10, 5, 5, 10, 10, 5, 30, 30, 30, 30, 10, 10], "dependencies": ["logging", "typing", "enum", "requests", "json", "builtins", "abc", "json.encoder", "requests.models", "types"], "hash": "83ae061ff18a52d03f98514dbba008d2", "id": "dataset", "ignore_all": false, "interface_hash": "029d6cd5fccde7da1a1e16a4cfd31966", "mtime": 1590222810, "options": {"allow_redefinition": false, "allow_untyped_globals": false, "always_false": [], "always_true": [], "bazel": false, "check_untyped_defs": false, "disallow_any_decorated": false, "disallow_any_explicit": false, "disallow_any_expr": false, "disallow_any_generics": false, "disallow_any_unimported": false, "disallow_incomplete_defs": false, "disallow_subclassing_any": false, "disallow_untyped_calls": false, "disallow_untyped_decorators": false, "disallow_untyped_defs": false, "follow_imports": "normal", "follow_imports_for_stubs": false, "ignore_errors": false, "ignore_missing_imports": true, "implicit_reexport": true, "local_partial_types": false, "mypyc": false, "no_implicit_optional": false, "platform": "darwin", "plugins": [], "show_none_errors": true, "strict_equality": false, "strict_optional": true, "strict_optional_whitelist": null, "warn_no_return": true, "warn_return_any": false, "warn_unreachable": false, "warn_unused_ignores": false}, "path": "dataset.py", "plugin_data": null, "size": 10489, "suppressed": ["pandas", "numpy"], "version_id": "0.761"}
Loading

0 comments on commit ae56fe2

Please sign in to comment.