Merge pull request #422 from eclipse/kaybee-importer-doc

KB-Importer Docs

docs/mkdocs.yml

@@ -65,6 +65,7 @@ nav:
     - Tutorial: vuln_db/tutorials/vuln_db_tutorial.md
     - Manual:
       - 'Intro': vuln_db/manuals/index.md
+      - 'KB Importer': vuln_db/manuals/kb_importer.md
       - 'Patch Analyzer' : vuln_db/manuals/patch_analyzer.md
       - 'Patch Lib Analyzer' : vuln_db/manuals/patch_lib_analyzer.md
       - 'Manual Assessment' : vuln_db/manuals/manual_assessment.md

docs/public/content/vuln_db/manuals/index.md

@@ -10,6 +10,6 @@ An up-to-date list of all vulnerabilities currently comprised in the database ca
 
 Detailed information for a given vulnerability can be obtained using the following link, whereby `<foo>` has to be replaced by the vulnerability identifier:  `@@ADDRESS@@/backend/bugs/<foo>`
 
-The [Patch Analyzer](../../vuln_db/manuals/patch_analyzer/) module can be used to add vulnerabilities to the knowledge base.
+The [Patch Analyzer](../../vuln_db/manuals/patch_analyzer/) and [KB Importer](../../vuln_db/manuals/kb_importer/)  module can be used to add vulnerabilities to the knowledge base.
 
 In the current implementation, the vulnerability detection is based on the containment of the fully-qualified name of a construct changed to fix a vulnerability in a library. The information whether the library contains the vulnerable or fixed version of the construct is computed asynchronously by the [Patch Lib Analyzer](../../vuln_db/manuals/patch_lib_analyzer/).

docs/public/content/vuln_db/manuals/kb_importer.md

@@ -0,0 +1,120 @@
+# KB(Knowledge Base) Importer
+
+KB-Importer is a java application which is used to import vulnerabilities in Steady Backend from [Project KB](https://github.com/SAP/project-kb). 
+
+!!! info "Project KB"
+    [Project KB](https://github.com/SAP/project-kb) supports the creation, management and aggregation of a distributed, collaborative knowledge base of vulnerabilities that affect open-source software. 
+
+This guide talks about gathering the data in Steady from [Project KB](https://github.com/SAP/project-kb).
+
+## Setup
+1. Use the following link to setup and configure Project KB - [installing-the-kaybee-tool](https://sap.github.io/project-kb/getting_started/#installation).
+2. Update the variables in `kaybeeconf.yaml` configuration file. The variables are 
+    * `USER_TOKEN` - Token specified in your Steady .env file.
+    * `BACKEND_URL` - URL of the rest-backend service (Mandatory).
+    * `CIA_URL` - URL of the deployed rest-lib-util service (This is used if you are uploading the affected artifacts).
+    * `KB_IMPORTER_PATH` - Jar path of your kb-importer module.
+
+## Usage
+1. Run `kaybee merge` to get the vulnerablity information from Project-KB.
+2. Run `kaybee export -t steady` which outputs a shell script `steady.sh` which is is then run to update all vulnerability data in Steady.
+3. To get specific vulnerability information run `kaybee export -t steady -f .kaybee/merged/CVE-2019-0191/statement.yaml` which outputs a shell script `steady.sh` which is is then run to update `CVE-2019-0191` vulnerability data in Steady.
+
+## Technical Information
+This information is to know how KB-Importer works in detail. The directory structure and the files are automatically generated by the shell script `steady.sh` as shown above. 
+
+To add a vulnerability manually using KB-Importer the folloing folder structure and command has to be used:
+
+Command Line arguments for KB-Importer:
+
+* `directory (-d) <arg>` : A directory containing commits information for a vulnerability. More details regarding the directory structure is shown below.
+* `overwrite (-o)` : Overwrite the analysis of a vulnerability if it already exists in the backend.
+* `upload (-u)` : Upload construct changes. Without the -u option, the resulting JSON will be saved to file for you to review (e.g., to check that the list of construct changes is not empty).
+* `verbose (-v)` : Verbose mode to get more detailed logs.
+
+Other useful/required configurations:
+
+* `vulas.shared.backend.serviceUrl` : URL of the deployed rest-backend service (Mandatory).
+* `vulas.shared.cia.serviceUrl` : URL of the deployed rest-lib-util service (This is used if you are uploading the affected artifacts as shown in the below example).
+
+Directory structure should be as follows:
+```
+vulnerability_dir          #vulnerability directory specified with option -d. Note: The directory name need not be the same as the vulnerability id
+├── commit1_dir            #commit directory. Note: The directory name need not be the same as the commit id
+│   ├── before             #directory containing the files/folders before the vulnerability fix
+│   ├── after              #directory containing the files/folders after the vulnerability fix
+│   └── metadata.json      #commit metadata file for each commit. More info can be found below
+├── commit2_dir            #mutiple commit directories if there are mutiple commits for a vulnerability fix
+│   ├── before
+│   ├── after
+│   └── metadata.json
+└── metadata.json          #vulnerability metadata file
+```
+
+Vulnerability `metadata.json` example:
+```json
+{ 
+  "vulnerability_id": "CVE-2016-1000338", 
+  "aliases": ["VULN-1000338","VULN-0002"],
+  "affected_artifacts": [
+    { 
+      "id": "pkg:maven/org.bouncycastle/[email protected]",
+      "reason": "Manual review",
+      "affected": true
+    },
+    { 
+      "id": "pkg:maven/org.bouncycastle/[email protected]",
+      "reason": "Manual review",
+      "affected": false
+    }
+  ],
+  "notes": [
+    {
+      "links": [
+        "https://www.bouncycastle.org/releasenotes.html",
+        "https://ubuntu.com/security/notices/USN-3727-1"
+      ],
+      "text": "In Bouncy Castle JCE Provider version 1.55 and earlier the DSA does not fully validate ASN.1 encoding of signature on verification. It is possible to inject extra elements in the sequence making up the signature and still have it validate, which in some cases may allow the introduction of 'invisible' data into a signed structure."
+    },
+    {
+      "links": [
+        "https://lists.debian.org/debian-lts-announce/2018/07/msg00009.html"
+      ],
+      "text": "It was discovered that Bouncy Castle incorrectly handled certain crypto algorithms. A remote attacker could possibly use these issues to obtain sensitive information, including private keys."
+    }
+  ]
+}
+```
+
+Information about the fields of vulnerability `metadata.json` file:
+
+ * `vulnerability_id`: Vulnerability ID.
+ * `aliases`: Aliases used for the vulnerability id.
+ * `affected_artifacts`: Affected Libraries for the vulnerability.
+ * `notes`: This is an array of `links` and `text`. Here, the links array consists of links which provide comprehensive vulnerability information and the text is the vulnerability description. (optional, it must be provided for vulnerabilities not available from the NVD).
+
+
+Commit metadata.json example file:
+```json
+{
+  "repository" : "https://github.com/bcgit/bc-java",
+  "branch" : "DEFAULT_BRANCH",
+  "timestamp" : "1476490865",
+  "commit_id" : "843c2e60f67d71faf81d236f448ebbe56c62c647"
+}
+```
+
+Information about the fields of commit metadata.json file:
+
+ * `repository`: URL of the VCS repository hosting the library project.
+ * `branch`: Branch name.
+ * `timestamp`: Timestamp when the commit was made.
+ * `commit_id`: VCS commit id.
+
+ Example run for kb-importer:
+```sh
+java -Dvulas.shared.backend.header.X-Vulas-Client-Token=<token> -Dvulas.shared.cia.serviceUrl=<cia-url> -Dvulas.shared.backend.serviceUrl=<backend-url> -jar kb-importer-3.1.14-SNAPSHOT-jar-with-dependencies.jar -d ./CVEdata/CVE-2016-1000338 -v -u -o
+```
+
+## Limitation
+[Project KB](https://github.com/SAP/project-kb) only supports Java specific vulnerabilities. Python vulnerabilities will be supported once [For python, __init__ files are to be checked out](https://github.com/SAP/project-kb/issues/36) is fixed.