Skip to content

Commit

Permalink
Progress calculation (#1579)
Browse files Browse the repository at this point in the history
* Add: count alive and dead hosts found during the scan and return the results

All information in the Results struct allows to calculate the scan progress.

For testing with a running Redis server, check that the nvticache is already
loaded (use the old openvas -u or nasl-cli feed update). Add example items for results and status

```
$ redis-cli -s /var/run/redis-openvas/redis.sock
redis /var/run/redis-openvas/redis.sock> info keyspace
db0:keys=1,expires=0,avg_ttl=0
db1:keys=57685,expires=0,avg_ttl=0
db2:keys=429536,expires=0,avg_ttl=0
redis /var/run/redis-openvas/redis.sock> SELECT 3
OK
redis /var/run/redis-openvas/redis.sock[3]> LPUSH internal/results "LOG|||127.0.0.1||| localhost ||||||||| HOST_START" "ERRMSG|||127.0.0.1||| localhost ||||||1.2.3.4.5.6||| NVT timeout" "ALARM|||127.0.0.1||| example.com |||22/tcp|||12.11.10.9.8.7||| Something wrong|||/var/lib/lib1.jar" "DEADHOST||| ||| ||| ||| |||3" "HOST_COUNT||| ||| ||| ||| |||12" "DEADHOST||| ||| ||| ||| |||1" "HOSTS_EXCLUDED||| ||| ||| ||| |||4"

(integer) 7
redis /var/run/redis-openvas/redis.sock[3]> LPUSH internal/status 127.0.0.1/0/1000 127.0.0.2/15/1000 127.0.0.3/750/1000 127.0.0.2/0/-1 127.0.0.4/500/1000 127.0.0.1/128/1000 127.0.0.4/1000/1000 127.0.0.5/0/-1
(integer) 8
redis /var/run/redis-openvas/redis.sock[3]> info keyspace
db0:keys=1,expires=0,avg_ttl=0
db1:keys=57685,expires=0,avg_ttl=0
db2:keys=429536,expires=0,avg_ttl=0
db3:keys=2,expires=0,avg_ttl=0
```

And run the following test, which initialize a RedisHelper struct with
both dbs, cache(2) and main kb (3), and get results and scan status
```
use std::sync::{Arc, Mutex};
use openvas::{
    openvas_redis::RedisHelper, result_collector::ResultHelper,
};
use redis_storage::{NameSpaceSelector, RedisCtx};

async fn main() -> Result<(), std::fmt::Error> {

    let redis_url = "unix:///run/redis-openvas/redis.sock";
    let rctx = RedisCtx::open(redis_url, &[NameSpaceSelector::Fix(3)]).unwrap();
    let cache = RedisCtx::open(redis_url, &[NameSpaceSelector::Key("nvticache")]).unwrap();

    let rc = RedisHelper::<RedisCtx>::new(Arc::new(Mutex::new(cache)), Arc::new(Mutex::new(rctx)));
    let mut p = ResultHelper::init(rc);

    p.get_results().await.unwrap();
    p.get_status().await.unwrap();

    if let Ok(r) = Arc::as_ref(&p.results).lock() {
        println!("{:?}", r);
    }
    Ok(())
}
```

* Doc: document progress calculation
  • Loading branch information
jjnicola authored Feb 29, 2024
1 parent bb7ae5b commit e0e6dac
Show file tree
Hide file tree
Showing 3 changed files with 288 additions and 58 deletions.
75 changes: 75 additions & 0 deletions rust/doc/faq/progress-calculation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# How do I calculate a scan progress?

The required information for the progress calculation will be always provided by `openvasd`, via the route `scan/{scan-id}/status`.
You get a json object which looks like as following:
```
{
"start_time": 1709111465,
"end_time": 1709111496,
"status": "succeeded",
"host_info": {
"all": 12,
"excluded": 3,
"dead": 2,
"alive": 1,
"queued": 0,
"finished": 1,
"scanning": [
"127.0.0.1": 12,
"127.0.0.3": 75,
]
}
}
```

Then, with this information you can calculate the scan progress with the following suggested formula:

```
scan_progress = (sum_of_scanning_hosts_values + 100 * finished)
/ (all - dead)
```
For the example given above, the progress is:

```
scan_progress = (12 + 75 + 100 * 1) / (12 - 2) = 18.7 %
```

## Special case for resume task

When you resume a task, the finished hosts should not be scanned again. Therefore they are sent in the list of excluded hosts. Please read the documentation for [resume scan](resume-scan.md).
In this case, a resume scan with some finished hosts, should not start with a progress with 0%, but a progress according with the already finished hosts.

Then, imagine that the scan of example above, with an initial target of 15 hosts, was stopped/interrupted and you want to resume it. It has an already finished hosts. This hosts is added to the list of `excluded hosts`.
At the beginning of the resumed scan you have:

```
{
"start_time": 1709111465,
"end_time": 1709111496,
"status": "succeeded",
"host_info": {
"all": 11,
"excluded": 4,
"dead": 2,
"alive": 1,
"queued": 0,
"finished": 1,
"scanning": [
]
}
}
```

As you already know the amount of previously finished hosts, you use this value for the progress calculation as well. You have to add the previously finished hosts to the total amount of hosts and add them to the `finished` host as well, because in the end, they were already scanned and finished.
The suggested formula for calculating the scan progress of a resume task:
```
scan_progress = (sum_of_scanning_hosts_values + 100 * (finished + previously_finished))
/ (all + previously_finished - dead)
```
Then, the scan progress for a resume task will start from:

```
scan_progress = (0 + 100 * (0 + 1) / (11 + 1 - 2) = 10 %
```
114 changes: 114 additions & 0 deletions rust/doc/misc/progress-calculation-details.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# How do I calculate a scan progress?

The scan progress is calculated in base of the amount of launched plugins over the total amount of plugins per host. This would be easiest case, when a scan has a single host target.

Other cases change the equation, like having many hosts, excluded hosts and dead hosts.

The required information for the calculation will be always provided by `openvasd`, but is a client task to keep track of the numbers.

### Excluded hosts

`openvasd` provides the amount of excluded hosts, which can differ from the amount sent by the client. This happens because openvasd can find a duplicate host in the list, or a host which is not system allowed.

Then, the amount of excluded hosts to be used for the progress calculation, is the one provided by `openvasd`.

### Total hosts

Is the amount of hosts to be scanned after a clean up. This means, after removing duplicated, excluded, unresolveble and not allowed hosts. I sent once at the beginning of the scan.

### Dead hosts

With `Boreas` it is possible to know at the beginning of the scan, how many alive hosts are in the target and how many are dead. However, this number can change if later a host which is found alive at the scan start, dies during the the scan.

So, the client receives a message at the beginning with the number of dead hosts, but this number must be updated each time a new dead hosts is found during the scan.

### Alive hosts

Is the amount of hosts which were already scanned and successfully finished.

### Current scan hosts

`openvasd` sends to the client a list of the current scanned hosts and its progress. The host progress, as explained above, is the the amount of launched plugins over the total amount of plugins.

Once the host progress reaches the 100%, the host is removed from the list and adds +1 to the `alive hosts` counter.

The same happens with a dead hosts when it is found as dead during the scan. The dead hosts is removed from this list and adds +1 to the `dead hosts` counter.

## Example:

Suppose a target with 15 hosts and 3 from the list are excluded. `openvas` found a total of 12 hosts. Also, during the alive test scan with `Boreas`, or even during the scan, 2 hosts where found dead.
One hosts is already scanned, and 2 hosts are currently being scanned.

```
{
"start_time": 1709111465,
"end_time": 1709111496,
"status": "succeeded",
"host_info": {
"all": 12,
"excluded": 3,
"dead": 2,
"alive": 1,
"queued": 0,
"finished": 1,
"scanning": [
"127.0.0.1": 12,
"127.0.0.3": 75,
]
}
}
```

Then, with this information you can calculate the scan progress with the following formula:

```
scan_progress = (sum_of_host_scanning_values + 100 * alive)
/ (all - dead)
```
For this example, the progress is:

```
scan_progress = (12 + 75 + 100 * 1) / (12 - 2) = 18.7 %
```

## Special case for resume task

When a resume task, the finished hosts should not be scanned again. Therefore they are sent in the list of excluded hosts. Please read the documentation for [resume scan](resume-scan.md).
In this case, a resume scan with some finished hosts, should not start with a progress with 0%, but a progress according with the already finished hosts.

Then, imagine that the scan of example above, with an initial target of 15 hosts, was stopped/interrupted and you want to resume it. It has an already finished hosts. This hosts is added to the list of `excluded hosts`.
At the beginning of the resumed scan you have:

```
{
"start_time": 1709111465,
"end_time": 1709111496,
"status": "succeeded",
"host_info": {
"all": 11,
"excluded": 4,
"dead": 2,
"alive": 1,
"queued": 0,
"finished": 1,
"scanning": [
]
}
}
```

As you already know the amount of finished hosts, you use this value for the progress calculation. You have to add the finished hosts to the total amount of hosts and the add them also to the `count_alive`, because in the end, they were already scanned and finished.
The suggested formula for calculating the scan progress of a resume task:
```
scan_progress = (sum_of_host_status_values + 100 * (count_alive + finished))
/ (total_hosts + finished - count_dead)
```
Then, the scan progress for a resume task will start from:

```
scan_progress = (0 + 100 * (0 + 1) / (11 + 1 - 2) = 10 %
```

Loading

0 comments on commit e0e6dac

Please sign in to comment.