Extend system meta (#1598)

* wip: add posture checks structs * add netbird version check * Refactor posture checks and add version checks * Add posture check activities (#1445) * Integrate Endpoints for Posture Checks (#1432) * wip: add posture checks structs * add netbird version check * Refactor posture checks and add version checks * Implement posture and version checks in API models * Refactor API models and enhance posture check functionality * wip: add posture checks endpoints * go mod tidy * Reference the posture checks by id's in policy * Add posture checks management to server * Add posture checks management mocks * implement posture checks handlers * Add posture checks to account copy and fix tests * Refactor posture checks validation * wip: Add posture checks handler tests * Add JSON encoding support to posture checks * Encode posture checks to correct api response object * Refactored posture checks implementation to align with the new API schema * Refactor structure of `Checks` from slice to map * Cleanup * Add posture check activities (#1445) * Revert map to use list of checks * Add posture check activity events * Refactor posture check initialization in account test * Improve the handling of version range in posture check * Fix tests and linter * Remove max_version from NBVersionCheck * Added unit tests for NBVersionCheck * go mod tidy * Extend policy endpoint with posture checks (#1450) * Implement posture and version checks in API models * go mod tidy * Allow attaching posture checks to policy * Update error message for linked posture check on deleting * Refactor PostureCheck and Checks structures * go mod tidy * Add validation for non-existing posture checks * fix unit tests * use Wt version * Remove the enabled field, as posture check will now automatically be activated by default when attaching to a policy * wip: add posture checks structs * add netbird version check * Refactor posture checks and add version checks * Add posture check activities (#1445) * Integrate Endpoints for Posture Checks (#1432) * wip: add posture checks structs * add netbird version check * Refactor posture checks and add version checks * Implement posture and version checks in API models * Refactor API models and enhance posture check functionality * wip: add posture checks endpoints * go mod tidy * Reference the posture checks by id's in policy * Add posture checks management to server * Add posture checks management mocks * implement posture checks handlers * Add posture checks to account copy and fix tests * Refactor posture checks validation * wip: Add posture checks handler tests * Add JSON encoding support to posture checks * Encode posture checks to correct api response object * Refactored posture checks implementation to align with the new API schema * Refactor structure of `Checks` from slice to map * Cleanup * Add posture check activities (#1445) * Revert map to use list of checks * Add posture check activity events * Refactor posture check initialization in account test * Improve the handling of version range in posture check * Fix tests and linter * Remove max_version from NBVersionCheck * Added unit tests for NBVersionCheck * go mod tidy * Extend policy endpoint with posture checks (#1450) * Implement posture and version checks in API models * go mod tidy * Allow attaching posture checks to policy * Update error message for linked posture check on deleting * Refactor PostureCheck and Checks structures * go mod tidy * Add validation for non-existing posture checks * fix unit tests * use Wt version * Remove the enabled field, as posture check will now automatically be activated by default when attaching to a policy * Extend network map generation with posture checks (#1466) * Apply posture checks to network map generation * run policy posture checks on peers to connect * Refactor and streamline policy posture check process for peers to connect. * Add posture checks testing in a network map * Remove redundant nil check in policy.go * Refactor peer validation check in policy.go * Update 'Check' function signature and use logger for version check * Refactor posture checks run on sources and updated the validation func * Update peer validation * fix tests * improved test coverage for policy posture check * Refactoring * Extend NetBird agent to collect kernel version (#1495) * Add KernelVersion field to LoginRequest * Add KernelVersion to system info retrieval * Fix tests * Remove Core field from system info * Replace Core field with new OSVersion field in system info * Added WMI dependency to info_windows.go * Add OS Version posture checks (#1479) * Initial support of Geolocation service (#1491) * Add Geo Location posture check (#1500) * wip: implement geolocation check * add geo location posture checks to posture api * Merge branch 'feature/posture-checks' into geo-posture-check * Remove CityGeoNameID and update required fields in API * Add geoLocation checks to posture checks handler tests * Implement geo location-based checks for peers * Update test values and embed location struct in peer system * add support for country wide checks * initialize country code regex once * Fix peer meta core compability with older clients (#1515) * Refactor extraction of OSVersion in grpcserver * Ignore lint check * Fix peer meta core compability with older management (#1532) * Revert core field deprecation * fix tests * Extend peer meta with location information (#1517) This PR uses the geolocation service to resolve IP to location. The lookup happens once on the first connection - when a client calls the Sync func. The location is stored as part of the peer: * Add Locations endpoints (#1516) * add locations endpoints * Add sqlite3 check and database generation in geolite script * Add SQLite storage for geolocation data * Refactor file existence check into a separate function * Integrate geolocation services into management application * Refactoring * Refactor city retrieval to include Geonames ID * Add signature verification for GeoLite2 database download * Change to in-memory database for geolocation store * Merge manager to geolocation * Update GetAllCountries to return Country name and iso code * fix tests * Add reload to SqliteStore * Add geoname indexes * move db file check to connectDB * Add concurrency safety to SQL queries and database reloading The commit adds mutex locks to the GetAllCountries and GetCitiesByCountry functions to ensure thread-safety during database queries. Additionally, it introduces a mechanism to safely close the old database connection before a new connection is established upon reloading, which improves the reliability of database operations. Lastly, it moves the checking of database file existence to the connectDB function. * Add sha256 sum check to geolocation store before reload * Use read lock * Check SHA256 twice when reload geonames db --------- Co-authored-by: Yury Gargay <[email protected]> * Add tests and validation for empty peer location in GeoLocationCheck (#1546) * Disallow Geo check creation/update without configured Geo DB (#1548) * Fix shared access to in memory copy of geonames.db (#1550) * Trim suffix in when evaluate Min Kernel Version in OS check * Add Valid Peer Windows Kernel version test * Add Geolocation handler tests (#1556) * Implement user admin checks in posture checks * Add geolocation handler tests * Mark initGeolocationTestData as helper func * Add error handling to geolocation database closure * Add cleanup function to close geolocation resources * Simplify checks definition serialisation (#1555) * Regenerate network map on posture check update (#1563) * change network state and generate map on posture check update * Refactoring * Make city name optional (#1575) * Do not return empty city name * Validate action param of geo location checks (#1577) We only support allow and deny * Switch realip middleware to upstream (#1578) * Be more silent in download-geolite2.sh script * Fix geonames db reload (#1580) * Ensure posture check name uniqueness when create (#1594) * Enhance the management of posture checks (#1595) * add a correct min version and kernel for os posture check example * handle error when geo or location db is nil * expose all peer location details in api response * Check for nil geolocation manager only * Validate posture check before save * bump open api version * add peer location fields to toPeerListItemResponse * Feautre/extend sys meta (#1536) * Collect network addresses * Add Linux sys product info * Fix peer meta comparison * Collect sys info on mac * Add windows sys info * Fix test * Fix test * Fix grpc client * Ignore test * Fix test * Collect IPv6 addresses * Change the IP to IP + net * fix tests * Use netip on server side * Serialize netip to json * Extend Peer metadata with cloud detection (#1552) * add cloud detection + test binary * test windows exe * Collect IPv6 addresses * Change the IP to IP + net * switch to forked cloud detect lib * new test builds * new GCE build * discontinue using library but local copy instead * fix imports * remove openstack check * add hierarchy to cloud check * merge IBM and SoftLayer * close resp bodies and use os lib for file reading * close more resp bodies * fix error check logic * parallelize IBM checks * fix response value * go mod tidy * include context + change kubernetes detection * add context in info functions * extract platform into separate field * fix imports * add missing wmi import --------- Co-authored-by: Zoltan Papp <[email protected]> --------- Co-authored-by: pascal-fischer <[email protected]> * generate proto * remove test binaries --------- Co-authored-by: bcmmbaga <[email protected]> Co-authored-by: Yury Gargay <[email protected]> Co-authored-by: Zoltan Papp <[email protected]>
netbirdio · Feb 20, 2024 · 51f133f · 51f133f
1 parent d5338c0
commit 51f133f
Show file tree

Hide file tree

Showing 30 changed files with 1,357 additions and 331 deletions.
diff --git a/client/system/detect_cloud/alibabacloud.go b/client/system/detect_cloud/alibabacloud.go
@@ -0,0 +1,24 @@
+package detect_cloud
+
+import (
+	"context"
+	"net/http"
+)
+
+func detectAlibabaCloud(ctx context.Context) string {
+	req, err := http.NewRequestWithContext(ctx, "GET", "http://100.100.100.200/latest/", nil)
+	if err != nil {
+		return ""
+	}
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return ""
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode == http.StatusOK {
+		return "Alibaba Cloud"
+	}
+	return ""
+}
diff --git a/client/system/detect_cloud/aws.go b/client/system/detect_cloud/aws.go
@@ -0,0 +1,57 @@
+package detect_cloud
+
+import (
+	"context"
+	"net/http"
+)
+
+func detectAWS(ctx context.Context) string {
+	v1ResultChan := make(chan bool, 1)
+	v2ResultChan := make(chan bool, 1)
+
+	go func() {
+		v1ResultChan <- detectAWSIDMSv1(ctx)
+	}()
+
+	go func() {
+		v2ResultChan <- detectAWSIDMSv2(ctx)
+	}()
+
+	v1Result, v2Result := <-v1ResultChan, <-v2ResultChan
+
+	if v1Result || v2Result {
+		return "Amazon Web Services"
+	}
+	return ""
+}
+
+func detectAWSIDMSv1(ctx context.Context) bool {
+	req, err := http.NewRequestWithContext(ctx, "GET", "http://169.254.169.254/latest/", nil)
+	if err != nil {
+		return false
+	}
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return false
+	}
+	defer resp.Body.Close()
+
+	return resp.StatusCode == http.StatusOK
+}
+
+func detectAWSIDMSv2(ctx context.Context) bool {
+	req, err := http.NewRequestWithContext(ctx, "PUT", "http://169.254.169.254/latest/api/token", nil)
+	if err != nil {
+		return false
+	}
+	req.Header.Set("X-aws-ec2-metadata-token-ttl-seconds", "21600")
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return false
+	}
+	defer resp.Body.Close()
+
+	return resp.StatusCode == http.StatusOK
+}
diff --git a/client/system/detect_cloud/azure.go b/client/system/detect_cloud/azure.go
@@ -0,0 +1,25 @@
+package detect_cloud
+
+import (
+	"context"
+	"net/http"
+)
+
+func detectAzure(ctx context.Context) string {
+	req, err := http.NewRequestWithContext(ctx, "GET", "http://169.254.169.254/metadata/instance?api-version=2021-02-01", nil)
+	if err != nil {
+		return ""
+	}
+	req.Header.Set("Metadata", "true")
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return ""
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode == http.StatusOK {
+		return "Microsoft Azure"
+	}
+	return ""
+}
diff --git a/client/system/detect_cloud/detect.go b/client/system/detect_cloud/detect.go
@@ -0,0 +1,65 @@
+package detect_cloud
+
+import (
+	"context"
+	"net/http"
+	"sync"
+	"time"
+)
+
+/*
+	This packages is inspired by the work of the original author (https://github.com/perlogix), but it has been modified to fit the needs of the project.
+	Original project: https://github.com/perlogix/libdetectcloud
+*/
+
+var hc = &http.Client{Timeout: 300 * time.Millisecond}
+
+func Detect(ctx context.Context) string {
+	subCtx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	funcs := []func(context.Context) string{
+		detectAlibabaCloud,
+		detectAWS,
+		detectAzure,
+		detectDigitalOcean,
+		detectGCP,
+		detectOracle,
+		detectIBMCloud,
+		detectSoftlayer,
+		detectVultr,
+	}
+
+	results := make(chan string, len(funcs))
+
+	var wg sync.WaitGroup
+
+	for _, fn := range funcs {
+		wg.Add(1)
+		go func(f func(context.Context) string) {
+			defer wg.Done()
+			select {
+			case <-subCtx.Done():
+				return
+			default:
+				if result := f(ctx); result != "" {
+					results <- result
+					cancel()
+				}
+			}
+		}(fn)
+	}
+
+	go func() {
+		wg.Wait()
+		close(results)
+	}()
+
+	for result := range results {
+		if result != "" {
+			return result
+		}
+	}
+
+	return ""
+}
diff --git a/client/system/detect_cloud/digitalocean.go b/client/system/detect_cloud/digitalocean.go
@@ -0,0 +1,24 @@
+package detect_cloud
+
+import (
+	"context"
+	"net/http"
+)
+
+func detectDigitalOcean(ctx context.Context) string {
+	req, err := http.NewRequestWithContext(ctx, "GET", "http://169.254.169.254/metadata/v1/", nil)
+	if err != nil {
+		return ""
+	}
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return ""
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode == http.StatusOK {
+		return "Digital Ocean"
+	}
+	return ""
+}
diff --git a/client/system/detect_cloud/gcp.go b/client/system/detect_cloud/gcp.go
@@ -0,0 +1,25 @@
+package detect_cloud
+
+import (
+	"context"
+	"net/http"
+)
+
+func detectGCP(ctx context.Context) string {
+	req, err := http.NewRequestWithContext(ctx, "GET", "http://metadata.google.internal", nil)
+	if err != nil {
+		return ""
+	}
+	req.Header.Add("Metadata-Flavor", "Google")
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return ""
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode == http.StatusOK {
+		return "Google Cloud Platform"
+	}
+	return ""
+}
diff --git a/client/system/detect_cloud/ibmcloud.go b/client/system/detect_cloud/ibmcloud.go
@@ -0,0 +1,54 @@
+package detect_cloud
+
+import (
+	"context"
+	"net/http"
+)
+
+func detectIBMCloud(ctx context.Context) string {
+	v1ResultChan := make(chan bool, 1)
+	v2ResultChan := make(chan bool, 1)
+
+	go func() {
+		v1ResultChan <- detectIBMSecure(ctx)
+	}()
+
+	go func() {
+		v2ResultChan <- detectIBM(ctx)
+	}()
+
+	v1Result, v2Result := <-v1ResultChan, <-v2ResultChan
+
+	if v1Result || v2Result {
+		return "IBM Cloud"
+	}
+	return ""
+}
+
+func detectIBMSecure(ctx context.Context) bool {
+	req, err := http.NewRequestWithContext(ctx, "PUT", "https://api.metadata.cloud.ibm.com/instance_identity/v1/token", nil)
+	if err != nil {
+		return false
+	}
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return false
+	}
+	defer resp.Body.Close()
+	return resp.StatusCode == http.StatusOK
+}
+
+func detectIBM(ctx context.Context) bool {
+	req, err := http.NewRequestWithContext(ctx, "PUT", "http://api.metadata.cloud.ibm.com/instance_identity/v1/token", nil)
+	if err != nil {
+		return false
+	}
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return false
+	}
+	defer resp.Body.Close()
+	return resp.StatusCode == http.StatusOK
+}
diff --git a/client/system/detect_cloud/oracle.go b/client/system/detect_cloud/oracle.go
@@ -0,0 +1,56 @@
+package detect_cloud
+
+import (
+	"context"
+	"net/http"
+)
+
+func detectOracle(ctx context.Context) string {
+	v1ResultChan := make(chan bool, 1)
+	v2ResultChan := make(chan bool, 1)
+
+	go func() {
+		v1ResultChan <- detectOracleIDMSv1(ctx)
+	}()
+
+	go func() {
+		v2ResultChan <- detectOracleIDMSv2(ctx)
+	}()
+
+	v1Result, v2Result := <-v1ResultChan, <-v2ResultChan
+
+	if v1Result || v2Result {
+		return "Oracle"
+	}
+	return ""
+}
+
+func detectOracleIDMSv1(ctx context.Context) bool {
+	req, err := http.NewRequestWithContext(ctx, "GET", "http://169.254.169.254/opc/v1/instance/", nil)
+	if err != nil {
+		return false
+	}
+	req.Header.Add("Authorization", "Bearer Oracle")
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return false
+	}
+	defer resp.Body.Close()
+	return resp.StatusCode == http.StatusOK
+}
+
+func detectOracleIDMSv2(ctx context.Context) bool {
+	req, err := http.NewRequestWithContext(ctx, "GET", "http://169.254.169.254/opc/v2/instance/", nil)
+	if err != nil {
+		return false
+	}
+	req.Header.Add("Authorization", "Bearer Oracle")
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return false
+	}
+	defer resp.Body.Close()
+	return resp.StatusCode == http.StatusOK
+}
diff --git a/client/system/detect_cloud/softlayer.go b/client/system/detect_cloud/softlayer.go
@@ -0,0 +1,25 @@
+package detect_cloud
+
+import (
+	"context"
+	"net/http"
+)
+
+func detectSoftlayer(ctx context.Context) string {
+	req, err := http.NewRequestWithContext(ctx, "GET", "https://api.service.softlayer.com/rest/v3/SoftLayer_Resource_Metadata/UserMetadata.txt", nil)
+	if err != nil {
+		return ""
+	}
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return ""
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode == http.StatusOK {
+		// Since SoftLayer was acquired by IBM, we should return "IBM Cloud"
+		return "IBM Cloud"
+	}
+	return ""
+}
diff --git a/client/system/detect_cloud/vultr.go b/client/system/detect_cloud/vultr.go
@@ -0,0 +1,24 @@
+package detect_cloud
+
+import (
+	"context"
+	"net/http"
+)
+
+func detectVultr(ctx context.Context) string {
+	req, err := http.NewRequestWithContext(ctx, "GET", "http://169.254.169.254/v1.json", nil)
+	if err != nil {
+		return ""
+	}
+
+	resp, err := hc.Do(req)
+	if err != nil {
+		return ""
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode == http.StatusOK {
+		return "Vultr"
+	}
+	return ""
+}