Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* feat(otel): wip pktvisor otel receiver * feat(otel): wip move logic to pktvisor implementation * feat(otel): wip use pktvisor otel factory * feat(otel): wip use pktvisor otel factory * test(otel): fix unit and integration tests * feat(otel): fix remove go.mod unnecessary dependencies * feat(orb-ui): change layout to two cards in a row, fields per row per card. * fix(sinks): fix status code response of sink ceration * feat(otel): organize folder structure to pktvisor otel receiver * feat(otel): Fix load config unit test * fix(orb-ui): Missing trailling space in scss file * fix(orb-ui): fix username property missing * feat(otel): Fix orb-agent compilation fail * fix(orb-ui): send/retrieve company and fullName using metadata field * test(otel): wip setting up integration tests with pktvisor container * test(otel): crate pktvisor docker config * test(otel): wip making basic pktvisor config works * fix(sinks): reformat code * feat(orb-ui): Add reset-password page and move request-password component to proper folder * fix(orb-ui): sink list loading bar always reloading * fix(orb-ui): pages never stop reloading * fix for agent groups list page * fix for agents list page * fix for datasets list page * fix for agent policies list page * feature(sinks): add service metrics to api * 558 dataset tests (#597) * added agent groups feature test * fix: breadcrumb 'Home' to 'User Profile' * test(otel): wip make pktvisor container run (but still not scrapping) * test(otel): wip integration test pktvisor container * feat(policies): add service metrics to api * test(otel): wip integration test pktvisor * feature(sinks): improvements on service metrics * test(otel): fix integration tests with pktvisor (problems exposing the container localhost) * test(otel) minor corrections * fix(orb-ui): fix display of agent tags on agent group details view * fix(orb-ui): fix line break on third step - misaligned select field * fix(orb-ui): remove dataset view 'tags' field * fix(orb-ui): remove required * from key/value tag pair in sink add/edit pages * fix(orb-ui): remove required * from sink details view tag field * feat(otel-exporter): bootstraping otlp exporter for orb agent * fix(orb-ui): Remove red asterisk '*' required tags for agents in view and details page * test(otel-exporter): create unit test to otel exporter config * fix(orb-ui): fix error msg * fix(orb-ui): Remove required asterisk '*' from version field in agent policy detail view * feat(otel): create a config to decide which way to scrape metrics * fix(orb-ui): fix not displaying agent matches on creation * feat(otel): code review changes * fix(orb-ui): fix menu items order in navigation menu * feat(otel): code review changes * fix(orb-ui): fix agent view copy command button not changing icon after click * test(otel): create unit test to otlp exporter * test(otel): using otlp exporter on orb agent * fix(orb-ui): #625 fix order on dashboard page (#637) * fix(orb-ui): fix order of orb entities on dashboard page * fix(orb-ui): fix wrong row ordering * english correction * feat(otel): wip code sanitization * feat(otel): wip extract code to new methods * feat(otel): Create otlp exporter starter on pktvisor startup. * enhancement(orb-ui): improve home page card ordering for user actions * fix(dataset): Fix http patch to put on edition. * feat(otel): Make otel configuration parameters more readable when configuring pktvisor * bump to 0.11.0-develop * feat(otel-exporter): Fix merge conflicts with ui * fix ui conflicts * fix missing PKTVISOR_TAG in GitHub actions. bring go builds up to 1.17 (#664) * fix missing PKTVISOR_TAG in GitHub actions * bring go builds up to 1.17 * add agent README * test(sinks): increase unit test coverage at data access layer * test(sinks): increase service unit test coverage * test(sinks): increase endpoint unit tests coverage * add dnstap input (#669) * fix(orb-ui): fix button outside form scope not submitting forgotPassword fix(orb-ui): override nbResetPassword resetPassword method * feat(otel-exporter): create otel otlp exporter starter * test(sinks): increase endpoint unit test coverage * bug(sinker): Applying changes from pktvisor/types.go to sinker/pktvisor/types.go to fix prom particle generation * test(sinks): fix test cases description * fix(orb-ui): remove option to choose backend type for agent policy * fix(orb-ui): tap & input of agent policy fix: use input predefined in tap configuration fix: show input in dropdown selector for tap fix: fail gracefully - add label stating there are no taps listed for when there are no agents running. Links to Agents List Page. * fix(orb-ui): fix handler configuration json not being posted * bug(sinker): wip sinker console output * bug(sinker): wip remove unused console output * bug(sinker): wip ugly way to solve the topGeoLoc and topASN problem * testing agents through the ui (#667) * testing agents through the ui * bug(sinker): wip solving TopGeoLoc using regex * test(sinks): fix postgres unit test description and fix error return for update non-existing sink * test(sinks): fix expected error of postgres unit test cases * add the indirect go mods, ensure static link during CGO build. * bug(sinker): wip TopGeoLoc * bug(sinker): wip using regex to get exceptions to prometheus conversion * bug(sinker): return version to unknown on version.go file * bug(sinker): fixing go.mod file * fix(orb-ui): Page not responding after clicking on Register nor logging into dashboard * feat(sinker): remove unnecessary comments and console writers * feat(orb-ui): add advanced options to tap configuration and filter configuration in agent policies. * feat(fleet): remove last_db_data and agent_metadata from agent list endpoint * feat(orb-ui): mark text where user could/should change interface * feat(orb-ui): * remove odd <hr/> breaks * add labels to each section of input/tap configuration * feat(orb-ui): change color for marked text * remove codeql analysis for now * feat(orb-agent): extract pktvisor otel receiver / exporter functions to a new file * feat(orb-agent): Remove policy from agent local repo on removal action * 679 second policy (#683) * apply two policies to an agent Co-authored-by: Stéfano de Faria <[email protected]> * fix ui conflicts * fix conflicts with python-test * fix(orb-ui): chain observables for dataset related entities and display name of such entities when retrieved * fix(orb-ui): Fix wrong spacing between sink display and elements above it * fix(orb-ui): keep sinks list instead of only their names * refactor(orb-ui): cleanup unused imports and code style * refactor(orb-ui): implement ngOnDestroy to unsubscribe from observables * fix(orb-ui): remove option to filter dataset by tags * recovering some changes that I noticed that shold be on previous PR (#724) * refactor(orb-ui): refactor pactsafe integration * refactor(orb-ui): finish integration with pactsafe * show only checkbox with 'agreement' label * remove minified pactsafe wrapper * contract is now shown when user clicks link * int_tests - automate account registration with credentials required (email and password) if they are not yet registered (#726) * register an account using email and password if credentials are not registered yet * fix(orb-ui): Policy Details View (#734) * wip * fix(orb-ui): Add more details on Agent Policy Details View * add handlers section to display name of each handler * add input type and tap name section * feat(orb-ui): list handlers as accordion items * show handler details inside accordion body, represented in JSON * Feat/orb agent debug arg (#728) * feat(orb-agent): Create debug parameters to agent * feat(orb-agent): Use right debug configuration args * fix(orb-ui): #737 register button not working (#740) * fix(orb-ui): register button not working * fix(orb-ui): enable sandbox when not production build * fix(orb-ui): register button not working - _ps module not available * Bug/mqtt incoming stoped error (#742) * feat(orb-ui): add advanced options to tap configuration and filter configuration in agent policies. * feat(orb-ui): * remove odd <hr/> breaks * add labels to each section of input/tap configuration * recovering some changes that I noticed that shold be on previous PR (#724) * fix - upgrade paho.mqtt version and setup auto reconnect configuration * remove ui e python-test * merge fix with ui and python-test * bug(orb-agent) Change keep alive and ping timeout configuration for bigger times Co-authored-by: gpazuch <[email protected]> Co-authored-by: manrodrigues <[email protected]> * fix(orb-ui): disable ps contract box if not prod (#741) * fix(orb-ui): register button not working * fix(orb-ui): enable sandbox when not production build * fix(orb-ui): register button not working - _ps module not available * fix(orb-ui): hide contract box when development and not production * fix(orb-ui): remove unused imports (#744) * feat(orb-ui): Display warning when no tags added for agent (#735) * Show message in agent add and agent list page warning user that agent has no tags to it. * Also fix string for tag color on sink page * Automated tests for delete a policy applied to an agent (#743) Automated tests for delete a policy applied to an agent * fix(orb-ui): Add missing required asterisk (*) on handler label field (#753) * doc(policy): update dataset and agent policy creation request (#635) * Feat/orb policy associate schema version (#732) * feat(orb-agent): Create schema version field on policy and receive on policy creation * feat(orb-agent): Update openapi spec to reflect schema version new field * feat(fleet): Create schema version on fleet backends return * feat(policy): Changes and enhancements after code review * Bug/orb sinker ipv6 metrics (#755) * bug(sinker): change colon for semicolon when creating prom labels (avoid problem of ipv6 hex names, which has colons) * bug(sinker) change split method on label creation * bug(sinker): improve log messages when prometheus components fail to be created * bug(sinker): change log message type to error * bug(sinker): change colon for semicolon on quatile lable * delete dataset tests (#759) * delete dataset tests * Tests/login (#761) * automated tests for login * fix(orb-ui): add 'e.g.: ' to examples (#757) * fix(orb-ui): fix error message during register (#758) * authentication module no longer expects valid token on /users endpoint * remove code for redirection on /users endpoint response - force auth * fix(orb-ui): fix agent policy not able to apply when filter is empty (#762) * fix(orb-ui): expose pactsafe env vars to docker space in local builds (#770) * fix(orb-ui): expose pactsafe env vars to docker space in local builds * building orb-ui via `make ui` -> make sure PS_SID and PS_GROUP_KEY are present in the environment. * Fix whitespace * fix(orb-ui): Fix environment setup When reading from environment, setenv.js would check if property is available, and determine wheter to produce the output to the environment file supplied to Angular App. Since Dockerfile was patched to pass down such variables, they're now always present in the docker env., though set as empty string ''. Setenv.js check changed to string comparisson. * fix(orb-ui): check only that PS can be enabled, no longer check for prod * refactor(orb-agent): enhancement on log messages (#750) * refactor(orb-agent): enhancement on log messages Create more wrappers for errors messages related to mqtt errors * feat(orb-agent): remove unnecessary param version on agent creation * Enhancement on logs for orb-agent policy removal * security(org-agent): remove from logs key which is used to log on mqtt * Fix policies migrations error (#774) * Fix migrations error setting valid value for schema_version column creation * bug(policies) make dafault value for schema_version 1.0 * Bug/orb mqtt agent group order creation (#772) * wip investigation of the problem * wip to solve bug on agent group order creation * Wip create connection with agent group on agent creation * Remove docker-compose auth changes which is unrelated this feature * Fix unit test on agent view (missing mock implemetation for RetrieveAllByAgent method) * bug(fleet): logs improvements and removal of unnecessary commets * Change agent_group_id to group_id to keep consistency * Handle error for retrieve agent group when connect agents to group channel * New version 0.12.0-develop (#777) * changing is_credentials_registered variable after register the account (#786) * get correct agent through UI (#787) * Remove orb_tag validation on agent edition (#789) * change title of prometheus configuration (#790) * Feat/orb agent handle multiple datasets (#781) * feat(orb-agent): make orb-agent handle with multiple datasets for the same policy (avoid apply policy multiple times) * initial logic for updating a policy * Create logic to handle policy updated based on version * create a timeout parameter on pktvisor request method * Enhancements after code review * removing duplicated tests (#794) * automated tests: covering integration scenarios (#799) * automated tests: covering integration scenarios * Feature/orb policies increase unit tests coverage (#782) * test(policies): improvements on unit tests coverage of data acess layer * test(policies): improvements on service unit test coverage * test(policies): improvements on unit test coverage of policies API * Orb - tests documentation (#785) * orb_int_tests_description * Feat/orb policy last modify (#800) * Fix schema_version empty valeu * create last_modified field to registry the date of policy edition * create last modified field on endpoint response * Refactor/orb sinker change top k metrics label (#795) * wip refactor prom particle label for top k metrics * minor corrections * create sinker parser for top n metrics * remove unnecessary console prints * changes requested on code review * Bootstraping unit tests to sinker process metrics | mapped dhcp metrics * Enhancements on unit tests * remove migrations from this PR * Tests/integration tests (#803) * automated tests: covering integration scenarios * fix(orb-ui): fix advancedOptions filter (#783) * fix filter advancedOptions, enabling correct rendering of advanced and non-advanced options for agent policies * Feat/orb agent reset (#816) * wip orb-agent reset logic * wip create endpoint to receive orb-agent reset request * create logic for backend reset enhancements on logs for backend reset procedure create comms for receive agent reset message * rollback build info version to unknown * changes after code review * build debug agent in ci * Avoiding NaN on prometheus sending metrics every 1 minute even when its 0 (#818) * set pktvisor port test (#823) * (hotfix): node/types specify version 12.20.42 - 12.20.43 breaks other deps (#824) * Fix dataset overriting to empty when edit a policy (#826) * Feat/orb policy last modify (#819) * Fix schema_version empty valeu * create last_modified field to registry the date of policy edition * create last modified field on endpoint response * Add last modified field for list policies endpoint * fix typo on agent tags help message (#834) * add undersocre bofore ASN (#835) * Bug/orb agent tag edition (#828) * Fix dataset overriting to empty when edit a policy * wip solve orb-agent tag edition problem * enhancements on logs for group unsubscrition * wip tags edition * code sanitization * fix group info struct name * changes after code review * fix(orb-ui): (#837) * require two names, space separated, for full user name * send full user name as signer id when contract agreement * pactsafe confirmation email set to true - user should receive a copy of contract * fix(orb-ui): (#838) * agent provisioning command to use 'develop' tagged docker image -> ns1labs/orb-agent:develop * creating unit test and fixing topASN label (#847) * fix(orb-ui): agent policy fixes (#843) * feat(orb-ui): agent policy handler add modal * wip - agent policy adjustments * fix(orb-ui): qname validator and other fixes * fix(orb-ui): #487 last modified field on policies list page (#844) * fix last modified date on agent policy list * agent subscription to multiple groups (orb-tags) and tag scenarios * agent subscription to multiple groups (orb-tags) and tag scenarios * fix(orb-ui): fix regex pattern for fullname in register form (#845) * Bug/orb mix tags (#848) * trying approach with group tags * merge tag columns to fix bug on matching tags * fix(orb-ui): fix agent policy edit failing (#855) * fix(orb-ui): Add cancel button to all stepper controls in forms (#846) * fix(orb-ui): Add cancel button to all stepper controls in forms * Agent Create/Edit * Agent Group/Edit * Agent Policy Create/Edit * Dataset Create/Edit * Sink Create/Edit * fix(orb-ui): fix strings standardization and codestyle * add separate orb-agent workflow * add push step to new orb-agent workflow * fix(orb-ui): fix validator for having at least one module in agent policy (#858) * Tests/integration tests (#871) * orb agent tags editing * Feature/orb fleet increase unit tests coverage (#801) * test(fleet): improvement on unit tests coverage of data access layer * test(fleet): improvements on agent service unit tests coverage * test(fleet): improvements on api unit tests coverage * test(fleet): increase unit tests coverage of fleet service * test(fleet): first unit test of comms service * test(fleet): improvements on service and endpoint unit tests and minor corrections on endpoint response * Test(fleet): unit tests for comms service * test(fleet): add a mock nats PubSub * edit agent name and tag scenarios (#872) * Test/orb fleet unit test mix tags (#868) * create scenarios to test mix of agent and orb tags * create more test cases for matching agents * new test case for unmatching tags * fix expected return of matching agents feature on postgres's unit test (#874) * fix broken unit tests on fleet agents (#877) * Tests/editing group tags name description (#876) * editing groups' tags, name and description * Tests/updating documentation (#881) * uptading test documentation * Bug/agent group subscription fail (#857) * wip solve agent_group bug editing tag process * connect new matching agents to group channel * fix policy apply after tags editing * handle dataset when editing agent groups * ignore conflicts on channel subscription * fix unit tests fail * create a fullList param to handle with agent policies * refactor handleAgentPolicies (reducing cyclomaticy complexity) * extract remove policy to a new method * create loop to remove policies * changes after code review (handling errors) * feat(orb-ui): Agent View Enhancements (#856) * wip * fix(orb-ui): some css and page fixes removing unecessary pieces * fix taps json output prettifier and text color * remove unecessary links to manage and edit agent properties * few more fixes and css * feat(orb-ui): * add policy state to header * fix issues with header format * fix(orb-ui): improvements to css overall * fix(orb-ui): fix agent provisioning command key for agent view * fix(orb-ui): update command provisioning card * wip * fix(orb-ui): fix observable chaining * couple fixes on provisioning command icon * fix reload and add agent group list with view and edit links * css adjustments * fix prefer interfaces over types * fix css detail on agent group link * fix(orb-ui): ironclad email issues (#888) * wip * fix code styling * update node version * update config for localhost https server for testing * wip * restore angular.json * fix(orb-ui): agent group add page matching agents table status (#891) * fix status color * fix time format for last_hb * code style fixes * fix(orb-ui): agent group matching agents list issues (#893) * wip * fix time format for last_hb * code style fixes * wip * fix bug when updating tags for agent groupings * fix(orb-ui): register redirect issue (#898) * wip - this return shouldn't be here and is certainly preventing the re-route and authentication * wip * on refresh page, _ps would become undefined * fix(orb-ui): lint missing new line (#911) * including advanced net, dhcp and dns policies to automated tests (#908) * including advanced policies to automated tests * Tests/tagging scenario (#910) * tagging scenarios with smoke, sanity and fail * fix(orb-ui): change provisioning command to be rendered as is. (#912) * Bug/orb agent status (#884) * wip solve agent_group bug editing tag process * connect new matching agents to group channel * fix policy apply after tags editing * handle dataset when editing agent groups * ignore conflicts on channel subscription * fix unit tests fail * create ticket to check agent state every 60 seconds * wip graceful agent stop * create function to check agent communication * fix broken unit tests * wip * changes after code review * call ticker stop on routine stopping * fix(orb-ui): Datatable pagination and sorting issues (#913) * query all sinks at once, filter client side, sort client side. * dynamic pagination and virtual scroll enabled. * column width NOT FLUID * table height and width with better fluidness * improve tag filter - use , or ; to split search * fix time format to display timezone and 24hour format * fix: remove extra line from provisioning command on agent view (#917) * fix(orb-ui): fix correct prop tracking 'version' (#920) * add N/A and tooltip for policies that do not have version property defined yet * show correct property 'version' on agent policies list page * fix(orb-ui): define custom tag comparator for ngx-datatables (#921) * fix order by matching agents in agent group list page * Bug missing owner_id redis sinker update (#918) * fix redis message without owner id * fix unit test * fix(orb-ui): fix handler modal button positions (#924) * fix issue #865 button order is wrong in handler modal * policy editing (#922) Edit policy name, description, handler, host_specification, bpf_filter_expression, pcap_source, only_qname_suffix, only_rcode * hotfix: fix agent list data retrieval (#925) * fix(orb-ui): (#932) * fix add metadata with fullName and company * verify that _ps is available and if not, ignore calls * fix: remote write url instead of remote host property title (#933) * Feat/orb 851 unlink removed sink from dataset and inactivate if in case (#887) * feat(dataset): invalidate dataset and unlink deleted sink * feat(dataset): add unit tests to DeleteSinkFromDataset and InactivateDatasetBySinkID * feat(dataset): fix dataset inactivate by sink ID * feat(dataset): enhancement on invalidating datasets with no sinks linked * feat(dataset): fix method name doesn't match in DeleteSinkFromAllDatasets metrics labels, fix method signature and name and add unit tests to inactivate dataset by id service * feat(dataset): rename DeleteSinkFromAllDatasets to DeleteSinkFromAllDatasetsInternal method and InactivateDatasetById to InactivateDatasetByIdInternal method * feat(dataset): fix order of InactivateDatasetByIDInternal arguments to be consistent with preexistings * test(sinker): add more unit test for sinker parser (#873) * test(sinker): add more unit tests for sinker parser * test(sinker): add unit tests for DNS and packets parser * test(sinker): add unit tests for rates and period parser * fix page object for agent view (#934) * check company and full name for registered accounts (#930) * check company and full name for registered accounts * Bug/orb fleet fail subscribe only one group (#902) * fix subscription fail when agent has only one group * wip * avoid call two times policyreq * make policy sanitization action more explicity * Feat/orb broken agent reset (#929) * Create logic to start broken backends * Create logic do restart broken agent * make pktvisor state handler on restart more explicity * change view to get mix of agents (#915) * change view to get mix of agents * solve problem to connect to channel when using auto provisioning command * code sanitization * handle db transaction on failure * Remove unnecessary transaction management and create migrations for view update * fix(orb-ui): fix broken dataset details view (#939) * method for retrieving sinks not updated after sinkService refactor. * fix(policy): set version number to display zero after creating (#926) * fix(orb-ui): fix agent view datasets showing in wrong policy nest (#943) * fix(orb-ui): fix wrong conditional in version templatecell (#944) * Tests/scenarios adjustments (#953) * adjustments made to avoid false failure * Set 0.13.0-develop release (#955) Co-authored-by: dscabral <[email protected]> Co-authored-by: gpazuch <[email protected]> Co-authored-by: mclcavalcante <[email protected]> Co-authored-by: manrodrigues <[email protected]> Co-authored-by: dscabral <[email protected]> Co-authored-by: Shannon Weyrick <[email protected]> Co-authored-by: Stéfano de Faria <[email protected]>
- Loading branch information
8 people
authored
Mar 15, 2022
1 parent
40afd52
commit f858423