Use sessions config and GeoIP download failure (#801)

* Adding user session config management Fixed Maxmind GeoIP download. Fixed tilookup to honor ¨progress¨ parameter * Fixing future annotations errors in tests * Remove spurious bandit warnings * Fixing docs build errors * More doc errors * Updating version to 2.14.0 * Fixing name of Timeline-01.png * Fixing some grammatical errors * Fixing comments from review. * Fixing unit test break for mp_user_session Adding ruff to pre-commit-config.yaml
microsoft · Oct 21, 2024 · e9adaff · e9adaff
1 parent 4d715ad
commit e9adaff
Show file tree

Hide file tree

Showing 50 changed files with 860 additions and 152 deletions.
diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml
@@ -204,7 +204,7 @@ jobs:
         if: ${{ always() }}
       - name: bandit
         run: |
-          bandit -x tests -r -s B303,B404,B603,B607,B608 msticpy
+          bandit -x tests -r -s B303,B404,B603,B607,B608,B113 msticpy
         if: ${{ always() }}
       - name: flake8
         run: |

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -41,10 +41,16 @@ repos:
       - id: pydocstyle
         args:
           - --convention=numpy
-  # - repo: https://github.com/pre-commit/mirrors-mypy
-  #   rev: v1.0.1
-  #   hooks:
-  #     - id: mypy
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    # Ruff version.
+    rev: v0.7.0
+    hooks:
+      # Run the linter.
+      - id: ruff
+        types_or: [ python, pyi, jupyter ]
+        args:
+        - msticpy
+        - --fix
   - repo: local
     hooks:
       - id: check_reqs_all

diff --git a/conda/conda-reqs-dev.txt b/conda/conda-reqs-dev.txt
@@ -26,6 +26,7 @@ pyroma>=3.1
 pytest-check>=1.0.1
 pytest-xdist>=2.5.0
 respx>=0.20.1
+ruff>=0.6.6
 sphinx_rtd_theme>=0.5.1
 sphinx>=2.1.2
 virtualenv

diff --git a/docs/requirements.txt b/docs/requirements.txt
@@ -1,4 +1,8 @@
 attrs>=18.2.0
+azure.mgmt.network
+azure.mgmt.resource
+azure.mgmt.monitor
+azure.mgmt.compute
 cryptography
 deprecated>=1.2.4
 docutils<0.22.0

diff --git a/docs/source/GettingStarted.rst b/docs/source/GettingStarted.rst
@@ -11,6 +11,7 @@ Getting Started
    getting_started/PackageSummary
    getting_started/msticpyconfig
    getting_started/SettingsEditor
+   getting_started/UserSessionConfig
    getting_started/JupyterAndSecurity
    getting_started/JupyterAndAzureSentinel
    getting_started/AzureAuthentication
diff --git a/docs/source/api/msticpy.init.mp_user_session.rst b/docs/source/api/msticpy.init.mp_user_session.rst
@@ -0,0 +1,7 @@
+msticpy.init.mp\_user\_session module
+=====================================
+
+.. automodule:: msticpy.init.mp_user_session
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/docs/source/api/msticpy.init.rst b/docs/source/api/msticpy.init.rst
@@ -26,6 +26,7 @@ Submodules
    msticpy.init.logging
    msticpy.init.mp_pandas_accessors
    msticpy.init.mp_plugins
+   msticpy.init.mp_user_session
    msticpy.init.nbinit
    msticpy.init.nbmagics
    msticpy.init.pivot

diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -269,6 +269,7 @@
     "nest_asyncio",
     "networkx",
     "openpyxl",
+    "packaging",
     "panel",
     "passivetotal",
     "pygeohash",

diff --git a/docs/source/data_acquisition/GeoIPLookups.rst b/docs/source/data_acquisition/GeoIPLookups.rst
@@ -33,6 +33,10 @@ service. Again, the paid tiers offer greater accuracy, more detailed
 information and higher throughput. Please check out their site for more
 details.
 
+.. note:: The free tier of IPStack is now extremely limited and not
+  practical for use other than brief experimentation. If you want to
+  use this option, you should sign up for a paid tier of service.
+
 Importing the GeoIP classes
 ---------------------------
 
@@ -61,7 +65,7 @@ Maxmind Geo-IP Lite Lookup Class
 
 See :py:class:`GeoLiteLookup<msticpy.context.geoip.GeoLiteLookup>`
 
-.. note:: Maxmind now require an API Key to download database
+.. note:: Maxmind requires an Account ID and API Key to download database
    updates. You can create a free account or opt for a paid tier,
    which gives you greater accuracy and more features.
 
@@ -78,11 +82,14 @@ The example shown here shows part of the ``OtherProviders`` section of
 msticpyconfig.yaml. You can specify an API key in the ``AuthKey`` setting.
 For example, ``AuthKey: abcd424246789`` or use a reference to an
 environment variable holding the key value.
-The API key you need to specify in the ``AuthKey`` setting is you MaxMind
-License Key that can be found on the MaxMind website under Account > Services.
+The API key you need to specify in the ``AuthKey`` setting is your MaxMind
+License Key that can be found on the MaxMind website under Account/Services.
+Set the ``AccountID`` field to your MaxMind Account ID. (this is typically
+not a secret value but you can opt to store this in an environment variable
+or Azure Key Vault).
 
 The DBFolder setting specifies a folder where the downloaded Maxmind
-database files will be stored and referenced from. Thefolder path
+database files will be stored and referenced from. The folder path
 can be prefixed with "~" to specify a path relative to the current
 users home directory (this works cross-platform).
 
@@ -95,6 +102,7 @@ users home directory (this works cross-platform).
     OtherProviders:
       GeoIPLite:
         Args:
+          AccountID: "1234567"
           AuthKey:
             EnvironmentVar: "MAXMIND_AUTH"
           DBFolder: "~/.msticpy"
@@ -121,7 +129,7 @@ not work reliably cross-platform.
 
 .. code:: ipython3
 
-    iplocation = GeoLiteLookup(api_key="mykey", db_folder="/tmp/mmdb")
+    iplocation = GeoLiteLookup(api_key="mykey", account_id="1234567", db_folder="/tmp/mmdb")
 
 
 GeoLite Usage
@@ -138,6 +146,7 @@ You can also supply options to customize the behavior of the
 local maxmind database.
 
 * ``api_key``: described above
+* ``account_id``: described above
 * ``db_folder`` : Specify custom path containing local Maxmind city
   database. If not specified, download to .msticpy dir under user's home
   directory.