Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elasticsearch enum module enhancements #18310

Merged
merged 6 commits into from
Aug 24, 2023
Merged

Conversation

h00die
Copy link
Contributor

@h00die h00die commented Aug 23, 2023

The original elasticsearch module had a few issues:

  1. it didn't use any auth and couldn't have any added
  2. it checked for indicies via aliases and that was it, missing substantial amounts of potential info gathering

This PR enhances the module to:

  1. accept creds
  2. pull version info against the host
  3. Pull info about the cluster
  4. pull info about each of the nodes in the cluster
  5. enumerate users
  6. allow the user to specify a number of rows to query in each index, and save them as a CSV for data sampling.

Verification

  • Install the docker/docker-compose ES noted in the docs, this will populate some data so you test everything appropriately
  • Start msfconsole
  • use auxiliary/scanner/elasticsearch/enum
  • set ssl false
  • set username elastic
  • set password esbackup-password
  • set rhosts 127.0.0.1
  • run
  • Verify you get tables with server version info, cluster info, node info, user info, and data from tables

@h00die
Copy link
Contributor Author

h00die commented Aug 23, 2023

@jmartin-tech I believe this is more of a 'gather' module than a 'scanner' at this point, and think it should be moved, but wanted to get other's thoughts.

@jmartin-tech
Copy link
Contributor

I would agree, this makes sense as gather I don't see the Scanner mixin bringing much to this module since support for RHOSTS would now allow range based processing. Should probably do a little validation though to be sure there is not some additional magic info stored in the db that gets lost when converted to gather.

Copy link
Contributor

@smcintyre-r7 smcintyre-r7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a bunch of comments mostly related to newish patterns in MSF. The three themes were:

When making a request with send_request_cgi (or raw really):

  1. #normalize_uri should be used with the TARGETURI datastore option so the module can be fully targeted towards a system with a non-default root path
  2. There should not be any exception handling, it's only necessary to check that res is nil and the status code
  3. JSON data should be parsed with res.get_json_document

modules/auxiliary/gather/elasticsearch_enum.rb Outdated Show resolved Hide resolved
modules/auxiliary/gather/elasticsearch_enum.rb Outdated Show resolved Hide resolved
modules/auxiliary/gather/elasticsearch_enum.rb Outdated Show resolved Hide resolved
modules/auxiliary/gather/elasticsearch_enum.rb Outdated Show resolved Hide resolved
modules/auxiliary/gather/elasticsearch_enum.rb Outdated Show resolved Hide resolved
modules/auxiliary/gather/elasticsearch_enum.rb Outdated Show resolved Hide resolved
modules/auxiliary/gather/elasticsearch_enum.rb Outdated Show resolved Hide resolved
modules/auxiliary/gather/elasticsearch_enum.rb Outdated Show resolved Hide resolved
Comment on lines 80 to 89
elastic_table = Rex::Text::Table.new(
'Header' => "#{index} Data",
'Indent' => 2,
# we know at least 1 row since we wouldn't query an index w/o a row
'Columns' => json_body.dig('hits', 'hits')[0]['_source'].keys
)

json_body.dig('hits', 'hits').each do |hash|
elastic_table << hash['_source'].values
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While testing this module I ran into this stack trace here where hash['_source'].values.length was less than json_body.dig('hits', 'hits')[0]['_source'].keys.

Here's the stack trace:

[+] .kibana-event-log-7.9.1-000030 Data
===================================

  event                                           message            @timestamp                ecs                   kibana
  -----                                           -------            ----------                ---                   ------
  {"provider"=>"eventLog", "action"=>"starting"}  eventLog starting  2023-06-09T15:12:52.818Z  {"version"=>"1.5.0"}  {"server_uuid"=>"480b1a88-e0c9-43f3-87c9-8734570c0acf"}

[-] Auxiliary failed: Rex::RuntimeError Invalid number of columns!
[-] Call stack:
[-]   /home/smcintyre/.rvm/gems/ruby-3.0.4@metasploit-framework/gems/rex-text-0.2.52/lib/rex/text/wrapped_table.rb:194:in `add_row'
[-]   /home/smcintyre/.rvm/gems/ruby-3.0.4@metasploit-framework/gems/rex-text-0.2.52/lib/rex/text/wrapped_table.rb:186:in `<<'
[-]   /home/smcintyre/Repositories/metasploit-framework.pr/modules/auxiliary/gather/elasticsearch_enum.rb:88:in `block in get_results'
[-]   /home/smcintyre/Repositories/metasploit-framework.pr/modules/auxiliary/gather/elasticsearch_enum.rb:87:in `each'
[-]   /home/smcintyre/Repositories/metasploit-framework.pr/modules/auxiliary/gather/elasticsearch_enum.rb:87:in `get_results'
[-]   /home/smcintyre/Repositories/metasploit-framework.pr/modules/auxiliary/gather/elasticsearch_enum.rb:165:in `block in get_indices'
[-]   /home/smcintyre/Repositories/metasploit-framework.pr/modules/auxiliary/gather/elasticsearch_enum.rb:164:in `each'
[-]   /home/smcintyre/Repositories/metasploit-framework.pr/modules/auxiliary/gather/elasticsearch_enum.rb:164:in `get_indices'
[-]   /home/smcintyre/Repositories/metasploit-framework.pr/modules/auxiliary/gather/elasticsearch_enum.rb:397:in `run'
[*] Auxiliary module execution completed
msf6 auxiliary(gather/elasticsearch_enum) > 

And here's what I did to fix it:

Suggested change
elastic_table = Rex::Text::Table.new(
'Header' => "#{index} Data",
'Indent' => 2,
# we know at least 1 row since we wouldn't query an index w/o a row
'Columns' => json_body.dig('hits', 'hits')[0]['_source'].keys
)
json_body.dig('hits', 'hits').each do |hash|
elastic_table << hash['_source'].values
end
columns = json_body.dig('hits', 'hits')[0]['_source'].keys
elastic_table = Rex::Text::Table.new(
'Header' => "#{index} Data",
'Indent' => 2,
# we know at least 1 row since we wouldn't query an index w/o a row
'Columns' => columns
)
json_body.dig('hits', 'hits').each do |hash|
elastic_table << columns.map { |column| hash['_source'][column] }
end

Copy link
Contributor

@smcintyre-r7 smcintyre-r7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So when I target an elasticsearch instance I use for things that has some data in it, the modules output is effectively unustable. The columns span so wide and the output scrolls way off the screen. I'd recommend simply printing less and not going into pulling the individual records themselves.

Comment on lines +374 to +383
json_body.each do |username, attributes|
elastic_table << [
username,
attributes['roles'],
attributes['email'],
attributes['metadata'],
attributes['enabled'],
]
end
print_good(elastic_table.to_s)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting another exception here. I don't think I have any users setup.

[-] Auxiliary failed: NoMethodError undefined method `each' for nil:NilClass
[-] Call stack:
[-]   /home/smcintyre/Repositories/metasploit-framework.pr/modules/auxiliary/gather/elasticsearch_enum.rb:375:in `get_users'
[-]   /home/smcintyre/Repositories/metasploit-framework.pr/modules/auxiliary/gather/elasticsearch_enum.rb:399:in `run'
[*] Auxiliary module execution completed
msf6 auxiliary(gather/elasticsearch_enum) > 

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more likely the user you gave didn't have permission to view that data, which is a good case to test

@smcintyre-r7 smcintyre-r7 self-assigned this Aug 24, 2023
Copy link
Contributor

@smcintyre-r7 smcintyre-r7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for those updates, it's working much better now so I'll get this landed.

msf6 auxiliary(gather/elasticsearch_enum) > run
[*] Running module against 192.168.249.3

[+] Elastic Information
===================

  Name          Cluster Name    Version  Build Type  Lucene Version
  ----          ------------    -------  ----------  --------------
  b6b69aad1328  docker-cluster  7.9.1    docker      8.6.2

[+] Node Information
================

  IP         Transport Port  HTTP Port       Version  Name          Uptime  Ram Usage    Node Role  Master  CPU Load  Disk Usage
  --         --------------  ---------       -------  ----          ------  ---------    ---------  ------  --------  ----------
  10.88.0.3  9300            10.88.0.3:9200  7.9.1    b6b69aad1328  25.1d   3.3gb/3.8gb  dilmrt     *       1%        29.6gb/49.1gb

[+] Cluster Information
===================

  Cluster Name    Status  Number of Nodes
  ------------    ------  ---------------
  docker-cluster  yellow  1

[+] Indicies Information
====================

  Name                            Health  Status  UUID                    Documents  Storage Usage (MB)
  ----                            ------  ------  ----                    ---------  ------------------
  .apm-agent-configuration        green   open    HWuN0mCoRrS9p7dF6GmGgQ  0          0MB
  .apm-custom-link                green   open    QgNjk6s4Tpm8nu5bI84tUQ  0          0MB
  .async-search                   green   open    NLQRQdm1SqmSUPbWbQVBRg  0          0MB
  .kibana-event-log-7.9.1-000030  green   open    UsoR3jPrROifPpbBg77Yiw  1          0MB
  .kibana-event-log-7.9.1-000031  green   open    IGbdcS29SUiUt5byJ4sHUg  1          0MB
  .kibana-event-log-7.9.1-000032  green   open    tR_JqMG4Tv68ctolHynA_Q  0          0MB
  .kibana-event-log-7.9.1-000033  green   open    pZYjn3Z3REmlllDcKx35Vw  0          0MB
  .kibana_1                       green   open    YGu3riAEQqeqy322DEP4Qg  243        10MB
  .kibana_task_manager_1          green   open    KDPL7nunRQ6rQYgj_Qc9PQ  6          11MB
  .reporting-2020-09-13           green   open    Thglhp-8TmODvy22KBUC-g  1          0MB
  .reporting-2021-06-20           green   open    eEP_GjxnQ_mEEee9SfBUbQ  1          0MB
  .reporting-2022-04-03           green   open    Vbi_VAl2TeiC3tqUWUWNqg  2          0MB
  .reporting-2022-11-06           green   open    dO0O8FPbTl2v28FpRrJDog  3          0MB
  .reporting-2022-11-13           green   open    PO-9xMNHQFuGjX0whtkR-A  1          0MB
  .reporting-2022-11-27           green   open    y9yZlMyFTK-u1xRJfd79hQ  1          0MB
  comicsdb                        yellow  open    2oEl-DaARy-Gq9jTaLiBFA  2460       0MB
  github-api-v3.issues            yellow  open    mU0kc2N1TWekqIZaHFFaUw  21469      68MB
  github-api-v3.pull-requests     yellow  open    -aX4LXhyTZKnTCoMUxM95Q  14482      151MB
  github-api-v3.repositories      yellow  open    sX0RvG1rReOeLiNB1A5psQ  73         0MB
  metasploit.modules              yellow  open    sGRIQZD5TS-XVkW9CX3-Gg  4452       6MB

[+] .kibana-event-log-7.9.1-000031 data stored to /home/smcintyre/.msf4/loot/20230824183003_default_192.168.249.3_elasticserch.ind_274242.csv
[+] .kibana-event-log-7.9.1-000030 data stored to /home/smcintyre/.msf4/loot/20230824183003_default_192.168.249.3_elasticserch.ind_984103.csv
[+] metasploit.modules data stored to /home/smcintyre/.msf4/loot/20230824183003_default_192.168.249.3_elasticserch.ind_701298.csv
[+] .reporting-2022-11-13 data stored to /home/smcintyre/.msf4/loot/20230824183003_default_192.168.249.3_elasticserch.ind_577512.csv
[+] comicsdb data stored to /home/smcintyre/.msf4/loot/20230824183003_default_192.168.249.3_elasticserch.ind_265916.csv
[+] github-api-v3.pull-requests data stored to /home/smcintyre/.msf4/loot/20230824183004_default_192.168.249.3_elasticserch.ind_230802.csv
[+] .reporting-2020-09-13 data stored to /home/smcintyre/.msf4/loot/20230824183004_default_192.168.249.3_elasticserch.ind_800440.csv
[+] .kibana_1 data stored to /home/smcintyre/.msf4/loot/20230824183014_default_192.168.249.3_elasticserch.ind_704807.csv
[+] .reporting-2021-06-20 data stored to /home/smcintyre/.msf4/loot/20230824183016_default_192.168.249.3_elasticserch.ind_102443.csv
[+] .reporting-2022-04-03 data stored to /home/smcintyre/.msf4/loot/20230824183016_default_192.168.249.3_elasticserch.ind_838023.csv
[+] .reporting-2022-11-06 data stored to /home/smcintyre/.msf4/loot/20230824183017_default_192.168.249.3_elasticserch.ind_740292.csv
[+] .kibana_task_manager_1 data stored to /home/smcintyre/.msf4/loot/20230824183017_default_192.168.249.3_elasticserch.ind_244783.csv
[+] .reporting-2022-11-27 data stored to /home/smcintyre/.msf4/loot/20230824183017_default_192.168.249.3_elasticserch.ind_408942.csv
[+] github-api-v3.repositories data stored to /home/smcintyre/.msf4/loot/20230824183017_default_192.168.249.3_elasticserch.ind_716946.csv
[+] github-api-v3.issues data stored to /home/smcintyre/.msf4/loot/20230824183017_default_192.168.249.3_elasticserch.ind_354774.csv
[-] Unable to pull user data
[*] Auxiliary module execution completed

@smcintyre-r7 smcintyre-r7 merged commit b830587 into rapid7:master Aug 24, 2023
@smcintyre-r7
Copy link
Contributor

Release Notes

This updates the Elasticsearch auxiliary module. It has been renamed to elastic_enum, accepts credentials and will store data to disk that is pulled from the target.

@h00die h00die deleted the elastic_enum branch August 25, 2023 01:20
@bwatters-r7 bwatters-r7 added rn-modules release notes for new or majorly enhanced modules rn-enhancement release notes enhancement and removed rn-modules release notes for new or majorly enhanced modules labels Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants