NOTE: forked from https://github.com/Digital-Identity-Labs/mdqt
MDQT is small library and commandline tool to query MDQ services for SAML metadata.
You could do this with curl and xmlsec1 but it's a little more convenient to use mdqt instead.
MDQT also has features for managing local metadata files, to help when running an MDQ service or a Shibboleth IdP or SP.
MDQ currently provides these features:
- Downloading single entities, lists or aggregates
- Signature verification
- Validating metadata against SAML2 schema
- Saving metadata to disk
- Extracting entity IDs from both aggregate and individual metadata files
- Renaming metadata files to their entity ID sha1 hashes (for use with LocalDynamicMetadataProvider)
- Creating sha1 hash symlinks to metadata files (also for use with Local Dynamic Metadata)
- Listing the entity IDs of downloaded metadata files
- Showing the full URL of an entity
- Caching entity metadata and using Gzip compression
MDQ is a simple HTTP-based standard for looking up individual SAML entity metadata. Rather than regularly downloading large metadata aggregates containing thousands of entity descriptions, an IdP or SP can download the metadata for an individual entity whenever it is needed.
The UK Access Management Federation has a useful page explaining MDQ
MDQT is tested on recent MacOS and Linux, and should work with Ruby 3.0.0 or later and recent JRuby releases.
To install system-wide on your default Ruby, use
sudo gem install mdqt
If using a per-user Ruby via rbenv, asdf or similar, you'll need
gem install mdqt
To add MDQT to a Ruby project include this line in your application's Gemfile
gem 'mdqt'and then execute:
bundle
(Experimental) See the instructions at MDQT-Container
MDQT can check that metadata has not been tampered with by verifying its signature. Some MDQ services use unencrypted HTTP connections and rely on signed metadata.
MDQT supports signature verification but requires a Ruby library called Nokogiri to do the hard work. Nokogiri is fast and useful but can sometimes be awkward to install for non-developers (it can sometimes require a C development environment and various XML libraries). In most cases Nokogiri will install automatically, without problems, when you install MDQT, but if you encounter any problems installing Nokogiri the Installing Nokogiri documentation is very helpful.
You can see a list of commandline options by typing:
mdqt help
To see more information about a command, use the --help option after the command or type help <command>:
mdqt help get
You can specify the MDQ service with a commandline option:
mdqt get --service https://mdq.example.com/mdq http://entity.ac.uk/shibboleth
It's more convenient to set an environment variable to specify a default MDQ
service. Set MDQT_SERVICE or MDQ_BASE_URL to the base URL of your MDQ service.
export MDQT_SERVICE=https://mdq.example.com/mdq
mdqt get http://entity.ac.uk/shibboleth
mdqt get http://example.org/service
Finally, if you don't specify an MDQ service with --service or MDQT_SERVICE then mdqt might be
able to guess your local NREN's MDQ service. Do not do this in production!
If an MDQ service is known to MDQT it can be selected using an alias:
mdqt get --service incommon http://entity.edu/shibboleth
You can see known services and their aliases using mdqt services
Downloading entity metadata to STDOUT:
mdqt get https://test-idp.ukfederation.org.uk/idp/shibboleth
Using the sha1 hashed version of entity IDs requires quotes or escaping in some shells:
mdqt get "{sha1}52e2065fc0d53744e8d4ee2c2f30696ebfc5def9"
mdqt get \{sha1\}52e2065fc0d53744e8d4ee2c2f30696ebfc5def9
mdqt get [sha1]52e2065fc0d53744e8d4ee2c2f30696ebfc5def9
Requesting all metadata from an MDQ endpoint is done by specifying --all:
mdqt get --all
Caching can be enabled using --cache. At the moment the mdqt executable
only supports caching to disk. It will create a cache directory in your temporary
directory.
mdqt get --cache --service https://mdq.example.com/mdq http://entity.ac.uk/shibboleth
Caching is now on by default. To force a single command to not use the cache, include --reset
mdqt get --reset --service https://mdq.example.com/mdq http://entity.ac.uk/shibboleth
You can clear the cache by using the reset command:
mdqt reset
If you have enabled verification by installing xmldsig (and have downloaded and checked a suitable
certificate for your MDQ server) you can require verification by passing
they verify-with flag with the path of your certificate.
mdqt get --verify-with myfederation.pem https://indiid.net/idp/shibboleth
It's possible to pass more than one certificate by separating them with commas
mdqt get --verify-with myfederation.pem,previous.pem https://indiid.net/idp/shibboleth
Basic XML correctness and validation against SAML2 Metadata schema can be enabled with the
--validate switch:
mdqt get --validate https://indiid.net/idp/shibboleth
If you need to check metadata that has already been downloaded then try the check
command:
mdqt check metadata.xml # Just validate
mdqt check --verify-with myfederation.pem metadata.xml # Verify signature too
You shouldn't need to validate XML from a trusted MDQ service such as one run by a national federation. You should however always verify the signature of XML sent over an unencrypyted HTTP connection, and probably even over HTTPS. MDQT's validation check is mostly for use when writing or debugging your own MDQ service.
The simplest way to save metadata is to redirect output from the get command:
mdqt get http://entity.ac.uk/shibboleth > metadata.xml
MDQT also offers the --save-to option to write all metadata into a directory
mdqt get http://entity.ac.uk/shibboleth --save-to metadata_directory
The --save-to option requires a directory to be specified. All files will be saved
with a name based on their transformed identifier (sha1 hash) such as
77603e0cbda1e00d50373ca8ca20a375f5d1f171.xml
For more information about current settings, download results, and so on, add
--verbose to commands:
$mdqt get --verbose http://entity.ac.uk/shibboleth
To convert normal URI entity IDs into MDQ SHA1 hashed transformed identifiers use the transform command:
mdqt transform http://example.org/service
Transforming identifiers that have already been transformed should not re-transform them.
To see more details of what is being sent and received by a get command add the --explain flag
mdqt get --explain --service https://mdq.example.com/mdq http://entity.ac.uk/shibboleth
MDQT will then show a table of sent and received headers which may be useful when debugging servers.
To extract a list of all entity IDs from a file:
mdqt entities metadata.xml
mdqt entities --sha1 metadata.xml
To create sha1 symlinks to a metadata file:
mdqt ln example_idp.xml
To rename a file to its entity ID sha1 has:
mdqt rename example_idp.xml
To list the entity IDs of files in a directory:
mdqt ls
To list all entities available at an MDQ service:
mdqt list
To show the MDQ services known to MDQT, and their aliases:
mdqt services
To show the full MDQ URL of an entity
mdqt url http://entity.ac.uk/shibboleth
MDQT can accept input on stdin, allowing composition and pipelining
cat list_of_ids.txt | mdqt url
mdqt list | grep cern.ch | mdqt get --save-to cern_metadata/ --list | mdqt ls
- SAML Library is a set of scripts to help with metadata-related tasks, written by Tom Scavo of Internet2. Some of the scripts provide similar functionality to MDQT, and are designed to be piped together.
Please don't! We originally had plans to include a usable generic library was part of MDQT but unless there's new demand for it that's now unlikely to happen. However, we do now have a set of libraries for the Elixir language, based around Smee - not very helpful for Ruby projects but possibly of use for new projects.
After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install.
Bug reports and pull requests are welcome on GitHub at https://github.com/Digital-Identity-Labs/mdqt. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
The gem is available as open source under the terms of the MIT License.