Skip to content

Latest commit

 

History

History
111 lines (84 loc) · 3.26 KB

configuration.md

File metadata and controls

111 lines (84 loc) · 3.26 KB
Raw

Configuration

The configuration of the Archive Interface service is done in two files. The REST API utilizes CherryPy and review of their configuration documentation is recommended. The service configuration file is a INI formatted file containing configuration for accessing the HSM backend.

CherryPy Configuration File

An example of Archive Interface server CherryPy configuration:

[global]
log.screen: True
log.access_file: 'access.log'
log.error_file: 'error.log'
server.socket_host: '0.0.0.0'
server.socket_port: 8080

[/]
request.dispatch: cherrypy.dispatch.MethodDispatcher()
tools.response_headers.on: True
tools.response_headers.headers: [('Content-Type', 'application/json')]

Service Configuration File

The service configuration is an INI file and an example is as follows:

[posix]
; use id2filename method when accessing a posix endpoint
use_id2filename = false

[hpss]
; IBM HPSS HSM settings
auth = /var/hpss/etc/hpss.unix.keytab
sitename = example.com
user = hpss.unix

[hsm_sideband]
; Oracle HSM Sideband database settings (MySQL)
sam_qfs_prefix = /tmp/path
schema = schema_name
host = host
user = user
password = pass
port = 3306

ID Mapping to File Names

The Pacifica software depends on a flat ID space for indexing files. This needs to map to filenames on the backend storage in a nice way. To limit the number of files in a single directory (or number of directories in a directory) we use the algorithm in archiveinterface.id2filename. This takes a number and breaks it into bytes. Each byte is then represented in hex and used to build the directory tree.

For example id2filename(12345) becomes /39/3039 on the backend file system. See more documentation in the id2filename module.

Running It

There are two ways of running the archive interface, using CherryPy builtin server and uWSGI to run the server.

CherryPy does provide a server and this works for some workloads.

pacifica-archiveinterface -t posix -p 8080 -a 127.0.0.1 --prefix /path

uWSGI is recommended for this service as it performs a lot better for higher throughput workloads.

export PAI_BACKEND_TYPE=posix
export PAI_PREFIX=/path
uwsgi --http-socket :8080 --master -p 8 --module pacifica.archiveinterface.wsgi

Post Deployment Testing

Inside the post_deployment_test directory there is a file called deployment_test.py This file will run a series of tests against a deployed archive interface. The test are ordered so that they post, stage, status, and get files properly. There are a few global variables at the top of the file that need to be adjusted to each deployment.

Variables

export ARCHIVE_URL='http://127.0.0.1:8080/'
export LARGE_FILE_SIZE=$(( 1024 * 1024 * 1024))
export MANY_FILES_TEST_COUNT=1000
  • ARCHIVE_URL is the URL to the newly deployed archive_interface
  • LARGE_FILE_SIZE is the size of the large file to test with (default 1Gib)
  • MANY_FILES_TEST_COUNT is the number of small files to spam (default 1000)

Running

pytest -v post_deployment_tests/deployment_test.py

Output will be the status of the tests against the archive interface