Send Sir Perceval on a quest to retrieve and gather data from software repositories.
usage: perceval [-g] <backend> [<args>] | --help | --version | --list
Send Sir Perceval on a quest to retrieve and gather data from software
repositories.
Repositories are reached using specific backends. The most common backends
are:
askbot Fetch questions and answers from Askbot site
bugzilla Fetch bugs from a Bugzilla server
bugzillarest Fetch bugs from a Bugzilla server (>=5.0) using its REST API
confluence Fetch contents from a Confluence server
discourse Fetch posts from Discourse site
dockerhub Fetch repository data from Docker Hub site
gerrit Fetch reviews from a Gerrit server
git Fetch commits from Git
github Fetch issues, pull requests and repository information from GitHub
gitlab Fetch issues, merge requests from GitLab
gitter Fetch messages from a Gitter room
googlehits Fetch hits from Google API
groupsio Fetch messages from Groups.io
hyperkitty Fetch messages from a HyperKitty archiver
jenkins Fetch builds from a Jenkins server
jira Fetch issues from JIRA issue tracker
launchpad Fetch issues from Launchpad issue tracker
mattermost Fetch posts from a Mattermost server
mbox Fetch messages from MBox files
mediawiki Fetch pages and revisions from a MediaWiki site
meetup Fetch events from a Meetup group
nntp Fetch articles from a NNTP news group
pagure Fetch issues from Pagure
phabricator Fetch tasks from a Phabricator site
pipermail Fetch messages from a Pipermail archiver
redmine Fetch issues from a Redmine server
rocketchat Fetch messages from a Rocket.Chat channel
rss Fetch entries from a RSS feed server
slack Fetch messages from a Slack channel
stackexchange Fetch questions from StackExchange sites
supybot Fetch messages from Supybot log files
telegram Fetch messages from the Telegram server
twitter Fetch tweets from the Twitter Search API
optional arguments:
-h, --help show this help message and exit
-v, --version show version
-g, --debug set debug mode on
-l, --list show available backends
Run 'perceval <backend> --help' to get information about a specific backend.
- Python >= 3.8
- Poetry >= 1.2
- git
- build-essential
You will also need some other libraries for running the tool, you can find the whole list of dependencies in pyproject.toml file.
- build-essentials
Build-essentials is a package that contains a set of tools to compile and build software. It is required to work with Debian packages.
$ sudo apt-get install build-essential
- git
Git is a version control system that allows you to keep track of changes in your code. It is required to work with Git repositories.
$ sudo apt-get install git
There are several ways to install Perceval on your system: packages or source code using Poetry or pip or using Docker.
Perceval can be installed using pip, a tool for installing Python packages. To do it, run the next command:
$ pip install perceval
To install from the source code you will need to clone the repository first:
$ git clone https://github.com/chaoss/grimoirelab-perceval
$ cd grimoirelab-perceval
Then use pip or Poetry to install the package along with its dependencies.
To install the package from local directory run the following command:
$ pip install .
In case you are a developer, you should install perceval in editable mode:
$ pip install -e .
We use poetry for dependency management and packaging. You can install it following its documentation. Once you have installed it, you can install perceval and the dependencies in a project isolated environment using:
$ poetry install
To spaw a new shell within the virtual environment use:
$ poetry shell
A Perceval Docker image is available at DockerHub.
Detailed information on how to run and/or build this image can be found here.
Documentation is generated automatically in the ReadTheDocs Perceval site.
If you use Perceval in your research papers, please refer to Perceval: software project data at your will -- Pre-print:
Dueñas, S., Cosentino, V., Robles, G., & Gonzalez-Barahona, J. M. (2018, May). Perceval: software project data at your will. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings (pp. 1-4). ACM.
@inproceedings{duenas2018perceval,
title={Perceval: software project data at your will},
author={Due{\~n}as, Santiago and Cosentino, Valerio and Robles, Gregorio and Gonzalez-Barahona, Jesus M},
booktitle={Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings},
pages={1--4},
year={2018},
organization={ACM}
}
$ perceval askbot 'http://askbot.org/' --from-date '2016-01-01'
To fetch bugs from Bugzilla, you have two options:
a) Use the traditional backend
$ perceval bugzilla 'https://bugzilla.redhat.com/' --backend-user user --backend-password pass --from-date '2016-01-01'
b) Use the REST API backend for Buzilla 5.0 (or higher) servers. We strongly recommend this backend when data is fetched from version servers >=5.0 because the retrieval process is much faster.
$ perceval bugzillarest 'https://bugzilla.mozilla.org/' --backend-user user --backend-password pass --from-date '2016-01-01'
$ perceval confluence 'https://wiki.opnfv.org/' --from-date '2016-01-01'
$ perceval discourse 'https://foro.mozilla-hispano.org/' --from-date '2016-01-01'
$ perceval dockerhub grimoirelab perceval
To run gerrit, you will need an authorized SSH private key:
$ eval `ssh-agent -s`
$ ssh-add ~/.ssh/id_rsa
Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)
To run the backend, execute the next command:
$ perceval gerrit --user user 'review.openstack.org' --from-date '2016-01-01'
To run this backend execute the next command. Take into account that to run this backend Git program has to be installed on your system.
$ perceval git 'https://github.com/chaoss/grimoirelab-perceval.git' --from-date '2016-01-01'
To run the backend against a private git repository, you must pass the credentials directly in the URL:
$ perceval git https://<username>:<password>@repository-url
For example, for private GitHub repositories:
$ perceval git https://<username>:<api-token>@github.com/chaoss/grimoirelab-perceval
Git backend can also work with a Git log file as input. We recommend to use the next command to get the most complete log file.
$ git log --raw --numstat --pretty=fuller --decorate=full --parents --reverse --topo-order -M -C -c --remotes=origin --all > /tmp/gitlog.log
Then, to run the backend, just execute any of the next commands:
$ perceval git --git-log '/tmp/gitlog.log' 'file:///myrepo.git'
or
$ perceval git '/tmp/gitlog.log'
$ perceval github elastic logstash --from-date '2016-01-01'
The GitHub backend accepts the categories issue, pull_request and
repository which allow to fetch the specific data.
$ perceval github --category issue elastic logstash
$ perceval gitlab fdroid fdroiddata -t $GITLAB_TOKEN --from-date '2016-01-01'
$ perceval gitter -t 'abcdefghi' --from-date '2020-03-18' 'jenkinsci' 'jenkins'
$ perceval googlehits "bitergia grimoirelab"
$ perceval groupsio 'updates' -e '<[email protected]>' -p 'my-password' --from-date '2016-01-01'
In order to fetch the data from a group, you should first subscribe to it via the Groups.io website. In case you want to know the group names where you are subscribed, you can use the following script: https://gist.github.com/valeriocos/ad33a0b9b2d13a8336230c8c59df3c55
$ perceval hyperkitty 'https://lists.mailman3.org/archives/list/[email protected]' --from-date 2017-01-01
$ perceval jenkins 'https://build.opnfv.org/ci/'
$ perceval jira 'https://tickets.puppetlabs.com' --project PUP --from-date '2016-01-01'
$ perceval launchpad ubuntu --from-date '2016-01-01'
$ perceval mattermost 'http://mattermost.example.com' jgw7jdmjkjf19ffkwnw59i5f9e --from-date '2016-01-01' -t 'abcdefghijk'
$ perceval mbox 'http://example.com' /tmp/mboxes/
$ perceval mediawiki 'https://wiki.mozilla.org' --from-date '2016-06-30'
$ perceval meetup 'Software-Development-Analytics' --from-date '2016-06-01' -t abcdefghijk
$ perceval nntp 'news.mozilla.org' 'mozilla.dev.project-link' --offset 10
$ perceval pagure '389-ds-base' --from-date '2020-03-06'
$ perceval phabricator 'https://secure.phabricator.com/' -t 123456789abcefe
$ perceval pipermail 'https://mail.gnome.org/archives/libart-hackers/'
Pipermail also is able to fetch data from Apache's mod_box interface:
$ perceval pipermail 'http://mail-archives.apache.org/mod_mbox/httpd-dev/'
$ perceval redmine 'https://www.redmine.org/' --from-date '2016-01-01' -t abcdefghijk
Rocket.Chat backend needs an API token and a User Id to authenticate to the server.
$ perceval rocketchat -t 'abchdefghij' -u '1234abcd' --from-date '2020-05-02' https://open.rocket.chat general
$ perceval rss 'https://blog.bitergia.com/feed/'
Slack backend requires an API token for authentication. Slack apps can be used
to generate and configure this API token. The scopes required by a Slack app for
the backend are channels:history, channels:read and users:read. To know
more about Slack apps and its integration please refer the Slack apps
documentation. For more information about
the scopes required by a Slack app please refer the Scopes and permissions
documentation.
The following script can also be used to generate an OAuth2 token to access the Slack API.
$ perceval slack C0001 --from-date 2016-01-12 -t abcedefghijk
$ perceval stackexchange --site stackoverflow --tagged python --from-date '2016-01-01' -t abcdabcdabcdabcd
$ perceval supybot 'http://channel.example.com' /tmp/supybot/
Telegram backend needs an API token to authenticate the bot. In addition and in order to fetch messages from a group or channel, privacy settings must be disabled. To know how to create a bot, to obtain its token and to configure it please read the Telegram Bots docs pages.
Note that the messages are available on the Telegram server until the bot fetches them, but they will not be kept longer than 24 hours.
$ perceval telegram mybot -t 12345678abcdefgh --chats 1 2 -10
Twitter backend needs a bearer token to authenticate the requests. It can be obtained using the code available on GistGitHub: https://gist.github.com/valeriocos/7d4d28f72f53fbce49f1512ba77ef5f6
$ perceval twitter grimoirelab -t 12345678abcdefgh
Some backends are implemented in a seperate repository but not merged into chaoss/grimoirelab-perceval due to long-run maintainence reasons. Please feel free to check the backends and contact the maintainers for any issues or questions related to them.
- Bundle for Puppet, Inc. ecosystem: chaoss/grimoirelab-perceval-puppet
- Bundle for OPNFV ecosystem: chaoss/grimoirelab-perceval-opnfv
- Bundle for Mozilla ecosystem: chaoss/grimoirelab-perceval-mozilla
- Bundle for FINOS ecosystem: Bitergia/grimoirelab-perceval-finos
- Weblate backend: chaoss/grimoirelab-perceval-weblate
- Zulip backend: vchrombie/grimoirelab-perceval-zulip
- OSF backend: gitlab.com/open-rit/perceval-osf
- Gitee backend: grimoirelab-gitee/grimoirelab-perceval-gitee
- Airtable backend: perceval-backends/grimoirelab-perceval-airtable
- Bitbucket backend: perceval-backends/grimoirelab-perceval-bitbucket
Perceval comes with a comprehensive list of unit tests. To run them, in addition
to the dependencies installed with Perceval, you need httpretty.
Licensed under GNU General Public License (GPL), version 3 or later.