GitHub 1C Enterprise Daily Trending

thuml/Time-Series-Library

2024-08-21T01:27:46Z

A Library for Advanced Deep Time Series Models.

Time Series Library (TSLib)

TSLib is an open-source library for deep learning researchers, especially for deep time series analysis.

We provide a neat code base to evaluate advanced deep time series models or develop your model, which covers five mainstream tasks: long- and short-term forecasting, imputation, anomaly detection, and classification.

🚩News (2024.07) We wrote a comprehensive survey of [Deep Time Series Models] with a rigorous benchmark based on TSLib. In this paper, we summarized the design principles of current time series models supported by insightful experiments, hoping to be helpful to future research.

🚩News (2024.04) Many thanks for the great work from frecklebars. The famous sequenctial model Mamba has been included in our library. See this file, where you need to install mamba_ssm with pip at first.

🚩News (2024.03) Given the inconsistent look-back length of various papers, we split the long-term forecasting in the leaderboard into two categories: Look-Back-96 and Look-Back-Searching. We recommend researchers read TimeMixer, which includes both look-back length settings in experiments for scientific rigor.

🚩News (2023.10) We add an implementation to iTransformer, which is the state-of-the-art model for long-term forecasting. The official code and complete scripts of iTransformer can be found here.

🚩News (2023.09) We added a detailed tutorial for TimesNet and this library, which is quite friendly to beginners of deep time series analysis.

🚩News (2023.02) We release the TSlib as a comprehensive benchmark and code base for time series models, which is extended from our previous GitHub repository Autoformer.

Leaderboard for Time Series Analysis

Till March 2024, the top three models for five different tasks are:

Model Ranking	Long-term Forecasting Look-Back-96	Long-term Forecasting Look-Back-Searching	Short-term Forecasting	Imputation	Classification	Anomaly Detection
🥇 1st	iTransformer	TimeMixer	TimesNet	TimesNet	TimesNet	TimesNet
🥈 2nd	TimeMixer	PatchTST	Non-stationary Transformer	Non-stationary Transformer	Non-stationary Transformer	FEDformer
🥉 3rd	TimesNet	DLinear	FEDformer	Autoformer	Informer	Autoformer

Note: We will keep updating this leaderboard. If you have proposed advanced and awesome models, you can send us your paper/code link or raise a pull request. We will add them to this repo and update the leaderboard as soon as possible.

Compared models of this leaderboard. ☑ means that their codes have already been included in this repo.

TimeMixer - TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting [ICLR 2024] [Code].
TSMixer - TSMixer: An All-MLP Architecture for Time Series Forecasting [arXiv 2023] [Code]
iTransformer - iTransformer: Inverted Transformers Are Effective for Time Series Forecasting [ICLR 2024] [Code].
PatchTST - A Time Series is Worth 64 Words: Long-term Forecasting with Transformers [ICLR 2023] [Code].
TimesNet - TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis [ICLR 2023] [Code].
DLinear - Are Transformers Effective for Time Series Forecasting? [AAAI 2023] [Code].
LightTS - Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures [arXiv 2022] [Code].
ETSformer - ETSformer: Exponential Smoothing Transformers for Time-series Forecasting [arXiv 2022] [Code].
Non-stationary Transformer - Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting [NeurIPS 2022] [Code].
FEDformer - FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting [ICML 2022] [Code].
Pyraformer - Pyraformer: Low-complexity Pyramidal Attention for Long-range Time Series Modeling and Forecasting [ICLR 2022] [Code].
Autoformer - Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting [NeurIPS 2021] [Code].
Informer - Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting [AAAI 2021] [Code].
Reformer - Reformer: The Efficient Transformer [ICLR 2020] [Code].
Transformer - Attention is All You Need [NeurIPS 2017] [Code].

See our latest paper [TimesNet] for the comprehensive benchmark. We will release a real-time updated online version soon.

Newly added baselines. We will add them to the leaderboard after a comprehensive evaluation.

Mamba - Mamba: Linear-Time Sequence Modeling with Selective State Spaces [arXiv 2023] [Code]
SegRNN - SegRNN: Segment Recurrent Neural Network for Long-Term Time Series Forecasting [arXiv 2023] [Code].
Koopa - Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors [NeurIPS 2023] [Code].
FreTS - Frequency-domain MLPs are More Effective Learners in Time Series Forecasting [NeurIPS 2023] [Code].
TiDE - Long-term Forecasting with TiDE: Time-series Dense Encoder [arXiv 2023] [Code].
FiLM - FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting [NeurIPS 2022][Code].
MICN - MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting [ICLR 2023][Code].
Crossformer - Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting [ICLR 2023][Code].
TFT - Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting [arXiv 2019][Code].

Usage

Install Python 3.8. For convenience, execute the following command.

pip install -r requirements.txt

Prepare Data. You can obtain the well pre-processed datasets from [Google Drive] or [Baidu Drive], Then place the downloaded data in the folder./dataset. Here is a summary of supported datasets.

Train and evaluate model. We provide the experiment scripts for all benchmarks under the folder ./scripts/. You can reproduce the experiment results as the following examples:

# long-term forecast
bash ./scripts/long_term_forecast/ETT_script/TimesNet_ETTh1.sh
# short-term forecast
bash ./scripts/short_term_forecast/TimesNet_M4.sh
# imputation
bash ./scripts/imputation/ETT_script/TimesNet_ETTh1.sh
# anomaly detection
bash ./scripts/anomaly_detection/PSM/TimesNet.sh
# classification
bash ./scripts/classification/TimesNet.sh

Develop your own model.

Add the model file to the folder ./models. You can follow the ./models/Transformer.py.
Include the newly added model in the Exp_Basic.model_dict of ./exp/exp_basic.py.
Create the corresponding scripts under the folder ./scripts.

Note: The original code for the classification task can be found here. It is hard to fuse all five tasks in one library. We are still working on this task.

Citation

If you find this repo useful, please cite our paper.

@inproceedings{wu2023timesnet,
  title={TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis},
  author={Haixu Wu and Tengge Hu and Yong Liu and Hang Zhou and Jianmin Wang and Mingsheng Long},
  booktitle={International Conference on Learning Representations},
  year={2023},
}

@article{wang2024tssurvey,
  title={Deep Time Series Models: A Comprehensive Survey and Benchmark},
  author={Yuxuan Wang and Haixu Wu and Jiaxiang Dong and Yong Liu and Mingsheng Long and Jianmin Wang},
  booktitle={arXiv preprint arXiv:2407.13278},
  year={2024},
}

Contact

If you have any questions or suggestions, feel free to contact our maintenance team:

Current:

Haixu Wu (Ph.D. student, wuhx23@mails.tsinghua.edu.cn)
Yong Liu (Ph.D. student, liuyong21@mails.tsinghua.edu.cn)
Yuxuan Wang (Ph.D. student, wangyuxu22@mails.tsinghua.edu.cn)
Huikun Weng (Undergraduate, wenghk22@mails.tsinghua.edu.cn)

Tengge Hu (Master student, htg21@mails.tsinghua.edu.cn)
Haoran Zhang (Master student, z-hr20@mails.tsinghua.edu.cn)
Jiawei Guo (Undergraduate, guo-jw21@mails.tsinghua.edu.cn)

Or describe it in Issues.

Acknowledgement

This project is supported by the National Key R&D Program of China (2021YFB1715200).

This library is constructed based on the following repos:

Forecasting: https://github.com/thuml/Autoformer.
Anomaly Detection: https://github.com/thuml/Anomaly-Transformer.
Classification: https://github.com/thuml/Flowformer.

All the experiment datasets are public, and we obtain them from the following links:

Long-term Forecasting and Imputation: https://github.com/thuml/Autoformer.
Short-term Forecasting: https://github.com/ServiceNow/N-BEATS.
Anomaly Detection: https://github.com/thuml/Anomaly-Transformer.
Classification: https://www.timeseriesclassification.com/.

All Thanks To Our Contributors

LibreTranslate/LibreTranslate

2024-08-21T01:27:46Z

Free and Open Source Machine Translation API. Self-hosted, offline capable and easy to setup.

LibreTranslate

Try it online! | API Docs | Community Forum

Free and Open Source Machine Translation API, entirely self-hosted. Unlike other APIs, it doesn't rely on proprietary providers such as Google or Azure to perform translations. Instead, its translation engine is powered by the open source Argos Translate library.

Try it online! | API Docs

API Examples

Simple

Request:

const res = await fetch("https://libretranslate.com/translate", {
  method: "POST",
  body: JSON.stringify({
    q: "Hello!",
    source: "en",
    target: "es"
  }),
  headers: { "Content-Type": "application/json" }
});

console.log(await res.json());

Response:

{
    "translatedText": "¡Hola!"
}

List of language codes: https://libretranslate.com/languages

Auto Detect Language

Request:

const res = await fetch("https://libretranslate.com/translate", {
  method: "POST",
  body: JSON.stringify({
    q: "Ciao!",
    source: "auto",
    target: "en"
  }),
  headers: { "Content-Type": "application/json" }
});

console.log(await res.json());

Response:

{
    "detectedLanguage": {
        "confidence": 83,
        "language": "it"
    },
    "translatedText": "Bye!"
}

HTML

Request:

const res = await fetch("https://libretranslate.com/translate", {
  method: "POST",
  body: JSON.stringify({
    q: '<p class="green">Hello!</p>',
    source: "en",
    target: "es",
    format: "html"
  }),
  headers: { "Content-Type": "application/json" }
});

console.log(await res.json());

Response:

{
    "translatedText": "<p class=\"green\">¡Hola!</p>"
}

Alternative Translations

Request:

const res = await fetch("https://libretranslate.com/translate", {
	method: "POST",
	body: JSON.stringify({
		q: "Hello",
		source: "en",
		target: "it",
		format: "text",
		alternatives: 3
	}),
	headers: { "Content-Type": "application/json" }
});

console.log(await res.json());

Response:

{
    "alternatives": [
        "Salve",
        "Pronto"
    ],
    "translatedText": "Ciao"
}

Install and Run

You can run your own API server with just a few lines of setup!

Make sure you have Python installed (3.8 or higher is recommended), then simply run:

pip install libretranslate
libretranslate [args]

Then open a web browser to http://localhost:5000

On Ubuntu 20.04 you can also use the install script available at https://github.com/argosopentech/LibreTranslate-init

Run with Docker

You can also run the application with docker:

Linux/macOS

./run.sh [args]

Windows

run.bat [args]

Build and Run

See CONTRIBUTING.md for information on how to build and run the project yourself.

CUDA

You can use hardware acceleration to speed up translations on a GPU machine with CUDA 11.2 and nvidia-docker installed.

Run this version with:

docker compose -f docker-compose.cuda.yml up -d --build

Arguments

Arguments passed to the process or set via environment variables are split into two kinds.

Settings or runtime flags used to toggle specific runmodes or disable parts of the application. These act as toggle when added or removed.
Configuration parameters to set various limits and configure the application. These require a parameter to be passed to function, if removed the default parameters are used.

Settings / Flags

Argument	Description	Default Setting	Env. name
--debug	Enable debug environment	`Disabled`	LT_DEBUG
--ssl	Whether to enable SSL	`Disabled`	LT_SSL
--api-keys	Enable API keys database for per-client rate limits when --req-limit is reached	`Don't use API keys`	LT_API_KEYS
--require-api-key-origin	Require use of an API key for programmatic access to the API, unless the request origin matches this domain	`No restrictions on domain origin`	LT_REQUIRE_API_KEY_ORIGIN
--require-api-key-secret	Require use of an API key for programmatic access to the API, unless the client also sends a secret match	`No secrets required`	LT_REQUIRE_API_KEY_SECRET
--suggestions	Allow user suggestions	`Disabled`	LT_SUGGESTIONS
--disable-files-translation	Disable files translation	`File translation allowed`	LT_DISABLE_FILES_TRANSLATION
--disable-web-ui	Disable web ui	`Web Ui enabled`	LT_DISABLE_WEB_UI
--update-models	Update language models at startup	`Only on if no models found`	LT_UPDATE_MODELS
--metrics	Enable the /metrics endpoint for exporting Prometheus usage metrics	`Disabled`	LT_METRICS

Configuration Parameters

Argument	Description	Default Parameter	Env. name
--host	Set host to bind the server to	`127.0.0.1`	LT_HOST
--port	Set port to bind the server to	`5000`	LT_PORT
--char-limit	Set character limit	`No limit`	LT_CHAR_LIMIT
--req-limit	Set maximum number of requests per minute per client (outside of limits set by api keys)	`No limit`	LT_REQ_LIMIT
--req-limit-storage	Storage URI to use for request limit data storage. See Flask Limiter	`memory://`	LT_REQ_LIMIT_STORAGE
--req-time-cost	Considers a time cost (in seconds) for request limiting purposes. If a request takes 10 seconds and this value is set to 5, the request cost is either 2 or the actual request cost (whichever is greater).	`No time cost`	LT_REQ_TIME_COST
--batch-limit	Set maximum number of texts to translate in a batch request	`No limit`	LT_BATCH_LIMIT
--ga-id	Enable Google Analytics on the API client page by providing an ID	`Empty (no tracking)`	LT_GA_ID
--frontend-language-source	Set frontend default language - source	`auto`	LT_FRONTEND_LANGUAGE_SOURCE
--frontend-language-target	Set frontend default language - target	`locale` (match site's locale)	LT_FRONTEND_LANGUAGE_TARGET
--frontend-timeout	Set frontend translation timeout	`500`	LT_FRONTEND_TIMEOUT
--api-keys-db-path	Use a specific path inside the container for the local database. Can be absolute or relative	`db/api_keys.db`	LT_API_KEYS_DB_PATH
--api-keys-remote	Use this remote endpoint to query for valid API keys instead of using the local database	`Empty (use local db instead)`	LT_API_KEYS_REMOTE
--get-api-key-link	Show a link in the UI where to direct users to get an API key	`Empty (no link shown on web ui)`	LT_GET_API_KEY_LINK
--shared-storage	Shared storage URI to use for multi-process data sharing (e.g. when using gunicorn)	`memory://`	LT_SHARED_STORAGE
--secondary	Mark this instance as a secondary instance to avoid conflicts with the primary node in multi-node setups	`Primary node`	LT_SECONDARY
--load-only	Set available languages	`Empty (use all from argostranslate)`	LT_LOAD_ONLY
--threads	Set number of threads	`4`	LT_THREADS
--metrics-auth-token	Protect the /metrics endpoint by allowing only clients that have a valid Authorization Bearer token	`Empty (no auth required)`	LT_METRICS_AUTH_TOKEN
--url-prefix	Add prefix to URL: example.com:5000/url-prefix/	`/`	LT_URL_PREFIX

Notes:

Each argument has an equivalent environment variable that can be used instead. The env. variables overwrite the default values but have lower priority than the command arguments and are particularly useful if used with Docker. The environment variable names are the upper-snake-case of the equivalent command argument's name with a LT prefix.
To configure requirement for api key to use, set --req-limit to 0 and add the --api-keys flag. Requests made without a proper api key will be rejected.
Setting --update-models will update models regardless of whether updates are available or not.

Update

Software

If you installed with pip:

pip install -U libretranslate

If you're using docker:

docker pull libretranslate/libretranslate

Language Models

Start the program with the --update-models argument. For example: libretranslate --update-models or ./run.sh --update-models.

Alternatively you can also run the scripts/install_models.py script.

Run with WSGI and Gunicorn

pip install gunicorn
gunicorn --bind 0.0.0.0:5000 'wsgi:app'

You can pass application arguments directly to Gunicorn via:

gunicorn --bind 0.0.0.0:5000 'wsgi:app(api_keys=True)'

Kubernetes Deployment

See Medium article by JM Robles and the improved k8s.yaml by @rasos.

Helm Chart

Based on @rasos work you can now install LibreTranslate on Kubernetes using Helm.

A Helm chart is now available in the helm-chart repository where you can find more details.

You can quickly install LibreTranslate on Kubernetes using Helm with the following command:

helm repo add libretranslate https://libretranslate.github.io/helm-chart/
helm repo update
helm search repo libretranslate

helm install libretranslate libretranslate/libretranslate --namespace libretranslate --create-namespace

Manage API Keys

LibreTranslate supports per-user limit quotas, e.g. you can issue API keys to users so that they can enjoy higher requests limits per minute (if you also set --req-limit). By default all users are rate-limited based on --req-limit, but passing an optional api_key parameter to the REST endpoints allows a user to enjoy higher request limits. You can also specify different character limits that bypass the default --char-limit value on a per-key basis.

To use API keys simply start LibreTranslate with the --api-keys option. If you modified the API keys database path with the option --api-keys-db-path, you must specify the path with the same argument flag when using the ltmanage keys command.

Add New Keys

To issue a new API key with 120 requests per minute limits:

ltmanage keys add 120

To issue a new API key with 120 requests per minute and a maximum of 5,000 characters per request:

ltmanage keys add 120 --char-limit 5000

If you changed the API keys database path:

ltmanage keys --api-keys-db-path path/to/db/dbName.db add 120

Remove Keys

ltmanage keys remove <api-key>

View Keys

ltmanage keys

Prometheus Metrics

LibreTranslate has Prometheus exporter capabilities when you pass the --metrics argument at startup (disabled by default). When metrics are enabled, a /metrics endpoint is mounted on the instance:

http://localhost:5000/metrics

# HELP libretranslate_http_requests_in_flight Multiprocess metric
# TYPE libretranslate_http_requests_in_flight gauge
libretranslate_http_requests_in_flight{api_key="",endpoint="/translate",request_ip="127.0.0.1"} 0.0
# HELP libretranslate_http_request_duration_seconds Multiprocess metric
# TYPE libretranslate_http_request_duration_seconds summary
libretranslate_http_request_duration_seconds_count{api_key="",endpoint="/translate",request_ip="127.0.0.1",status="200"} 0.0
libretranslate_http_request_duration_seconds_sum{api_key="",endpoint="/translate",request_ip="127.0.0.1",status="200"} 0.0

You can then configure prometheus.yml to read the metrics:

scrape_configs:
  - job_name: "libretranslate"

    # Needed only if you use --metrics-auth-token
    #authorization:
      #credentials: "mytoken"

    static_configs:
      - targets: ["localhost:5000"]

To secure the /metrics endpoint you can also use --metrics-auth-token mytoken.

If you use Gunicorn, make sure to create a directory for storing multiprocess data metrics and set PROMETHEUS_MULTIPROC_DIR:

mkdir -p /tmp/prometheus_data
rm /tmp/prometheus_data/*
export PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus_data
gunicorn -c scripts/gunicorn_conf.py --bind 0.0.0.0:5000 'wsgi:app(metrics=True)'

Language Bindings

You can use the LibreTranslate API using the following bindings:

Rust: https://github.com/DefunctLizard/libretranslate-rs
Node.js: https://github.com/franciscop/translate
TypeScript: https://github.com/tderflinger/libretranslate-ts
.Net: https://github.com/sigaloid/LibreTranslate.Net
Go: https://github.com/SnakeSel/libretranslate
Python: https://github.com/argosopentech/LibreTranslate-py
PHP: https://github.com/jefs42/libretranslate
C++: https://github.com/argosopentech/LibreTranslate-cpp
Swift: https://github.com/wacumov/libretranslate
Unix: https://github.com/argosopentech/LibreTranslate-sh
Shell: https://github.com/Hayao0819/Hayao-Tools/tree/master/libretranslate-sh
Java: https://github.com/suuft/libretranslate-java
Ruby: https://github.com/noesya/libretranslate
R: https://github.com/myanesp/libretranslateR

Discourse Plugin

You can use the official discourse translator plugin to translate Discourse topics with LibreTranslate. To install it simply modify /var/discourse/containers/app.yml:

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-translator
    ...

Then issue ./launcher rebuild app. From the Discourse's admin panel then select "LibreTranslate" as a translation provider and set the relevant endpoint configurations.

See it in action on this page.

Mobile Apps

LibreTranslator is an Android app available on the Play Store and in the F-Droid store that uses the LibreTranslate API.
LiTranslate is an iOS app available on the App Store that uses the LibreTranslate API.

Web browser

minbrowser is a web browser with integrated LibreTranslate support.
A LibreTranslate Firefox addon is currently a work in progress.

Mirrors

This is a list of public LibreTranslate instances, some require an API key. If you want to add a new URL, please open a pull request.

URL	API Key Required	Links
libretranslate.com	✔	[ Get API Key ] [ Service Status ]
translate.terraprint.co	-
trans.zillyhuhn.com	-
translate.lotigara.ru	-

TOR/i2p Mirrors

URL
lt.vernccvbvyi5qhfzyqengccj7lkove6bjot2xhh5kajhwvidqafczrad.onion
lt.vern.i2p

Adding New Language Models

You have two options to create new language models:

Most of the training data is from Opus, which is an open source parallel corpus. Check also NLLU

Localization

The LibreTranslate Web UI is available in all the languages for which LibreTranslate can translate to. It can also (roughly) translate itself! Some languages might not appear in the UI since they haven't been reviewed by a human yet. You can enable all languages by turning on --debug mode.

To help improve or review the UI translations:

Go to https://hosted.weblate.org/projects/libretranslate/app/#translations. All changes are automatically pushed to this repository.
Once all strings have been reviewed/edited, open a pull request and change libretranslate/locales/{code}/meta.json:

{
 "name": "<Language>",
 "reviewed": true <-- Change this from false to true
}

UI Languages

Language	Reviewed	Weblate Link
Arabic		Edit
Azerbaijani		Edit
Chinese		Edit
Chinese (Traditional)		Edit
Czech	✔	Edit
Danish		Edit
Dutch		Edit
English	✔	Edit
Esperanto	✔	Edit
Finnish		Edit
French	✔	Edit
German	✔	Edit
Greek		Edit
Hebrew		Edit
Hindi		Edit
Hungarian		Edit
Indonesian		Edit
Irish		Edit
Italian	✔	Edit
Japanese		Edit
Kabyle		Edit
Korean	✔	Edit
Occitan		Edit
Persian		Edit
Polish		Edit
Portuguese	✔	Edit
Russian	✔	Edit
Slovak		Edit
Spanish	✔	Edit
Swedish		Edit
Turkish		Edit
Ukrainian	✔	Edit
Vietnamese		Edit

Roadmap

Help us by opening a pull request!

Language bindings for every computer language
Improved translations

Any other idea is welcome also.

FAQ

Can I use your API server at libretranslate.com for my application in production?

In short, yes, but only if you buy an API key. You can always run LibreTranslate for free on your own server of course.

Some translations on libretranslate.com are different than the self-hosted ones. Why?

By default language models are loaded from the argos-index. Sometimes we deploy models on libretranslate.com that haven't been added to the argos-index yet, such as those converted from OPUS (thread)

Where are the language models saved?

In $HOME/.local/share/argos-translate/packages. On Windows that's C:\Users\youruser\.local\share\argos-translate\packages.

Can I use LibreTranslate behind a reverse proxy, like Apache2 or Caddy?

Yes, here are config examples for Apache2 and Caddy that redirect a subdomain (with HTTPS certificate) to LibreTranslate running on a docker at localhost.

sudo docker run -ti --rm -p 127.0.0.1:5000:5000 libretranslate/libretranslate

You can remove 127.0.0.1 on the above command if you want to be able to access it from domain.tld:5000, in addition to subdomain.domain.tld (this can be helpful to determine if there is an issue with Apache2 or the docker container).

Add --restart unless-stopped if you want this docker to start on boot, unless manually stopped.

Apache config

Replace [YOUR_DOMAIN] with your full domain; for example, translate.domain.tld or libretranslate.domain.tld.

Remove # on the ErrorLog and CustomLog lines to log requests.

#Libretranslate

#Redirect http to https
<VirtualHost *:80>
    ServerName http://[YOUR_DOMAIN]
    Redirect / https://[YOUR_DOMAIN]
    # ErrorLog ${APACHE_LOG_DIR}/error.log
    # CustomLog ${APACHE_LOG_DIR}/tr-access.log combined
 </VirtualHost>

#https
<VirtualHost *:443>
    ServerName https://[YOUR_DOMAIN]

    ProxyPass / http://127.0.0.1:5000/
    ProxyPassReverse / http://127.0.0.1:5000/
    ProxyPreserveHost On

    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/[YOUR_DOMAIN]/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/[YOUR_DOMAIN]/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/[YOUR_DOMAIN]/fullchain.pem

    # ErrorLog ${APACHE_LOG_DIR}/tr-error.log
    # CustomLog ${APACHE_LOG_DIR}/tr-access.log combined
</VirtualHost>

Add this to an existing site config, or a new file in /etc/apache2/sites-available/new-site.conf and run sudo a2ensite new-site.conf.

To get a HTTPS subdomain certificate, install certbot (snap), run sudo certbot certonly --manual --preferred-challenges dns and enter your information (with subdomain.domain.tld as the domain). Add a DNS TXT record with your domain registrar when asked. This will save your certificate and key to /etc/letsencrypt/live/{subdomain.domain.tld}/. Alternatively, comment the SSL lines out if you don't want to use HTTPS.

Caddy config

Replace [YOUR_DOMAIN] with your full domain; for example, translate.domain.tld or libretranslate.domain.tld.

#Libretranslate
[YOUR_DOMAIN] {
  reverse_proxy localhost:5000
}

Add this to an existing Caddyfile or save it as Caddyfile in any directory and run sudo caddy reload in that same directory.

NGINX config

Replace [YOUR_DOMAIN] with your full domain; for example, translate.domain.tld or libretranslate.domain.tld.

Remove # on the access_log and error_log lines to disable logging.

server {
  listen 80;
  server_name [YOUR_DOMAIN];
  return 301 https://$server_name$request_uri;
}

server {
  listen 443 http2 ssl;
  server_name [YOUR_DOMAIN];

  #access_log off;
  #error_log off;

  # SSL Section
  ssl_certificate /etc/letsencrypt/live/[YOUR_DOMAIN]/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/[YOUR_DOMAIN]/privkey.pem;

  ssl_protocols TLSv1.2 TLSv1.3;

  # Using the recommended cipher suite from: https://wiki.mozilla.org/Security/Server_Side_TLS
  ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';

  ssl_session_timeout 10m;
  ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
  ssl_session_tickets off;

  # Specifies a curve for ECDHE ciphers.
  ssl_ecdh_curve prime256v1;
  # Server should determine the ciphers, not the client
  ssl_prefer_server_ciphers on;


  # Header section
  add_header Strict-Transport-Security  "max-age=31536000; includeSubDomains; preload" always;
  add_header Referrer-Policy            "strict-origin" always;

  add_header X-Frame-Options            "SAMEORIGIN"    always;
  add_header X-XSS-Protection           "1; mode=block" always;
  add_header X-Content-Type-Options     "nosniff"       always;
  add_header X-Download-Options         "noopen"        always;
  add_header X-Robots-Tag               "none"          always;

  add_header Feature-Policy             "microphone 'none'; camera 'none'; geolocation 'none';"  always;
  # Newer header but not everywhere supported
  add_header Permissions-Policy         "microphone=(), camera=(), geolocation=()" always;

  # Remove X-Powered-By, which is an information leak
  fastcgi_hide_header X-Powered-By;

  # Do not send nginx server header
  server_tokens off;

  # GZIP Section
  gzip on;
  gzip_disable "msie6";

  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_min_length 256;
  gzip_types text/xml text/javascript font/ttf font/eot font/otf application/x-javascript application/atom+xml application/javascript application/json application/manifest+json application/rss+xml application/x-web-app-manifest+json application/xhtml+xml application/xml image/svg+xml image/x-icon text/css text/plain;

  location / {
      proxy_pass http://127.0.0.1:5000/;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      client_max_body_size 0;
  }
}

Add this to an existing NGINX config or save it as libretranslate in the /etc/nginx/site-enabled directory and run sudo nginx -s reload.

Can I do batch translations?

Yes, pass an array of strings instead of a string to the q field:

const res = await fetch("https://libretranslate.com/translate", {
  method: "POST",
  body: JSON.stringify({
    q: ["Hello", "world"],
    source: "en",
    target: "es"
  }),
  headers: { "Content-Type": "application/json" }
});

console.log(await res.json());
// {
//     "translatedText": [
//         "Hola",
//         "mundo"
//     ]
// }

Contributing

We welcome contributions! Here's some ideas:

Train a new language model using Locomotive. For example, we want to train improved neural networks for German and many other languages.
Can you beat the performance of our language models? Train a new one and let's compare it. To submit your model make a post on the community forum with a link to download your .argosmodel file and some sample text that your model has translated.
Pick an issue to work on.

Credits

This work is largely possible thanks to Argos Translate, which powers the translation engine.

License

GNU Affero General Public License v3

Trademark

See Trademark Guidelines

toss/es-toolkit

2024-08-21T01:27:46Z

A modern JavaScript utility library that's 2-3 times faster and up to 97% smaller—a major upgrade to lodash.

es-toolkit ·

English | 한국어 | 简体中文 | 日本語

es-toolkit is a state-of-the-art, high-performance JavaScript utility library with a small bundle size and strong type annotations.

es-toolkit offers a variety of everyday utility functions with modern implementations, such as debounce, delay, chunk, sum, and pick.
Designed with performance in mind, es-toolkit achieves 2-3× better performance in modern JavaScript environments.
es-toolkit supports tree shaking out of the box, and reduces JavaScript code by up to 97% compared to other libraries.
es-toolkit includes built-in TypeScript support, with straightforward yet robust types. It also provides useful type guards such as isNotNil.
es-toolkit is battle-tested with 100% test coverage, ensuring reliability and robustness.

Examples

// import from '@es-toolkit/es-toolkit' in jsr.
import { debounce, chunk } from 'es-toolkit';

const debouncedLog = debounce(message => {
  console.log(message);
}, 300);

// This call will be debounced
debouncedLog('Hello, world!');

const array = [1, 2, 3, 4, 5, 6];
const chunkedArray = chunk(array, 2);

console.log(chunkedArray);
// Output: [[1, 2], [3, 4], [5, 6]]

Contributing

We welcome contribution from everyone in the community. Read below for detailed contribution guide.

CONTRIBUTING

License

MIT © Viva Republica, Inc. See LICENSE for details.

krayin/laravel-crm

2024-08-22T01:28:22Z

Free & Opensource Laravel CRM solution for SMEs and Enterprises for complete customer lifecycle management.

berichan/ACNHMobileSpawner

2024-08-22T01:28:28Z

Animal Crossing New Horizons real-time multi-tool built in Unity.

ACNHSpawner

For mobile and desktop, despite the name.

Multi-tool app for Animal Crossing: New Horizons built in Unity. Designed to be used while you are playing the game so you don't have to manually edit saves on a PC. Confirmed working on Windows, Mac, Linux, Android and iOS.

『あつまれどうぶつの森』でリアルタイムにアイテムをインジェクトしたり島を編集したりゲーム内の値を変更したり等のカスタマイズを行うことができるAndroid/ iOS(Windows/ macOS/ Linuxも可)向けユーティリティ。

Requires a Switch running custom firmware with the sysmodule sys-botbase or USB-Botbase installed.

It currently supports the following in-play actions:

Injecting and deleting inventory items. Supports all players 1-8.
Changing amount of miles, bells in your bank and in your wallet (inventory).
Changing and replacing villagers using the perfect villager database.
Changing the turnip buy/sell prices and fluctuations.
Place and bulk spawn items to your map, a few presets exist within the app. You may also find and replace items.
Spawning internal items* such as the donut. A list of all internal items is here.
Hex editing raw RAM bytes. This can be used in any Switch game, not just Animal Crossing.
Removing certain items from your map, such as weeds, trees, flowers, spoiled turnips, etc.
Saving, sharing and loading certain New Horizons file types: *.nhi (inventory), *.nhv (villager) and *.nhvh (villager house).
Refresh items on the floor of your island, and logs people coming in during the time the refresher was running.
Freeze certain values (villagers, inventory, map etc)
Create teleports so you can easily move between common areas.

Refer to the Wiki for help and troubleshooting.

Based heavily on NHSE.

You run this at your own risk, I'm not responsible for anything. Please check the license for full details before using the app or source.

*Please do not use this app to ruin the experience of other players, be responsible! Do not trade or use items from the internal list in local or online play- this should go without saying, but they will ruin the experience of other players as they are not easily removable on a non-cfw console.

Discord help server

You may ask for help in the support server if you're running into trouble. Please read the wiki first if you haven't already done so.

Builds

Each major release is built for Windows, Android, MacOSX and iOS. You may download the compiled builds here.

iOS builds are auto-built and untested, but I've been told they work.

Screenshots

Video guide

Click the image above.

Notes

Some code was deleted and had to be rebuilt using ILSpy. I've done my best to clean up the classes affected, but they will be uncommented, minus ILSpy warnings I've kept in just to know what was affected.

chrxh/alien

2024-08-21T01:28:38Z

ALIEN is a CUDA-powered artificial life simulation program.

ALIEN - Explore worlds of artificial life

Artificial LIfe ENvironment (ALIEN) is an artificial life simulation tool based on a specialized 2D particle engine in CUDA for soft bodies and fluids. Each simulated body consists of a network of particles that can be upgraded with higher-level functions, ranging from pure information processing capabilities to physical equipment (such as sensors, muscles, weapons, constructors, etc.) whose executions are orchestrated by neural networks. The bodies can be thought of as agents or digital organisms operating in a common environment. Their blueprints can be stored in genomes and passed on to offspring.

The simulation code is written entirely in CUDA and optimized for large-scale real-time simulations with millions of particles. The development is driven by the desire to better understand the conditions for (pre-)biotic evolution and the growing complexity of biological systems. An important goal is to make the simulator user-friendly through a modern user interface, visually appealing rendering and a playful approach.

Please join our Discord server as a place for discussions, new developments and feedback around ALIEN and artificial life in general.

Demo video: Emerging Ecosystems | Winner of the ALIFE 2024 Virtual Creatures Competition

⚡ Main features

Physics and graphics engine

Particles for simulating soft and rigid body mechanics, fluids, heat dissipation, damage, adhesion etc.
Real-time user interactions with running simulations
Simulation runs entirely on GPU via CUDA
Rendering and post-processing via OpenGL using CUDA-OpenGL interoperability

https://user-images.githubusercontent.com/73127001/229868357-131fa71f-d03d-45db-ac76-9d192f5464af.mp4

Artificial Life engine extensions

Multi-cellular organisms are simulated as particle networks
Genetic system and cell by cell construction of offspring
Neural networks for controlling higher-level functions (e.g. sensors and muscles)
Various colors may be used to customize cell types according to own specifications
Support for spatially varying simulation parameters

https://user-images.githubusercontent.com/73127001/229569056-0db6562b-0147-43c8-a977-5f12c1b6277b.mp4

Extensive editing tools

Graph editor for manipulating every particle and connection
Freehand and geometric drawing tools
Genetic editor for designing customized organisms
Mass-operations and (up/down) scaling functions

Networking

Built-in simulation browser
Download and upload simulation files
Upvote simulations by giving stars

❓ But for what is this useful

A first attempt to answer: Feed your curiosity by watching evolution at work! As soon as self-replicating machines come into play and mutations are turned on, the simulation itself does everything.
Perhaps the most honest answer: Fun! It is almost like a game with a pretty fast and realistic physics engine. You can make hundreds of thousands of machines accelerate and destroy with the mouse cursor. It feels like playing god in your own universe with your own rules. Different render styles and a visual editor offer fascinating insights into the events. There are a lot of videos on the YouTube channel for illustration.
A more academic answer: A tool to tackle fundamental questions of how complexity or life-like structure may arise from simple components. How do entire ecosystems adapt to environmental changes and find a new equilibrium? How to find conditions that allow open-ended evolution?
A tool for generative art: Evolution is a creative force that leads to ever new forms and behaviors.

📘 Documentation

A documentation for the previous major version, which introduces the reader to the simulator with tutorial-like articles, can be found at alien-project.gitbook.io/docs. Please notice that many of the information therein are no longer up to date. The latest version includes a brief documentation and user guidance in the program itself via help windows and tooltips.

Further information and artwork:

🖥️ Minimal system requirements

An Nvidia graphics card with compute capability 6.0 or higher is needed. Please check https://en.wikipedia.org/wiki/CUDA#GPUs_supported.

💽 Installer

Installer for Windows: alien-installer.msi (updated: 2024-08-17)

In the case that the program crashes for an unknown reason, please refer to the troubleshooting section below.

🔨 How to build the sources

The build process is mostly automated using the cross-platform CMake build system and the vcpkg package manager, which is included as a Git submodule.

Getting the sources

To obtain the sources, please open a command prompt in a suitable directory (which should not contain whitespace characters) and enter the following command:

git clone --recursive https://github.com/chrxh/alien.git

Note: The --recursive parameter is necessary to check out the vcpkg submodule as well. Besides that, submodules are not normally updated by the standard git pull command. Instead, you need to write git pull --recurse-submodules.

Build instructions

Prerequisites: CUDA Toolkit 11.2+ and a toolchain for CMake (e.g. GCC 9.x+ or MSVC v142+).

Build steps:

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build . --config Release -j8

If everything goes well, the ALIEN executable can be found under the build directory in ./alien or .\Release\alien.exe depending on the used toolchain and platform. It is important to start ALIEN directly from the build folder, otherwise it will not find the resource folder.

There are reported build issues with (updated: 2024-06-22)

GCC 13 (version 11 should work)
Visual Studio 17.10 (version 17.9 should work)
CUDA 12.5 (version 12.4 should work)

⌨️ Command-line interface

This repository also contains a CLI for ALIEN. It can be used to run simulations without using a GUI. This is useful for performance measurements as well as for automatic execution and evaluation of simulations for different parameters. The CLI takes the simulation file, along with its parameters and the number of time steps, as input. It then provides the resulting simulation file and the statistics (as a CSV file) as output. For example,

.\cli.exe -i example.sim -o output.sim -t 1000

runs the simulation file example.sim for 1000 time steps.

🔎 Troubleshooting

Please make sure that:

You have an NVIDIA graphics card with compute capability 6.0 or higher (for example GeForce 10 series).
You have the latest NVIDIA graphics driver installed.
The name of the installation directory (including the parent directories) should not contain non-English characters. If this is not fulfilled, please re-install ALIEN to a suitable directory. Do not move the files manually. If you use Windows, make also sure that you install ALIEN with a Windows user that contains no non-English characters. If this is not the case, a new Windows user could be created to solve this problem.
ALIEN needs write access to its own directory. This should normally be the case.
If you have multiple graphics cards, please check that your primary monitor is connected to the CUDA-powered card. ALIEN uses the same graphics card for computation as well as rendering and chooses the one with the highest compute capability.
If you possess both integrated and dedicated graphics cards, please ensure that the alien-executable is configured to use your high-performance graphics card. On Windows you need to access the 'Graphics settings,' add 'alien.exe' to the list, click 'Options,' and choose 'High performance'.

If these conditions are not met, ALIEN may crash unexpectedly. If the conditions are met and the error still occurs, please start ALIEN with the command line parameter -d, try to reproduce the error and then create a GitHub issue on https://github.com/chrxh/alien/issues where the log.txt is attached.

🌌 Screenshots

Different plant-like populations around a radiation source

Close-up of different types of organisms so that their cell networks can be seen

Different swarms attacking an ecosystem

Genome editor

🧩 Contributing to the project

Contributions to the project are very welcome. The most convenient way is to communicate via GitHub Issues, Pull requests or the Discussion forum depending on the subject. For example, it could be

Providing new content (simulation or genome files)
Producing or sharing media files
Reporting of bugs, wanted features, questions or feedback via GitHub Issues or in the Discussion forum.
Pull requests for bug fixes, code cleanings, optimizations or minor tweaks. If you want to implement new features, refactorings or other major changes, please use the Discussion forum for consultation and coordination in advance.
Extensions or corrections to the alien-docs. It has its own repository.

A short architectural overview of the source code can be found in the documentation.

💎 Credits and dependencies

ALIEN has been initiated, mainly developed and maintained by Christian Heinemann. Thanks to all the others who contributed to this repository:

The following external libraries are used:

Free icons and icon font:

IconFontCppHeaders
Iconduck (Noto Emoji by Google, Apache License 2.0)
Iconfinder (Bogdan Rosu Creative, CC BY 4.0)
People icons created by Freepik - Flaticon (Flaticon license)

🧾 License

ALIEN is licensed under the BSD 3-Clause license.

unbit/uwsgi

2024-08-21T01:28:34Z

uWSGI application server container

The uWSGI project

For official documentation check: https://uwsgi-docs.readthedocs.io/en/latest/

Note: The project is in maintenance mode (only bugfixes and updates for new languages apis)

Do not expect quick answers on github issues and/or pull requests (sorry for that) A big thanks to all of the users and contributors since 2009.

uWSGI development has been sponsored by:

http://unbit.it https://www.pythonanywhere.com/ https://lincolnloop.com/ https://yourlabs.io/oss https://fili.com

chipweinberger/flutter_blue_plus

2024-08-21T01:29:12Z

Flutter plugin for connecting and communicationg with Bluetooth Low Energy devices, on Android and iOS

Note: this plugin is continuous work from FlutterBlue.

Migrating from FlutterBlue? See Migration Guide

Introduction

FlutterBluePlus is a Bluetooth Low Energy plugin for Flutter.

It supports BLE Central Role only (most common).

If you need BLE Peripheral Role, you should check out FlutterBlePeripheral.

Tutorial

If you are new to Bluetooth, you should start by reading BLE tutorials.

❗ Bluetooth Classic is not supported ❗

i.e. Arduino HC-05 & HC-06, speakers, headphones, mice, keyboards, gamepads, and more are not supported. These all use Bluetooth Classic.

Also, iBeacons are not supported on iOS. Apple requires you to use CoreLocation.

Cross-Platform Bluetooth Low Energy

FlutterBluePlus supports nearly every feature on all supported platforms: iOS, macOS, Android.

FlutterBluePlus was written to be simple, robust, and easy to understand.

No Dependencies

FlutterBluePlus has zero dependencies besides Flutter, Android, iOS, and macOS themselves.

This makes FlutterBluePlus very stable, and easy to maintain.

Other BLE Libraries

These other libraries are worth considering. They support all platforms, but fewer features.

⭐ Stars ⭐

Please star this repo & on pub.dev. We all benefit from having a larger community.

Discord 💬

There is a community Discord server. (Link)

Example

FlutterBluePlus has a beautiful example app, useful to debug issues.

cd ./example
flutter run

Usage

🔥 Error Handling 🔥

Flutter Blue Plus takes error handling seriously.

Every error returned by the native platform is checked and thrown as an exception where appropriate. See Reference for a list of throwable functions.

Streams: Streams returned by FlutterBluePlus never emit any errors and never close. There's no need to handle onError or onDone for stream.listen(...). The one exception is FlutterBluePlus.scanResults, which you should handle onError.

Set Log Level

// if your terminal doesn't support color you'll see annoying logs like `\x1B[1;35m`
FlutterBluePlus.setLogLevel(LogLevel.verbose, color:false)

Setting LogLevel.verbose shows all data in and out.

⚫ = function name

🟣 = args to platform

🟡 = data from platform

Bluetooth On & Off

Note: On iOS, a "This app would like to use Bluetooth" system dialogue appears on first call to any FlutterBluePlus method.

// first, check if bluetooth is supported by your hardware
// Note: The platform is initialized on the first call to any FlutterBluePlus method.
if (await FlutterBluePlus.isSupported == false) {
    print("Bluetooth not supported by this device");
    return;
}

// handle bluetooth on & off
// note: for iOS the initial state is typically BluetoothAdapterState.unknown
// note: if you have permissions issues you will get stuck at BluetoothAdapterState.unauthorized
var subscription = FlutterBluePlus.adapterState.listen((BluetoothAdapterState state) {
    print(state);
    if (state == BluetoothAdapterState.on) {
        // usually start scanning, connecting, etc
    } else {
        // show an error to the user, etc
    }
});

// turn on bluetooth ourself if we can
// for iOS, the user controls bluetooth enable/disable
if (Platform.isAndroid) {
    await FlutterBluePlus.turnOn();
}

// cancel to prevent duplicate listeners
subscription.cancel();

Scan for devices

If your device is not found, see Common Problems.

Note: It is recommended to set scan filters to reduce main thread & platform channel usage.

// listen to scan results
// Note: `onScanResults` only returns live scan results, i.e. during scanning. Use
//  `scanResults` if you want live scan results *or* the results from a previous scan.
var subscription = FlutterBluePlus.onScanResults.listen((results) {
        if (results.isNotEmpty) {
            ScanResult r = results.last; // the most recently found device
            print('${r.device.remoteId}: "${r.advertisementData.advName}" found!');
        }
    },
    onError: (e) => print(e),
);

// cleanup: cancel subscription when scanning stops
FlutterBluePlus.cancelWhenScanComplete(subscription);

// Wait for Bluetooth enabled & permission granted
// In your real app you should use `FlutterBluePlus.adapterState.listen` to handle all states
await FlutterBluePlus.adapterState.where((val) => val == BluetoothAdapterState.on).first;

// Start scanning w/ timeout
// Optional: use `stopScan()` as an alternative to timeout
await FlutterBluePlus.startScan(
  withServices:[Guid("180D")], // match any of the specified services
  withNames:["Bluno"], // *or* any of the specified names
  timeout: Duration(seconds:15));

// wait for scanning to stop
await FlutterBluePlus.isScanning.where((val) => val == false).first;

Connect to a device

// listen for disconnection
var subscription = device.connectionState.listen((BluetoothConnectionState state) async {
    if (state == BluetoothConnectionState.disconnected) {
        // 1. typically, start a periodic timer that tries to 
        //    reconnect, or just call connect() again right now
        // 2. you must always re-discover services after disconnection!
        print("${device.disconnectReason?.code} ${device.disconnectReason?.description}");
    }
});

// cleanup: cancel subscription when disconnected
//   - [delayed] This option is only meant for `connectionState` subscriptions.  
//     When `true`, we cancel after a small delay. This ensures the `connectionState` 
//     listener receives the `disconnected` event.
//   - [next] if true, the the stream will be canceled only on the *next* disconnection,
//     not the current disconnection. This is useful if you setup your subscriptions
//     before you connect.
device.cancelWhenDisconnected(subscription, delayed:true, next:true);

// Connect to the device
await device.connect();

// Disconnect from device
await device.disconnect();

// cancel to prevent duplicate listeners
subscription.cancel();

Auto Connect

Connects whenever your device is found.

// enable auto connect
//  - note: autoConnect is incompatible with mtu argument, so you must call requestMtu yourself
await device.connect(autoConnect:true, mtu:null)

// wait until connection
//  - when using autoConnect, connect() always returns immediately, so we must
//    explicity listen to `device.connectionState` to know when connection occurs 
await device.connectionState.where((val) => val == BluetoothConnectionState.connected).first;

// disable auto connect
await device.disconnect()

Save Device

To save a device between app restarts, just write the remoteId to a file.

Now you can connect without needing to scan again, like so:

final String remoteId = await File('/remoteId.txt').readAsString();
var device = BluetoothDevice.fromId(remoteId);
// AutoConnect is convenient because it does not "time out"
// even if the device is not available / turned off.
await device.connect(autoConnect: true);

MTU

On Android, we request an mtu of 512 by default during connection (see: connect function arguments).

On iOS & macOS, the mtu is negotiated automatically, typically 135 to 255.

final subscription = device.mtu.listen((int mtu) {
    // iOS: initial value is always 23, but iOS will quickly negotiate a higher value
    print("mtu $mtu");
});

// cleanup: cancel subscription when disconnected
device.cancelWhenDisconnected(subscription);

// You can also manually change the mtu yourself.
if (Platform.isAndroid) {
    await device.requestMtu(512);
}

Discover services

// Note: You must call discoverServices after every re-connection!
List<BluetoothService> services = await device.discoverServices();
services.forEach((service) {
    // do something with service
});

Read Characteristics

// Reads all characteristics
var characteristics = service.characteristics;
for(BluetoothCharacteristic c in characteristics) {
    if (c.properties.read) {
        List<int> value = await c.read();
        print(value);
    }
}

Write Characteristic

// Writes to a characteristic
await c.write([0x12, 0x34]);

allowLongWrite: To write large characteristics (up to 512 bytes) regardless of mtu, use allowLongWrite:

/// allowLongWrite should be used with caution. 
///   1. it can only be used *with* response to avoid data loss
///   2. the peripheral device must support the 'long write' ble protocol.
///   3. Interrupted transfers can leave the characteristic in a partially written state
///   4. If the mtu is small, it is very very slow.
await c.write(data, allowLongWrite:true);

splitWrite: To write lots of data (unlimited), you can define the splitWrite function.

import 'dart:math';
// split write should be used with caution.
//    1. due to splitting, `characteristic.read()` will return partial data.
//    2. it can only be used *with* response to avoid data loss
//    3. The characteristic must be designed to support split data
extension splitWrite on BluetoothCharacteristic {
  Future<void> splitWrite(List<int> value, {int timeout = 15}) async {
    int chunk = device.mtuNow - 3; // 3 bytes ble overhead
    for (int i = 0; i < value.length; i += chunk) {
      List<int> subvalue = value.sublist(i, min(i + chunk, value.length));
      await write(subvalue, withoutResponse:false, timeout: timeout);
    }
  }
}

Subscribe to a characteristic

If onValueReceived is never called, see Common Problems in the README.

final subscription = characteristic.onValueReceived.listen((value) {
    // onValueReceived is updated:
    //   - anytime read() is called
    //   - anytime a notification arrives (if subscribed)
});

// cleanup: cancel subscription when disconnected
device.cancelWhenDisconnected(subscription);

// subscribe
// Note: If a characteristic supports both **notifications** and **indications**,
// it will default to **notifications**. This matches how CoreBluetooth works on iOS.
await characteristic.setNotifyValue(true);

Last Value Stream

lastValueStream is an alternative to onValueReceived. It emits a value any time the characteristic changes, including writes.

It is very convenient for simple characteristics that support both WRITE and READ (and/or NOTIFY). e.g. a "light switch toggle" characteristic.

final subscription = characteristic.lastValueStream.listen((value) {
    // lastValueStream` is updated:
    //   - anytime read() is called
    //   - anytime write() is called
    //   - anytime a notification arrives (if subscribed)
    //   - also when first listened to, it re-emits the last value for convenience.
});

// cleanup: cancel subscription when disconnected
device.cancelWhenDisconnected(subscription);

// enable notifications
await characteristic.setNotifyValue(true);

Read and write descriptors

// Reads all descriptors
var descriptors = characteristic.descriptors;
for(BluetoothDescriptor d in descriptors) {
    List<int> value = await d.read();
    print(value);
}

// Writes to a descriptor
await d.write([0x12, 0x34])

Services Changed Characteristic

FlutterBluePlus automatically listens to the Services Changed Characteristic (0x2A05)

In FlutterBluePlus, we call it onServicesReset because you must re-discover services.

// - uses the GAP Services Changed characteristic (0x2A05)
// - you must call discoverServices() again
device.onServicesReset.listen(() async {
    print("Services Reset");
    await device.discoverServices();
});

Get Connected Devices

Get devices currently connected to your app.

List<BluetoothDevice> devs = FlutterBluePlus.connectedDevices;
for (var d in devs) {
    print(d);
}

Get System Devices

Get devices connected to the system by any app.

Note: you must connect your app to them before you can communicate with them.

List<BluetoothDevice> devs = await FlutterBluePlus.systemDevices;
for (var d in devs) {
    await d.connect(); // Must connect *our* app to the device
    await d.discoverServices();
}

Create Bond (Android Only)

Note: calling this is usually not necessary!! The platform will do it automatically.

However, you can force the popup to show sooner.

final bsSubscription = device.bondState.listen((value) {
    print("$value prev:{$device.prevBondState}");
});

// cleanup: cancel subscription when disconnected
device.cancelWhenDisconnected(bsSubscription);

// Force the bonding popup to show now (Android Only) 
await device.createBond();

// remove bond
await device.removeBond();

Events API

Access streams from all devices simultaneously.

There are streams for:

events.onConnectionStateChanged
events.onMtuChanged
events.onReadRssi
events.onServicesReset
events.onDiscoveredServices
events.onCharacteristicReceived
events.onCharacteristicWritten
events.onDescriptorRead
events.onDescriptorWritten
events.onNameChanged (iOS Only)
events.onBondStateChanged (Android Only)

// listen to *any device* connection state changes 
FlutterBluePlus.events.onConnectionStateChanged.listen((event)) {
    print('${event.device} ${event.connectionState}');
}

Mocking

To mock FlutterBluePlus for development, refer to the Mocking Guide.

Getting Started

Change the minSdkVersion for Android

flutter_blue_plus is compatible only from version 21 of Android SDK so you should change this in android/app/build.gradle:

android {
  defaultConfig {
     minSdkVersion: 21

Add permissions for Android (No Location)

In the android/app/src/main/AndroidManifest.xml add:

<!-- Tell Google Play Store that your app uses Bluetooth LE
     Set android:required="true" if bluetooth is necessary -->
<uses-feature android:name="android.hardware.bluetooth_le" android:required="false" />

<!-- New Bluetooth permissions in Android 12
https://developer.android.com/about/versions/12/features/bluetooth-permissions -->
<uses-permission android:name="android.permission.BLUETOOTH_SCAN" android:usesPermissionFlags="neverForLocation" />
<uses-permission android:name="android.permission.BLUETOOTH_CONNECT" />

<!-- legacy for Android 11 or lower -->
<uses-permission android:name="android.permission.BLUETOOTH" android:maxSdkVersion="30" />
<uses-permission android:name="android.permission.BLUETOOTH_ADMIN" android:maxSdkVersion="30" />
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" android:maxSdkVersion="30"/>

<!-- legacy for Android 9 or lower -->
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" android:maxSdkVersion="28" />

Add permissions for Android (With Fine Location)

If you want to use Bluetooth to determine location, or support iBeacons.

In the android/app/src/main/AndroidManifest.xml add:

<!-- Tell Google Play Store that your app uses Bluetooth LE
     Set android:required="true" if bluetooth is necessary -->
<uses-feature android:name="android.hardware.bluetooth_le" android:required="false" />

<!-- New Bluetooth permissions in Android 12
https://developer.android.com/about/versions/12/features/bluetooth-permissions -->
<uses-permission android:name="android.permission.BLUETOOTH_SCAN"/>
<uses-permission android:name="android.permission.BLUETOOTH_CONNECT" />
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />

<!-- legacy for Android 11 or lower -->
<uses-permission android:name="android.permission.BLUETOOTH" android:maxSdkVersion="30" />
<uses-permission android:name="android.permission.BLUETOOTH_ADMIN" android:maxSdkVersion="30" />

<!-- legacy for Android 9 or lower -->
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" android:maxSdkVersion="28" />

And set androidUsesFineLocation when scanning:

// Start scanning
flutterBlue.startScan(timeout: Duration(seconds: 4), androidUsesFineLocation: true);

Android Proguard

Add the following line in your project/android/app/proguard-rules.pro file:

-keep class com.lib.flutter_blue_plus.* { *; }

to avoid seeing the following kind errors in your release builds:

PlatformException(startScan, Field androidScanMode_ for m0.e0 not found. Known fields are
 [private int m0.e0.q, private b3.b0$i m0.e0.r, private boolean m0.e0.s, private static final m0.e0 m0.e0.t,
 private static volatile b3.a1 m0.e0.u], java.lang.RuntimeException: Field androidScanMode_ for m0.e0 not found

Add permissions for iOS

In the ios/Runner/Info.plist let’s add:

<dict>
    <key>NSBluetoothAlwaysUsageDescription</key>
    <string>This app needs Bluetooth to function</string>

For location permissions on iOS see more at: https://developer.apple.com/documentation/corelocation/requesting_authorization_for_location_services

Add permissions for macOS

Make sure you have granted access to the Bluetooth hardware:

Xcode -> Runners -> Targets -> Runner-> Signing & Capabilities -> App Sandbox -> Hardware -> Enable Bluetooth

Using Ble in App Background

This is an advanced use case. FlutterBluePlus does not support everything. You may have to fork it. PRs are welcome.

iOS

Documentation: https://developer.apple.com/library/archive/documentation/NetworkingInternetWeb/Conceptual/CoreBluetooth_concepts/CoreBluetoothBackgroundProcessingForIOSApps/PerformingTasksWhileYourAppIsInTheBackground.html

Add the following to your Info.plist

<key>UIBackgroundModes</key>
<array>
    <string>bluetooth-central</string>
</array>

When this key-value pair is included in the app’s Info.plist file, the system wakes up your app to process ble read, write, and subscription events.

You may also have to use https://pub.dev/packages/workmanager

Note: Upon being woken up, an app has around 10 seconds to complete a task. Apps that spend too much time executing in the background can be throttled back by the system or killed.

Android

You can try using https://pub.dev/packages/flutter_foreground_task or possibly https://pub.dev/packages/workmanager

Reference

🌀 = Stream ⚡ = synchronous

FlutterBluePlus API

	Android	iOS	Throws	Description
setLogLevel	✅	✅		Configure plugin log level
setOptions	✅	✅		Set configurable bluetooth options
isSupported	✅	✅		Checks whether the device supports Bluetooth
turnOn	✅		🔥	Turns on the bluetooth adapter
adapterStateNow ⚡	✅	✅		Current state of the bluetooth adapter
adapterState 🌀	✅	✅		Stream of on & off states of the bluetooth adapter
startScan	✅	✅	🔥	Starts a scan for Ble devices
stopScan	✅	✅	🔥	Stop an existing scan for Ble devices
onScanResults 🌀	✅	✅		Stream of live scan results
scanResults 🌀	✅	✅		Stream of live scan results or previous results
lastScanResults ⚡	✅	✅		The most recent scan results
isScanning 🌀	✅	✅		Stream of current scanning state
isScanningNow ⚡	✅	✅		Is a scan currently running?
connectedDevices ⚡	✅	✅		List of devices connected to your app
systemDevices	✅	✅	🔥	List of devices connected to the system, even by other apps
getPhySupport	✅		🔥	Get supported bluetooth phy codings

FlutterBluePlus Events API

	Android	iOS	Description
events.onConnectionStateChanged 🌀	✅	✅	Stream of connection changes of all devices
events.onMtuChanged 🌀	✅	✅	Stream of mtu changes of all devices
events.onReadRssi 🌀	✅	✅	Stream of rssi reads of all devices
events.onServicesReset 🌀	✅	✅	Stream of services resets of all devices
events.onDiscoveredServices 🌀	✅	✅	Stream of services discovered of all devices
events.onCharacteristicReceived 🌀	✅	✅	Stream of characteristic value reads of all devices
events.onCharacteristicWritten 🌀	✅	✅	Stream of characteristic value writes of all devices
events.onDescriptorRead 🌀	✅	✅	Stream of descriptor value reads of all devices
events.onDescriptorWritten 🌀	✅	✅	Stream of descriptor value writes of all devices
events.onBondStateChanged 🌀	✅		Stream of android bond state changes of all devices
events.onNameChanged 🌀		✅	Stream of iOS name changes of all devices

BluetoothDevice API

	Android	iOS	Throws	Description
platformName ⚡	✅	✅		The platform preferred name of the device
advName ⚡	✅	✅		The advertised name of the device found during scanning
connect	✅	✅	🔥	Establishes a connection to the device
disconnect	✅	✅	🔥	Cancels an active or pending connection to the device
isConnected ⚡	✅	✅		Is this device currently connected to your app?
isDisonnected ⚡	✅	✅		Is this device currently disconnected from your app?
connectionState 🌀	✅	✅		Stream of connection changes for the Bluetooth Device
discoverServices	✅	✅	🔥	Discover services
servicesList ⚡	✅	✅		The current list of available services
onServicesReset 🌀	✅	✅		The services changed & must be rediscovered
mtu 🌀	✅	✅		Stream of current mtu value + changes
mtuNow ⚡	✅	✅		The current mtu value
readRssi	✅	✅	🔥	Read RSSI from a connected device
requestMtu	✅		🔥	Request to change the MTU for the device
requestConnectionPriority	✅		🔥	Request to update a high priority, low latency connection
bondState 🌀	✅			Stream of device bond state. Can be useful on Android
createBond	✅		🔥	Force a system pairing dialogue to show, if needed
removeBond	✅		🔥	Remove Bluetooth Bond of device
setPreferredPhy	✅		🔥	Set preferred RX and TX phy for connection and phy options
clearGattCache	✅		🔥	Clear android cache of service discovery results

BluetoothCharacteristic API

	Android	iOS	Throws	Description
uuid ⚡	✅	✅		The uuid of characteristic
read	✅	✅	🔥	Retrieves the value of the characteristic
write	✅	✅	🔥	Writes the value of the characteristic
setNotifyValue	✅	✅	🔥	Sets notifications or indications on the characteristic
isNotifying ⚡	✅	✅		Are notifications or indications currently enabled
onValueReceived 🌀	✅	✅		Stream of characteristic value updates received from the device
lastValue ⚡	✅	✅		The most recent value of the characteristic
lastValueStream 🌀	✅	✅		Stream of onValueReceived + writes

BluetoothDescriptor API

	Android	iOS	Throws	Description
uuid ⚡	✅	✅		The uuid of descriptor
read	✅	✅	🔥	Retrieves the value of the descriptor
write	✅	✅	🔥	Writes the value of the descriptor
onValueReceived 🌀	✅	✅		Stream of descriptor value reads & writes
lastValue ⚡	✅	✅		The most recent value of the descriptor
lastValueStream 🌀	✅	✅		Stream of onValueReceived + writes

Debugging

The easiest way to debug issues in FlutterBluePlus is to make your own local copy.

cd /user/downloads
git clone https://github.com/boskokg/flutter_blue_plus.git

then in pubspec.yaml add the repo by path:

  flutter_blue_plus:
    path: /user/downloads/flutter_blue_plus

Now you can edit the FlutterBluePlus code yourself.

Common Problems

Many common problems are easily solved.

Adapter:

Scanning:

Connecting:

Reading & Writing:

Subscriptions:

Android Errors:

Flutter Errors:

MissingPluginException(No implementation found for method XXXX ...)

"bluetooth must be turned on"

You need to wait for the bluetooth adapter to fully turn on.

await FlutterBluePlus.adapterState.where((state) => state == BluetoothAdapterState.on).first;

You can also use FlutterBluePlus.adapterState.listen(...). See Usage.

adapterState is not 'on' but my Bluetooth is on

For iOS:

adapterState always starts as unknown. You need to wait longer for the service to initialize. Use this code:

// wait for actual adapter state, up to 3 seconds
Set<BluetoothAdapterState> inProgress = {BluetoothAdapterState.unknown, BluetoothAdapterState.turningOn};
var adapterState = FlutterBluePlus.adapterState.where((v) => !inProgress.contains(v)).first;
await adapterState.timeout(const Duration(seconds: 3)).onError((error, stackTrace) {
   throw Exception("Could not determine Bluetooth state. ${FlutterBluePlus.adapterStateNow}");
});

// check adapter state
if (FlutterBluePlus.adapterStateNow != BluetoothAdapterState.on) {
   throw Exception("Bluetooth Is Not On. ${FlutterBluePlus.adapterStateNow}");
}

If adapterState is unavailable, you must add access to Bluetooth Hardware in the app's Xcode settings. See Getting Started.

For Android:

Check that your device supports Bluetooth & has permissions.

adapterState is called multiple times

You are forgetting to cancel the original FlutterBluePlus.adapterState.listen resulting in multiple listeners.

// tip: using ??= makes it easy to only make new listener when currently null
final subscription ??= FlutterBluePlus.adapterState.listen((value) {
    // ...
});

// also, make sure you cancel the subscription when done!
subscription.cancel()

Scanning does not find my device

1. you're using an emulator

Use a physical device.

2. try using another ble scanner app

iOS: nRF Connect
Android: BLE Scanner

Install a BLE scanner app on your phone. Can it find your device?

3. your device uses bluetooth classic, not BLE.

Headphones, speakers, keyboards, mice, gamepads, & printers all use Bluetooth Classic.

These devices may be found in System Settings, but they cannot be connected to by FlutterBluePlus. FlutterBluePlus only supports Bluetooth Low Energy.

4. your device stopped advertising.

you might need to reboot your device
you might need to put your device in "discovery mode"
your phone may have already connected automatically
another app may have already connected to your device
another phone may have already connected to your device

Try looking through system devices:

// search system devices. i.e. any device connected to by *any* app
List<BluetoothDevice> system = await FlutterBluePlus.systemDevices;
for (var d in system) {
    print('${r.device.platformName} already connected to! ${r.device.remoteId}');
    if (d.platformName == "myBleDevice") {
         await r.connect(); // must connect our app
    }
}

5. your scan filters are wrong.

try removing all scan filters
for withServices to work, your device must actively advertise the serviceUUIDs it supports

6. Android: you're calling startScan too often

On Adroid you can only call startScan 5 times per 30 second period. This is a platform restriction.

Scanned device never goes away

This is expected.

You must set the removeIfGone scan option if you want the device to go away when no longer available.

iBeacons Not Showing

iOS:

iOS does not support iBeacons using CoreBluetooth. You must find a plugin meant for CoreLocation.

Android:

you need to enable location permissions, see Getting Started
you must pass androidUsesFineLocation:true to the startScan method.

Connection fails

1. Your ble device may be low battery

Bluetooth can become erratic when your peripheral device is low on battery.

2. Your ble device may have refused the connection or have a bug

Connection is a two-way process. Your ble device may be misconfigured.

3. You may be on the edge of the Bluetooth range.

The signal is too weak, or there are a lot of devices causing radio interference.

4. Some phones have an issue connecting while scanning.

The Huawei P8 Lite is one of the reported phones to have this issue. Try stopping your scanner before connecting.

5. Try restarting your phone

Bluetooth is a complicated system service, and can enter a bad state.

connectionState is called multiple times

You are forgetting to cancel the original device.connectionState.listen resulting in multiple listeners.

// tip: using ??= makes it easy to only make new listener when currently null
final subscription ??= FlutterBluePlus.device.connectionState.listen((value) {
    // ...
});

// also, make sure you cancel the subscription when done!
subscription.cancel()

The remoteId is different on Android versus iOS & macOS

This is expected. There is no way to avoid it.

For privacy, iOS & macOS use a randomly generated uuid. This uuid will periodically change.

e.g. 6920a902-ba0e-4a13-a35f-6bc91161c517

Android uses the mac address of the bluetooth device. It never changes.

e.g. 05:A4:22:31:F7:ED

iOS: "[Error] The connection has timed out unexpectedly."

You can google this error. It is a common iOS ble error code.

It means your device stopped working. FlutterBluePlus cannot fix it.

List of Bluetooth GATT Errors

These GATT error codes are part of the BLE Specification.

These are responses from your ble device because you are sending an invalid request.

FlutterBluePlus cannot fix these errors. You are doing something wrong & your device is responding with an error.

GATT errors as they appear on iOS:

apple-code: 1  | The handle is invalid.
apple-code: 2  | Reading is not permitted.
apple-code: 3  | Writing is not permitted.
apple-code: 4  | The command is invalid.
apple-code: 6  | The request is not supported.
apple-code: 7  | The offset is invalid.
apple-code: 8  | Authorization is insufficient.
apple-code: 9  | The prepare queue is full.
apple-code: 10 | The attribute could not be found.
apple-code: 11 | The attribute is not long.
apple-code: 12 | The encryption key size is insufficient.
apple-code: 13 | The value's length is invalid.
apple-code: 14 | Unlikely error.
apple-code: 15 | Encryption is insufficient.
apple-code: 16 | The group type is unsupported.
apple-code: 17 | Resources are insufficient.
apple-code: 18 | Unknown ATT error.

GATT errors as they appear on Android:

android-code: 1  | GATT_INVALID_HANDLE
android-code: 2  | GATT_READ_NOT_PERMITTED
android-code: 3  | GATT_WRITE_NOT_PERMITTED
android-code: 4  | GATT_INVALID_PDU
android-code: 5  | GATT_INSUFFICIENT_AUTHENTICATION
android-code: 6  | GATT_REQUEST_NOT_SUPPORTED
android-code: 7  | GATT_INVALID_OFFSET
android-code: 8  | GATT_INSUFFICIENT_AUTHORIZATION
android-code: 9  | GATT_PREPARE_QUEUE_FULL
android-code: 10 | GATT_ATTR_NOT_FOUND
android-code: 11 | GATT_ATTR_NOT_LONG
android-code: 12 | GATT_INSUFFICIENT_KEY_SIZE
android-code: 13 | GATT_INVALID_ATTRIBUTE_LENGTH
android-code: 14 | GATT_UNLIKELY
android-code: 15 | GATT_INSUFFICIENT_ENCRYPTION
android-code: 16 | GATT_UNSUPPORTED_GROUP
android-code: 17 | GATT_INSUFFICIENT_RESOURCES

Descriptions:

1   | Invalid Handle                 | The attribute handle given was not valid on this server.
2   | Read Not Permitted             | The attribute cannot be read.
3   | Write Not Permitted            | The attribute cannot be written.
4   | Invalid PDU                    | The attribute PDU was invalid.
5   | Insufficient Authentication    | The attribute requires authentication before it can be read or written.
6   | Request Not Supported          | Attribute server does not support the request received from the client.
7   | Invalid Offset                 | Offset specified was past the end of the attribute.
8   | Insufficient Authorization     | The attribute requires an authorization before it can be read or written.
9   | Prepare Queue Full             | Too many prepare writes have been queued.
10  | Attribute Not Found            | No attribute found within the given attribute handle range.
11  | Attribute Not Long             | The attribute cannot be read or written using the Read Blob or Write Blob requests.
12  | Insufficient Key Size          | The Encryption Key Size used for encrypting this link is insufficient.
13  | Invalid Attribute Value Length | The attribute value length is invalid for the operation.
14  | Unlikely Error                 | The request has encountered an unlikely error and cannot be completed.
15  | Insufficient Encryption        | The attribute requires encryption before it can be read or written.
16  | Unsupported Group Type         | The attribute type is not a supported grouping as defined by a higher layer.
17  | Insufficient Resources         | Insufficient Resources to complete the request.

characteristic write fails

First, check the List of Bluetooth GATT Errors for your error.

1. your bluetooth device turned off, or is out of range

If your device turns off or crashes during a write, it will cause a failure.

2. Your Bluetooth device has bugs

Maybe your device crashed, or is not sending a response due to software bugs.

3. there is radio interference

Bluetooth is wireless and will not always work.

Characteristic read fails

First, check the List of Bluetooth GATT Errors for your error.

1. your bluetooth device turned off, or is out of range

If your device turns off or crashes during a read, it will cause a failure.

2. Your Bluetooth device has bugs

Maybe your device crashed, or is not sending a response due to software bugs.

3. there is radio interference

Bluetooth is wireless and will not always work.

onValueReceived is never called (or lastValueStream)

1. you are not calling the right function

lastValueStream is called for await chr.read() & await chr.write() & await chr.setNotifyValue(true)

onValueReceived is only called for await chr.read() & await chr.setNotifyValue(true)

2. your device has nothing to send

If you are using await chr.setNotifyValue(true), your device chooses when to send data.

Try interacting with your device to get it to send new data.

3. your device has bugs

Try rebooting your ble device.

Some ble devices have buggy software and stop sending data

onValueReceived data is split up (or lastValueStream)

Verify that the mtu is large enough to hold your message.

device.mtu

If it still happens, it is a problem with your peripheral device.

onValueReceived is called with duplicate data (or lastValueStream)

You are probably forgetting to cancel the original chr.onValueReceived.listen resulting in multiple listens.

The easiest solution is to use device.cancelWhenDisconnected(subscription) to cancel device subscriptions.

final subscription = chr.onValueReceived.listen((value) {
    // ...
});

// make sure you have this line!
device.cancelWhenDisconnected(subscription);

await characteristic.setNotifyValue(true);

ANDROID_SPECIFIC_ERROR

There is no 100% solution.

FBP already has mitigations for this error, but Android will still fail with this code randomly.

The recommended solution is to catch the error, and retry.

android pairing popup appears twice

This is a bug in android itself.

You can call createBond() yourself just after connecting and this will resolve the issue.

MissingPluginException(No implementation found for method XXXX ...)

If you just added flutter_blue_plus to your pubspec.yaml, a hot reload / hot restart is not enough.

You need to fully stop your app and run again so that the native plugins are loaded.

Also try flutter clean.

youegraillot/lidarr-on-steroids

2024-08-21T01:29:21Z

Lidarr with some muscles thanks to deemix

lidarr-on-steroids

I'm looking for maintainers, pm me if you're interested !

This repository bundles a modded version of Lidarr and Deemix into a docker image featuring :

Native Deemix integration as an indexer and downloader for Lidarr
Automatic Lidarr and Deemix configuration
Automatic conversion from any format with ffmpeg
Podman compatibility with rootless mode

This allows an easy deployment, with the advantage of having a direct control over Deemix indexing and downloader capacities into Lidarr :

Usage

Parameters

Parameter	Function
`-p 8686`	Lidarr WebUI
`-p 6595`	Deemix WebUI
`-e PUID=1000`	for UserID
`-e PGID=1000`	for GroupID
`-e AUTOCONFIG=true`	Enable automatic configuration - see below for explanation
`-e FLAC2CUSTOM_ARGS=""`	Sets arguments used when calling flac2custom.sh
`-e CLEAN_DOWNLOADS=true`	Enable cleaning empty folders in /downloads
`-v /config`	Configuration files for Lidarr.
`-v /config_deemix`	Configuration files for Deemix.
`-v /downloads`	Path to your download folder for music.
`-v /music`	Music files.

Docker Run

docker run \
  --name lidarr \
  -p 8686:8686 \
  -p 6595:6595 \
  -v <path>:/config \
  -v <path>:/config_deemix \
  -v <path>:/downloads \
  -v <path>:/music \
  --restart unless-stopped \
 youegraillot/lidarr-on-steroids

Docker Compose

version: "3"
services:
  lidarr:
    image: youegraillot/lidarr-on-steroids
    restart: unless-stopped
    ports:
      - "8686:8686" # Lidarr web UI
      - "6595:6595" # Deemix web UI
    volumes:
      - <path>:/config
      - <path>:/config_deemix
      - <path>:/downloads
      - <path>:/music

Automatic configuration

Deemix comes with optimal settings allowing Lidarr integration, in particular regarding the folder structure ("createCDFolder" is required for this to work). DEEMIX_SINGLE_USER environment variable is also set to true to allow the setup script to read the corresponding ARL.

The setup service will install the Deemix plugin. This requires Lidarr to be restarted once.

In AUTOCONFIG mode (default), the only manual manipulation you'll only have to fill your Deezer credentials in Deemix web UI (port 6595 by default). Once the /config_deemix/login.json is filled with the resulting ARL, the setup will be able to create the following :

/music root folder if no other root folder is configured
Delay profile allowing Deemix to be used by automatic search
Deemix as an indexer
Deemix as a download client
Flac2Custom script connection if FLAC2CUSTOM_ARGS is set
clean-downloads script connection to keep your downloads folder clean after each imports

In case you don't want the automagical part (which is really the only value of this image), just set AUTOCONFIG environment variable to false.

Audio files conversion

The image uses a modded version of lidarr-flac2mp3 allowing conversion from any format.

To enable conversion on Lidarr import, create a new Connection to a Custom Script. You can also provide your own custom conversion script, see lidarr-flac2mp3 for more information.

In AUTOCONFIG, if FLAC2CUSTOM_ARGS is set and no other connection to flac2* is found, this step done for you :

Acknowledgment

This project is just a compilation of various tools made possible by these projects :

Lidarr and especially ta264 for the plugin integration
lidarr-flac2mp3 for the format conversion script
Deemix for the downloader backend
hotio for the base docker image

Alternatively, you could use Deemixrr which pretty much offers the same functionalities without the *starr of the various Sonarr forks.

MTES-MCT/ecobalyse

2024-08-22T01:29:37Z

Accélerer la mise en place de l'affichage environnemental

sourcegraph/sourcegraph-public-snapshot

2024-08-22T01:30:06Z

Code AI platform with Code Search & Cody

Docs • Contributing • Twitter • Discord

Sourcegraph makes it easy to read, write, and fix code—even in big, complex codebases.

Code search: Search all of your repositories across all branches and all code hosts.
Code intelligence: Navigate code, find references, see code owners, trace history, and more.
Fix and refactor: Roll out large-scale changes to many repositories at once and track big migrations.

Getting started

Development

Refer to the Developing Sourcegraph guide to get started.

Documentation

The doc directory has additional documentation for developing and understanding Sourcegraph:

Architecture: high-level architecture
Database setup: database best practices
Go style guide
Documentation style guide
GraphQL API: useful tips when modifying the GraphQL API
Contributing

License

This repository contains primarily non-OSS-licensed files. See LICENSE.

adnanh/webhook

2024-08-22T01:30:06Z

webhook is a lightweight incoming webhook server to run shell commands

What is webhook?

webhook is a lightweight configurable tool written in Go, that allows you to easily create HTTP endpoints (hooks) on your server, which you can use to execute configured commands. You can also pass data from the HTTP request (such as headers, payload or query variables) to your commands. webhook also allows you to specify rules which have to be satisfied in order for the hook to be triggered.

For example, if you're using Github or Bitbucket, you can use webhook to set up a hook that runs a redeploy script for your project on your staging server, whenever you push changes to the master branch of your project.

If you use Mattermost or Slack, you can set up an "Outgoing webhook integration" or "Slash command" to run various commands on your server, which can then report back directly to you or your channels using the "Incoming webhook integrations", or the appropriate response body.

webhook aims to do nothing more than it should do, and that is:

receive the request,
parse the headers, payload and query variables,
check if the specified rules for the hook are satisfied,
and finally, pass the specified arguments to the specified command via command line arguments or via environment variables.

Everything else is the responsibility of the command's author.

Not what you're looking for?


Scriptable webhook gateway to safely run your custom builds, deploys, and proxy scripts on your servers.	Inspect, monitor and replay webhooks without the back and forth troubleshooting.

Getting started

Installation

Building from source

To get started, first make sure you've properly set up your Go 1.14 or newer environment and then run

$ go build github.com/adnanh/webhook

to build the latest version of the webhook.

Using package manager

Snap store

Ubuntu

If you are using Ubuntu linux (17.04 or later), you can install webhook using sudo apt-get install webhook which will install community packaged version.

Debian

If you are using Debian linux ("stretch" or later), you can install webhook using sudo apt-get install webhook which will install community packaged version (thanks @freeekanayaka) from https://packages.debian.org/sid/webhook

FreeBSD

If you are using FreeBSD, you can install webhook using pkg install webhook.

Download prebuilt binaries

Prebuilt binaries for different architectures are available at GitHub Releases.

Configuration

Next step is to define some hooks you want webhook to serve. webhook supports JSON or YAML configuration files, but we'll focus primarily on JSON in the following example. Begin by creating an empty file named hooks.json. This file will contain an array of hooks the webhook will serve. Check Hook definition page to see the detailed description of what properties a hook can contain, and how to use them.

Let's define a simple hook named redeploy-webhook that will run a redeploy script located in /var/scripts/redeploy.sh. Make sure that your bash script has #!/bin/sh shebang on top.

Our hooks.json file will now look like this:

[
  {
    "id": "redeploy-webhook",
    "execute-command": "/var/scripts/redeploy.sh",
    "command-working-directory": "/var/webhook"
  }
]

NOTE: If you prefer YAML, the equivalent hooks.yaml file would be:

- id: redeploy-webhook
  execute-command: "/var/scripts/redeploy.sh"
  command-working-directory: "/var/webhook"

You can now run webhook using

$ /path/to/webhook -hooks hooks.json -verbose

It will start up on default port 9000 and will provide you with one HTTP endpoint

http://yourserver:9000/hooks/redeploy-webhook

Check webhook parameters page to see how to override the ip, port and other settings such as hook hotreload, verbose output, etc, when starting the webhook.

By performing a simple HTTP GET or POST request to that endpoint, your specified redeploy script would be executed. Neat!

However, hook defined like that could pose a security threat to your system, because anyone who knows your endpoint, can send a request and execute your command. To prevent that, you can use the "trigger-rule" property for your hook, to specify the exact circumstances under which the hook would be triggered. For example, you can use them to add a secret that you must supply as a parameter in order to successfully trigger the hook. Please check out the Hook rules page for detailed list of available rules and their usage.

Multipart Form Data

webhook provides limited support the parsing of multipart form data. Multipart form data can contain two types of parts: values and files. All form values are automatically added to the payload scope. Use the parse-parameters-as-json settings to parse a given value as JSON. All files are ignored unless they match one of the following criteria:

The Content-Type header is application/json.
The part is named in the parse-parameters-as-json setting.

In either case, the given file part will be parsed as JSON and added to the payload map.

Templates

webhook can parse the hooks configuration file as a Go template when given the -template CLI parameter. See the Templates page for more details on template usage.

Using HTTPS

webhook by default serves hooks using http. If you want webhook to serve secure content using https, you can use the -secure flag while starting webhook. Files containing a certificate and matching private key for the server must be provided using the -cert /path/to/cert.pem and -key /path/to/key.pem flags. If the certificate is signed by a certificate authority, the cert file should be the concatenation of the server's certificate followed by the CA's certificate.

TLS version and cipher suite selection flags are available from the command line. To list available cipher suites, use the -list-cipher-suites flag. The -tls-min-version flag can be used with -list-cipher-suites.

CORS Headers

If you want to set CORS headers, you can use the -header name=value flag while starting webhook to set the appropriate CORS headers that will be returned with each response.

Interested in running webhook inside of a Docker container?

You can use one of the following Docker images, or create your own (please read this discussion):

almir/webhook
roxedus/webhook
thecatlady/webhook
lwlook/webhook - This setup allows direct access to the Docker host, providing a streamlined and efficient way to manage webhooks.

Examples

Check out Hook examples page for more complex examples of hooks.

Guides featuring webhook

Plex 2 Telegram by @psyhomb
Webhook & JIRA by @perfecto25
Trigger Ansible AWX job runs on SCM (e.g. git) commit by @jpmens
Deploy using GitHub webhooks by @awea
Setting up Automatic Deployment and Builds Using Webhooks by Will Browning
Auto deploy your Node.js app on push to GitHub in 3 simple steps by Karolis Rusenas
Automate Static Site Deployments with Salt, Git, and Webhooks by Linode
Using Prometheus to Automatically Scale WebLogic Clusters on Kubernetes by Marina Kogan
Github Pages and Jekyll - A New Platform for LACNIC Labs by Carlos Martínez Cagnazzo
How to Deploy React Apps Using Webhooks and Integrating Slack on Ubuntu by Arslan Ud Din Shafiq
Private webhooks by Thomas
Adventures in webhooks by Drake
GitHub pro tips by Spencer Lyon
XiaoMi Vacuum + Amazon Button = Dash Cleaning by c0mmensal
Set up Automated Deployments From Github With Webhook by Maxim Orlov
Kick Me Now with Webhooks By Dave Cottlehuber
VIDEO: Gitlab CI/CD configuration using Docker and adnanh/webhook to deploy on VPS - Tutorial #1 by Yes! Let's Learn Software Engineering
Integrate automatic deployment in 20 minutes using webhooks + Nginx setup by Anksus
Automatically redeploy your static blog with Gitea, Uberspace & Webhook by Arran
Automatically Updating My Zola Site Using a Webhook by Óscar Fernández
...
Want to add your own? Open an Issue or create a PR :-)

Community Contributions

See the webhook-contrib repository for a collections of tools and helpers related to webhook that have been contributed by the webhook community.

Need help?

Check out existing issues to see if someone else also had the same problem, or open a new one.

Support active development

By contributing

This project exists thanks to all the people who contribute. Contribute!.

By giving money

Thank you to all our backers!

License

The MIT License (MIT)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

crytic/echidna

2024-08-22T01:30:24Z

Ethereum smart contract fuzzer

Echidna: A Fast Smart Contract Fuzzer

Echidna is a weird creature that eats bugs and is highly electrosensitive (with apologies to Jacob Stanley)

More seriously, Echidna is a Haskell program designed for fuzzing/property-based testing of Ethereum smart contracts. It uses sophisticated grammar-based fuzzing campaigns based on a contract ABI to falsify user-defined predicates or Solidity assertions. We designed Echidna with modularity in mind, so it can be easily extended to include new mutations or test specific contracts in specific cases.

Features

Generates inputs tailored to your actual code
Optional corpus collection, mutation and coverage guidance to find deeper bugs
Powered by Slither to extract useful information before the fuzzing campaign
Source code integration to identify which lines are covered after the fuzzing campaign
Interactive terminal UI, text-only or JSON output
Automatic test case minimization for quick triage
Seamless integration into the development workflow
Maximum gas usage reporting of the fuzzing campaign
Support for a complex contract initialization with Etheno and Truffle

.. and a beautiful high-resolution handcrafted logo.

Usage

Executing the test runner

The core Echidna functionality is an executable called echidna, which takes a contract and a list of invariants (properties that should always remain true) as input. For each invariant, it generates random sequences of calls to the contract and checks if the invariant holds. If it can find some way to falsify the invariant, it prints the call sequence that does so. If it can't, you have some assurance the contract is safe.

Writing invariants

Invariants are expressed as Solidity functions with names that begin with echidna_, have no arguments, and return a boolean. For example, if you have some balance variable that should never go below 20, you can write an extra function in your contract like this one:

function echidna_check_balance() public returns (bool) {
    return(balance >= 20);
}

To check these invariants, run:

$ echidna myContract.sol

An example contract with tests can be found tests/solidity/basic/flags.sol. To run it, you should execute:

$ echidna tests/solidity/basic/flags.sol

Echidna should find a call sequence that falsifies echidna_sometimesfalse and should be unable to find a falsifying input for echidna_alwaystrue.

Collecting and visualizing coverage

After finishing a campaign, Echidna can save a coverage maximizing corpus in a special directory specified with the corpusDir config option. This directory will contain two entries: (1) a directory named coverage with JSON files that can be replayed by Echidna and (2) a plain-text file named covered.txt, a copy of the source code with coverage annotations.

If you run tests/solidity/basic/flags.sol example, Echidna will save a few files serialized transactions in the coverage directory and a covered.$(date +%s).txt file with the following lines:

*r  |  function set0(int val) public returns (bool){
*   |    if (val % 100 == 0)
*   |      flag0 = false;
  }

*r  |  function set1(int val) public returns (bool){
*   |    if (val % 10 == 0 && !flag0)
*   |      flag1 = false;
  }

Our tool signals each execution trace in the corpus with the following "line marker":

* if an execution ended with a STOP
r if an execution ended with a REVERT
o if an execution ended with an out-of-gas error
e if an execution ended with any other error (zero division, assertion failure, etc)

Support for smart contract build systems

Echidna can test contracts compiled with different smart contract build systems, including Truffle or hardhat using crytic-compile. To invoke echidna with the current compilation framework, use echidna ..

On top of that, Echidna supports two modes of testing complex contracts. Firstly, one can describe an initialization procedure with Truffle and Etheno and use that as the base state for Echidna. Secondly, Echidna can call into any contract with a known ABI by passing in the corresponding Solidity source in the CLI. Use allContracts: true in your config to turn this on.

Crash course on Echidna

Our Building Secure Smart Contracts repository contains a crash course on Echidna, including examples, lessons and exercises.

Using Echidna in a GitHub Actions workflow

There is an Echidna action which can be used to run echidna as part of a GitHub Actions workflow. Please refer to the crytic/echidna-action repository for usage instructions and examples.

Configuration options

Echidna's CLI can be used to choose the contract to test and load a configuration file.

$ echidna contract.sol --contract TEST --config config.yaml

The configuration file allows users to choose EVM and test generation parameters. An example of a complete and annotated config file with the default options can be found at tests/solidity/basic/default.yaml. More detailed documentation on the configuration options is available in our wiki.

Echidna supports three different output drivers. There is the default text driver, a json driver, and a none driver, which should suppress all stdout output. The JSON driver reports the overall campaign as follows.

Campaign = {
  "success"      : bool,
  "error"        : string?,
  "tests"        : [Test],
  "seed"         : number,
  "coverage"     : Coverage,
  "gas_info"     : [GasInfo]
}
Test = {
  "contract"     : string,
  "name"         : string,
  "status"       : string,
  "error"        : string?,
  "testType"     : string,
  "transactions" : [Transaction]?
}
Transaction = {
  "contract"     : string,
  "function"     : string,
  "arguments"    : [string]?,
  "gas"          : number,
  "gasprice"     : number
}

Coverage is a dict describing certain coverage-increasing calls. Each GasInfo entry is a tuple that describes how maximal gas usage was achieved, and is also not too important. These interfaces are subject to change to be slightly more user-friendly at a later date. testType will either be property or assertion, and status always takes on either fuzzing, shrinking, solved, passed, or error.

Debugging Performance Problems

One way to diagnose Echidna's performance issues is to run echidna with profiling on. To run Echidna with basic profiling, add +RTS -p -s to your original echidna command:

$ nix develop # alternatively nix-shell
$ cabal --enable-profiling run echidna -- ... +RTS -p -s
$ less echidna.prof

This produces a report file (echidna.prof), that shows which functions take up the most CPU and memory usage.

If the basic profiling doesn't help, you can use more advanced profiling techniques.

Common causes for performance issues that we observed:

Costly functions called in hot paths
Lazy data constructors that accumulate thunks
Inefficient data structures used in hot paths

Checking for these is a good place to start. If you suspect some computation is too lazy and leaks memory, you can use force from Control.DeepSeq to make sure it gets evaluated.

Limitations and known issues

EVM emulation and testing are hard. Echidna has some limitations in the latest release. Some of these are inherited from hevm while some are results from design/performance decisions or simply bugs in our code. We list them here including their corresponding issue and the status ("wont fix", "on hold", "in review", "fixed"). Issues that are "fixed" are expected to be included in the next Echidna release.

Description	Issue	Status
Vyper support is limited	#652	wont fix
Limited library support for testing	#651	wont fix

Installation

Precompiled binaries

Before starting, make sure Slither is installed (pip3 install slither-analyzer --user). If you want to quickly test Echidna in Linux or MacOS, we provide statically linked Linux binaries built on Ubuntu and mostly static MacOS binaries on our releases page. You can also grab the same type of binaries from our CI pipeline, just click the commit to find binaries for Linux or MacOS.

Homebrew (macOS / Linux)

If you have Homebrew installed on your Mac or Linux machine, you can install Echidna and all of its dependencies (Slither, crytic-compile) by running brew install echidna.

You can also compile and install the latest master branch code by running brew install --HEAD echidna

You can get further information in the echidna Homebrew Formula page. The formula itself is maintained as part of the homebrew-core repository

Docker container

If you prefer to use a pre-built Docker container, check out our docker package, which is auto-built via GitHub Actions. The echidna container is based on ubuntu:focal and it is meant to be a small yet flexible enough image to use Echidna on. It provides a pre-built version of echidna, as well as slither, crytic-compile, solc-select and nvm under 200 MB.

Note that the container images currently only build on x86 systems. Running them on ARM devices, such as Mac M1 systems, is not recommended due to the performance loss incurred by the CPU emulation.

Different tags are available for the Docker container image:

Tag	Build in tag
`vx.y.z`	Build corresponding to release `vx.y.z`
`latest`	Latest Echidna tagged release.
`edge`	Most recent commit on the default branch.
`testing-foo`	Testing build based on the `foo` branch.

To run the container with the latest Echidna version interactively, you can use something like the following command. It will map the current directory as /src inside the container, and give you a shell where you can use echidna:

$ docker run --rm -it -v `pwd`:/src ghcr.io/crytic/echidna/echidna

Otherwise, if you want to locally build the latest version of Echidna, we recommend using Docker. From within a clone of this repository, run the following command to build the Docker container image:

$ docker build -t echidna -f docker/Dockerfile --target final-ubuntu .

Then, you can run the echidna image locally. For example, to install solc 0.5.7 and check tests/solidity/basic/flags.sol, you can run:

$ docker run -it -v `pwd`:/src echidna bash -c "solc-select install 0.5.7 && solc-select use 0.5.7 && echidna /src/tests/solidity/basic/flags.sol"

Building using Stack

If you'd prefer to build from source, use Stack. stack install should build and compile echidna in ~/.local/bin. You will need to link against libreadline and libsecp256k1 (built with recovery enabled), which should be installed with the package manager of your choosing. You also need to install the latest release of libff. Refer to our CI tests for guidance.

Some Linux distributions do not ship static libraries for certain things that Haskell needs, e.g. Arch Linux, which will cause stack build to fail with linking errors because we use the -static flag. In that case, use --flag echidna:-static to produce a dynamically linked binary.

If you're getting errors building related to linking, try tinkering with --extra-include-dirs and --extra-lib-dirs.

Building using Nix (works natively on Apple M1 systems)

Nix users can install the latest Echidna with:

$ nix-env -i -f https://github.com/crytic/echidna/tarball/master

With flakes enabled, you can run Echidna straight from this repo:

$ nix run github:crytic/echidna # master
$ nix run github:crytic/echidna/v2.1.1 # specific ref (tag/branch/commit)

To build a standalone release for non-Nix macOS systems, the following will build Echidna in a mostly static binary. This can also be used on Linux systems to produce a fully static binary.

$ nix build .#echidna-redistributable

Nix will automatically install all the dependencies required for development including crytic-compile and solc. A quick way to start developing Echidna:

$ git clone https://github.com/crytic/echidna
$ cd echidna
$ nix develop # alternatively nix-shell
[nix-shell]$ cabal run echidna
[nix-shell]$ cabal run tests
[nix-shell]$ cabal new-repl

Public use of Echidna

Property testing suites

This is a partial list of smart contracts projects that use Echidna for testing:

Security reviews

The following shows public security reviews that used Echidna to uncover vulnerabilities

Trophies

The following security vulnerabilities were found by Echidna. If you found a security vulnerability using our tool, please submit a PR with the relevant information.

Project	Vulnerability	Date
0x Protocol	If an order cannot be filled, then it cannot be canceled	Oct 2019
0x Protocol	If an order can be partially filled with zero, then it can be partially filled with one token	Oct 2019
0x Protocol	The cobbdouglas function does not revert when valid input parameters are used	Oct 2019
Balancer Core	An attacker cannot steal assets from a public pool	Jan 2020
Balancer Core	An attacker cannot generate free pool tokens with joinPool	Jan 2020
Balancer Core	Calling joinPool-exitPool does not lead to free pool tokens	Jan 2020
Balancer Core	Calling exitswapExternAmountOut does not lead to free assets	Jan 2020
Liquity Dollar	Closing troves require to hold the full amount of LUSD minted	Dec 2020
Liquity Dollar	Troves can be improperly removed	Dec 2020
Liquity Dollar	Initial redeem can revert unexpectedly	Dec 2020
Liquity Dollar	Redeem without redemptions might still return success	Dec 2020
Origin Dollar	Users are allowed to transfer more tokens that they have	Nov 2020
Origin Dollar	User balances can be larger than total supply	Nov 2020
Yield Protocol	Arithmetic computation for buying and selling tokens is imprecise	Aug 2020

Research

We can also use Echidna to reproduce research examples from smart contract fuzzing papers to show how quickly it can find the solution. All of these can be solved, in a few seconds to one or two minutes on a laptop computer.

Source	Code
Using automatic analysis tools with MakerDAO contracts	SimpleDSChief
Integer precision bug in Sigma Prime	VerifyFunWithNumbers
Learning to Fuzz from Symbolic Execution with Application to Smart Contracts	Crowdsale
Harvey: A Greybox Fuzzer for Smart Contracts	Foo, Baz

Academic Publications

Paper Title	Venue	Publication Date
echidna-parade: Diverse multicore smart contract fuzzing	ISSTA 2021	July 2021
Echidna: Effective, usable, and fast fuzzing for smart contracts	ISSTA 2020	July 2020
Echidna: A Practical Smart Contract Fuzzer	FC 2020	Feb 2020

If you are using Echidna for academic work, consider applying to the Crytic $10k Research Prize.

Getting help

Feel free to stop by our #ethereum slack channel in Empire Hacking for help using or extending Echidna.

Get started by reviewing these simple Echidna invariants
Considering emailing the Echidna development team directly for more detailed questions

License

Echidna is licensed and distributed under the AGPLv3 license.

LMS-Community/slimserver-platforms

2024-08-22T01:30:43Z

Platform-specific build tools for Lyrion Music Server

DImuthuUpe/AndroidPdfViewer

2024-08-21T01:30:42Z

Android view for displaying PDFs rendered with PdfiumAndroid

Change of ownership and looking for contributors!

The ownership of the project was recently changed and we are actively looking for contributors to bring the project back to track. Please visit

Android PdfViewer

AndroidPdfViewer 1.x is available on AndroidPdfViewerV1 repo, where can be developed independently. Version 1.x uses different engine for drawing document on canvas, so if you don't like 2.x version, try 1.x.

Library for displaying PDF documents on Android, with animations, gestures, zoom and double tap support. It is based on PdfiumAndroid for decoding PDF files. Works on API 11 (Android 3.0) and higher. Licensed under Apache License 2.0.

What's new in 3.2.0-beta.1?

Merge PR #714 with optimized page load
Merge PR #776 with fix for max & min zoom level
Merge PR #722 with fix for showing right position when view size changed
Merge PR #703 with fix for too many threads
Merge PR #702 with fix for memory leak
Merge PR #689 with possibility to disable long click
Merge PR #628 with fix for hiding scroll handle
Merge PR #627 with fitEachPage option
Merge PR #638 and #406 with fixed NPE
Merge PR #780 with README fix
Update compile SDK and support library to 28
Update Gradle and Gradle Plugin

Changes in 3.0 API

Replaced Contants.PRELOAD_COUNT with PRELOAD_OFFSET
Removed PDFView#fitToWidth() (variant without arguments)
Removed Configurator#invalidPageColor(int) method as invalid pages are not rendered
Removed page size parameters from OnRenderListener#onInitiallyRendered(int) method, as document may have different page sizes
Removed PDFView#setSwipeVertical() method

Installation

Add to build.gradle:

implementation 'com.github.barteksc:android-pdf-viewer:3.2.0-beta.1'

or if you want to use more stable version:

implementation 'com.github.barteksc:android-pdf-viewer:2.8.2'

Library is available in jcenter repository, probably it'll be in Maven Central soon.

ProGuard

If you are using ProGuard, add following rule to proguard config file:

-keep class com.shockwave.**

Include PDFView in your layout

<com.github.barteksc.pdfviewer.PDFView
        android:id="@+id/pdfView"
        android:layout_width="match_parent"
        android:layout_height="match_parent"/>

Load a PDF file

All available options with default values:

pdfView.fromUri(Uri)
or
pdfView.fromFile(File)
or
pdfView.fromBytes(byte[])
or
pdfView.fromStream(InputStream) // stream is written to bytearray - native code cannot use Java Streams
or
pdfView.fromSource(DocumentSource)
or
pdfView.fromAsset(String)
    .pages(0, 2, 1, 3, 3, 3) // all pages are displayed by default
    .enableSwipe(true) // allows to block changing pages using swipe
    .swipeHorizontal(false)
    .enableDoubletap(true)
    .defaultPage(0)
    // allows to draw something on the current page, usually visible in the middle of the screen
    .onDraw(onDrawListener)
    // allows to draw something on all pages, separately for every page. Called only for visible pages
    .onDrawAll(onDrawListener)
    .onLoad(onLoadCompleteListener) // called after document is loaded and starts to be rendered
    .onPageChange(onPageChangeListener)
    .onPageScroll(onPageScrollListener)
    .onError(onErrorListener)
    .onPageError(onPageErrorListener)
    .onRender(onRenderListener) // called after document is rendered for the first time
    // called on single tap, return true if handled, false to toggle scroll handle visibility
    .onTap(onTapListener)
    .onLongPress(onLongPressListener)
    .enableAnnotationRendering(false) // render annotations (such as comments, colors or forms)
    .password(null)
    .scrollHandle(null)
    .enableAntialiasing(true) // improve rendering a little bit on low-res screens
    // spacing between pages in dp. To define spacing color, set view background
    .spacing(0)
    .autoSpacing(false) // add dynamic spacing to fit each page on its own on the screen
    .linkHandler(DefaultLinkHandler)
    .pageFitPolicy(FitPolicy.WIDTH) // mode to fit pages in the view
    .fitEachPage(false) // fit each page to the view, else smaller pages are scaled relative to largest page.
    .pageSnap(false) // snap pages to screen boundaries
    .pageFling(false) // make a fling change only a single page like ViewPager
    .nightMode(false) // toggle night mode
    .load();

pages is optional, it allows you to filter and order the pages of the PDF as you need

Scroll handle

Scroll handle is replacement for ScrollBar from 1.x branch.

From version 2.1.0 putting PDFView in RelativeLayout to use ScrollHandle is not required, you can use any layout.

To use scroll handle just register it using method Configurator#scrollHandle(). This method accepts implementations of ScrollHandle interface.

There is default implementation shipped with AndroidPdfViewer, and you can use it with .scrollHandle(new DefaultScrollHandle(this)). DefaultScrollHandle is placed on the right (when scrolling vertically) or on the bottom (when scrolling horizontally). By using constructor with second argument (new DefaultScrollHandle(this, true)), handle can be placed left or top.

You can also create custom scroll handles, just implement ScrollHandle interface. All methods are documented as Javadoc comments on interface source.

Document sources

Version 2.3.0 introduced document sources, which are just providers for PDF documents. Every provider implements DocumentSource interface. Predefined providers are available in com.github.barteksc.pdfviewer.source package and can be used as samples for creating custom ones.

Predefined providers can be used with shorthand methods:

pdfView.fromUri(Uri)
pdfView.fromFile(File)
pdfView.fromBytes(byte[])
pdfView.fromStream(InputStream)
pdfView.fromAsset(String)

Custom providers may be used with pdfView.fromSource(DocumentSource) method.

Links

Version 3.0.0 introduced support for links in PDF documents. By default, DefaultLinkHandler is used and clicking on link that references page in same document causes jump to destination page and clicking on link that targets some URI causes opening it in default application.

You can also create custom link handlers, just implement LinkHandler interface and set it using Configurator#linkHandler(LinkHandler) method. Take a look at DefaultLinkHandler source to implement custom behavior.

Pages fit policy

Since version 3.0.0, library supports fitting pages into the screen in 3 modes:

WIDTH - width of widest page is equal to screen width
HEIGHT - height of highest page is equal to screen height
BOTH - based on widest and highest pages, every page is scaled to be fully visible on screen

Apart from selected policy, every page is scaled to have size relative to other pages.

Fit policy can be set using Configurator#pageFitPolicy(FitPolicy). Default policy is WIDTH.

Additional options

Bitmap quality

By default, generated bitmaps are compressed with RGB_565 format to reduce memory consumption. Rendering with ARGB_8888 can be forced by using pdfView.useBestQuality(true) method.

Double tap zooming

There are three zoom levels: min (default 1), mid (default 1.75) and max (default 3). On first double tap, view is zoomed to mid level, on second to max level, and on third returns to min level. If you are between mid and max levels, double tapping causes zooming to max and so on.

Zoom levels can be changed using following methods:

void setMinZoom(float zoom);
void setMidZoom(float zoom);
void setMaxZoom(float zoom);

Possible questions

Why resulting apk is so big?

Android PdfViewer depends on PdfiumAndroid, which is set of native libraries (almost 16 MB) for many architectures. Apk must contain all this libraries to run on every device available on market. Fortunately, Google Play allows us to upload multiple apks, e.g. one per every architecture. There is good article on automatically splitting your application into multiple apks, available here. Most important section is Improving multiple APKs creation and versionCode handling with APK Splits, but whole article is worth reading. You only need to do this in your application, no need for forking PdfiumAndroid or so.

Why I cannot open PDF from URL?

Downloading files is long running process which must be aware of Activity lifecycle, must support some configuration, data cleanup and caching, so creating such module will probably end up as new library.

How can I show last opened page after configuration change?

You have to store current page number and then set it with pdfView.defaultPage(page), refer to sample app

How can I fit document to screen width (eg. on orientation change)?

Use FitPolicy.WIDTH policy or add following snippet when you want to fit desired page in document with different page sizes:

Configurator.onRender(new OnRenderListener() {
    @Override
    public void onInitiallyRendered(int pages, float pageWidth, float pageHeight) {
        pdfView.fitToWidth(pageIndex);
    }
});

How can I scroll through single pages like a ViewPager?

You can use a combination of the following settings to get scroll and fling behaviour similar to a ViewPager:

    .swipeHorizontal(true)
    .pageSnap(true)
    .autoSpacing(true)
    .pageFling(true)

One more thing

If you have any suggestions on making this lib better, write me, create issue or write some code and send pull request.

License

Created with the help of android-pdfview by Joan Zapata

Copyright 2017 Bartosz Schiller

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

bluewave-labs/bluewave-uptime

2024-08-22T01:30:53Z

BlueWave Uptime Monitor application

BlueWave Uptime

An open source server monitoring application

BlueWave Uptime is an open source server monitoring application used to track the operational status and performance of servers and websites. It regularly checks whether a server/website is accessible and performs optimally, providing real-time alerts and reports on the monitored services' availability, downtime, and response time.

Features

Completely open source, deployable on your servers
Website monitoring
Port monitoring
Ping monitoring
Incidents at a glance
Page speed monitoring
E-mail notifications
Scheduled maintenance (in the works)

Roadmap (short term):

Memory, disk and CPU monitoring
3rd party integrations
DNS monitoring
SSL monitoring

Roadmap (long term):

Status pages

Tech stack

Contributing

We love contributors. Here's how you can contribute:

Check Contributor's guideline.
Have a look at our Figma designs here. We encourage you to copy to your own Figma page, then work on it as it is read-only.
Open an issue if you believe you've encountered a bug
Make a pull request to add new features/make quality-of-life improvements/fix bugs.

Made with contrib.rocks.

Also check other developer and contributor-friendly projects of BlueWave:

Getting Started

Clone this repository to your local machine

Quickstart for Developers
Docker Compose
Installation (Client)
Configuration(Client)
- Environment
Getting Started (Server)
Endpoints
Auth
- POST /api/v1/auth/register
- POST /api/v1/auth/login
- PUT /api/v1/auth/user/{userId}
- GET /api/v1/auth/users
- GET /api/v1/auth/users/admin
- POST /api/v1/auth/invite
- POST /api/v1/auth/invite/verify/
- POST /api/v1/auth/recovery/request
- POST /api/v1/auth/recovery/validate
- POST /api/v1/auth/recovery/reset
Monitors
- GET /api/v1/monitors
- GET /api/v1/monitors/{monitorId}
- GET /api/v1/monitors/user/{userId}?limit
- POST /api/v1/monitors
- PUT /api/v1/monitors/{monitorId}
- DELETE /api/v1/monitors/{monitorId}
- DELETE /api/v1/monitors/all
Checks
- POST /api/v1/checks/{monitorId}
- GET /api/v1/checks/{monitorId}
- DELETE /api/v1/checks/{monitorId}
Alerts
- POST /api/v1/alerts/{monitorId}
- GET /api/v1/alerts/user/{userId}
- GET /api/v1/alerts/monitor/{monitorId}
- GET /api/v1/alerts/{alertId}
- PUT /api/v1/alerts/{alertId}
- DELETE /api/v1/alerts/{alertId}
Page Speed
- POST /api/v1/pagespeed/:monitorId
- GET /api/v1/pagespeed/:monitorId
- DELETE /api/v1/pagespeed/:monitorId
Maintenance Window
- POST /api/v1/maintenance-window/:monitorId
- GET /api/v1/maintenance-window/user/:userId
- GET /api/v1/maintenance-window/monitor/:monitorId
Error Handling
Contributors

Quickstart for Developers

MAKE SURE YOU CD TO THE SPECIFIED DIRECTORIES AS PATHS IN COMMANDS ARE RELATIVE

Cloning and Initial Setup

Clone this repository
Checkout the develop branch git checkout develop

Docker Images Setup

CD to the Docker directory
Run docker run -d -p 6379:6379 -v $(pwd)/redis/data:/data --name uptime_redis uptime_redis
Run docker run -d -p 27017:27017 -v $(pwd)/mongo/data:/data/db --name uptime_database_mongo uptime_database_mongo

Server Setup

CD to Server directory, run npm install
While in Server directory, create a .env file with the required environmental variables
While in the Server directory, run npm run dev

Client Setup

CD to Client directory run npm install
While in the Client directory, create a .env file with the required environmental variables
While in the Client cirectory run npm run dev

Access Application

Client is running at localhost:5173
Server is running at localhost:5000

Docker Compose

The fastest way to start the application is to use our Dockerfiles and Docker Compose.

To get the application up and running you need to:

In the Docker directory run the build script build_images.sh to build docker images for the client, server, Redis database, and MongoDB database.
In the Docker directory, create a server.env file with the requried environtmental variables for the server. Sample file:

CLIENT_HOST="http://localhost:5173"
JWT_SECRET=<jwt_secret>
DB_TYPE="MongoDB"
DB_CONNECTION_STRING="mongodb://mongodb:27017/uptime_db"
REDIS_HOST="redis"
REDIS_PORT=6379
TOKEN_TTL="99d"
PAGESPEED_API_KEY=<api_key>
SYSTEM_EMAIL_HOST="smtp.gmail.com"
SYSTEM_EMAIL_PORT=465
SYSTEM_EMAIL_ADDRESS=<system_email>
SYSTEM_EMAIL_PASSWORD=<system_email_password>

In the Client directory, create a client.env file with the required environtmental variables for the client. Sample file:

VITE_APP_API_BASE_URL="http://localhost:5000/api/v1"
VITE_APP_API_LOG_LEVEL="debug"

In the Docker directory run docker compose up to run the docker-compose.yaml file and start all four images.

That's it, the application is ready to use on port 80.

Client

Installation

Change directory to the Client directory
Install all dependencies by running npm install

Configuration

Environmental Variables

ENV Variable Name	Required/Optional	Type	Description	Accepted Values
VITE_APP_API_BASE_URL	Required	`string`	Base URL of server	{host}/api/v1
VITE_APP_LOG_LEVEL	Optional	`string`	Log level	`"none"`\|`"error"` \| `"warn"` \|

Starting Development Server

Run npm run dev to start the development server.

Getting Started (Server)

Manual Install

Install Server

Change directory to the Server directory
Install all dependencies by running npm install

Environmental Variables

Configure the server with the following environmental variables:

ENV Variable Name	Required/Optional	Type	Description	Accepted Values
CLIENT_HOST	Required	`string`	Frontend Host
JWT_SECRET	Required	`string`	JWT secret
DB_TYPE	Optional	`string`	Specify DB to use	`MongoDB \| FakeDB`
DB_CONNECTION_STRING	Required	`string`	Specifies URL for MongoDB Database
PORT	Optional	`integer`	Specifies Port for Server
LOGIN_PAGE_URL	Required	`string`	Login url to be used in emailing service
REDIS_HOST	Required	`string`	Host address for Redis database
REDIS_PORT	Required	`integer`	Port for Redis database
TOKEN_TTL	Optional	`string`	Time for token to live	In vercel/ms format https://github.com/vercel/ms
PAGESPEED_API_KEY	Optional	`string`	API Key for PageSpeed requests
SYSTEM_EMAIL_HOST	Required	`string`	Host to send System Emails From
SYSTEM_EMAIL_PORT	Required	`number`	Port for System Email Host
SYSTEM_EMAIL_ADDRESS	Required	`string`	System Email Address
SYSTEM_EMAIL_PASSWORD	Required	`string`	System Email Password

Databases

This project requires a number of databases to run:

Main database for the application. This project includes an implementation for a MongoDB database as well as a MongoDB Docker image.
A Redis database is required for the Queue implementation in the PingService. This project includes a Redis docker image.

You may use the included Dockerfiles to spin up databases quickly if you wish.

(Optional) Dockerised Databases

Dockerfiles for the server and databases are located in the Docker directory

MongoDB Image

Location: Docker/mongoDB.Dockerfile

The Docker/mongo/data directory should be mounted to the MongoDB container in order to persist data.

From the Docker directory run

Build the image: docker build -f mongoDB.Dockerfile -t uptime_database_mongo .
Run the docker image: docker run -d -p 27017:27017 -v $(pwd)/mongo/data:/data/db --name uptime_database_mongo uptime_database_mongo

Redis Image

Location Docker/redis.Dockerfile

the Docker/redis/data directory should be mounted to the Redis container in order to persist data.

From the Docker directory run

Build the image: docker build -f redis.Dockerfile -t uptime_redis .
Run the image: docker run -d -p 6379:6379 -v $(pwd)/redis/data:/data --name uptime_redis uptime_redis

Starting the Development Server

run npm run dev to start the development server

run node index.js to start server

Endpoints

All endpoints return a response in this format:

Name	Type	Notes
success	`boolean`	Success or failure of request
msg	`string`	Message describing response
data	`Object`	Arbitrary Payload

Example:

{success: true, msg: "Successful Request", data: {test: testData}}

Data Types

User

Name	Type	Notes
firstname	`string`	First name
lastname	`string`	Last name
email	`string`	User's email
profilePicUrl	`string`	URL to User's picture
isActive	`boolean`	Default to `true`
isVerified	`boolean`	Default to `false`
updated_at	`Date`	Last update time
created_at	`Date`	Time created at

Monitor

Name	Type	Notes
userId	`string`	Unique ID identifying monitor creator
name	`string`	Name of the monitor
description	`string`	Description of the monitor
url	`string`	Url the monitor will ping
isActive	`boolean`	Whether or not the monitor is active
interval	`integer`	Interval with which to ping monitor (ms)
updatedAt	`Date`	Last time the monitor was updated
CreatedAt	`Date`	When the monitor was updated

Check

Name	Type	Notes
monitorId	`string`	Unique ID for the monitor
status	`boolean`	Indicates the service is Up or Down
responseTime	`integer`	Indicates the response time of the service (ms)
statusCode	`integer`	Status Code returned from the service
message	`string`	Message returned from the service
updatedAt	`Date`	Last time the check was updated
CreatedAt	`Date`	When the check was created

Alert

Name	Type	Notes
checkId	`string`	Unique ID for the associated check
monitorId	`string`	Unique ID for the associated monitor
userId	`string`	Unique ID for the associated user
status	`boolean`	Indicates the service is Up or Down
message	`string`	Message for the user about the down service
notifiedStatus	`boolean`	Indicates whether the user is notified
acknowledgeStatus	`boolean`	Indicates whether the user acknowledged the alert
updatedAt	`Date`	Last time the alert was updated
CreatedAt	`Date`	When the alert was created

Auth

POST /api/v1/auth/register

Method/Headers

Method/Headers Value

Method POST

content-type multipart/form-data

Method/Headers	Value
Method	POST
content-type	multipart/form-data

Form

Name Type Notes

firstName string

lastName string

email string Valid email address

password string Min 8 chars, One Upper, one number, one special

role Array<string> Array of user roles

Name	Type	Notes
firstName	`string`
lastName	`string`
email	`string`	Valid email address
password	`string`	Min 8 chars, One Upper, one number, one special
role	`Array<string>`	Array of user roles

Response Payload

Type Notes

User User data

JWT JSON web token

Type	Notes
User	User data
JWT	JSON web token

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/auth/register \
  --header 'Content-Type: multipart/form-data' \
  --form firstName=Alex \
  --form lastName=Hollidaty \
  --form email=ajhollid@gmail.com \
  --form 'password=Testtest1!' \
  --form 'role[]=admin'

Sample Response

{
  "success": true,
  "msg": "User created successfully",
  "data": {
    "user": {
      "_id": "66a1425b873da2207443f192",
      "firstName": "First Name",
      "lastName": "Last Name",
      "email": "name@gmail.com",
      "isActive": true,
      "isVerified": false,
      "role": ["admin"],
      "createdAt": "2024-07-24T18:05:15.852Z",
      "updatedAt": "2024-07-24T18:05:15.852Z",
      "__v": 0
    },
    "token": "<token>"
  }
}

POST /api/v1/auth/login

Method/Headers

Method/Headers Value

Method POST

content-type application/json

Method/Headers	Value
Method	POST
content-type	application/json

Body

Name Type Notes

email string Valid email address

password string

Name	Type	Notes
email	`string`	Valid email address
password	`string`

Response Payload

Type Notes

User User data

JWT JSON web token

Type	Notes
User	User data
JWT	JSON web token

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/auth/login \
  --header 'Authorization: Bearer undefined' \
  --header 'Content-Type: application/json' \
  --data '{
	"email" : "name@gmail.com",
	"password": "Testtest1!"
}'

Sample response

{
{
	"success": true,
	"msg": "User logged in successfully",
	"data": {
		"user": {
      "_id": "66a1425b873da2207443f192",
      "firstName": "First Name",
      "lastName": "Last Name",
      "email": "name@gmail.com",
      "isActive": true,
      "isVerified": false,
      "role": ["admin"],
      "createdAt": "2024-07-24T18:05:15.852Z",
      "updatedAt": "2024-07-24T18:05:15.852Z",
      "__v": 0
		},
		"token": "<token>"
	}
}
}

POST/api/v1/auth/user/{userId}

Method/Headers

Method/Headers Value

Method POST

content-type multipart/form-data

Method/Headers	Value
Method	POST
content-type	multipart/form-data

Form

Name Type Notes

firstName string Optional

lastName string Optional

profileIame file Optional

password string Required to change password

newPassword string Required to change password

Name	Type	Notes
firstName	`string`	Optional
lastName	`string`	Optional
profileIame	`file`	Optional
password	`string`	Required to change password
newPassword	`string`	Required to change password

Response Payload

Type Notes

User Returns the updated user

Type	Notes
`User`	Returns the updated user

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/auth/user/66a1425b873da2207443f192 \
  --header 'Authorization: <bearer_token>' \
  --header 'Content-Type: multipart/form-data' \
  --form firstName=Test \
  --form lastName=Test \
  --form profileImage=@/home/user/Desktop/cat.jpg \
  --form 'newPassword=Testtest1!' \
  --form 'password=Testtest2!'

Sample Response

{
  "success": true,
  "msg": "User updated successfully",
  "data": {
    "_id": "66a1425b873da2207443f192",
    "firstName": "First name",
    "lastName": "Last name",
    "email": "name@gmail.com",
    "isActive": true,
    "isVerified": false,
    "role": ["admin"],
    "createdAt": "2024-07-24T18:05:15.852Z",
    "updatedAt": "2024-07-24T18:31:32.314Z",
    "__v": 0,
    "avatarImage": "<Base64 Image>"
  }
}

GET/api/v1/auth/users

Method/Headers

Method/Headers Value

Method GET

content-type application/json

Method/Headers	Value
Method	GET
content-type	application/json

Response Payload

Type Notes

Array<User> Returns an array containing all users

Type	Notes
`Array<User>`	Returns an array containing all users

Sample CURL request

curl --request GET \
  --url http://localhost:5000/api/v1/auth/users \
  --header 'Authorization: <bearer_token>\

Sample Resonse

{
  "success": true,
  "msg": "Got all users",
  "data": [
    {
      "_id": "669e90072d5663d25808bc7b",
      "firstName": "First name",
      "lastName": "Last name",
      "email": "name@gmail.com",
      "isActive": true,
      "isVerified": false,
      "role": ["admin"],
      "createdAt": "2024-07-22T16:59:51.695Z",
      "updatedAt": "2024-07-22T16:59:51.695Z",
      "__v": 0
    }
  ]
}

POST/api/v1/auth/recovery/request

Method/Headers

Method/Headers Value

Method POST

content-type application/json

Method/Headers	Value
Method	POST
content-type	application/json

Body

Name Type Notes

email string User's email

Name	Type	Notes
email	`string`	User's email

Response Payload

Type Notes

RecoveryToken Returns a recovery token if email found

Type	Notes
`RecoveryToken`	Returns a recovery token if email found

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/auth/recovery/request \
  --header 'Content-Type: application/json' \
  --data '{
	"email" : "name@gmail.com"
}'

Sample Response

{
  "success": true,
  "msg": "Created recovery token",
  "data": {
    "email": "name@gmail.com",
    "token": "f519da5e4a9be40cfc3c0fde97e60c0e6d17bdaa613f5ba537a45073f3865193",
    "_id": "6668878263587f30748e968e",
    "expiry": "2024-06-11T17:21:06.984Z",
    "createdAt": "2024-06-11T17:21:06.985Z",
    "updatedAt": "2024-06-11T17:21:06.985Z",
    "__v": 0
  }
}

POST/api/v1/auth/recovery/validate

Method/Headers

Method/Headers Value

Method POST

content-type application/json

Method/Headers	Value
Method	POST
content-type	application/json

Body

Name Type Notes

recoveryToken string Token issued in /recovery/request

Name	Type	Notes
recoveryToken	`string`	Token issued in `/recovery/request`

Response Payload

Type Notes

RecoveryToken Returns the recovery token

Type	Notes
`RecoveryToken`	Returns the recovery token

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/auth/recovery/validate \
  --header 'Content-Type: application/json' \
  --data '{
	"recoveryToken" : "f519da5e4a9be40cfc3c0fde97e60c0e6d17bdaa613f5ba537a45073f3865193"
}'

Sample Response

{
  "success": true,
  "msg": "Token is valid",
  "data": {
    "_id": "6668894263587f30748e969a",
    "email": "name@gmail.com",
    "token": "457d9926b24dedf613f120eeb524ef00ac45b3f0fc5c70bd25b1cc8aa83a64a0",
    "expiry": "2024-06-11T17:28:34.349Z",
    "createdAt": "2024-06-11T17:28:34.349Z",
    "updatedAt": "2024-06-11T17:28:34.349Z",
    "__v": 0
  }
}

POST/api/v1/auth/recovery/reset

Method/Headers

Method/Headers Value

Method POST

content-type application/json

Method/Headers	Value
Method	POST
content-type	application/json

Body

Name Type Notes

recoveryToken string Token issued returned by /recovery/validate

password string User's new password`

Name	Type	Notes
recoveryToken	`string`	Token issued returned by `/recovery/validate`
password	`string`	User's new password`

Response Payload

Type Notes

User Returns the updated user

Type	Notes
`User`	Returns the updated user

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/auth/recovery/reset \
  --header 'Content-Type: application/json' \
  --data '{
	"recoveryToken" : "f519da5e4a9be40cfc3c0fde97e60c0e6d17bdaa613f5ba537a45073f3865193",
	"password": "testtest"
}'

Sample Response

{
  "success": true,
  "msg": "Password reset",
  "data": {
    "_id": "66675891cb17336d84c25d9f",
    "firstname": "First Name",
    "lastname": "Last Name",
    "email": "name@gmail.com",
    "isActive": true,
    "isVerified": false,
    "createdAt": "2024-06-10T19:48:33.863Z",
    "updatedAt": "2024-06-11T17:21:22.289Z",
    "__v": 0
  }
}

Monitors

GET /api/v1/monitors

Method/Headers

Method/Headers Value

Method GET

content-type application/json

Method/Headers	Value
Method	GET
content-type	application/json

Response Payload

Type Notes

Array<Monitor> Array of all monitors

Type	Notes
`Array<Monitor>`	Array of all monitors

Sample cURL Request

curl --request GET \
  --url http://localhost:5000/api/v1/monitors \
  --header '<bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Monitors found",
  "data": [
    {
      "_id": "664d070786e62625ac612ca1",
      "userId": "6645079aae0b439371913972",
      "name": "Wha3",
      "description": "Description",
      "url": "https://monitor0.com",
      "isActive": true,
      "interval": 60000,
      "createdAt": "2024-05-21T20:41:43.051Z",
      "updatedAt": "2024-05-21T20:45:10.496Z",
      "__v": 0
    },
    {
      "_id": "664e5ccf189c864800debc16",
      "userId": "6645079aae0b439371913972",
      "name": "Inserting a new Monitor",
      "description": "Description",
      "url": "https://monitor0.com",
      "isActive": true,
      "interval": 60000,
      "createdAt": "2024-05-22T20:59:59.295Z",
      "updatedAt": "2024-05-22T20:59:59.295Z",
      "__v": 0
    }
  ]
}

GET /api/v1/monitor/{id}?status

Method/Headers

Method/Headers Value

Method GET

content-type application/json

Method/Headers	Value
Method	GET
content-type	application/json

Query Params

Name Type Required Notes

status boolean Optional Check status

limit number Optional Number of checks to return

sortOrder string Optional desc:Newest -> Oldest, asc: Oldest -> Newest

Name	Type	Required	Notes
status	`boolean`	Optional	Check status
limit	`number`	Optional	Number of checks to return
sortOrder	`string`	Optional	`desc`:Newest -> Oldest, `asc`: Oldest -> Newest

Response Payload

Type Notes

Monitor Single monitor with the id in the request parameter

Type	Notes
`Monitor`	Single monitor with the id in the request parameter

Sample cURL Request

curl --request GET \
  --url http://localhost:5000/api/v1/monitors/664d070786e62625ac612ca1?status=true?limit=0 \
  --header '<bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Got monitor by Id successfully",
  "data": {
    "_id": "6671eb54f7040ece47892f53",
    "userId": "666c9146c9bfa20db790b1df",
    "name": "Google Monitor",
    "description": "Google",
    "type": "http",
    "url": "https://www.google.com/404",
    "isActive": true,
    "interval": 10000,
    "createdAt": "2024-06-18T20:17:24.112Z",
    "updatedAt": "2024-06-18T20:17:24.112Z",
    "__v": 0,
    "checks": [
      {
        "_id": "6671eb5af7040ece47892f61",
        "monitorId": "6671eb54f7040ece47892f53",
        "status": false,
        "responseTime": 145,
        "expiry": "2024-06-18T20:17:30.246Z",
        "statusCode": 404,
        "createdAt": "2024-06-18T20:17:30.246Z",
        "updatedAt": "2024-06-18T20:17:30.246Z",
        "__v": 0
      },
      {
        "_id": "6671eb64f7040ece47892f6b",
        "monitorId": "6671eb54f7040ece47892f53",
        "status": false,
        "responseTime": 170,
        "expiry": "2024-06-18T20:17:40.209Z",
        "statusCode": 404,
        "createdAt": "2024-06-18T20:17:40.210Z",
        "updatedAt": "2024-06-18T20:17:40.210Z",
        "__v": 0
      }
    ]
  }
}

GET /api/v1/monitors/user/{userId}?limit

Method/Headers

Method/Headers Value

Method GET

content-type application/json

Query Params

Name Type Required Notes

status boolean Optional Check status

type string Optional Multiple allowed: http | ping | pagespeed

limit number Optional Monitor status

sortOrder string Optional desc:Newest -> Oldest, asc: Oldest -> Newest

Response Payload

Type Notes

Array<Monitor> Array of monitors created by user with specified UserID

Sample cURL Request

curl --request GET \
  --url http://localhost:5000/api/v1/monitors/user/6645079aae0b439371913972?limit=25 \
  --header '<bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Got monitor for 666c9146c9bfa20db790b1df successfully\"",
  "data": [
    {
      "_id": "6671eb54f7040ece47892f53",
      "userId": "666c9146c9bfa20db790b1df",
      "name": "Google Monitor",
      "description": "Google",
      "type": "http",
      "url": "https://www.google.com/404",
      "isActive": true,
      "interval": 10000,
      "createdAt": "2024-06-18T20:17:24.112Z",
      "updatedAt": "2024-06-18T20:17:24.112Z",
      "__v": 0,
      "checks": [
        {
          "_id": "6671eb5af7040ece47892f61",
          "monitorId": "6671eb54f7040ece47892f53",
          "status": false,
          "responseTime": 145,
          "expiry": "2024-06-18T20:17:30.246Z",
          "statusCode": 404,
          "createdAt": "2024-06-18T20:17:30.246Z",
          "updatedAt": "2024-06-18T20:17:30.246Z",
          "__v": 0
        }
      ]
    }
  ]
}

POST/api/v1/monitors

Method/Headers

Method/Headers Value

Method POST

content-type application/json

Body

Name Type Notes Accepted Values

userId string UserId of current user

name string Monitor name

description string Monitor Description

type string Valid email address "ping"|"http"|pagespeed

url string URL of service or IP

isActive boolean

interval number In ms

Response Payload

Type Notes

Monitor Returns newly created Monitor

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/monitors \
  --header <bearer_token> \
  --header 'Content-Type: application/json' \
  --data '{
      "userId": "66675891cb17336d84c25d9f",
			"name": "Ping Google",
			"description": "Google",
 			"type": "ping",
			"url": "8.8.8.8",
			"isActive": true,
			"interval": 5000}'

Sample Response

{
  "success": true,
  "msg": "Monitor created",
  "data": {
    "userId": "6645079aae0b439371913972",
    "name": "Inserting a new Monitor",
    "description": "Description",
    "url": "https://monitor0.com",
    "isActive": true,
    "interval": 60000,
    "_id": "664e5ccf189c864800debc16",
    "createdAt": "2024-05-22T20:59:59.295Z",
    "updatedAt": "2024-05-22T20:59:59.295Z",
    "__v": 0
  }
}

POST/api/v1/monitors/delete/{monitorId}

Method/Headers

Method/Headers Value

Method POST

content-type application/json

Response Payload

Type Notes

None No payload returned

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/monitors/delete/664e632a7a3ee9d620761938 \
  --header '<bearer_token>' \
  --header 'Content-Type: application/json' \

Sample Response

{
  "success": true,
  "msg": "Monitor deleted"
}

POST/api/v1/monitors/edit/{monitorId}

Method/Headers

Method/Headers Value

Method POST

content-type application/json

Body

Name Type Notes Accepted Values

name string Monitor name

description string Monitor Description

interval number In ms

Response Payload

Type Notes

Monitor Returns the updated monitor

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/monitors/edit/664e5ccf189c864800debc16 \
  --header '<bearer_token' \
  --header 'Content-Type: application/json' \
  --data '
		{
			"name": "Edited monitor",
			"description": "Description",
			"interval": 60000
		}'

Sample Response

{
  "success": true,
  "msg": "Monitor edited",
  "data": {
    "_id": "664e5ccf189c864800debc16",
    "userId": "6645079aae0b439371913972",
    "name": "Edited monitor",
    "description": "Description",
    "url": "https://monitor0.com",
    "isActive": true,
    "interval": 60000,
    "createdAt": "2024-05-22T20:59:59.295Z",
    "updatedAt": "2024-05-22T21:34:33.893Z",
    "__v": 0
  }
}

Checks

POST/api/v1/checks/{monitorId}

Method/Headers

Method/Headers Value

Method POST

Response Payload

Type Notes

Check Returns newly created check

Body

Name Type Notes

monitorId string Monitor associated with Check

status boolean true for up and false for down

responseTime number How long it took the server to respond

statusCode number HTTP Status code of response

message string

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/checks/66562414035c4ce6a8a610ac \
  --header 'Authorization: <bearer_token>' \
  --header 'Content-Type: application/json' \
  --data '{
	"monitorId": "66562414035c4ce6a8a610ac",
	"status": true,
	"responseTime": 1,
	"statusCode": 200,
	"message": "good"
}'

Sample Response

{
  "success": true,
  "msg": "Check created",
  "data": {
    "monitorId": "66562414035c4ce6a8a610ac",
    "status": true,
    "responseTime": 1,
    "statusCode": 200,
    "message": "good",
    "_id": "66576decba9f70148ea1f354",
    "createdAt": "2024-05-29T18:03:24.445Z",
    "updatedAt": "2024-05-29T18:03:24.445Z",
    "__v": 0
  }
}

GET/api/v1/checks/{monitorId}

Method/Headers

Method/Headers Value

Method GET

Response Payload

Type Notes

Array<Checks> Array of Check objects

Sample CURL request

curl --request GET \
  --url http://localhost:5000/api/v1/checks/66562414035c4ce6a8a610ac \
  --header 'Authorization: <bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Checks retrieved",
  "data": [
    {
      "_id": "66576c0194e11c0d4409d3c1",
      "monitorId": "66562414035c4ce6a8a610ac",
      "status": true,
      "responseTime": 1,
      "statusCode": 200,
      "message": "good",
      "createdAt": "2024-05-29T17:55:13.581Z",
      "updatedAt": "2024-05-29T17:55:13.581Z",
      "__v": 0
    },
    {
      "_id": "66576c0994e11c0d4409d3c5",
      "monitorId": "66562414035c4ce6a8a610ac",
      "status": true,
      "responseTime": 2,
      "statusCode": 200,
      "message": "good",
      "createdAt": "2024-05-29T17:55:21.127Z",
      "updatedAt": "2024-05-29T17:55:21.127Z",
      "__v": 0
    }
  ]
}

POST/api/v1/checks/delete/{monitorId}

Method/Headers

Method/Headers Value

Method POST

Response Payload

Type Notes

Object {deletedCount: n} Returns an object showing how many items deleted

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/checks/delete/66562414035c4ce6a8a610ac \
  --header 'Authorization: <bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Checks deleted",
  "data": {
    "deletedCount": 3
  }
}

Alerts

POST/api/v1/alerts/{monitorId}

Method/Headers

Method/Headers Value

Method POST

Response Payload

Type Notes

Alert Returns newly created Alert

Body

"checkId": "66577a3fd16dcf7c1ce35148",
"monitorId": "6657789ebf6766ee8e2d2edb",
"userId": "6654d1a2634754f789e1f115",
"status": false,
"message": "This is a test alert",
"notifiedStatus": "false",
"acknowledgeStatus": false

Name Type Notes

checkId string Id of Check associated with Alert

monitorId string Id of Monitor associated with Alert

userId string Id of User associated with Alert

status boolean Status of Alert

message string Alert message

notifiedStatus boolean

acknowledgeStatus boolean

Sample CURL request

Sample Response

GET/api/v1/alerts/user/{userId}

Method/Headers

Method/Headers Value

Method GET

Response Payload

Type Notes

Array<Alert> Returns all Alert created by a User

Sample CURL request

curl --request GET \
  --url http://localhost:5000/api/v1/alerts/user/6654d1a2634754f789e1f115 \
  --header 'Authorization: <bearer_token>'

Sample Response

{
  "success": true,
  "msg": "Got alerts",
  "data": [
    {
      "_id": "6657813d809adfded891a6b7",
      "checkId": "66577a3fd16dcf7c1ce35148",
      "monitorId": "6657789ebf6766ee8e2d2edb",
      "userId": "6654d1a2634754f789e1f115",
      "status": false,
      "message": "This is a test alert",
      "notifiedStatus": false,
      "acknowledgeStatus": false,
      "createdAt": "2024-05-29T19:25:49.317Z",
      "updatedAt": "2024-05-29T19:25:49.317Z",
      "__v": 0
    }
  ]
}

GET/api/v1/alerts/monitor/{monitorId}

Method/Headers

Method/Headers Value

Method GET

Response Payload

Type Notes

Array<Alert> Returns an array of Alert belonging to a specified Monitor

Sample CURL request

curl --request GET \
  --url http://localhost:5000/api/v1/alerts/monitor/6657789ebf6766ee8e2d2edb \
  --header 'Authorization: <bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Got alerts by Monitor",
  "data": [
    {
      "_id": "6657813d809adfded891a6b7",
      "checkId": "66577a3fd16dcf7c1ce35148",
      "monitorId": "6657789ebf6766ee8e2d2edb",
      "userId": "6654d1a2634754f789e1f115",
      "status": false,
      "message": "This is a test alert",
      "notifiedStatus": false,
      "acknowledgeStatus": false,
      "createdAt": "2024-05-29T19:25:49.317Z",
      "updatedAt": "2024-05-29T19:25:49.317Z",
      "__v": 0
    }
  ]
}

GET/api/v1/alerts/{alertId}

Method/Headers

Method/Headers Value

Method GET

Response Payload

Type Notes

Alert Returns specified Alert

Sample CURL request

curl --request GET \
  --url http://localhost:5000/api/v1/alerts/66577ddae5ff3c91437d0887 \
  --header 'Authorization: <bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Got Alert By alertID",
  "data": {
    "_id": "66577ddae5ff3c91437d0887",
    "checkId": "66577a3fd16dcf7c1ce35148",
    "monitorId": "6657789ebf6766ee8e2d2edb",
    "userId": "6654d1a2634754f789e1f115",
    "status": false,
    "message": "This is a test alert",
    "notifiedStatus": false,
    "acknowledgeStatus": false,
    "createdAt": "2024-05-29T19:11:22.205Z",
    "updatedAt": "2024-05-29T19:11:22.205Z",
    "__v": 0
  }
}

POST/api/v1/alerts/edit/{alertId}

Method/Headers

Method/Headers Value

Method POST

Response Payload

Type Notes

Alert Returns edited Alert

Body

Name Type Notes

checkId string ID of Check associated with Alert

monitorId string ID of Monitor id associated with Alert

userId string ID of User associated with Alert

status boolean Alert status

message string Alert message

notifiedStatus boolean

acknowledgeStatus boolean

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/alerts/edit/66577ddae5ff3c91437d0887 \
  --header 'Authorization: <bearer_token>' \
  --header 'Content-Type: application/json' \
  --data '{
	"acknowledgeStatus": true
}'

Sample Response

{
  "success": true,
  "msg": "Edited alert",
  "data": {
    "_id": "66577ddae5ff3c91437d0887",
    "checkId": "66577a3fd16dcf7c1ce35148",
    "monitorId": "6657789ebf6766ee8e2d2edb",
    "userId": "6654d1a2634754f789e1f115",
    "status": false,
    "message": "This is a test alert",
    "notifiedStatus": false,
    "acknowledgeStatus": true,
    "createdAt": "2024-05-29T19:11:22.205Z",
    "updatedAt": "2024-05-29T19:12:23.951Z",
    "__v": 0
  }
}

POST/api/v1/alerts/delete/{alertId}

Method/Headers

Method/Headers Value

Method POST

Response Payload

Type Notes

Alert Returns the deleted Alert

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/alerts/delete/66577ddae5ff3c91437d0887 \
  --header 'Authorization: <bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Deleted alert",
  "data": {
    "_id": "66577ddae5ff3c91437d0887",
    "checkId": "66577a3fd16dcf7c1ce35148",
    "monitorId": "6657789ebf6766ee8e2d2edb",
    "userId": "6654d1a2634754f789e1f115",
    "status": false,
    "message": "This is a test alert",
    "notifiedStatus": false,
    "acknowledgeStatus": true,
    "createdAt": "2024-05-29T19:11:22.205Z",
    "updatedAt": "2024-05-29T19:12:23.951Z",
    "__v": 0
  }
}

Checks

POST/api/v1/checks/{monitorId}

Method/Headers

Method/Headers Value

Method POST

Response Payload

Type Notes

Check Returns newly created check

Body

Name Type Notes

monitorId string Monitor associated with Check

status boolean true for up and false for down

responseTime number How long it took the server to respond

statusCode number HTTP Status code of response

message string

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/checks/66562414035c4ce6a8a610ac \
  --header 'Authorization: <bearer_token>' \
  --header 'Content-Type: application/json' \
  --data '{
	"monitorId": "66562414035c4ce6a8a610ac",
	"status": true,
	"responseTime": 1,
	"statusCode": 200,
	"message": "good"
}'

Sample Response

{
  "success": true,
  "msg": "Check created",
  "data": {
    "monitorId": "66562414035c4ce6a8a610ac",
    "status": true,
    "responseTime": 1,
    "statusCode": 200,
    "message": "good",
    "_id": "66576decba9f70148ea1f354",
    "createdAt": "2024-05-29T18:03:24.445Z",
    "updatedAt": "2024-05-29T18:03:24.445Z",
    "__v": 0
  }
}

GET/api/v1/checks/{monitorId}

Method/Headers

Method/Headers Value

Method GET

Response Payload

Type Notes

Array<Checks> Array of Check objects

Sample CURL request

curl --request GET \
  --url http://localhost:5000/api/v1/checks/66562414035c4ce6a8a610ac \
  --header 'Authorization: <bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Checks retrieved",
  "data": [
    {
      "_id": "66576c0194e11c0d4409d3c1",
      "monitorId": "66562414035c4ce6a8a610ac",
      "status": true,
      "responseTime": 1,
      "statusCode": 200,
      "message": "good",
      "createdAt": "2024-05-29T17:55:13.581Z",
      "updatedAt": "2024-05-29T17:55:13.581Z",
      "__v": 0
    },
    {
      "_id": "66576c0994e11c0d4409d3c5",
      "monitorId": "66562414035c4ce6a8a610ac",
      "status": true,
      "responseTime": 2,
      "statusCode": 200,
      "message": "good",
      "createdAt": "2024-05-29T17:55:21.127Z",
      "updatedAt": "2024-05-29T17:55:21.127Z",
      "__v": 0
    }
  ]
}

POST/api/v1/checks/delete/{monitorId}

Method/Headers

Method/Headers Value

Method POST

Response Payload

Type Notes

Object {deletedCount: n} Returns an object showing how many items deleted

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/checks/delete/66562414035c4ce6a8a610ac \
  --header 'Authorization: <bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Checks deleted",
  "data": {
    "deletedCount": 3
  }
}

Alerts

POST/api/v1/alerts/{monitorId}

Method/Headers

Method/Headers Value

Method POST

Response Payload

Type Notes

Alert Returns newly created Alert

Body

"checkId": "66577a3fd16dcf7c1ce35148",
"monitorId": "6657789ebf6766ee8e2d2edb",
"userId": "6654d1a2634754f789e1f115",
"status": false,
"message": "This is a test alert",
"notifiedStatus": "false",
"acknowledgeStatus": false

Name Type Notes

checkId string Id of Check associated with Alert

monitorId string Id of Monitor associated with Alert

userId string Id of User associated with Alert

status boolean Status of Alert

message string Alert message

notifiedStatus boolean

acknowledgeStatus boolean

Sample CURL request

Sample Response

GET/api/v1/alerts/user/{userId}

Method/Headers

Method/Headers Value

Method GET

Response Payload

Type Notes

Array<Alert> Returns all Alert created by a User

Sample CURL request

curl --request GET \
  --url http://localhost:5000/api/v1/alerts/user/6654d1a2634754f789e1f115 \
  --header 'Authorization: <bearer_token>'

Sample Response

{
  "success": true,
  "msg": "Got alerts",
  "data": [
    {
      "_id": "6657813d809adfded891a6b7",
      "checkId": "66577a3fd16dcf7c1ce35148",
      "monitorId": "6657789ebf6766ee8e2d2edb",
      "userId": "6654d1a2634754f789e1f115",
      "status": false,
      "message": "This is a test alert",
      "notifiedStatus": false,
      "acknowledgeStatus": false,
      "createdAt": "2024-05-29T19:25:49.317Z",
      "updatedAt": "2024-05-29T19:25:49.317Z",
      "__v": 0
    }
  ]
}

GET/api/v1/alerts/monitor/{monitorId}

Method/Headers

Method/Headers Value

Method GET

Response Payload

Type Notes

Array<Alert> Returns an array of Alert belonging to a specified Monitor

Sample CURL request

curl --request GET \
  --url http://localhost:5000/api/v1/alerts/monitor/6657789ebf6766ee8e2d2edb \
  --header 'Authorization: <bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Got alerts by Monitor",
  "data": [
    {
      "_id": "6657813d809adfded891a6b7",
      "checkId": "66577a3fd16dcf7c1ce35148",
      "monitorId": "6657789ebf6766ee8e2d2edb",
      "userId": "6654d1a2634754f789e1f115",
      "status": false,
      "message": "This is a test alert",
      "notifiedStatus": false,
      "acknowledgeStatus": false,
      "createdAt": "2024-05-29T19:25:49.317Z",
      "updatedAt": "2024-05-29T19:25:49.317Z",
      "__v": 0
    }
  ]
}

GET/api/v1/alerts/{alertId}

Method/Headers

Method/Headers Value

Method GET

Response Payload

Type Notes

Alert Returns specified Alert

Sample CURL request

curl --request GET \
  --url http://localhost:5000/api/v1/alerts/66577ddae5ff3c91437d0887 \
  --header 'Authorization: <bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Got Alert By alertID",
  "data": {
    "_id": "66577ddae5ff3c91437d0887",
    "checkId": "66577a3fd16dcf7c1ce35148",
    "monitorId": "6657789ebf6766ee8e2d2edb",
    "userId": "6654d1a2634754f789e1f115",
    "status": false,
    "message": "This is a test alert",
    "notifiedStatus": false,
    "acknowledgeStatus": false,
    "createdAt": "2024-05-29T19:11:22.205Z",
    "updatedAt": "2024-05-29T19:11:22.205Z",
    "__v": 0
  }
}

POST/api/v1/alerts/edit/{alertId}

Method/Headers

Method/Headers Value

Method POST

Response Payload

Type Notes

Alert Returns edited Alert

Body

Name Type Notes

checkId string ID of Check associated with Alert

monitorId string ID of Monitor id associated with Alert

userId string ID of User associated with Alert

status boolean Alert status

message string Alert message

notifiedStatus boolean

acknowledgeStatus boolean

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/alerts/edit/66577ddae5ff3c91437d0887 \
  --header 'Authorization: <bearer_token>' \
  --header 'Content-Type: application/json' \
  --data '{
	"acknowledgeStatus": true
}'

Sample Response

{
  "success": true,
  "msg": "Edited alert",
  "data": {
    "_id": "66577ddae5ff3c91437d0887",
    "checkId": "66577a3fd16dcf7c1ce35148",
    "monitorId": "6657789ebf6766ee8e2d2edb",
    "userId": "6654d1a2634754f789e1f115",
    "status": false,
    "message": "This is a test alert",
    "notifiedStatus": false,
    "acknowledgeStatus": true,
    "createdAt": "2024-05-29T19:11:22.205Z",
    "updatedAt": "2024-05-29T19:12:23.951Z",
    "__v": 0
  }
}

POST/api/v1/alerts/delete/{alertId}

Method/Headers

Method/Headers Value

Method POST

Response Payload

Type Notes

Alert Returns the deleted Alert

Sample CURL request

curl --request POST \
  --url http://localhost:5000/api/v1/alerts/delete/66577ddae5ff3c91437d0887 \
  --header 'Authorization: <bearer_token>' \

Sample Response

{
  "success": true,
  "msg": "Deleted alert",
  "data": {
    "_id": "66577ddae5ff3c91437d0887",
    "checkId": "66577a3fd16dcf7c1ce35148",
    "monitorId": "6657789ebf6766ee8e2d2edb",
    "userId": "6654d1a2634754f789e1f115",
    "status": false,
    "message": "This is a test alert",
    "notifiedStatus": false,
    "acknowledgeStatus": true,
    "createdAt": "2024-05-29T19:11:22.205Z",
    "updatedAt": "2024-05-29T19:12:23.951Z",
    "__v": 0
  }
}

Error handling

Errors are returned in a standard format:

{"success": false, "msg": "No token provided"}

Errors are handled by error handling middleware and should be thrown with the following parameters

Name	Type	Default	Notes
status	`integer`	500	Standard HTTP codes
message	`string`	"Something went wrong"	An error message
service	`string`	"Unknown Service"	Name of service that threw the error

Example:

const myRoute = async(req, res, next) => {
  try{
    const result = myRiskyOperationHere();
  }
  catch(error){
    error.status = 404
    error.message = "Resource not found"
    error.service = service name
    next(error)
    return;
  }
}

Errors should not be handled at the controller level and should be left to the middleware to handle.

ReactiveBayes/GraphPPL.jl

2024-08-22T01:31:06Z

DSL for probabilistic models specification and probabilistic programming.

anthropics/prompt-eng-interactive-tutorial

2024-08-21T01:31:01Z

Anthropic's Interactive Prompt Engineering Tutorial

Welcome to Anthropic's Prompt Engineering Interactive Tutorial

Course introduction and goals

This course is intended to provide you with a comprehensive step-by-step understanding of how to engineer optimal prompts within Claude.

After completing this course, you will be able to:

Master the basic structure of a good prompt
Recognize common failure modes and learn the '80/20' techniques to address them
Understand Claude's strengths and weaknesses
Build strong prompts from scratch for common use cases

Course structure and content

This course is structured to allow you many chances to practice writing and troubleshooting prompts yourself. The course is broken up into 9 chapters with accompanying exercises, as well as an appendix of even more advanced methods. It is intended for you to work through the course in chapter order.

Each lesson has an "Example Playground" area at the bottom where you are free to experiment with the examples in the lesson and see for yourself how changing prompts can change Claude's responses. There is also an answer key.

Note: This tutorial uses our smallest, fastest, and cheapest model, Claude 3 Haiku. Anthropic has two other models, Claude 3 Sonnet and Claude 3 Opus, which are more intelligent than Haiku, with Opus being the most intelligent.

This tutorial also exists on Google Sheets using Anthropic's Claude for Sheets extension. We recommend using that version as it is more user friendly.

When you are ready to begin, go to 01_Basic Prompt Structure to proceed.

Each chapter consists of a lesson and a set of exercises.

Beginner

Chapter 1: Basic Prompt Structure
Chapter 2: Being Clear and Direct
Chapter 3: Assigning Roles

Intermediate

Chapter 4: Separating Data from Instructions
Chapter 5: Formatting Output & Speaking for Claude
Chapter 6: Precognition (Thinking Step by Step)
Chapter 7: Using Examples

Advanced

Chapter 8: Avoiding Hallucinations
Chapter 9: Building Complex Prompts (Industry Use Cases)
- Complex Prompts from Scratch - Chatbot
- Complex Prompts for Legal Services
- Exercise: Complex Prompts for Financial Services
- Exercise: Complex Prompts for Coding
- Congratulations & Next Steps
Appendix: Beyond Standard Prompting
- Chaining Prompts
- Tool Use
- Search & Retrieval

Kotlin/anko

2024-08-22T01:31:20Z

Pleasant Android application development

⚠ Anko is deprecated. Please see this page for more information.

Anko is a Kotlin library which makes Android application development faster and easier. It makes your code clean and easy to read, and lets you forget about rough edges of the Android SDK for Java.

Anko consists of several parts:

Anko Commons: a lightweight library full of helpers for intents, dialogs, logging and so on;
Anko Layouts: a fast and type-safe way to write dynamic Android layouts;
Anko SQLite: a query DSL and parser collection for Android SQLite;
Anko Coroutines: utilities based on the kotlinx.coroutines library.

Anko Commons

Anko Commons is a "toolbox" for Kotlin Android developer. The library contains a lot of helpers for Android SDK, including, but not limited to:

Intents (wiki);
Dialogs and toasts (wiki);
Logging (wiki);
Resources and dimensions (wiki).

Anko Layouts (wiki)

Anko Layouts is a DSL for writing dynamic Android layouts. Here is a simple UI written with Anko DSL:

verticalLayout {
    val name = editText()
    button("Say Hello") {
        onClick { toast("Hello, ${name.text}!") }
    }
}

The code above creates a button inside a LinearLayout and attaches an OnClickListener to that button. Moreover, onClick accepts a suspend lambda, so you can write your asynchronous code right inside the listener!

Note that this is the complete layout code. No XML is required!

Anko has a DSL for ConstraintLayout since v0.10.4

There is also a plugin for Android Studio that supports previewing Anko DSL layouts.

Anko SQLite (wiki)

Have you ever been tired of parsing SQLite query results using Android cursors? Anko SQLite provides lots of helpers to simplify working with SQLite databases.

For example, here is how you can fetch the list of users with a particular name:

fun getUsers(db: ManagedSQLiteOpenHelper): List<User> = db.use {
    db.select("Users")
            .whereSimple("family_name = ?", "John")
            .doExec()
            .parseList(UserParser)
}

Anko Coroutines (wiki)

Anko Coroutines is based on the kotlinx.coroutines library and provides:

bg() function that executes your code in a common pool.
asReference() function which creates a weak reference wrapper. By default, a coroutine holds references to captured objects until it is finished or canceled. If your asynchronous framework does not support cancellation, the values you use inside the asynchronous block can be leaked. asReference() protects you from this.

Using Anko

Gradle-based project

Anko has a meta-dependency which plugs in all available features (including Commons, Layouts, SQLite) into your project at once:

dependencies {
    implementation "org.jetbrains.anko:anko:$anko_version"
}

Make sure that you have the $anko_version settled in your gradle file at the project level:

ext.anko_version='0.10.8'

If you only need some of the features, you can reference any of Anko's parts:

dependencies {
    // Anko Commons
    implementation "org.jetbrains.anko:anko-commons:$anko_version"

    // Anko Layouts
    implementation "org.jetbrains.anko:anko-sdk25:$anko_version" // sdk15, sdk19, sdk21, sdk23 are also available
    implementation "org.jetbrains.anko:anko-appcompat-v7:$anko_version"

    // Coroutine listeners for Anko Layouts
    implementation "org.jetbrains.anko:anko-sdk25-coroutines:$anko_version"
    implementation "org.jetbrains.anko:anko-appcompat-v7-coroutines:$anko_version"

    // Anko SQLite
    implementation "org.jetbrains.anko:anko-sqlite:$anko_version"
}

There are also a number of artifacts for the Android support libraries:

dependencies {
    // Appcompat-v7 (only Anko Commons)
    implementation "org.jetbrains.anko:anko-appcompat-v7-commons:$anko_version"

    // Appcompat-v7 (Anko Layouts)
    implementation "org.jetbrains.anko:anko-appcompat-v7:$anko_version"
    implementation "org.jetbrains.anko:anko-coroutines:$anko_version"

    // CardView-v7
    implementation "org.jetbrains.anko:anko-cardview-v7:$anko_version"

    // Design
    implementation "org.jetbrains.anko:anko-design:$anko_version"
    implementation "org.jetbrains.anko:anko-design-coroutines:$anko_version"

    // GridLayout-v7
    implementation "org.jetbrains.anko:anko-gridlayout-v7:$anko_version"

    // Percent
    implementation "org.jetbrains.anko:anko-percent:$anko_version"

    // RecyclerView-v7
    implementation "org.jetbrains.anko:anko-recyclerview-v7:$anko_version"
    implementation "org.jetbrains.anko:anko-recyclerview-v7-coroutines:$anko_version"

    // Support-v4 (only Anko Commons)
    implementation "org.jetbrains.anko:anko-support-v4-commons:$anko_version"

    // Support-v4 (Anko Layouts)
    implementation "org.jetbrains.anko:anko-support-v4:$anko_version"

    // ConstraintLayout
    implementation "org.jetbrains.anko:anko-constraint-layout:$anko_version"
}

There is an example project showing how to include Anko library into your Android Gradle project.

IntelliJ IDEA project

If your project is not based on Gradle, just attach the required JARs from the jcenter repository as the library dependencies and that's it.

Contributing

The best way to submit a patch is to send us a pull request. Before submitting the pull request, make sure all existing tests are passing, and add the new test if it is required.

If you want to add new functionality, please file a new proposal issue first to make sure that it is not in progress already. If you have any questions, feel free to create a question issue.

Instructions for building Anko are available in the Wiki.

MathIsFun0/Cryptid

2024-08-21T01:31:25Z

An unbalanced Balatro mod

Cryptid

An unbalanced Balatro mod. Note that this is currently a heavy work in progress.

Note: Cryptid requires Steamodded 1.0.0-Alpha and Talisman.

Cryptid currently adds:

107 new jokers
43 new decks
24 new stakes
20 new boss blinds
18 new code cards
18 new vouchers
9 new spectral cards
9 new tags
7 new challenges
5 new editions
4 new planet cards
4 new booster packs
2 new joker rarities
2 new tarot cards
1 new consumable type
1 new enhancement
1 new seal

Official Discord

Video Installation Guide

OP-TEE/build

2024-08-21T01:31:33Z

CLI to install, manage & troubleshoot Kubernetes clusters running Cilium

open-telemetry/semantic-conventions

2024-08-22T01:32:40Z

Defines standards for generating consistent, accessible telemetry across a variety of domains

PwC-IR/Office-365-Extractor

2024-08-21T01:32:56Z

The Office 365 Extractor is a tool that allows for complete and reliable extraction of the Unified Audit Log (UAL)

Office 365 Extractor

This script makes it possible to extract log data out of an Office365 environment. The script created by us consist out of four main options, which enable the investigator to easily extract logging out of an Office365 environment.

Show available log sources and amount of logging
Extract all audit logging
Extract group audit logging
Extract Specific audit logging (advanced mode)

Show available log sources and amount of logging

Pretty straightforward a search is executed and the total number of logs within the
set timeframe will be displayed and written to a csv file called "Amount_Of_Audit_Logs.csv" the file is prefixed with a random number to prevent duplicates.

Extract all audit logs

Extract all audit logs" this option wil get all available audit logs within the set timeframe and written out to a file called AuditRecords.CSV.

Extract group logging

Extract a group of logs. You can for example extract all Exchange or Azure logging in one go

Extract specific audit logs

Extract specific audit logs" Use this option if you want to extract a subset of the audit logs. To configure what logs will be extracted the tool needs to
be configured with the required Record Types. A full list of recordtypes can be found at the bottom of this page.
The output files will be writen in a directory called 'Log_Directory" and will be given the name of their recordtype e.g. (ExchangeItem_AuditRecords.csv)

Prerequisites

-Exchange Online PowerShell V2 Module
-PowerShell
-Office365 account with privileges to access/extract audit logging
-One of the following windows versions:
Windows 10, Windows 8.1, Windows 8, or Windows 7 Service Pack 1 (SP1)
Windows Server 2019, Windows Server 2016, Windows Server 2012 R2, Windows Server 2012, or Windows Server 2008 R2 SP1

You have to be assigned the View-Only Audit Logs or Audit Logs role in Exchange Online to search the Office 365 audit log. By default, these roles are assigned to the Compliance Management and Organization Management role groups on the Permissions page in the Exchange admin center. To give a user the ability to search the Office 365 audit log with the minimum level of privileges, you can create a custom role group in Exchange Online, add the View-Only Audit Logs or Audit Logs role, and then add the user as a member of the new role group. For more information, see Manage role groups in Exchange Online. https://docs.microsoft.com/en-us/office365/securitycompliance/search-the-audit-log-in-security-and-compliance)

Install Exchange Online Powershell V2 Module

1. Start Windows PowerShell with the “Run as administrator” option
2. Install PowerShellGet Module. To install the ExchangeOnlineManagement module, you need PowerShellGet 2.0 or later version. Else, you end up with an error: Run: Install-Module PowerShellGet -Force
3. Run the following cmdlet to install Exchange Online PowerShell V2 Module: Install-Module –Name ExchangeOnlineManagement

How to use the script

1. Download the Office365_Extractor.ps1
2. Right click on the script and press "Run with PowerShell".
3. Now pick any of the options in the menu.
4. The logs will be written to the logdirectory in the folder where the script is located.

Output

Amount_Of_Audit_Logs.csv:
Will show what logs are available and how many for each RecordType.
AuditLog.txt:
The AuditLog stores valuable information for debugging.
AuditRecords.csv:
When all logs are extracted they will be written to this file.
[RecordType]__AuditRecords:
When extracting specific RecordTypes, logs are sorted on RecordType and written to a CSV file.
The name of this file is the RecordType + _AuditRecords.

Available RecordTypes

ExchangeAdmin
ExchangeItem
ExchangeItemGroup
SharePoint
SyntheticProbe
SharePointFileOperation
OneDrive
AzureActiveDirectory
AzureActiveDirectoryAccountLogon
DataCenterSecurityCmdlet
ComplianceDLPSharePoint
Sway
ComplianceDLPExchange
SharePointSharingOperation
AzureActiveDirectoryStsLogon
SkypeForBusinessPSTNUsage
SkypeForBusinessUsersBlocked
SecurityComplianceCenterEOPCmdlet
ExchangeAggregatedOperation
PowerBIAudit
CRM
Yammer
SkypeForBusinessCmdlets
Discovery
MicrosoftTeams
ThreatIntelligence
MailSubmission
MicrosoftFlow
AeD
MicrosoftStream
ComplianceDLPSharePointClassification
ThreatFinder
Project
SharePointListOperation
SharePointCommentOperation
DataGovernance
Kaizala
SecurityComplianceAlerts
ThreatIntelligenceUrl
SecurityComplianceInsights
MIPLabel
WorkplaceAnalytics
PowerAppsApp
PowerAppsPlan
ThreatIntelligenceAtpContent
LabelExplorer
TeamsHealthcare
ExchangeItemAggregated
HygieneEvent
DataInsightsRestApiAudit
InformationBarrierPolicyApplication
SharePointListItemOperation
SharePointContentTypeOperation
SharePointFieldOperation
MicrosoftTeamsAdmin
HRSignal
MicrosoftTeamsDevice
MicrosoftTeamsAnalytics
InformationWorkerProtection
Campaign
DLPEndpoint
AirInvestigation
Quarantine
MicrosoftForms
LabelContentExplorer
ApplicationAudit
ComplianceSupervisionExchange
CustomerKeyServiceEncryption
OfficeNative
MipAutoLabelSharePointItem
MipAutoLabelSharePointPolicyLocation
MicrosoftTeamsShifts
MipAutoLabelExchangeItem
CortanaBriefing
Search
WDATPAlerts
MDATPAudit

Frequently Asked Questions

If I enable mailbox auditing now can I see historical records?
No, additionaly if you enable auditing now it will take up to 24 hours before events will be logged.

I logged into a mailbox with auditing turned on but I don't see my events?
It can take up to 24 hours before an event is stored in the UAL.

Which date format does the script accepts as input?
The script will tell what the correct date format is. For Start and End data variables it will show between brackets what the format is (yyyy-MM-dd).

Do I need to configure the time period?
No if you don't specify a time period the script will use the default If you don't include a timestamp in the value for the StartDate or EndDate parameters, the default timestamp 12:00 AM (midnight) is used.

What about timestamps?
The audit logs are in UTC, and they will be exported as such

What is the retention period?
Office 365 E3 - Audit records are retained for 90 days. That means you can search the audit log for activities that were performed within the last 90 days.

Office 365 E5 - Audit records are retained for 365 days (one year). That means you can search the audit log for activities that were performed within the last year. Retaining audit records for one year is also available for users that are assigned an E3/Exchange Online Plan 1 license and have an Office 365 Advanced Compliance add-on license.

Can this script also acquire Message Trace Logs?
At the moment it cannot, but there are several open-source scripts available that can help you with getting the MTL One example can be found here: https://gallery.technet.microsoft.com/scriptcenter/Export-Mail-logs-to-CSV-d5b6c2d6

Known errors

Import-PSSession : No command proxies have been created, because all of the requested remote....
This error is caused when the script did not close correctly and an active session will be running in the background. The script tries to import/load all modules again, but this is not necessary since it is already loaded. This error message has no impact on the script and will be gone when the open session gets closed. This can be done by restarting the PowerShell Windows or entering the following command: Get-PSSession | Remove-PSSession

Audit logging is enabled in the Office 365 environment but no logs are getting displayed?
The user must be assigned an Office 365 E5 license. Alternatively, users with an Office 365 E1 or E3 license can be assigned an Advanced eDiscovery standalone license. Administrators and compliance officers who are assigned to cases and use Advanced eDiscovery to analyze data don't need an E5 license.

Audit log search argument start date should be after
The start date should be earlier then the end date.

New-PSSession: [outlook.office365.com] Connecting to remove server outlook.office365.com failed with the following error message: Access is denied.
The password/username combination are incorrect or the user has not enough privileges to extract the audit logging.

Custom script was developed by Joey Rentenaar and Korstiaan Stam from PwC Netherlands Incident Response team.
Idea is based on a script developed by Tehnoon Raza from Microsoft:
(https://blogs.msdn.microsoft.com/tehnoonr/2018/01/26/retrieving-office-365-audit-data-using-powershell/).

thuml/Time-Series-Library

2024-08-21T01:33:07Z

Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

 ___       _       _ _ _                  _     _____              _ _               ____        _ 
|_ _|_ __ | |_ ___| | (_) __ _  ___ _ __ | |_  |_   _| __ __ _  __| (_)_ __   __ _  | __ )  ___ | |_
 | || '_ \| __/ _ \ | | |/ _` |/ _ \ '_ \| __|   | || '__/ _` |/ _` | | '_ \ / _` | |  _ \ / _ \| __|
 | || | | | ||  __/ | | | (_| |  __/ | | | |_    | || | | (_| | (_| | | | | | (_| | | |_) | (_) | |_ 
|___|_| |_|\__\___|_|_|_|\__, |\___|_| |_|\__|   |_||_|  \__,_|\__,_|_|_| |_|\__, | |____/ \___/ \__|
                         |___/                                               |___/                   
₿   Ξ   ₳   ₮   ✕   ◎   ●   Ð   Ł   Ƀ   Ⱥ   ∞   ξ   ◈   ꜩ   ɱ   ε   ɨ   Ɓ   Μ   Đ  ⓩ  Ο   Ӿ   Ɍ  ȿ

📈 Intelligent Trading Signals 📉 https://t.me/intelligent_trading_signals

Intelligent trading bot

The project is aimed at developing an intelligent trading bot for automated trading cryptocurrencies using state-of-the-art machine learning (ML) algorithms and feature engineering. The project provides the following major functionalities:

Defining derived features using custom (Python) functions including technical indicators
Analyzing historic data and training machine learning models in batch off-line mode
Analyzing the predicted scores and choosing best signal parameters
Signaling service which is regularly requests new data from the exchange and generates buy-sell signals by applying the previously trained models in on-line mode
Trading service which does real trading by buying or selling the assets according to the generated signals

Intelligent trading channel

The signaling service is running in cloud and sends its signals to this Telegram channel:

📈 Intelligent Trading Signals 📉 https://t.me/intelligent_trading_signals

Everybody can subscribe to the channel to get the impression about the signals this bot generates.

Currently, the bot is configured using the following parameters:

Exchange: Binance
Cryptocurrency: ₿ Bitcoin
Analysis frequency: 1 minute (currently the only option)
Score between -1 and +1. <0 means likely to decrease, and >0 means likely to increase
Filter: notifications are sent only if score is greater than ±0.20 (may change)
One increase/decrease sign is added for each step of 0.05 (exceeding the filter threshold)

There are silent periods when the score in lower than the threshold and no notifications are sent to the channel. If the score is greater than the threshold, then every minute a notification is sent which looks like

₿ 24.518 📉📉📉 Score: -0.26

The first number is the latest close price. The score -0.26 means that it is very likely to see the price lower than the current close price.

If the score exceeds some threshold specified in the model then buy or sell signal is generated which means that it is a good time to do a trade. Such notifications look as follows:

🟢 BUY: ₿ 24,033 Score: +0.34

Training machine learning models (offline)

For the signaler service to work, a number of ML models must be trained and the model files available for the service. All scripts run in batch mode by loading some input data and storing some output files. The batch scripts are located in the scripts module.

If everything is configured then the following scripts have to be executed:

python -m scripts.download_binance -c config.json
python -m scripts.merge -c config.json
python -m scripts.features -c config.json
python -m scripts.labels -c config.json
python -m scripts.train -c config.json
python -m scripts.signals -c config.json
python -m scripts.train_signals -c config.json

Without a configuration file the scripts will use the default parameters which is useful for testing purposes and not intended for showing good performance. Use sample configuration files which are provided for each release like config-sample-v0.6.0.jsonc.

Downloading and merging source data

The main configuration parameter for the both scripts is a list of sources in data_sources. One entry in this list specifies a data source as well as column_prefix used to distinguish columns with the same name from different sources.

Download the latest historic data: python -m scripts.download_binance -c config.json
- It uses Binance API but you can use any other data source or download data manually using other scripts
Merge several historic datasets into one dataset: python -m scripts.merge -c config.json
- This script solves two problems: 1) there could be other sources like depth data or futures 2) a data source may have gaps so we need to produce a regular time raster in the output file

Generate features

This script is intended for computing derived features:

Script: python -m scripts.features -c config.json
Currently it runs in non-incremental model by computing features for all available input records (and not only for the latest update), and hence it may take hours for complex configurations
The script loads merged input data, applies feature generation procedures and stores all derived features in an output file
Not all generated features will be used for training and prediction. For the train/predict phases, a separate list of features is specified
Feature functions get additional parameters like windows from the config section
The same features must be used for on-line feature generation (in the service when they are generated for a micro-batch) and off-line feature generation.

The list of features to be generated is configured via feature_sets list in the configuration file. How features are generated is defined by the feature generator each having some parameters specified in its config section.

talib feature generator relies on the TA-lib technical analysis library. Here an example of its configuration: "config": {"columns": ["close"], "functions": ["SMA"], "windows": [5, 10, 15]}
itbstats feature generator implements functions which can be found in tsfresh like scipy_skew, scipy_kurtosis, lsbm (longest strike below mean), fmax (first location of maximum), mean, std, area, slope. Here are typical parameters: "config": {"columns": ["close"], "functions": ["skew", "fmax"], "windows": [5, 10, 15]}
itblib feature generator implemented in ITB but most of its features can be generated (much faster) via talib
tsfresh generates functions from the tsfresh library

Generate labels

This script is similar to feature generation because it adds new columns to the input file. However, these columns describe something that we want to predict and what is not known when executing in online mode. For example, it could be price increase in future:

Script: python -m scripts.labels -c config.json
The script loads features, computes label columns and stores the result in output file
Not all generated labels have to be used. The labels to be used for training are specified in a separate list

The list of labels to be generated is configured via label_sets list in the configuration. One label set points to the function which generates additional columns. Their configuration is very similar to feature configurations.

highlow label generator returns True if the price is higher than the specified threshold within some future horizon
highlow2 Computes future increases (decreases) with the conditions that there are no significant decreases (increases) before that. Here is its typical configuration: "config": {"columns": ["close", "high", "low"], "function": "high", "thresholds": [1.0, 1.5, 2.0], "tolerance": 0.2, "horizon": 10080, "names": ["first_high_10", "first_high_15", "first_high_20"]}
topbot Deprecated
topbot2 Computes maximum and minimum values (labeled as True). Every labelled maximum (minimum) is guaranteed to be surrounded by minimums (maximums) lower (higher) than the specified level. The required minimum difference between adjacent minimums and maximums is specified via level parameters. The tolerance parameter allows for including also points close to the maximum/minimum. Here is a typical configuration: "config": {"columns": "close", "function": "bot", "level": 0.02, "tolerances": [0.1, 0.2], "names": ["bot2_1", "bot2_2"]}

Train prediction models

This script uses the specified input features and labels to train several ML models:

Script: python -m scripts.train -c config.json
Hyper-parameter tuning is not part of this procedure - they are supposed to be known
The algorithm descriptions and hyper-parameters are specified in the model store
The results are stored as multiple model files in the model folder. File names are equal to the predicted column names and have this pattern: (label_name, algorithm_name)
This script trains models for all specified labels and all specified algorithms
The script also generates prediction-metrics.txt file with the prediction scores for all models

Configuration:

Models and hyper-parameters are described in model_store.py
Features to be used for training are specified in train_features
List of labels is specified in labels
List of algorithms is specified in algorithms

Aggregation and post-processing

The goal of this step is to aggregate the prediction scores generated by different algorithms for different labels. The result is one score which is supposed to be consumed by the signal rules on the next step. The aggregation parameters are specified in the score_aggregation section. The buy_labels and sell_labels specify input prediction scores processed by the aggregation procedure. window is the number of previous steps used for rolling aggregation and combine is a way how two score types (buy and labels) are combined into one output score.

Signal generation

The score generated by the aggregation procedure is some number and the goal of signal rules is to make the trading decisions: buy, sell or do nothing. The parameters of the signal rules are described in the trade_model.

Train signal models

This script simulates trades using many buy-sell signal parameters and then chooses the best performing signal parameters:

Script: python -m scripts.train_signals -c config.json

Prediction online based on trained models (service)

This script starts a service which periodically executes one and the same task: load latest data, generate features, make predictions, generate signals, notify subscribers:

Start script: python -m service.server -c config.json
The service assumes that the models were trained using the features specified in the configuration
The service uses credentials to access the exchange which are specified in the configuration

Hyper-parameter tuning

There are two problems:

How to choose best hyper-parameters for ML models. This problem is solved in the classical way, e.g., by grid search. For example, for Gradient Boosting, we train the model on the same data using different hyper-parameters and then select those showing the best score. This approach has one drawback - we optimize it for the best score which is not trading performance. This means that the trading performance is not guaranteed to be good (and in fact it will not be good). Therefore, we use this score as an intermediate feature with the goal to optimize trading performance on later stages.
If we compute the final aggregated score (like +0.21), then the question is should we buy, sell or do nothing? In fact, it is the most difficult question. To help answer it, additional scripts were developed for backtesting and optimizing buy-sell signal generation:
- Generate rolling predictions which simulates what we do by regularly re-training the models and using them for prediction: python -m scripts.predict_rolling -c config.json
- Train signal models for choosing the best thresholds for sell-buy signals producing the best performance on historic data: python -m scripts.train_signals -c config.json

Configuration parameters

The configuration parameters are specified in two files:

service.App.py in the config field of the App class
-c config.jsom argument to the services and scripts. The values from this config file will overwrite those in the App.config when this file is loaded into a script or service

Here are some most important fields (in both App.py and config.json):

data_folder - location of data files which are needed only for batch offline scripts
symbol it is a trading pair like BTCUSDT
Analyzer parameters. These mainly columns names.
- labels List of column names which are treated as labels. If you define a new label used for training and then for prediction then you need to specify its name here
- algorithms List of algorithm names used for training
- train_features List of all column names used as input features for training and prediction.
Signers:
- buy_labels and sell_labels Lists of predicted columns used for signals
- trade_model Parameters of the signaler (mainly some thresholds)
trader is a section for trader parameters. Currently, not thoroughly tested.
collector These parameter section is intended for data collection services. There are two types of data collection services: synchronous with regular requests to the data provider and asynchronous streaming service which subscribes to the data provider and gets notifications as soon as new data is available. They are working but not thoroughly tested and integrated into the main service. The current main usage pattern relies on manual batch data updates, feature generation and model training. One reason for having these data collection services is 1) to have faster updates 2) to have data not available in normal API like order book (there exist some features which use this data but they are not integrated into the main workflow).

See sample configuration files and comments in App.config for more details.

Signaler service

Every minute, the signaler performs the following steps to make a prediction about whether the price is likely to increase or decrease:

Retrieve the latest data from the server and update the current data window which includes some history (the history length is defined by a configuration parameter)
Compute derived features based on the nearest history collected (which now includes the latest data). The features to be computed are described in the configuration file and are exactly the same as used in batch mode during model training
Apply several (previously trained) ML models by forecasting some future values (not necessarily prices) which are also treated as (more complex) derived features. We apply several forecasting models (currently, Gradient Boosting, Neural network, and Linear regression) to several target variables (labels)
Aggregate the results of forecasting produced by different ML models and compute the final signal score which reflects the strength of the upward or downward trend. Here we use many previously computed scores as inputs and derive one output score. Currently, it is implemented as an aggregation procedure but it could be based on a dedicated ML model trained on previously collected scores and the target variable. Positive score means growth and negative score means fall
Use the final score for notifications

Notes:

The final result of the signaler is the score (between -1 and +1). The score should be used for further decisions about buying or selling by taking into account other parameters and data sources
For the signaler service to work, trained models have to be available and stored in the "MODELS" folder. The models are trained in batch mode and the process is described in the corresponding section.

Starting the service: python3 -m service.server -c config.json

Trader

The trader is working but not thoroughly debugged, particularly, not tested for stability and reliability. Therefore, it should be considered a prototype with basic functionality. It is currently integrated with the Signaler but in a better design should be a separate service.

Related projects

https://github.com/CryptoSignal/Crypto-Signal Github.com/CryptoSignal - #1 Quant Trading & Technical Analysis Bot
https://github.com/tensortrade-org/tensortrade An open source reinforcement learning framework for training, evaluating, and deploying robust trading agents
https://github.com/Superalgos/Superalgos Free, open-source crypto trading bot, automated bitcoin / cryptocurrency trading software, algorithmic trading bots. Visually design your crypto trading bot, leveraging an integrated charting system, data-mining, backtesting, paper trading, and multi-server crypto bot deployments
https://github.com/kieran-mackle/AutoTrader A Python-based development platform for automated trading systems - from backtesting to optimisation to livetrading
https://github.com/areed1192/python-trading-robot A trading robot, that can submit basic orders in an automated fashion using the TD API
https://github.com/jmrichardson/tuneta Intelligently optimizes technical indicators and optionally selects the least intercorrelated for use in machine learning models
https://github.com/Erfaniaa/binance-futures-trading-bot Easy-to-use multi-strategic automatic trading for Binance Futures with Telegram integration
https://github.com/smileinnovation/cryptocurrency-trading How to make profits in cryptocurrency trading with machine learning

Backtesting

External integrations

https://github.com/ccxt/ccxt A JavaScript / Python / PHP cryptocurrency trading API with support for more than 100 bitcoin/altcoin exchanges
https://github.com/aiogram/aiogram Is a pretty simple and fully asynchronous framework for Telegram Bot API
https://github.com/sammchardy/python-binance

YvanYin/Metric3D

2024-08-21T01:33:07Z

The repo for "Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image" and "Metric3Dv2: A Versatile Monocular Geometric Foundation Model..."

🚀 Metric3D Project 🚀

Official PyTorch implementation of Metric3Dv1 and Metric3Dv2:

[1] Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image

[2] Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation

🏆 Champion in CVPR2023 Monocular Depth Estimation Challenge

News

[2024/8] Metric3Dv2 is accepted by TPAMI!
[2024/7/5] Our stable-diffusion alternative GeoWizard has now been accepted by ECCV 2024! Check NOW the repository and paper for the finest-grained geometry ever! 🎉🎉🎉
[2024/6/25] Json files for KITTI datasets now available! Refer to Training for more details
[2024/6/3] ONNX is supported! We appreciate @xenova for their remarkable efforts!
[2024/4/25] Weights for ViT-giant2 model released!
[2024/4/11] Training codes are released!
[2024/3/18] HuggingFace 🤗 GPU version updated!
[2024/3/18] Project page released!
[2024/3/18] Metric3D V2 models released, supporting metric depth and surface normal now!
[2023/8/10] Inference codes, pre-trained weights, and demo released.
[2023/7] Metric3D accepted by ICCV 2023!
[2023/4] The Champion of 2nd Monocular Depth Estimation Challenge in CVPR 2023

🌼 Abstract

Metric3D is a strong and robust geometry foundation model for high-quality and zero-shot metric depth and surface normal estimation from a single image. It excels at solving in-the-wild scene reconstruction. It can directly help you measure the size of structures from a single image. Now it achieves SOTA performance on over 10 depth and normal benchmarks.

📝 Benchmarks

Metric Depth

Our models rank 1st on the routing KITTI and NYU benchmarks.

	Backbone	KITTI δ1 ↑	KITTI δ2 ↑	KITTI AbsRel ↓	KITTI RMSE ↓	KITTI RMS_log ↓	NYU δ1 ↑	NYU δ2 ↑	NYU AbsRel ↓	NYU RMSE ↓	NYU log10 ↓
ZoeDepth	ViT-Large	0.971	0.995	0.053	2.281	0.082	0.953	0.995	0.077	0.277	0.033
ZeroDepth	ResNet-18	0.968	0.996	0.057	2.087	0.083	0.954	0.995	0.074	0.269	0.103
IEBins	SwinT-Large	0.978	0.998	0.050	2.011	0.075	0.936	0.992	0.087	0.314	0.031
DepthAnything	ViT-Large	0.982	0.998	0.046	1.985	0.069	0.984	0.998	0.056	0.206	0.024
Ours	ViT-Large	0.985	0.998	0.044	1.985	0.064	0.989	0.998	0.047	0.183	0.020
Ours	ViT-giant2	0.989	0.998	0.039	1.766	0.060	0.987	0.997	0.045	0.187	0.015

Affine-invariant Depth

Even compared to recent affine-invariant depth methods (Marigold and Depth Anything), our metric-depth (and normal) models still show superior performance.

	#Data for Pretrain and Train	KITTI Absrel ↓	KITTI δ1 ↑	NYUv2 AbsRel ↓	NYUv2 δ1 ↑	DIODE-Full AbsRel ↓	DIODE-Full δ1 ↑	Eth3d AbsRel ↓	Eth3d δ1 ↑
OmniData (v2, ViT-L)	1.3M + 12.2M	0.069	0.948	0.074	0.945	0.149	0.835	0.166	0.778
MariGold (LDMv2)	5B + 74K	0.099	0.916	0.055	0.961	0.308	0.773	0.127	0.960
DepthAnything (ViT-L)	142M + 63M	0.076	0.947	0.043	0.981	0.277	0.759	0.065	0.882
Ours (ViT-L)	142M + 16M	0.042	0.979	0.042	0.980	0.141	0.882	0.042	0.987
Ours (ViT-g)	142M + 16M	0.043	0.982	0.043	0.981	0.136	0.895	0.042	0.983

Surface Normal

Our models also show powerful performance on normal benchmarks.

	NYU 11.25° ↑	NYU Mean ↓	NYU RMS ↓	ScanNet 11.25° ↑	ScanNet Mean ↓	ScanNet RMS ↓	iBims 11.25° ↑	iBims Mean ↓	iBims RMS ↓
EESNU	0.597	16.0	24.7	0.711	11.8	20.3	0.585	20.0	-
IronDepth	-	-	-	-	-	-	0.431	25.3	37.4
PolyMax	0.656	13.1	20.4	-	-	-	-	-	-
Ours (ViT-L)	0.688	12.0	19.2	0.760	9.9	16.4	0.694	19.4	34.9
Ours (ViT-g)	0.662	13.2	20.2	0.778	9.2	15.3	0.697	19.6	35.2

🌈 DEMOs

Zero-shot monocular metric depth & surface normal

Zero-shot metric 3D recovery

Improving monocular SLAM

🔨 Installation

One-line Installation

For the ViT models, use the following environment：

pip install -r requirements_v2.txt

For ConvNeXt-L, it is

pip install -r requirements_v1.txt

dataset annotation components

With off-the-shelf depth datasets, we need to generate json annotaions in compatible with this dataset, which is organized by:

dict(
	'files':list(
		dict(
			'rgb': 'data/kitti_demo/rgb/xxx.png',
			'depth': 'data/kitti_demo/depth/xxx.png',
			'depth_scale': 1000.0 # the depth scale of gt depth img.
			'cam_in': [fx, fy, cx, cy],
		),

		dict(
			...
		),

		...
	)
)

To generate such annotations, please refer to the "Inference" section.

configs

In mono/configs we provide different config setups.

Intrinsics of the canonical camera is set bellow:

    canonical_space = dict(
        img_size=(512, 960),
        focal_length=1000.0,
    ),

where cx and cy is set to be half of the image size.

Inference settings are defined as

    depth_range=(0, 1),
    depth_normalize=(0.3, 150),
    crop_size = (512, 1088),

where the images will be first resized as the crop_size and then fed into the model.

✈️ Training

Please refer to training/README.md. Now we provide complete json files for KITTI fine-tuning.

✈️ Inference

News: Improved ONNX support with dynamic shapes (Feature owned by @xenova. Appreciate for this outstanding contribution 🚩🚩🚩)

Now the onnx supports are availble for all three models with varying shapes. Refer to issue117 for more details.

Improved ONNX Checkpoints Available now

	Encoder	Decoder	Link
v2-S-ONNX	DINO2reg-ViT-Small	RAFT-4iter	Download 🤗
v2-L-ONNX	DINO2reg-ViT-Large	RAFT-8iter	Download 🤗
v2-g-ONNX	DINO2reg-ViT-giant2	RAFT-8iter	Download 🤗

One additional reminder for using these onnx models is reported by @norbertlink.

News: Pytorch Hub is supported

Now you can use Metric3D via Pytorch Hub with just few lines of code:

import torch
model = torch.hub.load('yvanyin/metric3d', 'metric3d_vit_small', pretrain=True)
pred_depth, confidence, output_dict = model.inference({'input': rgb})
pred_normal = output_dict['prediction_normal'][:, :3, :, :] # only available for Metric3Dv2 i.e., ViT models
normal_confidence = output_dict['prediction_normal'][:, 3, :, :] # see https://arxiv.org/abs/2109.09881 for details

Supported models: metric3d_convnext_tiny, metric3d_convnext_large, metric3d_vit_small, metric3d_vit_large, metric3d_vit_giant2.

We also provided a minimal working example in hubconf.py, which hopefully makes everything clearer.

News: ONNX Exportation and Inference are supported

We also provided a flexible working example in metric3d_onnx_export.py to export the Pytorch Hub model to ONNX format. We could test with the following commands:

# Export the model to ONNX model
python3 onnx/metric_3d_onnx_export.py metric3d_vit_small # metric3d_vit_large/metric3d_convnext_large

# Test the inference of the ONNX model
python3 onnx/test_onnx.py metric3d_vit_small.onnx

ros2_vision_inference provides a Python example, showcasing a pipeline from image to point clouds and integrated into ROS2 systems.

Download Checkpoint

	Encoder	Decoder	Link
v1-T	ConvNeXt-Tiny	Hourglass-Decoder	Download 🤗
v1-L	ConvNeXt-Large	Hourglass-Decoder	Download
v2-S	DINO2reg-ViT-Small	RAFT-4iter	Download
v2-L	DINO2reg-ViT-Large	RAFT-8iter	Download
v2-g	DINO2reg-ViT-giant2	RAFT-8iter	Download 🤗

Dataset Mode

put the trained ckpt file model.pth in weight/.
generate data annotation by following the code data/gene_annos_kitti_demo.py, which includes 'rgb', (optional) 'intrinsic', (optional) 'depth', (optional) 'depth_scale'.
change the 'test_data_path' in test_*.sh to the *.json path.
run source test_kitti.sh or source test_nyu.sh.

In-the-Wild Mode

put the trained ckpt file model.pth in weight/.
change the 'test_data_path' in test.sh to the image folder path.
run source test_vit.sh for transformers and source test.sh for convnets. As no intrinsics are provided, we provided by default 9 settings of focal length.

Metric3D and Droid-Slam

If you are interested in combining metric3D and monocular visual slam system to achieve the metric slam, you can refer to this repo.

❓ Q & A

Q1: Why depth maps look good but pointclouds are distorted?

Because the focal length is not properly set! Please find a proper focal length by modifying codes here yourself.

Q2: Why the pointclouds are too slow to be generated?

Because the images are too large! Use smaller ones instead.

Q3: Why predicted depth maps are not satisfactory?

First be sure all black padding regions at image boundaries are cropped out. Then please try again. Besides, metric 3D is not almighty. Some objects (chandeliers, drones...) / camera views (aerial view, bev...) do not occur frequently in the training datasets. We will going deeper into this and release more powerful solutions.

📧 Citation

@article{hu2024metric3d,
  title={Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation},
  author={Hu, Mu and Yin, Wei and Zhang, Chi and Cai, Zhipeng and Long, Xiaoxiao and Chen, Hao and Wang, Kaixuan and Yu, Gang and Shen, Chunhua and Shen, Shaojie},
  journal={arXiv preprint arXiv:2404.15506},
  year={2024}
}

@article{yin2023metric,
  title={Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image},
  author={Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, Chunhua Shen},
  booktitle={ICCV},
  year={2023}
}

License and Contact

The Metric 3D code is under a 2-clause BSD License. For further commercial inquiries, please contact Dr. Wei Yin [yvanwy@outlook.com] and Mr. Mu Hu [mhuam@connect.ust.hk].

NVlabs/VILA

2024-08-22T01:33:20Z

VILA - a multi-image visual language model with training, inference and evaluation recipe, deployable from cloud to edge (Jetson Orin and laptops)

VILA: On Pre-training for Visual Language Models

VILA arxiv / VILA Demo / VILA Huggingface

💡 Introduction

VILA is a visual language model (VLM) pretrained with interleaved image-text data at scale, enabling video understanding and multi-image understanding capabilities. VILA is deployable on the edge by AWQ 4bit quantization and TinyChat framework. We find: (1) image-text pairs are not enough, interleaved image-text is essential; (2) unfreezing LLM during interleaved image-text pre-training enables in-context learning; (3)re-blending text-only instruction data is crucial to boost both VLM and text-only performance; (4) token compression extends #video frames. VILA unveils appealing capabilities, including: video reasoning, in-context learning, visual chain-of-thought, and better world knowledge.

💡 News

[2024/08] We release LongVILA that supports long video understanding (Captioning, QA, Needle-in-a-Haystack) up to 1024 frames.
[2024/07] VILA1.5 also ranks 1st place (OSS model) on MLVU test leaderboard.
[2024/06] VILA1.5 is now the best open sourced VLM on MMMU leaderboard and Video-MME leaderboard!
[2024/05] We release VILA-1.5, which offers video understanding capability. VILA-1.5 comes with four model sizes: 3B/8B/13B/40B.
[2024/05] We release AWQ-quantized 4bit VILA-1.5 models. VILA-1.5 is efficiently deployable on diverse NVIDIA GPUs (A100, 4090, 4070 Laptop, Orin, Orin Nano) by TinyChat and TensorRT-LLM backends.
[2024/03] VILA has been accepted by CVPR 2024!
[2024/02] We release AWQ-quantized 4bit VILA models, deployable on Jetson Orin and laptops through TinyChat and TinyChatEngine.
[2024/02] VILA is released. We propose interleaved image-text pretraining that enables multi-image VLM. VILA comes with impressive in-context learning capabilities. We open source everything: including training code, evaluation code, datasets, model ckpts.
[2023/12] Paper is on Arxiv!

Performance

Image QA Benchmarks

$~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$	Prec.	VQAv2	GQA	VizWiz	SQA-I	VQA-T	POPE	MME	MMB	MMB-CN	SEED	SEED-I	MMMU (val)	MMMU (test)	llava-bench	MM-Vet	Average
VILA1.5-3B	fp16	80.4	61.5	53.5	69.0	60.4	85.9	1442.44	63.4	52.7	60.9	67.9	33.3	30.8	75.9	35.4	60.2
VILA1.5-3B-AWQ	int4	80.0	61.1	53.8	67.8	60.4	85.9	1437.34	63.3	51.4	59.8	66.6	32.7	31.1	75.0	37.3	59.9
VILA1.5-3B-S2	fp16	79.8	61.4	61.3	69.6	63.4	85.3	1431.65	62.8	52.2	60.0	66.4	32.8	31.3	76.7	38.6	60.9
VILA1.5-3B-S2-AWQ	int4	79.4	61.3	62.3	69.2	63.0	85.8	1417.06	61.6	51.5	59.1	65.7	33.4	30.4	77.1	36.7	60.5
Llama-3-VILA1.5-8B	fp16	80.9	61.9	58.7	79.9	66.3	84.4	1577.01	72.3	66.2	64.2	71.4	36.9	36.0	80.0	38.3	65.1
Llama-3-VILA1.5-8B-AWQ	int4	80.3	61.7	59.3	79.0	65.4	82.9	1593.65	71.0	64.9	64.0	71.1	36.0	36.1	79.0	37.2	64.5
VILA1.5-13B	fp16	82.8	64.3	62.6	80.1	65.0	86.3	1569.55	74.9	66.3	65.1	72.6	37.9	33.6	80.8	44.3	66.3
VILA1.5-13B-AWQ	int4	82.7	64.5	63.3	79.7	64.7	86.7	1531.35	74.7	66.7	65.1	72.6	37.8	34.0	81.9	46.4	66.5
VILA1.5-40B	fp16	84.3	64.6	62.2	87.2	73.6	87.3	1726.82	82.4	80.2	69.1	75.8	51.9	46.9	81.3	53.0	72.4
VILA1.5-40B-AWQ	int4	84.1	64.4	61.3	86.7	73.2	88.2	1714.79	83.2	79.6	68.9	75.6	49.3	46.2	83.0	51.4	72.1

^{NOTE: VQAV2 and VizWiz are test-dev, the average accuracy is calculated over all datasets and MME numbers are divided by 20.}

Video QA Benchmarks

$~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$	Prec.	Perception Test	ActivityNet	MSVD	MSRVTT	TGIF	EgoSchema (test)	CinePile
VILA1.5-3B	fp16	47	50.2	76.6	57.5	51.7	42.6	37.9
VILA1.5-3B-S2	fp16	49.7	50.7	76.9	57.6	51.7
Llama-3-VILA1.5-8B	fp16	54.1	54.3	78.3	60.1	54.1	50.4	48.7
VILA1.5-13B	fp16	53.6	54.7	77.9	60.2	56	52.2	50.1
VILA1.5-40B	fp16	54	58	80.1	63	58.2	58.7	51.3

Inference speed ( Token/sec )

$~~~~~~$	Precision	A100	4090	Orin
VILA1.5-3B	fp16	104.6	137.6	25.4
VILA1.5-3B-AWQ	int4	182.8	215.5	42.5
VILA1.5-3B-S2	fp16	104.3	137.2	24.6
VILA1.5-3B-S2-AWQ	int4	180.2	219.3	40.1
Llama-3-VILA1.5-8B	fp16	74.9	57.4	10.2
Llama-3-VILA1.5-8B-AWQ	int4	168.9	150.2	28.7
VILA1.5-13B	fp16	50.9	OOM	6.1
VILA1.5-13B-AWQ	int4	115.9	105.7	20.6
VILA1.5-40B	fp16	OOM	OOM	--
VILA1.5-40B-AWQ	int4	57.0	OOM	--

^{NOTE: Measured using the TinyChat backend at batch size = 1.}

VILA Examples

Video captioning

https://github.com/Efficient-Large-Model/VILA/assets/156256291/c9520943-2478-4f97-bc95-121d625018a6

Prompt: Elaborate on the visual and narrative elements of the video in detail.

Caption: The video shows a person's hands working on a white surface. They are folding a piece of fabric with a checkered pattern in shades of blue and white. The fabric is being folded into a smaller, more compact shape. The person's fingernails are painted red, and they are wearing a black and red garment. There are also a ruler and a pencil on the surface, suggesting that measurements and precision are involved in the process.

In context learning

Multi-image reasoning

VILA on Jetson Orin

https://github.com/Efficient-Large-Model/VILA/assets/7783214/6079374c-0787-4bc4-b9c6-e1524b4c9dc4

VILA on RTX 4090

https://github.com/Efficient-Large-Model/VILA/assets/7783214/80c47742-e873-4080-ad7d-d17c4700539f

Installation

./environment_setup.sh vila

Training

VILA training contains three steps, for specific hyperparameters, please check out the scripts/v1_5 folder:

Step-1: Alignment

We utilize LLaVA-CC3M-Pretrain-595K dataset to align the textual and visual modalities.

The stage 1 script takes in two parameters and it can run on a single 8xA100 node. BASE_MODEL_PATH points to a online or local huggingface repository, such as NousResearch/Llama-2-7b-hf. OUTPUT_NAME points to a target directory under checkpoints, which will save the trained multimodal projector afterwards.

bash scripts/v1_5/paper/1_mm_align.sh [BASE_MODEL_PATH] [OUTPUT_NAME]

Step-2: Pretraining

We use MMC4 and Coyo dataset to train VLM with interleaved image-text pairs.

bash scripts/v1_5/paper/2_pretrain_mmc4_coyo.sh [CODE_PATH] [BASE_MODEL_PATH] [STAGE1_PATH] [OUTPUT_NAME]

The stage 2 script takes in four arguments. CODE_PATH is the absolute path to our VILA codebase, BASE_MODEL_PATH has similar meaning to what is presented in the stage 1 script. STAGE1_PATH points to the OUTPUT_NAME of stage 1 (i.e. where the stage 1 checkpoint is stored). OUTPUT_NAME is the desired folder name under checkpoints that saves the pretraining checkpoint. The script we provided for this stage is executed on slurm, and we expect it to execute on 16 nodes (128 GPUs).

Step-3: Supervised fine-tuning

This is the last stage of VILA training, in which we tune the model to follow multimodal instructions on a subset of M3IT, FLAN and ShareGPT4V. This stage runs on a 8xA100 node.

bash scripts/v1_5/paper/3_sft.sh [STAGE2_PATH] [OUTPUT_NAME]

The stage 3 script takes in two arguments. STAGE2_PATH points to the OUTPUT_NAME of the stage 2 script (i.e. where the stage 2 checkpoint is stored). OUTPUT_NAME is the desired folder name under checkpoints that stores the final checkpoint.

Evaluations

Image Benchmarks

You can follow Llava1.5 eval to download all datasets. After downloading all datasets, please put them under playground/data/eval.

Please make the following changes to the MME evaluation script. Please search for:

data_path = "MME_Benchmark_release_version"

and replace it with:

data_path = os.path.join(script_dir, "MME_Benchmark_release_version")

We provide a push-the-button script to perform evaluation on all 10 datasets that do not require GPT-assisted evaluation:

./scripts/v1_5/eval/eval_all.sh [CHECKPOINT_PATH] [MODEL_NAME] [CONV_MODE]

This script takes in two parameters, CHECKPOINT_PATH points to the stage 3 model checkpoint, and MODEL_NAME will be the name of evaluation results.

VQAv2 and Vizwiz evaluations are hosted on eval.ai. You need to register an account and create a team to be able to submit eval.

MMBench and MMBench_CN eval are hosted on another evaluation server. Make sure you change the name of the file before submitting, otherwise the server caches results and will always return wrong result to you.

We provide a quick script to automatically organize the prediction files that need to be submitted to servers:

python scripts/v1_5/eval/copy_predictions.py [MODEL_NAME]

You will be able to find the predictions under playground/data/predictions_upload/[MODEL_NAME] after executing this script.

Video Benchmarks

Please follow the evaluation steps in Video-LLaVA for dataset preparation.

./scripts/v1_5/eval/video_chatgpt/run_all.sh [CHECKPOINT_PATH] [MODEL_NAME] [CONV_MODE]
./scripts/v1_5/eval/video_chatgpt/eval_all.sh [MODEL_NAME]

Inference

We provide snippets for quick inference with user prompts and images.

Llama-3-VILA1.5-8B inference:

python -W ignore llava/eval/run_vila.py \
    --model-path Efficient-Large-Model/Llama-3-VILA1.5-8b \
    --conv-mode llama_3 \
    --query "<image>\n Please describe the traffic condition." \
    --image-file "av.png"

VILA1.5-40B inference:

python -W ignore llava/eval/run_vila.py \
    --model-path Efficient-Large-Model/VILA1.5-40b \
    --conv-mode hermes-2 \
    --query "<image>\n Please describe the traffic condition." \
    --image-file "av.png"

VILA1.5-3B video inference:

python -W ignore llava/eval/run_vila.py \
    --model-path Efficient-Large-Model/VILA1.5-3b \
    --conv-mode vicuna_v1 \
    --query "<video>\n Please describe this video." \
    --video-file "demo.mp4"

Quantization and Deployment

Our VILA models are quantized by AWQ into 4 bits for efficient inference on the edge. We provide a push-the-button script to quantize VILA with AWQ.

Running VILA on desktop GPUs and edge GPUs

We support AWQ-quantized 4bit VILA on GPU platforms via TinyChat. We provide a tutorial to run the model with TinyChat after quantization. We also provide an instruction to launch a Gradio server (powered by TinyChat and AWQ) to serve 4-bit quantized VILA models.

Running VILA on laptops

We further support our AWQ-quantized 4bit VILA models on various CPU platforms with both x86 and ARM architectures with our TinyChatEngine. We also provide a detailed tutorial to help the users deploy VILA on different CPUs.

Running VILA API server

A simple API server has been provided to serve VILA models. The server is built on top of FastAPI and Huggingface Transformers. The server can be run with the following command:

With CLI

python -W ignore server.py \
    --port 8000 \
    --model-path Efficient-Large-Model/VILA1.5-3B \
    --conv-mode vicuna_v1

With Docker

docker build -t vila-server:latest .
docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
    -v ./hub:/root/.cache/huggingface/hub \
    -it --rm -p 8000:8000 \
    -e VILA_MODEL_PATH=Efficient-Large-Model/VILA1.5-3B \
    -e VILA_CONV_MODE=vicuna_v1 \
    vila-server:latest

Then you can call the endpoint with the OpenAI SDK as follows:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000",
    api_key="fake-key",
)
response = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://blog.logomyway.com/wp-content/uploads/2022/01/NVIDIA-logo.jpg",
                        # Or you can pass in a base64 encoded image
                        # "url": "data:image/png;base64,<base64_encoded_image>",
                    },
                },
            ],
        }
    ],
    max_tokens=300,
    model="VILA1.5-3B",
    # You can pass in extra parameters as follows
    extra_body={"num_beams": 1, "use_cache": False},
)
print(response.choices[0].message.content)

^{NOTE: This API server is intended for evaluation purposes only and has not been optimized for production use. It has only been tested on A100 and H100 GPUs.}

Checkpoints

We release VILA1.5-3B, VILA1.5-3B-S2, Llama-3-VILA1.5-8B, VILA1.5-13B, VILA1.5-40B and the 4-bit AWQ-quantized models VILA1.5-3B-AWQ, VILA1.5-3B-S2-AWQ, Llama-3-VILA1.5-8B-AWQ, VILA1.5-13B-AWQ, VILA1.5-40B-AWQ.

🔒 License

The code is released under the Apache 2.0 license as found in the LICENSE file.
The pretrained weights are released under the CC-BY-NC-SA-4.0 license.
The service is a research preview intended for non-commercial use only, and is subject to the following licenses and terms:
- Model License of LLaMA. For LLAMA3-VILA checkpoints terms of use, please refer to the LLAMA3 License for additional details.
- Terms of Use of the data generated by OpenAI
- Dataset Licenses for each one used during training.

Team


*Yao Lu: Nvidia	*Hongxu Yin: Nvidia	*Ji Lin: OpenAI (work done at Nvidia and MIT)
Wei Ping: Nvidia	Pavlo Molchanov: Nvidia	Andrew Tao: Nvidia
Haotian Tang: MIT	Shang Yang: MIT	Ligeng Zhu: Nvidia, MIT
Wei-Chen Wang: MIT	Fuzhao Xue: Nvidia, NUS	Yunhao Fang: Nvidia, UCSD
Yukang Chen: Nvidia, CUHK	Zhuoyang Zhang: Nvidia, Tsinghua Univ.	Yue Shen: Nvidia
Wei-Ming Chen: Nvidia	Huizi Mao: Nvidia	Baifeng Shi: Nvidia, UC Berkeley
Jan Kautz: Nvidia	Mohammad Shoeybi: Nvidia	Song Han: Nvidia, MIT

Citations

@misc{lin2023vila,
      title={VILA: On Pre-training for Visual Language Models},
      author={Ji Lin and Hongxu Yin and Wei Ping and Yao Lu and Pavlo Molchanov and Andrew Tao and Huizi Mao and Jan Kautz and Mohammad Shoeybi and Song Han},
      year={2023},
      eprint={2312.07533},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

LLaVA: the codebase we built upon. Thanks for their wonderful work.
InternVL: for open-sourcing InternViT (used in VILA1.5-40b) and the InternVL-SFT data blend (inspired by LLaVA-1.6) used in all VILA1.5 models.
Vicuna: the amazing open-sourced large language model!
Video-ChatGPT: we borrowed video evaluation script from this repository.
MMC4, COYO-700M, M3IT, OpenORCA/FLAN, ShareGPT4V, WIT, GSM8K-ScRel, VisualGenome, VCR, ScienceQA, Shot2Story, Youcook2, Vatex, ShareGPT-Video for providing datasets used in this research.

firstbatchxyz/dkn-compute-node

2024-08-22T01:33:42Z

Compute Node of Dria Knowledge Network.

Dria Compute Node

Dria Compute Node serves the computation results within Dria Knowledge Network.

About

A Dria Compute Node is a unit of computation within the Dria Knowledge Network. It's purpose is to process tasks given by the Dria Admin Node, and receive rewards for providing correct results.

To get started, setup your envrionment and then see usage to run the node.

Tasks

Compute nodes can technically do any arbitrary task, from computing the square root of a given number to finding LLM outputs from a given prompt, or validating an LLM's output with respect to knowledge available on the web accessed via tools.

Ping/Pong: Dria Admin Node broadcasts ping messages at a set interval, it is a required duty of the compute node to respond with a pong to these so that they can be included in the list of available nodes for task assignment. These tasks will respect the type of model provided within the pong message, e.g. if a task requires gpt-4o and you are running phi3, you won't be selected for that task.
Workflows: Each task is given in the form of a workflow, based on Ollama Workflows (see repository for more information). In simple terms, each workflow defines the agentic behavior of an LLM, all captured in a single JSON file, and can represent things ranging from simple LLM generations to iterative web searching.

Requirements

The compute node is a very lightweight process, with few MBs of memory usage along with an image size of less than ~65MBs. If you are using Ollama, you will need the memory to run large models locally, which depend on the model's size that you are willing to.

You need the following applications to run compute node:

Git: We will use git to clone the repository from GitHub, and pull latest changes for updates later.
Docker: Our services will make use of Docker so that the node can run on any machine.

[!TIP]

You can check if you have these via:
which git
which docker

Setup

To be able to run a node, we need to make a few simple preparations. Follow the steps below one by one.

1. Clone the repository

This repository has the necessary setup to run the node, so start by cloning it using the command below:

git clone https://github.com/firstbatchxyz/dkn-compute-node
cd dkn-compute-node

2. Prepare Environment Variables

Dria Compute Node makes use of several environment variables. Create a .env file, and copy the environment variables as given in .env.example. We will fill out the missing parts in a moment.

cp .env.example .env

[!NOTE]

DKN_ADMIN_PUBLIC_KEY is used to verify that the tasks are given by certain nodes, so that your node does not work for tasks given to the network by untrusted people. You don't need to change this, simply copy and paste it to your .env.

[!TIP]

While adding anything to your .env, you can do it without leaving the terminal. For example, suppose you want to set VALUE to some KEY, you can do it as:
echo "KEY=VALUE" >> .env
If you would like to view the .env without leaving the terminal, you can do:
cat .env

3. Prepare Ethereum Wallet

Dria makes use of the same Ethereum wallet, that is the recipient of your hard-earned rewards! Place your private key at DKN_WALLET_SECRET_KEY in .env without the 0x prefix. It should look something like:

DKN_WALLET_SECRET_KEY=ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80

[!CAUTION]

Always make sure your private key is within the .gitignore'd .env file, nowhere else! To be even safer, you can use a throwaway wallet, you can always transfer your rewards to a main wallet afterwards.

4. Setup LLM Provider

For the final step, we need to make sure we can serve LLM requests.

For OpenAI

If you will be using OpenAI to serve its models, you need to have an API key in the environment. Simply set the key within your .env:

OPENAI_API_KEY=<YOUR_KEY>

For Ollama

Of course, first you have to install Ollama; see their download page. Then, you must first pull a small embedding model that is used internally.

ollama pull hellord/mxbai-embed-large-v1:f16

For the models that you choose (see list of models just below here) you can download them with same command. Note that if your model size is large, pulling them may take a while.

# example for phi3:3.8b
ollama pull phi3:3.8b

[!TIP]

Alternatively, you can set OLLAMA_AUTO_PULL=true in the .env so that the compute node will always download the missing models for you.

Optional Services

Based on presence of API keys, Ollama Workflows may use more superior services instead of free alternatives, e.g. Serper instead of DuckDuckGo or Jina without rate-limit instead of with rate-limit. Add these within your .env as:

SERPER_API_KEY=<key-here>
JINA_API_KEY=<key-here>

Usage

With all setup steps above completed, we are ready to start a node!

1. Choose Model(s)

Based on the resources of your machine, you must decide which models that you will be running locally. For example, you can use OpenAI with their models, not running anything locally at all; or you can use Ollama with several models loaded to disk, and only one loaded to memory during its respective task. Available models (see here for latest) are:

Ollama Models

adrienbrault/nous-hermes2theta-llama3-8b:q8_0
phi3:14b-medium-4k-instruct-q4_1
phi3:14b-medium-128k-instruct-q4_1
phi3:3.8b
llama3.1:latest

OpenAI Models

gpt-3.5-turbo
gpt-4-turbo
gpt-4o
gpt-4o-mini

[!TIP]

If you are using Ollama, make sure you have pulled the required models, as specified in the section above!

2. Start Docker

Our node will be running within a Docker container, so we should make sure that Docker is running before the next step. You can launch Docker via its desktop application, or a command such as:

sudo systemctl start docker

[!NOTE]

You don't need to do this step if Docker is already running in the background.

3. Run Node

It's time to run our compute node. We have a starter script that makes this much easier, you can see available commadns with:

See the available commands with:

chmod +x start.sh
./start.sh --help

Simply run the script with the model names provided, such as:

./start.sh -m=llama3.1:latest -m=gpt-3.5-turbo

Start script will run the containers in the background. You can check their logs either via the terminal or from Docker Desktop.

Running in Debug Mode

To print DEBUG-level logs for the compute node, you can add --dev argument to the start script. For example:

./start.sh -m=gpt-4o-mini --dev

Running in debug mode will also allow you to see behind the scenes of Ollama Workflows, i.e. you can see the reasoning of the LLM as it executes the task.

Similarly, you can run in trace mode with --trace to see trace logs, which cover low-level logs from the p2p client.

4. Looking at Logs

To see your logs, you can go to Docker Desktop and see the running containers and find dkn-compute-node. There, open the containers within the compose (click on > to the left) and click on any of the container to see its logs.

Alternatively, you can use docker compose logs such as below:

docker compose logs -f compute  # compute node logs
docker compose logs -f ollama   # ollama logs

The -f option is so that you can track the logs from terminal. If you prefer to simply check the latest logs, you can use a command such as:

# logs from last 1 hour
docker compose logs --since=1h compute

# logs from last 30 minutes
docker compose logs --since=30m compute

5. Stopping the Node

When you start your node with ./start.sh, it will wait for you in the same terminal to do CTRL+C before stopping. Once you do that, the containers will be stopped and removed. You can also kill the containers manually, doing CTRL+C afterwards will do nothing in such a case.

[!NOTE]

Sometimes it may not immediately exit whilst executing a task, if you REALLY need to quite the process you can kill it manually.

Using Ollama

If you don't have Ollama installed, you can ignore this section.

If you have Ollama installed already (e.g. via brew install ollama) then you must indicate that you will be using that Ollama, instead of a Docker container. To do this, we set the provide the argument --local-ollama=true which is true by default. With this, the compute node will use the Ollama server on your machine, instead of a Docker container.

If the Ollama server is not running, the start script will initiate it with ollama serve and terminate it when the node is being stopped.

If --local-ollama=false or the local Ollama server is reachable, the compute node will use a Docker Compose service for it.

[!TIP]

There are three Docker Compose Ollama options: ollama-cpu, ollama-cuda, and ollama-rocm. The start script will decide which option to use based on the host machine's GPU specifications.

# Run with local ollama
./start.sh -m=phi3 --local-ollama=true

Additional Static Nodes

You can add additional relay nodes & bootstrap nodes from environment, using the DKN_RELAY_NODES and DKN_BOOTSTRAP_NODES variables respectively. Simply write the Multiaddr string of the static nodes as comma-separated values, and the compute node will pick them up at the start.

Releases

We have 3 types of releases:

Versioned: With each release, a versioned image is deployed on Docker hub with the version tag :vX.X.X.
Latest: As usual, the latest version is kept under :latest tag.
Development: On each push to master, a new image is created with :unstable.

See deployed images on Docker Hub.

Development

If you have a feature that you would like to add with respect to its respective issue, or a bug fix, feel free to fork & create a PR!

If you would like to run the node from source (which is really handy during development), you can use our shorthand scripts within the Makefile. You can see the available commands with:

make help

You will need OpenSSL installed as well, see shorthand commands here. While running Ollama elsewhere (if you are using it) or with an OpenAI API key provided, you can run the compute node with:

make run      # info-level logs
make debug    # debug-level logs

Testing & Benchmarking

You can the tests as follows:

make test         # unit tests
make test-ollama  # Ollama tests (requires a running Ollama client)

To measure the speed of some Ollama models we have a benchmark that uses some models for a few prompts:

cargo run --release --example ollama

You can also benchmark these models using a larger task list at a given path, with the following command:

JSON_PATH="./path/to/your.json" cargo run --release --example ollama

Documentation

Open crate docs using:

make docs

Styling

Lint and format with:

make lint   # clippy
make format # rustfmt

Profiling

To create a flamegraph of the application, do:

make profile

This will create a profiling build that inherits release mode, except with debug information.

[!NOTE]

Profiling requires superuser access.

License

This project is licensed under the Apache License 2.0.

bazel-contrib/rules_nodejs

2024-08-21T01:34:15Z

NodeJS toolchain for Bazel.

brave/leo

2024-08-21T01:34:19Z

Design tokens for the Brave's design system known as Leo

pointfreeco/swift-navigation

2024-08-21T01:34:22Z

Bringing simple and powerful navigation tools to all Swift platforms, inspired by SwiftUI.

Swift Navigation

Bringing simple and powerful navigation tools to all Swift platforms, inspired by SwiftUI.

Overview

This library contains a suite of tools that form the foundation for building powerful state management and navigation APIs for Apple platforms, such as SwiftUI, UIKit, and AppKit, as well as for non-Apple platforms, such as Windows, Linux, Wasm, and more.

The SwiftNavigation library forms the foundation that more advanced tools can be built upon, such as the UIKitNavigation and SwiftUINavigation libraries. There are two primary tools provided:

observe: Minimally observe changes in a model.
UIBinding: Two-way binding for connecting navigation and UI components to an observable model.

In addition to these tools there are some supplementary concepts that allow you to build more powerful tools, such as UITransaction, which associates animations and other data with state changes, and UINavigationPath, which is a type-erased stack of data that helps in describing stack-based navigation.

All of these tools form the foundation for how one can build more powerful and robust tools for SwiftUI, UIKit, AppKit, and even non-Apple platforms.

SwiftUI

[!IMPORTANT] To get access to the tools described below you must depend on the SwiftNavigation package and import the SwiftUINavigation library.

SwiftUI already comes with incredibly powerful navigation APIs, but there are a few areas lacking that can be filled. In particular, driving navigation from enum state so that you can have compile-time guarantees that only one destination can be active at a time.

For example, suppose you have a feature that can present a sheet for creating an item, drill-down to a view for editing an item, and can present an alert for confirming to delete an item. One can technically model this with 3 separate optionals:

@Observable
class FeatureModel {
  var addItem: AddItemModel?
  var deleteItemAlertIsPresented: Bool
  var editItem: EditItemModel?
}

And then in the view one can use the sheet, navigationDestination and alert view modifiers to describe the type of navigation:

.sheet(item: $model.addItem) { addItemModel in
  AddItemView(model: addItemModel)
}
.alert("Delete?", isPresented: $model.deleteItemAlertIsPresented) {
  Button("Yes", role: .destructive) { /* ... */ }
  Button("No", role: .cancel) {}
}
.navigationDestination(item: $model.editItem) { editItemModel in
  EditItemModel(model: editItemModel)
}

This works great at first, but also introduces a lot of unnecessary complexity into your feature. These 3 optionals means that there are technically 8 different states: All can be nil, one can be non-nil, two could be non-nil, or all three could be non-nil. But only 4 of those states are valid: either all are nil or exactly one is non-nil.

By allowing these 4 other invalid states we can accidentally tell SwiftUI to both present a sheet and alert at the same time, but that is not a valid thing to do in SwiftUI, and SwiftUI will even print a message to the console letting you know that in the future it may actually crash your app.

Luckily Swift comes with the perfect tool for dealing with this kind of situation: enums! They allow you to concisely define a type that can be one of many cases. So, we can refactor our 3 optionals as an enum with 3 cases, and then hold onto a single piece of optional state:

@Observable
class FeatureModel {
  var destination: Destination?

  enum Destination {
    case addItem(AddItemModel)
    case deleteItemAlert
    case editItem(EditItemModel)
  }
}

This is more concise, and we get compile-time verification that at most one destination can be active at a time. However, SwiftUI does not come with the tools to drive navigation from this model. This is where the SwiftUINavigation tools becomes useful.

We start by annotating the Destination enum with the @CasePathable macro, which allows one to refer to the cases of an enum with dot-syntax just like one does with structs and properties:

+@CasePathable
 enum Destination {
   // ...
 }

And now one can use simple dot-chaining syntax to derive a binding from a particular case of the destination property:

.sheet(item: $model.destination.addItem) { addItemModel in
  AddItemView(model: addItemModel)
}
.alert("Delete?", isPresented: Binding($model.destination.deleteItemAlert)) {
  Button("Yes", role: .destructive) { /* ... */ }
  Button("No", role: .cancel) {}
}
.navigationDestination(item: $model.destination.editItem) { editItemModel in
  EditItemView(model: editItemModel)
}

[!NOTE] For the alert we are using the special Binding initializer that turns a Binding<Void?> into a Binding<Bool>.

We now have a concise way of describing all of the destinations a feature can navigate to, and we can still use SwiftUI's navigation APIs.

UIKit

[!IMPORTANT] To get access to the tools described below you must depend on the SwiftNavigation package and import the UIKitNavigation library.

Unlike SwiftUI, UIKit does not come with state-driven navigation tools. Its navigation tools are "fire-and-forget", meaning you simply invoke a method to trigger a navigation, but there is no representation of that in your feature's state.

For example, to present a sheet from a button press one can simply do:

let button = UIButton(type: .system, primaryAction: UIAction { [weak self] _ in
  present(SettingsViewController(), animated: true)
})

This makes it easy to get started with navigation, but as SwiftUI has taught us, it is incredibly powerful to be able to drive navigation from state. It allows you to encapsulate more of your feature's logic in an isolated and testable domain, and it unlocks deep linking for free since one just needs to construct a piece of state that represents where you want to navigate to, hand it to SwiftUI, and let SwiftUI handle the rest.

The UIKitNavigation library brings a powerful suite of navigation tools to UIKit that are heavily inspired by SwiftUI. For example, if you have a feature model like the one discussed above in the SwiftUI section:

@Observable
class FeatureModel {
  var destination: Destination?

  enum Destination {
    case addItem(AddItemModel)
    case deleteItemAlert
    case editItem(EditItemModel)
  }
}

…then one can drive navigation in a view controller using tools in the library:

class FeatureViewController: UIViewController {
  @UIBindable var model: FeatureModel

  func viewDidLoad() {
    super.viewDidLoad()

    // Set up view hierarchy

    present(item: $model.destination.addItem) { addItemModel in
      AddItemViewController(model: addItemModel)
    }
    present(isPresented: Binding($model.destination.deleteItemAlert)) {
      let alert = UIAlertController(title: "Delete?", message: message, preferredStyle: .alert)
      alert.addAction(UIAlertAction(title: "Yes", style: .destructive))
      alert.addAction(UIAlertAction(title: "No", style: .cancel))
      return alert
    }
    navigationDestination(item: $model.destination.editItem) { editItemModel in
      EditItemViewController(model: editItemModel)
    }
  }
}

By using the libraries navigation tools we can be guaranteed that the model will be kept in sync with the view. When the state becomes non-nil the corresponding form of navigation will be triggered, and when the presented view is dismissed, the state will be nil'd out.

Another powerful aspect of SwiftUI is its ability to update its UI whenever state in an observable model changes. And thanks to Swift's observation tools this can be done done implicitly and minimally: whichever fields are accessed in the body of the view are automatically tracked so that when they change the view updates.

Our UIKitNavigation library comes with a tool that brings this power to UIKit, and it's called observe:

observe { [weak self] in
  guard let self else { return }
  
  countLabel.text = "Count: \(model.count)"
  factLabel.isHidden = model.fact == nil 
  if let fact = model.fact {
    factLabel.text = fact
  }
  activityIndicator.isHidden = !model.isLoadingFact
}

Whichever fields are accessed inside observe (such as count, fact and isLoadingFact above) are automatically tracked, so that whenever they are mutated the trailing closure of observe will be invoked again, allowing us to update the UI with the freshest data.

All of these tools are built on top of Swift's powerful Observation framework. However, that framework only works on newer versions of Apple's platforms: iOS 17+, macOS 14+, tvOS 17+ and watchOS 10+. However, thanks to our back-port of Swift's observation tools (see Perception), you can make use of our tools right away, going all the way back to the iOS 13 era of platforms.

Non-Apple platforms

The tools provided by this library can also form the foundation of building navigation tools for non-Apple platforms, such as Windows, Linux, Wasm and more. We do not currently provide any such tools at this moment, but it is possible for them to be built externally.

For example, in Wasm it is possible to use the observe(isolation:_:)-93yzu function to observe changes to a model and update the DOM:

import JavaScriptKit

var countLabel = document.createElement("span")
_ = document.body.appendChild(countLabel)

let token = observe {
  countLabel.innerText = .string("Count: \(model.count)")
}

And it's possible to drive navigation from state, such as an alert:

alert(isPresented: $model.isShowingErrorAlert) {
  "Something went wrong"
}

And you can build more advanced tools for presenting and dismissing <dialog>'s in the browser.

Examples

This repo comes with lots of examples to demonstrate how to solve common and complex navigation problems with the library. Check out this directory to see them all, including:

Case Studies: A collection of SwiftUI and UIKit case studies demonstrating this library's APIs.
Inventory: A multi-screen application with lists, sheets, popovers and alerts, all driven by state and deep-linkable.

Learn More

Swift Navigation's tools were motivated and designed over the course of many episodes on Point-Free, a video series exploring functional programming and the Swift language, hosted by Brandon Williams and Stephen Celis.

You can watch all of the episodes here.

Community

If you want to discuss this library or have a question about how to use it to solve a particular problem, there are a number of places you can discuss with fellow Point-Free enthusiasts:

For long-form discussions, we recommend the discussions tab of this repo.
For casual chat, we recommend the Point-Free Community slack.

Installation

You can add Swift Navigation to an Xcode project by adding it as a package dependency.

https://github.com/pointfreeco/swift-navigation

If you want to use Swift Navigation in a SwiftPM project, it's as simple as adding it to a dependencies clause in your Package.swift:

dependencies: [
  .package(url: "https://github.com/pointfreeco/swift-navigation", from: "2.0.0")
]

Documentation

The latest documentation for the Swift Navigation APIs is available here.

License

This library is released under the MIT license. See LICENSE for details.

mattpocock/total-typescript-monorepo

2024-08-22T01:34:48Z

The home of all Matt's internal tooling

Name	Type	Notes	Accepted Values
userId	`string`	UserId of current user
name	`string`	Monitor name
description	`string`	Monitor Description
type	`string`	Valid email address	`"ping"`\|`"http"`\|`pagespeed`
url	`string`	URL of service or IP
isActive	`boolean`
interval	`number`	In ms

Name	Type	Notes
monitorId	`string`	Monitor associated with Check
status	`boolean`	`true` for up and `false` for down
responseTime	`number`	How long it took the server to respond
statusCode	`number`	HTTP Status code of response
message	`string`

Name	Type	Notes
checkId	`string`	Id of `Check` associated with `Alert`
monitorId	`string`	Id of `Monitor` associated with `Alert`
userId	`string`	Id of `User` associated with `Alert`
status	`boolean`	Status of `Alert`
message	`string`	`Alert` message
notifiedStatus	`boolean`
acknowledgeStatus	`boolean`