GitHub 1C Enterprise Daily Trending

microsoft/promptflow

2023-10-10T01:23:53Z

👨‍💼 Headshot AI - Professional Headshots with AI

Introducing Headshot AI, an open-source project from Leap AI that generates Professional AI Headshots in minutes.

How It Works

Live demo here.

The app is powered by:

🚀 Leap AI for AI model training
🚀 Leap AI to generate headshots
▲ Next.js for app and landing page
🔋 Supabase for DB & Auth
📩 Resend (optional) to email user when headshots are ready
⭐️ Shadcn with Tailwind CSS for styles
▲ Vercel for deployments
💳 Stripe for billing

Just clone, configure, deploy and you have a Headshot AI SaaS in a box.

Running Locally

To create your own Headshot AI app, follow these steps:

To setup Supabase/Vercel and your github repo, click on the Vercel Deploy Button and follow the steps.

IMPORTANT: In the Supabase integration step: Make sure you leave the Create sample tables option checked. This might take a few minutes to complete.

The Vercel Deployment will create a new repository with this template on your GitHub account and guide your through a new Supabase project creation. The Supabase Vercel Deploy Integration will set up the necessary Supabase environment variables and run the SQL migrations to set up the Database schema on your account. You can inspect the created tables in your project's Table editor.

This will create the tables with their respective columns and RLS policies:

credits
images
models
samples

Clone your newly created repo:

git clone {{your-repo-name}}

Enter your newly created repo's directory:

cd {{your-repo-name}}

Install dependencies:

For npm:
```
npm install
```
For yarn:
```
yarn
```
Magic Link Auth (Supabase)

In your supabase dashboard, make sure to update the email template for magic link correctly. You can use the following template:

Make sure to setup your site URL and redirect urls in the supabase dashboard under Authentication -> URL Configuration. For example: Site URL: https://headshots-starter.vercel.app Redirect URL: https://headshots-starter.vercel.app/**

<h2>Magic Link</h2>
<p>Follow this link to login:</p>
<p><a href="{{ .SiteURL }}/auth/confirm?token_hash={{ .TokenHash }}&type=email">Log In</a></p>

Create a Leap AI account

In your .env.local file:
- Fill in your_api_key with your Leap API key
- Fill in your-hosted-url/leap/train-webhook with https://{your-hosted-url}/leap/train-webhook
- Fill in your-hosted-url/leap/image-webhook with https://{your-hosted-url}/leap/image-webhook
- Fill in your-webhook-secret with any arbitrary URL friendly string eg.shadf892yr398hq23h
Create a Resend account (Optional)
- Fill in your-resend-api-key with your Resend API Key if you wish to use Resend to email users when their model has finished training.
Configure Stripe to bill users on a credit basis. (Optional)

The current setup is for a credit based system. 1 credit = 1 model train.

To enable Stripe billing, you will need to fill out the following fields in your .env.local file:
- STRIPE_SECRET_KEY=your-stripe-secret-key
- STRIPE_WEBHOOK_SECRET=your-stripe-webhook-secret
- STRIPE_PRICE_ID_ONE_CREDIT=your-stripe-price-id-one-credit
- STRIPE_PRICE_ID_THREE_CREDITS=your-stripe-price-id-three-credit
- STRIPE_PRICE_ID_FIVE_CREDITS=your-stripe-price-id-five-credit
- NEXT_PUBLIC_STRIPE_IS_ENABLED=false # set to true to enable Stripe payments
You need to do multiple things to get Stripe working:
- Get your Stripe API secret key from the Stripe Dashboard
- Create a Stripe Webhook that will point to your hosted URL. The webhook should be listening for the checkout.session.completed event. The webhook should point to your-hosted-url/stripe/subscription-webhook.
- Create a Stripe Price for each credit package you want to offer.
- Create a Stripe Pricing Table and replace the script @/components/stripe/StripeTable.tsx with your own values. It should look like this:
```
<stripe-pricing-table
  pricing-table-id="your-stripe-pricing-table-id"
  publishable-key="your-stripe-publishable-key"
  client-reference-id={user.id}
  customer-email={user.email}
></stripe-pricing-table>
```
Here are the products you need to create to get Stripe working with our example, checkout the images Here

To create them go on the Stripe dashboard, search for Product Catalog and then click on the add product button on the top right of the screen. You will need to create 3 products, one for each credit package as shown in the images before. We set them to One time payments, but you can change that if you want to and you can set the price too. After creating the products make sure to update the variables in the .env.local [your-stripe-price-id-one-credit, your-stripe-price-id-three-credit, your-stripe-price-id-five-credit] with their respective price ids, each price id is found in the product page at the bottom.
Start the development server:

For npm:
```
npm run dev
```
For yarn:
```
yarn dev
```
Visit http://localhost:3000 in your browser to see the running app.

One-Click Deploy

Default deploy using Vercel:

Deployment also supported on Replit.

How To Get Good Results

The image samples used to teach the model what your face looks like are critical. Garbage in = garbage out.

Enforce close-ups of faces and consider cropping so that the face is centered.
Enforce images with only one person in the frame.
Avoid accessories in samples like sunglasses and hats.
Ensure the face is clearly visible. (For face detection, consider using tools like Cloudinary API).

If you get distorted results with multiple faces, repeated subjects, multiple limbs, etc, make sure to follow these steps and minimize the chance of this happening:

Make sure any samples uploaded are the same 1:1 height / width aspect ratio, for example 512x512, 1024x1024, etc.
Avoid multiple people in the samples uploaded.
Add "double torso, totem pole" to the negative prompt when generating.
Make sure your dimensions when generating are also 1:1 with the same height / width ratios of the samples.

For more information on how to improve quality, read the blog here.

Additional Use-Cases

Headshot AI can be easily adapted to support many other use-cases on Leap AI including:

AI Avatars

Pet Portraits

Product Shots
Food Photography

Icons
Style-Consistent Assets

& more!

Contributing

We welcome collaboration and appreciate your contribution to Headshot AI. If you have suggestions for improvement or significant changes in mind, feel free to open an issue!

If you want to contribute to the codebase make sure you create a new branch and open a pull request that points to dev.

Resources and Support

Discord Community: Leap Discord
Help Email: help@tryleap.ai

License

Headshot AI is released under the MIT License.

microsoft/ToRA

2023-10-10T01:23:53Z

ToRA is a series of Tool-integrated Reasoning LLM Agents designed to solve challenging mathematical reasoning problems by interacting with tools.

ToRA: A Tool-Integrated Reasoning Agent

[🌐 Website] • [📜 Paper] • [🤗 HF Models] • [🐱 GitHub]
[🐦 Twitter] • [💬 Reddit] • [🍀 Unofficial Blog]

Repo for "ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving"

Figure 1: Comparing ToRA with baselines on LLaMA-2 base models from 7B to 70B.

🔥 News

[2023/10/08] 🔥🔥🔥 All ToRA models released at 🤗 HuggingFace!!!
[2023/09/29] ToRA paper, repo, and website released.

💡 Introduction

ToRA is a series of Tool-integrated Reasoning Agents designed to solve challenging mathematical reasoning problems by interacting with tools, e.g., computation libraries and symbolic solvers. ToRA series seamlessly integrate natural language reasoning with the utilization of external tools, thereby amalgamating the analytical prowess of language and the computational efficiency of external tools.

Model	Size	GSM8k	MATH	AVG@10 math tasks^†
GPT-4	-	92.0	42.5	78.3
GPT-4 (PAL)	-	94.2	51.8	86.4
ToRA-7B	7B	68.8	40.1	62.4
ToRA-Code-7B	7B	72.6	44.6	66.5
ToRA-13B	13B	72.7	43.0	65.9
ToRA-Code-13B	13B	75.8	48.1	71.3
ToRA-Code-34B^*	34B	80.7	51.0	74.8
ToRA-70B	70B	84.3	49.7	76.9

^*ToRA-Code-34B is currently the first and only open-source model to achieve over 50% accuracy (pass@1) on the MATH dataset, which significantly outperforms GPT-4’s CoT result (51.0 vs. 42.5), and is competitive with GPT-4 solving problems with programs. By open-sourcing our codes and models, we hope more breakthroughs will come!
^†10 math tasks include GSM8k, MATH, GSM-Hard, SVAMP, TabMWP, ASDiv, SingleEQ, SingleOP, AddSub, and MultiArith.

Tool-Integrated Reasoning

Figure 2: A basic example of single-round tool interaction, which interleaves rationales with program-based tool use.

ToRA Training Pipeline

Figure 3: Training ToRA contains ① Imitation Learning, and ② output space shaping.

🚀 Quick Start

⚙️ Setup

We recommend using Conda to manage your environment. We use vLLM (0.1.4) to accelerate inference. Run the following commands to setup your environment:

git clone https://github.com/microsoft/ToRA.git && cd ToRA/src
conda create -n tora python=3.10
conda activate tora
pip install torch==2.0.1 --index-url https://download.pytorch.org/whl/cu118 # CUDA 11.8 for example
pip install -r requirements.txt

🪁 Inference

We provide a script for inference, simply config the MODEL_NAME_OR_PATH and DATA in src/scripts/infer.sh and run the following command:

bash scritps/infer.sh

We also open-source the model outputs from our best models (ToRA-Code-34B and ToRA-70B) in the src/outputs/ folder.

⚖️ Evaluation

The src/eval/grader.py file contains the grading logic that assesses the accuracy of the predicted answer by comparing it to the ground truth. This logic is developed based on the Hendrycks' MATH grading system, which we have manually verified on the MATH dataset to minimize false positives and false negatives.

To evaluate the predicted answer, run the following command:

python -m eval.evaluate \
    --data_name "math" \
    --prompt_type "tora" \
    --file_path "outputs/llm-agents/tora-code-34b-v1.0/math/test_tora_-1_seed0_t0.0_s0_e5000.jsonl" \
    --execute

then you will get:

Num samples: 5000
Num scores: 5000
Timeout samples: 0
Empty samples: 2
Mean score: [51.0]
Type scores: {'Algebra': 67.3, 'Counting & Probability': 42.2, 'Geometry': 26.1, 'Intermediate Algebra': 40.0, 'Number Theory': 59.3, 'Prealgebra': 63.8, 'Precalculus': 34.2}

⚡️ Training

Due to some restrictions, ToRA-Corpus 16k is under review and cannot be released immediately. However, we open-source our complete training scripts as well as output space shaping pipelines for the community, and you may construct your own dataset for training.

To train a model, run the following command:

bash scritps/train.sh codellama 7b

☕️ Citation

If you find this repository helpful, please consider citing our paper:

@misc{gou2023tora,
      title={ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving}, 
      author={Zhibin Gou and Zhihong Shao and Yeyun Gong and yelong shen and Yujiu Yang and Minlie Huang and Nan Duan and Weizhu Chen},
      year={2023},
      eprint={2309.17452},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

🍀 Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

🌟 Star History

chatchat-space/Langchain-Chatchat

2023-10-10T01:23:53Z

Langchain-Chatchat（原Langchain-ChatGLM）基于 Langchain 与 ChatGLM 等语言模型的本地知识库问答 | Langchain-Chatchat (formerly langchain-ChatGLM), local knowledge based LLM (like ChatGLM) QA app with langchain

🌍 READ THIS IN ENGLISH

📃 LangChain-Chatchat (原 Langchain-ChatGLM): 基于 Langchain 与 ChatGLM 等大语言模型的本地知识库问答应用实现。

介绍

🤖️ 一种利用 langchain 思想实现的基于本地知识库的问答应用，目标期望建立一套对中文场景与开源模型支持友好、可离线运行的知识库问答解决方案。

💡 受 GanymedeNil 的项目 document.ai 和 AlexZhangji 创建的 ChatGLM-6B Pull Request 启发，建立了全流程可使用开源模型实现的本地知识库问答应用。本项目的最新版本中通过使用 FastChat 接入 Vicuna, Alpaca, LLaMA, Koala, RWKV 等模型，依托于 langchain 框架支持通过基于 FastAPI 提供的 API 调用服务，或使用基于 Streamlit 的 WebUI 进行操作。

✅ 依托于本项目支持的开源 LLM 与 Embedding 模型，本项目可实现全部使用开源模型离线私有部署。与此同时，本项目也支持 OpenAI GPT API 的调用，并将在后续持续扩充对各类模型及模型 API 的接入。

⛓️ 本项目实现原理如下图所示，过程包括加载文件 -> 读取文本 -> 文本分割 -> 文本向量化 -> 问句向量化 -> 在文本向量中匹配出与问句向量最相似的 top k个 -> 匹配出的文本作为上下文和问题一起添加到 prompt中 -> 提交给 LLM生成回答。

📺 原理介绍视频

从文档处理角度来看，实现流程如下：

🚩 本项目未涉及微调、训练过程，但可利用微调或训练对本项目效果进行优化。

🌐 AutoDL 镜像中 v9 版本所使用代码已更新至本项目 v0.2.5 版本。 🐳 Docker 镜像

💻 一行命令运行 Docker 🌲：

docker run -d --gpus all -p 80:8501 registry.cn-beijing.aliyuncs.com/chatchat/chatchat:0.2.5

环境最低要求

想顺利运行本代码，请按照以下的最低要求进行配置：

Python版本: >= 3.8.5, < 3.11
Cuda版本: >= 11.7
强烈推荐使用Python3.10，部分Agent功能可能没有完全支持Python3.10以下版本。

如果想要顺利在GPU运行本地模型(int4版本)，你至少需要以下的硬件配置:

chatglm2-6b & LLaMA-7B 最低显存要求: 7GB 推荐显卡: RTX 3060, RTX 2060
LLaMA-13B 最低显存要求: 11GB 推荐显卡: RTX 2060 12GB, RTX3060 12GB, RTX3080, RTXA2000
Qwen-14B-Chat 最低显存要求: 13GB 推荐显卡: RTX 3090
LLaMA-30B 最低显存要求: 22GB 推荐显卡：RTX A5000,RTX 3090,RTX 4090,RTX 6000,Tesla V100,RTX Tesla P40
LLaMA-65B 最低显存要求: 40GB 推荐显卡：A100,A40,A6000

如果是int8 则显存x1.5 fp16 x2.5的要求如：使用fp16 推理Qwen-7B-Chat 模型则需要使用16GB显存。

以上仅为估算，实际情况以nvidia-smi占用为准。

变更日志

参见版本更新日志。

从 0.1.x 升级过来的用户请注意，需要按照开发部署过程操作，将现有知识库迁移到新格式，具体见知识库初始化与迁移。

`0.2.0` 版本与 `0.1.x` 版本区别

使用 FastChat 提供开源 LLM 模型的 API，以 OpenAI API 接口形式接入，提升 LLM 模型加载效果；
使用 langchain 中已有 Chain 的实现，便于后续接入不同类型 Chain，并将对 Agent 接入开展测试；
使用 FastAPI 提供 API 服务，全部接口可在 FastAPI 自动生成的 docs 中开展测试，且所有对话接口支持通过参数设置流式或非流式输出；
使用 Streamlit 提供 WebUI 服务，可选是否基于 API 服务启动 WebUI，增加会话管理，可以自定义会话主题并切换，且后续可支持不同形式输出内容的显示；
项目中默认 LLM 模型改为 THUDM/chatglm2-6b，默认 Embedding 模型改为 moka-ai/m3e-base，文件加载方式与文段划分方式也有调整，后续将重新实现上下文扩充，并增加可选设置；
项目中扩充了对不同类型向量库的支持，除支持 FAISS 向量库外，还提供 Milvus, PGVector 向量库的接入；
项目中搜索引擎对话，除 Bing 搜索外，增加 DuckDuckGo 搜索选项，DuckDuckGo 搜索无需配置 API Key，在可访问国外服务环境下可直接使用。

模型支持

本项目中默认使用的 LLM 模型为 THUDM/chatglm2-6b，默认使用的 Embedding 模型为 moka-ai/m3e-base 为例。

LLM 模型支持

本项目最新版本中支持接入本地模型与在线 LLM API。

本地 LLM 模型接入基于 FastChat 实现，支持模型如下：

meta-llama/Llama-2-7b-chat-hf
Vicuna, Alpaca, LLaMA, Koala
BlinkDL/RWKV-4-Raven
camel-ai/CAMEL-13B-Combined-Data
databricks/dolly-v2-12b
FreedomIntelligence/phoenix-inst-chat-7b
h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b
lcw99/polyglot-ko-12.8b-chang-instruct-chat
lmsys/fastchat-t5-3b-v1.0
mosaicml/mpt-7b-chat
Neutralzz/BiLLa-7B-SFT
nomic-ai/gpt4all-13b-snoozy
NousResearch/Nous-Hermes-13b
openaccess-ai-collective/manticore-13b-chat-pyg
OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5
project-baize/baize-v2-7b
Salesforce/codet5p-6b
StabilityAI/stablelm-tuned-alpha-7b
THUDM/chatglm-6b
THUDM/chatglm2-6b
tiiuae/falcon-40b
timdettmers/guanaco-33b-merged
togethercomputer/RedPajama-INCITE-7B-Chat
WizardLM/WizardLM-13B-V1.0
WizardLM/WizardCoder-15B-V1.0
baichuan-inc/baichuan-7B
internlm/internlm-chat-7b
Qwen/Qwen-7B-Chat/Qwen-14B-Chat
HuggingFaceH4/starchat-beta
FlagAlpha/Llama2-Chinese-13b-Chat and others
BAAI/AquilaChat-7B
all models of OpenOrca
Spicyboros + airoboros 2.2
VMware's OpenLLaMa OpenInstruct
baichuan2-7b/baichuan2-13b
任何 EleutherAI 的 pythia 模型，如 pythia-6.9b
在以上模型基础上训练的任何 Peft 适配器。为了激活，模型路径中必须有 peft 。注意：如果加载多个peft模型，你可以通过在任何模型工作器中设置环境变量 PEFT_SHARE_BASE_WEIGHTS=true 来使它们共享基础模型的权重。

以上模型支持列表可能随 FastChat 更新而持续更新，可参考 FastChat 已支持模型列表。

除本地模型外，本项目也支持直接接入 OpenAI API、智谱AI等在线模型，具体设置可参考 configs/model_configs.py.example 中的 llm_model_dict 的配置信息。

在线 LLM 模型目前已支持：

项目中默认使用的 LLM 类型为 THUDM/chatglm2-6b，如需使用其他 LLM 类型，请在 [configs/model_config.py] 中对 llm_model_dict 和 LLM_MODEL 进行修改。

Embedding 模型支持

本项目支持调用 HuggingFace 中的 Embedding 模型，已支持的 Embedding 模型如下：

项目中默认使用的 Embedding 类型为 sensenova/piccolo-base-zh，如需使用其他 Embedding 类型，请在 [configs/model_config.py] 中对 embedding_model_dict 和 EMBEDDING_MODEL 进行修改。

Text Splitter 个性化支持

本项目支持调用 Langchain 的 Text Splitter 分词器以及基于此改进的自定义分词器，已支持的 Text Splitter 类型如下：

CharacterTextSplitter
LatexTextSplitter
MarkdownHeaderTextSplitter
MarkdownTextSplitter
NLTKTextSplitter
PythonCodeTextSplitter
RecursiveCharacterTextSplitter
SentenceTransformersTokenTextSplitter
SpacyTextSplitter

已经支持的定制分词器如下：

项目中默认使用的 Text Splitter 类型为 ChineseRecursiveTextSplitter，如需使用其他 Text Splitter 类型，请在 [configs/model_config.py] 中对 text_splitter_dict 和 TEXT_SPLITTER 进行修改。

关于如何使用自定义分词器和贡献自己的分词器，可以参考Text Splitter 贡献说明。

Agent生态

基础的Agent

在本版本中，我们实现了一个简单的基于OpenAI的React的Agent模型，目前，经过我们测试，仅有以下两个模型支持：

OpenAI GPT4
ChatGLM2-130B

目前版本的Agent仍然需要对提示词进行大量调试，调试位置

构建自己的Agent工具

详见自定义Agent说明

Docker 部署

🐳 Docker 镜像地址: registry.cn-beijing.aliyuncs.com/chatchat/chatchat:0.2.5)

docker run -d --gpus all -p 80:8501 registry.cn-beijing.aliyuncs.com/chatchat/chatchat:0.2.5

该版本镜像大小 35.3GB，使用 v0.2.5，以 nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04 为基础镜像
该版本内置两个 embedding 模型：m3e-large，text2vec-bge-large-chinese，默认启用后者，内置 chatglm2-6b-32k
该版本目标为方便一键部署使用，请确保您已经在Linux发行版上安装了NVIDIA驱动程序
请注意，您不需要在主机系统上安装CUDA工具包，但需要安装 NVIDIA Driver 以及 NVIDIA Container Toolkit，请参考安装指南
首次拉取和启动均需要一定时间，首次启动时请参照下图使用 docker logs -f <container id> 查看日志
如遇到启动过程卡在 Waiting.. 步骤，建议使用 docker exec -it <container id> bash 进入 /logs/ 目录查看对应阶段日志

开发部署

软件需求

本项目已在 Python 3.8.1 - 3.10，CUDA 11.7 环境下完成测试。已在 Windows、ARM 架构的 macOS、Linux 系统中完成测试。

1. 开发环境准备

参见开发环境准备。

请注意： 0.2.5 及更新版本的依赖包与 0.1.x 版本依赖包可能发生冲突，强烈建议新建环境后重新安装依赖包。

2. 下载模型至本地

如需在本地或离线环境下运行本项目，需要首先将项目所需的模型下载至本地，通常开源 LLM 与 Embedding 模型可以从 HuggingFace 下载。

以本项目中默认使用的 LLM 模型 THUDM/chatglm2-6b 与 Embedding 模型 moka-ai/m3e-base 为例：

下载模型需要先安装Git LFS，然后运行

$ git clone https://huggingface.co/THUDM/chatglm2-6b

$ git clone https://huggingface.co/moka-ai/m3e-base

3. 设置配置项

复制模型相关参数配置模板文件 configs/model_config.py.example 存储至项目路径下 ./configs 路径下，并重命名为 model_config.py。

复制服务相关参数配置模板文件 configs/server_config.py.example 存储至项目路径下 ./configs 路径下，并重命名为 server_config.py。

在开始执行 Web UI 或命令行交互前，请先检查 configs/model_config.py 和 configs/server_config.py 中的各项模型参数设计是否符合需求：

请确认已下载至本地的 LLM 模型本地存储路径写在 llm_model_dict 对应模型的 local_model_path 属性中，如:

"chatglm2-6b": "/Users/xxx/Downloads/chatglm2-6b",

请确认已下载至本地的 Embedding 模型本地存储路径写在 embedding_model_dict 对应模型位置，如：

"m3e-base": "/Users/xxx/Downloads/m3e-base",

请确认本地分词器路径是否已经填写，如：

text_splitter_dict = {
    "ChineseRecursiveTextSplitter": {
        "source": "huggingface",  ## 选择tiktoken则使用openai的方法,不填写则默认为字符长度切割方法。
        "tokenizer_name_or_path": "", ## 空格不填则默认使用大模型的分词器。 
    }
}

如果你选择使用OpenAI的Embedding模型，请将模型的 key写入 embedding_model_dict中。使用该模型，你需要能够访问OpenAI官的API，或设置代理。

4. 知识库初始化与迁移

当前项目的知识库信息存储在数据库中，在正式运行项目之前请先初始化数据库（我们强烈建议您在执行操作前备份您的知识文件）。

如果您是从 0.1.x 版本升级过来的用户，针对已建立的知识库，请确认知识库的向量库类型、Embedding 模型与 configs/model_config.py 中默认设置一致，如无变化只需以下命令将现有知识库信息添加到数据库即可：
```
$ python init_database.py
```
如果您是第一次运行本项目，知识库尚未建立，或者配置文件中的知识库类型、嵌入模型发生变化，或者之前的向量库没有开启 normalize_L2，需要以下命令初始化或重建知识库：
```
$ python init_database.py --recreate-vs
```

5. 一键启动 API 服务或 Web UI

5.1 启动命令

一键启动脚本 startup.py,一键启动所有 Fastchat 服务、API 服务、WebUI 服务，示例代码：

$ python startup.py -a

并可使用 Ctrl + C 直接关闭所有运行服务。如果一次结束不了，可以多按几次。

可选参数包括 -a (或--all-webui), --all-api, --llm-api, -c (或--controller), --openai-api, -m (或--model-worker), --api, --webui，其中：

--all-webui 为一键启动 WebUI 所有依赖服务；
--all-api 为一键启动 API 所有依赖服务；
--llm-api 为一键启动 Fastchat 所有依赖的 LLM 服务；
--openai-api 为仅启动 FastChat 的 controller 和 openai-api-server 服务；
其他为单独服务启动选项。

5.2 启动非默认模型

若想指定非默认模型，需要用 --model-name 选项，示例：

$ python startup.py --all-webui --model-name Qwen-7B-Chat

更多信息可通过 python startup.py -h查看。

5.3 多卡加载

项目支持多卡加载，需在 startup.py 中的 create_model_worker_app 函数中，修改如下三个参数:

gpus=None, 
num_gpus= 1, 
max_gpu_memory="20GiB"

其中，gpus 控制使用的显卡的ID，例如 "0,1";

num_gpus 控制使用的卡数;

max_gpu_memory 控制每个卡使用的显存容量。

注1：server_config.py的FSCHAT_MODEL_WORKERS字典中也增加了相关配置，如有需要也可通过修改FSCHAT_MODEL_WORKERS字典中对应参数实现多卡加载。

注2：少数情况下，gpus参数会不生效，此时需要通过设置环境变量CUDA_VISIBLE_DEVICES来指定torch可见的gpu,示例代码：

CUDA_VISIBLE_DEVICES=0,1 python startup.py -a

5.4 PEFT 加载(包括lora,p-tuning,prefix tuning, prompt tuning,ia3等)

本项目基于 FastChat 加载 LLM 服务，故需以 FastChat 加载 PEFT 路径，即保证路径名称里必须有 peft 这个词，配置文件的名字为 adapter_config.json，peft 路径下包含.bin 格式的 PEFT 权重，peft路径在startup.py中create_model_worker_app函数的args.model_names中指定，并开启环境变量PEFT_SHARE_BASE_WEIGHTS=true参数。

注：如果上述方式启动失败，则需要以标准的fastchat服务启动方式分步启动，分步启动步骤参考第六节，PEFT加载详细步骤参考加载lora微调后模型失效，

5.5 注意事项：

1. startup 脚本用多进程方式启动各模块的服务，可能会导致打印顺序问题，请等待全部服务发起后再调用，并根据默认或指定端口调用服务（默认 LLM API 服务端口：127.0.0.1:8888,默认 API 服务端口：127.0.0.1:7861,默认 WebUI 服务端口：本机IP：8501)

2.服务启动时间示设备不同而不同，约 3-10 分钟，如长时间没有启动请前往 ./logs目录下监控日志，定位问题。

3. 在Linux上使用ctrl+C退出可能会由于linux的多进程机制导致multiprocessing遗留孤儿进程，可通过shutdown_all.sh进行退出

5.6 启动界面示例：

FastAPI docs 界面

webui启动界面示例：

Web UI 对话界面：

Web UI 知识库管理页面：

常见问题

参见常见问题。

路线图

Langchain 应用
- 本地数据接入
  - 接入非结构化文档
    - .md
    - .txt
    - .docx
  - 结构化数据接入
    - .csv
    - .xlsx
  - 分词及召回
    - 接入不同类型 TextSplitter
    - 优化依据中文标点符号设计的 ChineseTextSplitter
    - 重新实现上下文拼接召回
  - 本地网页接入
  - SQL 接入
  - 知识图谱/图数据库接入
- 搜索引擎接入
  - Bing 搜索
  - DuckDuckGo 搜索
- Agent 实现
  - 基础React形式的Agent实现，包括调用计算器等
  - Langchain 自带的Agent实现和调用
  - 更多模型的Agent支持
  - 更多工具
LLM 模型接入
- 支持通过调用 FastChat api 调用 llm
- 支持 ChatGLM API 等 LLM API 的接入
Embedding 模型接入
- 支持调用 HuggingFace 中各开源 Emebdding 模型
- 支持 OpenAI Embedding API 等 Embedding API 的接入
基于 FastAPI 的 API 方式调用
Web UI
- 基于 Streamlit 的 Web UI

项目交流群

🎉 langchain-Chatchat 项目微信交流群，如果你也对本项目感兴趣，欢迎加入群聊参与讨论交流。

关注我们

🎉 langchain-Chatchat 项目官方公众号，欢迎扫码关注。

deaaprizal/unofficial-trust

2023-10-10T01:24:47Z

video: https://youtu.be/_pFe43G_jV4

segiqt/cs2-ahk-hack

2023-10-10T01:24:51Z

A working in progress internal cheat for CS2 written in ahk :p

xwxbug/autoit-cn

2023-10-11T01:24:39Z

AutoIt script software translate and enhancement.(EN2CHS)

shuhongfan/NavicatCracker

2023-10-10T01:25:01Z

navicat 激活 Navicat所有版本破解 A keygen for Navicat 16 Navicat16破解 Navicat Keygen Navicat 16 Keygen注册机(NavicatCracker) Navicat 16 破解 Navicat Premium 16 破解 Navicat破解绿色版 MySQL

XuanchenLin/NanUI

2023-10-10T01:25:35Z

NanUI is an open source .NET project for .NET developers who want to create desktop applications with HTML, CSS and JavaScript.

The WinFormium Project
(A.K.A. NanUI)

Easily buid powerful WinForm applications with HTML, CSS and JavaScript.

WinFormium

点击[此处]切换到简体中文仓库首页。

NanUI has always been in the testing stage. After 9 years, it is time to release the official v1.0 version of NanUI! When version 1.0 is released, the new name of the project will be officially launched - WinFormium

⭐ About

WinFormium is a open source framework on .NET platform for creating user interface of WinForm Applicaitons using HTML5, CSS3, and JavaScript. It is based on the Xilium.CefGlue project, which is a .NET wrapper around the Chromium Embedded Framework.

If you are looking for a framework for creating a WinForm application with a modern user interface, WinFormium is a good choice. you can use HTML, CSS, and JavaScript to create a user interface, and use C# to write the business logic of the application.

Please give WinFormium project a star⭐ if you like it.

If this project helps, please consider funding it.

🖥️ Requirements

For Development

.NET Framework 4.6.2 or higher / .NET 6.0 or higher
Visual Studio 2019 or higher (VS2022 is recommended)

For Deployment

Microsoft Windows 7 Service Pack 1 or higher
.Net Framework 4.6.2 or higher
.NET 6.0 for Windows 7 and higher.
.NET 7.0/8.0 for Windows 10 and higher.

This is a Windows Only framework, it can not run on Linux or Mac OS.

The minimum supported Windows is Windows 7 Service Pack 1, and some features (such as DirectComposition Offscreen Rendering) are not supported on Windows 7.

🧰 Getting Started

Create a simple WinFormium Application by following the steps below:

1. Create a WinForm Application by default template.

2. Install WinFormium NuGet Package

Open the NuGet Package Manager to install or use NuGet Package Manager Console, and run the following command to install WinFormium nuget package:

PM> Install-Package NetDimension.NanUI

Install the dependencies of Chromium Embedded Framework that WinFormium depends on:

PM> Install-Package NetDimension.NanUI.Runtime

3. A basic WinFormium application requires the following code:

Modify the code in the Program.cs file as follows:

using WinFormium;

class Program
{
    [STAThread]
    static void Main(string[] args)
    {
        var builder = WinFormiumApp.CreateBuilder();

        builder.UseWinFormiumApp<MyApp>();

        var app = builder.Build();

        app.Run();
    }
}

Create a class implements WinFormiumStartup for configuring the application:

using WinFormium;

class MyAPP : WinFormiumStartup
{
    protected override MainWindowCreationAction? UseMainWindow(MainWindowOptions opts)
    {
        // Configure the main window of this application
        return opts.UseMainFormium<MyWindow>();
    }

    protected override void WinFormiumMain(string[] args)
    {
        // The codes in Main function should be here, this function only runs in Main process. So it can prevent the codes in Main process running in sub-processes.
        ApplicationConfiguration.Initialize();
    }

    protected override void ConfigurationChromiumEmbedded(ChromiumEnvironmentBuiler cef)
    {
        // Configure the Chromium Embedded Framework here
    }

    protected override void ConfigureServices(IServiceCollection services)
    {
        // Configure the services of this application here
    }
}

Create a class implements Formium for configuring the main window of the application:

using WinFormium;
using WinFormium.Forms;

class MyWindow : Formium
{
    public MyWindow()
    {
        Url = "https://www.google.com";
    }

    protected override FormStyle ConfigureWindowStyle(WindowStyleBuilder builder)
    {
        // Configure the style of the window here or leave it blank to use the default style

        var style = builder.UseSystemForm();

        style.TitleBar = false;

        style.DefaultAppTitle = "My first WinFomrim app";

        return style;
    }
}

4. Build and run your WinFormium application

📖 Documentation

For more info please see - Documentation or Wiki

🤖 Demos

Minimal WinFormium App - Introduction to the basic usage of WinFormium.

🔗 Third-Party References & Tools

CEF - https://bitbucket.org/chromiumembedded/cef
Xilium.CefGlue - https://gitlab.com/xiliumhq/chromiumembedded/cefglue/
Vanara.Library - https://github.com/dahall/Vanara/
Vortice.Windows - https://github.com/amerkoleci/Vortice.Windows
SkiaSharp - https://github.com/mono/SkiaSharp
React - https://github.com/facebook/react
React-Router - https://github.com/remix-run/react-router
Vite - https://github.com/vitejs/vite

🏆 Inspirations

I was inspired by the following songs and albums when creating this version of WinFormium.

Strandels - Chance Of Rain
One Direction - What a Feeling (Made In The A.M.)
Thomas Rhett - VHS (Center Point Road)
Sammy Kershaw - She Don't Know She's Beautiful (Haunted Heart)
Chrissy Steele - Two Bodies (Magnet To Steele)
Halestorm - I Like It Heavy (Into the Wild Life)
Joan Jett & The Blackhearts - I Hate Myself for Loving You (Up Your Alley)

microsoft/typechat.net

2023-10-10T01:25:35Z

TypeChat.NET

TypeChat.NET is an experimental project from the Microsoft Semantic Kernel team. TypeChat.NET brings the ideas of TypeChat to .NET.

TypeChat.NET provides cross platform libraries that help you build natural language interfaces with language models using strong types, type validation and simple type safe programs (plans). Strong typing may help make software that uses language models more deterministic and reliable.

// Translates user intent into strongly typed Calendar Actions
var model = new LanguageModel(Config.LoadOpenAI());
var translator = new JsonTranslator<CalendarActions>(model);

// Translate natural language request 
CalendarActions actions = await translator.TranslateAsync(requestText);

TypeChat.NET is in active development with frequent updates. The framework will evolve as the team explores the space and incorporates feedback. Supported scenarios are shown in the included Examples. Documentation will also continue to improve. When in doubt, please look at the code.

Assemblies

TypeChat.NET currently consists of the following assemblies:

Microsoft.TypeChat: Classes that translate user intent into strongly typed and validated objects.
Microsoft.TypeChat.Program: Classes to synthesize, validate and run JSON programs.
Microsoft.TypeChat.SemanticKernel: Integration with Microsoft Semantic Kernel for language models, plugins and embeddings.

Microsoft.TypeChat

TypeChat uses language models to translate user intent into JSON that conforms to a schema. This JSON is then validated and deserialized into a typed object. Additional constraint checking is applied as needed. Validation errors are sent back to the language model, which uses them to repair the Json it originally returned.

TypeChat provides:

Json translation, validation, repair and deserialization.
Extensibility: interfaces for customizing schemas, validators and prompts.
Schema export: classes to generate schema for the .NET Type you want to translate to. Exported schema includes dependencies and base classes. The exported schema is specified using Typescript, which can concisely express schema for JSON objects.
- Support for common scenarios shown in TypeChat examples. When you encounter limitations (such as how generics are currently exported), you can supply schema text, such as Typescript authored by hand.
- Helper attributes for Vocabularies and Comments. Vocabularies are string tables that constrain the values that can be assigned to string properties. Dynamic loading of vocabularies enables scenarios where they vary at runtime.
- Note: Like TypeChat, TypeChat.NET has only been tested with schema specified in Typescript.

[Comment("Milks currently in stock")]
public class Milks
{
    [JsonVocab("whole milk | two percent milk | nonfat milk | soy milk | almond milk")]
    public string Name { get; set; }
}

Microsoft.TypeChat.Program

TypeChat.Program translates natural language requests into simple programs (Plans), represented as JSON.

JSON programs can be thought of as a DSL or Plan, expressed in JSON, with an associated grammar that is enforced. JSON programs can be type checked against the APIs they target. They can be then be run using an interpreter, or compiled into .NET code. Both mechanisms enforce type safety.

TypeChat.Program includes:

Program Translator: translates user intent into programs that follow the Program Grammar
Program Interpreter: runs programs generated by ProgramTranslator using an interpreter.
Program Compiler: uses the dynamic language runtime (DLR) to compile programs/plans with type checking. Compilation diagnostics are used to repair programs.

// Translates user intent into typed Programs that call methods on a Math API
var model = new LanguageModel(Config.LoadOpenAI());
var api = new MathAPI();
var translator = new ProgramTranslator<IMathAPI>(model, api);

// Translate natural language request
Program program = await translator.TranslateAsync(requestText);
// Run the program
program.Run(api);

Microsoft.TypeChat.SemanticKernel

TypeChat.SemanticKernel provides default bindings for language models, plugins and embeddings to Typechat.NET and TypeChat.NET examples.

TypeChat.SemanticKernel include classes for:

Json Programs for Plugins: turn registered plugins into APIs that Json programs can target. See the Plugins Example.
Language model and embeddings access: all TypeChat examples use the Semantic Kernel to call models and generate embeddings.

Getting Started

Prerequisite: OpenAI

OpenAI Language Models: TypeChat.NET and its examples currently require familiarity with and access to language models from OpenAI.
TypeChat.NET has been tested with and supports the following models:
- gpt-35-turbo
- gpt-4
- ada-002
Some examples and scenarios will work best with gpt-4
Since TypeChat.NET uses the Semantic Kernel, models from other providers may be used for experimentation.

Building

Visual Studio 2022.
- Load typechat.sln from the root directory.
- Restore packages
- Build
dotnet build
- Launch a command prompt / terminal
- Go to the root directory of the project
- dotnet build Typechat.sln

Nuget Packages

Microsoft.Typechat

dotnet add package Microsoft.TypeChat
dotnet add package Microsoft.TypeChat.SemanticKernel

Please ensure that you have installed both packages above.

Microsoft.TypeChat.Program

dotnet add package Microsoft.TypeChat.Program

Examples

To see TypeChat.NET in action, explore the Example projects and TypeChat.Examples Library.

Each example includes an input.txt with sample input. Pass the input file as an argument to run the example in batch mode.

The sections below describe which examples will best introduce which concept. Some examples or scenarios may work best with gpt-4.

Hello World

The Sentiment example is TypeChat's Hello World and a minimal introduction to JsonTranslator.

JsonTranslator

The following examples demonstrate how to use JsonTranslator, Schemas and Vocabularies:

CoffeeShop: Natural language ordering at a coffee shop. Demonstrates a complex schema with polymorphic deserialzation.
Calendar: Transform language into calendar actions
Restaurant: Order processing at a pizza restaurant

Hierarchical schemas

MultiSchema: dynamically route user intent to other 'sub-apps'
SchemaHierarchy: A Json Translator than uses multiple child JsonTranslators. For each user request, it picks the semantically nearest child translator and routes the input to it.
TextClassification and VectorTextIndex show how to build a simple classifiers to route input.

Json Programs

Math: How to turn user requests into simple calculator programs
Plugins: How to translate user intent into programs programs that call Semantic Kernel Plugins

Interactive agents

HealthData: how to use an interactive bot to collect a user's health information.
Agent Classes for working with interactive agents that have history. These classes emonstrate how TypeChat.NET may be used for strongly typed interactions with message passing agents or bots. These agents can include features such as built in interaction history.

Api Key and Configuration

To use TypeChat.net or run the examples, you need an API key for an OpenAI service. Azure OpenAI and the OpenAI service are both supported.

Configure Api Key for examples

Go to the examples folder in the solution
Make a copy of the appSettings.json file and name it appSettings.Development.json. Ensure it is in the same folder as appSettings.json
appSettings.Development.json is a local development only override of the settings in appSettings.json and is never checked in.
Add your Api Key to appSettings.Development.json.

A typical appSettings.Development.json will look like this:

// For Azure OpenAI service
{
  "OpenAI": {
    "Azure": true,
    "ApiKey": "YOUR API KEY",
    "Endpoint": "https://YOUR_RESOURCE_NAME.openai.azure.com",
    "Model": "gpt-35-turbo"  // Name of Azure deployment
  }
}

// For OpenAI Service:
{
  "OpenAI": {
    "Azure": false,
    "ApiKey": "YOUR API KEY",
    "Endpoint": "https://api.openai.com/v1/chat/completions",
    "Model": "gpt-3.5-turbo"  // Name of OpenAI model
  }
}

OpenAIConfig

TypeChat examples accesses language models using the LanguageModel class. The OpenAIConfig class supplies configuration for LanguageModel. You initialize OpenAIConfig from your application's configuration, from a Json file or from environment variables.

See OpenAIConfig.cs for a list of :

Configurable properties
Supported environment variables.

// Your configuration 
OpenAIConfig config = Config.LoadOpenAI();
// Or Json file
OpenAIConfig config = OpenAIConfig.LoadFromJsonFile(...);
// Or from config
config = OpenAIConfig.FromEnvironment();

var model = new LanguageModel(config);

Using Semantic Kernel directly

You can also initialize LanguageModel using an IKernel object you created using a KernelBuilder.

const string modelName = "gpt-35-turbo";
new LanguageModel(_kernel.GetService<IChatCompletion>(modelName), modelName);

Using your own client

TypeChat accesses language models using the ILanguageModel interface. LanguageModel implements ILanguageModel.

You can use your own model client by implementing ILanguageModel.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

License

Licensed under the MIT license.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

bepu/bepuphysics2

2023-10-10T01:25:35Z

Pure C# 3D real time physics simulation library, now with a higher version number.

bepuphysics v2

This is the repo for the bepuphysics v2 library, a complete rewrite of the C# 3d rigid body physics engine BEPUphysics v1.

The BepuPhysics and BepuUtilities libraries target .NET 6 and should work on any supported platform. The demos application, Demos.sln, uses DX11 by default. There is also a Demos.GL.sln that uses OpenGL and should run on other platforms. The demos can be run from the command line (in the repo root directory) with dotnet run --project Demos/Demos.csproj -c Release or dotnet run --project Demos.GL/Demos.csproj -c Release.

The physics engine heavily uses System.Numerics.Vectors types, so to get good performance, you'll need a compiler which can consume those types (like RyuJIT).

To build the source, you'll need a recent version of Visual Studio with the .NET desktop development workload installed. Demos.sln references all relevant projects. For more information, see Building.

Features

Spheres, capsules, boxes, triangles, cylinders, and convex hulls
Compounds of the above
Meshes
A whole bunch of constraint types
Newts
Linear and angular continuous collision detection
Extremely low cost sleep states for resting bodies
Efficient scene-wide ray and sweep queries
Character controller example
At least somewhat extensible collision pipeline, with example custom voxel collidable
Highly nonidiomatic APIs
Super speediness
And a bunch of other miscellaneous stuff!

Links

Report bugs on the issues tab.

Use the discussions tab for... discussions. And questions.

By user request, there's a discord server. I'll be focusing on github for long-form content, but if you like discord, now you can discord.

Continuous Collision Detection

Contributing

Change log

Upgrading from v1, concept mapping

Packaging and Versioning

Check the roadmap for a high level look at where things are going.

If you have too many dollars, we are willing to consume them through github sponsors.

defold/defold

2023-10-10T01:25:50Z

Defold is a completely free to use game engine for development of desktop, mobile and web games.

Defold

Repository for the Defold engine, editor and command line tools.

Supported by

Folder Structure

build_tools - Build configuration and build tools used by build scripts
ci - Continuous integration files for GitHub CI (more info)
com.dynamo.cr - Bob
engine - Engine
editor - Editor
packages - External packages
scripts - Build and utility scripts
share - Misc shared stuff used by other tools. Waf build-scripts, valgrind suppression files, etc.

Setup and Build

Setup Engine

Follow the setup guide to install all of the tools needed to build the Defold engine.

Build Engine

Follow the build instructions to build the engine and command line tools.

Setup, Build and Run Editor

Follow the instructions in the editor folder.

Engine Overview

An overview of the engine architecture and additional engine information can be viewed here.

Platform Specifics

Releasing a new version

The release process is documented here.

Complying with licenses

A full list of third party software licenses along with information on how to give attribution and include the licenses in your game can be found in the COMPLYING WITH LICENSES document in the Defold repository on GitHub.

reduz/larvita3

2023-10-11T01:25:21Z

Larvita 3 Game Engine

Larvita3

Larvita 3 Game Engine

This game engine is the precursor of Godot Engine developed by Ariel Manzur and Juan Linietsky. It was used for a couple of games and prototypes, most notably (Atmosphir)[https://www.youtube.com/watch?v=DvQsmPC27VU].

Compilation

I have no idea, please update instructions. I think it went something like:

scons bin/script_exec

Usage

The editor is written in Lua (this is a Lua game engine). You open the editor like this:

cd bin
script_exec editor

To execute games:

script_exec game_folder

Games are implemented by inheriting SceneMainLoop from Lua, then doing whathever you want.

Documentation

There is none.

Origins

Our previous engine was called "Larvotor" (used for Regnum Online). This was originally a 3D engine aimed to run on the Nintendo 3DS, hence the name "Larvita" (Larvita 1 was a prototype engine for that platform). Life happened and we ended up getting to work for clients wanting to make PC games.

An explanation of Larvita 3 design and how we moved from it to Godot can be read here.

OpenIPC/firmware

2023-10-11T01:25:21Z

Alternative IP Camera firmware from an open community

OpenIPC firmware

(based on Buildroot)

OpenIPC is a rapidly developing open source alternative firmware for popular IP cameras from an open community.

Historically, OpenIPC firmware only supported SoC manufactured by HiSilicon, but as the development continues, the list of supported processors expands. Today, it also includes chips from Ambarella, Anyka, Fullhan, Goke, GrainMedia, Ingenic, MStar, Novatek, SigmaStar, XiongMai, and is expected to grow further.

More information about the project is available in our website and on the wiki.

Support

OpenIPC offers two levels of support.

Free support through the community (via chat).
Paid commercial support (from the team of developers).

Please consider subscribing for paid commercial support if you intend to use our product for business. As a paid customer, you will get technical support and maintenance services directly from our skilled team. Your bug reports and feature requests will get prioritized attention and expedited solutions. It's a win-win strategy for both parties, that would contribute to the stability your business, and help core developers to work on the project full-time.

If you have any specific questions concerning our project, feel free to contact us.

Participating and Contribution

If you like what we do, and willing to intensify the development, please consider participating.

You can improve existing code and send us patches. You can add new features missing from our code.

You can help us to write a better documentation, proofread and correct our websites.

You can just donate some money to cover the cost of development and long-term maintaining of what we believe is going to be the most stable, flexible, and open IP Network Camera Framework for users like yourself.

You can make a financial contribution to the project at Open Collective.

Thank you.

linxinrao/Shamrock

2023-10-11T01:25:21Z

☘ 基于Xposed实现的Onebot11/12标准QQBot框架

下载 | 部署 | 接口 | 文档 | 加群

简介

☘ 基于 Xposed 实现 OneBot 标准的 QQ 机器人框架

本项目仅提供学习与交流用途，请在24小时内删除。
本项目目的是研究 Xposed 和 LSPosed 框架的使用。 Epic 框架开发相关知识。
如有违反法律，请联系删除。

兼容说明

本项目基于 go-cqhttp 的文档进行开发实现，未来将会支持 Docker 部署(现在也可以)。
在 qsign 死亡(报废)的情况下，Shamrock 支持作为 NTR 提供 sign API 服务。

权限声明

如出现未在此处声明的权限，请警惕 Shamrock 是否被修改/植入恶意代码

网络访问权限: Shamrock 进程需要使用 HTTP API 来进行一些操作
Hook 系统框架: 为了保证息屏状态下服务不被杀死，Shamrock 需要 Hook 系统框架
后台启动 Activity: 自动唤醒 QQ 时需要使用

语音解码

请参考文档中语音支持部分

开源协议

本项目使用 GPL-3.0 协议开放源代码

Shamrock - OneBot standard QQ robot framework based on Xposed implementation
Copyright (C) 2023 Shamrock Team
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

贡献者

Starklandxyz/contract

2023-10-10T01:26:14Z

vnovikov746/ShifterMan

2023-10-11T01:26:06Z

ShifterMan Project -The ultimate tool to organize shift schedule

ftravers/soap-helper

2023-10-11T01:26:13Z

sush2023/URC

2023-10-10T01:26:36Z

Project Pheonix Object detector node

ISC-Project-Phoenix/obj_tracker

2023-10-11T01:26:11Z

Project Phoenix Object Tracking Node

omicreativedev/hacktoberfest2023

2023-10-10T01:27:08Z

⭐ 200 Star Goal 👆🏽 My first time maintaining a repo for #hacktoberfest (YAY!) — Website with Theme Switcher. You can see the deployment below.

Due to something breaking— The code in this repository had a big update. Because so many forks are out of sync, some actions are failing. If you are working on a fork, please backup your work and resync. Since almost all open issues are in modules, you can you drop your themes/pages in the synced repo.

hacktoberfest2023 - ⭐ Goal 200 Stars!

📢 Announcement: You do NOT need to work on issues in this repo. You can also just FORK IT and start working on a page or theme without approval. Of course you are ALSO welcome to work on issues!

Read the Docs.
Give a ⭐
Don't break anything.

🥳👉🏽 Follow me on Github to stay notified of NEXT YEAR's #Hacktoberfest Repos!

About The Project

My first attempt at trying to maintain a repo for #hacktoberfest (YAY!) Your stars help this repo qualify for syndication libraries. Please star BEFORE you are merged! Please give me 24 hours to respond to issues or pull requests. 😎

This project is a simple website with a theme switcher and page switcher in Javascript. You can drop the theme switcher and/or page switcher into your project and it's highly customizable.

Flat file list of themes located in a theme folder so you can make multiple css files for your site!
Flat file list of pages located in a pages folder so you can make webpages, presentation slides, activities, etc.
Magically ✨🧙‍♂️✨ switch your fancy theme to an Accessible Color Scheme or Darkmode 🐱‍👤

🤩

☝ Although this project is intended to be beginner friendly 💗, it should not be easy 🚫. Because some people might be really advanced at C++ or Python but never made a webpage before, I'm trusting that people don't use this repository for an easy pull request. Tiny PRs won't be merged such as typos, spaces, spam without a significant contribution. Please stretch your limits 🤓 and challenge yourself 💪🏽.

For total beginners, I will be listing ongoing issues and enhancement ideas that you can partake in. If you have an idea or suggestion, feel free to raise an issue or start a Discussions. This site is very basic, and I made it after taking a few lessons on freeCodeCamp. 💖💖💖 Anyone can do it!

Stack

Contributors - Your Image Will Show Here! OPEN FIRST Pull Request 🎉

(updates every 7 days)

About Hacktoberfest 2023

Register here for Hacktoberfest
Get four pull requests (PRs) merged between October 1st-31st (Yes, you can do all 4 here!)
Earn DIGITAL-SWAG 🔥 from Digital Ocean ⚡

This Project Is Perfect For Your First Pull Request

Read the README.md
Look at how the Website works
Read about CONTRIBUTING.md
Read the CODE_OF_CONDUCT.md
Find or Raise an issue and wait for approval -OR- just get started making a page or theme.
Learn more about the fork workflow here 🏹
Fork the repository and make your changes
Create your PR!
Wait for feedback or approval (Tweet me or DM on X or Discord for faster response)
Get Merged 🎉🎈🥳
CLICK TO TWEET A RECOGNITION BADGE FOR CONTRIBUTING

Don't Forget!

⭐ STAR this repository!
👽 Follow me on Github
🐤 Follow me on ❌itter
👾 Friend me on DISCORD

COPYLEFT

This code / website is CC0 but the information on user created pages is personal and may be covered by additional rights such as namesake, tradedress, etc. As such, you are welcome to download this whole project but only the project is CC0, not the "personal data" within profile pages. Themes created are part of the opensource contribution but pages (such as the profiles in the /pages folder) must be WIPED of all personal identifying information (name, face, links) and any photographs which I don't have rights to grant. The LAYOUTS of those pages are opensource contributions and CC0. Anyone may adapt this project for their own repository to start a Hacktoberfest Repo of their own 💗. If you choose to do this, make a PULL REQUEST and list your repository (based on this one) below.

COOL PROJECTS TO VISIT BASED ON THIS ONE! 👇🏽👇🏽👇🏽

Your Project Link Here! ⬅⬅⬅

THANK YOU! 💗💗💗

The following repsitories are teaching me a lot about open source, Git, maintaining a repo for Hacktoberfest and more.

Git Into Open Source vy Lissy93
Hacktoberfest2023 by OssamaMehmood

DEPLOYMENT STATUS

This may take awhile after a large update...

lazyjinchuriki/pokedex

2023-10-10T01:27:08Z

For Course3 Jenkins Getting Started - fork of https://github.com/spring-projects/spring-petclinic

Spring PetClinic Sample Application

Understanding the Spring Petclinic application with a few diagrams

See the presentation here

Running petclinic locally

Petclinic is a Spring Boot application built using Maven or Gradle. You can build a jar file and run it from the command line (it should work just as well with Java 17 or newer):

git clone https://github.com/spring-projects/spring-petclinic.git
cd spring-petclinic
./mvnw package
java -jar target/*.jar

You can then access petclinic at http://localhost:8080/

Or you can run it from Maven directly using the Spring Boot Maven plugin. If you do this, it will pick up changes that you make in the project immediately (changes to Java source files require a compile as well - most people use an IDE for this):

./mvnw spring-boot:run

NOTE: If you prefer to use Gradle, you can build the app using ./gradlew build and look for the jar file in build/libs.

Building a Container

There is no Dockerfile in this project. You can build a container image (if you have a docker daemon) using the Spring Boot build plugin:

./mvnw spring-boot:build-image

In case you find a bug/suggested improvement for Spring Petclinic

Our issue tracker is available here

Database configuration

In its default configuration, Petclinic uses an in-memory database (H2) which gets populated at startup with data. The h2 console is exposed at http://localhost:8080/h2-console, and it is possible to inspect the content of the database using the jdbc:h2:mem:testdb url.

A similar setup is provided for MySQL and PostgreSQL if a persistent database configuration is needed. Note that whenever the database type changes, the app needs to run with a different profile: spring.profiles.active=mysql for MySQL or spring.profiles.active=postgres for PostgreSQL.

You can start MySQL or PostgreSQL locally with whatever installer works for your OS or use docker:

docker run -e MYSQL_USER=petclinic -e MYSQL_PASSWORD=petclinic -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=petclinic -p 3306:3306 mysql:8.0

docker run -e POSTGRES_USER=petclinic -e POSTGRES_PASSWORD=petclinic -e POSTGRES_DB=petclinic -p 5432:5432 postgres:15.2

Further documentation is provided for MySQL and for PostgreSQL.

Instead of vanilla docker you can also use the provided docker-compose.yml file to start the database containers. Each one has a profile just like the Spring profile:

$ docker-compose --profile mysql up

$ docker-compose --profile postgres up

Test Applications

At development time we recommend you use the test applications set up as main() methods in PetClinicIntegrationTests (using the default H2 database and also adding Spring Boot devtools), MySqlTestApplication and PostgresIntegrationTests. These are set up so that you can run the apps in your IDE and get fast feedback, and also run the same classes as integration tests against the respective database. The MySql integration tests use Testcontainers to start the database in a Docker container, and the Postgres tests use Docker Compose to do the same thing.

Compiling the CSS

There is a petclinic.css in src/main/resources/static/resources/css. It was generated from the petclinic.scss source, combined with the Bootstrap library. If you make changes to the scss, or upgrade Bootstrap, you will need to re-compile the CSS resources using the Maven profile "css", i.e. ./mvnw package -P css. There is no build profile for Gradle to compile the CSS.

Working with Petclinic in your IDE

Prerequisites

The following items should be installed in your system:

Java 17 or newer (full JDK, not a JRE).
git command line tool
Your preferred IDE
- Eclipse with the m2e plugin. Note: when m2e is available, there is an m2 icon in Help -> About dialog. If m2e is not there, follow the install process here
- Spring Tools Suite (STS)
- IntelliJ IDEA
- VS Code

Steps:

On the command line run:

git clone https://github.com/spring-projects/spring-petclinic.git

Inside Eclipse or STS:
```
File -> Import -> Maven -> Existing Maven project
```
Then either build on the command line ./mvnw generate-resources or use the Eclipse launcher (right click on project and Run As -> Maven install) to generate the css. Run the application main method by right-clicking on it and choosing Run As -> Java Application.
Inside IntelliJ IDEA In the main menu, choose File -> Open and select the Petclinic pom.xml. Click on the Open button.

CSS files are generated from the Maven build. You can build them on the command line ./mvnw generate-resources or right-click on the spring-petclinic project then Maven -> Generates sources and Update Folders.

A run configuration named PetClinicApplication should have been created for you if you're using a recent Ultimate version. Otherwise, run the application by right-clicking on the PetClinicApplication main class and choosing Run 'PetClinicApplication'.
Navigate to Petclinic

Visit http://localhost:8080 in your browser.

Looking for something in particular?

Spring Boot Configuration	Class or Java property files
The Main Class	PetClinicApplication
Properties Files	application.properties
Caching	CacheConfiguration

Interesting Spring Petclinic branches and forks

The Spring Petclinic "main" branch in the spring-projects GitHub org is the "canonical" implementation based on Spring Boot and Thymeleaf. There are quite a few forks in the GitHub org spring-petclinic. If you are interested in using a different technology stack to implement the Pet Clinic, please join the community there.

Interaction with other open source projects

One of the best parts about working on the Spring Petclinic application is that we have the opportunity to work in direct contact with many Open Source projects. We found bugs/suggested improvements on various topics such as Spring, Spring Data, Bean Validation and even Eclipse! In many cases, they've been fixed/implemented in just a few days. Here is a list of them:

Name	Issue
Spring JDBC: simplify usage of NamedParameterJdbcTemplate	SPR-10256 and SPR-10257
Bean Validation / Hibernate Validator: simplify Maven dependencies and backward compatibility	HV-790 and HV-792
Spring Data: provide more flexibility when working with JPQL queries	DATAJPA-292

Contributing

The issue tracker is the preferred channel for bug reports, features requests and submitting pull requests.

For pull requests, editor preferences are available in the editor config for easy use in common text editors. Read more and download plugins at https://editorconfig.org. If you have not previously done so, please fill out and submit the Contributor License Agreement.

License

The Spring PetClinic sample application is released under version 2.0 of the Apache License.

samipak458/Simple-Library-Managment

2023-10-11T01:26:46Z

A simple library management project that is easy to use and helps learners understand the basic concept of JS.

Simple Library Management

A simple library management project is an Open Source Website built using HTML, CSS, JavaScript and Bootstrap and is easy to use and helps learners understand the basic concept of JS. In this project, the user has a provision to add book details like book name, author name, book URL, ISBN, edition, publication date, the status of reading the book, and book genre through the web page. In addition, the user has a provision to search for the available books in the library by the book name, author name, or type. If book details are present in the LocalStorage, the search details are displayed on the web page. Users also have a provision to delete a specific book or all books. The user can choose his favorites and edit any book details very quickly.

In this project, you will learn how to implement add, remove/delete, search, and save functionalities. You will also learn how to deploy, or publish, your web site to Azure so that you can show it off to your friends.

Visit Website:

You can try the website at Online Library

Functionalities :

Input the book Name, Author, URL, Edition, ISBN, Publication Date, Status, and Book Type to add the book. If the book name is missing, it will show you an error. As I used LocalStorage, if you refresh or close the tab, your books will not disappear. You can clear the bookshelf just by clicking on clear shelf. You can also search books by book name, author, and type.

Adding

Adding Book in Shelf:

Adding Book Without Entering Book Name:

Clearing Book Shelf:

Searching

Search Book By Type:

Search Book By Author:

Search Book By Name:

Sample working

https://user-images.githubusercontent.com/53844369/195153604-9a5adf2d-67be-4fc2-82d7-f2307d517e00.mp4

Tech Stack

How to use :

Clone the repository
Go to the directory
Run index.html file
Enter book name, author and type of book
Press add book

Code Of Conduct 📜

To maintain a safe and inclusive space for everyone to learn and grow, contributors are advised to follow the Code of Conduct.

Contribution is fun! ✌🏼

If you have any feedback or suggestions please reach out to me.

In order to make a hassle-free environment, I implore you all (while contributing) to follow the instructions Contributing Guidelines!

You can create a issue and mention there , which new features or extension can make this Project more good.

Show some ❤️ by starring⭐ this awesome Repository!

OUR VALUABLE CONTRIBUTORS✨

AOSSIE-Org/Resonate

2023-10-10T01:27:34Z

A modern E-Commerce mobile application built with Flutter, TDD, Clean Architecture, and BLoC.

Flutter TDD Clean Architecture E-Commerce App - EShop

Welcome to the Flutter-TDD-Clean-Architecture-E-Commerce-App GitHub repository! This project is a showcase of modern mobile app development practices, leveraging the power of Flutter, Test-Driven Development (TDD), Clean Architecture, and the BLoC (Business Logic Component) package. Built using the latest version of Flutter 3, this E-Commerce application exemplifies best practices for building scalable, maintainable, and efficient Flutter apps.

Key Features:

Test-Driven Development (TDD): This project emphasizes the importance of writing tests before writing the actual code. It ensures that the application's logic is thoroughly tested, enhancing reliability and maintainability.
Clean Architecture: The app follows a clean and modular architecture that separates concerns into different layers: Presentation, Domain, and Data. This architecture promotes code reusability and makes it easier to adapt to changes in the future.
BLoC State Management: The app utilizes the BLoC pattern for state management. BLoC helps manage the flow of data and business logic in a clean and reactive manner, improving overall app performance.
E-Commerce Functionality: The app showcases a variety of E-Commerce features, such as product browsing, searching, cart and purchasing. Users can explore products, add them to their cart, and complete transactions seamlessly.

Feature	UseCases
Product	Get Product UseCase
Category	Get Cached Category UseCase Get Remote Category UseCase Filter Category UseCase
Cart	Get Cached Cart UseCase Get Remote Cart UseCase Add Cart Item UseCase Sync Cart UseCase
User	Get Cached User UseCase SignIn UseCase SignUp UseCase SignOut UseCase
Delivery Info	Get Cached Delivery Info UseCase Get Remote Delivery Info UseCase Add Delivery Info UseCase
Order	Get Orders UseCase Add Order UseCase

Demo Sample

Contributing:

We welcome contributions from the Flutter community to make this project even better. Whether you are interested in adding new features, fixing bugs, or improving documentation, your contributions are highly appreciated. Please refer to the contribution guidelines in the repository for more details on how to get involved.

Getting Started

To get started with this project, follow the instructions in the README to set up your development environment and run the app locally. You can also explore the project's architecture, tests, and documentation to gain insights into building robust Flutter apps.

We hope this Flutter-TDD-Clean-Architecture-E-Commerce-App serves as a valuable resource for both Flutter enthusiasts and developers looking to learn about TDD, clean architecture, and BLoC in the context of mobile app development. Happy coding!

Installation

Clone the repo

git clone https://github.com/Sameera-Perera/Flutter-TDD-Clean-Architecture-E-Commerce-App.git

Install packages
```
flutter pub get
```
Run app
```
flutter run lib/main.dart
```
Run test
```
flutter test
```

For help getting started with Flutter, view our online documentation.

License

Distributed under the MIT License. See LICENSE for more information.

JasmineRickards/Wasteland-After-Dark

2023-10-10T01:27:50Z

jfloff/alpine-python

2023-10-10T01:27:57Z

A small, more complete, Python Docker image based on Alpine Linux.

alpine-python

A small Python Docker image based on Alpine Linux.

alpine-python

NOTES:

onbuild images install the requirements.txt of your project from the get go. This allows you to cache your requirements right in the build. Make sure you are in the same directory of your requirements.txt file.

Why?

The default docker python images are too big, much larger than they need to be. Hence I built this simple image based on docker-alpine, that has everything needed for the most common python projects - including python-dev (which is not common in most minimal alpine python packages).

REPOSITORY	TAG	SIZE
jfloff/alpine-python	2.7-slim	60MB
python	2.7-slim	120MB
python	2.7-alpine	61.2MB

jfloff/alpine-python	2.7	235MB
python	2.7	912MB

jfloff/alpine-python	3.6-slim	76.3MB
python	3.6-slim	138MB
python	3.6-alpine	79MB

jfloff/alpine-python	3.6	252MB
python	3.6	922MB

jfloff/alpine-python	3.7-slim	80.4MB
python	3.7-slim	86.7MB
python	3.7-alpine	143MB

jfloff/alpine-python	3.7	256MB
python	3.7	927MB

Perhaps this could be even smaller, but I'm not an Alpine guru. Feel free to post a PR.

Details

Installs build-base and python-dev, allowing the use of more advanced packages such as gevent
Installs bash allowing interaction with the container
Just like the main python docker image, it creates useful symlinks that are expected to exist, e.g. python3 > python, pip2.7 > pip, etc.)
Added testing and community repositories to Alpine's /etc/apk/repositories file

Usage

This image runs python command on docker run. You can either specify your own command, e.g:

docker run --rm -ti jfloff/alpine-python python hello.py

You can also access bash inside the container:

docker run --rm -ti jfloff/alpine-python bash

Usage of `onbuild` images

These images can be used to bake your dependencies into an image by extending the plain python images. To do so, create a custom Dockerfile like this:

FROM jfloff/alpine-python:3.6-onbuild

# for a flask server
EXPOSE 5000
CMD python manage.py runserver

Don't forget to build that Dockerfile:

docker build --rm=true -t jfloff/app .

docker run --rm -t jfloff/app

Personally, I build an extended Dockerfile version (like shown above), and mount my specific application inside the container:

docker run --rm -v "$(pwd)":/home/app -w /home/app -p 5000:5000 -ti jfloff/app

Usage of `slim` images

These images are very small to download, and can install requirements at run-time via flags. The install only happens the first time the container is run, and dependencies can be baked in (see Creating Images).

Via `docker run`

These images can be run in multiple ways. With no arguments, it will run python interactively:

docker run --rm -ti jfloff/alpine-python:2.7-slim

If you specify a command, they will run that:

docker run --rm -ti jfloff/alpine-python:2.7-slim python hello.py

Pip Dependencies

Pip dependencies can be installed by the -p switch, or a requirements.txt file.

If the file is at /requirements.txt it will be automatically read for dependencies. If not, use the -P or -r switch to specify a file.

# This runs interactive Python with 'simplejson' and 'requests' installed
docker run --rm -ti jfloff/alpine-python:2.7-slim -p simplejson -p requests

# Don't forget to add '--' after your dependencies to run a custom command:
docker run --rm -ti jfloff/alpine-python:2.7-slim -p simplejson -p requests -- python hello.py

# This accomplishes the same thing by mounting a requirements.txt in:
echo 'simplejson' > requirements.txt
echo 'requests' > requirements.txt
docker run --rm -ti \
  -v requirements.txt:/requirements.txt \
  jfloff/alpine-python:2.7-slim python hello.py

# This does too, but with the file somewhere else:
echo 'simplejson requests' > myapp/requirements.txt
docker run --rm -ti \
  -v myapp:/usr/src/app \
  jfloff/alpine-python:2.7-slim \
    -r /usr/src/app/requirements.txt \
    -- python /usr/src/app/hello.py

Run-Time Dependencies

Alpine package dependencies can be installed by the -a switch, or an apk-requirements.txt file.

If the file is at /apk-requirements.txt it will be automatically read for dependencies. If not, use the -A switch to specify a file.

You can also try installing some Python modules via this method, but it is possible for Pip to interfere if it detects a version problem.

# Unknown why you'd need to do this, but you can!
docker run --rm -ti jfloff/alpine-python:2.7-slim -a openssl -- python hello.py

# This installs libxml2 module faster than via Pip, but then Pip reinstalls it because Ajenti's dependencies make it think it's the wrong version.
docker run --rm -ti jfloff/alpine-python:2.7-slim -a py-libxml2 -p ajenti

Build-Time Dependencies

Build-time Alpine package dependencies (such as compile headers) can be installed by the -b switch, or a build-requirements.txt file. They will be removed after the dependencies are installed to save space.

If the file is at /build-requirements.txt it will be automatically read for dependencies. If not, use the -B switch to specify a file.

build-base, linux-headers and python-dev are always build dependencies, you don't need to include them.

docker run --rm -ti jfloff/alpine-python:2.7-slim \
  -p gevent \
  -p libxml2 \
  -b libxslt-dev \
  -b libxml-dev \
  -- python hello.py

Creating Images

Similar to the onbuild images, dependencies can be baked into a new image by using a custom Dockerfile, e.g:

FROM jfloff/alpine-python:2.7-slim
RUN /entrypoint.sh \
  -p ajenti-panel \
  -p ajenti.plugin.dashboard \
  -p ajenti.plugin.settings \
  -p ajenti.plugin.plugins \
  -b libxml2-dev \
  -b libxslt-dev \
  -b libffi-dev \
  -b openssl-dev \
&& echo
CMD ["ajenti-panel"]
# you won't be able to add more dependencies later though-- see 'Debugging'

Debugging

The /entrypoint.sh script that manages dependencies in the slim images creates an empty file, /requirements.installed, telling the script not to install any dependencies after the container's first run. Removing this file will allow the script to work again if it is needed.

You can use the -x flag to see everything the /entrypoint.sh script is doing.

You can also access bash inside the container:

docker run --rm -ti jfloff/alpine-python:2.7-slim bash

Additional Arguments

-q: silences output from /entrypoint.sh -x: turns on Bash debugging, making the output very verbose.

Ecosystem

These are some of the images that use jfloff/alpine-python as base image. If you have another image that uses this as base image, please submit an issue or PR for it to be added. Image has to be published on Docker Hub.

jfloff/alscipy : image with common packages for Science in Alpine Python.
jfloff/pywfm : image from the python wrapper for Steffen Rendle's factorization machines library libFM.
bismuthfoundation/Bismuth-Docker : node and associated services, from scratch crypto-currency with Python codebase.

Contribution

Feel free to contribute with whatever you feel like this image is missing. There is also some changes that happen often like, updating Alpine or Python versions. Do not forget that this repo folders mirror Python version and not Alpine versions.

License

The code in this repository, unless otherwise noted, is MIT licensed. See the LICENSE file in this repository.

atmlvs/scripts

2023-10-10T01:27:57Z

untils in general task

Scripts

Untils in general tasks.

Description:

create target directories using YEAR.
read date time from EXIF as part of photo name.
use system creation time for photos with limit EXIF.
copy photos from src directory to corresponding target directories.

tested on Mac OS X 10.8.2/Python 2.7.3

aesmail/kaffy

2023-10-11T01:28:53Z

Powerfully simple admin package for phoenix applications

ur5r/barsoom_app

2023-10-11T01:29:14Z

Web-Machine Sample Application runs on windows 7 Erlang R16A

raffomania/spooky-shady-skeletons

2023-10-10T01:29:46Z

A simple rps-online clone in godot

dandeliondino/terrain-autotiler

2023-10-11T01:29:45Z

An advanced terrain tile matching algorithm for Godot 4.

Azure/azure-container-networking

2023-10-11T01:30:12Z

Azure Container Networking Solutions for Linux and Windows Containers

Microsoft Azure Container Networking

Azure Network Policy Manager Conformance
Cyclonus Network Policy Suite
Kubernetes Network Policy E2E

Overview

This repository contains container networking services and plugins for Linux and Windows containers running on Azure:

Azure CNI network and IPAM plugins for Kubernetes.
Azure CNM (libnetwork) network and IPAM plugins for Docker Engine. (MAINTENANCE MODE)
Azure NPM - Kubernetes Network Policy Manager (Linux and (preview) Windows Server 2022)

The azure-vnet network plugins connect containers to your Azure VNET, to take advantage of Azure SDN capabilities. The azure-vnet-ipam IPAM plugins provide address management functionality for container IP addresses allocated from Azure VNET address space.

The following environments are supported:

Microsoft Azure: Available in all Azure regions.

Plugins are offered as part of Azure Kubernetes Service (AKS), as well as for individual Azure IaaS VMs. For Kubernetes clusters created by aks-engine, the deployment and configuration of both plugins on both Linux and Windows nodes is automatic and default.

The next generation of Azure CNI Plugin is powered by Cilium. Learn more at Azure CNI Powered By Cilium

Documentation

See Documentation for more information and examples.

Build

This repository builds on Windows and Linux. Build plugins directly from the source code for the latest version.

$ git clone https://github.com/Azure/azure-container-networking
$ cd azure-container-networking
$ make all-binaries

Then follow the instructions for the plugin in Documentation.

Contributions

Contributions in the form of bug reports, feature requests and PRs are always welcome.

Please follow these steps before submitting a PR:

Create an issue describing the bug or feature request.
Clone the repository and create a topic branch.
Make changes, adding new tests for new functionality.
Submit a PR.

License

See LICENSE.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

geb/geb-seconf-2012-slides

2023-10-10T01:30:34Z

d3scomp/NPRG014

2023-10-10T01:30:34Z

Supplementary materials for the course of "Concepts of Modern Programming Languages" (NPRG014) taught at Charles University, Prague, Czech Republic.

nlad218/ShredNStyle

2023-10-11T01:30:45Z

Zelenya/elephants

2023-10-11T01:30:48Z

How to use PostgreSQL with Haskell. Elephantine Library Review 2023

As of October 2023, there are around a dozen mature PostgreSQL libraries, all levels of abstractions, from low-level queries to the type level. More than enough for anybody. (Shout out to everyone who says Haskell has no libraries.)

Which one to use? Let’s see. Foreach library, we’ll talk about features and basics like writing queries, complexity, pitfalls, (everyone’s favorite topic) errors, and so on.

We assume you are familiar with the fundamentals of SQL and PostgreSQL.

Mise en place

Before integrating with the database, let’s discuss the data and the setup.

💡 If you want to follow allow at home, the repository contains all the imports and data types — we omit most of them from the tutorial for simplicity.

Data

Let’s imagine that we’re building a (tiny) warehouse management system:

Warehouse has multiple products, some quantity of each.
Product has a label and a description.
Product can belong to categories (many-to-many).
Category has a label.

The scripts/create_tables.sql contains all the definitions:

CREATE TABLE product (
    id SERIAL PRIMARY KEY,
    label TEXT NOT NULL,
    description TEXT,
    UNIQUE (label)
);

CREATE TABLE category (
    id SERIAL PRIMARY KEY,
    label TEXT NOT NULL,
    UNIQUE (label)
);

CREATE TABLE product_category (
    category_id INT NOT NULL,
    product_id INT NOT NULL,
    PRIMARY KEY (category_id, product_id),
    FOREIGN KEY (product_id) REFERENCES product(id),
    FOREIGN KEY (category_id) REFERENCES category(id)
);

CREATE TABLE warehouse (
    id SERIAL PRIMARY KEY, 
    product_id INT NOT NULL,
    quantity INT NOT NULL,
    created TIMESTAMP,
    modified TIMESTAMP,
    FOREIGN KEY (product_id) REFERENCES product(id)
);

Postgres server

Here are a few things you need to know to play along at home.

🐘 If you don’t care, don’t like docker, or already have an easily-accessible postgres server, feel free to skip this section.

First, install docker.

docker compose up (docker-compose up on older versions) starts PostgreSQL (see docker-compose.yml) and initializes the databases and tables (using the scripts/create_tables.sql). It mounts the data to the postgres-data/ (in case you need to wipe it or something).

Some of the hardcoded things that the code relies on:

environment:
  - POSTGRES_DB=warehouse
  - POSTGRES_USER=postgres
  - POSTGRES_PASSWORD=password
ports:
  - 5432:5432

💡 These hardcoded values are also hardcoded in the code in Hardcoded.hs (for laziness reasons)

You can connect to the container and run arbitrary queries using psql:

docker exec -it elephants-postgres-1 psql -U postgres -d warehouse

Note: elephants-postgres-1 is a container name, which might be different for you; check with docker ps to get the correct container id (or name). We also pass a couple of flags: -U postgres for the user name and -d warehouse for the database name.

docker compose down to stop and remove the containers.

Project Overview

If you have stack installed, stack build to build and stack run to run.

💡 We use lts-21.7 (ghc-9.4.5), published on 2023-08-14.

🤷 To build the backend, you might need the libpq development libraries installed (e.g., libpq-dev on Debian-based distributions).

Extensions overview

Note that most of the libraries rely on using various extensions. Here is a quick overview of the most important ones.

OverloadedStrings

Used to simplify the construction of query values — we can use literal strings, like "SELECT * FROM user", instead of manually constructing the whole type; for example, Query . toByteString . stringUtf8 $ "SELECT * FROM user" (see Query in postgresql-simple).

TemplateHaskell

Template Haskell (TH) is fairly useful for generating boilerplate code. Some libraries provide the TH machinery to derive typeclass instances and/or generate custom type-safe data types at the compile time.

DeriveAnyClass and DeriveGeneric enable alternative ways to get free typeclass instances.

We’ll use DerivingStrategies to make the derivation explicit, for example:

data Category = Category {label :: Text}
  deriving (Show, Generic)           -- Derive Generic instance,
  deriving anyclass (ToRow, FromRow) -- used for these instances

QuasiQuotes

Some libraries (e.g., postgresql-simple) provide quasi quoters for less tedious sql construction in Haskell code. For example:

[sql| select label from product |]

DuplicateRecordFields

Only one postgres library requires it. But it’s a stylistic choice — we use this extension because we have multiple data types with a label field.

OverloadedRecordDot

To access record fields as well as specific columns of the tables (for example, product.label)

Note that the project uses GHC2021, which includes DeriveGeneric, TypeApplications, and many other extensions required for some libraries to work. Also, for the tutorial's sake, we’ll use the included ScopedTypeVariables to demonstrate some intermediate types.

`postgresql-simple`

Let’s start simple. postgresql-simple describes itself as “Mid-Level PostgreSQL client library“.

In other words, we’ll write raw sql queries, and the library will deal with security and stuff.

To get started, we add postgresql-simple to dependencies. We’re using v0.7.0.0 published in 2023.

How to connect to a database

We use connect to acquire a connection. It accepts ConnectInfo, which we can get by using defaultConnectInfo and overriding some defaults.

getConnection :: IO Connection
getConnection =
  connect $
    defaultConnectInfo
      { connectHost = Hardcoded.host
      , connectDatabase = Hardcoded.database
      , connectUser = Hardcoded.user
      , connectPassword = Hardcoded.password
      }

Eventually, we have to close connection. But you will probably not need to do it manually because you can use withConnect, bracket, or (better) a connection pool.

The library doesn’t support pools, but you can use the resource-pool package (or something similar).

How to modify data

We use execute and execute_ to insert, update, and delete data.

The version with the _ suffix is simpler — it doesn’t perform any query substitutions. We can use it with hardcoded values or with straightforward queries such as truncating the tables:

cleanUp :: Connection -> IO ()
cleanUp connection =
  void $ execute_ connection "truncate warehouse, product_category, product, category"

Both execute functions return the number of affected rows, which isn’t relevant in case of truncate (it’s 0) and safe to ignore (void ignores the result of evaluation).

We can use execute to make a proper insert and pass some values for substitutions. The simplest way is to pass a tuple:

insert1 <-
  execute
    connection
    "insert into product (label, description) values (?, ?)"
    ("Wood Screw Kit 1" :: Text, "245-pieces" :: Text)

Sometimes, we must be explicit about types; for example, in cases like this, when we use string literals with OverloadedStrings or numeric literals (like 245).

Because there is no tuple of 1, the library provides a custom type Only:

insert2 <-
  execute
    connection
    "insert into product (label) values (?)"
    (Only "Wood Screw Kit 2" :: Only Text)

Alternatively, we can use lists for any number of values:

insert3 <-
  execute
    connection
    "insert into product (label) values (?)"
    ["Wood Screw Kit 3" :: Text]

But preferable, we use dedicated types:

execute
  connection
  "insert into product (label, description) values (?, ?)"
  (BasicProduct "Wood Screw Kit 4" (Just "245-pieces"))

A record can be turned into a list of substitutions via the ToRow typeclass, which is derivable using GHC generics:

data BasicProduct = BasicProduct {label :: Text, description :: Maybe Text}
  deriving (Show, Generic)
  deriving anyclass (ToRow, FromRow)

If we want to modify multiple rows, we can use executeMany:

insert5 <-
 executeMany
   connection
   insert into category (label) values (?)"
   [Category "Screws", Category "Wood Screws", Category "Concrete Screws"]

How to query data

The execute functions can’t return any results (other than the number of affected rows), so we have to use the query functions.

Similar to execute_, query_ takes a query with no substitutes:

query1 :: [(Int64, Text, Maybe Text)] <-
  query_ connection "select id, label, description from product"

Note that we must be explicit about return types — the library can’t guess what we want. In this case, we expect an id of type Int64 that corresponds to Serial, required Text label, and optional description.

We can specify a record return type if we derive FromRow (recall ToRow from the previous section). For example, let’s get a BasicProduct list by label using query:

query2 :: [BasicProduct] <-
  query 
    connection 
    "select label, description from product where label = ? "
    (Only "Wood Screw Kit 2" :: Only Text)

If we want to use the in-clause, the library provides a dedicated wrapper:

query3 :: [BasicProduct] <-
  query connection "select label, description from product where label in ?" $
    Only (In ["Wood Screw Kit 2" :: Text, "Wood Screw Kit 3"])

How to use transactions

Imagine we want to atomically insert a new listing: product, category, and quantity. This touches multiple tables and requires a transaction. Additionally, because we have a many-to-many relationship, we must first insert the product and category and then use their new ids to create a mapping.

We can use returning to get ids of created rows:

productIds :: [Only Int64] <-
  query
    connection
    "insert into product (label, description) values (?, ?) returning id"
    (BasicProduct "Drywall Screws Set" (Just "8000pcs"))

categoryIds :: [Only Int64] <-
  query
    connection
    "insert into category (label) values (?) returning id"
    (Category "Drywall Screws")

Note that we must use query and not execute because these queries return results.

We can use withTransaction to wrap multiple queries in a transaction:

withTransaction connection $ do
  productIds :: [Only Int64] <- query ...
  categoryIds :: [Only Int64] <- query ...

  void $ case (productIds, categoryIds) of
    ([Only productId], [Only categoryId]) -> do
      _ <-
        execute
          connection
          "insert into warehouse (product_id, quantity, created, modified) values (?, ?, now(), now())"
          (productId, 10 :: Int)

      execute
        connection
        "insert into product_category (category_id, product_id) values (?, ?)"
        (categoryId, productId)
    _ -> 
      throwIO $ userError "Failed to insert product/category"

Any error will rollback the transaction (and the exception will be rethrown). In this example, we throw an explicit error if we don’t get the expected ids for some reason.

Note that in case of a sql error, the exception will not only rollback the transaction but, if uncaught, will propagate further (killing everything on its way and potentially crashing the whole app). So, we should (at least) wrap transactions in the exception handler(s); we’ll see how to do this later.

When you need to, you can also use granular transaction functions: begin, commit, and rollback.

How to query using joins

To read all these tables at once, we need to query using a few joins. The library provides a quasi-quoter that makes writing big queries easier — we can format the query and not worry about whitespaces:

result :: [Listing] <-
  query
    connection
    [sql|
      select
        w.quantity,
        p.label,
        p.description,
        c.label
      from warehouse as w
      inner join product as p on w.product_id = p.id
      left outer join product_category as pc on p.id = pc.product_id
      left outer join category as c on c.id = pc.category_id
      where w.quantity > (?)|]
    [3 :: Int]

Errors

In postgresql-simple, all the programmer errors (in sql or library usage) are (runtime) exceptions.

If the query string is not formatted correctly, we get FormatError. For instance, if we have a mismatching number of substitutions (? and actual values):

execute
  connection
  "INSERT INTO category (label) VALUES (?)"
  ("One" :: Text, "Two" :: Text)

FormatError {fmtMessage = "1 single '?' characters, but 2 parameters", fmtQuery = "INSERT INTO category (label) VALUES (?)", fmtParams = ["One","Two"]}

Similarly, on the return side, if the number of columns doesn’t match the number of elements in the result type (in a list, a tuple, or a record), we get ResultError. The most likely variants are Incompatible and UnexpectedNull.

If we forget to wrap a nullable type on the Haskell side, we get UnexpectedNull. For instance, if we try to get description (which is nullable) as Text and not Maybe Text:

let result :: IO [(Text, Text)] = query_ connection "select label, description from product"

UnexpectedNull {errSQLType = "text", errSQLTableOid = Just (Oid 16386), errSQLField = "description", errHaskellType = "Text", errMessage = ""}

If we mistype the types, we get Incompatible. For instance, if we try to parse just id into BasicProduct:

let result :: IO [BasicProduct] = query_ connection "select id from product"

Incompatible {errSQLType = "int4", errSQLTableOid = Just (Oid 16386), errSQLField = "id", errHaskellType = "Text", errMessage = "types incompatible"}

On top of that, if we misuse the library — by mistaking query for execute or vice verse — we get QueryError. For example, if we use execute with insert query that has returning:

execute_
  connection
  "INSERT INTO category (label) VALUES (Screws) returning id"

QueryError {qeMessage = "execute resulted in Col 1-column result", qeQuery = "INSERT INTO category (label) VALUES ('Deck Screws') returning id"}

And last but not least, any sql errors from postgres, will come back as SqlError:

let result :: IO [BasicProduct] = query_ connection "select I have no idea what I'm doing"

Wrong sql: SqlError {sqlState = "42601", sqlExecStatus = FatalError, sqlErrorMsg = "syntax error at or near \"no\"", sqlErrorDetail = "", sqlErrorHint = ""}

The errors are pretty good but still not the most descriptive — if you try to write big queries, you have to concentrate on projecting the error information to the query.

Resources

The docs are also simple; the library covers all the primary blocks, describes the functions, and provides some examples. Outside, a few blog posts cover similar things, mainly targeting beginners.

And you don’t need more than that — if you know how to write one simple query, you know how to write them all.

Migrations

The library has a companion package, postgresql-migration.

🗂️ This is a fork of the archived postgresql-simple-migration.

In summary

postgresql-simple is a library for all levels and a great option if you love writing sql by hand and don’t need reusability.

It doesn’t parse or validate the queries, so we must pay attention to what we write: sql queries, haskell types (type-safety is our responsibility), the order of parameters, and so on.

`hasql`

The next “obvious” step is to add more type-safety.

According to the readme, Hasql “is a highly efficient PostgreSQL driver for Haskell with a typesafe yet flexible mapping API; it is production-ready, actively maintained, and the API is pretty stable. It's used by many companies and most notably by the Postgrest project.“

Hasql is an ecosystem of libraries. To keep it simple, let’s limit ourselves to core hasql, hasql-transaction, and hasql-th. We’re using hasql 1.6.3.2 published in 2023.

We’ll also use contravariant-extras, vector, profunctors, and tuple packages to make a few things tidier (this isn’t required; it’s all copy-paste anyway).

💡 (It’s not very important, but) We assume you’ve seen the part on postgresql-simple, which covers the same topics but at a slower pace.

How to connect to a database

First, we get a connection:

Right connection <- getConnection

getConnection :: IO (Either ConnectionError Connection)
getConnection =
  acquire $ settings Hardcoded.host Hardcoded.portNumber Hardcoded.user Hardcoded.password Hardcoded.database

Note the Either. But for now, let’s just pattern-match and not worry about possible errors…

In reality/production, we should probably use hasql-pool to work with a pool of connections.

How to modify data

Let’s see the leading players through the clean-up query:

cleanUp :: Connection -> IO (Either QueryError ())
cleanUp connection = run cleanUpSession connection
 where
  cleanUpSession :: Session ()
  cleanUpSession = statement () cleanUpStatement

  cleanUpStatement :: Statement () ()
  cleanUpStatement = Statement rawSql E.noParams D.noResult True

  rawSql = "truncate warehouse, product_category, product, category"

Session is a batch of actions to be executed in the context of a connection (a query).
Statement is a specification of a strictly single-statement query, which can be parameterized and prepared (how to make a query).
Statement consists of SQL template, params encoder, result decoder, and a flag that determines whether it’s prepared.
statement creates a Session from a Statement and input parameters.
run executes a bunch of commands (statements) on the provided connection.

💡 Remember that you can see the complete code in the repo.

We have a simple query with no parameters and no result — we don’t need to encode or decode anything. That’s what E.noParams D.noResult for. If we want to pass parameters, we need to supply a decoder.

The first option, is tuples of primitive types and manually written decoders:

insertProductSql = "insert into product (label, description) values ($1, $2)"

insertProduct1 :: Statement (Text, Maybe Text) Int64
insertProduct1 = Statement insertProductSql rawParams D.rowsAffected True

rawParams =
  (fst >$< E.param (E.nonNullable E.text))
    <> (snd >$< E.param (E.nullable E.text))

statement ("Wood Screw Kit 1", Just "245-pieces") insertProduct1

rawParams is the encoder for our parameters. We use contramap operator (>$<) and append (<>) to compose multiple parameters. D.rowsAffected is the decoder for the result when we want to return the number of affected rows.

💡 Instead of fst and snd, you can use the contrazip family of functions from the contravariant-extras package to reduce boilerplate.

Another option, is using records:

insertProduct2 :: Statement BasicProduct Int64
insertProduct2 = Statement insertProductSql basicProductParams D.rowsAffected True

basicProductParams :: E.Params BasicProduct
basicProductParams =
  ((.label) >$< E.param (E.nonNullable E.text))
    <> ((.description) >$< E.param (E.nullable E.text))

statement (BasicProduct "Wood Screw Kit 2" Nothing) insertProduct2

If we want to modify multiple rows, we have to use the postgres unnest function:

insertManyCategories :: Statement (Vector Category) Int64
insertManyCategories = Statement insertManyCategoriesSql categoryParams D.rowsAffected True

insertManyCategoriesSql = "insert into category (label) select * from unnest ($1)"

categoryParams :: E.Params (Vector Category)
categoryParams =
  E.param
    $ E.nonNullable
    $ E.array
    $ E.dimension List.foldl'
    $ categoryArray

categoryArray :: E.Array Category
categoryArray = (.label) >$< (E.element $ E.nonNullable E.text)

categoryParams is an encoder that allows us to pass a vector of categories to insert.

let categories = [Category "Screws", Category "Wood Screws", Category "Concrete Screws"]
statement (fromList categories) insertManyCategories

Note that unnest is more efficient than executing a single-row insert statement multiple times.

How to query data

Querying data is similar:

session1 :: Session [(Int64, Text, Maybe Text)]
session1 =
  statement ()
    $ Statement
      "select id, label, description from product"
      E.noParams
      decoder1
      True

decoder1 =
  D.rowList
    $ (,,)
    <$> D.column (D.nonNullable D.int8)
    <*> D.column (D.nonNullable D.text)
    <*> D.column (D.nullable D.text)

We need to provide a decoder for the result (to specify how each row results maps into the expected type). If this sounds tedious, we can ask Template Haskell to do the work for us:

In this case, we use singletonStatement that expects one result. There are other variants that we’ll see later.

session2 :: Session (Text, Maybe Text)
session2 = statement () statement2

statement2 :: Statement () (Text, Maybe Text)
statement2 =
  [singletonStatement|
    select label :: text, description :: text? from product limit 1
  |]

We write the query and specify the types, hasql-th handles the codecs for us.

But we still need to handle the conversions if we use custom types instead of tuples. The result of the statement has a Profunctor instance, which allows us to modify (input) parameters and (output) results. In other words, we use lmap to map parameters, rmap — result, and dimap — both. For example, let’s return BasicProduct:

session3 :: Session (Maybe BasicProduct)
session3 = statement "Wood Screw Kit 2" statement3

statement3 :: Statement Text (Maybe BasicProduct)
statement3 =
  rmap
    (fmap (uncurryN BasicProduct))
    [maybeStatement|
      select label :: text, description :: text?
      from product
      where label = $1 :: text
    |]

💡 (fmap (uncurryN BasicProduct)) is a concise way to write the following (using tuples package):

(\result -> fmap (\(a, b) -> (BasicProduct a b)) result)

hasql doesn’t have "special support" for an array as a parameter for the IN operator, we should use Any:

session4 :: Session (Vector BasicProduct)
session4 = statement (fromList ["Wood Screw Kit 1", "Wood Screw Kit 2"]) statement4

statement4 :: Statement (Vector Text) (Vector BasicProduct)
statement4 =
  rmap
    (fmap (uncurryN BasicProduct))
    [vectorStatement|
      select label :: text, description :: text?
      from product
      where label = ANY($1 :: text[])
    |]

How to use transactions

We can use returning to get ids of created rows:

insertProduct :: Statement (Text, Maybe Text) Int64
insertProduct =
  [singletonStatement|
    insert into product (label, description) values ($1 :: text, $2 :: text?) returning id :: int8
  |]

To wrap multiple queries in a transaction, we can use hasql-transaction. First, we compose the statements:

insertAll :: FullProduct -> Transaction Int64
insertAll listing = do
  productId <- Transaction.statement (listing.label, listing.description) insertProduct
  categoryId <- Transaction.statement listing.category insertCategory
  _ <- Transaction.statement (productId) insertListing
  ids <- Transaction.statement (productId, categoryId) insertMapping
  pure ids

insertProduct :: Statement (Text, Maybe Text) Int64

insertCategory :: Statement Text Int64

insertListing :: Statement Int64 ()

insertMapping :: Statement (Int64, Int64) Int64

Then we run the transaction using the relevant isolation level and mode:

insertWithTransaction :: Connection -> IO ()
insertWithTransaction connection = do
  let listing = FullProduct "Drywall Screws Set" (Just "8000pcs") "Drywall Screws"
  mapping <- run (transaction Serializable Write $ insertAll listing) connection
  putStrLn $ "Insert with transaction: " <> show mapping

How to query using joins

We can query these tables using a few joins. There should be nothing unexpected here:

listings :: Statement Int32 (Vector Listing)
listings =
  rmap
    (fmap (uncurryN Listing))
    [vectorStatement|
    select
        w.quantity :: int,
        p.label :: text,
        p.description :: text?,
        c.label :: text?
      from warehouse as w
      inner join product as p on w.product_id = p.id
      left outer join product_category as pc on p.id = pc.product_id
      left outer join category as c on c.id = pc.category_id
      where w.quantity > $1 :: int4
  |]

Errors

We’ve been neglecting this until now, but all error reporting is explicit and is presented using Either.

💡 Just a reminder, don’t ignore errors. And don’t pattern match only on Right, or you will end up with this:

user error (Pattern match failure in 'do' block at ...)

The other good thing is that the hasql-th parser is pretty good at error reporting and catching typos at compile time (and most of the time, it’s more explicit than postgres’ syntax error at or near). This won’t compile:

[singletonStatement|
    select I have no idea what I'm doing
|]

The library doesn’t accept (doesn’t compile) if you forget to specify one of the types. For instance, if we omit type of label, we get a somewhat generic error:

[singletonStatement|
    select label, description :: text? from product
|]

Result expression is missing a typecast

This ensures that most input and result type (including nullability) mismatches are caught in the compile time. For example, if we forget an input type and return a wrong result type:

statement :: Statement () (Text, Int32)
statement =
  [singletonStatement|
    select label :: text, description :: text?
--                                       ^^^^^ 
--  Couldn't match type ‘Int32’ with ‘Maybe Text’
    from product where label = $1 :: text
--                                   ^^^^
--  Couldn't match type ‘Text’ with ‘()’
  |]

However, we’re not safe from programming errors. We should use correct statement functions not to get a runtime error. For example, if we use singletonStatement for statements that might not return a result (instead of maybeStatement):

do
  failure <- run (statement () failsBecauseNoResults) connection
  putStrLn $ "Wrong statement function: " <> show failure
 where
  failsBecauseNoResults :: Statement () (Text)
  failsBecauseNoResults =
    [singletonStatement|
        select label :: text from product where 1 = 0
    |]

Wrong statement function: Left (QueryError "SELECT label :: text FROM product WHERE 1 = 0" [] (ResultError (UnexpectedAmountOfRows 0)))

Or if we use singletonStatement with () result (instead of resultlessStatement):

do
  failure <- run (statement () failsBecauseResultless) connection
  putStrLn $ "Wrong statement function: " <> show failure
 where
  failsBecauseResultless :: Statement () ()
  failsBecauseResultless =
    [singletonStatement|
      insert into product (label) values ('this insert fails')
    |]

Wrong statement function: Left (QueryError "INSERT INTO product (label) VALUES ('this insert fails')" [] (ResultError (UnexpectedResult "Unexpected result status: CommandOk")))

In case of runtime sql error, for instance, if we violate a constraint, we get a similar error:

inserProduct :: Statement Text ()
inserProduct =
  [singletonStatement|
    insert into product (label) values ($1 :: text)
  |]

run (statement "Duplicate screw" inserProduct) connection
  >> run (statement "Duplicate screw" inserProduct) connection

Wrong statement function (Left): QueryError "INSERT INTO product (label) VALUES ('Duplicate')" [] (ResultError (ServerError "23505" "duplicate key value violates unique constraint \"product_label_key\"" (Just "Key (label)=(Duplicate screw) already exists.") Nothing Nothing))

Resources

Core readme has a good overview and example. The library has simple docs, a couple of tutorials, and talks from the author.

Migrations

hasql-migrations tool is a port of postgresql-simple-migration for use with hasql.

In summary

Overall, hasql is a great choice for writing raw sql queries with more type safety and compile-time syntax checks. The ecosystem comes with other whistles like connection pools and transactions.

The TemplateHaskell module and compile-time checks are optional — if you want, you can deal with the encoders and decoders yourself.

The library requires basic/intermediate knowledge of Haskell and ecosystems. To be comfortable and productive, you must be familiar with vectors, contravariant functors, etc. Other than that, the library is relatively straightforward.

`persistent` + `esqueleto`

If that was not enough, it’s time to move to the type level.

According to the readme, ”Persistent's goal is to catch every possible error at compile-time, and it comes close to that. It is also designed to be adaptable to any datastore”. As a result, ”a major limitation for SQL databases is that the persistent library does not directly provide joins”.

However, we can use Esqueleto (”a bare bones, type-safe EDSL for SQL queries”) with Persistent's serialization to write type-safe SQL queries. It’s unlikely that you want to use Persistent by itself with SQL, so let’s use and review them together.

We’re using persistent (2.14.5.1), persistent-postgresql (2.13.5.2), and esqueleto (3.5.10.1), all published in 2023. Additionally, we’ll use the experimental style, which will become the new "default" in esqueleto-4.0.0.0.

We’ll also use mtl, monad-logger, unliftio-core, time, and exceptions.

The libraries require additional extensions: DataKinds, GADTs, TypeFamilies, and UndecidableInstances.

💡 (It’s not very important, but) We assume you’ve seen the part on postgresql-simple, which covers the same topics but at a slower pace.

How to connect to a database

Database.Persist.Postgresql provides various ways to connect to postgres with and without a connection pool.

First, we need a libpq connection string, which looks like this "host=localhost port=5432 user=postgres dbname=warehouse password=password".

We create the pool and run actions on it using withPostgresqlPool and passing the connection string, number of connections, and action(s) to be executed. We use liftSqlPersistMPool to run an action/transaction on a pool. And “finally”, use runNoLoggingT (runStdoutLoggingT, or alternative) to run with appropriate logging.

runNoLoggingT $ P.withPostgresqlPool Hardcoded.connectionString 3 $ \pool -> do
    le runWithPool = flip liftSqlPersistMPool pool
    runWithPool transaction1
    runWithPool transaction2
    ...

💡 We can use runStdoutLoggingT to see what sql queries get executed.

How to define tables

Persistent takes care of creating and matching Haskell datatypes and PersistEntity instances; we need to declare the entities by passing them to mkPersist:

mkPersist
  sqlSettings
  [persistLowerCase|
  Product
    label Text
    description Text Maybe
    UniqueLabel label
    deriving Eq Show
  Category
    label Text
    UniqueCategory label
    deriving Eq Show
  ProductCategory
    productId ProductId
    categoryId CategoryId
    Primary productId categoryId
    deriving Eq Show
  Warehouse
    productId ProductId
    quantity Int
    created UTCTime default=CURRENT_TIME
    modified UTCTime default=CURRENT_TIME
    deriving Eq Show
|]

persistLowerCase states that SomeTable corresponds to the SQL table some_table.

How to modify data

Even though it’s not encouraged, we can always execute raw sql; for example, we can truncate tables with rawExecute:

cleanUp :: (MonadIO m) => SqlPersistT m ()
cleanUp = rawExecute "truncate warehouse, product_category, product, category" []

What’s SqlPersistT m ()? Let’s say it’s something that can be executed with runWithPool and returns ().

💡 Note that we can also use deleteWhere to delete all the records from a table:

deleteWhere ([] :: [Filter Product]))

Because we’ve done all the groundwork, we use records right away (no tuples):

insertStuff :: (MonadIO m) => SqlPersistT m ()
insertStuff = do
  newId <- insert $ Product "Wood Screw Kit 1" (Just "245-pieces")
  liftIO $ putStrLn $ "Insert 1: " <> show newId

  newIds <- insertMany [Category "Screws", Category "Wood Screws", Category "Concrete Screws"]
  liftIO $ putStrLn $ "Insert 2: " <> show newIds

That’s it! Persistent is concise when it comes to inserts. Note that insert returns the id, and insertMany returns multiple ids.

We can use liftIO to execute IO operations like printing “inside” SqlPersistT.

How to query data

This is the part where esqueleto comes in.

The first query takes a label and returns a list of product entities:

query1 :: Text -> SqlPersistT m [Entity Product]
query1 label = select $ do
  aProduct <- from $ table @Product
  where_ (aProduct.label ==. val label)
  pure aProduct

It returns an Entity instead of a value — an Entity combines a database id and a value.

This is an experimental syntax that mimics sql. We use the TypeApplications extensions to make the table explicit, OverloadedRecordDot to select the field/column value, the ==. operator to check for equality, and val to “lift” haskell value into “sql query land”.

💡 Note that there are other alternatives for field projections (instead of OverloadedRecordDot), such as the (^.) operator and OverloadedLabels.

We can select multiple labels using in_:

query2 :: [Text] -> SqlPersistT m [Entity Product]
query2 lables = select $ do
  aProduct <- from $ table @Product
  where_ $ aProduct.label `in_` valList lables
  pure aProduct

How to use transactions

We’ve been kind-of using transactions all this time. Everything inside a single call to liftSqlPersistMPool (and other versions, with and without pooling) runs in a single transaction.

insertWithTransaction :: (MonadIO m, MonadCatch m) => SqlPersistT m ()
insertWithTransaction = handle (\(SomeException _) -> pure ()) $ do
  productId <- insert $ Product "Drywall Screws Set" (Just "8000pcs")
  categoryId <- insert $ Category "Drywall Screws"
  time <- liftIO getCurrentTime
  _ <- insert_ $ Warehouse productId 10 time time
  _ <- insert_ $ ProductCategory productId categoryId
  liftIO $ putStrLn $ "Insert with transaction"

This time, we handle exceptions (any SomeException).

💡 We generally want to split the queries into transactions and catch exceptions on each transaction. We dive deeper into error handling in the errors section.

How to query using joins

And this is the part where experimental syntax comes in handy:

query quantity = select $ do
  (warehouse :& aProduct :& _ :& category) <-
    from
      $ table @Warehouse
      `innerJoin` table @Product
      `on` do \(w :& p) -> w.productId ==. p.id
      `LeftOuterJoin` table @ProductCategory
      `on` do \(_ :& p :& pc) -> just p.id ==. pc.productId
      `LeftOuterJoin` table @Category
      `on` do \(_ :& _ :& pc :& c) -> pc.categoryId ==. c.id
  where_ (warehouse.quantity >. val quantity)
  pure $ (warehouse.quantity, aProduct.label, aProduct.description, category.label)

The on clauses are attached directly to the relevant join. The ON clause lambda has all the available tables — only the tables we have already joined into are in scope.

We use the :& operator to pattern match against the joined tables. We use _ placeholder to ignore the previous references to the table.

This generates this query:

SELECT 
  "warehouse"."quantity", 
  "product"."label", 
  "product"."description", 
  "category"."label" 
FROM 
  "warehouse" 
  INNER JOIN "product" ON "warehouse"."product_id" = "product"."id" 
  LEFT OUTER JOIN "product_category" ON "product"."id" = "product_category"."product_id" 
  LEFT OUTER JOIN "category" ON "product_category"."category_id" = "category"."id" 
WHERE 
  "warehouse"."quantity" > ?

Errors

It’s possible to write type-checked queries that fail at runtime, but most typical sql errors are caught as compile-time errors.

Sometimes, mistakes in queries will result in error messages that refer to library internals (for example, you might see PersistUniqueRead backend0, Database.Esqueleto.Internal.Internal.SqlExpr, PersistRecordBackend backend val, ‘BaseBackend backend0’, ‘SqlBackend’). This takes some time to get used to. Help the type inference, and it will help you.

Nobody is safe from runtime sql errors. For example, if we violate the uniqueness constraint, we get an exception that we need to deal with:

errors :: (MonadIO m, MonadCatch m) => SqlPersistT m ()
errors = do
  let duplicateScrew = Product "Duplicate screw" Nothing
  void $ insert duplicateScrew
  (void $ insert duplicateScrew)
    `catch` (\(SomeException err) -> liftIO $ putStrLn $ "Caught SQL Error: " <> displayException err)

Caught SQL Error: SqlError {sqlState = "23505", sqlExecStatus = FatalError, sqlErrorMsg = "duplicate key value violates unique constraint \"product_label_key\"", sqlErrorDetail = "Key (label)=(Duplicate screw) already exists.", sqlErrorHint = ""}

Note that we use the exceptions package to handle exceptions. (We don’t use the exceptions from Control.Exception as we did in postgresql-simple because we don’t want to be limited to IO).

Resources

persistent is well documented as part of the yesod book, and esqueleto has good readme and docs. The catch is that you have to keep an eye on multiple packages simultaneously.

On top of that, (currently) esqueleto supports legacy and experimental syntax, and you have to be aware that some tutorials and examples use less safe legacy syntax (or a mix of both) — the good news is that the compiler will warn you if you’re on the wrong path.

Migrations

persistent can automatically create tables and do migrations. However, the book discourages that:

“Using automated database migrations is only recommended in development environments. Allowing your application to modify your database schema in a production environment is very strongly discouraged.”

In summary

You should consider persistent with esqueleto if you mainly have a lot of simple queries, are tired of writing raw sql, but want moderately more type-safe and composable sql.

The persistent is a (very) generalized library, meaning you should be comfortable using abstractions. And you should be familiar with mtl, monad-logger, lifting/unlifting IO, and so on.

“Most kinds of errors committed when writing SQL are caught as compile-time errors — although it is possible to write type-checked esqueleto queries that fail at runtime”

If you look around, some tutorials and comparisons might say that esqueleto joins might lead to to runtime errors. Don’t worry — this refers to legacy syntax — use new/experimental syntax.

`beam`

Tired of sql and Template Haskell?

Beam “is a highly-general library for accessing any kind of database with Haskell”. Beam makes extensive use of GHC's Generics mechanism — no Template Haskell.

First, install beam-core (0.10.1.0 released in 2023) and beam-postgres(0.5.3.1).

A few additional extensions: GADTs and TypeFamilies.

beam-postgres is built on top of postgresql-simple, which is used for connection management, transaction support, serialization, and deserialization.

💡 We assume that you’ve seen the part on postgresql-simple.

How to connect to a database

We use postgresql-simple straight away. Reminder:

connectionInfo :: ConnectInfo
connectionInfo =
  defaultConnectInfo
    { connectHost = Hardcoded.host
    , connectDatabase = Hardcoded.database
    , connectUser = Hardcoded.user
    , connectPassword = Hardcoded.password
    }

Simple.withConnect connectionInfo $ \connection -> do
    doFoo connection
    doBar connection

How to define tables

Let’s look at the definition of the product table:

data ProductT f = Product
  { id :: Columnar f Int64
  , label :: Columnar f Text
  , description :: Columnar f (Maybe Text)
  }
  deriving (Generic)
  deriving anyclass (Beamable)

type Product = ProductT Identity
deriving instance Show Product

instance Table ProductT where
  data PrimaryKey ProductT f = ProductId (Columnar f Int64)
    deriving (Generic)
    deriving anyclass (Beamable)
  primaryKey = ProductId . (.id)

ProductT is a beam table. All beam tables must implement the Beamable typeclass (derived via generics) and the Table typeclass. The Table instance declares the type of primary keys for the table and a function that extracts them. We can use Product to construct values of type Product.

💡 For details, see beam tutorial.

All the other tables look quite similar; see the repo for the rest of the boilerplate. One interesting bit is foreign keys / referencing other primary keys; for example, product_id and category_id in the mapping table look like are defined as PrimaryKey ProductT f (not Columnar f Int64):

data ProductCategoryT f = ProductCategory
  { product_id :: PrimaryKey ProductT f
  , category_id :: PrimaryKey CategoryT f
  }
  deriving (Generic)
  deriving anyclass (Beamable)

After declaring all the tables, we describe our database:

data WarehouseDb f = WarehouseDb
  { product :: f (TableEntity ProductT)
  , category :: f (TableEntity CategoryT)
  , product_category :: f (TableEntity ProductCategoryT)
  , warehouse :: f (TableEntity WarehouseT)
  }
  deriving (Generic)
  deriving anyclass (Database Postgres)

warehouseDb :: DatabaseSettings Postgres WarehouseDb
warehouseDb =
  defaultDbSettings
    `withDbModification` dbModification
      { product_category =
          modifyTableFields
            tableModification
              { category_id = CategoryId (fieldNamed "category_id")
              , product_id = ProductId (fieldNamed "product_id")
              }
      , warehouse =
          modifyTableFields @WarehouseT
            tableModification
              { product_id = ProductId (fieldNamed "product_id")
              }
      }

WarehouseDb needs to define all the tables and an instance of Database _.

💡 Note that you don’t need to hardcode Postgres and can keep the database more generic.

If you don’t have an existing database, you might get away with only defaultDbSettings as DatabaseSettings. Beam can guess a lot about the tables if we follow their conventions. But we need to override a few generated table fields in our case.

Remember that we have a couple of foreign keys? Beam adds a suffix __id to these, meaning if we have a record field product_id, generated queries will try to use the column product_id__id. So, we must override these in the product_category* mapping and warehouse tables.

💡 See beam defaults for more information.

How to modify data

For raw queries, we can use postgresql-simple:

cleanUp :: Connection -> IO ()
cleanUp connection =
  void $ Simple.execute_ connection "truncate warehouse, product_category, product, category"

Let’s insert some products:

insert1 :: Connection -> IO ()
insert1 connection =
  runBeamPostgres connection
    $ runInsert
    $ insert (warehouseDb.product)
    $ insertValues
      [ Product 1 "Wood Screw Kit 1" (Just "245-pieces")
      , Product 2 "Wood Screw Kit 2" Nothing
      ]

We construct the statement using insert, which accepts a table and values. We use insertValues to supply concrete values (including ids). runInsert runs the statement (in MonadBeam), which runBeamPostgres executes using the given connection.

💡 Note that we can use runBeamPostgresDebug putStrLn instead of runBeamPostgres to see what sql queries get executed.

runInsert doesn’t return anything (no affected rows, no ids, nothing). When we want some confirmation back, we can use runInsertReturningList:

insert2 :: Connection -> IO ()
insert2 connection = do
  result :: [Category] <-
    runBeamPostgres connection
      $ runInsertReturningList
      $ insert (warehouseDb.category)
      $ insertExpressions
        [ Category default_ "Screws"
        , Category default_ "Wood Screws"
        , Category default_ "Concrete Screws"
        ]

  putStrLn $ "Inserted categories: " <> show result

We can use insertExpressions function to insert arbitrary sql expressions. In this case, we pass default_ to ask the database to give us default ids.

How to query data

Instead of talking about Q monads and MonadBeam, let’s look at the examples. First, query all the products:

query1 :: (MonadBeam Postgres m) => m [Product]
query1 = do
  let allProducts = all_ (warehouseDb.product)
  runSelectReturningList $ select allProducts

runBeamPostgres connection query

Important bits:

build a query;
pass it into select;
run it in MonadBeam (using runSelectReturningList, runSelectReturningOne, etc);
execute using runBeamPostgres connection.

For example, to build a query, we can use all_ to introduce all entries of a table together with guard_ to filter the results:

query2 label = runSelectReturningList $ select $ do
  aProduct <- all_ warehouseDb.product
  guard_ (aProduct.label ==. val_ label)
  pure (aProduct.label, aProduct.description)

filter_ is built on top of guard_ and allows us to use the in_ clause:

query3 labels =
  runSelectReturningList
    $ select
    $ filter_ (\p -> p.label `in_` predicate)
    $ all_ warehouseDb.product
 where
  predicate = val_ <$> labels

Note that we use val_ to “lift” haskell values into “sql query land”.

How to use transactions

We use postgresql-simple for transactions:

insertWithTransaction :: Connection -> IO ()
insertWithTransaction connection = Simple.withTransaction connection $ do
  [newProduct] :: [Product] <-
    runBeamPostgres connection
      $ runInsertReturningList
      $ insert (warehouseDb.product)
      $ insertExpressions [Product default_ "Drywall Screws Set" (just_ "8000pcs")]

  [newCategory] <-
    runBeamPostgres connection
      $ runInsertReturningList
      $ insert (warehouseDb.category)
      $ insertExpressions [Category default_ "Drywall Screws"]

  runBeamPostgresDebug putStrLn connection
    $ runInsert
    $ insert (warehouseDb.product_category)
    $ insertValues [ProductCategory (pk newProduct) (pk newCategory)]

  runBeamPostgres connection
    $ runInsert
    $ insert (warehouseDb.warehouse)
    $ insertExpressions [Warehouse default_ (val_ (pk newProduct)) 10 currentTimestamp_ currentTimestamp_]

  putStrLn $ "Insert with transaction"

We use currentTimestamp_ to ask the database for the current time and pk to get the entity's primary key. For example, we pass pk newProduct into the ProductCategory mapping.

How to query using joins

There are various ways to get data from multiple tables using Beam.

For example, we can use related_ to get all entries of the given table referenced by the given primary key and leftJoin_ to introduce a table using a left join:

query1 quantity = runBeamPostgres connection
  $ runSelectReturningList
  $ select
  $ do
    warehouse <- all_ warehouseDb.warehouse
    aProduct <- related_ warehouseDb.product warehouse.product_id
    mapping <-
      leftJoin_
        (all_ warehouseDb.product_category)
        (\pc -> pc.product_id ==. primaryKey aProduct)
    category <-
      leftJoin_
        (all_ warehouseDb.category)
        (\c -> just_ (primaryKey c) ==. mapping.category_id)
    guard_ (warehouse.quantity >. quantity)
    pure (warehouse.quantity, aProduct.label, aProduct.description, category.label)

Which generates the following query:

SELECT 
  "t0"."quantity" AS "res0", 
  "t1"."label" AS "res1", 
  "t1"."description" AS "res2", 
  "t3"."label" AS "res3" 
FROM 
  "warehouse" AS "t0" 
  INNER JOIN "product" AS "t1" ON ("t0"."product_id") = ("t1"."id") 
  LEFT JOIN "product_category" AS "t2" ON ("t2"."product_id") = ("t1"."id") 
  LEFT JOIN "category" AS "t3" ON ("t3"."id") IS NOT DISTINCT 
FROM 
  ("t2"."category_id") 
WHERE 
  ("t0"."quantity") > (3)

We can also use the manyToMany_ construct to fetch sides of a many-to-many relationship.

productCategoryRelationship :: ManyToMany Postgres WarehouseDb ProductT CategoryT
productCategoryRelationship =
  manyToMany_ (warehouseDb.product_category) (.product_id) (.category_id)

query2 quantity = runBeamPostgres connection
  $ runSelectReturningList
  $ select
  $ do
    warehouse <- all_ warehouseDb.warehouse
    products <- related_ warehouseDb.product warehouse.product_id
    categories <- all_ warehouseDb.category
    (aProduct, category) <- productCategoryRelationship (pure products) (pure categories)
    guard_ (warehouse.quantity >. quantity)
    pure (warehouse.quantity, aProduct.label, aProduct.description, category.label)

Which generates the following query:

SELECT 
  "t0"."quantity" AS "res0", 
  "t1"."label" AS "res1", 
  "t1"."description" AS "res2", 
  "t2"."label" AS "res3" 
FROM 
  "warehouse" AS "t0" 
  INNER JOIN "product" AS "t1" ON ("t0"."product_id") = ("t1"."id") 
  CROSS JOIN "category" AS "t2" 
  INNER JOIN "product_category" AS "t3" ON (
    ("t3"."product_id") = ("t1"."id")
  ) 
  AND (
    ("t3"."category_id") = ("t2"."id")
  ) 
WHERE 
  ("t0"."quantity") > (3)

Errors

It’s not possible to write an invalid sql query, but this comes at a cost — compile-time errors.

For example, once we forgot to pass a parameter, and this resulted in:

Couldn't match expected type: Q Postgres
                                    WarehouseDb
                                    QBaseScope
                                    a0
                with actual type: Q Postgres
                                    WarehouseDb
                                    s0
                                    (ProductT (QExpr Postgres s0))
                                  -> Q Postgres WarehouseDb s0 (CategoryT (QExpr Postgres s0))
                                  -> Q Postgres
                                       WarehouseDb
                                       s0
                                       (ProductT (QExpr Postgres s0),
                                       CategoryT (QExpr Postgres s0))

Runtime sql errors are still there, re-exported from postgresql-simple. Review the relevant error section if you need a reminder.

errors :: Connection -> IO ()
errors connection = do
  insertDuplicateScrew
  insertDuplicateScrew
    `catch` (\err@SqlError{} -> putStrLn $ "Caught SQL Error: " <> displayException err)
 where
  insertDuplicateScrew =
    runBeamPostgres connection
      $ runInsert
      $ insert (warehouseDb.product)
      $ insertExpressions [Product default_ "Duplicate screw" nothing_]

Caught SQL Error: SqlError {sqlState = "23505", sqlExecStatus = FatalError, sqlErrorMsg = "duplicate key value violates unique constraint \"product_label_key\"", sqlErrorDetail = "Key (label)=(Duplicate screw) already exists.", sqlErrorHint = ""}

Resources

Beam has you covered — it comes with an overview, quick-start guide, tutorial, user guide, and hackage docs.

Spoiler alert: Beam is likely the best-documented library reviewed in this tutorial.

Migrations

The beam-migrate package provides a migrations framework.

“The beam-migrate tool can generate a beam schema from a pre-existing database, manage migrations for several production databases, automatically generate migrations between two schemas, and much more.”

In summary

beam states that if the query compiles, it will generate proper code. Beam uses the GHC Haskell type system and nothing else — no Template Haskell. You don’t have to write raw sql or sql like code. After defining some boilerplate, you write and compose queries in a straightforward Haskell style and get valid SQL.

Regarding complexity, let the types do the talking:

 manyToMany_
  :: ( Database be db, Table joinThrough
     , Table left, Table right
     , Sql92SelectSanityCheck syntax
     , IsSql92SelectSyntax syntax
     , SqlEq (QExpr (Sql92SelectExpressionSyntax syntax) s) (PrimaryKey left (QExpr (Sql92SelectExpressionSyntax syntax) s))
     , SqlEq (QExpr (Sql92SelectExpressionSyntax syntax) s) (PrimaryKey right (QExpr (Sql92SelectExpressionSyntax syntax) s)) )
  => DatabaseEntity be db (TableEntity joinThrough)
  -> (joinThrough (QExpr (Sql92SelectExpressionSyntax syntax) s) -> PrimaryKey left (QExpr (Sql92SelectExpressionSyntax syntax) s))
  -> (joinThrough (QExpr (Sql92SelectExpressionSyntax syntax) s) -> PrimaryKey right (QExpr (Sql92SelectExpressionSyntax syntax) s))
  -> Q syntax db s (left (QExpr (Sql92SelectExpressionSyntax syntax) s)) -> Q syntax db s (right (QExpr (Sql92SelectExpressionSyntax syntax) s))
  -> Q syntax db s (left (QExpr (Sql92SelectExpressionSyntax syntax) s), right (QExpr (Sql92SelectExpressionSyntax syntax) s))

`squeal`

Okay, what if we did something quite similar but quite different?

Squeal “is a type-safe embedding of PostgreSQL in Haskell”, which means “that Squeal embeds both SQL terms and SQL types into Haskell at the term and type levels respectively. This leads to a very high level of type-safety”.

Install squeal-postgresql (0.9.1.3 released in 2023) and generics-sop, which the library uses for generic encodings of Haskell tuples and records.

Enable: DataKinds, GADTs, and OverloadedLabels

💡 (It’s not very important, but) We assume you’ve seen the part on postgresql-simple, which covers the same topics but at a slower pace.

How to connect to a database

We pass libpq connection string (e.g., "host=localhost port=5432 user=postgres dbname=warehouse password=password") to withConnection:

withConnection Hardcoded.connectionString $ 
  doFoo 
    & pqThen doBar
    & pqThen doBaz

We can also create a connection pool using createConnectionPool and use the pool with usingConnectionPool.

How to define tables

First, we define table columns and constraints:

type ProductColumns =
  '[ "id" ::: 'Def :=> 'NotNull 'PGint4
   , "label" ::: 'NoDef :=> 'NotNull 'PGtext
   , "description" ::: 'NoDef :=> 'Null 'PGtext
   ]

type ProductConstraints = '["pk_product" ::: 'PrimaryKey '["id"]]

'Def means that DEFAULT is available for inserts and updates, 'NoDef — unavailable. We specify nullability with 'NotNull and 'Null and the primary key with 'PrimaryKey.

We use the ::: type operators to pair a Symbol with schema types, constraints, column types, etc. We use :=> to specify constraints as well as optionality.

All the other tables look pretty similar (with additional 'ForeignKey constraints here and there); see the repo for the rest of the boilerplate.

Then, we define a schema:

type Schema =
  '[ "product" ::: 'Table (ProductConstraints :=> ProductColumns)
   , "category" ::: 'Table (CategoryConstraints :=> CategoryColumns)
   , "product_category" ::: 'Table (ProductCategoryConstraints :=> ProductCategoryColumns)
   , "warehouse" ::: 'Table (WarehouseConstraints :=> WarehouseColumns)
   ]

type DB = Public Schema

We use generics to convert between Haskell and PostgreSQL values:

import qualified Generics.SOP as SOP
import qualified GHC.Generics as GHC

data BasicProduct = BasicProduct {label :: Text, description :: Maybe Text}
  deriving stock (Show, GHC.Generic)
  deriving anyclass (SOP.Generic, SOP.HasDatatypeInfo)

The SOP.Generic and SOP.HasDatatypeInfo instances allow us to encode and decode BasicProducts.

How to modify data

We can execute raw statements:

cleanUp :: PQ DB DB IO ()
cleanUp =
  execute_ teardown
 where
  teardown :: Statement db () ()
  teardown = manipulation $ UnsafeManipulation "truncate warehouse, product_category, product, category"

Manipulation represents update, insert, and delete statements.

We can specify the schema changes by using concrete PQ; when the schema doesn't change, we can use MonadPQ constraint (e.g., cleanUp :: (MonadPQ DB m) => m ()). In the end, we’ll turn either into IO:

withConnection Hardcoded.connectionString
  $ cleanUp

Let’s insert a product:

insertProduct :: Statement DB BasicProduct ()
insertProduct =
  manipulation
    $ insertInto_
      #product
      (Values_ (Default `as` #id :* Set (param @1) `as` #label :* Set (param @2) `as` #description))

A Statement is either a Manipulation or a Query that can be run in a MonadPQ.

We use manipulation and insertInto_ to construct an insert. We pass a table and what to insert. Values_ describes a single n-ary product, where we must match all the columns. We can use Default value for id and set the rest using relevant parameters.

And then, we use executePrepared_ to run a statement that returns nothing. The function prepares the statement and runs it on each element.

insertStuff :: (MonadPQ DB m) => m ()
insertStuff = do
  executePrepared_
    insertProduct
    [ BasicProduct "Wood Screw Kit 1" (Just "245-pieces")
    , BasicProduct "Wood Screw Kit 2" Nothing
    ]

insertInto_ is a specialized version of insertInto with OnConflictDoRaise (what to do in case of conflict) and no ReturningClause (what to return). ReturningClause returns a value based on each row; for example, we can use it to return the created id:

insertCategory :: Statement DB Category (Only Int32)
insertCategory =
  manipulation
    $ insertInto
      #category
      (Values_ (Default `as` #id :* Set (param @1) `as` #label))
      OnConflictDoRaise
      (Returning_ (#id `as` #fromOnly))

Note that we have to use Only and #fromOnly, because we can’t use primitive types (because they don’t have named labels that the library relies on).

This time we have to use executePrepared, which returns a list of results:

insertStuff :: (MonadPQ DB m, MonadIO m) => m ()
insertStuff = do
  result :: [Result (Only Int32)] <-
    executePrepared insertCategory [Category "Screws", Category "Wood Screws", Category "Concrete Screws"]
  rows <- traverse getRows result
  liftIO $ putStrLn $ "Inserted categories: " <> show rows

We use getRows to get all rows from a Result.

How to query data

To retrieve data, we also write Statements, this time using query and select_:

query1 :: Statement DB () BasicProduct
query1 =
  query
    $ select_
      (#product ! #label :* #product ! #description)
      (from (table #product))

The query returns all the products from the table.

💡 Note that we can use printSQL to print statements and see what sql queries get executed.

💡 We can’t return tuples or primitive types because they don't have named fields. You must define a new datatype and derive Squeal typeclasses to return something new.

If you try using tuples, you get an error:

The type `(Text, Text)' is not a record type.
It has no labelled fields.

And then we execute the query:

insertStuff :: (MonadPQ DB m, MonadIO m) => m ()
insertStuff = do
  result1 <- execute query1
  rows1 <- getRows result1
  liftIO $ putStrLn $ "Query 1: " <> show rows1

We can select specific fields and narrow down the results:

query2 :: Statement DB (Only Text) BasicProduct
query2 =
  query
    $ select_
      (#product ! #label :* #product ! #description)
      (from (table #product) & where_ (#product ! #label .== (param @1)))

We use where_ to filter the rows and .== to compare for equality.

This time, we use executeParams to pass the parameters into the statement:

queryData :: PQ DB DB IO ()
queryData = do
  result2 <- executeParams query2 (Only "Wood Screw Kit 1") >>= getRows
  liftIO $ putStrLn $ "Query 2: " <> show result2

We can also use in_:

query3 labels =
  query
    $ select_
      (#product ! #label :* #product ! #description)
      (from (table #product) & where_ (#product ! #label `in_` labels))

do
  (result3 :: [BasicProduct]) <- execute (query3 ["Wood Screw Kit 2", "Wood Screw Kit 3"]) >>= getRows
  liftIO $ putStrLn $ "Query 3: " <> show result3

How to use transactions

We can wrap computation in transactionally_:

insertWithTransaction :: PQ DB DB IO ()
insertWithTransaction =
  transactionally_
    ( do
        result1 <- executePrepared insertProduct [BasicProduct "Drywall Screws Set" (Just "8000pcs")]
        productIds <- join <$> traverse getRows result1

        result2 <- executePrepared insertCategory [Category "Drywall Screws"]
        categoryIds <- join <$> traverse getRows result2

        case (productIds, categoryIds) of
          ([Only productId], [Only categoryId]) -> do
            executePrepared_ insertProductCategory [(productId, categoryId)]
            executePrepared_ insertListing [(productId, 10)]
          _ ->
            throwM $ userError "Failed to insert product/category"
    )
    >> liftIO (putStrLn $ "Insert with transaction")

In case of exception, it rollbacks the transaction and rethrows the exception.

How to query using joins

We use innerJoin and leftOuterJoin to join the tables:

query1 :: Statement DB (Only Int32) Listing
query1 =
  query
    $ select_
      (#w ! #quantity `as` #quantity :* #p ! #label `as` #label :* #p ! #description `as` #description :* #c ! #label `as` #category)
      ( from
          ( table (#warehouse `as` #w)
              & innerJoin
                (table (#product `as` #p))
                (#w ! #product_id .== #p ! #id)
              & leftOuterJoin
                (table (#product_category `as` #pc))
                (#pc ! #product_id .== #p ! #id)
              & leftOuterJoin
                (table (#category `as` #c))
                (#c ! #id .== #pc ! #category_id)
          )
          & where_ (#w ! #quantity .> (param @1))
      )

Which generates:

SELECT "w"."quantity"    AS "quantity",
       "p"."label"       AS "label",
       "p"."description" AS "description",
       "c"."label"       AS "category"
FROM   "warehouse" AS "w"
       inner join "product" AS "p"
               ON ( "w"."product_id" = "p"."id" )
       left outer join "product_category" AS "pc"
                    ON ( "pc"."product_id" = "p"."id" )
       left outer join "category" AS "c"
                    ON ( "c"."id" = "pc"."category_id" )
WHERE  ( "w"."quantity" > ( $1 :: int4 ) )

Errors

If you forget or mistype anything, most of the time, the error messages are rarely simple.

Sometimes, they overwhelm:

_ :: NP
(Aliased
(Optional
(Expression
'Ungrouped
'[]
'[]
'["public"
::: '["product" ::: 'Table (ProductConstraints :=> ProductColumns),
"category" ::: 'Table (CategoryConstraints :=> CategoryColumns),
"product_category"
::: 'Table (ProductCategoryConstraints :=> ProductCategoryColumns),
"warehouse"
::: 'Table (WarehouseConstraints :=> WarehouseColumns)]]
'[ 'NotNull 'PGtext, 'Null 'PGtext]
from0)))
'["description" ::: ('NoDef :=> 'Null 'PGtext)]
Where: ‘from0’ is an ambiguous type variable

Sometimes, they leak:

Couldn't match type: TupleOf (TupleCodeOf Text (SOP.Code Text))
               with: null10 'PGtext : xs0

Ambiguous type variable ‘y0’ arising from a use of ‘manipulation’
prevents the constraint ‘(SOP.Generic y0)’ from being solved.

Couldn't match type: '["description"
::: ('NoDef :=> 'Null 'PGtext)]
with: '[]

Sometimes, they really leak:

Couldn't match type: records-sop-0.1.1.1:Generics.SOP.Record.ExtractTypesFromRecordCode
                       (records-sop-0.1.1.1:Generics.SOP.Record.ToRecordCode_Datatype
                          y (SOP.DatatypeInfoOf y) (SOP.Code y))
                with: records-sop-0.1.1.1:Generics.SOP.Record.GetSingleton
                       (SOP.Code y)
  arising from a use of ‘manipulation’

But when it comes to runtime SQL errors, the library provides a convenient SquealException for exceptions that Squeal can throw and a nice API for working with them built on top of exceptions. For example, we can use catchSqueal:

errors :: PQ DB DB IO ()
errors = do
  insertDuplicateScrew
  insertDuplicateScrew
    `catchSqueal` (\err -> liftIO $ putStrLn $ "Caught Squeal/SQL Error: " <> displayException err)
 where
  insertDuplicateScrew = executePrepared_ insertProduct [BasicProduct "Duplicate screw" Nothing]
  insertProduct =
    manipulation
      $ insertInto_
        #product
        (Values_ (Default `as` #id :* Set (param @1) `as` #label :* Set (param @2) `as` #description))

Resources

The library comes with a quickstart and Core Concepts Handbook.

Migrations

The library has a Migration module to change the database schema over time. They support linear, pure or impure, one-way or rewindable migrations.

In summary

Squeal is another type-safe postgres library not suitable for beginners. You should be comfortable working on the type level, reading generic-related errors, etc. The library uses generic encodings (generics-sop) of records/tuples, which keep getting into the error messages.

`opaleye`

Okay, what if we did something quite similar but quite different?

Opaleye is “an SQL-generating DSL targeting PostgreSQL. Allows Postgres queries to be written within Haskell in a typesafe and composable fashion”.

Install opaleye (0.10.1.0 released in 2023) and product-profunctors, which the library uses under the hood.

opaleye is built on top of postgresql-simple, which is used for connection management, transaction support, serialization, and deserialization.

💡 We assume you’ve seen the part on postgresql-simple.

How to connect to a database

We use postgresql-simple straight away. Reminder:

connectionInfo :: ConnectInfo
connectionInfo =
  defaultConnectInfo
    { connectHost = Hardcoded.host
    , connectDatabase = Hardcoded.database
    , connectUser = Hardcoded.user
    , connectPassword = Hardcoded.password
    }

Simple.withConnect connectionInfo $ \connection -> do
    doFoo connection
    doBar connection

How to define tables

We define a table using the table function — specify the table name and the type of fields.

warehouseTable ::
  Table
    (Maybe (Field SqlInt4), ProductIdField, Field SqlInt4, Maybe (Field SqlTimestamp), Maybe (Field SqlTimestamp))
    (Field SqlInt4, ProductIdField, Field SqlInt4, Field SqlTimestamp, Field SqlTimestamp)
warehouseTable =
  table "warehouse"
    $ p5
      ( tableField "id"
      , pProductId (ProductId (tableField "product_id"))
      , tableField "quantity"
      , tableField "created"
      , tableField "modified"
      )

Table takes write fields and view fields; for example, the first parameter (id) is auto-generated, so it’s optional on write (so we specify Maybe (Field SqlInt4)) but always there on view/read (so we specify Field SqlInt4).

p5 is a tiny glue function from product-profunctors; the number corresponds to the tuple arity (number of columns). We don’t need to know more than that.

tableField infers a required or an optional field depending on the write type.

(We’ll cover ProductId a bit later)

We can also use records instead of tuples; for example, for product:

data Product' a b c = Product {pId :: a, pLabel :: b, pDescription :: c}
type Product = Product' ProductId Text (Maybe Text)
deriving instance Show Product

type ProductFieldWrite = Product' (ProductId' (Maybe (Field SqlInt4))) (Field SqlText) (FieldNullable SqlText)
type ProductField = Product' (ProductIdField) (Field SqlText) (FieldNullable SqlText)

Note that we prefix field names because we’ll have some derivable code that can’t handle duplicate record fields.

Product' is polymorphic in all its fields. We’ll use Product in “normal” code and ProductField when interacting with a database. Because id is optional on write, we distinguish between ProductFieldWrite and ProductField.

We indicate nullable fields with FieldNullable, which will be converted into Maybe when executed.

We need some typeclass instances, which we can get with Template Haskell:

$(makeAdaptorAndInstance "pProduct" ''Product')

💡 If you’d rather write these by hand, see Data.Profunctor.Product.TH.

And then, we define the table:

productTable :: Table ProductFieldWrite ProductField
productTable =
  table "product"
    $ pProduct
      Product
        { pId = pProductId $ ProductId $ tableField "id"
        , pLabel = tableField "label"
        , pDescription = tableField "description"
        }

Note that instead of pN, we use pProduct, which we just generated with TH.

The library’s basic tutorial suggests using newtypes for ids. For example, we use one for the product id:

newtype ProductId' a = ProductId a
$(makeAdaptorAndInstance "pProductId" ''ProductId')
type ProductId = ProductId' Int
deriving instance Show ProductId

type ProductIdField = ProductId' (Field SqlInt4)

See the repo for the rest of the boilerplate.

How to modify data

For raw queries, we can use postgresql-simple:

cleanUp :: Connection -> IO ()
cleanUp connection =
  void $ Simple.execute_ connection "truncate warehouse, product_category, product, category"

Otherwise, we create an insert by using Insert:

insert1 :: Insert [ProductId]
insert1 =
  Insert
    { iTable = productTable
    , iRows =
        [ Product (ProductId Nothing) "Wood Screw Kit 1" (maybeToNullable Nothing)
        , Product (ProductId Nothing) "Wood Screw Kit 2" (maybeToNullable (Just "245-pieces"))
        ]
    , iReturning = rReturning (\(p :: ProductField) -> p.pId)
    , iOnConflict = Nothing
    }

We specify the tables, rows to be inserted, conflict-handling strategy, and what to return. In this case, we return product id using rReturning.

And then we run the insert with runInsert:

insertStuff :: Connection -> IO ()
insertStuff connection = do
  result1 <- runInsert connection insert1
  putStrLn $ "Inserted products: " <> show result1

If we want to return the number of affected rows, we can use rCount:

insert2 :: Insert Int64
insert2 =
  Insert
    { iTable = categoryTable
    , iRows = [Category (CategoryId 1) "Screws", Category (CategoryId 2) "Wood Screws", Category (CategoryId 3) "Concrete Screws"]
    , iReturning = rCount
    , iOnConflict = Nothing
    }

How to query data

The basic select is simple:

selectProduct :: Select ProductField
selectProduct = selectTable productTable

💡 Note that we can use showSql to print Select and see what sql queries get executed.

We run a select with runSelect:

queryData :: Connection -> IO ()
queryData connection = do
  result1 :: [Product] <- runSelect connection selectProduct
  putStrLn $ "Query 1: " <> show result1

runSelect “converts a "record" of Opaleye fields to a list of "records" of Haskell values.” We must specify the return type ([Product]) to help type inference.

We can select specific fields and narrow down the results:

select2 :: Select (Field SqlText, FieldNullable SqlText)
select2 = do
  (Product _ aLabel description) <- selectProduct
  where_ (aLabel .== "Wood Screw Kit 2")
  pure (aLabel, description)

result2 :: [(Text, Maybe Text)] <- runSelect connection select2

We use where_ to filter the rows and .== to compare for equality. We can also use in_:

select3 :: Select (Field SqlText)
select3 = do
  p <- selectProduct
  where_ $ in_ ["Wood Screw Kit 2", "Wood Screw Kit 3"] p.pLabel
  pure (p.pLabel)

result3 :: [Text] <- runSelect connection select3

How to use transactions

We use postgresql-simple for transactions:

insertWithTransaction :: Connection -> IO ()
insertWithTransaction connection = Simple.withTransaction connection $ do
  [newProduct] :: [ProductId] <-
    runInsert connection
      $ Insert
        { iTable = productTable
        , iRows = [Product (ProductId Nothing) "Drywall Screws Set" (maybeToNullable $ Just "8000pcs")]
        , iReturning = rReturning (\(p :: ProductField) -> p.pId)
        , iOnConflict = Nothing
        }

  [newCategory] :: [CategoryId] <-
    runInsert connection
      $ Insert
        { iTable = categoryTable
        , iRows = [Category (CategoryId 123) "Drywall Screws"]
        , iReturning = rReturning (.cId)
        , iOnConflict = Nothing
        }

  void
    $ runInsert connection
    $ Insert
      { iTable = productCategoryTable
      , iRows = [Mapping (toFields newProduct) (toFields newCategory)]
      , iReturning = rCount
      , iOnConflict = Nothing
      }

  void
    $ runInsert connection
    $ Insert
      { iTable = warehouseTable
      , iRows = [(Nothing, (toFields newProduct), 10, Nothing, Nothing)]
      , iReturning = rCount
      , iOnConflict = Nothing
      }

  putStrLn $ "Insert with transaction"

How to query using joins

Opaleye provides a couple of APIs for joins. They recommend using where_ directly for inner joins and optional for left/right joins. Which gives us something like this:

join :: Select (Field SqlInt4, Field SqlText, FieldNullable SqlText, MaybeFields (Field SqlText))
join = do
  (_, wProductId, quantity, _, _) <- selectTable warehouseTable
  p <- selectTable productTable
  mpc <- optional $ do
    pc <- selectTable productCategoryTable
    where_ $ pc.productId .=== p.pId
    pure pc
  mc <- optional $ do
    c <- selectTable categoryTable
    where_ $ isJustAnd mpc $ \pc -> c.cId .=== pc.categoryId
    pure c

  where_ $ wProductId .=== p.pId
  where_ $ quantity .> 3

  let category = cLabel <$> mc
  pure (quantity, p.pLabel, p.pDescription, category)

💡 Note that we use isJustAnd that will be added to Opaleye in the future.
isJustAnd :: MaybeFields a -> (a -> Field SqlBool) -> Field SqlBool
isJustAnd ma cond = matchMaybe ma $ \case
  Nothing -> sqlBool False
  Just a -> cond a

Which generates:

SELECT
"quantity2_1" as "result1_7",
"label1_2" as "result2_7",
"description2_2" as "result3_7",
NOT (("rebind0_6") IS NULL) as "result4_7",
"label1_5" as "result5_7"
FROM (SELECT
      *
      FROM (SELECT *
            FROM
            (SELECT *
             FROM
             (SELECT
              *
              FROM (SELECT
                    "id" as "id0_1",
                    "product_id" as "product_id1_1",
                    "quantity" as "quantity2_1",
                    "created" as "created3_1",
                    "modified" as "modified4_1"
                    FROM "warehouse" as "T1") as "T1",
                   LATERAL
                   (SELECT
                    "id" as "id0_2",
                    "label" as "label1_2",
                    "description" as "description2_2"
                    FROM "product" as "T1") as "T2") as "T1"
             LEFT OUTER JOIN
             LATERAL
             (SELECT
              TRUE as "rebind0_4",
              *
              FROM (SELECT
                    *
                    FROM (SELECT
                          "product_id" as "product_id0_3",
                          "category_id" as "category_id1_3"
                          FROM "product_category" as "T1") as "T1"
                    WHERE (("product_id0_3") = ("id0_2"))) as "T1") as "T2"
             ON
             TRUE) as "T1"
            LEFT OUTER JOIN
            LATERAL
            (SELECT
             TRUE as "rebind0_6",
             *
             FROM (SELECT
                   *
                   FROM (SELECT
                         "id" as "id0_5",
                         "label" as "label1_5"
                         FROM "category" as "T1") as "T1"
                   WHERE (CASE WHEN NOT (("rebind0_4") IS NULL) THEN ("id0_5") = ("category_id1_3") ELSE CAST(FALSE AS boolean) END)) as "T1") as "T2"
            ON
            TRUE) as "T1"
      WHERE (("quantity2_1") > (CAST(3 AS integer))) AND (("product_id1_1") = ("id0_2"))) as "T1"

Errors

Once again, type-safety and query validation equal compilation errors.

But because there isn’t much “type-level magic”, we only need to occasionally help the compiler with type inference. And it’s mainly about input and return types — not intermediate/internal library structures. For example, [Product] in this snippet:

result :: [Product] <- runSelect connection selectProduct
 where
  selectProduct = selectTable productTable

Sometimes, if you don’t specify enough types, profunctors show up:

Ambiguous type variable ‘haskells0’ arising from a use of ‘runSelect’
prevents the constraint ‘(Default
                            FromFields fields0 Product)’ from being solved.
Probable fix: use a type annotation to specify what ‘haskells0’ should be.

Runtime sql errors are again from postgresql-simple. Review the relevant error section if you need a reminder.

errors :: Connection -> IO ()
errors connection = do
  insertDuplicateScrew
  insertDuplicateScrew
    `catch` (\err@SqlError{} -> putStrLn $ "Caught SQL Error: " <> displayException err)
 where
  insertDuplicateScrew =
    void
      $ runInsert connection
      $ Insert
        { iTable = productTable
        , iRows = [Product (ProductId Nothing) "Duplicate screw" (maybeToNullable Nothing)]
        , iReturning = rCount
        , iOnConflict = Nothing
        }

Resources

There are a couple of basic tutorials in the repo and some external ones.

One thing to remember: sometimes, the library provides multiple ways of doing things (for example, left joins using optional vs. deprecated leftJoin or monadic vs. arrow syntax), and documentation/tutorials can do it one way or even deprecated way.

Migrations

Opaleye assumes a database already exists — no support for migrations or creating tables and databases.

In summary

Opaleye allows us to define tables and write type-safe postgres queries using Haskell code.

The library uses product-profunctors and typeclasses. Both only come up in copy-pasteable boilerplate and when you under-specify the return types. No deep knowledge is required.

`rel8`

Okay, what if we did something quite similar but quite different?

Rel8 “is a Haskell library for interacting with PostgreSQL databases”, which aims to be concise, inferrable, and familiar.

For the database connection, instead of postgresql-simple, rel8 uses Hasql.

Install rel8 (1.4.1.0 released in 2023), hasql, and hasql-transaction.

We bring back the TypeFamilies extension and (in case you haven’t already) DuplicateRecordFields. The latter is required to disambiguate the record fields when working with inserts, updates, and deletes…

💡 We assume you’ve seen the parts on postgresql-simple, hasql, and opaleye.

How to connect to a database

We use Hasql. Reminder:

Right connection <- getConnection

getConnection :: IO (Either ConnectionError Connection)
getConnection =
  acquire $ settings Hardcoded.host Hardcoded.portNumber Hardcoded.user Hardcoded.password Hardcoded.database

How to define tables

First, we describe the structural mapping of the tables. Take for instance Product:

newtype ProductId = ProductId Int64
  deriving newtype (DBEq, DBType, Eq, Show)

data Product f = Product
  { id :: Column f ProductId
  , label :: Column f Text
  , description :: Column f (Maybe Text)
  }
  deriving (Generic)
  deriving anyclass (Rel8able)

deriving stock instance (f ~ Result) => Show (Product f)

We define fields with Column and derive the Rel8able instance. We also declare a newtype for product id with a few instances.

Imagine that the last line is just deriving (Show).

Then, we describe a TableSchema for each table. The relevant table looks like this:

productSchema :: TableSchema (Product Name)
productSchema =
  TableSchema
    { name = "product"
    , schema = Nothing
    , columns =
        Product
          { id = "id"
          , label = "label"
          , description = "description"
          }
    }

Note that defining columns looks repetitive — we can use some generics machinery to get that information from the Rel8able:

productSchema :: TableSchema (Product Name)
productSchema =
  TableSchema
    { name = "product"
    , schema = Nothing
    , columns = namesFromLabels @(Product Name)
    }

💡 namesFromLabels generates a table schema where every column name corresponds precisely to the field's name. Alternatively, we can use namesFromLabelsWith.

See the repo for the rest of the boilerplate.

How to modify data

For raw queries, we can use Hasql:

cleanUp :: Connection -> IO (Either QueryError ())
cleanUp connection = run cleanUpSession connection
 where
  cleanUpSession = statement () $ Statement rawSql E.noParams D.noResult True
  rawSql = "truncate warehouse, product_category, product, category"

Otherwise, we create Insert:

insert1 :: Statement () [ProductId]
insert1 =
  insert
    $ Insert
      { into = productSchema
      , rows =
          values
            [ Product unsafeDefault "Wood Screw Kit 1" null
            , Product unsafeDefault "Wood Screw Kit 2" (lit $ Just "245-pieces")
            ]
      , returning = Projection (.id)
      , onConflict = Abort
      }

We’ve seen this in Opaleye’s insert: the table, rows to insert, conflict-handling strategy, and what to return.

We use unsafeDefault for sql DEFAULT, lit to turn Haskell values into expressions, and values to construct a query out of the given rows.

💡 Note that unsafeDefault is named unsafe for a reason; see the docs.

And run this like any other Hasql statement:

result1 <- run (statement () insert1) connection

If we want to return the number of affected rows, we can use NumberOfRowsAffected:

Insert
  { into = categorySchema
  , rows =
      values
        [ Category unsafeDefault "Screws"
        , Category unsafeDefault "Wood Screws"
        , Category unsafeDefault "Concrete Screws"
        ]
  , returning = NumberOfRowsAffected
  , onConflict = Abort
  }

How to query data

We build select statements using Query. We select all rows from a table using each and turn (run) the query into Statement using select:

select1 :: Statement () [Product Result]
select1 = select $ each productSchema

💡 Note that we can use showQuery to print sql queries that will be executed.

And once again we run the statement:

result1 <- run (statement () select1) connection

We can select specific fields and narrow down the results:

select2 :: Statement () [(Text, Maybe Text)]
select2 = select $ do
  p <- each productSchema
  where_ $ p.label ==. "Wood Screw Kit 2"
  pure (p.label, p.description)

We use where_ to filter the rows and ==. to compare for equality. We can also use in_:

select3 :: Statement () [Text]
select3 = select $ do
  p <- each productSchema
  where_ $ p.label `in_` ["Wood Screw Kit 2", "Wood Screw Kit 3"]
  pure p.label

Note that the order of parameters is different from Opaleye.

How to use transactions

We use Hasql for transactions:

insertWithTransaction :: Connection -> IO ()
insertWithTransaction connection = do
  result <- run (transaction Serializable Write insertAll) connection
  putStrLn $ "Insert with transaction: " <> show result
 where
  insertAll = do
    productIds <-
      Transaction.statement ()
        $ insert
        $ Insert
          { into = productSchema
          , rows = values [Product unsafeDefault "Drywall Screws Set" (lit $ Just "8000pcs")]
          , returning = Projection (.id)
          , onConflict = Abort
          }

    -- insert category
    -- insert mapping
    -- insert warehouse listing

How to query using joins

Rel8 doesn’t have a specific join operation — we use where_ (or filter) to filter the results and optional to do what outer joins do.

queryWithJoins :: Connection -> IO ()
queryWithJoins connection = do
  result1 <- run (statement () join) connection
  putStrLn $ "Query with join: " <> show result1
 where
  join :: Statement () [(Int32, Text, Maybe Text, Maybe Text)]
  join = select joinQuery

  joinQuery = do
    w <- each warehouseSchema
    p <- productsInWarehouse w
    pc <- optional $ mappingsForProduct p
    c <- traverseMaybeTable categoriesForMapping pc
    where_ $ w.quantity >. 3
    let category = maybeTable null (nullify . (.label)) c
    pure (w.quantity, p.label, p.description, category)

  productsInWarehouse :: Warehouse Expr -> Query (Product Expr)
  productsInWarehouse w =
    each productSchema >>= filter (\p -> p.id ==. w.product_id)

  mappingsForProduct :: Product Expr -> Query (ProductCategory Expr)
  mappingsForProduct p = do
    each productCategorySchema >>= filter (\pc -> pc.product_id ==. p.id)

  categoriesForMapping :: ProductCategory Expr -> Query (Category Expr)
  categoriesForMapping pc =
    each categorySchema >>= filter (\c -> c.id ==. pc.category_id)

We extract “each join” into a specialized function to make the code cleaner (according to the Rel8 tutorials). We use optional and traverseMaybeTable to account for the partiality of queries. MaybeTable results from an outer join, which we unwrap with maybeTable.

filter is an alternative way to write where clauses.

The generated query:

SELECT
CAST("quantity2_1" AS int4) as "_1",
CAST("label1_3" AS text) as "_2",
CAST("description2_3" AS text) as "_3",
CAST(CASE WHEN ("rebind0_8") IS NULL THEN CAST(NULL AS text) ELSE "label1_12" END AS text) as "_4"
FROM (SELECT
      *
      FROM (SELECT *
            FROM
            (SELECT *
             FROM
             (SELECT
              *
              FROM (SELECT
                    "id" as "id0_1",
                    "product_id" as "product_id1_1",
                    "quantity" as "quantity2_1",
                    "created" as "created3_1",
                    "modified" as "modified4_1"
                    FROM "warehouse" as "T1") as "T1",
                   LATERAL
                   (SELECT
                    "id" as "id0_3",
                    "label" as "label1_3",
                    "description" as "description2_3"
                    FROM "product" as "T1") as "T2"
              WHERE (("id0_3") = ("product_id1_1"))) as "T1"
             LEFT OUTER JOIN
             LATERAL
             (SELECT
              TRUE as "rebind0_8",
              *
              FROM (SELECT
                    *
                    FROM (SELECT
                          "product_id" as "product_id0_6",
                          "category_id" as "category_id1_6"
                          FROM "product_category" as "T1") as "T1"
                    WHERE (("product_id0_6") = ("id0_3"))) as "T1") as "T2"
             ON
             TRUE) as "T1"
            LEFT OUTER JOIN
            LATERAL
            (SELECT
             TRUE as "rebind0_14",
             *
             FROM (SELECT
                   *
                   FROM (SELECT
                         0) as "T1",
                        LATERAL
                        (SELECT
                         "id" as "id0_12",
                         "label" as "label1_12"
                         FROM "category" as "T1") as "T2"
                   WHERE (("id0_12") = ("category_id1_6")) AND (("rebind0_8") IS NOT NULL)) as "T1") as "T2"
            ON
            TRUE) as "T1"
      WHERE (("quantity2_1") > (CAST(3 AS int4))) AND (((("rebind0_14") IS NULL) AND (("rebind0_8") IS NULL)) OR ((("rebind0_14") = ("rebind0_8")) AND (COALESCE(("rebind0_14") = ("rebind0_8"),FALSE))))) as "T1"

Which looks similar to the relevant Opaleye query in the previous section.

Errors

On top of type-safety, according to the docs, “Rel8 aims to have excellent and predictable type inference”. And they deliver — type inference rarely needs any guidance, and the compilation errors are pretty good.

Although it’s possible to introduce runtime errors using unsafe operations like unsafeDefault, the name is explicit, well documented, and has proper alternatives.

Runtime errors come from Hasql — all error-reporting is explicit and is presented using Either. As a reminder, violating the constraint returns a familiar error:

errors :: Connection -> IO ()
errors connection = do
  Left failure <-
    run insertDuplicateScrew connection
      >> run insertDuplicateScrew connection
  putStrLn $ "Constraint violation (Left): " <> show failure
 where
  insertDuplicateScrew =
    statement ()
      $ insert
      $ Insert
        { into = productSchema
        , rows = values [Product unsafeDefault "Duplicate screw" null]
        , returning = NumberOfRowsAffected
        , onConflict = Abort
        }

Constraint violation (Left): QueryError "INSERT INTO \"product\" (\"id\",\n \"label\",\n \"description\")\nVALUES\n(DEFAULT,CAST(E'Duplicate screw' AS text),CAST(NULL AS text))" [] (ResultError (ServerError "23505" "duplicate key value violates unique constraint \"product_label_key\"" (Just "Key (label)=(Duplicate screw) already exists.") Nothing Nothing))

Resources

Rel8 has the Getting Started tutorial, the Concepts documentation, the cookbook, and good API docs. This would have been one of the best coverages, but unfortunately, some basic snippets (like running selects or constructing inserts) aren’t valid anymore.

Also, you have to keep in mind hasql.

Migrations

Rel8 assumes a database already exists — no support for migrations or creating tables and databases.

In summary

Rel8 also allows us to write type-safe postgres queries using concise, inferrable, and familiar Haskell code. It builds on top of opaleye and hasql, and you must be somewhat familiar with the latter.

`selda`

Okay, what if we did something quite similar but quite different?

Selda “is a Haskell library for interacting with SQL-based relational databases” (PostgreSQL or SQLite). “The library was inspired by LINQ and Opaleye.”

Install selda (0.5.2.0 released in 2022) and selda-postgresql.

Enable OverloadedLabels.

How to connect to a database

Create connection info:

connectionInfo :: PGConnectInfo
connectionInfo =
  PGConnectInfo
    { pgHost = Hardcoded.host
    , pgPort = Hardcoded.portNumber
    , pgDatabase = Hardcoded.database
    , pgUsername = Just Hardcoded.user
    , pgPassword = Just Hardcoded.password
    , pgSchema = Nothing
    }

And use it with withPostgreSQL:

withPostgreSQL connectionInfo $ do
  doFoo
  doBar

How to define tables

First, we declare normal types and derive SqlRow, for example, for product:

data Product = Product
  { id :: ID Product
  , label :: Text
  , description :: Maybe Text
  }
  deriving (Generic, Show)
  deriving anyclass (SqlRow)

Then, we use table to declare a table:

productTable :: Table Product
productTable = table "product" [#id :- autoPrimary]

We specify constraints by linking selectors of the table to the definitions. We use autoPrimary for auto-incrementing primary keys, primary for regular primary keys, and foreignKey for foreign keys:

mappingTable :: Table ProductCategory
mappingTable =
  table
    "product_category"
    [ #product_id :- foreignKey productTable #id
    , #category_id :- foreignKey categoryTable #id
    ]

See the repo for the rest of the boilerplate.

How to modify data

We can use rawStm from Database.Selda.Unsafe to execute raw queries:

cleanUp :: SeldaM PG ()
cleanUp =
  rawStm "truncate warehouse, product_category, product, category"

SeldaM is an alias for SeldaT IO, SeldaT is a Selda computation — a concrete implementation (of MonadSelda) with Selda SQL capabilities.

At the end we’ll turn it into IO:

withPostgreSQL connectionInfo $ do
  cleanUp

To insert data, we can use insert_ that doesn’t return anything, insert that returns the number of inserted rows, and insertWithPK that returns the primary key of the last inserted row.

insertStuff :: SeldaM PG ()
insertStuff = do
  productId <-
    insertWithPK
      productTable
      [ Product def "Wood Screw Kit 1" (Just "245-pieces")
      , Product def "Wood Screw Kit 2" Nothing
      ]
  liftIO $ putStrLn $ "Inserted product with id: " <> show productId

  rows <-
    insert
      categoryTable
      [Category def "Screws", Category def "Wood Screws", Category def "Concrete Screws"]
  liftIO $ putStrLn $ "Inserted categories: " <> show rows

We use def when we want to use the default value, which is the case with ids.

How to query data

We can get all the rows from the given table using select:

  selectProduct :: Query t (Row t Product)
  selectProduct = select productTable

💡 Note that we can use compile from Database.Selda.Debug to print sql queries that will be executed.

And execute the query with query:

queryData :: SeldaT PG IO ()
queryData = do
  result1 <- query selectProduct
  liftIO $ putStrLn $ "Query 1: " <> show result1

🤷 Somehow, here, Selda didn’t want to read/parse back the ids it just inserted:

elephants-exe: [SELDA BUG] fromSql: RowID column with non-int value: SqlInt32...

If we change the type from ID Foo to Int32, the select works, but then insert with auto-incremental primary keys and other functionality doesn’t 🤷

So let’s ignore this for now because other queries work fine.

We can select specific fields and narrow down the results:

select2 :: Query t (Col t Text :*: Col t (Maybe Text))
select2 = do
  p <- selectProduct
  restrict (p ! #label .== "Wood Screw Kit 2")
  pure (p ! #label :*: p ! #description)

Query is parameterized over a scope parameter t, ensuring that queries are always well-scoped, but we don’t have to worry about it now.

We use ! with selectors to extract a column, restrict to filter the rows, and .== to compare for equality. A result is an inductive tuple — one or more values separated by the :*: data constructor.

We can also use isIn:

select3 = do
  p <- selectProduct
  restrict (p ! #label `isIn` ["Wood Screw Kit 2", "Wood Screw Kit 3"])
  pure (p ! #label)

How to use transactions

We use transaction:

insertWithTransaction :: SeldaT PG IO ()
insertWithTransaction = transaction $ do
  productId <- insertWithPK productTable [Product def "Drywall Screws Set" (Just "8000pcs")]
  categoryId <- insertWithPK categoryTable [Category def "Drywall Screws"]
  insert_ mappingTable [ProductCategory productId categoryId]
  insert_ warehouseTable [Warehouse def productId 10 def def]
  liftIO $ putStrLn $ "Insert with transaction"

How to query using joins

We use restrict and leftJoin to query with joins:

join :: Query s (Col s Int32 :*: (Col s Text :*: (Col s (Maybe Text) :*: Col s (Coalesce (Maybe Text)))))
join = do
  w <- select warehouseTable
  p <- select productTable
  restrict (w ! #product_id .== p ! #id)

  pc <- leftJoin (\pc -> pc ! #product_id .== p ! #id) (select mappingTable)
  c <- leftJoin (\c -> just (c ! #id) .== pc ? #category_id) (select categoryTable)

  pure (w ! #quantity :*: p ! #label :*: p ! #description :*: c ? #label)

We use ? to extract a column from the nullable row.

The generated query:

SELECT 
  "quantity_2", 
  "label_6", 
  "description_7", 
  "label_13_15" 
FROM 
  (
    SELECT 
      "id_12_14", 
      "label_13_15", 
      "category_id_9_11", 
      "label_6", 
      "description_7", 
      "quantity_2" 
    FROM 
      (
        SELECT 
          "product_id_8_10", 
          "category_id_9_11", 
          "id_5", 
          "label_6", 
          "description_7", 
          "quantity_2" 
        FROM 
          (
            SELECT 
              "id_5", 
              "label_6", 
              "description_7", 
              "product_id_1", 
              "quantity_2" 
            FROM 
              (
                SELECT 
                  "product_id" AS "product_id_1", 
                  "quantity" AS "quantity_2" 
                FROM 
                  "warehouse"
              ) AS q0, 
              (
                SELECT 
                  "id" AS "id_5", 
                  "label" AS "label_6", 
                  "description" AS "description_7" 
                FROM 
                  "product"
              ) AS q1 
            WHERE 
              ("product_id_1" = "id_5")
          ) AS q3 
          LEFT JOIN (
            SELECT 
              "product_id_8" AS "product_id_8_10", 
              "category_id_9" AS "category_id_9_11" 
            FROM 
              (
                SELECT 
                  "product_id" AS "product_id_8", 
                  "category_id" AS "category_id_9" 
                FROM 
                  "product_category"
              ) AS q2
          ) AS q4 ON "product_id_8_10" = "id_5"
      ) AS q6 
      LEFT JOIN (
        SELECT 
          "id_12" AS "id_12_14", 
          "label_13" AS "label_13_15" 
        FROM 
          (
            SELECT 
              "id" AS "id_12", 
              "label" AS "label_13" 
            FROM 
              "product"
          ) AS q5
      ) AS q7 ON (
        Cast("id_12_14" AS INT)
      ) = "category_id_9_11"
  ) AS q8

Errors

From Selda’s tutorial: “While the types keep queries nice and safe, Haskell's type errors can be a bit daunting even under the best circumstances.” In practice, type inference rarely needed guidance, and the compilation errors were relatively clear.

The only problem we’ve encountered was the mismatch of ID and SqlInt32.

All Selda functions may throw SeldaError:

errors :: SeldaM PG ()
errors = do
  insertDuplicateScrew
  insertDuplicateScrew
    `catch` (\(err :: SeldaError) -> liftIO $ putStrLn $ "Caught Selda Error: " <> displayException err)
 where
  insertDuplicateScrew = insert_ productTable [Product def "Duplicate screw" Nothing]

elephants-exe: SqlError "error executing query INSERT INTO \"product_category\" (\"product_id\", \"category_id\") VALUES ($1, $2)': ERROR: insert or update on table \"product_category\" violates foreign key constraint \"product_category_category_id_fkey\"\nDETAIL: Key (category_id)=(748) is not present in table \"category\".\n"

Resources

Selda comes with a simple overview and example. There is also a tutorial.

Migrations

The library has a Migrations module for upgrading a table from one schema to another.

In summary

Selda allows us to write type-safe queries in a linear, natural style.

Depending on your experience and situation, you can use SeldaM straight, or you may need to get familiar with mtl, exceptions, lifting/unlifting IO, etc.

Honorable mentions

At some point, I ran out of steam and didn’t have the energy to make these things work.

I didn’t manage to connect to the database using postgresql-typed. And I didn’t manage to build Haskell Relational Record and groundhog with ghc 9.4.5. Neither had releases this year. Also, there are hdbc and postgresql-libpq (you might have noticed the libpq connection string we used here and there).

All of these sound fun. But we had enough for now. Maybe we’ll revisit and extend it later.

Okay, so which PostgreSQL library should I use with Haskell?

I can’t tell you which library to use. I don’t know myself. But I can tell you the questions you (and your team) should consider:

Do you want to write raw sql queries? or query builder? or use some sort of ORM? Do you want to learn a new DSL?
How type-safe do you want to be? How readable/optimizable the generated queries should be?
Do you need bells and whistles, like built-in connection pools or migrations?
How comfortable are you with Haskell and type-level? Is this something you can afford?
Do you need to be database agnostic?

Also, this is just the tip of the iceberg; we haven’t discussed specific features, performance, compilation speed… And spoiler alert: generics, template haskell, and type-families don’t come for free.

We aren’t really at the point where people care to compare the performance. But you don’t know unless you measure. Feel free to explore and get some community karma.

Another excellent contribution opportunity is documentation and tutorials, especially if you have a favorite library and want to convince others to consider it.

Wait, is it actually bad that we have so many libraries?

vrthra/markdown-ieee-pdf

2023-10-11T01:30:48Z

A skeletal project to convert a paper written in markdown format to IEEE style paper submission in pdf.

README

Introduction

This repository contains materials to generate a paper in pdf in ieee standard.

Layout

src

This directory contains the source code in pandoc markdown wiki format.

src/paper.md The paper itself in markdown.
src/read.md The references used in the paper.
src/meta The meta information such as title, author etc.
src/req The ieee format specific files for generating pdf from pandoc markdown.

build

This directory contains the generated pdf output.

Building

Requirements

You will need pandoc to process the files. If pdf generation is intented, pandoc internally uses latex to generate the pdf files.

If you are on ubuntu, these can be accomplished with

sudo apt-get install pandoc

sudo apt-get install texlive-full

Generate

Run "make" from the top directory to generate build/ieee_jrnl.pdf

Settings

Ensure that these are run in your terminal.

git config --global diff.tool vimdiff git config --global difftool.prompt false git config --global alias.d difftool

tydar/yahmdp

2023-10-11T01:30:48Z

Yet Another Haskell Markdown Parser

And not a very good one. Deviates from the specification in some ways, some features are not implemented. I'll document these shortcomings more extensively soon.

Usage

showProcessed $ parseMarkdown "test string"

Actual input/output hopefully coming soon.

Shortcomings

No links or images yet.
No nested lists, or block-type Markdown in lists.
Unordered lists can only begin with a +, no - or *.
No escaping of characters, only limited encoding.
May not give the correct parse every time. I don't have a test suite yet, so.
Probably a lot of other things I haven't seen because I haven't tested extextensively. But I did it. I wrote a Markdown parser-ish using only ReadP combinators. That was a bad idea.

weka/terraform-azure-weka

2023-10-10T01:30:57Z

Terraform module for configuring an integration with Lacework and AWS for agentless scanning

nsbno/terraform-aws-alarm-setup

2023-10-11T01:31:08Z

Setup your alarms to redirect to the right places

CanciuCostin/android-spyware

2023-10-10T01:31:07Z

Machine-Learning-Engineering-with-Python-Second-Edition

This is the code repository for the Second Edition of Machine Learning Engineering with Python, published by Packt.

More details are below, pick up your copy today!

Manage the production life cycle of machine learning models using MLOps with practical examples

The author of this book is - Andrew Peter McMahon

About the book

The Second Edition of Machine Learning Engineering with Python is the practical guide that MLOps and ML engineers need to build solutions to real-world problems. It will provide you with the skills you need to stay ahead in this rapidly evolving field. The book takes an examples-based approach to help you develop your skills and covers the technical concepts, implementation patterns, and development methodologies you need. You'll explore the key steps of the ML development lifecycle and create your own standardized "model factory" for training and retraining models. You'll learn to employ concepts like CI/CD and how to detect different types of drift. Get hands-on with the latest in deployment architectures and discover methods for scaling up your solutions. This edition goes deeper into all aspects of ML engineering and MLOps, with an emphasis on the latest open-source and cloud-based technologies. This includes a completely revamped approach to advanced pipelining and orchestration techniques. With a new chapter on deep learning, generative AI, and LLMOps, you will learn to use tools like LangChain, PyTorch, and Hugging Face to leverage LLMs for supercharged analysis. You will explore AI assistants like GitHub Copilot to become more productive, then dive deep into the engineering considerations of working with deep learning.

Key Takeaways

Plan and manage end-to-end ML development projects
Explore deep learning, LLMs, and LLMOps to leverage generative AI
Use Python to package your ML tools and scale up your solutions
Get to grips with Apache Spark, Kubernetes, and Ray
Build and run ML pipelines with Apache Airflow, ZenML, and Kubeflow
Detect drift and build retraining mechanisms into your solutions
Improve error handling with control flows and vulnerability scanning
Host and build ML microservices and batch processes running on AWS

What's New

This second edition goes deeper into ML engineering and MLOps to provide a solid foundation as well as hands-on examples. As well as covering traditional package managers, conda and pip, the second edition shows you how to manage complex dependencies with Poetry. You’ll go beyond creating general ML pipelines with Airflow by building advanced pipelines with ZenML and Kubeflow. The second edition introduces you to Ray, a Python native distributed computing framework, to help ML engineers meet the needs of massive data and massively scalable ML systems. There is a new chapter on Large Language Models (LLMs), generative AI, and LLM Operations (LLMOps), using PyTorch, Hugging Face, and GitHub Copilot and consume an LLM via an API using LangChain.

Outline and Chapter Summary

The book covers a wide variety of ML Engineering (MLE) and ML Operations (MLOps) topics to help you understand the tools, techniques, and processes you can apply to engineer your ML solutions, with an emphasis on introducing the key concepts so that you can build on them in your future work. The aim is to develop fundamentals and a broad understanding that will stand the test of time. We cover everything from how to organize your ML team, to software development methodologies and best practices, to automating model building through to packaging your ML code, how to deploy your ML pipelines to a variety of different targets, and then how to scale your workloads for large batch runs. We also discuss, in an entirely new chapter for this second edition, the exciting world of applying ML engineering and MLOps to deep learning and generative AI, including how to start building solutions using LLMs and the new field of LLMOps. Although a far greater emphasis has been placed on using open-source tooling, many examples will also leverage services and solutions from Amazon Web Services (AWS). Machine Learning Engineering with Python, Second Edition will help you to navigate the challenges of taking ML to production and give you the confidence to start applying MLOps in your projects. I hope you enjoy it!

Chapter 01, Introduction to ML Engineering

Chapter 1 of "Machine Learning Engineering with Python, Second Edition" provides a comprehensive introduction to the realm of ML engineering and operations. It begins by elucidating the core concepts of ML engineering and MLOps and underscores their importance in the dynamic landscape of ML. The chapter delves into the roles within ML teams and lays out the challenges inherent in ML engineering and MLOps. Moreover, it acknowledges the rapid evolution of ML, with advancements in modeling techniques and technology stacks, necessitating a more profound exploration of various topics to effectively navigate this complex field.

The chapter's primary goal is to equip readers with essential tools and techniques for creating production-ready ML systems in Python. It promises to cover fundamental areas like project management, Python ML package development, and the creation and deployment of reusable training and monitoring pipelines. In addition, it discusses modern tooling, deployment architectures, and scalability using AWS and cloud-agnostic tools. It also introduces readers to transformers and LLMs, exemplified through Hugging Face and OpenAI APIs. Ultimately, the chapter aims to empower readers with a robust foundation for confidently tackling the challenges of ML engineering, regardless of their chosen tools, encouraging further exploration and self-study in pursuit of a successful career in ML engineering.

Key Insights:

Dynamic Nature of ML : The chapter highlights how the field of machine learning has evolved significantly in recent years, with the emergence of more powerful modeling techniques, complex technology stacks, and new frameworks. This evolution underscores the importance of staying up-to-date with the latest trends and tools in ML engineering.
ML Engineering Foundation : The chapter emphasizes the importance of building a strong foundation in ML engineering. It introduces the core topics that readers will explore in the book, such as project management, Python ML package development, and the creation of training and monitoring pipelines. This foundation is crucial for building production-ready ML systems.
Role of ML Teams : It provides insights into the various roles within ML teams and explains how these roles complement each other. Understanding these roles is essential for assembling well-resourced teams that can effectively tackle ML projects.
Challenges in Real-World ML : The chapter discusses the challenges of building ML products within real-world organizations. It underscores the need to estimate value reasonably and communicate effectively with stakeholders, highlighting the practical aspects of ML engineering beyond just technical skills.
High-Level System Design : Readers are introduced to the concept of high-level ML system design for common business problems. This sets the stage for the technical details to be covered in later chapters and provides a glimpse into what ML solutions should look like from a design perspective.
Motivation for Exploration : The chapter serves as a motivation for readers to engage with the material in the book and encourages them to embark on a path of exploration and self-study. It emphasizes that a solid conceptual foundation is essential for a successful career as an ML engineer.

Chapter 02, The Machine Learning Development Process

Chapter 2, "The Machine Learning Development Process," is a comprehensive exploration of organizing and executing successful ML engineering projects. It begins by discussing various development methodologies like Agile, Scrum, and CRISP-DM. The chapter introduces a project methodology developed by the author, which is referred to throughout the book. It also covers essential concepts like continuous integration/continuous deployment (CI/CD) and developer tools. The chapter's central focus is on delineating how to divide the work for a prosperous ML software engineering project, offering insights into the process, workflow, and necessary tools, accompanied by real ML code examples. It places a particular emphasis on the "four-step" methodology proposed by the author, encompassing the steps of discovery, playing, development, and deployment, comparing it with the popular CRISP-DM methodology in data science.

The chapter provides detailed guidance on setting up tools, version control strategies, CI/CD for ML projects, and potential execution environments for ML solutions. By the chapter's end, readers are well-prepared for Python ML engineering projects, laying the foundation for subsequent chapters. Additionally, it highlights that the concepts discussed here can be applied not only to ML projects but also to other Python software engineering endeavors, emphasizing the versatility of the knowledge presented. Overall, this chapter establishes a robust groundwork for the development and deployment of ML solutions while offering a broader perspective on software engineering practices in Python.

Key Insights:

Organizing ML Projects : The chapter emphasizes the importance of structuring and organizing ML projects effectively. It outlines a four-step methodology (Discover, Play, Develop, Deploy) that serves as a framework for managing ML projects.
Development Methodologies : It introduces various development methodologies such as Agile, Scrum, and CRISP-DM. The focus is on understanding how these methodologies can be adapted to suit the unique challenges of ML engineering.
Continuous Integration and Deployment (CI/CD): The chapter discusses CI/CD practices, highlighting their significance in ML projects. It provides practical guidance on setting up CI/CD pipelines, including automated model validation, using tools like GitHub Actions.
Tooling and Version Control : It covers essential tools for ML project development and version control strategies. This includes setting up tools for code development, change tracking, and managing ML projects efficiently.
Project Management Methodologies : The chapter introduces the CRISP-DM methodology, widely used in data science. It compares this methodology with Agile and Waterfall, offering insights into when and how to apply them in ML projects.
Versatility of Concepts : It underscores that the principles discussed are not limited to ML projects alone but can be applied to various Python software engineering endeavors. This highlights the broader applicability of the knowledge presented.
Testing and Automation : The chapter delves into the importance of testing ML code and how to automate testing as part of CI/CD pipelines. It extends the concept to continuous model performance testing and continuous model training.
Project Setup : Practical steps for setting up tools, environment management, and DevOps and MLOps workflows are detailed. Readers are equipped with the knowledge needed to prepare for Python ML engineering projects.

Chapter 03, From Model to Model Factory,

Chapter 3, titled "From Model to Model Factory," dives into the critical process of standardizing, systematizing, and automating ML model training and deployment. The chapter introduces the concept of the "model factory," a methodology designed for repeatable model creation and validation. It underscores the importance of automating and scaling the challenging task of model training and fine-tuning for production systems. The chapter covers fundamental theoretical concepts necessary for understanding ML models, explores various types of drift detection, and discusses criteria for triggering model retraining.

The chapter begins by emphasizing the central question: How can the intricate process of model training and fine-tuning be automated, reproduced, and scaled for production? It proceeds to provide a comprehensive overview of training different ML models, combining both theoretical and practical insights. The motivation for model retraining, driven by the concept of model performance drift, is elucidated. Additionally, the chapter delves into feature engineering, a crucial aspect of ML tasks, and explains the optimization problems at the heart of ML. It presents various tools for tackling these optimization problems, such as manual model definition, hyperparameter tuning, and automated ML (AutoML). The chapter also guides readers on interfacing with MLflow APIs for model management and introduces the concept of pipelines to streamline model training steps. Ultimately, the chapter lays the groundwork for understanding how to assemble these components into a cohesive solution, setting the stage for subsequent chapters on packaging and deployment.

Key Insights:

The Model Factory Concept : The chapter introduces the concept of the "model factory," emphasizing the need for a systematic and automated approach to model training and deployment. This approach aims to make the process repeatable and scalable for production systems.
Model Retraining for Drift : It highlights the importance of model retraining due to the phenomenon of model performance drift. ML models won't perform well indefinitely, and retraining is essential to maintain their effectiveness over time.
Feature Engineering : The chapter explores the critical role of feature engineering in ML tasks. Feature engineering involves transforming raw data into a format that ML models can understand and learn from effectively.
Understanding Model Training : Readers gain insights into the technical details of training ML models, including an in-depth look at how ML models learn. The chapter breaks down the complexity of model training into comprehensible components.
Automating Model Training : It discusses various levels of abstraction for automating the model training process, from manual model definition to hyperparameter tuning and AutoML. Examples of libraries and tools for these tasks are provided.
Drift Detection : The chapter covers the concept of drift detection, which involves monitoring how ML models and the data they use evolve over time. It includes practical examples of drift detection using packages like Alibi Detect and Evidently.
Model Persistence** : It addresses the persistence of ML models, which is crucial for saving and serving models in production systems.
MLflow Model Registry : The chapter introduces the use of MLflow's Model Registry for programmatically managing the staging of ML models.
Defining Training Pipelines : It explains how to define training pipelines using libraries like Scikit-Learn and Spark ML. Pipelines are essential for organizing and automating the model training workflow.
Foundation for Deployment : The chapter establishes a solid foundation for understanding how to package and deploy ML models effectively, setting the stage for subsequent chapters.

Chapter 04, Packaging Up

Chapter 4, "Packaging Up," delves into the practical aspects of programming in Python, specifically focusing on how to code effectively and package your code for reuse across multiple projects, including those involving ML. It emphasizes that the techniques and methodologies discussed can be applied to various Python development activities throughout an ML project's life cycle. The chapter begins with a recap of fundamental Python programming concepts and moves on to address coding standards, quality code writing, and the distinction between object-oriented and functional programming in Python. It also highlights the importance of testing, logging, error handling, and not reinventing the wheel in the development process. The chapter offers insights into how to package and distribute your code across different platforms and use cases.

This chapter essentially serves as a guide to best practices when creating Python packages for ML solutions. It covers key principles of Python programming, coding standards, and various techniques to ensure high-quality code. The discussion extends to packaging code for distribution, testing, and robust error handling. The chapter culminates by stressing the significance of reusing existing functionality rather than duplicating efforts. As it sets the stage for the next chapter on deployment, it equips readers with the necessary knowledge to prepare their code for deployment on appropriate infrastructure and tools.

Key Insights:

Coding Best Practices : The chapter emphasizes the importance of coding best practices in Python. It highlights the need for writing clean, high-quality code that is both readable and maintainable, regardless of the specific project, including ML tasks.
Coding Standards : It discusses the significance of adhering to coding standards and guidelines. Consistency in code style and structure, often enforced through tools like linters, contributes to code quality and collaboration.
Object-Oriented vs. Functional Programming : The chapter explores the differences between object-oriented and functional programming paradigms in Python. It discusses where each approach can be leveraged effectively in ML projects.
Package Development : It delves into the process of packaging code for reuse across multiple platforms and projects. Readers learn about various tools and setups for packaging code, including Makefiles and Poetry.
Testing and Error Handling : The chapter underscores the importance of robust testing, logging, and error handling. These are essential components of code that not only function correctly but are also diagnosable when issues arise.
Avoiding Redundancy : It emphasizes the principle of not reinventing the wheel. Instead of recreating functionality that already exists in the Python ecosystem, developers should leverage existing libraries and packages to streamline their work.
Preparation for Deployment : The chapter serves as a foundational guide for preparing code for deployment. It lays the groundwork for the next chapter, which will focus on deploying scripts, packages, libraries, and apps on appropriate infrastructure and tools.

Chapter 05, Deployment Patterns and Tools

Chapter 5, "Deployment Patterns and Tools," delves into the crucial process of deploying ML solutions into real-world production environments. This chapter serves as a bridge between the development of ML models and their practical implementation. It highlights the challenges and importance of transitioning from proof-of-concept to scalable and impactful ML solutions. The chapter begins by focusing on fundamental concepts related to system design and architecture, providing insights into how to build solutions that can be easily scaled and extended. It then introduces the concept of containerization and its role in abstracting application code from specific infrastructure, enhancing portability. A practical example of deploying an ML microservice on AWS is presented. The chapter subsequently returns to the topic of building robust pipelines for end-to-end ML solutions, expanding on the discussion from the previous chapter. It introduces tools like Apache Airflow, ZenML, and Kubeflow for building and orchestrating ML pipelines, offering readers a comprehensive understanding of deployment possibilities.

This chapter equips readers with the knowledge and tools necessary to navigate the complexities of deploying ML solutions effectively. It covers key concepts related to system architecture, containerization, cloud deployment, and pipeline orchestration. By the end of the chapter, readers gain confidence in their ability to deploy and orchestrate complex ML solutions using a variety of software tools. The chapter emphasizes the practical aspects of turning ML models into impactful, production-ready systems that can generate real value.

Key Insights:

Deployment Challenges : The chapter underscores that deploying ML solutions is a challenging but crucial step in the ML development lifecycle. Successful deployment can make the difference between creating value and mere hype.
System Design and Architecture : It emphasizes the significance of designing and architecting ML systems effectively. Understanding how to develop solutions that can be seamlessly scaled and extended is a fundamental aspect of deployment.
Containerization : The chapter introduces containerization as a key concept. It explains how containerization abstracts application code from specific infrastructure, enabling portability across various environments.
AWS Microservice Deployment : A practical example demonstrates how to deploy an ML microservice on AWS. This offers readers insight into the practical aspects of cloud-based ML deployment.
Pipeline Orchestration : Building on the previous chapter, the discussion on pipeline orchestration continues. Tools like Apache Airflow, ZenML, and Kubeflow are introduced for orchestrating data engineering, ML, and MLOps pipelines.
Scalability: The chapter sets the stage for scaling ML solutions to handle large volumes of data and high-throughput calculations, hinting at future considerations for ML scalability.
Practical Application : Throughout the chapter, the practical application of deployment concepts and tools is highlighted, ensuring that readers can apply their knowledge effectively in real-world ML projects.
Value Creation: Successful deployment is framed as the bridge between ML development and creating real value for customers or colleagues. It emphasizes that the deployment phase is where ML solutions have a tangible impact.

Chapter 06, Scaling Up

Chapter 6, "Scaling Up," addresses the critical challenge of developing ML solutions that can handle large datasets and high-frequency computations. It recognizes that while running simple ML models on a small scale is suitable for initial exploration and proof of concept, it's inadequate when dealing with massive volumes of data or numerous models. The chapter introduces Apache Spark and Ray frameworks, explaining their inner workings and how to leverage them for scalable ML solutions. It emphasizes the importance of adopting a different approach, mindset, and toolkit for such scenarios. Practical examples illustrate the use of these frameworks for processing large batches of data, and there's an introduction to serverless applications for scaling inference endpoints and containerized ML applications with Kubernetes. The chapter equips readers with the knowledge and tools needed to scale ML solutions to handle larger datasets effectively.

The key takeaways from this chapter are a deeper understanding of Apache Spark and Ray frameworks for distributed computing, including coding patterns and syntax, the significance of User-Defined Functions (UDFs) for scalable ML workflows, and insights into scaling ML solutions through serverless architectures, containerization, and parallel computing with Ray. The chapter sets the stage for future discussions on scaling ML models, particularly deep learning models and large language models, and emphasizes the importance of these scaling concepts for the rest of the book.

Key Insights:

Challenges of Large-Scale Data : The chapter acknowledges the limitations of running simple machine learning models on small datasets and highlights the challenges of dealing with large volumes of data and high-frequency computations.
Importance of Scalability : It underscores the importance of adopting a different approach and toolkit when scaling up ML solutions to handle massive datasets or numerous models.
Apache Spark and Rayv : The chapter provides a detailed exploration of two popular frameworks, Apache Spark, and Ray, for distributed computing. It covers coding patterns, syntax, and the use of UDFs for scalable ML workflows.
Cloud-Based Scaling : Readers gain insights into using Apache Spark on the cloud, specifically through AWS Elastic MapReduce (EMR), to scale ML solutions.
Serverless Architectures : The chapter introduces the concept of serverless architectures and demonstrates how to build an ML model serving services using AWS Lambda, emphasizing scalability.
Containerization and Kubernetes : It discusses horizontal scaling of ML pipelines using containerization and Kubernetes, providing an overview of these tools' benefits and their role in real-time workloads.
Ray Parallel Computing : Readers learn about the Ray parallel computing framework and its ability to scale compute on heterogeneous clusters for supercharging ML workflows.
Preparation for Future Chapters : The chapter sets the stage for upcoming discussions on scaling deep learning models, including LLMs, and highlights the importance of the scaling concepts covered in this chapter for the rest of the book.
Prerequisites for Scaling : It emphasizes that the knowledge and techniques discussed in this chapter are prerequisites for understanding and effectively utilizing the concepts presented in subsequent chapters.

Chapter 07, Deep Learning, Generative AI, and LLMOps

In Chapter 7, "Deep Learning, Generative AI and LLMOps," the book delves into the rapidly evolving landscape of machine learning and artificial intelligence. It acknowledges the profound changes in the ML and AI realms, with the proliferation of generative artificial intelligence (generative AI or GenAI) tools and large LLMs like ChatGPT, Bing AI, Google Bard, and DALL-E. The chapter aims to guide aspiring ML engineers through this dynamic landscape, providing a comprehensive understanding of the core concepts and foundations necessary to navigate this brave new world effectively.

The chapter commences by revisiting the fundamental algorithmic approach of deep learning, exploring its theoretical underpinnings, and offering insights into building and hosting deep learning models. It then shifts its focus to GenAI and, more specifically, LLMs, such as ChatGPT. It deep dives into the workings and approaches behind these powerful text models, setting the stage for understanding the unique challenges and opportunities they bring. Additionally, the chapter introduces the concept of LLMOps, emphasizing how ML engineering and MLOps principles can be applied to LLMs and highlighting the nascent state of best practices in this domain. Overall, this chapter equips readers with a strong foundation in deep learning, GenAI, LLMs, and the evolving field of LLMOps, positioning them to navigate this transformative landscape confidently.

Key Insights:

The Rapid Evolution of AI and ML : The chapter highlights the rapid pace of development in the AI and ML field. It acknowledges the introduction of generative AI and LLMs like ChatGPT, reflecting the ongoing innovation and growth in AI technology.
Foundational Knowledge : It emphasizes the importance of building a strong foundation in core AI and ML concepts. Readers are guided through the theoretical aspects of deep learning, ensuring they have a solid understanding of the fundamental principles underpinning these technologies.
Focus on LLMs : The chapter gives special attention to LLMs, such as ChatGPT, and explores their significance in the AI landscape. It discusses their design principles, behaviors, and practical applications, providing valuable insights for ML engineers.
Introduction to LLMOps : LLMOps, the application of ML engineering and MLOps to LLMs, is introduced as an emerging field. The chapter outlines its core components and emerging best practices, highlighting that this area is still evolving.
Preparation for Real-World Applications : By offering a deep understanding of deep learning, GenAI, and LLMs, the chapter prepares ML engineers for real-world applications of these technologies. It equips them to harness the potential of AI and ML in their projects effectively.
Nurturing a Learning Mindset : The chapter emphasizes that the AI and ML landscape is continually evolving. ML engineers are encouraged to adopt a learning mindset to keep up with the rapid advancements and contribute to the field's growth.

Chapter 08, Building an Example ML Microservice

Chapter 8, "Building an Example ML Microservice," serves as a practical culmination of the concepts and techniques covered throughout the book. It focuses on creating a machine learning microservice for a forecasting solution, using tools like FastAPI, Docker, and Kubernetes to bring together the knowledge acquired earlier.

The chapter begins by introducing the forecasting problem, emphasizing the importance of understanding the scenario and making key decisions to address it. It then delves into the design of the forecasting service, highlighting the complexities that may arise in real-world ML engineering projects. The selection of appropriate tools is discussed, considering factors like task suitability and developer familiarity. The chapter proceeds to cover training at scale, serving models using FastAPI, and containerizing and deploying the solution to Kubernetes. Each topic provides a practical walk-through, offering readers a valuable reference for tackling similar challenges in their own ML microservices projects. By the end of the chapter, readers gain a clear understanding of how to leverage the tools and techniques acquired throughout the book to build robust ML microservices for diverse business problems, bridging the gap between theory and real-world application.

Key Insights:

Realistic Problem Solving : The chapter demonstrates the importance of understanding the business problem thoroughly before designing a solution. It highlights the need for engineers to make informed decisions that align with the problem's requirements, which often involve dynamically triggered forecasting algorithms.
Tool Selection : A critical aspect of solving real-world ML problems is selecting the right tools for the job. The chapter discusses the criteria for choosing appropriate tools, considering factors like task suitability and developer familiarity. This insight helps readers make informed choices when building ML solutions.
Microservice Architecture : The chapter guides readers through the design of a microservice architecture to handle various aspects of ML solutions, such as event handling, model training, storage, and predictions. This architectural approach addresses the complexities of real-world ML engineering projects.
Practical Implementation : The practical walk-through provided for training models at scale, serving models with FastAPI, and containerizing and deploying the solution to Kubernetes offers readers a concrete example of applying ML engineering concepts and tools to build a robust microservice.

Chapter 09, Building an Extract, Transform, Machine Learning Use Case

Chapter 9, "Building an Extract, Transform, Machine Learning Use Case," extends the book's lessons by providing a detailed example of a batch-processing ML system. This use case integrates standard ML algorithms with LLMs and LLMOps, demonstrating the application of these concepts in a real-world scenario. The chapter revolves around clustering taxi ride data and performing natural language processing (NLP) on contextual text data using the Extract, Transform, Machine Learning (ETML) pattern. It explores the decision-making process, tool selection, and execution of the solution, offering readers practical insights into tackling complex batch ML engineering challenges.

By examining the integration of concepts from previous chapters, such as the ML development process, model packaging, deployment patterns, and orchestration with tools like Apache Airflow, readers gain a holistic understanding of how to approach real-world ML engineering projects. This chapter empowers ML engineers with the knowledge and skills needed to apply ETML patterns, making it an invaluable resource for building robust, batch-processing ML solutions.

Key Insights:

ETML Pattern : The chapter introduces the ETML pattern, which is a common approach in ML solutions. It emphasizes the importance of structuring ML projects to efficiently retrieve, preprocess, and apply machine learning to data in a batch-processing manner. ETML helps in dealing with complex tasks like clustering taxi ride data and performing NLP on textual data.
Integration of Concepts : Chapter 9 integrates concepts and techniques from earlier chapters, showcasing how ML engineers can leverage the knowledge gained throughout the book. It combines elements from the ML development process, packaging models, deployment patterns, and orchestration using Apache Airflow. This integration highlights the importance of a comprehensive approach to solving real-world ML engineering challenges.
Advanced Tooling : The chapter delves into the selection of tools and methodologies for building ML solutions effectively. It discusses the use of libraries like Scikit-learn, AWS boto3, and OpenAI APIs for complex functionality. Moreover, it explores advanced features of Apache Airflow, illustrating how to orchestrate ML tasks in a resilient manner.
Real-World Problem Solving : Through the example scenario of clustering taxi ride data and applying NLP, the chapter demonstrates how to address practical ML engineering challenges. It emphasizes the importance of understanding business requirements and translating them into technical solutions.
Completion of Book : Chapter 9 marks the conclusion of the book, summarizing the wide range of topics covered in the field of ML engineering. It encourages readers to embrace the exciting opportunities in this evolving field and highlights the growing demand for ML engineering skills.

Software and hardware list

Chapter	Software required	Free/Proprietary	Can code testing be performed using a trial?	Cost of the software	Download Links to the software	Hardware specifications	OS required
1	Anaconda (>=v22.0) or Miniconda (>=v22.0)	Free	N/A	N/A	https://www.anaconda.com/download	N/A	MacOS, Windows, or Linux
2	• PyCharm or VSCode • Git and GitHub account • Atlassian JIRA • AWS Account	• Proprietary with free version available • Free • Proprietary with free version available • Proprietary with CLI, etc. free	• N/A • N/A • Yes • Yes	• N/A • N/A • N/A • Free tier, followed by pay as you go	• https://www.jetbrains.com/pycharm/download •https://code.visualstudio.com/download • https://www.atlassian.com/git/tutorials/install-git • https://github.com/ •https://www.atlassian.com/software/jira • https://aws.amazon.com/	N/A	MacOS, Windows, or Linux
3	• MLFlow • Tensorflow • PyTorch	• Free • Free • Free	• N/A • N/A • Yes • Yes	N/A	• https://mlflow.org/docs/latest/quickstart.html •https://www.tensorflow.org/install • https://pytorch.org/get-started/locally/	N/A	MacOS, Windows, or Linux
4	Make	Free	N/A	N/A	• https://formulae.brew.sh/formula/make • Linux: Pre-installed • Recommend you work in Windows Subsystem for Linux and then Make should be available	N/A	MacOS, Windows, or Linux
5	• Docker (v20.10) • Kind (>=v0.20) • Kubeflow Pipelines SDK (v1.8) • ZenML (>=0.40) • Apache Airflow (>=v2.6.0)	Free	N/A	N/A	• https://docs.docker.com/engine/install/ • https://kind.sigs.k8s.io • https://www.kubeflow.org/docs/started/installing-kubeflow/ • https://docs.zenml.io/getting-started/installation • https://airflow.apache.org/docs/apache-airflow/stable/installation/	N/A	MacOS, Windows, or Linux
6	• Apache Spark (>=v3.0) • Ray	• Free • Free	N/A	N/A	• https://spark.apache.org/downloads.html • https://docs.ray.io/en/latest/ray-overview/installation.html	N/A	MacOS, Windows, or Linux (beta)
7	• OpenAI API Account	Proprietary	Requires an OpenAI account and API key	Model dependent: $0.0015-0.12/1k tokens	https://openai.com/blog/openai-api	N/A	MacOS, Windows, or Linux
8	• Minikube (v1.30.0)	Free	N/A	N/A	https://minikube.sigs.k8s.io/docs/start/	N/A	MacOS, Windows, or Linux
9	• Apache Airflow (>=v2.6.0)	Free	N/A	N/A	https://airflow.apache.org/docs/apache-airflow/stable/installation/	N/A	MacOS, Windows, or Linux

Know more on the Discord server

You can get more engaged on the discord server for more latest updates and discussions in the community at Discord

Download a free PDF

If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost. Simply click on the link to claim your free PDF. Free-Ebook

We also provide a PDF file that has color images of the screenshots/diagrams used in this book at GraphicBundle

Get to know the Author

Andrew Peter McMahon has spent years building high-impact ML products across a variety of industries. He is currently Head of MLOps for NatWest Group in the UK and has a PhD in theoretical condensed matter physics from Imperial College London. He is an active blogger, speaker, podcast guest, and leading voice in the MLOps community. He is co-host of the AI Right podcast and was named ‘Rising Star of the Year’ at the 2022 British Data Awards and ‘Data Scientist of the Year’ by the Data Science Foundation in 2019.

Other Related Books

yash19sinha/coffee-bean

2023-10-10T01:31:07Z

Welcome to the "Coffee-Bean" Coffee Shop project! We're on a quest to blend coffee and code like never before. Join us for a latte-tude of coding fun. All skill levels are welcome to make this project as espresso-nal as it can be! Grab your digital mugs and join us.

Coffee-Bean Coffee Shop Website ☕

Welcome to the Coffee-Bean Coffee Shop Website project! We're on a journey to blend coffee and code like never before. Join us and help make our website as brew-tiful as your favorite coffee blend.

Contributing

We welcome contributions from everyone! Please review our Contribution Guidelines to get started.

Code Style Guidelines

Follow consistent coding style, including indentation and naming conventions.
Add comments for complex logic to enhance code readability.
Make sure your code is free from linting errors.

Reporting Issues

If you find a bug, have a suggestion, or want to discuss something related to the project, please open an issue on the GitHub repository. Be sure to include a clear description and, if applicable, steps to reproduce the issue.

Communication

Feel free to reach out to us through GitHub issues, discussions, or our community forum for any questions or ideas. We're excited to collaborate with you!

Contributors

A big shoutout to the fantastic contributors who have helped make this project a success. Your dedication and hard work are greatly appreciated!

Thanks to all Contributors

Made with contrib.rocks. Would you like to see your name here? Contribute to our project today!

Thank you for being a part of Coffee-Bean's coffee-fueled journey. ☕ Happy coding!

🌟 Don't forget to star ⭐ this repository if you find it brew-tiful and fork 🍴 it to make your own cup of coding magic! Let's spread the love for coffee and code together.

Code of Conduct

This repository adheres to the guidelines outlined in the Code of Conduct.

Devalphaspace/Dark-theme-portfolio

2023-10-11T01:31:20Z

nvim-treesitter/nvim-treesitter-context

2023-10-11T01:31:57Z

Show code context

DiamondHunters/NodeInject_Hook_example

2023-10-10T01:32:11Z

🗿 Mini jQuery alternative. Dependency-free animations. Pairs with htmx. Locality of Behavior. Use one element or arrays transparently. Vanilla querySelector() but better!

🗿 Surreal - tiny jQuery alternative for vanilla JS + inline Locality of Behavior

(Art by shahabalizadeh)

Why does this exist?

For devs who love ergonomics! You may appreciate Surreal if:

You want to stay as close as possible to Vanilla JS.
Hate typing document.querySelector over.. and over..
Hate typing addEventListener over.. and over..
Really wish document.querySelectorAll had Array functions..
Really wish this would work in any inline <script> tag
Enjoyed using jQuery selector syntax.
Animations, timelines, tweens with no extra libraries.
Only 320 lines. No build step. No dependencies.
Pairs well with htmx
Want fewer layers, less complexity. Are aware of the cargo cult. ✈️

✨ What does it add to Javascript?

⚡️ Locality of Behavior (LoB) Use me() inside <script>
- Get an element without creating a unique name: No .class or #id needed!
- this but better!
- Want me in your CSS <style> tags, too? See our companion script
🔗 Call chaining, jQuery style.
♻️ Functions work seamlessly on 1 element or arrays of elements!
- All functions can use: me(), any(), NodeList, HTMLElement (..or arrays of these!)
- Get 1 element: me()
- ..or many elements: any()
- me() or any() can chain with any Surreal function.
  - me() can be used directly as a single element (like querySelector() or $())
  - any() can use: for / forEach / filter / map (like querySelectorAll() or $())
🌗 No forced style. Use: classAdd or class_add or addClass or add_class
- Use camelCase (Javascript) or snake_case (Python, Rust, PHP, Ruby, SQL, CSS).

🤔 Why use `me()` / `any()` instead of `$()`

💡 We solve the classic jQuery code bloat problem: Am I getting 1 element or an array of elements?
- me() is guaranteed to return 1 element (or first found, or null).
- any() is guaranteed to return an array (or empty array).
- No more checks = you write less code. Bonus: Code reads more like self-documenting english.

👁️ How does it look?

Do surreal things with Locality of Behavior like:

<label for="file-input" >
  <div class="uploader"></div>
  <script>
    me().on("dragover", ev => { halt(ev); me(ev).classAdd('.hover'); console.log("Files in drop zone.") })
    me().on("dragleave", ev => { halt(ev); me(ev).classAdd('.hover'); console.log("Files left drop zone.") })
    me().on("drop", ev => { halt(ev); me(ev).classRemove('.hover').classAdd('.loading'); me('#file-input').attribute('files', ev.dataTransfer.files); me('#form').trigger('change') })
  </script>
</label>

See the Live Example! Then view source.

🎁 Install

Surreal is only 320 lines. No build step. No dependencies.

📥 Download into your project, and add <script src="/surreal.js"></script> in your <head>

Or, 🌐 use the CDN: <script src="https://cdn.jsdelivr.net/gh/gnat/surreal/surreal.js"></script>

⚡ Usage

🔍️ DOM Selection

Select one element: me(...)
- Can be any of:
  - CSS selector: ".button", "#header", "h1", "body > .block"
  - Variables: body, e, some_element
  - Events: event.target will be used.
  - Surreal selectors: me(),any()
  - Adding a start= parameter provides a starting DOM location to select from. Default is document
    - ▶️ any('button', start='header').classAdd('red')
- me() Get current element for Locality of Behavior in <script> without an explicit .class or #id
- me("body") Gets <body>
- me(".button") Gets the first <div class="button">...</div>. To get all of them use any()
Select one or more elements as an array: any(...)
- Similar to me() but guaranteed to return an array (or empty array).
- any(".foo") Gets all matching elements, such as: <div class="foo">...</div>
- Feel free to convert between arrays of elements and single elements: any(me()), me(any(".something"))

🔥 DOM Functions

♻️ All functions work on single elements or arrays of elements.
🔗 Start a chain using me() and any()
- 🟢 Style A me().classAdd('red') ⭐ Chain style, recommended!
- 🟠 Style B: classAdd(me(), 'red')
🌐 Global conveniences help you write less code.
- globalsAdd() will automatically warn about any clobbering issues.
  - If you prefer no conveniences, just delete globalsAdd()
    - me().classAdd('red') with globalsAdd() removed: $.me().classAdd('red')
    - classAdd(me(), 'red') with globalsAdd() removed: $.classAdd($.me(), 'red')

See: Quick Start and Reference and No Surreal Needed

⚡ Quick Start

Add a class
- me().classAdd('red')
- any("button").classAdd('red')
Events
- me().on("click", ev => me(ev).fadeOut() )
- on(any('button'), 'click', ev => { me(ev).styles('color: red') })
Run functions over elements.
- any('button').run(_ => { alert(_) })
Styles / CSS
- me().styles('color: red')
- me().styles({ 'color':'red', 'background':'blue' })
Attributes
- me().attribute('active', true)

Timeline animations without any libraries.

<div>I change color every second.
  <script>
    // Every second animate something new.
    me().on("click", async ev => {
      me(ev).styles({ "transition": "background 1s" })
      await sleep(1000)
      me(ev).styles({ "background": "red" })
      await sleep(1000)
      me(ev).styles({ "background": "green" })
      await sleep(1000)
      me(ev).styles({ "background": "blue" })
      await sleep(1000)
      me(ev).styles({ "background": "none" })
      await sleep(1000)
      me(ev).remove()
    })
  </script>
</div>

<div>I fade out and remove myself.
  <script>me().on("click", ev => { me(ev).fadeOut() })</script>
</div>

<div>I change color every second.
  <script>
    // Run immediately.
    (async (e = me()) => {
      me(e).styles({ "transition": "background 1s" })
      await sleep(1000)
      me(e).styles({ "background": "red" })
      await sleep(1000)
      me(e).styles({ "background": "green" })
      await sleep(1000)
      me(e).styles({ "background": "blue" })
      await sleep(1000)
      me(e).styles({ "background": "none" })
      await sleep(1000)
      me(e).remove()
    })()
  </script>
</div>

<script>
  // Run immediately, for every <button> globally!
  (async () => {
    any("button").fadeOut()
  })()
</script>

Array methods

any('button')?.forEach(...)
any('button')?.map(...)

👁️ Functions

Looking for DOM Selectors? Looking for stuff we recommend doing in vanilla JS?

🧭 Legend

🔗 Chainable off me() and any()
🌐 Global convenience helper.
▶️ Runnable example.
🔌 Built-in Plugin

👁️ At a glance

🔗 run
- It's forEach but less wordy and works on single elements, too!
- ▶️ me().run(e => { alert(e) })
- ▶️ any('button').run(e => { alert(e) })
🔗 remove
- ▶️ me().remove()
- ▶️ any('button').remove()
🔗 classAdd 🔁 class_add 🔁 addClass 🔁 add_class
- ▶️ me().classAdd('active')
- Leading . is optional for all class functions, and is removed automatically.
  - These are the same: me().classAdd('active') 🔁 me().classAdd('.active')
🔗 classRemove 🔁 class_remove 🔁 removeClass 🔁 remove_class
- ▶️ me().classRemove('active')
🔗 classToggle 🔁 class_toggle 🔁 toggleClass 🔁 toggle_class
- ▶️ me().classToggle('active')
🔗 styles
- ▶️ me().styles('color: red') Add style.
- ▶️ me().styles({ 'color':'red', 'background':'blue' }) Add multiple styles.
- ▶️ me().styles({ 'background':null }) Remove style.
🔗 attribute 🔁 attributes 🔁 attr
- Get: ▶️ me().attribute('data-x')
  - Get is only for single elements. For many, wrap the call in any(...).run(...) or any(...).forEach(...).
- Set: ▶️me().attribute('data-x', true)
- Set multiple: ▶️ me().attribute({ 'data-x':'yes', 'data-y':'no' })
- Remove: ▶️ me().attribute('data-x', null)
- Remove multiple: ▶️ me().attribute({ 'data-x': null, 'data-y':null })
🔗 trigger
- ▶️ me().trigger('hello')
- Wraps dispatchEvent
🔗 on
- ▶️ me().on('click', ev => { me(ev).styles('background', 'red') })
- Wraps addEventListener
🔗 off
- ▶️ me().remove('click')
- Wraps removeEventListener
🔗 offAll
- ▶️ me().offAll()
🌐 sleep
- ▶️ await sleep(1000, ev => { alert(ev) })
- async version of setTimeout
- Wonderful for animation timelines.
🌐 tick
- ▶️ await tick()
- await version of rAF / requestAnimationFrame.
- Animation tick. Waits 1 frame.
- Great if you need to wait for events to propagate.
🌐 rAF
- ▶️ rAF(e => { return e })
- Animation tick. Fires when 1 frame has passed. Alias of requestAnimationFrame
- Great if you need to wait for events to propagate.
🌐 rIC
- ▶️ rIC(e => { return e })
- Great time to compute. Fires function when JS is idle. Alias of requestIdleCallback
🌐 halt
- ▶️ halt(event)
- Great to prevent default browser behavior: such as displaying an image vs letting JS handle it.
- Wrapper for preventDefault
🌐 createElement 🔁 create_element
- ▶️ e_new = createElement("div"); me().prepend(e_new)
- Alias of vanilla document.createElement
🌐 onloadAdd 🔁 onload_add 🔁 addOnload 🔁 add_onload
- ▶️ onloadAdd(_ => { alert("loaded!"); })
- Execute after the DOM is ready. Similar to jquery ready()
- Queues functions onto window.onload
- Why? So you don't overwrite window.onload, also predictable sequential loading!

🔌 Built-in Plugins

Effects

Build effects with me().styles({...}) with timelines using CSS transitioned await or callbacks.

Common effects included:

🔗 fadeOut 🔁 fade_out
- Fade out and remove element.
- Keep element with remove=false.
- ▶️ me().fadeOut()
- ▶️ me().fadeOut(ev => { alert("Faded out!") }, 3000) Over 3 seconds then call function.
🔗 fadeIn 🔁 fade_in
- Fade in existing element which has opacity: 0
- ▶️ me().fadeIn()
- ▶️ me().fadeIn(ev => { alert("Faded in!") }, 3000) Over 3 seconds then call function.

🔮 No Surreal Needed

More often than not, Vanilla JS is the easiest way!

Logging

🌐 console.log() console.warn() console.error()
Event logging: ▶️ monitorEvents(me()) See: Chrome Blog

Benchmarking / Time It!

▶️ console.time('name')
▶️ console.timeEnd('name')

Text / HTML Content

▶️ me().textContent = "hello world"
- XSS Safe! See: MDN
▶️ me().innerHTML = "<p>hello world</p>"
▶️ me().innerText = "hello world"

Children

▶️ me().children
▶️ me().children.hidden = true

Append / Prepend elements.

▶️ me().prepend(new_element)
▶️ me().appendChild(new_element)
▶️ me().insertBefore(element, other_element.firstChild)
▶️ me().insertAdjacentHTML("beforebegin", new_element)

💎 Conventions & Tips

_ = for temporary or unused variables. Keep it short and sweet!
e, el, elt = element
e, ev, evt = event
f, fn = function
Many things can be done in vanilla HTML / CSS (ex: dropdowns).
Find where your change touches the least code.
Simplicity and ergonomics tend to have exponential payoff.

🔌 Adding a function

Feel free to modify Surreal for a particular project any way you like- but you can use this system to effortlessly merge functions with new versions.

Add your function

var $thing = {
  test(e, name) {
    console.log(`Hello ${name} from ${e}`)
    return e
  }
}
$ = {...$, ...$thing}

Is your function chainable? Add to sugar()

$.sugars['test'] = (name) => { return $.test($._e, name) }

Your function will be added globally by globalsAdd() If you do not want this (ex: name clash), add it to the restricted list.

Refer to an existing function to see how to make your function work with 1 or many elements.

Make an issue or pull request if you think people would like to use it! If it's useful enough we'll want it in core.

⭐ Awesome Surreal examples, plugins, and resources: awesome-surreal !

📚️ Inspired by

jQuery for the chainable syntax we all love.
BlingBling.js for modern minimalism.
Bliss.js for a focus on single elements and extensibility.
Hyperscript for Locality of Behavior and awesome ergonomics.
Shout out to Umbrella, Cash, Zepto- Not quite as ergonomic. Requires build step to extend.

🌘 Future

Always more example.html goodies!
Automated browser testing perhaps with:

QiuChenlyOpenSource/QQFlacMusicDownloader

2023-10-11T01:32:22Z

[秋城落叶] QQ 音乐源无损歌曲下载

大纲

介绍

Create & Design By QiuChenly

这是一个批量下载 QQ 音乐/酷我音乐/网易云会员无损音质歌曲的脚本,技术含量并不是很大,仅供参考。

参考是让你参考代码，不是让你想着法儿去白嫖。今天你不愿意掏198的数字专辑费用，明天你就会失去赚到198万的机会。

眼界决定了人生未来，格局决定了人生上限。哥们主打的就是一个三观。

鲁迅曾经说过:舍小我成大我,我不入地狱谁入地狱?

所以白嫖的事情我来，付钱你来。

前端技术:
Vue3+TS+Pinia+ElementUI Plus

后端技术:
Python3.11.2 + Flask + Concurrency协程

开发环境:
BenQ Laptop Core Duo 双核 1.2Ghz + 2GB DDR3

OS : Windows 7 x64
Env: Python 3.11.2
IDE: Windows Notepad

教程只能解决使用方法上的问题，解决不了dinner的问题。

已知问题

Windows用户可能体验不是很好，因为很多适配坑。
QQ音乐搜不到,是因为搜索频率过快。别问我为什么才搜两次就频繁了，你问qq。
下载歌曲后前端页面没有提示，看命令行窗口就知道有没有下载成功了。

关于网易云登录功能的声明:

第三方网易云API服务器来自于互联网，开发者不承认且不担保其具有官方服务器性质，并对该服务器可能会泄露用户信息的可能性保持怀疑。
此服务器不可信，请使用完后修改网易云账号密码避免账号失窃。
本项目不会保存任何个人隐私信息，但是不对使用的第三方服务器安全性/隐私性做保证。
使用者因使用本服务导致的数据泄露账号失窃等问题由使用者独自承担后果，开发者概不负责。
使用本服务即表示你同意以上条款，对于使用者在明知或不知以上条款情况下使用本项目所造成的任何数据/泄露隐私泄漏的后果由使用者自行承担，开发者不对任何数据安全性作保证。
本项目为完全公益性项目，谢绝任何形式打赏与钱财赠与，以前不会接受以后也不会接受。
使用本项目所造成的一切数据泄露，账号失窃，财产损失等严重后果有用户自行承担风险，开发者已经对用户可能所受到的攻击和安全问题做到了尽可能的提示与告知。
如果本项目侵犯了你的合法权益，请电子邮件到qiuchenly@outlook.com,我将第一时间删库跑路。
总结: 号没了跟我无关

前后端源码文件夹

前端界面源代码: WebSourceCode

后端Flask主代码: flaskSystem

提交代码时源码顶部的版权注释尽量不要修改，这是我IDE自动生成的，每次都会自动更新。如果你需要保留你的修改记录，可以通过Google风格的注释附加在函数注释上:

特别功能

功能	状态	附加说明
酷我音乐	✅	支持Flac和320KbpsMP3下载根据网友梨花喵的加密算法获取解析
FreeMyMP3	⌛️	狠狠的加密 v2算法已破解下载接口参数有问题暂时不可用
咪咕音乐	✅	可以下载Flac/320KbpsMP3歌曲
QQ音乐	✅	已经修复官方无损解析接口
网易云音乐	❌	版权问题灰色歌曲没有CDN资源缓存无法下载

基于web的友好界面出来啦

已支持元数据+内嵌歌词封面信息写入

目前仅适配: 目前已完整支持全部常见元数据补全 MP3/FLAC/HiRes格式文件补全

咪咕音乐
酷我音乐
QQ音乐 (最全面支持只要能写入的元数据信息我都写入了包括专辑/歌手/专辑高清图像/专辑简介/流派/歌曲语言...)

即将增加新功能

功能现在还没法用，不要乱点。你点了是没有反应的。

本地音乐元数据匹配
下载音乐时自动匹配元数据过滤系统

使用方法

0. 代码更新频繁,切记及时更新

有问题先下载最新的代码再看看是不是已经被解决了，而不是拿着几天前的代码问我为什么启动不了。你怎么不问问你自己一年前追的女神怎么今天跟别人酒店里泡芙都流出来了，为什么你还没追到呢？

楼主一般情况下脾气是很温和的，但是架不住小可爱太多了。提的也是可爱dinner问题，令人忍俊不禁莞尔一笑。

只有你学校的老师才会耐心解决你的问题，我只对我的学生负责，我不对小可爱负责。

1. 安装环境

首先安装最新的 python3.11.2 到你的操作系统里。

Windows用户强制安装此版本: 点击下载 -> Python3.11.2

Windows用户须知: 不是Windows用户不用看这一段小作文

首先我在此明确表示我的观点: Windows就是全宇宙最优秀的操作系统。

1.卸载电脑上所有的python版本,哪怕是3.10都不行.
如果你会玩python,那么你可以不用看这个提示.
如果你就是个纯纯的新手,我的建议是认真看一下.

2.Windows应用商店的残疾Python版本不要安装.
装完出问题然后还要明知故问,我会为你送上优美的赞歌.
计算机技术基础不应该由我给你补.

3.不要把项目放在文件夹路径有空格的目录下
比如你非要把下载好的代码文件放到
D:/Program Files(x86)/Test 123/AB CD/
这种存在空格的路径下面,运行报错再来问我那你确实注定要被我赞美称颂.

4.题外话
你要实在不会用,就真别用了吧.
但是你用不了就别到处发"这东西根本没吊用"之类的言论
你dinner不代表别人dinner。
球球了,折磨你自己可以,别折磨我.

windows用户如果有任何体验上的问题，请提交代码Merge。不要一句“不行没法用”来反馈。哥们看到这种可爱issues直接给你关了。
让我们共建和谐互联网大家庭。
最后，Microsoft Windows我测你的码。

macOS用户: 系统自带的python3.9版本就够了不需要另外安装

以下所有操作皆默认假设CMD/Powershell/Bash/zsh的当前目录在：

(Windows)
D:/Downloads/QQFlacMusicDownloader-master/
在文件夹空白处按住shift+鼠标右击,点击"在此处打开PowerShell"。
如果你很懂，cd命令进入上面的目录也行。

(macOS/Unix/Linux)
~/Download/QQFlacMusicDownloader-master/
终端cd进入上面的目录.

安装依赖包如果出现 404 错误或者太慢，可以用下面的代码切换到清华大学服务器安装。

# 设置python的依赖安装镜像服务器为清华大学服务器
pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

# 安装软件依赖必须包
pip3 install -r requirements.txt

2. 进入软件包目录下启动软件

终端/控制台进入到本文件所在的目录执行以下指令:

python3 MainServer.py

由于Windows的可爱设计,在Windows上可能是

python MainServer.py

如果你是windows10/11,那么系统很有可能会让你安装Windows应用商店的Python残疾版本，如果你不知道该不该安装的话，看看上面Windows用户须知。

启动后应该能看到这些信息，红色警告不是错误，无知可以但不要怀疑哥们的实力。看到类似于下面的输出提示即表示你启动成功。没看到类似下面的信息说明没成功。

没成功你也别急，截图+操作环境+操作步骤详细给出一份报告，而不是用几个字"本地环境实测无法启动"来概括。恕我直言，你不是文曲星，没那本事几个字概括核心要点就不要概括，我理都不想理你。

成功后google chrome/Microsoft Edge/Safari浏览器打开http://127.0.0.1:8899即可打开新世界。

只在Edge浏览器测试UI通过，建议全部使用Microsoft Edge浏览器访问。

高级搜索方法

目前 仅支持 QQ音乐/咪咕音乐搜索。

指令	例子	id从哪找	作用
(album) b:+专辑ID			(QQ音乐)获取一张专辑里的歌曲
(album) b:+专辑ID			(咪咕)获取一张专辑里的歌曲
(playlist) p:+歌单ID			获取一张歌单里的所有歌曲,需要注意的是列表下面的分页是无效的不要切换页面因为他一次性是加载的整个歌单列表
(id) id:+歌曲ID			有时候有些二逼非主流歌曲名称是及其之难以用输入法扣出来的，这个时候可以用id直接获取到这首歌。不想用那些二逼非主流歌曲做演示，用我最爱的杰伦演示给你们看看得了
(toplist) t:+榜单ID			获取一张榜单里的歌曲

2023.4.14

修复歌曲关键词过滤不可定制化搜索的问题，满足搜索部分土嗨神曲的需求。现在可以在搜索界面关闭关键词过滤或者在首页自定义过滤关键词。

新增歌曲元数据刮削内嵌

开发中，目前效果不等于最终效果。

本功能用于为本地音乐文件写入元数据功能，实现内嵌歌词文件/歌手/专辑/封面图片/专辑序号/发布年代等元数据信息，避免播放时一片空白无法加载歌曲信息的问题出现。主要使用以下接口:

iTunes 专辑信息接口 + 歌词信息 + 苹果特有的4K超清专辑封面图
QQ 音乐专辑信息接口 + 歌词信息 + 800x800低清专辑封面图

梗小鬼提问时间
Q: 为什么不用酷我/咪咕接口？
A: 因为老子高兴。iTunes+QQ基本上全球音乐元数据都全了，怎么，不够你用是吧？

为了保证数据精确度，仅支持完整匹配以下文件夹结构:
/周杰伦/专辑列表文件夹/周杰伦 - 一路向北.flac

例子：

或已经内置歌手专辑与歌曲名称的flac文件，对于这类文件将从iTunes/QQ中搜索补全剩余专辑信息。（从本项目下载的歌曲均自动已存在此信息）

效果预览：灰色无任何信息的歌曲就是没有自动匹配元数据的，写入了元数据的歌曲将会拥有精美封面图和完整的歌曲专辑元数据信息---这一切都本地存储在音乐文件中！

新增网易云盘歌曲信息纠错

可以搜索歌曲或指定歌曲ID来关联你上传到网易云盘里的歌曲。在使用前需要打开网易云登录界面一次更新Cookie才可以访问到你的云盘数据。加载时由于一次加载1000条数据，所以加载缓慢，请等几秒钟。

确认匹配

通过你设置的ID或者搜索到的歌曲来绑定这首歌的实际歌曲数据。如果你想取消某首歌的匹配可以点击取消匹配。

取消匹配

取消网易云自动/手动匹配的错误歌曲信息。有时候我们上传的歌曲信息网易云会识别错误，所以我们可以取消匹配或者手动更新。

歌曲ID如何获得？

Docker镜像部署

可以通过自己自己打包 Docker 进行部署，也可以使用本项目打包好的容器进行部署

自己打包 Docker 进行部署执行以下命令：

docker build -t dockerimage .

使用本项目打包好的容器进行部署：

docker pull registry.cn-hangzhou.aliyuncs.com/music_downloader/qq_flac_music_downloader

Docker 镜像部署需要进行端口映射，可以采用以下命令进行端口映射：

（注意：用你的本地使用目录替换下方“本地目录” 如 E:\music）

docker run -p 127.0.0.1:8899:8899 -v 本地目录:/workspace/music -it dockerimage:latest

更新方式：先运行 docker ps -a 查看容器名称

然后

docker stop 容器名称
docker rm 容器名称

最后，上面docker pull和docker run的代码重新执行一遍

docker-compose部署方式

本地新建txt，重命名为docker-compose.yml （不会修改后缀请百度）

复制以下内容，同样注意替换“本地目录”

或者你直接下载项目中的docker-compose.yml，然后自行修改本地目录

version: "3"
services:
  downloader:
    image: registry.cn-hangzhou.aliyuncs.com/music_downloader/qq_flac_music_downloader
    container_name: music
    network_mode: bridge
    volumes:
      - 本地目录:/workspace/music
    ports:
      - "127.0.0.1:8899:8899"
    restart: always

然后打开cmd命令行,cd到docker-compose.yml所在目录 docker-compose up -d

需要更新的时候，也是cd到docker-compose.yml所在目录

docker-compose pull
docker-compose up -d

相对docker，更新比较简单，所以个人比较推荐使用docker-compose的方式

自定义端口设置

部分设备会存在 8889 端口被占用的情况，部署时可自定义端口，终端/控制台进入到本文件所在的目录执行以下指令:

# []内为可选参数
python3 MainServer.py [--port 8999]

docker 部署可自行切换映射本地端口，以解决端口被占用情况。

免责声明

禁止任何形式的商业用途，包括但不仅限于售卖/打赏/获利，不得使用本代码进行任何形式的牟利/贩卖/传播，再次强调仅供个人私下研究学习技术使用，有条件者请支持正版音乐！律师函请发给提供这些音乐资源解析服务的网站运营方，本项目仅以纯粹的技术目的去学习研究，如有侵犯到任何人的合法权利，请致信qiuchenly@outlook.com，我将在第一时间删库跑路

本项目基于 GPL V3.0 许可证发行，以下协议是对于 GPL V3.0 的补充，如有冲突，以以下协议为准。

词语约定：本协议中的“本项目”指QQFlacMusicDownloader项目；“使用者”指签署本协议的使用者；“官方音乐平台”指对本项目内置的包括酷我、网易云、QQ音乐、咪咕等音乐源的官方平台统称；“版权数据”指包括但不限于图像、音频、名字等在内的他人拥有所属版权的数据。

本项目的数据来源原理是从各官方音乐平台的公开服务器中拉取数据，经过对数据简单地筛选与合并后进行展示，因此本项目不对数据的准确性负责。使用本项目的过程中可能会产生版权数据，对于这些版权数据，本项目不拥有它们的所有权，为了避免造成侵权，使用者务必在24小时内清除使用本项目的过程中所产生的版权数据。本项目内的官方音乐平台别名为本项目内对官方音乐平台的一个称呼，不包含恶意，如果官方音乐平台觉得不妥，可联系本项目更改或移除。本项目内使用的部分包括但不限于字体、图片等资源来源于互联网，如果出现侵权可联系本项目移除。由于使用本项目产生的包括由于本协议或由于使用或无法使用本项目而引起的任何性质的任何直接、间接、特殊、偶然或结果性损害（包括但不限于因商誉损失、停工、计算机故障或故障引起的损害赔偿，或任何及所有其他商业损害或损失）由使用者负责。本项目完全免费，仅供个人私下小范围研究交流学习 python 技术使用, 且开源发布于 GitHub 面向全世界人用作对技术的学习交流，本项目不对项目内的技术可能存在违反当地法律法规的行为作保证，禁止在违反当地法律法规的情况下使用本项目，对于使用者在明知或不知当地法律法规不允许的情况下使用本项目所造成的任何违法违规行为由使用者承担，本项目不承担由此造成的任何直接、间接、特殊、偶然或结果性责任。若你使用了本项目，将代表你接受以上协议。

音乐平台不易，请尊重版权，支持正版。

致谢

感谢所有项目的参与/贡献者，也感谢提出issues的各位用户。

其他资料

早期接口 QMD Apk的逆向过程

名词解释

dinner: 低能
我测你m: 一眼顶真鉴定为纯纯的范剑

SciML/StructuralIdentifiability.jl

2023-10-10T01:32:44Z

Fast and automatic structural identifiability software for ODE systems

m-dadej/MarSwitching.jl

2023-10-10T01:32:44Z

MarSwitching.jl: Julia package for Markov Switching dynamic models

mit-han-lab/torchquantum

2023-10-11T01:32:58Z

A PyTorch-based framework for Quantum Simulation, Quantum Machine Learning, Quantum Neural Networks, Parameterized Quantum Circuits with support for easy deployments on real quantum computers.

A PyTorch Library for Quantum Simulation and Quantum Machine Learning

Faster, Scalable, Easy Debugging, Easy Deployment on Real Machine

👋 Welcome

What it is doing

Quantum simulation framework based on PyTorch. It supports statevector simulation and pulse simulation (coming soon) on GPUs. It can scale up to the simulation of 30+ qubits with multiple GPUs.

Who will benefit

Researchers on quantum algorithm design, parameterized quantum circuit training, quantum optimal control, quantum machine learning, quantum neural networks.

Differences from Qiskit/Pennylane

Dynamic computation graph, automatic gradient computation, fast GPU support, batch model tersorized processing.

News

v0.1.7 Available!
Join our Slack for real time support!
Welcome to contribute! Please contact us or post in the forum if you want to have new examples implemented by TorchQuantum or any other questions.
Qmlsys website goes online: qmlsys.mit.edu and torchquantum.org

Features

Easy construction and simulation of quantum circuits in PyTorch
Dynamic computation graph for easy debugging
Gradient support via autograd
Batch mode inference and training on CPU/GPU
Easy deployment on real quantum devices such as IBMQ
Easy hybrid classical-quantum model construction
(coming soon) pulse-level simulation

Installation

git clone https://github.com/mit-han-lab/torchquantum.git
cd torchquantum
pip install --editable .

Basic Usage

import torchquantum as tq
import torchquantum.functional as tqf

qdev = tq.QuantumDevice(n_wires=2, bsz=5, device="cpu", record_op=True) # use device='cuda' for GPU

# use qdev.op
qdev.h(wires=0)
qdev.cnot(wires=[0, 1])

# use tqf
tqf.h(qdev, wires=1)
tqf.x(qdev, wires=1)

# use tq.Operator
op = tq.RX(has_params=True, trainable=True, init_params=0.5)
op(qdev, wires=0)

# print the current state (dynamic computation graph supported)
print(qdev)

# obtain the qasm string
from torchquantum.plugin import op_history2qasm
print(op_history2qasm(qdev.n_wires, qdev.op_history))

# measure the state on z basis
print(tq.measure(qdev, n_shots=1024))

# obtain the expval on a observable by stochastic sampling (doable on simulator and real quantum hardware)
from torchquantum.measurement import expval_joint_sampling
expval_sampling = expval_joint_sampling(qdev, 'ZX', n_shots=1024)
print(expval_sampling)

# obtain the expval on a observable by analytical computation (only doable on classical simulator)
from torchquantum.measurement import expval_joint_analytical
expval = expval_joint_analytical(qdev, 'ZX')
print(expval)

# obtain gradients of expval w.r.t. trainable parameters
expval[0].backward()
print(op.params.grad)


# Apply gates to qdev with tq.QuantumModule
ops = [
    {'name': 'hadamard', 'wires': 0}, 
    {'name': 'cnot', 'wires': [0, 1]},
    {'name': 'rx', 'wires': 0, 'params': 0.5, 'trainable': True},
    {'name': 'u3', 'wires': 0, 'params': [0.1, 0.2, 0.3], 'trainable': True},
    {'name': 'h', 'wires': 1, 'inverse': True}
]

qmodule = tq.QuantumModule.from_op_history(ops)
qmodule(qdev)

Guide to the examples

We also prepare many example and tutorials using TorchQuantum.

For beginning level, you may check QNN for MNIST, Quantum Convolution (Quanvolution) and Quantum Kernel Method, and Quantum Regression.

For intermediate level, you may check Amplitude Encoding for MNIST, Clifford gate QNN, Save and Load QNN models, PauliSum Operation, How to convert tq to Qiskit.

For expert, you may check Parameter Shift on-chip Training, VQA Gradient Pruning, VQE, VQA for State Prepration, QAOA (Quantum Approximate Optimization Algorithm).

Usage

Construct parameterized quantum circuit models as simple as constructing a normal pytorch model.

import torch.nn as nn
import torch.nn.functional as F
import torchquantum as tq
import torchquantum.functional as tqf

class QFCModel(nn.Module):
  def __init__(self):
    super().__init__()
    self.n_wires = 4
    self.measure = tq.MeasureAll(tq.PauliZ)

    self.encoder_gates = [tqf.rx] * 4 + [tqf.ry] * 4 + \
                         [tqf.rz] * 4 + [tqf.rx] * 4
    self.rx0 = tq.RX(has_params=True, trainable=True)
    self.ry0 = tq.RY(has_params=True, trainable=True)
    self.rz0 = tq.RZ(has_params=True, trainable=True)
    self.crx0 = tq.CRX(has_params=True, trainable=True)

  def forward(self, x):
    bsz = x.shape[0]
    # down-sample the image
    x = F.avg_pool2d(x, 6).view(bsz, 16)

    # create a quantum device to run the gates
    qdev = tq.QuantumDevice(n_wires=self.n_wires, bsz=bsz, device=x.device)

    # encode the classical image to quantum domain
    for k, gate in enumerate(self.encoder_gates):
      gate(qdev, wires=k % self.n_wires, params=x[:, k])

    # add some trainable gates (need to instantiate ahead of time)
    self.rx0(qdev, wires=0)
    self.ry0(qdev, wires=1)
    self.rz0(qdev, wires=3)
    self.crx0(qdev, wires=[0, 2])

    # add some more non-parameterized gates (add on-the-fly)
    qdev.h(wires=3)
    qdev.sx(wires=2)
    qdev.cnot(wires=[3, 0])
    qdev.qubitunitary(wires=[1, 2], params=[[1, 0, 0, 0],
                                            [0, 1, 0, 0],
                                            [0, 0, 0, 1j],
                                            [0, 0, -1j, 0]])

    # perform measurement to get expectations (back to classical domain)
    x = self.measure(qdev).reshape(bsz, 2, 2)

    # classification
    x = x.sum(-1).squeeze()
    x = F.log_softmax(x, dim=1)

    return x

VQE Example

Train a quantum circuit to perform VQE task. Quito quantum computer as in simple_vqe.py script:

cd examples/vqe
python vqe.py

MNIST Example

Train a quantum circuit to perform MNIST classification task and deploy on the real IBM Quito quantum computer as in mnist_example.py script:

cd examples/mnist
python mnist.py

Files

File	Description
devices.py	QuantumDevice class which stores the statevector
encoding.py	Encoding layers to encode classical values to quantum domain
functional.py	Quantum gate functions
operators.py	Quantum gate classes
layers.py	Layer templates such as RandomLayer
measure.py	Measurement of quantum states to get classical values
graph.py	Quantum gate graph used in static mode
super_layer.py	Layer templates for SuperCircuits
plugins/qiskit*	Convertors and processors for easy deployment on IBMQ
examples/	More examples for training QML and VQE models

Coding Style

torchquantum uses pre-commit hooks to ensure Python style consistency and prevent common mistakes in its codebase.

To enable it pre-commit hooks please reproduce:

pip install pre-commit
pre-commit install

Papers using TorchQuantum

Manuscripts

Dependencies

3.9 >= Python >= 3.7 (Python 3.10 may have the concurrent package issue for Qiskit)
PyTorch >= 1.8.0
configargparse >= 0.14
GPU model training requires NVIDIA GPUs

Contact

TorchQuantum Forum

Hanrui Wang hanrui@mit.edu

Contributors

Jiannan Cao, Jessica Ding, Jiai Gu, Song Han, Zhirui Hu, Zirui Li, Zhiding Liang, Pengyu Liu, Yilian Liu, Mohammadreza Tavasoli, Hanrui Wang, Zhepeng Wang, Zhuoyang Ye

Citation

@inproceedings{hanruiwang2022quantumnas,
    title     = {Quantumnas: Noise-adaptive search for robust quantum circuits},
    author    = {Wang, Hanrui and Ding, Yongshan and Gu, Jiaqi and Li, Zirui and Lin, Yujun and Pan, David Z and Chong, Frederic T and Han, Song},
    booktitle = {The 28th IEEE International Symposium on High-Performance Computer Architecture (HPCA-28)},
    year      = {2022}
}

manan-shxrma/Robosocnith_database

2023-10-11T01:32:58Z

This repo is used as a database for upcoming projects and a central repository

Robosocnith_database

This is dedicated repository for maintaining souce code of Robotics Society, NITH. The maintainers are not responsible for any plagiarism inadvertently crept into this repository.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

Built With

(back to top)

Getting Started

To run any of the Code/Program follow this steps to run it locally on your machine. To get a local copy up and running follow these simple example steps.

Prerequisites

Based on your choice of program choose your desired compiler to be installed on your machine

Arduino
C++
C

Usage

Use this Repository to learn and prepare for your internships and campus placements. This repository will contain most of the Repository which will be helpful to students.

For more examples, please refer to the Documentation

(back to top)

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

Fork the Project
Create your Feature Branch (git checkout -b feature/AmazingFeature)
Commit your Changes (git commit -m 'Add some AmazingFeature')
Push to the Branch (git push origin feature/AmazingFeature)
Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

devexpert-io/desafio-oct-2023

2023-10-11T01:33:26Z

https://devexpert.io/desafio-ac

mustfaibra/QPayMultiplatform

2023-10-11T01:33:26Z

QPay is a fintech app built entirely with Compose Multiplatform and considered a use case for unifying & providing financial services on both Android and iOS platforms while leveraging the power of Native APIs.

Quick Pay

Quick Pay - QPay for short - is a simple use case for a Fintech app for Onboarding new customers and providing Services, built entirely with Compose Multiplatform. This cross-platform project is considered a use case for unifying & providing financial services on both Android and iOS platforms while leveraging the power of Native.

Targeted Platforms For Now

Android Platform
iOS Platform

Screenshots

Tech & Libs Used Until Now

The project used the following:

Kotlin Programming Language
Compose Mutliplatform Framework
Decompose for State's Management, thanks Arkadii
Moko Resources for sharing resources & handling painters on both platforms.
Koin for handling dependency Injection on Mutliplatform.
Kotlinx Datetime for Mutliplatform
InsetX for handling windows insets - status bar and navigation bar - on both platforms.

Resources & Projects Until Now

Here are some projects that I found helpful when building this project:

Moko Sharing resources Demo by the great Phillip Lackner.
Music App by Abdulbasit, thank you so much!

UI Credits

The great app UI design is from QPay on Behance, by the talented Designer Alaa.

What's Next?

The remaining screens appears on the app UI design on the shared Behance link on the credit section is built and ready, only waiting for some clean up.

The following features is coming, feel free to contribute!

Camera For Capturing ID and Identity Verification : Currently the app is asking for Camera permission successfully on both platforms, but the camera feature itself is not implemented yet.
Login Screen for exited users.
QR Code Scanner to support Scan To Send Money for both Android & iOS.

Final words!

I really hope you find this repository helpful, if so, don't forget to give a star ⭐️!

You can also follow me on

Mustachedom/md-drugs

2023-10-10T01:33:58Z

An in depth free drug script for QBCore

Support

if you have any issues, you can come to my discord and throw a ticket in. I hope i can get it fixed ASAP Rocky

My Community Discord Invite Link

Credits

This was all possible because of creators and devs in Project Sloth, JixelTay, and Samuel's development. I used the globalstates and spawning of props from Samuel's christmas script (I asked permission before release)

Samuel's Tebex.

Project Sloth has been a pioneer of how the fivem community should be. Helping the community by creating free, easy to install, and fun scripts. Along with allowing the community to publish free scripts. and Jimathy and Tay are the sole reason I learned(ish) how to code. When I started back last year, I had no idea what I was doing and their scripts are easy to read, edit, and see how things actually function. without those scripts I would forget 1000x more commas and not understand how code communicates. If you don't have any of their scripts. you are missing out.

Jim's Tebex

I am not affiliated with any of these creators above but I have to give a shout out to the community when I can.

Big Credit to Samuel's Development for the way they did globalstates. That was the main inspiration to the beginning of this script.
Big Credit to Project Sloth as a team and also the Project Sloth Community. This was only made because of the big library of Free scripts published by them and their community. Buy Them coffee or something.
Big Credit to Bostra. if you have been around you may have seen him post some free scripts and help people. There was definetly times I dm'd him saying what the fuck am I doing and he helped keep me sane.
Big Credit to Feisty. She made all the images that look good. Seriously you can tell which ones I forgot to ask her for.

Dependencies

How to install like a fuckin bosssssss

STEP 1 Add items To qb-core - > shared -> items.lua

------------------ md-drugs
["coke"] 		 	 	 	     = {["name"] = "coke",           				["label"] = "Raw Cocaine", 				["weight"] = 1000,		["type"] = "item", 		["image"] = "coke.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,    ["description"] = "Processed cocaine"},
 ["coca_leaf"] 		 	 	 	 = {["name"] = "coca_leaf",           			["label"] = "Cocaine leaves",	 		["weight"] = 1500,		["type"] = "item", 		["image"] = "coca_leaf.png", 			["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,    ["description"] = "Cocaine leaves that must be processed !"},
 ["poppyresin"] 		 	 	 = {["name"] = "poppyresin",           			["label"] = "Poppy resin",	 			["weight"] = 2000, 		["type"] = "item", 		["image"] = "poppyresin.png", 			["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,    ["description"] = "It sticks to your fingers when you handle it."},
 ["heroin"] 		 	 	 	 = {["name"] = "heroin",           				["label"] = "Weak Heroin Powder",	 				["weight"] = 500, 		["type"] = "item", 		["image"] = "loosecoke.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = "Really addictive depressant..."},
 ["bakingsoda"] 		 	 	 = {["name"] = "bakingsoda",           		    ["label"] = "Baking Soda",	 		    ["weight"] = 300, 		["type"] = "item", 		["image"] = "bakingsoda.png", 		    ["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = "Household Baking Soda!"},
 ['loosecoke'] 			 		 = {['name'] = 'loosecoke', 			  		['label'] = 'loose coke', 				['weight'] = 100, 		['type'] = 'item', 		['image'] = 'loosecoke.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = 'Paper made specifically for encasing and smoking tobacco or cannabis.'},
['loosecokestagetwo'] 			 = {['name'] = 'loosecokestagetwo', 			  		['label'] = 'More Pure Loose Coke', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'loosecokestagetwo.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['loosecokestagethree'] 		 = {['name'] = 'loosecokestagethree', 			  		['label'] = 'Purest Loose Coke', 				['weight'] = 100, 		['type'] = 'item', 		['image'] = 'loosecokestagethree.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['cokebaggystagetwo'] 			 = {['name'] = 'cokebaggystagetwo', 			  	  	['label'] = 'Bag of Good Coke', 				['weight'] = 100, 		['type'] = 'item', 		['image'] = 'cocaine_baggystagetwo.png', 		['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,    ['combinable'] = nil,   ['description'] = 'To get happy real quick'},
['cokebaggystagethree'] 		 = {['name'] = 'cokebaggystagethree', 			  	  	['label'] = 'Bag of Great Coke', 				['weight'] = 100, 		['type'] = 'item', 		['image'] = 'cocaine_baggystagethree.png', 		['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,    ['combinable'] = nil,   ['description'] = 'To get happy real quick'},
["cokestagetwo"] 		 	 	 = {["name"] = "cokestagetwo",           				["label"] = "Better Raw Cocaine", 				["weight"] = 100,		["type"] = "item", 		["image"] = "cokestagetwo.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,   ["expire"] = 90,  ["description"] = "Processed cocaine"},
["cokestagethree"] 		 	 	 = {["name"] = "cokestagethree",           				["label"] = "Best Raw Cocaine", 				["weight"] = 100,		["type"] = "item", 		["image"] = "cokestagethree.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,   ["expire"] = 90,  ["description"] = "Processed cocaine"},
 
 
 ----md-lsd
 ['lysergic_acid'] 			 	 = {['name'] = 'lysergic_acid', 			  		['label'] = 'Lysergic Acid', 				['weight'] = 100, 		['type'] = 'item', 		['image'] = 'lysergic_acid.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['diethylamide'] 			 	 = {['name'] = 'diethylamide', 			  			['label'] = 'Diethylamide', 				['weight'] = 100, 		['type'] = 'item', 		['image'] = 'diethylamide.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['lsd_one_vial'] 			 	 = {['name'] = 'lsd_one_vial', 			  			['label'] = 'Weakest LSD Vial', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'lsd_one_vial.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['lsd_vial_two'] 			 	 = {['name'] = 'lsd_vial_two', 			  			['label'] = 'Slightly Better LSD Vial', 	['weight'] = 100, 		['type'] = 'item', 		['image'] = 'lsd_vial_two.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['lsd_vial_three'] 			 	 = {['name'] = 'lsd_vial_three', 			  		['label'] = 'Good LSD', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'lsd_vial_three.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['lsd_vial_four'] 			 	 = {['name'] = 'lsd_vial_four', 			  		['label'] = 'Really Good LSD', 				['weight'] = 100, 		['type'] = 'item', 		['image'] = 'lsd_vial_four.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['lsd_vial_five'] 			 	 = {['name'] = 'lsd_vial_five', 			  		['label'] = 'Amazing LSD', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'lsd_vial_five.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['lsd_vial_six'] 			 	 = {['name'] = 'lsd_vial_six', 			  			['label'] = 'Purest LSD', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'lsd_vial_six.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['tab_paper'] 			 		 = {['name'] = 'tab_paper', 			  			['label'] = 'Tab Paper', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'tab_paper.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['smileyfacesheet'] 			 = {['name'] = 'smileyfacesheet', 			  		['label'] = 'Smiley Face Sheet', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'smileysheet.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['wildcherrysheet'] 			 = {['name'] = 'wildcherrysheet', 			  		['label'] = 'Wild Cherry Sheet', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'wildcherrysheet.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['yinyangsheet'] 			     = {['name'] = 'yinyangsheet', 			  			['label'] = 'Yin and Yang Sheet', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'yinyangsheet.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['pineapplesheet'] 			 	 = {['name'] = 'pineapplesheet', 			  		['label'] = 'Pineapple Sheet', 				['weight'] = 100, 		['type'] = 'item', 		['image'] = 'pineapplesheet.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['bartsheet'] 			 		 = {['name'] = 'bartsheet', 			  			['label'] = 'Bart Simpson Sheet', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'bartsheet.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['gratefuldeadsheet'] 			 = {['name'] = 'gratefuldeadsheet', 			  	['label'] = 'Grateful Dead Sheet', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'gratefuldeadsheet.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['smiley_tabs'] 			     = {['name'] = 'smiley_tabs', 			  			['label'] = 'Smiley tabs', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'smiley_tabs.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['wildcherry_tabs'] 			 = {['name'] = 'wildcherry_tabs', 			  		['label'] = 'Wild Cherry Tabs', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'wildcherry_tabs.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['yinyang_tabs'] 			     = {['name'] = 'yinyang_tabs', 			  			['label'] = 'Yin and Yang Tabs', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'yinyang_tabs.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['pineapple_tabs'] 			     = {['name'] = 'pineapple_tabs', 			  		['label'] = 'Pineapple Tabs', 				['weight'] = 100, 		['type'] = 'item', 		['image'] = 'pineapple_tabs.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['bart_tabs'] 			 		 = {['name'] = 'bart_tabs', 			  			['label'] = 'Bart Simpson Tabs', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'bart_tabs.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['gratefuldead_tabs'] 			 = {['name'] = 'gratefuldead_tabs', 			  	['label'] = 'Grateful Dead Tabs', 			['weight'] = 100, 		['type'] = 'item', 		['image'] = 'gratefuldead_tabs.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = 'DONT USE IF YOU GET SEIZURES'},
['lsdlabkit'] 			 		= {['name'] = 'lsdlabkit', 			  				['label'] = 'LSD Mixing Table', 			['weight'] = 1000, 		['type'] = 'item', 		['image'] = 'labkit.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['dirtylsdlabkit'] 			 	= {['name'] = 'dirtylsdlabkit', 			  		['label'] = 'Dirty LSD Mixing Table', 			['weight'] = 1000, 		['type'] = 'item', 		['image'] = 'labkit_dirty.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},


---- md heroin
["heroinstagetwo"] 		 	 	 	 = {["name"] = "heroinstagetwo",           			["label"] = "Better Heroin",	 		["weight"] = 250, 		["type"] = "item", 		["image"] = "heroinpowderstagetwo.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
["heroinstagethree"] 		 	 	 = {["name"] = "heroinstagethree",           		["label"] = "Best Heroin",	 			["weight"] = 250, 		["type"] = "item", 		["image"] = "heroinpowderstagethree.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
["heroincut"] 		 	 	 		 = {["name"] = "heroincut",           				["label"] = "Cut Heroin",	 			["weight"] = 250, 		["type"] = "item", 		["image"] = "heroinpowder.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,   	     ["description"] = ""},
["heroincutstagetwo"] 		 	 	 = {["name"] = "heroincutstagetwo",           		["label"] = "Better Cut Heroin",	 	["weight"] = 250, 		["type"] = "item", 		["image"] = "heroinpowderstagethree.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,   		 ["description"] = ""},
["heroincutstagethree"] 		 	 = {["name"] = "heroincutstagethree",           	["label"] = "Best Cut Heroin",	 		["weight"] = 250, 		["type"] = "item", 		["image"] = "heroinpowderstagetwo.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,  	     ["description"] = ""},
["heroinlabkit"] 		 	 	 	 = {["name"] = "heroinlabkit",           			["label"] = "Heroin Lab Kit",	 		["weight"] = 250, 		["type"] = "item", 		["image"] = "labkit.png", 				["unique"] = false, 	["useable"] = true, 	["shouldClose"] = false,   ["combinable"] = nil,   		 ["description"] = ""},
["dirtyheroinlabkit"] 		 	 	 = {["name"] = "dirtyheroinlabkit",           		["label"] = "Dirty heroin Lab Kit",	 	["weight"] = 250, 		["type"] = "item", 		["image"] = "labkit_dirty.png", 		["unique"] = false, 	["useable"] = true, 	["shouldClose"] = false,   ["combinable"] = nil,   		 ["description"] = ""},
["heroinvial"] 		 	 	 		 = {["name"] = "heroinvial",           				["label"] = "Vial Of Heroin",	 		["weight"] = 250, 		["type"] = "item", 		["image"] = "heroin.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
["heroinvialstagetwo"] 		 	 	 = {["name"] = "heroinvialstagetwo",           		["label"] = "Better Vial of Heroin",	["weight"] = 250, 		["type"] = "item", 		["image"] = "heroinstagetwo.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
["heroinvialstagethree"] 		 	 = {["name"] = "heroinvialstagethree",           	["label"] = "Best Vial Of Heroin",	 	["weight"] = 250, 		["type"] = "item", 		["image"] = "heroinstagethree.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
["heroin_ready"] 		 	 	 	 = {["name"] = "heroin_ready",           			["label"] = "Syringe Of Heroin",	 	["weight"] = 250, 		["type"] = "item", 		["image"] = "heroin_ready.png", 				["unique"] = false, 	["useable"] = true, 	["shouldClose"] = false,   ["combinable"] = nil,   		 ["description"] = ""},
["heroin_readystagetwo"] 		 	 = {["name"] = "heroin_readystagetwo",           	["label"] = "Syringe Of Better Heroin",	 ["weight"] = 250, 		["type"] = "item", 		["image"] = "heroin_readystagetwo.png", 				["unique"] = false, 	["useable"] = true, 	["shouldClose"] = false,   ["combinable"] = nil,         ["description"] = ""},
["heroin_readystagethree"] 		 	 = {["name"] = "heroin_readystagethree",           	["label"] = "Syringe Of Best Heroin",	 ["weight"] = 250, 		["type"] = "item", 		["image"] = "heroin_readystagethree.png", 				["unique"] = false, 	["useable"] = true, 	["shouldClose"] = false,   ["combinable"] = nil,         ["description"] = ""},
["emptyvial"] 				     = {["name"] = "emptyvial", 			    	["label"] = "empty vial", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "emptyvial.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ['combinable'] = nil,					 ["description"] = ""},
["needle"] 		 					 = {["name"] = "needle",           					["label"] = "Syringe",	 				["weight"] = 250, 		["type"] = "item", 		["image"] = "syringe.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,         ["description"] = ""},


---- md crack
["crackrock"] 		 	 	 		 = {["name"] = "crackrock",           				["label"] = "Crack Rock",	 			["weight"] = 250, 		["type"] = "item", 		["image"] = "crackrock1.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
["crackrockstagetwo"] 		 	 	 = {["name"] = "crackrockstagetwo",           		["label"] = "Better Crack Rock",	 	["weight"] = 250, 		["type"] = "item", 		["image"] = "crackrock2.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
["crackrockstagethree"] 		 	 = {["name"] = "crackrockstagethree",           	["label"] = "Best Crack Rock",	 		["weight"] = 250, 		["type"] = "item", 		["image"] = "crackrock3.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,   	     ["description"] = ""},
["baggedcracked"] 		 	 	 	 = {["name"] = "baggedcracked",           			["label"] = "Bag Of Crack",	 			["weight"] = 250, 		["type"] = "item", 		["image"] = "crackbag1.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,   		 ["description"] = ""},
["baggedcrackedstagetwo"] 		 	 = {["name"] = "baggedcrackedstagetwo",           	["label"] = "Better Bag Of Crack",	 	["weight"] = 250, 		["type"] = "item", 		["image"] = "crackbag2.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,  	     ["description"] = ""},
["baggedcrackedstagethree"] 		 = {["name"] = "baggedcrackedstagethree",           ["label"] = "Best Bag Of Crack",	 	["weight"] = 250, 		["type"] = "item", 		["image"] = "crackbag3.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,  	     ["description"] = ""},
---- shrooms 
["shrooms"] 		 = {["name"] = "shrooms",           ["label"] = "Shrooms",	 	["weight"] = 250, 		["type"] = "item", 		["image"] = "shrooms.png", 				["unique"] = false, 	["useable"] = true, 	["shouldClose"] = false,   ["combinable"] = nil,  	     ["description"] = ""},

-- pharma
['prescription_pad'] 		        	= {['name'] = 'prescription_pad', 			    ['label'] = 'Prescription Pad',            	['weight'] = 10,      ['type'] = 'item',       ['image'] = 'prescriptionpad.png',         	        ['unique'] = true,     ['useable'] = true,     ['shouldClose'] = false,     ['combinable'] = nil,   ['description'] = ''},
["vicodin_prescription"] 		 	 	 		 = {["name"] = "vicodin_prescription",           				["label"] = "Vicodin Prescription",	 			["weight"] = 250, 		["type"] = "item", 		["image"] = "vicodinprescription.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
["adderal_prescription"] 		 	 	 		 = {["name"] = "adderal_prescription",           				["label"] = "Adderal Prescription",	 			["weight"] = 250, 		["type"] = "item", 		["image"] = "adderalprescription.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
["morphine_prescription"] 		 	 	 		 = {["name"] = "morphine_prescription",           				["label"] = "Morphine Prescription",	 			["weight"] = 250, 		["type"] = "item", 		["image"] = "morphineprescription.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
["xanax_prescription"] 		 	 	 		 = {["name"] = "xanax_prescription",           				["label"] = "Xanax Prescription",	 			["weight"] = 250, 		["type"] = "item", 		["image"] = "xanaxprescription.png", 				["unique"] = false, 	["useable"] = false, 	["shouldClose"] = false,   ["combinable"] = nil,     ["description"] = ""},
['adderal'] 			     = {['name'] = 'adderal', 			  			['label'] = 'Adderal', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'adderal.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['vicodin'] 			     = {['name'] = 'vicodin', 			  			['label'] = 'Vicodin', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'vicodin.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['morphine'] 			     = {['name'] = 'morphine', 			  			['label'] = 'Morphine', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'morphine.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['xanax'] 			     = {['name'] = 'xanax', 			  			['label'] = 'Xanax', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'xanax.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['adderalbottle'] 			     = {['name'] = 'adderalbottle', 			  			['label'] = 'Adderal Bottle', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'pillbottle.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['vicodinbottle'] 			     = {['name'] = 'vicodinbottle', 			  			['label'] = 'Vicodin Bottle', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'pillbottle.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['morphinebottle'] 			     = {['name'] = 'morphinebottle', 			  			['label'] = 'Morphine Bottle', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'pillbottle.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
['xanaxbottle'] 			     = {['name'] = 'xanaxbottle', 			  			['label'] = 'Xanax Bottle', 					['weight'] = 100, 		['type'] = 'item', 		['image'] = 'pillbottle.png', 			['unique'] = false, 	['useable'] = true, 	['shouldClose'] = true,	   ['combinable'] = nil, 					 ['description'] = ''},
---------------- XTC
['isosafrole'] 			 		 = {['name'] = 'isosafrole', 			  		['label'] = 'isosafrole', 						 ['weight'] = 100, 		['type'] = 'item', 		['image'] = 'isosafrole.png', 			['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil,					 ['description'] = ''},
['mdp2p'] 			 			 = {['name'] = 'mdp2p', 			  			['label'] = 'mdp2p', 							 ['weight'] = 100, 		['type'] = 'item', 		['image'] = 'mdp2p.png', 				['unique'] = false, 	['useable'] = false, 	['shouldClose'] = true,	   ['combinable'] = nil,				     ['description'] = ''},
["raw_xtc"] 				     = {["name"] = "raw_xtc", 			    		["label"] = "Raw XTC", 			       			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "raw_xtc.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["singlepress"] 				 = {["name"] = "singlepress", 			    	["label"] = "Single Pill Press", 			     ['weight'] = 100, 		["type"] = "item", 		["image"] = "pillpress.png", 		    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_xtc"] 				     = {["name"] = "white_xtc", 			    	["label"] = "Single Stack White XTC", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedwhite.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_xtc2"] 				     = {["name"] = "white_xtc2", 			    	["label"] = "Dual Stack White XTC", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedwhite.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_xtc3"] 				     = {["name"] = "white_xtc3", 			    	["label"] = "Triple Stack White XTC", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedwhite.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_xtc4"] 				     = {["name"] = "white_xtc4", 			    	["label"] = "Quad Stack White XTC", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedwhite.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_xtc"] 				     = {["name"] = "red_xtc", 			    		["label"] = "Single Stack Red XTC", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedred.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_xtc2"] 				     = {["name"] = "red_xtc2", 			    		["label"] = "Dual Stack Red XTC", 			     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedred.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_xtc3"] 				     = {["name"] = "red_xtc3", 			    		["label"] = "Triple Stack Red XTC", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedred.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_xtc4"] 				     = {["name"] = "red_xtc4", 			    		["label"] = "Quad Stack Red XTC", 			     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedred.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_xtc"] 				     = {["name"] = "orange_xtc", 			    	["label"] = "Single Stack Orange XTC", 	         ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedorange.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_xtc2"] 				 = {["name"] = "orange_xtc2", 			   		["label"] = "Dual Stack Orange XTC", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedorange.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_xtc3"] 				 = {["name"] = "orange_xtc3", 			   		["label"] = "Triple Stack Orange XTC", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedorange.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_xtc4"] 				 = {["name"] = "orange_xtc4", 			   		["label"] = "Quad Stack Orange XTC", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedorange.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_xtc"] 				     = {["name"] = "blue_xtc", 			    		["label"] = "Single Stack Blue XTC", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedblue.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_xtc2"] 				     = {["name"] = "blue_xtc2", 			    	["label"] = "Dual Stack Blue XTC", 			     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedblue.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_xtc3"] 				     = {["name"] = "blue_xtc3", 			    	["label"] = "Triple Stack Blue XTC", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedblue.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_xtc4"] 				     = {["name"] = "blue_xtc4", 			    	["label"] = "Quad Stack Blue XTC", 			     ['weight'] = 100, 		["type"] = "item", 		["image"] = "unstampedblue.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_playboys"] 				 = {["name"] = "white_playboys", 			    ["label"] = "Single Stack White Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_white.png", 		["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_playboys2"] 			 = {["name"] = "white_playboys2", 			    ["label"] = "Dual Stack White Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_white.png", 		["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_playboys3"] 			 = {["name"] = "white_playboys3", 			    ["label"] = "Triple Stack White Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_white.png", 		["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_playboys4"] 			 = {["name"] = "white_playboys4", 			    ["label"] = "Quad Stack White Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_white.png", 		["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_playboys"] 				 = {["name"] = "blue_playboys", 			    ["label"] = "Single Stack blue Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_playboys2"] 				 = {["name"] = "blue_playboys2", 			    ["label"] = "Dual Stack blue Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_playboys3"] 				 = {["name"] = "blue_playboys3", 			    ["label"] = "Triple Stack blue Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_playboys4"] 				 = {["name"] = "blue_playboys4", 			    ["label"] = "Quad Stack blue Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_playboys"] 				 = {["name"] = "red_playboys", 			    	["label"] = "Single Stack red Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_playboys2"] 				 = {["name"] = "red_playboys2", 			    ["label"] = "Dual Stack red Playboys", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_playboys3"] 				 = {["name"] = "red_playboys3", 			    ["label"] = "Triple Stack red Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_playboys4"] 				 = {["name"] = "red_playboys4", 			    ["label"] = "Quad Stack red Playboys", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_playboys"] 			 = {["name"] = "orange_playboys", 			    ["label"] = "Single Stack orange Playboys", 	 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_orange.png", 		["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_playboys2"] 			 = {["name"] = "orange_playboys2", 			    ["label"] = "Dual Stack orange Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_orange.png", 		["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_playboys3"] 			 = {["name"] = "orange_playboys3", 			    ["label"] = "Triple Stack orange Playboys", 	 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_orange.png", 		["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_playboys4"] 			 = {["name"] = "orange_playboys4", 			    ["label"] = "Quad Stack orange Playboys", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "playboy_orange.png", 		["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_aliens"] 				 = {["name"] = "white_aliens", 			    	["label"] = "Single Stack White aliens", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_aliens2"] 				 = {["name"] = "white_aliens2", 			    ["label"] = "Dual Stack White aliens", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_aliens3"] 				 = {["name"] = "white_aliens3", 			    ["label"] = "Triple Stack White aliens", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_aliens4"] 				 = {["name"] = "white_aliens4", 			    ["label"] = "Quad Stack White aliens", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_aliens"] 				 = {["name"] = "blue_aliens", 			    	["label"] = "Single Stack blue aliens", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_aliens2"] 				 = {["name"] = "blue_aliens2", 			    	["label"] = "Dual Stack blue aliens", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_aliens3"] 				 = {["name"] = "blue_aliens3", 			    	["label"] = "Triple Stack blue aliens", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_aliens4"] 				 = {["name"] = "blue_aliens4", 			    	["label"] = "Quad Stack blue aliens", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_aliens"] 					 = {["name"] = "red_aliens", 			    	["label"] = "Single Stack red aliens", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_aliens2"] 				 = {["name"] = "red_aliens2", 			    	["label"] = "Dual Stack red aliens", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_aliens3"] 				 = {["name"] = "red_aliens3", 			    	["label"] = "Triple Stack red aliens", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_aliens4"] 				 = {["name"] = "red_aliens4", 			    	["label"] = "Quad Stack red aliens", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_aliens"] 				 = {["name"] = "orange_aliens", 			    ["label"] = "Single Stack orange aliens", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_aliens2"] 				 = {["name"] = "orange_aliens2", 			    ["label"] = "Dual Stack orange aliens", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_aliens3"] 				 = {["name"] = "orange_aliens3", 			    ["label"] = "Triple Stack orange aliens", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_aliens4"] 				 = {["name"] = "orange_aliens4", 			    ["label"] = "Quad Stack orange aliens", 		 ['weight'] = 100, 		["type"] = "item", 		["image"] = "alien_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_pl"] 					 = {["name"] = "white_pl", 			    		["label"] = "Single Stack White pl", 			 ['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_white.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_pl2"] 					 = {["name"] = "white_pl2", 			    	["label"] = "Dual Stack White pl", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_white.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_pl3"] 					 = {["name"] = "white_pl3", 			    	["label"] = "Triple Stack White pl", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_white.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_pl4"] 					 = {["name"] = "white_pl4", 			    	["label"] = "Quad Stack White pl", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_white.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_pl"] 					 = {["name"] = "blue_pl", 			    		["label"] = "Single Stack blue pl", 		 	['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_blue.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_pl2"] 					 = {["name"] = "blue_pl2", 			    		["label"] = "Dual Stack blue pl", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_blue.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_pl3"] 					 = {["name"] = "blue_pl3", 			    		["label"] = "Triple Stack blue pl", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_blue.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_pl4"] 					 = {["name"] = "blue_pl4", 			    		["label"] = "Quad Stack blue pl", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_blue.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_pl"] 						 = {["name"] = "red_pl", 			    		["label"] = "Single Stack red pl", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_red.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_pl2"] 					 = {["name"] = "red_pl2", 			    		["label"] = "Dual Stack red pl", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_red.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_pl3"] 					 = {["name"] = "red_pl3", 			    		["label"] = "Triple Stack red pl", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_red.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_pl4"] 					 = {["name"] = "red_pl4", 			    		["label"] = "Quad Stack red pl", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_red.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_pl"] 					 = {["name"] = "orange_pl", 			    	["label"] = "Single Stack orange pl", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_pl2"] 					 = {["name"] = "orange_pl2", 			    	["label"] = "Dual Stack orange pl", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_pl3"] 					 = {["name"] = "orange_pl3", 			    	["label"] = "Triple Stack orange pl", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_pl4"] 					 = {["name"] = "orange_pl4", 			    	["label"] = "Quad Stack orange pl", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "PL_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_trolls"] 				 = {["name"] = "white_trolls", 			    	["label"] = "Single Stack White trolls", 		['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_trolls2"] 				 = {["name"] = "white_trolls2", 			    ["label"] = "Dual Stack White trolls", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_trolls3"] 				 = {["name"] = "white_trolls3", 			    ["label"] = "Tritrollse Stack White trolls", 	['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_trolls4"] 				 = {["name"] = "white_trolls4", 			    ["label"] = "Quad Stack White trolls", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_trolls"] 				 = {["name"] = "blue_trolls", 			    	["label"] = "Single Stack blue trolls", 		['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_trolls2"] 				 = {["name"] = "blue_trolls2", 			    	["label"] = "Dual Stack blue trolls", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_trolls3"] 				 = {["name"] = "blue_trolls3", 			    	["label"] = "Tritrollse Stack blue trolls",     ['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_trolls4"] 				 = {["name"] = "blue_trolls4", 			    	["label"] = "Quad Stack blue trolls", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_trolls"] 					 = {["name"] = "red_trolls", 			    	["label"] = "Single Stack red trolls", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_trolls2"] 				 = {["name"] = "red_trolls2", 			    	["label"] = "Dual Stack red trolls", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_trolls3"] 				 = {["name"] = "red_trolls3", 			    	["label"] = "Tritrollse Stack red trolls", 		['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_trolls4"] 				 = {["name"] = "red_trolls4", 			    	["label"] = "Quad Stack red trolls", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_trolls"] 				 = {["name"] = "orange_trolls", 			    ["label"] = "Single Stack orange trolls", 	    ['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_trolls2"] 				 = {["name"] = "orange_trolls2", 			    ["label"] = "Dual Stack orange trolls", 	    ['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_trolls3"] 				 = {["name"] = "orange_trolls3", 			    ["label"] = "Tritrollse Stack orange trolls", 	['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_trolls4"] 				 = {["name"] = "orange_trolls4", 			    ["label"] = "Quad Stack orange trolls", 		['weight'] = 100, 		["type"] = "item", 		["image"] = "troll_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_cats"] 					 = {["name"] = "white_cats", 			    	["label"] = "Single Stack White cats", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_cats2"] 				 = {["name"] = "white_cats2", 			    	["label"] = "Dual Stack White cats", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_cats3"] 				 = {["name"] = "white_cats3", 			    	["label"] = "Tricatse Stack White cats", 		['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["white_cats4"] 				 = {["name"] = "white_cats4", 			    	["label"] = "Quad Stack White cats", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_white.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_cats"] 					 = {["name"] = "blue_cats", 			    	["label"] = "Single Stack blue cats", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_cats2"] 					 = {["name"] = "blue_cats2", 			    	["label"] = "Dual Stack blue cats", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_cats3"] 					 = {["name"] = "blue_cats3", 			    	["label"] = "Tricatse Stack blue cats", 		['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["blue_cats4"] 					 = {["name"] = "blue_cats4", 			    	["label"] = "Quad Stack blue cats", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_blue.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_cats"] 					 = {["name"] = "red_cats", 			    		["label"] = "Single Stack red cats", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_cats2"] 					 = {["name"] = "red_cats2", 			    	["label"] = "Dual Stack red cats", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_cats3"] 					 = {["name"] = "red_cats3", 			    	["label"] = "Tricatse Stack red cats", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["red_cats4"] 					 = {["name"] = "red_cats4", 			    	["label"] = "Quad Stack red cats", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_red.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_cats"] 				 = {["name"] = "orange_cats", 			    	["label"] = "Single Stack orange cats", 	    ['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_cats2"] 				 = {["name"] = "orange_cats2", 			    	["label"] = "Dual Stack orange cats", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_cats3"] 				 = {["name"] = "orange_cats3", 			    	["label"] = "Tricatse Stack orange cats", 		['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["orange_cats4"] 				 = {["name"] = "orange_cats4", 			    	["label"] = "Quad Stack orange cats", 			['weight'] = 100, 		["type"] = "item", 		["image"] = "kitty_orange.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["dualpress"] 				 	 = {["name"] = "dualpress", 			    	["label"] = "Dual Pill Press", 			        ['weight'] = 100, 		["type"] = "item", 		["image"] = "pillpress.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["triplepress"] 				 = {["name"] = "triplepress", 			    	["label"] = "Triple Pill Press", 			    ['weight'] = 100, 		["type"] = "item", 		["image"] = "pillpress.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["quadpress"] 					 = {["name"] = "quadpress", 			    	["label"] = "Quad Pill Press", 			        ['weight'] = 100, 		["type"] = "item", 		["image"] = "pillpress.png", 			["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["spores"] 				     = {["name"] = "spores", 			    	["label"] = "spores", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "shrooms.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["cokeburner"] 				     = {["name"] = "cokeburner", 			    	["label"] = "Nokia Burner", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "cokeburner.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["crackburner"] 				     = {["name"] = "crackburner", 			    	["label"] = "Juke Burner", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "crackburner.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["heroinburner"] 				     = {["name"] = "heroinburner", 			    	["label"] = "Razr Burner", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "heroinburner.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["lsdburner"] 				     = {["name"] = "lsdburner", 			    	["label"] = "SideKick Burner", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "lsdburner.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["cactusbulb"] 				     = {["name"] = "cactusbulb", 			    	["label"] = "Cactus Bulb", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "cactusbulb.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},
["driedmescaline"] 				     = {["name"] = "driedmescaline", 			    	["label"] = "Mescaline", 		     ['weight'] = 100, 		["type"] = "item", 		["image"] = "driedmescaline.png", 			    ["unique"] = false, 	["useable"] = true, 	["shouldClose"] = true,	   ["combinable"] = nil,                     ["description"] = ""},

STEP 2 Add Images from HTML folder to your inventory script images location

STEP 3 delete qb-drugs

STEP 4 INSIDE OF THE CONFIG change all the locations for everything. All locations have been changed to make a preview video easier and with how big of a script this is, Im not going to config it for every city. Its simple vec3 and vec4s. Make your city unique so its not in the same location as every other city.

STEP 5 in qb-radial menu search for

event = 'qb-drugs:client:cornerselling',

and switch it to this

event = 'md-drugs:client:cornerselling',

STEP 6 in qb-adminmenu look for

QBCore.Functions.CreateCallback('test:getdealers', function(_, cb)
    cb(exports['qb-drugs']:GetDealers())
end)

in the server.lua and change it to

QBCore.Functions.CreateCallback('test:getdealers', function(_, cb)
    cb(exports['md-drugs']:GetDealers())
end)

STEP 7 in Cornerselling.lua line 30 and line 148 of deliveries.lua on the client side for both place your Police Alert inside that function. for example something like ps-dispatch will look like this

local function PoliceCall()
   local currentPos = GetEntityCoords(PlayerPedId())
                TriggerServerEvent("dispatch:server:notify", {
                dispatchcodename = "susactivity", -- has to match the codes in sv_dispatchcodes.lua so that it generates the right blip
                dispatchCode = "10-66",
                origin = {
                    x = currentPos.x,
                    y = currentPos.y,
                    z = currentPos.z
                },
                dispatchMessage = "Oxy Runs",
                job = {"LEO", "police"} -- has to match the recipientList in sv_dispatchcodes.lua
            })
end

sanathks/workspace.nvim

2023-10-11T01:34:03Z

Manage projects within workspaces using Tmux sessions

workspace.nvim

workspace.nvim is a Neovim plugin that allows you to manage Tmux sessions for your projects and workspaces in a simple and efficient way.

Introduction

Inspired by ThePrimeagen's tmux-sessionizer

Features

Create and manage Tmux sessions for different projects and workspaces.
Easily switch between Tmux sessions associated with your projects.

Installation

lazy.nvim

{
  "sanathks/workspace.nvim",
  dependencies = {"nvim-telescope/telescope.nvim"},
  config = function()
    require("workspace").setup({
      workspaces = {
        { name = "Work",  path = "~/projects/work",  keymap = { "<leader>w" } },
        { name = "Hobby", path = "~/projects/hobby", keymap = { "<leader>p" } },
      }
    })
  end,
}

Switch between tmux sessions

 local workspace = require("workspace")
 vim.keymap.set('n', '<leader>ps', workspace.tmux_sessions)

Customize the session name generation

with the tmux_session_name_generator option you can provide a custom session name generator, which allows you to make the session name unique across multiple workspaces.

function(project_name, workspace_name)
  local suffix = string.sub(workspace_name, 1, 2)
  local session_name = string.upper(project_name) .. "_" .. suffix
   return session_name
end

Contributing

Contributions are welcome! If you find any issues or have ideas for improvements, please open an issue or submit a pull request.

brainstorm-tools/brainstorm3

2023-10-10T01:34:28Z

Brainstorm software: MEG, EEG, fNIRS, ECoG, sEEG and electrophysiology

Bitcaster/helm-charts

2023-10-11T01:35:02Z

Forks of existent or own charts

profmaad/librabbitmq-objc

2023-10-10T01:35:49Z

Objective-C wrapper for librabbitmq-c

librabbitmq-objc: Objective-C wrapper for librabbitmq-c

Description

This is a simple Objective-C wrapper for librabbitmq-c, a C AMQP library from the people developing the RabbitMQ AMQP server.

Requirements

Cocoa
librabbitmq-c including headers

Build

There is nothing to build. Just include the source and header files into your Xcode project and link it againt librabbitmq-c.

Usage

Establish a connection the server using AMQPConnection
Open a communication channel using AMQPConnection - (AMQPChannel*)openChannel
Create queues using AMQPQueue or exchanges using AMQPExchange

Connect queues to exchanges using AMQPQueue:

 - (void)bindToExchange:(AMQPExchange*)theExchange
                withKey:(NSString*)bindingKey

To receive messages, use an AMQPConsumer bound to a queue:

    - (id)initForQueue:(AMQPQueue*)theQueue
             onChannel:(AMQPChannel*)theChannel
   useAcknowledgements:(BOOL)ack
           isExclusive:(BOOL)exclusive
  receiveLocalMessages:(BOOL)local

For ease of use, there is a complete threaded consumer implementation at AMQPConsumerThread using a AMQPConsumerThreadDelegate to handle received messages

To publish messages on an exchange, use AMQPExchange:

 - (void)publishMessage:(NSString*)body
        usingRoutingKey:(NSString*)theRoutingKey

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see http://www.gnu.org/licenses/.

JanX2/JXLS

2023-10-10T01:35:49Z

A Matcher Framework for Objective-C/Cocoa

Expecta

A matcher framework for Objective-C and Cocoa.

FEATURES

The main advantage of using Expecta over other matcher frameworks is that you do not have to specify the data types. Also, the syntax of Expecta matchers is much more readable and does not suffer from parenthesitis.

expect(@"foo").to.equal(@"foo"); // `to` is a syntactic sugar and can be safely omitted.
expect(foo).notTo.equal(1);
expect([bar isBar]).to.equal(YES);
expect(baz).to.equal(3.14159);

Expecta is framework-agnostic: it works well with XCTest and XCTest-compatible test frameworks such as Specta, or Kiwi.

MATCHERS

expect(x).to.equal(y); compares objects or primitives x and y and passes if they are identical (==) or equivalent isEqual:).

expect(x).to.beIdenticalTo(y); compares objects x and y and passes if they are identical and have the same memory address.

expect(x).to.beNil(); passes if x is nil.

expect(x).to.beTruthy(); passes if x evaluates to true (non-zero).

expect(x).to.beFalsy(); passes if x evaluates to false (zero).

expect(x).to.contain(y); passes if an instance of NSArray or NSString x contains y.

expect(x).to.beSupersetOf(y); passes if an instance of NSArray, NSSet, NSDictionary or NSOrderedSet x contains all elements of y.

expect(x).to.haveCountOf(y); passes if an instance of NSArray, NSSet, NSDictionary or NSString x has a count or length of y.

expect(x).to.beEmpty(); passes if an instance of NSArray, NSSet, NSDictionary or NSString x has a count or length of .

expect(x).to.beInstanceOf([Foo class]); passes if x is an instance of a class Foo.

expect(x).to.beKindOf([Foo class]); passes if x is an instance of a class Foo or if x is an instance of any class that inherits from the class Foo.

expect([Foo class]).to.beSubclassOf([Bar class]); passes if the class Foo is a subclass of the class Bar or if it is identical to the class Bar. Use beKindOf() for class clusters.

expect(x).to.beLessThan(y); passes if x is less than y.

expect(x).to.beLessThanOrEqualTo(y); passes if x is less than or equal to y.

expect(x).to.beGreaterThan(y); passes if x is greater than y.

expect(x).to.beGreaterThanOrEqualTo(y); passes if x is greater than or equal to y.

expect(x).to.beInTheRangeOf(y,z); passes if x is in the range of y and z.

expect(x).to.beCloseTo(y); passes if x is close to y.

expect(x).to.beCloseToWithin(y, z); passes if x is close to y within z.

expect(^{ /* code */ }).to.raise(@"ExceptionName"); passes if a given block of code raises an exception named ExceptionName.

expect(^{ /* code */ }).to.raiseAny(); passes if a given block of code raises any exception.

expect(x).to.conformTo(y); passes if x conforms to the protocol y.

expect(x).to.respondTo(y); passes if x responds to the selector y.

expect(^{ /* code */ }).to.notify(@"NotificationName"); passes if a given block of code generates an NSNotification named NotificationName.

expect(^{ /* code */ }).to.notify(notification); passes if a given block of code generates an NSNotification equal to the passed notification.

expect(x).to.beginWith(y); passes if an instance of NSString, NSArray, or NSOrderedSet x begins with y. Also liased by startWith

expect(x).to.endWith(y); passes if an instance of NSString, NSArray, or NSOrderedSet x ends with y.

expect(x).to.match(y); passes if an instance of NSString x matches regular expression (given as NSString) y one or more times.

Inverting Matchers

Every matcher's criteria can be inverted by prepending .notTo or .toNot:

expect(x).notTo.equal(y); compares objects or primitives x and y and passes if they are not equivalent.

Asynchronous Testing

Every matcher can be made to perform asynchronous testing by prepending .will, .willNot or after(...):

expect(x).will.beNil(); passes if x becomes nil before the default timeout.

expect(x).willNot.beNil(); passes if x becomes non-nil before the default timeout.

expect(x).after(3).to.beNil(); passes if x becoms nil after 3.0 seconds.

expect(x).after(2.5).notTo.equal(42); passes if x doesn't equal 42 after 2.5 seconds.

The default timeout is 1.0 second and is used for all matchers if not otherwise specified. This setting can be changed by calling [Expecta setAsynchronousTestTimeout:x], where x is the desired timeout in seconds.

describe(@"Foo", ^{
  beforeAll(^{
    // All asynchronous matching using `will` and `willNot`
    // will have a timeout of 2.0 seconds
    [Expecta setAsynchronousTestTimeout:2];
  });

  it(@"will not be nil", ^{
    // Test case where default timeout is used
    expect(foo).willNot.beNil();
  });

  it(@"should equal 42 after 3 seconds", ^{
    // Signle case where timeout differs from the default
    expect(foo).after(3).to.equal(42);
  });
});

Forced Failing

You can fail a test by using the failure attribute. This can be used to test branching.

failure(@"This should not happen"); outright fails a test.

WRITING NEW MATCHERS

Writing a new matcher is easy with special macros provided by Expecta. Take a look at how .beKindOf() matcher is defined:

EXPMatchers+beKindOf.h

#import "Expecta.h"

EXPMatcherInterface(beKindOf, (Class expected));
// 1st argument is the name of the matcher function
// 2nd argument is the list of arguments that may be passed in the function
// call.
// Multiple arguments are fine. (e.g. (int foo, float bar))

#define beAKindOf beKindOf

EXPMatchers+beKindOf.m

#import "EXPMatchers+beKindOf.h"

EXPMatcherImplementationBegin(beKindOf, (Class expected)) {
  BOOL actualIsNil = (actual == nil);
  BOOL expectedIsNil = (expected == nil);

  prerequisite(^BOOL {
    return !(actualIsNil || expectedIsNil);
    // Return `NO` if matcher should fail whether or not the result is inverted
    // using `.Not`.
  });

  match(^BOOL {
    return [actual isKindOfClass:expected];
    // Return `YES` if the matcher should pass, `NO` if it should not.
    // The actual value/object is passed as `actual`.
    // Please note that primitive values will be wrapped in NSNumber/NSValue.
  });

  failureMessageForTo(^NSString * {
    if (actualIsNil)
      return @"the actual value is nil/null";
    if (expectedIsNil)
      return @"the expected value is nil/null";
    return [NSString
        stringWithFormat:@"expected: a kind of %@, "
                          "got: an instance of %@, which is not a kind of %@",
                         [expected class], [actual class], [expected class]];
    // Return the message to be displayed when the match function returns `YES`.
  });

  failureMessageForNotTo(^NSString * {
    if (actualIsNil)
      return @"the actual value is nil/null";
    if (expectedIsNil)
      return @"the expected value is nil/null";
    return [NSString
        stringWithFormat:@"expected: not a kind of %@, "
                          "got: an instance of %@, which is a kind of %@",
                         [expected class], [actual class], [expected class]];
    // Return the message to be displayed when the match function returns `NO`.
  });
}
EXPMatcherImplementationEnd

DYNAMIC PREDICATE MATCHERS

It is possible to add predicate matchers by simply defining the matcher interface, with the matcher implementation being handled at runtime by delegating to the predicate method on your object.

For instance, if you have the following class:

@interface LightSwitch : NSObject
@property (nonatomic, assign, getter=isTurnedOn) BOOL turnedOn;
@end

@implementation LightSwitch
@synthesize turnedOn;
@end

The normal way to write an assertion that the switch is turned on would be:

expect([lightSwitch isTurnedOn]).to.beTruthy();

However, if we define a custom predicate matcher:

EXPMatcherInterface(isTurnedOn, (void));

(Note: we haven't defined the matcher implementation, just it's interface)

You can now write your assertion as follows:

expect(lightSwitch).isTurnedOn();

INSTALLATION

You can setup Expecta using CocoaPods, Carthage or completely manually.

CocoaPods

Add Expecta to your project's Podfile:

target :MyApp do
# your app dependencies

  target :MyAppTests do
    inherit! search_paths

    pod 'Expecta', '~> 1.0'
  end
end

Carthage

Add Expecta to your project's Cartfile.private:
```
github "specta/expecta" "master"
```
Run carthage update in your project directory.
Drag the appropriate Expecta.framework for your platform (located in Carthage/Build/) into your application’s Xcode project, and add it to your test target(s).
Run pod update or pod install in your project directory.

Setting Up Manually

Clone Expecta from Github.
Run rake in your project directory to build the frameworks and libraries.
Add a Cocoa or Cocoa Touch Unit Testing Bundle target to your Xcode project if you don't already have one.
For OS X projects, copy and add Expecta.framework in the Products/osx folder to your project's test target.

For iOS projects, copy and add Expecta.framework in the Products/ios folder to your project's test target.

You can also use libExpecta.a if you prefer to link Expecta as a static library — iOS 7.x and below require this.
Add -ObjC and -all_load to the Other Linker Flags build setting for the test target in your Xcode project.

You can now use Expecta in your test classes by adding the following import:

@import Expecta; // If you're using Expecta.framework

// OR

#import <Expecta/Expecta.h> // If you're using the static library, or the framework

STATUS

Expecta, and Specta are considered done projects, there are no plans for active development on the project at the moment aside from ensuring future Xcode compatability. Therefore it is a stable dependency, but will not be moving into the Swift world. If you are looking for that, we recommend you consider Quick and Nimble.

Contribution Guidelines

Please use only spaces and indent 2 spaces at a time.
Please prefix instance variable names with a single underscore (_).
Please prefix custom classes and functions defined in the global scope with EXP.

License

casamagalhaes/varejofacil-sdk-delphi

2023-10-10T01:36:31Z

shopware/shopware

2023-10-10T01:36:40Z

Shopware 6 is an open commerce platform based on Symfony Framework and Vue and supported by a worldwide community and more than 1.500 community extensions

Shopware

Modern open source e-Commerce

Shopware 6 is an open headless commerce platform powered by Symfony 6.2 and Vue.js 2.6 that is used by thousands of shops and supported by a huge worldwide community of developers, agencies and merchants.

If you like Shopware 6, give us a ⭐️ on Github

🙋‍♂️ Be part of shopware! ‍ We are hiring! 🙋
🌎 Discover our website
🧩 Browse more than 2.000 apps that are already available
📖 Learn how to develop apps and everything else about the tech behind shopware
🉐 Translate Shopware or help by contributing to existing languages
👍 Follow us on Twitter to get updates
🛠 Report bugs or add feature ideas in our issue tracker
🗨 Help and get helped in our - Community forum
👪 Exchange with more than 5.000 shopware developers in our - Slack
🕹 Download Shopware or start playing with Dockware

Open Commerce platform

We are always open, you are always free

With Shopware you control your destiny. Always having access to the source, you own it. No lock-ins, no compromises, no limits. Your freedom to grow.

Open commerce empowers you to seize the collective force of our community and partnerships. You benefit from a worldwide network of established developers, looking into the future of ecommerce and continuously advancing the platform in order to meet the latest market developments. It is exactly this focus that makes the difference and enables our merchants to grow strongly and sustainably in the ever-evolving world of digital commerce.

Innovative and future-oriented: Open, headless, API-first, cloud & self-managed

Seamless integration into your system landscape

Established technologies such as Symfony and vue.js

Business Model Composer

Plug and play your business model:

Adapt fast and test various business models on-time: Digital events, subscription services, consultations, and highly customisable goods – whether to buy or to rent. There are no limits to the power of human imagination. Dare to dream, we'll make it happen!

The Flow-builder makes the implementation of business processes easy

Run different business models on-time with one solution

Compose your unique business flow with no-code/low-code

Set your own rules and pioneer into new territory

Composable Customer Experience

Engage your customers across all channels with the perfect harmony of content and commerce

Designing content is fast and intuitive with the Shopping Experiences.

Strengthen your brand ID with our flawless integration of content and commerce

Convert casual touch-points into fundamental trust-points

Fly high and create the next-level user experience with Shopware's PWA

Under the hood

The build-in Shopware 6 Storefront is based on Twig and Bootstrap. Two well known and easy to learn frameworks, making the creation of templates a breeze and bringing many advantages for merchants & developers:

Lightweight and fast storefront
Works perfectly on all end devices
Reduced costs & effort required to make customisations, thanks to being built upon basic code & development standards that can be easily and widely used
Flexible basis for creating your own custom themes
Clear product presentation – filter according to properties and variants
Numerous sorting options for product lists in the storefront
Customer ratings can be handled conveniently in the administration and easily published in the storefront

Together with our friends of VueStorefront we also provide an open source PWA frontend called "Shopware PWA":

What makes Shopware PWA unique:

Quick integration & drop-in replacement for standard storefront
Unrestricted creativity for frontend developers
Modern tech-stack
Made for enterprise-level complexity

The Shopware 6 Administration is based on Vue.js and twig.js, making the creation of new modules fast and easy. Get started with the design documentation.

Extension-system

Shopware 6 is a lean and extremely flexible product that can be easily adapted to meet your requirements – and the Plugin Manager is your command centre for managing the apps and themes added to your shop. Using this module, you can install, purchase, update or delete apps.

You can easily build your own apps and extensions by following our developer documentation

Demo

You can easily setup shopware by:

Download shopware and run it on your server
using Dockware
or simply setup a cloud shop and tryout everything

Technology

Shopware 6 uses Symfony as the standard framework, while the administration is completely based on Vue.js. In relying on technological standards, we’ve made it even easier to work with Shopware – while reducing dependency on specialised knowledge. Our goal is to make it as comfortable as possible to get started with Shopware by providing you with various resources, completely free of charge.

API First

We want commerce to take place where people are; independent of place, time and end device. Following to the API-first approach, Shopware 6 provides retailers with the technological foundation to effortlessly build retail strategies across channels and devices.

The chart shows how the Shopware Platform fits into your enterprise. It provides web frontends for management and for commerce through a multitude of sales channels. It comes with a set of user facing interfaces and provides the ability to connect to your own infrastructure and outside services through REST-APIs.

More information can be found in the documentation.

Shopware 6 repository structure

Shopware 6 consists of multiple repositories, two of them are important to you:

shopware/platform is a mono repository
- This is where the shopware core is developed. You need it as dependency in your projects
- This is where you can participate in the development of Shopware through pull requests
- It's split into multiple repositories for production setups, all read-only
A template based on Symfony flex, which you can use to start your own project
- This is where your journey with shopware starts
- Installation see below!

Quickstart / Installation

A full installation guide covering different dev environments is available in the docs.

For the impatient reader, here is a tl;dr using docker and symfony cli.

Let's start by creating a new project:

> composer create-project shopware/production:dev-flex project

You now have the application template for the Shopware Platform in the directory project, we now change into it:

> cd project

Now we start our service containers:

> docker compose up -d

And install Shopware with the following command:

symfony console system:install --basic-setup --drop-database --create-database -f

Start the webserver:

symfony server:start -d

To be sure that the installation succeeded, just open the following URL in your favorite browser: localhost:8000

Now you're all set to start developing your first plugin.

Roadmap

You make the roadmap!

Shopware 6 will continue to evolve together with you and your feedback. This is our number one priority!

With openness as one of our core values, we will always provide you with a transparent overview of our product development. The Shopware Roadmap shows you what we are working on, what we want to tackle next, and what visions we have for the future.

Take a look at the current roadmap here.

Our community is our strongest asset

In today’s information-based world, you cannot thrive in closed systems. Black boxes and vendor lock-in models hurt innovation – and belong in the dark ages of ecommerce.

The future of IT is all about collaboration. At Shopware, we believe that the best ecommerce solution can only be developed in constant exchange with the people that use it every day. This is why we made a clear promise to the open source approach and embrace everyone willing to participate. We consider our community to be our greatest strength; not our competitor, like many companies tend to do.

We believe that our open source edition is our strongest asset and that we need, now more than ever, to invest in our ecosystem of partners and developers. So that we can work together to collaborate across backgrounds, experiences and ideas and mutually benefit from the software that results.

Join the community now 🖤

Discuss: forum.shopware.com
Slack: slack.shopware.com
Follow us on Twitter: @ShopwareDevs

Subscribe to our developer newsletter and get updates about:

Releases
Upcoming breaking changes
Important documentation changes and updates
Community events
Relevant blog articles

Subscribe now

Give us feedback

Moritz Naczenski
Community Manager
Twitter: @m_naczenski

Niklas Dzösch
Developer Evangelist
developer@shopware.com
Twitter: @ndzoesch

Ecosystem

Our Shopware Ecosystem gives you all the information you need to dive deep into the Shopware universe.

Shopware Community Store

Whether plugin, theme or marketing tool: You can easily extend the functionality of your shop with over 1,500 available apps and extensions in the Community Store.

store.shopware.com

Academy

Do you want to become a Shopware expert or get a sneak peek into the software? Find a training session that is individually tailored to your interests.

shopware.com/academy/

Contribution

First of all - Every contribution is meaningful, so thank you for participating.

You want to participate in the development of Shopware? There are many ways to contribute:

Submitting pull requests
Reporting issues on the issue tracker
Discuss shopware on e.g Slack or our forum
Write a translation for shopware on crowdin

You have a question regarding contribution, or you want to contribute in another way?

Please write us an email: contributors@shopware.com

Installation of Shopware for contributors

The installation process is slightly different for working on the platform. It's not necessary to create a new project and install the platform as a dependency. Instead, we will work on it directly.

Let's start by cloning the platform repository

> git clone git@github.com:shopware/platform.git shopware-platform

Change directory into our newly cloned project:

> cd shopware-platform

Install the Composer dependencies

> composer update

Now we start our service containers:

> docker compose up -d

Now, set up the Shopware environment with the following command:

> ./bin/console system:setup

URL to your /public folder should be: http://localhost:8000

For the database details, use the following:

Host: 127.0.0.1 User: root Password: root Database: shopware

At the end, the system will create a .env file containing the provided configuration.

Now we install Shopware with:

> composer setup

This will run the migrations, install dependencies and build the assets.

Start the webserver:

symfony server:start -d

To be sure that the installation succeeded, just open the following URL in your favorite browser: localhost:8000

Code Contribution

If you have decided to contribute code to Shopware and become a member of the Shopware community, we appreciate your hard work and want to handle it with the most possible respect. To ensure the quality of our code and our products we have created a guideline we all should endorse to. It helps you and us to collaborate. Following these guidelines will help us to integrate your changes in our daily workflow.

Read more in our contribution guideline or in our short HowTo contribute code.

The Shopware CLA

When submitting your code to Shopware you automatically need to sign our CLA (Contributor License Agreement). This CLA ensures that Shopware will stay an open and living product. In short, you give the explicit right to use your code in Shopware to shopware AG.

Reporting security issues

Please have a look at our security policy.

License

Shopware 6 is completely free and released under the MIT License.

Authors & Contributors

Shopware is built with the help of our community.

You can find an overview of everyone who contributed to the platform repository in the official github overview. Additionally there are numerous people contributing to the ecosystem through activities not related to the codebase. Thank you all for being part of this!

DanWin/le-chat-php

2023-10-10T01:36:40Z

A PHP Chat based on LE CHAT

General Information:

This is a PHP Chat based on LE CHAT v.1.14. An up-to-date copy of this script can be downloaded at https://github.com/DanWin/le-chat-php The original perl LE CHAT script by Lucky Eddie can be downloaded at this github fork. If you add your own cool features or have a feature request, please tell me and I will add them, if I like them. Please also let me know about any bugs you find in the code, so I can fix them. Now a piece of information about the origin of the name "LE CHAT" copied from the original script: The "LE" in the name you can take as "Lucky Eddie", or since the script was meant to be lean and easy on server resources, as "Light Edition". It may even be the French word for "the" if you prefer. Translated from French to English, "le chat" means: "the cat".

Features:

Optimized for Tor
No JavaScript needed
Cookies supported, but not needed
Captcha
Multiple languages
Members and guests
Waiting room for guests
Moderatoral approval of new guests
Public, member, moderator and admin only chats
Private messages
Multi-line messages
Change font, colour and refresh rate in profile settings
Autologout when inactive for some time
Image embedding
Notes for admins and moderators
Clone the chat to have multiple tabs
Kick chatters
Clean selected messages
Clean the whole room
Plain text message filter
Regex message filter
And more

Installation Instructions:

You'll need to have php with intl, gettext, pdo, pcre, mbstring and date extension, and a web-server installed. You will also need the pdo_sqlite, pdo_mysql or pdo_pgsql extension, depending on which database you choose. Optionally, you can install:

the gd extension for the captcha feature
the json extension for save/restore
a memcached server and the memcached extension and change the configuration to use memcached. This will lessen the database load a bit.
a MySQL or PostgreSQL server to use as an external database instead of SQLite
the libsodium extension (PHP >= 7.2) for encryption of messages and notes in the database When you have everything installed and use MySQL or PostgreSQL, you'll have to create a database and a user for the chat. Then edit the configuration at the bottom of the script to reflect the appropriate database settings and to modify the chat settings the way you like them. Then copy the script to your web-server directory and call the script in your browser with a parameter like this: http://(server)/(script-name).php?action=setup Now you can create the Superadmin account. With this account you can administer the chat and add new members and set the guest access. As soon as you are done with the setup, all necessary database tables will be created and the chat can be used. Note: If you updated the script, please visit http://(server)/(script-name).php?action=setup again, to make sure, that any database changes are applied and no errors occur.

Translating:

Translations are managed in Weblate. If you prefer manually submitting translations, the script update-translations.sh can be used to update the language template and translation files from source. It will generate the file locale/le-chat-php.pot which you can then use as basis to create a new language file in YOUR_LANG_CODE/LC_MESSAGES/le-chat-php.po and edit it with a translation program, such as Poedit. Once you are done, you can open a pull request, or email me, to include the translation.

Regex:

Yes, the chat supports regular expression filtering of messages. As regex tends to be difficult for most people, I decided to give it an extra section here. Regex is very powerful and can be used to filter messages that contain certain expressions and replace them with something else. It can be used e.g. to turn BB Code into html, so it is possible to use BB Code in the chat to format messages. To do this, use this Regex-Match \[(u|b)\](.*?)\[\/\1\] and this Regex-Replace <$1>$2</$1> and your text will be [b]bold[/b] or [u]underlined[/u]. You can also use smilies by using this Regex-Match (?-i::(cry|eek|lol|sad|smile|surprised|wink):) and this Regex-Replace <img src="/pictures/$1.gif" alt=":$1:"> And now if you enter 😄 an image with the smiley will be loaded from your server at /pictures/smile.gif. The following should be escaped by putting \ in front of it, if you are trying to match one of these characters / \ ^ . $ | ( ) [ ] * + ? { } ,. I used / as delimiter, so you will have to escape that, too. The only options I used is i to make the regex case insensitive. If you want to test your regex, before applying you can use this site and enter your Regex and Replacement there and click on preg_replace. If you never used regex before, check out this starting guide to begin with regular expressions.

mitre/stockpile

2023-10-10T01:37:21Z

USA Visa Appointment Checker

Hello, Visa Seeker!

This script automates the process of checking for available visa interview appointments.

It logs into your account on https://ais.usvisa-info.com using your username and password, and then scans the appointment page for availability.

You can run the script at regular intervals to receive timely alerts on new appointment slots.

Optionally, you can configure it to send email notifications about appointment availability to you or someone else.

Rest assured, the script runs locally on your computer, ensuring that your login details are not shared.

How to run

Download the visa-appointment-checker.ps1 file onto your computer. You can also clone this repo if you know Git.
Open up your preferred terminal app, like Windows Terminal or iTerm.
In your terminal, run this command to enter the Powershell shell: pwsh. If this command seems unavailable, restart your terminal app.
Navigate to the directory where you placed the visa-appointment-checker.ps1 file.
Run this command and make sure you provide your username, password, and schedule id ./visa-appointment-checker.ps1 -email "hello@world.com" -password "12abc" -scheduleId 456677

Prerequisites

You must download and install Powershell to run this script on your computer.

Dowload it here if you don't have it already.

More Help

.PARAMETER email
The email that you use to log in at https://ais.usvisa-info.com/en-tr/iv/users/sign_in

.PARAMETER password
The password that you use to log in at https://ais.usvisa-info.com/en-tr/iv/users/sign_in

.PARAMETER scheduleId
ScheduleId comes from the URL where you make the visa payment https://ais.usvisa-info.com/en-tr/iv/schedule/{scheduleId}/payment

.EXAMPLE
.\visa-appointment-checker.ps1 -email "hello@world.com" -password "12abc" -scheduleId 456677

yourwishismine1989/PowerShell_Getting-Started

2023-10-11T01:37:07Z

A quick how to with your first script example

PowerShell_Getting-Started

A quick how to with your first script example

install PowerShell

Setup linux distro (tested on Ubuntu 22.04 on 09/26/2023)

https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-linux?view=powershell-7.3

Setup Windows (tested on Windows 11 on 09/26/2023)

https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows?view=powershell-7.3

hello-world (tested on Ubuntu 22.04 on 09/26/2023)

/usr/bin/pwsh
touch hello-world.ps1
nano ./hello-world.ps1
[String]$message = "Hello World"
Write-Host -BackgroundColor $bgClr -ForegroundColor $fgClr "$message"
CTRL-o -> ENTER
CTRL-x
./hello-world.ps1

codewario/PocketLibraryImages

2023-10-11T01:37:07Z

This tool will prepare an image library for use with cartridges (and openFPGA cores if Analogue ever supports them) on Analogue OS

Analogue Pocket Library Image Generator

This is an interactive tool to assist with downloading image libraries from libretro-thumbnail and converting them to Analogue OS' .bin format.

Note that at this time image libraries only seem to work with cartridge-based games.

Note about OpenFPGA Cores

Presumably, OpenFPGA cores would follow the same image library format when displaying game details. However, Analogue has yet to support Library images for titles launched through OpenFPGA cores. Until such a time that OpenFPGA cores support Library images, using these image libraries with ROMs launched through OpenFPGA cores cannot be supported.

Quick Start

To get started, run AnalogueOSLibraryImageGenerator.ps1 in PowerShell. An interactive menu will guide you. It is recommended to read the Introduction before using the tool for the first time to get a feel for how it works.

By default the output is minimal. You can use the following flags to control whether to show additional output when launching the script. Note that these flags will increase the conversion time a bit as writing to the console is not an instant operation:

-OutputConvertedFiles: Output the files which are converted to the console
-Verbose: Additional debugging output, useful when troubleshooting the script
-Verbosity: Minimal, Extra, or Noisy. Controls how much information is displayed when -Verbose is set. No effect otherwise. Minimal is the default.

What game libraries can I convert?

At this time of writing only GBA, GB, GBC, and GG images are useful for conversion, but any libretro-thumbnail repository should be compatible. This ensures future compatibility as official converters are released or if Library images ever become supported on openFPGA cores.

Why didn't it generate a thumbnail for one of my games?

Note that since this tool relies on both libretro and DAT files, it may miss thumbnails for games which don't have a libretro-thumbnail image or if entries are not found in your DAT file. For the latter, if you know the CRC of your cart or ROM, you can add to your DAT file at any time using the text editor of your choice. In particular, rom hacks are likely going to be missing from the DAT file you pull from DAT-O-MATIC, but there are a few missing retail games or revisions of them as well.

Keep in mind that DAT-O-MATIC does split some classes of games into their own System. For example, GBA video cartridges have a special system classification and are not included in the normal DAT with most actual games.

Where can I run this script?

This tool is only supported on Windows as it relies on .NET types which are not available on MacOS or Linux. You may see some remnants in the source of a cross-platform direction, but this was quashed when I found the built-in namespaces for image manipulation are Windows-only or deprecated in non-Windows environments.

I am aware that there are third-party libs/tools that can be leveraged for image manipulation, however, one of the goals of this project is portability and not requiring additional items to be installed. It is also why this script crawls the GitHub site to obtain the zip URLs instead of requiring git to be installed for cloning libretro-thumbnails and its submodules directly.

That said, if anybody wants to contribute such functionality for Mac OS or Linux, pull requests are welcomed. Just try to keep any external dependencies to a "standard enough" minimum.

Getting a "connection closed" error when downloading one of the `libretro-thumbnails` image libraries

There's not a whole lot I can do about this, I hit this sometimes too when trying to convert the GBA library due to its size. GitHub terminates the code archive download if it runs for longer than 20 minutes, in my experience. This is likely due to the fact that GitHub creates code archives on-demand, and the time the exported archive remains cached is limited.

As a workaround when this happens, if you choose Download Console Image Library (Manual), and provide a file path (instead of a URL) to an archive that you've either:

exported yourself from a cloned copy of the git repo; or
been able to obtain from libretro-thumbnails yourself via other means

then the script will work with the local archive you've pointed to instead of attempting to obtain it from GitHub. Note that this script expects a .zip file; .tar.gz is not supported.

I would just like an image library please

Check out the latest Release page for the most recent sample image packs.

If you have a problem with the latest version, all Releases have sample packs provided, so you may try one of those. You can find older stable versions of the script along with image packs for supported games at the time of release as well as prior versions of the script. Don't forget if you have a problem with the latest sample packs to please create an issue on the project's issue tracker.

math-comp/hierarchy-builder

2023-10-11T01:37:29Z

High level commands to declare a hierarchy based on packed classes

haru52/haru52.github.io

2023-10-11T01:37:38Z

haru website.

muresan/learning-puppet

2023-10-11T01:37:37Z

learning puppet

jfryman/puppet-puppet

2023-10-11T01:37:37Z

Puppet module to manage Puppet (meta!)

icebourg/puppet-users

2023-10-11T01:37:37Z

A puppet repo for distributed user management.

Weixin-Liang/LLM-scientific-feedback

2023-10-10T01:37:46Z

🌐 FinGLM

赛题链接 | 赛题宣传页

📃 FinGLM: 致力于构建一个开放的、公益的、持久的金融大模型项目，利用开源开放来促进「AI+金融」。

🚀 目录

📖 项目介绍

📈 一个旨在深度解析上市公司年报的对话交互智能系统。面对金融文本中的专业术语与暗含信息，我们致力于用AI实现专家级别的金融分析。

🚀 在AI领域，虽然已在文本对话取得进展，但真正的金融交互场景仍然是一个巨大挑战。多方机构联手举办此次竞赛，探索金融领域AI的边界。

📘 上市公司年报为投资者呈现了公司的经营状况、财务状况和未来规划。专业知识是解读的关键，而我们的目标是通过AI技术让这一过程变得更简单、更准确。

🛠 项目框架

1. 数据准备流程

PDF 转 TXT：
- 转换为 TXT 格式。
- 保留表格并合并单元格。
数据切分：
- 基础信息：例如公司名称等。
- 财务数据：例如资产负债表等。
- 综合信息：例如财务指标等。
数据处理：
- 计算基础公式：如营业成本率等。
- 计算增长率。
- 计算行业均值和排名。
存入数据库：
- 存入 SQL、Mongo 和 ES 中。
- 包括建表及存储。

2. 模型微调流程

数据分类：如 SQL 数据、ES 数据等。
选择微调策略：例如 ptuningv2、lora等。
执行微调：根据选定策略。

3. 问答流程

输入问题：用户输入问题。
Prompt 准备：根据问题生成 prompt。
生成查询语句：基于 GPU 使用率选择生成方法。
查询数据库：并返回结果。
答案生成：结合问题和查询结果生成答案。

🌱 开源路线图

1. 开源策略

1) 赛事转型

比赛转型为学习赛，允许任何人学习使用。

2) 数据开源

目前开源数据有 70G/1w+份年报数据、10000条人工标注评测数据等。
后续我们也将根据项目需求，持续迭代更新数据。

3) 方案/代码/模型开源

经同意，结婚买房代代韭菜、Chatglm反卷总局、nsddd、馒头科技、南哪都队、龙盈战队、安硕硕眼探企等团队的方案、代码、模型完全开源，纳入FinGLM项目。
我们将长期维护优化 FinGLM 项目，提供便捷解决方案。

4) 开放交流

以上团队成员将共同维护 FinGLM项目，确保项目持续迭代。我们也欢迎更多团队来共同贡献问题和方案。
我们将不定期组织线上、线下交流，将更优秀的技术推广给每个项目成员。
FinGLM 开源项目出于完全公益目的，欢迎所有开发者申请加入，当然我们会进行严格审核。

5) 学习教程

基于FinGLM项目的开发，我们将整合并制作以下（包含且不限于）学习教程.
- 数据预处理教程
- 数据库使用教程
- GLM的使用教程
- Prompt编写教程
- 模型微调数据准备
- 模型微调技巧和步骤
- 全流程落地

6) 项目资源池

为了维持项目的健康发展，部分项目组织者（个人和企业）提供 10 万元作为开源项目资金池，以及提供项目算力、数据和模型支持。
我们欢迎所有受益于本项目的个人或单位来赞助本项目，包括且不限于以上内容，欢迎联系我们。

2. 开源进度

第一期：

组织 SMP 2023 ChatGLM 金融大模型挑战赛
比赛数据集开源
- 年报数据集
  - PDF 文件。包含 11588 份 2019 年至 2021 年期间的部分上市公司年度报告。
  - TXT 文件。利用 pdf2txt.py 对 PDF 文件解析而来。
  - HTML 文件。
- 数据库接入
  - sqlite
  - mongodb
决赛项目开源：

第二期：

微调Fintune
完善nl2sql
增加学习教程
- 数据预处理教程
- 数据库使用教程
- GLM的使用教程
- Prompt编写教程
- 模型微调数据准备
- 模型微调技巧和步骤
- 全流程落地
问答系统异常处理
提供详细使用手册
文档注释完善

🏆 开源项目

0. FinGLM_all

[PPT] [视频][代码]

本项目为安硕硕眼探企团队，根据自己的项目以及其他几队的项目整合而成。后续我们也将围绕此项目进行持续迭代升级。

📚 数据集

1. 年报数据集

我们开源的数据集涵盖了2019-2021年期间部分上市公司的年度报告。该数据集共包含 11588 个详尽的 PDF 文件（list）。您可以利用这些PDF文件的内容来构建您需要的数据库或者向量库。为了避免计算资源浪费，我们也将相应的文件转换成 TXT文件和 HTML文件，供大家使用。

PDF下载

大小：69GB 文件格式：pdf文件文件数量：11588

git加载

# 要求安装 git lfs
git clone http://www.modelscope.cn/datasets/modelscope/chatglm_llm_fintech_raw_dataset.git

sdk加载

# Note: 
# 1. 【重要】请将modelscope sdk升级到v1.7.2rc0，执行： pip3 install "modelscope==1.7.2rc0" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
# 2. 【重要】datasets版本限制为 >=2.8.0, <=2.13.0，可执行： pip3 install datasets==2.13.0

from modelscope.msdatasets import MsDataset

# 使用流式方式加载「推荐」
# 无需全量加载到cache，随下随处理
# 其中，通过设置 stream_batch_size 可以使用batch的方式加载

ds = MsDataset.load('chatglm_llm_fintech_raw_dataset', split='train', use_streaming=True, stream_batch_size=1)
for item in ds:
    print(item)

# 加载结果示例（单条，pdf:FILE字段值为该pdf文件本地缓存路径，文件名做了SHA转码，可以直接打开） 
{'name': ['2020-03-24__北京鼎汉技术集团股份有限公司__300011__鼎汉技术__2019年__年度报告.pdf'], 'pdf:FILE': ['~/.cache/modelscope/hub/datasets/modelscope/chatglm_llm_fintech_raw_dataset/master/data_files/430da7c46fb80d4d095a57b4fb223258ffa1afe8bf53d0484e3f2650f5904b5c']}


# 备注: 
1. 自定义缓存路径，可以自行设置cache_dir参数，即 MsDataset.load(..., cache_dir='/to/your/path')
2. 补充数据加载（从9493条增加到11588条），sdk加载注意事项
    a) 删除缓存中的csv映射文件(默认路径为)： ~/.cache/modelscope/hub/datasets/modelscope/chatglm_llm_fintech_raw_dataset/master/data_files/732dc4f3b18fc52380371636931af4c8
    b) 使用MsDataset.load(...) 加载，默认会reuse已下载过的文件，不会重复下载。

TXT下载

Note: pdf转txt格式文件，方便大家复用（有个文件损坏了，所以总数比pdf少1个，共11587 个）

# Linux
wget https://sail-moe.oss-cn-hangzhou.aliyuncs.com/open_data/hackathon_chatglm_fintech/alltxt.zip

# Windows示例
Invoke-WebRequest -Uri https://sail-moe.oss-cn-hangzhou.aliyuncs.com/open_data/hackathon_chatglm_fintech/alltxt.zip -OutFile D:\\alltxt.zip

HTML下载

Note: pdf转html格式文件，方便大家复用（有个文件损坏了，所以总数比pdf少，共11582 个）

# Linux
wget https://sail-moe.oss-cn-hangzhou.aliyuncs.com/open_data/hackathon_chatglm_fintech/allhtml.zip

# Windows示例
Invoke-WebRequest -Uri https://sail-moe.oss-cn-hangzhou.aliyuncs.com/open_data/hackathon_chatglm_fintech/allhtml.zip -OutFile D:\\allhtml.zip

使用建议

以下是我们推荐的处理步骤：

1、PDF文本和表格提取：您可以使用如pdfplumber、pdfminer等工具包提取PDF文件中的文本和表格数据。

2、数据切分：根据PDF文件的目录、子目录和章节信息，对内容进行精确的切块处理。

3、构建基础金融数据库：依据金融知识和PDF内容，设计专业的金融数据库字段和格式。例如，定义资产负债表、现金流量表和利润表等。

4、信息提取：使用大模型的信息提取能力和NLP技术来抽取对应的金融字段信息。例如，请使用json方式输出目录的内容，其中章节的名称作为key，页码作为value。同时，请详细地抽取表格内的数据，以JSON格式输出。

5、构建金融知识问答库：结合构建的金融数据库，应用大模型构建基础的金融问答库。例如，

{"question"："某公司2021年的财务费用为多少元？", "answer": "某公司2021年的财务费用为XXXX元。"}
prompt:用多种句式修改question及answer的内容。

{"question":"为什么财务费用可以是负的？", "answer": ""}
prompt：请模仿上面的question给出100个类似的问题与对应的答案，用json输出。

6、构建向量库：借助于如Word2Vec、Text2Vec等技术，从原始文本数据中提取出语义向量。使用pgvector这种基于PostgreSQL的扩展来存储和索引这些向量，从而建立起一个可供高效查询的大规模向量库。

7、应用：结合向量库、大模型、langchain等工具，提升应用效果。

2. 标注数据

在 SMP 2023 ChatGLM 金融大模型挑战赛中我们分别进行了初赛、复赛A、复赛B、复赛C。针对这几轮比赛，我们分别人工标注了相关数据，累计总共有 10000 条。

[初赛数据] ：5000 条
[复赛 A 数据] ：2000 条
[复赛 B 数据] ：2000 条
[复赛 C 数据] ：1000 条

数据示例：

{"ID": 1,
"question": "2019年中国工商银行财务费用是多少元?",
"answer":"2019年中国工商银行财务费用是12345678.9元。"}

{"ID": 2,
"question": "工商银行2019年营业外支出和营业外收入分别是多少元?",
"answer": "工商银行2019年营业外支出为12345678.9元，营业外收入为2345678.9元。"}

{"ID":3,
"question": "中国工商银行2021年净利润增长率是多少?保留2位小数。",
"answer": "中国工商银行2020年净利润为12345678.90元，2021年净利润为22345678.90元，根据公式，净利润增长率=(净利润-上年净利润)/上年净利润，得出结果中国工商银行2021年净利润增长率81.00%。" }

与此同时，我们也针对比赛撰写了评测代码。我们依据：


\begin{array}{ll}
max_{similar}(sentence1,sentence2,sentence3), & 无基础信息及关键词\\
0.25+0.25+max_{similar}(sentence1,sentence2,sentence3)*0.5, & 基础信息正确，关键词正确 \\
0.25 + 0 + max_{similar}(sentence1,sentence2,sentence3)*0.5, & 基础信息正确，关键词错误\\
0, & 基础信息错误

\end{array}

评测示例：

{"question": "2019年中国工商银行财务费用是多少元?",

"prompt": {"财务费用": "12345678.9元", "key_word":"财务费用、2019", "prom_answer": "12345678.9元"},

"answer": [

"2019年中国工商银行财务费用是12345678.9元。",

"2019年工商银行财务费用是12345678.9元。",

"中国工商银行2019年的财务费用是12345678.9元。" ]

}

评测计算示例：

答案一：工商银行2019年财务费用是12345678.9元。

most similar sentences:

2019年工商银行财务费用是12345678.9元。 (Score: 0.9915)

中国工商银行2019年的财务费用是12345678.9元。 (Score: 0.9820)

2019年中国工商银行财务费用是12345678.9元。 (Score: 0.9720)

评分：0.25+0.25+0.9915*0.5=0.9958分。

评分解释：prom_answer正确、包含所有key_word、相似度最高0.9915。

答案二：2019年中国工商银行财务费用是335768.91元。

评分：0分。

评分解释：prom_answer错误不得分。

答案三：12345678.9元。

most similar sentences:

2019年工商银行财务费用是12345678.9元。 (Score: 0.6488)

2019年中国工商银行财务费用是12345678.9元。 (Score: 0.6409)

中国工商银行2019年的财务费用是12345678.9元。 (Score: 0.6191)

评分：0.25+0+0.6488*0.5=0.5744分。

评分解释：prom_answer正确、未包含所有key_word、相似度最高0.6488。

📊 项目问答演示

{"id": 0, "question": "2021年其他流动资产第12高的是哪家上市公司？", "answer": "2021年其他流动资产第12高的公司是苏美达股份有限公司。"}
{"id": 1, "question": "注册地址在重庆的上市公司中，2021年营业收入大于5亿的有多少家？", "answer": "2021年注册在重庆，营业收入大于5亿的公司一共有4家。"}
{"id": 2, "question": "广东华特气体股份有限公司2021年的职工总人数为？", "answer": "2021年广东华特气体股份有限公司职工总人数是1044人。"}
{"id": 3, "question": "在保留两位小数的情况下，请计算出金钼股份2019年的流动负债比率", "answer": "2019金钼股份流动负债比率是61.10%。其中流动负债是1068418275.97元；总负债是1748627619.69元；"}
{"id": 4, "question": "2019年负债总金额最高的上市公司为？", "answer": "2019年负债合计最高的是上海汽车集团股份有限公司。"}
{"id": 5, "question": "2019年总资产最高的前五家上市公司是哪些家？", "answer": "2019年资产总计最高前五家是上海汽车集团股份有限公司、中远海运控股股份有限公司、国投电力控股股份有限公司、华域汽车系统股份有限公司、广州汽车集团股份有限公司。"}
{"id": 6, "question": "2020年营业收入最高的3家并且曾经在宁波注册的上市公司是？金额是？", "answer": "注册在宁波，2020年营业收入最高的3家是宁波均胜电子股份有限公司营业收入47889837616.15元；宁波建工股份有限公司营业收入19796854240.57元；宁波继峰汽车零部件股份有限公司营业收入15732749552.37元。"}
{"id": 7, "question": "注册地址在苏州的上市公司中，2020年利润总额大于5亿的有多少家？", "answer": "2020年注册在苏州，利润总额大于5亿的公司一共有2家。"}
{"id": 8, "question": "浙江运达风电股份有限公司在2019年的时候应收款项融资是多少元？", "answer": "2019年浙江运达风电股份有限公司应收款项融资是51086824.07元。"}
{"id": 9, "question": "神驰机电股份有限公司2020年的注册地址为？", "answer": "2020年神驰机电股份有限公司注册地址是重庆市北碚区童家溪镇同兴北路200号。"}
{"id": 10, "question": "2019年山东惠发食品股份有限公司营业外支出和营业外收入分别是多少元？", "answer": "2019年山东惠发食品股份有限公司营业外收入是1018122.97元；营业外支出是2513885.46元。"}
{"id": 11, "question": "福建广生堂药业股份有限公司2020年年报中提及的财务费用增长率具体是什么？", "answer": "2020福建广生堂药业股份有限公司财务费用增长率是34.33%。其中，财务费用是7766850.48元；上年财务费用是5781839.51元。"}
{"id": 12, "question": "华灿光电股份有限公司2021年的法定代表人与上年相比相同吗？", "answer": "不相同，华灿光电股份有限公司2020年法定代表人是俞信华，2021年法定代表人是郭瑾。"}
{"id": 13, "question": "请具体描述一下2020年仲景食品控股股东是否发生变更。", "answer": "2020年，仲景食品控股股东没有发生变更。"}
{"id": 14, "question": "什么是其他债权投资？", "answer": "其他债权投资是指企业或机构投资者通过购买债券、贷款、定期存款等金融产品获得的固定收益。这些金融产品通常由政府、公司或其他机构发行，具有一定的信用等级和风险。\n\n其他债权投资是企业或机构投资组合中的一部分，通常用于稳定收益和分散风险。与股票投资相比，其他债权投资的风险较低，但收益也相对较低。\n\n其他债权投资的管理和投资策略与其他资产类别类似，包括分散投资、风险控制、收益最大化等。然而，由于其他债权投资的种类繁多，其投资和管理也存在一定的特殊性。"}

🤝 贡献者

以下是为本项目做出贡献的团队和个人：

🌟 安硕硕眼探企
🌟 馒头科技
🌟 南哪都队
🌟 Chatglm反卷总局
🌟 nsddd
🌟 龙盈战队
🌟 结婚买房代代韭菜
🌟 小打小闹
🌟 东北大土豆
🌟 ... 更多贡献者

FinGLM 开源项目出于完全公益目的，欢迎所有开发者申请加入，当然我们会进行严格审核。如有意向，请填写表单。

免责声明

本项目相关资源仅供研究、交流使用，一般不建议用于商业用途；如用于商业用途，由此所带来的法律风险，请自行承担。

涉及到模型商业使用问题，请务必遵循相关模型的协议，例如 ChatGLM-6B。

🔍 项目联系

项目交流群

开源赞助

开源贡献

microsoft/ToRA

2023-10-10T01:37:46Z

ToRA is a series of Tool-integrated Reasoning LLM Agents designed to solve challenging mathematical reasoning problems by interacting with tools.

ToRA: A Tool-Integrated Reasoning Agent

[🌐 Website] • [📜 Paper] • [🤗 HF Models] • [🐱 GitHub]
[🐦 Twitter] • [💬 Reddit] • [🍀 Unofficial Blog]

Repo for "ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving"

Figure 1: Comparing ToRA with baselines on LLaMA-2 base models from 7B to 70B.

🔥 News

[2023/10/08] 🔥🔥🔥 All ToRA models released at 🤗 HuggingFace!!!
[2023/09/29] ToRA paper, repo, and website released.

💡 Introduction

Model	Size	GSM8k	MATH	AVG@10 math tasks^†
GPT-4	-	92.0	42.5	78.3
GPT-4 (PAL)	-	94.2	51.8	86.4
ToRA-7B	7B	68.8	40.1	62.4
ToRA-Code-7B	7B	72.6	44.6	66.5
ToRA-13B	13B	72.7	43.0	65.9
ToRA-Code-13B	13B	75.8	48.1	71.3
ToRA-Code-34B^*	34B	80.7	51.0	74.8
ToRA-70B	70B	84.3	49.7	76.9

^*ToRA-Code-34B is currently the first and only open-source model to achieve over 50% accuracy (pass@1) on the MATH dataset, which significantly outperforms GPT-4’s CoT result (51.0 vs. 42.5), and is competitive with GPT-4 solving problems with programs. By open-sourcing our codes and models, we hope more breakthroughs will come!
^†10 math tasks include GSM8k, MATH, GSM-Hard, SVAMP, TabMWP, ASDiv, SingleEQ, SingleOP, AddSub, and MultiArith.

Tool-Integrated Reasoning

Figure 2: A basic example of single-round tool interaction, which interleaves rationales with program-based tool use.

ToRA Training Pipeline

Figure 3: Training ToRA contains ① Imitation Learning, and ② output space shaping.

🚀 Quick Start

⚙️ Setup

We recommend using Conda to manage your environment. We use vLLM (0.1.4) to accelerate inference. Run the following commands to setup your environment:

git clone https://github.com/microsoft/ToRA.git && cd ToRA/src
conda create -n tora python=3.10
conda activate tora
pip install torch==2.0.1 --index-url https://download.pytorch.org/whl/cu118 # CUDA 11.8 for example
pip install -r requirements.txt

🪁 Inference

We provide a script for inference, simply config the MODEL_NAME_OR_PATH and DATA in src/scripts/infer.sh and run the following command:

bash scritps/infer.sh

We also open-source the model outputs from our best models (ToRA-Code-34B and ToRA-70B) in the src/outputs/ folder.

⚖️ Evaluation

To evaluate the predicted answer, run the following command:

python -m eval.evaluate \
    --data_name "math" \
    --prompt_type "tora" \
    --file_path "outputs/llm-agents/tora-code-34b-v1.0/math/test_tora_-1_seed0_t0.0_s0_e5000.jsonl" \
    --execute

then you will get:

Num samples: 5000
Num scores: 5000
Timeout samples: 0
Empty samples: 2
Mean score: [51.0]
Type scores: {'Algebra': 67.3, 'Counting & Probability': 42.2, 'Geometry': 26.1, 'Intermediate Algebra': 40.0, 'Number Theory': 59.3, 'Prealgebra': 63.8, 'Precalculus': 34.2}

⚡️ Training

To train a model, run the following command:

bash scritps/train.sh codellama 7b

☕️ Citation

If you find this repository helpful, please consider citing our paper:

@misc{gou2023tora,
      title={ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving}, 
      author={Zhibin Gou and Zhihong Shao and Yeyun Gong and yelong shen and Yujiu Yang and Minlie Huang and Nan Duan and Weizhu Chen},
      year={2023},
      eprint={2309.17452},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

🍀 Contributing

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

🌟 Star History

cvg/glue-factory

2023-10-10T01:37:46Z

Finetune llama2-70b and codellama on MacBook Air without quantization

slowllama

Fine-tune Llama2 and CodeLLama models, including 70B/35B on Apple M1/M2 devices (for example, Macbook Air or Mac Mini) or consumer nVidia GPUs.

slowllama is not using any quantization. Instead, it offloads parts of model to SSD or main memory on both forward/backward passes. In contrast with training large models from scratch (unattainable) or inference, where we are likely to care about interactivity, we can still get something finetuned if you let it run for a while.

Current version is using LoRA to limit the updates to a smaller set of parameters. First version supported full finetuning as well, but I decided to remove it for now, more on that below.

Finetuning is the only focus, there's nothing special done for inference, consider llama.cpp.

For CUDA-specific experiments, see report on a10.

It is all very experimental, but even more so for CUDA.

Example

Tests were done on Apple M1 with 16Gb memory and Apple M2 with 24Gb memory.

In order to fine-tune llama2 model we need to:

Install dependencies: pip install torch sentencepiece numpy. Optional: install pip install fewlines for weight/gradient distribution logging.
Clone llama2 and follow instructions to download the models. The script will download tokenizer as well. tokenizer.model should be put into the same directory as llama model itself. Use codellama for CodeLLama models. Example folder structure could look like:

/parent/
    /slowllama/...   # <- this repo
    /codellama/...   # <-- this is Meta's codellama repository.
    /llama-2-7b/...  # <- put tokenizer.model here
    /llama-2-13b/... # <- and here
    /llama-2-70b/... # <- and here as well
    /CodeLlama-34b-Python/... # and here

Let's start with a tiny example. It is an intro to the description of another open-source project - cubestat. Text is short enough to just be included as part of the prompt, but it's ok as an illustration and you can read it in seconds youself. As I just published that project recently, there's no way original llama would know anything about it.

Asking base llama2-7b to complete the prompt "Cubestat reports the following metrics: " results in "1) the number of cubes in the system, 2) the number of cubes that are in the process of being created".

First step is to transform the model to the sequential format more suitable for loading to/from storage block-by-block.

python prepare_model.py

Modify the input/output paths in the script itself.

Now we can try not-finetuned llama2:

python test_gen.py ../llama7b mps # use path to transformed model here

Now let's finetune the 7b model. finetune.py is a very simple script which trains LoRA weights based on the plaintext data. There are some settings you could change here, like sequence length, batch size, learning rate, dropout rate, number of iterations. Current settings are pretty much a guess, change this if desired. Adjust accordingly. Currently it uses AdamW optimizer.

python finetune.py

Here's train dataset loss:

2023-09-10 22:05:35,569 backprop done, loss after forward pass = 2.9539270401000977
2023-09-10 22:06:08,022 backprop done, loss after forward pass = 2.9073102474212646
2023-09-10 22:06:40,223 backprop done, loss after forward pass = 2.7192320823669434
2023-09-10 22:07:12,468 backprop done, loss after forward pass = 2.7223477363586426
2023-09-10 22:07:44,626 backprop done, loss after forward pass = 2.5889995098114014
2023-09-10 22:08:16,899 backprop done, loss after forward pass = 2.4459967613220215
2023-09-10 22:08:49,072 backprop done, loss after forward pass = 2.3632657527923584
2023-09-10 22:09:21,335 backprop done, loss after forward pass = 2.250361442565918
2023-09-10 22:09:53,511 backprop done, loss after forward pass = 2.165428638458252
2023-09-10 22:10:25,738 backprop done, loss after forward pass = 2.031874656677246
2023-09-10 22:13:45,794 backprop done, loss after forward pass = 1.8926434516906738
2023-09-10 22:14:18,049 backprop done, loss after forward pass = 1.7222942113876343
2023-09-10 22:14:50,243 backprop done, loss after forward pass = 1.58726966381073
2023-09-10 22:15:22,405 backprop done, loss after forward pass = 1.4983913898468018
2023-09-10 22:15:54,598 backprop done, loss after forward pass = 1.296463131904602
2023-09-10 22:16:26,909 backprop done, loss after forward pass = 1.3328818082809448
2023-09-10 22:16:59,031 backprop done, loss after forward pass = 1.0978631973266602
2023-09-10 22:17:31,200 backprop done, loss after forward pass = 1.018444538116455
2023-09-10 22:18:03,406 backprop done, loss after forward pass = 0.8421685099601746
2023-09-10 22:18:35,673 backprop done, loss after forward pass = 0.7168515920639038
2023-09-10 22:21:55,482 backprop done, loss after forward pass = 0.7870235443115234

I didn't add a validation set for this data, instead I just checked what would the fine-tuned model produce for the same prompt.

At ~10 iteration we get the following reasonable output: Cubestat reports the following metrics: 1. CPU usage, 2. Memory usage, 3. Disk usage

At ~20 iteration another output is produced:

0 - Cubestat reports the following metrics: CPU utilization: Efficiency and Performance cores. Shows as percentage.

Maybe we were overfitting already at this point.

Running completion with newly produced lora checkpoint can be done like this:

python test_gen.py ../llama7b mps ./out/state_dict_19.pth

How does it work?

For all versions the process is roughly the same.

First, we need to be able to load a model which requires more RAM than we have and save it back in sequential format. We create model instance with all large modules' weights offloaded to SSD - all of the transformer blocks, token embeddings and output linear layer. After that we load model shards one by one, for each shard iterate over all modules, update corresponding subset of its weights and save it back.

Doing forward path is easy - we just load modules when we need and pass the output forward.

Backward pass is a little more tricky, in a way we have to run forward pass twice. The way it's currently implemented is:

Do a forward pass while also saving inputs to each offloaded block to the SSD. The goal of the first forward pass is to compute the final loss and cache inputs to each offloaded block.
Then, do a manual backward gradient propagation. We start from the last block, re-run each block once again (forward, to build autograd graph) with the same input we cached on step (1). After that we run backward pass within that block only, and pass the gradient for the input to the next (previous?) block. As we use LoRA, only LoRA gradients are being saved. LoRA weights are not offloaded to disk, always staying on RAM/GPU. Important: we also need to save and restore random number generation state before evaluating each offloaded module. During training we use dropout, and randomly switched off neurons should be the same on both forward passes.
After that we run optimizer step on LoRA weights and save them separately if needed.

Original llama2 weights are in bfloat16, but mps backend doesn't support that type natively, so we do computation in float32 instead.

Experimental version of slowllama which can be still found here was capable of doing full finetuning and update all weights pretty much the same way. I've temporarily removed that feature to preserve the lifespan of SSDs, as frequent write operations can degrade performance over time. Reading from SSDs isn't an issue, but they do have a write limit. Limit is typically high enough for normal usage, but in the case of full finetunining we'll have to write ~150Gb per one iteration/weight update of 70B variant, assuming stateless optimizer and no gradient accumulation. With AdamW we'll have to save/update another 150Gb more of optimizer state per iteration. If, for example, we assume 1Pb of writes before SSD will start having issues, even 100 iterations of finetuning would incur significant cost/risk. For machines with GPUs and large amount of RAM we can skip the disk entirely and offload to RAM only. It should be possible to bring full finetuning back for main-memory-only offload. On the other hand, if everything fits into memory, there's no need to do whole 'evaluate twice' thing, might just use fairscale instead and only move tensors between GPU/CPU.

Experiments

Llama2 7B finetune on M1 Mini (16Gb memory):

Here we can see resource utilization for 1 full iteration on 7B model - forward and manual backward passes. Each column == 1 second. A few notes:

GPU is reasonably well utilized;
First forward pass has lower GPU utilization and spends more time on IO as we need to both read weights and write cached inputs/outputs
Backward (combined?) pass achieves very high GPU utilization, close to 100%
As we move along layers back and forth, right after each 'direction switch' we process layers in LIFO order. Thus in the beginning of both forward and backward pass we don't have to access disk, weights are being cached and we don't see disk reads.

batch_size/seq_len - works ok with, say, 2048 seq_len and batch_size = 2.

Llama2 70B finetune on M1 Mini (16Gb memory)

The chart here has different granularity - each column is 30 seconds. Input data was also different - it is the readme file you are reading now. I didn't have enough free space on disk to store both original weights (140Gb) + weights in sequential format we use (another 140Gb). In order to still be able to finetune this model, I stored original weights on much slower external SD card, as we need to read them only once. Weights in sequential format on fast internal SSD. With batch size = 16 and sequence length = 128 it was taking ~25-30 min per iteration.

As we can see, GPU utilization doesn't look that great - we might be able to benefit from prefetching next transformer block, assuming we have enough memory for storing 2 layers. Memory utilization peaked at around 80% of 16Gb.

Loss over time:

2023-09-13 17:30:28,731 backprop done, loss after forward pass = 2.431253433227539
2023-09-13 18:00:00,133 backprop done, loss after forward pass = 2.604712963104248
2023-09-13 18:29:36,473 backprop done, loss after forward pass = 2.6277880668640137
2023-09-13 19:00:40,463 backprop done, loss after forward pass = 2.408756971359253
2023-09-13 19:29:55,974 backprop done, loss after forward pass = 2.6121537685394287
2023-09-13 19:59:04,849 backprop done, loss after forward pass = 2.428431987762451
2023-09-13 20:27:03,760 backprop done, loss after forward pass = 2.4040215015411377
2023-09-13 20:55:56,969 backprop done, loss after forward pass = 2.158071279525757
2023-09-13 21:25:04,615 backprop done, loss after forward pass = 2.3459620475769043
2023-09-13 21:54:07,128 backprop done, loss after forward pass = 2.2933709621429443
2023-09-13 23:18:57,588 backprop done, loss after forward pass = 2.273494243621826
2023-09-13 23:48:05,310 backprop done, loss after forward pass = 2.4055371284484863
2023-09-14 00:17:19,113 backprop done, loss after forward pass = 2.2604546546936035
2023-09-14 00:46:31,872 backprop done, loss after forward pass = 2.552386522293091
2023-09-14 01:15:45,731 backprop done, loss after forward pass = 2.297588586807251
2023-09-14 01:44:51,640 backprop done, loss after forward pass = 2.1217401027679443
2023-09-14 02:14:09,033 backprop done, loss after forward pass = 1.9815442562103271
2023-09-14 02:43:09,114 backprop done, loss after forward pass = 2.020181179046631
2023-09-14 03:12:17,966 backprop done, loss after forward pass = 2.0041542053222656
2023-09-14 03:41:20,649 backprop done, loss after forward pass = 1.9396495819091797
2023-09-14 05:06:31,414 backprop done, loss after forward pass = 2.1592249870300293
2023-09-14 05:35:39,080 backprop done, loss after forward pass = 1.976989984512329
2023-09-14 06:04:57,859 backprop done, loss after forward pass = 1.7638890743255615
2023-09-14 06:34:06,953 backprop done, loss after forward pass = 1.9829202890396118
2023-09-14 07:03:18,661 backprop done, loss after forward pass = 1.754631519317627
2023-09-14 07:32:26,179 backprop done, loss after forward pass = 2.027863025665283
2023-09-14 08:01:37,546 backprop done, loss after forward pass = 1.8579339981079102
2023-09-14 08:30:41,689 backprop done, loss after forward pass = 1.7934837341308594
2023-09-14 08:59:55,921 backprop done, loss after forward pass = 1.794022798538208
2023-09-14 09:28:59,690 backprop done, loss after forward pass = 1.750269889831543
2023-09-14 10:56:19,282 backprop done, loss after forward pass = 1.4310824871063232
2023-09-14 11:25:28,462 backprop done, loss after forward pass = 1.6895856857299805
2023-09-14 11:54:39,973 backprop done, loss after forward pass = 1.5074403285980225
2023-09-14 12:23:42,604 backprop done, loss after forward pass = 1.6695624589920044
2023-09-14 12:53:00,535 backprop done, loss after forward pass = 1.4220315217971802
2023-09-14 13:22:15,685 backprop done, loss after forward pass = 1.5720497369766235
2023-09-14 13:51:30,744 backprop done, loss after forward pass = 1.544579267501831
2023-09-14 14:20:44,482 backprop done, loss after forward pass = 1.2813694477081299
2023-09-14 14:50:03,384 backprop done, loss after forward pass = 1.2990479469299316
2023-09-14 15:19:09,620 backprop done, loss after forward pass = 1.0500637292861938

We used prompt 'slowllama is a ', and here you can see the completions:

before any weight update: slowllama is a 24 year old (DOB: December 25, 1994) pure-blood witch
after 10 iterations: slowllama is a 24 year old (DOB: December 25, 1994) pure-blood witch
after 20 iterations: slowllama is a 70B model trained on the same data as llama.70b, but with a different training setup.
after 30 iterations: slowllama is a 2022 fork of llama2, which is a 2021 fork of llama, which is a 2020 fork
after 40 iterations: slowllama is a 2-stage finetuning implementation for llama2.

Current setup is probably too slow for 70B model finetuning on old mac mini M1. It would be interesting to try it on more recent hardware (say, M2 Max / M2 Pro), implement prefetch/async save and see how it's going to work.

merging LoRA weights back

In order to merge LoRA checkpoint back to the model in original format, we can do the following:

# confirm that old model is producing wrong output
python test_gen.py ../llama-2-7b mps

# ...
# 0 - slowllama is a 24 year old (DOB: May 1, 1997) pure-blood witch 

# check what would be the output for finetuned model by passing path to checkpoint
python test_gen.py ../llama-2-7b mps ./data/state_dict_29.pth

# ...
# 0 - slowllama is a 100% static, 100% offline, 100% open source, 100% free,

# now run merge. we need to pass: 
#   - original model path
#   - new path for new model
#   - lora checkpoint path 
#   - optionally number of model shards (default = 1)
python merge_lora.py ../llama-2-7b ./data/state_dict_29.pth ../llama-2-7b-out

# copy tokenizer model over:
cp ../llama-2-7b/tokenizer.model ../llama-2-7b-out/

# now run new model with no extra checkpoint, observe new output, same as in combined model: 
python test_gen.py ../llama-2-7b-out mps

# ...
# 0 - slowllama is a 100% static, 100% offline, 100% open source, 100% free,

Now the ../llama-2-7b-out can be used in exactly same way as original llama2 for further quantization, inference, etc.

Project structure

Just a few files with no dependencies other than torch, numpy and sentencepiece for tokenizer.

blackbox_model.py -- model definition and manual backprop implementation. It's based on model.py from llama2.c, also MIT licenced.
finetune.py - script which does the training
loader.py - manual loading/saving of large llama2 models
utils.py - small utility functions, including saving/loading random generator state for different devices.
test_gen.py - greedily complete the prompt. Takes base weights + trained LoRA weights as input. Useful for sanity checks.
blackbox.py - module wrapper which offloads the module to disk or main memory.
plot_lora.py - logging utility, writes LoRA weights and gradient distribution to logfile. Requires fewlines. If fewlines is not installed, does nothing.
merge_lora.py - merge original weights + lora weights in the original format which can then be used directly.
prepare_model.py - script to transform sharded model to sequentially split model.

TODO:

[ ] masking
[ ] more generic train routine
    [ ] pause/resume from LoRA snapshot
    [ ] do not create LoRA layers on prepare, only on finetune?
[ ] how to make it work with fp16 on Apple?
[ ] optimizations - prefetch the next layer/input, save asyncronously, etc;
[ ] gradient accumulation
[ ] plot something like memory requirement for (batch_size , seq_len)
[ ] combined RAM/disk offload - 200Gb RAM is rarity.
[ ] tests, cleanup and comments;
[ ] progress tracking for everything;
[ ] quantization beyond 16 bit?
[ ] configurable weight tying;
[ ] double check RNG state correctness.

References

Contact

{github handle} @ gmail.com

fhdsl/AnVIL_Collection

2023-10-10T01:38:17Z

📚 An auto-generating collection of all materials related to the AnVIL and GDSCN projects

📚 AnVIL & GDSCN Collection

This site was created from the AnVIL Template, which is based on the OTTR Template

What is the AnVIL Collection?

AnVIL Collection uses the Github API to gather jhudsl and fhdsl organization repositories that we have worked on. It renders the table in a markdown-readable format on the website.

What repositories are included?

The Collection only includes repositories that meet the following criteria:

Are public
Have a homepage listed
Have a description listed
Have "anvil" and/or "gdscn" as tags

How often is the Collection updated?

The collection is rendered nightly at 3am EST.

How can I add my new course/content to the AnVIL Collection?

Your repository must be created under the jhudsl or fhdsl organizations and meet the above criteria. For more information see the Contribute section.

For repositories under other organizations, you can fork the repository into the jhudsl or fhdsl organizations. Example

I want to use the raw Collection data!

You can use the following link:

https://raw.githubusercontent.com/fhdsl/AnVIL_Collection/main/resources/collection.tsv

Encountering problems?

If you are encountering any problems with this course, please file a GitHub issue.

All materials in this course are licensed under a Creative Commons Attribution 4.0 International License unless noted otherwise.

GaelleMG/starflock_firexaq

2023-10-10T01:38:17Z

2022 NASA International Space Apps Challenge

starflock_firexaq

2022 NASA International Space Apps Challenge

red-hat-data-services/ods-ci

2023-10-11T01:38:48Z

odh qe tier tests

gocardless/statesman

2023-10-10T01:39:22Z

A statesmanlike state machine library.

For our policy on compatibility with Ruby and Rails versions, see COMPATIBILITY.md.

Statesman is an opinionated state machine library designed to provide a robust audit trail and data integrity. It decouples the state machine logic from the underlying model and allows for easy composition with one or more model classes.

As such, the design of statesman is a little different from other state machine libraries:

State behaviour is defined in a separate, "state machine" class, rather than added directly onto a model. State machines are then instantiated with the model to which they should apply.
State transitions are also modelled as a class, which can optionally be persisted to the database for a full audit history. This audit history can include JSON metadata set during a transition.
Database indices are used to offer database-level transaction duplication protection.

Installation

To get started, just add Statesman to your Gemfile, and then run bundle:

gem 'statesman', '~> 10.0.0'

Usage

First, create a state machine based on Statesman::Machine:

class OrderStateMachine
  include Statesman::Machine

  state :pending, initial: true
  state :checking_out
  state :purchased
  state :shipped
  state :cancelled
  state :failed
  state :refunded

  transition from: :pending,      to: [:checking_out, :cancelled]
  transition from: :checking_out, to: [:purchased, :cancelled]
  transition from: :purchased,    to: [:shipped, :failed]
  transition from: :shipped,      to: :refunded

  guard_transition(to: :checking_out) do |order|
    order.products_in_stock?
  end

  before_transition(from: :checking_out, to: :cancelled) do |order, transition|
    order.reallocate_stock
  end

  before_transition(to: :purchased) do |order, transition|
    PaymentService.new(order).submit
  end

  after_transition(to: :purchased) do |order, transition|
    MailerService.order_confirmation(order).deliver
  end
end

Then, link it to your model:

class Order < ActiveRecord::Base
  has_many :order_transitions, autosave: false

  include Statesman::Adapters::ActiveRecordQueries[
    transition_class: OrderTransition,
    initial_state: :pending
  ]

  def state_machine
    @state_machine ||= OrderStateMachine.new(self, transition_class: OrderTransition)
  end
end

Next, you'll need to create a further model to represent state transitions:

class OrderTransition < ActiveRecord::Base
  include Statesman::Adapters::ActiveRecordTransition

  validates :to_state, inclusion: { in: OrderStateMachine.states }

  belongs_to :order, inverse_of: :order_transitions
end

Now, you can start working with your state machine:

Order.first.state_machine.current_state # => "pending"
Order.first.state_machine.allowed_transitions # => ["checking_out", "cancelled"]
Order.first.state_machine.can_transition_to?(:cancelled) # => true/false
Order.first.state_machine.transition_to(:cancelled, optional: :metadata) # => true/false
Order.first.state_machine.transition_to!(:cancelled) # => true/exception
Order.first.state_machine.last_transition # => transition model or nil
Order.first.state_machine.last_transition_to(:pending) # => transition model or nil

Order.in_state(:cancelled) # => [#<Order id: "123">]
Order.not_in_state(:checking_out) # => [#<Order id: "123">]

If you'd like, you can also define a template for a generic state machine, then alter classes which extend it as required:

module Template
  def define_states
    state :a, initial: true
    state :b
    state :c
  end

  def define_transitions
    transition from: :a, to: :b
    transition from: :b, to: :c
    transition from: :c, to: :a
  end
end

class Circular
  include Statesman::Machine
  extend Template
  
  define_states
  define_transitions
end

class Linear
  include Statesman::Machine
  extend Template
  
  define_states
  define_transitions
  
  remove_transitions from: :c, to: :a
end

class Shorter
  include Statesman::Machine
  extend Template

  define_states
  define_transitions

  remove_state :c
end

Persistence

By default Statesman stores transition history in memory only. It can be persisted by configuring Statesman to use a different adapter. For example, for ActiveRecord within Rails:

config/initializers/statesman.rb:

Statesman.configure do
  storage_adapter(Statesman::Adapters::ActiveRecord)
end

Generate the transition model:

$ rails g statesman:active_record_transition Order OrderTransition

Your transition class should include Statesman::Adapters::ActiveRecordTransition if you're using the ActiveRecord adapter.

If you're using the ActiveRecord adapter and decide not to include the default updated_at column in your transition table, you'll need to configure the updated_timestamp_column option on the transition class, setting it to another column name (e.g. :updated_on) or nil.

And add an association from the parent model:

app/models/order.rb:

class Order < ActiveRecord::Base
  has_many :transitions, class_name: "OrderTransition", autosave: false

  # Initialize the state machine
  def state_machine
    @state_machine ||= OrderStateMachine.new(self, transition_class: OrderTransition,
                                                   association_name: :transitions)
  end

  # Optionally delegate some methods

  delegate :can_transition_to?,
           :current_state, :history, :last_transition, :last_transition_to,
           :transition_to!, :transition_to, :in_state?, to: :state_machine
end

Using PostgreSQL JSON column

By default, Statesman uses serialize to store the metadata in JSON format. It is also possible to use the PostgreSQL JSON column if you are using Rails 4 or 5. To do that

Change metadata column type in the transition model migration to json or jsonb

# Before
t.text :metadata, default: "{}"
# After (Rails 4)
t.json :metadata, default: "{}"
# After (Rails 5)
t.json :metadata, default: {}

Remove the include Statesman::Adapters::ActiveRecordTransition statement from your transition model. (If you want to customise your transition class's "updated timestamp column", as described above, you should define a .updated_timestamp_column method on your class and return the name of the column as a symbol, or nil if you don't want to record an updated timestamp on transitions.)

Configuration

`storage_adapter`

Statesman.configure do
  storage_adapter(Statesman::Adapters::ActiveRecord)
end

Statesman defaults to storing transitions in memory. If you're using rails, you can instead configure it to persist transitions to the database by using the ActiveRecord adapter.

Statesman will fallback to memory unless you specify a transition_class when instantiating your state machine. This allows you to only persist transitions on certain state machines in your app.

Class methods

`Machine.state`

Machine.state(:some_state, initial: true)
Machine.state(:another_state)

Define a new state and optionally mark as the initial state.

`Machine.transition`

Machine.transition(from: :some_state, to: :another_state)

Define a transition rule. Both method parameters are required, to can also be an array of states (.transition(from: :some_state, to: [:another_state, :some_other_state])).

`Machine.guard_transition`

Machine.guard_transition(from: :some_state, to: :another_state) do |object|
  object.some_boolean?
end

Define a guard. to and from parameters are optional, a nil parameter means guard all transitions. The passed block should evaluate to a boolean and must be idempotent as it could be called many times. The guard will pass when it evaluates to a truthy value and fail when it evaluates to a falsey value (nil or false).

`Machine.before_transition`

Machine.before_transition(from: :some_state, to: :another_state) do |object|
  object.side_effect
end

Define a callback to run before a transition. to and from parameters are optional, a nil parameter means run before all transitions. This callback can have side-effects as it will only be run once immediately before the transition.

`Machine.after_transition`

Machine.after_transition(from: :some_state, to: :another_state) do |object, transition|
  object.side_effect
end

Define a callback to run after a successful transition. to and from parameters are optional, a nil parameter means run after all transitions. The model object and transition object are passed as arguments to the callback. This callback can have side-effects as it will only be run once immediately after the transition.

If you specify after_commit: true, the callback will be executed once the transition has been committed to the database.

`Machine.after_transition_failure`

Machine.after_transition_failure(from: :some_state, to: :another_state) do |object, exception|
  Logger.info("transition to #{exception.to} failed for #{object.id}")
end

Define a callback to run if Statesman::TransitionFailedError is raised during the execution of transition callbacks. to and from parameters are optional, a nil parameter means run after all transitions. The model object, and exception are passed as arguments to the callback. This is executed outside of the transaction wrapping other callbacks. If using transition! the exception is re-raised after these callbacks are executed.

`Machine.after_guard_failure`

Machine.after_guard_failure(from: :some_state, to: :another_state) do |object, exception|
  Logger.info("guard failed during transition to #{exception.to} for #{object.id}")
end

Define a callback to run if Statesman::GuardFailedError is raised during the execution of guard callbacks. to and from parameters are optional, a nil parameter means run after all transitions. The model object, and exception are passed as arguments to the callback. This is executed outside of the transaction wrapping other callbacks. If using transition! the exception is re-raised after these callbacks are executed.

`Machine.new`

my_machine = Machine.new(my_model, transition_class: MyTransitionModel)

Initialize a new state machine instance. my_model is required. If using the ActiveRecord adapter my_model should have a has_many association with MyTransitionModel.

`Machine.retry_conflicts`

Machine.retry_conflicts { instance.transition_to(:new_state) }

Automatically retry the given block if a TransitionConflictError is raised. If you know you want to retry a transition if it fails due to a race condition call it from within this block. Takes an (optional) argument for the maximum number of retry attempts (defaults to 1).

`Machine.states`

Returns an array of all possible state names as strings.

`Machine.successors`

Returns a hash of states and the states it is valid for them to transition to.

Machine.successors

{
  "pending" => ["checking_out", "cancelled"],
  "checking_out" => ["purchased", "cancelled"],
  "purchased" => ["shipped", "failed"],
  "shipped" => ["refunded"]
}

Instance methods

`Machine#current_state`

Returns the current state based on existing transition objects.

Takes an optional keyword argument to force a reload of data from the database. e.g current_state(force_reload: true)

`Machine#in_state?(:state_1, :state_2, ...)`

Returns true if the machine is in any of the given states.

`Machine#history`

Returns a sorted array of all transition objects.

`Machine#last_transition`

Returns the most recent transition object.

`Machine#last_transition_to(:state)`

Returns the most recent transition object to a given state.

`Machine#allowed_transitions`

Returns an array of states you can transition_to from current state.

`Machine#can_transition_to?(:state)`

Returns true if the current state can transition to the passed state and all applicable guards pass.

`Machine#transition_to!(:state)`

Transition to the passed state, returning true on success. Raises Statesman::GuardFailedError or Statesman::TransitionFailedError on failure.

`Machine#transition_to(:state)`

Transition to the passed state, returning true on success. Swallows all Statesman exceptions and returns false on failure. (NB. if your guard or callback code throws an exception, it will not be caught.)

Errors

Initialization errors

These errors are raised when the Machine and/or Model is initialized. A simple spec like

expect { OrderStateMachine.new(Order.new, transition_class: OrderTransition) }.to_not raise_error

will expose these errors as part of your test suite

InvalidStateError

Raised if:

Attempting to define a transition without a to state.
Attempting to define a transition with a non-existent state.
Attempting to define multiple states as initial.

InvalidTransitionError

Raised if:

Attempting to define a callback from a state that has no valid transitions (A terminal state).
Attempting to define a callback to the initial state if that state has no transitions to it.
Attempting to define a callback with from and to where any of the pairs have no transition between them.

InvalidCallbackError

Raised if:

Attempting to define a callback without a block.

UnserializedMetadataError

Raised if:

ActiveRecord is configured to not serialize the metadata attribute into to Database column backing it. See the Using PostgreSQL JSON column section.

IncompatibleSerializationError

Raised if:

There is a mismatch between the column type of the metadata in the Database and the model. See the Using PostgreSQL JSON column section.

MissingTransitionAssociation

Raised if:

The model that Statesman::Adapters::ActiveRecordQueries is included in does not have a has_many association to the transition_class.

Runtime errors

These errors are raised by transition_to!. Using transition_to will supress GuardFailedError and TransitionFailedError and return false instead.

GuardFailedError

Raised if:

A guard callback between from and to state returned a falsey value.

TransitionFailedError

Raised if:

A transition is attempted but current_state -> new_state is not a valid pair.

TransitionConflictError

Raised if:

A database conflict affecting the sort_key or most_recent columns occurs when attempting a transition. Retried automatically if it occurs wrapped in retry_conflicts.

Model scopes

A mixin is provided for the ActiveRecord adapter which adds scopes to easily find all models currently in (or not in) a given state. Include it into your model and passing in transition_class and initial_state as options.

In 4.1.2 and below, these two options had to be defined as methods on the model, but 5.0.0 and above allow this style of configuration as well. The old method pollutes the model with extra class methods, and is deprecated, to be removed in 6.0.0.

class Order < ActiveRecord::Base
  has_many :order_transitions, autosave: false
  include Statesman::Adapters::ActiveRecordQueries[
    transition_class: OrderTransition,
    initial_state: OrderStateMachine.initial_state
  ]
end

If the transition class-name differs from the association name, you will also need to pass transition_name as an option:

class Order < ActiveRecord::Base
  has_many :transitions, class_name: "OrderTransition", autosave: false
  include Statesman::Adapters::ActiveRecordQueries[
    transition_class: OrderTransition,
    initial_state: OrderStateMachine.initial_state,
    transition_name: :transitions
  ]
end

`Model.in_state(:state_1, :state_2, etc)`

Returns all models currently in any of the supplied states.

`Model.not_in_state(:state_1, :state_2, etc)`

Returns all models not currently in any of the supplied states.

`Model.most_recent_transition_join`

This joins the model to its most recent transition whatever that may be. We expose this method to ease use of ActiveRecord's or e.g

Model.in_state(:state_1).or(
  Model.most_recent_transition_join.where(model_field: 123)
)

Frequently Asked Questions

Storing the state on the model object

If you wish to store the model state on the model directly, you can keep it up to date using an after_transition hook. Combine it with the after_commit option to ensure the model state will only be saved once the transition has made it irreversibly to the database:

after_transition(after_commit: true) do |model, transition|
  model.state = transition.to_state
  model.save!
end

You could also use a calculated column or view in your database.

Accessing metadata from the last transition

Given a field foo that was stored in the metadata, you can access it like so:

model_instance.state_machine.last_transition.metadata["foo"]

Events

Used to using a state machine with "events"? Support for events is provided by the statesman-events gem. Once that's included in your Gemfile you can include event functionality in your state machine as follows:

class OrderStateMachine
  include Statesman::Machine
  include Statesman::Events

  ...
end

Deleting records.

If you need to delete the Parent model regularly you will need to change either the association deletion behaviour or add a DELETE CASCADE condition to foreign key in your database.

E.g

has_many :order_transitions, autosave: false, dependent: :destroy

or when migrating the transition model

add_foreign_key :order_transitions, :orders, on_delete: :cascade

Testing Statesman Implementations

This answer was abstracted from this issue.

At GoCardless we focus on testing that:

guards correctly prevent / allow transitions
callbacks execute when expected and perform the expected actions

Testing Guards

Guards can be tested by asserting that transition_to! does or does not raise a Statesman::GuardFailedError:

describe "guards" do
  it "cannot transition from state foo to state bar" do
    expect { some_model.transition_to!(:bar) }.to raise_error(Statesman::GuardFailedError)
  end

  it "can transition from state foo to state baz" do
    expect { some_model.transition_to!(:baz) }.to_not raise_error
  end
end

Testing Callbacks

Callbacks are tested by asserting that the action they perform occurs:

describe "some callback" do
  it "adds one to the count property on the model" do
    expect { some_model.transition_to!(:some_state) }.
      to change { some_model.reload.count }.
      by(1)
  end
end

Compatibility with type checkers

Including ActiveRecordQueries to your model can cause issues with type checkers such as Sorbet, this is because this technically is using a dynamic include, which is not supported by Sorbet.

To avoid these issues you can instead include the TypeSafeActiveRecordQueries module and pass in configuration.

class Order < ActiveRecord::Base
  has_many :order_transitions, autosave: false

  include Statesman::Adapters::TypeSafeActiveRecordQueries

  configure_state_machine transition_class: OrderTransition,
                          initial_state: :pending

  def state_machine
    @state_machine ||= OrderStateMachine.new(self, transition_class: OrderTransition)
  end
end

Third-party extensions

statesman-sequel - An adapter to make Statesman work with Sequel

GoCardless ♥ open source. If you do too, come join us.

rubocop/rubocop-minitest

2023-10-10T01:39:22Z

A Ruby gem for communicating with the Twilio API and generating TwiML

twilio-ruby

Documentation

The documentation for the Twilio API can be found here.

The individual releases here.

Versions

twilio-ruby uses a modified version of Semantic Versioning for all changes. See this document for details.

Supported Ruby Versions

This library supports the following Ruby implementations:

Ruby 2.4
Ruby 2.5
Ruby 2.6
Ruby 2.7
Ruby 3.0
Ruby 3.1
Ruby 3.2
JRuby 9.2
JRuby 9.3
JRuby 9.4

Migrating from 5.x

Upgrade Guide

Installation

To install using Bundler grab the latest stable version:

gem 'twilio-ruby', '~> 6.7.1'

To manually install twilio-ruby via Rubygems simply gem install:

gem install twilio-ruby -v 6.7.1

To build and install the development branch yourself from the latest source:

git clone git@github.com:twilio/twilio-ruby.git
cd twilio-ruby
make install

Info If the command line gives you an error message that says Permission Denied, try running the above commands with sudo.

For example: sudo gem install twilio-ruby

Test your installation

To make sure the installation was successful, try sending yourself an SMS message, like this:

require "twilio-ruby"

# Your Account SID and Auth Token from console.twilio.com
account_sid = "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
auth_token = "your_auth_token"

@client = Twilio::REST::Client.new account_sid, auth_token
message = @client.messages.create(
  body: "Hello from Ruby",
  to: "+12345678901",  # Text this number
  from: "+15005550006", # From a valid Twilio number
)

puts message.sid

Warning It's okay to hardcode your credentials when testing locally, but you should use environment variables to keep them secret before committing any code or deploying to production. Check out How to Set Environment Variables for more information.

Usage

Authenticate the Client

require 'twilio-ruby'

# Your Account SID and Auth Token from console.twilio.com
account_sid = 'ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
auth_token = 'yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'

# Initialize the Twilio Client with your credentials
@client = Twilio::REST::Client.new account_sid, auth_token

Use An API Key

require 'twilio-ruby'

# Your Account SID from console.twilio.com
account_sid = 'ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
# API Key from twilio.com/console/project/api-keys
api_key_sid = 'zzzzzzzzzzzzzzzzzzzzzz'
api_key_secret = 'yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'

# set up a client to talk to the Twilio REST API using an API Key
@client = Twilio::REST::Client.new api_key_sid, api_key_secret, account_sid

Specify a Region and/or Edge

To take advantage of Twilio's Global Infrastructure, specify the target Region and/or Edge for the client:

# set up a client to talk to the Twilio REST API over a specific region and edge
@client = Twilio::REST::Client.new account_sid, auth_token, nil, 'au1'
@client.edge = 'sydney'

# you may also specify the region and/or edge after client creation
@client = Twilio::REST::Client.new account_sid, auth_token
@client.region = 'au1'
@client.edge = 'sydney'

This will result in the hostname transforming from api.twilio.com to api.sydney.au1.twilio.com.

Make a Call

@client.calls.create(
  from: '+14159341234',
  to: '+16105557069',
  url: 'http://example.com'
)

Send an SMS

@client.messages.create(
  from: '+14159341234',
  to: '+16105557069',
  body: 'Hey there!'
)

List your SMS Messages

@client.messages.list(limit: 20)

Fetch a single SMS message by Sid

# put the message sid you want to retrieve here:
message_sid = 'SMxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
@client.messages(message_sid).fetch

Iterate through records

The library automatically handles paging for you. Collections, such as calls and messages, have list and stream methods that page under the hood. With both list and stream, you can specify the number of records you want to receive (limit) and the maximum size you want each page fetch to be (page_size). The library will then handle the task for you.

list eagerly fetches all records and returns them as a list, whereas stream returns an enumerator and lazily retrieves pages of records as you iterate over the collection. You can also page manually using the page method.

For more information about these methods, view the auto-generated library docs.

require 'twilio-ruby'

account_sid = 'ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
auth_token = 'your_auth_token'

@client = Twilio::REST::Client.new(account_sid, auth_token)

@client.calls.list
       .each do |call|
         puts call.direction
       end

Enable Debug logging

In order to enable debug logging, pass in a 'logger' instance to the client with the level set to at least 'DEBUG'

@client = Twilio::REST::Client.new account_sid, auth_token
myLogger = Logger.new(STDOUT)
myLogger.level = Logger::DEBUG
@client.logger = myLogger

@client = Twilio::REST::Client.new account_sid, auth_token
myLogger = Logger.new('my_log.log')
myLogger.level = Logger::DEBUG
@client.logger = myLogger

Handle Exceptions {#exceptions}

If the Twilio API returns a 400 or a 500 level HTTP response, the twilio-ruby library will throw a Twilio::REST::RestError. 400-level errors are normal during API operation (“Invalid number”, “Cannot deliver SMS to that number”, for example) and should be handled appropriately.

require 'twilio-ruby'

account_sid = 'ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
auth_token = 'your_auth_token'

@client = Twilio::REST::Client.new account_sid, auth_token

begin
  messages = @client.messages.list(limit: 20)
rescue Twilio::REST::RestError => e
  puts e.message
end

Debug API requests

To assist with debugging, the library allows you to access the underlying request and response objects. This capability is built into the default HTTP client that ships with the library.

For example, you can retrieve the status code of the last response like so:

require 'rubygems' # Not necessary with ruby 1.9 but included for completeness
require 'twilio-ruby'

# Your Account SID and Auth Token from console.twilio.com
account_sid = 'ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
auth_token = 'your_auth_token'

@client = Twilio::REST::Client.new(account_sid, auth_token)

@message = @client.messages.create(
  to: '+14158675309',
  from: '+14258675310',
  body: 'Ahoy!'
)

# Retrieve the status code of the last response from the HTTP client
puts @client.http_client.last_response.status_code

Customize your HTTP Client

twilio-ruby uses Faraday to make HTTP requests. You can tell Twilio::REST::Client to use any of the Faraday adapters like so:

@client.http_client.adapter = :typhoeus

To use a custom HTTP client with this helper library, please see the advanced example of how to do so.

To apply customizations such as middleware, you can use the configure_connection method like so:

@client.http_client.configure_connection do |faraday|
  faraday.use SomeMiddleware
end

Get started With Client Capability Tokens

If you just need to generate a Capability Token for use with Twilio Client, you can do this:

require 'twilio-ruby'

# put your own account credentials here:
account_sid = 'ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
auth_token = 'yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'

# set up
capability = Twilio::JWT::ClientCapability.new account_sid, auth_token

# allow outgoing calls to an application
outgoing_scope = Twilio::JWT::ClientCapability::OutgoingClientScope.new 'AP11111111111111111111111111111111'
capability.add_scope(outgoing_scope)

# allow incoming calls to 'andrew'
incoming_scope = Twilio::JWT::ClientCapability::IncomingClientScope.new 'andrew'
capability.add_scope(incoming_scope)

# generate the token string
@token = capability.to_s

There is a slightly more detailed document in the Capability section of the wiki.

Generate TwiML

To control phone calls, your application needs to output TwiML.

You can construct a TwiML response like this:

require 'twilio-ruby'

response = Twilio::TwiML::VoiceResponse.new do |r|
  r.say(message: 'hello there', voice: 'alice')
  r.dial(caller_id: '+14159992222') do |d|
    d.client 'jenny'
  end
end

# print the result
puts response.to_s

This will print the following (except for the whitespace):

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Say voice="alice">hello there</Say>
  <Dial callerId="+14159992222">
    <Client>jenny</Client>
  </Dial>
</Response>

Docker Image

The Dockerfile present in this repository and its respective twilio/twilio-ruby Docker image are currently used by Twilio for testing purposes only.

Getting help

If you need help installing or using the library, please check the Twilio Support Help Center first, and file a support ticket if you don't find an answer to your question.

If you've instead found a bug in the library or would like new features added, go ahead and open issues or pull requests against this repo!

fuzzland/ityfuzz

2023-10-10T01:39:12Z

Blazing Fast Bytecode-Level Hybrid Fuzzer for Smart Contracts

ItyFuzz 🍦

Fast hybrid fuzzer for EVM & MoveVM (WIP) smart contracts.

You can generate exploits instantly by just providing the contract address:

Tool / Research Paper / Fuzzing EVM Contracts / Fuzzing Move Contracts

Statistics & Comparison

Time taken for finding vulnerabilities / generating exploits:

Project Name	Vulnerability	Mythril	SMARTIAN	Slither	ItyFuzz
AES	Business Logic	Inf	Unsupported	No	4hrs
Carrot	Arbitrary External Call	17s	11s	Yes	1s
Olympus	Access Control	36s	Inf	Yes	1s
MUMUG	Price Manipulation	Inf	Unsupported	No	18hrs
Omni	Reentrancy	Inf	Unsupported	Yes*	22hrs
Verilog CTF-2	Reentrancy	Inf	Unsupported	Yes*	3s

_{* Slither only finds the reentrancy location, but not how to leverage reentrancy to trigger final buggy code. The output also contains significant amount of false positives.}

Test Coverage:

Dataset	SMARTIAN	Echidna	ItyFuzz
B1	97.1%	47.1%	99.2%
B2	86.2%	82.9%	95.4%
Tests	Unsupported	52.9%	100%

_{* B1 and B2 contain 72 single-contract projects from SMARTIAN artifacts. Tests are the projects in tests directory. The coverage is calculated as (instruction covered) / (total instruction - dead code).}

Installation

ityfuzzup (Recommended)

ityfuzzup is a script that automatically installs all dependencies and builds ItyFuzz.

curl -L https://ity.fuzz.land/ | bash

Docker

Install Docker and run docker image suitable for your system architecture:

docker pull fuzzland/ityfuzz:stable
docker run -p 8000:8000 fuzzland/ityfuzz:stable

Then, you can visit the interface at http://localhost:8000

_{Note: The container uses public ETH RPC, may time out / be slow}

Build from Source

You first need to install Rust through https://rustup.rs/

You need to have libssl-dev (OpenSSL) and libz3-dev (refer to Z3 Installation section for instruction) installed.

git clone https://github.com/fuzzland/ityfuzz.git && cd ityfuzz
git submodule update --recursive --init
cargo build --release

You can enable certain debug gates in Cargo.toml

solc is needed for compiling smart contracts. You can use solc-select tool to manage the version of solc.

Run (EVM)

Compile Smart Contracts:

cd ./tests/evm/multi-contract/
# include the library from ./solidity_utils for example
solc *.sol -o . --bin --abi --overwrite --base-path ../../../

Run Fuzzer:

ityfuzz evm -t '../tests/evm/multi-contract/*'

Demo

Verilog CTF Challenge 2 tests/verilog-2/

Flashloan attack + Reentrancy. The target is to reach line 34 in Bounty.sol.

Exact Exploit:

0. Borrow k MATIC such that k > balance() / 10
1. depositMATIC() with k MATIC
2. redeem(k * 1e18) -- reentrancy contract --> getBounty()
3. Return k MATIC

Use fuzzer to detect the vulnerability and generate the exploit (takes 0 - 200s):

# build contracts in tests/evm/verilog-2/
solc *.sol -o . --bin --abi --overwrite --base-path ../../../
# run fuzzer 
ityfuzz evm -f -t "../tests/evm/verilog-2/*"

-f flag enables automated flashloan, which hooks all ERC20 external calls and make any users to have infinite balance.

Fuzz a Project (Offline)

You can fuzz a project by providing a path to the project directory.

ityfuzz evm -t '[DIR_PATH]/*'

ItyFuzz would attempt to deploy all artifacts in the directory to a blockchain with no other smart contracts.

Specifically, the project directory should contain a few [X].abi and [X].bin files. For example, to fuzz a contract named main.sol, you should ensure main.abi and main.bin exist in the project directory. The fuzzer will automatically detect the contracts in directory, the correlation between them (see tests/evm/multi-contract), and fuzz them.

Optionally, if ItyFuzz fails to infer the correlation between contracts, you can add a [X].address, where [X] is the contract name, to specify the address of the contract.

Caveats:

Keep in mind that ItyFuzz is fuzzing on a clean blockchain, so you should ensure all related contracts (e.g., ERC20 token, Uniswap, etc.) are deployed to the blockchain before fuzzing.
If your smart contract requires constructor arguments, please refer to below Constructor Arguments section.

Fuzz a Project (Online)

Ityfuzz will prioritize the use of environment variables ETH_RPC_URL as the RPC endpoint, otherwise it will use built-in public RPCs.

Rebuild with flashloan_v2 (only supported in onchain) enabled to get better result.

python3 -c 'content=open("Cargo.toml").read().replace("default = [", "default = [\"flashloan_v2\",");open("Cargo.toml","w").write(content);'
cargo build --release

You can fuzz a project by providing an address, a block, and a chain type.

ityfuzz evm -o -t [TARGET_ADDR] --onchain-block-number [BLOCK] -c [CHAIN_TYPE] --onchain-etherscan-api-key [Etherscan API Key]

Example: Fuzzing WETH contract (0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2) on Ethereum mainnet at latest block.

ityfuzz evm -o -t 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 --onchain-block-number 0 -c ETH --onchain-etherscan-api-key PXUUKVEQ7Y4VCQYPQC2CEK4CAKF8SG7MVF

Fuzzing with flashloan and oracles enabled:

ityfuzz evm -o -t 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 --onchain-block-number 0 -c ETH -f -i -p --onchain-etherscan-api-key PXUUKVEQ7Y4VCQYPQC2CEK4CAKF8SG7MVF

ItyFuzz would pull the ABI of the contract from Etherscan and fuzz it. If ItyFuzz encounters an unknown slot in the memory, it would pull the slot from chain RPC. If ItyFuzz encounters calls to external unknown contract, it would pull the bytecode and ABI of that contract. If its ABI is not available, ItyFuzz would not send any transaction to that contract.

Constructor Arguments

ItyFuzz provides two methods to pass in constructor arguments. These arguments are necessary for initializing the state of the contract when deployed.

Method 1: CLI Arguments

The first method is to pass in the constructor arguments directly as CLI arguments.

When you run ItyFuzz using the CLI, you can include the --constructor-args flag followed by a string that specifies the arguments for each constructor.

The format is as follows:

ityfuzz evm -t 'tests/evm/multi-contract/*' --constructor-args "ContractName:arg1,arg2,...;AnotherContract:arg1,arg2,..;"

For example, if you have two contracts, main and main2, both having a bytes32 and a uint256 as constructor arguments, you would pass them in like this:

ityfuzz evm -t 'tests/evm/multi-contract/*' --constructor-args "main:1,0x6100000000000000000000000000000000000000000000000000000000000000;main2:2,0x6200000000000000000000000000000000000000000000000000000000000000;"

Method 2: Server Forwarding

The second method is to use our server to forward requests to a user-specified RPC, and cli will fetch the constructor arguments from the transactions sent to the RPC.

Firstly, go to the /server directory, and install the necessary packages:

cd /server
npm install

Then, start the server using the following command:

node app.js

By default, the server will forward requests to http://localhost:8545, which is the default address for Ganache, if you do not have a local blockchain running, you can use Ganache to start one. If you wish to forward requests to another location, you can specify the address as a command-line argument like so:

node app.js http://localhost:8546

Once the server is running, you can deploy your contract to localhost:5001 using a tool of your choice.

For example, you can use Foundry to deploy your contract through the server:

forge create src/flashloan.sol:main2 --rpc-url http://127.0.0.1:5001 --private-key 0x0000000000000000000000000000000000000000000000000000000000000000 --constructor-args "1" "0x6100000000000000000000000000000000000000000000000000000000000000"

Finally, you can fetch the constructor arguments using the --fetch-tx-data flag:

ityfuzz evm -t 'tests/evm/multi-contract/*' --fetch-tx-data

ItyFuzz will fetch the constructor arguments from the transactions forwarded to the RPC through the server.

Concolic Execution Support (Experimental)

Concolic execution can be performed on certain testcases on the fly during fuzzing. It is particularly useful for fuzzing code with complex if-conditions. You can add --concolic to args to make fuzzer conduct concolic execution. You can also add --concolic-caller to args to make fuzzer solve for callers.

Example:

cd tests/evm/concolic-1/ && solc *.sol -o . --bin --abi --overwrite --base-path ../../../ && ../../../
ityfuzz evm -t 'tests/evm/concolic-1/*' --concolic --concolic-caller

Finding Custom Bugs (EVM)

You can simply insert bug() or typed_bug(string message) in your contract to report a condition when bug is found.

For instance, a simple case can be written as follows:

function buy_token() public {
    if (msg.sender != owner) {
        bug();
    }
}

The implementation of bug() is as follows:

library FuzzLand {
    event AssertionFailed(string message);
  
    function bug() internal {
        emit AssertionFailed("Bug");
    }
  
    function typed_bug(string memory data) internal {
        emit AssertionFailed(data);
    }

}

function bug()  {
    FuzzLand.bug();
}

function typed_bug(string memory data)  {
    FuzzLand.typed_bug(data);
}

You can either paste the code above into your contract or import it from solidity_utils/lib.sol, if you are using bug or typed_bug.

Echidna Support

Any contracts bearing functions starting with echidna_ will be treated as invariants and will be tested by ItyFuzz. If it returns false, the fuzzer will report a bug.

function echidna_test() public {
    assert(false);
}

Scribble Support

Scribble is a tool for writing specifications for Solidity contracts. ItyFuzz supports Scribble annotations after it is compiled by scribble.

For example, the following contract has a Scribble annotation that specifies the return value of inc:

contract Foo {
    /// #if_succeeds {:msg "P1"} y == x + 2;
    function inc(uint x) public pure returns (uint y) {
        return x+1;
    }
}

You need to compile the contract using scribble and pass the compiled contract to ItyFuzz

Note that you must add --no-assert to the scribble command. Otherwise, ItyFuzz will not detect any bugs.

scribble test.sol --output-mode flat --output compiled.sol --no-assert

Then compile with solc and run ItyFuzz:

solc compiled.sol --bin --abi --overwrite -o build
ityfuzz evm -t "build/*" [More Arguments]

Test Coverage

ItyFuzz can collect instruction and branch coverage information for all the contracts it fuzzes. You simply need to append --replay-file [WORKDIR]/corpus/*_replayable to collect all these information.

ityfuzz evm -t [Targets] [Options Used During Fuzzing] --replay-file '[WORKDIR]/corpus/*_replayable'

Example:

ityfuzz evm -t 'tests/evm/multi-contract/*' --replay-file 'work_dir/corpus/*_replayable'

You may add source map information to the targets to get more accurate coverage information and uncovered source code. To get source map information, you simply need to append --combined-json bin-runtime,srcmap-runtime to the solc command when building the targets.

# run in your target building directory (where you run solc)
solc [Options Used During Building] --combined-json bin-runtime,srcmap-runtime

Example:

# build contracts in tests/evm/verilog-2/
solc *.sol -o . --bin --abi --overwrite --base-path ../../ --combined-json bin-runtime,srcmap-runtime

Rarely, ItyFuzz has trouble to figure out the source code location. You may supply the absolute path to the base location (what you passed to solc's --base-path or if you didn't pass anything, it is the building directory) to ItyFuzz.

ityfuzz evm -t [Targets] [Options Used During Fuzzing] --replay-file '[WORKDIR]/corpus/*_replayable' --base-path [ABSOLUTE PATH TO BASE LOCATION]

Example:

# note that we used --base-path ../../ when building the targets so it is /home/user/ityfuzz/tests/evm/verilog-2/../../
ityfuzz evm -t 'tests/evm/multi-contract/*' --replay-file 'work_dir/corpus/*_replayable' --base-path /home/user/ityfuzz

We do not track coverage of static calls (view, pure functions) by default!

Building With Move (Sui) Support

Build with feature sui_support in ./Cargo.toml to enable Move support.

# add sui_support feature to Cargo.toml
python3 -c 'content=open("Cargo.toml").read().replace("default = [", "default = [\"sui_support\",");open("Cargo.toml","w").write(content);'

# build ItyFuzz with sui_support feature
cargo build --release

You may also want to install sui-cli to build Move contracts.

Run (Move)

Compile the contracts with sui move build and run ItyFuzz:

# build example contract that contains a bug
cd ./tests/move/share_object
sui move build

# get back to ItyFuzz and run fuzzing on the built contract
cd ../../../
ityfuzz move -t "./tests/move/share_object/build"

Reporting Bugs (Move)

You can emit an event of `` in your contract to report a condition when bug is found.

// define the event struct
use sui::event;

struct AAAA__fuzzland_move_bug has drop, copy, store {
    info: u64
}

... 
    // inside function
    event::emit(AAAA__fuzzland_move_bug { info: 1 });
...

An example contract that report a bug can be found in tests/move/share_object/sources/test.move.

Troubleshooting

Z3 Installation

macOS

git clone https://github.com/Z3Prover/z3 && cd z3
python scripts/mk_make.py --prefix=/usr/local
cd build && make -j64 && sudo make install

If the build command still fails for not finding z3.h, do export Z3_SYS_Z3_HEADER=/usr/local/include/z3.h

Or you can use

brew install z3

Ubuntu

apt install libz3-dev

Onchain Fetching

ItyFuzz attempts to fetch storage from blockchain nodes when SLOAD is encountered and the target is uninitialized. There are three ways of fetching:

OneByOne: fetch one slot at a time. This is the default mode. It is slow but never fails.
All: fetch all slots at once using custom API eth_getStorageAll on our nodes. This is the fastest mode, but it may fail if the contract is too large.
Dump: dump storage using debug API debug_storageRangeAt. This only works for ETH (for now) and fails most of the time.

Citation

@inproceedings{10.1145/3597926.3598059,
  author = {Shou, Chaofan and Tan, Shangyin and Sen, Koushik},
  title = {ItyFuzz: Snapshot-Based Fuzzer for Smart Contract},
  year = {2023},
  isbn = {9798400702211},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3597926.3598059},
  doi = {10.1145/3597926.3598059},
  booktitle = {Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis},
  pages = {322–333},
  numpages = {12},
  location = {Seattle, WA, USA},
  series = {ISSTA 2023}
}

Acknowledgement

This work was supported in part by NSF grants CCF-1900968, CCF1908870, and CNS1817122 and SKY Lab industrial sponsors and affiliates Astronomer, Google, IBM, Intel, Lacework, Microsoft, Mohamed Bin Zayed University of Artificial Intelligence, Nexla, Samsung SDS, Uber, and VMware. Any opinions, findings, conclusions, or recommendations in this repo do not necessarily reflect the position or the policy of the sponsors.

Grants:

Grants	Description
	Grants from Sui Foundation for building Move and chain-specific support
	Grants from Web3 Foundation for building Substrate pallets and Ink! support

huachaohuang/sigma

2023-10-11T01:39:15Z

The Sigma programming language

The Sigma Programming Language

This is an experimental project to build a modern SQL alternative called Sigma.

Sigma tries to combine the power of relational calculus and common programming languages.

Usage

cargo run

This will run an interactive shell like this:

Sigma 0.0.1
>>>

Then input some expressions and see what happens!

There is a built-in json module. You can load some json file and process it with Sigma:

>>> import json
>>> list = json.load("path/to/file.json")

Statements

Syntax:

Expression [';'] |
'import' Name [';'] |
CreateClass |
DeleteClass

Create class statement

Syntax:

'create' 'class' Name '{'
    (Name ':' TypeName ',')*
    ['id' Name (',' Name)*]
'}'

TypeName = 'bool' | 'i64' | 'f64' | 'text'

Example:

create class User { name: text, score: i64, id name }

You must specify the id fields for a class. The type of id fields can only be i64 or text for now. In this example, the "name" field will be used as the id of the class. By default, if the id is not specified, a field name "id" will be used as the id. For example:

create class User { id: text, score: i64 }

In this case the "id" field will be used as the id of the class.

After a class is created, the name of the class will represent a set of instances of that class. Then you can manipulate the class set like other collections:

>>> create class User { name: text, score: i64, id name }
>>> into User insert { name: "a", score: 1 }, { name: "b", score: 2 }
2
>>> from u in User select u.score
[2, 1]
>>> from u in User update u.score *= 2
2
>>> User
{
{name: "a", score: 2}
{name: "b", score: 4}
}

Delete class statement

Syntax:

'delete' 'class' Name

Expressions

Sigma is a strongly-typed scripting language. There is no implicit type conversions. For example, you can not add an integer with a boolean or a floating-point number.

Literal expressions

Null

TODO

Boolean

A boolean expression consists of one of the keywords true or false.

Numeric

Integer numbers

An integer literal should match the regular expression [0-9]+. For now, integer literal will be evaluated to 64bit signed integers at runtime.

Example:

>>> 123
123

Floating-point numbers

A floatint-point literal should match the regular expression [0-9]+\.[0-9]+([eE][+-]?[0-9]+)?. For now, floating-point literals will be evaluated to 64bit double-precision floating-point numbers at runtime.

Example:

>>> 1.2
1.2

Arbitrary-precision numbers

TODO

Text strings

A string literal should match the regular expression "([^"\\]|\\.)*".

>>> "abc"
"abc"
>>> "ab\"cd\n"
"ab\"cd\n"

Operator expressions

Binary operators

Syntax:

Expression '+' Expression |
Expression '-' Expression |
Expression '*' Expression |
Expression '/' Expression |
Expression '%' Expression |
Expression '|' Expression |
Expression '^' Expression |
Expression '&' Expression |

Example:

>>> 1 + 2
3
>>> true | false
true
>>> true & false
false

Negation operators

Syntax:

'!' Expression |
'-' Expression

Example:

>>> !true
false
>>> -1234
-1234

Comparison operators

Syntax:

Expression '==' Expression |
Expression '!=' Expression |
Expression '<' Expression |
Expression '<=' Expression |
Expression '>' Expression |
Expression '>=' Expression |
Expression 'in' Expression |
Expression 'not' 'in' Expression

Example:

>>> 1 == 1
true
>>> 1 != 1
false
>>> 1 in [1, 2]
true
>>> 1 not in [1, 2]
false

List and index expressions

List

List is a ordered sequence of objects.

Syntax:

'[' (Expression ',')* Expression? ']'

Example:

>>> []
[]
>>> [1, 2 + 3]
[1, 5]
>>> [1, 2, 3,]
[1, 2, 3]

Index

Syntax:

Expression '[' Expression ']'

Example:

>>> [1, 2, 3][0]
1

Hash and field expressions

Hash

Hash is a set of unordered name/value pairs. Hash field name can be an identifier or a string.

Syntax:

'{' (Name ':' Expression ',')* (Name ':' Expression)? '}'

Example:

>>> {}
{}
>>> {a: 1 + 1, b: "2"}
{a: 2, b: "2"}
>>> {foo: "foo", "foo bar": "foo bar"}
{foo: "foo", foo bar: "foo bar"}

Field

Syntax:

Expression '.' Name

Example:

>>> x = {foo: "bar", "foo bar": "foo bar"}
>>> x.foo
"bar"
>>> x."foo bar"
"foo bar"

Assignment

Syntax:

Expression '=' Expression

>>> a = [1, 2]
>>> a[0] = 2
>>> a
[2, 2]
>>> b = {x: 1, y: 1}
>>> b.x = 2
>>> b
{x: 2, y: 1}

Collection expressions

There is two kinds of collection for now: list and class.

Insert

Syntax:

'into' Expression 'insert' (Expression ',')* Expression ','?

Example:

>>> list = [1, 2]
>>> into list insert 3, 4
2
>>> list
>>> [1, 2, 3, 4]

Update

Syntax:

'from' Name 'in' Expression ['where' Expression] 'update' Expression

Example:

>>> list = [1, 2, 3]
>>> from x in list update x *= 2
3
>>> list
[2, 4, 6]
>>> from x in list where x == 4 update x += 1
1
>>> list
[2, 5, 6]

Delete

Syntax:

'from' Name 'in' Expression ['where' Expression] 'delete' Name

Example:

>>> list = [{a: 1, b: 2}, {a: 3, b: 4}]
>>> from x in list delete x
2
>>> list
[]
>>> list = [{a: 1, b: 2}, {a: 3, b: 4}]
>>> from x in list where x.a < 2 delete x
1
>>> list
[{a: 3, b: 4}]

Select

Syntax:

'from' Name 'in' Expression
['join' Name 'in' Expression ['on' Expression]]
['where' Expression]
['limit' Expression]
['select' Expression]

Example:

>>> list = from x in [1, 2, 3, 4] select { a: x, b: x * 2 }
[{a: 1, b: 2}, {a: 2, b: 4}, {a: 3, b: 6}, {a: 4, b: 8}]
>>> from x in list where x.a < 4
[{a: 1, b: 2}, {a: 2, b: 4}, {a: 3, b: 6}]
>>> from x in list where x.a < 4 limit 1
[{a: 1, b: 2}]
>>> from x in list where x.a < 4 limit 1..
[{a: 2, b: 4}, {a: 3, b: 6}]
>>> from x in list where x.a < 4 limit 1..2
[{a: 2, b: 4}]
>>> from x in list select x.a
[1, 2, 3, 4]
>>> from x in list where x.a < 4 limit 2 select { a: x.a, c: x.a + x.b }
[{a: 1, c: 3}, {a: 2, c: 6}]
>>> list2 = from x in [3, 4, 5, 6] select { a: x, c: x * 3 }
[{a: 3, c: 9}, {a: 4, c: 12}, {a: 5, c: 15}, {a: 6, c: 18}]
>>> from x1 in list join x2 in list2 on x1.a == x2.a
[{x1: {a: 3, b: 6}, x2: {a: 3, c: 9}}, {x1: {a: 4, b: 8}, x2: {a: 4, c: 12}}]
>>> from x1 in list join x2 in list2 on x1.a == x2.a where x1.a % 2 == 0 select { a: x1.a, b: x1.b, c: x2.c }
[{a: 4, b: 8, c: 12}]

Nested expressions are also possible:

>>> ids = [1, 2, 3, 4]
[1, 2, 3, 4]
>>> list = from x in ids select { a: x, b: ids }
[{a: 1, b: [1, 2, 3, 4]}, {a: 2, b: [1, 2, 3, 4]}, {a: 3, b: [1, 2, 3, 4]}, {a: 4, b: [1, 2, 3, 4]}]
>>> from x in list where x.a % 2 == 0 select { a: x.a, b: from i in x.b where i % 2 == 0 }
[{a: 2, b: [2, 4]}, {a: 4, b: [2, 4]}]

pgcentralfoundation/pgrx

2023-10-11T01:39:15Z

Build Postgres Extensions with Rust!

`pgrx`

Build Postgres Extensions with Rust!

pgrx is a framework for developing PostgreSQL extensions in Rust and strives to be as idiomatic and safe as possible.

pgrx supports Postgres v11-v15.

Feel free to join our Discord Server.

Key Features

A fully managed development environment with cargo-pgrx
- cargo pgrx new: Create new extensions quickly
- cargo pgrx init: Install new (or register existing) PostgreSQL installs
- cargo pgrx run: Run your extension and interactively test it in psql (or pgcli)
- cargo pgrx test: Unit-test your extension across multiple PostgreSQL versions
- cargo pgrx package: Create installation packages for your extension
- More in the README.md!
Target Multiple Postgres Versions
- Support Postgres v11-v15 from the same codebase
- Use Rust feature gating to use version-specific APIs
- Seamlessly test against all versions
Automatic Schema Generation
- Implement extensions entirely in Rust
- Automatic mapping for many Rust types into PostgreSQL
- SQL schemas generated automatically (or manually via cargo pgrx schema)
- Include custom SQL with extension_sql! & extension_sql_file!
Safety First
- Translates Rust panic!s into Postgres ERRORs that abort the transaction, not the process
- Memory Management follows Rust's drop semantics, even in the face of panic! and elog(ERROR)
- #[pg_guard] procedural macro to ensure the above
- Postgres Datums are Option<T> where T: FromDatum
  - NULL Datums are safely represented as Option::<T>::None
First-class UDF support
- Annotate functions with #[pg_extern] to expose them to Postgres
- Return pgrx::iter::SetOfIterator<'a, T> for RETURNS SETOF
- Return pgrx::iter::TableIterator<'a, T> for RETURNS TABLE (...)
- Create trigger functions with #[pg_trigger]
Easy Custom Types
- #[derive(PostgresType)] to use a Rust struct as a Postgres type
  - By default, represented as a CBOR-encoded object in-memory/on-disk, and JSON as human-readable
  - Provide custom in-memory/on-disk/human-readable representations
- #[derive(PostgresEnum)] to use a Rust enum as a Postgres enum
- Composite types supported with the pgrx::composite_type!("Sample") macro
Server Programming Interface (SPI)
- Safe access into SPI
- Transparently return owned Datums from an SPI context
Advanced Features
- Safe access to Postgres' MemoryContext system via pgrx::PgMemoryContexts
- Executor/planner/transaction/subtransaction hooks
- Safely use Postgres-provided pointers with pgrx::PgBox<T> (akin to alloc::boxed::Box<T>)
- #[pg_guard] proc-macro for guarding extern "C" Rust functions that need to be passed into Postgres
- Access Postgres' logging system through eprintln!-like macros
- Direct unsafe access to large parts of Postgres internals via the pgrx::pg_sys module
- New features added regularly!

System Requirements

A Rust toolchain: rustc, cargo, and rustfmt. The recommended way to get these is from https://rustup.rs †
git
libclang 5.0 or greater (required by bindgen)
- Ubuntu: apt install libclang-dev or apt install clang
- RHEL: yum install clang
tar
bzip2
GCC 7 or newer
PostgreSQL's build dependencies ‡

† PGRX has no MSRV policy, thus may require the latest stable version of Rust, available via Rustup

‡ A local PostgreSQL server installation is not required. cargo pgrx can download and compile PostgreSQL versions on its own.

How to: GCC 7 on CentOS 7

In order to use GCC 7, install scl and enter the GCC 7 development environment:

yum install centos-release-scl
yum install devtoolset-7
scl enable devtoolset-7 bash

Getting Started

First install the cargo-pgrx sub-command and initialize the development environment:

cargo install --locked cargo-pgrx
cargo pgrx init

The init command downloads currently supported PostgreSQL versions, compiles them to ~/.pgrx/, and runs initdb. It's also possible to use an existing (user-writable) PostgreSQL install, or install a subset of versions, see the README.md of cargo-pgrx for details.

cargo pgrx new my_extension
cd my_extension

This will create a new directory for the extension crate.

$ tree 
.
├── Cargo.toml
├── my_extension.control
├── sql
└── src
    └── lib.rs

2 directories, 3 files

The new extension includes an example, so you can go ahead and run it right away.

cargo pgrx run

This compiles the extension to a shared library, copies it to the specified Postgres installation, starts that Postgres instance and connects you to a database named the same as the extension.

Once cargo-pgrx drops us into psql we can load the extension and do a SELECT on the example function.

my_extension=# CREATE EXTENSION my_extension;
CREATE EXTENSION

my_extension=# SELECT hello_my_extension();
 hello_my_extension
---------------------
 Hello, my_extension
(1 row)

For more details on how to manage pgrx extensions see Managing pgrx extensions.

Upgrading

You can upgrade your current cargo-pgrx installation by passing the --force flag to cargo install:

cargo install --force --locked cargo-pgrx

As new Postgres versions are supported by pgrx, you can re-run the pgrx init process to download and compile them:

cargo pgrx init

Mapping of Postgres types to Rust

Postgres Type	Rust Type (as `Option<T>`)
`bytea`	`Vec<u8>` or `&[u8]` (zero-copy)
`text`	`String` or `&str` (zero-copy)
`varchar`	`String` or `&str` (zero-copy) or `char`
`"char"`	`i8`
`smallint`	`i16`
`integer`	`i32`
`bigint`	`i64`
`oid`	`u32`
`real`	`f32`
`double precision`	`f64`
`bool`	`bool`
`json`	`pgrx::Json(serde_json::Value)`
`jsonb`	`pgrx::JsonB(serde_json::Value)`
`date`	`pgrx::Date`
`time`	`pgrx::Time`
`timestamp`	`pgrx::Timestamp`
`time with time zone`	`pgrx::TimeWithTimeZone`
`timestamp with time zone`	`pgrx::TimestampWithTimeZone`
`anyarray`	`pgrx::AnyArray`
`anyelement`	`pgrx::AnyElement`
`box`	`pgrx::pg_sys::BOX`
`point`	`pgrx::pg_sys::Point`
`tid`	`pgrx::pg_sys::ItemPointerData`
`cstring`	`&core::ffi::CStr`
`inet`	`pgrx::Inet(String)` -- TODO: needs better support
`numeric`	`pgrx::Numeric<P, S> or pgrx::AnyNumeric`
`void`	`()`
`ARRAY[]::<type>`	`Vec<Option<T>>` or `pgrx::Array<T>` (zero-copy)
`int4range`	`pgrx::Range<i32>`
`int8range`	`pgrx::Range<i64>`
`numrange`	`pgrx::Range<Numeric<P, S>>` or `pgrx::Range<AnyRange>`
`daterange`	`pgrx::Range<pgrx::Date>`
`tsrange`	`pgrx::Range<pgrx::Timestamp>`
`tstzrange`	`pgrx::Range<pgrx::TimestampWithTimeZone>`
`NULL`	`Option::None`
`internal`	`pgrx::PgBox<T>` where `T` is any Rust/Postgres struct
`uuid`	`pgrx::Uuid([u8; 16])`

There are also IntoDatum and FromDatum traits for implementing additional type conversions, along with #[derive(PostgresType)] and #[derive(PostgresEnum)] for automatic conversion of custom types.

Note that text and varchar are converted to &str or String, so PGRX assumes any Postgres database you use it with has UTF-8-compatible encoding. Currently, PGRX will panic if it detects this is incorrect, to inform you, the programmer, that you were wrong. However, it is best to not rely on this behavior, as UTF-8 validation can be a performance hazard. This problem was previously assumed to simply not happen, and PGRX may decide to change the details of how it does UTF-8 validation checks in the future in order to mitigate performance hazards.

The default Postgres server encoding is SQL_ASCII, and it guarantees neither ASCII nor UTF-8 (as Postgres will then accept but ignore non-ASCII bytes). For best results, always use PGRX with UTF-8, and set database encodings explicitly upon database creation.

Digging Deeper

Caveats & Known Issues

There's probably more than are listed here, but a primary things of note are:

Threading is not really supported. Postgres is strictly single-threaded. As such, if you do venture into using threads, those threads MUST NOT call any internal Postgres function, or otherwise use any Postgres-provided pointer. There's also a potential problem with Postgres' use of sigprocmask. This was being discussed on the -hackers list, even with a patch provided, but the conversation seems to have stalled (https://www.postgresql.org/message-id/flat/5EF20168.2040508%40anastigmatix.net#4533edb74194d30adfa04a6a2ce635ba).
How to correctly interact with Postgres in an async context remains unexplored.
pgrx wraps a lot of unsafe code, some of which has poorly-defined safety conditions. It may be easy to induce illogical and undesirable behaviors even from safe code with pgrx, and some of these wrappers may be fundamentally unsound. Please report any issues that may arise.
Not all of Postgres' internals are included or even wrapped. This isn't due to it not being possible, it's simply due to it being an incredibly large task. If you identify internal Postgres APIs you need, open an issue and we'll get them exposed, at least through the pgrx::pg_sys module.
Windows is not supported. It could be, but will require a bit of work with cargo-pgrx and figuring out how to compile pgrx's "cshim" static library.
Sessions started before ALTER EXTENSION my_extension UPDATE; will continue to see the old version of my_extension. New sessions will see the updated version of the extension.
pgrx is used by many "in production", but it is not "1.0.0" or above, despite that being recommended by SemVer for production-quality software. This is because there are many unresolved soundness and ergonomics questions that will likely require breaking changes to resolve, in some cases requiring cutting-edge Rust features to be able to expose sound interfaces. While a 1.0.0 release is intended at some point, it seems prudent to wait until it seems like a 2.0.0 release would not be needed the next week and the remaining questions can be deferred.

TODO

There's a few things on our immediate TODO list

Automatic extension schema upgrade scripts
Improved unit testing framework
Better/Safer API for Datum management
Improved generated bindings organization
Safely wrap more Postgres internal APIs
More examples -- especially around memory management and the various derive macros #[derive(PostgresType/Enum)]

Feature Flags

PGRX has optional feature flags for Rust code that do not involve configuring the version of Postgres used, but rather extend additional support for other kinds of Rust code. These are not included by default.

"unsafe-postgres": Allow compilation for Postgres forks that have a different ABI

As of Postgres v15, forks are allowed to specify they use a different ABI than canonical Postgres. Since pgrx makes countless assumptions about Postgres' internal ABI it is not possible for it to guarantee that a compiled pgrx extension will probably execute within such a Postgres fork. You, dear compiler runner, can make this guarantee for yourself by specifying the unsafe-postgres feature flag. Otherwise, a pgrx extension will fail to compile with an error similar to:

error[E0080]: evaluation of constant value failed
   --> pgrx/src/lib.rs:151:5
    |
151 | /     assert!(
152 | |         same_slice(pg_sys::FMGR_ABI_EXTRA, b"xPostgreSQL\0"),
153 | |         "Unsupported Postgres ABI. Perhaps you need `--features unsafe-postgres`?",
154 | |     );
    | |_____^ the evaluated program panicked at 'Unsupported Postgres ABI. Perhaps you need `--features unsafe-postgres`?', pgrx/src/lib.rs:151:5
    |

Contributing

We are most definitely open to contributions of any kind. Bug Reports, Feature Requests, Documentation, and even sponsorships.

If you'd like to contribute code via a Pull Request, please make it against our develop branch. The master branch is no longer used.

Providing wrappers for Postgres' internals is not a straightforward task, and completely wrapping it is going to take quite a bit of time. pgrx is generally ready for use now, and it will continue to be developed as time goes on. Your feedback about what you'd like to be able to do with pgrx is greatly appreciated.

Hacking

If you're hacking on pgrx and want to ensure your test will run correctly, you need to have the current implementation of cargo-pgrx (from the revision you're working on) in your PATH.

An easy way would be to install cargo-local-install:

cargo install cargo-local-install

and then run cargo local-install to install cargo-pgrx as specified in top-level's Cargo.toml.

Don't forget to prepend /path/to/pgrx/bin to your PATH!

This approach can also be used in extensions to ensure a matching version of cargo-pgrx is used.

License

Portions Copyright 2019-2021 ZomboDB, LLC.  
Portions Copyright 2021-2023 Technology Concepts & Design, Inc.
Portions Copyright 2023 PgCentral Foundation, Inc.

All rights reserved.
Use of this source code is governed by the MIT license that can be found in the LICENSE file.

ucb-bar/shuttle

2023-10-10T01:39:39Z

A Rocket-based RISC-V superscalar in-order core

Shuttle: A Rocket-based Superscalar In-order RISC-V Core

Shuttle is a Rocket-based superscalar in-order RISC-V core, supporting the base RV64IMAFDC instruction set with supervisor and user-mode. Shuttle is a 6-stage core that can be configured to be dual, three, or quad-issue, although dual-issue is the most sensible design point. Shuttle is not designed to meet any power, performance, or area targets. It exists purely as a demonstrative example of another RISC-V CPU design point.

The superscalar microarchitecture presents the most advantages for 1) floating-point kernels and 2) RoCC accelerator kernels, as scalar control code can execute concurrently with floating point or RoCC instructions, maintaining high utilization of those units.

Shuttle is tape-out proven, and has similar physical design complexity as Rocket.

ucb-bar/bar-fetchers

2023-10-10T01:39:39Z

Regular crossword resolver

Resolver for the following regular crossword: http://www.mit.edu/~puzzle/2013/coinheist.com/rubik/a_regular_crossword/grid.pdf

Install

git clone git@github.com:jcdelmas/regular-crossword-resolver.git

Run

sbt run

H4K3R13/The-Hello-World-Project

2023-10-11T01:39:28Z

This repository hosts programs that output 'Hello World' in various programming languages. 🌍✨

prayag17/JellySkin

2023-10-10T01:39:53Z

Vibrante/minimal Jellyfin CSS using custom Icons and more!!, created for Jellyfin web 🎞️, can be used by just one line.

Lego6245/hundoLeaderboard

2023-10-11T01:39:48Z

A Leaderboard for the recipes at home project

ChrisTitusTech/ArchTitus

2023-10-10T01:39:59Z

朱雀仿宋/朱雀宋朝/Zhuque Fangsong: An open-source Fansong typeface project

朱雀仿宋

English | 繁體中文 | 日本語

「朱雀仿宋」是璇玑造字的开源仿宋字体计划，志在最终提供高质量的、支持多语言的正文仿宋解决方案。

仿宋作为「宋黑仿楷」中文四大基础字体之一，长期缺乏开源许可证下的选择。我们希望「朱雀仿宋」可以弥补这一空缺，为开源软件与 TeX 社区贡献一份力。

上世纪初，仿宋活字一度品类繁盛，开发、应用风行一时。然而时过境迁，市面上的仿宋最终收敛于「华丰真宋」一系，这也是今天大多数人所熟悉的仿宋样貌。「朱雀仿宋」则以民国仿宋活字「南宋」为蓝本，在满足现代排印需求的前提下，尝试为正文仿宋字库带来一缕新鲜空气。

当前我们发布的「朱雀仿宋」仍为「预览测试版」，尚处于项目早期阶段；无论是设计风格、字数，还是字形、排版质量，都不代表正式版情况。因此，

「朱雀」的未来，由你定义

如果您有任何问题、意见建议、合作意向，我们也欢迎您加入「朱雀仿宋反馈」微信群；或移步 Issues 提出。

璇玑正在尝试一种全新中文字库设计模式。得益于我们开发的最新技术，璇玑的设计师可以直接将人工智能作为设计工具去使用，我们也因此能以迭代的方式开发「朱雀」。

即便有技术加持，我们宏伟的目标也让「朱雀」项目有着难以想象的工作量。为了让项目推进更快、更可持续，我们也希望能获得更多的支持。

下载

版本与下载链接将会更新在 Releases 发布页。

设计故事

开发一套开源仿宋字库的想法最早始于 2021 年初，是继「未来荧黑」项目之后，我们认为更有意义、也更具难度的挑战。随着设计研究的深入与技术进步，特别是去年以来 AIGC 技术的爆发，实现这一想法在今日遂成可能。

「朱雀仿宋」改刻自民国活字「南宋」，试图为这套沉睡数十年的字体赋予新生。「南宋」于 1932 年由「百宋铸字印刷局」推出，韩佑之设计原稿，邹根培雕刻字模。一经面世，即广泛用于书籍、期刊排印。「南宋」与大多数正文字体的设定不同，重心略靠右下；结构中宫收紧、撇捺舒展，笔画的粗细变化强烈；含蓄秀美而充满力量感，在排印标题和正文时展现出浓厚的「书卷气」。

在「南宋」活字的基础上，「朱雀仿宋」稍稍削弱了笔画的对比度，适当地规整了字形结构与版面效果，同时最大程度地保留了「南宋」的独特气质与活字趣味。

「朱雀」是中国传统文化中「四象」之一，南方七宿之神。「朱雀」之名，与「南宋」渊源以及「璇玑造字」的品牌名相合。

参考文献

孙明远：《中国近现代平面设计和文字设计发展历程研究——从一八〇五年至一九四九年》，厦门大学出版社，2021 年第 1 版，第 168～172 页
周博：《中国现代文字设计图史》，北京大学出版社，2018 年第 1 版，第 122～124 页

许可证

您可以：

免费商用本字体，无论是用于印刷、LOGO、软件、网页还是视频。
复制与再分发本字体。

您不可以：

单独销售字体的原始版本或是修改版本。

授权方式以 SIL Open Font License 1.1 的英文原文为准。

我们恳请您暂时不要在「朱雀仿宋」正式版发布之前大量传播非「朱雀」团队所修改的本字体。如果您有改进「朱雀仿宋」意愿，我们非常欢迎您提出宝贵的意见建议——包括与我们讨论如何参与到朱雀的设计工作。

朱雀的西文与符号暂采用 Alegreya 的一个外插值实例。Alegreya 由 Huerta Tipográfica 的 Juan Pablo del Peral 创作，同样以 SIL Open Font License 1.1 发布。

「朱雀」团队

主创：余俊益

假名与符号（制作中）：韩泳思

技术：殷叶航、余俊益

关于璇玑造字

「璇玑造字」是智琮科技的字库品牌。我们以传统文化为立足基本，从自然风物汲取设计灵感，藉前沿科技更新文字风景。无论技术如何变迁，我们将始终把「人」置于第一位，关心设计者的创意与读者的体验。

欢迎你来和璇玑造字一起塑造属于未来的文字！

致谢

特别鸣谢：特里王，在「南宋」活字收集、整理、研究上的贡献。
Siphercase 补充了部分国际音标字形。
落霞孤鹜的「霞鹜文楷」、Fontworks Klee One、思源宋体，这些开源字体是本项目人与机在设计过程中的写法参考。

r1vs3c/auto-bspwm

2023-10-10T01:39:59Z

Bash script that automates the setup of a professional hacking environment for Kali Linux using the tiled window manager bspwm.

auto-bspwm

This is a Bash script that automates the setup of a professional hacking environment for Kali Linux using the tiled window manager bspwm.

Installation

Install available updates.

sudo apt update
sudo apt upgrade -y

Clone the repository and navigate to it.

git clone https://github.com/r1vs3c/auto-bspwm.git
cd auto-bspwm

Grant execution permissions to the script.

chmod +x setup.sh

Execute the script.

./setup.sh

After the script has finished, you will be prompted to restart the system. Once you have rebooted, select bspwm as the window manager and then log in.

Overview of the environment

Keyboard shortcuts

Windows + Enter: Open a terminal emulator window (kitty).
Windows + W: Close the current window.
Windows + Alt + R: Restart the bspwm configuration.
Windows + Alt + Q: Log out.
Windows + (⬆⬅⬇➡): Navigate through windows in the current workspace.
Windows + D: Open Rofi. Press Esc to exit.
Windows + (1,2,3,4,5,6,7,8,9,0): Switch to the respective workspace.
Windows + T: Change the current window to tile mode.
Windows + M: Toggle the current window to "full" mode (doesn't occupy the polybar). Press the same keys to return to tile mode.
Windows + F: Change the current window to fullscreen mode (occupies the entire screen, including the polybar).
Windows + S: Change the current window to floating mode.
Windows + Shift + (1,2,3,4,5,6,7,8,9,0): Move the current window to another workspace.
Windows + Alt + (⬆⬅⬇➡): Resize the current window (only works if it's in floating mode).
Windows + Ctrl + (⬆⬅⬆➡): Change the position of the current window (only works in floating mode).
Windows + Shift + F: Open Firefox.
Windows + Shift + B: Open Burpsuite.
Ctrl + Alt + L: Lock the screen.
Ctrl + Shift + ⬆⬇: Increase/decrease volume.
Ctrl + Shift + M: Mute/unmute volume.
Windows + Ctrl + Alt + (⬆⬅⬇➡): Show a preselection and then open a window (kitty, Firefox, File manager, etc.).
- Windows + Ctrl + Alt + Space: Undo the preselection.
Ctrl + Shift + Enter: Open a sub-window in the current window.
Ctrl + Shift + T: Open a tab in the current window.
Ctrl + Shift + Z: Zoom in on the current sub-window.
Ctrl + (⬆⬅⬇➡): Navigate through sub-windows in the current window.
Ctrl + Shift + W: Close the current sub-window or tab.
Ctrl + Shift + R: Resize the current sub-window. Afterward, use:
- W for 'Wider'
- N for 'Narrower'
- T for 'Taller'
- S for 'Shorter'
- R for 'Reset'
- Esc to quit resize mode.
Ctrl + Shift + L: Toggle the arrangement of sub-windows.
Ctrl + Shift + C: Copy to the clipboard.
Ctrl + Shift + V: Paste from the clipboard.
F1: Copy to buffer A.
F2: Paste from buffer A.
F3: Copy to buffer B.
F4: Paste from buffer B.

Software

This configuration uses the following software:

WM: bspwm
Hotkey: sxhkd
Locker: i3lock-fancy
Shell: zsh
Bars: polybar
Compositor: picom
File Manager: thunar
Fonts: iosevka and hack
Application Launcher: rofi
Browsers: firefox
Terminals: kitty and qterminal
Static Wallpaper: feh
Color Scheme: pywal
Screenshot: flameshot

Credits

This environment has been inspired by the functionalities of S4vitar's environment.

oneinstack/oneinstack

2023-10-10T01:39:59Z

s6 overlay for containers (includes execline, s6-linux-utils & a custom init)

Table of Contents

s6-overlay

s6-overlay is an easy-to-install (just extract a tarball or two!) set of scripts and utilities allowing you to use existing Docker images while using s6 as a pid 1 for your container and process supervisor for your services.

Quickstart

Build the following Dockerfile and try it out:

# Use your favorite image
FROM ubuntu
ARG S6_OVERLAY_VERSION=3.1.5.0

RUN apt-get update && apt-get install -y nginx xz-utils
RUN echo "daemon off;" >> /etc/nginx/nginx.conf
CMD ["/usr/sbin/nginx"]

ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz
ENTRYPOINT ["/init"]

docker-host $ docker build -t demo .
docker-host $ docker run --name s6demo -d -p 80:80 demo
docker-host $ docker top s6demo acxf
PID                 TTY                 STAT                TIME                COMMAND
11735               ?                   Ss                  0:00                \_ s6-svscan
11772               ?                   S                   0:00                \_ s6-supervise
11773               ?                   Ss                  0:00                | \_ s6-linux-init-s
11771               ?                   Ss                  0:00                \_ rc.init
11812               ?                   S                   0:00                | \_ nginx
11814               ?                   S                   0:00                | \_ nginx
11816               ?                   S                   0:00                | \_ nginx
11813               ?                   S                   0:00                | \_ nginx
11815               ?                   S                   0:00                | \_ nginx
11779               ?                   S                   0:00                \_ s6-supervise
11785               ?                   Ss                  0:00                | \_ s6-ipcserverd
11778               ?                   S                   0:00                \_ s6-supervise
docker-host $ curl --head http://127.0.0.1/
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Mon, 17 Jan 2022 13:33:58 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Mon, 17 Jan 2022 13:32:11 GMT
Connection: keep-alive
ETag: "61e56fdb-264"
Accept-Ranges: bytes

Compatibility with v2

If you're migrating from a previous version of s6-overlay (v2) to the new version (v3), you may need to make some changes to your services or the way you use s6-overlay in order for everything to work smoothly. This document tries to be accurate on how v3 works, but we have a separate page listing the main differences, and things you're likely to notice. Please read it if you're in this situation!

Goals

The project has the following goals:

Be usable on top of any Docker image
Make it easy to create new images, that will operate like any other images
Provide users with a turnkey s6 installation that will give them a stable pid 1, a fast and orderly init sequence and shutdown sequence, and the power of process supervision and automatically rotated logs.

Features

A simple init process which allows the end-user to execute tasks like initialization (cont-init.d), finalization (cont-finish.d) and their own services with dependencies between them
The s6-overlay provides proper PID 1 functionality
- You'll never have zombie processes hanging around in your container, they will be properly cleaned up.
Multiple processes in a single container
Able to operate in "The Docker Way"
Usable with all base images - Ubuntu, CentOS, Fedora, Alpine, Busybox...
Distributed as a small number of .tar.xz files depending on what exact functionality you need - to keep your image's number of layers small.
A whole set of utilities included in s6 and s6-portable-utils. They include handy and composable utilities which make our lives much, much easier.
Log rotating out-of-the-box through logutil-service which uses s6-log under the hood.
Some support for Docker's USER directive, to run your whole process tree as a specific user. Not compatible with all features, details in the notes section.

The Docker Way?

One of the oft-repeated Docker mantras is "one process per container", but we disagree. There's nothing inherently bad about running multiple processes in a container. The more abstract "one thing per container" is our policy - a container should do one thing, such as "run a chat service" or "run gitlab." This may involve multiple processes, which is fine.

The other reason image authors shy away from process supervisors is they believe a process supervisor must restart failed services, meaning the Docker container will never die.

This does effectively break the Docker ecosystem - most images run one process that will exit when there's an error. By exiting on error, you allow the system administrator to handle failures however they prefer. If your image will never exit, you now need some alternative method of error recovery and failure notification.

Our policy is that if "the thing" fails, then the container should fail, too. We do this by determining which processes can restart, and which should bring down the container. For example, if cron or syslog fails, your container can most likely restart it without any ill effects, but if ejabberd fails, the container should exit so the system administrator can take action.

Our interpretation of "The Docker Way" is thus:

Containers should do one thing
Containers should stop when that thing stops

and our init system is designed to do exactly that. Your images will behave like other Docker images and fit in with the existing ecosystem of images.

See "Writing an optional finish script" under the Usage section for details on stopping "the thing."

Init stages

Our overlay init is a properly customized one to run appropriately in containerized environments. This section briefly explains how stages work but if you want to know how a complete init system should work, you can read this article: How to run s6-svscan as process 1

stage 1: Its purpose is to set up the image to execute the supervision tree which will handle all the auxiliary services, and to launch stage 2. Stage 1 is where all the black magic happens, all the container setup details that we handle for you so that you don't have to care about them.
stage 2: This is where most of the end-user provided files are meant to be executed:
1. Execute legacy oneshot user scripts contained in /etc/cont-init.d.
2. Run user s6-rc services declared in /etc/s6-overlay/s6-rc.d, following dependencies
3. Copy legacy longrun user services (/etc/services.d) to a temporary directory and have s6 start (and supervise) them.
stage 3: This is the shutdown stage. When the container is supposed to exit, it will:
1. Send a TERM signal to all legacy longrun services and, if required, wait for them to exit.
2. Bring down user s6-rc services in an orderly fashion.
3. Run any finalization scripts contained in /etc/cont-finish.d.
4. Send all remaining processes a TERM signal. There should not be any remaining processes anyway.
5. Sleep for a small grace time, to allow stray processes to exit cleanly.
6. Send all processes a KILL signal. Then the container exits.

Installation

s6-overlay comes as a set of tarballs that you can extract onto your image. The tarballs you need are a function of the image you use; most people will need the first two, and the other ones are extras you can use at your convenience.

s6-overlay-noarch.tar.xz: this tarball contains the scripts implementing the overlay. We call it "noarch" because it is architecture- independent: it only contains scripts and other text files. Everyone who wants to run s6-overlay needs to extract this tarball.
s6-overlay-x86_64.tar.xz: replace x86_64 with your system's architecture. This tarball contains all the necessary binaries from the s6 ecosystem, all linked statically and out of the way of your image's binaries. Unless you know for sure that your image already comes with all the packages providing the binaries used in the overlay, you need to extract this tarball.
s6-overlay-symlinks-noarch.tar.xz: this tarball contains symlinks to the s6-overlay scripts so they are accessible via /usr/bin. It is normally not needed, all the scripts are accessible via the PATH environment variable, but if you have old user scripts containing shebangs such as #!/usr/bin/with-contenv, installing these symlinks will make them work.
s6-overlay-symlinks-arch.tar.xz: this tarball contains symlinks to the binaries from the s6 ecosystem provided by the second tarball, to make them accessible via /usr/bin. It is normally not needed, but if you have old user scripts containing shebangs such as #!/usr/bin/execlineb, installing these symlinks will make them work.
syslogd-overlay-noarch.tar.xz: this tarball contains definitions for a syslogd service. If you are running daemons that cannot log to stderr to take advantage of the s6 logging infrastructure, but hardcode the use of the old syslog() mechanism, you can extract this tarball, and your container will run a lightweight emulation of a syslogd daemon, so your syslog logs will be caught and stored to disk.

To install those tarballs, add lines to your Dockerfile that correspond to the functionality you want to install. For instance, most people would use the following:

ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz

Make sure to preserve file permissions when extracting (i.e. to use the -p option to tar.)

Usage

The project is distributed as a set of standard .tar.xz files, which you extract at the root of your image. (You need the xz-utils package for tar to understand .tar.xz files; it is available in every distribution, but not always in the default container images, so you may need to apt install xz-utils or apk add xz-utils, or equivalent, before you can expand the archives.)

Afterwards, set your ENTRYPOINT to /init.

Right now, we recommend using Docker's ADD directive instead of running wget or curl in a RUN directive - Docker is able to handle the https URL when you use ADD, whereas your base image might not be able to use https, or might not even have wget or curl installed at all.

From there, you have a couple of options:

If you want the container to exit when your program exits: run the program as your image's CMD.
If you want the container to run until told to exit, and your program to be supervised by s6: write a service script for your program.

Using `CMD`

Using CMD is a convenient way to take advantage of the overlay. Your CMD can be given at build time in the Dockerfile, or at run time on the command line, either way is fine. It will be run as a normal process in the environment set up by s6; when it fails or exits, the container will shut down cleanly and exit. You can run interactive programs in this manner: only the CMD will receive your interactive command, the support processes will be unimpacted.

For example:

FROM busybox
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz
ENTRYPOINT ["/init"]

docker-host $ docker build -t s6demo .
docker-host $ docker run -ti s6demo /bin/sh
/package/admin/s6-overlay/libexec/preinit: notice: /var/run is not a symlink to /run, fixing it
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 /package/admin/s6/command/s6-svscan -d4 -- /run/service
   17 root      0:00 {rc.init} /bin/sh -e /run/s6/basedir/scripts/rc.init top /bin/sh
   18 root      0:00 s6-supervise s6-linux-init-shutdownd
   20 root      0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B
   24 root      0:00 s6-supervise s6rc-fdholder
   25 root      0:00 s6-supervise s6rc-oneshot-runner
   31 root      0:00 /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /packa
   58 root      0:00 /bin/sh
   66 root      0:00 ps
/ # exit
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
docker-host $

Writing a service script

The other way to use a container with s6-overlay is to make your services supervised. You can supervise any number of services; usually they're just support services for the main daemon you run as a CMD, but if that's what you want, nothing prevents you from having an empty CMD and running your main daemon as a supervised service as well. In that case, the daemon will be restarted by s6 whenever it exits; the container will only stop when you tell it to do so, either via a docker stop command, or from inside the container with the /run/s6/basedir/bin/halt command.

There are two ways of making a supervised service. The old way, which is still supported, is to make a "pure s6" service directory. Create a directory with the name of your service in /etc/services.d and put an executable run file into it; this is the file in which you'll put your long-lived process execution. For details of supervision of service directories, and how you can configure how s6 handles your daemon, you can take a look at the servicedir documentation. A simple example would look like this:

/etc/services.d/myapp/run:

#!/command/execlineb -P
nginx -g "daemon off;"

The new way is to make an s6-rc source definition directory in the /etc/s6-overlay/s6-rc.d directory, and add the name of that directory to the user bundle, i.e. create an empty file with the same name in the /etc/s6-overlay/s6-rc.d/user/contents.d directory. The format of a source definition directory is described in this page. Note that you can define longruns, i.e. daemons that will get supervised by s6 just like with the /etc/services.d method, but also oneshots, i.e. programs that will run once and exit. Your main service is probably a longrun, not a oneshot: you probably need a daemon to stick around.

The advantage of this new format is that it allows you to define dependencies between services: if B depends on A, then A will start first, then B will start when A is ready, and when the container is told to exit, B will stop first, then A. If you have a complex architecture where various processes depends on one another, or simply where you have to mix oneshots and longruns in a precise order, this may be for you.

The example above could be rewritten this way:

/etc/s6-overlay/s6-rc.d/myapp/type:

longrun

/etc/s6-overlay/s6-rc.d/myapp/run:

#!/command/execlineb -P
nginx -g "daemon off;"

/etc/s6-overlay/s6-rc.d/user/contents.d/myapp: empty file. (This adds myapp to the set of services that s6-rc will start at container boot.)

/etc/s6-overlay/s6-rc.d/myapp/dependencies.d/base: empty file. (This tells s6-rc to only start myapp when all the base services are ready: it prevents race conditions.)

We encourage you to switch to the new format, but if you don't need its benefits, you can stick with regular service directories in /etc/services.d, it will work just as well.

Setting the exit code of the container to the exit code of your main service

If you run your main service as a CMD, you have nothing to do: when your CMD exits, or when you run docker stop, the container will naturally exit with the same exit code as your service. (Be aware, however, that in the docker stop case, your service will get a SIGTERM, in which case the exit code will entirely depend on how your service handles it - it could trap it and exit 0, trap it and exit something else, or not trap it and let the shell exit its own code for it - normally 130.)

If you run your main service as a supervised service, however, things are different, and you need to tell the container what code to exit with when you send it a docker stop command. To do that, you need to write a finish script:

If your service is a legacy service in /etc/services.d, you need an executable /etc/services.d/myapp/finish script.
If your service is an s6-rc one, you need a /etc/s6-overlay/s6-rc.d/myapp/finish file containing your script (the file may or may not be executable).

This finish script will be run when your service exits, and will take two arguments:

The first argument will be the exit code of your service, or 256 if your service was killed by an uncaught signal.
The second argument is only meaningful if your service was killed by an uncaught signal, and contains the number of said signal.

In the finish script, you need to write the container exit code you want to the /run/s6-linux-init-container-results/exitcode file - and that's it.

For instance, the finish script for the myapp service above could be something like this:

#!/bin/sh

if test "$1" -eq 256 ; then
  e=$((128 + $2))
else
  e="$1"
fi

echo "$e" > /run/s6-linux-init-container-results/exitcode

When you send a docker stop command to your container, the myapp service will be killed and this script will be run; it will write either myapp's exit code (if myapp catches the TERM signal) or 130 (if myapp does not catch the TERM signal) to the special /run/s6-linux-init-container-results/exitcode file, which will be read by s6-overlay at the end of the container shutdown procedure, and your container will exit with that value.

Fixing ownership and permissions

This section describes a functionality from the versions of s6-overlay that are anterior to v3. fix-attrs is still supported in v3, but is deprecated, for several reasons: one of them is that it's generally not good policy to change ownership dynamically when it can be done statically. Another reason is that it doesn't work with USER containers. Instead of fix-attrs, we now recommend you to take care of ownership and permissions on host mounts offline, before running the container. This should be done in your Dockerfile, when you have all the needed information.

That said, here is what we wrote for previous versions and that is still applicable today (but please stop depending on it):

Sometimes it's interesting to fix ownership & permissions before proceeding because, for example, you have mounted/mapped a host folder inside your container. Our overlay provides a way to tackle this issue using files in /etc/fix-attrs.d. This is the pattern format followed by fix-attrs files:

path recurse account fmode dmode

path: File or dir path.
recurse: (Set to true or false) If a folder is found, recurse through all containing files & folders in it.
account: Target account. It's possible to default to fallback uid:gid if the account isn't found. For example, nobody,32768:32768 would try to use the nobody account first, then fallback to uid 32768 instead. If, for instance, daemon account is UID=2 and GID=2, these are the possible values for account field:
- daemon: UID=2 GID=2
- daemon,3:4: UID=2 GID=2
- 2:2,3:4: UID=2 GID=2
- daemon:11111,3:4: UID=2 GID=11111
- 11111:daemon,3:4: UID=11111 GID=2
- daemon:daemon,3:4: UID=2 GID=2
- daemon:unexisting,3:4: UID=2 GID=4
- unexisting:daemon,3:4: UID=3 GID=2
- 11111:11111,3:4: UID=11111 GID=11111
fmode: Target file mode. For example, 0644.
dmode: Target dir/folder mode. For example, 0755.

Here you have some working examples:

/etc/fix-attrs.d/01-mysql-data-dir:

/var/lib/mysql true mysql 0600 0700

/etc/fix-attrs.d/02-mysql-log-dirs:

/var/log/mysql-error-logs true nobody,32768:32768 0644 2700
/var/log/mysql-general-logs true nobody,32768:32768 0644 2700
/var/log/mysql-slow-query-logs true nobody,32768:32768 0644 2700

Executing initialization and finalization tasks

Here is the old way of doing it:

After fixing attributes (through /etc/fix-attrs.d/) and before starting user provided services (through s6-rc or /etc/services.d) our overlay will execute all the scripts found in /etc/cont-init.d, for example:

/etc/cont-init.d/02-confd-onetime:

#!/command/execlineb -P

with-contenv
s6-envuidgid nginx
multisubstitute
{
  import -u -D0 UID
  import -u -D0 GID
  import -u CONFD_PREFIX
  define CONFD_CHECK_CMD "/usr/sbin/nginx -t -c {{ .src }}"
}
confd --onetime --prefix="${CONFD_PREFIX}" --tmpl-uid="${UID}" --tmpl-gid="${GID}" --tmpl-src="/etc/nginx/nginx.conf.tmpl" --tmpl-dest="/etc/nginx/nginx.conf" --tmpl-check-cmd="${CONFD_CHECK_CMD}" etcd

This way is still supported. However, there is now a more generic and efficient way to do it: writing your oneshot initialization and finalization tasks as s6-rc services, by adding service definition directories in /etc/s6-overlay/s6-rc.d, making them part of the user bundle (so they are actually started when the container boots), and making them depend on the base bundle (so they are only started after base).

All the information on s6-rc can be found here.

When the container is started, the operations are performed in this order:

(deprecated) Attribute fixing is performed according to files in /etc/fix-attrs.d.
(legacy) One-shot initialization scripts in /etc/cont-init.d are run sequentially.
Services in the user bundle are started by s6-rc, in an order defined by dependencies. Services can be oneshots (initialization tasks) or longruns (daemons that will run throughout the container's lifetime). If the services depend on base, they are guaranteed to start at this point and not earlier; if they do not, they might have been started earlier, which may cause race conditions - so it's recommended to always make them depend on base.
(legacy) Longrun services in /etc/services.d are started.
Services in the user2 bundle with the correct dependency are started. (Most people don't need to use this; if you are not sure, stick to the user bundle.)

When the container is stopped, either because the admin sent a stop command or because the CMD exited, the operations are performed in the reverse order:

Services in the user2 bundle with the correct dependency are stopped.
(legacy) Longrun services in /etc/services.d are stopped.
All s6-rc services are stopped, in an order defined by dependencies. For oneshots, that means that the down script in the source definition directory is executed; that's how s6-rc can perform finalization tasks.
(legacy) One shot finalization scripts in /etc/cont-finish.d are run sequentially.

The point of the user2 bundle is to allow user services declared in it to start after the /etc/services.d ones; but in order to do so, every service in user2 needs to declare a dependency to legacy-services. In other words, for a service foobar to start late, you need to:

Define it in /etc/s6-overlay/s6-rc.d/foobar like any other s6-rc service.
Add an /etc/s6-overlay/s6-rc.d/foobar/dependencies.d/legacy-services file
Add an /etc/s6-overlay/s6-rc.d/user2/contents.d/foobar file.

That will ensure that foobar will start after everything in /etc/services.d.

Writing an optional finish script

By default, services created in /etc/services.d will automatically restart. If a service should bring the container down, you should probably run it as a CMD instead; but if you'd rather run it as a supervised service, then you'll need to write a finish script, which will be run when the service is down; to make the container stop, the /run/s6/basedir/bin/halt command must be invoked. Here's an example finish script:

/etc/services.d/myapp/finish:

#!/command/execlineb -S0

foreground { redirfd -w 1 /run/s6-linux-init-container-results/exitcode echo 0 }
/run/s6/basedir/bin/halt

The first line of the script writes 0 to the /run/s6-linux-init-container-results/exitcode file. The second line stops the container. When you stop the container via the /run/s6/basedir/bin/halt command run from inside the container, /run/s6-linux-init-container-results/exitcode is read and its contents are used as the exit code for the docker run command that launched the container. If the file doesn't exist, or if the container is stopped with docker stop or another reason, that exit code defaults to 0.

It is possible to do more advanced operations in a finish script. For example, here's a script from that only brings down the service when it exits nonzero:

/etc/services.d/myapp/finish:

#!/command/execlineb -S1
if { eltest ${1} -ne 0 -a ${1} -ne 256 }
/run/s6/basedir/bin/halt

Note that in general, finish scripts should only be used for local cleanups after a daemon dies. If a service is so important that the container needs to stop when it dies, we really recommend running it as the CMD.

Logging

Every service can have its dedicated logger. A logger is a s6 service that automatically reads from the stdout of your service, and logs the data to an automatically rotated file in the place you want. Note that daemons usually log to stderr, not stdout, so you should probably start your service's run script with exec 2>&1 in shell, or with fdmove -c 2 1 in execline, in order to catch stderr.

s6-overlay provides a utility called logutil-service which is a wrapper over the s6-log program. This helper does the following:

read how s6-log should proceed reading the logging script contained in S6_LOGGING_SCRIPT
drop privileges to the nobody user (defaulting to 65534:65534 if it doesn't exist)
clean all the environments variables
execute into s6-log.

s6-log will then run forever, reading data from your service and writing it to the directory you specified to logutil-service.

Please note:

Since the privileges are dropped automatically, there is no need to switch users with s6-setuidgid
You should ensure the log folder either:
- exists, and is writable by the nobody user
- does not exist, but the parent folder is writable by the nobody user.

You can create log folders in cont-init.d scripts, or as s6-rc oneshots. Here is an example of a logged service myapp implemented the old way:

/etc/cont-init.d/myapp-log-prepare:

#!/bin/sh -e
mkdir -p /var/log/myapp
chown nobody:nogroup /var/log/myapp
chmod 02755 /var/log/myapp

/etc/services.d/myapp/run:

#!/bin/sh
exec 2>&1
exec mydaemon-in-the-foreground-and-logging-to-stderr

/etc/services.d/myapp/log/run:

#!/bin/sh
exec logutil-service /var/log/myapp

And here is the same service, myapp, implemented in s6-rc.

/etc/s6-overlay/s6-rc.d/myapp-log-prepare/dependencies.d/base: empty file

/etc/s6-overlay/s6-rc.d/myapp-log-prepare/type:

oneshot

/etc/s6-overlay/s6-rc.d/myapp-log-prepare/up:

if { mkdir -p /var/log/myapp }
if { chown nobody:nogroup /var/log/myapp }
chmod 02755 /var/log/myapp

(Click here for an explanation of the weird syntax or if you don't understand why your `up` file isn't working.)

(Beginning of the detailed section.)

So, the up and down files are special: they're not shell scripts, but single command lines interpreted by execlineb. You should not have to worry about execline; you should only remember that an up file contains a single command line. So if you need a script with several instructions, here's how to do it:

Write your script in the language of your choice, in a location of your choice
Make it executable
Call that script in the up file.

Here is how you would normally proceed to write the up file for myapp-log-prepare:

/etc/s6-overlay/s6-rc.d/myapp-log-prepare/up:

/etc/s6-overlay/scripts/myapp-log-prepare

/etc/s6-overlay/scripts/myapp-log-prepare: (needs to be executable)

#!/bin/sh -e
mkdir -p /var/log/myapp
chown nobody:nogroup /var/log/myapp
chmod 02755 /var/log/myapp

The location of the actual script is arbitrary, it just needs to match what you're writing in the up file.

But here, it just so happens that the script is simple enough that it can fit entirely in the up file without making it too complex or too difficult to understand. So, we chose to include it as an example to show that there's more that you can do with up files, if you are so inclined. You can read the full documentation for the execline language here.

(End of the detailed section, click the triangle above again to collapse.)

/etc/s6-overlay/s6-rc.d/myapp/dependencies.d/base: empty file

/etc/s6-overlay/s6-rc.d/myapp-log/dependencies.d/myapp-log-prepare: empty file

/etc/s6-overlay/s6-rc.d/myapp/type:

longrun

/etc/s6-overlay/s6-rc.d/myapp/run:

#!/bin/sh
exec 2>&1
exec mydaemon-in-the-foreground-and-logging-to-stderr

/etc/s6-overlay/s6-rc.d/myapp/producer-for:

myapp-log

/etc/s6-overlay/s6-rc.d/myapp-log/type:

longrun

/etc/s6-overlay/s6-rc.d/myapp-log/run:

#!/bin/sh
exec logutil-service /var/log/myapp

/etc/s6-overlay/s6-rc.d/myapp-log/consumer-for:

myapp

/etc/s6-overlay/s6-rc.d/myapp-log/pipeline-name:

myapp-pipeline

/etc/s6-overlay/s6-rc.d/user/contents.d/myapp-pipeline: empty file

That's a lot of files! A summary of what it all means is:

myapp-log-prepare is a oneshot, preparing the logging directory. It is a dependency of myapp-log, so it will be started before myapp-log.
myapp is a producer for myapp-log and myapp-log is a consumer for myapp, so what myapp writes to its stdout will go to myapp-log's stdin. Both are longruns, i.e. daemons that will be supervised by s6.
The myapp | myapp-log pipeline is given a name, myapp-pipeline, and this name is declared as a part of the user bundle, so it will be started when the container starts.
myapp-log-prepare, myapp-log and myapp all depend on the base bundle, which means they will only be started when the system is actually ready to start them.

It really accomplishes the same things as the /etc/cont-init.d plus /etc/services.d method, but it's a lot cleaner underneath, and can handle much more complex dependency graphs, so whenever you get the opportunity, we recommend you familiarize yourself with the s6-rc way of declaring your services and your loggers. The full syntax of a service definition directory, including declaring whether your service is a longrun or a oneshot, declaring pipelines, adding service-specific timeouts if you need them, etc., can be found here.

Dropping privileges

When it comes to executing a service, no matter whether it's a service or a logger, a good practice is to drop privileges before executing it. s6 already includes utilities to do exactly these kind of things:

In execline:

#!/command/execlineb -P
s6-setuidgid daemon
myservice

In sh:

#!/bin/sh
exec s6-setuidgid daemon myservice

If you want to know more about these utilities, please take a look at: s6-setuidgid, s6-envuidgid, and s6-applyuidgid.

Container environment

If you want your custom script to have container environments available: you can use the with-contenv helper, which will push all of those into your execution environment, for example:

/etc/cont-init.d/01-contenv-example:

#!/command/with-contenv sh
env

This script will output the contents of your container environment.

Read-Only Root Filesystem

Recent versions of Docker allow running containers with a read-only root filesystem. If your container is in such a case, you should set S6_READ_ONLY_ROOT=1 to inform s6-overlay that it should not attempt to write to certain areas - instead, it will perform copies into a tmpfs mounted on /run.

Note that s6-overlay assumes that:

/run exists and is writable. If it is not, it will attempt to mount a tmpfs there.
/var/run is a symbolic link to /run, for compatibility with previous versions. If it is not, it will make it so.

In general your default docker settings should already provide a suitable tmpfs in /run.

Customizing s6-overlay behaviour

It is possible somehow to tweak s6-overlay's behaviour by providing an already predefined set of environment variables to the execution context:

PATH (default = /command:/usr/bin:/bin): this is the default PATH that all the services in the container, including the CMD, will have. Set this variable if you have a lot of services that depend on binaries stored in another directory, e.g. /usr/sbin. Note that /command, /usr/bin and /bin will always be added to that path if they're not already in the one you provide.
S6_KEEP_ENV (default = 0): if set, then environment is not reset and whole supervision tree sees original set of env vars. It switches with-contenv into a nop.
S6_LOGGING (default = 0):
- 0: Outputs everything to stdout/stderr.
- 1: Uses an internal catch-all logger and persists everything on it, it is located in /var/log/s6-uncaught-logs. Anything run as a CMD is still output to stdout/stderr.
- 2: Uses an internal catch-all logger and persists everything on it, including the output of CMD. Absolutely nothing is written to stdout/stderr.
S6_CATCHALL_USER (default = root): if set, and if S6_LOGGING is 1 or 2, then the catch-all logger is run as this user, which must be defined in your image's /etc/passwd. Every bit of privilege separation helps a little with security.
S6_BEHAVIOUR_IF_STAGE2_FAILS (default = 0): determines what the container should do if one of the service scripts fails. This includes:
- if anything fails in fix-attrs
- if any old-style /etc/cont-init.d or new-style s6-rc oneshot fails
- if any old-style /etc/services.d or new-style s6-rc longrun is marked as expecting readiness notification, and fails to become ready in the allotted time (see S6_CMD_WAIT_FOR_SERVICES_MAXTIME below). The valid values for S6_BEHAVIOUR_IF_STAGE2_FAILS are the following:
- 0: Continue silently even if a script has failed.
- 1: Continue but warn with an annoying error message.
- 2: Stop the container.
S6_KILL_FINISH_MAXTIME (default = 5000): How long (in milliseconds) the system should wait, at shutdown time, for a script in /etc/cont-finish.d to finish naturally. After this duration, the script will be sent a SIGKILL. Bear in mind that scripts in /etc/cont.finish.d are run sequentially, and the shutdown sequence will potentially wait for S6_KILL_FINISH_MAXTIME milliseconds for each script.
S6_SERVICES_READYTIME (default = 50): With services declared in /etc/services.d, there is an unavoidable race condition between the moment when services are started and the moment when they can be tested for readiness. To avoid that race, we sleep a little time, by default 50 milliseconds, before testing for readiness. If your machine is slow or very busy, you may get errors looking like s6-svwait: fatal: unable to s6_svstatus_read: No such file or directory. In that case, you should increase the sleeping time, by declaring it (in milliseconds) in the S6_SERVICES_READYTIME variable. Note that it only concerns /etc/services.d; s6-rc is immune to the race condition.
S6_SERVICES_GRACETIME (default = 3000): How long (in milliseconds) s6 should wait, at shutdown time, for services declared in /etc/services.d to die before proceeding with the rest of the shutdown.
S6_KILL_GRACETIME (default = 3000): How long (in milliseconds) s6 should wait, at the end of the shutdown procedure when all the processes have received a TERM signal, for them to die before sending a KILL signal to make sure they're dead.
S6_LOGGING_SCRIPT (default = "n20 s1000000 T"): This env decides what to log and how, by default every line will prepend with ISO8601, rotated when the current logging file reaches 1mb and archived, at most, with 20 files.
S6_CMD_ARG0 (default = not set): Value of this env var will be prepended to any CMD args passed by docker. Use it if you are migrating an existing image to s6-overlay and want to make it a drop-in replacement: setting this variable to the value of a previously used ENTRYPOINT will help you transition.
S6_CMD_USE_TERMINAL (default = 0): Set this value to 1 if you have a CMD that needs a terminal for its output (typically when you're running your container with docker run -it), and you have set S6_LOGGING to a nonzero value. This setting will make your CMD actually output to your terminal; the drawback is that its output will not be logged. By default (when this variable is 0 or not set), the stdout and stderr of your CMD are logged when S6_LOGGING is nonzero, which means they go to a pipe even if you're running it in an interactive terminal.
S6_FIX_ATTRS_HIDDEN (default = 0): Controls how fix-attrs.d scripts process files and directories.
- 0: Hidden files and directories are excluded.
- 1: All files and directories are processed.
S6_CMD_WAIT_FOR_SERVICES (default = 0): By default when the container starts, services in /etc/services.d will be started and execution will proceed to starting the user2 bundle and the CMD, if any of these is defined. If S6_CMD_WAIT_FOR_SERVICES is nonzero, however, the container starting sequence will wait until the services in /etc/services.d are ready before proceeding with the rest of the sequence. Note that this is only significant if the services in /etc/services.d notify their readiness to s6.
S6_CMD_WAIT_FOR_SERVICES_MAXTIME (default = 5000): The maximum time (in milliseconds) the services could take to bring up before proceding to CMD executing. Note that this value also includes the time setting up legacy container initialization (/etc/cont-init.d) and services (/etc/services.d), and that it is taken into account even if you are not running a CMD. In other words: no matter whether you're running a CMD or not, if you have scripts in /etc/cont-init.d that take a long time to run, you should set this variable to either 0, or a value high enough so that your scripts have time to finish without s6-overlay interrupting them and diagnosing an error.
S6_READ_ONLY_ROOT (default = 0): When running in a container whose root filesystem is read-only, set this env to 1 to inform init stage 2 that it should copy user-provided initialization scripts from /etc to /run/s6/etc before it attempts to change permissions, etc. See Read-Only Root Filesystem for more information.
S6_SYNC_DISKS (default = 0): Set this env to 1 to inform init stage 3 that it should attempt to sync filesystems before stopping the container. Note: this will likely sync all filesystems on the host.
S6_STAGE2_HOOK (default = none): If this variable exists, its contents will be interpreted as a shell excerpt that will be run in the early stage 2, before services are started. This can be used, for instance, to dynamically patch the service database at run-time right before it is compiled and run. The wrong value can prevent your container from running or endanger your security, so only use this if you know exactly what you are doing. When in doubt, leave this variable undefined.
S6_VERBOSITY (default = 2): controls the verbosity of s6-rc, and potentially other tools, at container start and stop time. The default, 2, is normally verbose: it will list the service start and stop operations. You can make the container quieter by decreasing this number: 1 will only print warnings and errors, and 0 will only print errors. You can also make the container more verbose, i.e. print tracing and debug information, by increasing this number up to 5, but the output will quickly become very noisy, and most people shouldn't need this.
S6_CMD_RECEIVE_SIGNALS (default = 0): decides whether signals sent to the container should be sent to the container's pid 1 or to the CMD. By default, when you perform for instance a docker stop, a TERM signal will be sent to the container's pid 1, which will trigger the full container shutdown sequence - but if a CMD is present, it will be among the last processes to be killed, only when everything else is down and the container is about to exit. If this variable is 1 or more, signals are diverted from pid 1 to the CMD, which means that docker stop will send a SIGTERM to the CMD instead, and the container will only trigger its shutdown procedure when the CMD is dead. Note that only SIGTERM, SIGQUIT, SIGINT, SIGUSR1, SIGUSR2, SIGPWR and SIGWINCH are diverted; other signals either are ignored or cannot be diverted and are necessarily handled by pid 1. Please be aware that using this option may prevent interactive CMDs from working at all - in other words, if you're running an interactive CMD in a terminal, don't set this variable; but that should be fine since in this case you already have interactive ways of stopping your CMD.

syslog

If software running in your container requires syslog, extract the syslogd-overlay-noarch.tar.xz tarball: that will give you a small syslogd emulation. Logs will be found under various subdirectories of /var/log/syslogd, for instance messages will be found in the /var/log/syslogd/messages/ directory, the latest logs being available in the /var/log/syslogd/messages/current file. Logging directories are used rather than files so that logs can be automatically rotated without race conditions (that is a feature of s6-log).

It is recommended to add syslog and sysllog users to your image, for privilege separation; the syslogd emulation processes will run as these users if they exist. Otherwise they will default to 32760:32760 and 32761:32761, numeric uids/gids that may already exist on your system.

Performance

The noarch and symlinks tarballs are all tiny. The biggest tarball is the one that contains the binaries; it's around 650 kB.
Uncompressed on a tmpfs, the overlay scripts use about 120 kB, and the binaries for x86_64 use about 5.7 MB.
We haven't yet measured the time it takes for the container to be up and running once you run docker run, but you will notice it's fast. Faster than previous versions of s6-overlay, with fewer delays. And if you convert your /etc/cont-init.d scripts to the s6-rc format, they will be able to run in parallel, so you will gain even more performance. If you have benchmarks, please send them to us!

Verifying Downloads

The s6-overlay releases have a checksum files you can use to verify the download using SHA256:

ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz.sha256 /tmp
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz.sha256 /tmp
RUN cd /tmp && sha256sum -c *.sha256

Notes

`USER` directive

As of version 3.1.5.0, s6-overlay has limited support for running as a user other than root:

Tools like fix-attrs and logutil-service are unlikely to work (they rely on being able to change UIDs).
The syslogd emulation will not work.

Generally speaking, if you're running a simple container with a main application and one or two support services, you may benefit from the USER directive if that is your preferred way of running containers. However, if you're running more than a few services, or daemons that expect a real system with complete Unix infrastructure, then USER is probably not a good idea and you would benefit more from using privilege separation between services in your container.

Releases

Over on the releases tab, we have a number of tarballs:

s6-overlay-noarch.tar.xz: the s6-overlay scripts.
s6-overlay-${arch}.tar.xz: the binaries for platform ${arch}. They are statically compiled and will work with any Linux distribution.
s6-overlay-symlinks-noarch.tar.xz: /usr/bin symlinks to the s6-overlay scripts. Totally optional.
s6-overlay-symlinks-arch.tar.xz: /usr/bin symlinks to the skaware binaries. Totally optional.
syslogd-overlay-noarch.tar.xz: the syslogd emulation. Totally optional.
s6-overlay-${version}.tar.xz: the s6-overlay source. Download it if you want to build s6-overlay yourself.

We have binaries for at least x86_64, aarch64, arm32, i486, i686, riscv64, and s390x. The full list of supported arches can be found in conf/toolchains.

Which architecture to use depending on your TARGETARCH

The ${arch} part in the s6-overlay-${arch}.tar.xz tarball uses the naming conventions of gcc, which are not the ones that Docker uses. (Everyone does something different in this field depending on their needs, and no solution is better than any other, but the Docker one is worse than others because its naming is inconsistent. The gcc convention is better for us because it simplifies our builds greatly and makes them more maintainable.)

The following table should help you find the right tarball for you if you're using the TARGETARCH value provided by Docker:

${TARGETARCH}	${arch}	Notes
amd64	x86_64
arm64	aarch64
arm/v7	arm	armv7 with soft-float
arm/v6	armhf	Raspberry Pi 1
386	i686	i486 for very old hw
riscv64	riscv64
s390x	s390x

If you need another architecture, ask us and we'll try to make a toolchain for it. In particular, we know that armv7 is a mess and needs a flurry of options depending on your precise target (and this is one of the reasons why the Docker naming system isn't good, although arguably the gcc naming system isn't much better on that aspect).

Contributing

Any way you want! Open issues, open PRs, we welcome all contributors!

Building the overlay yourself

Download the [s6-overlay source].
Check the conf/defaults.mk file for variables you may want to change. Do not modify the file itself.
Call make followed by your variable assignments. Example: make ARCH=riscv64-linux-musl to build the overlay for RISCV64.
The tarballs will be in the output subdirectory, unless you changed the OUTPUT variable.

Upgrade Notes

Please see CHANGELOG.

pharo-graphics/Roassal

2023-10-11T01:40:24Z

The Roassal Visualization Engine

cockroachdb/molt-helm-charts

2023-10-10T01:40:27Z

Migrate Off Legacy Things (MOLT) Helm charts

chainhackers/ethonline-2023

2023-10-10T01:40:35Z

yunusdefi/contract

2023-10-10T01:40:35Z

Template project for zksync-hardhat

ruchva/code-helpers

2023-10-11T01:40:39Z

METODOS, QUERYS, VARIOS SALVAVIDAS

matkoniecz/AIAI

2023-10-11T01:40:38Z

OpenTTD AI. Official topic at TTforums - http://www.tt-forums.net/viewtopic.php?f=65&t=47298

melsman/mlkit

2023-10-11T01:40:44Z

Standard ML Compiler and Toolkit

WardPearce/Purplix.io

2023-10-10T01:41:03Z

Purplix is an open-source collection of tools dedicated to user privacy and creating trust with your audience.

Mitcheljager/svelte-tiny-slider

2023-10-10T01:41:03Z

Demo and Docs: https://mitcheljager.github.io/svelte-tiny-slider/

jamezmca/hughjass

2023-10-10T01:41:03Z

SvelteKit & TailwindCSS web portfolio

root3nl/SupportApp

2023-10-10T01:41:10Z

The Support App is developed by Root3, specialized in managing Apple devices. Root3 offers consultancy and support for organizations to get the most out of their Apple devices and is based in The Netherlands (Halfweg).

macOS Support App

Introduction

The Support app is a macOS menu bar app built for organizations to:

Help users and helpdesks to see basic diagnostic information at a glance and proactively notify them to easily fix small issues.
Offer shortcuts to easily access support channels or other company resources such as a website or a file server
Give users a modern and native macOS app with your corporate identity

The app is developed by Root3, specialized in managing Apple devices. Root3 offers managed workplaces, consultancy and support for organizations to get the most out of their Apple devices and is based in The Netherlands (Halfweg).

Root3 already had a basic in-house support app written in Objective-C and decided to completely rewrite it in Swift using SwiftUI with an all-new design that looks great on macOS Big Sur. We’ve learned that SwiftUI is the perfect way of creating great looking apps for all Apple platforms with minimal effort. In the development process we decided to make it generic so other organizations can take advantage of it and contribute to the Mac admins community.

The easiest and recommended way to configure the app is using a Configuration Profile and your MDM solution.

Requirements

macOS 11.0.1 or higher
Any MDM solution supporting custom Configuration Profiles

Download

Support App

Package Installer (includes LaunchAgent): Download

Application (zipped): Download

See the MDM deployment section below for more info.

SupportHelper

Package Installer (includes LaunchDaemon): Download

TestFlight

You can participate in beta versions of Support App using TestFlight. This requires macOS 12 or higher.

Join TestFlight

Note There may not always be a TestFlight version available.

Technologies

Written in Swift using SwiftUI
All icons are SF Symbols
Built for and compatible with macOS 11.0 and higher
Native support for Apple Silicon
Dark Mode support
Colors are matched with your macOS accent color (blue by default)
MDM support to configure your own branding such as a custom title, logo, SF Symbols and contact methods
Notarized
Sandboxed
Localized in English, Dutch, French, German and Spanish

Features

Menu Bar Icon

The Menu Bar Icon can be customized to your own image such as a PNG with Alpha Channel, an image from URL or an SF Symbol. Any image will be shown as template to match the rest of the Menu Bar Extras. Icons larger than 22 points will automatically be resized to the recommended 16 points and the aspect ration will be preserved. Optionally a notification badge can overlay the icon to attract the user's attention when an Apple Software Update is available or any other warning was triggered. Please check the preference key StatusBarIconNotifierEnabled.

Note When using a local file, make sure to put the image in a folder accessible from the App Sandbox. We recommend a subfolder in /Library/Application Support/ such as /Library/Application Support/MyOrganization

Note When using an file from URL, the image will be downloaded once when the Support App opens and will be used for subsequent launches of the app, to avoid unnecessary downloads and use cases where the Mac has no internet connection at startup. To use a new icon, the URL must be changed for the Support App to trigger a new download.

Title and logo

The row above the buttons allow a custom title and company logo. The title supports both text and Emoji. On macOS Monterey and higher, it supports Markdown as well. The logo supports a remote URL, an SF Symbol and several local images types like PNG, JPEG and ICNS and will be resized to a maximum height of 48 points. The original aspect ratio will be retained. A PNG with alpha channel is advised to get variable transparency around your logo.

Logo options

Here are the available for the Logo:

Remote URL (macOS 12 and later): https://URL_TO_IMAGE
SF Symbol (macOS 12 and later): SF=SF_SYMBOL_NAME_HERE or SF=SF_SYMBOL_NAME_HERE,color=COLOR_OPTION_HERE. Available color options: auto, multicolor, hierarchical or a custom HEX color code such as #9ACEA4
Local file: /PATH_TO_LOCAL_FILE

Note When using a local file, make sure to put the image in a folder accessible from the App Sandbox. We recommend a subfolder in /Library/Application Support/ such as /Library/Application Support/MyOrganization

Color

All the circles around the symbols have the macOS accent color and will dynamically change with the user's setting in System Preferences --> General. If desired, this color can be customised matching your corporate colors. We recommend keeping the macOS accent color when the color of your choice is too light, as text will be difficult to read.

Diagnostic information

There are a couple of info items with diagnostics available to choose from. A total of four items will be displayed in the top four buttons. Available items:

Computer Name (default): The current computer name will be displayed here. Especially helpful when your organisation has a difficult naming convention and users want to do things like AirDrop.
macOS version (default): The current version of macOS including major, minor and patch version as well as the marketing name. The marketing name will be easier to understand for your end users. A notification badge will be shown when an Apple Software Update is available. Clicking on this item shows more details and allows the user to initiate the update in System Settings.
Last Reboot (default): The current uptime. When troubleshooting some issue, the first thing you would like to do is a reboot when the uptime is high. The optional preference key UptimeDaysLimit can be used to configure the maximum amount of uptime days recommended by the organization. Exceeding this limit results in a badge counter with exclamation mark in the info item.
Storage Used (default): The storage percentage used on the boot drive. When hovering with the mouse, the available storage is shown. Clicking on this item opens the macOS built-in Storage Management app. The optional preference key StorageLimit can be used to configure the maximum percentage of used storage recommended by the organization. Exceeding this limit results in a badge counter with exclamation mark in the info item.
Network: The active network interface type (Wi-Fi or Ethernet) along with the local IPv4 address. The icon indicates the connection type, Wi-Fi or Ethernet. Clicking on this item opens the Network preference pane in System Preferences. On macOS 13 or earlier, the current SSID name is shown. Due to privacy restrictions, macOS 14 and later show "Wi-Fi" instead of the SSID name.
Mac Password: Shows when the user's password expires and supports both local and Active Directory accounts by default. Alternative supported user sources are Jamf Connect, Kerberos SSO Extension and NoMAD. Shows a warning when the expiry reaches the value set in the optional key PasswordExpiryLimit. The text label in the item can be modified using the preference key PasswordLabel.
Extension A and B: Support App Extensions to show any information. The title, icon must be configured and optionally a link to open an App, URL, User Command or Privileged Command/Script. The value below the title of the Extension must be populated by setting a preference key using a script or command. Extensions can also trigger an orange notification badge alert in both the Extension and menu bar icon. See How to populate Support App Extensions for more information.

App, link or command shortcuts

The buttons in the 3rd and 4th row behave as shortcuts to applications or links. Both rows are flexible and can show two or three buttons. The total amount of configurable buttons is possible: 0, 2, 3, 4, 5, 6. You can configure five variables for every of these buttons:

Title: Button label
Subtitle (now shown if not configured): An extra string to display when the user hovers over the button
Type: The link type the button should open. The following action types are available:
- App
- URL
- Command
Link: Application, URL or command/script to execute:
- App: Bundle Identifier of the app
- URL: Link to a webpage or other links that would normaly work like PROTOCOL://URL
- Command: Zsh command or path to a script. Be aware that this will be executed as the user and therefore has its limitations
- DistributedNotification: Zsh command or path to a script to run with elevated privileges (requires SupportHelper)
Symbol: The symbol shown in the button, see the SF Symbols section how to use these symbols

The rows with all configurable items enabled are shown in the screenshot below:

Info items and Support App Extensions in GREEN
Buttons in ORANGE

Footer Text

A footer text can optionally be used to put some additional text at the bottom of the Support App. This supports both text and Emoji. On macOS Monterey and higher, it supports Markdown. Use the preference key FooterText to configure the footer.

Notification Icon

The icon shown in alerts and the about window can be modified by using the preference key NotificationIcon.

See an example below:

Note When using a local file, make sure to put the image in a folder accessible from the App Sandbox. We recommend a subfolder in /Library/Application Support/ such as /Library/Application Support/MyOrganization

Note When using an file from URL, the image will be downloaded once when the Support App opens and will be used for subsequent launches of the app, to avoid unnecessary downloads and use cases where the Mac has no internet connection at startup. To use a new icon, the URL must be changed for the Support App to trigger a new download.

Note Modifying the app icon when it is not running would compromise the App Sandbox and we decided not to implement this. We suggest hiding the app by running the following command: sudo chflags hidden "/Applications/Support.app"

Welcome Screen

An informational window can optionally be shown when the Support App is opened for the first time. It explains the key features to the user before all data is shown. This can be set using the preference key ShowWelcomeScreen.

See an example below:

Software Update integration

The Support App shows the current version of macOS and a notification badge if there is an update or upgrade available. Clicking the info item shows a popover with more details like the name of the update(s) available and also allows organizations to add a custom text. This can be used to provide more context and explain the user about the organization's update policy or anything else. The text string supports Markdown to style it further and include links. Please check Built-in local variables for an example using Markdown and variables.

The popover allows the user to open System Settings and install the update or upgrade. If there is no update or upgrade available, the popover simply shows "Your Mac is up to date".

Note When a deferral is set using the preference key forceDelayedMajorSoftwareUpdates in the domain com.apple.applicationaccess, major macOS updates will automatically be hidden indefinitely until the key is removed or set to false. The amount of days configured for the deferral are ignored. Due to limitations and complexity, it is not supported to automatically show the macOS major update once the deferral days are passed. This behaviour replaces the HideMajorUpdates key, previously available in version 2.5 and earlier. More info here: https://developer.apple.com/documentation/devicemanagement/restrictions

Configuration

The configuration of the Support app is optimized for use with your MDM solution. The easiest way to configure the app is using a Configuration Profile so you can use whatever MDM solution you like, as long as it supports custom Configuration Profiles.

Some preference keys like the icon and status bar icon point to a file location. Due to the sandboxed characteristic of the app, not all file locations are allowed. We suggest putting the files in a folder within Application Support such as /Library/Application Support/Your Company/ where the app can read the contents. Other supported file locations can be found in Apple’s documentation about App Sandbox: https://developer.apple.com/library/archive/documentation/Security/Conceptual/AppSandboxDesignGuide/AppSandboxInDepth/AppSandboxInDepth.html#//apple_ref/doc/uid/TP40011183-CH3-SW17

Preference domain: nl.root3.support

Below are all available preference keys:

General

All general settings

Preference key	Type	Default value	Description	Example
Title	String	Support	Text shown in the top left corner when the app opens.	“Your Company Name“, “IT Helpdesk“ etc.
Logo	String	App Icon	Remote URL, SF Symbol or path to the logo shown in the top right corner when the app opens. Scales to 48 points maximum height. A subfolder in `/Library/Application Support/` is the recommended location due to sandboxing	`/Library/Application Support/Your Company/logo.png` or `https://domain.tld/url_to_image.png`
LogoDarkMode	String	App Icon	Remote URL, SF Symbol or path to the logo shown in the top right corner when the app opens for Dark Mode. Scales to 48 points maximum height. A subfolder in `/Library/Application Support/` is the recommended location due to sandboxing	`/Library/Application Support/Your Company/logo_darkmode.png` or `https://domain.tld/url_to_image.png`
NotificationIcon	String	App Icon	Remote URL or path to a custom square image to be shown in alerts and the about window.	`/Library/Application Support/Your Company/logo.png` or `https://domain.tld/url_to_image.png`
StatusBarIcon	String	Root3 Logo	Remote URL or path to the status bar icon shown in the menu bar. Recommended: PNG, 16x16 points. Icons larger than 22 points will automatically be resized to 16 points. A subfolder in `/Library/Application Support/` is the recommended location due to sandboxing	`/Library/Application Support/Your Company/statusbaricon.png` or `https://domain.tld/url_to_image.png`
StatusBarIconSFSymbol	String	Root3 Logo	Custom status bar icon using an SF Symbol. Ignored when StatusBarIcon is also set	“lifepreserver”
StatusBarIconNotifierEnabled	Boolean	false	Shows a small notification badge in the Status Bar Icon when an info items triggers a warning or notification	true
UpdateText	String	-	The text shown below the software update details popover	"Your organization requires you to update as soon as possible. More info..."
CustomColor	String	macOS Accent Color	Custom color for all symbols. Leave empty to use macOS Accent Color. We recommend not to use a very light color as text may become hard to read	HEX color in RGB format like "#8cc63f"
CustomColorDarkMode	String	macOS Accent Color	Custom color for all symbols in Dark Mode. Leave empty to use macOS Accent Color or CustomColor if specified. We recommend not to use a very dark color as text may become hard to read	HEX color in RGB format like "#8cc63f"
HideFirstRowInfoItems	Boolean	false	Hides the first row of info items.	true
HideSecondRowInfoItems	Boolean	false	Hides the second row of info items.	true
HideThirdRowInfoItems	Boolean	false	Hides the third row of info items.	true
HideFirstRowButtons	Boolean	false	Hides the first row of configurable items.	true
HideSecondRowButtons	Boolean	false	Hides the second row of configurable items.	true
ErrorMessage	String	Please contact IT support	Shown when clicking an action results in an error	"Please contact the servicedesk", "Please contact COMPANY_NAME"
ShowWelcomeScreen	Boolean	false	Shows the welcome screen when the Support App is opened for the first time.	true
FooterText	String	-	Text shown at the bottom as footnote	"Provided by your IT department with ❤️"
OpenAtLogin	Boolean	false	Launch Support (non-PKG) automatically at login and keep it open (macOS 13 and higher). This setting is ignored if a legacy LaunchAgent is installed/active. Keep disabled if you don't want to open Support at login or use your own LaunchAgent	false

Info items

Configuration of the top four items with diagnostic information.

Preference key	Type	Default value	Description	Example
InfoItemOne	String	ComputerName	Info item shown in the upper left corner	"ComputerName", "MacOSVersion", "Network", "Password", "Storage", "Uptime", "ExtensionA" or "ExtensionB"
InfoItemTwo	String	MacOSVersion	Info item shown in the upper right corner	"ComputerName", "MacOSVersion", "Network", "Password", "Storage", "Uptime", "ExtensionA" or "ExtensionB"
InfoItemThree	String	Uptime	Info item shown in the second row left	"ComputerName", "MacOSVersion", "Network", "Password", "Storage", "Uptime", "ExtensionA" or "ExtensionB"
InfoItemFour	String	Storage	Info item shown in the second row right	"ComputerName", "MacOSVersion", "Network", "Password", "Storage", "Uptime", "ExtensionA" or "ExtensionB"
InfoItemFive	String	-	Info item shown in the third row left	"ComputerName", "MacOSVersion", "Network", "Password", "Storage", "Uptime", "ExtensionA" or "ExtensionB"
InfoItemSix	String	-	Info item shown in the third row right	"ComputerName", "MacOSVersion", "Network", "Password", "Storage", "Uptime", "ExtensionA" or "ExtensionB"
UptimeDaysLimit	Integer	0 (Disabled)	Days of uptime after which a notification badge is shown, disabled by default	7
PasswordType	String	Apple	The account type to use with the Password info item: local user account (Apple), Jamf Connect, Kerberos SSO Extension or NoMAD	"Apple", "JamfConnect", "KerberosSSO" or "Nomad"
PasswordExpiryLimit	Integer	0 (Disabled)	Days until password expiry after which a notification badge is shown, disabled by default	14
PasswordLabel	String	Mac Password	Alternative text label shown in the Password info item	"AD Password", "Company Password"
StorageLimit	Integer	0 (Disabled)	Percentage of storage used after which a notification badge is shown, disabled by default	80

First row of configurable items: Item left

Preference key	Type	Default value	Description	Example
FirstRowTitleLeft	String	Remote Support	The text shown in the button label.	“Share My Screen”, “TeamViewer“, “Software Updates“ “My core application” etc.
FirstRowSubtitleLeft	String	-	Subtitle text will appear under title when the user hovers over the button. Ignored if left empty.	“Click to open“, “Share your screen“
FirstRowTypeLeft	String	App	Type of link the item should open. Can be anything like screen sharing tools, company stores, file servers or core applications in your organization.	App, URL, Command or DistributedNotification (Privileged command/script)
FirstRowLinkLeft	String	com.apple.ScreenSharing	The Bundle Identifier of the App, URL or command to open.	“com.teamviewer.TeamViewerQS“ (App), “x-apple.systempreferences:com.apple.preferences.softwareupdate“ (URL)
FirstRowSymbolLeft	String	cursorarrow	The SF Symbol shown in the button.	“binoculars.fill”, “cursorarrow.click.2” or any other SF Symbol. Please check the SF Symbols section.

First row of configurable items: Item middle

Preference key	Type	Default value	Description	Example
FirstRowTitleMiddle	String	-	The text shown in the button label.	“Self Service“, “App Store“
FirstRowSubtitleMiddle	String	-	Subtitle text will appear under title when the user hovers over the button. Ignored if left empty.	“Click to open”, “Download apps“
FirstRowTypeMiddle	String	URL	Type of link the item should open. Can be anything like screen sharing tools, company stores, file servers or core applications in your organization.	App, URL, Command or DistributedNotification (Privileged command/script)
FirstRowLinkMiddle	String	-	The Bundle Identifier of the App, URL or command to open.	“com.jamfsoftware.selfservice.mac”
FirstRowSymbolMiddle	String	-	The SF Symbol shown in the button.	“briefcase.fill”, “bag.circle”, “giftcard.fill”, “gift.circle” or any other SF Symbol. Please check the SF Symbols section.

First row of configurable items: Item right

Preference key	Type	Default value	Description	Example
FirstRowTitleRight	String	Company Store	The text shown in the button label.	“Self Service“, “App Store“
FirstRowSubtitleRight	String	-	Subtitle text will appear under title when the user hovers over the button. Ignored if left empty.	“Click to open”, “Download apps“
FirstRowTypeRight	String	App	Type of link the item should open. Can be anything like screen sharing tools, company stores, file servers or core applications in your organization.	App, URL, Command or DistributedNotification (Privileged command/script)
FirstRowLinkRight	String	com.apple.AppStore	The Bundle Identifier of the App, URL or command to open.	“com.jamfsoftware.selfservice.mac”
FirstRowSymbolRight	String	cart.fill	The SF Symbol shown in the button.	“briefcase.fill”, “bag.circle”, “giftcard.fill”, “gift.circle” or any other SF Symbol. Please check the SF Symbols section.

Second row of configurable items: Item left

Preference key	Type	Default value	Description	Example
SecondRowTitleLeft	String	Support Ticket	The text shown in the button label.	“Create ticket”, “Open incident“
SecondRowSubtitleLeft	String	-	Subtitle text will replace the title when the user hovers over the button. Ignored if left empty.	“support.company.tld”, “Now”, “Create“
SecondRowTypeLeft	String	URL	Type of link the item should open. Can be anything like screen sharing tools, company stores, file servers or core applications in your organization.	App, URL, Command or DistributedNotification (Privileged command/script)
SecondRowLinkLeft	String	https://yourticketsystem.tld	The Bundle Identifier of the App, URL or command to open.	“https://yourticketsystem.tld”, “mailto:support@company.tld”, “tel:+31000000000” or “smb://yourfileserver.tld”
SecondRowSymbolLeft	String	ticket	The SF Symbol shown in the button.	“lifepreserver”, “person.fill.questionmark” or any other SF Symbol. Please check the SF Symbols section.

Second row of configurable items: Item middle

Preference key	Type	Default value	Description	Example
SecondRowTitleMiddle	String	-	The text shown in the button label.	“Send email”
SecondRowSubtitleMiddle	String	-	Subtitle text will replace the title when the user hovers over the button. Ignored if left empty.	“support@company.tld”, “Now”
SecondRowTypeMiddle	String	URL	Type of link the item should open. Can be anything like screen sharing tools, company stores, file servers or core applications in your organization.	App, URL, Command or DistributedNotification (Privileged command/script)
SecondRowLinkMiddle	String	-	The Bundle Identifier of the App, URL or command to open.	“https://yourticketsystem.tld”, “mailto:support@company.tld”, “tel:+31000000000” or “smb://yourfileserver.tld”
SecondRowSymbolMiddle	String	-	The SF Symbol shown in the button.	“paperplane”, “arrowshape.turn.up.right.fill” or any other SF Symbol. Please check the SF Symbols section.

Second row of configurable items: Item right

Preference key	Type	Default value	Description	Example
SecondRowTitleRight	String	Phone	The text shown in the button label.	“Call Helpdesk“, “Phone“
SecondRowSubtitleRight	String	-	Subtitle text will replace the title when the user hovers over the button. Ignored if left empty.	“+31 00 000 00 00”, “Now”, “Call“
SecondRowTypeRight	String	URL	Type of link the item should open. Can be anything like screen sharing tools, company stores, file servers or core applications in your organization.	App, URL, Command or DistributedNotification (Privileged command/script)
SecondRowLinkRight	String	tel:+31000000000	The Bundle Identifier of the App, URL or command to open.	“https://yourticketsystem.tld”, “mailto:support@company.tld”, “tel:+31000000000” or “smb://yourfileserver.tld”
SecondRowSymbolRight	String	phone	The SF Symbol shown in the button.	“iphone.homebutton”, “megaphone” or any other SF Symbol. Please check the SF Symbols section.

Advanced configuration

Support App Extensions

Support App Extensions enable administrators to create custom info items and populate those with output from scripts or commands. You can use your MDM solution to run scripts or commands to populate the Support App Extensions. Optionally we provide SupportHelper to run scripts or commands everytime the Support App popover appears to make sure data is up to date. Please read Privileged commands or scripts (SupportHelper) down below for more info.

Below are the preference keys to enable Support App Extensions:

Preference key	Type	Default value	Description	Example
ExtensionTitleA	String	-	The title shown in the extension.	"Last Check-In", "Compliance"
ExtensionSymbolA	String	-	The SF Symbol shown in the extension.	"clock.badge.checkmark.fill",
ExtensionTypeA	String	App	Type of link the item should open. Can be anything like screen sharing tools, company stores, file servers or core applications in your organization.	App, URL, Command or DistributedNotification (Privileged command/script)
ExtensionLinkA	String	-	The Bundle Identifier of the App, URL or command to open.	`defaults write /Library/Preferences/nl.root3.support.plist ExtensionLoadingA -bool true; /usr/local/bin/jamf policy;` `/usr/local/bin/jamf_last_check-in_time.zsh` or any other action you prefer by clicking on the Extension
ExtensionTitleB	String	-	The title shown in the extension.	"Account Privileges"
ExtensionSymbolB	String	-	The SF Symbol shown in the extension.	"wallet.pass.fill"
ExtensionTypeB	String	App	Type of link the item should open. Can be anything like screen sharing tools, company stores, file servers or core applications in your organization.	App, URL, Command or DistributedNotification (Privileged command/script)
ExtensionLinkB	String	-	The Bundle Identifier of the App, URL or command to open.	`/usr/local/bin/sap_privileges_change_permissions.zsh` or any other action you prefer by clicking on the Extension
OnAppearAction	String	-	Path to script script or command to be executed when the Support App is opened by clicking on the menu bar item. The SupportHelper is required for this feature.	`/usr/local/bin/runs_when_support_appears.zsh` such as `/usr/local/bin/user_permissions.zsh` or `/usr/local/bin/jamf_last_check-in_time.zsh` or `/usr/local/bin/mscp_compliance_status.sh`

Warning Both Support App Extensions have other preference keys ExtensionValueA and ExtensionValueB but those keys are meant to be dynamically set and changed by a script or command, not by MDM. Once set, the default placeholder will disappear and show the output from the preference keys.

How to populate Support App Extensions

Support App Extensions must be populated by setting the value in a preference key within the preference domain nl.root3.support. This can be achieved by running custom scripts from your MDM solution or using the OnAppearAction combined with SupportHelper. This last option will allow you to update the Support App Extension values every time the Support App popover appears by running the script.

Create a custom script and populate the desired value by running the following command: sudo defaults write /Library/Preferences/nl.root3.support.plist ExtensionValueA -string "OUTPUT_VALUE_HERE"
Add the following command to show a placeholder while getting the value: sudo defaults write /Library/Preferences/nl.root3.support.plist ExtensionValueA -string "KeyPlaceholder"
Add the following command at the beginning of the script to show a spinning indicator while getting the value: sudo defaults write /Library/Preferences/nl.root3.support.plist ExtensionLoadingA -bool true
Add the following command at the end of the script to stop the spinning indicator view: sudo defaults write /Library/Preferences/nl.root3.support.plist ExtensionLoadingA -bool false

Below a simple example script including loading effect and placeholder while loading

#!/bin/zsh

# Start spinning indicator
defaults write /Library/Preferences/nl.root3.support.plist ExtensionLoadingA -bool true

# Show placeholder value while loading
defaults write /Library/Preferences/nl.root3.support.plist ExtensionValueA -string "KeyPlaceholder"

# Keep loading effect active for 0.5 seconds
sleep 0.5

# Get output value
command_output=$(PUT_COMMAND_TO_GET_OUTPUT_HERE)

# Set output value
defaults write /Library/Preferences/nl.root3.support.plist ExtensionValueA -string "${command_output}"

# Trigger an orange warning notification badge depending on the output you decide
if [[ "${command_output}" == "OUTPUT_IS_BAD" ]]; then
 defaults write /Library/Preferences/nl.root3.support.plist ExtensionAlertA -bool true
else
 defaults write /Library/Preferences/nl.root3.support.plist ExtensionAlertA -bool false
fi

# Stop spinning indicator
defaults write /Library/Preferences/nl.root3.support.plist ExtensionLoadingA -bool false

Note When using more than one Support App Extension combined with OnAppearAction, it's best to update the values in one script instead of chaining multiple scripts to have the best experience

Note Please do not forget to make the script executable: sudo chmod +x /PATH/TO/SCRIPT

Privileged commands or scripts with SupportHelper

To allow commands or scripts to be executed with root privileges, the SupportHelper is available optionally. This utility is built on Distributed Notifications to allow inter-app communication between the Support App and the SupportHelper. The Support App notifies SupportHelper and the message contains the preference key set in the Configuration Profile with the command or path to the script. SupportHelper listens for new messages using a LaunchDaemon and executes the command or script by requesting the command or path to the script from the Configuration Profile.

Below an example to force a MDM check-in using SupportHelper and a custom script:

More information about Distributed Notifications: https://developer.apple.com/documentation/foundation/distributednotificationcenter

Use Cases

There are a couple of use cases SupportHelper can help with. For example run a command or script with root privileges:

Every time the Support App popover appears, populate Support App Extensions using OnAppearAction
Extension Attributes (Jamf Pro) by adding the commands to populate the Support App Extension to the EA:
- Show device compliance information, such as the macOS Security Compliance Project Failed Results Count
Executing a background task by clicking on a configurable button. Some examples:
- Request an MDM check-in or inventory depending on your MDM solution
- Requesting temporary admin permissions (for example in conjuntion with SAP Privileges)
- Collecting logs such as sudo sysdiagnose and sending the output somewhere else
- Run device compliance remediation, such as the macOS Security Compliance Project Remediation Script
- Any other action requiring root privileges, especially when users have standard permissions

File locations

The SupportHelper installer places two files:

LaunchDaemon: /Library/LaunchDaemons/nl.root3.support.helper.plist

Binary: /usr/local/bin/SupportHelper

Security considerations

As SupportHelper is able to execute scripts or commands with root privileges, it needs to be used responsibly. For most deployments, SupportHelper will not be needed and we recommend deploying the Support App without SupportHelper. If you're unsure or unfamiliar with this concept, DO NOT use SupportHelper. This utility is separated from the Support App to avoid compromising the app-sandbox as well.

Note Only values from a Configuration Profile will be used. Values set by defaults write will be ignored as it imposes a security risk.

Variables

You can use variables to dynamically populate text fields, like the title, footer, buttons or any other text field. You have the option to use Local Variables built-in the Support App and are MDM agnostic, or use MDM specific variables if available in your MDM solution.

Note Using Built-in Local Variables or MDM variables depends on the use case you want to achieve or the available variables and you may use both if needed.

Built-in local variables

The Support App supports local variables with device and user details and work independently from your MDM solution.

The following built-in local variables are available with an example:

$LocalComputerName: the current computer name/hostname
$LocalModelName: the model name, like MacBook Air (M2, 2022). Apple Silicon only
$LocalModelShortName: the short model name like MacBook or iMac
$LocalFullName: the full name of the local macOS user account
$LocalUserName: the username of the local macOS user account
$LocalMacosVersion: the macOS version, like 13.4.1
$LocalMacosVersionName: the macOS version marketing name, like Ventura or Sonoma
$LocalSerialNumber: the devices serial number
$LocalIpAddress: the current IP address
$LocalUpdatesAvailable: the number of updates available

Examples

Set title to: "Hi $LocalFullName!"
Set FooterText to: "Provided by IT with ❤️\nSerial Number: $LocalSerialNumber"
Set UpdateText including Markdown features to: "Your $LocalModelShortName has **$LocalUpdatesAvailable update(s)** available. Please update as soon as possible.\n[More info](https://LINK-TO-UPDATE-POLICY)"

Note Built-in local variables are case sensitive

MDM variables

Jamf Pro variables

When using Jamf Pro as the MDM solution, variables from Jamf Pro can also be used in the Configuration Profile values to dynamically populate text fields.

Example

Set title to "Hi $FULLNAME!"

More information about Jamf variables: https://learn.jamf.com/bundle/jamf-pro-documentation-current/page/Computer_Configuration_Profiles.html

How to use SF Symbols

We choose to go all the way with SF Symbols as these good looking icons are designed by Apple and give the app a native look and feel. All icons have a symbol name which you can use in the Configuration Profile. As these icons are built into macOS, it automatically shows the correct icon.

Download SF Symbols here
Select the icon you’d like to use
Copy the symbol name and paste into your Configuration Profile

MDM deployment

It is recommended to deploy the Configuration Profile first before installing the Support app.

Jamf Pro custom JSON schema

A JSON Schema for Jamf Pro is provided for easy configuration of all the preference keys without creating/modifying a custom Configuration Profile in XML format. Download the JSON file here

More information about the JSON Schema feature in Jamf Pro: https://docs.jamf.com/technical-papers/jamf-pro/json-schema/10.19.0/Overview.html

Installer or app bundle

Depending on your preference or MDM solution you can use either the installer or zipped app bundle. The installer includes a LaunchAgent and is the recommended method to make sure the app stays open and relaunches automatically.

Sample LaunchAgent

A sample LaunchAgent to always keep the app alive is provided here

Sample Configuration Profile

A sample Configuration Profile you can edit to your preferences is provided here

Background Item Management

A sample Configuration Profile is provided (both signed and unsigned) for macOS 13 and higher to avoid users from disabling the LaunchAgent in System Settings > General > Login Items. The profile uses the Root3 Team ID to only allow signed software from Root3. Samples

Logging

Logs can be viewed from Console or Terminal by filtering the subsystems nl.root3.support (Support App) and nl.root3.support.helper (SupportHelper).

An example to stream current logs in Terminal for troubleshooting:

log stream --debug --info --predicate 'subsystem contains "nl.root3.support"'

Known issues

All available software updates (minor and major) are shown in the menu bar icon and the macOS version info item, even when the update is deferred using a Restrictions Configuration Profile from MDM. macOS collects all available updates in /Library/Preferences/com.apple.SoftwareUpdate.plist regardless of any deferral configurations. Only major OS updates can be hidden using the HideMajorUpdates key for macOS 12.3 and later.
Buttons may keep a hovered state when mouse cursor moves fast: FB8212902 (resolved in macOS Monterey)
When Jamf Connect is used as password type, clicking "Change Now" does not allow the user to open the Jamf Connect change password window, but instead triggers an alert. Jamf Connect does not support a URL Scheme for opening the Change Password window. Please upvote this feature request: https://ideas.jamf.com/ideas/JN-I-16087

Changelog

Version 2.4

Support App Extensions: introducing a new way to make your own custom info items and provide relevant information to your end users and create actions. Extensions can show anything you want and can be populated using scripts or commands, for example using your MDM solution. There is support for two Extensions to use in the info item rows.
SupportHelper (separately available): there is now support for privileged scripts and commands to run directly from a button in the Support App while preserving the App Sandbox. An optional utility called SupportHelper will be required and handles the execution on behalf of the Support App. This is built on the Distributed Notifications framework to allow inter-app communication. SupportHelper is a separate installer and is not included in the standard Support App installer.
Optional info items row: an extra row with info items or Extensions can now be enabled. When enabling this third row, it will show the Password and Network info items by default. The row can show any info item or Extension.
New password sources: the Password info item can now be configured to show password expiry information from Jamf Connect, Apple’s Kerberos SSO Extension or NoMAD. Set preference key "PasswordType" to "JamfConnect", "KerberosSSO" or "Nomad".
The title text now supports markdown just like the footer text.
Items no longer show a hover effect when no link is configured, allowing buttons to be static without a clickable action. This applies to both configurable buttons and Extensions

Version 2.3

Welcome Screen: an informational window can now optionally be shown when the Support App is opened for the first time. It explains the key features to the user.
Preference key ‘StorageLimit’ added to configure the maximum percentage of used storage recommended by the organization. When the limit is reached, a badge counter with exclamation mark will be shown in the Storage tile. Also a little orange notification badge can overlay the menu bar icon when the preference key ‘StatusBarIconNotifierEnabled’ is set to ‘true’.
The Network info item now shows either the current SSID or Ethernet as the title instead of ‘IP Address’.
Preference key ‘NotificationIcon’ added to configure a custom square image to be shown in alerts and the about window.
The notification badge in the StatusBarItem can now show either orange or red, depending on the current state. An available software update will overrule orange warnings and will show a red notification badge.
A footer option is added to put some additional text at the bottom of the Support App. This supports both text and Emoji. On macOS Monterey and higher, it supports Markdown.
French localization is added
macOS Monterey compatibility
Bug fixes and improvements

Version 2.2

Modulair info items: you can now configure any of the top four info items to any of the six available options:
- IP Address (NEW)
- Mac Password(NEW)
- Computer Name
- macOS Version
- Last Reboot
- Storage
IP Address is added as a configurable info item. It shows the current IPv4 address. The icon will show the connection type, Wi-Fi or Ethernet. Clicking the item will open Network in System Preferences.
Mac Password is added as a configurable info item. It shows the days until the Mac password expires. Clicking the item will open Accounts in System Preferences. Local accounts and Active Directory accounts are supported. The text label in the item can be modified using the preference key ‘PasswordLabel’.
Preference key ‘UptimeDaysLimit’ added to configure the maximum amount of uptime days recommended by the organization. When the limit is reached, a badge counter with exclamation mark will be shown in the Last Reboot tile. Also a little notification badge can overlay the menu bar icon when the preference key ‘StatusBarIconNotifierEnabled’ is set to ‘true’
Preference key ‘PasswordExpiryLimit’ added to configure the minimum amount of days before the user should change the Mac password. When the limit is reached, a badge counter with exclamation mark will be shown in the Mac Password tile. Also a little notification badge can overlay the menu bar icon when the preference key ‘StatusBarIconNotifierEnabled’ is set to ‘true’.
Flexible rows of for App, URL or Command: The middle button is now optional, offering a two or three button row. The middle button are enabled by setting the following keys: FirstRowTitleMiddle or SecondRowTitleMiddle. Together with HideFirstRow and HideSecondRow, the total amount of configurable buttons is possible: 0, 2, 3, 4, 5, 6. As a result, keys for the middle buttons no longer have default values and the app shows two buttons by default;
Computer Name info item: The Mac model name and introduction year is shown when hovering over Computer Name. Clicking the item opens ‘About This Mac’
Small UI tweaks for Dark Mode. Buttons now have a small border and are more distinguishable
Performance improvements and bug fixes

Version 2.1

Preference key LogoDarkMode is added to provide a separate logo for Dark Mode
Preference key CustomColorDarkMode is added to set a separate custom color for Dark Mode
The number of available Apple Software Updates will now be shown in a badge counter in the macOS version tile. Also a little notification badge can overlay the menu bar icon when the preference key ‘StatusBarIconNotifierEnabled’ is set to ‘true’
Clicking on the macOS version tile will now open the Software Update preference pane in System Preferences
Running basic zsh commands as the user can now be used as an action by setting the LinkType to “Command”
Changes to the menu bar icon will now be observed and will automatically be applied without restarting the app
Preference key ErrorMessage is added to provide a custom error message when clicking an App, URL or Command results in an error
The app’s icon is changed to a more generic looking icon
Default error message is improved
Unified logging is implemented for the subsystem “nl.root3.support” to be able to capture errors when using commands or scripts
Fixed a localization issue for error alerts
Fixed an issue where some functions kept running in the background, causing more CPU time than required

Privacy policy

We value your privacy. To protect your privacy, the Support App does not collect or send any personal data. The only outgoing network request is to send the computer's serial number to an Apple API to request the model's marketing name. This information is only kept within the app and is never collected elsewhere.

Note and disclaimer

Root3 developed this application as a side project to add additional value for our customers
The application can be used free of charge and is provided ‘as is’, without any warranty
Comments and feature request are appreciated. Please file an issue on Github

Jinxiansen/SwiftUI

2023-10-10T01:41:10Z

`SwiftUI` Framework Learning and Usage Guide. 🚀

This article refers to SwiftUI apple example and records the results of the exploration here, I hope to be helpful to you.

For the content described in this article, by default you have some experience based on Swift language development, so it will not describe every detail in detail; if you have doubts about Swift syntax, you can learn Swift Grammar.

When learning and using SwiftUI, if you have any questions, you can join the SwiftUI QQ Group: 18552966 to discuss communication.

中文版🇨🇳

Recommended Preview: Windows 11 desktop client implemented using SwiftUI.

⭐️ Stargazers over time

💻 Requirements

macOS 10.15
Xcode 11.0
iOS 13.0

Directory：

Basic View

Text
Image
- Image
- WebImage
Button
Picker
- Picker
- DatePicker
- Toggle
- Slider
- Stepper
- SegmentedControl
Special Views
- WebView
- UIViewController

Layout

Stacks
- HStack
- VStack
- ZStack
List
- List
- ScrollView
- ForEach
Container Views
- Group
- GroupBox
- Section
- Form
Architectural Views
Alert
- Alert
- Modal
- Popover
- Sheet
- ActionSheet

State and Data Flow

Bindings
- Binding
Data-Dependent Views
Environment Values
- Environment
- EnvironmentValues
ENavigation Models
- DynamicNavigationDestinationLink
Preferences
- LocalizedStringKey
Transactions
- Transaction

Gestures

Basic View

Text

Text is used to display one or more lines of text content with the same effect as UILabel, but it is even better.

If you want to create Text, just create it with Text("SwiftUI"); With chained syntax, you can also add multiple attributes to the text, such as fonts, colors, shadows, spacing between top left and right, and so on.

Example:


Text("SwiftUI")
    .foregroundColor(.orange)
    .bold()
    .font(.system(.largeTitle))
    .fontWeight(.medium)
    .italic()
    .shadow(color: .black, radius: 1, x: 0, y: 2)

View running results

HStack and VStack controls are used to host multiple views, as mentioned later.

🔝

TextField

TextField is used to add a normal input box, which is often used as a text input.

Example：


TextField(self.$name, placeholder: self.nameText, onEditingChanged: { changed in
    print("onEditing: \(changed)")
}) {
    print("userName: \(self.name)")
    self.endEditing(true)
}}
.padding(10)
.frame(height: 50)
.textFieldStyle(RoundedBorderTextFieldStyle())
.padding(EdgeInsets(top: 0, leading: 20, bottom: 0, trailing: 20))

View running results

🔝

SecureField

SecureField is generally used as a password input. It is used in the same way as TextField. The example and the running effect are the same as TextField.

Image

The Image control is used to display images, example:


Image("icon")
    .resizable()
    .frame(width: 100,
           height: 100,
           alignment: .center)

View running results

🔝

WebImage

webImage is used to download the web image, use the URLSession to download the original Image after successful download; you can also use Kingfisher in the downloadWebImage function .

Example：


var body: some View {
        Image(uiImage: self.uiImage ?? placeholderImage)
            .resizable()
            .onAppear(perform: downloadWebImage)
            .frame(width: 80,
                   height: 80,
                   alignment: .center)
            .onTapGesture {
                print("Tap ")
        }
    }

View running results

🔝

Button

Button is used to respond to click events.

Example:


Button(action: {
    print("Tap")
}) {
   Text("I'm a Button")
}

View running results

🔝

PullDownButton

Waiting for release.

ItemBasedPopUpButton

Waiting for release.

NavigationButton

NavigationButtonPage is used to push to the next navigation page.

Example:


NavigationLink(destination: NavigationButtonPage()) {
            Text("NavigationButton").bold()
                .foregroundColor(.orange)
                .font(.largeTitle)
            }
    .navigationBarTitle(Text("Page"))

View running results

🔝

PresentationButton is deprecated

PresentationButton ~~is used to pop up a page.~~ has deprecated, please use NavigationLink

🔝

EditButton

EditButton is used to trigger the editing state, just use the navigationBarItems setting when using it.

Example:


navigationBarItems(trailing: EditButton())

View running results

🔝

PasteButton

Waiting for release.

Picker

Picker can customize the selector of the data source.

Example:


Picker(selection: $leftIndex, label: Text("Picker")) {
    ForEach(0..<leftSource.count) {
        Text(self.leftSource[$0]).tag($0)
    }
    }.frame(width: UIScreen.main.bounds.width/2)

View running results

🔝

DatePicker

DatePicker is used to select the absolute date of the control.

Example:


DatePicker(selection: $server.date, 
in: server.spaceDate, 
displayedComponents: .date, label: {
    Text("")
})

View running results

🔝

Toggle

Toggle is used to switch the selected state, for example:


Toggle(isOn: $isOn) {
    Text("State: \(self.isOn == true ? "Open":"open")")
}.padding(20)

View running results

🔝

Slider

Slider A control for selecting values from a finite range of values, example:


Slider(value: $data.rating)

View running results

🔝

Stepper

Stepper is used to increase or decrease the value, example:


Stepper(value: $value, step: 2, onEditingChanged: { c in
    print(c)
}) {
    Text("Stepper Value: \(self.value)")
    }.padding(50)

View running results

🔝

SegmentedControl is deprecated

SegmentedControl is used for segmentation condition selection, example:


SegmentedControl(selection: $currentIndex) {
    ForEach(0..<items.count) { index in
        Text(self.items[index]).tag(index)
    }
    }.tapAction {
        print("currentIndex: \(self.currentIndex)")
}

View running results

🔝

WebView

WebView is used to display an open web page, example:


struct WebViewPage : UIViewRepresentable {
    func makeUIView(context: Context) -> WKWebView  {
        return WKWebView()
    }
    func updateUIView(_ uiView: WKWebView, context: Context) {
        let req = URLRequest(url: URL(string: "https://www.apple.com")!)
        uiView.load(req)
    }
}

View running results

🔝

UIViewController

UIViewController is used to display the UIViewController that opens UIKit in SwiftUI and opens the SwiftUI View in UIViewController.

Example:

First define:


struct ControllerPage<T: UIViewController> : UIViewControllerRepresentable {
    
    typealias UIViewControllerType = UIViewController
    
    func makeUIViewController(context: UIViewControllerRepresentableContext<ControllerPage>) -> UIViewController {
        return T()
    }
    
    func updateUIViewController(_ uiViewController: UIViewController, context: UIViewControllerRepresentableContext<ControllerPage>) {
        debugPrint("\(#function)：\(type(of: T.self))")
    }
    
}

Then use this:


NavigationButton(destination: ControllerPage<UIKitController>()) {
    PageRow(title: "UIViewController",subTitle: "Open UIViewController")

}

View running results

🔝

Layout

HStack

HStack is used to arrange the subviews on a horizontal line.

Example:


HStack {
    Text("made in China.")
    Divider() // Just add a line.
    Text("the People's Republic Of China.")
}

View running results

🔝

VStack

VStack is used to arrange the subviews on a vertical line.

Example:


VStack {
    Text("made in China.")
    Divider() // Just add a line.
    Text("the People's Republic Of China.")
}

View running results

🔝

ZStack

ZStack is used to override the subview, aligned on two axes.

Example:


ZStack {
    Text("made in China.")
    Divider() // Just add a line.
    Text("the People's Republic Of China.")
}

View running results

🔝

List

List list container to display a list of data.

Examples:


List(0..<5) { item in
    Text("Hello World !")
}.navigationBarTitle(Text("List"), displayMode: .large)

View running results

🔝

ScrollView

ScrollView is a scroll view container.

Example:


ScrollView {
    Text("SwiftUI").padding(20)
    Divider()
    Image("icon").resizable()
        .frame(width: 300, height: 300, alignment: .center)
    Divider()
    Text("Views and ... user interface.")
    }
    .border(Color.gray.gradient, width: 1)
            .cornerRadius(10)
            .padding(10)
            .navigationBarTitle(Text("ScrollView"))

View running results

🔝

ForEach

ForEach is used to present a view based on a collection of existing data.

Example:


let data = (0..<5)
var body: some View {
    ForEach(data) { e in
        Text("Hello \(e)")
            .bold()
            .font(.system(size: 25, design: .monospaced))
            .padding(5)
}

View running results

🔝

Group

Group is used to aggregate multiple views, and the properties set on the Group will be applied to each child view.

Example:


Group {
        Text("Hello World !")
        Text("Hello World !")
      }

View running results

🔝

GroupBox

Waiting for release.

Section

Section is used to create the header/footer view content, which is generally used in conjunction with the List component.

Example:


Section(header: Text("I'm header"), footer: Text("I'm footer")) {
    ForEach(0..<3) {
        Text("Hello \($0)")
    }
}

View running results

Form

Form A container for grouping controls used for data entry, such as in settings or inspectors.

Example:


Form {
   TextField("First Name", text: $firstName)
   TextField("Last Name", text: $lastName)
}

View running results

🔝

NavigationView

NavigationView is used to create a view container that contains the top navigation bar.

Example:


NavigationView {
            Text("🧚‍♂️🧚‍♀️🧜‍♂️🧜‍♀️🧞‍♂️🧞‍♀️").blur(radius: 5)
            Text("Swifter Swifter")
            .bold()
                .foregroundColor(.orange)
                .font(.largeTitle)
        }
    .navigationBarTitle(Text("NavigationView"))

View running results

🔝

TabView

TabView is used to create a view container that contains the bottom ** TabBar**.

Example:


TabView(selection: $index) {
    ForEach(0..<imgs.count) { item in
        TabItemPage(index: item)
            .tabItem{
                Image(self.imgs[item])
                Text("\(item)")
        }
        .tag(item)
    }
}

View running results

🔝

HSplitView

Waiting for release.

VSplitView

Waiting for release.

🔝

Alert

Alert is used to display a bullet reminder that needs to be associated with a trigger event.

Example:


alert(isPresented: $showAlert, content: {
            Alert(title: Text("确定要支付这100000000美元吗？"),
                  message: Text("请谨慎操作\n一旦确认，钱款将立即转入对方账户"),
                  primaryButton: .destructive(Text("确认")) { print("转出中...") },
                  secondaryButton: .cancel())
        }).navigationBarTitle(Text("Alert"))

View running results

🔝

ActionSheet

ActionSheet is used to pop up a selection box.

Example:


ActionSheet(title: Text("Title"),
            message: Text("Message"),
            buttons:
    [.default(Text("Default"), onTrigger: {
        print("Default")
        self.showSheet = false
    }),.destructive(Text("destructive"), onTrigger: {
        print("destructive")
        self.showSheet = false
    }),.cancel({
        print("Cancel")
        self.showSheet = false
    })])

usage：


.actionSheet(isPresented: $showSheet, content: {sheet})

View running results

🔝

Modal is used to pop up a view.

Example:


Modal(Text("Modal View"),onDismiss: {
    print("View Dismiss !")
})

View running results

🔝

Popover

Popover is used to pop up a view, see the results below.

Example:


.popover(isPresented: $showPop, content: {
                ImagePage()
        })

View running results

🔝

📎 About

The code involved in the above example is in this repository code. It is recommended to download and run the view.
If you have better usage and suggestions about SwiftUI, look forward to sharing it!
If there are omissions and errors in the examples in this article, please create a Issue !

✉️ Contacts

email : hi@jinxiansen.com

微博 : @晋先森

📄 License

SwiftUI is released under the MIT license. See LICENSE for details.

web-auth/symfony-webauthn-demo

2023-10-10T01:42:05Z

Demo Application using Symfony 6, Tailwind, FrankenPHP, AssetMapper and Webauthn

piyushgarg-dev/review-app

2023-10-10T01:42:12Z

YouTube Music Desktop App bundled with custom plugins (and built-in ad blocker / downloader)

YouTube Music

Electron wrapper around YouTube Music featuring:

Native look & feel, aims at keeping the original interface
Framework for custom plugins: change YouTube Music to your needs (style, content, features), enable/disable plugins in one click

Download

You can check out the latest release to quickly find the latest version.

Arch Linux

Install the youtube-music-bin package from the AUR. For AUR installation instructions, take a look at this wiki page.

MacOS

If you get an error "is damaged and can’t be opened." when launching the app, run the following in the Terminal:

xattr -cr /Applications/YouTube\ Music.app

Windows

You can use the Scoop package manager to install the youtube-music package from the extras bucket.

scoop bucket add extras
scoop install extras/youtube-music

Alternately you can use Winget, Windows 11s official CLI package manager to install the th-ch.YouTubeMusic package.

Note: Microsoft Defender SmartScreen might block the installation since it is from an "unknown publisher". This is also true for the manual installation when trying to run the executable(.exe) after a manual download here on github (same file).

winget install th-ch.YouTubeMusic

How to install without a network connection? (in Windows)

Download the *.nsis.7z file for your device architecture in release page.
- x64 for 64-bit Windows
- ia32 for 32-bit Windows
- arm64 for ARM64 Windows
Download installer in release page. (*-Setup.exe)
Place them in the same directory.
Run the installer.

Available plugins:

Ad Blocker: Block all ads and tracking out of the box
Audio Compressor: Apply compression to audio (lowers the volume of the loudest parts of the signal and raises the volume of the softest parts)
Blur Nav Bar: makes navigation bar transparent and blurry
Bypass age restrictions: bypass YouTube's age verification
Captions selector: Enable captions
Compact sidebar: Always set the sidebar in compact mode
Crossfade: Crossfade between songs
Disable Autoplay: Makes every song start in "paused" mode
Discord: Show your friends what you listen to with Rich Presence
Downloader: downloads MP3 directly from the interface (youtube-dl)
Exponential Volume: Makes the volume slider exponential so it's easier to select lower volumes.
In-App Menu: gives bars a fancy, dark look

(see this post if you have problem accessing the menu after enabling this plugin and hide-menu option)
Last.fm: Scrobbles support
Lyrics Genius: Adds lyrics support for most songs
Navigation: Next/Back navigation arrows directly integrated in the interface, like in your favorite browser
No Google Login: Remove Google login buttons and links from the interface
Notifications: Display a notification when a song starts playing (interactive notifications are available on windows)
Picture in picture: allows to switch the app to picture-in-picture mode
Playback Speed: Listen fast, listen slow! Adds a slider that controls song speed
Precise Volume: Control the volume precisely using mousewheel/hotkeys, with a custom hud and customizable volume steps
Quality Changer: Allows changing the video quality with a button on the video overlay
Shortcuts: Allows setting global hotkeys for playback (play/pause/next/previous) + disable media osd by overriding media keys + enable Ctrl/CMD + F to search + enable linux mpris support for mediakeys + custom hotkeys for advanced users
Skip-Silences - Automatically skip silenced sections
SponsorBlock: Automatically Skips non-music parts like intro/outro or parts of music videos where the song isn't playing
Taskbar Media Control: Control playback from your Windows taskbar
Touchbar: Custom TouchBar layout for macOS
Tuna-OBS: Integration with OBS's plugin Tuna
Video Toggle: Adds a button to switch between Video/Song mode. can also optionally remove the whole video tab
Visualizer: Different music visualizers

Auto confirm when paused (Always Enabled): disable the "Continue Watching?" popup that pause music after a certain time

If Hide Menu option is on - you can show the menu with the alt key (or ` [backtick] if using the in-app-menu plugin)

Themes

You can load CSS files to change the look of the application (Options > Visual Tweaks > Themes).

Some predefined themes are available in https://github.com/kerichdev/themes-for-ytmdesktop-player.

Dev

git clone https://github.com/th-ch/youtube-music
cd youtube-music
npm ci
npm run start

Build your own plugins

Using plugins, you can:

manipulate the app - the BrowserWindow from electron is passed to the plugin handler
change the front by manipulating the HTML/CSS

Creating a plugin

Create a folder in plugins/YOUR-PLUGIN-NAME:

if you need to manipulate the BrowserWindow, create a file with the following template:

// file: back.ts
export default (win: Electron.BrowserWindow, config: ConfigType<'YOUR-PLUGIN-NAME'>) => {
  // something
};

then, register the plugin in index.ts:

import yourPlugin from './plugins/YOUR-PLUGIN-NAME/back';

// ...

const mainPlugins = {
  // ...
  'YOUR-PLUGIN-NAME': yourPlugin,
};

if you need to change the front, create a file with the following template:

// file: front.ts
export default (config: ConfigType<'YOUR-PLUGIN-NAME'>) => {
  // This function will be called as a preload script
  // So you can use front features like `document.querySelector`
};

then, register the plugin in preload.ts:

import yourPlugin from './plugins/YOUR-PLUGIN-NAME/front';

const rendererPlugins: PluginMapper<'renderer'> = {
  // ...
  'YOUR-PLUGIN-NAME': yourPlugin,
};

Common use cases

injecting custom CSS: create a style.css file in the same folder then:

import path from 'node:path';
import { injectCSS } from '../utils';

// back.ts
export default (win: Electron.BrowserWindow) => {
  injectCSS(win.webContents, path.join(__dirname, 'style.css'));
};

changing the HTML:

// front.ts
export default () => {
  // Remove the login button
  document.querySelector(".sign-in-link.ytmusic-nav-bar").remove();
};

communicating between the front and back: can be done using the ipcMain module from electron. See utils.js file and example in navigation plugin.

Build

Clone the repo
Run npm i to install dependencies
Run npm run build:OS

npm run dist:win - Windows
npm run dist:linux - Linux
npm run dist:mac - MacOS

Builds the app for macOS, Linux, and Windows, using electron-builder.

Tests

npm run test

Uses Playwright to test the app.

License

MIT © th-ch

Vanessa219/vditor

2023-10-10T01:42:12Z

♏ 一款浏览器端的 Markdown 编辑器，支持所见即所得（富文本）、即时渲染（类似 Typora）和分屏预览模式。An In-browser Markdown editor, support WYSIWYG (Rich Text), Instant Rendering (Typora-like) and Split View modes.

易于使用的 Markdown 编辑器，为适配不同的应用场景而生

English | Demo

💡 简介

Vditor 是一款浏览器端的 Markdown 编辑器，支持所见即所得、即时渲染（类似 Typora）和分屏预览模式。它使用 TypeScript 实现，支持原生 JavaScript 以及 Vue、React、Angular 和 Svelte 等框架。

欢迎到 Vditor 官方讨论区了解更多。同时也欢迎关注 B3log 开源社区微信公众号 B3log开源：

🗺️ 背景

随着 Markdown 排版方式的普及，越来越多的应用开始集成 Markdown 编辑器。目前主流可集成的 Markdown 编辑器现状如下：

有的仅支持分屏预览，即编辑区和预览区分离
有的同时支持所见即所得和分屏预览，但所见即所得模式下不能完整支持 Markdown 语法排版
几乎没有类似 Typora 的即时渲染

而这三点恰好对应了三种应用场景：

分屏预览：适配传统的 Markdown 使用场景，适合大屏下编辑排版
所见即所得：对不熟悉 Markdown 的用户友好，熟悉 Markdown 的用户也可以无缝使用
即时渲染：理论上这是最为优雅的 Markdown 编辑方式，让熟悉 Markdown 的用户能够更专注于内容创作

所以，一个能够适配应用场景的 Markdown 编辑器至关重要，它需要考虑到：

传统 Markdown 用户的使用场景，提供分屏预览
富文本编辑用户的使用场景，提供所见即所得
高阶 Markdown 用户的使用场景，提供即时渲染

Vditor 在这些方面做了努力，希望能为现代化的通用 Markdown 编辑领域做出一些贡献。

✨ 特性

支持三种编辑模式：所见即所得（wysiwyg）、即时渲染（ir）、分屏预览（sv）
支持大纲、数学公式、脑图、图表、流程图、甘特图、时序图、五线谱、多媒体、语音阅读、标题锚点、代码高亮及复制、graphviz 渲染、plantumlUML图
导出、图片懒加载、任务列表、多平台预览、多主题切换、复制到微信公众号/知乎功能
实现 CommonMark 和 GFM 规范，可对 Markdown 进行格式化和语法树查看，并支持10+项配置
工具栏包含 36+ 项操作，除支持扩展外还可对每一项中的快捷键、提示、提示位置、图标、点击事件、类名、子工具栏进行自定义
表情/at/话题等自动补全扩展
可使用拖拽、剪切板粘贴上传，显示实时上传进度，支持 CORS 跨域上传
实时保存内容，防止意外丢失
录音支持，用户可直接发布语音
粘贴 HTML 自动转换为 Markdown，如粘贴中包含外链图片可通过指定接口上传到服务器
支持主窗口大小拖拽、字符计数
多主题支持，内置黑白绿三套主题
多语言支持，内置中、英、韩文本地化
支持主流浏览器，对移动端友好

🔮 编辑模式

所见即所得（WYSIWYG）

所见即所得模式对不熟悉 Markdown 的用户较为友好，熟悉 Markdown 的话也可以无缝使用。

即时渲染（IR）

即时渲染模式对熟悉 Typora 的用户应该不会感到陌生，理论上这是最优雅的 Markdown 编辑方式。

分屏预览（SV）

传统的分屏预览模式适合大屏下的 Markdown 编辑。

🍱 语法支持

所有 CommonMark 语法：分隔线、ATX 标题、Setext 标题、缩进代码块、围栏代码块、HTML 块、链接引用定义、段落、块引用、列表、反斜杠转义、HTML 实体、行级代码、强调、加粗、链接、图片、行级 HTML、硬换行、软换行和纯文本。
所有 GFM 语法：表格、任务列表项、删除线、自动链接、XSS 过滤
常用 Markdown 扩展语法：脚注、ToC、自定义标题 ID
图表语法
- 流程图、时序图、甘特图，通过 Mermaid 支持
- Graphviz
- 折线图、饼图、脑图等，通过 ECharts 支持
五线谱：通过 abc.js 支持
数学公式：数学公式块、行级数学公式，通过 MathJax 和 KaTeX 支持
YAML Front Matter
中文语境优化
- 中西文之间插入空格
- 术语拼写修正
- 中文后跟英文逗号句号等标点替换为中文对应标点

以上大部分特性可以通过开关配置是否启用，开发者可根据自己的应用场景选择搭配。

🗃 案例

Sym 一款用 Java 实现的现代化社区（论坛/BBS/社交网络/博客）平台
Solo & Pipe B3log 分布式社区的博客端节点，欢迎加入下一代社区网络
Tditor 基于React、Vditor、Springboot，一款打造极致文字创作体验的在线Markdown编辑平台
Arya 基于 Vue、Vditor，所构建的在线 Markdown 编辑器
更多案例

🛠️ 使用文档

CommonJS

安装依赖

npm install vditor --save

在代码中引入并初始化对象，可参考 index.js

import Vditor from 'vditor'
import "~vditor/src/assets/less/index"

const vditor = new Vditor(id, {options...})

HTML script

在 HTML 中插入 CSS 和 JavaScript，可参考 demo

<!-- ⚠️生产环境请指定版本号，如 https://unpkg.com/vditor@x.x.x/dist... -->
<link rel="stylesheet" href="https://unpkg.com/vditor/dist/index.css" />
<script src="https://unpkg.com/vditor/dist/index.min.js"></script>

示例代码

主题

编辑器主题

编辑器所展现的外观。内置classic，dark 2 套主题。

编辑器初始化时可通过 options.theme 设置内置主题
初始化完成后可通过 setTheme 更新编辑器主题
可通过修改 index.less 中的变量对主题颜色进行定制
可参考现有结构和类名在原有基础上进行修改

内容主题

Markdown 输出的 HTML 所展现的外观。内置 ant-design, light，dark，wechat 4 套主题。支持内容主题扩展接口。

需在显示元素上添加 class="vditor-reset"
编辑器初始化时可通过 options.preview.theme 设置内置或自己开发的主题列表
内容渲染初始化时可通过 IPreviewOptions.theme 设置内置或自己开发的主题
初始化完成后可通过 setTheme 或 setContentTheme 更新内容主题

代码主题

代码块所展现的外观。内置 github 等 36 套主题。

编辑器初始化时可通过 options.preview.hljs 对代码块样式、行号、是否启用进行设置
内容渲染初始化时可通过 IPreviewOptions.hljs 对代码块样式、行号、是否启用进行设置
初始化完成后可通过 setTheme 或 setCodeTheme 更新代码主题

API

id

可填入元素 id 或元素自身 HTMLElement

⚠️：当填入元素自身的 HTMLElement 时需设置 options.cache.id 或将 options.cache.enable 设置为 false

options

	说明	默认值
i18n	多语言，参见 ITips	-
undoDelay	历史记录间隔	-
after	编辑器异步渲染完成后的回调方法	-
height	编辑器总高度	'auto'
minHeight	编辑区域最小高度	-
width	编辑器总宽度，支持 %	'auto'
placeholder	输入区域为空时的提示	''
lang	语言种类：en_US, fr_FR, pt_BR, ja_JP, ko_KR, ru_RU, sv_SE, zh_CN, zh_TW	'zh_CN'
input(value: string)	输入后触发	-
focus(value: string)	聚焦后触发	-
blur(value: string)	失焦后触发	-
keydown(event: KeyboardEvent)	按下后触发	-
esc(value: string)	`esc` 按下后触发	-
ctrlEnter(value: string)	`⌘/ctrl+enter` 按下后触发	-
select(value: string)	编辑器中选中文字后触发	-
tab	`tab` 键操作字符串，支持 `\t` 及任意字符串	-
typewriterMode	是否启用打字机模式	false
cdn	配置自建 CDN 地址	`https://unpkg.com/vditor@${VDITOR_VERSION}`
mode	可选模式：sv, ir, wysiwyg	'ir'
debugger	是否显示日志	false
value	编辑器初始化值	''
theme	主题：classic, dark	'classic'
icon	图标风格：ant, material	'ant'

options.toolbar

工具栏，可使用 name 进行简写： toolbar: ['emoji', 'br', 'bold', '|', 'line'] 。默认值参见 src/ts/util/Options.ts
name 可枚举为： emoji , headings , bold , italic , strike , | , line , quote , list , ordered-list , check ,outdent ,indent , code , inline-code , insert-after , insert-before ,undo , redo , upload , link , table , record , edit-mode , both , preview , fullscreen , outline , code-theme , content-theme , export, devtools , info , help , br
当 name 不在枚举中时，可以添加自定义按钮，格式如下：

new Vditor('vditor', {
  toolbar: [
    {
      hotkey: '⇧⌘S',
      name: 'sponsor',
      tipPosition: 's',
      tip: '成为赞助者',
      className: 'right',
      icon: '<svg t="1589994565028" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="2808" width="32" height="32"><path d="M506.6 423.6m-29.8 0a29.8 29.8 0 1 0 59.6 0 29.8 29.8 0 1 0-59.6 0Z" fill="#0F0F0F" p-id="2809"></path><path d="M717.8 114.5c-83.5 0-158.4 65.4-211.2 122-52.7-56.6-127.7-122-211.2-122-159.5 0-273.9 129.3-273.9 288.9C21.5 562.9 429.3 913 506.6 913s485.1-350.1 485.1-509.7c0.1-159.5-114.4-288.8-273.9-288.8z" fill="#FAFCFB" p-id="2810"></path><path d="M506.6 926c-22 0-61-20.1-116-59.6-51.5-37-109.9-86.4-164.6-139-65.4-63-217.5-220.6-217.5-324 0-81.4 28.6-157.1 80.6-213.1 53.2-57.2 126.4-88.8 206.3-88.8 40 0 81.8 14.1 124.2 41.9 28.1 18.4 56.6 42.8 86.9 74.2 30.3-31.5 58.9-55.8 86.9-74.2 42.5-27.8 84.3-41.9 124.2-41.9 79.9 0 153.2 31.5 206.3 88.8 52 56 80.6 131.7 80.6 213.1 0 103.4-152.1 261-217.5 324-54.6 52.6-113.1 102-164.6 139-54.8 39.5-93.8 59.6-115.8 59.6zM295.4 127.5c-72.6 0-139.1 28.6-187.3 80.4-47.5 51.2-73.7 120.6-73.7 195.4 0 64.8 78.3 178.9 209.6 305.3 53.8 51.8 111.2 100.3 161.7 136.6 56.1 40.4 88.9 54.8 100.9 54.8s44.7-14.4 100.9-54.8c50.5-36.3 108-84.9 161.7-136.6 131.2-126.4 209.6-240.5 209.6-305.3 0-74.9-26.2-144.2-73.7-195.4-48.2-51.9-114.7-80.4-187.3-80.4-61.8 0-127.8 38.5-201.7 117.9-2.5 2.6-5.9 4.1-9.5 4.1s-7.1-1.5-9.5-4.1C423.2 166 357.2 127.5 295.4 127.5z" fill="#141414" p-id="2811"></path><path d="M353.9 415.6m-33.8 0a33.8 33.8 0 1 0 67.6 0 33.8 33.8 0 1 0-67.6 0Z" fill="#0F0F0F" p-id="2812"></path><path d="M659.3 415.6m-33.8 0a33.8 33.8 0 1 0 67.6 0 33.8 33.8 0 1 0-67.6 0Z" fill="#0F0F0F" p-id="2813"></path><path d="M411.6 538.5c0 52.3 42.8 95 95 95 52.3 0 95-42.8 95-95v-31.7h-190v31.7z" fill="#5B5143" p-id="2814"></path><path d="M506.6 646.5c-59.6 0-108-48.5-108-108v-31.7c0-7.2 5.8-13 13-13h190.1c7.2 0 13 5.8 13 13v31.7c0 59.5-48.5 108-108.1 108z m-82-126.7v18.7c0 45.2 36.8 82 82 82s82-36.8 82-82v-18.7h-164z" fill="#141414" p-id="2815"></path><path d="M450.4 578.9a54.7 27.5 0 1 0 109.4 0 54.7 27.5 0 1 0-109.4 0Z" fill="#EA64F9" p-id="2816"></path><path d="M256 502.7a32.1 27.5 0 1 0 64.2 0 32.1 27.5 0 1 0-64.2 0Z" fill="#EFAFF9" p-id="2817"></path><path d="M703.3 502.7a32.1 27.5 0 1 0 64.2 0 32.1 27.5 0 1 0-64.2 0Z" fill="#EFAFF9" p-id="2818"></path></svg>',
      click () {alert('捐赠地址：https://ld246.com/sponsor')},
    }],
})

	说明	默认值
name	唯一标示	-
icon	svg 图标	-
tip	提示	-
tipPosition	提示位置：'n', 'ne', 'nw', 's', 'se', 'sw', 'w', 'e'	-
hotkey	快捷键，格式为`⇧⌘`/`⌘`/`⌥⌘`	-
suffix	插入编辑器中的后缀	-
prefix	插入编辑器中的前缀	-
click(event: Event, vditor: IVditor)	自定义按钮点击时触发的事件	-
className	样式名	''
toolbar?: Array<options.toolbar>	子菜单	-

options.toolbarConfig

	说明	默认值
hide	是否隐藏工具栏	false
pin	是否固定工具栏	false

options.counter

	说明	默认值
enable	是否启用计数器	false
after(length: number, counter: options.counter): void	字数统计回调	-
max	允许输入的最大值	-
type	统计类型：'markdown', 'text'	'markdown'

options.cache

	说明	默认值
enable	是否使用 localStorage 进行缓存	true
id	缓存 key，第一个参数为元素且启用缓存时必填	-
after(html: string): string	缓存后的回调	-

options.comment

⚠️：仅支持 wysiwyg 模式

	说明	默认值
enable	是否启用评论模式	false
add(id: string, text: string, commentsData: ICommentsData[])	添加评论回调	-
remove(ids: string[])	删除评论回调	-
scroll(top: number)	滚动回调	-
adjustTop(commentsData: ICommentsData[])	文档修改时，适配评论高度	-

options.preview

	说明	默认值
delay	预览 debounce 毫秒间隔	1000
maxWidth	预览区域最大宽度	800
mode	显示模式：both, editor	'both'
url	md 解析请求	-
parse(element: HTMLElement)	预览回调	-
transform(html: string): string	渲染之前回调	-

options.preview.hljs

	说明	默认值
defaultLang	未指定语言时默认使用该语言	''
enable	是否启用代码高亮	true
style	可选值参见Chroma	`github`
lineNumber	是否启用行号	false
langs	自定义指定语言	CODE_LANGUAGES

options.preview.markdown

	说明	默认值
autoSpace	自动空格	false
gfmAutoLink	自动链接	true
fixTermTypo	自动矫正术语	false
toc	插入目录	false
footnotes	脚注	true
codeBlockPreview	wysiwyg 和 ir 模式下是否对代码块进行渲染	true
mathBlockPreview	wysiwyg 和 ir 模式下是否对数学公式进行渲染	true
paragraphBeginningSpace	段落开头空两个	false
sanitize	是否启用过滤 XSS	true
listStyle	为列表添加 data-style 属性	false
linkBase	链接相对路径前缀	''
linkPrefix	链接强制前缀	''
mark	启用 mark 标记	false

options.preview.theme

	说明	默认值
current	当前主题	"light"
list	可选主题列表	{ "ant-design": "Ant Design", dark: "Dark", light: "Light", wechat: "WeChat" }
path	主题样式地址	`https://unpkg.com/vditor@${VDITOR_VERSION}/dist/css/content-theme`

options.preview.math

	说明	默认值
inlineDigit	内联数学公式起始 $ 后是否允许数字	false
macros	使用 MathJax 渲染时传入的宏定义	{}
engine	数学公式渲染引擎：KaTeX, MathJax	'KaTeX'
mathJaxOptions	数学公式渲染引擎为 MathJax 时的参数	-

options.preview.actions?: Array<IPreviewAction | IPreviewActionCustom>

默认值为 ["desktop", "tablet", "mobile", "mp-wechat", "zhihu"]。可从默认值中挑选进行配置，也可使用以下字段进行自定制开发。

	说明	默认值
key	按钮唯一标识，不能为空	-
text	按钮文字	-
tooltip	提示	-
className	按钮类名	-
click(key: string)	按钮点击回调事件	-

options.image

	说明	默认值
isPreview	是否预览图片	true
preview(bom: Element) => void	图片预览处理	-

options.link

	说明	默认值
isOpen	是否打开链接地址	true
click(bom: Element) => void	点击链接事件	-

options.hint

	说明	默认值
parse	是否进行 md 解析	true
delay	提示 debounce 毫秒间隔	200
emoji	默认表情，可从lute/emoji_map 中选取，也可自定义	{ '+1': '👍', '-1': '👎', 'heart': '❤️', 'cold_sweat': '😰' }
emojiTail	常用表情提示	-
emojiPath	表情图片地址	`https://unpkg.com/vditor@${VDITOR_VERSION}/dist/images/emoji`
extend: IHintExtend[]	对 @/话题等关键字自动补全的扩展	[]

interface IHintData {
  html: string;
  value: string;
}

interface IHintExtend {
    key: string;

    hint?(value: string): IHintData[] | Promise<IHintData[]>;
}

options.upload

文件上传的数据结构如下。后端返回的数据结构不一致时，可使用 format 进行转换。

// POST data
xhr.send(formData);  // formData = FormData.append("file[]", File)
// return data
{
 "msg": "",
 "code": 0,
 "data": {
 "errFiles": ['filename', 'filename2'],
 "succMap": {
   "filename3": "filepath3",
   "filename3": "filepath3"
   }
 }
}

为了防止站外图片失效， linkToImgUrl 可将剪贴板中的站外图片地址传到服务器端进行保存处理，其数据结构如下：

// POST data
xhr.send(JSON.stringify({url: src})); // src 为站外图片地址
// return data
{
 msg: '',
 code: 0,
 data : {
   originalURL: '',
   url: ''
 }
}

success，format，error 不会同时触发，具体调用情况如下：

if (xhr.status === 200) {
    if (vditor.options.upload.success) {
        vditor.options.upload.success(editorElement, xhr.responseText);
    } else {
        let responseText = xhr.responseText;
        if (vditor.options.upload.format) {
            responseText = vditor.options.upload.format(files as File [], xhr.responseText);
        }
        genUploadedLabel(responseText, vditor);
    }
} else {
    if (vditor.options.upload.error) {
        vditor.options.upload.error(xhr.responseText);
    } else {
        vditor.tip.show(xhr.responseText);
    }
}

	说明	默认值
url	上传 url，为空则不会触发上传相关事件	''
max	上传文件最大 Byte	10 * 1024 * 1024
linkToImgUrl	剪切板中包含图片地址时，使用此 url 重新上传	''
linkToImgCallback(responseText: string)	图片地址上传回调	-
linkToImgFormat(responseText: string): string	对图片地址上传的返回值进行格式化	-
success(editor: HTMLPreElement, msg: string)	上传成功回调	-
error(msg: string)	上传失败回调	-
token	CORS 上传验证，头为 X-Upload-Token	-
withCredentials	跨站点访问控制	false
headers	请求头设置	-
filename(name: string): string	文件名安全处理	name => name.replace(/\W/g, '')
accept	文件上传类型，同input accept	-
validate(files: File[]) => string \| boolean	校验，成功时返回 true 否则返回错误信息	-
handler(files: File[]) => string \| null \| Promise \| Promise	自定义上传，当发生错误时返回错误信息	-
format(files: File[], responseText: string): string	对服务端返回的数据进行转换，以满足内置的数据结构	-
file(files: File[]): File[] \| Promise<File[]>	将上传的文件处理后再返回	-
setHeaders(): { [key: string]: string }	上传前使用返回值设置头	-
extraData: { [key: string]: string \| Blob }	为 FormData 添加额外的参数	-
multiple	上传文件是否为多个	true
fieldName	上传字段名称	'file[]'

options.resize

	说明	默认值
enable	是否支持大小拖拽	false
position	拖拽栏位置：'top', 'bottom'	'bottom'
after(height: number)	拖拽结束的回调	-

options.classes

	说明	默认值
preview	预览元素上的 className	''

options.fullscreen

	说明	默认值
index	全屏层级	90

options.outline

	说明	默认值
enable	初始化是否展现大纲	false
position	大纲位置：'left', 'right'	'left'

methods

	说明
exportJSON(markdown: string)	根据 Markdown 获取对应 JSON
getValue()	获取 Markdown 内容
getHTML()	获取 HTML 内容
insertValue(value: string, render = true)	在焦点处插入内容，并默认进行 Markdown 渲染
focus()	聚焦到编辑器
blur()	让编辑器失焦
disabled()	禁用编辑器
enable()	解除编辑器禁用
getSelection(): string	返回选中的字符串
setValue(markdown: string, clearStack = false)	设置编辑器内容且选中清空历史栈
clearStack()	清空撤销和重做记录栈
renderPreview(value?: string)	设置预览区域内容
getCursorPosition():{top: number, left: number}	获取焦点位置
deleteValue()	删除选中内容
updateValue(value: string)	更新选中内容
isUploading()	上传是否还在进行中
clearCache()	清除缓存
disabledCache()	禁用缓存
enableCache()	启用缓存
html2md(value: string)	HTML 转 md
tip(text: string, time: number)	消息提示。time 为 0 将一直显示
setPreviewMode(mode: "both" \| "editor")	设置预览模式
setTheme(theme: "dark" \| "classic", contentTheme?: string, codeTheme?: string, contentThemePath?: string)	设置主题、内容主题及代码块风格
getCurrentMode(): string	获取编辑器当前编辑模式
destroy()	销毁编辑器
getCommentIds(): {id: string, top: number}[]	获取所有评论
hlCommentIds(ids: string[])	高亮评论
unHlCommentIds(ids: string[])	取消评论高亮
removeCommentIds(removeIds: string[])	删除评论

static methods

不需要进行编辑操作时，仅需引入 method.min.js 后如下直接调用

Vditor.mermaidRender(document)

import VditorPreview from 'vditor/dist/method.min'
VditorPreview.mermaidRender(document)

需要对页面中的 Markdown 进行渲染时可直接调用 preview 方法，参数如下：

previewElement: HTMLDivElement,   // 使用该元素进行渲染
markdown: string,  // 需要渲染的 markdown 原文
options?: IPreviewOptions {
  mode: "dark" | "light";
  anchor?: number;  // 为标题添加锚点 0：不渲染；1：渲染于标题前；2：渲染于标题后，默认 0
  customEmoji?: { [key: string]: string };    // 自定义 emoji，默认为 {}
  lang?: (keyof II18nLang);    // 语言，默认为 'zh_CN'
  emojiPath?: string;    // 表情图片路径
  hljs?: IHljs; // 参见 options.preview.hljs
  speech?: {  // 对选中后的内容进行阅读
    enable?: boolean,
  };
  math?: IMath; // 数学公式渲染配置
  cdn?: string; // 自建 CDN 地址
  transform?(html: string): string; // 在渲染前进行的回调方法
  after?(); // 渲染完成后的回调
  lazyLoadImage?: string; // 设置为 Loading 图片地址后将启用图片的懒加载
  markdown?: options.preview.markdown;
  theme?: options.preview.theme;
  renderers?: ILuteRender; // 自定义渲染 https://ld246.com/article/1588412297062
}

⚠️ method.min.js 和 index.min.js 不可同时引入

	说明
previewImage(oldImgElement: HTMLImageElement, lang: keyof II18n = "zh_CN", theme = "classic")	点击图片预览
mermaidRender(element: HTMLElement, cdn = options.cdn, theme = options.theme)	流程图/时序图/甘特图
flowchartRender(element: HTMLElement, cdn = options.cdn)	flowchart 渲染
codeRender(element: HTMLElement)	为 element 中的代码块添加复制按钮
chartRender(element: (HTMLElement \| Document) = document, cdn = options.cdn, theme = options.theme)	图表渲染
mindmapRender(element: (HTMLElement \| Document) = document, cdn = options.cdn, theme = options.theme)	脑图渲染
plantumlRender(element: (HTMLElement \| Document) = document, cdn = options.cdn)	plantuml 渲染
abcRender(element: (HTMLElement \| Document) = document, cdn = options.cdn)	五线谱渲染
md2html(mdText: string, options?: IPreviewOptions): Promise<string>	Markdown 文本转换为 HTML，该方法需使用异步编程
preview(previewElement: HTMLDivElement, markdown: string, options?: IPreviewOptions)	页面 Markdown 文章渲染
highlightRender(hljsOption?: IHljs, element?: HTMLElement \| Document, cdn = options.cdn)	为 element 中的代码块进行高亮渲染
mediaRender(element: HTMLElement)	为特定链接分别渲染为视频、音频、嵌入的 iframe
mathRender(element: HTMLElement, options?: {cdn?: string, math?: IMath})	对数学公式进行渲染
speechRender(element: HTMLElement, lang?: (keyof II18nLang))	对选中的文字进行阅读
graphvizRender(element: HTMLElement, cdn?: string)	对 graphviz 进行渲染
outlineRender(contentElement: HTMLElement, targetElement: Element)	对大纲进行渲染
lazyLoadImageRender(element: (HTMLElement \| Document) = document)	对启用懒加载的图片进行渲染
setCodeTheme(codeTheme: string, cdn = options.cdn)	设置代码主题，codeTheme 参见 options.preview.hljs.style
setContentTheme(contentTheme: string, path: string)	设置内容主题，contentTheme 参见 options.preview.theme.list

🏗 开发文档

原理相关

环境

安装 node LTS 版本
下载最新代码并解压
根目录运行 npm install
npm run start 启动本地服务器，打开 http://localhost:9000
修改代码
npm run build 打包代码到 dist 目录

CDN 切换

由于使用了按需加载的机制，默认 CDN 为 https://unpkg.com/vditor@版本号

如果代码有修改或需要使用自建 CDN 的话，可按以下步骤进行操作：

初始化时，需对 options 及 IPreviewOptions 中的 cdn，emojiPath, themes 进行配置
highlightRender , mathRender , abcRender , chartRender , mermaidRender， flowchartRender，mindmapRender，graphvizRender，setCodeTheme，setContentTheme 方法中需添加 cdn 参数
将 build 成功的 dist 目录或 jsDelivr 中的 dist 目录拷贝至正确的位置

升级

版本升级时请仔细阅读 CHANGELOG 中的升级部分

Ⓜ️ Markdown 使用指南

🏘️ 社区

📄 授权

Vditor 使用 MIT 开源协议。

🙏 鸣谢

Lute：🎼 一款结构化的 Markdown 引擎，支持 Go 和 JavaScript
highlight.js：JavaScript syntax highlighter
mermaid：Generation of diagram and flowchart from text in a similar manner as Markdown
incubator-echarts：A powerful, interactive charting and visualization library for browser
abcjs：JavaScript library for rendering standard music notation in a browser

📽️ 历史

我们在开发 Sym 的初期是直接使用 WYSIWYG 富文本编辑器的。那时候基于 HTML 的编辑器非常流行，项目中引用起来也很方便，也符合用户当时的使用习惯。

后来，Markdown 的崛起逐步改变了大家的排版方式。再加上我们其他几个项目都是面向程序员用户的，所以迁移到 md 上也是大势所趋。我们选择了 CodeMirror，这是一款优秀的编辑器，它对开发者提供了丰富的编程接口，对各种浏览器的兼容性也比较好。

再后来，随着我们项目业务需求方面的沉淀，使用 CodeMirror 有时候会感到比较“笨重”。比如要实现 @自动完成用户名列表、插入 Emoji、上传文件等就需要比较深入的二次开发，而这些业务需求恰恰是很多项目场景共有且必备的。

终于，我们决定开始在 Sym 中自己实现编辑器。随着几个版本的迭代，Sym 的编辑器也日趋成熟。在我们运营的社区链滴上陆续有人问我们是否能将编辑器单独抽离出来提供给大家使用。与此同时，我们的前端主程 V 同学对于维护分散在各个项目中的编辑器也感到有点力不从心，外加对 TypeScript 的好感，所以就决定使用 ts 来实现一个全新的浏览器端 md 编辑器。

于是，Vditor 就这样诞生了。

piyushgarg-dev/review-app-api

2023-10-10T01:42:12Z

⚡ Empowering JavaScript with native platform APIs. ✨ Best of all worlds (TypeScript, Swift, Objective C, Kotlin, Java). Use what you love ❤️ Angular, Capacitor, Ionic, React, Solid, Svelte, Vue with: SwiftUI, Jetpack Compose, Flutter and you name it compatible.

NativeScript empowers you to access native APIs from JavaScript directly. Currently iOS and Android runtimes are provided for rich mobile development across a variety of diverse use cases.

Contribute

Setup your local development environment
Clone to contribute:

$ git clone https://github.com/NativeScript/NativeScript.git
$ cd NativeScript

# setup workspace for development
$ npm run setup

# list all available commands to run
$ npm start

We love you and your pull requests 🤗. Please follow our contributing guide and see our code of governance to become as involved as you want to be.

@nativescript/*

@nativescript/core Core iOS/Android for NativeScript.
@nativescript/types Types for both iOS/Android below wrapped up as a convenience. Most commonly used.
@nativescript/types-ios Types for iOS.
@nativescript/types-android Types for Android.
@nativescript/types-minimal A very minimal set of types for only the latest Android and iOS sdks. Most commonly used to optimize Web-based IDE's which auto load all type declarations from node_modules.
@nativescript/ui-mobile-base UI mobile base native classes used by core.
@nativescript/webpack Webpack build utilities and configs used by NativeScript apps.

Other source repos

Outside the source centralized in this repo, NativeScript consists of a few other source repos. Here are the major ones:

iOS Runtime NativeScript iOS Runtime: Empowers JavaScript code to be executed on iOS devices written in a mix of C++, Objective-C, and Swift.
Android Runtime NativeScript Android Runtim: Empowers JavaScript code to be executed on Android devices written in a mix of C++, Java and Kotlin.
CLI NativeScript command-line interface empowering you to create, build, and run apps using NativeScript.
Docs NativeScript documentation available at http://docs.nativescript.org/ written in Markdown.
Plugins A plugin workspace with several useful plugins.
Firebase Modular Firebase 🔥 implementation for supported platforms.
ML Kit Google's ML Kit SDKs for supported platforms.
Payments In-App Purchase, Subscriptions, Google Pay and Apple Pay.
Artwork Want to use our logo or colors? Feel free to use any of our ready-to-use media material.

Made with ❤️

github/how-engineering-communicates

2023-10-10T01:23:49Z

LLMs interview notes and answers:该仓库主要记录大模型（LLMs）算法工程师相关的面试题和参考答案

grananqvist/Awesome-Quant-Machine-Learning-Trading

2023-10-10T01:23:49Z

Quant/Algorithm trading resources with an emphasis on Machine Learning

chao325/MaoTai_GUIT

2023-10-11T01:23:29Z

JD京东抢购、京东抢茅台Windows端、开箱即用无需配置环境。开发在即（开源协议采用Apache License）抢茅台外挂，茅台脚本

burmajam/dotfiles

2023-10-11T01:42:33Z

Mine dotfiles

instant-markdown/vim-instant-markdown

2023-10-11T01:42:33Z

Instant Markdown previews from Vim

Wizarrrr/wizarr

2023-10-11T01:42:43Z

Wizarr is an advanced user invitation and management system for Jellyfin, Plex, Emby etc.

tr1ckydev/chameleon

2023-10-10T01:43:50Z

Language server for GLSL (autocomplete, goto-definition, formatter, and more)

GitHub 1C Enterprise Daily Trending

microsoft/promptflow

👨‍💼 Headshot AI - Professional Headshots with AI

How It Works

Running Locally

One-Click Deploy

How To Get Good Results

Additional Use-Cases

Contributing

Resources and Support

License

microsoft/ToRA

ToRA: A Tool-Integrated Reasoning Agent

🔥 News

💡 Introduction

Tool-Integrated Reasoning

ToRA Training Pipeline

🚀 Quick Start

⚙️ Setup

🪁 Inference

⚖️ Evaluation

⚡️ Training

☕️ Citation

🍀 Contributing

🌟 Star History

chatchat-space/Langchain-Chatchat

目录

介绍

环境最低要求

变更日志

0.2.0 版本与 0.1.x 版本区别

模型支持

LLM 模型支持

Embedding 模型支持

Text Splitter 个性化支持

Agent生态

基础的Agent

构建自己的Agent工具

Docker 部署

开发部署

软件需求

1. 开发环境准备

2. 下载模型至本地

3. 设置配置项

4. 知识库初始化与迁移

5. 一键启动 API 服务或 Web UI

5.1 启动命令

5.2 启动非默认模型

5.3 多卡加载

5.4 PEFT 加载(包括lora,p-tuning,prefix tuning, prompt tuning,ia3等)

5.5 注意事项：

5.6 启动界面示例：

常见问题

路线图

项目交流群

关注我们

deaaprizal/unofficial-trust

segiqt/cs2-ahk-hack

xwxbug/autoit-cn

shuhongfan/NavicatCracker

XuanchenLin/NanUI

The WinFormium Project(A.K.A. NanUI)

WinFormium

⭐ About

🖥️ Requirements

🧰 Getting Started

📖 Documentation

🤖 Demos

🔗 Third-Party References & Tools

🏆 Inspirations

microsoft/typechat.net

TypeChat.NET

Assemblies

Microsoft.TypeChat

Microsoft.TypeChat.Program

Microsoft.TypeChat.SemanticKernel

Getting Started

Prerequisite: OpenAI

Building

Nuget Packages

`0.2.0` 版本与 `0.1.x` 版本区别

The WinFormium Project
(A.K.A. NanUI)