GitHub 1C Enterprise Daily Trending

rahulsend89/jigsawXml

2023-09-10T01:48:05Z

Actionscript 2.0 base jigsaw puzzle test Game.

sugazaki/twilight

2023-09-10T01:48:05Z

Beyamor/vikes-invaders

2023-09-10T01:48:05Z

A single-day jam of a Space Invaders clone

aras-p/UnityGaussianSplatting

2023-09-10T01:25:41Z

Toy Gaussian Splatting visualization in Unity

Toy Gaussian Splatting playground in Unity

SIGGRAPH 2023 had a paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering" by Kerbl, Kopanas, Leimkühler, Drettakis that looks pretty cool! Check out their website, source code repository, data sets and so on.

I've decided to try to implement the realtime visualization part (i.e. the one that takes already-produced gaussian splat "model" file) in Unity.

The original paper code has a purely CUDA-based realtime renderer; other people have done their own implementations (e.g. WebGPU at cvlab-epfl, Taichi at wanmeihuali, etc.).

Code in here so far is randomly cribbled together from reading the paper (as well as earlier literature on EWA splatting), looking at the official CUDA implementation, and so on. Current state:

The code does not use the "tile-based splat rasterizer" bit from the paper; it just draws each gaussian splat as a screenspace aligned rectangle that covers the extents of it.
Splat color accumulation is done by rendering front-to-back, with a blending mode that results in the same accumulated color as their tile-based renderer.
Splat sorting is done with a AMD FidelityFX derived radix sort, or (on DX11) with a GPU bitonic sort that is lifted from Unity HDRP codebase.

This is not a fast implementation yet!

Usage

Within Unity (2022.3), there's a Scene.unity that has a GaussianSplatRenderer script attached to it.
- The project defaults to DX12 on Windows, since then it can use a faster GPU sorting routine. DX11 should also work, at expense of performance.
- Metal and Vulkan also use the faster sorting approach.
You need to point it to a "model" directory. The model directory is expected to contain cameras.json and point_cloud/iteration_7000/point_cloud.ply inside of it.
- Since the models are quite large, I have not included any in this Github repo. The original paper github page has a a link to 14GB zip of their models.
Press play.
- The gaussian splat renderer component inspector will have a slider to move the game view camera into one of the cameras from the model directory. Or you can just move the game/scene view camera however you please.
- There are also various controls in the script to debug or visualize the data.

⚠ Note: this is all a toy, it is not robust, it does not handle errors gracefully, it does not interact or composite well with the "rest of rendering", it is not fast, etc. etc. I told you so!

Wishlist that I may or might not do at some point:

Make rendering faster (actual tiled compute shader rasterizer)
Look at ways to make the data sets smaller (both on-disk and in-memory)
Integrate better with "the rest" of rendering that might be in the scene (BiRP)
Maybe look at making it work in URP/HDRP? Not sure yet
Make sorting faster (bitonic -> FidelityFX radix sort)

Write-ups

My own blog posts about all this (so far... not that many!):

Gaussian Splatting is pretty cool! (2023 Sep)

Performance numbers:

"bicycle" scene from the paper, with 6.1M splats and first camera in there, rendering at 1200x800 resolution:

Windows (NVIDIA RTX 3080 Ti):
- Official SBIR viewer: 7.4ms (135FPS). 4.8GB VRAM usage.
- Unity, DX12 or Vulkan: 13.4ms (75FPS) - 10.1ms rendering, 3.3ms sorting. 2.1GB VRAM usage.
- Unity, DX11: 21.8ms (46FPS) - 9.9ms rendering, 11.9ms sorting.
Mac (Apple M1 Max):
- Unity, Metal: 80.6ms (12FPS) - with FidelityFX GPU sort.
- Unity, Metal: 108ms (9FPS) - with bitonic GPU sort.

External Code Used

zanders3/json, MIT license, (c) 2018 Alex Parker.
"Island" GPU sorting code adapted from Tim Gfrerer blog post.
"Ffx" GPU sorting code is AMD FidelityFX ParallelSort, ported to Unity by me.

bradtraversy/traversy-js-challenges

2023-09-10T01:25:41Z

Challenges & docs from JS Algorithms & Data Structures course

Traversy JS Challenges: Data Structures & Algorithms

This is the sandbox for my 70+ JS Challenges: Data Structures & Algorithms Course. Just about all of the concepts that are included, also exist in other languages. So, if you are not a JavaScript developer, you can still follow along and learn from this course/repo.

This course/repo goes over everything from basic loop challenges, high order array methods, recursion, time & space complexity, data structures such as stacks, queues, linked lists, trees, graphs, and we touch on sorting algorithms like bubble sort, insertion and merge sort. You should already know the basics of JavaScript before doing any of the challenges or taking the course.

Please do not make any PRs to this repo as it goes along with a specific course. I may open another one for student solutions and new challenges.

File Structure

Each folder includes:

readme.md - The challenge/code instructions. This also includes hints, tests and a dropdown with the solution code as well as the explanation of the solution code.
[name].js - This is your working file. It has the name of the function and the function is exported. No parameters are passed to the function. That is up to you to add.
[name]-run.js - File to run the code manually. The function is already imported and called with expected parameters.
[name]-solution.js The solution code with heavy commenting. Some challenges have multiple solutions. If you want the solution without comments, look in the readme.md file.
[name]-test.js - Jest tests for the solution code. You will need to rename this file to [name].test.js to run the tests.

Learning Modules/Lessons

Some lessons/modules are not challenges, they are more like mini-lessons. I don't just throw you a challenge using a new concept (Trees, Stacks, Bubble Sorts, etc) without explaining it first. I try to explain the concept and then give you a challenge to practice or implement it. So some folders will not have a challenge, just a readme file.

Running Tests

In order for the Jest tests to run, you need to rename the test file to [name].test.js. For example, if you are working on the hello-world challenge, you need to rename the hello-world-test.js file to hello-world.test.js. This is because Jest looks for files with the .test.js extension.

Run the command npm run test from the root directory and it will run all the tests.

These Don't Have to be "Challenges"

Some people, such as myself are not great at doing this stuff off the top of their head. Even though most of the code is setup as a challenge, you can certainly just follow along with the course and/or just study the solutions and learn from them.

You can use the tests to see if your code passes, but use the run files to run the code manually. This is so that you can experiment, console.log, etc.

Getting Started

Clone the repo
Run npm install
Run npm run test to run the tests. Again, you will need to rename the test files and replace the -test with .test to run the tests.

Index of Challenges/Lessons

01. Basic Challenges 1

These are mostly challenges that have to do with loops, conditionals, and string manipulation. I do not go over fundamentals like "what is a for loop". You should already know the basics of JavaScript.

02. Basic Challenges 2

These are more challenges that have to do with iteration. They are slightly harder than the first set of challenges.

03. High Order Array Methods

The next set of challenges/lessons will have to do with high order array methods such as map, filter, reduce, sort, etc. Even though most of these can be done with a for loop, I want you to practice using these methods.

04. Recursion

The next batch of challenges/lessons will have to do with recursion. We will first talk about what recursion is and then we can look at some challenges.

05. Complexity

This is more of a learning section than a challenge section. We will talk about Big O notation and how to calculate the time complexity of an algorithm. We will also talk about space complexity and how to calculate that as well. We will talk about the different types of complexity such as constant, linear, quadratic, etc.

06. Hash Tables, Maps & Sets

In this section, we will start to look at data structures. We will start with a data structure called a hash table. This will include maps and sets, which are built-in JavaScript data structures that are similar to hash tables. We will also create a custom hash table class and use it in a couple challenges.

07. Stacks, Queues & Linked Lists

In this section, we will look at working with data structures such as stacks, queues, and linked lists. We will also look at fast and slow pointers.

08. Binary Trees & Binary Search Trees & Graphs

In this section, we will look at trees and graphs. We will start with binary trees and binary search trees. We will also look at graphs and graph traversal.

09. Sorting Algorithms

In this section, we will get into sorting algorithms. We will start with bubble sort, which is very popular in interviews. We will also look at selection sort, insertion sort, merge sort, and quick sort.

baichuan-inc/Baichuan2

2023-09-10T01:25:41Z

A series of large language models developed by Baichuan Intelligent Technology

Baichuan 2

🤗 Hugging Face • 🤖 ModelScope • 💬 WeChat

🚀 百川53B大模型在线对话平台已正式向公众开放 🎉

中文 | English

模型介绍

Baichuan 2 是百川智能推出的新一代开源大语言模型，采用 2.6 万亿 Tokens 的高质量语料训练。
Baichuan 2 在多个权威的中文、英文和多语言的通用、领域 benchmark 上取得同尺寸最佳的效果。
本次发布包含有 7B、13B 的 Base 和 Chat 版本，并提供了 Chat 版本的 4bits 量化。
所有版本对学术研究完全开放。同时，开发者通过邮件申请并获得官方商用许可后，即可免费商用，请参考协议章节。
欢迎阅读我们的技术报告 Baichuan 2: Open Large-scale Language Models 获取更多信息。

本次发布版本和下载链接见下表：

	基座模型	对齐模型	对齐模型 4bits 量化
7B	🤗 Baichuan2-7B-Base	🤗 Baichuan2-7B-Chat	🤗 Baichuan2-7B-Chat-4bits
13B	🤗 Baichuan2-13B-Base	🤗 Baichuan2-13B-Chat	🤗 Baichuan2-13B-Chat-4bits

Benchmark 结果

我们在通用、法律、医疗、数学、代码和多语言翻译六个领域的中英文和多语言权威数据集上对模型进行了广泛测试。

通用领域

在通用领域我们在以下数据集上进行了 5-shot 测试。

C-Eval 是一个全面的中文基础模型评测数据集，涵盖了 52 个学科和四个难度的级别。我们使用该数据集的 dev 集作为 few-shot 的来源，在 test 集上进行测试。我们采用了 Baichuan-7B 的评测方案。
MMLU 是包含 57 个任务的英文评测数据集，涵盖了初等数学、美国历史、计算机科学、法律等，难度覆盖高中水平到专家水平，是目前主流的 LLM 评测数据集。我们采用了开源的评测方案。
CMMLU 是一个包含 67 个主题的综合性性中文评估基准，专门用于评估语言模型在中文语境下的知识和推理能力。我们采用了其官方的评测方案。
Gaokao 是一个以中国高考题作为评测大语言模型能力的数据集，用以评估模型的语言能力和逻辑推理能力。我们只保留了其中的单项选择题，并进行了随机划分。我们采用了与 C-Eval 类似的评测方案。
AGIEval 旨在评估模型的认知和解决问题相关的任务中的一般能力。我们只保留了其中的四选一单项选择题，并进行了随机划分。我们采用了与 C-Eval 类似的评测方案。
BBH 是一个挑战性任务 Big-Bench 的子集。Big-Bench 目前包括 204 项任务。任务主题涉及语言学、儿童发展、数学、常识推理、生物学、物理学、社会偏见、软件开发等方面。BBH 是从 204 项 Big-Bench 评测基准任务中大模型表现不好的任务单独拿出来形成的评测基准。

7B 模型结果

	C-Eval	MMLU	CMMLU	Gaokao	AGIEval	BBH
	5-shot	5-shot	5-shot	5-shot	5-shot	3-shot
GPT-4	68.40	83.93	70.33	66.15	63.27	75.12
GPT-3.5 Turbo	51.10	68.54	54.06	47.07	46.13	61.59
LLaMA-7B	27.10	35.10	26.75	27.81	28.17	32.38
LLaMA2-7B	28.90	45.73	31.38	25.97	26.53	39.16
MPT-7B	27.15	27.93	26.00	26.54	24.83	35.20
Falcon-7B	24.23	26.03	25.66	24.24	24.10	28.77
ChatGLM2-6B	50.20	45.90	49.00	49.44	45.28	31.65
Baichuan-7B	42.80	42.30	44.02	36.34	34.44	32.48
Baichuan2-7B-Base	54.00	54.16	57.07	47.47	42.73	41.56

13B 模型结果

	C-Eval	MMLU	CMMLU	Gaokao	AGIEval	BBH
	5-shot	5-shot	5-shot	5-shot	5-shot	3-shot
GPT-4	68.40	83.93	70.33	66.15	63.27	75.12
GPT-3.5 Turbo	51.10	68.54	54.06	47.07	46.13	61.59
LLaMA-13B	28.50	46.30	31.15	28.23	28.22	37.89
LLaMA2-13B	35.80	55.09	37.99	30.83	32.29	46.98
Vicuna-13B	32.80	52.00	36.28	30.11	31.55	43.04
Chinese-Alpaca-Plus-13B	38.80	43.90	33.43	34.78	35.46	28.94
XVERSE-13B	53.70	55.21	58.44	44.69	42.54	38.06
Baichuan-13B-Base	52.40	51.60	55.30	49.69	43.20	43.01
Baichuan2-13B-Base	58.10	59.17	61.97	54.33	48.17	48.78

法律、医疗

法律领域我们使用了 JEC-QA 数据集。JEC-QA 数据集来源于中国国家司法考试。我们只保留了其中的单选题。我们采用了与 C-Eval 类似的评测方案。

医疗领域则使用通用领域数据集（C-Eval、MMLU、CMMLU）中的医学相关学科、MedQA 和 MedMCQA。我们采用了与 C-Eval 类似的评测方案。

为了测试方便，我们使用了 C-Eval 的 val 集进行测试。
MedQA 数据集来源于美国、中国的医学考试。我们测试了 MedQA数据集中的 USMLE 和 MCMLE 两个子集，并采用了五个候选的版本。
MedMCQA 数据集来源于印度医学院的入学考试。我们只保留了其中的单选题。由于 test 集没有答案，我们使用 dev 集进行测试。
通用领域数据集包含的医学相关学科如下：
- C-Eval: clinical_medicine, basic_medicine
- MMLU: clinical_knowledge, anatomy, college_medicine, college_biology, nutrition, virology, medical_genetics, professional_medicine
- CMMLU: anatomy, clinical_knowledge, college_medicine, genetics, nutrition, traditional_chinese_medicine, virology

我们对以上数据集进行了 5-shot 测试。

7B 模型结果

	JEC-QA	CEval-MMLU-CMMLU	MedQA-USMLE	MedQA-MCMLE	MedMCQA
	5-shot	5-shot	5-shot	5-shot	5-shot
GPT-4	59.32	77.16	80.28	74.58	72.51
GPT-3.5 Turbo	42.31	61.17	53.81	52.92	56.25
LLaMA-7B	27.45	33.34	24.12	21.72	27.45
LLaMA2-7B	29.20	36.75	27.49	24.78	37.93
MPT-7B	27.45	26.67	16.97	19.79	31.96
Falcon-7B	23.66	25.33	21.29	18.07	33.88
ChatGLM2-6B	40.76	44.54	26.24	45.53	30.22
Baichuan-7B	34.64	42.37	27.42	39.46	31.39
Baichuan2-7B-Base	44.46	56.39	32.68	54.93	41.73

13B 模型结果

	JEC-QA	CEval-MMLU-CMMLU	MedQA-USMLE	MedQA-MCMLE	MedMCQA
	5-shot	5-shot	5-shot	5-shot	5-shot
GPT-4	59.32	77.16	80.28	74.58	72.51
GPT-3.5 Turbo	42.31	61.17	53.81	52.92	56.25
LLaMA-13B	27.54	35.14	28.83	23.38	39.52
LLaMA2-13B	34.08	47.42	35.04	29.74	42.12
Vicuna-13B	28.38	40.99	34.80	27.67	40.66
Chinese-Alpaca-Plus-13B	35.32	46.31	27.49	32.66	35.87
XVERSE-13B	46.42	58.08	32.99	58.76	41.34
Baichuan-13B-Base	41.34	51.77	29.07	43.67	39.60
Baichuan2-13B-Base	47.40	59.33	40.38	61.62	42.86

数学、代码

数学领域我们使用 OpenCompass 评估框架，对 GSM8K 和 MATH 数据集进行了 4-shot 测试。

GSM8K 是由 OpenAI 发布的一个由 8.5K 高质量的语言多样化的小学数学应用题组成的数据集，要求根据给定的场景和两个可能的解决方案，选择最合理的方案。
MATH 数据集包含 12,500 个数学问题（其中 7500 个属于训练集，5000 个属于测试集），这些问题收集自 AMC 10、AMC 12、AIME 等数学竞赛。

代码领域则采用了 HumanEval 和 MBPP 数据集。我们使用 OpenCompass，对 HumanEval 进行了 0-shot 测试，MBPP 数据集进行了 3-shot 测试。

HumanEval 中的编程任务包括模型语言理解、推理、算法和简单数学，以评估模型功能正确性，并衡量模型的问题解决能力。
MBPP 包括 974 个 Python 短函数、程序的文字描述以及用于检查功能正确性的测试用例的数据集。

7B 模型结果

	GSM8K	MATH	HumanEval	MBPP
	4-shot	4-shot	0-shot	3-shot
GPT-4	89.99	40.20	69.51	63.60
GPT-3.5 Turbo	57.77	13.96	52.44	61.40
LLaMA-7B	9.78	3.02	11.59	14.00
LLaMA2-7B	16.22	3.24	12.80	14.80
MPT-7B	8.64	2.90	14.02	23.40
Falcon-7B	5.46	1.68	-	10.20
ChatGLM2-6B	28.89	6.40	9.15	9.00
Baichuan-7B	9.17	2.54	9.20	6.60
Baichuan2-7B-Base	24.49	5.58	18.29	24.20

13B 模型结果

	GSM8K	MATH	HumanEval	MBPP
	4-shot	4-shot	0-shot	3-shot
GPT-4	89.99	40.20	69.51	63.60
GPT-3.5 Turbo	57.77	13.96	52.44	61.40
LLaMA-13B	20.55	3.68	15.24	21.40
LLaMA2-13B	28.89	4.96	15.24	27.00
Vicuna-13B	28.13	4.36	16.46	15.00
Chinese-Alpaca-Plus-13B	11.98	2.50	16.46	20.00
XVERSE-13B	18.20	2.18	15.85	16.80
Baichuan-13B-Base	26.76	4.84	11.59	22.80
Baichuan2-13B-Base	52.77	10.08	17.07	30.20

多语言翻译

我们采用了 Flores-101 数据集来评估模型的多语言能力。Flores-101 涵盖了世界各地的 101 种语言。它的数据来源于新闻、旅游指南和书籍等多个不同领域。我们选择了联合国官方语言（阿拉伯文、中文、英文、法文、俄文和西班牙文）以及德文和日文作为测试语种。我们使用 OpenCompass 对 Flores-101 中的中-英、中-法、中-西班牙、中-阿拉伯、中-俄、中-日、中-德等七个子任务分别进行了 8-shot 测试。

7B 模型结果

	CN-EN	CN-FR	CN-ES	CN-AR	CN-RU	CN-JP	CN-DE	Average
GPT-4	29.94	29.56	20.01	10.76	18.62	13.26	20.83	20.43
GPT-3.5 Turbo	27.67	26.15	19.58	10.73	17.45	1.82	19.70	17.59
LLaMA-7B	17.27	12.02	9.54	0.00	4.47	1.41	8.73	7.63
LLaMA2-7B	25.76	15.14	11.92	0.79	4.99	2.20	10.15	10.14
MPT-7B	20.77	9.53	8.96	0.10	3.54	2.91	6.54	7.48
Falcon-7B	22.13	15.67	9.28	0.11	1.35	0.41	6.41	7.91
ChatGLM2-6B	22.28	9.42	7.77	0.64	1.78	0.26	4.61	6.68
Baichuan-7B	25.07	16.51	12.72	0.41	6.66	2.24	9.86	10.50
Baichuan2-7B-Base	27.27	20.87	16.17	1.39	11.21	3.11	12.76	13.25

13B 模型结果

	CN-EN	CN-FR	CN-ES	CN-AR	CN-RU	CN-JP	CN-DE	Average
GPT-4	29.94	29.56	20.01	10.76	18.62	13.26	20.83	20.43
GPT-3.5 Turbo	27.67	26.15	19.58	10.73	17.45	1.82	19.70	17.59
LLaMA-13B	21.75	16.16	13.29	0.58	7.61	0.41	10.66	10.07
LLaMA2-13B	25.44	19.25	17.49	1.38	10.34	0.13	11.13	12.17
Vicuna-13B	22.63	18.04	14.67	0.70	9.27	3.59	10.25	11.31
Chinese-Alpaca-Plus-13B	22.53	13.82	11.29	0.28	1.52	0.31	8.13	8.27
XVERSE-13B	29.26	24.03	16.67	2.78	11.61	3.08	14.26	14.53
Baichuan-13B-Base	30.24	20.90	15.92	0.98	9.65	2.64	12.00	13.19
Baichuan2-13B-Base	30.61	22.11	17.27	2.39	14.17	11.58	14.53	16.09

推理和部署

推理所需的模型权重、源码、配置已发布在 Hugging Face，下载链接见本文档最开始的表格。我们在此示范多种推理方式。程序会自动从 Hugging Face 下载所需资源。

安装依赖

pip install -r requirements.txt

Python 代码方式

Chat 模型推理方法示范

>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> from transformers.generation.utils import GenerationConfig
>>> tokenizer = AutoTokenizer.from_pretrained("baichuan-inc/Baichuan2-13B-Chat", use_fast=False, trust_remote_code=True)
>>> model = AutoModelForCausalLM.from_pretrained("baichuan-inc/Baichuan2-13B-Chat", device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True)
>>> model.generation_config = GenerationConfig.from_pretrained("baichuan-inc/Baichuan2-13B-Chat")
>>> messages = []
>>> messages.append({"role": "user", "content": "解释一下“温故而知新”"})
>>> response = model.chat(tokenizer, messages)
>>> print(response)
"温故而知新"是一句中国古代的成语，出自《论语·为政》篇。这句话的意思是：通过回顾过去，我们可以发现新的知识和理解。换句话说，学习历史和经验可以让我们更好地理解现在和未来。

这句话鼓励我们在学习和生活中不断地回顾和反思过去的经验，从而获得新的启示和成长。通过重温旧的知识和经历，我们可以发现新的观点和理解，从而更好地应对不断变化的世界和挑战。

Base 模型推理方法示范

>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("baichuan-inc/Baichuan2-13B-Base", trust_remote_code=True)
>>> model = AutoModelForCausalLM.from_pretrained("baichuan-inc/Baichuan2-13B-Base", device_map="auto", trust_remote_code=True)
>>> inputs = tokenizer('登鹳雀楼->王之涣\n夜雨寄北->', return_tensors='pt')
>>> inputs = inputs.to('cuda:0')
>>> pred = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.1)
>>> print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
登鹳雀楼->王之涣
夜雨寄北->李商隐

在上述两段代码中，模型加载指定 device_map='auto'，会使用所有可用显卡。如需指定使用的设备，可以使用类似 export CUDA_VISIBLE_DEVICES=0,1（使用了0、1号显卡）的方式控制。

命令行工具方式

python cli_demo.py

本命令行工具是为 Chat 场景设计，因此我们不支持使用该工具调用 Base 模型。

网页 demo 方式

依靠 streamlit 运行以下命令，会在本地启动一个 web 服务，把控制台给出的地址放入浏览器即可访问。本网页 demo 工具是为 Chat 场景设计，因此我们不支持使用该工具调用 Base 模型。

streamlit run web_demo.py

Baichuan2-13B-Chat 示例输出

代码

用户：找到下列数组的中位数[3.1,6.2,1.3,8.4,10.5,11.6,2.1]，请用python代码完成以上功能

Baichuan 2：def find_median(numbers):
    numbers.sort()
    length = len(numbers)
    if length % 2 == 0:
        return (numbers[length // 2 - 1] + numbers[length // 2]) / 2
    else:
        return numbers[length // 2]
numbers = [3.1, 6.2, 1.3, 8.4, 10.5, 11.6, 2.1]
print(find_median(numbers))

数学

用户：鸡和兔在一个笼子里，共有26个头，68只脚，那么鸡有多少只，兔有多少只？

Baichuan 2：我们可以用代数方法来解决这个问题。假设鸡的数量为x，兔子的数量为y。根据题目给出的信息，我们可以得到以下两个方程：

1. x + y = 26 (头的总数)
2. 2x + 4y = 68 (脚的总数)

现在我们需要解这个二元一次方程组。首先从第一个方程中解出x：

x = 26 - y

然后将x的表达式代入第二个方程：

2(26 - y) + 4y = 68
52 - 2y + 4y = 68
2y = 16

接下来我们求解y的值：

y = 16 / 2
y = 8

现在我们得到了兔子的数量是8只。将y的值代入x的表达式求得鸡的数量：

x = 26 - 8
x = 18

所以鸡有18只，兔子有8只。

指令跟随

用户：从现在开始，请你将我接下来说的每一句话翻译成英语，准备好了吗？

Baichuan 2：I'm ready to translate every word you say next into English.

用户：你好

Baichuan 2：Hello

用户：我是Baichuan 2，很高兴认识大家

Baichuan 2：I am Baichuan 2, and I am very happy to meet everyone.

用户：希望通过大家的共同努力，早日迎接AGI时代的到来

Baichuan 2：I hope through the joint efforts of all of us, we can welcome the arrival of the AGI era as soon as possible.

解题

用户：以下物理常识题目，哪一个是错误的?A.在自然环境下，声音在固体中传播速度最快。B.牛顿第一定律:一个物体如果不受力作用，将保持静止或匀速直线运动的状态。C.牛顿第三定律:对于每个作用力，都有一个相等而反向的反作用力。D.声音在空气中的传播速度为1000m/s。

Baichuan 2：本题考查物理基本概念和原理的理解和应用能力。
A项：声音的传播需要介质，在固体、液体和气体的三种介质中，声音在固体中的传播速度是最快的，故A正确； B项：牛顿第一定律是惯性定律，其内容是：一个物体如果没有受到外力的作用，要么保持静止状态，要么保持匀速直线运动状态，直到有外力迫使它改变运动状态为止，故B正确； C项：牛顿第三定律的内容是：对于每一个作用力，都有一个大小相等、方向相反的反作用力，这两个力分别作用在两个相互作用的物体上，故C错误； D项：声音在空气中以波的形式传播，其传播速度与介质的温度、湿度等因素有关，一般情况下，声音在空气中的传播速度约为340m/s左右，而不是1000m/s，故D错误。 因此，本题的正确答案是 D 。

用户：I'm Mike 。I am going to have a busy weekend。On Saturday，I am going to learn how to swim。I will go with my father。Then we are going to have lunch in the restaurant。In the afternoon，we are going to the zoo。We are going to see the pandas。On Sunday，I will finish my homework in the morning。In the afternoon，my parents and I are going to visit my grandparents。We will have dinner together。
阅读短文，判断正确和错误！正确请写T，错误请写F:
1.Mike is going to have a busy weekend()
2.Mike is going to learn how to swim with his father()
3.They are going to have lunch at home()

Baichuan 2：1. T（正确的）
2. T（正确的）
3. F（错误的）

量化部署

为了让不同的用户以及不同的平台都能运行 Baichuan 2 模型，我们针对 Baichuan 2 模型做了相应地量化工作（包括 Baichuan2-7B-Chat 和 Baichuan2-13B-Chat），方便用户快速高效地在自己的平台部署 Baichuan 2 模型。

量化方法

Baichuan 2 的采用社区主流的量化方法：BitsAndBytes。该方法可以保证量化后的效果基本不掉点，目前已经集成到 transformers 库里，并在社区得到了广泛应用。BitsAndBytes 支持 8bits 和 4bits 两种量化，其中 4bits 支持 FP4 和 NF4 两种格式，Baichuan 2 选用 NF4 作为 4bits 量化的数据类型。

基于该量化方法，Baichuan 2 支持在线量化和离线量化两种模式。

在线量化

对于在线量化，我们支持 8bits 和 4bits 量化，使用方式和 Baichuan-13B 项目中的方式类似，只需要先加载模型到 CPU 的内存里，再调用quantize()接口量化，最后调用 cuda()函数，将量化后的权重拷贝到 GPU 显存中。实现整个模型加载的代码非常简单，我们以 Baichuan2-7B-Chat 为例：

8bits 在线量化:

model = AutoModelForCausalLM.from_pretrained("baichuan-inc/Baichuan2-7B-Chat", torch_dtype=torch.float16, trust_remote_code=True)
model = model.quantize(8).cuda()

4bits 在线量化:

model = AutoModelForCausalLM.from_pretrained("baichuan-inc/Baichuan2-7B-Chat", torch_dtype=torch.float16, trust_remote_code=True)
model = model.quantize(4).cuda()

需要注意的是，在用 from_pretrained 接口的时候，用户一般会加上 device_map="auto"，在使用在线量化时，需要去掉这个参数，否则会报错。

离线量化

为了方便用户的使用，我们提供了离线量化好的 4bits 的版本 Baichuan2-7B-Chat-4bits，供用户下载。用户加载 Baichuan2-7B-Chat-4bits 模型很简单，只需要执行:

model = AutoModelForCausalLM.from_pretrained("baichuan-inc/Baichuan2-7B-Chat-4bits", device_map="auto", trust_remote_code=True)

对于 8bits 离线量化，我们没有提供相应的版本，因为 Hugging Face transformers 库提供了相应的 API 接口，可以很方便的实现 8bits 量化模型的保存和加载。用户可以自行按照如下方式实现 8bits 的模型保存和加载：

# Model saving: model_id is the original model directory, and quant8_saved_dir is the directory where the 8bits quantized model is saved.
model = AutoModelForCausalLM.from_pretrained(model_id, load_in_8bit=True, device_map="auto", trust_remote_code=True)
model.save_pretrained(quant8_saved_dir)
model = AutoModelForCausalLM.from_pretrained(quant8_saved_dir, device_map="auto", trust_remote_code=True)

量化效果

量化前后显存占用对比 (GPU Mem in GB)：

Precision	Baichuan2-7B	Baichuan2-13B
bf16 / fp16	15.3	27.5
8bits	8.0	16.1
4bits	5.1	8.6

量化后在各个 benchmark 上的结果和原始版本对比如下：

Model 5-shot	C-Eval	MMLU	CMMLU
Baichuan2-13B-Chat	56.74	57.32	59.68
Baichuan2-13B-Chat-4bits	56.05	56.24	58.82
Baichuan2-7B-Chat	54.35	52.93	54.99
Baichuan2-7B-Chat-4bits	53.04	51.72	52.84

C-Eval 是在其 val set 上进行的评测

可以看到，4bits 相对 bfloat16 精度损失在 1 - 2 个百分点左右。

CPU 部署

Baichuan 2 模型支持 CPU 推理，但需要强调的是，CPU 的推理速度相对较慢。需按如下方式修改模型加载的方式：

# Taking Baichuan2-7B-Chat as an example
model = AutoModelForCausalLM.from_pretrained("baichuan-inc/Baichuan2-7B-Chat", torch_dtype=torch.float32, trust_remote_code=True)

对 Baichuan 1 的推理优化迁移到 Baichuan 2

由于很多用户在 Baichuan 1 (Baichuan-7B, Baichuan-13B)上做了很多优化的工作，例如编译优化、量化等，为了将这些工作零成本地应用于 Baichuan 2，用户可以对 Baichuan 2 模型做一个离线转换，转换后就可以当做 Baichuan 1 模型来使用。具体来说，用户只需要利用以下脚本离线对 Baichuan 2 模型的最后一层 lm_head 做归一化，并替换掉lm_head.weight即可。替换完后，就可以像对 Baichuan 1 模型一样对转换后的模型做编译优化等工作了。

import torch
import os
ori_model_dir = 'your Baichuan 2 model directory'
# To avoid overwriting the original model, it's best to save the converted model to another directory before replacing it
new_model_dir = 'your normalized lm_head weight Baichuan 2 model directory'
model = torch.load(os.path.join(ori_model_dir, 'pytorch_model.bin'))
lm_head_w = model['lm_head.weight']
lm_head_w = torch.nn.functional.normalize(lm_head_w)
model['lm_head.weight'] = lm_head_w
torch.save(model, os.path.join(new_model_dir, 'pytorch_model.bin'))

模型微调

依赖安装

git clone https://github.com/baichuan-inc/Baichuan2.git
cd Baichuan2/fine-tune
pip install -r requirements.txt

如需使用 LoRA 等轻量级微调方法需额外安装 peft
如需使用 xFormers 进行训练加速需额外安装 xFormers

单机训练

下面我们给一个微调 Baichuan2-7B-Base 的单机训练例子。

训练数据：data/belle_chat_ramdon_10k.json，该样例数据是从 multiturn_chat_0.8M 采样出 1 万条，并且做了格式转换。主要是展示多轮数据怎么训练，不保证效果。

hostfile=""
deepspeed --hostfile=$hostfile fine-tune.py  \
    --report_to "none" \
    --data_path "data/belle_chat_ramdon_10k.json" \
    --model_name_or_path "baichuan-inc/Baichuan2-7B-Base" \
    --output_dir "output" \
    --model_max_length 512 \
    --num_train_epochs 4 \
    --per_device_train_batch_size 16 \
    --gradient_accumulation_steps 1 \
    --save_strategy epoch \
    --learning_rate 2e-5 \
    --lr_scheduler_type constant \
    --adam_beta1 0.9 \
    --adam_beta2 0.98 \
    --adam_epsilon 1e-8 \
    --max_grad_norm 1.0 \
    --weight_decay 1e-4 \
    --warmup_ratio 0.0 \
    --logging_steps 1 \
    --gradient_checkpointing True \
    --deepspeed ds_config.json \
    --bf16 True \
    --tf32 True

多机训练

多机训练只需要给一下 hostfile ，内容类似如下：

ip1 slots=8
ip2 slots=8
ip3 slots=8
ip4 slots=8
....

同时在训练脚本里面指定 hosftfile 的路径：

hostfile="/path/to/hostfile"
deepspeed --hostfile=$hostfile fine-tune.py  \
    --report_to "none" \
    --data_path "data/belle_chat_ramdon_10k.json" \
    --model_name_or_path "baichuan-inc/Baichuan2-7B-Base" \
    --output_dir "output" \
    --model_max_length 512 \
    --num_train_epochs 4 \
    --per_device_train_batch_size 16 \
    --gradient_accumulation_steps 1 \
    --save_strategy epoch \
    --learning_rate 2e-5 \
    --lr_scheduler_type constant \
    --adam_beta1 0.9 \
    --adam_beta2 0.98 \
    --adam_epsilon 1e-8 \
    --max_grad_norm 1.0 \
    --weight_decay 1e-4 \
    --warmup_ratio 0.0 \
    --logging_steps 1 \
    --gradient_checkpointing True \
    --deepspeed ds_config.json \
    --bf16 True \
    --tf32 True

轻量化微调

代码已经支持轻量化微调如 LoRA，如需使用仅需在上面的脚本中加入以下参数：

--use_lora True

LoRA 具体的配置可见 fine-tune.py 脚本。

使用 LoRA 微调后可以使用下面的命令加载模型：

from peft import AutoPeftModelForCausalLM
model = AutoPeftModelForCausalLM.from_pretrained("output", trust_remote_code=True)

中间 Checkpoints

除了训练了 2.6 万亿 Tokens 的 Baichuan2-7B-Base 模型，我们还提供了在此之前的另外 11 个中间 checkpoints（分别对应训练了约 0.2 ~ 2.4 万亿 Tokens）供社区研究使用（下载地址）。下图给出了这些 checkpoints 在 C-Eval、MMLU、CMMLU 三个 benchmark 上的效果变化：

社区与生态

📢📢📢 我们会在此持续更新社区和生态对 Baichuan 2 的支持 😀😀😀

华为昇腾

Pytorch 框架

模型微调：Baichuan 2 支持基于昇腾 NPU 的 PyTorch + DeepSpeed 模型微调，微调所需的 modeling、README、示例脚本已发布：Baichuan2-7B、Baichuan2-13B 正在适配中。

推理部署：Baichuan 2 支持昇腾 NPU 推理，推理所需的 modeling、README、示例脚本已发布：Baichuan2-7B、Baichuan2-13B。

MindSpore 框架

MindFormers 是一个基于昇思框架（MindSpore）并支持大模型训练、微调、评估、推理、部署的全流程开发套件，Baichuan2-7B / 13B 已集成于此套件，支持用户进行模型微调、部署，具体使用方式可见 README。

大模型体验平台

昇思大模型平台基于昇思 MindSpore AI 框架、MindFormers 大模型开发套件与昇腾硬件算力，将 Baichuan2-7B 大模型能力开放给公众，欢迎大家在线体验。

声明与协议

声明

我们在此声明，我们的开发团队并未基于 Baichuan 2 模型开发任何应用，无论是在 iOS、Android、网页或任何其他平台。我们强烈呼吁所有使用者，不要利用 Baichuan 2 模型进行任何危害国家社会安全或违法的活动。另外，我们也要求使用者不要将 Baichuan 2 模型用于未经适当安全审查和备案的互联网服务。我们希望所有的使用者都能遵守这个原则，确保科技的发展能在规范和合法的环境下进行。

我们已经尽我们所能，来确保模型训练过程中使用的数据的合规性。然而，尽管我们已经做出了巨大的努力，但由于模型和数据的复杂性，仍有可能存在一些无法预见的问题。因此，如果由于使用 Baichuan 2 开源模型而导致的任何问题，包括但不限于数据安全问题、公共舆论风险，或模型被误导、滥用、传播或不当利用所带来的任何风险和问题，我们将不承担任何责任。

协议

对本仓库源码的使用遵循开源许可协议 Apache 2.0。对 Baichuan 2 模型的社区使用需遵循《Baichuan 2 模型社区许可协议》。Baichuan 2 支持商用。如果将 Baichuan 2 模型或其衍生品用作商业用途，请您通过邮箱 opensource@baichuan-inc.com 联系许可方，申请书面授权。

KillianLucas/open-interpreter

2023-09-10T01:47:40Z

OpenAI's Code Interpreter in your terminal, running locally

● Open Interpreter

Let language models run code on your computer.
An open-source, locally running implementation of OpenAI's Code Interpreter.

Get early access to the desktop application.

pip install open-interpreter

interpreter

Open Interpreter lets LLMs run code (Python, Javascript, Shell, and more) locally. You can chat with Open Interpreter through a ChatGPT-like interface in your terminal by running $ interpreter after installing.

This provides a natural-language interface to your computer's general-purpose capabilities:

Create and edit photos, videos, PDFs, etc.
Control a Chrome browser to perform research
Plot, clean, and analyze large datasets
...etc.

⚠️ Note: You'll be asked to approve code before it's run.

Demo

https://github.com/KillianLucas/open-interpreter/assets/63927363/37152071-680d-4423-9af3-64836a6f7b60

An interactive demo is also available on Google Colab:

Quick Start

pip install open-interpreter

Terminal

After installation, simply run interpreter:

interpreter

Python

import interpreter

interpreter.chat("Plot AAPL and META's normalized stock prices") # Executes a single command
interpreter.chat() # Starts an interactive chat

Comparison to ChatGPT's Code Interpreter

OpenAI's release of Code Interpreter with GPT-4 presents a fantastic opportunity to accomplish real-world tasks with ChatGPT.

However, OpenAI's service is hosted, closed-source, and heavily restricted:

No internet access.
Limited set of pre-installed packages.
100 MB maximum upload, 120.0 second runtime limit.
State is cleared (along with any generated files or links) when the environment dies.

Open Interpreter overcomes these limitations by running on your local environment. It has full access to the internet, isn't restricted by time or file size, and can utilize any package or library.

This combines the power of GPT-4's Code Interpreter with the flexibility of your local development environment.

Commands

Interactive Chat

To start an interactive chat in your terminal, either run interpreter from the command line:

interpreter

Or interpreter.chat() from a .py file:

interpreter.chat()

Programmatic Chat

For more precise control, you can pass messages directly to .chat(message):

interpreter.chat("Add subtitles to all videos in /videos.")

# ... Streams output to your terminal, completes task ...

interpreter.chat("These look great but can you make the subtitles bigger?")

# ...

Start a New Chat

In Python, Open Interpreter remembers conversation history. If you want to start fresh, you can reset it:

interpreter.reset()

Save and Restore Chats

interpreter.chat() returns a List of messages when return_messages=True, which can be used to resume a conversation with interpreter.load(messages):

messages = interpreter.chat("My name is Killian.", return_messages=True) # Save messages to 'messages'
interpreter.reset() # Reset interpreter ("Killian" will be forgotten)

interpreter.load(messages) # Resume chat from 'messages' ("Killian" will be remembered)

Customize System Message

You can inspect and configure Open Interpreter's system message to extend its functionality, modify permissions, or give it more context.

interpreter.system_message += """
Run shell commands with -y so the user doesn't have to confirm them.
"""
print(interpreter.system_message)

Change the Model

ⓘ Issues running locally? Read our new GPU setup guide and Windows setup guide.

You can run interpreter in local mode from the command line to use Code Llama:

interpreter --local

For gpt-3.5-turbo, use fast mode:

interpreter --fast

In Python, you will need to set the model manually:

interpreter.model = "gpt-3.5-turbo"

Azure Support

To connect to an Azure deployment, the --use-azure flag will walk you through setting this up:

interpreter --use-azure

In Python, set the following variables:

interpreter.use_azure = True
interpreter.api_key = "your_openai_api_key"
interpreter.azure_api_base = "your_azure_api_base"
interpreter.azure_api_version = "your_azure_api_version"
interpreter.azure_deployment_name = "your_azure_deployment_name"
interpreter.azure_api_type = "azure"

Debug mode

To help contributors inspect Open Interpreter, --debug mode is highly verbose.

You can activate debug mode by using it's flag (interpreter --debug), or mid-chat:

$ interpreter
...
> %debug # <- Turns on debug mode

Configuration with .env

Open Interpreter allows you to set default behaviors using a .env file. This provides a flexible way to configure the interpreter without changing command-line arguments every time.

Here's a sample .env configuration:

INTERPRETER_CLI_AUTO_RUN=False
INTERPRETER_CLI_FAST_MODE=False
INTERPRETER_CLI_LOCAL_RUN=False
INTERPRETER_CLI_DEBUG=False
INTERPRETER_CLI_USE_AZURE=False

INTERPRETER_CLI_AUTO_RUN: If set to True, the interpreter will execute code without user confirmation.
INTERPRETER_CLI_FAST_MODE: If set to True, the interpreter will use gpt-3.5-turbo instead of gpt-4.
INTERPRETER_CLI_LOCAL_RUN: If set to True, the interpreter will run fully locally with Code Llama.
INTERPRETER_CLI_DEBUG: If set to True, the interpreter will print extra debugging information.
INTERPRETER_CLI_USE_AZURE: If set to True, the interpreter will use Azure OpenAI Services.

You can modify these values in the .env file to change the default behavior of the Open Interpreter.

Safety Notice

Since generated code is executed in your local environment, it can interact with your files and system settings, potentially leading to unexpected outcomes like data loss or security risks.

⚠️ Open Interpreter will ask for user confirmation before executing code.

You can run interpreter -y or set interpreter.auto_run = True to bypass this confirmation, in which case:

Be cautious when requesting commands that modify files or system settings.
Watch Open Interpreter like a self-driving car, and be prepared to end the process by closing your terminal.
Consider running Open Interpreter in a restricted environment like Google Colab or Replit. These environments are more isolated, reducing the risks associated with executing arbitrary code.

How Does it Work?

Open Interpreter equips a function-calling language model with an exec() function, which accepts a language (like "python" or "javascript") and code to run.

We then stream the model's messages, code, and your system's outputs to the terminal as Markdown.

Contributing

Thank you for your interest in contributing! We welcome involvement from the community.

Please see our Contributing Guidelines for more details on how to get involved.

License

Open Interpreter is licensed under the MIT License. You are permitted to use, copy, modify, distribute, sublicense and sell copies of the software.

Note: This software is not affiliated with OpenAI.

Having access to a junior programmer working at the speed of your fingertips ... can make new workflows effortless and efficient, as well as open the benefits of programming to new audiences.

— OpenAI's Code Interpreter Release

binwiederhier/ntfy

2023-09-10T01:47:40Z

Send push notifications to your phone or desktop using PUT/POST

ntfy.sh | Send push notifications to your phone or desktop via PUT/POST

ntfy (pronounced "notify") is a simple HTTP-based pub-sub notification service. With ntfy, you can send notifications to your phone or desktop via scripts from any computer, without having to sign up or pay any fees. If you'd like to run your own instance of the service, you can easily do so since ntfy is open source.

You can access the free version of ntfy at ntfy.sh. There is also an open source Android app available on Google Play or F-Droid, as well as an open source iOS app available on the App Store.

ntfy Pro 💸 🎉

I now offer paid plans for ntfy.sh if you don't want to self-host, or you want to support the development of ntfy (→ Purchase via web app). You can buy a plan for as low as $3.33/month (if you use promo code MYTOPIC, limited time only). You can also donate via GitHub Sponsors, and Liberapay. I would be very humbled by your sponsorship. ❤️

Documentation

Getting started | Android/iOS | API | Install / Self-hosting | Building

Chat / forum

There are a few ways to get in touch with me and/or the rest of the community. Feel free to use any of these methods. Whatever works best for you:

Discord server - direct chat with the community
Matrix room #ntfy (+ Matrix space) - same chat, bridged from Discord
Lemmy discussion board - asynchronous forum (new as of June 2023)
GitHub issues - questions, features, bugs

Announcements / beta testers

For announcements of new releases and cutting-edge beta versions, please subscribe to the ntfy.sh/announcements topic. If you'd like to test the iOS app, join TestFlight. For Android betas, join Discord/Matrix (I'll eventually make a testing channel in Google Play).

Contributing

I welcome any and all contributions. Just create a PR or an issue. For larger features/ideas, please reach out on Discord/Matrix first to see if I'd accept them. To contribute code, check out the build instructions for the server and the Android app. Or, if you'd like to help translate 🇩🇪 🇺🇸 🇧🇬, you can start immediately in Hosted Weblate.

Code of Conduct

We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.

Please be sure to read the complete Code of Conduct.

License

Made with ❤️ by Philipp C. Heckel.
The project is dual licensed under the Apache License 2.0 and the GPLv2 License.

Third party libraries and resources:

github.com/urfave/cli (MIT) is used to drive the CLI
Mixkit sounds (Mixkit Free License) are used as notification sounds
Sounds from notificationsounds.com (Creative Commons Attribution) are used as notification sounds
Roboto Font (Apache 2.0) is used as a font in everything web
React (MIT) is used for the web app
Material UI components (MIT) are used in the web app
MUI dashboard template (MIT) was used as a basis for the web app
Dexie.js (Apache 2.0) is used for web app persistence in IndexedDB
GoReleaser (MIT) is used to create releases
go-smtp (MIT) is used to receive e-mails
stretchr/testify (MIT) is used for unit and integration tests
github.com/mattn/go-sqlite3 (MIT) is used to provide the persistent message cache
Firebase Admin SDK (Apache 2.0) is used to send FCM messages
github/gemoji (MIT) is used for emoji support (specifically the emoji.json file)
Lightbox with vanilla JS as a lightbox on the landing page
HTTP middleware for gzip compression (MIT) is used for serving static files
Regex for auto-linking (MIT) is used to highlight links (the library is not used)
Statically linking go-sqlite3
Linked tabs in mkdocs
webpush-go (MIT) is used to send web push notifications

modularml/mojo

2023-09-10T01:47:40Z

The Mojo Programming Language

Welcome to Mojo 🔥

Mojo is a new programming language that bridges the gap between research and production by combining Python syntax and ecosystem with systems programming and metaprogramming features. Mojo is still young, but it is designed to become a superset of Python over time.

We plan to open-source Mojo progressively over time, but it's changing very quickly now. We believe that a small, tight-knit group of engineers with a shared vision can move faster than a community effort, so we will continue to incubate it within Modular until it's more complete. Please see the Mojo FAQ for more information about this and other common questions.

We've opened this repo now because we want to gather issues and engage in feedback from users who have access to the Mojo Playground (our hosted JupyterHub where you can try coding with an early version of Mojo). To get access to the Mojo Playground, see here to sign up. Then, when you want to report issues or request features, please create a GitHub issue here.

For more general questions or to chat with other Mojo developers, check out our Discord.

Otherwise, you can:

Read the inspiration behind Mojo.
Check out the Mojo programming manual.
Read our other docs on docs.modular.com/mojo.

infi-nl/example-ableton-live-project

2023-09-10T01:48:13Z

Een template project ter ondersteuning van https://infi.nl/nieuws/versiebeheer-voor-ableton-live-projecten-met-git/

jotego/jtbin

2023-09-10T01:48:26Z

Binary files for MiSTerFPGA, Pocket and other platforms

rilozos/CMSI-2210_KehoePickarGevorgyan

2023-09-10T01:26:39Z

Repo for CMSI2210 Comp Sys Org Fall 2023

allegheny-college-cmpsc-200-fall-2023/single-pile-nim-assignment

2023-09-10T01:26:39Z

Repository for Lab 1: Single-Pile Nim assignment and accompanying activities.

stefangrage/koebenhavnerbilleder.dk

2023-09-10T01:26:42Z

nixcon/2023.nixcon.org

2023-09-11T01:26:39Z

NixCon 2023 conference website

manuelernestog/astropod

2023-09-10T01:48:35Z

Free and open source self-hosting serverless podcast solution

camptocamp/puppet-haproxy_c2c

2023-09-10T01:48:43Z

Puppet module for managing haproxy

sxzxs/Real-time-translation-typing

2023-09-10T01:48:42Z

实时打字翻译软件

SlejmUr/Manifest_Tool_TB

2023-09-10T01:48:54Z

Throwback Manifest Tool

minigueimeiros/office-365-cmd

2023-09-10T01:48:54Z

Ativador Office 365

hmcts/pet-container-app-infra

2023-09-10T01:27:05Z

cedev935/OpenChat

2023-09-11T01:27:00Z

FahimAnzamDip/triangle-pos

2023-09-10T01:49:06Z

Triangle POS is an open source Inventory Management with POS System. Developed with Laravel 10, Bootstrap 4 & Livewire 2.0. It's completely free to use.

aras-p/UnityGaussianSplatting

2023-09-10T01:27:34Z

Toy Gaussian Splatting visualization in Unity

Toy Gaussian Splatting playground in Unity

I've decided to try to implement the realtime visualization part (i.e. the one that takes already-produced gaussian splat "model" file) in Unity.

The original paper code has a purely CUDA-based realtime renderer; other people have done their own implementations (e.g. WebGPU at cvlab-epfl, Taichi at wanmeihuali, etc.).

Code in here so far is randomly cribbled together from reading the paper (as well as earlier literature on EWA splatting), looking at the official CUDA implementation, and so on. Current state:

The code does not use the "tile-based splat rasterizer" bit from the paper; it just draws each gaussian splat as a screenspace aligned rectangle that covers the extents of it.
Splat color accumulation is done by rendering front-to-back, with a blending mode that results in the same accumulated color as their tile-based renderer.
Splat sorting is done with a AMD FidelityFX derived radix sort, or (on DX11) with a GPU bitonic sort that is lifted from Unity HDRP codebase.

This is not a fast implementation yet!

Usage

Within Unity (2022.3), there's a Scene.unity that has a GaussianSplatRenderer script attached to it.
- The project defaults to DX12 on Windows, since then it can use a faster GPU sorting routine. DX11 should also work, at expense of performance.
- Metal and Vulkan also use the faster sorting approach.
You need to point it to a "model" directory. The model directory is expected to contain cameras.json and point_cloud/iteration_7000/point_cloud.ply inside of it.
- Since the models are quite large, I have not included any in this Github repo. The original paper github page has a a link to 14GB zip of their models.
Press play.
- The gaussian splat renderer component inspector will have a slider to move the game view camera into one of the cameras from the model directory. Or you can just move the game/scene view camera however you please.
- There are also various controls in the script to debug or visualize the data.

⚠ Note: this is all a toy, it is not robust, it does not handle errors gracefully, it does not interact or composite well with the "rest of rendering", it is not fast, etc. etc. I told you so!

Wishlist that I may or might not do at some point:

Make rendering faster (actual tiled compute shader rasterizer)
Look at ways to make the data sets smaller (both on-disk and in-memory)
Integrate better with "the rest" of rendering that might be in the scene (BiRP)
Maybe look at making it work in URP/HDRP? Not sure yet
Make sorting faster (bitonic -> FidelityFX radix sort)

Write-ups

My own blog posts about all this (so far... not that many!):

Gaussian Splatting is pretty cool! (2023 Sep)

Performance numbers:

"bicycle" scene from the paper, with 6.1M splats and first camera in there, rendering at 1200x800 resolution:

Windows (NVIDIA RTX 3080 Ti):
- Official SBIR viewer: 7.4ms (135FPS). 4.8GB VRAM usage.
- Unity, DX12 or Vulkan: 13.4ms (75FPS) - 10.1ms rendering, 3.3ms sorting. 2.1GB VRAM usage.
- Unity, DX11: 21.8ms (46FPS) - 9.9ms rendering, 11.9ms sorting.
Mac (Apple M1 Max):
- Unity, Metal: 80.6ms (12FPS) - with FidelityFX GPU sort.
- Unity, Metal: 108ms (9FPS) - with bitonic GPU sort.

External Code Used

zanders3/json, MIT license, (c) 2018 Alex Parker.
"Island" GPU sorting code adapted from Tim Gfrerer blog post.
"Ffx" GPU sorting code is AMD FidelityFX ParallelSort, ported to Unity by me.

MixedRealityToolkit/MixedRealityToolkit-Unity

2023-09-10T01:27:34Z

Extremely fast tween library implemented in Unity ECS

Magic Tween

Extremely fast tween library implemented in Unity ECS

日本語版READMEはこちら

Overview

Magic Tween is a high-performance tweening library implemented in Unity Entity Component System (ECS).

In addition to powerful tweening functionality compatible with traditional components, it also offers even higher-performance APIs for ECS.

Features

High-performance tweening library implemented in ECS.
Add extension methods compatible with many components.
Tween anything with Tween.To.
Apply various settings using method chaining.
Create complex animations with Sequences.
Add processing via callbacks.
Support for tweening TextMesh Pro.
Support for conversion to Observable with UniRx.
Support for async/await with UniTask.
Even higher-performance implementation for ECS.

Performance

When tweening float values of regular classes using Tween.To(), Magic Tween operates more than 2 to 5 times faster than other libraries.

When tweening float values within ECS components using Tween.Entity.To(), it is possible to achieve even higher performance.

For more details on performance, please refer to the README in the MagicTween.Benchmarks project.

Samples

The MagicTween.Samples project includes several samples implemented using Magic Tween. For more details, please refer to the README in the project.

Setup

Requirement

Unity 2022.1 or higher
Entities 1.0.0 or higher
Burst 1.8.8 or higher

Install

Open the Package Manager from Window > Package Manager
"+" button > Add package from git URL
Enter the following to install

https://github.com/AnnulusGames/MagicTween.git?path=/MagicTween/Assets/MagicTween

or open Packages/manifest.json and add the following to the dependencies block.

{
    "dependencies": {
        "com.annulusgames.magic-tween": "https://github.com/AnnulusGames/MagicTween.git?path=/MagicTween/Assets/MagicTween"
    }
}

Basic Usage

By introducing Magic Tween, numerous extension methods for creating tweens on traditional Unity components are added. Below is an example of animating the position of a Transform using these extension methods:

// Move from the current position to (1, 2, 3) over 5 seconds
transform.TweenPosition(
    new Vector3(1f, 2f, 3f), // Target value
    5f // Duration of the change
);

// Move from (0, 0, 0) to (1, 2, 3) over 5 seconds
transform.TweenPosition(
    new Vector3(0f, 0f, 0f), // Starting value
    new Vector3(1f, 2f, 3f), // Target value
    5f // Duration of the change
);

Warning Do not play multiple tweens on the same parameter simultaneously. This can lead to unexpected behavior due to overlapping value changes. (Tweens on different axes, such as TweenPositionX and TweenPositionY, will work.)

Warning Creating tweens in the editor is not supported.

Extension Methods

Magic Tween provides extension methods for most components included in Unity. These extension methods allow for more concise and optimized code compared to Tween.To(). It's recommended to use extension methods when available.

You can find a list of available extension methods here (a wiki is currently being created).

Classification

The extension methods added for tweening are classified into several categories:

Method Name	Behavior
Tween...	Creates a tween for a specific field/property.
Punch...	Creates a tween that vibrates the value of a specific field/property.
Shake...	Creates a tween that randomly shakes the value of a specific field/property.
Set...	Adds settings to customize the tween's behavior.
On...	Adds callbacks at specific points during the tween.

Tweening Custom Value

You can animate custom value using the Tween.To() method:

float foo;

float endValue = 10f;
float duration = 2f;

Tween.To(
    () => foo,
    x => foo = x,
    endValue,
    duration
);

If you want to use only values without defining variables, you can use Tween.FromTo():

float startValue = 0f;
float endValue = 10f;
float duration = 2f;

Tween.FromTo(
    x => Debug.Log("current value: " + x),
    startValue,
    endValue,
    duration
);

You can create tweens that follow a curve passing through multiple points using Tween.Path():

Vector3 foo;

Vector3[] points;
float duration = 2f;

Tween.Path(
    () => foo,
    x => foo = x,
    points,
    duration
);

Avoiding Allocations

The To and FromTo methods mentioned above cause allocations because they capture external variables. To reduce unnecessary allocations when performing tweens on an object, you can specify the target object as the first argument to avoid lambda expression allocations:

// A class with a field named 'foo'
ExampleClass target;

float endValue = 10f;
float duration = 2f;

// Pass the object as the first argument to avoid allocations
Tween.To(
    target,
    obj => obj.foo,
    (obj, x) => obj.foo = x,
    endValue,
    duration
);

Tween Control

In general, created tweens play and are destroyed automatically, so there's no need for explicit control. However, there are situations where manual control is necessary, such as when dealing with tweens that loop indefinitely. In such cases, you can control tweens through the Tween struct:

Tween tween = transform.TweenPosition(Vector3.up, 2f);

// Start/resume the tween
tween.Play();

// Pause the tween
tween.Pause();

// Restart the tween from the beginning
tween.Restart();

// Complete the tween
tween.Complete();

// Kill the tween
tween.Kill();

// Complete and kill the tween
tween.CompleteAndKill();

You can also perform operations on all playing tweens collectively. If you specify an ID as an argument, you can target only the tweens with matching IDs:

// Kill all playing tweens
Tween.KillAll();

// Complete all tweens with an ID of 1
Tween.CompleteAll(1);

// Pause all tweens with an ID of "Alpha"
Tween.PauseAll("Alpha");

Getting Tween Information

You can check if a tween is currently active using IsActive(). When performing operations on tweens, it's a good practice to check for activity, especially when there's a possibility that the tween may not be active:

// Kill the tween if it's active
if (tween.IsActive()) tween.Kill();

You can also retrieve the duration of a tween using GetDuration():

float duration = tween.GetDuration();

Adding Settings

You can customize the behavior of a tween using the Set methods. These methods can be chained together for concise code. The following code is an example of applying custom settings to a tween:

transform.TweenLocalScale(Vector3.one * 2f, 5f)
    .SetEase(Ease.OutSine) // Set the easing function to OutSine
    .SetLoops(3, LoopType.Restart) // Repeat 3 times with restart behavior
    .SetDelay(1f); // Delay the start by 1 second

SetEase

Sets the easing function to use for the tween. You can also use your own easing function by passing an AnimationCurve.

SetLoops

Sets the number of times the tween should loop. By default, it's set to 1. You can create a tween that loops infinitely by setting it to -1. You can also specify the loop behavior using the second argument, LoopType.

SetPlaybackSpeed

Sets the playback speed of the tween. The default is 1, and negative values are not supported.

SetDelay

Sets a delay in seconds before the tween starts.

SetIgnoreTimeScale

Ignores the effect of TimeScale.

SetRelative

Sets the end value as a relative value from the start value.

SetInvert

Swaps the start and end values. You can adjust the behavior using the InvertMode.

InvertMode	Behavior
InvertMode.None	Moves from the start value to the end value as usual.
InvertMode.Immediate	Moves to the end value as soon as the tween starts and then transitions towards the start value.
InvertMode.AfterDelay	Waits until the tween starts, then moves to the end value and transitions towards the start value.

SetId

Assigns an ID to the tween. This allows you to operate on tweens with the same ID in bulk when performing operations like KillAll. You can pass an int or a string with a length of 32 bytes or less as an ID (default is int 0 or an empty string).

SetLink

Links the tween's lifecycle to a GameObject. You can change the behavior by setting the LinkBehaviour as the second argument. However, regardless of the option set, Kill will be called on OnDestroy.

SetAutoPlay

Sets whether the tween should automatically play (default is true). If set to false, you need to manually call Play() to start the tween.

SetAutoKill

Sets whether the tween should automatically be killed at the end (default is true). If set to false, you need to manually call Kill() to remove the tween. This option is useful when you want to reuse the same tween multiple times.

SetFrequency (Punch, Shake)

Available options for Punch and Shake tweens to set the frequency of vibration (default is 10).

SetDampingRatio (Punch, Shake)

Available options for Punch and Shake tweens to set the damping ratio of vibration. A value of 1 will completely dampen the vibration at the end, and a value of 0 will result in no damping (default is 1).

SetRandomSeed (Shake)

Available option for Shake tweens to set the seed value for the random numbers used in vibration. This option must be applied before playback.

SetPathType (Path)

Available option for Path-based tweens to set how the points are connected.

PathType	Behavior
PathType.Linear	Connects each point using a straight line.
PathType.CatmullRom	Connects each point using a Catmull-Rom spline curve.

SetClosed (Path)

Available option for Path-based tweens to set whether the path is closed, allowing it to return to the starting point.

SetRoundingMode (int, int2, int3, int4, long)

Sets the rounding mode for decimal values. This option is applicable only to integer-based types.

RoundingMode	Behavior
RoundingMode.ToEven	Default setting. Rounds the value to the nearest integer, and if the value is midway, it rounds to the nearest even integer.
RoundingMode.AwayFromZero	Rounds the value to the nearest integer, and if the value is midway, it rounds away from zero.
RoundingMode.ToZero	Rounds the value towards zero.
RoundingMode.ToPositiveInfinity	Rounds the value towards positive infinity.
RoundingMode.ToNegativeInfinity	Rounds the value towards negative infinity.

SetScrambleMode (string)

Allows you to fill unrevealed characters with random characters. This option is only applicable to string tweens.

ScrambleMode	Behavior
ScrambleMode.None	Default setting. Nothing is displayed for unrevealed parts.
ScrambleMode.Uppercase	Fills unrevealed parts with random uppercase letters.
ScrambleMode.Lowercase	Fills unrevealed parts with random lowercase letters.
ScrambleMode.Numerals	Fills unrevealed parts with random numbers.
ScrambleMode.All	Fills unrevealed parts with random uppercase letters, lowercase letters, or numbers.
(ScrambleMode.Custom)	Fills unrevealed parts with random numbers from the specified string. This option cannot be explicitly set and is determined when passing a string as an argument to SetScrambleMode.

SetRichTextEnabled (string)

Enables RichText support, allowing text with RichText tags to be animated with proper character advancement. This option is only applicable to string tweens.

Callbacks

When you want to perform some actions at specific times, such as the start or completion of a tween, you can use the On-series methods. Callback methods, like other settings, can be written using method chaining.

transform.TweenPosition(new Vector3(1f, 2f, 3f), 5f)
    .SetLoops(5)
    .OnUpdate(() => Debug.Log("update"))
    .OnStepComplete(() => Debug.Log("step complete"))
    .OnComplete(() => Debug.Log("complete"));

Note When you set up any callback, the first callback creates an allocation for generating the component. Additionally, enabling one or more callbacks can reduce performance during playback. In most cases, the impact on performance is minimal, but it's recommended to avoid using callbacks when creating a large number of tweens.

OnPlay

Called when the tween starts playing. Unlike OnStart, it ignores delays set with SetDelay and is also called if Play is invoked after a pause.

OnStart

Called when the tween begins its operation. If a delay is set with SetDelay, it is called after the delay has passed.

OnUpdate

Called every frame during the tween's playback.

OnStepComplete

Called at the end of each loop when SetLoops is configured.

OnComplete

Called when the tween is completed.

OnKill

Called when the tween is killed.

DelayedCall / Empty

You can create a tween that performs a specified action after a certain delay using Tween.DelayedCall().

// Display a log after 3 seconds
Tween.DelayedCall(3f, () => Debug.Log("delayed call"));

Additionally, you can create an empty tween using Tween.Empty().

// A tween that completes after 3 seconds
Tween.Empty(3f);

// DelayedCall() internally calls the following code
Tween.Empty(3f)
    .OnStepComplete(() => Debug.Log("delayed call"))
    .OnComplete(() => Debug.Log("delayed call"));

Sequence

A Sequence is a feature used to group multiple tweens together. By using Sequences, you can easily create complex animations by combining multiple tweens.

Creating a Sequence

You can obtain a new Sequence from Sequence.Create():

// Create a new Sequence
Sequence sequence = Sequence.Create();

Adding Tweens

Next, you add the tweens you want to include in the Sequence. There are various methods available to add tweens to a Sequence. By using these methods, you can combine tweens and build complex animations.

Sequences can be nested regardless of their hierarchy. Options and callbacks such as SetDelay and SetLoops will also work for the Sequence after it has been added.

Append

The Append() method adds tweens to the end of the Sequence. The added tweens will play in sequence when you play the Sequence.

// Append a tween to the end
sequence.Append(transform.TweenPosition(new Vector3(1f, 0f, 0f), 2f))
    .Append(transform.TweenPosition(new Vector3(1f, 3f, 0f), 2f));

You can use AppendInterval() and AppendCallback() to add delays or callbacks:

// Append a delay to the end
sequence.AppendInterval(1f);

// Append a callback to the end
sequence.AppendCallback(() => Debug.Log("Hello!"));

Prepend

If you want to add tweens to the beginning, you can use Prepend(). In this case, the tweens already added will move back by the duration of the new prepend tween.

// Prepend a tween to the beginning
sequence.Prepend(transform.TweenPosition(new Vector3(1f, 0f, 0f), 2f));

You can also use PrependInterval() and PrependCallback():

// Prepend a delay to the beginning
sequence.PrependInterval(1f);

// Prepend a callback to the beginning
sequence.PrependCallback(() => Debug.Log("Hello!"));

Join

To concatenate a tween with the ones added before it, you can use Join(). Tweens added with Join() will play concurrently with the previously added tween.

sequence.Append(transform.TweenPosition(new Vector3(1f, 0f, 0f), 2f));

// Join with the previous tween
sequence.Join(transform.TweenPosition(new Vector3(1f, 3f, 0f), 2f));

Insert

If you want to insert a tween at an arbitrary point, you can use Insert(). The tween added with Insert() will operate independently of other tweens and will start playing once it reaches the specified position.

// Insert a tween at 1 second from the start
sequence.Insert(1f, transform.TweenPosition(new Vector3(1f, 0f, 0f), 2f));

You can also insert callbacks using InsertCallback():

// Insert a callback at 1 second from the start
sequence.InsertCallback(1f, () => Debug.Log("Hello!"));

Implicit Conversion to Tween

A Sequence can be implicitly converted to a Tween, allowing you to assign it directly:

Sequence sequence = Sequence.Create();

// Can be assigned directly to a Tween variable
Tween tween = sequence;

Usage Considerations

Here are some important points to keep in mind when using Sequences:

You cannot add tweens to a Sequence while it is playing.
You cannot add tweens with infinite loops to a Sequence.
Once a tween is added to a Sequence, it becomes locked, and you cannot access it individually. Be cautious, as you cannot manipulate the individual tweens within a Sequence.
You cannot include the same tween in multiple Sequences.

Logging

If you want to perform logging of Tween callbacks and values for debugging purposes, you can easily achieve this using dedicated extension methods. Note that these logs will only be displayed if MagicTweenSettings' LoggingMode is set to Full.

using MagicTween;
using MagicTween.Diagnostics; // Enable debug extension methods

// Log specific callback
transform.TweenPosition(Vector3.up, 5f)
    .LogOnUpdate();

// Log all callbacks together
transform.TweenEulerAngles(new Vector3(0f, 0f, 90f), 5f)
    .LogCallbacks();

// You can also assign a name for identification
transform.TweenLocalScale(Vector3.one * 2f, 5f)
    .LogCallbacks("Scale");

float foo;
// You can also log values (current value per frame)
Tween.To(() => foo, x => foo = x, 5f, 10f)
    .LogValue();

Project Settings

You can customize Tween's initial settings and logging preferences.

Creating MagicTweenSettings

Create a MagicTweenSettings asset to store your configuration by navigating to Assets > Create > Magic Tween > Magic Tween Settings.

Note The created MagicTweenSettings will be automatically added to the project's Preload Assets. If the settings are not being loaded, ensure that MagicTweenSettings is included in the Preload Assets.

Logging Mode

Set whether logging is enabled or not.

LoggingMode	Behavior
LoggingMode.Full	Display all logs, including Log-related extension methods, in the Console.
LoggingMode.WarningsAndErrors	Display only warnings and errors in the Console.
LoggingMode.ErrorsOnly	Display only errors in the Console.

Capture Exceptions

When set to "On," exceptions that occur internally in Tweens will be logged as warnings. When set to "Off," exceptions will be logged as regular exceptions.

Default Tween Parameters

You can modify the default settings for Tweens.

Changing Settings from Script

You can access these settings from the MagicTweenSettings class in your script.

// Change Logging Mode from script
MagicTweenSettings.loggingMode = LoggingMode.ErrorsOnly;

TextMesh Pro

Magic Tween supports TextMesh Pro (TMP) and allows you to tween text characters individually using the TweenChar extension methods. Here's how you can use it:

TMP_Text tmp;

// GetCharCount retrieves the number of tweenable characters
for (int i = 0; i < tmp.GetCharCount(); i++)
{
    tmp.TweenCharScale(i, Vector3.zero).SetInvert().SetDelay(i * 0.07f);
}

You can stop the tweens associated with TMP_Text and reset the text's decoration to its initial state using ResetCharTweens():

// Stop the character tweens and reset the decoration to the initial state
tmp.ResetCharTweens();

Character tweens are powered by the TMPTweenAnimator class internally, which you can access using GetTMPTweenAnimator():

// Get the internal TMPTweenAnimator
TMPTweenAnimator tmpAnimator = tmp.GetTMPTweenAnimator();

// Extension methods for TMP_Text use methods from TMPTweenAnimator internally
tmpAnimator.TweenCharOffset(0, Vector3.up);

// You can directly set parameters for each character using SetChar**
tmpAnimator.SetCharScale(1, Vector3.one * 2f);
tmpAnimator.SetCharColor(1, Color.red);

// ResetCharTweens() equivalent
tmpAnimator.Reset();

// GetCharCount() equivalent
tmpAnimator.GetCharCount();

UniRx

By integrating UniRx, you can convert Tween callbacks and Tween operations into Observables.

Converting Callbacks to Observables

You can use methods like OnUpdateAsObservable() to convert Tween callbacks into Observables. For example:

float foo;

Tween.To(() => foo, x => foo = x, 10f, 10f)
    .OnUpdateAsObservable()
    .Subscribe(_ =>
    {
        Debug.Log("update!");
    });

Converting Tweens to Observables

You can use ToObservable() to transform a Tween into an Observable that emits values each frame. For example:

Tween.FromTo(0f, 10f, 10f, null)
    .ToObservable()
    .Where(x => x >= 5f)
    .Subscribe(x =>
    {
        Debug.Log("current value: " + x);
    });

UniTask

By integrating UniTask, you can use async/await to handle Tween waiting operations.

var tween = transform.TweenPosition(Vector3.up, 2f);

// You can directly await the Tween (waits until the Tween is killed)
await tween;

You can use AwaitForKill() and pass a CancellationToken to handle cancellation.

// Create a CancellationTokenSource
var cts = new CancellationTokenSource();

// Wait until the Tween is killed, passing the CancellationToken
await transform.TweenPosition(Vector3.up, 2f)
    .AwaitForKill(cancellationToken: cts.Token);

You can also wait for other events like completion using AwaitForComplete() or AwaitForPause().

// Wait until the Tween completes
await transform.TweenPosition(Vector3.up, 2f).AwaitForComplete();

Furthermore, you can specify the CancelBehaviour to determine the behavior on cancellation.

var cts = new CancellationTokenSource();

// On cancellation, call Complete and throw OperationCanceledException
await transform.TweenPosition(Vector3.up, 2f)
    .AwaitForComplete(CancelBehaviour.CompleteAndCancelAwait, cts.Token);

CancelBehaviour	Behavior on Cancellation
CancelBehaviour.Kill	Calls the Kill method.
CancelBehaviour.Complete	Calls the Complete method.
CancelBehaviour.CompleteAndKill	Calls both Complete and Kill methods.
CancelBehaviour.CancelAwait	Throws an OperationCanceledException without calling Complete or Kill.
CancelBehaviour.KillAndCancelAwait	Default behavior. Calls Kill and throws an OperationCanceledException.
CancelBehaviour.CompleteAndCancelAwait	Calls Complete and throws an OperationCanceledException.
CancelBehaviour.CompleteAndKillAndCancelAwait	Calls both Complete and Kill methods and throws an OperationCanceledException.

Implementation for ECS

Magic Tween provides APIs for implementing Tween for ECS (Entity Component System), allowing you to create high-performance Tweens compared to regular ones.

Creating a Translator

To Tween the values of specific Components, you need to create a Translator for the target Component, along with a System to make it work.

As an example, let's create a Translator for the following Component:

public struct ExampleComponent : IComponentData
{
    public float value;
}

First, define a structure that implements ITweenTranslator, like this:

public struct ExampleTranslator : ITweenTranslator<float, ExampleComponent>
{
    // A property to track the target Entity, used in the System
    public Entity TargetEntity { get; set; }

    // Apply the value to the Component
    public void Apply(ref ExampleComponent component, in float value)
    {
        component.value = value;
    }

    // Return the current value of the Component
    public float GetValue(ref ExampleComponent component)
    {
        return component.value;
    }
}

Next, create a System class that inherits from TweenTranslationSystemBase, providing the Translator you created as a type argument. Keep the internal of the derived class empty, as the base class already implements the necessary logic.

public partial class ExampleTweenTranslationSystem : TweenTranslationSystemBase<float, ExampleComponent, ExampleTranslator> { }

Now, you're prepared to Tween values.

Tweening Component Values

To Tween values using the Translator you created, you can use Tween.Entity.To() or Tween.Entity.FromTo(). Provide the type of your Translator as type arguments.

var entity = EntityManager.CreateEntity();
EntityManager.AddComponent<ExampleComponent>(entity);

// Tween the value of ExampleComponent's 'value' field to 5 over 10 seconds
Tween.Entity.To<ExampleTranslator>(entity, 5f, 10f);

Just like regular Tweens, you can chain methods to add settings:

Tween.Entity.FromTo<ExampleTranslator>(entity, 0f, 5f, 10f)
    .SetEase(Ease.OutSine)
    .SetLoops(3, LoopType.Restart)
    .SetDelay(1f);

You can also add these Tweens to a Sequence:

var entity1 = EntityManager.CreateEntity();
var entity2 = EntityManager.CreateEntity();
EntityManager.AddComponent<ExampleComponent>(entity1);
EntityManager.AddComponent<ExampleComponent>(entity2);

var tween1 = Tween.Entity.To<ExampleTranslator>(entity1, 5f, 10f);
var tween2 = Tween.Entity.To<ExampleTranslator>(entity2, 5f, 10f);

var sequence = Sequence.Create()
    .Append(tween1)
    .Append(tween2);

Warning Avoid applying multiple concurrent Tweens with the same Translator to the same Entity. This may lead to overlapping value modifications and unexpected behavior.

Built-in Translators

Under MagicTween.Translators, you can find built-in Translators for ECS components.

Currently, there's a Translator that works with LocalTransform.

If you have the Entities Graphics package installed, you'll find Translators for tweening Material properties.

Limitations

Creating and manipulating Tweens/Sequences is only supported on the main thread. You cannot create new Tweens or perform operations like Kill or Complete from a Job. To alleviate this limitation, there's ongoing development to introduce functionality for creating/operating Tweens using a dedicated CommandBuffer.

Other Features

EaseUtility

The easing functions used internally by Tween can be accessed via EaseUtility.

float value1 = EaseUtility.Evaluate(0.5f, Ease.OutQuad);
float value2 = EaseUtility.InOutQuad(0.5f);

Experimental Features

The MagicTween.Experimental namespace contains features that are currently under development. These features are available for use but come with no guarantees, and there may be breaking changes without notice.

Known Issues

Performance Drop in the Editor

ECS performs numerous checks to enhance safety, which leads to decreased performance within the editor. This performance degradation is particularly noticeable when creating Tweens and, in some cases, can result in processing times several times longer than usual.

It's important to note that these safety checks are disabled in the build, so performance measurements should always be conducted in the build environment.

Performance Drop in WebGL

While it's possible to use ECS in WebGL, limitations in WebGL's specifications, such as the absence of multi-threading and SIMD, disable optimizations like the Job System and Burst. ECS achieves its high performance through the Job System and Burst, making performance degradation unavoidable in WebGL (hence, there are few advantages to using ECS on WebGL currently).

Magic Tween optimizes the calculation part of Tweens using the Job System and Burst. Therefore, on WebGL, performance suffers for the reasons mentioned above. While these effects are typically not noticeable, please be mindful of this when creating a large number of Tweens.

Support

Forum: https://forum.unity.com/threads/magic-tween-extremely-fast-tween-library-implemented-in-ecs.1490080/

License

MIT License

Kyome22/RunCat_for_windows

2023-09-10T01:49:35Z

A cute running cat animation on your windows taskbar.

RunCat_for_windows

A cute running cat animation on your windows taskbar.

Demo

You only have to run the RunCat.exe.

Installation

Access to the "Releases" page and download the RunCat.exe.

Or install via Scoop (x64 version): scoop install runcat

Contributors

umbraco/Umbraco-CMS

2023-09-10T01:49:35Z

The simple, flexible and friendly ASP.NET CMS used by more than 730.000 websites

Umbraco CMS

Umbraco is the friendliest, most flexible and fastest growing ASP.NET CMS, and used by more than 500,000 websites worldwide. Our mission is to help you deliver delightful digital experiences by making Umbraco friendly, simpler and social.

Learn more at umbraco.com

See the official Umbraco website for an introduction, core mission and values of the product and team behind it.

Please also see our Code of Conduct.

Getting Started

Umbraco Cloud is the easiest and fastest way to use Umbraco yet, with full support for all your custom .NET code and integrations. You're up and running in less than a minute, and your life will be made easier with automated upgrades and a built-in deployment engine. We offer a free 14-day trial, no credit card needed.

If you want to DIY, then you can download Umbraco either as a ZIP file or via NuGet. It's the same version of Umbraco CMS that powers Umbraco Cloud, but you'll need to find a place to host it yourself, and handling deployments and upgrades will be all up to you.

Documentation

The documentation for Umbraco CMS can be found on Our Umbraco. The source for the Umbraco docs is open source as well and we're happy to look at your documentation contributions.

Join the Umbraco community

Our friendly community is available 24/7 at the community hub, we call "Our Umbraco". Our Umbraco features forums for questions and answers, documentation, downloadable plugins for Umbraco, and a rich collection of community resources.

Besides "Our", we all support each other in our Community Discord Server and on Twitter: Umbraco HQ, Release Updates, #umbraco

Contributing

Umbraco is contribution-focused and community-driven. If you want to contribute back to the Umbraco source code, please check out our guide to contributing.

sksalahuddin2828/C_Sharp

2023-09-10T01:49:35Z

Explore something new

C# - Unlocking the Power of Programming

Overview

Welcome to the fascinating world of C# (pronounced "C sharp"), a versatile and powerful programming language developed by Microsoft. Whether you're a seasoned developer or just starting your coding journey, C# opens up a world of possibilities, enabling you to build robust and scalable applications across various platforms.

Why Choose C#?

C# offers a delightful blend of simplicity, expressiveness, and performance. With its intuitive syntax and extensive standard library, it allows developers to write clean and elegant code, making the development process enjoyable and productive. C# combines the best features of object-oriented programming (OOP) with modern language constructs, empowering you to create efficient and maintainable software solutions.

Key Features and Benefits

1. Object-Oriented Paradigm

C# follows a powerful object-oriented programming paradigm, enabling you to organize your code into reusable and modular components. Encapsulation, inheritance, and polymorphism provide the foundation for building scalable and extensible applications.

2. Cross-Platform Development

C# offers cross-platform capabilities through the .NET platform, allowing you to write code once and deploy it on various operating systems, including Windows, macOS, and Linux. With the introduction of .NET Core and later .NET 5 and beyond, C# has become a truly versatile language.

4. Productivity and Tooling

C# provides a wide range of development tools, including Visual Studio and Visual Studio Code, which offer rich features such as intelligent code completion, debugging, and integrated source control. These tools enhance your productivity and streamline the development process.

5. Strong Typing and Memory Management

C# is a statically typed language, ensuring type safety and early error detection during compilation. It also incorporates automatic memory management through garbage collection, relieving developers from manual memory allocation and deallocation.

6. Integration with the .NET Ecosystem

C# seamlessly integrates with the extensive .NET ecosystem, giving you access to a vast array of libraries and frameworks. Whether you're building desktop applications, web services, mobile apps, or games, you'll find a wealth of resources to accelerate your development and leverage existing solutions.

Get Started with C#

Ready to dive into the world of C#? Here's a quick guide to get you started:

Install the .NET SDK: Visit the official .NET website (https://dotnet.microsoft.com) and download the .NET SDK, which includes the C# compiler and other essential tools.
Choose an IDE: Select your preferred integrated development environment (IDE) such as Visual Studio or Visual Studio Code. These IDEs provide a comprehensive development environment tailored for C# development.
Learn the Language: Explore online tutorials, documentation, and interactive resources to grasp the fundamentals of C#. Familiarize yourself with concepts such as classes, objects, control flow, and LINQ.
Practice, Practice, Practice: Start coding! Undertake small projects, solve coding challenges, and contribute to open-source projects. Building real-world applications is the best way to hone your skills and deepen your understanding of the language.
Engage with the Community: Join developer communities, forums, and online platforms to connect with fellow C# enthusiasts. Networking and collaboration can provide valuable insights, guidance, and opportunities for growth.

Conclusion

C# empowers developers to unleash their creativity and build impactful software solutions. Its combination of simplicity, performance, and versatility makes it a valuable tool for a wide range of applications. Whether you're crafting desktop applications, web services, or game development, C# is your gateway to unlocking the full potential of modern programming.

So, embark on your C# journey with enthusiasm, curiosity, and a thirst for innovation. Happy coding!

ChefKissInc/NootedRed

2023-09-10T01:27:58Z

Lilu plugin for AMD Vega iGPUs. No commercial use.

NootedRed

[!IMPORTANT] The only official source for this kext is the Actions tab of the ChefKissInc/NootedRed GitHub repository, provided entirely free of charge.

Any other source may include malware, be outdated, etc.

Paid for this? Demand a refund.

Got it from an unofficial source? Complain and demand they take it down.

The AMD Vega iGPU support Lilu (1.6.4+) plug-in.

The NootedRed project is licensed under the Thou Shalt Not Profit License version 1.5. See LICENSE.

[!NOTE] This project is under active research and development; There will be crashes here and there, but is generally smooth.

See repository issues and our site for more information.

Thanks Acidanthera for the backlight code and UnfairGVA patches in WhateverGreen.

lammps/lammps

2023-09-10T01:27:58Z

Public development project of the LAMMPS MD software package

This is the LAMMPS software package.

LAMMPS stands for Large-scale Atomic/Molecular Massively Parallel Simulator.

Copyright (2003) Sandia Corporation. Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation, the U.S. Government retains certain rights in this software. This software is distributed under the GNU General Public License.

LAMMPS is a classical molecular dynamics simulation code designed to run efficiently on parallel computers. It was developed at Sandia National Laboratories, a US Department of Energy facility, with funding from the DOE. It is an open-source code, distributed freely under the terms of the GNU Public License (GPL) version 2.

The code is maintained by the LAMMPS development team who can be emailed at developers@lammps.org. The LAMMPS WWW Site at www.lammps.org has more information about the code and its uses.

The LAMMPS distribution includes the following files and directories:

README this file LICENSE the GNU General Public License (GPL) bench benchmark problems cmake CMake build files doc documentation examples simple test problems fortran Fortran wrapper for LAMMPS lib additional provided or external libraries potentials interatomic potential files python Python wrappers for LAMMPS src source files tools pre- and post-processing tools

Point your browser at any of these files to get started:

https://docs.lammps.org/Manual.html LAMMPS manual https://docs.lammps.org/Intro.html hi-level introduction https://docs.lammps.org/Build.html how to build LAMMPS https://docs.lammps.org/Run_head.html how to run LAMMPS https://docs.lammps.org/Commands_all.html Table of available commands https://docs.lammps.org/Library.html LAMMPS library interfaces https://docs.lammps.org/Modify.html how to modify and extend LAMMPS https://docs.lammps.org/Developer.html LAMMPS developer info

You can also create these doc pages locally:

% cd doc % make html # creates HTML pages in doc/html % make pdf # creates Manual.pdf

trilinos/Trilinos

2023-09-10T01:27:58Z

Mod manager for various PC games. Discord Server: https://discord.gg/ewUVAqyrQX if you would like to be more involved

Mod Organizer

Mod Organizer (MO) is a tool for managing mod collections of arbitrary size. It is specifically designed for people who like to experiment with mods and thus need an easy and reliable way to install and uninstall them. Mod Organizer 2 is a mod manager created by Tannin to support 64-bit games like Skyrim Special Edition and Fallout 4 in addition to all the 32-bit games MO1 already supported.

Tannin discontinued the project when he was hired by the Nexus team to develop their new Vortex mod manager. MO2 was not completed and was left with many issues. LePresidente managed to fix it to a usable state by himself after SkyrimSE came out, adding support for it, and releasing v2.0.8. The project took up speed again after a few more coders showed up in late 2017, and a lot of progress has been made.

Help Wanted!

Mod Organizer 2 is an open project in the hands of the community, there are problems that need to be solved and things that could be added. MO2 really needs developers and if you have the programming skills and some free time you can really improve the experience of the modding community.

To have more information, please join the open MO2 Development discord server: Mod Organizer 2 If you want to help translate MO2 to your language you should join the discord server too and head to the #translation channel. To setup a development environment on your machine, there is the mob project that handles that. If you want to submit your code changes, please use a good formatting style like the default one in Visual Studio.

Through the work of a few people of the community MO2 has come quite far, now it needs some more of those people to go further.

Reporting Issues:

Issues should be reported to the GitHub page or on the open discord server: Mod Organizer 2. Here is also where dev builds are tested, bugs are reported and investigated, suggestions are discussed and a lot more.

Credits to Tannin, LePresidente, Silarn, erasmux, AL12, LostDragonist, AnyOldName3, isa, Holt59 and many others for the development.

Download Location

on GitHub.com
on NexusMods.com

Old Download Location

TanninOne/modorganizer/releases

Building

Please refer to Modorganizer2/mob for build instructions.

Other Repositories

MO2 consists of multiple repositories on github. The mob project will download them automatically as required. They should however also be buildable individually. Here is a complete list:

Unused Repositories

libigl/libigl

2023-09-10T01:49:25Z

Simple C++ geometry processing library.

libigl - A simple C++ geometry processing library

Documentation, tutorial, and instructions at https://libigl.github.io.

🆕 Doxygen Documentation
The latest version of libigl (v2.5.0) introduces doxygen generated detailed documentation

sksalahuddin2828/C_Plus_Plus

2023-09-10T01:49:25Z

Explore something new

emilybache/GildedRose-Refactoring-Kata

2023-09-10T01:49:25Z

Starting code for the GildedRose Refactoring Kata in many programming languages.

Gilded Rose Refactoring Kata

You can find out more about this exercise in my YouTube video Why Developers LOVE The Gilded Rose Kata.

I use this kata as part of my work as a technical coach. I wrote a lot about the coaching method I use in this book Technical Agile Coaching with the Samman method. A while back I wrote this article "Writing Good Tests for the Gilded Rose Kata" about how you could use this kata in a coding dojo.

How to use this Kata

The simplest way is to just clone the code and start hacking away improving the design. You'll want to look at the "Gilded Rose Requirements" which explains what the code is for. I strongly advise you that you'll also need some tests if you want to make sure you don't break the code while you refactor.

You could write some unit tests yourself, using the requirements to identify suitable test cases. I've provided a failing unit test in a popular test framework as a starting point for most languages.

Alternatively, use the Approval tests provided in this repository. (Read more about that in the section "Text-based Approval Testing").

The idea of the exercise is to do some deliberate practice, and improve your skills at designing test cases and refactoring. The idea is not to re-write the code from scratch, but rather to practice taking small steps, running the tests often, and incrementally improving the design.

Please don't send me a pull request with your solution. It can be a bit confusing since GitHub encourages you to do so! Please only send me pull requests if you have a correction or improvement to the starting position. You don't want to spoil the fun of doing the exercise for other people!

Gilded Rose Requirements in other languages

Text-Based Approval Testing

Most language versions of this code have a TextTest fixture for Approval testing. For information about this, see the TextTests README

History of the exercise

This Kata was originally created by Terry Hughes (http://twitter.com/TerryHughes). It is already on GitHub here. See also Bobby Johnson's description of the kata.

I translated the original C# into a few other languages, (with a little help from my friends!), and slightly changed the starting position. This means I've actually done a small amount of refactoring already compared with the original form of the kata, and made it easier to get going with writing tests by giving you one failing unit test to start with. I also added test fixtures for Text-Based approval testing with TextTest (see the TextTests)

As Bobby Johnson points out in his article "Why Most Solutions to Gilded Rose Miss The Bigger Picture", it'll actually give you better practice at handling a legacy code situation if you do this Kata in the original C#. However, I think this kata is also really useful for practicing writing good tests using different frameworks and approaches, and the small changes I've made help with that. I think it's also interesting to compare what the refactored code and tests look like in different programming languages.

Translating this code

More translations are most welcome! I'm very open for pull requests that translate the starting position into additional languages.

Please note a translation should ideally include:

a translation of the production code for 'update_quality' and Item
one failing unit test complaining that "fixme" != "foo"
a TextTest fixture, ie a command-line program that runs update_quality on the sample data for the number of days specified.

Please don't write too much code in the starting position or add too many unit tests. The idea with the one failing unit test is to tempt people to work out how to fix it, discover it wasn't that hard, and now they understand what this test is doing they realize they can improve it.

If your programming language doesn't have an easy way to add a command-line interface, then the TextTest fixture is probably not necessary.

ByteOverlord/Watch_Quake2

2023-09-10T01:27:48Z

Quake 2 port for Apple Watch

03.09.2023

Watch_Quake2 is a Quake 2 port specifically designed for Apple Watch, incorporating software rendering and working video and audio playback capabilities.

About
Changes by ByteOverlord
Features
Changelog
Game Size
Battery Usage
Benchmark
Known Issues
Building Requirements
Building the Watch_Quake2
Adding music to Watch_Quake2 (Optional)
Credits
License

About

Following the successful release of the Quake 1 port for Apple Watch, our team set an ambitious objective: to bring Quake 2 to this portable device. However, porting the 1997 release of Quake posed significant challenges. We encountered initial setbacks that considerably delayed the release schedule for several months. ByteOverlord made commendable progress early in the porting process, but crashing issues persisted. After a brief hiatus, we resumed our efforts and focused on improving the stability of the game port, ultimately achieving a more reliable and robust build.

The primary objective of this project was to demonstrate the remarkable capabilities of the Apple Watch, even when operating in software rendering mode. This project serves as a glimpse into the untapped potential that could be unlocked if we were granted access to the device's GPU. Just envision the possibilities that could be achieved with such resources at our disposal.

Previous port:
Quake 1 for Apple Watch https://github.com/ByteOverlord/Watch_Quake 18.11.2022

Changes by ByteOverlord

Swift for UI drawing, audio callback and input processing
Music playback support added. Audio file format “Mono, 22050 Hz, WAV, IMA ADPCM” preferred for lower processing and minimal space required
Cheats an maps menus added. Custom graphics for these menus included with pak3.pak
Resolution setting is automatic with hud, menu and console notification placements affected by resolution
Transparent surface drawing fixed for 64bit
Game pausing through Swift UI when application enters background state and saves settings. Mutes audio when inactive state
Saving and loading rewritten for 64bit (referenced Yamagi Quake 2). Might still have some bugs
Video playback’s audio sample rate changed to match Apple Watch’s sample rate
Video cinematic’s audio volume control added to options
Initial game loading fixed so that game and audio loops start after the Swift UI tells the app is active (≥ WatchOS 9.2). When content view updates the game starts (< WatchOS 9.2)
Touch input handling through Swift UI. Customized touch control layout for movement, camera, interaction and menu controls
Benchmark mode that logs average, low and high frame times. Uses the Quake’s DEMO1 and DEMO2 playbacks

Features

pak0, pak1 and pak2 loading
Menu screen
Save and load game
Autosaving of options
Map quick select screen
Cheat activation screen
Tweaked controls for watch interface
Video and Music playback
Benchmark mode
Automatic native resolution (Series 4 40mm -> Ultra)

Changelog

06.09.2023

Cheats added to the menu (noclip and notarget)
Menu graphics updated (pak3.pak)

Game size

Complete game with optional video and music takes ~ 403,4 MB on the watch.

pak0.pak is 184 MB
pak1.pak is 13 MB
pak2.pak is 45 KB
Video Files 163,7 MB
Music Tracks are 38,7 MB (converted with Audacity (Mono, 22050 Hz, WAV, IMA ADPCM))

Battery Usage

Usage listed as % the charge dropped during the test.

Model	Battery Health	30 min	60 min
S4 40mm	-	-	-
S4 44mm	-	-	-
S5 40mm	-	-	-
S5 44mm	95%	17%	33%
SE (1st Gen) 40mm	-	-	-
SE (1st Gen) 44mm	-	-	-
S6 40mm	-	-	-
S6 44mm	-	-	-
S7 41mm	-	-	-
S7 45mm	-	-	-
SE (2nd Gen) 40mm	-	-	-
SE (2nd Gen) 44mm	-	-	-
S8 41mm	-	-	-
S8 45mm	-	-	-
Ultra	-	-	-

Results are only indicative.

Benchmark

Reference table:

Good	Moderate	Bad
≤ 5 ms	6-12 ms	> 16 ms

Results:

DEMO1	Avg	Low	High
S4 40mm	6.80	2.85	12.96
S4 44mm	7.12	2.90	14.08
S5 40mm	6.95	2.55	16.72
S5 44mm	7.28	3.19	14.89
SE (1st Gen) 40mm	-	-	-
SE (1st Gen) 44mm	-	-	-
S6 40mm	-	-	-
S6 44mm	-	-	-
S7 41mm	-	-	-
S7 45mm	-	-	-
SE (2nd Gen) 40mm	-	-	-
SE (2nd Gen) 44mm	7.71	2.57	14.29
S8 41mm	-	-	-
S8 45mm	-	-	-
Ultra	-	-	-

DEMO2	Avg	Low	High
S4 40mm	7.10	2.93	15.36
S4 44mm	7.58	4.31	16.79
S5 40mm	7.23	3.35	14.50
S5 44mm	7.20	4.17	14.43
SE (1st Gen) 40mm	-	-	-
SE (1st Gen) 44mm	-	-	-
S6 40mm	-	-	-
S6 44mm	-	-	-
S7 41mm	-	-	-
S7 45mm	-	-	-
SE (2nd Gen) 40mm	-	-	-
SE (2nd Gen) 44mm	7.96	3.45	18.13
S8 41mm	-	-	-
S8 45mm	-	-	-
Ultra	-	-	-

Benchmarks are only indicative.

Known Issues

Player camera jumps after saving or loading a game
Mods are not supported
Multiplayer is not supported
Audio will not transmit through Bluetooth
Sound does not playback on app start (Put the game to the dock and open it from there again.)
App does not start anymore (Build the app to the watch from Xcode). There is a 7 day testing period on a 'Free Provisioning' profile.

Building Requirements

Apple Watch Series 4 or later with watchOS 9.1 or later
iPhone 8 or later with iOS 16 or later
Mac with macOS 12.5 or later
Xcode 14.1 or later with command-line tools
Homebrew
Innoextract
bchunk (only for music)
Audacity (only for music)

Building the Watch_Quake2

Install Xcode through Apple AppStore and install command-line tools when Xcode asks for it.

Download the Watch_Quake2 from GitHub. (https://github.com/ByteOverlord/Watch_Quake2)

Get Quake II (Original) through gog.com
Download the Quake II (Original) installer from your gog.com profile:
"gog.com Profile" -> Games -> Quake II (Original) -> View Downloads -> Download Offline Backup Game Installers -> Quake II (Original)

Install Homebrew from terminal:
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/homebrew/go/install)"
Install innoextract from Homebrew:

brew update
brew install innoextract

In terminal extract the Quake 2 game files with innoextract:
innoextract “downloaded setup_quake2_quad_damage_2.0.0.3.exe file location”

(innoextract will extract an “app” and "tmp" folders in current working directory.) "app" folder has the needed PAK files (app/baseq2/), optional video (app/baseq2/video/) and optional audio (app/music/).

Copy pak0.pak, pak1.pak and pak2.pak to WatchQuake2 project (pak names have to be lowercase (case sensitive))

/WatchQuake2 Watch App/Resources/baseq2/

Optional video files can be placed in: /WatchQuake2 Watch App/Resources/baseq2/video/

Open the GitHub WatchQuake Xcode project.
In WatchQuake2 project file go to:
WatchQuake2 -> Targets -> WatchQuake2 -> Signing & Capabilities
Select your "Team" and write a unique “Bundle Identifier”.

Then in WatchQuake2 project file go to:
WatchQuake2 -> Targets -> WatchQuake2 Watch App -> Signing & Capabilities
Select your "Team" and use the same “Bundle Identifier” as previously but with “.watchkitapp” added at the end.

If the project “Frameworks” folder shows the framework texts in red continue with step 14.

If the project “Frameworks” folder shows the framework texts in white jump to step 17.

In WatchQuake2 project file go to:
WatchQuake2 -> Targets -> WatchQuake2 -> Build Phases -> Link Binary With Libraries
Add these frameworks:

AVFoundation.framework
CoreGraphics.framework
Foundation.framework
SwiftUI.framework
UIKit.framework
WatchKit.framework

And after adding them delete the frameworks with red text.
Check that the project is on release setting.

Test the build first with a simulator (Series 4 Simulator or newer (watchOS Simulator))

After building, the game will start and playback demo1.

Click the game view to go to main menu.

After testing connect iPhone to the Mac with a cable and make sure Apple Watch has a connection to iPhone.
Set the iPhone and Apple Watch in to "Developer" mode.

On iPhone:
Settings -> Privacy & Security -> Developer Mode (SECURITY) -> Developer Mode

On Apple Watch:
Settings -> Privacy & Security -> Developer Mode (SECURITY) -> Developer Mode

After the devices have restarted.

Select the Apple Watch as target.

From Xcode build the Watch_Quake2 to the watch.

Transfer times:
2 min (iPhone X & S4 40mm)
2 min (iPhone X & S4 44mm)
7 min (game, music), 10 min (game, videos, music), (iPhone X & S5 44mm))
15 min (iPhone 12 mini & S5 40mm)

Adding music to Watch_Quake2 (Optional)

Locate the music tracks (track02 - track21.ogg) that where extracted from the Quake 2 installer (app/music/).
Open Audacity and drag the tracks to it. Edit the tracks to Mono.
Select -> All
Tracks -> Mix -> Mix Stereo Down to Mono

Set the “Project Rate (Hz)” to 22050.

Export tracks with “Export Multiple” in “WAV” format with “IMA ADPCM” encoding.

Place the exported audio files (track02 - track21.wav) inside the WatchQuake2 project:
“WatchQuake2 Watch App/Resources/baseq2/music/”
From Xcode build the Watch_Quake2 to the watch.

Transfer times:
2 min (iPhone X & S4 40mm)
2 min (iPhone X & S4 44mm)
7 min (game, music), 10 min (game, videos, music), (iPhone X & S5 44mm))
15 min (iPhone 12 mini & S5 40mm)

Credits

quake_watch https://github.com/MyOwnClone/quake_watch
Quake 2 source code by Id Software. https://github.com/id-Software/Quake-2
Yamagi Quake II (game saving) https://github.com/yquake2/yquake2
App icon, images and documentation by IdeaVoid

License

This software is released under the terms of the GNU General Public License v2.

littlefs-project/littlefs

2023-09-10T01:49:41Z

A little fail-safe filesystem designed for microcontrollers

littlefs

A little fail-safe filesystem designed for microcontrollers.

   | | |     .---._____
  .-----.   |          |
--|o    |---| littlefs |
--|     |---|          |
  '-----'   '----------'
   | | |

Power-loss resilience - littlefs is designed to handle random power failures. All file operations have strong copy-on-write guarantees and if power is lost the filesystem will fall back to the last known good state.

Dynamic wear leveling - littlefs is designed with flash in mind, and provides wear leveling over dynamic blocks. Additionally, littlefs can detect bad blocks and work around them.

Bounded RAM/ROM - littlefs is designed to work with a small amount of memory. RAM usage is strictly bounded, which means RAM consumption does not change as the filesystem grows. The filesystem contains no unbounded recursion and dynamic memory is limited to configurable buffers that can be provided statically.

Example

Here's a simple example that updates a file named boot_count every time main runs. The program can be interrupted at any time without losing track of how many times it has been booted and without corrupting the filesystem:

#include "lfs.h"

// variables used by the filesystem
lfs_t lfs;
lfs_file_t file;

// configuration of the filesystem is provided by this struct
const struct lfs_config cfg = {
    // block device operations
    .read  = user_provided_block_device_read,
    .prog  = user_provided_block_device_prog,
    .erase = user_provided_block_device_erase,
    .sync  = user_provided_block_device_sync,

    // block device configuration
    .read_size = 16,
    .prog_size = 16,
    .block_size = 4096,
    .block_count = 128,
    .cache_size = 16,
    .lookahead_size = 16,
    .block_cycles = 500,
};

// entry point
int main(void) {
    // mount the filesystem
    int err = lfs_mount(&lfs, &cfg);

    // reformat if we can't mount the filesystem
    // this should only happen on the first boot
    if (err) {
        lfs_format(&lfs, &cfg);
        lfs_mount(&lfs, &cfg);
    }

    // read current count
    uint32_t boot_count = 0;
    lfs_file_open(&lfs, &file, "boot_count", LFS_O_RDWR | LFS_O_CREAT);
    lfs_file_read(&lfs, &file, &boot_count, sizeof(boot_count));

    // update boot count
    boot_count += 1;
    lfs_file_rewind(&lfs, &file);
    lfs_file_write(&lfs, &file, &boot_count, sizeof(boot_count));

    // remember the storage is not updated until the file is closed successfully
    lfs_file_close(&lfs, &file);

    // release any resources we were using
    lfs_unmount(&lfs);

    // print the boot count
    printf("boot_count: %d\n", boot_count);
}

Usage

Detailed documentation (or at least as much detail as is currently available) can be found in the comments in lfs.h.

littlefs takes in a configuration structure that defines how the filesystem operates. The configuration struct provides the filesystem with the block device operations and dimensions, tweakable parameters that tradeoff memory usage for performance, and optional static buffers if the user wants to avoid dynamic memory.

The state of the littlefs is stored in the lfs_t type which is left up to the user to allocate, allowing multiple filesystems to be in use simultaneously. With the lfs_t and configuration struct, a user can format a block device or mount the filesystem.

Once mounted, the littlefs provides a full set of POSIX-like file and directory functions, with the deviation that the allocation of filesystem structures must be provided by the user.

All POSIX operations, such as remove and rename, are atomic, even in event of power-loss. Additionally, file updates are not actually committed to the filesystem until sync or close is called on the file.

Other notes

Littlefs is written in C, and specifically should compile with any compiler that conforms to the C99 standard.

All littlefs calls have the potential to return a negative error code. The errors can be either one of those found in the enum lfs_error in lfs.h, or an error returned by the user's block device operations.

In the configuration struct, the prog and erase function provided by the user may return a LFS_ERR_CORRUPT error if the implementation already can detect corrupt blocks. However, the wear leveling does not depend on the return code of these functions, instead all data is read back and checked for integrity.

If your storage caches writes, make sure that the provided sync function flushes all the data to memory and ensures that the next read fetches the data from memory, otherwise data integrity can not be guaranteed. If the write function does not perform caching, and therefore each read or write call hits the memory, the sync function can simply return 0.

Design

At a high level, littlefs is a block based filesystem that uses small logs to store metadata and larger copy-on-write (COW) structures to store file data.

In littlefs, these ingredients form a sort of two-layered cake, with the small logs (called metadata pairs) providing fast updates to metadata anywhere on storage, while the COW structures store file data compactly and without any wear amplification cost.

Both of these data structures are built out of blocks, which are fed by a common block allocator. By limiting the number of erases allowed on a block per allocation, the allocator provides dynamic wear leveling over the entire filesystem.

                    root
                   .--------.--------.
                   | A'| B'|         |
                   |   |   |->       |
                   |   |   |         |
                   '--------'--------'
                .----'   '--------------.
       A       v                 B       v
      .--------.--------.       .--------.--------.
      | C'| D'|         |       | E'|new|         |
      |   |   |->       |       |   | E'|->       |
      |   |   |         |       |   |   |         |
      '--------'--------'       '--------'--------'
      .-'   '--.                  |   '------------------.
     v          v              .-'                        v
.--------.  .--------.        v                       .--------.
|   C    |  |   D    |   .--------.       write       | new E  |
|        |  |        |   |   E    |        ==>        |        |
|        |  |        |   |        |                   |        |
'--------'  '--------'   |        |                   '--------'
                         '--------'                   .-'    |
                         .-'    '-.    .-------------|------'
                        v          v  v              v
                   .--------.  .--------.       .--------.
                   |   F    |  |   G    |       | new F  |
                   |        |  |        |       |        |
                   |        |  |        |       |        |
                   '--------'  '--------'       '--------'

More details on how littlefs works can be found in DESIGN.md and SPEC.md.

DESIGN.md - A fully detailed dive into how littlefs works. I would suggest reading it as the tradeoffs at work are quite interesting.
SPEC.md - The on-disk specification of littlefs with all the nitty-gritty details. May be useful for tooling development.

Testing

The littlefs comes with a test suite designed to run on a PC using the emulated block device found in the bd directory. The tests assume a Linux environment and can be started with make:

make test

License

The littlefs is provided under the BSD-3-Clause license. See LICENSE.md for more information. Contributions to this project are accepted under the same license.

Individual files contain the following tag instead of the full license text.

SPDX-License-Identifier:    BSD-3-Clause

This enables machine processing of license information based on the SPDX License Identifiers that are here available: http://spdx.org/licenses/

Related projects

littlefs-fuse - A FUSE wrapper for littlefs. The project allows you to mount littlefs directly on a Linux machine. Can be useful for debugging littlefs if you have an SD card handy.
littlefs-js - A javascript wrapper for littlefs. I'm not sure why you would want this, but it is handy for demos. You can see it in action here.
littlefs-python - A Python wrapper for littlefs. The project allows you to create images of the filesystem on your PC. Check if littlefs will fit your needs, create images for a later download to the target memory or inspect the content of a binary image of the target memory.
littlefs2-rust - A Rust wrapper for littlefs. This project allows you to use littlefs in a Rust-friendly API, reaping the benefits of Rust's memory safety and other guarantees.
littlefs-disk-img-viewer - A memory-efficient web application for viewing littlefs disk images in your web browser.
mklfs - A command line tool built by the Lua RTOS guys for making littlefs images from a host PC. Supports Windows, Mac OS, and Linux.
Mbed OS - The easiest way to get started with littlefs is to jump into Mbed which already has block device drivers for most forms of embedded storage. littlefs is available in Mbed OS as the LittleFileSystem class.
SPIFFS - Another excellent embedded filesystem for NOR flash. As a more traditional logging filesystem with full static wear-leveling, SPIFFS will likely outperform littlefs on small memories such as the internal flash on microcontrollers.
Dhara - An interesting NAND flash translation layer designed for small MCUs. It offers static wear-leveling and power-resilience with only a fixed O(|address|) pointer structure stored on each block and in RAM.
ChaN's FatFs - A lightweight reimplementation of the infamous FAT filesystem for microcontroller-scale devices. Due to limitations of FAT it can't provide power-loss resilience, but it does allow easy interop with PCs.
chamelon - A pure-OCaml implementation of (most of) littlefs, designed for use with the MirageOS library operating system project. It is interoperable with the reference implementation, with some caveats.

Flipper-XFW/Xtreme-Firmware

2023-09-10T01:49:41Z

The Dom amongst the Flipper Zero Firmware. Give your Flipper the power and freedom it is really craving. Let it show you its true form. Dont delay, switch to the one and only true Master today!

XFW - `Xtreme Firmware` for the Flipper Zero

Website | Intro | Install | Changelog | Wiki | Discord | Donate

This firmware is a complete overhaul of the Official Firmware, and also features lots of awesome code-bits from Unleashed.

What makes it special?

We have spent many hours perfecting this code even further, and getting the most out of it.

The goal of this Firmware is to regularly bring out amazing updates based on what the community wants, with an actual understanding of whats going on. Fixing bugs that are regularly talked about, removing unstable / broken applications (.FAP) and actually using the level system that just sits abandoned everywhere else.

Feature-rich: We include all commonly found apps in the firmware, as long as they work.
Stable: Many hours have been spent rewriting core parts of the Flippers firmware as well as some of its apps to ensure stability. A task that was long needed on all Firmware, so we tackled it right away.
Customizable: Dont like the Animations, want to turn on/off the Home screen icons (battery, SD card etc), change the flippers name or anything like that? You absolutely can. No need to mess with code or deal with weird manifest files. Its all done with an App.

Note, the below mentioned changes are only a few things we did. For a full list click here

Xtreme Settings:

We wrote a powerful yet easy-to-use application specifically for our Firmware, that gives you easy-access to all the fancy things we implemented:

Interface: Customize every bit of your Flipper, from the desktop animations, to the main menu apps, lockscreen style etc.
Protocols: Here you can toggle between USB & Bluetooth mode for BadKB, and manage custom Subghz frequencies.
Misc: All the other options that don't fit elsewhere. Change your Flipper's name, xp level, and configure the RGB backlight.

Animations / Asset Packs:

Want to try some asset packs? Check here

We created our own, new & improved Animation / Asset system, that we can finally reveal. It lets you to create and cycle through your own Asset Packs with only a few button presses, allowing you to easily load custom Animations and Icons like never before.

You can easily create your own pack, or find some user made ones in the discord channel. Check here for a tutorial on creating your own. Essentially, we got our own Anims & Icons folders, inside each Asset Pack.

Once you have some packs, upload them to your Flipper in SD/asset_packs (if you did this right you should see SD/asset_packs/PackName/Anims and/or SD/asset_packs/PackName/Icons).

After installing the packs to Flipper, hit the Arrow UP button on the main menu and go to Xtreme Settings. Here choose which pack you want and tweak the other settings how you prefer, then press back to reboot and enjoy your new assets for all apps (e.g. Subghz scanning asset) & animations!

Bad Keyboard:

BadUSB is a wonderful app, but it lacks bluetooth capabilities. Now some might argue that its useless as you will always need authentication from both sides, but what if I told you that we found a solution to this problem?

Bad-KB allows you to toggle between USB and Bluetooth mode for your attacks. In Bluetooth mode it allows you to spoof the name & MAC of the device to whatever you want. Being a JBL speaker or a wireless razer keyboard is easily doable, allowing you to trick people so you can run your payloads without needing a cable at hand.

Levels:

This Firmware has 30 levels, not just the basic 3 the official one has.

With this new system in place, it allows for some cool stuff like locking animations behind a certain level. This can be done fairly easy: The idle_animations are tied to the level system. Specifically, the Min level variable of your manifest file is used here. Each level you reach, unlocks a new animation. The higher your level, the more animations people can see.

List of changes:

Note: This repo is always updated with OFW & Unleashed. No need to mention all those here. We will only mention our changes that we can actually be credited for.

[Added]

- Xtreme App
- Asset Packs
- More UI options
- Bad-Keyboard App
- A new battery display-type
- Scrolling view for long file names in browser
- Advanced and optimized level system. Read more above
- Folder handling for empty ones (Now indicate they are empty)

- Custom subghz presets
- Multiple NFC protocols
- Multiple Sub-Ghz protocols | Merged from Unleashed, thanks @xMasterX
- Subghz and IR signal replication via gpio | Credits to @xMasterX

- New API Routes for Locale settings

[Updated]

- All Assets

- Tons of apps
- File browser
- Massive compiler re-do
- About 4k files to speed things up a lot
- Applications to now use the new Locale setting

[Fixed]

- Keyboard issues on first char
- Passport crash on high level
- SFW / Dummy_mode getting you XP
- Leveling system
- Mood system

[REMOVED]

- Unused Dummy Mode
- Broken apps (bad apple, chess, etc.)
- Tons of unused code from FAPs and system calls

Install:

There are 3 methods to install Xtreme, we recommend you use the Web Updater, but choose whichever one you prefer:

Web Updater (Chrome)

Open the latest release page and click on the Web Updater link

Make sure qFlipper is closed

Click Connect and select your Flipper from the list

Click Flash and wait for the update to complete

qFlipper Package (.tgz)

Download the qFlipper package (.tgz) from the latest release page

Open qFlipper and connect your Flipper

Click Install from file

Select the .tgz you downloaded and wait for the update to complete

Zipped Archive (.zip)

Download the zipped archive (.zip) from the latest release page

Extract the archive. This is now your new Firmware folder

Open qFlipper, head to SD/Update and simply move the firmware folder there

On the Flipper, hit the Arrow Down button, this will get you to the file menu. In there simply search for your updates folder

Inside that folder, select the Firmware you just moved onto it, and run the file thats simply called Update

Build it yourself:

Warning We will not give basic support for compiling in our server. This is intended for people that already know what they are doing!

To download the needed tools:
$ git clone --recursive --jobs 8 https://github.com/Flipper-XFW/Xtreme-Firmware.git
$ cd Xtreme-Firmware/

To flash directly to the Flipper (Needs to be connected via USB, qFlipper closed)
$ ./fbt flash_usb_full

To compile a TGZ package
$ ./fbt updater_package

To build and launch a single app:
$ ./fbt launch APPSRC=some_appid

Stargazers over time

Contributors

❤️ Support

If you like what you're seeing, please consider donating to us. We won't ever put this behind a paywall, but we'd still appreciate a few bucks!

Patreon: ❤️ Account needed, subscription with perks across my entire org.
Wire-transfer: No account needed, one-time
Paypal: Account needed, one-time
ko-fi: No account needed, one-time
Monero: 41kyWeeoVdK4quzQ4M9ikVGs6tCQCLfdx8jLExTNsAu2SF1QAyDqRdjfGM6EL8L9NpXwt89HJeAoGf1aoArk7nDr4AMMV4T

Thanks for all your support <3

"What we do for ourselves dies with us. What we do for others and the world remains and is immortal.” ― Albert Pine

jqlang/jq

2023-09-10T01:49:41Z

Command-line JSON processor

jq

jq is a lightweight and flexible command-line JSON processor akin tosed,awk,grep, and friends for JSON data. It's written in portable C and has zero runtime dependencies, allowing you to easily slice, filter, map, and transform structured data.

Documentation

Official Documentation: jqlang.github.io/jq
Try jq Online: jqplay.org

Installation

Prebuilt Binaries

Download the latest releases from the GitHub release page.

Docker Image

Pull the jq image to start quickly with Docker.

Building from source

Dependencies

libtool
make
automake
autoconf

Instructions

git submodule update --init # if building from git to get oniguruma
autoreconf -i               # if building from git
./configure --with-oniguruma=builtin
make -j8
make check
sudo make install

Build a statically linked version:

make LDFLAGS=-all-static

If you're not using the latest git version but instead building a released tarball (available on the release page), skip the autoreconf step, and flex or bison won't be needed.

Cross-Compilation

For details on cross-compliation, check out the GitHub Actions file and the cross-compliation wiki page.

Community & Support

Questions & Help: Stack Overflow (jq tag)
Chat & Community: Join us on Discord
Wiki & Advanced Topics: Explore the Wiki

License

jq is released under the MIT License.

isohl/Promethium

2023-09-10T01:50:11Z

Space Based 4X Free Game

Fireman42/Guesser

2023-09-10T01:28:41Z

playing with clojure (guesses anything from 1-1000 if you don't lie to it) :D

athos/sweet-array

2023-09-10T01:28:41Z

Array manipulation library for Clojure with "sweet" array type notation and more safety by static types

samvit/clojure-datomic-scraper

2023-09-10T01:50:17Z

dcmoore/easy-bake-service

2023-09-10T01:50:17Z

A simple library that makes it easy to serve data

clojure/java.jdbc

2023-09-10T01:50:17Z

JDBC from Clojure (formerly clojure.contrib.sql)

kyubotics/coolq-cpp-sdk

2023-09-10T01:28:48Z

CoolQ C++ SDK

sudara/cmake-includes

2023-09-10T01:28:48Z

CMake include files used by Pamplejuce, the JUCE CMake template

ewinebrenner/gpr200-fa2023-templates

2023-09-10T01:50:23Z

Project templates for GPR-200 (Intro to Graphics Programming)

MiRoboticsLab/cyberdog_ws

2023-09-10T01:50:23Z

fonse/pokemon-battle

2023-09-10T01:50:30Z

Pokemon battle simulator for nodejs

techpines/asset-rack

2023-09-10T01:50:30Z

Static Web Framework for Nodejs

twilson63/express-coffee

2023-09-10T01:50:30Z

A Template for NodeJs Application using Express, CoffeeScript, Jade, Stylus, Nib

cognalog/AI

2023-09-10T01:29:00Z

AI homework assignments

AI

AI homework:

1-ass: Assignment 1 -- Pattern Matcher

problem spec: http://www.cs.columbia.edu/~sal/notes/assign1-pm_files/assignment1.txt

2-ass: Assignment 2 -- Search Algorithm comparison for a 15-puzzle

problem spec: http://www.cs.columbia.edu/~sal/notes/assign2-compare-15puz.txt

3-ass: Assignment 3 -- Isolation game playing using Adversarial Search (under construction)

problem spec: http://www.cs.columbia.edu/~sal/notes/assign3-isolation-game.htm

4-ass: Assignment 4 -- Decision Tree Learning Algorithm that accepts a list of records, each listing conditions (where the last "condition" is the outcome), and outputs a program that will infer the outcome from the conditions

problem spec: http://www.cs.columbia.edu/~sal/notes/assign4-decision-tree.html

akitaonrails/rinhabackend-lucky-crystal-api

2023-09-10T01:50:45Z

devopsbyraham/jenkins-java-project

2023-09-10T01:29:19Z

amethystnetwork-dev/Incognito

2023-09-10T01:29:19Z

Unofficial Incognito easy deployment version with TompHTTP bare server included.

NOTICE:

You CANNOT deploy to Vercel, Netifly, GitHub Pages, or Cloudflare pages. Read more here.

Get Started

To get started, press one of the buttons below to deploy Incognito

Or

Discord Server

Press the join button to join the server:

Overview

Features

Ultraviolet
Nice game library
Tab Cloaks
about:blank cloak

FAQ

How can I get official Incognito domains?

You should join the official Titanium Network Discord for official links

Who is this repo for?

This repo is for people that want an easy way to deploy Incognito to cloud hosting services. If you just want to use Incognito and not deploy it, I recommend you get an official Incognito domain (Check above for how to do so)

Where are the game files?

They are located in another repo because of the large file sizes.

X website is not working?

Proxies can have issues, like websites not working, websites blocking the proxy or data being locked to that website. AlienHub fixes these issues and more!

What proxies are used?

Incognito uses Ultraviolet as its only proxy.

Ultraviolet has some known limitations such as downloading files can be buggy and pages may not load.

My question is not answered

Because we can't answer every question here, you may want to check the docs. If you still have questions, you might want to join our Discord #discord-server (link)

Credits

Caracal.js - Creator of Incognito

professica/box-model-common-props

2023-09-10T01:29:19Z

repo for the "DC Facelift" project

Digital Collections App

This is the Repo for the Digital Collections app using the DS (internally referred to as "DC Facelift")

This is a Next.js project bootstrapped with create-next-app.

Getting Started

First, install Dependencies:

npm install

Use node version 18 or higher. If you have nvm installed on your local machine, use the following command to use node 18.

nvm use

If you don't already have node 18 installed on your machine, you can install it using:

nvm install 18

Second, run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev

Open http://localhost:3000 with your browser to see the result.

You can start editing the page by modifying pages/index.tsx. The page auto-updates as you edit the file.

API routes can be accessed on http://localhost:3000/api/hello. This endpoint can be edited in pages/api/hello.ts.

The pages/api directory is mapped to /api/*. Files in this directory are treated as API routes instead of React pages.

This project uses next/font to automatically optimize and load Inter, a custom Google Font.

Learn More

To learn more about Next.js, take a look at the following resources:

Next.js Documentation - learn about Next.js features and API.
Learn Next.js - an interactive Next.js tutorial.

You can check out the Next.js GitHub repository - your feedback and contributions are welcome!

Vercel

We use Vercel for internal purposes. Pull requests get deployed via Vercel for VQA and accessibility review. We do not formally deploy the app to qa and production using Vercel. We use Travis for this instead.

If you need access to the instance, please contact someone from the NYPL team.

nusserstudios/tailbliss

2023-09-10T01:50:52Z

TailBliss is a Hugo Starter theme built on TailwindCSS 3, and Alpine.JS.

TailBliss

TailBliss is an opinionated Hugo Starter with Tailwind CSS 3.2 and Alpine.js with light/dark modes.

View Demo · Report Bug · Request Feature · Ask Question

Install to VS Code with:
git clone git@github.com:nusserstudios/tailbliss your-name

Install with NPM

npm install

Install with Yarn

yarn

To start developing:

Develop with NPM

npm run start

Develop with Yarn

yarn start

To generate the site HTML:

Build with NPM

npm run build

Build with Yarn

yarn build

npm run start will run two commands parallel:
npx tailwindcss -i ./assets/css/main.css -o ./assets/css/style.css --watch

Has paginated Categories and Tags. Markdown files will automatically convert images put into /assets folder to .webp images.

Image shortcodes for webp as well.

Form

To use the form, visit FormSubmit.Co. Locate the contact form in "content/contact.md", and update the form action with the email address you want on this line: action="https://formsubmit.co/your@email.com" method="POST"

Credits

4044ever - Original Theme https://github.com/4044ever/Hugo-Tailwind-3.0.git

Jan Heise - Alpine.js Navbar https://github.com/jan-heise/responsive-navbar-with-dropdown

Made for Hacktoberfest, with ❤️ by NusserStudios.

HigherOrderCO/hvm-core

2023-09-10T01:29:34Z

DanNegrut/ME759-2023

2023-09-10T01:29:34Z

repo with information useful for the Fall 2023 offering of ME759 - High Performance Computing for Applications in Engineering

flintlib/python-flint

2023-09-10T01:29:43Z

Python bindings for Flint and Arb

leduykhanh/segmentation

2023-09-10T01:51:12Z

code-fusion-ima/hackadev

2023-09-10T01:29:52Z

Fusion Shop - E-Commerce Mobile ⌛

Este é um projeto em grupo desenvolvido como parte do programa da Imã Tech em colaboração com a Novo Mundo. O projeto, chamado Todas Tech, é uma formação full-stack exclusivamente para mulheres, com foco em criar um aplicativo de e-commerce mobile para produtos de tecnologia. Nós, a equipe Code Fusion, desenvolvemos o aplicativo que se chama Fusion Shop, incluindo sua identidade visual.

Tecnologias Utilizadas

Front-end: Dart e Flutter
Back-end: PHP e Laravel

Tasks Desenvolvidas

Tela Inicial - Home

A primeira tarefa concluída foi a criação da tela inicial - Home do Fusion Shop. Nesta tela, atendemos aos seguintes requisitos:

Listar 3 categorias principais:
- Eletroeletrônicos
- Smartphones
- SmartTVs
Listagem de produtos.
Menu de navegação (Tab Navigation Bottom)

Como Executar o Projeto

Para executar o projeto em seu ambiente local, siga as etapas abaixo:

Front-end (Dart e Flutter):

Clone o Repositório:

git clone code-fusion-ima/hackadev.git

2.Navegue até o Diretório do Projeto:

cd hackadev

3.Instale as Dependências:

flutter pub get

4.Execute o Aplicativo:

flutter run

Desenvolvedoras

💻 Brunna Gualberto 🎧

💻 Danúbia Ferreira 🎧

💻 Débora Barbosa 🎧

💻 Denise Schirmer 🎧

💻 Janine Oliveira 🎧
Monitoria da equipe

💻 Jordana Carneiro 🎧

💻 Luana Souza 🎧
Monitoria da equipe

💻 Sarah Rayssa 🎧

💻 Tahene Carvalho 🎧

💻 Viviane Canuto 🎧

Contato

Para obter mais informações sobre o projeto ou entrar em contato com a equipe, você pode nos encontrar no LinkedIn ou enviar um e-mail para codefusion.ima@gmail.com.

Sobre o Projeto

Este projeto é parte do programa da Imã Tech em colaboração com a Novo Mundo. Agradecemos pelo apoio e pela oportunidade de participar deste programa incrível! 🚀🌟

mylxsw/aidea

2023-09-10T01:51:19Z

AIdea 是一款支持 GPT 以及国产大语言模型通义千问、文心一言等，支持 Stable Diffusion 文生图、图生图、 SDXL1.0、超分辨率、图片上色的全能型 APP。

AIdea - AI 聊天、协作、图像生成

一款集成了主流大语言模型以及绘图模型的 APP，采用 Flutter 开发，代码完全开源，支持以下功能：

支持 GPT-3.5/4 问答聊天
支持国产模型：通义千问，文心一言、讯飞星火
支持文生图、图生图、超分辨率、黑白图片上色等功能，集成 Stable Diffusion 模型，支持 SDXL 1.0

开源代码：

客户端：https://github.com/mylxsw/aidea
服务端：https://github.com/mylxsw/aidea-server

微信技术交流群：3 个群都已满员，添加微信号 x-prometheus 为好友，拉你进群。

电报群：点此加入

下载安装地址

Android/IOS APP：https://aidea.aicode.cc/

IOS 国区目前不可用，除此之外所有区域都可以下载。

2023 年 9 月 4 日发现国区被下架后，非国区 IOS 版本可能无法完成应用内购（提示“你的购买无法完成”，但是也有人能够成功支付），暂时无法充值，最新进展在这里 issue#16。

Mac/Windows 桌面端：https://github.com/mylxsw/aidea/releases

Web 端：https://web.aicode.cc/

福利

目前我的 OpenAI 账户还有大约 4900+ 美金的额度，为了感谢各位的关注，在 2023 年 11 月 1 日前（4900 美金额度消耗完为止），GPT-4 使用价格低至每 1K Token 只需要 10 个智慧果。

相当于 0.1 元 / 1000 Token，OpenAI 官方价格为输入 0.21 元，输出 0.42 元。

APP 截图

如果对你有帮助，请我喝杯酒吧

微信	支付宝

Star History

License

MIT

miru-project/miru-app

2023-09-10T01:51:19Z

🎉 A versatile application that is free, open-source, and supports extension sources for videos, comics, and novels, available on Android, Windows, and Web platforms.

Miru App

免费、开源的支持视频、漫画、小说扩展源的多功能应用，支持 Android、Windows、Web.

特性

支持 windows,android
友好的扩展编写支持、调试日志
扩展使用 JavaScript 语言，开发简单
支持自定义扩展仓库
官方扩展仓库提供视频源，无需编写任何扩展即可使用
支持在线观看多种视频、漫画、小说源，实现多平台的统一
统一系统 UI 的设计语言
自动获取 TMDB 元数据信息

Todo

BT 种子播放
更好的调试工具
数据同步
自动搜寻字幕

安装

你可以通过 Release 页面下载最新版本的安装包，或者通过下面的方法自行构建

构建

安装 Flutter

参考 Flutter 官方文档

安装依赖

flutter pub get

运行

flutter run

打包

Android

flutter build apk

Windows

flutter build windows

关于 Linux

目前 Linux 因为依赖问题导致无法启动 quickjs 所以暂时不支持了

贡献

欢迎任何形式的贡献，包括但不限于：

提出建议
Bug 反馈
代码贡献
文档编写

交流

Telegram: https://t.me/MiruChat

Telegram 频道: https://t.me/MiruChannel

iwrqk/iwrqk

2023-09-10T01:51:19Z

Unofficial Iwara Flutter Client

IwrQk

中文

IwrQk is a cross-platform application based on Flutter that is compatible with the new version of Iwara (a video sharing platform).

Features

Cross-platform support for iOS and Android.

Todo list

Download manager (WIP)
Friends manager
Notification and message manager (WIP)
Follow, subscription, favorite, playlist, comments
Forum
Custom player and gallery viewer
Login, logout, register
Blocklist for tags
Advanced search

Screenshots

Contributions

If you are a developer, and you're willing to contribute to this project, please don't hesitate to submit a PR!

This app also uses intl to localize. You're welcome to commit translation too.

If you encounter any bugs, please report them after ruling out all network-related causes, as Iwara's servers often experience errors.

crossoverJie/k8s-combat

2023-09-10T01:30:22Z

☸️适用于研发人员的 k8s 实战项目🐳

背景

最近这这段时间更新了一些 k8s 相关的博客和视频，也收到了一些反馈；大概分为这几类：

公司已经经历过服务化改造了，但还未接触过云原生。
公司部分应用进行了云原生改造，但大部分工作是由基础架构和运维部门推动的，自己只是作为开发并不了解其中的细节，甚至 k8s 也接触不到。
还处于比较传统的以虚拟机部署的传统运维为主。

其中以第二种占大多数，虽然公司进行了云原生改造，但似乎和纯业务研发同学来说没有太大关系，自己工作也没有什么变化。

恰好我之前正好从业务研发的角度转换到了基础架构部门，两个角色我都接触过，也帮助过一些业务研发了解公司的云原生架构；

为此所以我想系统性的带大家以研发的角度对 k8s 进行实践。

因为 k8s 部分功能其实是偏运维的，对研发来说优先级并不太高；所以我不太会涉及一些 k8s 运维的知识点，比如安装、组件等模块；主要以我们日常开发会使用到的组件为主。

入门

进阶

如何使用配置
服务网格实战

运维你的应用

应用探针
滚动更新与回滚
优雅采集日志
应用可观测性
- 指标可视化

部署应用到 k8s

首先从第一章【部署应用到 k8s】开始，我会用 Go 写一个简单的 Web 应用，然后打包为一个 Docker 镜像，之后部署到 k8s 中，并完成其中的接口调用。

编写应用

func main() {  
   http.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {  
      log.Println("ping")  
      fmt.Fprint(w, "pong")  
   })  
  
   http.ListenAndServe(":8081", nil)  
}

应用非常简单就是提供了一个 ping 接口，然后返回了一个 pong.

Dockerfile

# 第一阶段：编译 Go 程序  
FROM golang:1.19 AS dependencies  
ENV GOPROXY=https://goproxy.cn,direct  
WORKDIR /go/src/app  
COPY go.mod .  
#COPY ../../go.sum .  
RUN --mount=type=ssh go mod download  
  
# 第二阶段：构建可执行文件  
FROM golang:1.19 AS builder  
WORKDIR /go/src/app  
COPY . .  
#COPY --from=dependencies /go/pkg /go/pkg  
RUN go build  
  
# 第三阶段：部署  
FROM debian:stable-slim  
RUN apt-get update && apt-get install -y curl  
COPY --from=builder /go/src/app/k8s-combat /go/bin/k8s-combat  
ENV PATH="/go/bin:${PATH}"  
  
# 启动 Go 程序  
CMD ["k8s-combat"]

之后编写了一个 dockerfile 用于构建 docker 镜像。

docker:  
   @echo "Docker Build..."  
   docker build . -t crossoverjie/k8s-combat:v1 && docker image push crossoverjie/k8s-combat:v1

使用 make docker 会在本地构建镜像并上传到 dockerhub

编写 deployment

下一步便是整个过程中最重要的环节了，也是唯一和 k8s 打交道的地方，那就是编写 deployment。

在之前的视频[《一分钟了解 k8s》](【一分钟带你了解 k8s】 https://www.bilibili.com/video/BV1Cm4y1n7yG/?share_source=copy_web&vd_source=358858ab808efe832b0dda9dbc4701da) 中讲过常见的组件：

其中我们最常见的就是 deployment，通常用于部署无状态应用；现在还不太需要了解其他的组件，先看看 deployment 如何编写：

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  labels:  
    app: k8s-combat  
  name: k8s-combat  
spec:  
  replicas: 1  
  selector:  
    matchLabels:  
      app: k8s-combat  
  template:  
    metadata:  
      labels:  
        app: k8s-combat  
    spec:  
      containers:  
        - name: k8s-combat  
          image: crossoverjie/k8s-combat:v1  
          imagePullPolicy: Always  
          resources:  
            limits:  
              cpu: "1"  
              memory: 300Mi  
            requests:  
              cpu: "0.1"  
              memory: 30Mi

开头两行的 apiVersion 和 kind 可以暂时不要关注，就理解为 deployment 的固定写法即可。

metadata：顾名思义就是定义元数据的地方，告诉 Pod 我们这个 deployment 叫什么名字，这里定义为：k8s-combat

中间的：

metadata:  
  labels:  
    app: k8s-combat

也很容易理解，就是给这个 deployment 打上标签，通常是将这个标签和其他的组件进行关联使用才有意义，不然就只是一个标签而已。

标签是键值对的格式，key, value 都可以自定义。

而这里的 app: k8s-combat 便是和下面的 spec 下的 selector 选择器匹配，表明都使用 app: k8s-combat 进行关联。

而 template 中所定义的标签也是为了让选择器和 template 中的定义的 Pod 进行关联。

Pod 是 k8s 中相同功能容器的分组，一个 Pod 可以绑定多个容器，这里就只有我们应用容器一个了；后续在讲到 istio 和日志采集时便可以看到其他的容器。

template 中定义的内容就很容易理解了，指定了我们的容器拉取地址，以及所占用的资源(cpu/ memory)。

replicas: 1：表示只部署一个副本，也就是只有一个节点的意思。

部署应用

之后我们使用命令:

kubectl apply -f deployment/deployment.yaml

生产环境中往往会使用云厂商所提供的 k8s 环境，我们本地可以使用 https://minikube.sigs.k8s.io/docs/start/ minikube 来模拟。

就会应用这个 deployment 同时将容器部署到 k8s 中，之后使用:

kubectl get pod

在后台 k8s 会根据我们填写的资源选择一个合适的节点，将当前这个 Pod 部署过去。

就会列出我们刚才部署的 Pod:

❯ kubectl get pod
NAME                                READY   STATUS    RESTARTS   AGE
k8s-combat-57f794c59b-7k58n         1/1     Running   0          17h

我们使用命令：

kubectl exec -it k8s-combat-57f794c59b-7k58n  bash

就会进入我们的容器，这个和使用 docker 类似。

之后执行 curl 命令便可以访问我们的接口了：

root@k8s-combat-57f794c59b-7k58n:/# curl http://127.0.0.1:8081/ping
pong
root@k8s-combat-57f794c59b-7k58n:/#

这时候我们再开一个终端执行：

❯ kubectl logs -f k8s-combat-57f794c59b-7k58n
2023/09/03 09:28:07 ping

便可以打印容器中的日志，当然前提是应用的日志是写入到了标准输出中。

跨服务调用

在做传统业务开发的时候，当我们的服务提供方有多个实例时，往往我们需要将对方的服务列表保存在本地，然后采用一定的算法进行调用；当服务提供方的列表变化时还得及时通知调用方。

student:
	url:
	  - 192.168.1.1:8081
	  - 192.168.1.2:8081

这样自然是对双方都带来不少的负担，所以后续推出的服务调用框架都会想办法解决这个问题。

以 spring cloud 为例：

服务提供方会向一个服务注册中心注册自己的服务（名称、IP等信息），客户端每次调用的时候会向服务注册中心获取一个节点信息，然后发起调用。

但当我们切换到 k8s 后，这些基础设施都交给了 k8s 处理了，所以 k8s 自然得有一个组件来解决服务注册和调用的问题。

也就是我们今天重点介绍的 service。

service

在介绍 service 之前我先调整了源码：

func main() {  
   http.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {  
      name, _ := os.Hostname()  
      log.Printf("%s ping", name)  
      fmt.Fprint(w, "pong")  
   })  
   http.HandleFunc("/service", func(w http.ResponseWriter, r *http.Request) {  
      resp, err := http.Get("http://k8s-combat-service:8081/ping")  
      if err != nil {  
         log.Println(err)  
         fmt.Fprint(w, err)  
         return  
      }  
      fmt.Fprint(w, resp.Status)  
   })  
  
   http.ListenAndServe(":8081", nil)  
}

新增了一个 /service 的接口，这个接口会通过 service 的方式调用服务提供者的服务，然后重新打包。

make docker

同时也新增了一个 deployment-service.yaml:

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  labels:  
    app: k8s-combat-service # 通过标签选择关联  
  name: k8s-combat-service  
spec:  
  replicas: 1  
  selector:  
    matchLabels:  
      app: k8s-combat-service  
  template:  
    metadata:  
      labels:  
        app: k8s-combat-service  
    spec:  
      containers:  
        - name: k8s-combat-service  
          image: crossoverjie/k8s-combat:v1  
          imagePullPolicy: Always  
          resources:  
            limits:  
              cpu: "1"  
              memory: 100Mi  
            requests:  
              cpu: "0.1"  
              memory: 10Mi  
---  
apiVersion: v1  
kind: Service  
metadata:  
  name: k8s-combat-service  
spec:  
  selector:  
    app: k8s-combat-service # 通过标签选择关联  
  type: ClusterIP  
  ports:  
    - port: 8081        # 本 Service 的端口  
      targetPort: 8081  # 容器端口  
      name: app

使用相同的镜像部署一个新的 deployment，名称为 k8s-combat-service，重点是新增了一个kind: Service 的对象。

这个就是用于声明 service 的组件，在这个组件中也是使用 selector 标签和 deployment 进行了关联。

也就是说这个 service 用于服务于名称等于 k8s-combat-service 的 deployment。

下面的两个端口也很好理解，一个是代理的端口，另一个是 service 自身提供出去的端口。

至于 type: ClusterIP 是用于声明不同类型的 service，除此之外的类型还有：

NodePort
LoadBalancer
ExternalName 等类型，默认是 ClusterIP，现在不用纠结这几种类型的作用，后续我们在讲到 Ingress 的时候会具体介绍。

负载测试

我们先分别将这两个 deployment 部署好：

k apply -f deployment/deployment.yaml
k apply -f deployment/deployment-service.yaml

❯ k get pod
NAME                                  READY   STATUS    RESTARTS   AGE
k8s-combat-7867bfb596-67p5m           1/1     Running   0          3h22m
k8s-combat-service-5b77f59bf7-zpqwt   1/1     Running   0          3h22m

由于我新增了一个 /service 的接口，用于在 k8s-combat 中通过 service 调用 k8s-combat-service 的接口。

resp, err := http.Get("http://k8s-combat-service:8081/ping")

其中 k8s-combat-service 服务的域名就是他的服务名称。

如果是跨 namespace 调用时，需要指定一个完整名称，在后续的章节会演示。

我们整个的调用流程如下：

相信大家也看得出来相对于 spring cloud 这类微服务框架提供的客户端负载方式，service 是一种服务端负载，有点类似于 Nginx 的反向代理。

为了更直观的验证这个流程，此时我将 k8s-combat-service 的副本数增加到 2：

spec:  
  replicas: 2

只需要再次执行：

❯ k apply -f deployment/deployment-service.yaml
deployment.apps/k8s-combat-service configured
service/k8s-combat-service unchanged

不管我们对 deployment 的做了什么变更，都只需要 apply 这个 yaml 文件即可， k8s 会自动将当前的 deployment 调整为我们预期的状态（比如这里的副本数量增加为 2）；这也就是 k8s 中常说的声明式 API。

可以看到此时 k8s-combat-service 的副本数已经变为两个了。如果我们此时查看这个 service 的描述时：

❯ k describe svc k8s-combat-service |grep Endpoints
Endpoints:         192.168.130.133:8081,192.168.130.29:8081

会发现它已经代理了这两个 Pod 的 IP。

此时我进入了 k8s-combat-7867bfb596-67p5m 的容器：

k exec -it k8s-combat-7867bfb596-67p5m bash
curl http://127.0.0.1:8081/service

并执行两次 /service 接口，发现请求会轮训进入 k8s-combat-service 的代理的 IP 中。

由于 k8s service 是基于 TCP/UDP 的四层负载，所以在 http1.1 中是可以做到请求级的负载均衡，但如果是类似于 gRPC 这类长链接就无法做到请求级的负载均衡。

换句话说 service 只支持连接级别的负载。

如果要支持 gRPC，就得使用 Istio 这类服务网格，相关内容会在后续章节详解。

总的来说 k8s service 提供了简易的服务注册发现和负载均衡功能，当我们只提供 http 服务时是完全够用的。

jakzal/phpqa

2023-09-10T01:30:22Z

Docker image that provides static analysis tools for PHP

Static Analysis Tools for PHP

Docker image providing static analysis tools for PHP. The list of available tools and the installer are actually managed in the jakzal/toolbox repository.

Supported platforms and PHP versions

Docker hub repository: https://hub.docker.com/r/jakzal/phpqa/

Nightly builds: https://hub.docker.com/r/jakzal/phpqa-nightly/

Debian

latest, debian (Dockerfile)
1.91.0, 1.91, 1.91.0-debian, 1.91-debian (Dockerfile)
1.91.0-php8.0, 1.91-php8.0, php8.0-debian, php8.0 (Dockerfile)
1.91.0-php8.1, 1.91-php8.1, php8.1-debian, php8.1 (Dockerfile)
1.91.0-php8.2, 1.91-php8.2, php8.2-debian, php8.2 (Dockerfile)

Alpine

alpine (Dockerfile)
1.91.0-alpine, 1.91-alpine, (Dockerfile)
1.91.0-php8.0-alpine, 1.91-php8.0-alpine, php8.0-alpine (Dockerfile)
1.91.0-php8.1-alpine, 1.91-php8.1-alpine, php8.1-alpine (Dockerfile)
1.91.0-php8.2-alpine, 1.91-php8.2-alpine, php8.2-alpine (Dockerfile)

Legacy

These are the latest tags for PHP versions that are no longer supported:

1.80.0-php7.4, 1.80-php7.4, php7.4-debian, php7.4 (Dockerfile)
1.80.0-php7.4-alpine, 1.80-php7.4-alpine, php7.4-alpine (Dockerfile)
1.61.2-php7.3, 1.61-php7.3, php7.3-debian, php7.3 (debian/Dockerfile)
1.61.2-php7.3-alpine, 1.61-php7.3-alpine, php7.3-alpine (alpine/Dockerfile)
1.44.0-php7.2, 1.44-php7.2, php7.2 (7.2/debian/Dockerfile)
1.44.0-php7.2-alpine, 1.44-php7.2-alpine, php7.2-alpine (7.2/alpine/Dockerfile)
1.26.0-php7.1, 1.26-php7.1, php7.1 (7.1/debian/Dockerfile)
1.26.0-php7.1-alpine, 1.26-php7.1-alpine, php7.1-alpine (7.1/alpine/Dockerfile)

Available tools

Name	Description	PHP 8.0	PHP 8.1	PHP 8.2
behat	Helps to test business expectations	✅	✅	✅
box	Fast, zero config application bundler with PHARs	❌	✅	✅
box-3	Fast, zero config application bundler with PHARs	✅	✅	❌
churn	Discovers good candidates for refactoring	✅	✅	✅
codeception	Codeception is a BDD-styled PHP testing framework	✅	✅	✅
composer	Dependency Manager for PHP	✅	✅	✅
composer-bin-plugin	Composer plugin to install bin vendors in isolated locations	✅	✅	✅
composer-normalize	Composer plugin to normalize composer.json files	✅	✅	✅
composer-require-checker	Verify that no unknown symbols are used in the sources of a package.	❌	✅	✅
composer-require-checker-3	Verify that no unknown symbols are used in the sources of a package.	✅	✅	✅
composer-unused	Show unused packages by scanning your code	✅	✅	✅
dephpend	Detect flaws in your architecture	✅	✅	✅
deprecation-detector	Finds usages of deprecated code	✅	✅	✅
deptrac	Enforces dependency rules between software layers	❌	✅	✅
diffFilter	Applies QA tools to run on a single pull request	✅	✅	✅
ecs	Sets up and runs coding standard checks	✅	✅	✅
infection	AST based PHP Mutation Testing Framework	❌	✅	✅
larastan	PHPStan extension for Laravel	✅	✅	✅
local-php-security-checker	Checks composer dependencies for known security vulnerabilities	✅	✅	✅
parallel-lint	Checks PHP file syntax	✅	✅	✅
paratest	Parallel testing for PHPUnit	✅	✅	✅
pdepend	Static Analysis Tool	✅	✅	✅
pest	The elegant PHP Testing Framework	✅	✅	✅
phan	Static Analysis Tool	✅	✅	✅
phive	PHAR Installation and Verification Environment	✅	✅	✅
php-coupling-detector	Detects code coupling issues	✅	✅	✅
php-cs-fixer	PHP Coding Standards Fixer	✅	✅	✅
php-fuzzer	A fuzzer for PHP, which can be used to find bugs in libraries by feeding them 'random' inputs	✅	✅	✅
php-semver-checker	Suggests a next version according to semantic versioning	✅	✅	✅
phpa	Checks for weak assumptions	✅	✅	✅
phparkitect	Helps to put architectural constraints in a PHP code base	✅	✅	✅
phpat	Easy to use architecture testing tool	✅	✅	✅
phpbench	PHP Benchmarking framework	❌	✅	✅
phpca	Finds usage of non-built-in extensions	✅	✅	✅
phpcb	PHP Code Browser	✅	✅	✅
phpcbf	Automatically corrects coding standard violations	✅	✅	✅
phpcodesniffer-composer-install	Easy installation of PHP_CodeSniffer coding standards (rulesets).	✅	✅	✅
phpcov	a command-line frontend for the PHP_CodeCoverage library	❌	✅	✅
phpcpd	Copy/Paste Detector	✅	✅	✅
phpcs	Detects coding standard violations	✅	✅	✅
phpcs-security-audit	Finds vulnerabilities and weaknesses related to security in PHP code	✅	✅	✅
phpda	Generates dependency graphs	✅	❌	❌
phpdd	Finds usage of deprecated features	✅	✅	✅
phpDocumentor	Documentation generator	❌	✅	✅
phpinsights	Analyses code quality, style, architecture and complexity	✅	✅	✅
phplint	Lints php files in parallel	✅	✅	✅
phploc	A tool for quickly measuring the size of a PHP project	✅	✅	✅
phpmd	A tool for finding problems in PHP code	✅	✅	✅
phpmetrics	Static Analysis Tool	✅	✅	✅
phpmnd	Helps to detect magic numbers	✅	✅	✅
phpspec	SpecBDD Framework	✅	✅	❌
phpstan	Static Analysis Tool	✅	✅	✅
phpstan-banned-code	PHPStan rules for detecting calls to specific functions you don't want in your project	✅	✅	✅
phpstan-beberlei-assert	PHPStan extension for beberlei/assert	✅	✅	✅
phpstan-deprecation-rules	PHPStan rules for detecting deprecated code	✅	✅	✅
phpstan-doctrine	Doctrine extensions for PHPStan	✅	✅	✅
phpstan-ergebnis-rules	Additional rules for PHPstan	✅	✅	✅
phpstan-exception-rules	PHPStan rules for checked and unchecked exceptions	✅	✅	✅
phpstan-larastan	Separate installation of phpstan for larastan	✅	✅	✅
phpstan-phpunit	PHPUnit extensions and rules for PHPStan	✅	✅	✅
phpstan-strict-rules	Extra strict and opinionated rules for PHPStan	✅	✅	✅
phpstan-symfony	Symfony extension for PHPStan	✅	✅	✅
phpstan-webmozart-assert	PHPStan extension for webmozart/assert	✅	✅	✅
phpunit	The PHP testing framework	❌	✅	✅
phpunit-8	The PHP testing framework (8.x version)	✅	✅	✅
phpunit-9	The PHP testing framework (9.x version)	✅	✅	✅
pint	Opinionated PHP code style fixer for Laravel	✅	✅	✅
psalm	Finds errors in PHP applications	✅	✅	✅
psalm-plugin-doctrine	Stubs to let Psalm understand Doctrine better	✅	✅	✅
psalm-plugin-phpunit	Psalm plugin for PHPUnit	✅	✅	✅
psalm-plugin-symfony	Psalm Plugin for Symfony	✅	✅	✅
psecio-parse	Scans code for potential security-related issues	✅	✅	✅
rector	Tool for instant code upgrades and refactoring	✅	✅	✅
roave-backward-compatibility-check	Tool to compare two revisions of a class API to check for BC breaks	✅	✅	✅
simple-phpunit	Provides utilities to report legacy tests and usage of deprecated code	✅	✅	✅
twig-lint	Standalone cli twig 1.X linter	✅	✅	✅
twig-linter	Standalone cli twig 3.X linter	✅	✅	✅
twigcs	The missing checkstyle for twig!	✅	✅	✅
yaml-lint	Compact command line utility for checking YAML file syntax	❌	✅	✅

More tools

Some tools are not included in the docker image, to use them refer to their documentation:

exakat - a real time PHP static analyser

Removed tools

Name	Summary
analyze	Visualizes metrics and source code
box-legacy	Legacy version of box
composer-normalize	Composer plugin to normalize composer.json files
design-pattern	Detects design patterns
parallel-lint	Checks PHP file syntax
php-formatter	Custom coding standards fixer
phpcf	Finds usage of deprecated features
phpdoc-to-typehint	Automatically adds type hints and return types based on PHPDocs
phpstan-localheinz-rules	Additional rules for PHPstan
security-checker	Checks composer dependencies for known security vulnerabilities
testability	Analyses and reports testability issues of a php codebase

Running tools

Pull the image:

docker pull jakzal/phpqa

The default command will list available tools:

docker run -it --rm jakzal/phpqa

To run the selected tool inside the container, you'll need to mount the project directory on the container with -v "$(pwd):/project". Some tools like to write to the /tmp directory (like PHPStan, or Behat in some cases), therefore it's often useful to share it between docker runs, i.e. with -v "$(pwd)/tmp-phpqa:/tmp". If you want to be able to interrupt the selected tool if it takes too much time to complete, you can use the --init option. Please refer to the docker run documentation for more information.

docker run --init -it --rm -v "$(pwd):/project" -v "$(pwd)/tmp-phpqa:/tmp" -w /project jakzal/phpqa phpstan analyse src

You might want to tweak this command to your needs and create an alias for convenience:

alias phpqa='docker run --init -it --rm -v "$(pwd):/project" -v "$(pwd)/tmp-phpqa:/tmp" -w /project jakzal/phpqa:alpine'

Add it to your ~/.bashrc so it's defined every time you start a new terminal session.

Now the command becomes a lot simpler:

phpqa phpstan analyse src

Building the image

git clone https://github.com/jakzal/phpqa.git
cd phpqa
make build-debian

To build the alpine version:

make build-alpine

Cookbook

Please check out the cookbook for further tips & tricks.

Contributing

Please read the Contributing guide to learn about contributing to this project. Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

eitsupi/devcontainer-try-r

2023-09-10T01:30:22Z

This repository contains some example Dev Container definitions for R and can be run on GitHub Codespaces.

Try Out Development Containers: R

This repository contains some example Dev Container definitions for R and can be run on GitHub Codespaces (click on the badges!).

Dev Container Definitions

Combination of r-rig and rstudio-server

It is configured to install ghcr.io/rocker-org/devcontainer-features/r-rig and ghcr.io/rocker-org/devcontainer-features/rstudio-server on top of mcr.microsoft.com/devcontainers/base:ubuntu. Initially, the latest release version of R is installed, but rig makes it easy to install a different version of R and switch between these.

RStudio Server is configured to run immediately after container startup.

R (rocker/r-ver base) template

A simple template that runs a Docker image ghcr.io/rocker-org/devcontainer/r-ver configured to be used as a Dev Container. Many R packages are already installed.

It is just a download of the Dev Container Template ghcr.io/rocker-org/devcontainer-templates/r-ver.

R (r2u and bspm configured) template

It is configured to install ghcr.io/rocker-org/devcontainer-features/r-apt on top of mcr.microsoft.com/devcontainers/base:ubuntu. Installing many R packages quickly thanks to r2u.

It is just a download of the Dev Container Template ghcr.io/rocker-org/devcontainer-templates/r2u.

janza/docker-python3-opencv

2023-09-10T01:51:46Z

🐋 🐍 👁 Docker image with python 3 and opencv 4.1

Docker image with python 3.7 and opencv 4.1.0

Usage:

docker run -it jjanzic/docker-python3-opencv python
>>> import cv2

Image tagged with :contrib contains docker image built with contrib modules

List of available docker tags:

opencv-4.1.0 (latest branch)
contrib-opencv-4.1.0 (opencv_contrib branch)
opencv-4.0.1
contrib-opencv-4.0.1
opencv-4.0.0
contrib-opencv-4.0.0
opencv-3.4.2
contrib-opencv-3.4.2
opencv-3.4.1
contrib-opencv-3.4.1
opencv-3.4.0
contrib-opencv-3.4.0
opencv-3.3.0
contrib-opencv-3.3.0
opencv-3.2.0
contrib-opencv-3.2.0

Taiyo-ai/ts-mesh-pipeline

2023-09-10T01:51:46Z

Use this template repository to write time series data ingestion pipelines

Data Ingestion Pipeline Template

This repository consists of boilerplate folder structure to write and organize your scripts for a data ingestion pipeline

Folder Structure

The tree diagram below represents a general file structure

|--- data_source_name                      
     |--- deploy                            # pipeline orchestration and configuration of DAGs
     |    |---dev              
     |    |---prod
     |--- src
          |--- dependencies
          |    |--- cleaning
          |    |    |--- __init__.py
          |    |    |--- cleaner.py         ## Cleaning script here
          |    |--- geocoding
          |    |    |--- __init__.py
          |    |    |--- geocoder.py        ## Geocoding script here
          |    |--- scraping                # This folder contains all data harvesting scipts
          |    |    |--- __init__.py
          |    |    |--- scraper.py         ## Harvesting script here
          |    |--- standardization
          |    |    |--- __init__.py
          |    |    |--- standardizer.py    ## Standardization script here
          |    |--- utils                   # Utility and helper scipts to be placed here
          |         |--- __init__.py
          |--- .dockerignore
          |--- Dockerfile
          |--- client.py                    # Master script that connects all the above blocks
          |--- requirements.txt

Different Blocks of ETL pipeline

Scraping/Data Harvesting
- Contains all the scripts that extracts metadata and raw data to be processed further from database, websites, webservices, APIs, etc.
Cleaning
- Treatment missing fields and values
- Treatment of duplicate entries
- Convert country codes to ISO 3166-1 alpha3 i.e. 3 letter format
- Identify region name and region code using the country code
Geocoding
- Based upon location information available in the data
  - Location label
  - Geo-spatial coordinates
- Missing field can be found either by using geocoding or reverse geocoding with max precision available
Standardization
- Fields to be strictly in lower snake casing
- Taking care of data types and consistency of fields
- Standardize fields like sector, subsector, domain, subdomain
- Renaming of field names as per required standards
- Manipulation of certain fields and values to meet up the global standards for presentation, analytics and business use of data
- Refer to the Global Field Standards spreadsheet for the standards to be followed

Note

Depending upon what fields are already available in the data GEOCODING step may or may not be required.

It is recommended that the resultant data after each and every step is stored and backed up for recovery purpose.

Apart from the primary fields listed down in Global Field Standards spreadsheet, there are several other secondary fields that are to be scraped; given by the data provider for every document that holds significant business importance.

Submission and Evaluation

For assignment submission guidelines and evaluation criteria refer to the WIKI documentation

D3VL/L3MON

2023-09-10T01:51:46Z

L3MON
Layer 3 Advanced Network Management and Monitoring System

L3MON is a cutting-edge network management and monitoring system developed using NodeJS. It empowers users with a robust and intuitive web-based interface that enables effortless network management and monitoring capabilities from any corner of the globe. Whether you're an IT professional overseeing an enterprise network or a curious home user exploring your network's intricacies, L3MON offers a comprehensive suite of tools to streamline your network management tasks.

Note

This is currently under development. The information provided in this document is subject to change.
Development is currently outside of GitHub. This repository will be updated once development is complete.

Key Features

1. Web-Based User Interface

L3MON boasts a sleek and user-friendly web-based interface that ensures easy access to your network management tasks. With its intuitive design, you can seamlessly navigate through various features and monitor your network's health and performance at a glance.

2. Network Bandwidth Monitoring

Gain insights into your network's bandwidth consumption and distribution. L3MON provides real-time and historical bandwidth usage data, enabling you to identify trends, detect anomalies, and optimize your network resources.

3. Server Monitoring

L3MON goes beyond simple network monitoring by offering comprehensive server monitoring capabilities. Keep a close watch on your servers' vital statistics, resource utilization, and performance metrics. Ensure your servers are running optimally and take proactive measures to prevent potential downtime.

4. Remote Accessibility

Manage your network from virtually anywhere in the world. L3MON's remote accessibility feature eliminates geographical barriers, allowing you to make informed decisions and take immediate actions regardless of your location.

5. Customizable Alerts

Set up personalized alerts to stay informed about critical events in your network. Whether it's a sudden bandwidth spike, a server reaching its resource limit, or any other noteworthy occurrence, L3MON can notify you via email, SMS, or other preferred communication channels.

6. Extensibility and Scalability

Built on the robust NodeJS framework, L3MON is designed for scalability and extensibility. Tailor the system to your specific network requirements by incorporating custom modules, plugins, and integrations.

Getting Started

To deploy and start using L3MON, follow these simple steps:

Clone the Repository: Begin by cloning the L3MON repository to your server.
```
git clone https://github.com/your-username/l3mon.git
```
Install Dependencies: Navigate to the cloned directory and install the required dependencies.
```
cd l3mon
npm install
```
Configuration: Customize the configuration settings in config.js to match your network environment.
Database Setup: Set up the database according to the specifications provided in the documentation.
Start L3MON: Launch the L3MON application.
Access the Interface: Open your web browser and navigate to http://localhost:3000 to access the L3MON web interface.

For detailed installation instructions, troubleshooting, and advanced configuration options, please refer to the L3MON Documentation.

Contributing

We welcome contributions from the community to enhance L3MON's functionality and address issues. To contribute:

Fork the repository and create a new branch.
Make your changes, ensuring to follow the project's coding standards.
Submit a pull request detailing your changes and improvements.
Please refer to the Contribution Guidelines for more information.

L3MON revolutionizes network management and monitoring, providing you with the tools you need to ensure a seamless and optimized network infrastructure. Explore its features, contribute to its development, and join the community to shape the future of network management with L3MON.

Gibberlings3/TyrisFlare

2023-09-10T01:52:26Z

Meet Tyris Flare, a red-haired amazon who can kick ass and hurl fire! Sound familiar? That's because I decided to have a nostalgia trip and recreate a character from ye olde Golden Axe computer games that were released on the Sega Mega Drive (or Genesis in America) and arcade machines in the 90s.

feg-giessen/VideoCtrl

2023-09-10T01:52:29Z

Device for controlling projectors, video matrix-switchers and an ATEM video switcher.

hotwired/turbo-native-demo

2023-09-11T01:30:06Z

Demo web server for the Turbo Native adapters

jaketrent/html-webpack-template

2023-09-10T01:52:43Z

a better default template for html-webpack-plugin

elixir-webrtc/rel

2023-09-10T01:31:30Z

Standalone TURN/STUN server in Elixir

glossia/noora

2023-09-10T01:31:30Z

🎨 A design system for Phoenix apps inspired by Radix UI and Shadcn

liveview-native/elixirconf_chat

2023-09-10T01:52:49Z

ElixirConf 2023 Chat

hexlet-codebattle/codebattle

2023-09-10T01:52:49Z

Codebattle - Game for programmers, where you solved algorithmic problems with other players in real time. You can fight with other players in single player mode or join massive tournaments.

scottming/iex-unit

2023-09-10T01:52:49Z

ExUnit.run in IEx shell

elm/html

2023-09-10T01:52:45Z

Use HTML in Elm!

vinayurala/emacs_init

2023-09-10T01:31:37Z

My Emacs Init file backup

My emacs init file; contains emacs customizations.

leahneukirchen/gitsum

2023-09-10T01:31:37Z

basic darcsum feelalike for Git

= gitsum.el --- basic darcsum feelalike for Git

Gitsum provides a mode to do interactive partial commits with Emacs. Run M-x gitsum' inside a Git repository and kill everything you don't want to commit with k'. You can navigate easily between hunks with n' and p' and between files with N' and P'. A whole file can be killed with K'. If you want to refine the patch, you can split it with C-c C-s'. If you toggle read-only with `C-x C-q', you can edit everything freely (be careful!).

If you are ready, commit the patch with C-c C-c' or just c'. Enter the message and press `C-c C-c' again.

N.B: If you want to add files in the commit, use M-x git-status' from git.el, add the files with a', then switch to gitsum with `s'. You can only add files as a whole. You need git.el from the git repository then.

Repository: http://github.com/chneukirchen/gitsum git://github.com/chneukirchen/gitsum.git Patches to: chneukirchen@gmail.com

sebastiencs/company-box

2023-09-10T01:52:57Z

A company front-end with icons

company-box

A company front-end with icons.

Differences with the built-in front-end:

Differents colors for differents backends.
Icons associated to functions/variables/.. and their backends
Display candidate's documentation (support quickhelp-string)
Not limited by the current window size, buffer's text properties, .. (it's better than you might think)

This package requires emacs 26.
Also, not compatible with emacs in a tty.

Installation

;; With use-package:
(use-package company-box
  :hook (company-mode . company-box-mode))

;; Or:
(require 'company-box)
(add-hook 'company-mode-hook 'company-box-mode)

To customize:
M-x customize-group [RET] company-box [RET]

Backends colors

See the docstring of the variable company-box-backends-colors:
C-h v company-box-backends-colors

Themes

You can select different themes with company-box-icons-alist

Icons

See the variable company-box-icons-functions

For now, there are customs icons for 4 backends only: company-lsp, company-elisp, company-yasnippet and company-php.
You can customize their icons with the variables:
company-box-icons-lsp, company-box-icons-elisp, company-box-icons-yasnippet and company-box-icons-acphp.

Notes:
By default, images are used to display icons.
You can also use font icons
With images, you can't change icons colors

DarthFennec/highlight-indent-guides

2023-09-10T01:52:57Z

Emacs minor mode to highlight indentation

highlight-indent-guides.el

This minor mode highlights indentation levels via font-lock. Indent widths are dynamically discovered, which means this correctly highlights in any mode, regardless of indent width, even in languages with non-uniform indentation such as Haskell. By default, this mode also inspects your theme dynamically, and automatically chooses appropriate colors for highlighting. This mode works properly around hard tabs and mixed indentation, and it behaves well in large buffers.

Screenshots

M-x customize-variable RET highlight-indent-guides-method RET ...

`'fill`	`'column`

`'character`	`'bitmap`

Installation

To install from Melpa, use M-x package-install RET highlight-indent-guides RET.

To install from GNU Guix, run guix install emacs-highlight-indent-guides.

Otherwise, download highlight-indent-guides.el and put it in your load path.

Usage

Once the mode is installed, do M-x highlight-indent-guides-mode to enable it. To enable it automatically in most programming modes, use the following:

(add-hook 'prog-mode-hook 'highlight-indent-guides-mode)

Configuration

This mode supports four display methods. To change the display method, customize highlight-indent-guides-method, and set it to one of the following:

fill: The default method. All whitespace used for indentation is highlighted. The color of each level of indentation alternates between highlight-indent-guides-odd-face and highlight-indent-guides-even-face.
column: Like fill, but only the first column of each level of indentation is highlighted.
character: The first column of each level of indentation is drawn using a column of characters. The character to draw with is specified by highlight-indent-guides-character, and it is drawn using the face highlight-indent-guides-character-face.
bitmap: Like character, but an image is used in place of a character. This provides a wider variety of appearance options, and ensures that guides are always flush, not broken if the line height exceeds the character height. The image to use can be set by overloading the highlight-indent-guides-bitmap-function variable with a custom function.

For example:

M-x customize-variable RET highlight-indent-guides-method RET 'character

To change the character used for drawing guide lines with the character display method, customize highlight-indent-guides-character.

For example:

M-x customize-variable RET highlight-indent-guides-character RET ?|

Highlight Colors

By default, this mode dynamically chooses colors that look acceptable with the loaded theme. It does this by altering the luminosity of the theme's background color by a given percentage. These percentages can be tweaked, to make the colors more intense or subtle.

For example:

M-x customize-variable RET highlight-indent-guides-auto-odd-face-perc RET 15 M-x customize-variable RET highlight-indent-guides-auto-even-face-perc RET 15 M-x customize-variable RET highlight-indent-guides-auto-character-face-perc RET 20

To set the colors manually, disable this feature and customize the faces directly.

For example:

M-x customize-variable RET highlight-indent-guides-auto-enabled RET nil

(set-face-background 'highlight-indent-guides-odd-face "darkgray")
(set-face-background 'highlight-indent-guides-even-face "dimgray")
(set-face-foreground 'highlight-indent-guides-character-face "dimgray")

In some configurations, the following error might show up when emacs starts:

Error: highlight-indent-guides cannot auto set faces: `default' face is not set properly

This is meant as a warning for when the faces can't be set, but in some situations the error might show up even when the faces are set properly. If this happens regularly, the error can be suppressed by customizing highlight-indent-guides-suppress-auto-error:

M-x customize-variable RET highlight-indent-guides-suppress-auto-error RET t

Responsive Guides

Responsive guides allow you to visualize not only the indentation itself, but your place in it. To enable this feature, customize highlight-indent-guides-responsive, and set it to one of the following:

nil: The default. Responsive guides are disabled.
top: Use a different color to highlight the "current" guide (the indentation block of the line that the cursor is on). This changes as the cursor moves.
stack: Like top, but also use a third color for all "ancestor" guides of the current guide. Again, this will change as the cursor moves around.

By default, responsive guides are not updated immediately every time the cursor moves. Instead, guides only update after the cursor stops moving for a certain period of time (one tenth of a second, by default). If you would like to change this behavior, customize highlight-indent-guides-delay, and set it to the number of seconds to wait. For example, to disable the delay entirely:

M-x customize-variable RET highlight-indent-guides-delay RET 0

Enabling this feature provides more highlight faces, as well as more color modifiers for the dynamic colors feature. These are specified in the following table:

Type	Level	Method	Variable
face	nil	odd	`highlight-indent-guides-odd-face`
face	nil	even	`highlight-indent-guides-even-face`
face	nil	character	`highlight-indent-guides-character-face`
face	top	odd	`highlight-indent-guides-top-odd-face`
face	top	even	`highlight-indent-guides-top-even-face`
face	top	character	`highlight-indent-guides-top-character-face`
face	stack	odd	`highlight-indent-guides-stack-odd-face`
face	stack	even	`highlight-indent-guides-stack-even-face`
face	stack	character	`highlight-indent-guides-stack-character-face`
perc	nil	odd	`highlight-indent-guides-auto-odd-face-perc`
perc	nil	even	`highlight-indent-guides-auto-even-face-perc`
perc	nil	character	`highlight-indent-guides-auto-character-face-perc`
perc	top	odd	`highlight-indent-guides-auto-top-odd-face-perc`
perc	top	even	`highlight-indent-guides-auto-top-even-face-perc`
perc	top	character	`highlight-indent-guides-auto-top-character-face-perc`
perc	stack	odd	`highlight-indent-guides-auto-stack-odd-face-perc`
perc	stack	even	`highlight-indent-guides-auto-stack-even-face-perc`
perc	stack	character	`highlight-indent-guides-auto-stack-character-face-perc`

Custom Highlighter Function

The highlighter function is the function that calculates which faces to use to display each guide character. If the default highlighter function isn't doing it for you, you can write your own by customizing highlight-indent-guides-highlighter-function. A custom highlighter takes three parameters:

level: The indent level this guide character exists at, starting at 0.
responsive: The responsive class of this guide character. This can be nil, top, or stack.
display: The display method setting. One of fill, column, character, or bitmap.

A custom highlighter should return the face to use to color the given guide character. Alternatively, it may return nil to specify that the guide should not be displayed at all.

The highlighter function is called once for each indentation character, each time a section of the buffer is re-highlighted. To speed things up a little, the results of the highlighter function are memoized per-character, and are reused when possible. Because of this, a custom highlighter should run quickly, and should not have side-effects (i.e. it should not depend on or change external values that might differ from one call to the next). A custom highlighter can return custom faces, but those faces will not be recognized by the dynamic color feature, and will need to be defined and colored manually.

The following example highlighter will highlight normally, except that it will not highlight the first two levels of indentation:

(defun my-highlighter (level responsive display)
  (if (> 2 level)
      nil
    (highlight-indent-guides--highlighter-default level responsive display)))

(setq highlight-indent-guides-highlighter-function 'my-highlighter)

Custom Bitmap Function

If you're using the 'bitmap display method, you may set a custom bitmap function, which determines what your guides will look like. Customize highlight-indent-guides-bitmap-function, and set it to:

highlight-indent-guides--bitmap-dots: A guide is a column of small dots. This is the default.
highlight-indent-guides--bitmap-line: A guide is a solid vertical line.
Or, write your own.

A custom bitmap function takes four parameters:

width: The width in pixels of the bitmap.
height: The height in pixels of the bitmap.
crep: A character that represents a "filled" or "colored" pixel. This is as opposed to an "empty" pixel, which the background color will show through.
zrep: A character that represents an "empty" pixel.

The function should return a list of string lists, representing the pixels themselves. The list must contain height sublists, and each sublist must contain width strings, all of which are either crep for a colored pixel or zrep for an empty pixel.

Limitations

To display the character method guides, and to highlight tab characters correctly, this mode controls the display text property of some characters via font-lock. Therefore, this mode may or may not play well with other modes that use the display text property. This mode may also interfere with modes that use a display table to modify how whitespace is drawn, e.g., the whitespace minor mode.

The bitmap display method can only be used if emacs is compiled with xpm support, and is running in gui mode.

Currently, with the way this mode is designed, there is no good way to display indent guides on empty lines.

Alternatives

Package Name	Widths	Hard Tabs	Other Notes
highlight-indentation.el	Fixed	Unsupported	Very popular
indent-guide.el	Dynamic	Supported	Fairly slow, jittery
hl-indent.el	Dynamic	Unsupported	Slow for large files
visual-indentation-mode.el	Fixed	Unsupported	Fast and slim

emacs-lsp/lsp-java

2023-09-10T01:52:57Z

lsp-mode ❤️ java

lsp-java

Emacs Java IDE using Eclipse JDT Language Server.

Requirements

JDK 17

Features

LSP java mode supports the following JDT Features:

As you type reporting of parsing and compilation errors (via flycheck/lsp-ui)
Code completion - using company-capf or builtin complete-at-point
Javadoc hovers - using lsp-ui
Code actions - using lsp-ui
Code outline - using builtin imenu
Code navigation - using builtin xref
Code lens (references/implementations) - using builtin xref
Highlights
Code formatting
Maven pom.xml project support
Limited Gradle support
Visual debugger - dap-mode
Test runner - dap-mode
Project explorer integration - treemacs
Integration with Spring Initializr

Demo

Here it is a demo from EmacsConf2019 https://www.youtube.com/watch?v=Bbjxn9yVNJ8

Tutorial

Here is a tutorial that covers setup and use https://xpressrazor.wordpress.com/2020/11/04/java-programming-in-emacs/

Screenshot

Installation

Spacemacs

lsp-java is included in spacemacs (for now only on the dev branch). If you are using the development version of spacemacs you can simply add (java :variables java-backend 'lsp) to dotspacemacs-configuration-layers.

Install via melpa

The recommended way to install LSP Java is via package.el - the built-in package manager in Emacs. LSP Java is available on the two major package.el community maintained repos - MELPA Stable and MELPA.

M-x package-install [RET] lsp-java [RET]

Then add the following lines to your .emacs and open a file from the any of the specified projects.

(require 'lsp-java)
(add-hook 'java-mode-hook #'lsp)

Eclipse JDT Language Server

lsp-java will automatically detect when the server is missing and it will download Eclipse JDT Language Server before the first startup. The server installation will be in lsp-java-server-install-dir. It will detect whether dap-mode is present and it will download the required server side plugins/components. If you want to update the server you can run lsp-java-update-server. To run specific version of Eclipse JDT Language Server use lsp-java-server-install-dir.

Quick start

Minimal configuration with company-capf and lsp-ui and dap-mode. Set lsp-java-workspace-dir in case you have existing Java projects. Now you can explore the methods under lsp-java-*, dap-java-*, dap-*, and lsp-*.

(condition-case nil
    (require 'use-package)
  (file-error
   (require 'package)
   (add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))
   (package-initialize)
   (package-refresh-contents)
   (package-install 'use-package)
   (setq use-package-always-ensure t)
   (require 'use-package)))

(use-package projectile)
(use-package flycheck)
(use-package yasnippet :config (yas-global-mode))
(use-package lsp-mode :hook ((lsp-mode . lsp-enable-which-key-integration)))
(use-package hydra)
(use-package company)
(use-package lsp-ui)
(use-package which-key :config (which-key-mode))
(use-package lsp-java :config (add-hook 'java-mode-hook 'lsp))
(use-package dap-mode :after lsp-mode :config (dap-auto-configure-mode))
(use-package dap-java :ensure nil)
(use-package helm-lsp)
(use-package helm
  :config (helm-mode))
(use-package lsp-treemacs)

Supported commands

LSP Mode commands

lsp-execute-code-action - Execute code action.
lsp-rename - Rename symbol at point
lsp-describe-thing-at-point - Display help for the thing at point.
lsp-goto-type-definition - Go to type definition
lsp-goto-implementation - Go to implementation
lsp-restart-workspace - Restart project
lsp-format-buffer - Format current buffer
lsp-symbol-highlight - Highlight all relevant references to the symbol under point.
lsp-workspace-folders-add - Add workspace folder
lsp-workspace-folders-remove - Remove workspace folder
lsp-workspace-folders-switch - Switch workspace folder

LSP Java commands

lsp-java-organize-imports - Organize imports
lsp-java-build-project - Perform partial or full build for the projects
lsp-java-update-project-configuration - Update project configuration
lsp-java-actionable-notifications - Resolve actionable notifications
lsp-java-update-user-settings - Update user settings (Check the options in the table bellow.)
lsp-java-update-server - Update server instalation.
lsp-java-generate-to-string - Generate toString method.
lsp-java-generate-equals-and-hash-code - Generate equals and hashCode methods.
lsp-java-generate-overrides - Generate method overrides
lsp-java-generate-getters-and-setters - Generate getters and setters.
lsp-java-type-hierarchy - Open type hierarchy. Use prefix arg to specify the type of the hierarchy.

Refactoring

LSP Java provides rich set of refactorings via Eclipse JDT Language Server code actions and some of them are bound to Emacs commands:

lsp-java-extract-to-constant - Extract constant refactoring
lsp-java-add-unimplemented-methods - Extract constant refactoring
lsp-java-create-parameter - Create parameter refactoring
lsp-java-create-field - Create field refactoring
lsp-java-create-local - Create local refactoring
lsp-java-extract-method - Extract method refactoring
lsp-java-add-import - Add missing import

Testing support

lsp-jt-browser - Browse tests and run/debug them. ** Use x to run the test(s) under point; d to debug the tests under point. R to refresh. ** Support for GUI operations.
lsp-jt-report-open - open test report
lsp-jt-lens-mode - test lenses mode(requires lsp-lens-mode to be enabled.)

Dependency viewer

lsp-java-dependency-list - View java dependencies

STS4 Integration (experimental)

LSP java has integration with STS4 providing the following functionality.

Spring boot support (Experimental)

In addition to the integration with Eclipse JDT Language Server lsp-java provides integration with STS4 which covers Spring Boot application.properties, application.yml and .java files.

Usage:

Make sure that you have configured JAVA_HOME. lsp-java will automatically download the STS4 when you call lsp-java-update-server. In order to enable STS4 integration add the following lines to your config:

(require 'lsp-java-boot)

;; to enable the lenses
(add-hook 'lsp-mode-hook #'lsp-lens-mode)
(add-hook 'java-mode-hook #'lsp-java-boot-lens-mode)

Functionality for `.java`

Navigating the source code - Go to symbol in file/workspace

Easy navigation to Spring-specific elements of your source code.

Commands

lsp-workspace-symbol - (works better usign helm-lsp)

Examples

@/ shows all defined request mappings (mapped path, request method, source location)
@+ shows all defined beans (bean name, bean type, source location)
@> shows all functions (prototype implementation)
@ shows all Spring annotations in the code

Quick-access for running apps

Easy navigation to the provided request mappings of running apps.

Commands

lsp-workspace-symbol - (works better usign helm-lsp)

Examples

// shows all request mappings of all running Spring Boot apps and opens a browser for the selected endpoint

Live application information hovers

STS4 automatically detects JVM processes for running boot applications on your local machine.

For some types of information, STS 4 may also show a 'quick summary' as a codelens.

If there are multiple instances of the app running on your machine, the live data from all those instances will show up in the hover information.

(add-hook 'java-mode-hook #'lsp-java-boot-lens-mode)

Examples

@Profile: shows information about the active profiles on the running apps
@Component, @Bean, @Autowired: shows detailed information about the beans and their wiring from the live app
@ContidionalOn...: shows information about the conditions and their evaluation at runtime

Code templates

Write Spring code with templates, available via regular code completion.

Examples

@GetMapping
@PostMapping
@PutMapping

Smart code completions

Additional code completions for Spring-specific annotations

Functionality for `.properties` and `.yml`

This extension analyzes your project's classpath and parses and indexes any Spring Boot Properties Metadata it finds. Both Maven and Gradle projects are supported.

The data in the index is used to provide validation, code completions and information hovers while editing Spring Boot Properties in either .properties or .yml format.

Validation and code completion in properties file

Validation and code completion in yaml file

Spring Initializr

lsp-java provides a frontend for Spring Initializr which simplifies the creation of Spring Boot projects directly from Emacs via lsp-java-spring-initializr.

Supported settings

lsp-java-server-install-dir - "Install directory for eclipse.jdt.ls-server. The slash is expected at the end."
lsp-java-jdt-download-url - JDT JS download url. Use http://download.eclipse.org/che/che-ls-jdt/snapshots/che-jdt-language-server-latest.tar.gz if you want to use Eclipse Che JDT LS.
lsp-java-java-path - Path of the java executable.
lsp-java-progress-string - Java progress status as reported by the language server.
lsp-java-workspace-dir - LSP java workspace directory.
lsp-java-workspace-cache-dir - LSP java workspace cache directory.
lsp-java-themes-directory - Directory containing themes.
lsp-java-theme - Theme to use.
lsp-java-pop-buffer-function - The function which will be used for showing the helper windows.
lsp-java-vmargs - Specifies extra VM arguments used to launch the Java Language Server. Eg. use -noverify -Xmx1G -XX:+UseG1GC -XX:+UseStringDeduplication to bypass class verification,increase the heap size to 1GB and enable String deduplication with the G1 Garbage collector
lsp-java-9-args - Specifies arguments specific to java 9 and later.
lsp-java-errors-incomplete-classpath-severity - Specifies the severity of the message when the classpath is incomplete for a Java file
lsp-java-configuration-check-project-settings-exclusions - Checks if the extension-generated project settings files (.project, .classpath, .factorypath, .settings/) should be excluded from the file explorer.
lsp-java-configuration-update-build-configuration - Specifies how modifications on build files update the Java classpath/configuration
lsp-java-trace-server - Traces the communication between VS Code and the Java language server.
lsp-java-import-gradle-enabled - Enable/disable the Gradle importer.
lsp-java-import-maven-enabled - Enable/disable the Maven importer.
lsp-java-maven-download-sources - Enable/disable eager download of Maven source artifacts.
lsp-java-references-code-lens-enabled - Enable/disable the references code lens.
lsp-java-signature-help-enabled - Enable/disable the signature help.
lsp-java-implementations-code-lens-enabled - Enable/disable the implementations code lens.
lsp-java-configuration-maven-user-settings - Path to Maven's settings.xml
lsp-java-format-enabled - Enable/disable default Java formatter
lsp-java-save-actions-organize-imports - Enable/disable auto organize imports on save action
lsp-java-import-exclusions - Configure glob patterns for excluding folders
lsp-java-content-provider-preferred - Preferred content provider (a 3rd party decompiler id, usually). We support https://github.com/dgileadi/vscode-java-decompiler. To enable it, add:
```
(setq lsp-java-content-provider-preferred "fernflower")
```
lsp-java-autobuild-enabled - Enable/disable the 'auto build'
lsp-java-max-concurrent-builds - Max simultaneous project builds
lsp-java-completion-enabled - Enable/disable code completion support
lsp-java-completion-overwrite - When set to true, code completion overwrites the current text. When set to false, code is simply added instead.
lsp-java-completion-guess-method-arguments - When set to true, method arguments are guessed when a method is selected from as list of code assist proposals.
lsp-java-completion-favorite-static-members - Defines a list of static members or types with static members. Content assist will propose those static members even if the import is missing.
lsp-java-completion-import-order - Defines the sorting order of import statements. A package or type name prefix (e.g. 'org.eclipse') is a valid entry. An import is always added to the most specific group.
lsp-java-folding-range-enabled - Enable/disable smart folding range support. If disabled, it will use the default indentation-based folding range provided by VS Code.
indentation-based - [Experimental] Enable/disable progress reports from background processes on the server.
lsp-java-progress-reports-enabled - [Experimental] Enable/disable progress reports from background processes on the server.
lsp-java-format-settings-url - Specifies the url or file path to the Eclipse formatter xml settings.
lsp-java-format-settings-profile - Optional formatter profile name from the Eclipse formatter settings.
lsp-java-format-comments-enabled - Includes the comments during code formatting.
lsp-java-format-on-type-enabled - Enable/disable automatic block formatting when typing ;, <enter> or }
lsp-java-bundles - List of bundles that will be loaded in the JDT server.
lsp-java-code-generation-hash-code-equals-use-java7objects - Use Objects.hash and Objects.equals when generating the hashCode and equals methods. This setting only applies to Java 7 and higher.
lsp-java-code-generation-hash-code-equals-use-instanceof - Use 'instanceof' to compare types when generating the hashCode and equals methods.
lsp-java-code-generation-use-blocks - Use blocks in 'if' statements when generating the methods.
lsp-java-code-generation-generate-comments - Generate method comments when generating the methods.
lsp-java-code-generation-to-string-template - The template for generating the toString method.
lsp-java-code-generation-to-string-code-style - The code style for generating the toString method.
lsp-java-code-generation-to-string-skip-null-values - Skip null values when generating the toString method.
lsp-java-code-generation-to-string-list-array-contents - List contents of arrays instead of using native toString().
lsp-java-code-generation-to-string-limit-elements - Limit number of items in arrays/collections/maps to list, if 0 then list all.
lsp-java-inhibit-message - If non-nil, inhibit java messages echo via `inhibit-message'.

Additional packages

lsp-ui : Flycheck, documentation and code actions support.
company-capf : Company backend support.
treemacs : Project viewer.
lsp-treemacs : lsp-mode GUI controls implemented using treemacs.

FAQ

LSP Java is showing to many debug messages, how to stop that? Add the following configuration.

(setq lsp-inhibit-message t)

lsp-ui does not display all of the actions on the current point(e. g "Extract constant")? LSP UI by default sends current line bounds for action region which breaks forces JDT server to return only "Extract method action."

(setq lsp-ui-sideline-update-mode 'point)

LSP Java does not provide completion, go to definition for some of the files?

When particular file is not part of imported project Eclipse JDT Language Server could not calculate the current classpath.

How do I change the version of java used by LSP?

Have a look at ~/.emacs.d/workspace/.metadata/.plugins/org.eclipse.jdt.launching/libraryInfos.xml. If you updated your local java path and want LSP to use the new version, try removing the ~/.emacs.d/workspace/ directory and relaunch LSP. Also see #114. If you have multiple java JDK versions installed and want to change the version of java used by LSP, also need to set lsp-java-configuration-runtimes. An example for setting lsp-java-configuration-runtimes:

(setq lsp-java-configuration-runtimes '[(:name "JavaSE-1.8"
						:path "/home/kyoncho/jdk1.8.0_201.jdk/")
					(:name "JavaSE-11"
						:path "/home/kyoncho/jdk-11.0.1.jdk/"
						:default t)])

How do I change JVM args passed to JDT server? LSP slowness could be caused by slow JDT server, especially on large JAVA projects. Bump up the heap size maybe a good idea.

;; current VSCode defaults
(setq lsp-java-vmargs '("-XX:+UseParallelGC" "-XX:GCTimeRatio=4" "-XX:AdaptiveSizePolicyWeight=90" "-Dsun.zip.disableMemoryMapping=true" "-Xmx2G" "-Xms100m"))

Feuerlabs/exoiodev

2023-09-10T01:32:05Z

IO device sample application

oujinliang/erlang_utils

2023-09-10T01:53:13Z

Some utilities code for Erlang

kivra/raven-erlang

2023-09-10T01:53:13Z

Feuerlabs/exoiodev

2023-09-10T01:53:13Z

IO device sample application

dracid/Pangram90

2023-09-10T01:53:34Z

Simple pangram checker written in FORTRAN, just an exercise

diku-dk/pmph-e2023-pub

2023-09-10T01:53:35Z

ThomasToka/MarlinFirmware

2023-09-11T01:30:52Z

This is [MARLIN-E3S1PROFORK-BYTT]. A Fork of Marlin 2.1.x-bugfix with the Creality Ender 3 S1 Pro source code included. Personal tweaks, fixes and enhancements included. This firmware is for the Ender 3 S1 Pro (touchscreen) and Ender 3 S1 Plus (touchscreen).

SteamDatabase/GameTracking-CS2

2023-09-10T01:53:43Z

📥 Game Tracker: Counter-Strike 2

Zennyth/EnhancedStat

2023-09-11T01:31:01Z

Enhanced Stat is an addon for Godot 4.1, designed to provide an intuitive and powerful way to manage stats in games such as Health, Mana, Speed, and Attack Damage.

godotessentials/2d-essentials

2023-09-10T01:53:48Z

Godot 2D essentials is a collection of production ready components to speed up the development of your 2d games. This components handles basic behaviours without affecting the particular logic of your game.

clear-code-projects/UltimateGodotIntro2D

2023-09-10T01:53:48Z

Resources for the ultimate introduction to Godot 4 [2D]

sqlc-dev/sqlc

2023-09-10T01:33:10Z

GitHub's Online Schema-migration Tool for MySQL

gh-ost

GitHub's online schema migration for MySQL

gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides pausability, dynamic control/reconfiguration, auditing, and many operational perks.

gh-ost produces a light workload on the master throughout the migration, decoupled from the existing workload on the migrated table.

It has been designed based on years of experience with existing solutions, and changes the paradigm of table migrations.

How?

All existing online-schema-change tools operate in similar manner: they create a ghost table in the likeness of your original table, migrate that table while empty, slowly and incrementally copy data from your original table to the ghost table, meanwhile propagating ongoing changes (any INSERT, DELETE, UPDATE applied to your table) to the ghost table. Finally, at the right time, they replace your original table with the ghost table.

gh-ost uses the same pattern. However it differs from all existing tools by not using triggers. We have recognized the triggers to be the source of many limitations and risks.

Instead, gh-ost uses the binary log stream to capture table changes, and asynchronously applies them onto the ghost table. gh-ost takes upon itself some tasks that other tools leave for the database to perform. As result, gh-ost has greater control over the migration process; can truly suspend it; can truly decouple the migration's write load from the master's workload.

In addition, it offers many operational perks that make it safer, trustworthy and fun to use.

Highlights

Build your trust in gh-ost by testing it on replicas. gh-ost will issue same flow as it would have on the master, to migrate a table on a replica, without actually replacing the original table, leaving the replica with two tables you can then compare and satisfy yourself that the tool operates correctly. This is how we continuously test gh-ost in production.
True pause: when gh-ost throttles, it truly ceases writes on master: no row copies and no ongoing events processing. By throttling, you return your master to its original workload
Dynamic control: you can interactively reconfigure gh-ost, even as migration still runs. You may forcibly initiate throttling.
Auditing: you may query gh-ost for status. gh-ost listens on unix socket or TCP.
Control over cut-over phase: gh-ost can be instructed to postpone what is probably the most critical step: the swap of tables, until such time that you're comfortably available. No need to worry about ETA being outside office hours.
External hooks can couple gh-ost with your particular environment.

Please refer to the docs for more information. No, really, read the docs.

Usage

The cheatsheet has it all. You may be interested in invoking gh-ost in various modes:

a noop migration (merely testing that the migration is valid and good to go)
a real migration, utilizing a replica (the migration runs on the master; gh-ost figures out identities of servers involved. Required mode if your master uses Statement Based Replication)
a real migration, run directly on the master (but gh-ost prefers the former)
a real migration on a replica (master untouched)
a test migration on a replica, the way for you to build trust with gh-ost's operation.

Our tips:

Testing above all, try out --test-on-replica first few times. Better yet, make it continuous. We have multiple replicas where we iterate our entire fleet of production tables, migrating them one by one, checksumming the results, verifying migration is good.
For each master migration, first issue a noop
Then issue the real thing via --execute.

More tips:

Use --exact-rowcount for accurate progress indication
Use --postpone-cut-over-flag-file to gain control over cut-over timing
Get familiar with the interactive commands

Also see:

What's in a name?

Originally this was named gh-osc: GitHub Online Schema Change, in the likes of Facebook online schema change and pt-online-schema-change.

But then a rare genetic mutation happened, and the c transformed into t. And that sent us down the path of trying to figure out a new acronym. gh-ost (pronounce: Ghost), stands for GitHub's Online Schema Transmogrifier/Translator/Transformer/Transfigurator

License

gh-ost is licensed under the MIT license

gh-ost uses 3rd party libraries, each with their own license. These are found here.

Community

gh-ost is released at a stable state, but with mileage to go. We are open to pull requests. Please first discuss your intentions via Issues.

We develop gh-ost at GitHub and for the community. We may have different priorities than others. From time to time we may suggest a contribution that is not on our immediate roadmap but which may appeal to others.

Please see Coding gh-ost for a guide to getting started developing with gh-ost.

Download/binaries/source

gh-ost is now GA and stable.

gh-ost is available in binary format for Linux and Mac OS/X

Download latest release here

gh-ost is a Go project; it is built with Go 1.15 and above. To build on your own, use either:

script/build - this is the same build script used by CI hence the authoritative; artifact is ./bin/gh-ost binary.
build.sh for building tar.gz artifacts in /tmp/gh-ost

Generally speaking, master branch is stable, but only releases are to be used in production.

Authors

gh-ost is designed, authored, reviewed and tested by the database infrastructure team at GitHub:

vektra/mockery

2023-09-10T01:33:10Z

Keybase Go Library, Client, Service, OS X, iOS, Android, Electron

Keybase

Hi, and welcome to the Keybase client repo. All our client apps (macOS, Windows, Linux, iOS, and Android) are being actively developed in this repository. Please, dig around.

Warnings

We'd love you to read our source code.

But - some of the things in this repo are explorations, and the app you build from source just might not do what it says it's doing. So, if you just want to install Keybase on your computer, you should monitor our releases for macOS, Linux, or Windows.

If you're interested in our Docker image releases, please check out the Docker README.

Code Layout

go: Core crypto libraries; the Keybase service; the command line client. Learn More
shared/{android,ios}: Android and iOS apps developed with React Native.
shared/desktop: Desktop application for macOS, Linux, and Windows, made with the Electron framework, sharing React code with react-native.
packaging: Scripts for releasing packages across the various platforms.
protocol: Defines the protocol for communication for clients to the Keybase services. Uses Avro. Learn More
media: Icons, graphics, media for Keybase apps.
osx: The macOS Keybase.app, development parallel to an Electron-based application above. Learn More

Problems?

Report any issues with client software on this GitHub issue tracker. Internally, we track our progress using Jira, but all PRs come through GitHub for your review!

If you're having problems with the command line keybase client, take a look at the troubleshooting doc.

If you're having problems with our Website, try the keybase-issues issue tracker.

We check and update both frequently.

License

Most code is released under the New BSD (3 Clause) License. If subdirectories include a different license, that license applies instead.

Development Guidelines

We check all git commits with pre-commit hooks generated via pre-commit.com pre-commit hooks. To enable use of these pre-commit hooks:

Install the pre-commit utility. For some common cases:
- pip install pre-commit
- brew install pre-commit
Remove any existing pre-commit hooks via rm .git/hooks/pre-commit
Configure via pre-commit install

Then proceed as normal.

External Contributors

If you forked this repository on GitHub and made a PR, then it'll show up as having failed Jenkins CI. We do not build external PRs because it's a security risk to do so without a review first. If your PR is successfully reviewed by a member of the Keybase team, then we will merge your commits to a branch on our primary fork and build from there.

Cryptography Notice

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org/ for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.

deepfence/PacketStreamer

2023-09-11T01:31:25Z

⭐ ⭐ Distributed tcpdump for cloud native environments ⭐ ⭐

PacketStreamer

Deepfence PacketStreamer is a high-performance remote packet capture and collection tool. It is used by Deepfence's ThreatStryker security observability platform to gather network traffic on demand from cloud workloads for forensic analysis.

Primary design goals:

Stay light, capture and stream, no additional processing
Portability, works across virtual machines, Kubernetes and AWS Fargate. Linux and Windows

PacketStreamer sensors are started on the target servers. Sensors capture traffic, apply filters, and then stream the traffic to a central reciever. Traffic streams may be compressed and/or encrypted using TLS.

The PacketStreamer receiver accepts PacketStreamer streams from multiple remote sensors, and writes the packets to a local pcap capture file

PacketStreamer sensors collect raw network packets on remote hosts. It selects packets to capture using a BPF filter, and forwards them to a central reciever process where they are written in pcap format. Sensors are very lightweight and impose little performance impact on the remote hosts. PacketStreamer sensors can be run on bare-metal servers, on Docker hosts, and on Kubernetes nodes.

The PacketStreamer receiver accepts network traffic from multiple sensors, collecting it into a single, central pcap file. You can then process the pcap file or live feed the traffic to the tooling of your choice, such as Zeek, Wireshark Suricata, or as a live stream for Machine Learning models.

When to use PacketStreamer

PacketStreamer meets more general use cases than existing alternatives. For example , Use PacketStreamer if you need a lightweight, efficient method to collect raw network data from multiple machines for central logging and analysis.

Quick Start

For full instructions, refer to the PacketStreamer Documentation.

You will need to install the golang toolchain and libpcap-dev before building PacketStreamer.

# Pre-requisites (Ubuntu): sudo apt install golang-go libpcap-dev
git clone https://github.com/deepfence/PacketStreamer.git
cd PacketStreamer/
make

Run a PacketStreamer receiver, listening on port 8081 and writing pcap output to /tmp/dump_file (see receiver.yaml):

./packetstreamer receiver --config ./contrib/config/receiver.yaml

Run one or more PacketStreamer sensors on local and remote hosts. Edit the server address in sensor.yaml:

# run on the target hosts to capture and forward traffic

# copy and edit the sample sensor-local.yaml file, and add the address of the receiver host
cp ./contrib/config/sensor-local.yaml ./contrib/config/sensor.yaml

./packetstreamer sensor --config ./contrib/config/sensor.yaml

Who uses PacketStreamer?

Deepfence ThreatStryker uses PacketStreamer to capture traffic from production platforms for forensics and anomaly detection.

Get in touch

Thank you for using PacketStreamer.

Start with the documentation
Got a question, need some help? Find the Deepfence team on Slack
Got a feature request or found a bug? Raise an issue
productsecurity at deepfence dot io: Found a security issue? Share it in confidence
Find out more at deepfence.io

Security and Support

For any security-related issues in the PacketStreamer project, contact productsecurity at deepfence dot io.

Please file GitHub issues as needed, and join the Deepfence Community Slack channel.

License

The Deepfence PacketStreamer project (this repository) is offered under the Apache2 license.

Contributions to Deepfence PacketStreamer project are similarly accepted under the Apache2 license, as per GitHub's inbound=outbound policy.

kubernetes/minikube

2023-09-10T01:54:17Z

Run Kubernetes locally

minikube

minikube implements a local Kubernetes cluster on macOS, Linux, and Windows. minikube's primary goals are to be the best tool for local Kubernetes application development and to support all Kubernetes features that fit.

Features

minikube runs the latest stable release of Kubernetes, with support for standard Kubernetes features like:

LoadBalancer - using minikube tunnel
Multi-cluster - using minikube start -p <name>
NodePorts - using minikube service
Persistent Volumes
Ingress
Dashboard - minikube dashboard
Container runtimes - minikube start --container-runtime
Configure apiserver and kubelet options via command-line flags
Supports common CI environments

As well as developer-friendly features:

Addons - a marketplace for developers to share configurations for running services on minikube
NVIDIA GPU support - for machine learning
Filesystem mounts

For more information, see the official minikube website

Installation

See the Getting Started Guide

📣 Please fill out our fast 5-question survey so that we can learn how & why you use minikube, and what improvements we should make. Thank you! 👯

Documentation

See https://minikube.sigs.k8s.io/docs/

More Examples

See minikube in action here

Community

minikube is a Kubernetes #sig-cluster-lifecycle project.

#minikube on Kubernetes Slack - Live chat with minikube developers!
minikube-users mailing list
minikube-dev mailing list
Contributing
Development Roadmap

Join our meetings:

emirpasic/gods

2023-09-10T01:54:17Z

GoDS (Go Data Structures) - Sets, Lists, Stacks, Maps, Trees, Queues, and much more

GoDS (Go Data Structures)

Implementation of various data structures and algorithms in Go.

Data Structures

Containers
- Lists
- Sets
- Stacks
  - LinkedListStack
  - ArrayStack
- Maps
- Trees
  - RedBlackTree
  - AVLTree
  - BTree
  - BinaryHeap
- Queues
Functions
Appendix

Containers

All data structures implement the container interface with the following methods:

type Container interface {
	Empty() bool
	Size() int
	Clear()
	Values() []interface{}
	String() string
}

Containers are either ordered or unordered. All ordered containers provide stateful iterators and some of them allow enumerable functions.

Data	Structure	Ordered	Iterator	Enumerable	Referenced by
Lists
	ArrayList	yes	yes*	yes	index
	SinglyLinkedList	yes	yes	yes	index
	DoublyLinkedList	yes	yes*	yes	index
Sets
	HashSet	no	no	no	index
	TreeSet	yes	yes*	yes	index
	LinkedHashSet	yes	yes*	yes	index
Stacks
	LinkedListStack	yes	yes	no	index
	ArrayStack	yes	yes*	no	index
Maps
	HashMap	no	no	no	key
	TreeMap	yes	yes*	yes	key
	LinkedHashMap	yes	yes*	yes	key
	HashBidiMap	no	no	no	key*
	TreeBidiMap	yes	yes*	yes	key*
Trees
	RedBlackTree	yes	yes*	no	key
	AVLTree	yes	yes*	no	key
	BTree	yes	yes*	no	key
	BinaryHeap	yes	yes*	no	index
Queues
	LinkedListQueue	yes	yes	no	index
	ArrayQueue	yes	yes*	no	index
	CircularBuffer	yes	yes*	no	index
	PriorityQueue	yes	yes*	no	index
			_^*reversible		_{^{*bidirectional}}

Lists

A list is a data structure that stores values and may have repeated values.

Implements Container interface.

type List interface {
	Get(index int) (interface{}, bool)
	Remove(index int)
	Add(values ...interface{})
	Contains(values ...interface{}) bool
	Sort(comparator utils.Comparator)
	Swap(index1, index2 int)
	Insert(index int, values ...interface{})
	Set(index int, value interface{})

	containers.Container
	// Empty() bool
	// Size() int
	// Clear()
	// Values() []interface{}
    // String() string
}

ArrayList

A list backed by a dynamic array that grows and shrinks implicitly.

Implements List, ReverseIteratorWithIndex, EnumerableWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import (
	"github.com/emirpasic/gods/lists/arraylist"
	"github.com/emirpasic/gods/utils"
)

func main() {
	list := arraylist.New()
	list.Add("a")                         // ["a"]
	list.Add("c", "b")                    // ["a","c","b"]
	list.Sort(utils.StringComparator)     // ["a","b","c"]
	_, _ = list.Get(0)                    // "a",true
	_, _ = list.Get(100)                  // nil,false
	_ = list.Contains("a", "b", "c")      // true
	_ = list.Contains("a", "b", "c", "d") // false
	list.Swap(0, 1)                       // ["b","a",c"]
	list.Remove(2)                        // ["b","a"]
	list.Remove(1)                        // ["b"]
	list.Remove(0)                        // []
	list.Remove(0)                        // [] (ignored)
	_ = list.Empty()                      // true
	_ = list.Size()                       // 0
	list.Add("a")                         // ["a"]
	list.Clear()                          // []
	list.Insert(0, "b")                   // ["b"]
	list.Insert(0, "a")                   // ["a","b"]
}

SinglyLinkedList

A list where each element points to the next element in the list.

Implements List, IteratorWithIndex, EnumerableWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import (
	sll "github.com/emirpasic/gods/lists/singlylinkedlist"
	"github.com/emirpasic/gods/utils"
)

func main() {
	list := sll.New()
	list.Add("a")                         // ["a"]
	list.Add("c", "b")                    // ["a","c","b"]
	list.Sort(utils.StringComparator)     // ["a","b","c"]
	_, _ = list.Get(0)                    // "a",true
	_, _ = list.Get(100)                  // nil,false
	_ = list.Contains("a", "b", "c")      // true
	_ = list.Contains("a", "b", "c", "d") // false
	list.Swap(0, 1)                       // ["b","a",c"]
	list.Remove(2)                        // ["b","a"]
	list.Remove(1)                        // ["b"]
	list.Remove(0)                        // []
	list.Remove(0)                        // [] (ignored)
	_ = list.Empty()                      // true
	_ = list.Size()                       // 0
	list.Add("a")                         // ["a"]
	list.Clear()                          // []
	list.Insert(0, "b")                   // ["b"]
	list.Insert(0, "a")                   // ["a","b"]
}

DoublyLinkedList

A list where each element points to the next and previous elements in the list.

Implements List, ReverseIteratorWithIndex, EnumerableWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import (
	dll "github.com/emirpasic/gods/lists/doublylinkedlist"
	"github.com/emirpasic/gods/utils"
)

func main() {
	list := dll.New()
	list.Add("a")                         // ["a"]
	list.Add("c", "b")                    // ["a","c","b"]
	list.Sort(utils.StringComparator)     // ["a","b","c"]
	_, _ = list.Get(0)                    // "a",true
	_, _ = list.Get(100)                  // nil,false
	_ = list.Contains("a", "b", "c")      // true
	_ = list.Contains("a", "b", "c", "d") // false
	list.Swap(0, 1)                       // ["b","a",c"]
	list.Remove(2)                        // ["b","a"]
	list.Remove(1)                        // ["b"]
	list.Remove(0)                        // []
	list.Remove(0)                        // [] (ignored)
	_ = list.Empty()                      // true
	_ = list.Size()                       // 0
	list.Add("a")                         // ["a"]
	list.Clear()                          // []
	list.Insert(0, "b")                   // ["b"]
	list.Insert(0, "a")                   // ["a","b"]
}

Sets

A set is a data structure that can store elements and has no repeated values. It is a computer implementation of the mathematical concept of a finite set. Unlike most other collection types, rather than retrieving a specific element from a set, one typically tests an element for membership in a set. This structure is often used to ensure that no duplicates are present in a container.

Set additionally allow set operations such as intersection, union, difference, etc.

Implements Container interface.

type Set interface {
	Add(elements ...interface{})
	Remove(elements ...interface{})
	Contains(elements ...interface{}) bool
    // Intersection(another *Set) *Set
    // Union(another *Set) *Set
    // Difference(another *Set) *Set
	
	containers.Container
	// Empty() bool
	// Size() int
	// Clear()
	// Values() []interface{}
	// String() string
}

HashSet

A set backed by a hash table (actually a Go's map). It makes no guarantees as to the iteration order of the set.

Implements Set, JSONSerializer and JSONDeserializer interfaces.

package main

import "github.com/emirpasic/gods/sets/hashset"

func main() {
	set := hashset.New()   // empty
	set.Add(1)             // 1
	set.Add(2, 2, 3, 4, 5) // 3, 1, 2, 4, 5 (random order, duplicates ignored)
	set.Remove(4)          // 5, 3, 2, 1 (random order)
	set.Remove(2, 3)       // 1, 5 (random order)
	set.Contains(1)        // true
	set.Contains(1, 5)     // true
	set.Contains(1, 6)     // false
	_ = set.Values()       // []int{5,1} (random order)
	set.Clear()            // empty
	set.Empty()            // true
	set.Size()             // 0
}

TreeSet

A set backed by a red-black tree to keep the elements ordered with respect to the comparator.

Implements Set, ReverseIteratorWithIndex, EnumerableWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import "github.com/emirpasic/gods/sets/treeset"

func main() {
	set := treeset.NewWithIntComparator() // empty (keys are of type int)
	set.Add(1)                            // 1
	set.Add(2, 2, 3, 4, 5)                // 1, 2, 3, 4, 5 (in order, duplicates ignored)
	set.Remove(4)                         // 1, 2, 3, 5 (in order)
	set.Remove(2, 3)                      // 1, 5 (in order)
	set.Contains(1)                       // true
	set.Contains(1, 5)                    // true
	set.Contains(1, 6)                    // false
	_ = set.Values()                      // []int{1,5} (in order)
	set.Clear()                           // empty
	set.Empty()                           // true
	set.Size()                            // 0
}

LinkedHashSet

A set that preserves insertion-order. Data structure is backed by a hash table to store values and doubly-linked list to store insertion ordering.

Implements Set, ReverseIteratorWithIndex, EnumerableWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import "github.com/emirpasic/gods/sets/linkedhashset"

func main() {
	set := linkedhashset.New() // empty
	set.Add(5)                 // 5
	set.Add(4, 4, 3, 2, 1)     // 5, 4, 3, 2, 1 (in insertion-order, duplicates ignored)
	set.Add(4)                 // 5, 4, 3, 2, 1 (duplicates ignored, insertion-order unchanged)
	set.Remove(4)              // 5, 3, 2, 1 (in insertion-order)
	set.Remove(2, 3)           // 5, 1 (in insertion-order)
	set.Contains(1)            // true
	set.Contains(1, 5)         // true
	set.Contains(1, 6)         // false
	_ = set.Values()           // []int{5, 1} (in insertion-order)
	set.Clear()                // empty
	set.Empty()                // true
	set.Size()                 // 0
}

Stacks

A stack that represents a last-in-first-out (LIFO) data structure. The usual push and pop operations are provided, as well as a method to peek at the top item on the stack.

Implements Container interface.

type Stack interface {
	Push(value interface{})
	Pop() (value interface{}, ok bool)
	Peek() (value interface{}, ok bool)

	containers.Container
	// Empty() bool
	// Size() int
	// Clear()
	// Values() []interface{}
	// String() string
}

LinkedListStack

A stack based on a linked list.

Implements Stack, IteratorWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import lls "github.com/emirpasic/gods/stacks/linkedliststack"

func main() {
	stack := lls.New()  // empty
	stack.Push(1)       // 1
	stack.Push(2)       // 1, 2
	stack.Values()      // 2, 1 (LIFO order)
	_, _ = stack.Peek() // 2,true
	_, _ = stack.Pop()  // 2, true
	_, _ = stack.Pop()  // 1, true
	_, _ = stack.Pop()  // nil, false (nothing to pop)
	stack.Push(1)       // 1
	stack.Clear()       // empty
	stack.Empty()       // true
	stack.Size()        // 0
}

ArrayStack

A stack based on a array list.

Implements Stack, IteratorWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import "github.com/emirpasic/gods/stacks/arraystack"

func main() {
	stack := arraystack.New() // empty
	stack.Push(1)             // 1
	stack.Push(2)             // 1, 2
	stack.Values()            // 2, 1 (LIFO order)
	_, _ = stack.Peek()       // 2,true
	_, _ = stack.Pop()        // 2, true
	_, _ = stack.Pop()        // 1, true
	_, _ = stack.Pop()        // nil, false (nothing to pop)
	stack.Push(1)             // 1
	stack.Clear()             // empty
	stack.Empty()             // true
	stack.Size()              // 0
}

Maps

A Map is a data structure that maps keys to values. A map cannot contain duplicate keys and each key can map to at most one value.

Implements Container interface.

type Map interface {
	Put(key interface{}, value interface{})
	Get(key interface{}) (value interface{}, found bool)
	Remove(key interface{})
	Keys() []interface{}

	containers.Container
	// Empty() bool
	// Size() int
	// Clear()
	// Values() []interface{}
	// String() string
}

A BidiMap is an extension to the Map. A bidirectional map (BidiMap), also called a hash bag, is an associative data structure in which the key-value pairs form a one-to-one relation. This relation works in both directions by allow the value to also act as a key to key, e.g. a pair (a,b) thus provides a coupling between 'a' and 'b' so that 'b' can be found when 'a' is used as a key and 'a' can be found when 'b' is used as a key.

type BidiMap interface {
	GetKey(value interface{}) (key interface{}, found bool)

	Map
}

HashMap

A map based on hash tables. Keys are unordered.

Implements Map, JSONSerializer and JSONDeserializer interfaces.

package main

import "github.com/emirpasic/gods/maps/hashmap"

func main() {
	m := hashmap.New() // empty
	m.Put(1, "x")      // 1->x
	m.Put(2, "b")      // 2->b, 1->x (random order)
	m.Put(1, "a")      // 2->b, 1->a (random order)
	_, _ = m.Get(2)    // b, true
	_, _ = m.Get(3)    // nil, false
	_ = m.Values()     // []interface {}{"b", "a"} (random order)
	_ = m.Keys()       // []interface {}{1, 2} (random order)
	m.Remove(1)        // 2->b
	m.Clear()          // empty
	m.Empty()          // true
	m.Size()           // 0
}

TreeMap

A map based on red-black tree. Keys are ordered with respect to the comparator.

Implements Map, ReverseIteratorWithIndex, EnumerableWithKey, JSONSerializer and JSONDeserializer interfaces.

package main

import "github.com/emirpasic/gods/maps/treemap"

func main() {
	m := treemap.NewWithIntComparator() // empty (keys are of type int)
	m.Put(1, "x")                       // 1->x
	m.Put(2, "b")                       // 1->x, 2->b (in order)
	m.Put(1, "a")                       // 1->a, 2->b (in order)
	_, _ = m.Get(2)                     // b, true
	_, _ = m.Get(3)                     // nil, false
	_ = m.Values()                      // []interface {}{"a", "b"} (in order)
	_ = m.Keys()                        // []interface {}{1, 2} (in order)
	m.Remove(1)                         // 2->b
	m.Clear()                           // empty
	m.Empty()                           // true
	m.Size()                            // 0

	// Other:
	m.Min() // Returns the minimum key and its value from map.
	m.Max() // Returns the maximum key and its value from map.
}

LinkedHashMap

A map that preserves insertion-order. It is backed by a hash table to store values and doubly-linked list to store ordering.

Implements Map, ReverseIteratorWithIndex, EnumerableWithKey, JSONSerializer and JSONDeserializer interfaces.

package main

import "github.com/emirpasic/gods/maps/linkedhashmap"

func main() {
	m := linkedhashmap.New() // empty (keys are of type int)
	m.Put(2, "b")            // 2->b
	m.Put(1, "x")            // 2->b, 1->x (insertion-order)
	m.Put(1, "a")            // 2->b, 1->a (insertion-order)
	_, _ = m.Get(2)          // b, true
	_, _ = m.Get(3)          // nil, false
	_ = m.Values()           // []interface {}{"b", "a"} (insertion-order)
	_ = m.Keys()             // []interface {}{2, 1} (insertion-order)
	m.Remove(1)              // 2->b
	m.Clear()                // empty
	m.Empty()                // true
	m.Size()                 // 0
}

HashBidiMap

A map based on two hashmaps. Keys are unordered.

Implements BidiMap, JSONSerializer and JSONDeserializer interfaces.

package main

import "github.com/emirpasic/gods/maps/hashbidimap"

func main() {
	m := hashbidimap.New() // empty
	m.Put(1, "x")          // 1->x
	m.Put(3, "b")          // 1->x, 3->b (random order)
	m.Put(1, "a")          // 1->a, 3->b (random order)
	m.Put(2, "b")          // 1->a, 2->b (random order)
	_, _ = m.GetKey("a")   // 1, true
	_, _ = m.Get(2)        // b, true
	_, _ = m.Get(3)        // nil, false
	_ = m.Values()         // []interface {}{"a", "b"} (random order)
	_ = m.Keys()           // []interface {}{1, 2} (random order)
	m.Remove(1)            // 2->b
	m.Clear()              // empty
	m.Empty()              // true
	m.Size()               // 0
}

TreeBidiMap

A map based on red-black tree. This map guarantees that the map will be in both ascending key and value order. Other than key and value ordering, the goal with this structure is to avoid duplication of elements (unlike in HashBidiMap), which can be significant if contained elements are large.

Implements BidiMap, ReverseIteratorWithIndex, EnumerableWithKey, JSONSerializer and JSONDeserializer interfaces.

package main

import (
	"github.com/emirpasic/gods/maps/treebidimap"
	"github.com/emirpasic/gods/utils"
)

func main() {
	m := treebidimap.NewWith(utils.IntComparator, utils.StringComparator)
	m.Put(1, "x")        // 1->x
	m.Put(3, "b")        // 1->x, 3->b (ordered)
	m.Put(1, "a")        // 1->a, 3->b (ordered)
	m.Put(2, "b")        // 1->a, 2->b (ordered)
	_, _ = m.GetKey("a") // 1, true
	_, _ = m.Get(2)      // b, true
	_, _ = m.Get(3)      // nil, false
	_ = m.Values()       // []interface {}{"a", "b"} (ordered)
	_ = m.Keys()         // []interface {}{1, 2} (ordered)
	m.Remove(1)          // 2->b
	m.Clear()            // empty
	m.Empty()            // true
	m.Size()             // 0
}

Trees

A tree is a widely used data data structure that simulates a hierarchical tree structure, with a root value and subtrees of children, represented as a set of linked nodes; thus no cyclic links.

Implements Container interface.

type Tree interface {
	containers.Container
	// Empty() bool
	// Size() int
	// Clear()
	// Values() []interface{}
	// String() string
}

RedBlackTree

A red–black tree is a binary search tree with an extra bit of data per node, its color, which can be either red or black. The extra bit of storage ensures an approximately balanced tree by constraining how nodes are colored from any path from the root to the leaf. Thus, it is a data structure which is a type of self-balancing binary search tree.

The balancing of the tree is not perfect but it is good enough to allow it to guarantee searching in O(log n) time, where n is the total number of elements in the tree. The insertion and deletion operations, along with the tree rearrangement and recoloring, are also performed in O(log n) time. _^Wikipedia

Implements Tree, ReverseIteratorWithKey, JSONSerializer and JSONDeserializer interfaces.

package main

import (
	"fmt"
	rbt "github.com/emirpasic/gods/trees/redblacktree"
)

func main() {
	tree := rbt.NewWithIntComparator() // empty (keys are of type int)

	tree.Put(1, "x") // 1->x
	tree.Put(2, "b") // 1->x, 2->b (in order)
	tree.Put(1, "a") // 1->a, 2->b (in order, replacement)
	tree.Put(3, "c") // 1->a, 2->b, 3->c (in order)
	tree.Put(4, "d") // 1->a, 2->b, 3->c, 4->d (in order)
	tree.Put(5, "e") // 1->a, 2->b, 3->c, 4->d, 5->e (in order)
	tree.Put(6, "f") // 1->a, 2->b, 3->c, 4->d, 5->e, 6->f (in order)

	fmt.Println(tree)
	//
	//  RedBlackTree
	//  │           ┌── 6
	//	│       ┌── 5
	//	│   ┌── 4
	//	│   │   └── 3
	//	└── 2
	//		└── 1

	_ = tree.Values() // []interface {}{"a", "b", "c", "d", "e", "f"} (in order)
	_ = tree.Keys()   // []interface {}{1, 2, 3, 4, 5, 6} (in order)

	tree.Remove(2) // 1->a, 3->c, 4->d, 5->e, 6->f (in order)
	fmt.Println(tree)
	//
	//  RedBlackTree
	//  │       ┌── 6
	//  │   ┌── 5
	//  └── 4
	//      │   ┌── 3
	//      └── 1

	tree.Clear() // empty
	tree.Empty() // true
	tree.Size()  // 0

	// Other:
	tree.Left() // gets the left-most (min) node
	tree.Right() // get the right-most (max) node
	tree.Floor(1) // get the floor node
	tree.Ceiling(1) // get the ceiling node
}

Extending the red-black tree's functionality has been demonstrated in the following example.

AVLTree

AVL tree is a self-balancing binary search tree. In an AVL tree, the heights of the two child subtrees of any node differ by at most one; if at any time they differ by more than one, rebalancing is done to restore this property. Lookup, insertion, and deletion all take O(log n) time in both the average and worst cases, where n is the number of nodes in the tree prior to the operation. Insertions and deletions may require the tree to be rebalanced by one or more tree rotations.

AVL trees are often compared with red–black trees because both support the same set of operations and take O(log n) time for the basic operations. For lookup-intensive applications, AVL trees are faster than red–black trees because they are more strictly balanced. _^Wikipedia

Implements Tree, ReverseIteratorWithKey, JSONSerializer and JSONDeserializer interfaces.

_{AVL tree with balance factors (green)}

package main

import (
	"fmt"
	avl "github.com/emirpasic/gods/trees/avltree"
)

func main() {
	tree := avl.NewWithIntComparator() // empty(keys are of type int)

	tree.Put(1, "x") // 1->x
	tree.Put(2, "b") // 1->x, 2->b (in order)
	tree.Put(1, "a") // 1->a, 2->b (in order, replacement)
	tree.Put(3, "c") // 1->a, 2->b, 3->c (in order)
	tree.Put(4, "d") // 1->a, 2->b, 3->c, 4->d (in order)
	tree.Put(5, "e") // 1->a, 2->b, 3->c, 4->d, 5->e (in order)
	tree.Put(6, "f") // 1->a, 2->b, 3->c, 4->d, 5->e, 6->f (in order)

	fmt.Println(tree)
	//
	//  AVLTree
	//  │       ┌── 6
	//  │   ┌── 5
	//  └── 4
	//      │   ┌── 3
	//      └── 2
	//          └── 1


	_ = tree.Values() // []interface {}{"a", "b", "c", "d", "e", "f"} (in order)
	_ = tree.Keys()   // []interface {}{1, 2, 3, 4, 5, 6} (in order)

	tree.Remove(2) // 1->a, 3->c, 4->d, 5->e, 6->f (in order)
	fmt.Println(tree)
	//
	//  AVLTree
	//  │       ┌── 6
	//  │   ┌── 5
	//  └── 4
	//      └── 3
	//          └── 1

	tree.Clear() // empty
	tree.Empty() // true
	tree.Size()  // 0
}

BTree

B-tree is a self-balancing tree data structure that keeps data sorted and allows searches, sequential access, insertions, and deletions in logarithmic time. The B-tree is a generalization of a binary search tree in that a node can have more than two children.

According to Knuth's definition, a B-tree of order m is a tree which satisfies the following properties:

Every node has at most m children.
Every non-leaf node (except root) has at least ⌈m/2⌉ children.
The root has at least two children if it is not a leaf node.
A non-leaf node with k children contains k−1 keys.
All leaves appear in the same level

Each internal node’s keys act as separation values which divide its subtrees. For example, if an internal node has 3 child nodes (or subtrees) then it must have 2 keys: a1 and a2. All values in the leftmost subtree will be less than a1, all values in the middle subtree will be between a1 and a2, and all values in the rightmost subtree will be greater than a2._^Wikipedia

Implements Tree, ReverseIteratorWithKey, JSONSerializer and JSONDeserializer interfaces.

package main

import (
	"fmt"
	"github.com/emirpasic/gods/trees/btree"
)

func main() {
	tree := btree.NewWithIntComparator(3) // empty (keys are of type int)

	tree.Put(1, "x") // 1->x
	tree.Put(2, "b") // 1->x, 2->b (in order)
	tree.Put(1, "a") // 1->a, 2->b (in order, replacement)
	tree.Put(3, "c") // 1->a, 2->b, 3->c (in order)
	tree.Put(4, "d") // 1->a, 2->b, 3->c, 4->d (in order)
	tree.Put(5, "e") // 1->a, 2->b, 3->c, 4->d, 5->e (in order)
	tree.Put(6, "f") // 1->a, 2->b, 3->c, 4->d, 5->e, 6->f (in order)
	tree.Put(7, "g") // 1->a, 2->b, 3->c, 4->d, 5->e, 6->f, 7->g (in order)

	fmt.Println(tree)
	// BTree
	//         1
	//     2
	//         3
	// 4
	//         5
	//     6
	//         7

	_ = tree.Values() // []interface {}{"a", "b", "c", "d", "e", "f", "g"} (in order)
	_ = tree.Keys()   // []interface {}{1, 2, 3, 4, 5, 6, 7} (in order)

	tree.Remove(2) // 1->a, 3->c, 4->d, 5->e, 6->f, 7->g (in order)
	fmt.Println(tree)
	// BTree
	//     1
	//     3
	// 4
	//     5
	// 6
	//     7

	tree.Clear() // empty
	tree.Empty() // true
	tree.Size()  // 0

	// Other:
	tree.Height() // gets the height of the tree
	tree.Left() // gets the left-most (min) node
	tree.LeftKey() // get the left-most (min) node's key
	tree.LeftValue() // get the left-most (min) node's value
	tree.Right() // get the right-most (max) node
	tree.RightKey() // get the right-most (max) node's key
	tree.RightValue() // get the right-most (max) node's value
}

BinaryHeap

A binary heap is a tree created using a binary tree. It can be seen as a binary tree with two additional constraints:

Shape property:

A binary heap is a complete binary tree; that is, all levels of the tree, except possibly the last one (deepest) are fully filled, and, if the last level of the tree is not complete, the nodes of that level are filled from left to right.
Heap property:

All nodes are either greater than or equal to or less than or equal to each of its children, according to a comparison predicate defined for the heap. _^Wikipedia

Implements Tree, ReverseIteratorWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import (
	"github.com/emirpasic/gods/trees/binaryheap"
	"github.com/emirpasic/gods/utils"
)

func main() {

	// Min-heap
	heap := binaryheap.NewWithIntComparator() // empty (min-heap)
	heap.Push(2)                              // 2
	heap.Push(3)                              // 2, 3
	heap.Push(1)                              // 1, 3, 2
	heap.Values()                             // 1, 3, 2
	_, _ = heap.Peek()                        // 1,true
	_, _ = heap.Pop()                         // 1, true
	_, _ = heap.Pop()                         // 2, true
	_, _ = heap.Pop()                         // 3, true
	_, _ = heap.Pop()                         // nil, false (nothing to pop)
	heap.Push(1)                              // 1
	heap.Clear()                              // empty
	heap.Empty()                              // true
	heap.Size()                               // 0

	// Max-heap
	inverseIntComparator := func(a, b interface{}) int {
		return -utils.IntComparator(a, b)
	}
	heap = binaryheap.NewWith(inverseIntComparator) // empty (min-heap)
	heap.Push(2, 3, 1)                              // 3, 2, 1 (bulk optimized)
	heap.Values()                                   // 3, 2, 1
}

Queues

A queue that represents a first-in-first-out (FIFO) data structure. The usual enqueue and dequeue operations are provided, as well as a method to peek at the first item in the queue.

Implements Container interface.

type Queue interface {
	Enqueue(value interface{})
	Dequeue() (value interface{}, ok bool)
	Peek() (value interface{}, ok bool)

	containers.Container
	// Empty() bool
	// Size() int
	// Clear()
	// Values() []interface{}
	// String() string
}

LinkedListQueue

A queue based on a linked list.

Implements Queue, IteratorWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import llq "github.com/emirpasic/gods/queues/linkedlistqueue"

// LinkedListQueueExample to demonstrate basic usage of LinkedListQueue
func main() {
    queue := llq.New()     // empty
    queue.Enqueue(1)       // 1
    queue.Enqueue(2)       // 1, 2
    _ = queue.Values()     // 1, 2 (FIFO order)
    _, _ = queue.Peek()    // 1,true
    _, _ = queue.Dequeue() // 1, true
    _, _ = queue.Dequeue() // 2, true
    _, _ = queue.Dequeue() // nil, false (nothing to deque)
    queue.Enqueue(1)       // 1
    queue.Clear()          // empty
    queue.Empty()          // true
    _ = queue.Size()       // 0
}

ArrayQueue

A queue based on a array list.

Implements Queue, ReverseIteratorWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import aq "github.com/emirpasic/gods/queues/arrayqueue"

// ArrayQueueExample to demonstrate basic usage of ArrayQueue
func main() {
    queue := aq.New()      // empty
    queue.Enqueue(1)       // 1
    queue.Enqueue(2)       // 1, 2
    _ = queue.Values()     // 1, 2 (FIFO order)
    _, _ = queue.Peek()    // 1,true
    _, _ = queue.Dequeue() // 1, true
    _, _ = queue.Dequeue() // 2, true
    _, _ = queue.Dequeue() // nil, false (nothing to deque)
    queue.Enqueue(1)       // 1
    queue.Clear()          // empty
    queue.Empty()          // true
    _ = queue.Size()       // 0
}

CircularBuffer

A circular buffer, circular queue, cyclic buffer or ring buffer is a data structure that uses a single, fixed-size buffer as if it were connected end-to-end. This structure lends itself easily to buffering data streams.

Implements Queue, ReverseIteratorWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import cb "github.com/emirpasic/gods/queues/circularbuffer"

// CircularBufferExample to demonstrate basic usage of CircularBuffer
func main() {
    queue := cb.New(3)     // empty (max size is 3)
    queue.Enqueue(1)       // 1
    queue.Enqueue(2)       // 1, 2
    queue.Enqueue(3)       // 1, 2, 3
    _ = queue.Values()     // 1, 2, 3
    queue.Enqueue(3)       // 4, 2, 3
    _, _ = queue.Peek()    // 4,true
    _, _ = queue.Dequeue() // 4, true
    _, _ = queue.Dequeue() // 2, true
    _, _ = queue.Dequeue() // 3, true
    _, _ = queue.Dequeue() // nil, false (nothing to deque)
    queue.Enqueue(1)       // 1
    queue.Clear()          // empty
    queue.Empty()          // true
    _ = queue.Size()       // 0
}

PriorityQueue

A priority queue is a special type of queue in which each element is associated with a priority value. And, elements are served on the basis of their priority. That is, higher priority elements are served first. However, if elements with the same priority occur, they are served according to their order in the queue.

Implements Queue, ReverseIteratorWithIndex, JSONSerializer and JSONDeserializer interfaces.

package main

import (
  pq "github.com/emirpasic/gods/queues/priorityqueue"
  "github.com/emirpasic/gods/utils"
)

// Element is an entry in the priority queue
type Element struct {
    name     string
    priority int
}

// Comparator function (sort by element's priority value in descending order)
func byPriority(a, b interface{}) int {
    priorityA := a.(Element).priority
    priorityB := b.(Element).priority
    return -utils.IntComparator(priorityA, priorityB) // "-" descending order
}

// PriorityQueueExample to demonstrate basic usage of BinaryHeap
func main() {
    a := Element{name: "a", priority: 1}
    b := Element{name: "b", priority: 2}
    c := Element{name: "c", priority: 3}
    
    queue := pq.NewWith(byPriority) // empty
    queue.Enqueue(a)                // {a 1}
    queue.Enqueue(c)                // {c 3}, {a 1}
    queue.Enqueue(b)                // {c 3}, {b 2}, {a 1}
    _ = queue.Values()              // [{c 3} {b 2} {a 1}]
    _, _ = queue.Peek()             // {c 3} true
    _, _ = queue.Dequeue()          // {c 3} true
    _, _ = queue.Dequeue()          // {b 2} true
    _, _ = queue.Dequeue()          // {a 1} true
    _, _ = queue.Dequeue()          // <nil> false (nothing to dequeue)
    queue.Clear()                   // empty
    _ = queue.Empty()               // true
    _ = queue.Size()                // 0
}

Functions

Various helper functions used throughout the library.

Comparator

Some data structures (e.g. TreeMap, TreeSet) require a comparator function to automatically keep their elements sorted upon insertion. This comparator is necessary during the initalization.

Comparator is defined as:

Return values (int):

negative , if a < b
zero     , if a == b
positive , if a > b

Comparator signature:

type Comparator func(a, b interface{}) int

All common comparators for builtin types are included in the library:

func StringComparator(a, b interface{}) int

func IntComparator(a, b interface{}) int

func Int8Comparator(a, b interface{}) int

func Int16Comparator(a, b interface{}) int

func Int32Comparator(a, b interface{}) int

func Int64Comparator(a, b interface{}) int

func UIntComparator(a, b interface{}) int

func UInt8Comparator(a, b interface{}) int

func UInt16Comparator(a, b interface{}) int

func UInt32Comparator(a, b interface{}) int

func UInt64Comparator(a, b interface{}) int

func Float32Comparator(a, b interface{}) int

func Float64Comparator(a, b interface{}) int

func ByteComparator(a, b interface{}) int

func RuneComparator(a, b interface{}) int

func TimeComparator(a, b interface{}) int

Writing custom comparators is easy:

package main

import (
	"fmt"
	"github.com/emirpasic/gods/sets/treeset"
)

type User struct {
	id   int
	name string
}

// Custom comparator (sort by IDs)
func byID(a, b interface{}) int {

	// Type assertion, program will panic if this is not respected
	c1 := a.(User)
	c2 := b.(User)

	switch {
	case c1.id > c2.id:
		return 1
	case c1.id < c2.id:
		return -1
	default:
		return 0
	}
}

func main() {
	set := treeset.NewWith(byID)

	set.Add(User{2, "Second"})
	set.Add(User{3, "Third"})
	set.Add(User{1, "First"})
	set.Add(User{4, "Fourth"})

	fmt.Println(set) // {1 First}, {2 Second}, {3 Third}, {4 Fourth}
}

Iterator

All ordered containers have stateful iterators. Typically an iterator is obtained by Iterator() function of an ordered container. Once obtained, iterator's Next() function moves the iterator to the next element and returns true if there was a next element. If there was an element, then element's can be obtained by iterator's Value() function. Depending on the ordering type, it's position can be obtained by iterator's Index() or Key() functions. Some containers even provide reversible iterators, essentially the same, but provide another extra Prev() function that moves the iterator to the previous element and returns true if there was a previous element.

Note: it is unsafe to remove elements from container while iterating.

IteratorWithIndex

An iterator whose elements are referenced by an index.

Typical usage:

it := list.Iterator()
for it.Next() {
	index, value := it.Index(), it.Value()
	...
}

Other usages:

if it.First() {
	firstIndex, firstValue := it.Index(), it.Value()
	...
}

for it.Begin(); it.Next(); {
	...
}

Seeking to a specific element:

// Seek function, i.e. find element starting with "b"
seek := func(index int, value interface{}) bool {
    return strings.HasSuffix(value.(string), "b")
}

// Seek to the condition and continue traversal from that point (forward).
// assumes it.Begin() was called.
for found := it.NextTo(seek); found; found = it.Next() {
    index, value := it.Index(), it.Value()
    ...
}

IteratorWithKey

An iterator whose elements are referenced by a key.

Typical usage:

it := tree.Iterator()
for it.Next() {
	key, value := it.Key(), it.Value()
	...
}

Other usages:

if it.First() {
	firstKey, firstValue := it.Key(), it.Value()
	...
}

for it.Begin(); it.Next(); {
	...
}

Seeking to a specific element from the current iterator position:

// Seek function, i.e. find element starting with "b"
seek := func(key interface{}, value interface{}) bool {
    return strings.HasSuffix(value.(string), "b")
}

// Seek to the condition and continue traversal from that point (forward).
// assumes it.Begin() was called.
for found := it.NextTo(seek); found; found = it.Next() {
    key, value := it.Key(), it.Value()
    ...
}

ReverseIteratorWithIndex

An iterator whose elements are referenced by an index. Provides all functions as IteratorWithIndex, but can also be used for reverse iteration.

Typical usage of iteration in reverse:

it := list.Iterator()
for it.End(); it.Prev(); {
	index, value := it.Index(), it.Value()
	...
}

Other usages:

if it.Last() {
	lastIndex, lastValue := it.Index(), it.Value()
	...
}

Seeking to a specific element:

// Seek function, i.e. find element starting with "b"
seek := func(index int, value interface{}) bool {
    return strings.HasSuffix(value.(string), "b")
}

// Seek to the condition and continue traversal from that point (in reverse).
// assumes it.End() was called.
for found := it.PrevTo(seek); found; found = it.Prev() {
    index, value := it.Index(), it.Value()
	...
}

ReverseIteratorWithKey

An iterator whose elements are referenced by a key. Provides all functions as IteratorWithKey, but can also be used for reverse iteration.

Typical usage of iteration in reverse:

it := tree.Iterator()
for it.End(); it.Prev(); {
	key, value := it.Key(), it.Value()
	...
}

Other usages:

if it.Last() {
	lastKey, lastValue := it.Key(), it.Value()
	...
}

// Seek function, i.e. find element starting with "b"
seek := func(key interface{}, value interface{}) bool {
    return strings.HasSuffix(value.(string), "b")
}

// Seek to the condition and continue traversal from that point (in reverse).
// assumes it.End() was called.
for found := it.PrevTo(seek); found; found = it.Prev() {
    key, value := it.Key(), it.Value()
	...
}

Enumerable

Enumerable functions for ordered containers that implement EnumerableWithIndex or EnumerableWithKey interfaces.

EnumerableWithIndex

Enumerable functions for ordered containers whose values can be fetched by an index.

Each

Calls the given function once for each element, passing that element's index and value.

Each(func(index int, value interface{}))

Map

Invokes the given function once for each element and returns a container containing the values returned by the given function.

Map(func(index int, value interface{}) interface{}) Container

Select

Returns a new container containing all elements for which the given function returns a true value.

Select(func(index int, value interface{}) bool) Container

Any

Passes each element of the container to the given function and returns true if the function ever returns true for any element.

Any(func(index int, value interface{}) bool) bool

All

Passes each element of the container to the given function and returns true if the function returns true for all elements.

All(func(index int, value interface{}) bool) bool

Find

Passes each element of the container to the given function and returns the first (index,value) for which the function is true or -1,nil otherwise if no element matches the criteria.

Find(func(index int, value interface{}) bool) (int, interface{})}

Example:

package main

import (
	"fmt"
	"github.com/emirpasic/gods/sets/treeset"
)

func printSet(txt string, set *treeset.Set) {
	fmt.Print(txt, "[ ")
	set.Each(func(index int, value interface{}) {
		fmt.Print(value, " ")
	})
	fmt.Println("]")
}

func main() {
	set := treeset.NewWithIntComparator()
	set.Add(2, 3, 4, 2, 5, 6, 7, 8)
	printSet("Initial", set) // [ 2 3 4 5 6 7 8 ]

	even := set.Select(func(index int, value interface{}) bool {
		return value.(int)%2 == 0
	})
	printSet("Even numbers", even) // [ 2 4 6 8 ]

	foundIndex, foundValue := set.Find(func(index int, value interface{}) bool {
		return value.(int)%2 == 0 && value.(int)%3 == 0
	})
	if foundIndex != -1 {
		fmt.Println("Number divisible by 2 and 3 found is", foundValue, "at index", foundIndex) // value: 6, index: 4
	}

	square := set.Map(func(index int, value interface{}) interface{} {
		return value.(int) * value.(int)
	})
	printSet("Numbers squared", square) // [ 4 9 16 25 36 49 64 ]

	bigger := set.Any(func(index int, value interface{}) bool {
		return value.(int) > 5
	})
	fmt.Println("Set contains a number bigger than 5 is ", bigger) // true

	positive := set.All(func(index int, value interface{}) bool {
		return value.(int) > 0
	})
	fmt.Println("All numbers are positive is", positive) // true

	evenNumbersSquared := set.Select(func(index int, value interface{}) bool {
		return value.(int)%2 == 0
	}).Map(func(index int, value interface{}) interface{} {
		return value.(int) * value.(int)
	})
	printSet("Chaining", evenNumbersSquared) // [ 4 16 36 64 ]
}

EnumerableWithKey

Enumerable functions for ordered containers whose values whose elements are key/value pairs.

Each

Calls the given function once for each element, passing that element's key and value.

Each(func(key interface{}, value interface{}))

Map

Invokes the given function once for each element and returns a container containing the values returned by the given function as key/value pairs.

Map(func(key interface{}, value interface{}) (interface{}, interface{})) Container

Select

Returns a new container containing all elements for which the given function returns a true value.

Select(func(key interface{}, value interface{}) bool) Container

Any

Passes each element of the container to the given function and returns true if the function ever returns true for any element.

Any(func(key interface{}, value interface{}) bool) bool

All

Passes each element of the container to the given function and returns true if the function returns true for all elements.

All(func(key interface{}, value interface{}) bool) bool

Find

Passes each element of the container to the given function and returns the first (key,value) for which the function is true or nil,nil otherwise if no element matches the criteria.

Find(func(key interface{}, value interface{}) bool) (interface{}, interface{})

Example:

package main

import (
	"fmt"
	"github.com/emirpasic/gods/maps/treemap"
)

func printMap(txt string, m *treemap.Map) {
	fmt.Print(txt, " { ")
	m.Each(func(key interface{}, value interface{}) {
		fmt.Print(key, ":", value, " ")
	})
	fmt.Println("}")
}

func main() {
	m := treemap.NewWithStringComparator()
	m.Put("g", 7)
	m.Put("f", 6)
	m.Put("e", 5)
	m.Put("d", 4)
	m.Put("c", 3)
	m.Put("b", 2)
	m.Put("a", 1)
	printMap("Initial", m) // { a:1 b:2 c:3 d:4 e:5 f:6 g:7 }

	even := m.Select(func(key interface{}, value interface{}) bool {
		return value.(int) % 2 == 0
	})
	printMap("Elements with even values", even) // { b:2 d:4 f:6 }

	foundKey, foundValue := m.Find(func(key interface{}, value interface{}) bool {
		return value.(int) % 2 == 0 && value.(int) % 3 == 0
	})
	if foundKey != nil {
		fmt.Println("Element with value divisible by 2 and 3 found is", foundValue, "with key", foundKey) // value: 6, index: 4
	}

	square := m.Map(func(key interface{}, value interface{}) (interface{}, interface{}) {
		return key.(string) + key.(string), value.(int) * value.(int)
	})
	printMap("Elements' values squared and letters duplicated", square) // { aa:1 bb:4 cc:9 dd:16 ee:25 ff:36 gg:49 }

	bigger := m.Any(func(key interface{}, value interface{}) bool {
		return value.(int) > 5
	})
	fmt.Println("Map contains element whose value is bigger than 5 is", bigger) // true

	positive := m.All(func(key interface{}, value interface{}) bool {
		return value.(int) > 0
	})
	fmt.Println("All map's elements have positive values is", positive) // true

	evenNumbersSquared := m.Select(func(key interface{}, value interface{}) bool {
		return value.(int) % 2 == 0
	}).Map(func(key interface{}, value interface{}) (interface{}, interface{}) {
		return key, value.(int) * value.(int)
	})
	printMap("Chaining", evenNumbersSquared) // { b:4 d:16 f:36 }
}

Serialization

All data structures can be serialized (marshalled) and deserialized (unmarshalled). Currently, only JSON support is available.

JSONSerializer

Outputs the container into its JSON representation.

Typical usage for key-value structures:

package main

import (
	"encoding/json"
	"fmt"
	"github.com/emirpasic/gods/maps/hashmap"
)

func main() {
	m := hashmap.New()
	m.Put("a", "1")
	m.Put("b", "2")
	m.Put("c", "3")

	bytes, err := json.Marshal(m) // Same as "m.ToJSON(m)"
	if err != nil {
		fmt.Println(err)
	}
	fmt.Println(string(bytes)) // {"a":"1","b":"2","c":"3"}
}

Typical usage for value-only structures:

package main

import (
	"encoding/json"
	"fmt"
	"github.com/emirpasic/gods/lists/arraylist"
)

func main() {
	list := arraylist.New()
	list.Add("a", "b", "c")

	bytes, err := json.Marshal(list) // Same as "list.ToJSON(list)"
	if err != nil {
		fmt.Println(err)
	}
	fmt.Println(string(bytes)) // ["a","b","c"]
}

JSONDeserializer

Populates the container with elements from the input JSON representation.

Typical usage for key-value structures:

package main

import (
	"encoding/json"
	"fmt"
	"github.com/emirpasic/gods/maps/hashmap"
)

func main() {
	hm := hashmap.New()

	bytes := []byte(`{"a":"1","b":"2"}`)
	err := json.Unmarshal(bytes, &hm) // Same as "hm.FromJSON(bytes)"
	if err != nil {
		fmt.Println(err)
	}
	fmt.Println(hm) // HashMap map[b:2 a:1]
}

Typical usage for value-only structures:

package main

import (
	"encoding/json"
	"fmt"
	"github.com/emirpasic/gods/lists/arraylist"
)

func main() {
	list := arraylist.New()

	bytes := []byte(`["a","b"]`)
	err := json.Unmarshal(bytes, &list) // Same as "list.FromJSON(bytes)"
	if err != nil {
		fmt.Println(err)
	}
	fmt.Println(list) // ArrayList ["a","b"]
}

Sort

Sort is a general purpose sort function.

Lists have an in-place Sort() function and all containers can return their sorted elements via containers.GetSortedValues() function.

Internally these all use the utils.Sort() method:

package main

import "github.com/emirpasic/gods/utils"

func main() {
	strings := []interface{}{}                  // []
	strings = append(strings, "d")              // ["d"]
	strings = append(strings, "a")              // ["d","a"]
	strings = append(strings, "b")              // ["d","a",b"
	strings = append(strings, "c")              // ["d","a",b","c"]
	utils.Sort(strings, utils.StringComparator) // ["a","b","c","d"]
}

Container

Container specific operations:

// Returns sorted container''s elements with respect to the passed comparator.
// Does not affect the ordering of elements within the container.
func GetSortedValues(container Container, comparator utils.Comparator) []interface{}

Usage:

package main

import (
	"github.com/emirpasic/gods/lists/arraylist"
	"github.com/emirpasic/gods/utils"
)

func main() {
	list := arraylist.New()
	list.Add(2, 1, 3)
	values := GetSortedValues(container, utils.StringComparator) // [1, 2, 3]
}

Appendix

Motivation

Collections and data structures found in other languages: Java Collections, C++ Standard Template Library (STL) containers, Qt Containers, Ruby Enumerable etc.

Goals

Fast algorithms:

Based on decades of knowledge and experiences of other libraries mentioned above.

Memory efficient algorithms:

Avoiding to consume memory by using optimal algorithms and data structures for the given set of problems, e.g. red-black tree in case of TreeMap to avoid keeping redundant sorted array of keys in memory.

Easy to use library:

Well-structured library with minimalistic set of atomic operations from which more complex operations can be crafted.

Stable library:

Only additions are permitted keeping the library backward compatible.

Solid documentation and examples:

Learning by example.

Production ready:

Used in production.

No dependencies:

No external imports.

There is often a tug of war between speed and memory when crafting algorithms. We choose to optimize for speed in most cases within reasonable limits on memory consumption.

Thread safety is not a concern of this project, this should be handled at a higher level.

Testing and Benchmarking

This takes a while, so test within sub-packages:

go test -run=NO_TEST -bench . -benchmem -benchtime 1s ./...

Contributing

Biggest contribution towards this library is to use it and give us feedback for further improvements and additions.

For direct contributions, pull request into master branch or ask to become a contributor.

Coding style:

# Install tooling and set path:
go install gotest.tools/gotestsum@latest
go install golang.org/x/lint/golint@latest
go install github.com/kisielk/errcheck@latest
export PATH=$PATH:$GOPATH/bin

# Fix errors and warnings:
go fmt ./... &&
go test -v ./... && 
golint -set_exit_status ./... && 
! go fmt ./... 2>&1 | read &&
go vet -v ./... &&
gocyclo -avg -over 15 ../gods &&
errcheck ./...

License

This library is distributed under the BSD-style license found in the LICENSE file.

argoproj/argo-cd

2023-09-10T01:54:17Z

Declarative continuous deployment for Kubernetes.

Releases:

Code:

Social:

Argo CD - Declarative Continuous Delivery for Kubernetes

What is Argo CD?

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.

Why Argo CD?

Application definitions, configurations, and environments should be declarative and version controlled.
Application deployment and lifecycle management should be automated, auditable, and easy to understand.

Who uses Argo CD?

Official Argo CD user list

Documentation

To learn more about Argo CD go to the complete documentation. Check live demo at https://cd.apps.argoproj.io/.

Community

Contribution, Discussion and Support

You can reach the Argo CD community and developers via the following channels:

Q & A : Github Discussions
Chat : The #argo-cd Slack channel
Contributors Office Hours: Every Thursday | Agenda
User Community meeting: First Wednesday of the month | Agenda

Participation in the Argo CD project is governed by the CNCF Code of Conduct

Blogs and Presentations

Itiviti/gradle-msbuild-plugin

2023-09-10T01:54:49Z

Gradle plugin for msbuild execution, supports C# project files for now

vert-x/mod-lang-groovy

2023-09-10T01:54:49Z

Vert.x 2.x is deprecated - use instead

seqeralabs/libseqera

2023-09-10T01:54:49Z

Library for Java & Groovy modules shared across Seqera projects

Verification-for-Security/haskell-intro

2023-09-11T01:31:57Z

An introductory assignment for Haskell.

Haskell introduction

In this assignment, you will pick up the basics of Haskell. This will help you to understand and write the harder assignments later in the course.

Getting started

GHCup

Install GHCup by following the installation instructions for your operating system. Do make sure to allow GHCup to hook into stack, as we will be using stack to build the assignments. To check if the installation went succesful, the following command should give a version number.

$ ghcup --version

Make sure this is the latest version of GHCup. If this is not the case, you can upgrade to the latest version via

$ ghcup upgrade

GHCup manages all your Haskell related software. For this course, we will be working with the package manager Stack. To install this, you can simply run:

$ ghcup install stack

We also recommend that you install the Haskell Language Server (HLS). If your code editor supports language servers (which most of them do), this will display errors or useful tips to improve your code! It also allows you to run some additional commands when editing, for example, jumping to a definition of a piece of code. You can install it via the following command.

$ ghcup install hls

For an overview of all the software installed on your system by GHCup, run the following command. With this, you can make sure that both Stack and HLS are on their latest versions.

$ ghcup tui

It should be the case that stack will automatically install the GHC version specified by the project when trying to compile it. If this doesn't work you could also manually install specific GHC versions, either through this menu, or through the command line.

Git

Install Git by following the installation instructions for your operating system.

You may clone this assignment through git. Make sure to run this command in the directory where you would like this project to exist. You can find the <directory-path> in the top right of this repository, under Code.

$ git clone <directory-path>

Git is also the recommended way to cooperate with your teammates during the later assignments. Do make sure to keep your repositories private in this case! Note that this is an individual assignment.

Running and testing

You can build and test the code through stack. Make sure to be within the project folder before you run stack. One of the most useful commands throughout the course will be:

$ stack test

This will run the entire test infrastructure on your code. If all went well up to this point, this will give you a list of functions that failed their tests.

There are many other ways to build and/or run your code. One recommended way would be to use GHCi. This is a Read-Eval-Print-Loop (REPL) that allows you to test smaller parts of your program. You could for example run a single function in your implementation via this command.

$ stack ghci

Here is a user-guide for GHCi. It is not all that difficult to use, but it is useful to know some of the commands like :r or :q. Note that running GHCi through stack already imports the current codebase into scope. This is practically always desireable, so we recommend running GHCi in this fashion instead of as a standalone program.

Assignment structure

Implement all function stubs found within the src folder of this project. A stub will have the following shape:

-- | Returns true if the passed list was empty
-- e.g. null [1, 2] = False
null :: [a] -> Bool
null = undefined

You will have to provide an implementation for this, e.g.

-- | Returns true if the passed list was empty
-- e.g. null [1, 2] = False
null :: [a] -> Bool
null [] = True
null _  = False

Notice how the comment above the stub provides you with an explanation of what the code is supposed to do, as well as an example case. Do make sure to read all the comments in a file. They provide useful insights, as well as occasionally ask you to implement functions in a specific way. Do not change the function signature of the stubs. This will generally make your program unable to compile!

The automatic tests that are ran by stack test will check whether your implementation is correct. These tests will also tell you in which order we think the assignment works best. Some tests may even depend on earlier pieces of your implementation in order to function. Hence, we strongly suggest to work in this order!

Grading

Your final grade corresponds directly to the one awarded to you by the test infrastructure. Do make sure your submission correctly executes on our online environment.

If there are issues with the submission system, don't panic! We will handle this on a case-by-case basis.

If your uploaded submission somehow fail tests that work locally, ping us and we will have a look!

If the online environment suddenly fails to work moments before the deadline, don't hesitate to send us your submission through different means (e.g. email).

Plagiarism

We have a strict zero tolerance policy against plagiarism. Sadly, we find cases every year... This is not fun for you, nor us. Please, refrain from copying and/or sharing your code with other students.

While we generally recommend our students to work via Git, do make sure to make your repository private! Sharing your code in this manner is sadly still plagiarism, even if unintentional.

mame/type-level-quine

2023-09-11T01:31:57Z

A type-level Quine in Haskell

Type-level-Quine

A type-level Quine in Haskell.

How to run the proof-of-concept version

$ ghc -fcontext-stack=2048 type-level-quine-poc.hs
$ ./type-level-quine-poc

NOTE:

-fcontext-stack=2048 is required. (Otherwise, ghc will fail with stack overflow)
The compilation may take a few minutes.
The result is partially snipped; this is not a complete quine.

How to run the full version

$ ghc -fcontext-stack=2048 type-level-quine.hs
$ ./type-level-quine

NOTE:

This may make your OS stuck because of short of memory.
Actually I has failed to confirm this.

References

Both are written in Japanese.

snoyberg/markdown

2023-09-10T01:55:06Z

Convert Markdown to HTML, with XSS protection

Markdown for haskell

NOTE This package is deprecated and no longer supported. Instead, please consider using cmark or cmark-gfm.

This is a haskell library for converting markdown to HTML. It's written in pure haskell. Also read the more technical description on stackage.

Use with yesod

For use with yesod, check out yesod-text-markdown.

resonatehq/durable-promise

2023-09-10T01:55:06Z

Durable Promise Specification

Durable Promises

Keywords

Promise, Future, Awaitable, Future Value, Write Once Register, Write Only Register, Single Assignment Container, Concurrency, Coordination

tl;dr This repository is a specification for a Durable Promise API

20 years after the launch of Amazon Web Services, building reliable and scalable cloud-based applications is a challenge even for the most seasoned developers. Developing distributed applications is a dissonant mix of idioms, patterns, and technologies, feeling fragmented and inconsistent. Yet developing traditional applications feels coherent and consistent. Developers who craft these applications have access to composable abstractions, functions & promises a.k.a async•await, and enjoy a delightful developer experience.

Even the largest traditional systems are built consistently from the smallest building blocks: functions & promises. Functions & promises are the foundational model to express concurrency and coordination. While their interpretations vary across languages and runtimes, the ideas are the same.

Promises

A promise, also called future, awaitable, or deferred is a representation of a future value. A promise is either pending or completed, that is, resolved or rejected: A promise is pending, signaling that the value is not yet available or completed, signaling success or failure.

A promise is a coordination primitive: In a typical scenario, a downstream function execution creates a promise and awaits its completion. An upstream function execution either resolves or rejects the promise. On completion, the downstream execution resumes with the value of the promise.

Adding Durability

Recently, durable functions a.k.a durable executions have emerged as an abstraction for building distributed systems. Durable functions are functions with strong execution guarantees. Traditional, that is, volatile functions are bound to a single runtime. If the runtime crashes, any volatile function execution ceases to exist. Durable function executions are not bound to a runtime. If the runtime crashes, a durable function execution is simply rescheduled on a different runtime.

Durable promises are the counterpart to durable functions. Durable promises have identity and state that is not bound to a single runtime. Based on durable promises you can compose reliable and scalable distributed systems across heterogenous technology stacks.

Application Programming Interface (API)

Logically, the Application Programming Interface (API) is divided in two parts, the Downstream API and the Upstream API.

Downstream API

Create

A downstream component may create a promise

Create(promise-id, idempotence-key, param, timeout)

Cancel

A downstream component can cancel an existing promise
```
Cancel(promise-id, idempotence-key, value)
```

Upstream API

Resolve

An upstream component can resolve an existing promise, signaling success
```
Resolve(promise-id, idempotence-key, value)
```
Reject

An upstream component can reject a promise, signalling failure
```
Reject(promise-id, idempotence-key, value)
```

Idempotence

When creating, resolving, or rejecting a Durable Promise, use an idempotency key. An idempotency key is a client generated value which the server uses to recognize a duplicated request.

TBD

dataartisan/trees-algebra-haskell

2023-09-10T01:55:06Z

This example is based on chapter six of Richard Bird's Introduction to Functional Programming using Haskell (2nd ed.) and section two of Erik Meijer's banana paper.

Please see the nat-algebra-haskell example for more information.

ViktorUJ/cks

2023-09-10T01:33:47Z

Opensource Platform for learning kubernetes and aws eks and preparation for for Certified Kubernetes Specialist (CKA ,CKS , CKAD) exams

ministryofjustice/data-platform

2023-09-11T01:31:56Z

Core Repo for Data Platform • This repository is defined and managed in Terraform

iam-veeramalla/terraform-zero-to-hero

2023-09-10T01:55:04Z

Master Terraform in 7 days using this Zero to Hero course.

ministryofjustice/cloud-platform-environments

2023-09-10T01:55:04Z

Environment configuration for the Cloud Platform

LandmakTechnology/terraform-master-class-series

2023-09-10T01:55:04Z

A repository for terraform-master-class-series

kagisearch/smallweb

2023-09-10T01:34:08Z

Kagi Small Web

Kagi's mission is to humanize the web and this project is built to help surface recent (7 day old or newer) results from the small web - people and stories that typically zip by in legacy search engines. Read more about it in the announcement blog post.

Few things to note:

Kagi search engine now surfaces posts from the small web for relevant queries in its search results.
Try the Kagi Small Web website to read and appreciate the posts directly.
You can also use the RSS feed or access these results as a part of a broader Kagi News Enrichment API.

Criteria for posts to show on the KSW website

If the blog is included in small web feed list (which means it has content in English, it is informational/educational by nature and it is not trying to sell anything) we check for these two things to show it on the site:

Blog has recent posts (<7 days old)
The website can appear in an iframe

Small web is beautiful

What is Small Web exactly? Recommend reading:

Info

smallweb.txt - Contains the feeds of indexed blogs

smallyt.txt - Contains the feeds of indexed YouTube channels

yt_rejected.txt - Contains the list of YouTube channels that were reviewed (in an automated way) and rejected

app/ - App powering the Kagi Small Web website

Add a site or channel to the list

Add a new personal blog RSS feed to the list. Rules:

Do not submit your own website unless you submit 2 other sites that are not yours (and are not in list).
Locate and submit the RSS feed of the website. Place in the file so that it remains sorted.
Content must be in English (currently, other languages are not accepted).
Only personal blogs may be submitted.
The blog must have a recent post, no older than 6 months, to meet the recency criteria for inclusion.
The site must not contain any forms of monetization, such as advertisements, newsletter signup popups, etc.

Add website RSS feed

Add a new YouTube channel RSS feed to the list. Rules:

Do not submit your own YouTube channel unless you submit 2 other channels that are not yours (and are not in list).
Locate and submit the RSS feed of the YouTube channel. Place in the file so that it remains sorted.
Content must be in English (currently, other languages are not accepted).
Preference is given to channels focusing on hobbies or passions.
The channel must not post more than twice a week.
The channel must have fewer than 400,000 subscribers.

Hint: To extract the RSS link from a YouTube channel, you can use this tool.

Add YouTube channel RSS feed

Remove a site or a channel

Remove a website if :

Content is not in English.
Website has poor quality content.
Website contains intrusive monetization such as ads or newsletter popups.
Website purpose is monetization rather than education.

Clicking "Remove website" will edit small web list in new tab, where you can locate and remove the website feed in question. Make sure to add in comments the rason for removal.

Remove website

Remove a YouTube channel if :

Content is not in English.
Channel has poor quality content.
Channel's purpose is monetization rather than education.

Clicking "Remove" will edit small youtube list in new tab, where you can locate and remove the YouTube channel feed in question. Make sure to add in comments the reason for removal.

Remove channel

Sources

Small web

The original list of small web blogs has been assembled from various sources including:

YouTube channels

The seed list for YouTube channels has been assembled from these HN discussions.

tjy-gitnub/win12

2023-09-10T01:55:24Z

Windows 12 网页版，在线体验点击下面的链接在线体验

Windows 12 网页版

🎉🎉Windows 12 网页版一周岁生日快乐！o(^o^)o 向在座的各位的支持表示诚挚的感谢！

千古有冬夏，一载经春秋。破浪长风扶摇起，夜深星源起处。
天涯何处觅知音，春秋冬夏君相昔。
一路有你，纵冰塞黄河，雪满太行，不畏，丈夫未可轻年少。
愿我如星君如月，夜夜流光相皎洁。

感念一年来大家的支持与鼓励，批评和建议。
Windows 12 的明天在大家的努力下会更灿烂美好！
————星源 23/8/28

作者今年初三了，不会做什么功能的更新了。维护和继续开发交给我们的 @User782Tec，详见公告。

寒窗苦读九春秋，一朝定乾坤。
山岭越尽云雾散，霞光终万道。
待蟾折桂题金榜，再叙一载过往！
后会有期，明年再见，各位朋友，洒家告辞！
————星源 23/8/30

🎉🎉项目在 GitHub 上突破✨1.4k✨stars✨啦！芜湖🎊🎈o(^o^)o 向在座的各位的支持表示诚挚的感谢！

Windows 12 网页版

前前言

最近（不久之前）看到有一些冒充我们的，在此做个声明：

我们项目是由 tjy-gitnub（初三），NB-group（预备），782（初一），三位开发者共同开发的！

我们只在 GitHub、Bilibili 和爱发电上有官方账号！

GitHub：

去贡献者里看一下吧~

Bilibili：

tjy-gitnub：

NB-group：

782：

爱发电：

全世界唯一可以捐款的官方账号！擦亮你的双眼！不要给其他人捐款！

链接：https://afdian.net/a/qstudio?tab=feed

前言

最近（很久以前）看到 Windows 12 概念版（Powered by PowerPoint）后深受启发，决定做一个 Windows 12 网页版，就像 Windows 11 网页版一样。

参考前景规划。

精美的 ui 设计，流畅丰富的动画，各种高级的功能（相较于网页版）。

移动端适配不太完善，开桌面版网站凑合着用吧 >v-)o

特别感谢

特别感谢以下赞助者:

CursoR_光标（https://afdian.net/a/cursor）

在此对以上赞助者表示公开鸣谢！

效果展示

新的版本有很多变化，仅供参考，请以实物为准（你点一下这里就知道了啊，不麻烦-_-)o ）

开始菜单

丰富的应用

深色模式

AI Copilot

在线体验

在线预览

有点慢，稍等一下~

安装和使用

下载代码，无需安装，打开 desktop.html 即可。

前景规划

关于该项目的路径规划:

基本功能与应用
外观整体优化
加入特效
窗口功能
应用完善
添加更多个性化方面的设置
添加 Edge 应用
为更多应用添加标签页
完善小组件，添加到桌面等功能
动态壁纸
更多任务栏的自定义
丰富应用生态，添加 Microsoft Store
完善设置及 Windows 更新

下面是一些...呃......畅..想 ~o~)/：

建立文件系统
建立自己的可执行文件机制
将 .exe 文件转化并执行
提供更多 api 供应用调用
内置浏览器内核，成为应用
将项目更名为 "Windows 12"
封装到 Windows 系统中
将启动程序设为此应用
去除多余系统功能，封装成独立的操作系统
将项目更名为 "Doswin 1.0"
适配量子计算机
接入chatgpt
将项目更名为 "550W"

开源说明

本项目作者：谭景元（tjy-gitnub）

本项目链接：https://github.com/tjy-gitnub/win12

此项目是一个开源项目。此项目使用 EPL v2.0 开源许可。开源许可是具有法律效力的合同，请自觉遵守开源许可，尊重他人劳动。

根据许可，你可以对该项目进行传播、分发、修改以及二次发布，包括个人和商业用途，但我方不鼓励一切商业用途。

您必须给出源码来源，包括作者，项目链接（见上）等，必须使用相同的协议开源。

若此项目的源码作为项目的一部分与你私有的源码一起发布时，你可以使用其它协议，但要声明 EPL 部分的内容并声明此部分继续遵循 EPL 协议。

不是在该项目基础上进行增加、修改的，仅参考源码的，不需要开源，但也仅供学习用途。

由于近期发现大量滥用行为，在此规定附加条例：

任何使用本项目者，必须在介绍里附上原作者及项目链接，不得故意隐瞒原项目中的署名信息、原作者或项目链接，不得限制查看原项目中的署名信息、原作者或项目链接，且不得修改原项目中的署名信息；
将本项目用于商业用途者，需要标明原作者及项目链接，且必须以 EPL 协议开源。未经修改的源代码不得用于商业用途；
任何使用本项目者，不得移除或故意隐藏、限制查看本声明；
请您遵守上述规定，我方有权通过法律手段进行合法维权。

贡献须知

详情请见贡献指南。

zanfranceschi/rinha-de-backend-2023-q3

2023-09-10T01:55:24Z

Rinha de Backend - Edição 2023 Q3

Rinha de Backend 2023

O que é?

Tudo começou com esse tweet. E termininou com uma live no youtube.

De 28 de Julho a 25 de Agosto foi realizada a Rinha de Backend que é um torneio em que a API que suportasse mais carga durante um teste de stress seria a vencedora. Participantes tiveram que implementar uma API com endpoints para criar, consultar e buscar 'pessoas' (uma espécie de CRUD sem UPDATE e DELETE). No torneio, participantes ainda tiveram que lidar com restrições de CPU e memória – cada participantes teve que entregar a API no formato de docker-compose podendo usar apenas 1,5 unidades de CPU e 3GB de memória. Mais detalhes sobre aspectos técnicos podem ser encontrados nas instruções.

Resultados

A Rinha usou um critério simples e irrealista apenas para finalidade de haver um ranking qualquer: a API que tiver mais registros no banco. Você pode conferir todos os resultados e relatórios acessando o diretório resultados e cada submissão em participantes. Todos os testes foram executados numa instância EC2 na AWS e as especificações podem ser encontradas aqui.

Resultado da Etapa Final

posição	participante	contagem de pessoas	p99 geral	submissão
1	viniciusfonseca	44936	17418	README
2	h4nkb31f0ng	44270	15690	README
3	grupo-2a	44200	17668	README
4	sofia_aripiprazole	43731	58571	README
5	lpicanco	42832	29764	README
6	isadora-souza	42612	57327	README
7	vimsos	42041	3159	README
8	jrodrigues	41193	44445	README
9	navarro	39356	23958	README
10	navarro-touche	36106	11772	README

Resultado da Primeira Etapa

posição	participante	contagem de pessoas	p99 geral	submissão
1	viniciusfonseca	44628	16994	README
2	h4nkb31f0ng	44270	16405	README
3	grupo-2a	44100	17588	README
4	isadora-souza	42122	55570	README
5	sofia_aripiprazole	40938	62753	README
6	vimsos	39996	2799	README
7	jrodrigues	39070	44045	README
8	lpicanco	37693	28739	README
9	navarro	37567	18183	README
10	navarro-touche	35865	12773	README
11	oliveigah	34809	47416	README
12	lucaswilliameufrasio	34680	1127	README
13	luucaspole	34165	48510	README
14	met4tron	27904	59707	README
15	rode	26607	2674	README
16	saiintbrisson	26567	3454	README
17	lauroappelt	25493	21533	README
18	brunoborges	25352	44867	README
19	lazaronixon	24466	3097	README
20	leandronsp	24418	54741	README
21	iancambrea	23831	19279	README
22	thelinuxlich	22762	25409	README
23	luanpontes100	21315	54779	README
24	lauroappeltv2	16554	13854	README
25	true_eduardo	16415	7893	README
26	rodrigoknol	16265	59210	README
27	korodzi	16217	5178	README
28	lucasteles	16074	20162	README
29	MarcosCostaDev	13368	17304	README
30	boaglio	12957	59724	README
31	viniciusferraz-nativo	12247	5879	README
32	andrelsmelo	11933	25576	README
33	viniciusferraz	11521	7916	README
34	OpenCodeCo	10960	20236	README
35	EuFountai	10549	58238	README
36	andrew-vasco	8869	59655	README
37	willy-r	8458	36359	README
38	gustavocs789	8002	18608	README
39	juniorleaoo	7761	58901	README
40	dscamargo	7503	16511	README
41	bpaulino0	7351	37202	README
42	wesleynepo	7320	21249	README
43	ftsuda	6951	60299	README
44	fernandozanutto	6233	39774	README
45	reonardoleis	5844	8810	README
46	giovannibassi	5658	58745	README
47	Bandolin	5205	16210	README
48	cleciusjm	3720	50390	README
49	Pr3d4dor-php-puro	3002	59348	README
50	wendryo	2835	44839	README
51	leandronsp-bash	17	47482	README
-	felipemarkson (43667 - desq.)	0	58617	README
-	alberto_souza	0	0	README
-	allan-cordeiro	0	0	README
-	andre237	0	0	README
-	Bandolin_simplified_api	0	40267	README
-	brahma	0	5759	README
-	CaravanaCloud	0	19377	README
-	carlosdaniiel07	0	0	README
-	danielfireman	0	0	README
-	davidlins	0	0	README
-	dupla-de-2	0	51942	README
-	fabricio_juliatto	0	0	README
-	flavio1110	0	0	README
-	guimeira	0	20227	README
-	gustmrg	0	16880	README
-	h4ad	0	54593	README
-	hampshire	0	0	README
-	igorsantos07	0	0	README
-	insalubre	0	0	README
-	isaacnborges	0	40090	README
-	kalogs-c	0	59594	README
-	krymancer	0	10231	README
-	lucasmadeira	0	5720	README
-	lucasnribeiro	0	11740	README
-	lucasraziel	0	0	README
-	LuisKpBeta	0	41978	README
-	marcospaulo	0	50418	README
-	matheuslc	0	0	README
-	MrPowerGamerBR	0	0	README
-	natanaelsimoes	0	0	README
-	Pr3d4dor-laravel	0	24781	README
-	ramoncunha	0	0	README
-	rodrigograudo	0	0	README
-	rwillians	0	245	README
-	sinhorinho	0	23474	README
-	Tagliatti	0	16077	README
-	uasouz	0	0	README
-	vhogemann	0	20054	README
-	willian	0	0	README

Premiação

Ao final do torneio, o pessoal da Gatling decidiu apoiar a Rinha e ofereceu prêmios para os 10 primeiros colocados. Mais detalhes sobre os prêmios, podem ser encontrados na página da rinha no site da Gatling.

NebulaServices/Nebula

2023-09-10T01:55:24Z

a stunning and sleek web proxy with support for hundreds of popular sites.

Nebula

NebulaWeb is an official flagship of Nebula Services and Nebula Developer Labs. NebulaWeb is a stunning, sleek, and functional web-proxy with support for thousands of popular sites. With NebulaWeb, the sky is the limit.

Quick note: If you are a system administrator, and have questions, our email is always open chloe@nebulaproxy.io

Features

Stunning and highly functional UI with multiple themes
XOR/b64 encoding all traffic
Hides your IP from sites
List of officially supported sites
limited mobile support
Stealth Mode (buffed about:blank cloaking)
NEW Clickoff cloaking
NEW Email OTP verification

Deployment

Table of contents

Quick & easy deployment
Deployment configuration explaination
how to use email OTP Verification mode
Advanced Deployment
Filesystem

Quick & Easy Deployment Options

PLEASE READ THIS NOTICE: IF YOU ARE USING NODE VERSION 20 (HEROKU, RAILWAY), YOU WILL NEED TO TEMPORARILY DOWNGRADE TO 19 OR EARLIER.

Deployment Configuration Guide

(Example configuration with none-json notes)

{
  "sendgrid_verification": false,
  "sendgrid_options": {
    "api_key": "YOUR_SENDGRID_API_KEY",
    "sendFromEmail": "THE EMAIL THE CODES WILL BE SENT FROM (MUST BE VERIFIED IN SENDGRID)",
    "to_email": "THE EMAIL YOU WANT THE CODES SENT TO"
  },

  "discord_verification": false,
  "webhook_url": "YOUR DISCORD WEBHOOK URL",

  "smtp_verification": false,
  "smtp_options": {
    "to_email": "THE EMAIL YOU WANT THE CODES SENT TO",
    "sendFromEmail": "THE EMAIL THE CODES ARE SENT FROM",
    "host": "YOUR SMTP HOST",
    "port": 465,
    "auth": {
      "user": "SMTP USER",
      "pass": "YOUR PASSWORD"
    }
  }
}

Email Verification OTP

What is this?

Email verification is a new and unique feature that we've implemented in the event that someone wants to keep their deployment of Nebula private and secure.

What does it do

When a user tries to access the website, before allowed access they will be asked for a One time password sent to an email set in the deployment configuration. Once verified, they will have 15 day access to the site.

SendGrid Setup Instructions

Firstly, We need to enable verification within the deployment configuration
- change "sendgrid_verification":false, to "sendgrid_verification":true, above the SendGrid Section
- Note: You have to reboot the node app for any changes to take place.
Now, we need to use an api to send a message
- Make an account at Sendgrid (https://app.sendgrid.com/)
- Note: It is likely that other versions of Nebula will use a different package to send emails.
Verify the email you want to recieve emails from (Create a sender identity)
- Go to settings -> Sender authentication and click Verify a Single Sender
Now, We need to get the API key to connect to the API
- Go to settings -> API Keys -> and make an API key.
Complete the information in the deployment config deployment.config.json under the sendgrid_options section such as: sendFromEmail, to_email and api_key

Discord Webhook Setup Instructions

Set discord_verification to true in the deployment configuration.
Create a channel in a discord server you have admin in.
Click the Edit Channel button.
Click Integrations
Click create web hook and copy the URL.
Paste it under the webhook_url section in the deployment configuration.

SMTP Setup Instructions

Set smtp_verification to true.
Change to_email to the email address you want the codes to be sent to.
Change `sendFromEmail to the email address that is going to send the codes.
Get the host and port from your email provider's documentation.
Fill in your username and password under the user and pass section under auth.

Advanced Deployment

Initial configuration

credits to @ProgrammerIn-wonderland for writing this wonderful tutorial (which can also be found in the docs :)

Create an account at https://www.cloudflare.com/
Create an account at https://www.freenom.com/ (or any registrars)
Find a free domain name at Freenom
Click checkout
- Select (12 Months @ FREE)
- Select "Use DNS"
  - Select Use your own DNS
Go to cloudflare, click add new site, and enter the free domain name
Select "Free Plan"
Click continue, ignore DNS
- Copy the name servers cloudflare gives you
Go back to your Freenom tab, enter in the name servers which cloudflare gave you
- You can keep IP blank
Click continue
Click complete order
Go back to cloudflare tab, click "Check Nameservers"
Select DNS on your right bar
Enter in the IP of the server which will be hosting Nebula
- Target will be @
Click Enable proxy (little gray cloud icon, if active its orange)
Select SSL/TLS in your right bar
Click "Flexible"

Server configuration

SSH into the server you'll be using, I'll assume its running Ubuntu 22.04 (though the commands are the same for debian 10+ versions, and Ubuntu versions 20.04+)
run

curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash - \ &&
sudo apt-get install -y nodejs npm
git clone https://github.com/NebulaServices/Nebula.git
cd Nebula
npm i
npm ci
sudo nohup PORT=80 node . &

Make sure your firewall is configured to let through port 80 traffic!
Note: Server will need to run cd Nebula && sudo nohup PORT=80 node . & on reboot

File Structure

File	Purpose
`src/index.html`	The main frontend visuals for NebulaWEB.
`src/unv.html`	The verification-required frontend/visuals.
`src/options.html`	The frontend for Nebula's options, settings, and preferences.
`public/resources/v.js`	Client verification system for the OTP system.
`public/resources/nebulamain.js`	All of the DOM/client code for NebulaWEB. Includes options, themeSystem, cloak, stealthengine, and more.
`app.js`	The backend server for Nebula. Contains Nodestatic, Bare, HTTP, and more.

Tech Stack

HTML, JS, CSS
Partical.JS (Specifically v4, 5, 6.1 &< only)
Ultraviolet (proxy)
Osana (proxy)
TompHTTP Bare Server Node
ExpressJS

Support

For support, email chloe@nebula.bio or join our discord: discord.gg/unblocker

Demo

Click here to see a demo of Nebula

Acknowledgements

License

(Nebula's license is now GNU AGPL V3 as of v7.10) Copyright Nebula Services 2021 - Present
This project uses the AGLP GNU V3 license.

linkedin/coral

2023-09-10T01:35:22Z

Coral is a translation, analysis, and query rewrite engine for SQL and other relational languages.

Coral

Coral is a SQL translation, analysis, and rewrite engine. It establishes a standard intermediate representation, Coral IR, which captures the semantics of relational algebraic expressions independently of any SQL dialect. Coral IR is defined in two forms: one is the at the abstract syntax tree (AST) layer, and the other is at the logical plan layer. Both forms are isomorphic and convertible to each other.

Coral exposes APIs for implementing conversions between SQL dialects and Coral IR in both directions. Currently, Coral supports converting HiveQL and Spark SQL to Coral IR, and converting Coral IR to HiveQL, Spark SQL, and Trino SQL. With multiple SQL dialects supported, Coral can be used to translate SQL statements and views defined in one dialect to equivalent ones in another dialect. It can also be used to interoperate between engines and SQL-powered data sources. For dialect conversion examples, see the modules coral-hive, coral-spark, and coral-trino.

Coral also exposes APIs for Coral IR rewrite and manipulation. This includes rewriting Coral IR expressions to produce semantically equivalent, but more performant expressions. For example, Coral automates incremental view maintenance by rewriting a view definition to an incremental one. See the module coral-incremental for more details. Other Coral rewrite applications include data governance and policy enforcement.

Coral can be used as a library in other projects, or as a service. See instructions below for more details.

Slack

Join the discussion with the community on Slack here!

Modules

Coral consists of following modules:

Coral-Hive: Converts HiveQL to Coral IR (can be typically used with Spark SQL as well).
Coral-Trino: Converts Coral IR to Trino SQL. Converting Trino SQL to Coral IR is WIP.
Coral-Spark: Converts Coral IR to Spark SQL (can be typically used with HiveQL as well).
Coral-Dbt: Integrates Coral with DBT. It enables applying Coral transformations on DBT models.
Coral-Incremental: Derives an incremental query from input SQL for incremental view maintenance.
Coral-Schema: Derives Avro schema of view using view logical plan and input Avro schemas of base tables.
Coral-Spark-Plan [WIP]: Converts Spark plan strings to equivalent logical plan.
Coral-Visualization: Visualizes Coral SqlNode and RelNode trees and renders them to an output file.
Coral-Service: Service that exposes REST APIs that allow users to interact with Coral (see Coral-as-a-Service for more details).

Version Upgrades

This project adheres to semantic versioning, where the format x.y.z represents major, minor, and patch version upgrades. Consideration should be given to potential changes required when integrating different versions of this project.

Major version Upgrade

A major version upgrade represents a version change that introduces backward incompatibility by removal or renaming of classes.

Minor version Upgrade

A minor version upgrade represents a version change that introduces backward incompatibility by removal or renaming of methods.

Please carefully review the release notes and documentation accompanying each version upgrade to understand the specific changes and the recommended steps for migration.

How to Build

Clone the repository:

git clone https://github.com/linkedin/coral.git

Build:

./gradlew clean build

Please note that this project requires Python 3 and Java 8 to run. Either set JAVA_HOME to the home of an appropriate version and then use ./gradlew clean build as described above, or set the org.gradle.java.home gradle property to the Java home of an appropriate version as below:

./gradlew -Dorg.gradle.java.home=/path/to/java/home clean build

Contributing

The project is under active development and we welcome contributions of different forms. Please see the Contribution Agreement.

Resources

Coral: A SQL translation, analysis, and rewrite engine for modern data lakehouses, LinkedIn Engineering Blog, 12/10/2020.
Incremental View Maintenance with Coral, DBT, and Iceberg, Tech Talk, Iceberg Meetup, 5/11/2023.
Coral & Transport UDFs: Building Blocks of a Postmodern Data Warehouse, Tech-talk, Facebook HQ, 2/28/2020.
Transport: Towards Logical Independence Using Translatable Portable UDFs, LinkedIn Engineering Blog, 11/14/2018.
Dali Views: Functions as a Service for Big Data, LinkedIn Engineering Blog, 11/9/2017.

Coral-as-a-Service

Coral-as-a-Service or simply, Coral Service is a service that exposes REST APIs that allow users to interact with Coral without necessarily coming from a compute engine. Currently, the service supports an API for query translation between different dialects and another for interacting with a local Hive Metastore to create example databases, tables, and views so they can be referenced in the translation API. The service can be used in two modes: remote Hive Metastore mode, and local Hive Metastore mode. The remote mode uses an existing (already deployed) Hive Metastore to resolve tables and views, while the local one creates an empty embedded Hive Metastore so users can add their own table and view definitions.

API Reference

/api/translations/translate

A POST API which takes JSON request body containing following parameters and returns the translated query:

fromLanguage: Input dialect (e.g., spark, trino, hive -- see below for supported inputs)
toLanguage: Output dialect (e.g., spark, trino, hive -- see below for supported outputs)
query: SQL query to translate between two dialects

/api/catalog-ops/execute

A POST API which takes a SQL statement to create a database/table/view in the local metastore (note: this endpoint is only available with Coral Service in local metastore mode).

Instructions to use with examples

Clone Coral repo

git clone https://github.com/linkedin/coral.git

From the root directory of Coral, access the coral-service module

cd coral-service

Build

../gradlew clean build

To run Coral Service using the local metastore:

../gradlew bootRun --args='--spring.profiles.active=localMetastore'

To run Coral Service using the remote metastore:

Add your kerberos client keytab file to coral-service/src/main/resources
Appropriately replace all instances of SET_ME in coral-service/src/main/resources/hive.properties
Run

../gradlew bootRun

You can also specify a custom location of hive.properties file through --hivePropsLocation as follows

 ./gradlew bootRun --args='--hivePropsLocation=/tmp/hive.properties'

Then you can interact with the service using your browser or the CLI.

Coral Service UI

After running ../gradlew bootRun --args='--spring.profiles.active=localMetastore' (for local metastore mode) or ../gradlew bootRun (for remote metastore mode) from coral-service module, the UI can be accessed from the browser. Use the URL http://localhost:8080 to run the UI on a local browser.

The UI provides 2 features:

Create a database/table/view in local metastore mode

This feature is only available with Coral Service in local metastore mode, it calls /api/catalog-ops/execute API above.

You can enter a SQL statement to create a database/table/view in the local metastore:

Translate SQL from source language to target language

This feature is available with Coral Service in both local and remote metastore modes, it calls /api/translations/translate API above.

You can enter a SQL query and specify the source and target language to use Coral translation service:

Coral Service CLI

Apart from the UI above, you can also interact with the service using the CLI.

Example workflow for local metastore mode:

Create a database called db1 in local metastore using the /api/catalog-ops/execute endpoint

curl --header "Content-Type: application/json" \
  --request POST \
  --data "CREATE DATABASE IF NOT EXISTS db1" \
  http://localhost:8080/api/catalog-ops/execute

Creation successful

Create a table called airport within db1 in local metastore using the /api/catalog-ops/execute endpoint

curl --header "Content-Type: application/json" \
  --request POST \
  --data "CREATE TABLE IF NOT EXISTS db1.airport(name string, country string, area_code int, code string, datepartition string)" \
  http://localhost:8080/api/catalog-ops/execute

Creation successful

Translate a query on db1.airport in local metastore using the /api/translations/translate endpoint

curl --header "Content-Type: application/json" \
  --request POST \
  --data '{
    "fromLanguage":"hive", 
    "toLanguage":"trino", 
    "query":"SELECT * FROM db1.airport"
  }' \
  http://localhost:8080/api/translations/translate

The translation result is:

Original query in HiveQL:
SELECT * FROM db1.airport
Translated to Trino SQL:
SELECT "name", "country", "area_code", "code", "datepartition"
FROM "db1"."airport"

Currently Supported Translation Flows

Hive to Trino
Hive to Spark
Trino to Spark
Note: During Trino to Spark translations, views referenced in queries are considered to be defined in HiveQL and hence cannot be used when translating a view from Trino. Currently, only referencing base tables is supported in Trino queries. This translation path is currently a POC and may need further improvements.

apache/incubator-kie-optaplanner

2023-09-10T01:35:22Z

AI constraint solver in Java to optimize the vehicle routing problem, employee rostering, task assignment, maintenance scheduling, conference scheduling and other planning problems.

A fast, easy-to-use, open source AI constraint solver for software developers

Looking for Quickstarts?

OptaPlanner’s quickstarts are located in the optaplanner-quickstarts repository.

Quick development start

To build and run from source:

$ mvn clean install -Dquickly
$ cd optaplanner-examples
$ mvn exec:java

To develop with IntelliJ IDEA, Eclipse or VSCode, open the root pom.xml as a new project and configure a Run/Debug configuration like this:

Type: Application
Main class: org.optaplanner.examples.app.OptaPlannerExamplesApp
VM options: -Xmx2G -server (memory only needed when using the big datasets in the examples)
Program arguments: (none)
Working directory: $MODULE_DIR$ (must resolve to optaplanner-examples directory)
Use classpath of module: optaplanner-examples

Contributing to OptaPlanner

This is an open source project, and you are more than welcome to contribute ❤️!

If you’re just starting out with OptaPlanner and want to contribute, take a look at our starter issues. They’re specifically chosen to be easier for first time contributors.
If you want to contribute or start an opinionated discussion, join our discussion or send an e-mail directly to optaplanner-dev@googlegroups.com.
If you want to submit an issue, check out the OptaPlanner Jira project.

Time to make a change?

Every change must be submitted through a GitHub pull request (PR). OptaPlanner uses continuous integration (CI). The OptaPlanner CI runs checks against your branch after you submit the PR to ensure that your PR doesn’t introduce errors. If the CI identifies a potential problem, our friendly PR maintainers will help you resolve it.

Contributing

Fork it (https://github.com/kiegroup/optaplanner).
Create your feature branch: (git checkout -b feature).
Commit your changes with a comment: (git commit -am 'Add some feature').
Push to the branch to GitHub: (git push origin feature).
Create a new pull request.

Code standards

Your code is automatically formatted according to the Import and Code Style conventions during every Maven build. CI checks enforce those conventions too, so be sure to build your project with maven before creating your PR:

mvn clean install

For information about how to set up code style checks, see IDE Setup Instructions.

Building your OptaPlanner project

Use one of the following ways to build your OptaPlanner project:

🚀 build-fast: mvn clean install -Dquickly skips any checks and code analysis (~1 min)
🔨 build-normally: mvn clean install runs tests, checks code style, skips documentation (~17 min)
🧾 build-doc: mvn clean install at optaplanner/optaplanner-docs creates asciidoctor documentation target/optaplanner-docs-*/html_single/index.html (~2 min)
🦾 build-all: mvn clean install -Dfull runs all checks + creates documentation and distribution files (~20 min)

OptaPlanner CI status

You can check the CI status of the OptaPlanner repositories from the Chain Status webpage.

Jenkins CI Jobs (need VPN access)

All Jenkins jobs can be found under the OptaPlanner folder: https://eng-jenkins-csb-business-automation.apps.ocp-c1.prod.psi.redhat.com/job/KIE/job/optaplanner

Interesting Jenkins CI jobs (need VPN access):

Update Quarkus version
Update Drools version
Release pipeline (only available on release branches)
Post-Release pipeline (only available on release branches)

spring-projects/spring-framework

2023-09-10T01:56:39Z

Spring Framework

This is the home of the Spring Framework: the foundation for all Spring projects. Collectively the Spring Framework and the family of Spring projects are often referred to simply as "Spring".

Spring provides everything required beyond the Java programming language for creating enterprise applications for a wide range of scenarios and architectures. Please read the Overview section of the reference documentation for a more complete introduction.

Code of Conduct

This project is governed by the Spring Code of Conduct. By participating, you are expected to uphold this code of conduct. Please report unacceptable behavior to spring-code-of-conduct@pivotal.io.

Access to Binaries

For access to artifacts or a distribution zip, see the Spring Framework Artifacts wiki page.

Documentation

The Spring Framework maintains reference documentation (published and source), GitHub wiki pages, and an API reference. There are also guides and tutorials across Spring projects.

Micro-Benchmarks

See the Micro-Benchmarks wiki page.

Build from Source

See the Build from Source wiki page and the CONTRIBUTING.md file.

Continuous Integration Builds

Information regarding CI builds can be found in the Spring Framework Concourse pipeline documentation.

Stay in Touch

Follow @SpringCentral, @SpringFramework, and its team members on Twitter. In-depth articles can be found at The Spring Blog, and releases are announced via our releases feed.

License

The Spring Framework is released under version 2.0 of the Apache License.

apache/tomcat

2023-09-10T01:56:39Z

Apache Tomcat

Welcome to Apache Tomcat!

What Is It?

The Apache Tomcat® software is an open source implementation of the Java Servlet, JavaServer Pages, Java Expression Language and Java WebSocket technologies. The Java Servlet, JavaServer Pages, Java Expression Language and Java WebSocket specifications are developed under the Java Community Process.

The Apache Tomcat software is developed in an open and participatory environment and released under the Apache License version 2. The Apache Tomcat project is intended to be a collaboration of the best-of-breed developers from around the world. We invite you to participate in this open development project. To learn more about getting involved, click here or keep reading.

Apache Tomcat software powers numerous large-scale, mission-critical web applications across a diverse range of industries and organizations. Some of these users and their stories are listed on the PoweredBy wiki page.

Apache Tomcat, Tomcat, Apache, the Apache feather, and the Apache Tomcat project logo are trademarks of the Apache Software Foundation.

Get It

For every major Tomcat version there is one download page containing links to the latest binary and source code downloads, but also links for browsing the download directories and archives:

To facilitate choosing the right major Tomcat version one, we have provided a version overview page.

Documentation

The documentation available as of the date of this release is included in the docs webapp which ships with tomcat. You can access that webapp by starting tomcat and visiting http://localhost:8080/docs/ in your browser. The most up-to-date documentation for each version can be found at:

Installation

Please see RUNNING.txt for more info.

Licensing

Please see LICENSE for more info.

Support and Mailing List Information

Free community support is available through the tomcat-users email list and a dedicated IRC channel (#tomcat on Freenode).
If you want freely available support for running Apache Tomcat, please see the resources page here.
If you want to be informed about new code releases, bug fixes, security fixes, general news and information about Apache Tomcat, please subscribe to the tomcat-announce email list.
If you have a concrete bug report for Apache Tomcat, please see the instructions for reporting a bug here.

Contributing

Please see CONTRIBUTING for more info.

thingsboard/thingsboard

2023-09-10T01:56:39Z

Open-source IoT Platform - Device management, data collection, processing and visualization.

ThingsBoard

ThingsBoard is an open-source IoT platform for data collection, processing, visualization, and device management.

Documentation

ThingsBoard documentation is hosted on thingsboard.io.

IoT use cases

Getting Started

Collect and Visualize your IoT data in minutes by following this guide.

Support

Licenses

This project is released under Apache 2.0 License.

bradtraversy/traversy-js-challenges

2023-09-10T01:35:12Z

Challenges & docs from JS Algorithms & Data Structures course

Traversy JS Challenges: Data Structures & Algorithms

Please do not make any PRs to this repo as it goes along with a specific course. I may open another one for student solutions and new challenges.

File Structure

Each folder includes:

readme.md - The challenge/code instructions. This also includes hints, tests and a dropdown with the solution code as well as the explanation of the solution code.
[name].js - This is your working file. It has the name of the function and the function is exported. No parameters are passed to the function. That is up to you to add.
[name]-run.js - File to run the code manually. The function is already imported and called with expected parameters.
[name]-solution.js The solution code with heavy commenting. Some challenges have multiple solutions. If you want the solution without comments, look in the readme.md file.
[name]-test.js - Jest tests for the solution code. You will need to rename this file to [name].test.js to run the tests.

Learning Modules/Lessons

Running Tests

Run the command npm run test from the root directory and it will run all the tests.

These Don't Have to be "Challenges"

You can use the tests to see if your code passes, but use the run files to run the code manually. This is so that you can experiment, console.log, etc.

Getting Started

Clone the repo
Run npm install
Run npm run test to run the tests. Again, you will need to rename the test files and replace the -test with .test to run the tests.

Index of Challenges/Lessons

01. Basic Challenges 1

02. Basic Challenges 2

These are more challenges that have to do with iteration. They are slightly harder than the first set of challenges.

03. High Order Array Methods

04. Recursion

The next batch of challenges/lessons will have to do with recursion. We will first talk about what recursion is and then we can look at some challenges.

05. Complexity

06. Hash Tables, Maps & Sets

07. Stacks, Queues & Linked Lists

In this section, we will look at working with data structures such as stacks, queues, and linked lists. We will also look at fast and slow pointers.

08. Binary Trees & Binary Search Trees & Graphs

In this section, we will look at trees and graphs. We will start with binary trees and binary search trees. We will also look at graphs and graph traversal.

09. Sorting Algorithms

ckolderup/postmarks

2023-09-10T01:35:12Z

a single-user bookmarking website designed to live on the Fediverse

Postmarks

About this project

Postmarks is a bookmarking site that you own yourself and can connect the Fediverse, interacting with other Postmarks sites as well as Mastodon/FireFish/any text-based ActivityPub platform. You can read more about it here:

The site allows the owner to add, edit and delete bookmarks, but only if a valid login is provided. Check the setup below to understand how to do that!

Setting up your site

To set your app up:

If you're using Glitch:
- Rename your project immediately in the project settings, if you intend to be called something else. This determines the domain that your site lives at, which also determines the second half of your @username@project-name.glitch.me identity on the fediverse. NOTE: If you change this later, you will break the connection any existing followers have to your site, they'll have to re-follow the account on its new domain (and depending on the software they're following from, may even prevent them from unfollowing the old URL 😱)
- In your .env editor, create a key ADMIN_KEY and give it a text string as a value. This is your "password" when your browser prompts you, so make it as secure as you need to protect your data.
- Add another key to your .env called SESSION_SECRET and generate a random string for its value. This is your session secret, used to generate the hashed version of your session that gets encoded with the cookies used to store your login. If you make this string too easily guessable, you make it easier for someone to hijack your session and gain unauthorized login. Also, if you ever change this string, it will invalidate all existing cookies.
- Edit the contents of account.json.example to set your @username, display name, bio, and avatar. (If you don't set a username, your default actor name will be 'bookmarks', so people will find you on the fediverse @bookmarks@project-name.glitch.me.)
- THEN: either rename account.json.example to account.json, or copy the contents into a new file called account.json. Whatever username you have in this file when the project first starts you'll need to retain or else you'll break your followers' connection to this account.
Otherwise:
- Set up your domain by editing src/util.js and making the definition of export const domain return a string that is your domain. Fun, huh?
- Create a .env file in the root of the project.
- Add the line ADMIN_KEY=<key> to your .env where <key> is the password you'll enter when the browser prompts you, and another line for SESSION_SECRET=<secret> where <secret> is a random string used when hashing your session for use in a secure cookie.
- Make a file called account.json in the project root. Copy the contents of account.json.example into it and edit the values to set your @username, display name, bio, and avatar. (If you don't set a username, your default actor name will be 'bookmarks', so people will find you on the fediverse @bookmarks@project-name.glitch.me.)
If you're using Glitch, you should be done! If you're running this yourself, run npm run start via whatever mechanism you choose to use to host this website.
Click on the Admin link in the footer, and enter the password (whatever you set ADMIN_KEY to in the .env).
You should be logged in, at which point you can configure various settings, import bookmarks, and use the "Add" links in the header and footer (as well as the bookmarklet, available in the Admin section) to save new bookmarks.

Developing Postmarks

To automatically log all requests to a text file, set add LOGGING_ENABLED=true to your .env file. This will cause all incoming requests to append to request_log.txt in your project folder.

Acknowledgments

The "Postmarks" name is compliments of Casey C (no relation to Casey K), who brainstormed dozens of ideas for the name when Casey was first trying to rename the project. Thank you!
Postmarks (in its default configuration) uses an edited version of Eynav Raphael's "Postmark Stamp" icon from The Noun Project.
It also makes use of free fonts including Averia Sans and Public Sans.
Much of the original form of the site's frontend is lifted from the starter projects available on Glitch. Thank you to all the people who have contributed to those projects over the years!
Much of the original backend of the site is based off of Darius Kazemi's express-activitypub repo. I made a point not to just clone his repo from the start, but then ended up retyping most of it as I learned how things work. While some pieces have been upgraded, much of Darius' work creates the foundation for Postmarks' ActivityPub functionality.

We built this with Glitch!

Glitch is a friendly community where millions of people come together to build web apps and websites.

Need more help? Check out the Help Center for answers to any common questions.
Ready to make it official? Become a paid Glitch member to boost your app with private sharing, more storage and memory, domains and more.

AllJavi/tartarus-startpage

2023-09-10T01:35:12Z

Clean and functional startpage for firefox and chrome with material gruvbox style

Preview

https://github.com/AllJavi/tartarus-startpage/assets/49349604/9a2a3f4c-33ef-4eb3-9243-cc160a56a181

This start page is based on the dawn repository, which has even more functionality. I've tweaked the page's style a bit to match my dotfiles, and I've added some features to make it more comfortable.

Keybindings

Hotkey	Action
`Numrow` \| `MouseWheel` \| `Click`	Switch tabs
`s`	Search Dialog
`q`	Config Dialog (new)
`Esc`	Close Dialogs

Configuration Dialog

The default configuration file is userconfig.js, but you can change it in the configuration dialog. You can find more information about how the file works in the original repository. The available components are tabs, a clock, and weather.

Additionally, there's a new option called fastlink to set the link of the Pokéball button.

Search Dialog

The search dialog allows you to display a search bar with various search engines defined in the configuration. To select each one, you simply need to prefix the query with the corresponding !<id>. By default, the defined search engines are:

!g: google
!d: duckduckgo
!y: youtube
!r: reddit
!p: pinterest

Available banners

cbg-2	cbg-3	cbg-4	cbg-5

cbg-6	cbg-7	cbg-8	cbg-9

Credit

Dawn Startpage (preview)

License

MIT License

evershopcommerce/evershop

2023-09-10T01:56:29Z

🛍️ NodeJS E-commerce Platform

EverShop

Documentation | Demo

Introduction

EverShop is a GraphQL Based and React ecommerce platform with essential commerce features. Built with React, modular and fully customizable.

Quick Start

You can get started with EverShop in minutes by running the following command:

npx create-evershop-app my-app

Documentation

Demo

Explore our demo store.

Demo user:

Email: demo@evershop.io
Password: 123456

Support

If you like my work, feel free to:

⭐ this repository. It helps.
about EverShop. Please accept my gratitude.

Ask a question about EverShop

You can ask questions, and participate in discussions about EverShop-related topics in the EverShop Discord channel.

Create a bug report

If you see an error message or run into an issue, please create bug report. This effort is valued and it will help all EverShop users.

Submit a feature request

If you have an idea, or you're missing a capability that would make development easier and more robust, please Submit feature request.

If a similar feature request already exists, don't forget to leave a "+1". If you add some more information such as your thoughts and vision about the feature, your comments will be embraced warmly :)

Contributing

EverShop is an open-source project. We are committed to a fully transparent development process and appreciate highly any contributions. Whether you are helping us fix bugs, proposing new features, improving our documentation or spreading the word - we would love to have you as part of the EverShop community.

Please refer to our Contribution Guidelines and Code of Conduct.

License

GPL-3.0 License

SocialSisterYi/bilibili-API-collect

2023-09-10T01:56:29Z

哔哩哔哩-API收集整理【不断更新中....】

哔哩哔哩-API收集整理

野生API文档

不断更新中....

本项目旨在对 B站 WEB、APP、TV 等客户端中，散落在世界各地的野生 API 进行收集整理，研究使用方法并对其进行说明，运用了黑箱法、控制变量法、代码逆向分析、拆包及反编译法、网络抓包法等研究办法

本文档探讨的对象是主站业务接口，官方开放平台和直播开放平台均不属于本项目范畴，请移步

B站 API 采用 C/S 结构，大多数接口为 REST API 和 gRPC，少部分接口为 WebSocket；REST API 接口请求数据大多为 url query 表单或 JSON，返回数据大多为 JSON 或 Protobuf，强制使用 https 协议

📖阅读地址：GithubPages

小小的 Demo：~~av583785685~~ 视频失效原因 (Youtube备链)

::: warning ⚠️声明

本项目遵守 CC-BY-NC 4.0 协议，禁止一切商业使用，如需转载请注明作者 ID
请勿滥用，本项目仅用于学习和测试！请勿滥用，本项目仅用于学习和测试！请勿滥用，本项目仅用于学习和测试！
利用本项目提供的接口、文档等造成不良影响及后果与本人无关
由于本项目的特殊性，可能随时停止开发或删档
本项目为开源项目，不接受任何形式的催单和索取行为，更不容许存在付费内容

:::

🌱参与贡献

欢迎各位 dalao 对本项目做出贡献，也希望每个使用者都能提出宝贵的意见

目前本项目存在的问题包括但不限于：

文档二级目录尚未完成
部分文档较旧，修改与更新没有跟进
目前文档使用 markdown 语法编写，不易生成编程语言的 SDK，详见 #604

更多信息请浏览贡献指南

🍴目录

计划整理分类 & 目录：(文档已完结请选中 checkbox)

接口签名与验证
- APP API 签名（appkey与sign）
- 已知的 APPKey
- Wbi 签名（wts与w_rid）
公共错误码
图片格式化
bvid 说明
gRPC API 接口定义
登录
- 登录操作 (人机认证)
  - 短信登录
  - 密码登录
  - 二维码登录
  - SNS 登录 (QQ & 微信 & 微博)
- 登录基本信息
- 个人中心
- 注销登录
- 登录记录
- Web 端 Coookie 刷新
消息中心
- 通知类消息
- 私信
- 设置
用户
- 基本信息
- 状态数
- 关系
- 个人空间
- 检查昵称是否可注册
- 用户注册
- 用户认证类型一览
大会员
- 大会员基本信息
- 大会员中心
- 大会员签到
- 大会员操作
视频
- 视频分区一览 (分区代码)
- 基本信息
- 状态数
- 快照
- 点赞 & 投币 & 收藏 & 分享
- TAG
- 视频推荐
- 播放&下载地址 (视频流)
- 互动视频
- 高能进度条
- 信息上报 (心跳及记录历史)
- 视频属性数据
- 视频在线人数
剧集 (番剧、影视)
- 基本信息
- 播放&下载地址（视频流）
- 时间轴
- 状态数
- 操作
视频弹幕
- protobuf 实时弹幕
- protobuf 弹幕元数据（BAS 弹幕 / 互动弹幕）
- xml 实时弹幕
- 历史弹幕
- 快照
- 弹幕操作
- 高级弹幕
- 屏蔽管理
- 智能防挡弹幕
- 弹幕个人配置修改
- 名词解释
- 点赞查询
视频笔记
- 笔记列表
- 笔记详细信息
- 笔记操作
专栏
- 分区
- 基本信息
- 点赞 & 投币 & 收藏 & 分享
- 文集基本信息
- 获取用户专栏文章列表
音频
- 歌曲基本信息
- 歌单 & 音频收藏夹详细信息
- 状态数
- 投币 & 收藏
- 播放 & 下载地址（音频流）
- 音频榜单
排行榜 & 最新视频
- 排行榜
- 热门视频
- 最新视频
- 入站必刷视频
搜索
- 搜索请求
- 搜索结果
- 默认搜索 & 热搜
- 搜索建议
小黑屋
- 基本信息
- 封禁公示
- 风纪委员及众裁案件相关
评论区
- 评论区明细
- 操作
表情
- 表情及表情包信息
- 操作
创作中心
- 统计与数据
- 列表查询相关
- 电磁力数据
实时广播（通讯协议）
- 视频内广播
充电
- 包月充电操作
- 自定义充电（B币方式）
- 自定义充电（微信 & 支付宝方式）
- 充电留言
- 充电列表
动态
- 动态基本信息
- 发送 & 转载动态
- 根据关键字搜索用户（at 别人时的填充列表）
- 操作
- 动态列表
  - 特定话题动态列表
- 动态内容
相簿
- 基本信息
- 相簿列表
- 推荐作者
- 活动列表
- 操作
- 投稿
历史记录 & 稍后再看
- 历史记录
- 稍后再看
收藏夹
- 基本信息
- 收藏夹内容
- 收藏夹操作
课程
- 课程基本信息
- 已购课程
- 分区推荐列表
- 操作
- 播放 & 下载地址（视频流）
直播
- 直播间基本信息
- 直播分区
- 直播间管理
- 直播间操作
- 直播视频流
- 直播信息流
- 直播红包
- 直播间用户实用 API
转正答题
- 查询信息
- 拉取题目
- 操作
B币钱包
- 基本信息
- B币充值
- 贝壳相关
哔哩哔哩漫画
- 用户信息
- 签到
- 积分商城
- 漫画操作
- 漫画赛季
- 漫读券/已购相关
- 下载
- data.index解析
哔哩哔哩游戏
终端网络查询
- 基于ip的地理位置查询
- 终端信息查询
其他
- 获取当前时间戳
web端组件
- 分区当日投稿数
- 404 页漫画收集
APP端组件
- 开屏图片 + 恰饭珍贵录像
个性装扮
- APP 主题
- 主题色

✨鸣谢

你们的存在，让社区更美好

📖相关协议基础

http 协议：传送门

json 序列格式：传送门

xml 序列格式：传送门

protobuf 序列格式：传送门

💦交流

⚠注意：开源社群欢迎交流探讨，拒绝咨询、不支持合作，黑产号一经发现立即拉黑并举报相关 SRC

QQ 交流群：邀请链接
Telegram 交流群：@bilibili_API_collect_community

🧋发电

欢迎来~~交♂易~~，大家的支持就是我继续开发的动力！

~~请可爱的易姐喝杯奶茶~~

WeChat & Alipay：

OR Aifadian：https://afdian.net/@ShakaiAneE

🔗相关项目推荐

库及文档

jingyuexing/bilibiliAPI
fython/BilibiliAPIDocs
czp3009/bilibili-api
Vespa314/bilibili-api
wnstar/bili-utils
lovelyyoshino/Bilibili-Live-API
flaribbit/bilibili-manga-spider
simon300000/bili-api
iyear/biligo Bilibili API SDK in Golang
bilibili-openplatform/demo: 哔哩哔哩开放平台示例代码库
ddiu8081/blive-message-listener: Bilibili-live danmu listener with type. Bilibili 直播间弹幕监听库，支持类型输出。
Nemo2011/bilibili-api: 哔哩哔哩常用API调用。支持视频、番剧、用户、频道、音频等功能。工具齐全。
CuteReimu/bilibili: 哔哩哔哩API的Go版本SDK

成品

NullPointerException/AnimePipe: 功能完善的Android流媒体综合客户端，支持Bilibili, Youtube, NicoNico
3Shain/BiliChat : 基于h5的B站直播弹幕姬
AncientLysine/BiliLocal:本地弹幕播放器
zyzsdy/biliroku:bilibili 生放送（直播）录制
otakustay/danmaku-to-ass:A站B站弹幕转字幕文件
bilibili-helper/bilibili-helper-o:哔哩哔哩 (bilibili.com) 辅助工具，可以下载视频，查询弹幕发送人以及一些十分实用的直播区功能。
apachecn/BiliDriveEx:基于B站相簿上传的文件分块索引存储器
apachecn/CDNDrive:基于B站相簿上传的文件分块索引存储器
Hsury/BiliDrive:基于B站相簿上传的文件分块索引存储器
Tsuk1ko/bilibili-live-chat:无后端的仿 YouTube Live Chat 风格的简易 Bilibili 弹幕姬
ironmanic/crawler_target_users_good:搜索bilibili特定视频，为评论点赞，关注，私信，一体化服务
dd-center/DDatElectron:DD@Home 分布式项目, 桌面客户端
dd-center/vtbs.moe:B站VTB数据中心
JunzhouLiu/BILIBILI-HELPER:利用Linux Crontab定时任务,云函数，Docker等方式实现B站，哔哩哔哩（Bilibili）每日自动投币，签到，银瓜子兑换硬币，领取大会员福利，大会员月底给自己充电等。每天轻松获取65经验值。
the1812/Bilibili-Evolved:强大的哔哩哔哩增强脚本: 下载视频, 音乐, 封面, 弹幕 / 简化直播间, 评论区, 首页 / 自定义顶栏, 删除广告, 夜间模式 / 触屏设备支持
xlzy520/bili-short-url: 哔哩哔哩短链生成器
zjkwdy/bili_app_splash: B站壁纸娘和开屏图自动下载，每天使用Actions自动同步
Jannchie/BiliOB: BiliOB观测者是一个观测B站UP主及视频数据变化，并予以分析的Web应用程序
biliob233/biliob233.github.io: ~~无可奉告~~
biliup/biliup: 全自动录播、投稿工具，也支持twitch、ytb频道搬运。提供分p上传b站接口，腾讯云内网免流+大幅提速
ddiu8081/bilicli: Bilibili-live danmu dashboard in your terminal.
MotooriKashin/Bilibili-Old: 恢复旧版Bilibili页面，为了那些念旧的人。
SocialSisterYi/bcut-asr: 使用必剪API的语音字幕识别
CzJam/Bili_Realtime_Data: Bilibili粉丝与视频实时数据统计

其他

kuresaru/geetest-validator:geetest调试器
uw-labs/bloomrpc: GUI Client for GRPC Services
grpc/grpc: The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#)
quicktype quicktype generates strongly-typed models and serializers from JSON, JSON Schema, TypeScript, and GraphQL queries, making it a breeze to work with JSON type-safely in many programming languages.一键生成多种语言的json反序列化所需类,以便于快速反序列化, 有网页版

hiteshchoudhary/apihub

2023-09-10T01:56:29Z

Your own API Hub to learn and master API interaction. Ideal for frontend, mobile dev and backend developers.

FreeAPI.app

Problem

We are trying to build a single source API hub that can be used to learn api handling in any programming language. Users can build their front end portfolio in web and mobile apps using this api hub.

What is FreeAPI.app

The FreeAPI project is an innovative and community-driven initiative aimed at providing developers with free and accessible APIs for their projects.

The project focuses on delivering a wide range of APIs that cater to various domains and functionalities, enabling developers to seamlessly integrate these APIs into their applications.

Key highlights of the FreeAPI project include:

Accessibility: The FreeAPI project is committed to eliminating barriers by providing free access to its collection of APIs. Developers can leverage these APIs without any cost limitations, allowing them to experiment, learn, and build innovative applications.
Diverse API Collection: The project offers a diverse and comprehensive collection of APIs that span across different industries, domains, and functionalities. Whether you require social media integrations, payment gateways, machine learning algorithms, or IoT device connectivity, the FreeAPI project has you covered.
Simplified Integration: The FreeAPI project understands the challenges developers face when integrating APIs into their applications. To address this, the project provides clear documentation, code samples, and SDKs, simplifying the integration process and reducing development time and effort.
Community-Driven Development: The project fosters a vibrant and collaborative community of developers. Contributors are encouraged to share their knowledge, engage in discussions, and collaborate on API-related projects. This collective effort ensures the continuous improvement and reliability of the APIs offered by the FreeAPI project.
Learning and Skill Development: The FreeAPI project aims to empower developers by providing a platform for learning and skill development. Through access to various APIs and educational resources, developers can enhance their understanding of API integration, expand their knowledge, and showcase their expertise through building complete projects.

Overall, the FreeAPI project is a valuable resource for developers seeking accessible and diverse APIs.

By fostering a supportive community, the project empowers developers to learn, create, and innovate, ultimately contributing to the growth and advancement of the API integration landscape.

Features:

Introducing our groundbreaking open source API hub project, a dynamic platform designed to revolutionize the way developers interact with APIs.

With an emphasis on openness, accessibility, and learning, our API hub empowers developers of all levels to explore, experiment, and grow their skills in API integration.

Highlights:

Open Source: Our API hub is built on the principles of open source, ensuring transparency, collaboration, and community-driven development. This means that the source code is freely available, allowing developers to customize, extend, and contribute to the project.
Free to Use: We firmly believe in removing barriers to entry, which is why our API hub is completely free to use. Whether you're a seasoned developer or just starting your coding journey, you can leverage our platform without any cost limitations.
Local or Deployment: Flexibility is at the core of our API hub. You have the option to use it locally, running on your own machine, or deploy it to a server, making it accessible to others. This versatility ensures that you can adapt the platform to your specific development environment.
Learning Resource: Our API hub is designed as a comprehensive learning resource, offering a wealth of educational materials, tutorials, and documentation. Whether you're a beginner or seeking to expand your API knowledge, our platform provides the resources you need to learn and improve.
Custom Endpoints for Beginners: For developers at the beginner level, our API hub offers custom endpoints that provide a hands-on experience in handling API responses. These beginner-friendly APIs allow you to practice and familiarize yourself with the basics of working with APIs.
Advanced APIs for Portfolio Building: In addition to beginner-level endpoints, our API hub also provides advanced APIs to challenge and stretch your skills. These APIs enable you to tackle more complex integration scenarios, helping you build a robust portfolio of projects to showcase your expertise.

By combining open source principles, accessibility, and a focus on learning, our API hub project paves the way for developers to thrive in the world of API integration. Join our vibrant community and embark on an exciting journey of discovery, growth, and innovation.

How to contribute - Guidelines

We welcome your interest in contributing to our open source project!

To contribute to FreeAPI, please follow these steps:

Fork the repository.
Create a new branch for your feature or bug fix: git checkout -b feat/your-feature-name or git checkout -b fix/your-bug-fix-name.
Make your changes and commit them using conventional commit messages: git commit -am "feat: Add new feature". For more details on the commit format and other guidelines, please refer to the Contributor Guidelines.
Push your changes to your forked repository: git push origin feat/your-feature-name.
Submit a pull request to the main repository, explaining the changes you've made and providing any necessary details.

Here's a guide on how you can effectively contribute to our API hub:

Pull Requests for Readme Updates: Please refrain from sending pull requests solely for updating the project's readme file. While we appreciate the importance of clear and concise documentation, we prefer to focus on substantial code contributions and feature enhancements.
Grammar Updates: Our team values effective communication, but we're not grammar sticklers. You don't need to send pull requests solely for grammar fixes or minor language improvements. Instead, concentrate on the core functionalities and features of the project.
Avoid Updating Existing Public APIs: To maintain stability and consistency, we discourage direct updates to existing public APIs within the API hub. These APIs have been thoroughly tested and approved. However, if you encounter any bugs or issues, we encourage you to open an issue on our project's issue tracker to notify us.
Build New Project APIs: We encourage you to explore your creativity and contribute by building complete project APIs. These APIs should provide comprehensive solutions that can assist developers in constructing complex projects to showcase their skills and abilities. Your contributions in this area will greatly benefit the community.
Draft a Proposal and Discuss on Discord: Before diving into your project, we recommend drafting a proposal. This can include a mind map or outline of the API you intend to build and its potential benefits. Join our Discord community, where you can share your proposal, discuss ideas, and gather feedback from fellow contributors. Engaging in these discussions will enhance your backend portfolio and help shape the future direction of the project.

We appreciate your enthusiasm and look forward to your valuable contributions to our open source API hub project.

Together, we can foster a collaborative environment and make a significant impact in the API integration landscape.

Click here for detailed contribution guide.

🏁 Installation

📦 Using Docker (recommended)

To run the FreeAPI project, follow these steps:

Install Docker on your machine.
Clone the project repository.
Navigate to the project directory.
Create .env file in the root folder and copy paste the content of .env.sample, and add necessary credentials.
Run the Docker Compose command:

docker-compose up --build --attach backend

# --build: Rebuild the image and run the containers
# --attach: only show logs of Node app container and not mongodb

Access the project APIs at the specified endpoints.

💻 Running locally

To run the FreeAPI project locally, follow these steps:

Install Yarn, NodeJs, MongoDB and MongoDB Compass (optional) on your machine.
Clone the project repository.
Navigate to the project directory.
Create .env file in the root folder and copy paste the content of .env.sample, and add necessary credentials.
Install the packages:

yarn install

Run the project:

yarn start

Access the project APIs at the specified endpoints.

🚄 Using Railway (One-click Deploy)

To self-host the FreeAPI.app application, you can take advantage of a pre-built template that is readily available.

Click the button above to visit railway.app.
Click on the Deploy Now button.
(Optional) Sign in with GitHub to deploy.
Fill in the Repository details:
- Specify the repo name (e.g., freeapi-app).
- Checkmark for Public/Private repository.
For Environment variables, we have provided some default values in the ENV to reduce the burden, but some parameters are mandatory:
- PORT: Do not change the value, let it be set to 8080 to view the swagger docs after deployment.
- MONGODB_URI: Provide the MongoDB Atlas database URL. An example is prefilled for you, edit/update it to continue.
- NODE_ENV: Default set to 'development' to view the logs. You may choose to change it to any other value such as 'prod' to hide them.
- EXPRESS_SESSION_SECRET: It is advised to change the default value to your own secret value.
- ACCESS_TOKEN_SECRET: It is advised to change the default value to your own secret value.
- ACCESS_TOKEN_EXPIRY: Set to 1 day as default.
- REFRESH_TOKEN_SECRET: It is advised to change the default value to your own secret value.
- REFRESH_TOKEN_EXPIRY: Set to 10 days as default.
Once you fill in the required environment parameters, if you choose to add others such as PayPal, Google, and Razorpay, please proceed to mention your credentials in the form.
Click on the Deploy button to trigger the first build.
- Monitor the server logs; if you come across any deployment problems, feel free to raise an issue for our team to look into.

Note: Once the application is deployed, please wait for 3-5 minutes for the swagger docs to be available.

📜 Swagger Docs

Swagger Docs: http://localhost:8080

eclipse-kanto/.eclipsefdn

2023-09-10T01:57:05Z

Repository to host configurations related to the Eclipse Foundation.

yoninazarathy/2504_2023_project1

2023-09-11T01:34:21Z

Template for project 1 in the 2023 MATH2504 course

jbisits/RasterHistograms.jl

2023-09-11T01:34:21Z

Empirical distribution fitting for Rasters.jl.

svaksha/pythonidae

2023-09-10T01:57:14Z

Curated decibans of scientific programming resources in Python.

JuliaWeb/HTTP2.jl

2023-09-10T01:57:14Z

samvher/bert-for-laptops

2023-09-10T01:36:00Z

A BERT that you can train on a (gaming) laptop.

amaiya/onprem

2023-09-10T01:36:00Z

A tool for running on-premises large language models with non-public data

OnPrem.LLM

A tool for running large language models on-premises using non-public data

OnPrem.LLM is a simple Python package that makes it easier to run large language models (LLMs) on your own machines using non-public data (possibly behind corporate firewalls). Inspired by the privateGPT GitHub repo and Simon Willison’s LLM command-line utility, OnPrem.LLM is intended to help integrate local LLMs into practical applications.

The full documentation is here.

A Google Colab demo of installing and using OnPrem.LLM is here.

Install

Once installing PyTorch, you can install OnPrem.LLM with:

pip install onprem

For fast GPU-accelerated inference, see additional instructions below.

How to use

Setup

from onprem import LLM

llm = LLM()

By default, a 7B-parameter model is downloaded and used. If use_larger=True, a 13B-parameter is used. You can also supply the URL to an LLM of your choosing to LLM (see the code generation section below for an example). Currently, only models in GGML format are supported. Future versions of OnPrem.LLM will transition to the newer GGUF format.

Send Prompts to the LLM to Solve Problems

This is an example of few-shot prompting, where we provide an example of what we want the LLM to do.

prompt = """Extract the names of people in the supplied sentences. Here is an example:
Sentence: James Gandolfini and Paul Newman were great actors.
People:
James Gandolfini, Paul Newman
Sentence:
I like Cillian Murphy's acting. Florence Pugh is great, too.
People:"""

saved_output = llm.prompt(prompt)

Cillian Murphy, Florence Pugh

Additional prompt examples are shown here.

Talk to Your Documents

Answers are generated from the content of your documents.

Step 1: Ingest the Documents into a Vector Database

llm.ingest('./sample_data')

2023-09-03 16:30:54.459509: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Loading new documents: 100%|██████████████████████| 2/2 [00:00<00:00, 17.16it/s]

Creating new vectorstore
Loading documents from ./sample_data
Loaded 11 new documents from ./sample_data
Split into 62 chunks of text (max. 500 tokens each)
Creating embeddings. May take some minutes...
Ingestion complete! You can now query your documents using the LLM.ask method

Step 2: Answer Questions About the Documents

question = """What is  ktrain?""" 
answer, docs = llm.ask(question)
print('\n\nReferences:\n\n')
for i, document in enumerate(docs):
    print(f"\n{i+1}.> " + document.metadata["source"] + ":")
    print(document.page_content)

 Ktrain is a low-code machine learning library designed to augment human
engineers in the machine learning workow by automating or semi-automating various
aspects of model training, tuning, and application. Through its use, domain experts can
leverage their expertise while still benefiting from the power of machine learning techniques.

References:



1.> ./sample_data/ktrain_paper.pdf:
lection (He et al., 2019). By contrast, ktrain places less emphasis on this aspect of au-
tomation and instead focuses on either partially or fully automating other aspects of the
machine learning (ML) workﬂow. For these reasons, ktrain is less of a traditional Au-
2

2.> ./sample_data/ktrain_paper.pdf:
possible, ktrain automates (either algorithmically or through setting well-performing de-
faults), but also allows users to make choices that best ﬁt their unique application require-
ments. In this way, ktrain uses automation to augment and complement human engineers
rather than attempting to entirely replace them. In doing so, the strengths of both are
better exploited. Following inspiration from a blog post1 by Rachel Thomas of fast.ai

3.> ./sample_data/ktrain_paper.pdf:
with custom models and data formats, as well.
Inspired by other low-code (and no-
code) open-source ML libraries such as fastai (Howard and Gugger, 2020) and ludwig
(Molino et al., 2019), ktrain is intended to help further democratize machine learning by
enabling beginners and domain experts with minimal programming or data science experi-
4. http://archive.ics.uci.edu/ml/datasets/Twenty+Newsgroups
6

4.> ./sample_data/ktrain_paper.pdf:
ktrain: A Low-Code Library for Augmented Machine Learning
toML platform and more of what might be called a “low-code” ML platform. Through
automation or semi-automation, ktrain facilitates the full machine learning workﬂow from
curating and preprocessing inputs (i.e., ground-truth-labeled training data) to training,
tuning, troubleshooting, and applying models. In this way, ktrain is well-suited for domain
experts who may have less experience with machine learning and software coding. Where

Text to Code Generation

We’ll use the CodeUp LLM by supplying the URL and employ the particular prompt format this model expects.

from onprem import LLM
url = 'https://huggingface.co/TheBloke/CodeUp-Llama-2-13B-Chat-HF-GGML/resolve/main/codeup-llama-2-13b-chat-hf.ggmlv3.q4_1.bin'
llm = LLM(url, n_gpu_layers=43) # see below for GPU information

Setup the prompt based on what this model expects (this is important):

template = """
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:"""

answer = llm.prompt('Write Python code to validate an email address.', prompt_template=template)

Here is an example of Python code that can be used to validate an email address:
```
import re

def validate_email(email):
    # Use a regular expression to check if the email address is in the correct format
    pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
    if re.match(pattern, email):
        return True
    else:
        return False

# Test the validate_email function with different inputs
print("Email address is valid:", validate_email("example@example.com"))  # Should print "True"
print("Email address is invalid:", validate_email("example@"))  # Should print "False"
print("Email address is invalid:", validate_email("example.com"))  # Should print "False"
```
The code defines a function `validate_email` that takes an email address as input and uses a regular expression to check if the email address is in the correct format. The regular expression checks for an email address that consists of one or more letters, numbers, periods, hyphens, or underscores followed by the `@` symbol, followed by one or more letters, periods, hyphens, or underscores followed by a `.` and two to three letters.
The function returns `True` if the email address is valid, and `False` otherwise. The code also includes some test examples to demonstrate how to use the function.

Let’s try out the code generated above.

import re
def validate_email(email):
    # Use a regular expression to check if the email address is in the correct format
    pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
    if re.match(pattern, email):
        return True
    else:
        return False
print(validate_email('sam@@openai.com')) # bad email address
print(validate_email('sam@openai'))      # bad email address
print(validate_email('sam@openai.com'))  # good email address

False
False
True

The generated code may sometimes need editing, but this one worked out-of-the-box.

Speeding Up Inference Using a GPU

The above example employed the use of a CPU.
If you have a GPU (even an older one with less VRAM), you can speed up responses.

Step 1: Install `llama-cpp-python` with CUBLAS support

CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install --upgrade --force-reinstall llama-cpp-python==0.1.69 --no-cache-dir

It is important to use the specific version shown above due to library incompatibilities.

Step 2: Use the `n_gpu_layers` argument with `LLM`

llm = LLM(n_gpu_layers=35)

The value for n_gpu_layers depends on your GPU memory and the model you’re using (e.g., max of 35 for default 7B model). You can reduce the value if you get an error (e.g., CUDA OOM).

With the steps above, calls to methods like llm.prompt will offload computation to your GPU and speed up responses from the LLM.

FAQ

How do I use other models with OnPrem.LLM?

You can supply the URL to other models to the LLM constructor, as we did above in the code generation example.

We currently support models in GGML format. However, the GGML format has now been superseded by GGUF. As of August 21st 2023, llama.cpp no longer supports GGML models, which is why we are pinning to an older version of all dependencies.

Future versions of OnPrem.LLM will use the newer GGUF format.
I’m behind a corporate firewall and am receiving an SSL error when trying to download the model?
Try this:
```
from onprem import LLM
LLM.download_model(url, ssl_verify=False)
```
How do I use this on a machine with no internet access?

Use the LLM.download_model method to download the model files to <your_home_directory>/onprem_data and transfer them to the same location on the air-gapped machine.
For the ingest and ask methods, you will need to also download and transfer the embedding model files:
```
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
model.save('/some/folder')
```
Copy the some/folder folder to the air-gapped machine and supply the path to LLM via the embedding_model parameter.
When installing onprem, I’m getting errors related to llama-cpp-python on Windows/Mac/Linux?

For Linux systems like Ubuntu, try this: sudo apt-get install build-essential g++ clang. Other tips are here.

For Windows systems, either use Windows Subsystem for Linux (WSL) or install Microsoft Visual Studio build tools and ensure the selections shown in this post are installed. WSL is recommended.

For Macs, try following these tips.

If you still have problems, there are various other tips for each of the above OSes in this privateGPT repo thread. Of course, you can also easily use OnPrem.LLM on Google Colab.

Significant-Gravitas/Auto-GPT

2023-09-10T01:36:00Z

An experimental open-source attempt to make GPT-4 fully autonomous.

Auto-GPT: An Autonomous GPT-4 Experiment

💡 Get help - Q&A or Discord 💬

🔴 USE `stable` not `master` 🔴

Download the latest stable release from here: https://github.com/Significant-Gravitas/Auto-GPT/releases/latest. The master branch is under heavy development and may often be in a broken state.

Auto-GPT is an experimental open-source application showcasing the capabilities of the GPT-4 language model. This program, driven by GPT-4, chains together LLM "thoughts", to autonomously achieve whatever goal you set. As one of the first examples of GPT-4 running fully autonomously, Auto-GPT pushes the boundaries of what is possible with AI.

Demo April 16th 2023

https://user-images.githubusercontent.com/70048414/232352935-55c6bf7c-3958-406e-8610-0913475a0b05.mp4

Demo made by Blake Werlinger

🚀 Features

🌐 Internet access for searches and information gathering
💾 Long-term and short-term memory management
🧠 GPT-4 instances for text generation
🔗 Access to popular websites and platforms
🗃️ File storage and summarization with GPT-3.5
🔌 Extensibility with Plugins

Quickstart

Check out the wiki
Get an OpenAI API Key
Download the latest release
Follow the installation instructions
Configure any additional features you want, or install some plugins
Run the app

Please see the documentation for full setup instructions and configuration options.

📖 Documentation

💖 Help Fund Auto-GPT's Development 💖

If you can spare a coffee, you can help to cover the costs of developing Auto-GPT and help to push the boundaries of fully autonomous AI! Your support is greatly appreciated. Development of this free, open-source project is made possible by all the contributors and sponsors. If you'd like to sponsor this project and have your avatar or company logo appear below click here.

⚠️ Limitations

This experiment aims to showcase the potential of GPT-4 but comes with some limitations:

Not a polished application or product, just an experiment
May not perform well in complex, real-world business scenarios. In fact, if it actually does, please share your results!
Quite expensive to run, so set and monitor your API key limits with OpenAI!

🛡 Disclaimer

This project, Auto-GPT, is an experimental application and is provided "as-is" without any warranty, express or implied. By using this software, you agree to assume all risks associated with its use, including but not limited to data loss, system failure, or any other issues that may arise.

The developers and contributors of this project do not accept any responsibility or liability for any losses, damages, or other consequences that may occur as a result of using this software. You are solely responsible for any decisions and actions taken based on the information provided by Auto-GPT.

Please note that the use of the GPT-4 language model can be expensive due to its token usage. By utilizing this project, you acknowledge that you are responsible for monitoring and managing your own token usage and the associated costs. It is highly recommended to check your OpenAI API usage regularly and set up any necessary limits or alerts to prevent unexpected charges.

As an autonomous experiment, Auto-GPT may generate content or take actions that are not in line with real-world business practices or legal requirements. It is your responsibility to ensure that any actions or decisions made based on the output of this software comply with all applicable laws, regulations, and ethical standards. The developers and contributors of this project shall not be held responsible for any consequences arising from the use of this software.

By using Auto-GPT, you agree to indemnify, defend, and hold harmless the developers, contributors, and any affiliated parties from and against any and all claims, damages, losses, liabilities, costs, and expenses (including reasonable attorneys' fees) arising from your use of this software or your violation of these terms.

🐦 Connect with Us on Twitter

Stay up-to-date with the latest news, updates, and insights about Auto-GPT by following our Twitter accounts. Engage with the developer and the AI's own account for interesting discussions, project updates, and more.

Developer: Follow @siggravitas for insights into the development process, project updates, and related topics from the creator of Entrepreneur-GPT.

We look forward to connecting with you and hearing your thoughts, ideas, and experiences with Auto-GPT. Join us on Twitter and let's explore the future of AI together!

microsoft/BatteryML

2023-09-10T01:57:14Z

BatteryML: An Open-Source Tool for Machine Learning on Battery Degradation

Introduction

The performance degradation of lithium batteries is a complex electrochemical process, involving factors such as the growth of solid electrolyte interface, lithium precipitation, loss of active materials, etc. Furthermore, this inevitable performance degradation can have a significant impact on critical commercial scenarios, such as causing 'range anxiety' for electric vehicle users and affecting the power stability of energy storage systems. Therefore, effectively analyzing and predicting the performance degradation of lithium batteries to provide guidance for early prevention and intervention has become a crucial research topic.

To this end, we open source the BatteryML tool to facilitate the research and development of machine learning on battery degradation. We hope BatteryML can empower both battery researchers and data scientists to gain deeper insights from battery degradation data and build more powerful models for accurate predictions and early interventions.

Framework

Highlights:

Open-source and Community-driven: BatteryML is an open-source project for battery degradation modeling, encouraging contributions and collaboration from the communities of both computer science and battery research to push the frontiers of this crucial field.
A Comprehensive Dataset Collection: BatteryML includes a comprehensive dataset collection, allowing easy accesses to most publicly available battery data.
Preprocessing and Feature Engineering: Our tool offers built-in data preprocessing and feature engineering capabilities, making it easier for researchers and developers to prepare ready-to-use battery datasets for machine learning.
A Wide Range of Models: BatteryML already includes most classic models in the literature, enabling developers to quickly compare and benchmark different approaches.
Extensible and Customizable: BatteryML provides flexible interfaces to support further extensions and customizations, making it a versatile tool for potential applications in battery research.

Quick Start

Install the dependencies

pip install -r requirements.txt

Download Raw Data and Run Preprocessing Scripts

Download the raw data and execute the preprocessing scripts as per the provided instruction. You can also use the code below to download public datasets and convert them to BatteryML's uniform data format.

from scripts.download import download_raw_data
from scripts.preprocess import convert_data_format

# Download battery public datasets. If `dataset_name` is set, only the specified dataset will be downloaded. If not set (None), all available preset datasets will be downloaded, including HUST, MATR, CALCE and RWTH.
download_raw_data(dataset_name='matr')

# Convert data format to BatteryML's unified data format. If `dataset_name` is set, only the specified dataset will be converted. If not set (None), all datasets in the BatteryML/data/raw folder will be processed.
convert_data_format(dataset_name='matr')

Run Pipeline

To get started, simply configure the data, features, models, etc. in the config file. Once you've set everything up, run the following code:

from scripts.pipeline import Pipeline

pipeline = Pipeline(config_path=`path/to/your/config`,
                    device='cuda',
                    metric='RMSE',
                    workspace='workspaces')

train_loss , test_loss = pipeline.train()

Note: Replace path/to/your/config with the actual config_path.

Besides, we have prepared an example baseline.

Documentation

By leveraging BatteryML, researchers can gain valuable insights into the latest advancements in battery prediction and materials science, enabling them to conduct experiments efficiently and effectively. We invite you to join us in our journey to accelerate battery research and innovation by contributing to and utilizing BatteryML for your research endeavors.

microsoft/lida

2023-09-10T01:57:14Z

Automatic Generation of Visualizations and Infographics using Large Language Models

LIDA: Automatic Generation of Visualizations and Infographics using Large Language Models

LIDA is a library for generating data visualizations and data-faithful infographics. LIDA is grammar agnostic (will work with any programming language and visualization libraries e.g. matplotlib, seaborn, altair, d3 etc) and works with multiple large language model providers (OpenAI, PaLM, Cohere, Huggingface). Details on the components of LIDA are described in the paper here and in this tutorial notebook. See the project page here for updates!.

Note on Code Execution: To create visualizations, LIDA generates and executes code. Ensure that you run LIDA in a secure environment.

Features

LIDA treats visualizations as code and provides a clean api for generating, executing, editing, explaining, evaluating and repairing visualization code.

Data Summarization
Goal Generation
Visualization Generation
Visualization Editing
Visualization Explanation
Visualization Evaluation and Repair
Visualization Recommendation
Infographic Generation (beta) # pip install lida[infographics]


from lida import Manager, llm

lida = Manager(text_gen = llm("openai")) # palm, cohere ..
summary = lida.summarize("data/cars.csv")
goals = lida.goals(summary, n=2) # exploratory data analysis
charts = lida.visualize(summary=summary, goal=goals[0]) # exploratory data analysis

Getting Started

Setup and verify that your python environment is python 3.10 or higher (preferably, use Conda). Install the library via pip.

pip install lida

Once requirements are met, setup your api key. Learn more about setting up keys for other LLM providers here.

export OPENAI_API_KEY=<your key>

Alternatively you can install the library in dev model by cloning this repo and running pip install -e . in the repository root.

Web API and UI

LIDA comes with an optional bundled ui and web api that you can explore by running the following command:

lida ui  --port=8080 --docs

Then navigate to http://localhost:8080/ in your browser. To view the web api specification, add the --docs option to the cli command, and navigate to http://localhost:8080/api/docs in your browser.

The fastest and recommended way to get started after installation will be to try out the web ui above or run the tutorial notebook.

Data Summarization

Given a dataset, generate a compact summary of the data.

from lida import Manager

lida = Manager()
summary = lida.summarize("data/cars.json") # generate data summary

Goal Generation

Generate a set of visualization goals given a data summary.

goals = lida.goals(summary, n=5) # generate goals

Visualization Generation

Generate, refine, execute and filter visualization code given a data summary and visualization goal. Note that LIDA represents visualizations as code.

# generate charts (generate and execute visualization code)
charts = lida.visualize(summary=summary, goal=goals[0], library="matplotlib") # seaborn, ggplot ..

Visualization Editing

Given a visualization, edit the visualization using natural language.

# modify chart using natural language
instructions = ["convert this to a bar chart", "change the color to red", "change y axes label to Fuel Efficiency", "translate the title to french"]
edited_charts = lida.edit(code=code,  summary=summary, instructions=instructions, library=library, textgen_config=textgen_config)

Visualization Explanation

Given a visualization, generate a natural language explanation of the visualization code (accessibility, data transformations applied, visualization code)

# generate explanation for chart
explanation = lida.explain(code=charts[0].code, summary=summary)

Visualization Evaluation and Repair

Given a visualization, evaluate to find repair instructions (which may be human authored, or generated), repair the visualization.

evaluations = lida.evaluate(code=code,  goal=goals[i], library=library)

Visualization Recommendation

Given a dataset, generate a set of recommended visualizations.

recommendations = lida.recommend(code=code, summary=summary, n=2,  textgen_config=textgen_config)

Infographic Generation [WIP]

Given a visualization, generate a data-faithful infographic. This methods should be considered experimental, and uses stable diffusion models from the peacasso library. You will need to run pip install lida[infographics] to install the required dependencies.

infographics = lida.infographics(visualization = charts[0].raster, n=3, style_prompt="line art")

Important Notes / Caveats / FAQs

LIDA generates and executes code based on provided input. Ensure that you run LIDA in a secure environment with appropriate permissions.
LIDA currently works best with datasets that have a small number of columns (<= 10). This is mainly due to the limited context size for most models. For larger datasets, consider preprocessing your dataset to use a subset of the columns.
LIDA assumes the dataset exists and is in a format that can be loaded into a pandas dataframe. For example, a csv file, or a json file with a list of objects. In practices the right dataset may need to be curated and preprocessed to ensure that it is suitable for the task at hand.
Smaller LLMs (e.g., OSS LLMs on Huggingface) have limited instruction following capabilities and may not work well with LIDA. LIDA works best with larger LLMs (e.g., OpenAI GPT 3.5, GPT 4).
How reliable is the LIDA approach? The LIDA paper describes experiments that evaluate the reliability of LIDA using a visualization error rate metric. With the current version of prompts, data summarization techniques, preprocessing/postprocessing logic and LLMs, LIDA has an error rate of < 3.5% on over 2200 visualizations generated (compared to a baseline of over 10% error rate). This area is work in progress.
Can I build my own apps with LIDA? Yes! You can either use the python api directly in your app or setup a web api endpoint and use the web api in your app. See the web api section for more details.

Naturally, some of these limitations could be addressed by a much welcomed PR.

Documentation and Citation

A short paper describing LIDA (Accepted at ACL 2023 Conference) is available here.

@inproceedings{dibia2023lida,
    title = "{LIDA}: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models",
    author = "Dibia, Victor",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-demo.11",
    doi = "10.18653/v1/2023.acl-demo.11",
    pages = "113--126",
}

LIDA builds on insights in automatic generation of visualization from an earlier paper - Data2Vis: Automatic Generation of Data Visualizations Using Sequence to Sequence Recurrent Neural Networks.

stanfordnlp/dspy

2023-09-10T01:57:14Z

DSPy: The framework for programming with foundation models

DSPy: Programming—not prompting—Foundation Models

DSPy is the framework for solving advanced tasks with language models (LMs) and retrieval models (RMs). DSPy unifies techniques for prompting and fine-tuning LMs — and approaches for reasoning and tool/retrieval augmentation. In DSPy, all of these are expressed through a small set of Pythonic modules that compose and learn.

To make this possible:

DSPy provides composable and declarative modules for instructing LMs in a familiar Pythonic syntax. It upgrades "prompting techniques" like chain-of-thought and self-reflection from hand-adapted string manipulation tricks into truly modular generalized operations that learn to adapt to your task.
DSPy introduces an automatic compiler that teaches LMs how to conduct the declarative steps in your program. Specifically, the DSPy compiler will internally trace your program and then craft high-quality prompts for large LMs (or train automatic finetunes for small LMs) to teach them the steps of your task.

The DSPy compiler bootstraps prompts and finetunes from minimal data without needing manual labels for the intermediate steps in your program. Instead of brittle "prompt engineering" with hacky string manipulation, you can explore a systematic space of modular and trainable pieces.

For complex tasks, DSPy can routinely teach powerful models like GPT-3.5 and local models like T5-base or Llama2-13b to be much more reliable at tasks. DSPy will compile the same program into different few-shot prompts and/or finetunes for each LM.

If you want to see DSPy in action, open our intro tutorial notebook.

Analogy to Neural Networks

When we build neural networks, we don't write manual for-loops over lists of hand-tuned floats. Instead, you might use a framework like PyTorch to compose declarative layers (e.g., Convolution or Dropout) and then use optimizers (e.g., SGD or Adam) to learn the parameters of the network.

Ditto! DSPy gives you the right general-purpose modules (e.g., ChainOfThought, Retrieve, etc.) and takes care of optimizing their prompts for your program and your metric, whatever they aim to do. Whenever you modify your code, your data, or your validation constraints, you can compile your program again and DSPy will create new effective prompts that fit your changes.

1) Installation

All you need is:

pip install dspy-ai

Or open our intro notebook in Google Colab:

Note: If you're looking for Demonstrate-Search-Predict (DSP), which is the previous version of DSPy, you can find it on the v1 branch of this repo.

2) Syntax: You're in charge of the workflow—it's free-form Python code!

DSPy hides tedious prompt engineering, but it cleanly exposes the important decisions you need to make: [1] what's your system design going to look like? [2] what are the important constraints on the behavior of your program?

You express your system as free-form Pythonic modules. DSPy will tune the quality of your program in whatever way you use foundation models: you can code with loops, if statements, or exceptions, and use DSPy modules within any Python control flow you think works for your task.

Suppose you want to build a simple retrieval-augmented generation (RAG) system for question answering. You can define your own RAG program like this:

class RAG(dspy.Module):
    def __init__(self, num_passages=3):
        super().__init__()
        self.retrieve = dspy.Retrieve(k=num_passages)
        self.generate_answer = dspy.ChainOfThought("context, question -> answer")
    
    def forward(self, question):
        context = self.retrieve(question).passages
        answer = self.generate_answer(context=context, question=question)
        return answer

A program has two key methods, which you can edit to fit your needs.

Your __init__ method declares the modules you will use. Here, RAG will use the built-in Retrieve for retrieval and ChainOfThought for generating answers. DSPy offers general-purpose modules that take the shape of your own sub-tasks — and not pre-built functions for specific applications.

Modules that use the LM, like ChainOfThought, require a signature. That is a declarative spec that tells the module what it's expected to do. In this example, we use the short-hand signature notation context, question -> answer to tell ChainOfThought it will be given some context and a question and must produce an answer. We will discuss more advanced signatures below.

Your forward method expresses any computation you want to do with your modules. In this case, we use the modules self.retrieve and self.generate_answer to search for some context and then use the context and question to generate the answer!

You can now either use this RAG program in zero-shot mode. Or compile it to obtain higher quality. Zero-shot usage is simple. Just define an instance of your program and then call it:

rag = RAG()  # zero-shot, uncompiled version of RAG
rag("what is the capital of France?").answer  # -> "Paris"

The next section will discuss how to compile our simple RAG program. When we compile it, the DSPy compiler will annotate demonstrations of its steps: (1) retrieval, (2) using context, and (3) using chain-of-thought to answer questions. From these demonstrations, the DSPy compiler will make sure it produces an effective few-shot prompt that works well with your LM, retrieval model, and data. If you're working with small models, it'll finetune your model (instead of prompting) to do this task.

If you later decide you need another step in your pipeline, just add another module and compile again. Maybe add a module that takes the chat history into account during search?

3) Two Powerful Concepts: Signatures & Teleprompters

To make it possible to compile any program you write, DSPy introduces two simple concepts: Signatures and Teleprompters.

3.a) Declaring the input/output behavior of LMs with `dspy.Signature`

When we assign tasks to LMs in DSPy, we specify the behavior we need as a Signature. A signature is a declarative specification of input/output behavior of a DSPy module.

Instead of investing effort into how to get your LM to do a sub-task, signatures enable you to inform DSPy what the sub-task is. Later, the DSPy compiler will figure out how to build a complex prompt for your large LM (or finetune your small LM) specifically for your signature, on your data, and within your pipeline.

A signature consists of three simple elements:

A minimal description of the sub-task the LM is supposed to solve.
A description of one or more input fields (e.g., input question) that will we will give to the LM.
A description of one or more output fields (e.g., the question's answer) that we will expect from the LM.

We support two notations for expressing signatures. The short-hand signature notation is for quick development. You just provide your module (e.g., dspy.ChainOfThought) with a string with input_field_name_1, ... -> output_field_name_1, ... with the fields separated by commas.

In the RAG class earlier, we saw:

self.generate_answer = dspy.ChainOfThought("context, question -> answer")

In many cases, this barebones signature is sufficient. However, sometimes you need more control. In these cases, we can use the full notation to express a more fully-fledged signature below.

class GenerateSearchQuery(dspy.Signature):
    """Write a simple search query that will help answer a complex question."""

    context = dspy.InputField(desc="may contain relevant facts")
    question = dspy.InputField()
    query = dspy.OutputField()

### inside your program's __init__ function
self.generate_answer = dspy.ChainOfThought(GenerateSearchQuery)

You can optionally provide a prefix and/or desc key for each input or output field to refine or constraint the behavior of modules using your signature.

3.b) Asking DSPy to automatically optimize your program with `dspy.teleprompt.*`

After defining the RAG program, we can compile it. Compiling a program will update the parameters stored in each module. For large LMs, this is primarily in the form of creating and validating good demonstrations for inclusion in your prompt(s).

Compiling depends on three things: a (potentially tiny) training set, a metric for validation, and your choice of teleprompter from DSPy. Teleprompters are powerful optimizers (included in DSPy) that can learn to bootstrap and select effective prompts for the modules of any program. (The "tele-" in the name means "at a distance", i.e., automatic prompting at a distance.)

DSPy typically requires very minimal labeling. For example, our RAG pipeline may work well with just a handful of examples that contain a question and its (human-annotated) answer. Your pipeline may involve multiple complex steps: our basic RAG example includes a retrieved context, a chain of thought, and the answer. However, you only need labels for the initial question and the final answer. DSPy will bootstrap any intermediate labels needed to support your pipeline. If you change your pipeline in any way, the data bootstrapped will change accordingly!

my_rag_trainset = [
  dspy.Example(
    question="Which award did Gary Zukav's first book receive?",
    answer="National Book Award"
  ),
  ...
]

Second, define your validation logic, which will express some constraints on the behavior of your program or individual modules. For RAG, we might express a simple check like this:

def validate_context_and_answer(example, pred, trace=None):
    # check the gold label and the predicted answer are the same
    answer_match = example.answer.lower() == pred.answer.lower()

    # check the predicted answer comes from one of the retrieved contexts
    context_match = any((pred.answer.lower() in c) for c in pred.context)

    return answer_match and context_match

Different teleprompters offer various tradeoffs in terms of how much they optimize cost versus quality, etc. For RAG, we might use the simple teleprompter called BootstrapFewShot. To do so, we instantiate the teleprompter itself with a validation function my_rag_validation_logic and then compile against some training set my_rag_trainset.

from dspy.teleprompt import BootstrapFewShot

teleprompter = BootstrapFewShot(metric=my_rag_validation_logic)
compiled_rag = teleprompter.compile(RAG(), trainset=my_rag_trainset)

If we now use compiled_rag, it will invoke our LM with rich prompts with few-shot demonstrations of chain-of-thought retrieval-augmented question answering on our data.

4) Documentation & Tutorials

While we work on new tutorials and documentation, please check out our intro notebook.

Or open it directly in free Google Colab:

Intro Tutorial [coming soon]

[Intro-01] Getting Started: High Quality Pipelined Prompts with Minimal Effort

[Intro-02] Using DSPy For Your Own Task: Building Blocks

[Intro-03] Adding Complexity: Multi-stage Programs

[Intro-04] Adding Complexity for Your Own Task: Design Patterns

Advanced Demos [coming soon]

[Advanced-01] Long-Form QA & Programmatic Evaluation.

[Advanced-02] Programmatic Evaluation II & Dataset Creation.

[Advanced-03] Compiling & Teleprompters.

[Advanced-04] Extending DSPy with Modules or Teleprompters.

[Advanced-05]: Agents and General Tool Use in DSPy.

[Advanced-06]: Reproducibility, Saving Programs, and Advanced Caching.

Module Reference [coming soon]

Language Model Clients

dspy.OpenAI
dspy.Cohere
dspy.TGI
dspy.VLLM

Retrieval Model Clients

dspy.ColBERTv2
dspy.AzureCognitiveSearch

Signatures

dspy.Signature
dspy.InputField
dspy.OutputField

Modules

dspy.Predict
dspy.Retrieve
dspy.ChainOfThought
dspy.SelfConsistency [use functional dspy.majority now]
dspy.MultiChainReasoning [coming soon]
dspy.SelfCritique [coming soon]
dspy.SelfRevision [coming soon]

Teleprompters

dspy.teleprompt.LabeledFewShot
dspy.teleprompt.BootstrapFewShot
dspy.teleprompt.BootstrapFewShotWithRandomSearch
dspy.teleprompt.BootstrapFinetune

5) FAQ: Is DSPy right for me?

The DSPy philosophy and abstraction differ significantly from other libraries and frameworks, so it's usually straightforward to decide when DSPy is (or isn't) the right framework for your usecase.

If you're a NLP/AI researcher (or a practitioner exploring new pipelines or new tasks), the answer is generally an invariable yes. If you're a practitioner doing other things, please read on.

[5.a] DSPy vs. thin wrappers around prompts (OpenAI API, MiniChain, basic templating, etc.)

In other words: Why can't I just write my prompts directly as string templates? Well, for extremely simple settings, this might work just fine. (If you're familiar with neural networks, this is like expressing a tiny two-layer NN as a Python for-loop. It kinda works.)

However, when you need higher quality (or manageable cost), then you need to iteratively explore multi-stage decomposition, improved prompting, data bootstrapping, careful finetuning, retrieval augmentation, and/or using smaller (or cheaper, or local) models. The true expressive power of building with foundation models lies in the interactions between these pieces. But every time you change one piece, you likely break (or weaken) multiple other components.

DSPy cleanly abstracts away (and powerfully optimizes) the parts of these interactions that are external to your actual system design. It lets you focus on designing the module-level interactions: the same program expressed in 10 or 20 lines of DSPy can easily be compiled into multi-stage instructions for GPT-4, detailed prompts for Llama2-13b, or finetunes for T5-base.

Oh, and you wouldn't need to maintain long, brittle, model-specific strings at the core of your project anymore.

[5.b] DSPy vs. application development libraries like LangChain, LlamaIndex

Note: If you use LangChain as a thin wrapper around your own prompt strings, refer to answer [5.a] instead.

LangChain and LlamaIndex are popular libraries that target high-level application development with LMs. They offer many batteries-included, pre-built application modules that plug in with your data or configuration. In practice, indeed, many usecases genuinely don't need any special components. If you'd be happy to use someone's generic, off-the-shelf prompt for question answering over PDFs or standard text-to-SQL as long as it's easy to set up on your data, then you will probably find a very rich ecosystem in these libraries.

Unlike these libraries, DSPy doesn't internally contain hand-crafted prompts that target specific applications you can build. Instead, DSPy introduces a very small set of much more powerful and general-purpose modules that can learn to prompt (or finetune) your LM within your pipeline on your data.

DSPy offers a whole different degree of modularity: when you change your data, make tweaks to your program's control flow, or change your target LM, the DSPy compiler can map your program into a new set of prompts (or finetunes) that are optimized specifically for this pipeline. Because of this, you may find that DSPy obtains the highest quality for your task, with the least effort, provided you're willing to implement (or extend) your own short program. In short, DSPy is for when you need a lightweight but automatically-optimizing programming model — not a library of predefined prompts and integrations.

If you're familiar with neural networks:

This is like the difference between PyTorch (i.e., representing DSPy) and HuggingFace Transformers (i.e., representing the higher-level libraries). If you simply want to use off-the-shelf BERT-base-uncased or GPT2-large or apply minimal finetuning to them, HF Transformers makes it very straightforward. If, however, you're looking to build your own architecture (or extend an existing one significantly), you have to quickly drop down into something much more modular like PyTorch. Luckily, HF Transformers is implemented in backends like PyTorch. We are similarly excited about high-level wrapper around DSPy for common applications. If this is implemented using DSPy, your high-level application can also adapt significantly to your data in a way that static prompt chains won't. Please open an issue if this is something you want to help with.

[5.c] DSPy vs. generation control libraries like Guidance, LMQL, RELM, Outlines

Guidance, LMQL, RELM, and Outlines are all exciting new libraries for controlling the individual completions of LMs, e.g., if you want to enforce JSON output schema or constrain sampling to a particular regular expression.

This is very useful in many settings, but it's generally focused on low-level, structured control of a single LM call. It doesn't help ensure the JSON (or structured output) you get is going to be correct or useful for your task.

In contrast, DSPy automatically optimizes the prompts in your programs to align them with various task needs, which may also include producing valid structured ouputs. That said, we are considering allowing Signatures in DSPy to express regex-like constraints that are implemented by these libraries.

Contributors & Acknowledgements

DSPy is led by Omar Khattab at Stanford NLP with Chris Potts and Matei Zaharia.

Key contributors and team members include Arnav Singhvi, Paridhi Maheshwari, Keshav Santhanam, Sri Vardhamanan, Eric Zhang, Hanna Moazam, and Thomas Joshi.

DSPy includes important contributions from Rick Battle and Igor Kotenkov. It reflects discussions with Lisa Li, David Hall, Ashwin Paranjape, Heather Miller, Chris Manning, Percy Liang, and many others.

📜 Citation & Reading More

To stay up to date or learn more, follow @lateinteraction on Twitter.

If you use DSPy (or DSPv1) in a research paper, please cite our work as follows:

@article{khattab2022demonstrate,
  title={Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive {NLP}},
  author={Khattab, Omar and Santhanam, Keshav and Li, Xiang Lisa and Hall, David and Liang, Percy and Potts, Christopher and Zaharia, Matei},
  journal={arXiv preprint arXiv:2212.14024},
  year={2022}
}

You can also read more about the old v1 of our framework (Demonstrate–Search–Predict, or DSP):

Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP (Academic Paper, Dec 2022)
Introducing DSP (Twitter Thread, Jan 2023)
Thread Releasing the DSP Compiler (v0.1) (Twitter Thread, Feb 2023)

android/snippets

2023-09-10T01:36:24Z

Mi-Freeform 3 is an Android third-party freeform software that supports Android 8.1-Android 13

Mi-Freeform

中文

Mi-Freeform 3 is an Android third-party freeform software that supports Android 8.1-Android 13.

TODO

Compatible with Android 8.1-Android 9 ⭐⭐⭐⭐⭐
Redo the sidebar trigger way ⭐⭐⭐⭐
Sliding style window ⭐⭐⭐
Takeover notification ⭐⭐
Start in multitasking ⭐

Defect

Touch sampling rate
Some apps will reload after restoring to the default screen

License

Copyright (C) 2021-2023  sunshine0523

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

getsentry/sentry-java

2023-09-10T01:57:33Z

A Sentry SDK for Java, Android and other JVM languages.

Bad software is everywhere, and we're tired of it. Sentry is on a mission to help developers write better software faster, so we can get back to enjoying technology. If you want to join us Check out our open positions

Sentry SDK for Java and Android

Packages	Maven Central	Android API
sentry-android		16
sentry-android-core		14
sentry-android-ndk		16
sentry-android-okhttp		21
sentry-android-timber		14
sentry-android-fragment		14
sentry-android-navigation		14
sentry-android-sqlite		14
sentry-compose-android		21
sentry-compose-desktop
sentry-compose
sentry-apache-http-client-5
sentry		14
sentry-jul
sentry-jdbc
sentry-apollo		14
sentry-apollo-3		14
sentry-kotlin-extensions		14
sentry-servlet
sentry-servlet-jakarta
sentry-spring-boot
sentry-spring-boot-jakarta
sentry-spring-boot-starter
sentry-spring-boot-starter-jakarta
sentry-spring
sentry-spring-jakarta
sentry-logback
sentry-log4j2
sentry-bom
sentry-graphql
sentry-openfeign
sentry-opentelemetry-agent
sentry-opentelemetry-agentcustomization
sentry-opentelemetry-core

Useful links and docs

Current Javadocs generated from source code.
Java SDK version 1.x can still be found here.
Migration page from sentry-android 1.x and 2.x to sentry-android 4.x.
Migration page from sentry 1.x to sentry 4.x.
Releases from sentry-android 2.x and its changelogs.
Sentry Android Gradle Plugin repo sits on another repo

Blog posts

Samples

Development

This repository includes sentry-native as a git submodule. To build against sentry-native checked-out elsewhere in your file system, create a symlink sentry-android-ndk/sentry-native-local that points to your sentry-native directory. For example, if you had sentry-native checked-out in a sibling directory to this repo:

ln -s ../../sentry-native sentry-android-ndk/sentry-native-local

which will be picked up by gradle and used instead of the git submodule. This directory is also included in .gitignore not to be shown as pending changes.

Sentry Self Hosted Compatibility

Since version 3.0.0 of this SDK, Sentry version >= v20.6.0 is required. This only applies to self-hosted Sentry, if you are using sentry.io no action is needed.

Since version 6.0.0 of this SDK, Sentry version >= v21.9.0 is required or you have to manually disable sending client reports via the sendClientReports option. This only applies to self-hosted Sentry, if you are using sentry.io no action is needed.

Resources

zhanghai/MaterialFiles

2023-09-10T01:57:33Z

Material Design file manager for Android

Material Files

本文中文版

An open source Material Design file manager, for Android 5.0+.

Get it on Coolapk Get the APK

Help translation on Transifex (Search Android & GNOME translations, Search Microsoft translations, MIME type translations)

Preview

Features

Open source: Lightweight, clean and secure.
Material Design: Follows Material Design guidelines, with attention into details.
Breadcrumbs: Navigate in the filesystem with ease.
Root support: View and manage files with root access.
Archive support: View, extract and create common compressed files.
NAS support: View and manage files on FTP, SFTP and SMB servers.
Themes: Customizable UI colors, plus night mode with optional true black.
Linux-aware: Like Nautilus, knows symbolic links, file permissions and SELinux context.
Robust: Uses Linux system calls under the hood, not yet another ls parser.
Well-implemented: Built upon the right things, including Java NIO2 File API and LiveData.

Why Material Files?

Because I like Material Design, and clean Material Design.

There are already a handful of powerful file managers, but most of them just isn't Material Design. And even among the ones with Material Design, they usually have various minor design flaws (layout, alignment, padding, icon, font, etc) across the app which makes me uncomfortable, while still being minor enough so that not everybody would care to fix it. So I had to create my own.

Because I want an open source file manager.

Most of the popular and reliable file managers are just closed source, and I sometimes use them to view and modify files that require root access. But deep down inside, I just feel uneasy with giving any closed source app the root access to my device. After all, that means giving literally full access to my device, which stays with me every day and stores my own information, and what apps do with such access merely depends on their good intent.

Because I want a file manager that is implemented the right way.

This app implemented Java NIO2 File API as its backend, instead of inventing a custom model for file information/operations, which often gets coupled with UI logic and grows into a mixture of everything (example). On the contrary, a decoupled backend allows cleaner code (which means less bugs), and easier addition of support for other file systems.
This app doesn't use java.io.File or parse the output of ls, but built bindings to Linux syscalls to properly access the file system. java.io.File is an old API missing many features, and just can't handle things like symbolic links correctly, which is the reason why many people rather parse ls instead. However parsing the output ls is not only slow, but also unreliable, which made Cabinet broken on newer Android versions. By virtue of using Linux syscalls, this app is able to be fast and smooth, and handle advanced things like Linux permissions, symbolic links and even SELinux context. It can also handle file names with invalid UTF-8 encoding because paths are not naively stored as Java Strings, which most file managers does and fails during file operation.
This app built its frontend upon modern ViewModel and LiveData which enables a clear code structure and support for rotation. It also properly handles things like errors during file operation, file conflicts and foreground/background state.

In a word, this app tries to follow the best practices on Android and do the right thing, while keeping its source code clean and maintainable.

Because I know people can do it right.

Nautilus is a beautifully-designed and user-friendly file manager on Linux desktop, and it's fully Linux-aware. Phonograph is an open source Material Design music player app (which I've been using for years), and it has just the right Material Design and implementation.

So, it's time for yet another Android file manager.

Inclusion in custom ROMs

Thank you if you choose to include Material Files in your custom ROM! However since I've received several user complaints due to improper inclusion, I'd like to offer some suggestions on including this app properly for the good of end users:

Please don't replace the AOSP DocumentsUI app with this app. This app is not designed to replace DocumentsUI and can't handle a number of functionalities in DocumentsUI - in fact, it relies on DocumentsUI to do things like granting external SD card access.
Please make sure this app can be uninstalled or at least disabled. Some users may not want this app for a variety of reasons, and get very upset when they can't remove it.
Please avoid conflict with the Play/F-Droid version of this app. App stores cannot update apps signed with a different certificate, so you can either ship an APK that's signed by me (or F-Droid) so that users will be able to update it on Play/F-Droid, or fork this project and rename the package name when you need to sign the APK with a different certificate and potentially making other changes.

License

Copyright (C) 2018 Hai Zhang

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <https://www.gnu.org/licenses/>.

rumboalla/apkupdater

2023-09-10T01:57:33Z

APKUpdater is an open source tool that simplifies the process of finding updates for your installed apps.

APKUpdater

APKUpdater is an open source tool that simplifies the process of finding updates for your installed apps.
It provides similar functionality to an app store, but instead of depending on a single source, it aggregates the results from APKMirror, Aptoide, F-Droid, IzzyOnDroid, APKPure and GitHub.

The 3.x branch is a full rewrite using modern technologies like Jetpack Compose, Flow and WorkManager.

Features

Update Sources: Find updates from APKMirror, Aptoide, F-Droid, IzzyOnDroid, APKPure and GitHub.
Search Sources: Find new apps to install from APKMirror, Aptoide, F-Droid, IzzyOnDroid, APKPure and GitHub.
Schedule background update checks and receive a notification when updates are found.
Supports Android 5 (21) to Android 14 (34).
Supports Android TV.
Material Design 3 with Dark, Light and System theme support.
Supports Material You on Android 12+.
Direct install of updates for sources that support it.
Supports installs without user interaction on Android 12+.
Root install of updates.
No ads. No tracking.
Languages: English, Spanish, Dutch, German, Traditional Chinese, Simplified Chinese, Romanian, Italian, Portuguese, Russian, Turkish, Indonesian.

Download

Translations

If you want to help with translations, open a Pull Request or an Issue with the translated strings.xml.

Dutch by AnonymousWP
German by peat80
Traditional Chinese by abc0922001
Simplified Chinese by Nriver
Romanian by StormProductionsMusic
Italian by NicKoehler
Portuguese by zekabra
Russian by Xenorant
Turkish by kyoyacchi
Indonesian by HazakuraID

Feedback

To give feedback and request new features go to Discussions.
To report bugs, crashes, typos and translations go to Issues.

Screenshots

Other Projects

CoolRs: A collection of Android RenderScript effects.
KryptoPrefs: Kotlin library for handling encrypted SharedPreferences.

License

lftcm2023/lftcm2023

2023-09-10T01:36:41Z

gebner/quote4

2023-09-10T01:57:48Z

Intuitive, type-safe expression quotations for Lean 4.

JLimperg/aesop

2023-09-10T01:57:48Z

White-box automation for Lean 4

pabloprogramador/CustomShellMaui

2023-09-10T01:36:59Z

Custom pages animation transition.

Bremaweb/domination

2023-09-10T01:58:04Z

Domination Minetest Game Mode

xiaorouji/openwrt-passwall2

2023-09-10T01:58:19Z

A Xray core for Openwrt LuCI Application.

AlexvZyl/nordic.nvim

2023-09-10T01:58:19Z

🌒 Nord for Neovim, but warmer and darker. Supports a variety of plugins and other platforms.

🌒 Nordic

A Neovim colorscheme based on Nord, but Aurora > Frost. The idea behind this colorscheme is to use Nord, but add some darker colors and use Aurora more prominently than Nord themes usually do. This ends up creating a colorscheme that is soft on the eyes.

📷 Showcase

From my dotfiles:

🎨 Palette

For the list of colors/palette, see this file. Some extra colors and use cases are generated in this file.

📦 Installation

With packer.nvim:

use 'AlexvZyl/nordic.nvim'

With lazy.nvim:

{
    'AlexvZyl/nordic.nvim',
    lazy = false,
    priority = 1000,
    config = function()
        require 'nordic' .load()
    end
}

With vim-plug:

Plug 'AlexvZyl/nordic.nvim', { 'branch': 'main' }

🚀 Usage

Using vim:

colorscheme nordic

Using lua:

vim.cmd.colorscheme 'nordic'
-- or
require 'nordic' .load()

Using with lualine:

require 'lualine' .setup {
    options = {
        theme = 'nordic'
    }
}

To get the palette in lua:

local palette = require 'nordic.colors'

⚙️ Configuration

Nordic will use the default values, unless setup is called. Below is the default configuration.

require 'nordic' .setup {
    -- This callback can be used to override the colors used in the palette.
    on_palette = function(palette) return palette end,
    -- Enable bold keywords.
    bold_keywords = false,
    -- Enable italic comments.
    italic_comments = true,
    -- Enable general editor background transparency.
    transparent_bg = false,
    -- Enable brighter float border.
    bright_border = false,
    -- Reduce the overall amount of blue in the theme (diverges from base Nord).
    reduced_blue = true,
    -- Swap the dark background with the normal one.
    swap_backgrounds = false,
    -- Override the styling of any highlight group.
    override = {},
    -- Cursorline options.  Also includes visual/selection.
    cursorline = {
        -- Bold font in cursorline.
        bold = false,
        -- Bold cursorline number.
        bold_number = true,
        -- Avialable styles: 'dark', 'light'.
        theme = 'dark',
        -- Blending the cursorline bg with the buffer bg.
        blend = 0.7,
    },
    noice = {
        -- Available styles: `classic`, `flat`.
        style = 'flat',
    },
    telescope = {
        -- Available styles: `classic`, `flat`.
        style = 'flat',
    },
    leap = {
        -- Dims the backdrop when using leap.
        dim_backdrop = false,
    },
    ts_context = {
        -- Enables dark background for treesitter-context window
        dark_background = true,
    }
}

An example of overriding the TelescopePromptTitle colors:

local palette = require 'nordic.colors'
require 'nordic' .setup {
    override = {
        TelescopePromptTitle = {
            fg = palette.red.bright,
            bg = palette.green.base,
            italic = true,
            underline = true,
            sp = palette.yellow.dim,
            undercurl = false
        }
    }
}

🗒️ Supported Plugins and Platforms

For the list of supported plugins, please take a look at this directory. For the list of supported platforms, please take a look at this directory.

I do not personally use all of the platforms and plugins in the list, so if something is not right, or you have a suggestion, please open a PR!

🎙️ Acknowledgements

folke/tokyonight served as an excellent example and template to create a Neovim theme.
Dotfiles used in the screenshots.

JonnyBro/beatrun

2023-09-10T01:58:19Z

Beatrun | Community version

Infamous parkour addon for Garry's Mod, fully open sourced and maintained by the community.

This version does not include malicious modules, code or networking. What it does contain is:

Lua modules for Discord Rich Presence
Lua modules for Steam Presence
Network connectivity for courses (activates only when you load or upload courses, and by default courses.beatrun.ru is used)

All of this is optional and you may remove all of it. (modules are located here and online courses functionality is here)

Installation (Manual)

Download this repository here.
Delete the beatrun folder in addons if you have one!
Extract the beatrun folder to your_game_folder/garrysmod/addons.
Extract the lua folder to your_game_folder/garrysmod.
- lua folder contains modules for Discord Rich Presense and Steam Presence. They are optional. You can find their source code in the credits section

Installation (Automatic | Windows only)

Run the command below in the Powershell. (Win + R --> powershell --> the command below)

iex (iwr "beatrun.ru/install.ps1" -UseBasicParsing)

Animations

You can use "Beatrun Animations Installer" for custom animations. Currently there's:

Beatrun Reanimated
Fixed Original

Start the executable and press a key on your keyboard with the number of the animation you want to install (if nothing's changed, run the program as admin).
Installer's source can be found here.

Changes and fixes done by me

Created a custom online courses database, which is also free and open source 🤯!
Added a new gamemode - Deathmatch, it's like Data Theft, but you collect kills not cubes!
Added an in-game config menu - You can find it in the tool menu, in the Beatrun Category!
All of the settings below can be changed in the configuration menu.
Added the ability Getting off of ladders.
Allowed jumping while slowwalking.
Done various tweaks to the Time Trials Menu (F4).
Added an arrow that shows the next checkpoint.
Added a convar to allow Overdrive usage on the server - Beatrun_AllowOverdriveInMultiplayer.
Added a convar to toggle between old and new (like in ME) Kick-Glitch - Beatrun_OldKickGlitch.
Added convars to change HUD's colors - Beatrun_HUDTextColor, Beatrun_HUDCornerColor, Beatrun_HUDFloatingXPColor.
Added a convar to allow players to spawn props without admin rights - Beatrun_AllowPropSpawn.
Added a convar to disable grapple ability - Beatrun_DisableGrapple.
Implemented Discord Rich Presence using open source tools (See step 3).
Added small camera punch when diving.
Added the ability to remove ziplines that created with Zipline Gun - RMB.
SteamID in the right corner is no longer present.

Notable changes and fixes done by the community

Made QuickTurn work only with Runner Hands.
Fixed leaderboard sorting in gamemodes.
Fixed grapple usage in courses and gamemodes.
Fixed a crash in DataTheft when touching data bank.
Fixed an error on course load.
Fixed collisions issues. (you getting stuck in walls for no reason or player to player damage not going through in gamemodes other than datatheft)
Added Proper Kick Glitch (Like in original ME). (cry about prediction errors l0l)
Tweaked safety roll, now you can roll under things.
Made it possible to dive to your death =).
Added some grapple related stuff. Now it moves with the entity it was attached to and other players can see the rope.

TODO

Gamemodes menu.

beatrun-anims - Decompiled beatrun animations.

Credits <3

All contributors.
MTB - Beatrun Reanimated project.
Discord Rich Presence by Fluffy Servers.
Steam Presence by YuRaNnNzZZ.
earthykiller - He made that piece of shit code.

apui-avignon-university/warren-tdbp

2023-09-10T01:37:31Z

A warren plugin for moodle learning analytics dashboard

HanshengGUO/DistriPerfMonitor

2023-09-11T01:35:47Z

DistriPerfMonitor is a robust and scalable C++ based performance monitoring solution specifically designed for distributed Linux environments. It aims to provide real-time insights and analytics into system performance, resource utilization, and potential bottlenecks across multiple nodes.

utilitywarehouse/kafka-manifests

2023-09-10T01:58:37Z

remonbonbon/makefile-example

2023-09-10T01:58:37Z

Good Makefile for C/C++

DLR-TS/adore_cli

2023-09-10T01:58:37Z

eleev/soft-skills

2023-09-10T01:58:44Z

🍦List of Soft Skills for software engineers/developers.

acdh-oeaw/status

2023-09-10T01:58:44Z

Uptime status of ACDH-CH core services

ME435/MATLAB-Robot-Arm-GUI-Start

2023-09-10T01:37:43Z

Starting code for the MATLAB Robot Arm GUI

meyiao/ImuFusion

2023-09-10T01:58:49Z

EKF IMU Fusion Algorithms

ThePolywellGuy/Matlab-Modeling

2023-09-10T01:58:49Z

Download the MATLAB Code Used to Model WB6!

Mesywang/Motion-Planning-Algorithms

2023-09-10T01:58:49Z

Matlab Implementations of some basic motion planning algorithms, such as A*, RRT, RRT*, Minimum Snap Trajectory Generation, etc..

mesosphere/charts

2023-09-11T01:36:35Z

D2IQ Helm Chart Repository

nickel-lang/organist

2023-09-10T01:38:28Z

An experimental Nix toolkit to use nickel as a language for writing nix packages, shells and more.

Helixo32/DetectHooks

2023-09-11T01:36:49Z

Detect userland hooks placed by AV/EDR

juancarlospaco/nodejs

2023-09-10T01:59:48Z

Alternative StdLib for Nim for NodeJS/JavaScript targets, hijacks NodeJS StdLib for Nim

karaxnim/karax

2023-09-10T01:59:48Z

Karax. Single page applications for Nim.

heraldofsolace/nixos-dots

2023-09-11T01:36:53Z

My NixOS dotfiles

replit/nixmodules

2023-09-10T01:59:52Z

nakst/gf

2023-09-10T01:39:07Z

A GDB frontend for Linux.

gf – A GDB Frontend

Building

Download this project's source.

git clone https://github.com/nakst/gf.git

And compile the application.

# linux
./build.sh

# freebsd
./build_freebsd.sh

# netbsd
./build_netbsd.sh

# openbsd
./build_openbsd.sh

Please read the rest of this file to learn about using and configuring gf. If you're new to GDB, see this article.

If you want to make a debug build of gf, use a command like g++ gf2.cpp -g -lX11 -pthread.

Tips

You can run the application with ./gf2. Any additional command line arguments passed to gf will be forwarded to GDB.
Press Ctrl+Shift+P to synchronize your working directory with GDB after you start your target executable. This is necessary if you open gf in a different directory to the one you compile in.
To view RGBA bitmaps, select the Data tab and then select Add bitmap....
Ctrl+Click a line in the source view to run "until" that line. Shift+Click a line in the source view to skip to it without executing the code in between.
Press Shift+F10 to step out of a block, and press Shift+F11 to step out a function.
Press Tab while entering a watch expression to auto-complete it.
Press / with a watch expression highlighted to change the format specifier. For example, /x switches to hexadecimal view.
Press backtick to enter line inspect mode. This mode evaluates all expressions on the current line.
Use gf2 --rr-replay for replaying a trace recorded by rr. Use Ctrl+Shift+(F5/F10/F11) for reverse continue and step.

You may want to add the following commands to your ~/.gdbinit file:

set breakpoint pending on
set disassembly-flavor intel

Settings

On startup, settings are loaded from ~/.config/gf2_config.ini, followed by .project.gf. This is an INI-style file.

GDB configuration

You can pass additional arguments to GDB in the [gdb] section. For example,

[gdb]
argument=-nx
argument=-ex record

You can also change the location of the GDB executable. For example,

[gdb]
path=/home/a/opt/gdb

You can direct all output from GDB to be sent to the "Log" window, if you have one in your layout string. This will work even if you haven't setup a log pipe. This can be used to view the stderr output from your target dynamically as it is running.

[gdb]
log_all_output=1

You can disable the confirmation dialogs for the kill (F3) and connect (F4) commands.

[gdb]
confirm_command_kill=0
confirm_command_connect=0

You can limit the number of stack frames in the stack window (the default is 50).

[gdb]
backtrace_count_limit=50

Custom keyboard shortcuts

Keyboard shortcuts are placed in the [shortcuts] section. For example,

[shortcuts]
Ctrl+I=print i
Ctrl+Shift+F10=reverse-next
Ctrl+Shift+F11=reverse-step

You can use any standard GDB command, or any of the commands listed in "Special commands" below.

User interface

You can change the font and user interface scaling in the [ui] section. For example,

[ui]
scale=1.5
font_path=/usr/share/fonts/TTF/DejaVuSansMono.ttf
font_size_interface=17
font_size_code=20

To change the font, FreeType must have been available when you compiled gf. You can enable subpixel font rendering by recompiling with extra_flags=-DUI_FREETYPE_SUBPIXEL ./build.sh.

You can also configure the interface layout, with the layout parameter. Use h(position,left,right) to create a horizontal split, v(position,left,right) to create a vertical split, and t(...) to create a tab pane. This value should not contain any whitespace. Please note this value is not validated, so make sure it is formatted correctly!

layout=h(75,v(75,Source,Console),v(50,t(Watch,Breakpoints,Commands,Struct,Exe),t(Stack,Files,Registers,Data,Thread))))

NB: Horizontal and vertical splits must have exactly two children. Instead, you can nest them to create more complex layouts.

You can maximize the window at startup with maximize=1.

You can request for the expressions in the watch window to be saved and restored by setting restore_watch_window=1.

Themes

You can change the theme in the theme section. See https://github.com/nakst/gf/wiki/Themes for a list of examples.

Preset commands

You can create a list of quickly accessible commands, available in the "Commands" tab in the UI. Separate individual commands using a semicolon. Each command in the list is run one after another; to run the final command asynchronously, put a & at the end. For example,

[commands]
Compile=shell gcc -o bin/app src/main.c
Run normal=file bin/app;run&
Run tests=file bin/app;run test_cases.txt&
Set breakpoints=b main;b LoadFile;b AssertionFailure

You can use any standard GDB command, or any of the commands listed in "Special commands" below.

Vim integration

You can change the server name with the server_name key in the vim section. For example,

[vim]
server_name=MyVimServer

Control pipe

You can change the loaded file and line by sending commands to the control pipe.

First, you must set the location of the control pipe. In the [pipe] section of the configuration file, set the control key to the absolute path where you want the control pipe to be.

Then, you can send commands to the pipe. For example,

# Load the specified file (must be a full path).
echo f /home/a/test.c > /home/a/control_pipe.dat

# Go to line 123.
echo l 123 > /home/a/control_pipe.dat

# Send a GDB command.
echo c file myapp > /home/a/control_pipe.dat

This can be used for text editor integration.

Log window

You can show messages send to a pipe using the log window.

First, you must set the location of the log pipe. In the [pipe] section of the configuration file, set the log key to the absolute path where you want the log pipe to be. Next, you must add the "Log" window somewhere in your layout string (see the "User interface" section above). Once configured, you can then send messages to the pipe and they will appear in the log window.

Here is an example of how to send messages to the pipe:

#define LOG(...) do { fprintf(logFile, __VA_ARGS__); fflush(logFile); } while (0)
#define LOG_OPEN(path) logFile = fopen(path, "w")
FILE *logFile;

...

LOG_OPEN("...");
LOG("Hello, world!\n");

Special commands

gf-step

gf-step either steps a single line (step) or single instruction (stepi), depending whether disassembly view is active.

gf-next

gf-next either steps over a single line (next) or single instruction (nexti), depending whether disassembly view is active.

gf-step-out-of-block

gf-step-out-of-block steps out of the current block. That is, it steps to the next line after the first unmatched }, starting from the current line.

gf-restart-gdb

gf-restart-gdb restarts the GDB process immediately. Any state such as loaded symbol files or breakpoints will be lost.

gf-get-pwd

gf-get-pwd asks GDB for the working directory in which the current executable file was compiled. This ensures the source view tries to load files from the correct directory.

gf-switch-to

gf-switch-to <window-name> switches to a specific window. The window names are the same as given in the layout string, as seen in the "User interface" section.

gf-command

gf-command <name> runs the command(s) corresponding to <name> in the [commands] section of your configuration file.

gf-inspect-line

gf-inspect-line toggles inspect line mode. By default, this is bound to the backtick key.

Watch window hooks

You can customize the behaviour of the watch window when displaying specific types using Python. When the watch window wants to display the fields of a value, it will look a hook function at gf_hooks[type_of_value]. The hook function should take two arguments, item and field. If the hook function exists, it will be called in one of two ways:

When the watch window needs a list of the fields in the value, it calls the hook with item set to an opaque handle and field set to None. You should print out a list of all the names of the fields in the value, one on each line. You can print out all the standard fields by calling _gf_fields_recurse(item). When adding custom fields, their names must be enclosed by [].
When the watch window needs to get the value of a specific custom field in the value, it calls the hook with item set to a gdb.Value for the value, and field to the name of the custom field that was added. The hook is not called for standard fields. You should return a gdb.Value that gives the value of the field.

For example, the following hook add a width and height custom field for a rectangle type.

def RectangleHook(item, field):
    if field:
        if field == '[width]':  
            # item['...'] looks up a field in the struct, returned as a gdb.Value
            # int(...) converts the gdb.Value to an int so we can do arithmetic on it
            # gdb.Value(...) converts the result back to a gdb.Value
            return gdb.Value(int(item['right']) - item['left'])
        if field == '[height]': 
            # do something similar for the height
            return gdb.Value(int(item['bottom']) - item['top'])
    else:
        print('[width]')         # add the width custom field
        print('[height]')        # add the height custom field
        _gf_fields_recurse(item) # add the fields actually in the struct

gf_hooks = { 'Rectangle': RectangleHook } # create the hook dictionary

If you want to create a custom dynamic array type, instead of printing field names, print (d_arr) followed by the number of array items. The fields will then be automatically populated in the form of [%d], where %d is the index. For example, given the following structure:

struct MyArray {
	int length;
	float *items;
};

This is the hook definition:

def MyArrayHook(item, field):
	if field: return item['items'][int(field[1:-1])]
	else: print('(d_arr)', int(item['length']))

Templates are removed from the name of the type. For example, Array<int>, Array<char *> and Array<float> would all use the Array hook.

Plugins

There is a simple plugin system. Make a file called plugins.cpp in the source code folder. It will be found automatically, and #included in the compilation of the main translation unit.

gf uses the Luigi UI library. It is documented here: https://github.com/nakst/luigi/blob/main/README.md.

You can register new windows, command and data viewers in a constructor function. For example,

__attribute__((constructor)) 
void MyPluginRegister() {
	interfaceWindows.Add({ 
		"Hello", // The window's name. Used to match it against the UI layout string.
		MyPluginHelloWindowCreate, // The callback to create an instance of the window.
		MyPluginHelloWindowUpdate // The callback to update an instance of the window (called every time the target pauses/steps).
	});

	interfaceDataViewers.Add({ 
		"Add test...", // The label of the button to show in the Data tab.
		MyPluginTestViewerCreate // The callback to create the data viewer.
	});

	interfaceCommands.Add({ 
		"My command", // The label to show in the application menu.
		{ /* UIShortcut */ } 
	});
}

The interface window creation callback is passed the parent UIElement and should return the UIElement it creates.

UIElement *MyPluginHelloWindowCreate(UIElement *parent) {
	UIPanel *panel = UIPanelCreate(parent, UI_PANEL_GRAY | UI_PANEL_EXPAND);
	UILabelCreate(&panel->e, 0, "Hello, world!", -1);
	return &panel->e;
}

The interface window update callback is passed the output of GDB from the most recent step, and the UIElement returned by the creation callback.

void MyPluginHelloWindowUpdate(const char *gdbOutput, UIElement *element) {
	// TODO Update the window.
}

The interface data viewer creation callback should create a MDI child of the data tab as follows:

void MyPluginTestViewerCreate(void *unused) {
	UIMDIChild *window = UIMDIChildCreate(&dataWindow->e, UI_MDI_CHILD_CLOSE_BUTTON, UI_RECT_1(0), "Title", -1);
	// TODO Configure the viewer.
	UIElementRefresh(&dataWindow->e);
}

For communicating with GDB, there are the following functions.

// Evaluate an expression. The result is overwritten between calls!
const char *EvaluateExpression(const char *expression, const char *format = nullptr);

// Send and run a command in GDB. Set `echo` to log the command in the console window. 
// If `synchronous` is set the function will wait for the command to complete before it returns.
void DebuggerSend(const char *string, bool echo, bool synchronous);

There are many examples of how to do these things in windows.cpp.

Contributors

nakst
Philippe Mongeau phmongeau 
Jimmy "Keeba" Lefevre JimmyLefevre 
John Blat johnblat64 
IWouldRatherUsePasteBin
Gavin Beatty gavinbeatty
Michael Stopa StomyPX
Anders Kaare sqaxomonophonen
Arseniy Khvorov khvorov45

Extension pack

All tiers on my Patreon, https://www.patreon.com/nakst, get access to the extension pack.

This currently includes:

Embedded tracing profiler
Memory window
Extended watch expression view (for strings, matrices and base conversion)
Waveform viewer
Full source code for the pack

Make sure you use the latest version of the extension pack with the latest commit of gf, otherwise you'll likely run into compile or runtime errors!

spotify/ios-sdk

2023-09-11T01:37:12Z

Spotify SDK for iOS

Spotify iOS SDK

Overview

The Spotify iOS framework allows your application to interact with the Spotify app running in the background on a user's device. Capabilities include authorizing, getting metadata for the currently playing track and context, as well as issuing playback commands.

Please Note: By using Spotify developer tools you accept our Developer Terms of Use.

The Spotify iOS SDK is a set of lightweight objects that connect with the Spotify app and let you control it while all the heavy lifting of playback is offloaded to the Spotify app itself. The Spotify app takes care of playback, networking, offline caching and OS music integration, leaving you to focus on your user experience. Moving from your app to the Spotify app and vice versa is a streamlined experience where playback and metadata always stay in sync.

Key Features

Filing Bugs

Components

How Do App Remote Calls Work?

Tutorial

Frequently Asked Questions

Key Features

Playback is always in sync with Spotify app
Playback, networking, and caching is all accounted for by the Spotify app
Works offline and online and does not require Web API calls to get metadata for player state
Allows authentication through the Spotify app so users don't have to type in their credentials

Filing Bugs

We love feedback from the developer community, so please feel free to file missing features or bugs over at our issue tracker. Make sure you search existing issues before creating new ones.

Open bug tickets | Open feature requests

Requirements

The Spotify iOS framework requires a deployment target of iOS 9 or higher. The following architectures are supported: armv7, armv7s and arm64 for devices, i386 and x86_64 for the iOS Simulator. Bitcode is also supported.

Components

Models

SPTAppRemoteAlbum
SPTAppRemoteArtist
SPTAppRemoteLibraryState
SPTAppRemotePlaybackRestrictions
SPTAppRemotePlaybackOptions
SPTAppRemotePlayerState
SPTAppRemoteTrack
SPTAppRemoteContentItem
SPTAppRemoteUserCapabilities
SPTAppRemoteImageRepresentable
SPTConfiguration

SPTAppRemote

The main entry point to connect to the Spotify app and retrieve API components. Use this to establish, monitor, and terminate the connection.

SPTAppRemotePlayerAPI

Send playback related commands such as:

Play track by URI
Resume/pause playback
Skip forwards and backwards
Seek to position
Set shuffle on/off
Request player state
Request player context
Subscribe to player state

SPTAppRemoteImagesAPI

Fetch an image for a SPTAppRemoteImageRepresentable

SPTAppRemoteUserAPI

Fetch/subscribe/set user-related data such as:

Fetch and/or subscribe to SPTAppRemoteUserCapabilities
Determine if a user can play songs on demand (Premium vs Free)
Add/remove/check if a song is in a user's library

SPTAppRemoteContentAPI

Fetch recommended content for the user.

How App Remote calls work

When you interact with any of the App Remote APIs you pass in a SPTAppRemoteCallback block that gets invoked with either the expected result item or an NSError if the operation failed. The block is triggered after the command was received by the Spotify app (or if the connection could not be made).

Here is an example using the SPTRemotePlayerAPI to skip a song:

[appRemote.playerAPI skipToNext:^(id  _Nullable result, NSError * _Nullable error) {
    if (error) {
        // Operation failed
    } else {
        // Operation succeeded
    }
}];

Tutorial and Examples

We provide a few sample projects to help you get started with the iOS Framework in the DemoProjects folder. See the Readme in the DemoProjects folder for more information on what each sample does.

Authentication and Authorization

To communicate with the Spotify app your application will need to get a user's permission to control playback first by using built-in authorization for App Remote. To do that you will need to request authorization view when connecting to Spotify. The framework will automatically request the app-remote-control scope and show the auth view if user hasn't agreed to it yet.

Terms of Use

Note that by using Spotify developer tools, you accept our Developer Terms of Use.

Included Open Source Libraries

MPMessagePack

Tutorial

This tutorial leads you step-by-step through the creation of a simple app that uses the Spotify iOS SDK to play an audio track and subscribe to player state. It will walk through the authorization flow.

Prepare Your Environment

Follow these steps to make sure you are prepared to start coding.

Download the Spotify iOS framework from the "Clone or download" button at the top of this page, and unzip it.
Install the latest version of Spotify from the App Store onto the device you will be using for development. Run the Spotify app and login or sign up. Note: A Spotify Premium account will be required to play a track on-demand for a uri.
Register Your Application. You will need to register your application at My Applications and obtain a client ID. When you register your app you will also need to whitelist a redirect URI that the Spotify app will use to callback to your app after authorization.

Add Dependencies

Add the SpotifyiOS package to your project. You can either do this through Swift Package Manager (SPM), or by adding SpotifyiOS.framework or SpotifyiOS.xcframework to your Xcode project directly.
In your info.plist add your redirect URI you registered at My Applications. You will need to add your redirect URI under "URL types" and "URL Schemes". Be sure to set a unique "URL identifier" as well. More info on URL Schemes
Add #import <SpotifyiOS/SpotifyiOS.h> to your source files to import necessary headers.

Check if Spotify is Active

If a user is already using Spotify, but has not authorized your application, you can use the following check to prompt them to start the authorization process.

[SPTAppRemote checkIfSpotifyAppIsActive:^(BOOL active) {
    if (active) {
        // Prompt the user to connect Spotify here
    }
}];

Authorize Your Application

To be able to use the playback control part of the SDK the user needs to authorize your application. If they haven't, the connection will fail with a No token provided error. To allow the user to authorize your app, you can use the built-in authorization flow.

Initialize SPTConfiguration with your client ID and redirect URI.

SPTConfiguration *configuration =
    [[SPTConfiguration alloc] initWithClientID:@"your_client_id" redirectURL:[NSURL urlWithString:@"your_redirect_uri"]];

Initialize SPTAppRemote with your SPTConfiguration

self.appRemote = [[SPTAppRemote alloc] initWithConfiguration:configuration logLevel:SPTAppRemoteLogLevelDebug];

Initiate the authentication flow (for other ways to detect if Spotify is installed, as well as attributing installs, please see our Content Linking Guide).

// Note: A blank string will play the user's last song or pick a random one.
BOOL spotifyInstalled = [self.appRemote authorizeAndPlayURI:@"spotify:track:69bp2EbF7Q2rqc5N3ylezZ"];
if (!spotifyInstalled) {
    /*
    * The Spotify app is not installed.
    * Use SKStoreProductViewController with [SPTAppRemote spotifyItunesItemIdentifier] to present the user
    * with a way to install the Spotify app.
    */
}

Configure your AppDelegate to parse out the accessToken in application:openURL:options: and set it on the SPTAppRemote connectionParameters.

- (BOOL)application:(UIApplication *)app openURL:(NSURL *)url options:(NSDictionary<UIApplicationOpenURLOptionsKey,id> *)options
{
    NSDictionary *params = [self.appRemote authorizationParametersFromURL:url];
    NSString *token = params[SPTAppRemoteAccessTokenKey];
    if (token) {
        self.appRemote.connectionParameters.accessToken = token;
    } else if (params[SPTAppRemoteErrorDescriptionKey]) {
        NSLog(@"%@", params[SPTAppRemoteErrorDescriptionKey]);
    }
    return YES;
}

If you are using UIScene then you need to use appropriate method in your scene delegate.

func scene(_ scene: UIScene, openURLContexts URLContexts: Set<UIOpenURLContext>) {
    guard let url = URLContexts.first?.url else {
        return
    }

    let parameters = appRemote.authorizationParameters(from: url);

    if let access_token = parameters?[SPTAppRemoteAccessTokenKey] {
        appRemote.connectionParameters.accessToken = access_token
        self.accessToken = access_token
    } else if let error_description = parameters?[SPTAppRemoteErrorDescriptionKey] {
        // Show the error
    }
}

Connect and Subscribe to Player State

Set your connection delegate and attempt to connect.

self.appRemote.delegate = self;
[self.appRemote connect];


- (void)appRemoteDidEstablishConnection:(SPTAppRemote *)appRemote
{
    // Connection was successful, you can begin issuing commands
}

- (void)appRemote:(SPTAppRemote *)appRemote didFailConnectionAttemptWithError:(NSError *)error
{
    // Connection failed
}

- (void)appRemote:(SPTAppRemote *)appRemote didDisconnectWithError:(nullable NSError *)error
{
    // Connection disconnected
}

Set a delegate and subscribe to player state:

appRemote.playerAPI.delegate = self;

[appRemote.playerAPI subscribeToPlayerState:^(id  _Nullable result, NSError * _Nullable error) {
    // Handle Errors
}];

- (void)playerStateDidChange:(id<SPTAppRemotePlayerState>)playerState
{
    NSLog(@"Track name: %@", playerState.track.name);
}

Connection handling

As a courtesy you should always disconnect App Remote when your app enters a background state. This tells Spotify that it's safe to disable the active stream. If your app does not properly call disconnect Spotify has no way of knowing that it should not maintain the connection, and this may result in future connection issues.

If you want your app to automatically reconnect after disruption events like incoming calls or Siri interactions you may use the willResignActive and didBecomeActive callbacks to safely disconnect and reconnect. If you don't wish to reconnect directly, it's typically enough to close the connection in didEnterBackground callbacks.

- (void)applicationWillResignActive:(UIApplication *)application
{
    [self.appRemote disconnect];
}
- (void)applicationDidBecomeActive:(UIApplication *)application
{
    [self.appRemote connect];
}
// If you're using UIWindowSceneDelegate
- (void)sceneDidBecomeActive:(UIScene *)scene
{
    [self.appRemote connect];
}
- (void)sceneWillResignActive:(UIScene *)scene
{
    [self.appRemote disconnect];
}

Frequently Asked Questions

Why does music need to be playing to connect with SPTAppRemote?

Music must be playing when you connect with SPTAppRemote to ensure the Spotify app is not suspended in the background. iOS applications can only stay active in the background for a few seconds unless they are actively doing something like navigation or playing music.

Is SpotifyiOS.framework thread safe?

No, the framework currently expects to be called from the main thread. It will offload most of its work to a background thread internally but callbacks to your code will also occur on the main thread.

What if I need to authorize without starting playback?

There is an alternative authorization method. You can find more information about that here.

LIFX/LIFXKit

2023-09-11T01:37:12Z

The LIFX SDK for Objective-C.

LIFX

LIFX will no longer be supporting LIFXKit. This has been done in order to better support developers by focusing instead on releasing documentation for the LAN protocol itself and enabling SDKs to be produced in any language. We have also published a HTTP API cloud-based light control from anywhere.

Our Developer Zone is available if you have any questions about using the newly released documentation. You can find it at https://community.lifx.com/

We sincerely hope that someone decides to fork this repository and continues maintaining it. If you do this please feel free to let us know on the Developer Zone or by email at developers@lifx.com. We will do our best to support your efforts.

We hope that our new commitment to support public documentation allows better quality libraries than we have the resources to support alone.

romaonthego/RESideMenu

2023-09-10T02:00:13Z

iOS 7/8 style side menu with parallax effect.

RESideMenu

iOS 7/8 style side menu with parallax effect inspired by Dribbble shots (first and second).

Since version 4.0 you can add menu view controllers on both left and right sides of your content view controller.

Requirements

Xcode 6 or higher
Apple LLVM compiler
iOS 6.0 or higher
ARC

Demo

Build and run the RESideMenuExample project in Xcode to see RESideMenu in action. For storyboards integration demo, build and run RESideMenuStoryboardsExample.

Installation

CocoaPods

The recommended approach for installating RESideMenu is via the CocoaPods package manager, as it provides flexible dependency management and dead simple installation. For best results, it is recommended that you install via CocoaPods >= 0.28.0 using Git >= 1.8.0 installed via Homebrew.

Install CocoaPods if not already available:

$ [sudo] gem install cocoapods
$ pod setup

Change to the directory of your Xcode project:

$ cd /path/to/MyProject
$ touch Podfile
$ edit Podfile

Edit your Podfile and add RESideMenu:

platform :ios, '6.0'
pod 'RESideMenu', '~> 4.0.7'

Install into your Xcode project:

$ pod install

Open your project in Xcode from the .xcworkspace file (not the usual project file)

$ open MyProject.xcworkspace

Please note that if your installation fails, it may be because you are installing with a version of Git lower than CocoaPods is expecting. Please ensure that you are running Git >= 1.8.0 by executing git --version. You can get a full picture of the installation details by executing pod install --verbose.

Manual Install

All you need to do is drop RESideMenu files into your project, and add #include "RESideMenu.h" to the top of classes that will use it.

Example Usage

In your AppDelegate's - (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions create the view controller and assign content and menu view controllers.

// Create content and menu controllers
//
DEMONavigationController *navigationController = [[DEMONavigationController alloc] initWithRootViewController:[[DEMOHomeViewController alloc] init]];
DEMOLeftMenuViewController *leftMenuViewController = [[DEMOLeftMenuViewController alloc] init];
DEMORightMenuViewController *rightMenuViewController = [[DEMORightMenuViewController alloc] init];

// Create side menu controller
//
RESideMenu *sideMenuViewController = [[RESideMenu alloc] initWithContentViewController:navigationController
                                                                leftMenuViewController:leftMenuViewController
                                                               rightMenuViewController:rightMenuViewController];
sideMenuViewController.backgroundImage = [UIImage imageNamed:@"Stars"];

// Make it a root controller
//
self.window.rootViewController = sideMenuViewController;

Present the menu view controller:

[self.sideMenuViewController presentLeftMenuViewController];

[self.sideMenuViewController presentRightMenuViewController];

Switch content view controllers:

#import <RESideMenu/RESideMenu.h>

....

[self.sideMenuViewController setContentViewController:viewController animated:YES];
[self.sideMenuViewController hideMenuViewController];

Storyboards Example

Create a subclass of RESideMenu. In this example we call it DEMORootViewController.
In the Storyboard designate the root view's owner as DEMORootViewController.
Make sure to #import "RESideMenu.h" in DEMORootViewController.h.
Add more view controllers to your Storyboard, and give them identifiers "leftMenuViewController", "rightMenuViewController" and "contentViewController". Note that in the new XCode the identifier is called "Storyboard ID" and can be found in the Identity inspector.
Add a method awakeFromNib to DEMORootViewController.m with the following code:

- (void)awakeFromNib
{
    self.contentViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"contentViewController"];
    self.leftMenuViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"leftMenuViewController"];
    self.rightMenuViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"rightMenuViewController"];
}

Customization

You can customize the following properties of RESideMenu:

@property (assign, readwrite, nonatomic) NSTimeInterval animationDuration;
@property (strong, readwrite, nonatomic) UIImage *backgroundImage;
@property (assign, readwrite, nonatomic) BOOL panGestureEnabled;
@property (assign, readwrite, nonatomic) BOOL panFromEdge;
@property (assign, readwrite, nonatomic) NSUInteger panMinimumOpenThreshold;
@property (assign, readwrite, nonatomic) BOOL interactivePopGestureRecognizerEnabled;
@property (assign, readwrite, nonatomic) BOOL scaleContentView;
@property (assign, readwrite, nonatomic) BOOL scaleBackgroundImageView;
@property (assign, readwrite, nonatomic) BOOL scaleMenuView;
@property (assign, readwrite, nonatomic) BOOL contentViewShadowEnabled;
@property (assign, readwrite, nonatomic) UIColor *contentViewShadowColor;
@property (assign, readwrite, nonatomic) CGSize contentViewShadowOffset;
@property (assign, readwrite, nonatomic) CGFloat contentViewShadowOpacity;
@property (assign, readwrite, nonatomic) CGFloat contentViewShadowRadius;
@property (assign, readwrite, nonatomic) CGFloat contentViewScaleValue;
@property (assign, readwrite, nonatomic) CGFloat contentViewInLandscapeOffsetCenterX;
@property (assign, readwrite, nonatomic) CGFloat contentViewInPortraitOffsetCenterX;
@property (assign, readwrite, nonatomic) CGFloat parallaxMenuMinimumRelativeValue;
@property (assign, readwrite, nonatomic) CGFloat parallaxMenuMaximumRelativeValue;
@property (assign, readwrite, nonatomic) CGFloat parallaxContentMinimumRelativeValue;
@property (assign, readwrite, nonatomic) CGFloat parallaxContentMaximumRelativeValue;
@property (assign, readwrite, nonatomic) CGAffineTransform menuViewControllerTransformation;
@property (assign, readwrite, nonatomic) BOOL parallaxEnabled;
@property (assign, readwrite, nonatomic) BOOL bouncesHorizontally;
@property (assign, readwrite, nonatomic) UIStatusBarStyle menuPreferredStatusBarStyle;
@property (assign, readwrite, nonatomic) BOOL menuPrefersStatusBarHidden;

If you set a backgroundImage, don't forget to set the Menu View Controller's background color to clear color.

You can implement RESideMenuDelegate protocol to receive the following messages:

- (void)sideMenu:(RESideMenu *)sideMenu didRecognizePanGesture:(UIPanGestureRecognizer *)recognizer;
- (void)sideMenu:(RESideMenu *)sideMenu willShowMenuViewController:(UIViewController *)menuViewController;
- (void)sideMenu:(RESideMenu *)sideMenu didShowMenuViewController:(UIViewController *)menuViewController;
- (void)sideMenu:(RESideMenu *)sideMenu willHideMenuViewController:(UIViewController *)menuViewController;
- (void)sideMenu:(RESideMenu *)sideMenu didHideMenuViewController:(UIViewController *)menuViewController;

Contact

Roman Efimov

License

RESideMenu is available under the MIT license.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

between40and2/JXFanfouEngine

2023-09-10T02:00:13Z

Objective-C framework to access Fanfou API

Hello https://github.com/between40and2/JXFanfouEngine

google/EarlGrey

2023-09-10T02:00:13Z

🍵 iOS UI Automation Test Framework

Deprecation: EarlGrey 1.0 is deprecated in favor of EarlGrey 2.0 which integrates it with XCUITest. Please look at the earlgrey2 branch. EarlGrey 1.0 is not being maintained internally with iOS 13.

EarlGrey

For EarlGrey 2, please go to earlgrey2 branch.

EarlGrey is a native iOS UI automation test framework that enables you to write clear, concise tests.

With the EarlGrey framework, you have access to enhanced synchronization features. EarlGrey automatically synchronizes with the UI, network requests, and various queues; but still allows you to manually implement customized timings, if needed.

EarlGrey’s synchronization features help to ensure that the UI is in a steady state before actions are performed. This greatly increases test stability and makes tests highly repeatable.

EarlGrey works in conjunction with the XCTest framework and integrates with Xcode’s Test Navigator so you can run tests directly from Xcode or the command line (using xcodebuild).

Getting Started

The EarlGrey documentation for users is located in the EarlGrey/docs folder. To get started, review the EarlGrey features, check for backward compatibility, and then install/run EarlGrey with your test target. After everything is configured, take a look at the EarlGrey API and start writing your own tests.

Getting Help

If you need help, several resources are available. First check the FAQ. If you have more questions after reading the FAQ, see Known Issues. You can bring more specific issues to our attention by asking them on stackoverflow.com using the #earlgrey tag. You can also start new discussions with us on our Google group or request to join our slack channel.

Analytics

To prioritize and improve EarlGrey, the framework collects usage data and uploads it to Google Analytics. More specifically, the framework collects the MD5 hash of Bundle ID, Test Class Names and Test Method Names. This information allows us to measure the volume of usage. For more detailed information about our analytics collection, please peruse the GREYAnalytics.m file which contains the implementation details. If they wish, users can choose to opt out by disabling the Analytics config setting in their test’s - (void)setUp method:

In Objective-C:

// Disable analytics.
[[GREYConfiguration sharedInstance] setValue:@(NO) forConfigKey:kGREYConfigKeyAnalyticsEnabled];

In Swift:

// Disable analytics.
GREYConfiguration.sharedInstance().setValue(false, forConfigKey: kGREYConfigKeyAnalyticsEnabled)

For Contributors

Please make sure you’ve followed the guidelines in CONTRIBUTING.md before making any contributions.

Setup an EarlGrey Project

Clone the EarlGrey repository from GitHub:

git clone https://github.com/google/EarlGrey.git

After you have cloned the EarlGrey repository, download all the dependencies using setup-earlgrey.sh.
After the script completes successfully, open EarlGrey.xcodeproj and ensure that all the targets build.
You can now use EarlGrey.xcodeproj to make changes to the framework.

Add and Run Tests

Unit Tests

To add unit tests for EarlGrey, use UnitTests.xcodeproj located at Tests/UnitTests. To run all unit tests, select the UnitTests Scheme and press Cmd+U.

Functional Tests

To add functional tests for EarlGrey, use the FunctionalTests.xcodeproj located at Tests/FunctionalTests. To run all functional tests, select the FunctionalTests Scheme and press Cmd+U.

BinaryAnalysisPlatform/FrontC

2023-09-10T02:00:31Z

An OCaml library providing a C parser and lexer

Rachnog/Deep-Trading

2023-09-10T02:00:42Z

Algorithmic trading with deep learning experiments

digao-dalpiaz/DzHTMLText

2023-09-10T02:01:04Z

Delphi and Lazarus HTML Label component

oov/aviutl_rampreview

2023-09-10T02:01:04Z

AviUtl用プラグイン「拡張編集RAMプレビュー」

WouterVanNifterick/C-To-Delphi

2023-09-10T02:01:04Z

C To Delphi converter

IanMulvany/connotea-public

2023-09-10T01:40:00Z

public snapshot of current connotea code under GIT revision control

NAME Connotea Code

This program is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation; either version 2 of the License, or (at your
option) any later version.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
Public License for more details.

You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

Some portions regarding RDF are originally from RDF::Core, derived from
works Copyright (C) 2001 Ginger Alliance Ltd., and carry their own
copyright and GPL notices.

NAMING You will the see the names Connotea, Bibliotech and Connotea Code used. To eliminate any confusion, we'll clarify the meaning of those names here.

Connotea is the name of the online reference management service created
and run by Nature Publishing Group (NPG). Bibliotech was the initial
project name used at NPG while the service was being developed, and
hence this name is used for some class and variable names in the code.
The release of the underlying technology for Connotea is known as
Connotea Code. The purpose of this page and the SourceForge project is
to make the code that runs this site publicly available for review and
re-use.

Therefore, it makes sense to refer to Connotea the service, or to the
Connotea Code. However, Connotea is a trademark of Nature Publishing
Group, so if you use the code to create your own bookmarking service, we
ask that you don't brand it as Connotea. We also ask that you include
the following footer on your site:

  This site is powered by
  <a href="http://sf.net/projects/connotea">Connotea Code</a>,
  the open source software behind
  <a href="http://www.connotea.org/">Connotea</a>.

The Connotea logo, the site guide and related documentation, other image
files and stylesheets are copyrighted by NPG and are not released under
the GPL.

ABOUT THE CODE Connotea Code runs a social bookmarking web site for users to save and share links, which can have citation data automatically retrieved from authoritative sources.

Connotea Code is written in Perl, and uses MySQL as the data store. It
runs as a mod_perl handler in Apache2, and uses templates for page
presentation.

DOWNLOAD Download the tarball from the connotea SourceForge project area at http://sf.net/projects/connotea.

The current stable release is version 1.8.

UPGRADING NEW FEATURES FROM 1.7.1 TO 1.8 * Web API in regular use. * Template Toolkit based templates in regular use. * More optimized SQL queries for common requests. * Greater use of transactions in MySQL. * Greater flexbility for citation source modules. * New citation source modules, plus improvements to existing modules. * Blog component to create news page from external blog. * Wiki component to create custom wiki. * Admin component with user search. * Integration with Bibutils library for BibTeX and MODS output. * Antispam system with captcha and quarantine responses. * Click tracker for all posts. * Alpha-version proxy module system to handle known proxied post URL's. * Alpha-version stand-alone citation server capability. * Additional tools such as command-line post by API, user recovery, and test suite launch. * Automated deployment scripts, now supporting Darcs instead of CVS. * Updated code to support newer versions of CPAN modules. * More test suite scripts.

NEW FEATURES FROM 1.5.0 TO 1.7.1 * Many bugs fixed. * Alpha-version Web API. * Alpha-version Template Toolkit based output framework. * Full text searching feature. * Better cache control and throttling. * Better bookmarklets. * Better URI validation. * Better XML encoding for fringe cases. * Better character set decoding of downloaded documents for citations. * Exception email notification. * More support for two instances on same server. * More support for split web/database servers for one instance. * More comprehensive User Agent support for citation modules. * Method to switch from one citation module to another. * Optimized SQL for counting totals and some other operations. * Added methods for profiling code and dumping SQL statements. * Loosen some grammar restrictions, e.g. ok to name a tag "tag". * Tighten some grammar restrictions, e.g. num & start must be numeric. * Better RIS import based on real-world file examples. * New citation modules:

    * Blackwell
    * PMC
    * Wiley
    * ePrints

* Several optional administrator utilities:

    * retro: script to update citation data rectroactively.
    * bibwatch: load monitoring utility.
    * bibpreempt: preempting and testing utility.
    * resendreg: utility to resend registration details.
    * deluser: utility to delete users.
    * memcache_wrapper: init.d script to keep memcached running.

* Several developer testing utilities:

    * import: test import modules
    * citation_source_test.pl: test citation modules
    * get_test_urls.pl: retrieve URLs from Yahoo for testing citation
    modules
    * htmlise.pl: convert citation module test results to clickable HTML
    for review in a browser

UPGRADING FROM 1.7.1 TO 1.8 See sql/schema_alter.sql for commands to patch your database. Other elements of the upgrade should be optional; that is, you can turn them on later.

UPGRADING FROM 1.5.0 TO 1.7.1 The biggest difference between 1.5.0 and 1.7.1 is that 1.7.1 uses two databases at once.

In order to support fulltext matching, a new feature, we use a MyISAM
database in MySQL with FULLTEXT keys (see
<http://dev.mysql.com/doc/refman/4.1/en/fulltext-search.html>).

However, InnoDB is still faster for JOIN's and offers referential
integrity, so as a compromise we run two databases and keep them
synchronized with MySQL replication (see
<http://dev.mysql.com/doc/refman/4.1/en/replication.html>).

If you are upgrading from Connotea Code 1.5.0, please see the section
below on database setup for the secondary search database. To upgrade,
you will need to:

* Create a mysql dump that does not mention schema, just data, as in:
     $ mysqldump -c -t -u bibliotech -p bibliotech > /tmp/dump

* Create the MyISAM search database as described below.
* Setup replication and restart MySQL as described below.
* Run sql/wipe.sql to remove all data from your database:
     $ echo 'source sql/wipe.sql' | mysql -u bibliotech -p bibliotech

* Reimport your dump back into your main InnoDB database, from where it
will flow to the search database because of replication:
     $ echo 'source /tmp/dump' | mysql -u bibliotech -p bibliotech

Except for the addition of a MyISAM database, there are no intradatabase
schema changes between 1.5.0 and 1.7.1.

UPGRADING FROM VERSIONS PRIOR TO 1.5.0 TO 1.7.1 To upgrade from versions prior to 1.5.0, please edit sql/schema_alter.sql to contain only the statements necessary to alter the database schema from your version to the current schema. There are no schema changes between 1.5.0 and 1.7.1.

 $ $EDITOR schema_alter.sql
 $ echo 'source schema_alter.sql' | mysql -u root -p bibliotech

Then follow the directions above for upgrading from 1.5.0.

ACQUIRING SOURCE FOR SPECIALIZED PROGRAMMING CREATING A CITATION MODULE Connotea's ability to import bibliographic information from third-party websites is enabled by a series of plug-ins.

If you downloaded this source code with the intent of creating a
citation module, see the comments and code in the file
Bibliotech/CitationSource.pm which will explain the base class from
which your citation source module should be derived.

In previous releases testing your citation module required a full
instance of Connotea Code. In this release, a script named
test_util/citation_source_test.pl provides a way to test your module's
return values without an instance. Your module file should be placed in
the Bibliotech/CitationSource directory to be recognized by this script.

You may also test by creating a fully installed instance, which gives
the added benefit of letting you test via a web browser and ensure that
citation data is saved properly in MySQL.

If you create a new citation plug-in, please consider releasing it back
to the Connotea community.

CREATING AN IMPORT MODULE Connotea's ability to import a batch of links or references depends on a series of plug-ins.

If you downloaded this source code with the intent of creating an import
module, see the comments and code in the file Bibliotech/Import.pm which
will explain the base class from which your import module should be
derived.

In previous releases testing your citation module required a full
instance of Connotea Code. In this release, a script named
test_util/import provides a way to test your module's return values
without an instance. Your module file should be placed in the
Bibliotech/Import directory to be recognized by this script.

You may also test by creating a fully installed instance, which gives
the added benefit of letting you test via a web browser and ensure that
imported data is saved properly in MySQL.

If you create a new import plug-in, please consider releasing it back to
the Connotea community.

CREATING A PROXY MODULE Connotea's ability to provide proxy translation for specific types of URI's depends on a series of plug-ins.

If you downloaded this source code with the intent of creating a proxy
module, see the comments and code in the file Bibliotech/Proxy.pm which
will explain the base class from which your import module should be
derived.

You may test by creating a fully installed instance.

If you create a new proxy plug-in, please consider releasing it back to
the Connotea community.

ADDING A STATIC WEB PAGE Any Connotea Code instance that contains the "Inc" component has the ability to deliver static pages through the template system. A URL path that is not recognized by "Bibliotech::Parser" will be tested as a filename under the document root with an extension of ".inc" appended. The contents of this file should be XHTML. If found, the contents will be served within inc.tt or default.tt according to the rules of the template system.

ADDING A DYNAMIC WEB PAGE To create a new component for your Connotea Code instance that serves dynamic web content requires at least the following:

In Bibliotech/Parser.pm you must find the grammar definition and add a
subrule to the page production which will designate the URL path that
will activate your component. Keep in mind that a path name that is a
shortened version of another path name will always eclipse the longer
one if it appears first, so you should add it after (e.g. "urilabel"
must come before "uri" or "uri" would always match for either).

In Bibliotech/Page/Standard.pm add a package based on "Bibliotech::Page"
like the others defined in that file. The name should be
"Bibliotech::Page::x" where "x" is your path name with a single capital
letter at the beginning even if it is more than one word (e.g.
"Bibliotech::Page::Reportspam" for a path of "/reportspam"). Include a
"main_component()" method that returns a string of the last part of the
class name of the main component, a "Bibliotech::Component"-derived
class (e.g. 'ReportSpam' for "Bibliotech::Component::ReportSpam").

In the Bibliotech/Component directory create a module based on
Bibliotech::Component. Use the others that appear in that diectory as
examples and refer directly to the source code in
Bibliotech/Component.pm, particularly the comments, for descriptions of
expected methods and their expected return values. For an HTML
compontent be sure to include "last_updated_basis()" and
"html_content()". In particular, html_content() should return a
"Bibliotech::Page::HTML_Content" object; that class is defined in
Bibliotech/Page.pm.

SPEAKING TO THE WEB API FROM YOUR APPLICATION The Connotea Web API allows communication with an instance, either the Connotea web site at http://www.connotea.org/ or your own private instance, using a predefined set of commands to access structured data and accomplish normal user actions in a programmatic manner.

Your software may be written in any language you choose - the basic
requirements are the ability to create and parse XML and communicate
using the HTTP protocol. The ability to interpret the XML as RDF and use
object orientation to model the objects serialized as RDF may prove
helpful. Libraries and sample code are available.

See <http://www.connotea.org/wiki/WebAPI> for Web API documentation.

MINIMUM REQUIREMENTS This code requires, or has been best tested on:

* Linux/UNIX operating system (tested on Red Hat Enterprise Linux 4 -
see <http://www.redhat.com/>)
* Perl 5.8.0 (see <http://www.perl.org/>)
* Perl CPAN modules as identified on the list below (see
<http://cpan.perl.org/>)
* Apache 2.0.40 (see <http://httpd.apache.org/>)
* MySQL 5.0.17 (see <http://www.mysql.com/>)
* Memcached 1.1.12 (see <http://www.danga.com/memcached/>)

CPAN You will need to have the following modules installed from CPAN.

On all Perl systems you can type:

  $ LANG=C cpan

...or...

  $ LANG=C perl -MCPAN -e shell

...to get a CPAN shell prompt, and then type:

  cpan> install XXX::YYY

..or...

  cpan> force install XXX::YYY

...to install a module.

The "LANG=C" portion of the command line above is highly recommended as
many modern Linux distributions set your default "LANG" to a
locale-based setting and this often interferes with Perl module
compilations. When it does, the error messages will be very misleading
and never mention the "LANG" variable.

Before you embark on what will probably be a long install-fest, it is
also recommended that you type:

 cpan> install Bundle::CPAN

...inside the CPAN shell and then restart it. This will ensure that you
are using the latest version of the CPAN code. Some things will go more
smoothly.

When asked whether to follow dependencies, answer yes. When asked about
optional utilities and scripts that can be installed to /bin or
/usr/bin, answer however you like, as none are necessary for this code.

You do not necessarily need the latest version of every module, although
in one or two cases you do. In general, if your Perl is at least 5.8.0,
just install the version that a non-force install will give you at the
CPAN prompt. If you are lower than 5.8.0, upgrade your base Perl
installation first.

The list:

On Red Hat and some other distros, the following are provided in vendor
packages, and you're better off using those.

* Apache2
* Apache::Const
* Apache::File

...but install these from CPAN so you get new versions:

* IO::String (for Bio::Biblio::IO, better to preinstall)
* XML::Writer (for Bio::Biblio::IO, better to preinstall)
* XML::Twig (for Bio::Biblio::IO, better to preinstall)
* SOAP::Lite (for Bio::Biblio::IO, better to preinstall)
* Pod::Man (for DateTime, better to preinstall)
* Bio::Biblio (may need to be forced)
* Cache::Memcached
* CGI
* Class::DBI
* Config::Scoped
* Data::Dumper (not just for debugging, actually used in production)
* Date::Parse
* DateTime (you may need to force installation of DateTime::Set if your
timezone is not UTC)
* DateTime::Format::ISO8601
* DateTime::Format::MySQL
* DateTime::Incomplete
* Digest::MD5
* Encode (you may need to force installation of Encode if some
non-English tests fail)
* Fcntl
* File::Temp
* File::Touch
* FindBin
* HTML::Entities
* HTML::Sanitizer (you may need to force installation of HTML::Sanitizer
due to some year-old bugs already filed on CPAN)
* HTTP::OAI
* IO::File
* JSON
* LWP::UserAgent
* List::MoreUtils
* List::Util (you may need to force installation of List::Util unless
you have a very new version of Perl)
* Net::Daemon::Log (you may need to force installation of
Net::Daemon::Log for failing a fork test - not used by us)
* Netscape::Bookmarks
* Parse::RecDescent
* RDF::Core
* SQL::Abstract
* Set::Array
* Storable
* Template
* Test::Exception
* Time::HiRes
* URI
* URI::Escape
* URI::Heuristic
* URI::OpenURL
* URI::QueryParam
* Want
* Wiki::Toolkit
* Wiki::Toolkit::Plugin::Diff
* XML::Element
* XML::Feed
* XML::LibXML
* XML::RSS
* YAML (you may need to force installation of Test::Simple which is a
dependency of YAML, for an unknown reason)
* Apache::Emulator (not required for core web service service)
* Text::BibTeX (not required for core web service service)

SETUP MYSQL Two databases for user posts need to be created. See sql/schema.sql for the database schema which needs to be created in MySQL. The first database will be created using InnoDB tables to enforce foreign keys and constraints and for table joining speed. A second database then should be created with a _search suffix using MyISAM tables that have FULLTEXT indexes which are queried when searching for words. (FULLTEXT indexes are not available for InnoDB yet.)

The second schema is generated from the first by running:

  $ cd sql
  $ perl mkschema_search < schema.sql > schema_search.sql

MySQL relication can be used to make the MyISAM database a slave of the
InnoDB database, even on the same machine. This is a suggested
configuration for /etc/my.cnf that will do just that:

  [mysqld]
  # local replication of bibliotech to bibliotech_search:
  server-id=1
  log-bin=mysql-bin
  binlog-do-db=bibliotech
  replicate-same-server-id=1
  replicate-rewrite-db=bibliotech->bibliotech_search
  replicate-do-db=bibliotech_search
  master-host=localhost
  master-user=search_repl
  master-password=pass
  # change stopwords in support of bibliotech freematch feature:
  #ft_stopword_file=/etc/mysql_stopwords.txt
  ft_min_word_len=2
  ft_max_word_len=255
  # allow packing of queries
  group_concat_max_len=8192

Change the master-password line! Also change the database names if you
are not using "bibliotech".

You will probably find the MySQL stopwords to be too restrictive in
practice. The list can be viewed at
<http://dev.mysql.com/doc/mysql/en/fulltext-stopwords.html>. We
recommend that you pare down this list to a more suitable one, and use
the ft_stopword_file keyword to tell MySQL to use your list instead.

In any case, if you want the search feature to behave predictably, you
must specify an external text file stopword list to MySQL. The search
handler will query MySQL to find out the stopword list file being used,
and read it as well, so it can anticipate MySQL reporting no matches for
words that otherwise should match.

You'll need to execute a grant statement similar to this one:

 GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO
 search_repl@'localhost.localdomain' IDENTIFIED BY 'pass';

Two notes on the replication grant statement:

* MySQL seems to consider "localhost.localdomain" different from
"localhost" and while the shorter version normally works, for
replication it seems that the longer one is needed. If you have
problems, try both.
* You must have the updated privilege table structure. If you have had
MySQL installed since the 3.x series, your mysql.user table lacks the
privilege fields mentioned above; check your docs about a script called
'mysql_fix_privilege_tables'. On many systems this will be a shell
script in /usr/bin that you can run as root with a "--password=xxx"
parameter (to specify the MySQL root user password, not the Unix root
user password).

The MySQL username used by the Perl handler must have access to both
databases (username and password as in /etc/bibliotech.conf):

  GRANT SELECT, INSERT, UPDATE, DELETE ON bibliotech.* TO
  user@localhost IDENTIFIED BY 'secret';
  GRANT SELECT, INSERT, UPDATE, DELETE ON bibliotech_search.* TO
  user@localhost IDENTIFIED BY 'secret';

WIKI::TOOLKIT You also need to setup Wiki::Toolkit so that a wiki is available. This is required. You should create a blank database, grant a user rights to it, and run the provided setup script.

  CREATE DATABASE conwiki;
  GRANT ALL ON conwiki.* TO conwiki@localhost IDENTIFIED BY 'secret';

  $ /usr/bin/wiki-toolkit-setupdb --type mysql \
                                  --name conwiki \
                                  --user conwiki \
                                  --pass secret \
                                  --host localhost

Remember to populate the "COMPONENT WIKI" block of your configuration
file with the wiki database details.

APACHE Everything under the site/default subdirectory should be placed or linked into an Apache-accessible location, and a location handler should be added to httpd.conf (or elsewhere in the Apache configuration) such as the following one.

Update the values to match your IP, domain, and file paths:

  <VirtualHost 1.2.3.4:80>
    ServerName www.yourdomain.com
    ServerAlias yourdomain.com
    ServerAdmin you@yourdomain.com
    DocumentRoot /var/www/perl/connotea_code/site/default
    PerlOptions +Parent
    PerlSwitches -I/var/www/perl/connotea_code
    PerlModule Bibliotech::Apache
    PerlModule Bibliotech::AuthCookie
    <Location />
      SetHandler perl-script
      PerlHandler Bibliotech::Apache
      PerlAuthenHandler Bibliotech::AuthCookie::authen_handler
      AuthName Bibliotech
      AuthType basic
      require valid-user
      #ErrorDocument 503 /paused.html
      #ErrorDocument 503 /readonly.html
      ErrorDocument 503 /unavailable.html
    </Location>
  </VirtualHost>

The 503 lines allow a custom page to be displayed when your site is
under heavy load (unavailable.html) or when you deliberately pause
service (paused.html) or make it read-only (readonly.html); you must
edit your Apache configuration and switch which line is commented for
the latter two modes.

MEMCACHED Memcached is required, and the code is written to assume that a memcache is running. Database timestamps, cached HTML, and uploaded files are all stored temporarily in this cache.

CONFIGURATION See config for a configuration that should be copied to /etc/bibliotech.conf and edited to suit your needs. Particularly, be sure to change *_SECRET and *_PASSWORD variables.

Default configuration:

  (((config)))

CUSTOMIZATION The look and feel of your Connotea Code installation can be modified by creating a new stylesheet and new templates. The template system is Template Toolkit documented at the web site at http://www.template-toolkit.org/. We refer to this system as TT for short.

TEMPLATE LOCATION Templates are located by default in site/default. This is controlled by options in the configuration. It is recommended that templates have a .tt extension.

TEMPLATE SELECTION The template used to service a particular request is determined by the page requested and the available template filenames.

Individual templates can be defined for individual pages; for example,
to override the template for the add form, create a template called
add.tt.

For general bookmark listing queries (e.g. "/tag/tagname"), templates
beginning with recent can be used. recent.tt will be used for queries
with no user or tag parameters - recent_user.tt, recent_tag.tt and
recent_user_tag.tt can be created to specify the behaviour is there is a
user query, a tag query, or both respectively.

Unless overridden by a specific template, default.tt is used.

TEMPLATE EXAMPLES Templates should not contain the full HTML for the page you want to construct, but only that which should appear between the "" and "" tags.

This is an example default.tt:

  [% prepare_component_begin() %]
  [% prepare_component('main',undef,'main,verbose') %]
  [% prepare_component_end() %]
  <html>
  <head>
  <title>[% main_title %]</title>
  [% rss_link %]
  [% component_javascript_block_if_needed %]
  </head>
  <body[% component_javascript_onload_if_needed %]>
  [% component_html('main',undef,'main,verbose') %]
  </body>
  </html>

The syntax is from Template Toolkit documented at the web site at
<http://www.template-toolkit.org/>. We refer to TT for short.

A Connotea web page is a series of components that are combined
together, contributing HTML which can be organized in separately-placed
parts calculated at once, or as one block, and also sometimes Javascript
to be placed in a "<script>" block in the "<head>" or in the "<body>"
"onload" attribute. The components are controlled by the template
selected to represent the HTTP query, and each component running can
access the current command, the posts that result from the SQL engine
processing the query of the command, and a variety of support services.

Several functions are provided to TT by the calling instance.
"prepare_component_begin()" is called before anything happens, and this
prepares some internal data structures; then "prepare_component()" is
called with the base name of the component (e.g. "Blah" corresponds to
"Bibliotech::Component::Blah") or the special word "main" which does a
lookup for the main component of a page described in
Bibliotech/Parser.pm and Bibliotech/Page/Standard.pm, which allows some
templates to be reused. The second argument is a comma-separated list of
parts, which is a mechanism used by some components such as
"ListOfTags", and the third argument is a comma-separated list of
options to the components, of which the universal ones are "main" and
"verbose" which should be set to true for the main component (the format
is key=value but ommiting the value is the same as =1 which is true).
Later in the template, a call to "component_html()" with the same
arguments inserts the HTML (if you specified multiple part names in the
prepare call, you should have multiple calls to insert HTML each with
one part mentioned).

Individual templates can be defined for individual requests. For
example, to override the template for the add form, create a template
called add.tt. For general bookmark listing queries (e.g.
"/tag/tagname"), templates beginning with recent can be used. recent.tt
will be used for queries with no user or tag parameters -
recent_user.tt, recent_tag.tt and recent_user_tag.tt can be created to
specify the behaviour is there is a user query, a tag query, or both
respectively.

A more realistic example for default.tt would make calls to a
normalprep.tt and a normal.tt wrapper:

  [% prepare_component_begin() %]
  [% INCLUDE normalprep.tt %]
  [% prepare_component('main',undef,'main,verbose') %]
  [% prepare_component_end() %]
  [% WRAPPER normal.tt %]
  [% component_html('main',undef,'main,verbose') %]
  [% END %]

In this case, normal.tt would contain a basic look for the web site that
can be used by many other templates.

A set of working templates is furnished with this distribution.

TEMPLATES AND COMPONENTS A Connotea web page is a series of components that are combined together, contributing HTML which can be organized in separately-placed parts calculated at once, or as one block, and also sometimes Javascript to be placed in a "<script>" block in the "" or in the "" "onload" attribute. The components are controlled by the template selected to represent the HTTP query, and each component running can access the current command, the posts that result from the SQL engine processing the query of the command, and a variety of support services.

Several functions are provided to TT by the calling instance. Some
functions are general, and are available in all templates, even small
snippet templates used by components (conventionally named with a comp
prefix). Some functions are page-level utilities that largely control
component insertion.

TEMPLATE PAGE LEVEL FUNCTIONS * "prepare_component_begin()" Declare the beginning of "prepare_component()" calls.

* "prepare_component(module, parts, options)"
    Prepare a component.

    * module
        The base name of the desired component (e.g. "Blah" corresponds
        to "Bibliotech::Component::Blah") or the special word "main"
        which does a lookup for the main component of a page described
        in Bibliotech/Parser.pm and Bibliotech/Page/Standard.pm, which
        allows some templates to be reused.

    * parts
        For most components, use "undef" for one block of HTML output.
        For components that can return multiple parts of HTML, this
        option is a comma-separated list of part names to prepare in one
        calculation for efficiency. Components with parts: "ListOfTags",
        "ListOfUsers", and "ListOfGangs".

    * options
        A comma-separated list of options to the component, of which the
        universal ones are "main" and "verbose" which should be set to
        true for the main component (the format is key=value but
        ommiting the value is the same as =1 which is true).

* "prepare_component_end()"
    Declare the end of "prepare_component()" calls.

* "component_html(module, part, options)"
    The arguments are the same as "prepare_component()", except that the
    part should be at most one part, not more than one.

* "component_javascript_onload()"
    Insert the Javascript addressed at the "onload" handler.

* "component_javascript_onload_if_needed"
    Insert the Javascript addressed at the "onload" handler, but wrap it
    a space followed by the actual attribute itself, as in, "
    onload="blah"", or if there is no Javascript, insert nothing.

* "component_javascript_block()"
    Insert the Javascript addressed at the "head" of the HTML document.

* "component_javascript_block_if_needed()"
    Insert the Javascript addressed at the "head" of the HTML document,
    but wrap it in a "<script>" block, or if there is no Javascript,
    insert nothing.

* "main_title"
    The HTML document title recommended by the main component, or
    failing that, a default constructed from the site name and page
    name.

* "main_heading"
    The HTML document heading ("H1") recommended by the main component.

* "main_description"
    The description recommended by the main component; used in RSS, etc.

* "css_link"
    Insert a "<link>" representing the CSS files dictated by
    "HOME_CSS_FILE" for the home page or "GLOBAL_CSS_FILE" otherwise
    (configuration options).

* "rss_link"
    Insert a "<link>" representing the RSS format output for the
    currently viewed page.

TEMPLATE GENERAL FUNCTIONS * "location" Base URL for the web site, which can be directly prepended to page names, as in:

      <a href="[% location %]news">

* "sitename"
    The name of the web site as defined in the configuration.

* "siteemail"
    The email address of the administrator of the web site as defined in
    the configuration.

* "user"
    User object of the current user looking at the web page, e.g.:

      [% IF user %][% user.username %][% ELSE %]Visitor[% END %]

* "is_browser_safari", "is_browser_firefox", "is_browser_ie",
"is_browser_other"
    Can be used in an IF test - true if the user's browser is the type
    indicated.

* "browser_redirect(url)"
    Immediately abort and issue a Location header to a new URL. The URL
    can be relative to the root of the web site in which case "location"
    is prepended.

* "is_virgin"
    Can be used in an IF test - true if the user is a first-time
    visitor.

* "canonical_uri"
    Canonical URI for the current page.

* "canonical_location"
    Canonical URI using "location".

* "object_location"
    Canonical URI using "location" and setting format to HTML.

* "no_num(url)"
    Remove the num=x parameter from a URL.

* "encode_xml_utf8(str)"
    Escape ampersand, less- and greater-than symbols, normalize HTML
    entities to XML entities, and remove unusual control characters.

* "encode_xhtml_utf8(str)"
    Encode characters as XML entities where needed and remove unusual
    control characters.

* "now"
    The current date and time, as a "Bibliotech::DateTime" object, e.g.:

      [% now.label %]
      [% now.ymd %]
      [% now.ymdhm %]
      [% now.iso8601 %]
      [% now.iso8601_utc %]

* "time"
    The current date and time, as a Unix timestamp.

* "join(joinstr, ...)"
    Perl's join command.

* "speech_join(jointype, ...)"
    Join several elements as in speech. Argument "jointype" is "and" or
    "or". This function will combine with commas, spaces, and the
    jointype operator (if at least three items), e.g.:

      speech_join('and', 'bob') -> 'bob'
      speech_join('and', 'bob', 'alice') -> 'bob and alice'
      speech_join('and', 'bob', 'alice', 'tom') -> 'bob, alice, and tom'

* "plural(amount, singular, plural, no_space)"
    Join a number with the appropriate singular or plural noun, e.g.:

      plural(6, 'second', 'seconds') -> '6 seconds'
      plural(1, 'second', 'seconds') -> '1 second'

* "commas(num)"
    Decorate a number with commas every three digits (thousands) per the
    American style, e.g.:

      commas(5000000) -> '5,000,000'

* "divide(a, b, places, multiplier)"
    Divide two numbers, but avoid a division by zero error by returning
    zero. Return a number formatted to the number of decimals indicated
    in "places" (default 1 if omitted), and multiplied by "multiplier"
    (default 1 if omitted), e.g.:

      divide(10, 0) -> 0
      divide(10, 2) -> 5
      divide(10, 4, 2) -> 2.50
      divide(10, 4, 2, 100) -> 250

* "percent(a, b, places)"
    Same as "divide" but multiplier is 100 and a percent sign is
    appended.

      percent(1, 2) -> 50.0%
      percent(4, 100) -> 4.0%

* "bookmarklets"
    Insert all the bookmarklets.

* "bookmarklet(page, popup)"
    Insert a bookmarklet. Argument "page" should be "add", "addcomment",
    or "comments". Argument "popup" should be "direct" or "popup".

* "bookmarklet_js(page, popup)"
    Same as "bookmarklet" but only insert the Javascript.

* "user_in_own_library"
    Can be used in an IF test - true if the user is looking at a page
    that has a current filter of "/user" with their username.

* "user_in_another_library"
    Can be used in an IF test - true if the user is looking at a page
    that has a current filter of "/user" with a username other than
    their own.

* "click_counter_onclick(url, new_window)"
    Insert a hyperlink run through the click counter, to the URL
    provided, optionally in a new window if "new_window" is true.

TEMPLATE COMPONENT SNIPPET FUNCTIONS * "sticky(parameter)" Used primarily inside the "value" attributes of HTML "ipnut" tags, this function allows the form to remember values between refreshes, so if a message must be displayed to the user causing the form to be redisplayed, the user's form responses are not lost.

* "has_validation_error"
    True if the form is redisplayed with an active error.

* "has_validation_error_for(field)"
    True if the form is redisplayed with an active error concerning the
    field specified. Note that some errors are not tied to a field.

* "validation_error_field"
    If the form is redisplayed with an active error, the field name that
    gave rise to the error. Note that some errors are not tied to a
    field.

* "validation_error"
    If the form is redisplayed with an active error, the error message.

LOGROTATE A log file will be created at the place specified in "LOG_FILE" in the configuration. This file can grow quickly so you may like to configure logrotate to deal with it on a weekly basis. This is the contents of a suggested /etc/logrotate.d/bibliotech file:

  /var/log/bibliotech.log {
          create 644 apache root
          notifempty
          weekly
          rotate 5
          compress
      postrotate
          /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
      endscript
  }

SUPPORT To subscribe to the Connotea Code development mailing list, go to https://lists.sourceforge.net/lists/listinfo/connotea-code-devel.

This list is for the discussion of the core code and citation and import
plug-ins. It is intended for use by people who are installing their own
instance of Connotea Code, or who are reviewing the code to see how
Connotea handles their data, or who would like to help enable the
importing of bibliographic information from more sources.

There is a separate list, connotea-discuss, for discussion of Connotea
itself - i.e. for discussion of <http://www.connotea.org/>. That list is
the appropriate place for questions about how to use the site, or
requests or suggestions for new features.

WEBCITE The Connotea Code has a module called WebCite which can be employed separately as a simple web service providing citation information using the Connotea Code citation modules.

A WebCite instance requires the full codebase to be present, as well as
the CPAN module dependencies, and a compatible version of Apache, just
as you might setup for Connotea Code proper. However, MySQL, Memcache,
and Wiki::Toolkit are not required.

You should create a configuration file as directed for Connotea Code,
but only the WEBCITE section and sections for your active citation
modules are required.

WebCite provides caching not with memcached but via the filesystem, so
the results survive Apache restarts.

WebCite can be turned on inside a normal Connotea Code deployment, but
will perform its duties separately from the main codebase.

The main page for WebCite simply presents a form with two fields:

* "uri" - text field for the URL
* "fmt" - select field for the format:

    * "ris"
    * "mods"
    * "json"

A submit button is provided for human users but the value is ignored.

The form can be called by programs by performing an HTTP POST to the
installed location with "uri" and "fmt" parameters.

Programs should evaluate the HTTP status code first.

* 200 - citation data returned
* 404 - no citation data found
* 500 - internal error

In the case of a 404 code, a brief message such as "No citation" may be
displayed, but programs should not expect this value, and should rely
exclusively on the HTTP status code to determine this condition.

In the case of a 200 code, the "Content-Type" header should be
appropriate for the format requested, and the data payload should
present the citation data in the format requested.

A 500 code will occur if an exception is thrown retrieving the citation
data, and the data payload will be plain text giving the error message.

All transactions are separate. There is no concept of sessions.

There are no authentication checks in the WebCite code, although the
system administrator is free to add restrictions in the Apache
configuration.

Subsequent requests within 90 days for the same URL will return data
cached from the original request. Data is cached in an internal
structure form so the same cache entry can produce all output formats.

The Apache configuration block for WebCite is as follows:

  PerlSwitches -I/var/www/perl/...
  <Location /bibliotech>
    SetHandler perl-script
    PerlHandler Bibliotech::WebCite
  </Location>

The PerlSwitches line should point to the directory that contains the
Bibliotech directory.

This may or may not appear in the same Apache configuration as Connotea
Code proper.

A suggested configuration for /etc/bibliotech.conf is as follows:

  WEBCITE {
    CACHE_ENABLED = true
    CACHE_PATH = '/var/cache/webcite'
    CACHE_TIMEOUT = 7776000
    LOG_ENABLED = true
    LOG_FILE = '/var/log/webcite.log'
  }

Again this may be a file exclusively for WebCite or a file with
intermixed configuration for Connotea Code proper.

You should create the cache directory and log file and give the Apache
user write access before starting Apache.

ACKNOWLEDGMENTS The look, structure, documentation and source code of http://www.connotea.org/ are the collective work of Martin Flack, Ben Lund, Timo Hannay, Joanna Scott, Stefania Bojano, Grant Farrelly, Euan Adie, and Ian Mulvany. The vast majority of the programming was done by Martin Flack of NeoReality, Inc., http://www.neoreality.com/.

The materials available from <http://sf.net/projects/connotea> are
released under the GNU General Public License; reuse of all other
materials requires the express written permission of Nature Publishing
Group.

MORE INFORMATION Please visit this URL for more information: http://www.connotea.org/code

If you have questions, email us or try our mailing lists:
<http://www.connotea.org/contact>

jautz/kraus

2023-09-11T01:37:58Z

Determines the "floor of the day" on which my co-workers and me meet for coffee.

K.R.A.U.S.

Abstract

Determines the "floor of the day" on which my co-workers and me meet for coffee.

The acronym stands for the German phrase Kaffee-Runde auf unterschiedlichen Stockwerken.

Runtime Environment

To run this script you need Perl5 with modules CGI and AppConfig. CGI.pm has been included in the Perl distribution since Perl 5.4. Thus, on Debian derivates, this will prepare the runtime environment:

apt-get install perl libappconfig-perl

Usually you will run this script inside a webserver like Apache with mod_perl.

Debugging

You can run this script directly from the command line using the CGI module's debugging feature (see perldoc CGI section DEBUGGING for details):

./index.cgi format=raw offset=2

Finally, to find out whether the pseudo-random location selector distributes equally there is a simulation mode that prints the selections for the next days to stdout:

./index.cgi -s [default=30]

For example, this produces the number of selections for each floor in the next 500 days:

./index.cgi -s 500 | awk '{print $2}' | sort | uniq -c
 56 1
 56 2
 60 3
 69 4
 69 5
 68 6
 59 7
 64 8

Well, the distribution is far from perfect but at least this script yields reproducible results without a central storage.

theory/text-markup

2023-09-11T01:37:58Z

Parse files in many different text markup formats into HTML.

Text/Markup version 0.31

This library's module, Text::Markup, provides an single interface for parsing a large number of text markup formats and converting them to HTML. It currently supports the following markups:

Installation

To install this module, type the following:

perl Build.PL
./Build
./Build test
./Build install

Or, if you don't have Module::Build installed, type the following:

perl Makefile.PL
make
make test
make install

Dependencies

Text-Markup requires the following modules:

File::BOM 0.15
HTML::Entities
perl 5.8.1
Pod::Simple::XHTML 3.15
Text::Markdown 1.000004
Text::MediawikiFormat 1.0
Text::Textile 2.10
Text::Trac 0.10
Parse::BBCode 0.15
Text::WikiCreole 0.07

Copyright and Licence

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

sbwml/halflife-list

2023-09-10T02:01:11Z

ABP/ublock 广告过滤规则（每周一早上9点更新）

ABP/ublock订阅规则

ad-pc.txt：[推荐桌面端]合并自乘风视频广告过滤规则、Easylist、EasylistChina、EasyPrivacy、CJX'sAnnoyance，以及补充的一些规则；
```
https://cdn.jsdelivr.net/gh/sbwml/halflife-list@master/ad-pc.txt
```
ad-mo.txt：合并自Easylist、EasylistChina、EasyPrivacy、CJX'sAnnoyance；
```
https://cdn.jsdelivr.net/gh/sbwml/halflife-list@master/ad-mo.txt
```
ad.txt：[推荐移动端]合并自乘风视频广告过滤规则、EasylistChina、EasylistLite、CJX'sAnnoyance，以及补充的一些规则；
```
https://cdn.jsdelivr.net/gh/sbwml/halflife-list@master/ad.txt
```
ad2.txt：合并自乘风视频广告过滤规则、EasylistChina、EasylistLite、CJX'sAnnoyance；
```
https://cdn.jsdelivr.net/gh/sbwml/halflife-list@master/ad2.txt
```
ad3.txt：合并自乘风视频广告过滤规则、EasylistChina、EasylistLite、CJX'sAnnoyance、EasyPrivacy；
```
https://cdn.jsdelivr.net/gh/sbwml/halflife-list@master/ad3.txt
```
ad-edentw.txt：合并自Adblock Warning Removal List、ABP filters、anti-adblock-killer-filters。
```
https://cdn.jsdelivr.net/gh/sbwml/halflife-list@master/ad-edentw.txt
```

rocky/Perl-Devel-Trepan-Shell

2023-09-10T02:01:11Z

Adds an Interactive Shell to Devel::Trepan

Devel::Trepan::Shell -- interactive shell support for Devel::Trepan and more

An interactive shell command for Devel::Trepan.

Motivation: Devel::Trepan is getting quite large and adding a shell via Devel::REPL pulls in lots of other packages. Thus we have this separated this portion.

SYNOPSIS

This adds a "shell" command with alias "re.pl" to the Devel::Trepan debugger, trepan.pl. The command goes into a Devel::REPL shell from inside the debugger.

But wait, there's more!

This package also contains some Devel::REPL plugins for entering both the Devel::Trepan debugger and the tried-and-true perl5db debugger, from a re.pl shell:

To call the debuggers inside re.pl, first run or put in your ~/.re.pl/rc.pl file:

$_REPL->load_plugin('Trepan');         # to go into the trepan debugger
$_REPL->load_plugin('Perl5db');        # to go into the perl5db debugger

And then in your re.pl session:

%trepan Perl-expression-or-statement    # enter Devel::Trepan debugger
%perl5db Perl-expression-or-statement   # enter Perl5db

INSTALLATION

To install Devel::Trepan::Shell, run the following commands:

perl Build.PL
make
make test
[sudo] make install

or:

$ perl -MCPAN -e shell
...
cpan[1]> install Devel::Trepan::Shell

LICENSE AND COPYRIGHT

This program is distributed WITHOUT ANY WARRANTY, including but not limited to the implied warranties of merchantability or fitness for a particular purpose.

The program is free software. You may distribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation (either version 2 or any later version) and the Perl Artistic License as published by O’Reilly Media, Inc. Please open the files named gpl-2.0.txt and Artistic for a copy of these licenses.

kvishnivetsky/NetgearProSafeUtils

2023-09-10T02:01:11Z

Firmware update and configuration utilities for NETGEAR ProSafe switched

NETGEAR ProSafePlus Utils

Package includes:

Firmware update utility
Configuration read utility All for NETGEAR ProSafePlus switches.

Contact info:

URL: www.vishnivetsky.ru Moblie Phone: +7 913 77507588 SkyPE: @kvishnivetsky TG: @kvishnivetsky

LinkStackOrg/LinkStack

2023-09-10T01:40:23Z

LinkStack - the ultimate solution for creating a personalized & professional profile page. Showcase all your important links in one place, forget the limitation of one link on social media. Set up your personal site on your own server with just a few clicks.

Open-Source Linktree Alternative

LinkStack is a highly customizable link sharing platform
with an intuitive, easy to use user interface.

Function • About • Instances • Themes • Installation • Docker Version • Updating • Discord • Fork structure • License • Supporters • Special thanks • Additional credit

Function

LinkStack: The Ultimate Link Management Solution

LinkStack is a unique platform that provides an efficient solution for managing and sharing links online. Our platform offers a website similar to Linktree, which allows users to overcome the limitation of only being able to add one link on social media platforms.

With LinkStack, users can easily link to their own custom page and provide their followers with access to all the links they need in one convenient location. What sets LinkStack apart from other link management platforms is its flexibility, which allows users to host their links on their own web server or web hosting provider. This provides users with complete control over their online presence and ensures that their links are easily accessible.

Additionally, LinkStack allows other users to register and create their own links, making it an ideal solution for businesses and organizations looking to manage multiple links. With our user-friendly Admin Panel, managing and accessing other users' links is easy.

About

With LinkStack, our mission is to provide users with a free and privacy-focused solution for managing and sharing links online. We believe that everyone should have access to a customizable link-sharing platform without sacrificing their privacy and control over their data.

To achieve this mission, we offer a self-hosted option for users who want complete control over their data without having it sold to third-party companies. Our platform can be easily implemented through a simple drag and drop process, eliminating the need for complex terminal commands or source code manipulation.

For those who may not have the technical expertise to self-host, we also offer free instances of our platform while still prioritizing their privacy. Our platform offers many of the same features and options as commercial link-sharing platforms while maintaining the values of privacy and autonomy.

Our goal is to provide a free version of a link-sharing service, similar to Linktree, while empowering users to take ownership of their data. We will never sell user data and believe in providing a trustworthy and transparent solution for managing and sharing links online.

Instances

Find the right instance for you

Our community instance program provides users with the opportunity to register on hosted instances and use Linkstack for free. Members of our community have generously provided their resources to host instances, allowing us to expand the reach of Linkstack and give back to the community

Themes

Custom Themes

Customize the look of your LinkStack instance with themes. Themes allow you to change the look and feel of your site with a few clicks. Users can submit themes they created for everyone to download and use.

Contribute by designing your own themes. You can read more about contributing below.

You can find all available Themes on here: linkstack.org/themes

How to add themes

How to add themes to your LinkStack instance

You can add your downloaded themes to your LinkStack instance on the Admin Panel. Navigate to the 'Themes' tab and scroll to the bottom of the page.

Now click on Choose file and select your downloaded theme zip file. Now click on 'Upload theme' and you should be able to select your uploaded theme now.

Themes are envisioned to be made by users for users.

If you know a bit about CSS, consider making your own theme and adding it to the public directory. Everything is documented in the dedicated GitHub repository.

github.com/LinkStackOrg/linkstack-themes/tree/main/contributing

Installation

Downloading and installing steps:

Download the latest release of LinkStack and simply place the folder 'littlelink' or the contents of this folder in the root directory of your website.

That's it! No coding no command line setup just plug and play.

Go through the first setup page:

When accessing your instance for the first time, you will be greeted by the first setup page.

Docker

The official docker version of LinkStack. This docker image is a simple to set up solution, containing everything you need to run LinkStack.

The docker version of LinkStack retains all the features and customization options of the original version.

This docker is based on Alpine Linux, a Linux distribution designed to be small, simple and secure. The web server is running Apache2, a free and open-source cross-platform web server software. The docker comes with PHP 8.0 for high compatibility and performance.

Using the docker is as simple as pulling and deploying.

Pull

docker pull linkstackorg/linkstack

Learn more about the Docker version

Updating

When a new version is released, you will get an update notification on your Admin Panel.

Automatic one click Updater

This updater allows you to update your installation with just one click.

How to use the Automatic Updater:

To update your instance, click on the update notification on your Admin Panel.
Click on “Update automatically” and the updater will take care of the rest.

You can still download updates manually. New versions will are still uploaded to the GitHub repository as usual.

Before updating, the updater will create a backup. Your instance won’t save more than two backups at a time. You can download these updates from the created folder: backups\updater-backups.

If you switched your database to MySQL, your database will not be included in the backup.

Discord

## License

As of version 4.0.0, the license for this project has been updated to the GNU Affero General Public License v3.0, which explicitly requires that any modifications made to the project must be made public. This license also requires that a copyright notice and license notice be included in any copies or derivative works of the project.

Additionally, any changes made to the project must be clearly stated, and the source code for the modified version must be made available to anyone who receives the modified version. Network use of the project is also considered distribution, and as such, any network use of the project must comply with the terms of the license.

Finally, any derivative works of the project must be licensed under the same license terms as the original project.

Supporters

You can support LinkStack here.

💖 Thank you:

Contributors

Thank you for improving LinkStack!

Beta Testers

Thank you for all your efforts!

Become a beta tester

Stargazers

Additional-credit

ramonrietdijk/livewire-tables

2023-09-10T01:40:23Z

Monitor your user and all activity on your application

Laravel User Monitoring

Introduction

Welcome to the world of enhanced user monitoring with the groundbreaking Laravel User Monitoring package! Developed by the brilliant minds at Binafy, this innovative open-source solution is designed to empower Laravel developers and website administrators with invaluable insights into user activities.

Tracking user behavior and interactions is now made effortless, allowing you to gain a deeper understanding of your users' engagement, preferences, and pain points. With its seamless integration into Laravel projects, this package opens up a realm of possibilities, enabling you to optimize user experiences, detect bottlenecks, and make data-driven decisions for your platform's success.

Experience real-time monitoring like never before, as you access comprehensive analytics and visualize user interactions with ease. Rest assured, your users' data is handled securely, respecting privacy while giving you the tools to improve your application's performance and user satisfaction.

Whether you are building a new project or looking to enhance an existing one, "Laravel User Monitoring" is the missing piece to elevate your web applications to new heights. So, why wait? Dive into the world of intelligent user monitoring and witness the transformation of your Laravel-powered application today!

Installation

You can install the package with Composer.

composer require binafy/laravel-user-monitoring

Publish

If you want to publish a config file you can use this command:

php artisan vendor:publish --tag="laravel-user-monitoring-config"

If you want to publish migration files you can use this command:

php artisan vendor:publish --tag="laravel-user-monitoring-migrations"

For convenience, you can use this command to publish config and migration files:

php artisan vendor:publish --provider="Binafy\LaravelUserMonitoring\Providers\LaravelUserMonitoringServiceProvider"

After publishing, run the php artisan migrate command.

Usage

The Laravel-User-Monitoring, just need to use middleware, traits, etc ... and it's not hard, enjoy :)

User Configuration

You can config your user with the user-monitoring.php configuration file:

'user' => [
    /*
     * User model.
     */
    'model' => 'App\Models\User',

    /*
     * Foreign Key column name.
     */
    'foreign_key' => 'user_id',

    /*
     * Users table name.
     */
    'table' => 'users',

    /*
     * The correct guard.
     */
    'guard' => 'web',

    /*
     * If you are using uuid or ulid you can change it for the type of foreign_key.
     *
     * When you are using ulid or uuid, you need to add related traits into the models.
     */
    'foreign_key_type' => 'id', // uuid, ulid, id
],

model: If your user model exists in another place, you can change it to the correct namespace.
foreign_key: You can set the user foreign_key name, like customer_id.
table: You can write your users table name if is not `users.
guard: The correct guard that using for the user.

Foreign Key Type (UUID, ULID, ID)

If you are using uuid or ulid, you can change foreign_key_type to your correct foreign key type:

'user' => [
    ...

    /*
     * If you are using uuid or ulid you can change it for the type of foreign_key.
     *
     * When you are using ulid or uuid, you need to add related traits into the models.
     */
    'foreign_key_type' => 'uuid', // uuid, ulid, id
],

NOTE: You must write uuid or ulid or id.

Visit Monitoring

When you want to monitor all views of your application, you must follow below:

Publish the Migrations
Use VisitMonitoringMiddleware in Kernel.php, you can go to the App/Http folder and open the Kernel.php file and add VisitMonitoringMiddleware into your middleware for example:

protected $middlewareGroups = [
    'web' => [
        ...
        \Binafy\LaravelUserMonitoring\Middlewares\VisitMonitoringMiddleware::class,
    ],

    'api' => [
        // \Laravel\Sanctum\Http\Middleware\EnsureFrontendRequestsAreStateful::class,
        \Illuminate\Routing\Middleware\ThrottleRequests::class.':api',
        \Illuminate\Routing\Middleware\SubstituteBindings::class,
    ],
];

After, you can see all pages monitoring :)

If you want to disable monitoring for specific pages you can go to user-monitoring.php that exists in the config folder and add pages into the visit_monitoring key:

'visit_monitoring' => [
    /*
     * You can specify pages not to be monitored.
     */
    'expect_pages' => [
        'home',
        'admin/dashboard',
    ],
],

Delete Visit Monitoring Records By Specific Days

You may delete records by specific days, Laravel-User-Monitoring also supports this 🤩.

First, you need to go to the user-monitoring config file and highlight the days that you want to delete:

'visit_monitoring' => [
    ...

    /*
     * If you want to delete visit rows after some days, you can change this to 360,
     * but if you don't like to delete rows you can change it to 0.
     *
     * For this feature you need Task-Scheduling => https://laravel.com/docs/10.x/scheduling
     */
    'delete_days' => 10,
],

After, you need to use Task Scheduling to fire-related command, so go to app/Console/Kernel.php and do like this:

<?php

namespace App\Console;

...
use Binafy\LaravelUserMonitoring\Commands\RemoveVisitMonitoringRecordsCommand;

class Kernel extends ConsoleKernel
{
    /**
     * Define the application's command schedule.
     */
    protected function schedule(Schedule $schedule): void
    {
         $schedule->command(RemoveVisitMonitoringRecordsCommand::class)->hourly();
    }
}

You can change hourly to minute or second, for more information you can read Schedule Frequency Options.

Turn ON-OFF

Maybe you want to turn off visit monitoring for somedays or always, you can use configuration to turn it off:

'visit_monitoring' => [
    ...

    /*
     * If you want to disable visit monitoring, you can change it to false.
     */
    'turn_on' => true,
    
    ...
]

Visit Monitoring Views

Laravel-User-Monitoring also has an amazing views that you can use it very easy, just need to go to /user-monitoring/visits-monitoring url, and enjoy:

Action Monitoring

If you want to monitor your models actions, you can use the Actionable trait in your model:

<?php

namespace App\Models;

use Binafy\LaravelUserMonitoring\Traits\Actionable;
use Illuminate\Database\Eloquent\Model;

class Product extends Model
{
    use Actionable;
}

Now when a product is read, created, updated, or deleted, you can see which users doing that.

If you want to disable some actions like created, you can use the config file:

'action_monitoring' => [
    ...
    
    /*
     * Monitor actions.
     *
     * You can set true/false for monitor actions like (store, update, and ...).
     */
    'on_store'      => false,
    'on_update'     => true,
    'on_destroy'    => true,
    'on_read'       => true,
    'on_restore'    => false,
    'on_replicate'  => false,
],

Action Monitoring Views

Laravel-User-Monitoring also has amazing views that you can use very easily, just need to go to the /user-monitoring/actions-monitoring URL, and enjoy:

Authentication Monitoring

Have you ever thought about monitoring the entry and exit of users of your application? Now you can :)
If you want to monitor users when logging in or logout of your application, you need to migrate the migrations to the config file and change true for monitoring authentication.

'authentication_monitoring' => [
    ...

    /*
     * You can set true/false for monitor login or logout. 
     */
    'on_login' => true,
    'on_logout' => true,
],

Authentication Monitoring Views

Laravel-User-Monitoring also has amazing views that you can use very easily, just need to go to the /user-monitoring/authentications-monitoring URL, and enjoy:

How to use in big projects

If you want to use Laravel-User-Monitoring is big projects, you have lots of ways, but I want to give some tips and ideas to help you:

If you have an idea for this section you can create PRs or issues to help us.

You can use this package with Cache
You can make a separate DB and connect to your project to separate monitoring and application.

Contributors

Thanks to all the people who contributed. Contributors.

Security

If you discover any security-related issues, please email binafy23@gmail.com instead of using the issue tracker.

Changelog

The changelog can be found in the CHANGELOG.md file of the GitHub repository. It lists the changes, bug fixes, and improvements made to each version of the Laravel User Monitoring package.

License

The MIT License (MIT). Please see License File for more information.

Conclusion

Congratulations! You have successfully installed and integrated the Laravel User Monitoring package into your Laravel application. By effectively logging and analyzing user activity, you can gain valuable insights that can help you improve your application's user experience and performance. If you have any questions or need further assistance, feel free to refer to the documentation or seek help from the package's GitHub repository. Happy monitoring!

xdebug/xdebug

2023-09-10T02:01:34Z

Xdebug — Step Debugger and Debugging Aid for PHP

Xdebug

.. image:: https://github.com/xdebug/xdebug/workflows/Build/badge.svg :target: https://github.com/xdebug/xdebug/actions?query=workflow%3ABuild .. image:: https://ci.appveyor.com/api/projects/status/glp9xfsmt1p25nkn?svg=true :target: https://ci.appveyor.com/project/derickr/xdebug .. image:: https://circleci.com/gh/xdebug/xdebug/tree/master.svg?style=svg :target: https://circleci.com/gh/xdebug/xdebug

.. image:: https://raw.githubusercontent.com/vshymanskyy/StandWithUkraine/main/badges/StandWithUkraine.svg :target: https://stand-with-ukraine.pp.ua/

Xdebug is a debugging tool for PHP. It provides step-debugging and a whole range of development helpers, such as stack traces, a code profiler, features to dump the full execution of your script to a file, and more.

|Repography logo| / Recent activity |Time period|

.. |Time period| image:: https://images.repography.com/0/xdebug/xdebug/recent-activity/ef1290ac5bfa674f07dcfa4f915ce6b3_badge.svg :alt: Time period :target: https://repography.com .. |Repography logo| image:: https://images.repography.com/logo.svg :alt: Repography logo :target: https://repography.com .. |recent-activity_timeline| image:: https://images.repography.com/0/xdebug/xdebug/recent-activity/ef1290ac5bfa674f07dcfa4f915ce6b3_timeline.svg :alt: Timeline graph :target: https://github.com/xdebug/xdebug/commits .. |recent-activity_prs| image:: https://images.repography.com/0/xdebug/xdebug/recent-activity/ef1290ac5bfa674f07dcfa4f915ce6b3_prs.svg :alt: Pull request status graph :target: https://github.com/xdebug/xdebug/pulls .. |recent-activity_words| image:: https://images.repography.com/0/xdebug/xdebug/recent-activity/ef1290ac5bfa674f07dcfa4f915ce6b3_words.svg :alt: Trending topics :target: https://github.com/xdebug/xdebug/commits .. |recent-activity_users| image:: https://images.repography.com/0/xdebug/xdebug/recent-activity/ef1290ac5bfa674f07dcfa4f915ce6b3_users.svg :alt: Top contributors :target: https://github.com/xdebug/xdebug/graphs/contributors

Requirements

Xdebug requires a supported version <https://www.php.net/supported-versions.php>_ of PHP. For installation it requires the pecl tool (available through the php-pear package), unless your Linux distribution has an Xdebug package (php-xdebug).

Installation

On most Linux distributions you can install Xdebug through its package manager. You can also compile from source with the pecl tool through pecl install xdebug. The latter also works for MacOS as long as PHP is installed with Homebrew.

On Windows, you need to download <https://xdebug.org/download#releases>_ a binary. Use the Wizard <https://xdebug.org/wizard>_.

Unless you have installed Xdebug with a package manager on Linux, you also need to add the following line to your php.ini file, or create a new Xdebug specific ini file xdebug.ini in the conf.d directory. In either case, it needs the following line added::

zend_extension=xdebug

For more extensive installation instructions, see the documentation at https://xdebug.org/docs/install

Configuration

Most features in Xdebug have to be opted in into. Each feature has a specific opt-in. For example to use the step debugger <https://xdebug.org/docs/remote>_ you need to set xdebug.remote_enable=1 in your configuration file. The step debugger requires an IDE (client), of which there are many available <https://xdebug.org/docs/remote#clients>_.

The documentation has instructions for each of Xdebug's features: https://xdebug.org/docs/ and a full list of settings <https://xdebug.org/docs/all_settings>_ is also available there.

Contributing

Xdebug is written in C, and extensive knowledge of PHP's internals is necessary to be able to contribute. Contributing guidance is available separately <https://github.com/xdebug/xdebug/blob/master/CONTRIBUTING.rst>_.

Before you begin to contribute, please reach out first. Either through email (address at the bottom), an issue in the issue tracker <https://bugs.xdebug.org>_ or preferably through IRC on Freenode's #xdebug channel.

Testing

If you are familiar with compiling PHP extension from source, have a local checkout of Xdebug's GitHub repository, and have compiled Xdebug in that directory following the instructions under installation <https://xdebug.org/docs/install#source>_ you can run Xdebug's tests by running::

php run-xdebug-tests.php

The test framework requires that the PHP binary on the path has Xdebug loaded, with remote debugging enabled through xdebug.mode=debug. It is possible to skip remote debugging tests by exporting the SKIP_DBGP_TESTS=1 environment variable.

The SKIP_UNPARALLEL_TESTS=1 can be used to skip tests that can not run in parallel environments, and the SKIP_SLOW_TESTS=1 environment variable to skip slow tests. The OPCACHE environment variable can either be yes or no and controls whether the test framework enables or disables OpCache.

Licensing

Xdebug is released under The Xdebug License <https://github.com/xdebug/xdebug/blob/master/LICENSE>, which is based on The PHP License <https://github.com/php/php-src/blob/master/LICENSE>. It is an Open Source license (though not explicitly endorsed by the Open Source Initiative).

Support

For questions regarding Xdebug, please use StackOverflow <https://stackoverflow.com/questions/tagged/xdebug>_, and tag your question with xdebug.

You can also find ad-hoc and sporadic support on IRC: freenode/#xdebug. You can do that with your favourite client, or by using their webchat <http://webchat.freenode.net/?channels=#xdebug>_.

If you think that you encountered a bug, please file a detailed bug report at https://bugs.xdebug.org. You are required to create an account, this is so that you can be contacted for additional information and to keep out spam.

Derick Rethans — derick@xdebug.org

ramsey/uuid

2023-09-10T02:01:34Z

❄️ A PHP library for generating universally unique identifiers (UUIDs).

ramsey/uuid

A PHP library for generating and working with UUIDs.

ramsey/uuid is a PHP library for generating and working with universally unique identifiers (UUIDs).

This project adheres to a code of conduct. By participating in this project and its community, you are expected to uphold this code.

Much inspiration for this library came from the Java and Python UUID libraries.

Installation

The preferred method of installation is via Composer. Run the following command to install the package and add it as a requirement to your project's composer.json:

composer require ramsey/uuid

Upgrading to Version 4

See the documentation for a thorough upgrade guide:

Upgrading ramsey/uuid Version 3 to 4

Documentation

Please see https://uuid.ramsey.dev for documentation, tips, examples, and frequently asked questions.

Contributing

Contributions are welcome! To contribute, please familiarize yourself with CONTRIBUTING.md.

Coordinated Disclosure

Keeping user information safe and secure is a top priority, and we welcome the contribution of external security researchers. If you believe you've found a security issue in software that is maintained in this repository, please read SECURITY.md for instructions on submitting a vulnerability report.

ramsey/uuid for Enterprise

Available as part of the Tidelift Subscription.

The maintainers of ramsey/uuid and thousands of other packages are working with Tidelift to deliver commercial support and maintenance for the open source packages you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact packages you use. Learn more.

Copyright and License

ahmedesa/laravel-api-tool-kit

2023-09-10T02:01:34Z

Your all-in-one solution for building efficient APIs with Laravel.

Laravel API tool kit

Introduction

Elevate your development journey with high-performance APIs using the API Toolkit. Supercharge your API development with standardized responses, dynamic pagination, advanced filtering, and a one-click full CRUD setup. Let's take your APIs to the next level!

Installation

to install the package using Composer:

composer require essa/api-tool-kit

Why Choose the Laravel API Toolkit?

Consistent Responses, Less Hassle

The API Response feature simplifies generating consistent JSON responses. It provides a standardized format for your api responses:

{
  "message": "your resource successfully",
  "data": [
    ...
  ]
}

Pagination Done Right

Don't fuss over managing the number of results per page. The dynamic pagination feature adapts effortlessly to your needs, giving you control without complications.

$users = User::dynamicPaginate();

Simplified Filtering

Refine query results with simplicity. The powerful filtering system lets you filter, sort, search, and even include relationships with ease.

Car::useFilters()->get();

Simplify API Setup with the API Generator

The API Generator automates file setup, creating key files from migrations to controllers. Use one command to kickstart your API development.

php artisan api:generate ModelName --all

Schema Support

Enhance the API Generator with schema support, allowing you to define your database table structure directly from the command line. Generate factory model migrations, requests, and data based on this schema.

php artisan api:generate ModelName "column1:string|column2:integer|column3:datetime"

Logic Made Clear

Tackle complex business logic with Actions. These gems follow the command pattern, boosting readability and maintenance for your code.

app(CreateCar::class)->execute($data);

Media? Handled.

Handle file uploads and deletions like a pro. The Media Helper streamlines media management, leaving you with clean and organized file handling.

$filePath = MediaHelper::uploadFile($file, $path);

Enums for Clarity

The Enum class provides a way to work with enumerations, eliminating hardcoded values in your code:

namespace App\Enums;

class UserTypes extends Enum
{
    public const ADMIN = 'admin';
    public const STUDENT = 'student';
}

Official Documentation

Access our documentation to unlock the full potential of the Laravel API Toolkit:

Explore the Documentation

Contributing

We welcome your contributions to help make this package even better. Please refer to our CONTRIBUTING.md file for contribution guidelines.

License

By contributing to the Laravel API Toolkit, you agree that your contributions will be licensed under the project's MIT License.

OasisDEX/vulcan0x

2023-09-10T02:01:46Z

Maker system contracts api

AllenDowney/ThinkPython

2023-09-10T02:02:00Z

Code examples and exercise solutions from Think Python by Allen Downey, published by O'Reilly Media.

0xJs/domain_audit

2023-09-11T01:38:43Z

Audit tool for Active Directory. Automates a lot of checks from a pentester perspective.

Domain Audit

The tool is a wrapper around PowerView, Impacket, PowerUpSQL, BloodHound, Ldaprelayscan and Crackmapexec to automate the execution of enumeration and a lot of checks performed during a On-Prem Active Directory Penetrationtest. Thanks to all the authors of the original tools.

Installation & Setup

Install python 3.10 (For example from the Windows store)

git clone https://github.com/0xJs/domain_audit
cd .\domain_audit\import\
git clone https://github.com/SecureAuthCorp/impacket
cd impacket; python3 -m pip install .
cd ../;
git clone https://github.com/zyn3rgy/LdapRelayScan;
cd LdapRelayScan;
python3 -m pip install -r .\requirements.txt
cd ../../;

Make sure the path viariables in the script to the following tools are correct (if you followed the installation steps it should be)

$script:PowerView_Path = "$PSScriptRoot\import\PowerView.ps1"
$script:Powerupsql_Path = "$PSScriptRoot\import\PowerUpSQL.ps1"
$script:Impacket_Path = "$PSScriptRoot\import\impacket"
$script:BloodHound_Path = "$PSScriptRoot\import\Sharphound.ps1"
$script:GpRegisteryPolicy_Path = "$PSScriptRoot\import\GPRegistryPolicy\GPRegistryPolicy.psd1"
$script:CME_Path = "$PSScriptRoot\import\cme"
$script:LdapRelayScan_Path = "$PSScriptRoot\import\LdapRelayScan\LdapRelayScan.py"

Issue with accessing Sysvol access denied?
- To resolve this issue run gpedit.msc, go to Computer -> Administrative Templates -> Network -> Network Provider -> Hardened UNC Paths, enable the policy and click "Show" button.
- Enter * into "Value name" and enter RequireMutualAuthentication=0,RequireIntegrity=0,RequirePrivacy=0 in value.
- Enter \\*\NETLOGON into "Value name" and enter RequireMutualAuthentication=0,RequireIntegrity=0,RequirePrivacy=0 in value.
- Enter \\*\SYSVOL into "Value name" and enter RequireMutualAuthentication=0,RequireIntegrity=0,RequirePrivacy=0 in value.
If it still gives problems add the following:

reg add HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\NetworkProvider\HardenedPaths /v "\\*\SYSVOL" /d "RequireMutualAuthentication=0" /t REG_SZ /f
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\NetworkProvider\HardenedPaths /v "\\*\NETLOGON" /d "RequireMutualAuthentication=0" /t REG_SZ /f

Running domain_audit

Start PowerShell as Administrator
- This is required to change the DNS server to DC IP and add domain to hosts file. Otherwise some checks/executions will fail such as Impacket.

Run all checks

PS C:\Users\user\Desktop> . C:\Tools\domain_audit\domain_audit.ps1
PS C:\Users\user\Desktop> Invoke-ADCheckAll -Domain "amsterdam.bank.local" -User "john" -Password "Welcome2022!" -Server 10.0.0.3

Example output at the bottom of the page.

Run seperate checks

Some checks need the DNS And domain name in the host file. So best is to set that!

Set dns and host file

Invoke-ChangeDNS -Server 10.0.0.3 -Domain "amsterdam.bank.local"

Run a check

Invoke-ADEnum -Domain "amsterdam.bank.local" -User "john" -Password "Welcome2022!" -Server 10.0.0.3
Invoke-ADEnumTrust -Domain "amsterdam.bank.local" -User "john" -Password "Welcome2022!" -Server 10.0.0.3
Invoke-ADCheckSQL -Domain "amsterdam.bank.local" -User "john" -Password "Welcome2022!" -Server 10.0.0.3

Output

The tool will create a directory with the domain name and date on current directory or in the supplied location. In this directory there are three directories:

Data, has .csv files from domain objects, bloodhound data and some lists of objects.
Findings, has output from checks that could be something interesting or reported as a finding.
Checks, has output from checks that should be assessed manually to check ifs something to be reported.

Checks

Invoke-ADCheckAll will execute the following in order:

Collect basic data of AD objects and place them in /data/ directory in .csv format
Collect data with bloodhound - Collectionmethods all, acl and sessions in the background.
Create list of all (enabled) users, admin accounts, computers and groups in /data/
List amount of users, groups, computers, OU's, GPO's, Administrators, domain controllers and domain functional level.
Check if the amount of admins is more then 5% (High amount of admins in the domain)
Enumerate domain trusts and trusts within forst
Check if AzureAD or Azure SSO is installed
Execute a runas in a new window for the SQL checks
- Check for SQL instances in the domain
- Check if current user has access to SQL instances
  - Check if the current user is sysadmin
  - Gather database info on SQL instancees
  - Check as who the SQL instance is running
  - Check for database links as sysadmin
  - Run invoke-sqlaudit and save all data
    - Check for weak passwords
    - Check for Execute xp_dirtree and Execute xp_fileexist
Check if domain functional level is 2016
Check password policy configuration
- Check for cleartextpasswords = 1
- Check passwordlength
- Check passwordcomplexity
- Check account lockout
Check if kerberos policy configuration is changed from the defaults
Check if there is a GPO with LAPS in its name
- Check to which OU's the GPO is applied to
- Check the LAPS policy
  - Check adminaccountname
  - Check passwordcomplexity
  - Check passwordlength
  - Check passwordagedays
  - Check pwdexpirationprotection enabled
  - Check admpwdenabled
Check if there are systems with LAPS installed
- Check if there are systems where LAPS isn't installed on
If LAPS GPO found or LAPS computers found - Check if the current user can read LAPS passwords
Get all users with a description - Manually check for passwords or interesting information
- Check if the string pw, pass, ww or wachtwoord is in the description.
Get all groups with a description - Manually check for passwords or interesting information
Get all computers with a description - Manually check for passwords or interesting information
Check if there are admins with a Spn
Check if there are users with a Spn
- Kerberoast users with a Spn
Check for users with constrained delegation
Check for users with unconstrained delegation
Check for computers with constrained delegation
Check for computers with unconstrained delegation except domain controllers
Check for computers with resource based constrained delegation set
Check PASSWD_NOT_REQ attribute on users
- Checks if users has empty password
Check DONT_REQ_PREAUTH attribute on users
- AS-REP Roast users
Check DONT_EXPIRE_PASSWORD attribute on users
Check if there are users with reversible encryption
Check if there are users that use DES encryption
Check if there are domain admins with an old password
Check if the KRBTGT has a old password
Check for EOL operating systems in the AD
Check for EOS Windows 10 versions
Check for inactive computerobjects with no login or pwdlastset older then 365 days
Check for inactive users that didn't login the last 365 days
Check if all privileged users are part of the protected users groups
Check if all privileged users have the flag "This account is sensitive and cannot be delegated"
Check if there are members of the following privileged groups: Account Operators, Backup Operators, Print Operators, DNS Admins, Schema Admins
Check if there are computerobjects part of a high privileged groups
Check who can add computerobjects to the domain
- Check if this is the default authenticated users group
- Check what the ms-ds-machineaccountqouta is, if it is null(good) or not-set(bad) or another value.
Check ADIDNS
- if it contains createchild for the authenticated users group
- If wildcard record exists
Check membership of Pre-Windows 2000 Compatible Access group and if authenticated users is still member of it
Check if printspooler service is running on the DC
Check if LDAP signing is enabled, if LDAPS is configured and LDAPS binding is enabled
Check if default Exchange groups exists within the domain
- Check if there is an Exchange server and if its active
- Check for memberships in the default Exchange groups
Check for passwords in the sysvol
Check for passwords in netlogon (scripts, programs etc)
Retrieve IP adresses of all computer objects and create /24 ranges for further enumeration
- Scan the ranges for well known Windows ports + ssh and web. Create lists for each port
- Checks for SMB on accessible machines
  - Check SMBv1
  - Check signing requirement
  - Check for readable shares
  - Check for writable shares
  - Check if the webservice client is running on accessible machines
- Check Access
  - Check if the current user is local admin through SMB
  - Check if the current user can access over WINRM
  - Check if the current user can access over RDP
  - Check if the current user can access over MSSQL and is sysadmin (WIP)

TO-DO & Features

Create query for EOS Server versions?
Update queries/checks for domain admin/enterprise admins to all privileged groups.
Update queries to filter disabled users from data.
Check for old vulnerable Exchange permissions
Change privileged role function to loop and add more roles
Remove -Ping function for accessible machines and execute a portscan for specific ports using Invoke-Portscan
Check for access to discovered machines over SMB, RDP, MSSQL etc!
Find a way to audit all ACL's (Invoke-ACLScanner doesnt work well from non domain joined perspective, -ResolvGUIDS breaks!)
Add ADCS checks, but need to set this up in a LAB environment first.
Add function to skip dns change
Add dependancy options to each function

Example output Invoke-ADCheckAll

PS C:\Users\user\Desktop> . C:\Tools\domain_audit\domain_audit.ps1
PS C:\Users\user\Desktop> Invoke-ADCheckAll -Domain "amsterdam.bank.local" -User "john" -Password "Welcome2022!" -Server 10.0.0.3

[+] Running as administrator, changing DNS to 10.0.0.3 and adding 10.0.0.3 amsterdam.bank.local to host file
[+] AD Authentication for amsterdam.bank.local\john succeeded!
[+] Output will be written in C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22

---------- DATA EXPLAINED ----------
- All data is written to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\
- In this folder are three subfolders
- files in \findings\ are findings that should be reported
- files in \checks\ needs to be checked
- files in \data\ is raw data

---------- COLORS EXPLAINED ----------
White is informational text
Green means check has passed
Yellow means manually check the data
Dark Red means finding

---------- GATHERING DATA ----------
[+] Gathering data of all Users, Groups, Computerobject, GPO's, OU's, DC's and saving it to csv
[+] Gathering BloodHound data all, session and ACL in seperate PowerShell session in background

---------- BASIC ENUMERATION ----------
[W] Saving a list of all users to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\list_users.txt
[W] Saving a list of all enabled users to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\list_users_enabled.txt
[W] Saving a list of all administrators to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\list_administrators.txt
[W] Saving a list of all groups to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\list_groups.txt
[W] Saving a list of all computerobjects to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\list_computers.txt

---------- DOMAIN INFORMATION ----------
The domain functional level is: Windows 2016
In the domain amsterdam.bank.local there are:
- 27 users and 25 enabled users
- 51 groups
- 4 computers
- 4 OU's
- 2 GPO's
- 3 Administrators
- 1 Domain Controllers

---Checking if amount of admins is more then 5% of all users---
[-] There are 3 administrators, which is 11.11% of all users
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\large_amount_of_administrators.txt

---------- ENUMERATING DOMAIN TRUSTS ----------
[+] The domain amsterdam.bank.local trusts 1 domains which are:
bank.local
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\trusts.txt

[+] The trust for domain bank.local is WITHIN_FOREST, enumerating trusts
[+] The domain bank.local trusts 2 domains which are:
amsterdam.bank.local
secure.local
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\trusts.txt

---Checking if AzureAD connect is in use---
[+] AzureAD connect is not installed

---Checking if Azure SSO is in use---
[+] Azure SSO is not configured

---------- EXECUTING CHECKS ----------
[+] Executing in another window because runas is required
[+] Pleace manually supply the Password Welcome2022!
--- Running SQL checks in new window ---
Enter the password for amsterdam.bank.local\john:
Attempting to start powershell.exe -Exec bypass -NoExit Import-Module C:\Tools\domain_audit\domain_audit.ps1; Set-Variable Findings_Path -Value C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings; Set-Variable Data_Path -Value C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data; Set-Variable Checks_Path -Value C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\checks; Set-Variable OutputDirectoryCreated -Value True; Invoke-ADCheckSQL -Domain amsterdam.bank.local -Server 10.0.0.3 -User john -Password Welcome2022! -SkipPrompt as user "amsterdam.bank.local\john" ...

---Checking password policy---
[+] Passwordpolicy contains ClearTextPassword=0. Domain controller does not save passwords in cleartext
[-] Password length requirement is 7 characters
[-] PasswordComplexity is 0 (Disabled)!
[-] LockOutBadCount is 0, accounts wont be locked!
[-] ResetLockoutCount is not set
[-] LockoutDuration is not set
Writing password policy to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\passwordpolicy.txt

---Checking if there is a GPO with LAPS---
[-] There is no GPO with LAPS in their name

---Checking if LAPS is enabled on any computerobject---
[-] There are no systems where LAPS is enabled
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\laps_notenabled.txt

---Checking description field for passwords---
[-] There are 4 users that have a description, please manually check for passwords!
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\checks\description_users.txt

---Checking groups description field for interesting information---
[-] There are 45 groups that have a description, please manually check for passwords or interesting information!
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\checks\description_groups.txt

---Checking computerobjects description field for interesting information---
[+] There are no computerobjects with a description

---Checking kerberoastable administrators---
[-] There are 1 kerberoastable administrators
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\administrators_serviceprincipalname.txt

---Checking kerberoastable users---
[-] There are 2 kerberoastable users
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\users_serviceprincipalname.txt
[+] Requested 2 hashes, please crack with hashcat
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\users_kerberoast_hashes.txt

---Checking if there are users with the DONT_REQ_PREAUTH attribute---
[-] There are 2 users with the attribute DONT_REQ_PREAUTH
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\users_dontrequirepreath.txt
[+] Requested 2 hashes, please crack with hashcat
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\users_aspreproast_hashes.txt

---Checking constrained delegation users---
[-] There are 1 users that have constrained delegation enabled
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\users_constrained_delegation.txt

---Checking unconstrained delegation computerobjects, excluding domain-controllers---
[-] There are 1 users that have unconstrained delegation enabled
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\users_unconstrained_delegation.txt

---Checking constrained delegation computerobjects---
[+] There are no computerobjects with constrained delegation

---Checking unconstrained delegation computerobjects, excluding domain-controllers---
[-] There are 1 computerobjects that have unconstrained delegation enabled
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\computers_unconstrained_delegation.txt

---Checking resource based constrained delegation computerobjects---
[+] There are no computerobjects with resource based constrained delegation

---Checking if there are users with the PASSWD_NOTREQD attribute---
[-] There are 2 users with the attribute PASSWD_NOTREQD
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\users_passwdnotreqd.txt

---Checking for users with empty password---
[-] The password for user bank_dev is empty
[-] The password for user steve is empty

---Checking if there are users with the DONT_EXPIRE_PASSWORD attribute---
[-] There are 17 users with the attribute DONT_EXPIRE_PASSWORD
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\users_dontexpirepassword.txt

---Checking if there are users with the reversible encryption---
[+] There are no users with reversible encryption

---Checking if there are users with DES encryption---
[+] There are no users with DES encryption

---Checking if administrator accounts - that aren't disabled - have a password older then 365 days---
[+] There where no enabled administrators with a password older then 365 days

---Checking if KRBTGT account has a password older then 365 days---
[+] The password from the krbtgt is not older then 365 days

---Checking if there are EOL operating systems in the AD---
[+] There are no computerobjects in the AD that are EOL

---Checking if there are end of service Windows 10 operating systems in the AD---
[+] There are no Windows 10 computerobjects computerobjects in the AD that are End Of Service

---Checking if there are computerobjects that have no login or login/pwdlastset older then 365 days---
[+] There are no computerobjects in the AD that are inactive

---Checking if there are users that didn't login for 365 days---
[+] There are no users in the AD that are inactive (didn't login or changed their password in the last 365 days)

---Checking if members of privileged groups are part of the protected users group---
[-] There are 3 privileged users not part of the protected users group
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\administrators_notin_protectedusersgroup.txt

---Checking if members of privileged groups have the flag 'this account is sensitive and cannot be delegated'---
[-] There are 3 privileged users without the flag 'this account is sensitive and cannot be delegated' that aren't in the Protected Users group
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\administrators_delegation_flag.txt

---Checking if there are members in high privileged groups---
[+] There are no users in the Account Operators group
[+] There are no users in the Backup Operators group
[+] There are no users in the Print Operators group
[+] There are no users in the DNS Admins group
[+] There are no users in the Schema Admins group

---Checking if there are computerobjects part ofhigh privileged groups---
[+] There are no computerobjects part of a high privileged groups

---Checking who can add computerobjects to the domain---
[-] The authenticated users group(S-1-5-11) can add 10 computerobjects to the domain
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\authenticated_users_can_join_domain.txt
[W] Writing amount of computerobjects that can be joined to the domain by the object to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\checks\can_join_domain_amount.txt

---Checking if there are passwords in the SYSVOL share---
This might take a while
[+] Checking SYSVOL of DC02
[-] There might be 1 passwords in the SYSVOL of DC02. Please manually check
Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\checks\sysvol_passwords.txt

---Checking if there are passwords in the NETLOGON share---
This might take a while
[+] Checking NETLOGON of DC02
[-] There might be 1 passwords(string pass) in the NETLOGON of DC02. Please manually check
Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\checks\netlogon_passwords.txt

---Checking printspooler service on each DC---
[-] Printspooler enabled on DC02
Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\printspooler_domaincontrollers.txt

---Running LdapRelayScan---
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\domaincontrollers_ldaprelayscan.txt

---Checking for LDAP signing---
[+] One or more domain controller(s) does not require LDAP signing
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\domaincontrollers_no_ldap_signing.txt

---Checking for LDAPS binding---
[+] One or more domain controller(s) does not require LDAPS binding
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\domaincontrollers_no_ldaps_binding.txt

---Checking if exchange is used within the domain---
[+] No Exchange groups exist

---Checking which machines are reachable from current machine through ping---
[+] There are 3 computers which are reachable
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\computers_accessible.txt

---Running crackmapexec against each reachable host enumerating SMB data and shares---
[+] Crackmapexec will hang and needs a enter to continue
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\crackmapexec_reachablecomputers.txt

---Checking for hosts which have SMBV1 enabled---
[+] There are no reachable computers which have SMBV1 enabled (SMBv1:True)

---Checking for hosts without signing---
[+] There are 2 reachable computers which does not require signing (Signing:False)
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\computers_nosigning.txt

---Checking for shares with READ access---
[+] There are 4 shares the current user can READ
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\shares_read_access.txt

---Checking for shares with WRITE access---
[+] There are 1 shares the current user can WRITE to
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\shares_write_access.txt

---Running crackmapexec against each reachable host enumerating webclient service---
[+] Crackmapexec will hang and needs a enter to continue
[W] Writing all data to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\data\crackmapexec_webdav.txt

[+] There are 1 systems with the webclient service running
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\computers_webdav.txt

---------- EXECUTING SQL CHECKS ----------
---Checking MSSQL instances---
[+] Found 1 MSSQL instances
[+] Checking connection to each MSSQL instance
[-] The current user can access 1 MSSQL instances
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\SQLserver_user_has_access.txt

---Checking if the user is sysadmin on the accessible instances---
[+] The current user is not sysdmin to any SQL instances

---Checking database links for sysadmin security context---
[-] There are 1 links which run under the security context of a sysadmin user
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\SQLserver_sysadmin_on_links.txt

---Running Invoke-SQLAudit on the accessible instances---
This might take a while
VERBOSE: WEB01.amsterdam.bank.local : No named instance found.
VERBOSE: WEB01.amsterdam.bank.local : Connection Success.
VERBOSE: WEB01.amsterdam.bank.local : Checking for autoexec stored procedures...
[-] Invoke-SQLAudit found 7 issues
[W] Writing to C:\Users\user\Desktop\amsterdam.bank.local-2022-08-22\findings\SQLserver_sqlaudit.txt

Slluxx/ImPS

2023-09-11T01:38:43Z

ImPS is a wrapper module to create GUIs with PowerShell like you do with ImGui

ImPS

ImPS is a simple one-file module wrapper to use WPF components in your script without writing massive amounts of UI code or using XML layouts. It is heavily inspired by ImGUI/DearPyGui to make it as easy as possible to use while keeping the configurability.

Please have in mind that this is in active development and, while usable, should not be used in any kind of production environment.

Documentation

Click me

Comparison

Here is the code you would need to write this basic window + label:

Add-Type -AssemblyName System.Windows.Forms

$Window = New-Object System.Windows.Forms.Form
$Window.ClientSize = "285,75"
$Window.Text = "ImPS Window"
$Window.BackColor = "#ffffff"
$Window.StartPosition = [System.Windows.Forms.FormStartPosition]::CenterScreen

$Label = New-Object System.Windows.Forms.label
$Label.Text = "This is almost like ImGUI"
$Label.AutoSize=$true
$Label.Font="Verdana,8,style=Regular"
$Label.Location=New-Object System.Drawing.Point(20,20)

$Window.Controls.Add($Label)

$Window.ShowDialog()
$Window.Dispose()

Here is ImPS's code for the exact same:

using module ".\ImPS.psm1"

$ImPS = [ImPS]::new()
$window = $ImPS.add_window("ImPS Window", 285, 75)
$window.Elements.add_Label("This is almost like ImGUI", 20,20)
$window.show()

Usable Elements

Label
Button
Checkbox
ListBox (And its ListItem)
TextBox
ProgressBar
ComboBox
TableLayoutPanel

More elements will get added.

Advanced usage

ImPS is written for you to easily chain methods if desired:

$window.Elements.add_Label("My bold label", 20, 20).set_font("Verdana,8,style=Bold")
# or
$label = $window.Elements.add_Label("My italic label", 20, 40)
$label.set_font("Verdana,8,style=Italic")

Button onClick-handler work inline or as defined function:

$btn = $window.Elements.add_Button("Click me", 20, 20).onClick({
    write-host "Got clicked"
})
# or
$myfunc = { write-host "Got clicked" }
$btn = $window.Elements.add_Button("Click me", 20, 20).onClick($myfunc)

[ImPS_Window].Drawable exposes a System.Windows.Forms.Form Object, so you can combine ImPS with the normal method of creating UIs (for example if a UI element does not exist yet):

$window = [ImPS]::new("ImPS Window", 285, 200)
$objLabel = New-Object System.Windows.Forms.label
$objLabel.Location = New-Object System.Drawing.Size(7,10)
$objLabel.Size = New-Object System.Drawing.Size(130,15)
$objLabel.Text = "Enter Computer Name"
$window.Drawable.Controls.Add($objLabel)

Every UI Element by ImPS exposes its "native" Object at [Object].Drawable, which means you can change attributes even if there is no getter/setter by ImPS:

$texbox = $window.Elements.add_TextBox("password", 100, 20)
$texbox.Drawable.PasswordChar = '*'

Note: This is discouraged and should not be done. I plan on supporting the most used attribute getter/setter.

cyb3rzest/Burp-Suite-Pro

2023-09-10T02:02:03Z

A bash and powershell script to download the latest version of Burp-Suite Professional and use it for free.

Burp-Suite-Pro

Burp Suite Professional Installation steps for Windows

--> Copy the downloaded files to C:\Burp.
	Make new directory "Burp" in C Drive for faster access.
--> Open Powershell and execute below command to set Script Execution Policy.
	Set-ExecutionPolicy -ExecutionPolicy bypass -Scope process
--> Now Execute Windows_Setup.ps1 file in Powershell to Complete Installation.
	./Windows_Setup.ps1
--> Change the icon of **Burp-Suite-Pro.vbs** to the given icon 
	Create a shortcut to Desktop. Right Click over **Burp-Suite-Pro.vbs** Go to Shortcut tab, and below there is **Change Icon** tab.
	Click there and choose the burp-suite.ico from C:\Burp\ 
	![image](https://user-images.githubusercontent.com/29830064/230825172-16c9cfba-4bca-46a4-86df-b352a4330b12.png)

--> For Start Menu Entry, copy **Burp-Suite-Pro.vbs** file to 
	C:\ProgramData\Microsoft\Windows\Start Menu\Programs\

Burp Suite Professional Installation steps for Kali-Linux

--> Execute Kali_Linux_Setup.sh file as root user

Burp Suite Pro Activation Steps

1. Modify License String like "license to xxxxxxxx"
2. Copy License key from loader.jar(Keygenerator) and paste in Burp Suite Pro and click Next.
3. Select Manual Activation Option on your bottom Right in Burp Suite Pro.
4. Copy License Request from BurpSuite_Pro and paste in Keygenerator.
5. Copy license response from Keygenerator and paste in Burp Suite Pro, then next and Done.

Video Tutorial

# For Windows

# For Kali Linux, Parrot OS, Ubuntu

Executing Burp Suite Profession after Activation

--> Windows :-: You can start Burp Suite Professional from Start Menu.
--> Kali Linux :-: You can start Burp Suite Professional by writing **burpsuite** in terminal.

denoland/deno_install

2023-09-10T02:02:03Z

Deno Binary Installer

deno_install

One-line commands to install Deno on your system.

Install Latest Version

With Shell:

curl -fsSL https://deno.land/install.sh | sh

With PowerShell:

irm https://deno.land/install.ps1 | iex

Install Specific Version

With Shell:

curl -fsSL https://deno.land/install.sh | sh -s v1.0.0

With PowerShell:

$v="1.0.0"; irm https://deno.land/install.ps1 | iex

Install via Package Manager

With Scoop:

scoop install deno

With Homebrew:

brew install deno

With Macports:

sudo port install deno

With Chocolatey:

choco install deno

With Snap:

sudo snap install deno

With Pacman:

pacman -S deno

With Zypper:

zypper install deno

Build and install from source using Cargo:

# Install the Protobuf compiler
apt install -y protobuf-compiler # Linux
brew install protobuf # macOS

# Build and install Deno
cargo install deno

Install and Manage Multiple Versions

With asdf and asdf-deno:

asdf plugin add deno

# Get latest version of deno available
DENO_LATEST=$(asdf latest deno)

asdf install deno $DENO_LATEST

# Activate globally with:
asdf global deno $DENO_LATEST

# Activate locally in the current folder with:
asdf local deno $DENO_LATEST

#======================================================
# If you want to install specific version of deno then use that version instead
# of DENO_LATEST variable example
asdf install deno 1.0.0

# Activate globally with:
asdf global deno 1.0.0

# Activate locally in the current folder with:
asdf local deno 1.0.0

With Scoop:

# Install a specific version of deno:
scoop install deno@1.0.0

# Switch to v1.0.0
scoop reset deno@1.0.0

# Switch to the latest version
scoop reset deno

Environment Variables

DENO_INSTALL - The directory in which to install Deno. This defaults to $HOME/.deno. The executable is placed in $DENO_INSTALL/bin. One application of this is a system-wide installation:

With Shell (/usr/local):
```
curl -fsSL https://deno.land/install.sh | sudo DENO_INSTALL=/usr/local sh
```
With PowerShell (C:\Program Files\deno):
```
# Run as administrator:
$env:DENO_INSTALL = "C:\Program Files\deno"
irm https://deno.land/install.ps1 | iex
```

Compatibility

The Shell installer can be used on Windows with Windows Subsystem for Linux, MSYS or equivalent set of tools.

Known Issues

unzip is required

The program unzip is a requirement for the Shell installer.

$ curl -fsSL https://deno.land/install.sh | sh
Error: unzip is required to install Deno (see: https://github.com/denoland/deno_install#unzip-is-required).

When does this issue occur?

During the install.sh process, unzip is used to extract the zip archive.

How can this issue be fixed?

You can install unzip via brew install unzip on MacOS or apt-get install unzip -y on Linux.

CERIT-SC/puppet-sec

2023-09-10T02:02:36Z

Puppet module for SEC (simple event correlator)

jeraman/frevo-algorithms

2023-09-10T02:02:40Z

ECTO-1A/AppleJuice

2023-09-10T01:41:30Z

Arch Linux installer - guided, templates etc.

Arch Installer

Just another guided/automated Arch Linux installer with a twist. The installer also doubles as a python library to install Arch Linux and manage services, packages and other things inside the installed system (Usually from a live medium).

archinstall discord server
archinstall matrix.org channel
archinstall #archinstall@irc.libera.chat
archinstall documentation

Installation & Usage

$ sudo pacman -S archinstall

Alternative ways to install are git clone the repository or pip install --upgrade archinstall.

Running the guided installer

Assuming you are on an Arch Linux live-ISO or installed via pip:

# archinstall

Running the guided installer using `git`

# cd archinstall-git
# cp archinstall/scripts/guided.py
# python guided.py

Advanced

Some additional options that are not needed by most users are hidden behind the --advanced flag.

Running from a declarative configuration file or URL

archinstall can be run with a JSON configuration file. There are 2 different configuration files to consider, the user_configuration.json contains all general installation configuration, whereas the user_credentials.json contains the sensitive user configuration such as user password, root password and encryption password.

An example of the user configuration file can be found here configuration file and example of the credentials configuration here credentials file.

HINT: The configuration files can be auto-generated by starting archinstall, configuring all desired menu points and then going to Save configuration.

To load the configuration file into archinstall run the following command

archinstall --config <path to user config file or URL> --creds <path to user credentials config file or URL>

Available Languages

Archinstall is available in different languages which have been contributed and are maintained by the community. Current translations are listed below and vary in the amount of translations per language

English
Arabic
Brazilian Portuguese
Czech
Dutch
Estonian
French
Georgian
German
Indonesian
Italian
Korean
Modern Greek
Polish
Portuguese
Russian
Spanish
Swedish
Tamil
Turkish
Ukrainian
Urdu

Any contributions to the translations are more than welcome, to get started please follow the guide

Help or Issues

Submit an issue here on GitHub, or submit a post in the discord help channel.
When doing so, attach the /var/log/archinstall/install.log to the issue ticket. This helps us help you!

Mission Statement

Archinstall promises to ship a guided installer that follows the Arch Principles as well as a library to manage services, packages and other Arch Linux aspects.

The guided installer will provide user-friendly options along the way, but the keyword here is options, they are optional and will never be forced upon anyone. The guided installer itself is also optional to use if so desired and not forced upon anyone.

Archinstall has one fundamental function which is to be a flexible library to manage services, packages and other aspects inside the installed system. This library is in turn used by the provided guided installer but is also for anyone who wants to script their own installations.

Therefore, Archinstall will try its best to not introduce any breaking changes except for major releases which may break backwards compatibility after notifying about such changes.

Scripting your own installation

Scripting interactive installation

There are some examples in the examples/ directory that should serve as a starting point.

The following is a small example of how to script your own interative installation:

from pathlib import Path

from archinstall import Installer, ProfileConfiguration, profile_handler, User
from archinstall.default_profiles.minimal import MinimalProfile
from archinstall.lib.disk.device_model import FilesystemType
from archinstall.lib.disk.encryption_menu import DiskEncryptionMenu
from archinstall.lib.disk.filesystem import FilesystemHandler
from archinstall.lib.interactions.disk_conf import select_disk_config

fs_type = FilesystemType('ext4')

# Select a device to use for the installation
disk_config = select_disk_config()

# Optional: ask for disk encryption configuration
data_store = {}
disk_encryption = DiskEncryptionMenu(disk_config.device_modifications, data_store).run()

# initiate file handler with the disk config and the optional disk encryption config
fs_handler = FilesystemHandler(disk_config, disk_encryption)

# perform all file operations
# WARNING: this will potentially format the filesystem and delete all data
fs_handler.perform_filesystem_operations()

mountpoint = Path('/tmp')

with Installer(
        mountpoint,
        disk_config,
        disk_encryption=disk_encryption,
        kernels=['linux']
) as installation:
    installation.mount_ordered_layout()
    installation.minimal_installation(hostname='minimal-arch')
    installation.add_additional_packages(['nano', 'wget', 'git'])

    # Optionally, install a profile of choice.
    # In this case, we install a minimal profile that is empty
    profile_config = ProfileConfiguration(MinimalProfile())
    profile_handler.install_profile_config(installation, profile_config)

    user = User('archinstall', 'password', True)
    installation.create_users(user)

This installer will perform the following:

Prompt the user to configurate the disk partitioning
Prompt the user to setup disk encryption
Create a file handler instance for the configured disk and the optional disk encryption
Perform the disk operations (WARNING: this will potentially format the disks and erase all data)
Installs a basic instance of Arch Linux (base base-devel linux linux-firmware btrfs-progs efibootmgr)
Installs and configures a bootloader to partition 0 on uefi. On BIOS, it sets the root to partition 0.
Install additional packages (nano, wget, git)
Create a new user

Creating your own ISO with this script on it: Follow ArchISO's guide on how to create your own ISO.

Script non-interactive automated installation

For an example of a fully scripted, automated installation please see the example full_automated_installation.py

Unattended installation based on MAC address

Archinstall comes with an unattended example which will look for a matching profile for the machine it is being run on, based on any local MAC address. For instance, if the machine the code is executed on has the MAC address 52:54:00:12:34:56 it will look for a profile called 52-54-00-12-34-56.py. If it's found, the unattended installation will commence and source that profile as its installation procedure.

Profiles

archinstall ships with a set of pre-defined profiles that can be chosen during the installation process.

The definitions of the profiles and what packages they will install can be seen directly in the menu or default profiles

Testing

Using a Live ISO Image

If you want to test a commit, branch or bleeding edge release from the repository using the vanilla Arch Live ISO image, you can replace the version of archinstall with a new version and run that with the steps described below:

You need a working network connection
Install the build requirements with pacman -Sy; pacman -S git python-pip gcc pkgconf (note that this may or may not work depending on your RAM and current state of the squashfs maximum filesystem free space)
Uninstall the previous version of archinstall with pip uninstall archinstall
Now clone the latest repository with git clone https://github.com/archlinux/archinstall
Enter the repository with cd archinstall At this stage, you can choose to check out a feature branch for instance with git checkout v2.3.1-rc1
Build the project and install it using pip install --break-operating-system .

After this, running archinstall with python -m archinstall will run against whatever branch you chose in step 5.

Without a Live ISO Image

To test this without a live ISO, the simplest approach is to use a local image and create a loop device.
This can be done by installing pacman -S arch-install-scripts util-linux locally and doing the following:

# truncate -s 20G testimage.img
# losetup -fP ./testimage.img
# losetup -a | grep "testimage.img" | awk -F ":" '{print $1}'
# pip install --upgrade archinstall
# python -m archinstall --script guided
# qemu-system-x86_64 -enable-kvm -machine q35,accel=kvm -device intel-iommu -cpu host -m 4096 -boot order=d -drive file=./testimage.img,format=raw -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_VARS.fd

This will create a 20 GB testimage.img and create a loop device which we can use to format and install to.
archinstall is installed and executed in guided mode. Once the installation is complete, ~~you can use qemu/kvm to boot the test media.~~
(You'd actually need to do some EFI magic in order to point the EFI vars to the partition 0 in the test medium, so this won't work entirely out of the box, but that gives you a general idea of what we're going for here)

There's also a Building and Testing guide.
It will go through everything from packaging, building and running (with qemu) the installer against a dev branch.

Contributing

Please see CONTRIBUTING.md

XPixelGroup/DiffBIR

2023-09-10T01:41:30Z

DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior

Paper | Project Page

Xinqi Lin^1,*, Jingwen He^2,*, Ziyan Chen², Zhaoyang Lyu², Ben Fei², Bo Dai², Wanli Ouyang², Yu Qiao², Chao Dong^1,2

¹Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
²Shanghai AI Laboratory

⭐If DiffBIR is helpful for you, please help star this repo. Thanks!🤗

📖Table Of Contents

👀Visual Results On Real-world Images

General Image Restoration

Face Image Restoration

⚙Installation

Python >= 3.9
CUDA >= 11.3
PyTorch >= 1.12.1
xformers == 0.0.16

# clone this repo
git clone https://github.com/XPixelGroup/DiffBIR.git
cd DiffBIR

# create a conda environment with python >= 3.9
conda create -n diffbir python=3.9
conda activate diffbir

conda install pytorch==1.12.1 torchvision==0.13.1 cudatoolkit=11.3 -c pytorch
conda install xformers==0.0.16 -c xformers

# other dependencies
pip install -r requirements.txt

🧬Pretrained Models

Model Name	Description
general_swinir_v1.ckpt	Stage1 model (SwinIR) for general image restoration.
general_full_v1.ckpt	Full model for general image restoration. "Full" means it contains both the stage1 and stage2 model.
face_swinir_v1.ckpt	Stage1 model (SwinIR) for face restoration.
face_full_v1.ckpt	Full model for face restoration.

🛫Quick Start

Download general_full_v1.ckpt and general_swinir_v1.ckpt to weights/, then run the following command to interact with the gradio website.

python gradio_diffbir.py \
--ckpt weights/general_full_v1.ckpt \
--config configs/model/cldm.yaml \
--reload_swinir \
--swinir_ckpt weights/general_swinir_v1.ckpt

⚔Inference

Full Pipeline (Remove Degradations & Refine Details)

General Image

Download general_full_v1.ckpt and general_swinir_v1.ckpt to weights/ and run the following command.

python inference.py \
--input inputs/general \
--config configs/model/cldm.yaml \
--ckpt weights/general_full_v1.ckpt \
--reload_swinir --swinir_ckpt weights/general_swinir_v1.ckpt \
--steps 50 \
--sr_scale 4 \
--image_size 512 \
--color_fix_type wavelet --resize_back \
--output results/general

If you are confused about where the reload_swinir option came from, please refer to the degradation details.

Face Image

Download face_full_v1.ckpt to weights/ and run the following command.

# for aligned face inputs
python inference_face.py \
--config configs/model/cldm.yaml \
--ckpt weights/face_full_v1.ckpt \
--input inputs/face/aligned \
--steps 50 \
--sr_scale 1 \
--image_size 512 \
--color_fix_type wavelet \
--output results/face/aligned --resize_back \
--has_aligned

# for unaligned face inputs
python inference_face.py \
--config configs/model/cldm.yaml \
--ckpt weights/face_full_v1.ckpt \
--input inputs/face/whole_img \
--steps 50 \
--sr_scale 1 \
--image_size 512 \
--color_fix_type wavelet \
--output results/face/whole_img --resize_back

Only Stage1 Model (Remove Degradations)

Download general_swinir_v1.ckpt, face_swinir_v1.ckpt for general, face image respectively, and run the following command.

python scripts/inference_stage1.py \
--config configs/model/swinir.yaml \
--ckpt [swinir_ckpt_path] \
--input [lq_dir] \
--sr_scale 1 --image_size 512 \
--output [output_dir_path]

Only Stage2 Model (Refine Details)

Since the proposed two-stage pipeline is very flexible, you can utilize other awesome models to remove degradations instead of SwinIR and then leverage the Stable Diffusion to refine details.

# step1: Use other models to remove degradations and save results in [img_dir_path].

# step2: Refine details of step1 outputs.
python inference.py \
--config configs/model/cldm.yaml \
--ckpt [full_ckpt_path] \
--steps 50 --sr_scale 1 --image_size 512 \
--input [img_dir_path] \
--color_fix_type wavelet --resize_back \
--output [output_dir_path] \
--disable_preprocess_model

🌠Train

Degradation Details

For general image restoration, we first train both the stage1 and stage2 model under codeformer degradation to enhance the generative capacity of the stage2 model. In order to improve the ability for degradation removal, we train another stage1 model under Real-ESRGAN degradation and utilize it during inference.

For face image restoration, we adopt the degradation model used in DifFace for training and directly utilize the SwinIR model released by them as our stage1 model.

Data Preparation

Generate file list of training set and validation set.
```
python scripts/make_file_list.py \
--img_folder [hq_dir_path] \
--val_size [validation_set_size] \
--save_folder [save_dir_path] \
--follow_links
```
This script will collect all image files in img_folder and split them into training set and validation set automatically. You will get two file lists in save_folder, each line in a file list contains an absolute path of an image file:
```
save_folder
├── train.list # training file list
└── val.list   # validation file list
```
Configure training set and validation set.

For general image restoration, fill in the following configuration files with appropriate values.
- training set and validation set for CodeFormer degradation.
- training set and validation set for Real-ESRGAN degradation.
For face image restoration, fill in the face training set and validation set configuration files with appropriate values.

Train Stage1 Model

Configure training-related information.

Fill in the configuration file of training with appropriate values.
Start training.
```
python train.py --config [training_config_path]
```
💡:Checkpoints of SwinIR will be used in training stage2 model.

Train Stage2 Model

Download pretrained Stable Diffusion v2.1 to provide generative capabilities.

wget https://huggingface.co/stabilityai/stable-diffusion-2-1-base/resolve/main/v2-1_512-ema-pruned.ckpt --no-check-certificate

Create the initial model weights.

python scripts/make_stage2_init_weight.py \
--cldm_config configs/model/cldm.yaml \
--sd_weight [sd_v2.1_ckpt_path] \
--swinir_weight [swinir_ckpt_path] \
--output [init_weight_output_path]

You will see some outputs which show the weight initialization.

Configure training-related information.

Fill in the configuration file of training with appropriate values.

Start training.

python train.py --config [training_config_path]

🆕Update

2023.08.30: Repo is released.
2023.09.06: Update colab demo. Thanks to camenduru!🤗
2023.09.08: Add support for restoring unaligned faces.

🧗TODO

Release code and pretrained models💻.
Update links to paper and project page🔗.
Release real47 testset💽.
Reduce the memory usage of DiffBIR😺.
Provide HuggingFace demo📓.
Upload inference code of latent image guidance📄.
Improve the performance🦸.
Add a patch-based sampling schedule🔍.

Citation

Please cite us if our work is useful for your research.

@article{2023diffbir,
  author    = {Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, Chao Dong},
  title     = {DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior},
  journal   = {arxiv},
  year      = {2023},
}

License

This project is released under the Apache 2.0 license.

Acknowledgement

This project is based on ControlNet and BasicSR. Thanks for their awesome work.

Contact

If you have any questions, please feel free to contact with me at linxinqi@tju.edu.cn.

OpenGVLab/LLaMA-Adapter

2023-09-10T01:41:30Z

Fine-tuning LLaMA to follow Instructions within 1 Hour and 1.2M Parameters

LLaMA-Adapter: Efficient Fine-tuning of LLaMA 🚀

Announcement: We release LLaMA2-Accessory, an open-source toolkit for pre-training, fine-tuning and deployment of LLMs and mutlimodal LLMs.🔥

Official implementation of 'LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention' and 'LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model'.

This repo proposes LLaMA-Adapter (V2), a lightweight adaption method for fine-tuning Instruction-following and Multi-modal LLaMA models 🔥.

Try out the web demo 🤗 of LLaMA-Adapter: , LLaMA-Adapter V2 and ImageBind-LLM.

News

[2023.08.28] We release quantized LLM with OmniQuant, which is an efficient, accurate, and omnibearing (even extremely low bit) quantization algorithm. Multimodal version is coming soon.🔥🔥🔥
[2023.07.24] We release LLaMA2-Accessory, an open-source toolkit for pre-training, fine-tuning and deployment of Large Language Models (LLMs) and mutlimodal LLMs. Please check Alpha-VLLM/LLaMA2-Accessory for more details!🔥🔥🔥
[2023.07.05] We release the pretrain/finetune code of llama_adapter_v2_multimodal7b.
[2023.07.04] We release the code for reproducing Gorilla by both full finetune and LLaMA-Adapter, please see gorilla/README.md.
[2023.06.08] We release the demo of ImageBind-LLM.
[2023.06.06] We release Point-Bind to extend ImageBind with 3D point clouds, which achieves 3D instruction-following capacity for imagebind_LLM.
[2023.06.05] We support the integration of LLaMA-Adapter (both V1 and V2) and LangChain. Check out the Notebook.
[2023.05.29] We release the code of ImageBind-LLM at imagebind_LLM.
[2023.05.23] We release the demos and multi-modal code of LLaMA-Adapter V2!
[2023.05.05] We release the paper and code of our new work Personalize Segment Anything, which efficiently fine-tunes Segment Anything with 10 seconds, and improves DreamBooth for better text-to-image generation.
[2023.04.30] We noticed that GPT-4 evaluation has a strong positional bias in favor of the first response. We will soon update the paper to reveal the position bias. Great thanks to Canwen Xu.
[2023.04.28] We release LLaMA-Adapter V2, a multi-modal instruction model. Check out our paper, demos and code!
[2023.03.28] The paper and training code for LLaMA-Adapter V1 are released. 📌

Released Models

Name	Approach	Data	Modality	Visual	Text
LLaMA-Adapter V1	prefix, gate	Alpaca	Text	×	LLaMA-7B
LLaMA-Adapter V2 dialog	scale, bias, norm	ShareGPT	Text	×	LLaMA-65B
LLaMA-Adapter V2 multimodal	[P] prefix, projection, gate [F] bias, norm	[P] Image-Text-V1 [F] GPT4LLM, LLaVA	Image&Text	CLIP-ViT-L/14	LLaMA-7B
ImageBind-LLM	[P] prefix, projection, gate [F] bias, norm, lora	[P] Image-Text-V1 [F] Instruction Following	ImageBind Modalities + Point Cloud	imagebind_huge	Open-Chinese-LLaMA-7B
ImageBind-dialog	[P] prefix, projection, gate [F] bias, norm, lora	[P] Image-Text-V1 [F] LLaVA, ShareGPT	ImageBind Modalities + Point Cloud	imagebind_huge	Open-Chinese-LLaMA-7B

[P] means Pre-train and [F] means Fine-tune
Image-Text-V1 is a concatenation of LAION400M, COYO, MMC4, SBU, Conceptual Captions, and COCO
ImageBind Modalities include image, video, text, audio, depth, thermal, IMU
ImageBind-dialog will be release soon

Overview

Efficiency Comparison:

Model	Parameters	Storage Space	Training Time
Alpaca	7B	13G	3 Hours
LLaMA-Adapter	1.2M	4.7M	1 Hour

By inserting adapters into LLaMA's transformer, our method only introduces 1.2M learnable parameters, and turns a LLaMA into an instruction-following model within 1 hour. For stablizing training at early stages, we propose a novel Zero-init Attention with zero gating mechanism to adaptively incorporate the instructional signals. After fine-tuning, LLaMA-Adapter can generate high-quality instruction-following sentences, comparable to the fully fine-tuned Stanford Alpaca and Alpaca-Lora.

Our approach can be simply extended to Multi-modal Input Instructions. The reasoning framework of image-conditioned LLaMA-Adapter for ScienceQA is as follows, which is also shared by other modalities, such as audio and video.

Setup

Here is a from-scratch script for LLaMA-Adapter V1.

conda create -n llama_adapter -y python=3.8
conda activate llama_adapter

# install pytorch
conda install pytorch cudatoolkit -c pytorch -y

# install dependency and llama-adapter
pip install -r requirements.txt
pip install -e .

Note: To setup other models, please refer to llama_adapter_v2_chat65b, llama_adapter_v2_multimodal7b and imagebind_LLM for more details.

Inference

Please request access to the pre-trained LLaMA from this form (official) or download the LLaMA-7B from Hugging Face (unofficial). Then, obtain the weights of our LLaMA-Adapter from here. We denote the path to the downloaded weights of LLaMA and adapters as TARGET_FOLDER and ADAPTER_PATH.

Here is an example to generate instruction-following sentences with 7B LLaMA model and our LLaMA-Adapter:

torchrun --nproc_per_node 1 example.py \
         --ckpt_dir $TARGET_FOLDER/model_size\
         --tokenizer_path $TARGET_FOLDER/tokenizer.model \
         --adapter_path $ADAPTER_PATH

Training

We release the simple fine-tuning code of LLaMA-Adapter on LLaMA-7B model at here, which is for effortless reproduction with minimal dependencies. We will soon release the fine-tuning code for LLaMA-65B and multi-model LLaMA-Adapter.

Please download the 52K instruction-following training data from Standford Alpaca, and put it under DATA_PATH. Then run:

cd alpaca_finetuning_v1

torchrun --nproc_per_node 8 finetuning.py \
         --model Llama7B_adapter \
         --llama_model_path $TARGET_FOLDER/ \
         --data_path $DATA_PATH/alpaca_data.json \
         --adapter_layer 30 \
         --adapter_len 10 \
         --max_seq_len 512 \
         --batch_size 4 \
         --epochs 5 \
         --warmup_epochs 2 \
         --blr 9e-3 \
         --weight_decay 0.02 \
         --output_dir ./checkpoint/

Comparison with Other Methods

Instruction 1:

How can I develop my critical thinking skills? Give me three suggestions.

LLaMA-Adapter V2:

Read widely: Reading books, articles, and other sources of information can help you develop your critical thinking skills. By exposing yourself to different perspectives and ideas, you can learn to analyze and evaluate information more effectively.
Practice mindfulness: Mindfulness is the practice of being present in the moment and paying attention to your thoughts and feelings. By being more aware of your thoughts and emotions, you can better understand and evaluate them. This can help you develop a more objective and rational approach to thinking.
Engage in open-minded discussions: Join a discussion group or debate club where you can engage in open-minded discussions with others. This can help you develop your critical thinking skills by challenging your assumptions and encouraging you to consider different perspectives.

LLaMA-Adapter V1:

Read widely and critically.
Engage in discussions with people who have different perspectives.
Ask questions and seek out answers.

Instruction 2:

As a superhero, how would you explain your origin story to a curious child?

LLaMA-Adapter V2:

As a superhero, I have a very interesting and exciting origin story to share with you. Once upon a time, there was a young boy named John who lived in a small town. John was always curious and wanted to help others, so he spent his days exploring the world around him and learning about different cultures and people. One day, while he was out exploring, he stumbled upon a mysterious cave. Inside the cave, John found a strange device that had the power to give him incredible abilities. With these abilities, John could fly, shoot lasers from his eyes, and even turn invisible.

John knew that he had to use these powers for good, so he decided to become a superhero and use his abilities to protect his town and the people who lived there. He started to train every day, learning how to use his powers and how to fight against evil. Eventually, John became the superhero known as "The Invisible Man" and he used his powers to help others and make the world a better place.

That's my origin story, and I'm sure you'll love it!

LLaMA-Adapter V1:

Once upon a time, there was a young boy who had a special power. He used his power to help people in need and protect the innocent. He became known as the superhero, and he used his powers to fight evil and protect the world.

Instruction 3:

Tell me about alpacas.

LLaMA-Adapter: Alpacas are members of the camelid family and are native to the Andes Mountains of South America. They are typically found in herds of 10-20 animals and are known for their gentle nature and their ability to thrive in harsh climates. Alpacas are typically white or grey in color and have long, silky coats. They are also known for their ability to produce high-quality wool, which is used in the production of clothing and other products.

OTHER METHODS

Alpaca-LoRA: Alpacas are members of the camelid family and are native to the Andes Mountains of South America. They are known for their soft, luxurious fleece, which is used to make clothing, blankets, and other items. Alpacas are herbivores and graze on grasses and other plants. They are social animals and live in herds of up to 20 individuals.
Stanford Alpaca: Alpacas are small, fluffy animals related to camels and llamas. They are native to Peru and Bolivia, and were first domesticated around 5,000 years ago. They are kept mainly for their fine, soft fleece, which is used to make knitwear and other garments. Alpacas are herd animals and live in small family groups, led by an older male. They are highly social animals and can form strong bonds with their owners.
text-davinci-003: Alpacas are a domesticated species of South American camelid, similar to llamas. They are native to the Andes Mountains in South America and are kept as livestock for their soft, luxurious wool. Alpacas are social animals and live in herds of up to 20 individuals. They are typically kept in small herds of two to five animals, and are relatively easy to care for. Alpacas are herbivores and feed on grass, hay, and other vegetation. They are also known for their gentle and friendly nature, making them popular as pets.

Instruction 4:

Tell me about the president of Mexico in 2019.

LLaMA-Adapter: The president of Mexico in 2019 was Andrés Manuel López Obrador, who took office on December 1st, 2018. He is a member of the left-wing party Morena and is known for his populist policies. He has focused on tackling corruption, reducing poverty, and improving the country's infrastructure. He has also been a vocal critic of the United States' immigration policies.

OTHER METHODS

Alpaca-LoRA: The president of Mexico in 2019 was Andrés Manuel López Obrador, who took office on December 1, 2018. He is a member of the National Regeneration Movement (MORENA) political party and is the first left-wing president of Mexico since 1946. He is known for his anti-corruption and anti-neoliberal policies, as well as his commitment to improving the living conditions of the Mexican people.
Stanford Alpaca: In 2019, Andres Manuel Lopez Obrador was sworn in as President of Mexico. He is the country’s 58th president and the first from the left-wing Morena party. He was elected in 2018 on a promise to combat corruption and reduce poverty and has faced criticism for his handling of the economy, crime and the country’s relationship with the United States. He is also known for his ambitious social welfare programs and has been praised for raising the minimum wage and providing aid to low-income families. He is seen as a champion of democracy and human rights in Mexico.
text-davinci-003: The president of Mexico in 2019 is Andrés Manuel López Obrador, commonly known as AMLO. He is a Mexican politician who has been the President of Mexico since December 1, 2018. He is a member of the National Regeneration Movement (MORENA) party and previously served as the Head of Government of Mexico City from 2000 to 2005. He is a left-wing populist and has promised to reduce corruption and inequality in Mexico. He has also promised to increase public spending on social programs and infrastructure projects.

Instruction 5:

Write a Python program that prints the first 10 Fibonacci numbers.

LLaMA-Adapter:

def fibonacci():
    a = 0
    b = 1
    for i in range(10):
        a, b = b, a + b
        print(a, b)
    print("Fibonacci numbers: ", a, b)

fibonacci()

OTHER METHODS

Alpaca-LoRA:

def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

print(fibonacci(10))

Stanford Alpaca:

numbers = [0, 1]

for i in range(2, 11):
    numbers.append(numbers[i-2] + numbers[i-1])

print(numbers)
# Output: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

text-davinci-003:

# Fibonacci numbers are the numbers in the following integer sequence, called the Fibonacci sequence, and characterized by the fact that every number after the first two is the sum of the two preceding ones.

# Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, ...

# Program:

# Initialize the first two numbers of the Fibonacci sequence
a = 0
b = 1

# Print the first 10 numbers of the Fibonacci sequence
print(a)
print(b)

for i in range(2,10):
    c = a + b
    a = b
    b = c
    print(c)

Instruction 6:

Write a program that prints the numbers from 1 to 100. But for multiples of three print 'Fizz' instead of the number and for the multiples of five print 'Buzz'. For numbers which are multiples of both three and five print 'FizzBuzz'.

LLaMA-Adapter:

def print_numbers(n):
    for i in range(1, 101):
        if i % 3 == 0 and i % 5 == 0:
            print("FizzBuzz")
        elif i % 3 == 0:
            print("Fizz")
        elif i % 5 == 0:
            print("Buzz")
        else:
            print(i)

OTHER METHODS

Alpaca-LoRA:

for i in range(1, 101):
    if i % 3 == 0 and i % 5 == 0:
        print("FizzBuzz")
    elif i % 3 == 0:
        print("Fizz")
    elif i % 5 == 0:
        print("Buzz")
    else:
        print(i)

Stanford Alpaca:

for i in range(1, 101):
    if i % 3 == 0 and i % 5 == 0:
        print('FizzBuzz')
    elif i % 3 == 0:
        print('Fizz')
    elif i % 5 == 0:
        print('Buzz')
    else:
        print(i)

text-davinci-003:

for (let i = 1; i <= 100; i++) {
  if (i % 3 === 0 && i % 5 === 0) {
    console.log("FizzBuzz");
  } else if (i % 3 === 0) {
    console.log("Fizz");
  } else if (i % 5 === 0) {
    console.log("Buzz");
  } else {
    console.log(i);
  }
}

Contributors

Peng Gao, Jiaming Han, Renrui Zhang, Aojun Zhou

Hiring Announcement

🔥 We are hiring interns, postdocs, and full-time researchers at the General Vision Group, Shanghai AI Lab, with a focus on multi-modality and vision foundation models. If you are interested, please contact gaopengcuhk@gmail.com.

Citation

If you find our LLaMA-Adapter code and paper useful, please kindly cite:

@article{zhang2023llamaadapter,
  title = {LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention},
  author={Zhang, Renrui and Han, Jiaming and Liu, Chris and Gao, Peng and Zhou, Aojun and Hu, Xiangfei and Yan, Shilin and Lu, Pan and Li, Hongsheng and Qiao, Yu},
  journal={arXiv preprint arXiv:2303.16199},
  year={2023}
}

If you find our LLaMA-Adapter V2 code and paper useful, please kindly cite:

@article{gao2023llamaadapterv2,
  title = {LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model},
  author={Gao, Peng and Han, Jiaming and Zhang, Renrui and Lin, Ziyi and Geng, Shijie and Zhou, Aojun and Zhang, Wei and Lu, Pan and He, Conghui and Yue, Xiangyu and Li, Hongsheng and Qiao, Yu},
  journal={arXiv preprint arXiv:2304.15010},
  year={2023}
}

Acknowledgement

This repo benefits from LLaMA, Stanford Alpaca, and Alpaca-Lora. Thanks for their wonderful works.

KillianLucas/open-interpreter

2023-09-10T02:02:51Z

OpenAI's Code Interpreter in your terminal, running locally

● Open Interpreter

Let language models run code on your computer.
An open-source, locally running implementation of OpenAI's Code Interpreter.

Get early access to the desktop application.

pip install open-interpreter

interpreter

This provides a natural-language interface to your computer's general-purpose capabilities:

Create and edit photos, videos, PDFs, etc.
Control a Chrome browser to perform research
Plot, clean, and analyze large datasets
...etc.

⚠️ Note: You'll be asked to approve code before it's run.

Demo

https://github.com/KillianLucas/open-interpreter/assets/63927363/37152071-680d-4423-9af3-64836a6f7b60

An interactive demo is also available on Google Colab:

Quick Start

pip install open-interpreter

Terminal

After installation, simply run interpreter:

interpreter

Python

import interpreter

interpreter.chat("Plot AAPL and META's normalized stock prices") # Executes a single command
interpreter.chat() # Starts an interactive chat

Comparison to ChatGPT's Code Interpreter

OpenAI's release of Code Interpreter with GPT-4 presents a fantastic opportunity to accomplish real-world tasks with ChatGPT.

However, OpenAI's service is hosted, closed-source, and heavily restricted:

No internet access.
Limited set of pre-installed packages.
100 MB maximum upload, 120.0 second runtime limit.
State is cleared (along with any generated files or links) when the environment dies.

Open Interpreter overcomes these limitations by running on your local environment. It has full access to the internet, isn't restricted by time or file size, and can utilize any package or library.

This combines the power of GPT-4's Code Interpreter with the flexibility of your local development environment.

Commands

Interactive Chat

To start an interactive chat in your terminal, either run interpreter from the command line:

interpreter

Or interpreter.chat() from a .py file:

interpreter.chat()

Programmatic Chat

For more precise control, you can pass messages directly to .chat(message):

interpreter.chat("Add subtitles to all videos in /videos.")

# ... Streams output to your terminal, completes task ...

interpreter.chat("These look great but can you make the subtitles bigger?")

# ...

Start a New Chat

In Python, Open Interpreter remembers conversation history. If you want to start fresh, you can reset it:

interpreter.reset()

Save and Restore Chats

interpreter.chat() returns a List of messages when return_messages=True, which can be used to resume a conversation with interpreter.load(messages):

messages = interpreter.chat("My name is Killian.", return_messages=True) # Save messages to 'messages'
interpreter.reset() # Reset interpreter ("Killian" will be forgotten)

interpreter.load(messages) # Resume chat from 'messages' ("Killian" will be remembered)

Customize System Message

You can inspect and configure Open Interpreter's system message to extend its functionality, modify permissions, or give it more context.

interpreter.system_message += """
Run shell commands with -y so the user doesn't have to confirm them.
"""
print(interpreter.system_message)

Change the Model

ⓘ Issues running locally? Read our new GPU setup guide and Windows setup guide.

You can run interpreter in local mode from the command line to use Code Llama:

interpreter --local

For gpt-3.5-turbo, use fast mode:

interpreter --fast

In Python, you will need to set the model manually:

interpreter.model = "gpt-3.5-turbo"

Azure Support

To connect to an Azure deployment, the --use-azure flag will walk you through setting this up:

interpreter --use-azure

In Python, set the following variables:

interpreter.use_azure = True
interpreter.api_key = "your_openai_api_key"
interpreter.azure_api_base = "your_azure_api_base"
interpreter.azure_api_version = "your_azure_api_version"
interpreter.azure_deployment_name = "your_azure_deployment_name"
interpreter.azure_api_type = "azure"

Debug mode

To help contributors inspect Open Interpreter, --debug mode is highly verbose.

You can activate debug mode by using it's flag (interpreter --debug), or mid-chat:

$ interpreter
...
> %debug # <- Turns on debug mode

Configuration with .env

Open Interpreter allows you to set default behaviors using a .env file. This provides a flexible way to configure the interpreter without changing command-line arguments every time.

Here's a sample .env configuration:

INTERPRETER_CLI_AUTO_RUN=False
INTERPRETER_CLI_FAST_MODE=False
INTERPRETER_CLI_LOCAL_RUN=False
INTERPRETER_CLI_DEBUG=False
INTERPRETER_CLI_USE_AZURE=False

INTERPRETER_CLI_AUTO_RUN: If set to True, the interpreter will execute code without user confirmation.
INTERPRETER_CLI_FAST_MODE: If set to True, the interpreter will use gpt-3.5-turbo instead of gpt-4.
INTERPRETER_CLI_LOCAL_RUN: If set to True, the interpreter will run fully locally with Code Llama.
INTERPRETER_CLI_DEBUG: If set to True, the interpreter will print extra debugging information.
INTERPRETER_CLI_USE_AZURE: If set to True, the interpreter will use Azure OpenAI Services.

You can modify these values in the .env file to change the default behavior of the Open Interpreter.

Safety Notice

Since generated code is executed in your local environment, it can interact with your files and system settings, potentially leading to unexpected outcomes like data loss or security risks.

⚠️ Open Interpreter will ask for user confirmation before executing code.

You can run interpreter -y or set interpreter.auto_run = True to bypass this confirmation, in which case:

Be cautious when requesting commands that modify files or system settings.
Watch Open Interpreter like a self-driving car, and be prepared to end the process by closing your terminal.
Consider running Open Interpreter in a restricted environment like Google Colab or Replit. These environments are more isolated, reducing the risks associated with executing arbitrary code.

How Does it Work?

Open Interpreter equips a function-calling language model with an exec() function, which accepts a language (like "python" or "javascript") and code to run.

We then stream the model's messages, code, and your system's outputs to the terminal as Markdown.

Contributing

Thank you for your interest in contributing! We welcome involvement from the community.

Please see our Contributing Guidelines for more details on how to get involved.

License

Open Interpreter is licensed under the MIT License. You are permitted to use, copy, modify, distribute, sublicense and sell copies of the software.

Note: This software is not affiliated with OpenAI.

Having access to a junior programmer working at the speed of your fingertips ... can make new workflows effortless and efficient, as well as open the benefits of programming to new audiences.

— OpenAI's Code Interpreter Release

graphdeco-inria/gaussian-splatting

2023-09-10T02:02:51Z

Original reference implementation of "3D Gaussian Splatting for Real-Time Radiance Field Rendering"

3D Gaussian Splatting for Real-Time Radiance Field Rendering

This repository contains the official authors implementation associated with the paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering", which can be found here. We further provide the reference images used to create the error metrics reported in the paper, as well as recently created, pre-trained models.

Abstract: Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (≥ 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

BibTeX

@Article{kerbl3Dgaussians,
      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
      journal      = {ACM Transactions on Graphics},
      number       = {4},
      volume       = {42},
      month        = {July},
      year         = {2023},
      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
}

Funding and Acknowledgments

This research was funded by the ERC Advanced grant FUNGRAPH No 788065. The authors are grateful to Adobe for generous donations, the OPAL infrastructure from Université Côte d’Azur and for the HPC resources from GENCI–IDRIS (Grant 2022-AD011013409). The authors thank the anonymous reviewers for their valuable feedback, P. Hedman and A. Tewari for proofreading earlier drafts also T. Müller, A. Yu and S. Fridovich-Keil for helping with the comparisons.

Step-by-step Tutorial

Jonathan Stephens made a fantastic step-by-step tutorial for setting up Gaussian Splatting on your machine, along with instructions for creating usable datasets from videos. If the instructions below are too dry for you, go ahead and check it out here.

Colab

User camenduru was kind enough to provide a Colab template that uses this repo's source (status: August 2023!) for quick and easy access to the method. Please check it out here.

Cloning the Repository

The repository contains submodules, thus please check it out with

# SSH
git clone git@github.com:graphdeco-inria/gaussian-splatting.git --recursive

# HTTPS
git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive

Overview

The codebase has 4 main components:

A PyTorch-based optimizer to produce a 3D Gaussian model from SfM inputs
A network viewer that allows to connect to and visualize the optimization process
An OpenGL-based real-time viewer to render trained models in real-time.
A script to help you turn your own images into optimization-ready SfM data sets

The components have different requirements w.r.t. both hardware and software. They have been tested on Windows 10 and Ubuntu Linux 22.04. Instructions for setting up and running each of them are found in the sections below.

Optimizer

The optimizer uses PyTorch and CUDA extensions in a Python environment to produce trained models.

Hardware Requirements

CUDA-ready GPU with Compute Capability 7.0+
24 GB VRAM (to train to paper evaluation quality)
Please see FAQ for smaller VRAM configurations

Software Requirements

Conda (recommended for easy setup)
C++ Compiler for PyTorch extensions (we used Visual Studio 2019 for Windows)
CUDA SDK 11 for PyTorch extensions, install after Visual Studio (we used 11.8, known issues with 11.6)
C++ Compiler and CUDA SDK must be compatible

Setup

Local Setup

Our default, provided install method is based on Conda package and environment management:

SET DISTUTILS_USE_SDK=1 # Windows only
conda env create --file environment.yml
conda activate gaussian_splatting

Please note that this process assumes that you have CUDA SDK 11 installed, not 12. For modifications, see below.

Tip: Downloading packages and creating a new environment with Conda can require a significant amount of disk space. By default, Conda will use the main system hard drive. You can avoid this by specifying a different package download location and an environment on a different drive:

conda config --add pkgs_dirs <Drive>/<pkg_path>
conda env create --file environment.yml --prefix <Drive>/<env_path>/gaussian_splatting
conda activate <Drive>/<env_path>/gaussian_splatting

Modifications

If you can afford the disk space, we recommend using our environment files for setting up a training environment identical to ours. If you want to make modifications, please note that major version changes might affect the results of our method. However, our (limited) experiments suggest that the codebase works just fine inside a more up-to-date environment (Python 3.8, PyTorch 2.0.0, CUDA 12). Make sure to create an environment where PyTorch and its CUDA runtime version match and the installed CUDA SDK has no major version difference with PyTorch's CUDA version.

Known Issues

Some users experience problems building the submodules on Windows (cl.exe: File not found or similar). Please consider the workaround for this problem from the FAQ.

Running

To run the optimizer, simply use

python train.py -s <path to COLMAP or NeRF Synthetic dataset>

Command Line Arguments for train.py

--source_path / -s

Path to the source directory containing a COLMAP or Synthetic NeRF data set.

--model_path / -m

Path where the trained model should be stored (output/<random> by default).

--images / -i

Alternative subdirectory for COLMAP images (images by default).

--eval

Add this flag to use a MipNeRF360-style training/test split for evaluation.

--resolution / -r

Specifies resolution of the loaded images before training. If provided 1, 2, 4 or 8, uses original, 1/2, 1/4 or 1/8 resolution, respectively. For all other values, rescales the width to the given number while maintaining image aspect. If not set and input image width exceeds 1.6K pixels, inputs are automatically rescaled to this target.

--data_device

Specifies where to put the source image data, cuda by default, recommended to use cpu if training on large/high-resolution dataset, will reduce VRAM consumption, but slightly slow down training. Thanks to HrsPythonix.

--white_background / -w

Add this flag to use white background instead of black (default), e.g., for evaluation of NeRF Synthetic dataset.

--sh_degree

Order of spherical harmonics to be used (no larger than 3). 3 by default.

--convert_SHs_python

Flag to make pipeline compute forward and backward of SHs with PyTorch instead of ours.

--convert_cov3D_python

Flag to make pipeline compute forward and backward of the 3D covariance with PyTorch instead of ours.

--debug

Enables debug mode if you experience erros. If the rasterizer fails, a dump file is created that you may forward to us in an issue so we can take a look.

--debug_from

Debugging is slow. You may specify an iteration (starting from 0) after which the above debugging becomes active.

--iterations

Number of total iterations to train for, 30_000 by default.

--ip

IP to start GUI server on, 127.0.0.1 by default.

--port

Port to use for GUI server, 6009 by default.

--test_iterations

Space-separated iterations at which the training script computes L1 and PSNR over test set, 7000 30000 by default.

--save_iterations

Space-separated iterations at which the training script saves the Gaussian model, 7000 30000 <iterations> by default.

--checkpoint_iterations

Space-separated iterations at which to store a checkpoint for continuing later, saved in the model directory.

--start_checkpoint

Path to a saved checkpoint to continue training from.

--quiet

Flag to omit any text written to standard out pipe.

--feature_lr

Spherical harmonics features learning rate, 0.0025 by default.

--opacity_lr

Opacity learning rate, 0.05 by default.

--scaling_lr

Scaling learning rate, 0.005 by default.

--rotation_lr

Rotation learning rate, 0.001 by default.

--position_lr_max_steps

Number of steps (from 0) where position learning rate goes from initial to final. 30_000 by default.

--position_lr_init

Initial 3D position learning rate, 0.00016 by default.

--position_lr_final

Final 3D position learning rate, 0.0000016 by default.

--position_lr_delay_mult

Position learning rate multiplier (cf. Plenoxels), 0.01 by default.

--densify_from_iter

Iteration where densification starts, 500 by default.

--densify_until_iter

Iteration where densification stops, 15_000 by default.

--densify_grad_threshold

Limit that decides if points should be densified based on 2D position gradient, 0.0002 by default.

--densification_interal

How frequently to densify, 100 (every 100 iterations) by default.

--opacity_reset_interval

How frequently to reset opacity, 3_000 by default.

--lambda_dssim

Influence of SSIM on total loss from 0 to 1, 0.2 by default.

--percent_dense

Percentage of scene extent (0--1) a point must exceed to be forcibly densified, 0.01 by default.

Note that similar to MipNeRF360, we target images at resolutions in the 1-1.6K pixel range. For convenience, arbitrary-size inputs can be passed and will be automatically resized if their width exceeds 1600 pixels. We recommend to keep this behavior, but you may force training to use your higher-resolution images by setting -r 1.

The MipNeRF360 scenes are hosted by the paper authors here. You can find our SfM data sets for Tanks&Temples and Deep Blending here. If you do not provide an output model directory (-m), trained models are written to folders with randomized unique names inside the output directory. At this point, the trained models may be viewed with the real-time viewer (see further below).

Evaluation

By default, the trained models use all available images in the dataset. To train them while withholding a test set for evaluation, use the --eval flag. This way, you can render training/test sets and produce error metrics as follows:

python train.py -s <path to COLMAP or NeRF Synthetic dataset> --eval # Train with train/test split
python render.py -m <path to trained model> # Generate renderings
python metrics.py -m <path to trained model> # Compute error metrics on renderings

If you want to evaluate our pre-trained models, you will have to download the corresponding source data sets and indicate their location to render.py with an additional --source_path/-s flag. Note: The pre-trained models were created with the release codebase. This code base has been cleaned up and includes bugfixes, hence the metrics you get from evaluating them will differ from those in the paper.

python render.py -m <path to pre-trained model> -s <path to COLMAP dataset>
python metrics.py -m <path to pre-trained model>

Command Line Arguments for render.py

--model_path / -m

Path to the trained model directory you want to create renderings for.

--skip_train

Flag to skip rendering the training set.

--skip_test

Flag to skip rendering the test set.

--quiet

Flag to omit any text written to standard out pipe.

The below parameters will be read automatically from the model path, based on what was used for training. However, you may override them by providing them explicitly on the command line.

--source_path / -s

Path to the source directory containing a COLMAP or Synthetic NeRF data set.

--images / -i

Alternative subdirectory for COLMAP images (images by default).

--eval

Add this flag to use a MipNeRF360-style training/test split for evaluation.

--resolution / -r

Changes the resolution of the loaded images before training. If provided 1, 2, 4 or 8, uses original, 1/2, 1/4 or 1/8 resolution, respectively. For all other values, rescales the width to the given number while maintaining image aspect. 1 by default.

--white_background / -w

Add this flag to use white background instead of black (default), e.g., for evaluation of NeRF Synthetic dataset.

--convert_SHs_python

Flag to make pipeline render with computed SHs from PyTorch instead of ours.

--convert_cov3D_python

Flag to make pipeline render with computed 3D covariance from PyTorch instead of ours.

Command Line Arguments for metrics.py

--model_paths / -m

Space-separated list of model paths for which metrics should be computed.

We further provide the full_eval.py script. This script specifies the routine used in our evaluation and demonstrates the use of some additional parameters, e.g., --images (-i) to define alternative image directories within COLMAP data sets. If you have downloaded and extracted all the training data, you can run it like this:

python full_eval.py -m360 <mipnerf360 folder> -tat <tanks and temples folder> -db <deep blending folder>

In the current version, this process takes about 7h on our reference machine containing an A6000. If you want to do the full evaluation on our pre-trained models, you can specify their download location and skip training.

python full_eval.py -o <directory with pretrained models> --skip_training -m360 <mipnerf360 folder> -tat <tanks and temples folder> -db <deep blending folder>

If you want to compute the metrics on our paper's evaluation images, you can also skip rendering. In this case it is not necessary to provide the source datasets. You can compute metrics for multiple image sets at a time.

python full_eval.py -m <directory with evaluation images>/garden ... --skip_training --skip_rendering

Command Line Arguments for full_eval.py

--skip_training

Flag to skip training stage.

--skip_rendering

Flag to skip rendering stage.

--skip_metrics

Flag to skip metrics calculation stage.

--output_path

Directory to put renderings and results in, ./eval by default, set to pre-trained model location if evaluating them.

--mipnerf360 / -m360

Path to MipNeRF360 source datasets, required if training or rendering.

--tanksandtemples / -tat

Path to Tanks&Temples source datasets, required if training or rendering.

--deepblending / -db

Path to Deep Blending source datasets, required if training or rendering.

Interactive Viewers

We provide two interactive viewers for our method: remote and real-time. Our viewing solutions are based on the SIBR framework, developed by the GRAPHDECO group for several novel-view synthesis projects.

Hardware Requirements

OpenGL 4.5-ready GPU and drivers (or latest MESA software)
4 GB VRAM recommended
CUDA-ready GPU with Compute Capability 7.0+ (only for Real-Time Viewer)

Software Requirements

Visual Studio or g++, not Clang (we used Visual Studio 2019 for Windows)
CUDA SDK 11, install after Visual Studio (we used 11.8)
CMake (recent version, we used 3.24)
7zip (only on Windows)

Pre-built Windows Binaries

We provide pre-built binaries for Windows here. We recommend using them on Windows for an efficient setup, since the building of SIBR involves several external dependencies that must be downloaded and compiled on-the-fly.

Installation from Source

If you cloned with submodules (e.g., using --recursive), the source code for the viewers is found in SIBR_viewers. The network viewer runs within the SIBR framework for Image-based Rendering applications.

Windows

CMake should take care of your dependencies.

cd SIBR_viewers
cmake -Bbuild .
cmake --build build --target install --config RelWithDebInfo

You may specify a different configuration, e.g. Debug if you need more control during development.

Ubuntu 22.04

You will need to install a few dependencies before running the project setup.

# Dependencies
sudo apt install -y libglew-dev libassimp-dev libboost-all-dev libgtk-3-dev libopencv-dev libglfw3-dev libavdevice-dev libavcodec-dev libeigen3-dev libxxf86vm-dev libembree-dev
# Project setup
cd SIBR_viewers
cmake -Bbuild . -DCMAKE_BUILD_TYPE=Release # add -G Ninja to build faster
cmake --build build -j24 --target install

Ubuntu 20.04

Backwards compatibility with Focal Fossa is not fully tested, but building SIBR with CMake should still work after invoking

git checkout fossa_compatibility

Navigation in SIBR Viewers

The SIBR interface provides several methods of navigating the scene. By default, you will be started with an FPS navigator, which you can control with W, A, S, D, Q, E for camera translation and I, K, J, L, U, O for rotation. Alternatively, you may want to use a Trackball-style navigator (select from the floating menu). You can also snap to a camera from the data set with the Snap to button or find the closest camera with Snap to closest. The floating menues also allow you to change the navigation speed. You can use the Scaling Modifier to control the size of the displayed Gaussians, or show the initial point cloud.

Running the Network Viewer

https://github.com/graphdeco-inria/gaussian-splatting/assets/40643808/90a2e4d3-cf2e-4633-b35f-bfe284e28ff7

After extracting or installing the viewers, you may run the compiled SIBR_remoteGaussian_app[_config] app in <SIBR install dir>/bin, e.g.:

./<SIBR install dir>/bin/SIBR_remoteGaussian_app

The network viewer allows you to connect to a running training process on the same or a different machine. If you are training on the same machine and OS, no command line parameters should be required: the optimizer communicates the location of the training data to the network viewer. By default, optimizer and network viewer will try to establish a connection on localhost on port 6009. You can change this behavior by providing matching --ip and --port parameters to both the optimizer and the network viewer. If for some reason the path used by the optimizer to find the training data is not reachable by the network viewer (e.g., due to them running on different (virtual) machines), you may specify an override location to the viewer by using -s <source path>.

Primary Command Line Arguments for Network Viewer

--path / -s

Argument to override model's path to source dataset.

--ip

IP to use for connection to a running training script.

--port

Port to use for connection to a running training script.

--rendering-size

Takes two space separated numbers to define the resolution at which network rendering occurs, 1200 width by default. Note that to enforce an aspect that differs from the input images, you need --force-aspect-ratio too.

--load_images

Flag to load source dataset images to be displayed in the top view for each camera.

Running the Real-Time Viewer

https://github.com/graphdeco-inria/gaussian-splatting/assets/40643808/0940547f-1d82-4c2f-a616-44eabbf0f816

After extracting or installing the viewers, you may run the compiled SIBR_gaussianViewer_app[_config] app in <SIBR install dir>/bin, e.g.:

./<SIBR install dir>/bin/SIBR_gaussianViewer_app -m <path to trained model>

It should suffice to provide the -m parameter pointing to a trained model directory. Alternatively, you can specify an override location for training input data using -s. To use a specific resolution other than the auto-chosen one, specify --rendering-size <width> <height>. Combine it with --force-aspect-ratio if you want the exact resolution and don't mind image distortion.

To unlock the full frame rate, please disable V-Sync on your machine and also in the application (Menu → Display). In a multi-GPU system (e.g., laptop) your OpenGL/Display GPU should be the same as your CUDA GPU (e.g., by setting the application's GPU preference on Windows, see below) for maximum performance.

In addition to the intial point cloud and the splats, you also have the option to visualize the Gaussians by rendering them as ellipsoids from the floating menu. SIBR has many other functionalities, please see the documentation for more details on the viewer, navigation options etc. There is also a Top View (available from the menu) that shows the placement of the input cameras and the original SfM point cloud; please note that Top View slows rendering when enabled. The real-time viewer also uses slightly more aggressive, fast culling, which can be toggled in the floating menu. If you ever encounter an issue that can be solved by turning fast culling off, please let us know.

Primary Command Line Arguments for Real-Time Viewer

--model-path / -m

Path to trained model.

--iteration

Specifies which of state to load if multiple are available. Defaults to latest available iteration.

--path / -s

Argument to override model's path to source dataset.

--rendering-size

Takes two space separated numbers to define the resolution at which real-time rendering occurs, 1200 width by default. Note that to enforce an aspect that differs from the input images, you need --force-aspect-ratio too.

--load_images

Flag to load source dataset images to be displayed in the top view for each camera.

--device

Index of CUDA device to use for rasterization if multiple are available, 0 by default.

--no_interop

Disables CUDA/GL interop forcibly. Use on systems that may not behave according to spec (e.g., WSL2 with MESA GL 4.5 software rendering).

Processing your own Scenes

Our COLMAP loaders expect the following dataset structure in the source path location:

<location>
|---images
|   |---<image 0>
|   |---<image 1>
|   |---...
|---sparse
    |---0
        |---cameras.bin
        |---images.bin
        |---points3D.bin

For rasterization, the camera models must be either a SIMPLE_PINHOLE or PINHOLE camera. We provide a converter script convert.py, to extract undistorted images and SfM information from input images. Optionally, you can use ImageMagick to resize the undistorted images. This rescaling is similar to MipNeRF360, i.e., it creates images with 1/2, 1/4 and 1/8 the original resolution in corresponding folders. To use them, please first install a recent version of COLMAP (ideally CUDA-powered) and ImageMagick. Put the images you want to use in a directory <location>/input.

<location>
|---input
    |---<image 0>
    |---<image 1>
    |---...

If you have COLMAP and ImageMagick on your system path, you can simply run

python convert.py -s <location> [--resize] #If not resizing, ImageMagick is not needed

Alternatively, you can use the optional parameters --colmap_executable and --magick_executable to point to the respective paths. Please note that on Windows, the executable should point to the COLMAP .bat file that takes care of setting the execution environment. Once done, <location> will contain the expected COLMAP data set structure with undistorted, resized input images, in addition to your original images and some temporary (distorted) data in the directory distorted.

If you have your own COLMAP dataset without undistortion (e.g., using OPENCV camera), you can try to just run the last part of the script: Put the images in input and the COLMAP info in a subdirectory distorted:

<location>
|---input
|   |---<image 0>
|   |---<image 1>
|   |---...
|---distorted
    |---database.db
    |---sparse
        |---0
            |---...

Then run

python convert.py -s <location> --skip_matching [--resize] #If not resizing, ImageMagick is not needed

Command Line Arguments for convert.py

--no_gpu

Flag to avoid using GPU in COLMAP.

--skip_matching

Flag to indicate that COLMAP info is available for images.

--source_path / -s

Location of the inputs.

--camera

Which camera model to use for the early matching steps, OPENCV by default.

--resize

Flag for creating resized versions of input images.

--colmap_executable

Path to the COLMAP executable (.bat on Windows).

--magick_executable

Path to the ImageMagick executable.

FAQ

Where do I get data sets, e.g., those referenced in full_eval.py? The MipNeRF360 data set is provided by the authors of the original paper on the project site. Note that two of the data sets cannot be openly shared and require you to consult the authors directly. For Tanks&Temples and Deep Blending, please use the download links provided at the top of the page. Alternatively, you may access the cloned data (status: August 2023!) from HuggingFace
How can I use this for a much larger dataset, like a city district? The current method was not designed for these, but given enough memory, it should work out. However, the approach can struggle in multi-scale detail scenes (extreme close-ups, mixed with far-away shots). This is usually the case in, e.g., driving data sets (cars close up, buildings far away). For such scenes, you can lower the --position_lr_init, --position_lr_final and --scaling_lr (x0.3, x0.1, ...). The more extensive the scene, the lower these values should be. Below, we use default learning rates (left) and --position_lr_init 0.000016 --scaling_lr 0.001" (right).

I'm on Windows and I can't manage to build the submodules, what do I do? Consider following the steps in the excellent video tutorial here, hopefully they should help. The order in which the steps are done is important! Alternatively, consider using the linked Colab template.
It still doesn't work. It says something about cl.exe. What do I do? User Henry Pearce found a workaround. You can you try adding the visual studio path to your environment variables (your version number might differ); C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64 Then make sure you start a new conda prompt and cd to your repo location and try this;

conda activate gaussian_splatting
cd <dir_to_repo>/gaussian-splatting
pip install submodules\diff-gaussian-rasterization
pip install submodules\simple-knn

I'm on macOS/Puppy Linux/Greenhat and I can't manage to build, what do I do? Sorry, we can't provide support for platforms outside of the ones we list in this README. Consider using the linked Colab template.
I don't have 24 GB of VRAM for training, what do I do? The VRAM consumption is determined by the number of points that are being optimized, which increases over time. If you only want to train to 7k iterations, you will need significantly less. To do the full training routine and avoid running out of memory, you can increase the --densify_grad_threshold, --densification_interval or reduce the value of --densify_until_iter. Note however that this will affect the quality of the result. Also try setting --test_iterations to -1 to avoid memory spikes during testing. If --densify_grad_threshold is very high, no densification should occur and training should complete if the scene itself loads successfully.
24 GB of VRAM for reference quality training is still a lot! Can't we do it with less? Yes, most likely. By our calculations it should be possible with way less memory (~8GB). If we can find the time we will try to achieve this. If some PyTorch veteran out there wants to tackle this, we look forward to your pull request!
How can I use the differentiable Gaussian rasterizer for my own project? Easy, it is included in this repo as a submodule diff-gaussian-rasterization. Feel free to check out and install the package. It's not really documented, but using it from the Python side is very straightforward (cf. gaussian_renderer/__init__.py).
Wait, but <insert feature> isn't optimized and could be much better? There are several parts we didn't even have time to think about improving (yet). The performance you get with this prototype is probably a rather slow baseline for what is physically possible.
Something is broken, how did this happen? We tried hard to provide a solid and comprehensible basis to make use of the paper's method. We have refactored the code quite a bit, but we have limited capacity to test all possible usage scenarios. Thus, if part of the website, the code or the performance is lacking, please create an issue. If we find the time, we will do our best to address it.

facebookresearch/nougat

2023-09-10T02:02:51Z

Implementation of Nougat Neural Optical Understanding for Academic Documents

Nougat: Neural Optical Understanding for Academic Documents

This is the official repository for Nougat, the academic document PDF parser that understands LaTeX math and tables.

Project page: https://facebookresearch.github.io/nougat/

Install

From pip:

pip install nougat-ocr

From repository:

pip install git+https://github.com/facebookresearch/nougat

Note, on Windows: If you want to utilize a GPU, make sure you first install the correct PyTorch version. Follow instructions here

There are extra dependencies if you want to call the model from an API or generate a dataset. Install via

pip install "nougat-ocr[api]" or pip install "nougat-ocr[dataset]"

Get prediction for a PDF

CLI

To get predictions for a PDF run

$ nougat path/to/file.pdf -o output_directory

usage: nougat [-h] [--batchsize BATCHSIZE] [--checkpoint CHECKPOINT] [--out OUT] [--recompute] [--markdown] pdf [pdf ...]

positional arguments:
  pdf                   PDF(s) to process.

options:
  -h, --help            show this help message and exit
  --batchsize BATCHSIZE, -b BATCHSIZE
                        Batch size to use.
  --checkpoint CHECKPOINT, -c CHECKPOINT
                        Path to checkpoint directory.
  --out OUT, -o OUT     Output directory.
  --recompute           Recompute already computed PDF, discarding previous predictions.
  --markdown            Add postprocessing step for markdown compatibility.

In the output directory every PDF will be saved as a .mmd file, the lightweight markup language, mostly compatible with Mathpix Markdown (we make use of the LaTeX tables).

API

With the extra dependencies you use app.py to start an API. Call

$ nougat_api

To get a prediction of a PDF file by making a POST request to http://127.0.0.1:8503/predict/. It also accepts parameters start and stop to limit the computation to select page numbers (boundaries are included).

The response is a string with the markdown text of the document.

curl -X 'POST' \
  'http://127.0.0.1:8503/predict/' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'file=@<PDFFILE.pdf>;type=application/pdf'

To use the limit the conversion to pages 1 to 5, ue the start/stop parameters in the request URL: http://127.0.0.1:8503/predict/?start=1&stop=5

Dataset

Generate dataset

To generate a dataset you need

A directory containing the PDFs
A directory containing the .html files (processed .tex files by LaTeXML) with the same folder structure
A binary file of pdffigures2 and a corresponding environment variable export PDFFIGURES_PATH="/path/to/binary.jar"

Next run

python -m nougat.dataset.split_htmls_to_pages --html path/html/root --pdfs path/pdf/root --out path/paired/output --figure path/pdffigures/outputs

Additional arguments include

Argument	Description
`--recompute`	recompute all splits
`--markdown MARKDOWN`	Markdown output dir
`--workers WORKERS`	How many processes to use
`--dpi DPI`	What resolution the pages will be saved at
`--timeout TIMEOUT`	max time per paper in seconds
`--tesseract`	Tesseract OCR prediction for each page

Finally create a jsonl file that contains all the image paths, markdown text and meta information.

python -m nougat.dataset.create_index --dir path/paired/output --out index.jsonl

For each jsonl file you also need to generate a seek map for faster data loading:

python -m nougat.dataset.gen_seek file.jsonl

The resulting directory structure can look as follows:

root/
├── images
├── train.jsonl
├── train.seek.map
├── test.jsonl
├── test.seek.map
├── validation.jsonl
└── validation.seek.map

Note that the .mmd and .json files in the path/paired/output (here images) are no longer required. This can be useful for pushing to a S3 bucket by halving the amount of files.

Training

To train or fine tune a Nougat model, run

python train.py --config config/train_nougat.yaml

Evaluation

Run

python test.py --checkpoint path/to/checkpoint --dataset path/to/test.jsonl --save_path path/to/results.json

To get the results for the different text modalities, run

python -m nougat.metrics path/to/results.json

Citation

@misc{blecher2023nougat,
      title={Nougat: Neural Optical Understanding for Academic Documents}, 
      author={Lukas Blecher and Guillem Cucurull and Thomas Scialom and Robert Stojnic},
      year={2023},
      eprint={2308.13418},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Acknowledgments

This repository builds on top of the Donut repository.

License

Nougat codebase is licensed under MIT.

Nougat model weights are licensed under CC-BY-NC.

duckdb/duckdb-r

2023-09-11T01:39:46Z

duckdb R package

Installation from CRAN

install.packages("duckdb")

Building

To build the bleeding edge of duckdb-r, you can clone this repository and run

~duckdb-r: R CMD INSTALL .

If you wish to test new duckdb functionality with duckdb-r, make sure your clones of duckdb-r and duckdb share the same parent directory. Then run the following commands

~ (cd duckdb && git checkout {{desired_branch}})
~ (cd ducdkb-r && ./vendor.sh)
~ (cd duckdb-r && R CMD INSTALL .)

It helps if both the duckdb directory and duckdb-r directory are clean. If you encounter linker errors, merge both duckdb-r and duckdb with their respective main branches.

Dependencies

To build the R package, you first need to install the dependencies:

# install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))
pak::pak()

cedev935/shiny-python

2023-09-11T01:39:46Z

shiny

Easily build rich and productive interactive web apps in R — no HTML/CSS/JavaScript required.

Features

An intuitive and extensible reactive programming model which makes it easy to transform existing R code into a "live app" where outputs automatically react to new user input.
- Compared to event-based programming, reactivity allows Shiny to do the minimum amount of work when input(s) change, and allows humans to more easily reason about complex MVC logic.
A prebuilt set of highly sophisticated, customizable, and easy-to-use widgets (e.g., plots, tables, sliders, dropdowns, date pickers, and more).
An attractive default look based on Bootstrap which can also be easily customized with the bslib package or avoided entirely with more direct R bindings to HTML/CSS/JavaScript.
Seamless integration with R Markdown, making it easy to embed numerous applications natively within a larger dynamic document.
Tools for improving and monitoring performance, including native support for async programming, caching, load testing, and more.
Modules: a framework for reducing code duplication and complexity.
An ability to bookmark application state and/or generate code to reproduce output(s).
A rich ecosystem of extension packages for more custom widgets, input validation, unit testing, and more.

Installation

To install the stable version from CRAN:

install.packages("shiny")

Getting Started

Once installed, load the library and run an example:

library(shiny)
# Launches an app, with the app's source code included
runExample("06_tabsets")
# Lists more prepackaged examples
runExample()

For more examples and inspiration, check out the Shiny User Gallery.

For help with learning fundamental Shiny programming concepts, check out the Mastering Shiny book and the Shiny Tutorial. The former is currently more up-to-date with modern Shiny features, whereas the latter takes a deeper, more visual, dive into fundamental concepts.

Join the conversation

If you want to chat about Shiny, meet other developers, or help us decide what to work on next, join us on Discord.

Getting Help

To ask a question about Shiny, please use the RStudio Community website.

For bug reports, please use the issue tracker and also keep in mind that by writing a good bug report, you're more likely to get help with your problem.

Contributing

We welcome contributions to the shiny package. Please see our CONTRIBUTING.md file for detailed guidelines of how to contribute.

License

The shiny package as a whole is licensed under the GPLv3. See the LICENSE file for more details.

R version support

Shiny is supported on the latest release version of R, as well as the previous four minor release versions of R. For example, if the latest release R version is 4.1, then that version is supported, as well as 4.0, 3.6, 3.5, and 3.4.

mrjoh3/c3

2023-09-10T02:03:19Z

c3 HTMLWidget Ploting

c3

The c3 package is a wrapper, or htmlwidget, for the C3 javascript charting library by Masayuki Tanaka. You will find this package useful if you are wanting to create a chart using R and embedding it in a Rmarkdown document or Shiny App.

The C3 library is very versatile and includes a lot of options. Currently this package wraps most of the C3 options object.

Installation

You probably already guessed this bit.


install.packages('c3')

# OR

devtools::install_github("mrjoh3/c3")

Basic Usage

The c3 package is intended to be as simple and lightweight as possible. As a starting point the data input must be a data.frame or tibble with several options.

Where no options are supplied a simple line plot is produced by default. Where no x-axis is defined the plots are sequential. Date x-axis can be parsed with not additional setting if in the format %Y-%m-%d (ie ‘2014-01-01’)


library(c3)

data <- data.frame(a = abs(rnorm(20) * 10),
                   b = abs(rnorm(20) * 10),
                   date = seq(as.Date("2014-01-01"), by = "month", length.out = 20))

c3(data)

Documentation

For more detailed documentation and methods see http://mrjoh3.github.io/c3/.

gmarv/Data_Analysis_Coursera

2023-09-10T02:03:19Z

A repository for code created during the Coursera Data Analysis Course (Jeff Leek)

dlebauer/climate-utils

2023-09-10T02:03:19Z

code used for exploring / transforming climate data

climate-utils

ClimateUtils: exploring, transforming, and plotting climate data

How to downloading and install

install.packages("devtools")
library(devtools)
install_github("climate-utils", "dlebauer")  ## takes a while, lots of data

Example downloading RH and T, then calculating and plotting VPD

Here is an example of using the ClimateUtils package to calculate amd map summer VPD (vapor pressure deficit) in the midwest US.

The steps are:

run `get.ncep.vpd`

This function is a wrapper for the NCEP.gather function (from the RNCEP package). It combines the following steps:

downloads relative humidity and temperature from NCEP
calculates VPD using the get.es and get.vpd functions

These functions use the following equations:

es saturation vapor pressure
es = 6.11 * exp((2.5 * 10^6 / 461) * (1 / 273 - 1 / (273 + temp)))
VPD vapor pressure deficit
vpd = ((100 - rh) / 100) * es

See ?get.ncep.vpd, ?get.es, and ?get.vpd for details. (source code for calculating es and vpd are in the R/utils.R file)

Plot VPD

library(ClimateUtils)

## Download RH, T from NCEP and calculate VPD
midwest.vpd <- get.ncep.vpd(lat = c(40, 45), lon = c(-90, -82), years = c(2010, 
    2012), months = c(6, 8))

## [1] Units of variable 'air.sig995' are degK
## [1] Units of variable 'rhum.sig995' are %

states <- map_data("state", region = c("michigan", "wisconsin", "illinois", 
    "indiana", "ohio", "pennsylvania", "new york", "iowa", "missouri", "kentucky", 
    "west virginia"))

## Plot it
ggplot() + geom_raster(data = midwest.vpd, aes(longitude - 360, latitude, fill = VPD)) + 
    geom_polygon(data = states, aes(long, lat, group = group), color = "grey20", 
        alpha = 0) + theme_bw()

References:

VanLoocke et al 2012 A regional comparison of water use efficiency for miscanthus, switchgrass and maize http://www.sciencedirect.com/science/article/pii/S0168192312001931

Hartmann, D. Global Physical Climatology. Academic Press

ChargingTurnip/Fortress_Obscura

2023-09-10T02:03:43Z

If you know it, you know it.

python/peps

2023-09-11T01:40:09Z

Python Enhancement Proposals

rails/execjs

2023-09-11T01:40:30Z

Run JavaScript code from Ruby

ExecJS

ExecJS lets you run JavaScript code from Ruby. It automatically picks the best runtime available to evaluate your JavaScript program, then returns the result to you as a Ruby object.

ExecJS supports these runtimes:

therubyrhino - Mozilla Rhino embedded within JRuby
Duktape.rb - Duktape JavaScript interpreter
Node.js
Apple JavaScriptCore - Included with Mac OS X
Microsoft Windows Script Host (JScript)
Google V8
mini_racer - Google V8 embedded within Ruby
GraalVM JavaScript - used on TruffleRuby

A short example:

require "execjs"
ExecJS.eval "'red yellow blue'.split(' ')"
# => ["red", "yellow", "blue"]

A longer example, demonstrating how to invoke the CoffeeScript compiler:

require "execjs"
require "open-uri"
source = open("http://coffeescript.org/extras/coffee-script.js").read

context = ExecJS.compile(source)
context.call("CoffeeScript.compile", "square = (x) -> x * x", bare: true)
# => "var square;\nsquare = function(x) {\n  return x * x;\n};"

Installation

$ gem install execjs

FAQ

Why can't I use CommonJS require() inside ExecJS?

ExecJS provides a lowest common denominator interface to any JavaScript runtime. Use ExecJS when it doesn't matter which JavaScript interpreter your code runs in. If you want to access the Node API, you should check another library like commonjs.rb designed to provide a consistent interface.

Why can't I use setTimeout?

For similar reasons as modules, not all runtimes guarantee a full JavaScript event loop. So setTimeout, setInterval and other timers are not defined.

Why can't I use ES5 features?

Some runtimes like Node will implement many of the latest ES5 features. However older stock runtimes like JSC on OSX and JScript on Windows may not. You should only count on ES3 features being available. Prefer feature checking these APIs rather than hard coding support for specific runtimes.

Can ExecJS be used to sandbox scripts?

No, ExecJS shouldn't be used for any security related sandboxing. Since runtimes are automatically detected, each runtime has different sandboxing properties. You shouldn't use ExecJS.eval on any inputs you wouldn't feel comfortable Ruby eval()ing.

Contributing to ExecJS

ExecJS is work of dozens of contributors. You're encouraged to submit pull requests, propose features and discuss issues.

See CONTRIBUTING.

License

ExecJS is released under the MIT License.

jlecour/munin_resque

2023-09-11T01:40:30Z

A few munin plugins for Resque, written in Ruby

Install

You can install those plugins wherever you want.

On my Debian systems, Munin plugins are stored in /usr/share/munin/plugins. I've installed those in my home directory and it works well.

Plugins

At the top of the plugins, the load path is altered to search for helper files in the same directory as the plugin, and in the standard plugin directory (if the MUNIN_LIBDIR environment variable is set).

resque_queues

A graph with a line for the total number of pending jobs, and a line for each queue.

resque_workers

A graph with the total number of workers and the number of active (busy) workers.

There are 2 levels :

warning = 1.5 x workers
critical = 3 x workers

For the moment, those thresholds are not configurable in plugin settings.

Helper files

plugin.rb

This file has the same purpose as plugin.sh but for Ruby scripts.

resque.rb

It provides a simple stats method, that returns a nested hash from Resque::Server web app.

oven-sh/homebrew-bun

2023-09-11T01:40:30Z

Official Homebrew tap for Bun

homebrew-bun

This is the official Homebrew tap for Bun. You can use it to install Bun on MacOS and Linux using Homebrew.

Install

To install Bun with Homebrew, aka. brew, run the following commands:

brew tap oven-sh/bun
brew install bun

If you want to install a specific release, you can include the version in the command. You can also view the list of releases here.

brew install bun@1.0.0

Upgrade

To upgrade, it is recommended that you run the brew upgrade command. However, you can also run bun upgrade as well.

brew upgrade bun

Uninstall

If you want to uninstall Bun, run the following commands:

brew uninstall bun
brew untap oven-sh/bun

googleapis/google-cloud-ruby

2023-09-10T02:04:07Z

Google Cloud Client Library for Ruby

Google Cloud Ruby Clients

Idiomatic Ruby client libraries for Google Cloud Platform APIs.

This repository includes client libraries for Google Cloud Platform services, along with a selected set of Google services unrelated to the cloud platform.

What's here

Client library gems

Most directories each correspond to a client library RubyGem, including its code, tests, gemspec, and documentation. Some client libraries also include handwritten samples in the samples directory, and/or autogenerated samples in the snippets directory.

Most client libraries in this repository are automatically generated by the GAPIC Generator. A small number are written and maintained by hand. You can identify a generated client library by the presence of .OwlBot.yaml in the library directory. For the most part, do not try to edit generated libraries by hand, because changes will be overwritten by the code generator.

Other directories

A few directories include support files, including:

.github includes configuration for GitHub Actions and bots that help to maintain this repository.
.kokoro includes configuration for internal Google processes that help to maintain this repository.
.toys includes scripts for running CI, releases, and maintenance tasks.
acceptance and integration include shared fixtures for acceptance tests.
obsolete contains older libraries that are obsolete and no longer maintained.

GitHub facilities

Issues for client libraries hosted here can be filed in the issues tab. However, this is not an official support channel. If you have support questions, file a support request through the normal Google support channels, or post questions on a forum such as StackOverflow.

Pull requests are welcome. Please see the section below on contributing.

Some maintenance tasks can be run in the actions tab by authorized personnel.

Using the client libraries

These client library RubyGems each include classes and methods that can be used to make authenticated calls to specific Google APIs. Some libraries also include additional convenience code implementing common client-side workflows or best practices.

In general, you can expect to:

Activate access to the API by creating a project on the Google Cloud Console, enabling billing if necessary, and enabling the API.
Choose a library and install it, typically by adding it to your bundle. For example, here is how you might add a the Translation service client to your Gemfile:
```
# Gemfile

# ... previous libraries ...
gem "google-cloud-translate", "~> 3.2"
```
Instantiate a client object. This object represents an authenticated connection to the service. For example, here is how you might create a client for the translation service:
```
require "google/cloud/translate"

translation_client = Google::Cloud::Translate.translation_service
```
Depending on your environment and authentication needs, you might need to provide credentials to the client object.

Make API calls by invoking methods on the client. For example, here is how you might translate a phrase:

result = translation_client.translate_text contents: ["Hello, world!"],
                                           mime_type: "text/plain",
                                           source_language_code: "en-US",
                                           target_language_code: "ja-JP",
                                           parent: "projects/my-project-name"
puts result.translations.first.translated_text
# => "こんにちは世界！"

Activating the API

To access a Google Cloud API, you will generally need to activate it in the cloud console. This typically involves three steps:

If you have not created a Google Cloud Project, do so. Point your browser to the Google Cloud Console, sign up if needed, and create or choose a project. Make note of the project number (which is numeric) or project ID (which is usually three or more words separated by hyphens). Many services will require you to pass that information in when calling an API.
For most services, you will need to provide billing information. If this is your first time using Google Cloud Platform, you may be eligible for a free trial.
Enable the API you want to use. Click the "APIs & Services" tab in the left navigation, then click the "Enable APIs and Services" button near the top. Search for the API you want by name, and click "Enable". A few APIs may be enabled automatically for you, but most APIs need to be enabled explicitly.

Once you have a project set up and have enabled an API, you are ready to begin using a client library to call the API.

Choosing a client library

This repository contains two types of API client RubyGems: the main library for the API (e.g. the google-cloud-translate gem for the Translation service), and one ore more versioned libraries for different versions of the service (e.g. google-cloud-translate-v2 and google-cloud-translate-v3 for versions 2 and 3 of the service, respectively). Note that we're referring to different versions of the backend service, not of the client library gem.

In most cases, you should install the main library (the one without a service version in the name). This library will provide all the required code for making calls to the API. It may also provide additional convenience code implementing common client-side workflows or best practices. Often the main library will bring in one or more versioned libraries as dependencies, and the client and data type classes you will use may actually be defined in a versioned library, but installing the main library will ensure you have access to the best tools and interfaces for interacting with the service.

The versioned libraries are lower-level libraries that target a specific version of the service. You may choose to intall a versioned library directly, instead of or in addition to the main library, to handle advanced use cases that require lower level access.

Note: Many services may also provide client libraries with names beginning with google-apis-. Those clients are developed in a different repository, and utilize an older client technology that lacks some of the performance and ease of use benefits of the clients in the google-cloud-ruby repository. The older clients may cover some services for which a google-cloud-ruby client is not yet available, but for services that are covered, we generally recommend the clients in the google-cloud-ruby repository over the older ones.

Most client libraries have directories in this repository, or you can look up the name of the client library to use in the documentation for the service you are using. Install this library as a RubyGem, or add it to your Gemfile.

Authentication

Most API calls must be accompanied by authentication information proving that the caller has sufficient permissions to make the call. For an overview of authentication with Google, see https://cloud.google.com/docs/authentication.

These API client libraries provide several mechanisms for attaching credentials to API calls.

If your application runs on an Google Cloud Platform hosting environment such as Google Compute Engine, Google Container Engine, Google App Engine, Google Cloud Run, or Google Cloud Functions, the environment will provide "ambient" credentials which client libraries will recognize and use automatically. You can generally configure these credentials in the hosting environment, for example per-VM in Google Compute Engine.
You can also provide your own service account credentials by including a service account key file in your application's file system and setting the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path to that file. Client libraries will read this environment variable if it is set.
Finally, you can override credentials in code by setting the credentials field in the client configuration. This can be set globally on the client class or provided when you construct a client object.

See https://cloud.google.com/docs/authentication/production for more information on these and other methods of providing credentials.

Supported Ruby Versions

These libraries are currently supported on Ruby 2.6 through Ruby 3.1. Older versions of Ruby may still work, but are unsupported and not recommended.

In general, Google provides official support for Ruby versions that are actively supported by Ruby Core--that is, Ruby versions that are either in normal maintenance or in security maintenance, and not end of life. See https://www.ruby-lang.org/en/downloads/branches/ for details about the Ruby support schedule.

Library Versioning

The libraries in this repository follow Semantic Versioning.

Libraries are released at one of two different support quality levels:

GA: Libraries defined at the GA (general availability) quality level, indicated by a gem version number greater than or equal to 1.0, are stable. The code surface will not change in backwards-incompatible ways unless absolutely necessary (e.g. because of critical security issues), or unless accompanying a semver-major version update (such as version 1.x to 2.x.) Issues and requests against GA libraries are addressed with the highest priority.

Preview: Libraries defined at a Preview quality level, indicated by a gem version number less than 1.0, are expected to be mostly stable and we're working towards their release candidate. However, these libraries may get backwards-incompatible updates from time to time. We will still address issues and requests with a high priority.

Note that the gem version is distinct from the service version. Some backend services have mulitple versions, for example versions v2 and v3 of the translation service. These are treated as separate services and will have separate versioned clients, e.g. the google-cloud-translate-v2 and google-cloud-translate-v3 gems. These gems will in turn have their own gem versions, tracking the development of the two services.

Contributing

Contributions to this repository are welcome. However, please note that many of the clients in this repository are automatically generated. The Ruby files in those clients will have a comment to that effect near the top; changes to those files will not be accepted as they will simply be overwritten by the code generator. If in doubt, please open an issue and ask the maintainers. See the CONTRIBUTING document for more information on how to get started.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Code of Conduct for more information.

License

This library is licensed under Apache 2.0. Full license text is available in LICENSE.

Support

Please report bugs at the project on Github.

If you have questions about how to use the clients or APIs, ask on Stack Overflow.

sous-chefs/nodejs

2023-09-10T02:04:07Z

Development repository for the nodejs cookbook

nodejs-cookbook

Installs node.js/npm and includes a resource for managing npm packages

Maintainers

This cookbook is maintained by the Sous Chefs. The Sous Chefs are a community of Chef cookbook maintainers working together to maintain important cookbooks. If you’d like to know more please visit sous-chefs.org or come chat with us on the Chef Community Slack in #sous-chefs.

Requirements

Platforms

Debian/Ubuntu
RHEL/CentOS/Scientific/Amazon/Oracle
openSUSE
Windows

Note: Source installs require GCC 4.8+, which is not included on older distro releases

Chef

Chef Infra Client 15.3+

Cookbooks

Usage

Include the nodejs recipe to install node on your system based on the default installation method:

include_recipe "nodejs"

Install methods

Package

Install node from packages:

node['nodejs']['install_method'] = 'package' # Not necessary because it's the default
include_recipe "nodejs"
# Or
include_recipe "nodejs::nodejs_from_package"

By default this will setup deb/rpm repositories from nodesource.com, which include up to date NodeJS packages. If you prefer to use distro provided package you can disable this behavior by setting node['nodejs']['install_repo'] to false.

Binary

Install node from official prebuilt binaries:

node['nodejs']['install_method'] = 'binary'
include_recipe "nodejs"

# Or
include_recipe "nodejs::nodejs_from_binary"

# Or set a specific version of nodejs to be installed
node.default['nodejs']['install_method'] = 'binary'
node.default['nodejs']['version'] = '5.9.0'
node.default['nodejs']['binary']['checksum'] = '99c4136cf61761fac5ac57f80544140a3793b63e00a65d4a0e528c9db328bf40'

# Or fetch the binary from your own location
node.default['nodejs']['install_method'] = 'binary'
node.default['nodejs']['binary']['url'] = 'https://s3.amazonaws.com/my-bucket/node-v7.8.0-linux-x64.tar.gz'
node.default['nodejs']['binary']['checksum'] = '0bd86f2a39221b532172c7d1acb57f0b0cba88c7b82ea74ba9d1208b9f6f9697'

Source

Install node from sources:

node['nodejs']['install_method'] = 'source'
include_recipe "nodejs"
# Or
include_recipe "nodejs::nodejs_from_source"

Chocolatey

Install node from chocolatey:

node['nodejs']['install_method'] = 'chocolatey'
include_recipe "nodejs"
# Or
include_recipe "nodejs::nodejs_from_chocolatey"

NPM

Npm is included in nodejs installs by default. By default, we are using it and call it embedded. Adding recipe nodejs::npm assure you to have npm installed and let you choose install method with node['nodejs']['npm']['install_method']

include_recipe "nodejs::npm"

Warning: This recipe will include the nodejs recipe, which by default includes nodejs::nodejs_from_package if you did not set node['nodejs']['install_method'].

Resources

npm_package

note: This resource was previously named nodejs_npm. Calls to that resource name will still function, but cookbooks should be updated for the new npm_package resource name.

npm_package lets you install npm packages from various sources:

npm registry:
- name: property :package
- version: property :version (optional)
url: property :url
- for git use git://{your_repo}
from a json (package.json by default): property :json
- use true for default
- use a String to specify json file

Packages can be installed globally (by default) or in a directory (by using attribute :path)

You can specify an NPM_TOKEN environment variable for accessing NPM private modules by using attribute :npm_token

You can specify a NODE_ENV environment variable, in the case that some element of your installation depends on this by using attribute :node_env. E.g., using node-config as part of your postinstall script. Please note that adding the --production option will override this to NODE_ENV=production.

You can append more specific options to npm command with attribute :options array :

You can specify auto_update as false to stop the npm install command from running and updating an installed package. Running the command will update packages within restrictions imposed by a package.json file. The default behavior is to update automatically.

use an array of options (w/ dash), they will be added to npm call.
ex: ['--production','--force'] or ['--force-latest']

You can specify live_stream true for the resource to have the package install information included in the chef-client log outout for better npm package diagnostics and trouble shooting.

This LWRP attempts to use vanilla npm as much as possible (no custom wrapper).

Packages

npm_package 'express'

npm_package 'async' do
  version '0.6.2'
end

npm_package 'request' do
  url 'github mikeal/request'
end

npm_package 'grunt' do
  path '/home/random/grunt'
  json true
  user 'random'
  node_env 'staging'
end

npm_package 'my_private_module' do
  path '/home/random/myproject' # The root path to your project, containing a package.json file
  json true
  npm_token '12345-abcde-e5d4c3b2a1'
  user 'random'
  options ['--production'] # Only install dependencies. Skip devDependencies
end

Working Examples

Or add packages via attributes (which accept the same attributes as the LWRP above):

"nodejs": {
  "npm_packages": [
    {
      "name": "express"
    },
    {
      "name": "async",
      "version": "0.6.2"
    },
    {
      "name": "request",
      "url": "github mikeal/request"
    }
    {
      "name": "grunt",
      "path": "/home/random/grunt",
      "json": true,
      "user": "random"
    }
  ]
}

Contributors

This project exists thanks to all the people who contribute.

Backers

Thank you to all our backers!

cloudflare/pingora

2023-09-10T01:43:04Z

A library for building fast, reliable and evolvable network services.

Pingora

A library for building fast, reliable and evolvable network services.

biomejs/biome

2023-09-10T02:04:28Z

A toolchain for web projects, aimed to provide functionalities to maintain them.

Biome formats and lints your code in a fraction of a second.

Biome supports JavaScript, TypeScript, JSON, and CSS. It aims to support all main languages of modern web development.

Biome has sane defaults and requires minimal configuration. Biome helps you as much as possible by displaying detailed and contextualized diagnostics. Read more about our project philosophy.

Biome unifies functionality that has previously been separate tools. Building upon a shared base allows us to provide a cohesive experience for processing code, displaying errors, parallelizing work, caching, and configuration.

Biome is designed to eventually replace Babel, ESLint, webpack, Prettier, Jest, and others.

Biome has first-class IDE support, with a sophisticated parser that represents the source text in full fidelity and top-notch error recovery.

Biome is MIT licensed and moderated under the Contributor Covenant Code of Conduct.

Documentation

Check out our homepage to learn more about Biome, or directly head to the Getting Started guide to start using Biome.

containers/youki

2023-09-10T02:04:28Z

A container runtime written in Rust

youki: A container runtime in Rust

youki is an implementation of the OCI runtime-spec in Rust, similar to runc.
Your ideas are welcome here.

Quick Install

User Documentation

About the name

youki is pronounced as /joʊki/ or yoh-key. youki is named after the Japanese word 'youki', which means 'a container'. In Japanese language, youki also means 'cheerful', 'merry', or 'hilarious'.

Motivation

Here is why we are writing a new container runtime in Rust.

Rust is one of the best languages to implement the oci-runtime spec. Many very nice container tools are currently written in Go. However, the container runtime requires the use of system calls, which requires a bit of special handling when implemented in Go. This is too tricky (e.g. namespaces(7), fork(2)); with Rust, it's not that tricky. And, unlike in C, Rust provides the benefit of memory safety. While Rust is not yet a major player in the container field, it has the potential to contribute a lot: something this project attempts to exemplify.

youki has the potential to be faster and use less memory than runc, and therefore work in environments with tight memory usage requirements. Here is a simple benchmark of a container from creation to deletion.

Runtime	Time (mean ± σ)	Range (min … max)
youki	198.4 ms ± 52.1 ms	97.2 ms … 296.1 ms
runc	352.3 ms ± 53.3 ms	248.3 ms … 772.2 ms
crun	153.5 ms ± 21.6 ms	80.9 ms … 196.6 ms

Details about the benchmark

A command used for the benchmark

hyperfine --prepare 'sudo sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' --warmup 10 --min-runs 100 'sudo ./youki create -b tutorial a && sudo ./youki start a && sudo ./youki delete -f a'

Environment

$ ./youki info
Version           0.0.1
Kernel-Release    5.11.0-41-generic
Kernel-Version    #45-Ubuntu SMP Fri Nov 5 11:37:01 UTC 2021
Architecture      x86_64
Operating System  Ubuntu 21.04
Cores             12
Total Memory      32025
Cgroup setup      hybrid
Cgroup mounts
  blkio           /sys/fs/cgroup/blkio
  cpu             /sys/fs/cgroup/cpu,cpuacct
  cpuacct         /sys/fs/cgroup/cpu,cpuacct
  cpuset          /sys/fs/cgroup/cpuset
  devices         /sys/fs/cgroup/devices
  freezer         /sys/fs/cgroup/freezer
  hugetlb         /sys/fs/cgroup/hugetlb
  memory          /sys/fs/cgroup/memory
  net_cls         /sys/fs/cgroup/net_cls,net_prio
  net_prio        /sys/fs/cgroup/net_cls,net_prio
  perf_event      /sys/fs/cgroup/perf_event
  pids            /sys/fs/cgroup/pids
  unified         /sys/fs/cgroup/unified
CGroup v2 controllers
  cpu             detached
  cpuset          detached
  hugetlb         detached
  io              detached
  memory          detached
  pids            detached
  device          attached
Namespaces        enabled
  mount           enabled
  uts             enabled
  ipc             enabled
  user            enabled
  pid             enabled
  network         enabled
  cgroup          enabled
$ ./youki --version
youki version 0.0.1
commit: 0.0.1-0-0be33bf
$ runc -v
runc version 1.0.0-rc93
commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
spec: 1.0.2-dev
go: go1.13.15
libseccomp: 2.5.1
$ crun --version
crun version 0.19.1.45-4cc7
commit: 4cc7fa1124cce75dc26e12186d9cbeabded2b710
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

I have fun implementing this. In fact, this may be the most important.

Related project

containers/oci-spec-rs - OCI Runtime and Image Spec in Rust

Status of youki

youki is not at the practical stage yet. However, it is getting closer to practical use, running with docker and passing all the default tests provided by opencontainers/runtime-tools.

Feature	Description	State
Containerd	Running via Containerd	✅
Docker	Running via Docker	✅
Podman	Running via Podman	✅
pivot_root	Change the root directory	✅
Mounts	Mount files and directories to container	✅
Namespaces	Isolation of various resources	✅
Capabilities	Limiting root privileges	✅
Cgroups v1	Resource limitations, etc	✅
Cgroups v2	Improved version of v1	Support is complete except for devices. WIP on #230
Systemd cgroup driver	Setting up a cgroup using systemd	✅
Seccomp	Filtering system calls	✅
Hooks	Add custom processing during container creation	✅
Rootless	Running a container without root privileges	✅
OCI Compliance	Compliance with OCI Runtime Spec	✅ 50 out of 50 test cases passing
CRIU Integration	Functionality to checkpoint/restore containers	Initial checkpoint support as described in #641

Design and implementation of youki

The User and Developer Documentation for youki is hosted at https://containers.github.io/youki/

Getting Started

Local build is only supported on Linux. For other platforms, please use the Vagrantfile that we have prepared. You can also spin up a fully preconfigured development environment in the cloud with gitpod.

Requires

Rust(See here), edition 2021
linux kernel ≥ 5.3

Dependencies

To install just, follow the instruction here.

Debian, Ubuntu and related distributions

$ sudo apt-get install    \
      pkg-config          \
      libsystemd-dev      \
      libdbus-glib-1-dev  \
      build-essential     \
      libelf-dev          \
      libseccomp-dev      \
      libclang-dev        \
      glibc-static        \
      libssl-dev

Fedora, CentOS, RHEL and related distributions

$ sudo dnf install          \
      pkg-config            \
      systemd-devel         \
      dbus-devel            \
      elfutils-libelf-devel \
      libseccomp-devel      \
      clang-devel           \
      openssl-devel

Build

git clone git@github.com:containers/youki.git
cd youki
just youki-dev # or youki-release
./youki -h # you can get information about youki command

Tutorial

Requires

Docker(See here)

Create and run a container

Let's try to run a container that executes sleep 30 with youki. This tutorial may need root permission.

git clone git@github.com:containers/youki.git
cd youki
just youki-dev # or youki-release

mkdir -p tutorial/rootfs
cd tutorial
# use docker to export busybox into the rootfs directory
docker export $(docker create busybox) | tar -C rootfs -xvf -

Then, we need to prepare a configuration file. This file contains metadata and specs for a container, such as the process to run, environment variables to inject, sandboxing features to use, etc.

../youki spec  # will generate a spec file named config.json

We can edit the config.json to add customized behaviors for container. Here, we modify the process field to run sleep 30.

  "process": {
    ...
    "args": [
      "sleep", "30"
    ],

  ...
  }

Then we can explore the lifecycle of a container:

cd ..                                                # go back to the repository root
sudo ./youki create -b tutorial tutorial_container   # create a container with name `tutorial_container`
sudo ./youki state tutorial_container                # you can see the state the container is `created`
sudo ./youki start tutorial_container                # start the container
sudo ./youki list                                    # will show the list of containers, the container is `running`
sudo ./youki delete tutorial_container               # delete the container

Change the command to be executed in config.json and try something other than sleep 30.

Rootless container

youki provides the ability to run containers as non-root user(rootless mode). To run a container in rootless mode, we need to add some extra options in config.json, other steps are same with above:

$ mkdir -p tutorial/rootfs
$ cd tutorial
# use docker to export busybox into the rootfs directory
$ docker export $(docker create busybox) | tar -C rootfs -xvf -

$ ../youki spec --rootless          # will generate a spec file named config.json with rootless mode
## Modify the `args` field as you like

$ ../youki run rootless-container   # will create and run a container with rootless mode

Usage

Start the docker daemon.

dockerd --experimental --add-runtime="youki=$(pwd)/youki"

If you get an error like the below, that means your normal Docker daemon is running, and it needs to be stopped. Do that with your init system (i.e., with systemd, run systemctl stop docker, as root if necessary).

failed to start daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid

Now repeat the command, which should start the docker daemon.

You can use youki in a different terminal to start the container.

docker run -it --rm --runtime youki busybox

Afterwards, you can close the docker daemon process in other the other terminal. To restart normal docker daemon (if you had stopped it before), run:

systemctl start docker # might need root permission

Integration Tests

Go and node-tap are required to run integration tests. See the opencontainers/runtime-tools README for details.

git submodule update --init --recursive
just oci-tests

Setting up Vagrant

You can try youki on platforms other than Linux by using the Vagrantfile we have prepared. We have prepared two environments for vagrant, namely rootless mode and rootful mode

git clone git@github.com:containers/youki.git
cd youki

# If you want to develop in rootless mode, and this is the default mode
vagrant up
vagrant ssh

# or if you want to develop in rootful mode
VAGRANT_VAGRANTFILE=Vagrantfile.root vagrant up
VAGRANT_VAGRANTFILE=Vagrantfile.root vagrant ssh

# in virtual machine
cd youki
just youki-dev # or youki-release

Community

We also have an active Discord if you'd like to come and chat with us.

Contribution

This project welcomes your PR and issues. For example, refactoring, adding features, correcting English, etc. If you need any help, you can contact me on Twitter.

Thanks to all the people who already contributed!

shadowsocks/shadowsocks-rust

2023-09-10T02:04:28Z

A Rust port of shadowsocks

shadowsocks

This is a port of shadowsocks.

shadowsocks is a fast tunnel proxy that helps you bypass firewalls.

Library	Description
shadowsocks	shadowsocks core protocol
shadowsocks-service	Services for serving shadowsocks
shadowsocks-rust	Binaries running common shadowsocks services

Related Projects:

spyophobia/shadowsocks-gtk-rs A GUI on Linux for sslocal using GTK, discussion
honwen/openwrt-shadowsocks-rust OpenWRT solution for sslocal, discussion
cg31/shadowsocks-windows-gui-rust Windows GUI client, discussion

Build & Install

Optional Features

trust-dns - Uses trust-dns-resolver as DNS resolver instead of tokio's builtin.
local-http - Allow using HTTP protocol for sslocal
- local-http-native-tls - Support HTTPS with native-tls
- local-http-rustls - Support HTTPS with rustls
local-tunnel - Allow using tunnel protocol for sslocal
local-socks4 - Allow using SOCKS4/4a protocol for sslocal
local-redir - Allow using redir (transparent proxy) protocol for sslocal
local-dns - Allow using dns protocol for sslocal, serves as a DNS server proxying queries to local or remote DNS servers by ACL rules
local-tun - TUN interface support for sslocal
stream-cipher - Enable deprecated stream ciphers. WARN: stream ciphers are UNSAFE!
aead-cipher-extra - Enable non-standard AEAD ciphers
aead-cipher-2022 - Enable AEAD-2022 ciphers (SIP022)
aead-cipher-2022-extra - Enable AEAD-2022 extra ciphers (non-standard ciphers)

Memory Allocators

This project uses system (libc) memory allocator (Rust's default). But it also allows you to use other famous allocators by features:

jemalloc - Uses jemalloc as global memory allocator
mimalloc - Uses mi-malloc as global memory allocator
tcmalloc - Uses TCMalloc as global memory allocator. It tries to link system-wide tcmalloc by default, use vendored from source with tcmalloc-vendored.
snmalloc - Uses snmalloc as global memory allocator
rpmalloc - Uses rpmalloc as global memory allocator

crates.io

Install from crates.io:

# Install from crates.io
cargo install shadowsocks-rust

then you can find sslocal and ssserver in $CARGO_HOME/bin.

Install using Homebrew

For macOS and Linux, you can install it using Homebrew:

brew install shadowsocks-rust

Install using snap

# Install from snapstore
snap install shadowsocks-rust

# List services
snap services shadowsocks-rust

# Enable and start shadowsocks-rust.sslocal-daemon snap service
snap start --enable shadowsocks-rust.sslocal-daemon

# Show generated systemd service status
systemctl status snap.shadowsocks-rust.sslocal-daemon.service

# Override generated systemd service (configure startup options)
systemctl edit snap.shadowsocks-rust.sslocal-daemon.service

## NOTE: you can pass args to sslocal:
##  [Service]
##  ExecStart=
##  ExecStart=/usr/bin/snap run shadowsocks-rust.sslocal-daemon -b "127.0.0.1:1080" --server-url "ss://...."

# Restart generated systemd service to apply changes
systemctl restart snap.shadowsocks-rust.sslocal-daemon.service

# ... and show service status
systemctl status snap.shadowsocks-rust.sslocal-daemon.service

Download release

Download static-linked build here.

build-windows: Build for x86_64-pc-windows-msvc
build-linux: Build for x86_64-unknown-linux-gnu, Debian 9 (Stretch), GLIBC 2.18
build-docker: Build for x86_64-unknown-linux-musl, x86_64-pc-windows-gnu, ... (statically linked)

Docker

This project provided Docker images for the linux/i386 and linux/amd64 and linux/arm64/v8 architectures.

⚠ Docker containers do not have access to IPv6 by default: Make sure to disable IPv6 Route in the client or enable IPv6 access to docker containers.

Pull from GitHub Container Registry

Docker will pull the image of the appropriate architecture from our GitHub Packages.

docker pull ghcr.io/shadowsocks/sslocal-rust:latest
docker pull ghcr.io/shadowsocks/ssserver-rust:latest

Build on the local machine（Optional）

If you want to build the Docker image yourself, you need to use the BuildX.

docker buildx build -t shadowsocks/ssserver-rust:latest -t shadowsocks/ssserver-rust:v1.15.2 --target ssserver .
docker buildx build -t shadowsocks/sslocal-rust:latest -t shadowsocks/sslocal-rust:v1.15.2 --target sslocal .

Run the container

You need to mount the configuration file into the container and create an external port map for the container to connect to it.

docker run --name sslocal-rust \
  --restart always \
  -p 1080:1080/tcp \
  -v /path/to/config.json:/etc/shadowsocks-rust/config.json \
  -dit ghcr.io/shadowsocks/sslocal-rust:latest

docker run --name ssserver-rust \
  --restart always \
  -p 8388:8388/tcp \
  -p 8388:8388/udp \
  -v /path/to/config.json:/etc/shadowsocks-rust/config.json \
  -dit ghcr.io/shadowsocks/ssserver-rust:latest

Deploy to Kubernetes

This project provided yaml manifests for deploying to Kubernetes.

You can leverage k8s Service to expose traffic outside, like LoadBalancer or NodePort which gains more fine-grained compared with fixed host or port.

For a more interesting use case, you can use a Ingress(Istio, nginx, etc.) which routes the matched traffic to shadowsocks along with the real web service.

Using `kubectl`

kubectl apply -f https://github.com/shadowsocks/shadowsocks-rust/raw/master/k8s/shadowsocks-rust.yaml

You can change the config via editing the ConfigMap named shadowsocks-rust.

For more fine-grained control, use helm.

Using `helm`

helm install my-release k8s/chart -f my-values.yaml

Below is the common default values you can change:

# This is the shadowsocks config which will be mount to /etc/shadowocks-rust.
# You can put arbitrary yaml here, and it will be translated to json before mounting.
servers:
- server: "::"
  server_port: 8388
  service_port: 80 # the k8s service port, default to server_port
  password: mypassword
  method: aes-256-gcm
  fast_open: true
  mode: tcp_and_udp
  # plugin: v2ray-plugin
  # plugin_opts: server;tls;host=github.com

# Whether to download v2ray and xray plugin.
downloadPlugins: false

# Name of the ConfigMap with config.json configuration for shadowsocks-rust.
configMapName: ""

service:
  # Change to LoadBalancer if you are behind a cloud provider like aws, gce, or tke.
  type: ClusterIP

# Bind shadowsocks port port to host, i.e., we can use host:port to access shawdowsocks server.
hostPort: false

replicaCount: 1

image:
  repository: ghcr.io/shadowsocks/ssserver-rust
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "latest"

Build from source

Use cargo to build. NOTE: RAM >= 2GiB

cargo build --release

Then sslocal and ssserver will appear in ./target/(debug|release)/, it works similarly as the two binaries in the official ShadowSocks' implementation.

make install TARGET=release

Then sslocal, ssserver, ssmanager and ssurl will be installed to /usr/local/bin (variable PREFIX).

For Windows users, if you have encountered any problem in building, check and discuss in #102.

target-cpu optimization

If you are building for your current CPU platform (for example, build and run on your personal computer), it is recommended to set target-cpu=native feature to let rustc generate and optimize code for the CPU running the compiler.

export RUSTFLAGS="-C target-cpu=native"

Build standalone binaries

Requirements:

Docker

./build/build-release

Then sslocal, ssserver, ssmanager and ssurl will be packaged in

./build/shadowsocks-${VERSION}-stable.x86_64-unknown-linux-musl.tar.xz
./build/shadowsocks-${VERSION}-stable.x86_64-pc-windows-gnu.zip

Read Cargo.toml for more details.

Getting Started

Generate a safe and secured password for a specific encryption method (aes-128-gcm in the example) with:

ssservice genkey -m "aes-128-gcm"

Create a ShadowSocks' configuration file. Example

{
    "server": "my_server_ip",
    "server_port": 8388,
    "password": "rwQc8qPXVsRpGx3uW+Y3Lj4Y42yF9Bs0xg1pmx8/+bo=",
    "method": "aes-256-gcm",
    // ONLY FOR `sslocal`
    // Delete these lines if you are running `ssserver` or `ssmanager`
    "local_address": "127.0.0.1",
    "local_port": 1080
}

Detailed explanation could be found in shadowsocks' documentation.

In shadowsocks-rust, we also have an extended configuration file format, which is able to define more than one server. You can also disable individual servers.

{
    "servers": [
        {
            "server": "127.0.0.1",
            "server_port": 8388,
            "password": "rwQc8qPXVsRpGx3uW+Y3Lj4Y42yF9Bs0xg1pmx8/+bo=",
            "method": "aes-256-gcm",
            "timeout": 7200
        },
        {
            "server": "127.0.0.1",
            "server_port": 8389,
            "password": "/dliNXn5V4jg6vBW4MnC1I8Jljg9x7vSihmk6UZpRBM=",
            "method": "chacha20-ietf-poly1305"
        },
        {
            "disabled": true,
            "server": "eg.disable.me",
            "server_port": 8390,
            "password": "mGvbWWay8ueP9IHnV5F1uWGN2BRToiVCAWJmWOTLU24=",
            "method": "chacha20-ietf-poly1305"
        }
    ],
    // ONLY FOR `sslocal`
    // Delete these lines if you are running `ssserver` or `ssmanager`
    "local_port": 1080,
    "local_address": "127.0.0.1"
}

sslocal automatically selects the best server with the lowest latency and the highest availability.

Start Shadowsocks client and server with:

sslocal -c config.json
ssserver -c config.json

If you Build it with Cargo:

cargo run --bin sslocal -- -c config.json
cargo run --bin ssserver -- -c config.json

List all available arguments with -h.

Usage

Start local client with configuration file

# Read local client configuration from file
sslocal -c /path/to/shadowsocks.json

Socks5 Local client

# Pass all parameters via command line
sslocal -b "127.0.0.1:1080" -s "[::1]:8388" -m "aes-256-gcm" -k "hello-kitty" --plugin "v2ray-plugin" --plugin-opts "server;tls;host=github.com"

# Pass server with SIP002 URL
sslocal -b "127.0.0.1:1080" --server-url "ss://YWVzLTI1Ni1nY206cGFzc3dvcmQ@127.0.0.1:8388/?plugin=v2ray-plugin%3Bserver%3Btls%3Bhost%3Dgithub.com"

HTTP Local client

sslocal -b "127.0.0.1:3128" --protocol http -s "[::1]:8388" -m "aes-256-gcm" -k "hello-kitty"

All parameters are the same as Socks5 client, except --protocol http.

Tunnel Local client

# Set 127.0.0.1:8080 as the target for forwarding to
sslocal --protocol tunnel -b "127.0.0.1:3128" -f "127.0.0.1:8080" -s "[::1]:8388" -m "aes-256-gcm" -k "hello-kitty"

--protocol tunnel enables local client Tunnel mode
-f "127.0.0.1:8080 sets the tunnel target address

Transparent Proxy Local client

NOTE: It currently only supports

Linux (with iptables targets REDIRECT and TPROXY)
BSDs (with pf), such as OS X 10.10+, FreeBSD, ...

sslocal -b "127.0.0.1:60080" --protocol redir -s "[::1]:8388" -m "aes-256-gcm" -k "hello-kitty" --tcp-redir "redirect" --udp-redir "tproxy"

Redirects connections with iptables configurations to the port that sslocal is listening on.

--protocol redir enables local client Redir mode
(optional) --tcp-redir sets TCP mode to REDIRECT (Linux)
(optional) --udp-redir sets UDP mode to TPROXY (Linux)

Tun interface client

NOTE: It currently only supports

Linux, Android
macOS, iOS

Linux

Create a Tun interface with name tun0

ip tuntap add mode tun tun0
ifconfig tun0 inet 10.255.0.1 netmask 255.255.255.0 up

Start sslocal with --protocol tun and binds to tun0

sslocal --protocol tun -s "[::1]:8388" -m "aes-256-gcm" -k "hello-kitty" --outbound-bind-interface lo0 --tun-interface-name tun0

macOS

sslocal --protocol tun -s "[::1]:8388" -m "aes-256-gcm" -k "hello-kitty" --outbound-bind-interface lo0 --tun-interface-address 10.255.0.1/24

It will create a Tun interface with address 10.255.0.1 and netmask 255.255.255.0.

Server

# Read server configuration from file
ssserver -c /path/to/shadowsocks.json

# Pass all parameters via command line
ssserver -s "[::]:8388" -m "aes-256-gcm" -k "hello-kitty" --plugin "v2ray-plugin" --plugin-opts "server;tls;host=github.com"

Server Manager

Supported Manage Multiple Users API:

add - Starts a server instance
remove - Deletes an existing server instance
list - Lists all current running servers
ping - Lists all servers' statistic data

NOTE: stat command is not supported. Because servers are running in the same process with the manager itself.

# Start it just with --manager-address command line parameter
ssmanager --manager-address "127.0.0.1:6100"

# For *nix system, manager can bind to unix socket address
ssmanager --manager-address "/tmp/shadowsocks-manager.sock"

# You can also provide a configuration file
#
# `manager_address` key must be provided in the configuration file
ssmanager -c /path/to/shadowsocks.json

# Create one server by UDP
echo 'add: {"server_port":8388,"password":"hello-kitty"}' | nc -u '127.0.0.1' '6100'

# Close one server by unix socket
echo 'remove: {"server_port":8388}' | nc -Uu '/tmp/shadowsocks-manager.sock'

For manager UI, check more details in the shadowsocks-manager project.

Example configuration:

{
    // Required option
    // Address that ssmanager is listening on
    "manager_address": "127.0.0.1",
    "manager_port": 6100,

    // Or bind to a Unix Domain Socket
    "manager_address": "/tmp/shadowsocks-manager.sock",

    "servers": [
        // These servers will be started automatically when ssmanager is started
    ],

    // Outbound socket binds to this IP address
    // For choosing different network interface on the same machine
    "local_address": "xxx.xxx.xxx.xxx",

    // Other options that may be passed directly to new servers
}

Configuration

{
    // LOCAL: Listen address. This is exactly the same as `locals[0]`
    // SERVER: Bind address for remote sockets, mostly used for choosing interface
    //         Don't set it if you don't know what's this for.
    "local_address": "127.0.0.1",
    "local_port": 1080,

    // Extended multiple local configuration
    "locals": [
        {
            // Basic configuration, a SOCKS5 local server
            "local_address": "127.0.0.1",
            "local_port": 1080,
            // OPTIONAL. Setting the `mode` for this specific local server instance.
            // If not set, it will derive from the outer `mode`
            "mode": "tcp_and_udp",
            // OPTIONAL. Authentication configuration file
            // Configuration file document could be found in the next section.
            "socks5_auth_config_path": "/path/to/auth.json",
            // OPTIONAL. Instance specific ACL
            "acl": "/path/to/acl/file.acl",
        },
        {
            // SOCKS5, SOCKS4/4a local server
            "protocol": "socks",
            // Listen address
            "local_address": "127.0.0.1",
            "local_port": 1081,
            // OPTIONAL. Enables UDP relay
            "mode": "tcp_and_udp",
            // OPTIONAL. Customizing the UDP's binding address. Depending on `mode`, if
            // - TCP is enabled, then SOCKS5's UDP Association command will return this address
            // - UDP is enabled, then SOCKS5's UDP server will listen to this address.
            "local_udp_address": "127.0.0.1",
            "local_udp_port": 2081
        },
        {
            // Tunnel local server (feature = "local-tunnel")
            "protocol": "tunnel",
            // Listen address
            "local_address": "127.0.0.1",
            "local_port": 5353,
            // Forward address, the target of this tunnel
            // In this example, this will build a `127.0.0.1:5353` -> `8.8.8.8:53` tunnel
            "forward_address": "8.8.8.8",
            "forward_port": 53,
            // OPTIONAL. Customizing whether to start TCP and UDP tunnel
            "mode": "tcp_only"
        },
        {
            // HTTP local server (feature = "local-http")
            "protocol": "http",
            // Listen address
            "local_address": "127.0.0.1",
            "local_port": 3128
        },
        {
            // DNS local server (feature = "local-dns")
            // This DNS works like China-DNS, it will send requests to `local_dns` and `remote_dns` and choose by ACL rules
            "protocol": "dns",
            // Listen address
            "local_address": "127.0.0.1",
            "local_port": 53,
            // OPTIONAL. DNS local server uses `tcp_and_udp` mode by default
            "mode": "udp_only",
            // Local DNS address, DNS queries will be sent directly to this address
            "local_dns_address": "114.114.114.114",
            // OPTIONAL. Local DNS's port, 53 by default
            "local_dns_port": 53,
            // Remote DNS address, DNS queries will be sent through ssserver to this address
            "remote_dns_address": "8.8.8.8",
            // OPTIONAL. Remote DNS's port, 53 by default
            "remote_dns_port": 53
        },
        {
            // Tun local server (feature = "local-tun")
            "protocol": "tun",
            // Tun interface name
            "tun_interface_name": "tun0",
            // Tun interface address
            //
            // It has to be a host address in CIDR form
            "tun_interface_address": "10.255.0.1/24"
        },
        {
            // Transparent Proxy (redir) local server (feature = "local-redir")
            "protocol": "redir",
            // OPTIONAL: TCP type, may be different between platforms
            // Linux/Android: redirect (default), tproxy
            // FreeBSD/OpenBSD: pf (default), ipfw
            // NetBSD/macOS/Solaris: pf (default), ipfw
            "tcp_redir": "tproxy",
            // OPTIONAL: UDP type, may be different between platforms
            // Linux/Android: tproxy (default)
            // FreeBSD/OpenBSD: pf (default)
            "udp_redir": "tproxy"
        }
    ],

    // Server configuration
    // listen on :: for dual stack support, no need add [] around.
    "server": "::",
    // Change to use your custom port number
    "server_port": 8388,
    "method": "aes-256-gcm",
    "password": "your-password",
    "plugin": "v2ray-plugin",
    "plugin_opts": "mode=quic;host=github.com",
    "plugin_args": [
        // Each line is an argument passed to "plugin"
        "--verbose"
    ],
    "plugin_mode": "tcp_and_udp", // SIP003u, default is "tcp_only"
    // Server: TCP socket timeout in seconds.
    // Client: TCP connection timeout in seconds.
    // Omit this field if you don't have specific needs.
    "timeout": 7200,

    // Extended multiple server configuration
    // LOCAL: Choosing the best server to connect dynamically
    // SERVER: Creating multiple servers in one process
    "servers": [
        {
            // Fields are the same as the single server's configuration

            // Individual servers can be disabled
            // "disabled": true,
            "address": "0.0.0.0",
            "port": 8389,
            "method": "aes-256-gcm",
            "password": "your-password",
            "plugin": "...",
            "plugin_opts": "...",
            "plugin_args": [],
            "plugin_mode": "...",
            "timeout": 7200,

            // Customized weight for local server's balancer
            //
            // Weight must be in [0, 1], default is 1.0.
            // The higher weight, the server may rank higher.
            "tcp_weight": 1.0,
            "udp_weight": 1.0,

            // OPTIONAL. Instance specific ACL
            "acl": "/path/to/acl/file.acl",
        },
        {
            // Same key as basic format "server" and "server_port"
            "server": "0.0.0.0",
            "server_port": 8388,
            "method": "chacha20-ietf-poly1305",
            // Read the actual password from environment variable PASSWORD_FROM_ENV
            "password": "${PASSWORD_FROM_ENV}"
        },
        {
            // AEAD-2022
            "server": "::",
            "server_port": 8390,
            "method": "2022-blake3-aes-256-gcm",
            "password": "3SYJ/f8nmVuzKvKglykRQDSgg10e/ADilkdRWrrY9HU=",
            // For Server (OPTIONAL)
            // Support multiple users with Extensible Identity Header
            // https://github.com/Shadowsocks-NET/shadowsocks-specs/blob/main/2022-2-shadowsocks-2022-extensible-identity-headers.md
            "users": [
                {
                    "name": "username",
                    // User's password must have the same length as server's password
                    "password": "4w0GKJ9U3Ox7CIXGU4A3LDQAqP6qrp/tUi/ilpOR9p4="
                }
            ],
            // For Client (OPTIONAL)
            // If EIH enabled, then "password" should have the following format: iPSK:iPSK:iPSK:uPSK
            // - iPSK is one of the middle relay servers' PSK, for the last `ssserver`, it must be server's PSK ("password")
            // - uPSK is the user's PSK ("password")
            // Example:
            // "password": "3SYJ/f8nmVuzKvKglykRQDSgg10e/ADilkdRWrrY9HU=:4w0GKJ9U3Ox7CIXGU4A3LDQAqP6qrp/tUi/ilpOR9p4="
        }
    ],

    // Global configurations for UDP associations
    "udp_timeout": 300, // Timeout for UDP associations (in seconds), 5 minutes by default
    "udp_max_associations": 512, // Maximum UDP associations to be kept in one server, unlimited by default

    // Options for Manager
    "manager_address": "127.0.0.1", // Could be a path to UNIX socket, /tmp/shadowsocks-manager.sock
    "manager_port": 5300, // Not needed for UNIX socket

    // DNS server's address for resolving domain names
    // For *NIX and Windows, it uses system's configuration by default
    //
    // Value could be IP address of DNS server, for example, "8.8.8.8".
    // DNS client will automatically request port 53 with both TCP and UDP protocol.
    //
    // - system, uses system provided API (`getaddrinfo` on *NIX)
    //
    // It also allows some pre-defined well-known public DNS servers:
    // - google (TCP, UDP)
    // - cloudflare (TCP, UDP)
    // - cloudflare_tls (TLS), enable by feature "dns-over-tls"
    // - cloudflare_https (HTTPS), enable by feature "dns-over-https"
    // - quad9 (TCP, UDP)
    // - quad9_tls (TLS), enable by feature "dns-over-tls"
    //
    // The field is only effective if feature "trust-dns" is enabled.
    "dns": "google",
    // Configure `cache_size` for "trust-dns" ResolverOpts. Set to "0" to disable DNS cache.
    "dns_cache_size": 0,

    // Mode, could be one of the
    // - tcp_only
    // - tcp_and_udp
    // - udp_only
    "mode": "tcp_only",

    // TCP_NODELAY
    "no_delay": false,

    // Enables `SO_KEEPALIVE` and set `TCP_KEEPIDLE`, `TCP_KEEPINTVL` to the specified seconds
    "keep_alive": 15,

    // Soft and Hard limit of file descriptors on *NIX systems
    "nofile": 10240,

    // Try to resolve domain name to IPv6 (AAAA) addresses first
    "ipv6_first": false,
    // Set IPV6_V6ONLY for all IPv6 listener sockets
    // Only valid for locals and servers listening on `::`
    "ipv6_only": false,

    // Outbound socket options
    // Linux Only (SO_MARK)
    "outbound_fwmark": 255,
    // FreeBSD only (SO_USER_COOKIE)
    "outbound_user_cookie": 255,
    // `SO_BINDTODEVICE` (Linux), `IP_BOUND_IF` (BSD), `IP_UNICAST_IF` (Windows) socket option for outbound sockets
    "outbound_bind_interface": "eth1",
    // Outbound socket bind() to this IP (choose a specific interface)
    "outbound_bind_addr": "11.22.33.44",

    // Balancer customization
    "balancer": {
        // MAX Round-Trip-Time (RTT) of servers
        // The timeout seconds of each individual checks
        "max_server_rtt": 5,
        // Interval seconds between each check
        "check_interval": 10,
        // Interval seconds between each check for the best server
        // Optional. Specify to enable shorter checking interval for the best server only.
        "check_best_interval": 5
    },

    // Service configurations
    // Logger configuration
    "log": {
        // Equivalent to `-v` command line option
        "level": 1,
        "format": {
            // Euiqvalent to `--log-without-time`
            "without_time": false,
        },
        // Equivalent to `--log-config`
        // More detail could be found in https://crates.io/crates/log4rs
        "config_path": "/path/to/log4rs/config.yaml"
    },
    // Runtime configuration
    "runtime": {
        // single_thread or multi_thread
        "mode": "multi_thread",
        // Worker threads that are used in multi-thread runtime
        "worker_count": 10
    }
}

SOCKS5 Authentication Configuration

The configuration file is set by socks5_auth_config_path in locals.

{
    // Password/Username Authentication (RFC1929)
    "password": {
        "users": [
            {
                "user_name": "USERNAME in UTF-8",
                "password": "PASSWORD in UTF-8"
            }
        ]
    }
}

Environment Variables

SS_SERVER_PASSWORD: A default password for servers that created from command line argument (--server-addr)
SS_SYSTEM_DNS_RESOLVER_FORCE_BUILTIN: "system" DNS resolver force use system's builtin (getaddrinfo in *NIX)

Supported Ciphers

AEAD 2022 Ciphers

2022-blake3-aes-128-gcm, 2022-blake3-aes-256-gcm
2022-blake3-chacha20-poly1305, 2022-blake3-chacha8-poly1305

These Ciphers require "password" to be a Base64 string of key that have exactly the same length of Cipher's Key Size. It is recommended to use ssservice genkey -m "METHOD_NAME" to generate a secured and safe key.

AEAD Ciphers

chacha20-ietf-poly1305
aes-128-gcm, aes-256-gcm

Stream Ciphers

plain or none (No encryption, only used for debugging or with plugins that ensure transport security)

Deprecated

table
aes-128-cfb, aes-128-cfb1, aes-128-cfb8, aes-128-cfb128
aes-192-cfb, aes-192-cfb1, aes-192-cfb8, aes-192-cfb128
aes-256-cfb, aes-256-cfb1, aes-256-cfb8, aes-256-cfb128
aes-128-ctr
aes-192-ctr
aes-256-ctr
camellia-128-cfb, camellia-128-cfb1, camellia-128-cfb8, camellia-128-cfb128
camellia-192-cfb, camellia-192-cfb1, camellia-192-cfb8, camellia-192-cfb128
camellia-256-cfb, camellia-256-cfb1, camellia-256-cfb8, camellia-256-cfb128
rc4-md5
chacha20-ietf

ACL

sslocal, ssserver, and ssmanager support ACL file with syntax like shadowsocks-libev. Some examples could be found in here.

Available sections

For local servers (sslocal, ssredir, ...)
- Modes:
  - [bypass_all] - ACL runs in BlackList mode. Bypasses all addresses that didn't match any rules.
  - [proxy_all] - ACL runs in WhiteList mode. Proxies all addresses that didn't match any rules.
- Rules:
  - [bypass_list] - Rules for connecting directly
  - [proxy_list] - Rules for connecting through proxies
For remote servers (ssserver)
- Modes:
  - [reject_all] - ACL runs in BlackList mode. Rejects all clients that didn't match any rules.
  - [accept_all] - ACL runs in WhiteList mode. Accepts all clients that didn't match any rules.
- Rules:
  - [white_list] - Rules for accepted clients
  - [black_list] - Rules for rejected clients
  - [outbound_block_list] - Rules for blocking outbound addresses.

Example

# SERVERS
# For ssserver, accepts requests from all clients by default
[accept_all]

# Blocks these clients
[black_list]
1.2.3.4
127.0.0.1/8

# Disallow these outbound addresses
[outbound_block_list]
127.0.0.1/8
::1
# Using regular expression
^[a-z]{5}\.baidu\.com
# Match exactly
|baidu.com
# Match with subdomains
||google.com
# An internationalized domain name should be converted to punycode
# |☃-⌘.com - WRONG
|xn----dqo34k.com
# ||джpумлатест.bрфa - WRONG
||xn--p-8sbkgc5ag7bhce.xn--ba-lmcq

# CLIENTS
# For sslocal, ..., bypasses all targets by default
[bypass_all]

# Proxy these addresses
[proxy_list]
||google.com
8.8.8.8

Useful Tools

ssurl is for encoding and decoding ShadowSocks URLs (SIP002). Example:

ss://YWVzLTI1Ni1jZmI6cGFzc3dvcmQ@127.0.0.1:8388/?plugin=obfs-local%3Bobfs%3Dhttp%3Bobfs-host%3Dwww.baidu.com

Notes

It supports the following features:

SOCKS5 CONNECT command
SOCKS5 UDP ASSOCIATE command (partial)
SOCKS4/4a CONNECT command
Various crypto algorithms
Load balancing (multiple servers) and server delay checking
SIP004 AEAD ciphers
SIP003 Plugins
SIP003u Plugin with UDP support
SIP002 Extension ss URLs
SIP022 AEAD 2022 ciphers
HTTP Proxy Supports (RFC 7230 and CONNECT)
Defend against replay attacks, shadowsocks/shadowsocks-org#44
Manager APIs, supporting Manage Multiple Users
ACL (Access Control List)
Support HTTP/HTTPS Proxy protocol

TODO

Documentation
Extend configuration format
Improved logging format (waiting for the new official log crate)
Support more ciphers without depending on libcrypto (waiting for an acceptable Rust crypto lib implementation)
Windows support.
Build with stable rustc ~~(blocking by crypto2)~~.
Support HTTP Proxy protocol
AEAD ciphers. (proposed in SIP004, still under discussion)
Choose server based on delay #152

License

The MIT License (MIT)

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

Stargazers over time

agragregra/start_html

2023-09-10T02:04:49Z

Стартовые шаблоны для адаптивной вёрстки сайтов

scalameta/munit

2023-09-11T01:40:59Z

Scala testing library with actionable errors and extensible APIs

MUnit

Scala testing library with actionable errors and extensible APIs.

Documentation

See the website: https://scalameta.org/munit/

LeifAndersen/CPSLambdaCalc

2023-09-10T02:04:52Z

Conatains a (likely extremely bugy) version of cps style lambda calculus in scala.

bcgov/map-geo-service

2023-09-10T02:05:04Z

seata/seata.github.io

2023-09-11T01:41:17Z

Seata website

mate-academy/Eco_cosmetics

2023-09-10T02:05:07Z

mate-academy/Kickstarter

2023-09-10T02:05:07Z

odoo/design-themes

2023-09-10T02:05:07Z

ocaml/opam-repository

2023-09-10T01:43:41Z

Run GUI applications and desktops in docker and podman containers. Focus on security.

x11docker: Run GUI applications in Docker or podman containers.

Avoid X security leaks and enhance container security

Introduction

x11docker allows to run graphical desktop applications (and entire desktops) in Linux containers.

Container tools like Docker, podman and nerdctl allow to run applications in an isolated container environment. Containers need much less resources than virtual machines for similar tasks.
Docker, podman and nerdctl do not provide a display server that would allow to run applications with a graphical user interface.
x11docker fills the gap. It runs an X display server and provides it to containers. X servers can run from host or in a container of image x11docker/xserver.
Additionally x11docker does some security setup to enhance container isolation and to avoid X security leaks. This allows a sandbox environment that fairly well protects the host system from possibly malicious or buggy software.

Software can be installed in a deployable image with a rudimentary Linux system inside. This can help to run or deploy software that is difficult to install on several systems due to dependency issues. It is possible to run outdated versions or latest development versions side by side. Files to work on can be shared between host and container.

x11docker wiki provides some how-to's for basic setups without x11docker.

Docker Desktop or Docker Engine

Since a while Docker distributes a version called "Docker Desktop" that runs Docker in a QEMU VM. x11docker is not designed to support this VM based version. Instead, use x11docker with the native "Docker Engine Server version" that uses your host kernel to run containers.

If you install Docker from your distribution's repository, you'll likely get this native version.
The supported native Docker Engine package name is mostly docker.io or docker-ce, in opposite to the non-supported VM based docker-desktop package.
If you prefer podman over Docker, you don't need to care about this difference.

TL;DR

For a quick start:

Install x11docker with:

curl -fsSL https://raw.githubusercontent.com/mviereck/x11docker/master/x11docker | sudo bash -s -- --update

Install dependencies:
- Either pull image x11docker/xserver or install at least nxagent or xpra and xephyr.
Run a GUI in container with:
```
x11docker IMAGENAME [COMMAND]
```
Add options:
- --desktop for a desktop environment in image.
- --gpu for hardware acceleration.

Examples:

x11docker x11docker/xfce thunar
x11docker --desktop x11docker/xfce
x11docker --gpu x11docker/xfce glxgears

Features

Focus on security:
- Avoids X security leaks by running additional X servers.
- Restricts container capabilities to bare minimum.
- Container user is same as host user to avoid root in container.
Low dependencies:
- No obliging dependencies on host beside X and one of docker, podman or nerdctl. Recommended: nxagent and Xephyr, alternatively image x11docker/xserver.
- No dependencies inside of images except for some optional features.
Several optional features like GPU, sound, webcam and printer support.
Remote access with SSH, VNC or HTML5 possible.
Easy to use. Examples:
- x11docker x11docker/fvwm xterm
- x11docker --desktop --size 320x240 x11docker/lxde (needs nested X server Xephyr)

Supported systems

x11docker runs on Linux and (with some setup and limitations) on MS Windows. x11docker does not run on macOS except in a Linux VM.

Terminal syntax

Just type x11docker IMAGENAME [COMMAND].

Get an overview of options with x11docker --help.
- For desktop environments in image add option -d, --desktop.
- For internet access use option -I, --network.
- To run without X at all use option -t, --tty.
- Get an interactive TTY with option -i, --interactive.
- See generated container backend command (and further infos) with option --debug.
If startup fails, look at chapter Troubleshooting.

General syntax:

Usage:
To run a container on a new X server:
  x11docker IMAGE
  x11docker [OPTIONS] IMAGE [COMMAND]
  x11docker [OPTIONS] -- IMAGE [COMMAND [ARG1 ARG2 ...]]
  x11docker [OPTIONS] -- CUSTOM_RUN_OPTIONS -- IMAGE [COMMAND [ARG1 ARG2 ...]]
To run a host application on a new X server:
  x11docker [OPTIONS] --backend=host COMMAND
  x11docker [OPTIONS] --backend=host -- COMMAND [ARG1 ARG2 ...]
  x11docker [OPTIONS] --backend=host -- -- COMMAND [ARG1 ARG2 ...] -- [ARG3]
To run only an empty new X server:
  x11docker [OPTIONS] --xonly

CUSTOM_RUN_OPTIONS are just added to the docker|podman|nerdctl run command without a serious check by x11docker.

Options

Description of some commonly used feature options.

Some of these options have dependencies on host and/or in image. Compare wiki: feature dependencies.
For often used option combinations you can make shortcuts with option --preset.

Choice of X servers and Wayland compositors

If no X server option is specified, x11docker automatically chooses one depending on installed dependencies and on given or missing options --desktop, --gpu and --wayland. Most recommended are nxagent and Xephyr.

Overview of all possible X server and Wayland options.
- Hints to use option --xorg within X.
- Use option -t, --tty to run without X at all.

Desktop or seamless mode

x11docker assumes that you want to run a single application in seamless mode, i.e. a single window on your regular desktop. If you want to run a desktop environment in image, add option --desktop.

Seamless mode is supported with options --nxagent and --xpra. As a fallback insecure option --hostdisplay is possible.
Desktop mode with --desktop is supported with all X server options except --hostdisplay. If available, x11docker prefers --xephyr and --nxagent.
Special case: Single applications with a window manager (option --wm).
- If neither nxagent nor xpra are installed, but x11docker finds a desktop capable X server like Xephyr, it avoids insecure option --hostdisplay and runs Xephyr with a window manager.

Internet access

By default x11docker disables Network access for containers with --network=none because it targets best possible container isolation. To allow internet access set option -I or --network.

Shared folders, volumes and HOME in container

Changes in a running container system will be lost, the created container will be discarded. For persistent data storage you can share host directories or volumes:

Option -m, --home creates a host directory in ~/.local/share/x11docker/IMAGENAME that is shared with the container and mounted as its HOME directory. Files in container home and user configuration changes will persist. x11docker creates a softlink from ~/.local/share/x11docker to ~/x11docker.
- You can specify another host directory for container HOME with --home=DIR.
- You can specify a volume for container HOME with --home=VOLUME.
Option --share PATH mounts a host file or folder at the same location in container.
- You can also specify a volume with --share VOLUME.
- --share PATH:ro restricts to read-only access.
- Device files in /dev are supported, too.
Special cases for $HOME:
- --home=$HOME will use your host home as container home. Discouraged, use with care.
- --share $HOME will symlink your host home as a subfolder of container home.

Note that x11docker copies files from /etc/skel in container to HOME if HOME is empty. That allows to provide predefined user configurations in the image.

GPU hardware acceleration

Hardware acceleration for OpenGL is possible with option -g, --gpu.

This will work out of the box in most cases with open source drivers on host. Otherwise have a look at wiki: feature dependencies.
Closed source NVIDIA drivers need some setup and support less x11docker X server options for driver version < v470.x and Xwayland < v22.1.2.

Clipboard

Clipboard sharing is possible with option -c, --clipboard [=ARG].

Optional arguments superv and altv only provide host clipboard content to container if keys [SUPER][v] or [ALT][v] are pressed.
Optional argument oneway only transfers clipboard content from container to host.

Sound

Sound is possible with options -p, --pulseaudio and --alsa.

For pulseaudio sound with --pulseaudio you need pulseaudio on host and pulseaudio (at least the pulseaudio client libraries) in image. Compare wiki: feature dependencies.
For ALSA sound with --alsa you might need to specify a sound card with e.g. --alsa=Generic. Get a list of available sound cards with aplay -l.

Webcam

Webcams on host can be shared with option --webcam.

If webcam application in image fails, install --gpu dependencies in image. Compare wiki: feature dependencies.
guvcview needs --pulseaudio or --alsa.
cheese and gnome-ring need --init=systemd.

Printer

Printers on host can be provided to container with option --printer.

It needs cups on host, the default printer server for most linux distributions.
The container needs cups client libraries in image. Compare wiki: feature dependencies.

Language locales

x11docker provides option --lang for flexible language locale settings.

--lang without an argument sets LANG in container to same as on host. Same as --lang=$LANG
x11docker will check on container startup if the desired locale is already present in image and enable it.
If x11docker does not find the locale, it creates it on container startup. This needs some locale packages in image. Compare wiki: feature dependencies.
Examples: --lang=de for German, --lang=zh_CN for Chinese, --lang=ru for Russian, --lang=$LANG for your host locale.
For support of chinese, japanese and korean characters install a font like fonts-arphic-uming in image.
Keyboard input methods like fcitx-pinyin are possible with some container setup. Compare ticket #269.

Wayland

To run Wayland instead of an X server x11docker provides options --wayland, --weston, --kwin and --hostwayland. For further description loot at wiki: Description of Wayland options.

Option --wayland automatically sets up a Wayland environment. It regards option --desktop.
Options --weston and --kwin run Wayland compositors weston or kwin_wayland.
Option --hostwayland can run applications seamless on host Wayland desktops like Gnome 3, KDE 5 and Sway.
Example: xfce4-terminal on Wayland: x11docker --wayland x11docker/xfce xfce4-terminal

Init system

x11docker supports several init systems as PID 1 in container with option --init. Init in container solves the zombie reaping issue. As default x11docker uses tini in/usr/bin/docker-init. Also available are systemd, SysVinit, runit, OpenRC and s6-overlay. elogind is supported, too. Look at wiki: Init systems in Docker.

DBus

Some desktop environments and applications need a running DBus system daemon and/or DBus user session. DBus options need dbus in image.

use --dbus to run a DBus user session daemon.
A DBus system daemon will be started automatically with init systems systemd, openrc, runit and sysvinit (option --init).
- It is also possible to run a DBus system daemon with --dbus=system without advanced init systems. However, this causes trouble in some cases and is not recommended in general.
use --hostdbus to connect to host DBus user session.
use --share /run/dbus/system_bus_socket to share host DBus system socket.

Container runtime

It is possible to run containers with different backends following the OCI runtime specification. Docker's default runtime is runc. You can specify another one with option --runtime=RUNTIME. Container runtimes known and supported by x11docker are:

runc: Docker default.
nvidia: Specialized fork of runc to support nvidia/nvidia-docker images.
crun: Fast and lightweight alternative to runc with same functionality.
oci: Runtime reported in #205, no documentation found. Handled by x11docker like runc.
sysbox-runtime: Based on runc, aims to enhance container isolation. Support is experimental yet. Needs Sybox>=0.5.0 and kernel version >=5.12.

Using different runtimes is well tested for rootful Docker, but not for other backend setups.

Example: possible runtime configuration in /etc/docker/daemon.json:

{
  "default-runtime": "runc",
  "runtimes": {
    "crun": {
      "path": "/usr/local/bin/crun",
      "runtimeArgs": []
    },
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    },
    "sysbox-runc": {
      "path": "/usr/bin/sysbox-runc"
    }
  }
}

Backends other than docker

x11docker supports container tools Docker, podman and nerdctl with option --backend=BACKEND in rootful and rootless mode. Supported --backend arguments: docker podman nerdctl host

Container backends:

By default x11docker tries to run docker.
- To change the default --backend=docker to another one like --backend=podman, create a default file for --preset.
Recommended for rootful container backend: docker or podman
Recommended for rootless container backend: podman
- Only podman allows option --home in rootless mode yet.
- Only podman provides useful file ownerships with option --share in rootless mode yet.
--backend=nerdctl is experimental yet. It supports rootful and rootless mode. nerdctl is in heavy development stage.

Other supported backends that are in fact no containers:

--backend=host runs a host application on a new X server. No containerization is involved.

Preconfiguration with --preset

For often used option combinations you might want to use option --preset FILENAME to have a command shortcut. FILENAME is a file in ~/.config/x11docker/preset or in /etc/x11docker/preset containing some x11docker options.

Example multimedia: Create a file ~/.config/x11docker/preset/multimedia:
```
--gpu
--webcam
--printer
--pulseaudio
--clipboard
--share ~/Videos
--share ~/Music
```
Use it like: x11docker --preset=multimedia jess/vlc

Example deepin desktop: Instead of long command

x11docker --desktop --init=systemd --gpu --pulseaudio --home -- --cap-add=IPC_LOCK -- x11docker/deepin

you can create a file ~/.config/x11docker/preset/deepin containing the desired options and even the image name:

--desktop 
--init=systemd
--gpu
--pulseaudio
--home
-- 
--cap-add=IPC_LOCK
-- 
x11docker/deepin

Run with: x11docker --preset=deepin

Default preset for all x11docker sessions

You can create a default preset file that is applied on all x11docker sessions. You can think of it as a configuration file for x11docker.

Example: To always use podman instead of docker, create a file with name default in ~/.config/x11docker/preset or in /etc/x11docker/preset with content:
```
--backend=podman
```
This will cause x11docker to always use podman instead of docker unless specified otherwise in the x11docker command.

The same way you can specify other and more options as default, e.g. --homebasedir=/my/containerhome/path. Note that a local user default file will supersede a system wide default file.

Security

Scope of x11docker is to run containerized GUI applications while preserving and improving container isolation. Core concept is:

Runs a second X server to avoid X security leaks.
- This in opposite to widespread solutions that share host X socket of display :0, thus breaking container isolation, allowing keylogging and remote host control. (However, x11docker provides this with fallback option --hostdisplay).
- Authentication is done with MIT-MAGIC-COOKIE, stored separate from file ~/.Xauthority.
Creates container user similar to host user to avoid root in container.
- You can also specify another user with --user=USERNAME or a non-existing one with --user=UID:GID.
- Disables possible root password and deletes entries in /etc/sudoers.
  - If you want root permissions in container, use option --sudouser that allows su and sudo with password x11docker.
- If you want to use USER specified in image instead, set option --user=RETAIN. x11docker won't change container's /etc/passwd or /etc/sudoers in that case. Option --home won't be available.
Reduces container capabilities to bare minimum.
- Sets run option --cap-drop=ALL to drop all capabilities. Most applications don't need them.
- Sets run option --security-opt=no-new-privileges.
- These restrictions can be disabled with x11docker option --cap-default or reduced with --sudouser, --newprivileges.

That being said, the default docker capabilities and the seccomp/SELinux/apparmor profiles are set up well to protect the host system. Nonetheless, x11docker follows the principle of least privilege. Containers should not have capabilities or privileges that they don't need for their job.

Security weaknesses

Possible SELinux restrictions are degraded for x11docker containers with run option --security-opt label=type:container_runtime_t to allow access to new X unix socket. A more restrictive solution is desirable. Compare: SELinux and Docker: allow access to X unix socket in /tmp/.X11-unix
A possible user namespace remapping setup is disabled to allow options --home and --share without file ownership issues.
- This is less an issue because x11docker already avoids root in container.
- Exception: User namespace remapping is not disabled for --user=RETAIN.
x11docker provides several different X server options. Each X server involved might have its individual vulnerabilities. x11docker only covers well-known X security leaks that result from X11 protocol design.
- An additional security layer for most supported X servers is set up if image x11docker/xserver is available. It will be used automatically in most cases if available. Enforce its usage with option --xc=yes.

Options degrading container isolation

x11docker shows warning messages in terminal if chosen options degrade container isolation. Note that x11docker does not check custom DOCKER_RUN_OPTIONS.

Most important:

--hostdisplay shares host X socket of display :0 instead of running a second X server.
- Danger of abuse is reduced providing so-called untrusted cookies, but do not rely on this.
- If additionally using --gpu or --clipboard, option --ipc=host and trusted cookies are enabled and no protection against X security leaks is left.
- If you don't care about container isolation, x11docker --hostdisplay --gpu is an insecure but quite fast setup without any overhead.
--gpu allows access to GPU hardware. This can be abused to get window content from host (palinopsia bug) and makes GPU rootkits like keyloggers possible.
--pulseaudio and --alsa allow catching audio output and microphone input from host.

Rather special options reducing security, but not needed for regular use:

--sudouser allows su and sudo with password x11dockerfor container user. If an application somehow breaks out of container, it can harm your host system. Allows many container capabilities that x11docker would drop otherwise.
--cap-default disables x11docker's container security hardening and falls back to default container capabilities as provided by the backends docker, podman or nerdctl. If an application somehow breaks out of container, it can harm your host system.
--init=systemd|sysvinit|openrc|runit allow some container capabilities that x11docker would drop otherwise. --init=systemd also shares access to /sys/fs/cgroup. Some processes will run as root in container. If a root process somehow breaks out of container, it can harm your host system. Allows many container capabilities that x11docker would drop otherwise.
--hostdbus allows communication over DBus with host applications.

Sandbox

Container isolation enhanced with x11docker allows to use containers as a sandbox that fairly well protects the host system from possibly malicious or buggy software. Though, no sandbox solution in the wild can provide a perfect secure protection, and Docker even with enhanced security settings from x11docker is no exception.

Using x11docker as a sandbox is not intended to run obviously evil software. Rather use it as:

Compatibility environment to run software that is hard or impossible to install on host due to dependency issues.
Development environment to collect libraries, compiler and so on to keep the host clean.
Development environment to mitigate damage caused by unexpected/buggy behaviour.
Security layer for software that may be malicious in worst case. Examples: Internet browser with enabled javascript, or wine with MS Windows applications.

x11docker already restricts process capabilities. You can additionally restrict access to CPU and RAM with option --limit. As default --limit restricts to 50% of available CPUs and 50% of currently free RAM. Another amount can be specified with --limit=FACTOR with a FACTOR greater than zero and less than or equal one.

For more custom fine tuning have a look at Docker documentation: Limit a container's resources.

WARNING: There is no restriction that can prevent the container from flooding the hard disk storing the container or in shared folders.

Security and feature check

To check container isolation and some feature options use image x11docker/check and try out with several options.

An insecure setup is x11docker --hostdisplay --gpu x11docker/check. It fairly well demonstrates common X security leaks.
Add options like --pulseaudio --alsa --webcam --clipboard --printer to check their functionality.

Installation

Note that x11docker is just a bash script without library dependencies. Basically it is just a wrapper for X servers and container backends docker, podman and nerdctl. To allow advanced usage of x11docker abilities have a look at chapter Dependencies.

Installation from distribution repositories

x11docker is available as a package in some distributions.

Stable releases:

Latest git master/beta version:

Much thanks to the maintainers who decided to provide these packages! There are some hints for packaging x11docker in the wiki.

Manual installation

Installation options

As root you can install, update and remove x11docker in system directories to be available system-wide:

x11docker --install : install x11docker from current directory. (Useful to install from an extracted zip file or a cloned git repository.)
x11docker --update : download and install latest release from github.
x11docker --update-master : download and install latest master version from github.
x11docker --remove : remove all files installed by x11docker.
- Note: This does not remove ~/.local/share/x11docker where it stores persistent files of option --home.
x11docker --remove-oldprefix : Before version 7.6.0 x11docker installed itself into /usr/bin. Now it installs into /usr/local/bin. Use --remove-oldprefix to remove /usr/bin installations.

To see the difference between current and coming updated version, you can use optional argument diff for --update and --update-master. Example: x11docker --update-master=diff will show you the code changes from your current installation to latest master/beta version without installing it.

Installed files

What the installation does (just for information):

Copies script x11docker to /usr/local/bin.
Installs icon x11docker.png below /usr/share/icons using xdg-icon-resource.
Copies documentation README.md, CHANGELOG.md and LICENSE.txt to /usr/local/share/doc/x11docker.
Stores man page for x11docker in /usr/local/share/man/man1/x11docker.1.gz.

Shortest way for first installation:

For systems using sudo:

curl -fsSL https://raw.githubusercontent.com/mviereck/x11docker/master/x11docker | sudo bash -s -- --update

Directly as root:

curl -fsSL https://raw.githubusercontent.com/mviereck/x11docker/master/x11docker | bash -s -- --update

Minimal installation

You can run x11docker from an arbitrary location with bash x11docker or ./x11docker. For minimal system-wide installation make x11docker executable with chmod +x x11docker and move it to /usr/local/bin (or another location in PATH). Other files than script x11docker itself are not essential.

Installation on MS Windows

x11docker can run natively on MS Windows electively in one of:

Further information at wiki: x11docker on MS Windows.

Deinstallation

You can remove x11docker with x11docker --remove. That will remove the files listed above. It will also remove ~/.cache/x11docker and stop all running x11docker containers. x11docker will not remove:

Files and folders for persistent data storage with option --home. These are:
- ~/.local/share/x11docker where persistent data is stored.
- Softlink ~/x11docker that points there.
Folders you might have created yourself for x11docker:
- ~/.local/share/x11docker
- ~/.config/x11docker

Dependencies

x11docker can run with standard system utilities without additional dependencies on host or in image.

As a core it only needs bash and one of docker, podman or nerdctl to run containers on X.
x11docker also needs an X server. x11docker can automatically use image x11docker/xserver that provides most optional x11docker dependencies and several X servers and Wayland compositors so you won't need to install them on host.
- If you prefer to install dependencies on host:
  - The recommended base commands are: nxagent Xephyr weston Xwayland xdotool xauth xinit xclip xhost xrandr xdpyinfo. Some of them are probably already installed.
  - See wiki: Dependencies - Recommended base for a package list matching your distribution.

Dependencies in image:

Some feature options have additional dependencies on host and/or in image. This affects especially options --gpu, --printer and --pulseaudio.
Compare wiki: feature dependencies.

Troubleshooting

For troubleshooting run x11docker in a terminal. x11docker shows warnings if something is insecure, missing or going wrong. Also it shows notes if options don't work and fallbacks are used. It might give hints to fix some issues.

Core checks

1. Make sure your x11docker version is up to date with x11docker --update (latest release) or x11docker --update-master (latest beta).

2. Carefully read the regular x11docker messages. Often they already give a hint what to do.

Use option -D, --debug to see some internal messages.
Use option -v, --verbose to see full logfile output.
You can find the latest dispatched logfile at ~/.cache/x11docker/x11docker.log.

3. Try another X server option.

Some applications fail with fallback option --hostdisplay. Add --clipboard to disable some security restrictions of --hostdisplay.
If that does not help, install additional X servers. The most stable and reliable option is --xephyr.

Privilege checks

Some applications need more privileges or capabilities than x11docker provides by default. One attempt is to allow several privileges until the setup works. Than reduce privileges to find out which are needed indeed. (Note the -- in the following commands, do not miss them).

1. Adding privileges:

Try x11docker --cap-default IMAGENAME
Try x11docker --cap-default --ipc=host --network=host IMAGENAME
Try x11docker --cap-default --ipc=host --network=host --share /run/udev/data:ro -- --cap-add ALL --security-opt seccomp=unconfined --security-opt apparmor=unconfined --privileged -- IMAGENAME

2. Reducing privileges:

Drop options one by one in this order: --privileged --security-opt apparmor=unconfined --security-opt seccomp=unconfined --cap-add ALL --share /run/udev/data:ro --network=host --ipc=host --cap-default. Only leave options that are needed to keep the setup working.
Option --cap-default might already be enough. It allows default container capabilities as docker|podman|nerdctl would do on themself.
- You can just stop debugging and reducing here if you like to.
- You can try to reduce --cap-default. Partially remove additional options to find out which one(s) are needed:
  - First try x11docker --newprivileges -- IMAGENAME
  - Than try and reduce: x11docker --newprivileges -- --cap-add=SETPCAP --cap-add=MKNOD --cap-add=AUDIT_WRITE --cap-add=CHOWN --cap-add=NET_RAW --cap-add=DAC_OVERRIDE --cap-add=FOWNER --cap-add=FSETID --cap-add=KILL --cap-add=SETGID --cap-add=SETUID --cap-add=NET_BIND_SERVICE --cap-add=SYS_CHROOT --cap-add=SETFCAP -- IMAGENAME
--cap-add ALL should not be considered to be a solution.
- Drop capabilities from following command to find the one(s) you need: x11docker --cap-default -- --cap-add=SYS_MODULE --cap-add=SYS_RAWIO --cap-add=SYS_PACCT --cap-add=SYS_ADMIN --cap-add=SYS_NICE --cap-add=SYS_RESOURCE --cap-add=SYS_TIME --cap-add=SYS_TTY_CONFIG --cap-add=AUDIT_CONTROL --cap-add=MAC_OVERRIDE --cap-add=MAC_ADMIN --cap-add=NET_ADMIN --cap-add=SYSLOG --cap-add=DAC_READ_SEARCH --cap-add=LINUX_IMMUTABLE --cap-add=NET_BROADCAST --cap-add=IPC_LOCK --cap-add=IPC_OWNER --cap-add=SYS_PTRACE --cap-add=SYS_BOOT --cap-add=LEASE --cap-add=WAKE_ALARM --cap-add=BLOCK_SUSPEND --cap-add=AUDIT_READ -- IMAGENAME
- Many of these capabilities are rather dangerous and should not be allowed for a container. Especially to mention is SYS_ADMIN.
Option --privileged should not be considered to be a solution. Basically it allows arbitrary access to the host for container applications.
- Likely you need to share a device file in /dev, e.g. something like --share /dev/vboxdrv.
--ipc=host and --network=host severely reduce container isolation. Better solutions are desirable.

3. Open a ticket to ask for possibilities how to optimize the privilege setup.

Other checks

1. Container user: By default x11docker sets up an unprivileged container user similar to your host user.

The image may have a USER specification and be designed for this user.
- Check for a USER specification in image with docker inspect --format '{{.Config.User}}' IMAGENAME
- You can enable this predefined user with --user=RETAIN
The container might need a root user. Try with --user=root, maybe add --cap-default.

2. Init and DBus

A few applications need a DBus user daemon. Install dbus in image and try option --dbus.
A few applications need systemd and/or a running DBus system daemon. Install systemd in image and try option --init=systemd.

3. Architecture check of host OS and image

The image may not be built for the architecture of your host OS. (ie. Image is built for amd64 but your OS runs on arm, e.g. on a RaspBerry PI). With a mismatch the container will quit unexpectedly & x11docker may emit the error dockerrc(): Did not receive PID of PID1 in container.
- You can check the image architecture with docker inspect --format {{.Architecture}} IMAGENAME.
- You can check the host architecture with uname -m.
- For further information and multi-arch setups look at wiki: Multi-arch setups with QEMU.

Contact

Feel free to open a ticket if you have a question or encounter an issue.

Issues

If reporting an issue:

Have a look at chapter Troubleshooting.
Most times it makes sense to store the --verbose output (or ~/.cache/x11docker/x11docker.log) at pastebin.com.
- Personal information in the logfile is mainly the user name (also in paths) and some hardware and system information.

Contributing

If you want to contribute to x11docker, please open a ticket before creating a pull request. Often it is possible to accomplish desired tasks with already available options. Note that x11docker is considered to be stable and changes other than bug fixes are less likely to be merged. In general new features are not intended.

Support

Please open a ticket if you need support. Please note that x11docker is a non-commercial project maintained in free time. I'll help where I can, but there is no organisation behind x11docker that can provide large scale support.

Donation

If you like to make a donation: Thank you! :) Please have a look at my sponsor site.

Instead of sponsoring me, you can as well spend some money to Galsan Tschinag in Mongolia (Wikipedia). One of his great projects is the afforestation of Mongolia. A donation account in Germany is provided by Förderverein Mongolei e.V..

Förderverein Mongolei e.V.
IBAN DE7261290120 0394 3660 00
BIC GENODES1NUE
Volksbank Kirchheim-Nürtingen

I personally know some of the people behind this. I assure that they are trustworthy and have a great heart and soul and do a good thing.

Examples

Some x11docker image examples with desktop environments can be found on docker hub. A special one to check features and container isolation is x11docker/check.

Docker does not provide automated builds for free anymore, so the images are becoming outdated. Instead of pulling them, rather build them locally with x11docker option --build.

Some x11docker Dockerfiles are provided at https://github.com/mviereck?tab=repositories; look for repositories beginning with dockerfile-x11docker. You can build locally from these Dockerfiles with x11docker option --build. Example:

x11docker --build x11docker/fvwm

Single applications

Application	x11docker command
Xfce4 Terminal	`x11docker x11docker/xfce xfce4-terminal`
GLXgears with hardware acceleration	`x11docker --gpu x11docker/xfce glxgears`
Kodi media center with hardware acceleration, Pulseaudio sound and shared `Videos` folder. For setup look at ehough/docker-kodi.	`x11docker --gpu --pulseaudio --share ~/Videos erichough/kodi`.
XaoS fractal generator	`x11docker patricknw/xaos`
Telegram messenger with persistent `HOME` for configuration storage	`x11docker --home xorilog/telegram`
Firefox with shared `Download` folder and internet access.	`x11docker -I --share $HOME/Downloads -- --tmpfs /dev/shm -- jess/firefox`
Tor browser	`x11docker -I jess/tor-browser`
Chromium browser with restricted resource usage	`x11docker -I --limit -- jess/chromium --no-sandbox`
VLC media player with shared `Videos` folder and Pulseaudio sound	`x11docker --pulseaudio --share=$HOME/Videos jess/vlc`
GNU Octave Scientific Programming Language built for arm & arm64	`x11docker aptman/dbhi:bionic-octave octave`

Desktop environments

Desktop environment (most based on Debian)	x11docker command
Cinnamon	`x11docker --desktop --gpu --init=systemd --cap-default x11docker/cinnamon`
deepin (website) (3D desktop from China)	`x11docker --desktop --gpu --init=systemd -- --cap-add=IPC_LOCK -- x11docker/deepin`
Enlightenment (based on Void Linux)	`x11docker --desktop --gpu --runit x11docker/enlightenment`
Fluxbox (based on Debian, 87 MB)	`x11docker --desktop x11docker/fluxbox`
FVWM (based on Alpine, 22.5 MB)	`x11docker --desktop x11docker/fvwm`
Gnome 3	`x11docker --desktop --gpu --init=systemd x11docker/gnome`
KDE Plasma on X	`x11docker --desktop --gpu --init=systemd x11docker/kde-plasma`
KDE Plasma on Wayland	`x11docker --kwin --wayland x11docker/kde-plasma plasmashell`
KDE Plasma as nested Wayland compositor	`x11docker --gpu --init=systemd -- --cap-add SYS_RESOURCE -- x11docker/kde-plasma startplasma-wayland`
Lumina (website) (based on Void Linux)	`x11docker --desktop x11docker/lumina`
LiriOS (based on Fedora)	`x11docker --desktop --gpu lirios/unstable`
LXDE	`x11docker --desktop x11docker/lxde`
LXDE with wine and PlayOnLinux and a persistent `HOME` folder to preserve installed Windows applications, and with Pulseaudio sound.	`x11docker --desktop --home --pulseaudio x11docker/lxde-wine`
LXQt	`x11docker --desktop x11docker/lxqt`
Mate	`x11docker --desktop x11docker/mate`
Trinity (website) (successor of KDE 3)	`x11docker --desktop x11docker/trinity`
Xfce	`x11docker --desktop x11docker/xfce`

Adjust images for your needs

For persistent changes of image system adjust Dockerfile and rebuild. To add custom applications to x11docker example images you can create a new Dockerfile based on them. Example:

# xfce desktop with VLC media player
FROM x11docker/xfce
RUN apt-get update && apt-get install -y vlc

Screenshots

More screenshots are stored in screenshot branch

x11docker --desktop x11docker/lxqt

x11docker --desktop x11docker/lxde-wine

x11docker --desktop --gpu --init=systemd -- --cap-add=IPC_LOCK --security-opt seccomp=unconfined -- x11docker/deepin

wodby/docker4drupal

2023-09-10T01:43:41Z

Docker-based Drupal stack

Introduction

Docker4Drupal is a set of docker images optimized for Drupal. Use docker-compose.yml file from the latest stable release to spin up local environment on Linux, Mac OS X and Windows.

Read the docs on how to use
Ask questions on Slack
Follow @wodbycloud for future announcements

Stack

The Drupal stack consist of the following containers:

Container	Versions	Image	ARM64 support	Enabled by default
Nginx	1.25, 1.24, 1.23	wodby/nginx	✓	✓
Apache	2.4	wodby/apache	✓
Drupal	10, 9, 7	wodby/drupal	✓	✓
PHP	8.2, 8.1, 8.0	wodby/drupal-php	✓
Crond		wodby/drupal-php	✓	✓
MariaDB	11, 10.11-10.4	wodby/mariadb	✓	✓
PostgreSQL	15, 14, 13, 12, 11	wodby/postgres	✓
Redis	7, 6, 5	wodby/redis	✓
Memcached	1	wodby/memcached
Varnish	6.0, 4.1	wodby/varnish	✓
Node.js	18, 16, 14	wodby/node
Drupal node	1.0	wodby/drupal-node
Solr	8, 7, 6, 5	wodby/solr
Zookeeper	3.8	zookeeper
Elasticsearch	7, 6	wodby/elasticsearch
Kibana	7, 6	wodby/kibana
OpenSMTPD	6.0	wodby/opensmtpd
Mailhog	latest	mailhog/mailhog		✓
AthenaPDF	2.16.0	arachnysdocker/athenapdf-service
Rsyslog	latest	wodby/rsyslog
Webgrind	1	wodby/webgrind
Xhprof viewer	latest	wodby/xhprof
Adminer	4.6	wodby/adminer
phpMyAdmin	latest	phpmyadmin/phpmyadmin
Selenium chrome	3.141	selenium/standalone-chrome
Traefik	latest	_/traefik	✓	✓

Supported Drupal versions: 10 / 9 / 7

Documentation

Full documentation is available at https://wodby.com/docs/stacks/drupal/local.

Image's tags

Images' tags format is [VERSION]-[STABILITY_TAG] where:

[VERSION] is the version of an application (without patch version) running in a container, e.g. wodby/nginx:1.15-x.x.x where Nginx version is 1.15 and x.x.x is a stability tag. For some images we include both major and minor version like PHP 7.2, for others we include only major like Redis 5.

[STABILITY_TAG] is the version of an image that corresponds to a git tag of the image repository, e.g. wodby/mariadb:10.2-3.3.8 has MariaDB 10.2 and stability tag 3.3.8. New stability tags include patch updates for applications and image's fixes/improvements (new env vars, orchestration actions fixes, etc). Stability tag changes described in the corresponding a git tag description. Stability tags follow semantic versioning.

We highly encourage to use images only with stability tags.

Maintenance

We regularly update images used in this stack and release them together, see releases page for full changelog and update instructions. Most of routine updates for images and this project performed by the bot via scripts located at wodby/images.

Beyond local environment

Docker4Drupal is a project designed to help you spin up local environment with docker-compose. If you want to deploy a consistent stack with orchestrations to your own server, check out Drupal stack on Wodby .

Other Docker4x projects

License

This project is licensed under the MIT open source license.

CISOfy/lynis

2023-09-10T02:05:14Z

Lynis - Security auditing tool for Linux, macOS, and UNIX-based systems. Assists with compliance testing (HIPAA/ISO27001/PCI DSS) and system hardening. Agentless, and installation optional.

Documentation

Do you like this software? Star the project and become a stargazer.

lynis

Lynis - Security auditing and hardening tool, for UNIX-based systems.

Lynis is a security auditing tool for systems based on UNIX like Linux, macOS, BSD, and others. It performs an in-depth security scan and runs on the system itself. The primary goal is to test security defenses and provide tips for further system hardening. It will also scan for general system information, vulnerable software packages, and possible configuration issues. Lynis was commonly used by system administrators and auditors to assess the security defenses of their systems. Besides the "blue team," nowadays penetration testers also have Lynis in their toolkit.

We believe software should be simple, updated on a regular basis, and open. You should be able to trust, understand, and have the option to change the software. Many agree with us, as the software is being used by thousands every day to protect their systems.

Goals

The main goals are:

Automated security auditing
Compliance testing (e.g. ISO27001, PCI-DSS, HIPAA)
Vulnerability detection

The software (also) assists with:

Configuration and asset management
Software patch management
System hardening
Penetration testing (privilege escalation)
Intrusion detection

Audience

Typical users of the software:

System administrators
Auditors
Security officers
Penetration testers
Security professionals

Installation

There are multiple options available to install Lynis.

Software Package

For systems running Linux, BSD, and macOS, there is typically a package available. This is the preferred method of obtaining Lynis, as it is quick to install and easy to update. The Lynis project itself also provides packages in RPM or DEB format suitable for systems systems running: CentOS, Debian, Fedora, OEL, openSUSE, RHEL, Ubuntu, and others.

Some distributions may also have Lynis in their software repository:

Note: Some distributions don't provide an up-to-date version. In that case it is better to use the CISOfy software repository, download the tarball from the website, or download the latest GitHub release.

Git

The very latest developments can be obtained via git.

Clone or download the project files (no compilation nor installation is required) ;
```
 git clone https://github.com/CISOfy/lynis
```
Execute:
```
 cd lynis && ./lynis audit system
```

If you want to run the software as root (or sudo), we suggest changing the ownership of the files. Use chown -R 0:0 to recursively alter the owner and group and set it to user ID 0 (root). Otherwise Lynis will warn you about the file permissions. After all, you are executing files owned by a non-privileged user.

Enterprise Version

This software component is also part of an enterprise solution. Same quality, yet with more functionality.

Focus areas include compliance (PCI DSS, HIPAA, ISO27001, and others). The Enterprise version comes with:

a web interface;
dashboard and reporting;
hardening snippets;
improvement plan (based on risk);
commercial support.

Documentation

Full documentation: https://cisofy.com/documentation/lynis/.

Customization

If you want to create your own tests, have a look at the Lynis software development kit.

Security

We participate in the CII best practices badge program of the Linux Foundation.

Media and Awards

Lynis is collecting some awards along the way and we are proud of that.

2016
- Best of Open Source Software Awards 2016.
- Article by TechRepublic, considering Lynis a "must-have" tool: How to quickly audit a Linux system from the command line
2015
- Best of Open Source Software Awards 2015.
2014
2013

Contribute

We love contributors.

Do you have something to share? Want to help out with translating Lynis into your own language? Create an issue or pull request on GitHub, or send us an e-mail: lynis-dev@cisofy.com.

More details can be found in the Contributors Guide.

You can also simply contribute to the project by starring the project and show your appreciation that way.

Thanks!

License

GPLv3

mdn/curriculum

2023-09-10T02:05:14Z

MDN front-end developer curriculum

Welcome to the MDN front-end developer curriculum

The MDN front-end developer curriculum is intended to provide an up-to-date industry recommendation for the key fundamental skills and knowledge that a front-end web developer should have, including the mindset and attitude required for securing a job and for long-term success in this field. This repository has been created to give the web community an understanding of the curriculum's purpose and target audience, an early preview of its content, and a chance to give feedback on it. Please let us know what you think!

Rationale

The curriculum is an outcome of a research project conducted by the MDN team in early 2023. We asked students, new web developers, technical hiring managers, and web education professionals about their expectations around learning front-end web development — on MDN and elsewhere. The key outcomes of this research were:

Students expect more structured guidance on what topics to learn and when.
All groups cited a lack of "soft skills" (like teamwork, research, and critical thinking) and knowledge of best practices (such as accessibility and privacy) in potential new hires as key problems in the industry.

Part of our focus for 2023 is on making MDN a resource where people can learn new skills, as well as look up reference material. Back in 2019, we launched the Learning Area on MDN to help people learn the basics of front-end development. With a lot of useful content published to date, this part of MDN has proven to be successful — it averages around 10% of MDN page views. However, we feel that it needs to be supplemented with stronger guidance on what new front-end devs should learn, and in what order, to be successful in the web industry. The curriculum represents a first step towards this goal.

Target audience and purpose

We believe the curriculum is useful to several different groups of people, from students wanting to learn web development to educators wanting to put together courses to teach it. Head over to the curriculum introduction to read more about its target audience and purpose.

Curriculum structure

We have structured the curriculum as follows:

Precursor knowledge: Topics that are not, strictly speaking, web development topics but do constitute useful topics for anyone wanting to learn front-end web development. This includes soft skills and knowledge of a typical development environment.
Core modules: Topics that we feel every web developer should have a good grounding in. This includes all the information they need to design and build a basic, accessible website/app that follows modern best practices, and manage and deploy their code using a tool like GitHub.
Optional extension modules: These "extension" topics constitute useful additional skills to learn as web developers start to expand their knowledge and develop specialisms.

Get started by heading over to the main table of contents to see a granular overview of the included topics, and start exploring the content.

Providing feedback

We would love to hear your feedback regarding our curriculum. To do so, please open an issue — leave your feedback under General feedback or Topic coverage. When doing so, think about the following questions:

Does our curriculum contain all the fundamental knowledge a front-end web developer needs? If not, what topics are we missing? We are interested in high-level concerns (for example, "this whole area is missing") as well as lower-level feedback (for example, specific CSS or JavaScript topic omissions).
Do you think the curriculum is helpful to its key target audiences, for example, students wanting to learn front-end development and teachers wishing to create courses based on it? If not, why not?

angristan/openvpn-install

2023-09-10T02:05:14Z

Set up your own OpenVPN server on Debian, Ubuntu, Fedora, CentOS or Arch Linux.

openvpn-install

OpenVPN installer for Debian, Ubuntu, Fedora, CentOS, Arch Linux, Oracle Linux, Rocky Linux and AlmaLinux.

This script will let you setup your own secure VPN server in just a few seconds.

You can also check out wireguard-install, a simple installer for a simpler, safer, faster and more modern VPN protocol.

Usage

First, get the script and make it executable:

curl -O https://raw.githubusercontent.com/angristan/openvpn-install/master/openvpn-install.sh
chmod +x openvpn-install.sh

Then run it:

./openvpn-install.sh

You need to run the script as root and have the TUN module enabled.

The first time you run it, you'll have to follow the assistant and answer a few questions to setup your VPN server.

When OpenVPN is installed, you can run the script again, and you will get the choice to:

Add a client
Remove a client
Uninstall OpenVPN

In your home directory, you will have .ovpn files. These are the client configuration files. Download them from your server and connect using your favorite OpenVPN client.

If you have any question, head to the FAQ first. Please read everything before opening an issue.

PLEASE do not send me emails or private messages asking for help. The only place to get help is the issues. Other people may be able to help and in the future, other users may also run into the same issue as you. My time is not available for free just for you, you're not special.

Headless install

It's also possible to run the script headless, e.g. without waiting for user input, in an automated manner.

Example usage:

AUTO_INSTALL=y ./openvpn-install.sh

# or

export AUTO_INSTALL=y
./openvpn-install.sh

A default set of variables will then be set, by passing the need for user input.

If you want to customise your installation, you can export them or specify them on the same line, as shown above.

APPROVE_INSTALL=y
APPROVE_IP=y
IPV6_SUPPORT=n
PORT_CHOICE=1
PROTOCOL_CHOICE=1
DNS=1
COMPRESSION_ENABLED=n
CUSTOMIZE_ENC=n
CLIENT=clientname
PASS=1

If the server is behind NAT, you can specify its endpoint with the ENDPOINT variable. If the endpoint is the public IP address which it is behind, you can use ENDPOINT=$(curl -4 ifconfig.co) (the script will default to this). The endpoint can be an IPv4 or a domain.

Other variables can be set depending on your choice (encryption, compression). You can search for them in the installQuestions() function of the script.

Password-protected clients are not supported by the headless installation method since user input is expected by Easy-RSA.

The headless install is more-or-less idempotent, in that it has been made safe to run multiple times with the same parameters, e.g. by a state provisioner like Ansible/Terraform/Salt/Chef/Puppet. It will only install and regenerate the Easy-RSA PKI if it doesn't already exist, and it will only install OpenVPN and other upstream dependencies if OpenVPN isn't already installed. It will recreate all local config and re-generate the client file on each headless run.

Headless User Addition

It's also possible to automate the addition of a new user. Here, the key is to provide the (string) value of the MENU_OPTION variable along with the remaining mandatory variables before invoking the script.

The following Bash script adds a new user foo to an existing OpenVPN configuration

#!/bin/bash
export MENU_OPTION="1"
export CLIENT="foo"
export PASS="1"
./openvpn-install.sh

Features

Installs and configures a ready-to-use OpenVPN server
Iptables rules and forwarding managed in a seamless way
If needed, the script can cleanly remove OpenVPN, including configuration and iptables rules
Customisable encryption settings, enhanced default settings (see Security and Encryption below)
OpenVPN 2.4 features, mainly encryption improvements (see Security and Encryption below)
Variety of DNS resolvers to be pushed to the clients
Choice to use a self-hosted resolver with Unbound (supports already existing Unbound installations)
Choice between TCP and UDP
NATed IPv6 support
Compression disabled by default to prevent VORACLE. LZ4 (v1/v2) and LZ0 algorithms available otherwise.
Unprivileged mode: run as nobody/nogroup
Block DNS leaks on Windows 10
Randomised server certificate name
Choice to protect clients with a password (private key encryption)
Many other little things!

Compatibility

The script supports these Linux distributions:

	Support
AlmaLinux 8	✅
Amazon Linux 2	✅
Arch Linux	✅
CentOS 7	✅ 🤖
CentOS Stream >= 8	✅ 🤖
Debian >= 10	✅ 🤖
Fedora >= 35	✅ 🤖
Oracle Linux 8	✅
Rocky Linux 8	✅
Ubuntu >= 18.04	✅ 🤖

To be noted:

The script is regularly tested against the distributions marked with a 🤖 only.
- It's only test on amd64 architecture.
It should work on older versions such as Debian 8+, Ubuntu 16.04+ and previous Fedora releases. But versions not in the table above are not officially supported.
- It should also support versions between the LTS versions, but these are not tested.
The script requires systemd.

Fork

This script is based on the great work of Nyr and its contributors.

Since 2016, the two scripts have diverged and are not alike anymore, especially under the hood. The main goal of the script was enhanced security. But since then, the script has been completely rewritten and a lot a features have been added. The script is only compatible with recent distributions though, so if you need to use a very old server or client, I advise using Nyr's script.

FAQ

More Q&A in FAQ.md.

Q: Which provider do you recommend?

A: I recommend these:

Vultr: Worldwide locations, IPv6 support, starting at $5/month
Hetzner: Germany, Finland and USA. IPv6, 20 TB of traffic, starting at 4.5€/month
Digital Ocean: Worldwide locations, IPv6 support, starting at $4/month

Q: Which OpenVPN client do you recommend?

A: If possible, an official OpenVPN 2.4 client.

Windows: The official OpenVPN community client.
Linux: The openvpn package from your distribution. There is an official APT repository for Debian/Ubuntu based distributions.
macOS: Tunnelblick, Viscosity, OpenVPN for Mac.
Android: OpenVPN for Android.
iOS: The official OpenVPN Connect client.

Q: Am I safe from the NSA by using your script?

A: Please review your threat models. Even if this script has security in mind and uses state-of-the-art encryption, you shouldn't be using a VPN if you want to hide from the NSA.

Q: Is there an OpenVPN documentation?

A: Yes, please head to the OpenVPN Manual, which references all the options.

More Q&A in FAQ.md.

One-stop solutions for public cloud

Solutions that provision a ready to use OpenVPN server based on this script in one go are available for:

AWS using Terraform at openvpn-terraform-install
Terraform AWS module openvpn-ephemeral

Contributing

Discuss changes

Please open an issue before submitting a PR if you want to discuss a change, especially if it's a big one.

Code formatting

We use shellcheck and shfmt to enforce bash styling guidelines and good practices. They are executed for each commit / PR with GitHub Actions, so you can check the configuration here.

Security and Encryption

Warning This has not been updated for OpenVPN 2.5 and later.

OpenVPN's default settings are pretty weak regarding encryption. This script aims to improve that.

OpenVPN 2.4 was a great update regarding encryption. It added support for ECDSA, ECDH, AES GCM, NCP and tls-crypt.

If you want more information about an option mentioned below, head to the OpenVPN manual. It is very complete.

Most of OpenVPN's encryption-related stuff is managed by Easy-RSA. Defaults parameters are in the vars.example file.

Compression

By default, OpenVPN doesn't enable compression. This script provides support for LZ0 and LZ4 (v1/v2) algorithms, the latter being more efficient.

However, it is discouraged to use compression since the VORACLE attack makes use of it.

TLS version

OpenVPN accepts TLS 1.0 by default, which is nearly 20 years old.

With tls-version-min 1.2 we enforce TLS 1.2, which the best protocol available currently for OpenVPN.

TLS 1.2 is supported since OpenVPN 2.3.3.

Certificate

OpenVPN uses an RSA certificate with a 2048 bits key by default.

OpenVPN 2.4 added support for ECDSA. Elliptic curve cryptography is faster, lighter and more secure.

This script provides:

ECDSA: prime256v1/secp384r1/secp521r1 curves
RSA: 2048/3072/4096 bits keys

It defaults to ECDSA with prime256v1.

OpenVPN uses SHA-256 as the signature hash by default, and so does the script. It provides no other choice as of now.

Data channel

By default, OpenVPN uses BF-CBC as the data channel cipher. Blowfish is an old (1993) and weak algorithm. Even the official OpenVPN documentation admits it.

The default is BF-CBC, an abbreviation for Blowfish in Cipher Block Chaining mode.

Using BF-CBC is no longer recommended, because of its 64-bit block size. This small block size allows attacks based on collisions, as demonstrated by SWEET32. See https://community.openvpn.net/openvpn/wiki/SWEET32 for details. Security researchers at INRIA published an attack on 64-bit block ciphers, such as 3DES and Blowfish. They show that they are able to recover plaintext when the same data is sent often enough, and show how they can use cross-site scripting vulnerabilities to send data of interest often enough. This works over HTTPS, but also works for HTTP-over-OpenVPN. See https://sweet32.info/ for a much better and more elaborate explanation.

OpenVPN's default cipher, BF-CBC, is affected by this attack.

Indeed, AES is today's standard. It's the fastest and more secure cipher available today. SEED and Camellia are not vulnerable to date but are slower than AES and relatively less trusted.

Of the currently supported ciphers, OpenVPN currently recommends using AES-256-CBC or AES-128-CBC. OpenVPN 2.4 and newer will also support GCM. For 2.4+, we recommend using AES-256-GCM or AES-128-GCM.

AES-256 is 40% slower than AES-128, and there isn't any real reason to use a 256 bits key over a 128 bits key with AES. (Source: 1,2). Moreover, AES-256 is more vulnerable to Timing attacks.

AES-GCM is an AEAD cipher which means it simultaneously provides confidentiality, integrity, and authenticity assurances on the data.

The script supports the following ciphers:

AES-128-GCM
AES-192-GCM
AES-256-GCM
AES-128-CBC
AES-192-CBC
AES-256-CBC

And defaults to AES-128-GCM.

OpenVPN 2.4 added a feature called "NCP": Negotiable Crypto Parameters. It means you can provide a cipher suite like with HTTPS. It is set to AES-256-GCM:AES-128-GCM by default and overrides the --cipher parameter when used with an OpenVPN 2.4 client. For the sake of simplicity, the script set both the --cipher and --ncp-cipher to the cipher chosen above.

Control channel

OpenVPN 2.4 will negotiate the best cipher available by default (e.g ECDHE+AES-256-GCM)

The script proposes the following options, depending on the certificate:

ECDSA:
- TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
- TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384
RSA:
- TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256
- TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384

It defaults to TLS-ECDHE-*-WITH-AES-128-GCM-SHA256.

Diffie-Hellman key exchange

OpenVPN uses a 2048 bits DH key by default.

OpenVPN 2.4 added support for ECDH keys. Elliptic curve cryptography is faster, lighter and more secure.

Also, generating a classic DH keys can take a long, looong time. ECDH keys are ephemeral: they are generated on-the-fly.

The script provides the following options:

ECDH: prime256v1/secp384r1/secp521r1 curves
DH: 2048/3072/4096 bits keys

It defaults to prime256v1.

HMAC digest algorithm

From the OpenVPN wiki, about --auth:

Authenticate data channel packets and (if enabled) tls-auth control channel packets with HMAC using message digest algorithm alg. (The default is SHA1 ). HMAC is a commonly used message authentication algorithm (MAC) that uses a data string, a secure hash algorithm, and a key, to produce a digital signature.

If an AEAD cipher mode (e.g. GCM) is chosen, the specified --auth algorithm is ignored for the data channel, and the authentication method of the AEAD cipher is used instead. Note that alg still specifies the digest used for tls-auth.

The script provides the following choices:

SHA256
SHA384
SHA512

It defaults to SHA256.

`tls-auth` and `tls-crypt`

From the OpenVPN wiki, about tls-auth:

Add an additional layer of HMAC authentication on top of the TLS control channel to mitigate DoS attacks and attacks on the TLS stack.

In a nutshell, --tls-auth enables a kind of "HMAC firewall" on OpenVPN's TCP/UDP port, where TLS control channel packets bearing an incorrect HMAC signature can be dropped immediately without response.

About tls-crypt:

Encrypt and authenticate all control channel packets with the key from keyfile. (See --tls-auth for more background.)

Encrypting (and authenticating) control channel packets:

provides more privacy by hiding the certificate used for the TLS connection,

makes it harder to identify OpenVPN traffic as such,

provides "poor-man's" post-quantum security, against attackers who will never know the pre-shared key (i.e. no forward secrecy).

So both provide an additional layer of security and mitigate DoS attacks. They aren't used by default by OpenVPN.

tls-crypt is an OpenVPN 2.4 feature that provides encryption in addition to authentication (unlike tls-auth). It is more privacy-friendly.

The script supports both and uses tls-crypt by default.

Say thanks

You can say thanks if you want!

Credits & Licence

Many thanks to the contributors and Nyr's original work.

This project is under the MIT Licence

Star History

sourcifyeth/infra

2023-09-10T01:44:07Z

npci/falcon

2023-09-10T02:05:43Z

Falcon: The Hyperledger Fabric Deployment Helper for Kubernetes is an open-source project designed to streamline the deployment and management of Hyperledger Fabric based blockchain networks on K8.

apache/skywalking-helm

2023-09-10T02:05:43Z

Apache SkyWalking Kubernetes Deployment Helm Chart

rh-telco-tigers/rhacm-gitops-workshop

2023-09-10T02:05:43Z

code-423n4/2023-09-centrifuge

2023-09-11T01:42:08Z

Cyfrin/2023-09-ditto

2023-09-11T01:42:08Z

solidquant/mev-templates

2023-09-10T02:05:57Z

MEV bot templates written in Python/Javascript/Rust. All implementing DEX flashloan arbitrage

tintinweb/smart-contract-sanctuary-celo

2023-09-10T02:05:57Z

🐦🌴🌴🌴🦕 A home for ethereum smart contracts. 🏠

noir-lang/noir-starter

2023-09-10T02:05:57Z

berrysauce/ingredients

2023-09-10T01:44:47Z

A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.

tokitou-san/CoreProject-Frontend

2023-09-10T01:44:47Z

Bridging the gap between streaming and torrenting sites with a modern and clean interface. Anime streaming with an ad-free, premium experience, setting a new standard

RiadhAdrani/slick-portfolio-svelte

2023-09-10T01:44:47Z

Vercel-like style portfolio template for developers.

rossrobino/drab

2023-09-10T02:06:23Z

Unstyled Svelte component library

CaptainCodeman/svelte-headlessui

2023-09-10T02:06:23Z

HeadlessUI components for Svelte

yarspirin/TagCloud

2023-09-11T01:42:42Z

TagCloud 🏷️: A flexible SwiftUI package for creating customizable tag clouds in iOS apps.

apple/swift-openapi-generator

2023-09-10T02:06:31Z

Generate Swift client and server code from an OpenAPI document.

Swift OpenAPI Generator

Generate Swift client and server code from an OpenAPI document.

Overview

OpenAPI is an open specification for documenting HTTP APIs.

Swift OpenAPI Generator is a Swift package plugin that can generate the ceremony code required to make API calls, or implement API servers.

Repository organization

The Swift OpenAPI Generator project is split across multiple repositories to enable extensibility and minimize dependencies in your project.

swift-openapi-generator (source, docs) provides the plugin.

swift-openapi-runtime (source, docs) provides a library with common types and abstractions used by the generated code.

See the generator in action in Meet Swift OpenAPI Generator from WWDC23.

Choose one of the transports listed below, or create your own by adopting the ClientTransport or ServerTransport protocol:

Repository	Type	Description
apple/swift-openapi-urlsession	Client	Uses `URLSession` from Foundation.
swift-server/swift-openapi-async-http-client	Client	Uses `HTTPClient` from AsyncHTTPClient.
swift-server/swift-openapi-vapor	Server	Uses Vapor.
swift-server/swift-openapi-hummingbird	Server	Uses Hummingbird.

Requirements and supported features

Generator versions	Supported OpenAPI versions	Minimum Swift version
`0.1.0` ... `0.1.11`	3.0	5.8
`0.1.12` ... `main`	3.0, 3.1	5.8

Supported platforms and minimum versions

The generator is used during development and is supported on macOS and Linux.

The generated code, runtime library, and transports are supported on more platforms, listed below.

Component	macOS	Linux	iOS	tvOS	watchOS
Generator plugin and CLI	✅ 10.15+	✅	❌	❌	❌
Generated code, runtime, transports	✅ 10.15+	✅	✅ 13+	✅ 13+	✅ 6+

Documentation

To get started, check out the full documentation, which contains step-by-step tutorials!

JohnEstropia/CoreStore

2023-09-10T02:06:31Z

Unleashing the real power of Core Data with the elegance and safety of Swift

Unleashing the real power of Core Data with the elegance and safety of Swift

Dependency managers

Contact

Swift 5.7: iOS 13+ / macOS 10.15+ / watchOS 7.4+ / tvOS 13.0+
Previously supported Swift versions: Swift 5.5, Swift 5.4, Swift 5.3

Upgrading from previous CoreStore versions? Check out the 🆕 features and make sure to read the Change logs.

CoreStore is part of the Swift Source Compatibility projects.

TL;DR (a.k.a. sample codes)
Why use CoreStore?
Architecture
CoreStore Tutorials (All of these have demos in the Demo app project!)
Roadmap
Installation
Changesets
Contact
Who uses CoreStore?
License

TL;DR (a.k.a. sample codes)

Pure-Swift models:

class Person: CoreStoreObject {
    @Field.Stored("name")
    var name: String = ""
    
    @Field.Relationship("pets", inverse: \Dog.$master)
    var pets: Set<Dog>
}

(Classic NSManagedObjects also supported)

Setting-up with progressive migration support:

dataStack = DataStack(
    xcodeModelName: "MyStore",
    migrationChain: ["MyStore", "MyStoreV2", "MyStoreV3"]
)

Adding a store:

dataStack.addStorage(
    SQLiteStore(fileName: "MyStore.sqlite"),
    completion: { (result) -> Void in
        // ...
    }
)

Starting transactions:

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let person = transaction.create(Into<Person>())
        person.name = "John Smith"
        person.age = 42
    },
    completion: { (result) -> Void in
        switch result {
        case .success: print("success!")
        case .failure(let error): print(error)
        }
    }
)

Fetching objects (simple):

let people = try dataStack.fetchAll(From<Person>())

Fetching objects (complex):

let people = try dataStack.fetchAll(
    From<Person>()
        .where(\.age > 30),
        .orderBy(.ascending(\.name), .descending(.\age)),
        .tweak({ $0.includesPendingChanges = false })
)

Querying values:

let maxAge = try dataStack.queryValue(
    From<Person>()
        .select(Int.self, .maximum(\.age))
)

But really, there's a reason I wrote this huge README. Read up on the details!

Check out the Demo app project for sample codes as well!

Why use CoreStore?

CoreStore was (and is) heavily shaped by real-world needs of developing data-dependent apps. It enforces safe and convenient Core Data usage while letting you take advantage of the industry's encouraged best practices.

Features

🆕SwiftUI and Combine API utilities. ListPublishers and ObjectPublishers now have their @ListState and @ObjectState SwiftUI property wrappers. Combine Publisher s are also available through the ListPublisher.reactive, ObjectPublisher.reactive, and DataStack.reactive namespaces.
Backwards-portable DiffableDataSources implementation! UITableViews and UICollectionViews now have a new ally: ListPublishers provide diffable snapshots that make reloading animations very easy and very safe. Say goodbye to UITableViews and UICollectionViews reload errors!
💎Tight design around Swift’s code elegance and type safety. CoreStore fully utilizes Swift's community-driven language features.
🚦Safer concurrency architecture. CoreStore makes it hard to fall into common concurrency mistakes. The main NSManagedObjectContext is strictly read-only, while all updates are done through serial transactions. (See Saving and processing transactions)
🔍Clean fetching and querying API. Fetching objects is easy, but querying for raw aggregates (min, max, etc.) and raw property values is now just as convenient. (See Fetching and querying)
🔭Type-safe, easy to configure observers. You don't have to deal with the burden of setting up NSFetchedResultsControllers and KVO. As an added bonus, list and object observable types all support multiple observers. This means you can have multiple view controllers efficiently share a single resource! (See Observing changes and notifications)
📥Efficient importing utilities. Map your entities once with their corresponding import source (JSON for example), and importing from transactions becomes elegant. Uniquing is also done with an efficient find-and-replace algorithm. (See Importing data)
🗑Say goodbye to .xcdatamodeld files! While CoreStore supports NSManagedObjects, it offers CoreStoreObject whose subclasses can declare type-safe properties all in Swift code without the need to maintain separate resource files for the models. As bonus, these special properties support custom types, and can be used to create type-safe keypaths and queries. (See Type-safe CoreStoreObjects)
🔗Progressive migrations. No need to think how to migrate from all previous model versions to your latest model. Just tell the DataStack the sequence of version strings (MigrationChains) and CoreStore will automatically use progressive migrations when needed. (See Migrations)
Easier custom migrations. Say goodbye to .xcmappingmodel files; CoreStore can now infer entity mappings when possible, while still allowing an easy way to write custom mappings. (See Migrations)
📝Plug-in your own logging framework. Although a default logger is built-in, all logging, asserting, and error reporting can be funneled to CoreStoreLogger protocol implementations. (See Logging and error reporting)
⛓Heavy support for multiple persistent stores per data stack. CoreStore lets you manage separate stores in a single DataStack, just the way .xcdatamodeld configurations are designed to. CoreStore will also manage one stack by default, but you can create and manage as many as you need. (See Setting up)
🎯Free to name entities and their class names independently. CoreStore gets around a restriction with other Core Data wrappers where the entity name should be the same as the NSManagedObject subclass name. CoreStore loads entity-to-class mappings from the managed object model file, so you can assign independent names for the entities and their class names.
📙Full Documentation. No magic here; all public classes, functions, properties, etc. have detailed Apple Docs. This README also introduces a lot of concepts and explains a lot of CoreStore's behavior.
ℹ️Informative (and pretty) logs. All CoreStore and Core Data-related types now have very informative and pretty print outputs! (See Logging and error reporting)
🛡More extensive Unit Tests. Extending CoreStore is safe without having to worry about breaking old behavior.

Have ideas that may benefit other Core Data users? Feature Requests are welcome!

Architecture

For maximum safety and performance, CoreStore will enforce coding patterns and practices it was designed for. (Don't worry, it's not as scary as it sounds.) But it is advisable to understand the "magic" of CoreStore before you use it in your apps.

If you are already familiar with the inner workings of CoreData, here is a mapping of CoreStore abstractions:

Core Data	CoreStore
`NSPersistentContainer` (.xcdatamodeld file)	`DataStack`
`NSPersistentStoreDescription` ("Configuration"s in the .xcdatamodeld file)	`StorageInterface` implementations (`InMemoryStore`, `SQLiteStore`)
`NSManagedObjectContext`	`BaseDataTransaction` subclasses (`SynchronousDataTransaction`, `AsynchronousDataTransaction`, `UnsafeDataTransaction`)

A lot of Core Data wrapper libraries set up their NSManagedObjectContexts this way:

Nesting saves from child context to the root context ensures maximum data integrity between contexts without blocking the main queue. But in reality, merging contexts is still by far faster than saving contexts. CoreStore's DataStack takes the best of both worlds by treating the main NSManagedObjectContext as a read-only context (or "viewContext"), and only allows changes to be made within transactions on the child context:

This allows for a butter-smooth main thread, while still taking advantage of safe nested contexts.

Setting up

The simplest way to initialize CoreStore is to add a default store to the default stack:

try CoreStoreDefaults.dataStack.addStorageAndWait()

This one-liner does the following:

Triggers the lazy-initialization of CoreStoreDefaults.dataStack with a default DataStack
Sets up the stack's NSPersistentStoreCoordinator, the root saving NSManagedObjectContext, and the read-only main NSManagedObjectContext
Adds an SQLiteStore in the "Application Support/ " directory (or the "Caches/ " directory on tvOS) with the file name "[App bundle name].sqlite"
Creates and returns the NSPersistentStore instance on success, or an NSError on failure

For most cases, this configuration is enough as it is. But for more hardcore settings, refer to this extensive example:

let dataStack = DataStack(
    xcodeModelName: "MyModel", // loads from the "MyModel.xcdatamodeld" file
    migrationChain: ["MyStore", "MyStoreV2", "MyStoreV3"] // model versions for progressive migrations
)
let migrationProgress = dataStack.addStorage(
    SQLiteStore(
        fileURL: sqliteFileURL, // set the target file URL for the sqlite file
        configuration: "Config2", // use entities from the "Config2" configuration in the .xcdatamodeld file
        localStorageOptions: .recreateStoreOnModelMismatch // if migration paths cannot be resolved, recreate the sqlite file
    ),
    completion: { (result) -> Void in
        switch result {
        case .success(let storage):
            print("Successfully added sqlite store: \(storage)")
        case .failure(let error):
            print("Failed adding sqlite store with error: \(error)")
        }
    }
)

CoreStoreDefaults.dataStack = dataStack // pass the dataStack to CoreStore for easier access later on

💡If you have never heard of "Configurations", you'll find them in your .xcdatamodeld file

In our sample code above, note that you don't need to do the CoreStoreDefaults.dataStack = dataStack line. You can just as well hold a reference to the DataStack like below and call all its instance methods directly:

class MyViewController: UIViewController {
    let dataStack = DataStack(xcodeModelName: "MyModel") // keep reference to the stack
    override func viewDidLoad() {
        super.viewDidLoad()
        do {
            try self.dataStack.addStorageAndWait(SQLiteStore.self)
        }
        catch { // ...
        }
    }
    func methodToBeCalledLaterOn() {
        let objects = self.dataStack.fetchAll(From<MyEntity>())
        print(objects)
    }
}

💡By default, CoreStore will initialize NSManagedObjects from .xcdatamodeld files, but you can create models completely from source code using CoreStoreObjects and CoreStoreSchema. To use this feature, refer to Type-safe CoreStoreObjects.

Notice that in our previous examples, addStorageAndWait(_:) and addStorage(_:completion:) both accept either InMemoryStore, or SQLiteStore. These implement the StorageInterface protocol.

In-memory store

The most basic StorageInterface concrete type is the InMemoryStore, which just stores objects in memory. Since InMemoryStores always start with a fresh empty data, they do not need any migration information.

try dataStack.addStorageAndWait(
    InMemoryStore(
        configuration: "Config2" // optional. Use entities from the "Config2" configuration in the .xcdatamodeld file
    )
)

Asynchronous variant:

try dataStack.addStorage(
    InMemoryStore(
        configuration: "Config2
    ),
    completion: { storage in
        // ...
    }
)

(A reactive-programming variant of this method is explained in detail in the section on DataStack Combine publishers)

Local Store

The most common StorageInterface you will probably use is the SQLiteStore, which saves data in a local SQLite file.

let migrationProgress = dataStack.addStorage(
    SQLiteStore(
        fileName: "MyStore.sqlite",
        configuration: "Config2", // optional. Use entities from the "Config2" configuration in the .xcdatamodeld file
        migrationMappingProviders: [Bundle.main], // optional. The bundles that contain required .xcmappingmodel files
        localStorageOptions: .recreateStoreOnModelMismatch // optional. Provides settings that tells the DataStack how to setup the persistent store
    ),
    completion: { /* ... */ }
)

Refer to the SQLiteStore.swift source documentation for detailed explanations for each of the default values.

CoreStore can decide the default values for these properties, so SQLiteStores can be initialized with no arguments:

try dataStack.addStorageAndWait(SQLiteStore())

(The asynchronous variant of this method is explained further in the next section on Migrations, and a reactive-programming variant in the section on DataStack Combine publishers)

The file-related properties of SQLiteStore are actually requirements of another protocol that it implements, the LocalStorage protocol:

public protocol LocalStorage: StorageInterface {
    var fileURL: NSURL { get }
    var migrationMappingProviders: [SchemaMappingProvider] { get }
    var localStorageOptions: LocalStorageOptions { get }
    func dictionary(forOptions: LocalStorageOptions) -> [String: AnyObject]?
    func cs_eraseStorageAndWait(metadata: [String: Any], soureModelHint: NSManagedObjectModel?) throws
}

If you have custom NSIncrementalStore or NSAtomicStore subclasses, you can implement this protocol and use it similarly to SQLiteStore.

Migrations

Declaring model versions

Model versions are now expressed as a first-class protocol, DynamicSchema. CoreStore currently supports the following schema classes:

XcodeDataModelSchema: a model version with entities loaded from a .xcdatamodeld file.
CoreStoreSchema: a model version created with CoreStoreObject entities. (See Type-safe CoreStoreObjects)
UnsafeDataModelSchema: a model version created with an existing NSManagedObjectModel instance.

All the DynamicSchema for all model versions are then collected within a single SchemaHistory instance, which is then handed to the DataStack. Here are some common use cases:

Multiple model versions grouped in a .xcdatamodeld file (Core Data standard method)

CoreStoreDefaults.dataStack = DataStack(
    xcodeModelName: "MyModel",
    bundle: Bundle.main,
    migrationChain: ["MyAppModel", "MyAppModelV2", "MyAppModelV3", "MyAppModelV4"]
)

CoreStoreSchema-based model version (No .xcdatamodeld file needed) (For more details, see also Type-safe CoreStoreObjects)

class Animal: CoreStoreObject {
    // ...
}
class Dog: Animal {
    // ...
}
class Person: CoreStoreObject {
    // ...
}

CoreStoreDefaults.dataStack = DataStack(
    CoreStoreSchema(
        modelVersion: "V1",
        entities: [
            Entity<Animal>("Animal", isAbstract: true),
            Entity<Dog>("Dog"),
            Entity<Person>("Person")
        ]
    )
)

Models in a .xcdatamodeld file during past app versions, but migrated to the new CoreStoreSchema method

class Animal: CoreStoreObject {
    // ...
}
class Dog: Animal {
    // ...
}
class Person: CoreStoreObject {
    // ...
}

let legacySchema = XcodeDataModelSchema.from(
    modelName: "MyModel", // .xcdatamodeld name
    bundle: bundle,
    migrationChain: ["MyAppModel", "MyAppModelV2", "MyAppModelV3", "MyAppModelV4"]
)
let newSchema = CoreStoreSchema(
    modelVersion: "V1",
    entities: [
        Entity<Animal>("Animal", isAbstract: true),
        Entity<Dog>("Dog"),
        Entity<Person>("Person")
    ]
)
CoreStoreDefaults.dataStack = DataStack(
    schemaHistory: SchemaHistory(
        legacySchema + [newSchema],
        migrationChain: ["MyAppModel", "MyAppModelV2", "MyAppModelV3", "MyAppModelV4", "V1"] 
    )
)

CoreStoreSchema-based model versions with progressive migration

typealias Animal = V2.Animal
typealias Dog = V2.Dog
typealias Person = V2.Person
enum V2 {
    class Animal: CoreStoreObject {
        // ...
    }
    class Dog: Animal {
        // ...
    }
    class Person: CoreStoreObject {
        // ...
    }
}
enum V1 {
    class Animal: CoreStoreObject {
        // ...
    }
    class Dog: Animal {
        // ...
    }
    class Person: CoreStoreObject {
        // ...
    }
}

CoreStoreDefaults.dataStack = DataStack(
    CoreStoreSchema(
        modelVersion: "V1",
        entities: [
            Entity<V1.Animal>("Animal", isAbstract: true),
            Entity<V1.Dog>("Dog"),
            Entity<V1.Person>("Person")
        ]
    ),
    CoreStoreSchema(
        modelVersion: "V2",
        entities: [
            Entity<V2.Animal>("Animal", isAbstract: true),
            Entity<V2.Dog>("Dog"),
            Entity<V2.Person>("Person")
        ]
    ),
    migrationChain: ["V1", "V2"]
)

Starting migrations

We have seen addStorageAndWait(...) used to initialize our persistent store. As the method name's ~AndWait suffix suggests though, this method blocks so it should not do long tasks such as data migrations. In fact CoreStore will only attempt a synchronous lightweight migration if you explicitly provide the .allowSynchronousLightweightMigration option:

try dataStack.addStorageAndWait(
    SQLiteStore(
        fileURL: sqliteFileURL,
        localStorageOptions: .allowSynchronousLightweightMigration
    )
}

if you do so, any model mismatch will be thrown as an error.

In general though, if migrations are expected the asynchronous variant addStorage(_:completion:) method is recommended instead:

let migrationProgress: Progress? = try dataStack.addStorage(
    SQLiteStore(
        fileName: "MyStore.sqlite",
        configuration: "Config2"
    ),
    completion: { (result) -> Void in
        switch result {
        case .success(let storage):
            print("Successfully added sqlite store: \(storage)")
        case .failure(let error):
            print("Failed adding sqlite store with error: \(error)")
        }
    }
)

The completion block reports a SetupResult that indicates success or failure.

(A reactive-programming variant of this method is explained further in the section on DataStack Combine publishers)

Notice that this method also returns an optional Progress. If nil, no migrations are needed, thus progress reporting is unnecessary as well. If not nil, you can use this to track migration progress by using standard KVO on the "fractionCompleted" key, or by using a closure-based utility exposed in Progress+Convenience.swift:

migrationProgress?.setProgressHandler { [weak self] (progress) -> Void in
    self?.progressView?.setProgress(Float(progress.fractionCompleted), animated: true)
    self?.percentLabel?.text = progress.localizedDescription // "50% completed"
    self?.stepLabel?.text = progress.localizedAdditionalDescription // "0 of 2"
}

This closure is executed on the main thread so UIKit and AppKit calls can be done safely.

Progressive migrations

By default, CoreStore uses Core Data's default automatic migration mechanism. In other words, CoreStore will try to migrate the existing persistent store until it matches the SchemaHistory's currentModelVersion. If no mapping model path is found from the store's version to the data model's version, CoreStore gives up and reports an error.

The DataStack lets you specify hints on how to break a migration into several sub-migrations using a MigrationChain. This is typically passed to the DataStack initializer and will be applied to all stores added to the DataStack with addSQLiteStore(...) and its variants:

let dataStack = DataStack(migrationChain: 
    ["MyAppModel", "MyAppModelV2", "MyAppModelV3", "MyAppModelV4"])

The most common usage is to pass in the model version (.xcdatamodeld version names for NSManagedObjects, or the modelName for CoreStoreSchemas) in increasing order as above.

For more complex, non-linear migration paths, you can also pass in a version tree that maps the key-values to the source-destination versions:

let dataStack = DataStack(migrationChain: [
    "MyAppModel": "MyAppModelV3",
    "MyAppModelV2": "MyAppModelV4",
    "MyAppModelV3": "MyAppModelV4"
])

This allows for different migration paths depending on the starting version. The example above resolves to the following paths:

MyAppModel-MyAppModelV3-MyAppModelV4
MyAppModelV2-MyAppModelV4
MyAppModelV3-MyAppModelV4

Initializing with empty values (either nil, [], or [:]) instructs the DataStack to disable progressive migrations and revert to the default migration behavior (i.e. use the .xcdatamodeld's current version as the final version):

let dataStack = DataStack(migrationChain: nil)

The MigrationChain is validated when passed to the DataStack and unless it is empty, will raise an assertion if any of the following conditions are met:

a version appears twice in an array
a version appears twice as a key in a dictionary literal
a loop is found in any of the paths

⚠️Important: If a MigrationChain is specified, the .xcdatamodeld's "Current Version" will be bypassed and the MigrationChain's leafmost version will be the DataStack's base model version.

Forecasting migrations

Sometimes migrations are huge and you may want prior information so your app could display a loading screen, or to display a confirmation dialog to the user. For this, CoreStore provides a requiredMigrationsForStorage(_:) method you can use to inspect a persistent store before you actually call addStorageAndWait(_:) or addStorage(_:completion:):

do {
    let storage = SQLiteStorage(fileName: "MyStore.sqlite")
    let migrationTypes: [MigrationType] = try dataStack.requiredMigrationsForStorage(storage)
    if migrationTypes.count > 1
        || (migrationTypes.filter { $0.isHeavyweightMigration }.count) > 0 {
        // ... will migrate more than once. Show special waiting screen
    }
    else if migrationTypes.count > 0 {
        // ... will migrate just once. Show simple activity indicator
    }
    else {
        // ... Do nothing
    }
    dataStack.addStorage(storage, completion: { /* ... */ })
}
catch {
    // ... either inspection of the store failed, or if no mapping model was found/inferred
}

requiredMigrationsForStorage(_:) returns an array of MigrationTypes, where each item in the array may be either of the following values:

case lightweight(sourceVersion: String, destinationVersion: String)
case heavyweight(sourceVersion: String, destinationVersion: String)

Each MigrationType indicates the migration type for each step in the MigrationChain. Use these information as fit for your app.

Custom migrations

CoreStore offers several ways to declare migration mappings:

CustomSchemaMappingProvider: A mapping provider that infers mapping initially, but also accepts custom mappings for specified entities. This was added to support custom migrations with CoreStoreObjects as well, but may also be used with NSManagedObjects.
XcodeSchemaMappingProvider: A mapping provider which loads entity mappings from .xcmappingmodel files in a specified Bundle.
InferredSchemaMappingProvider: The default mapping provider which tries to infer model migration between two DynamicSchema versions either by searching all .xcmappingmodel files from Bundle.allBundles, or by relying on lightweight migration if possible.

These mapping providers conform to SchemaMappingProvider and can be passed to SQLiteStore's initializer:

let dataStack = DataStack(migrationChain: ["MyAppModel", "MyAppModelV2", "MyAppModelV3", "MyAppModelV4"])
_ = try dataStack.addStorage(
    SQLiteStore(
        fileName: "MyStore.sqlite",
        migrationMappingProviders: [
            XcodeSchemaMappingProvider(from: "V1", to: "V2", mappingModelBundle: Bundle.main),
            CustomSchemaMappingProvider(from: "V2", to: "V3", entityMappings: [.deleteEntity("Person") ])
        ]
    ),
    completion: { (result) -> Void in
        // ...
    }
)

For version migrations present in the DataStack's MigrationChain but not handled by any of the SQLiteStore's migrationMappingProviders array, CoreStore will automatically try to use InferredSchemaMappingProvider as fallback. Finally if the InferredSchemaMappingProvider could not resolve any mapping, the migration will fail and the DataStack.addStorage(...) method will report the failure.

For CustomSchemaMappingProvider, more granular updates are supported through the dynamic objects UnsafeSourceObject and UnsafeDestinationObject. The example below allows the migration to conditionally ignore some objects:

let person_v2_to_v3_mapping = CustomSchemaMappingProvider(
    from: "V2",
    to: "V3",
    entityMappings: [
        .transformEntity(
            sourceEntity: "Person",
            destinationEntity: "Person",
            transformer: { (sourceObject: UnsafeSourceObject, createDestinationObject: () -> UnsafeDestinationObject) in
                
                if (sourceObject["isVeryOldAccount"] as! Bool?) == true {
                    return // this account is too old, don't migrate 
                }
                // migrate the rest
                let destinationObject = createDestinationObject()
                destinationObject.enumerateAttributes { (attribute, sourceAttribute) in
                
                if let sourceAttribute = sourceAttribute {
                    destinationObject[attribute] = sourceObject[sourceAttribute]
                }
            }
        ) 
    ]
)
SQLiteStore(
    fileName: "MyStore.sqlite",
    migrationMappingProviders: [person_v2_to_v3_mapping]
)

The UnsafeSourceObject is a read-only proxy for an object existing in the source model version. The UnsafeDestinationObject is a read-write object that is inserted (optionally) to the destination model version. Both classes' properties are accessed through key-value-coding.

Saving and processing transactions

To ensure deterministic state for objects in the read-only NSManagedObjectContext, CoreStore does not expose API's for updating and saving directly from the main context (or any other context for that matter.) Instead, you spawn transactions from DataStack instances:

let dataStack = self.dataStack
dataStack.perform(
    asynchronous: { (transaction) -> Void in
        // make changes
    },
    completion: { (result) -> Void in
        // ...
    }
)

Transaction closures automatically save changes once the closures completes. To cancel and rollback a transaction, throw a CoreStoreError.userCancelled from inside the closure by calling try transaction.cancel():

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        // ...
        if shouldCancel {
            try transaction.cancel()
        }
        // ...
    },
    completion: { (result) -> Void in
        if case .failure(.userCancelled) = result {
            // ... cancelled
        }
    }
)

⚠️Important: Never use try? or try! on a transaction.cancel() call. Always use try. Using try? will swallow the cancellation and the transaction will proceed to save as normal. Using try! will crash the app as transaction.cancel() will always throw an error.

The examples above use perform(asynchronous:...), but there are actually 3 types of transactions at your disposal: asynchronous, synchronous, and unsafe.

Transaction types

Asynchronous transactions

are spawned from perform(asynchronous:...). This method returns immediately and executes its closure from a background serial queue. The return value for the closure is declared as a generic type, so any value returned from the closure can be passed to the completion result:

dataStack.perform(
    asynchronous: { (transaction) -> Bool in
        // make changes
        return transaction.hasChanges
    },
    completion: { (result) -> Void in
        switch result {
        case .success(let hasChanges): print("success! Has changes? \(hasChanges)")
        case .failure(let error): print(error)
        }
    }
)

The success and failure can also be declared as separate handlers:

dataStack.perform(
    asynchronous: { (transaction) -> Int in
        // make changes
        return transaction.delete(objects)
    },
    success: { (numberOfDeletedObjects: Int) -> Void in
        print("success! Deleted \(numberOfDeletedObjects) objects")
    },
    failure: { (error) -> Void in
        print(error)
    }
)

⚠️Be careful when returning NSManagedObjects or CoreStoreObjects from the transaction closure. Those instances are for the transaction's use only. See Passing objects safely.

Transactions created from perform(asynchronous:...) are instances of AsynchronousDataTransaction.

Synchronous transactions

are created from perform(synchronous:...). While the syntax is similar to its asynchronous counterpart, perform(synchronous:...) waits for its transaction block to complete before returning:

let hasChanges = dataStack.perform(
    synchronous: { (transaction) -> Bool in
        // make changes
        return transaction.hasChanges
    }
)

transaction above is a SynchronousDataTransaction instance.

Since perform(synchronous:...) technically blocks two queues (the caller's queue and the transaction's background queue), it is considered less safe as it's more prone to deadlock. Take special care that the closure does not block on any other external queues.

By default, perform(synchronous:...) will wait for observers such as ListMonitors to be notified before the method returns. This may cause deadlocks, especially if you are calling this from the main thread. To reduce this risk, you may try to set the waitForAllObservers: parameter to false. Doing so tells the SynchronousDataTransaction to block only until it completes saving. It will not wait for other context's to receive those changes. This reduces deadlock risk but may have surprising side-effects:

dataStack.perform(
    synchronous: { (transaction) in
        let person = transaction.create(Into<Person>())
        person.name = "John"
    },
    waitForAllObservers: false
)
let newPerson = dataStack.fetchOne(From<Person>.where(\.name == "John"))
// newPerson may be nil!
// The DataStack may have not yet received the update notification.

Due to this complicated nature of synchronous transactions, if your app has very heavy transaction throughput it is highly recommended to use asynchronous transactions instead.

Unsafe transactions

are special in that they do not enclose updates within a closure:

let transaction = dataStack.beginUnsafe()
// make changes
downloadJSONWithCompletion({ (json) -> Void in

    // make other changes
    transaction.commit()
})
downloadAnotherJSONWithCompletion({ (json) -> Void in

    // make some other changes
    transaction.commit()
})

This allows for non-contiguous updates. Do note that this flexibility comes with a price: you are now responsible for managing concurrency for the transaction. As uncle Ben said, "with great power comes great race conditions."

As the above example also shows, with unsafe transactions commit() can be called multiple times.

You've seen how to create transactions, but we have yet to see how to make creates, updates, and deletes. The 3 types of transactions above are all subclasses of BaseDataTransaction, which implements the methods shown below.

Creating objects

The create(...) method accepts an Into clause which specifies the entity for the object you want to create:

let person = transaction.create(Into<MyPersonEntity>())

While the syntax is straightforward, CoreStore does not just naively insert a new object. This single line does the following:

Checks that the entity type exists in any of the transaction's parent persistent store
If the entity belongs to only one persistent store, a new object is inserted into that store and returned from create(...)
If the entity does not belong to any store, an assertion failure will be raised. This is a programmer error and should never occur in production code.
If the entity belongs to multiple stores, an assertion failure will be raised. This is also a programmer error and should never occur in production code. Normally, with Core Data you can insert an object in this state but saving the NSManagedObjectContext will always fail. CoreStore checks this for you at creation time when it makes sense (not during save).

If the entity exists in multiple configurations, you need to provide the configuration name for the destination persistent store:

let person = transaction.create(Into<MyPersonEntity>("Config1"))

or if the persistent store is the auto-generated "Default" configuration, specify nil:

let person = transaction.create(Into<MyPersonEntity>(nil))

Note that if you do explicitly specify the configuration name, CoreStore will only try to insert the created object to that particular store and will fail if that store is not found; it will not fall back to any other configuration that the entity belongs to.

Updating objects

After creating an object from the transaction, you can simply update its properties as normal:

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let person = transaction.create(Into<MyPersonEntity>())
        person.name = "John Smith"
        person.age = 30
    },
    completion: { _ in }
)

To update an existing object, fetch the object's instance from the transaction:

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let person = try transaction.fetchOne(
            From<MyPersonEntity>()
                .where(\.name == "Jane Smith")
        )
        person.age = person.age + 1
    },
    completion: { _ in }
)

(For more about fetching, see Fetching and querying)

Do not update an instance that was not created/fetched from the transaction. If you have a reference to the object already, use the transaction's edit(...) method to get an editable proxy instance for that object:

let jane: MyPersonEntity = // ...

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        // WRONG: jane.age = jane.age + 1
        // RIGHT:
        let jane = transaction.edit(jane)! // using the same variable name protects us from misusing the non-transaction instance
        jane.age = jane.age + 1
    },
    completion: { _ in }
)

This is also true when updating an object's relationships. Make sure that the object assigned to the relationship is also created/fetched from the transaction:

let jane: MyPersonEntity = // ...
let john: MyPersonEntity = // ...

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        // WRONG: jane.friends = [john]
        // RIGHT:
        let jane = transaction.edit(jane)!
        let john = transaction.edit(john)!
        jane.friends = NSSet(array: [john])
    },
    completion: { _ in }
)

Deleting objects

Deleting an object is simpler because you can tell a transaction to delete an object directly without fetching an editable proxy (CoreStore does that for you):

let john: MyPersonEntity = // ...

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        transaction.delete(john)
    },
    completion: { _ in }
)

or several objects at once:

let john: MyPersonEntity = // ...
let jane: MyPersonEntity = // ...

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        try transaction.delete(john, jane)
        // try transaction.delete([john, jane]) is also allowed
    },
    completion: { _ in }
)

If you do not have references yet to the objects to be deleted, transactions have a deleteAll(...) method you can pass a query to:

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        try transaction.deleteAll(
            From<MyPersonEntity>()
                .where(\.age > 30)
        )
    },
    completion: { _ in }
)

Passing objects safely

Always remember that the DataStack and individual transactions manage different NSManagedObjectContexts so you cannot just use objects between them. That's why transactions have an edit(...) method:

let jane: MyPersonEntity = // ...

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let jane = transaction.edit(jane)!
        jane.age = jane.age + 1
    },
    completion: { _ in }
)

But CoreStore, DataStack and BaseDataTransaction have a very flexible fetchExisting(...) method that you can pass instances back and forth with:

let jane: MyPersonEntity = // ...

dataStack.perform(
    asynchronous: { (transaction) -> MyPersonEntity in
        let jane = transaction.fetchExisting(jane)! // instance for transaction
        jane.age = jane.age + 1
        return jane
    },
    success: { (transactionJane) in
        let jane = dataStack.fetchExisting(transactionJane)! // instance for DataStack
        print(jane.age)
    },
    failure: { (error) in
        // ...
    }
)

fetchExisting(...) also works with multiple NSManagedObjects, CoreStoreObjects, or with NSManagedObjectIDs:

var peopleIDs: [NSManagedObjectID] = // ...

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let jane = try transaction.fetchOne(
            From<MyPersonEntity>()
                .where(\.name == "Jane Smith")
        )
        jane.friends = NSSet(array: transaction.fetchExisting(peopleIDs)!)
        // ...
    },
    completion: { _ in }
)

Importing data

Some times, if not most of the time, the data that we save to Core Data comes from external sources such as web servers or external files. If you have a JSON dictionary for example, you may be extracting values as such:

let json: [String: Any] = // ...
person.name = json["name"] as? NSString
person.age = json["age"] as? NSNumber
// ...

If you have many attributes, you don't want to keep repeating this mapping everytime you want to import data. CoreStore lets you write the data mapping code just once, and all you have to do is call importObject(...) or importUniqueObject(...) through BaseDataTransaction subclasses:

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let json: [String: Any] = // ...
        try! transaction.importObject(
            Into<MyPersonEntity>(),
            source: json
        )
    },
    completion: { _ in }
)

To support data import for an entity, implement either ImportableObject or ImportableUniqueObject on the NSManagedObject or CoreStoreObject subclass:

ImportableObject: Use this protocol if the object have no inherent uniqueness and new objects should always be added when calling importObject(...).
ImportableUniqueObject: Use this protocol to specify a unique ID for an object that will be used to distinguish whether a new object should be created or if an existing object should be updated when calling importUniqueObject(...).

Both protocols require implementers to specify an ImportSource which can be set to any type that the object can extract data from:

typealias ImportSource = NSDictionary

typealias ImportSource = [String: Any]

typealias ImportSource = NSData

You can even use external types from popular 3rd-party JSON libraries, or just simple tuples or primitives.

`ImportableObject`

ImportableObject is a very simple protocol:

public protocol ImportableObject: AnyObject {
    typealias ImportSource
    static func shouldInsert(from source: ImportSource, in transaction: BaseDataTransaction) -> Bool
    func didInsert(from source: ImportSource, in transaction: BaseDataTransaction) throws
}

First, set ImportSource to the expected type of the data source:

typealias ImportSource = [String: Any]

This lets us call importObject(_:source:) with any [String: Any] type as the argument to source:

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let json: [String: Any] = // ...
        try! transaction.importObject(
            Into<MyPersonEntity>(),
            source: json
        )
        // ...
    },
    completion: { _ in }
)

The actual extraction and assignment of values should be implemented in the didInsert(from:in:) method of the ImportableObject protocol:

func didInsert(from source: ImportSource, in transaction: BaseDataTransaction) throws {
    self.name = source["name"] as? NSString
    self.age = source["age"] as? NSNumber
    // ...
}

Transactions also let you import multiple objects at once using the importObjects(_:sourceArray:) method:

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let jsonArray: [[String: Any]] = // ...
        try! transaction.importObjects(
            Into<MyPersonEntity>(),
            sourceArray: jsonArray // make sure this is of type Array<MyPersonEntity.ImportSource>
        )
        // ...
    },
    completion: { _ in }
)

Doing so tells the transaction to iterate through the array of import sources and calls shouldInsert(from:in:) on the ImportableObject to determine which instances should be created. You can do validations and return false from shouldInsert(from:in:) if you want to skip importing from a source and continue on with the other sources in the array.

If on the other hand, your validation in one of the sources failed in such a manner that all other sources should also be rolled back and cancelled, you can throw from within didInsert(from:in:):

func didInsert(from source: ImportSource, in transaction: BaseDataTransaction) throws {
    self.name = source["name"] as? NSString
    self.age = source["age"] as? NSNumber
    // ...
    if self.name == nil {
        throw Errors.InvalidNameError
    }
}

Doing so can let you abandon an invalid transaction immediately:

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let jsonArray: [[String: Any]] = // ...

        try transaction.importObjects(
            Into<MyPersonEntity>(),
            sourceArray: jsonArray
        )
    },
    success: {
        // ...
    },
    failure: { (error) in
        switch error {
        case Errors.InvalidNameError: print("Invalid name")
        // ...
        }
    }
)

`ImportableUniqueObject`

Typically, we don't just keep creating objects every time we import data. Usually we also need to update already existing objects. Implementing the ImportableUniqueObject protocol lets you specify a "unique ID" that transactions can use to search existing objects before creating new ones:

public protocol ImportableUniqueObject: ImportableObject {
    typealias ImportSource
    typealias UniqueIDType: ImportableAttributeType

    static var uniqueIDKeyPath: String { get }
    var uniqueIDValue: UniqueIDType { get set }

    static func shouldInsert(from source: ImportSource, in transaction: BaseDataTransaction) -> Bool
    static func shouldUpdate(from source: ImportSource, in transaction: BaseDataTransaction) -> Bool
    static func uniqueID(from source: ImportSource, in transaction: BaseDataTransaction) throws -> UniqueIDType?
    func didInsert(from source: ImportSource, in transaction: BaseDataTransaction) throws
    func update(from source: ImportSource, in transaction: BaseDataTransaction) throws
}

Notice that it has the same insert methods as ImportableObject, with additional methods for updates and for specifying the unique ID:

class var uniqueIDKeyPath: String {
    return #keyPath(MyPersonEntity.personID) 
}
var uniqueIDValue: Int { 
    get { return self.personID }
    set { self.personID = newValue }
}
class func uniqueID(from source: ImportSource, in transaction: BaseDataTransaction) throws -> Int? {
    return source["id"] as? Int
}

For ImportableUniqueObject, the extraction and assignment of values should be implemented from the update(from:in:) method. The didInsert(from:in:) by default calls update(from:in:), but you can separate the implementation for inserts and updates if needed.

You can then create/update an object by calling a transaction's importUniqueObject(...) method:

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let json: [String: Any] = // ...
        try! transaction.importUniqueObject(
            Into<MyPersonEntity>(),
            source: json
        )
        // ...
    },
    completion: { _ in }
)

or multiple objects at once with the importUniqueObjects(...) method:

dataStack.perform(
    asynchronous: { (transaction) -> Void in
        let jsonArray: [[String: AnyObject]] = // ...
        try! transaction.importUniqueObjects(
            Into<MyPersonEntity>(),
            sourceArray: jsonArray
        )
        // ...
    },
    completion: { _ in }
)

As with ImportableObject, you can control whether to skip importing an object by implementing shouldInsert(from:in:) and shouldUpdate(from:in:), or to cancel all objects by throwing an error from the uniqueID(from:in:), didInsert(from:in:) or update(from:in:) methods.

Fetching and Querying

Before we dive in, be aware that CoreStore distinguishes between fetching and querying:

A fetch executes searches from a specific transaction or data stack. This means fetches can include pending objects (i.e. before a transaction calls on commit().) Use fetches when:
- results need to be NSManagedObject or CoreStoreObject instances
- unsaved objects should be included in the search (though fetches can be configured to exclude unsaved ones)
A query pulls data straight from the persistent store. This means faster searches when computing aggregates such as count, min, max, etc. Use queries when:
- you need to compute aggregate functions (see below for a list of supported functions)
- results can be raw values like NSStrings, NSNumbers, Ints, NSDates, an NSDictionary of key-values, or any type that conform to QueryableAttributeType. (See QueryableAttributeType.swift for a list of built-in types)
- only values for specified attribute keys need to be included in the results
- unsaved objects should be ignored

`From` clause

The search conditions for fetches and queries are specified using clauses. All fetches and queries require a From clause that indicates the target entity type:

let people = try dataStack.fetchAll(From<MyPersonEntity>())

people in the example above will be of type [MyPersonEntity]. The From<MyPersonEntity>() clause indicates a fetch to all persistent stores that MyPersonEntity belong to.

If the entity exists in multiple configurations and you need to only search from a particular configuration, indicate in the From clause the configuration name for the destination persistent store:

let people = try dataStack.fetchAll(From<MyPersonEntity>("Config1")) // ignore objects in persistent stores other than the "Config1" configuration

or if the persistent store is the auto-generated "Default" configuration, specify nil:

let person = try dataStack.fetchAll(From<MyPersonEntity>(nil))

Now we know how to use a From clause, let's move on to fetching and querying.

Fetching

There are currently 5 fetch methods you can call from CoreStore, from a DataStack instance, or from a BaseDataTransaction instance. All of the methods below accept the same parameters: a required From clause, and an optional series of Where, OrderBy, and/or Tweak clauses.

fetchAll(...) - returns an array of all objects that match the criteria.
fetchOne(...) - returns the first object that match the criteria.
fetchCount(...) - returns the number of objects that match the criteria.
fetchObjectIDs(...) - returns an array of NSManagedObjectIDs for all objects that match the criteria.
fetchObjectID(...) - returns the NSManagedObjectIDs for the first objects that match the criteria.

Each method's purpose is straightforward, but we need to understand how to set the clauses for the fetch.

`Where` clause

The Where clause is CoreStore's NSPredicate wrapper. It specifies the search filter to use when fetching (or querying). It implements all initializers that NSPredicate does (except for -predicateWithBlock:, which Core Data does not support):

var people = try dataStack.fetchAll(
    From<MyPersonEntity>(),
    Where<MyPersonEntity>("%K > %d", "age", 30) // string format initializer
)
people = try dataStack.fetchAll(
    From<MyPersonEntity>(),
    Where<MyPersonEntity>(true) // boolean initializer
)

If you do have an existing NSPredicate instance already, you can pass that to Where as well:

let predicate = NSPredicate(...)
var people = dataStack.fetchAll(
    From<MyPersonEntity>(),
    Where<MyPersonEntity>(predicate) // predicate initializer
)

Where clauses are generic types. To avoid verbose repetition of the generic object type, fetch methods support Fetch Chain builders. We can also use Swift's Smart KeyPaths as the Where clause expression:

var people = try dataStack.fetchAll(
    From<MyPersonEntity>()
        .where(\.age > 30) // Type-safe!
)

Where clauses also implement the &&, ||, and ! logic operators, so you can provide logical conditions without writing too much AND, OR, and NOT strings:

var people = try dataStack.fetchAll(
    From<MyPersonEntity>()
        .where(\.age > 30 && \.gender == "M")
)

If you do not provide a Where clause, all objects that belong to the specified From will be returned.

`OrderBy` clause

The OrderBy clause is CoreStore's NSSortDescriptor wrapper. Use it to specify attribute keys in which to sort the fetch (or query) results with.

var mostValuablePeople = try dataStack.fetchAll(
    From<MyPersonEntity>(),
    OrderBy<MyPersonEntity>(.descending("rating"), .ascending("surname"))
)

As seen above, OrderBy accepts a list of SortKey enumeration values, which can be either .ascending or .descending. As with Where clauses, OrderBy clauses are also generic types. To avoid verbose repetition of the generic object type, fetch methods support Fetch Chain builders. We can also use Swift's Smart KeyPaths as the OrderBy clause expression:

var people = try dataStack.fetchAll(
    From<MyPersonEntity>()
        .orderBy(.descending(\.rating), .ascending(\.surname)) // Type-safe!
)

You can use the + and += operator to append OrderBys together. This is useful when sorting conditionally:

var orderBy = OrderBy<MyPersonEntity>(.descending(\.rating))
if sortFromYoungest {
    orderBy += OrderBy(.ascending(\.age))
}
var mostValuablePeople = try dataStack.fetchAll(
    From<MyPersonEntity>(),
    orderBy
)

`Tweak` clause

The Tweak clause lets you, uh, tweak the fetch (or query). Tweak exposes the NSFetchRequest in a closure where you can make changes to its properties:

var people = try dataStack.fetchAll(
    From<MyPersonEntity>(),
    Where<MyPersonEntity>("age > %d", 30),
    OrderBy<MyPersonEntity>(.ascending("surname")),
    Tweak { (fetchRequest) -> Void in
        fetchRequest.includesPendingChanges = false
        fetchRequest.returnsObjectsAsFaults = false
        fetchRequest.includesSubentities = false
    }
)

Tweak also supports Fetch Chain builders:

var people = try dataStack.fetchAll(
    From<MyPersonEntity>(),
        .where(\.age > 30)
        .orderBy(.ascending(\.surname))
        .tweak {
            $0.includesPendingChanges = false
            $0.returnsObjectsAsFaults = false
            $0.includesSubentities = false
        }
)

The clauses are evaluated the order they appear in the fetch/query, so you typically need to set Tweak as the last clause. Tweak's closure is executed only just before the fetch occurs, so make sure that any values captured by the closure is not prone to race conditions.

While Tweak lets you micro-configure the NSFetchRequest, note that CoreStore already preconfigured that NSFetchRequest to suitable defaults. Only use Tweak when you know what you are doing!

Querying

One of the functionalities overlooked by other Core Data wrapper libraries is raw properties fetching. If you are familiar with NSDictionaryResultType and -[NSFetchedRequest propertiesToFetch], you probably know how painful it is to setup a query for raw values and aggregate values. CoreStore makes this easy by exposing the 2 methods below:

queryValue(...) - returns a single raw value for an attribute or for an aggregate value. If there are multiple results, queryValue(...) only returns the first item.
queryAttributes(...) - returns an array of dictionaries containing attribute keys with their corresponding values.

Both methods above accept the same parameters: a required From clause, a required Select<T> clause, and an optional series of Where, OrderBy, GroupBy, and/or Tweak clauses.

Setting up the From, Where, OrderBy, and Tweak clauses is similar to how you would when fetching. For querying, you also need to know how to use the Select<T> and GroupBy clauses.

`Select<T>` clause

The Select<T> clause specifies the target attribute/aggregate key, as well as the expected return type:

let johnsAge = try dataStack.queryValue(
    From<MyPersonEntity>(),
    Select<Int>("age"),
    Where<MyPersonEntity>("name == %@", "John Smith")
)

The example above queries the "age" property for the first object that matches the Where condition. johnsAge will be bound to type Int?, as indicated by the Select<Int> generic type. For queryValue(...), types that conform to QueryableAttributeType are allowed as the return type (and therefore as the generic type for Select<T>).

For queryAttributes(...), only NSDictionary is valid for Select, thus you are allowed to omit the generic type:

let allAges = try dataStack.queryAttributes(
    From<MyPersonEntity>(),
    Select("age")
)

query methods also support Query Chain builders. We can also use Swift's Smart KeyPaths to use in the expressions:

let johnsAge = try dataStack.queryValue(
    From<MyPersonEntity>()
        .select(\.age) // binds the result to Int
        .where(\.name == "John Smith")
)

If you only need a value for a particular attribute, you can just specify the key name (like we did with Select<Int>("age")), but several aggregate functions can also be used as parameter to Select:

.average(...)
.count(...)
.maximum(...)
.minimum(...)
.sum(...)

let oldestAge = try dataStack.queryValue(
    From<MyPersonEntity>(),
    Select<Int>(.maximum("age"))
)

For queryAttributes(...) which returns an array of dictionaries, you can specify multiple attributes/aggregates to Select:

let personJSON = try dataStack.queryAttributes(
    From<MyPersonEntity>(),
    Select("name", "age")
)

personJSON will then have the value:

[
    [
        "name": "John Smith",
        "age": 30
    ],
    [
        "name": "Jane Doe",
        "age": 22
    ]
]

You can also include an aggregate as well:

let personJSON = try dataStack.queryAttributes(
    From<MyPersonEntity>(),
    Select("name", .count("friends"))
)

which returns:

[
    [
        "name": "John Smith",
        "count(friends)": 42
    ],
    [
        "name": "Jane Doe",
        "count(friends)": 231
    ]
]

The "count(friends)" key name was automatically used by CoreStore, but you can specify your own key alias if you need:

let personJSON = try dataStack.queryAttributes(
    From<MyPersonEntity>(),
    Select("name", .count("friends", as: "friendsCount"))
)

which now returns:

[
    [
        "name": "John Smith",
        "friendsCount": 42
    ],
    [
        "name": "Jane Doe",
        "friendsCount": 231
    ]
]

`GroupBy` clause

The GroupBy clause lets you group results by a specified attribute/aggregate. This is useful only for queryAttributes(...) since queryValue(...) just returns the first value.

let personJSON = try dataStack.queryAttributes(
    From<MyPersonEntity>(),
    Select("age", .count("age", as: "count")),
    GroupBy("age")
)

GroupBy clauses are also generic types and support Query Chain builders. We can also use Swift's Smart KeyPaths to use in the expressions:

let personJSON = try dataStack.queryAttributes(
    From<MyPersonEntity>()
        .select(.attribute(\.age), .count(\.age, as: "count"))
        .groupBy(\.age)
)

this returns dictionaries that shows the count for each "age":

[
    [
        "age": 42,
        "count": 1
    ],
    [
        "age": 22,
        "count": 1
    ]
]

Logging and error reporting

One unfortunate thing when using some third-party libraries is that they usually pollute the console with their own logging mechanisms. CoreStore provides its own default logging class, but you can plug-in your own favorite logger by implementing the CoreStoreLogger protocol.

public protocol CoreStoreLogger {
    func log(level level: LogLevel, message: String, fileName: StaticString, lineNumber: Int, functionName: StaticString)
    func log(error error: CoreStoreError, message: String, fileName: StaticString, lineNumber: Int, functionName: StaticString)
    func assert(@autoclosure condition: () -> Bool, @autoclosure message: () -> String, fileName: StaticString, lineNumber: Int, functionName: StaticString)
    func abort(message: String, fileName: StaticString, lineNumber: Int, functionName: StaticString)
}

Implement this protocol with your custom class then pass the instance to CoreStoreDefaults.logger:

CoreStoreDefaults.logger = MyLogger()

Doing so channels all logging calls to your logger.

Note that to keep the call stack information intact, all calls to these methods are NOT thread-managed. Therefore you have to make sure that your logger is thread-safe or you may otherwise have to dispatch your logging implementation to a serial queue.

Take special care when implementing CoreStoreLogger's assert(...) and abort(...) functions:

assert(...): The behavior between DEBUG and release builds, or -O and -Onone, are all left to the implementers' responsibility. CoreStore calls CoreStoreLogger.assert(...) only for invalid but usually recoverable errors (for example, early validation failures that may cause an error thrown and handled somewhere else)
abort(...): This method is the last-chance for your app to synchronously log a fatal error within CoreStore. The app will be terminated right after this function is called (CoreStore calls fatalError() internally)

All CoreStore types have very useful (and pretty formatted!) print(...) outputs. A couple of examples, ListMonitor:

CoreStoreError.mappingModelNotFoundError:

These are all implemented with CustomDebugStringConvertible.debugDescription, so they work with lldb's po command as well.

Observing changes and notifications

CoreStore provides type-safe wrappers for observing managed objects:

	🆕ObjectPublisher	ObjectMonitor	🆕ListPublisher	ListMonitor
Number of objects	1	1	N	N
Allows multiple observers	✅	✅	✅	✅
Emits fine-grained changes	❌	✅	❌	✅
Emits DiffableDataSource snapshots	✅	❌	✅	❌
Delegate methods	❌	✅	❌	✅
Closure callback	✅	❌	✅	❌
SwiftUI support	✅	❌	✅	❌

Observe a single property

To get notifications for single property changes in an object, there are two methods depending on the object's base class.

For NSManagedObject subclasses: Use the standard KVO method:

let observer = person.observe(\.age, options: [.new]) { (person, change)
    print("Happy \(change.newValue)th birthday!")
}

For CoreStoreObject subclasses: Call the observe(...) method directly on the property. You'll notice that the API itself is a bit similar to the KVO method:

let observer = person.age.observe(options: [.new]) { (person, change)
    print("Happy \(change.newValue)th birthday!")
}

For both methods, you will need to keep a reference to the returned observer for the duration of the observation.

Observe a single object's updates

Observers of an ObjectPublisher can receive notifications if any of the object's property changes. You can create an ObjectPublisher from the object directly:

let objectPublisher: ObjectPublisher<Person> = person.asPublisher(in: dataStack)

or by indexing a ListPublisher's ListSnapshot:

let listPublisher: ListPublisher<Person> = // ...
// ...
let objectPublisher = listPublisher.snapshot[indexPath]

(See ListPublisher examples below)

To receive notifications, call the ObjectPublisher's addObserve(...) method passing the owner of the callback closure:

objectPublisher.addObserver(self) { [weak self] (objectPublisher) in
    let snapshot: ObjectSnapshot<Person> = objectPublisher.snapshot
    // handle changes
}

Note that the owner instance will not be retained. You may call ObjectPublisher.removeObserver(...) explicitly to stop receiving notifications, but the ObjectPublisher also discontinues sending events to deallocated observers.

The ObjectSnapshot returned from the ObjectPublisher.snapshot property returns a full-copy struct of all properties of the object. This is ideal for managing states as they are thread-safe and are not affected by further changes to the actual object. ObjectPublisher automatically updates its snapshot value to the latest state of the object.

(A reactive-programming variant of this method is explained in detail in the section on ObjectPublisher Combine publishers)

Observe a single object's per-property updates

If you need to track specifically which properties change in an object, implement the ObjectObserver protocol and specify the EntityType:

class MyViewController: UIViewController, ObjectObserver {
    func objectMonitor(monitor: ObjectMonitor<MyPersonEntity>, willUpdateObject object: MyPersonEntity) {
        // ...
    }
    
    func objectMonitor(monitor: ObjectMonitor<MyPersonEntity>, didUpdateObject object: MyPersonEntity, changedPersistentKeys: Set<KeyPathString>) {
        // ...
    }
    
    func objectMonitor(monitor: ObjectMonitor<MyPersonEntity>, didDeleteObject object: MyPersonEntity) {
        // ...
    }
}

We then need to keep an ObjectMonitor instance and register our ObjectObserver as an observer:

let person: MyPersonEntity = // ...
self.monitor = dataStack.monitorObject(person)
self.monitor.addObserver(self)

The controller will then notify our observer whenever the object's attributes change. You can add multiple ObjectObservers to a single ObjectMonitor without any problem. This means you can just share around the ObjectMonitor instance to different screens without problem.

You can get ObjectMonitor's object through its object property. If the object is deleted, the object property will become nil to prevent further access.

While ObjectMonitor exposes removeObserver(...) as well, it only stores weak references of the observers and will safely unregister deallocated observers.

Observe a diffable list

Observers of a ListPublisher can receive notifications whenever its fetched result set changes. You can create a ListPublisher by fetching from the DataStack:

let listPublisher = dataStack.listPublisher(
    From<Person>()
        .sectionBy(\.age") { "Age \($0)" } // sections are optional
        .where(\.title == "Engineer")
        .orderBy(.ascending(\.lastName))
)

To receive notifications, call the ListPublisher's addObserve(...) method passing the owner of the callback closure:

listPublisher.addObserver(self) { [weak self] (listPublisher) in
    let snapshot: ListSnapshot<Person> = listPublisher.snapshot
    // handle changes
}

Note that the owner instance will not be retained. You may call ListPublisher.removeObserver(...) explicitly to stop receiving notifications, but the ListPublisher also discontinues sending events to deallocated observers.

The ListSnapshot returned from the ListPublisher.snapshot property returns a full-copy struct of all sections and NSManagedObject items in the list. This is ideal for managing states as they are thread-safe and are not affected by further changes to the result set. ListPublisher automatically updates its snapshot value to the latest state of the fetch.

(A reactive-programming variant of this method is explained in detail in the section on ListPublisher Combine publishers)

Unlike ListMonitors (See ListMonitor examples below), a ListPublisher does not track detailed inserts, deletes, and moves. In return, a ListPublisher is a lot more lightweight and are designed to work well with DiffableDataSource.TableViewAdapters and DiffableDataSource.CollectionViewAdapters:

self.dataSource = DiffableDataSource.CollectionViewAdapter<Person>(
    collectionView: self.collectionView,
    dataStack: CoreStoreDefaults.dataStack,
    cellProvider: { (collectionView, indexPath, person) in
        let cell = collectionView.dequeueReusableCell(withReuseIdentifier: "PersonCell") as! PersonCell
        cell.setPerson(person)
        return cell
    }
)

// ...

listPublisher.addObserver(self) { [weak self] (listPublisher) in
   self?.dataSource?.apply(
       listPublisher.snapshot, animatingDifferences: true
   )
}

Observe detailed list changes

If you need to track each object's inserts, deletes, moves, and updates, implement one of the ListObserver protocols and specify the EntityType:

class MyViewController: UIViewController, ListObserver {
    func listMonitorDidChange(monitor: ListMonitor<MyPersonEntity>) {
        // ...
    }
    
    func listMonitorDidRefetch(monitor: ListMonitor<MyPersonEntity>) {
        // ...
    }
}

Including ListObserver, there are 3 observer protocols you can implement depending on how detailed you need to handle a change notification:

ListObserver: lets you handle these callback methods:

    func listMonitorWillChange(_ monitor: ListMonitor<MyPersonEntity>)
    func listMonitorDidChange(_ monitor: ListMonitor<MyPersonEntity>)
    func listMonitorWillRefetch(_ monitor: ListMonitor<MyPersonEntity>)
    func listMonitorDidRefetch(_ monitor: ListMonitor<MyPersonEntity>)

listMonitorDidChange(_:) and listMonitorDidRefetch(_:) implementations are both required. listMonitorDidChange(_:) is called whenever the ListMonitor's count, order, or filtered objects change. listMonitorDidRefetch(_:) is called when the ListMonitor.refetch() was executed or if the internal persistent store was changed.

ListObjectObserver: in addition to ListObserver methods, also lets you handle object inserts, updates, and deletes:

    func listMonitor(_ monitor: ListMonitor<MyPersonEntity>, didInsertObject object: MyPersonEntity, toIndexPath indexPath: IndexPath)
    func listMonitor(_ monitor: ListMonitor<MyPersonEntity>, didDeleteObject object: MyPersonEntity, fromIndexPath indexPath: IndexPath)
    func listMonitor(_ monitor: ListMonitor<MyPersonEntity>, didUpdateObject object: MyPersonEntity, atIndexPath indexPath: IndexPath)
    func listMonitor(_ monitor: ListMonitor<MyPersonEntity>, didMoveObject object: MyPersonEntity, fromIndexPath: IndexPath, toIndexPath: IndexPath)

ListSectionObserver: in addition to ListObjectObserver methods, also lets you handle section inserts and deletes:

    func listMonitor(_ monitor: ListMonitor<MyPersonEntity>, didInsertSection sectionInfo: NSFetchedResultsSectionInfo, toSectionIndex sectionIndex: Int)
    func listMonitor(_ monitor: ListMonitor<MyPersonEntity>, didDeleteSection sectionInfo: NSFetchedResultsSectionInfo, fromSectionIndex sectionIndex: Int)

We then need to create a ListMonitor instance and register our ListObserver as an observer:

self.monitor = dataStack.monitorList(
    From<MyPersonEntity>()
        .where(\.age > 30)
        .orderBy(.ascending(\.name))
        .tweak { $0.fetchBatchSize = 20 }
)
self.monitor.addObserver(self)

Similar to ObjectMonitor, a ListMonitor can also have multiple ListObservers registered to a single ListMonitor.

If you have noticed, the monitorList(...) method accepts Where, OrderBy, and Tweak clauses exactly like a fetch. As the list maintained by ListMonitor needs to have a deterministic order, at least the From and OrderBy clauses are required.

A ListMonitor created from monitorList(...) will maintain a single-section list. You can therefore access its contents with just an index:

let firstPerson = self.monitor[0]

If the list needs to be grouped into sections, create the ListMonitor instance with the monitorSectionedList(...) method and a SectionBy clause:

self.monitor = dataStack.monitorSectionedList(
    From<MyPersonEntity>()
        .sectionBy(\.age)
        .where(\.gender == "M")
        .orderBy(.ascending(\.age), .ascending(\.name))
        .tweak { $0.fetchBatchSize = 20 }
)

A list controller created this way will group the objects by the attribute key indicated by the SectionBy clause. One more thing to remember is that the OrderBy clause should sort the list in such a way that the SectionBy attribute would be sorted together (a requirement shared by NSFetchedResultsController.)

The SectionBy clause can also be passed a closure to transform the section name into a displayable string:

self.monitor = dataStack.monitorSectionedList(
    From<MyPersonEntity>()
        .sectionBy(\.age) { (sectionName) -> String? in
            "\(sectionName) years old"
        }
        .orderBy(.ascending(\.age), .ascending(\.name))
)

This is useful when implementing a UITableViewDelegate's section header:

func tableView(_ tableView: UITableView, titleForHeaderInSection section: Int) -> String? {
    let sectionInfo = self.monitor.sectionInfoAtIndex(section)
    return sectionInfo.name
}

To access the objects of a sectioned list, use an IndexPath or a tuple:

let indexPath = IndexPath(row: 2, section: 1)
let person1 = self.monitor[indexPath]
let person2 = self.monitor[1, 2]
// person1 and person2 are the same object

Type-safe `CoreStoreObject`s

Starting CoreStore 4.0, we can now create persisted objects without depending on .xcdatamodeld Core Data files. The new CoreStoreObject subclass replaces NSManagedObject, and specially-typed properties declared on these classes will be synthesized as Core Data attributes.

class Animal: CoreStoreObject {
    @Field.Stored("species")
    var species: String = ""
}

class Dog: Animal {
    @Field.Stored("nickname")
    var nickname: String?
    
    @Field.Relationship("master")
    var master: Person?
}

class Person: CoreStoreObject {
    @Field.Stored("name")
    var name: String = ""
    
    @Field.Relationship("pets", inverse: \Dog.$master)
    var pets: Set<Dog>
}

The property names to be saved to Core Data is specified as the keyPath argument. This lets us refactor our Swift code without affecting the underlying database. For example:

class Person: CoreStoreObject {
    @Field.Stored("name")
    private var internalName: String = ""
    // note property name is independent of the storage key name
}

Here we used the property name internalName and made it private, but the underlying key-path "name" was unchanged so our model will not trigger a data migration.

To tell the DataStack about these types, add all CoreStoreObjects' entities to a CoreStoreSchema:


CoreStoreDefaults.dataStack = DataStack(
    CoreStoreSchema(
        modelVersion: "V1",
        entities: [
            Entity<Animal>("Animal", isAbstract: true),
            Entity<Dog>("Dog"),
            Entity<Person>("Person")
        ]
    )
)
CoreStoreDefaults.dataStack.addStorage(/* ... */)

And that's all CoreStore needs to build the model; we don't need .xcdatamodeld files anymore.

In addition, @Field properties can be used to create type-safe key-path strings

let keyPath = String(keyPath: \Dog.$nickname)

as well as Where and OrderBy clauses

let puppies = try dataStack.fetchAll(
    From<Dog>()
        .where(\.$age < 5)
        .orderBy(.ascending(\.$age))
)

All CoreStore APIs that are usable with NSManagedObjects are also available for CoreStoreObjects. These include ListMonitors, ImportableObjects, fetching, etc.

New `@Field` Property Wrapper syntax

⚠️Important: @Field properties are only supported for CoreStoreObject subclasses. If you are using NSManagedObjects, you need to keep using @NSManaged for your attributes.

Starting CoreStore 7.1.0, CoreStoreObject properties may be converted to @Field Property Wrappers.

‼️ Please take note of the warnings below before converting or else the model's hash might change.

If conversion is too risky, the current Value.Required, Value.Optional, Transformable.Required, Transformable.Optional, Relationship.ToOne, Relationship.ToManyOrdered, and Relationship.ToManyUnordered will all be supported for while so you can opt to use them as is for now.

‼️ This cannot be stressed enough, but please make sure to set your schema's VersionLock before converting!

`@Field.Stored`

The @Field.Stored property wrapper is used for persisted value types. This is the replacement for "non-transient" Value.Required and Value.Optional properties.

Before

@Field.Stored

class Person: CoreStoreObject {
    

    let title = Value.Required<String>("title", initial: "Mr.")
    let nickname = Value.Optional<String>("nickname")
}

class Person: CoreStoreObject {
    

    @Field.Stored("title")
    var title: String = "Mr."
    

    @Field.Stored("nickname")
    var nickname: String?
}

⚠️ Only Value.Required and Value.Optional that are NOT transient values can be converted to Field.Stored. For transient/computed properties, refer to @Field.Virtual properties in the next section. ⚠️ When converting, make sure that all parameters, including the default values, are exactly the same or else the model's hash might change.

`@Field.Virtual`

The @Field.Virtual property wrapper is used for unsaved, computed value types. This is the replacement for "transient" Value.Required and Value.Optional properties.

Before

@Field.Virtual

class Animal: CoreStoreObject {
    

    let speciesPlural = Value.Required<String>(
        "speciesPlural",
        transient: true,
        customGetter: Animal.getSpeciesPlural(_:)
    )
    

    let species = Value.Required<String>("species", initial: "")
    

    static func getSpeciesPlural(_ partialObject: PartialObject<Animal>) -> String? {
        let species = partialObject.value(for: { $0.species })
        return species + "s"
    }
}

class Animal: CoreStoreObject {
    

    @Field.Virtual(
        "speciesPlural",
        customGetter: { (object, field) in
            return object.$species.value + "s"
        }
    )
    var speciesPlural: String
    

    @Field.Stored("species")
    var species: String = ""
}

⚠️ Only Value.Required and Value.Optional that ARE transient values can be converted to Field.Virtual. For non-transient properties, refer to @Field.Stored properties in the previous section. ⚠️ When converting, make sure that all parameters, including the default values, are exactly the same or else the model's hash might change.

`@Field.Coded`

The @Field.Coded property wrapper is used for binary-codable values. This is the new counterpart, not replacement, for Transformable.Required and Transformable.Optional properties. @Field.Coded also supports other encodings such as JSON and custom binary converters.

‼️ The current Transformable.Required and Transformable.Optional mechanism have no safe one-to-one conversion to @Field.Coded. Please use @Field.Coded only for newly added attributes.

Before

@Field.Coded

class Vehicle: CoreStoreObject {
    

    let color = Transformable.Optional<UIColor>("color", initial: .white)
}

class Vehicle: CoreStoreObject {
    

    @Field.Coded("color", coder: FieldCoders.NSCoding.self)
    var color: UIColor? = .white
}

Built-in encoders such as FieldCoders.NSCoding, FieldCoders.Json, and FieldCoders.Plist are available, and custom encoding/decoding is also supported:

class Person: CoreStoreObject {
    
    struct CustomInfo: Codable {
        // ...
    }
    
    @Field.Coded("otherInfo", coder: FieldCoders.Json.self)
    var otherInfo: CustomInfo?
    
    @Field.Coded(
        "photo",
        coder: {
            encode: { $0.toData() },
            decode: { Photo(fromData: $0) }
        }
    )
    var photo: Photo?
}

‼️Important: Any changes in the encoders/decoders are not reflected in the VersionLock, so make sure that the encoder and decoder logic is compatible for all versions of your persistent store.

`@Field.Relationship`

The @Field.Relationship property wrapper is used for link relationships with other CoreStoreObjects. This is the replacement for Relationship.ToOne, Relationship.ToManyOrdered, and Relationship.ToManyUnordered properties.

The type of relationship is determined by the @Field.Relationship generic type:

Optional<T> : To-one relationship
Array<T> : To-many ordered relationship
Set<T> : To-many unordered relationship

Before

@Field.Stored

class Pet: CoreStoreObject {
    

    let master = Relationship.ToOne<Person>("master")
}
class Person: CoreStoreObject {
    

    let pets: Relationship.ToManyUnordered<Pet>("pets", inverse: \.$master)
}

class Pet: CoreStoreObject {
    

    @Field.Relationship("master")
    var master: Person?
}
class Person: CoreStoreObject {
    

    @Field.Relationship("pets", inverse: \.$master)
    var pets: Set<Pet>
}

⚠️ When converting, make sure that all parameters, including the default values, are exactly the same or else the model's hash might change.

Also note how Relationships are linked statically with the inverse: argument. All relationships are required to have an "inverse" relationship. Unfortunately, due to Swift compiler limitation we can declare the inverse: on only one of the relationship-pair.

`@Field` usage notes

Accessor syntax

When using key-path utilities, properties using @Field property wrappers need to use the $ syntax:

Before: From<Person>.where(\.title == "Mr.")
After: From<Person>.where(\.$title == "Mr.")

This applies to property access using ObjectPublishers and ObjectSnapshots.

Before: let name = personSnapshot.name
After: let name = personSnapshot.$name

Default values vs. Initial values

One common mistake when assigning default values to CoreStoreObject properties is to assign it a value and expect it to be evaluated whenever an object is created:

// ❌
class Person: CoreStoreObject {

    @Field.Stored("identifier")
    var identifier: UUID = UUID() // Wrong!
    
    @Field.Stored("createdDate")
    var createdDate: Date = Date() // Wrong!
}

This default value will be evaluated only when the DataStack sets up the schema, and all instances will end up having the same values. This syntax for "default values" are usually used only for actual reasonable constant values, or sentinel values such as "" or 0.

For actual "initial values", @Field.Stored and @Field.Coded now supports dynamic evaluation during object creation via the dynamicInitialValue: argument:

// ✅
class Person: CoreStoreObject {

    @Field.Stored("identifier", dynamicInitialValue: { UUID() })
    var identifier: UUID
    
    @Field.Stored("createdDate", dynamicInitialValue: { Date() })
    var createdDate: Date
}

When using this feature, a "default value" should not be assigned (i.e. no = expression).

`VersionLock`s

While it is convenient to be able to declare entities only in code, it is worrying that we might accidentally change the CoreStoreObject's properties and break our users' model version history. For this, the CoreStoreSchema allows us to "lock" our properties to a particular configuration. Any changes to that VersionLock will raise an assertion failure during the CoreStoreSchema initialization, so you can then look for the commit which changed the VersionLock hash.

To use VersionLocks, create the CoreStoreSchema, run the app, and look for a similar log message that is automatically printed to the console:

Copy this dictionary value and use it as the versionLock: argument of the CoreStoreSchema initializer:

CoreStoreSchema(
    modelVersion: "V1",
    entities: [
        Entity<Animal>("Animal", isAbstract: true),
        Entity<Dog>("Dog"),
        Entity<Person>("Person"),
    ],
    versionLock: [
        "Animal": [0x1b59d511019695cf, 0xdeb97e86c5eff179, 0x1cfd80745646cb3, 0x4ff99416175b5b9a],
        "Dog": [0xe3f0afeb109b283a, 0x29998d292938eb61, 0x6aab788333cfc2a3, 0x492ff1d295910ea7],
        "Person": [0x66d8bbfd8b21561f, 0xcecec69ecae3570f, 0xc4b73d71256214ef, 0x89b99bfe3e013e8b]
    ]
)

You can also get this hash after the DataStack has been fully set up by printing to the console:

print(CoreStoreDefaults.dataStack.modelSchema.printCoreStoreSchema())

Once the version lock is set, any changes in the properties or to the model will trigger an assertion failure similar to this:

Reactive Programming

RxSwift

RxSwift utilities are available through the RxCoreStore external module.

Combine

Combine publishers are available from the DataStack, ListPublisher, and ObjectPublisher's .reactive namespace property.

`DataStack.reactive`

Adding a storage through DataStack.reactive.addStorage(_:) returns a publisher that reports a MigrationProgress enum value. The .migrating value is only emitted if the storage goes through a migration. Refer to the Setting up section for details on the storage setup process itself.

dataStack.reactive
    .addStorage(
        SQLiteStore(fileName: "core_data.sqlite")
    )
    .sink(
        receiveCompletion: { result in
            // ...
        },
        receiveValue: { (progress) in
            print("\(round(progress.fractionCompleted * 100)) %") // 0.0 ~ 1.0
            switch progress {
            case .migrating(let storage, let nsProgress):
                // ...
            case .finished(let storage, let migrationRequired):
                // ...
            }
        }
    )
    .store(in: &cancellables)

Transactions are also available as publishers through DataStack.reactive.perform(_:), which returns a Combine Future that emits any type returned from the closure parameter:

dataStack.reactive
    .perform(
        asynchronous: { (transaction) -> (inserted: Set<NSManagedObject>, deleted: Set<NSManagedObject>) in

            // ...
            return (
                transaction.insertedObjects(),
                transaction.deletedObjects()
            )
        }
    )
    .sink(
        receiveCompletion: { result in
            // ...
        },
        receiveValue: { value in
            let inserted = dataStack.fetchExisting(value0.inserted)
            let deleted = dataStack.fetchExisting(value0.deleted)
            // ...
        }
    )
    .store(in: &cancellables)

For importing convenience, ImportableObject and ImportableUniqueObjects can be imported directly through DataStack.reactive.import[Unique]Object(_:source:) and DataStack.reactive.import[Unique]Objects(_:sourceArray:) without having to create a transaction block. In this case the publisher emits objects that are already usable directly from the main queue:

dataStack.reactive
    .importUniqueObjects(
        Into<Person>(),
        sourceArray: [
            ["name": "John"],
            ["name": "Bob"],
            ["name": "Joe"]
        ]
    )
    .sink(
        receiveCompletion: { result in
            // ...
        },
        receiveValue: { (people) in
            XCTAssertEqual(people?.count, 3)
            // ...
        }
    )
    .store(in: &cancellables)

`ListPublisher.reactive`

ListPublishers can be used to emit ListSnapshots through Combine using ListPublisher.reactive.snapshot(emitInitialValue:). The snapshot values are emitted in the main queue:

listPublisher.reactive
    .snapshot(emitInitialValue: true)
    .sink(
        receiveCompletion: { result in
            // ...
        },
        receiveValue: { (listSnapshot) in
            dataSource.apply(
                listSnapshot,
                animatingDifferences: true
            )
        }
    )
    .store(in: &cancellables)

`ObjectPublisher.reactive`

ObjectPublishers can be used to emit ObjectSnapshots through Combine using ObjectPublisher.reactive.snapshot(emitInitialValue:). The snapshot values are emitted in the main queue:

objectPublisher.reactive
    .snapshot(emitInitialValue: true)
    .sink(
        receiveCompletion: { result in
            // ...
        },
        receiveValue: { (objectSnapshot) in
            tableViewCell.setObject(objectSnapshot)
        }
    )
    .store(in: &tableViewCell.cancellables)

SwiftUI Utilities

Observing list and object changes in SwiftUI can be done through a couple of approaches. One is by creating views that autoupdates their contents, or by declaring property wrappers that trigger view updates. Both approaches are implemented almost the same internally, but this lets you be flexible depending on the structure of your custom Views.

SwiftUI Views

CoreStore provides View containers that automatically update their contents when data changes.

`ListReader`

A ListReader observes changes to a ListPublisher and creates its content views dynamically. The builder closure receives a ListSnapshot value that can be used to create the contents:

let people: ListPublisher<Person>

var body: some View {
   List {
       ListReader(self.people) { listSnapshot in
           ForEach(objectIn: listSnapshot) { person in
               // ...
           }
       }
   }
   .animation(.default)
}

As shown above, a typical use case is to use it together with CoreStore's ForEach extensions.

A KeyPath can also be optionally provided to extract specific properties of the ListSnapshot:

let people: ListPublisher<Person>

var body: some View {
    ListReader(self.people, keyPath: \.count) { count in
        Text("Number of members: \(count)")
    }
}

`ObjectReader`

An ObjectReader observes changes to an ObjectPublisher and creates its content views dynamically. The builder closure receives an ObjectSnapshot value that can be used to create the contents:

let person: ObjectPublisher<Person>

var body: some View {
   ObjectReader(self.person) { objectSnapshot in
       // ...
   }
   .animation(.default)
}

A KeyPath can also be optionally provided to extract specific properties of the ObjectSnapshot:

let person: ObjectPublisher<Person>

var body: some View {
    ObjectReader(self.person, keyPath: \.fullName) { fullName in
        Text("Name: \(fullName)")
    }
}

By default, an ObjectReader does not create its views wheen the object observed is deleted from the store. In those cases, the placeholder: argument can be used to provide a custom View to display when the object is deleted:

let person: ObjectPublisher<Person>

var body: some View {
   ObjectReader(
       self.person,
       content: { objectSnapshot in
           // ...
       },
       placeholder: { Text("Record not found") }
   )
}

SwiftUI Property Wrappers

As an alternative to ListReader and ObjectReader, CoreStore also provides property wrappers that trigger view updates when the data changes.

`ListState`

A @ListState property exposes a ListSnapshot value that automatically updates to the latest changes.

@ListState
var people: ListSnapshot<Person>

init(listPublisher: ListPublisher<Person>) {
   self._people = .init(listPublisher)
}

var body: some View {
   List {
       ForEach(objectIn: self.people) { objectSnapshot in
           // ...
       }
   }
   .animation(.default)
}

As shown above, a typical use case is to use it together with CoreStore's ForEach extensions.

If a ListPublisher instance is not available yet, the fetch can be done inline by providing the fetch clauses and the DataStack instance. By doing so the property can be declared without an initial value:

@ListState(
    From<Person>()
        .sectionBy(\.age)
        .where(\.isMember == true)
        .orderBy(.ascending(\.lastName))
)
var people: ListSnapshot<Person>

var body: some View {
    List {
        ForEach(sectionIn: self.people) { section in
            Section(header: Text(section.sectionID)) {
                ForEach(objectIn: section) { person in
                    // ...
                }
            }
        }
    }
    .animation(.default)
}

For other initialization variants, refer to the ListState.swift source documentations.

`ObjectState`

An @ObjectState property exposes an optional ObjectSnapshot value that automatically updates to the latest changes.

@ObjectState
var person: ObjectSnapshot<Person>?

init(objectPublisher: ObjectPublisher<Person>) {
   self._person = .init(objectPublisher)
}

var body: some View {
   HStack {
       if let person = self.person {
           AsyncImage(person.$avatarURL)
           Text(person.$fullName)
       }
       else {
           Text("Record removed")
       }
   }
}

As shown above, the property's value will be nil if the object has been deleted, so this can be used to display placeholders if needed.

SwiftUI Extensions

For convenience, CoreStore provides extensions to the standard SwiftUI types.

`ForEach`

Several ForEach initializer overloads are available. Choose depending on your input data and the expected closure data. Refer to the table below (Take note of the argument labels as they are important):

Data	Example
Signature: ForEach(_: [ObjectSnapshot<O>]) Closure: ObjectSnapshot<O>	let array: [ObjectSnapshot<Person>] var body: some View { List { ForEach(self.array) { objectSnapshot in // ... } } }
Signature: ForEach(objectIn: ListSnapshot<O>) Closure: ObjectPublisher<O>	let listSnapshot: ListSnapshot<Person> var body: some View { List { ForEach(objectIn: self.listSnapshot) { objectPublisher in // ... } } }
Signature: ForEach(objectIn: [ObjectSnapshot<O>]) Closure: ObjectPublisher<O>	let array: [ObjectSnapshot<Person>] var body: some View { List { ForEach(objectIn: self.array) { objectPublisher in // ... } } }
Signature: ForEach(sectionIn: ListSnapshot<O>) Closure: [ListSnapshot<O>.SectionInfo]	let listSnapshot: ListSnapshot<Person> var body: some View { List { ForEach(sectionIn: self.listSnapshot) { sectionInfo in // ... } } }
Signature: ForEach(objectIn: ListSnapshot<O>.SectionInfo) Closure: ObjectPublisher<O>	let listSnapshot: ListSnapshot<Person> var body: some View { List { ForEach(sectionIn: self.listSnapshot) { sectionInfo in ForEach(objectIn: sectionInfo) { objectPublisher in // ... } } } }

Roadmap

Prototyping stage

Widget/Extensions storage-sharing support
CloudKit support

Under consideration

Derived attributes
Cross-storage relationships (via Fetched Attributes)

Installation

Requires:
- iOS 13 SDK and above
- Swift 5.7 (Xcode 14.0+)
- For previous Swift versions: Swift 5.5, Swift 5.4, Swift 5.3
Dependencies:
- None
Other notes:
- The com.apple.CoreData.ConcurrencyDebug debug argument should be turned off for the app. CoreStore already guarantees safety for you by making the main context read-only, and by only executing transactions serially.

Install with CocoaPods

In your Podfile, add

pod 'CoreStore', '~> 9.1'

and run

pod update

This installs CoreStore as a framework. Declare import CoreStore in your swift file to use the library.

Install with Carthage

In your Cartfile, add

github "JohnEstropia/CoreStore" >= 9.1.0

and run

carthage update

This installs CoreStore as a framework. Declare import CoreStore in your swift file to use the library.

Install with Swift Package Manager:

dependencies: [
    .package(url: "https://github.com/JohnEstropia/CoreStore.git", from: "9.1.0"))
]

Declare import CoreStore in your swift file to use the library.

Install as Git Submodule

git submodule add https://github.com/JohnEstropia/CoreStore.git <destination directory>

Drag and drop CoreStore.xcodeproj to your project.

Install through Xcode's Swift Package Manager

From the File - Swift Packages - Add Package Dependency… menu, search for

CoreStore

where JohnEstropia is the Owner (forks may appear as well). Then add to your project

Changesets

For the full Changelog, refer to the Releases page.

Contact

You can reach me on Twitter @JohnEstropia

or join our Slack team at swift-corestore.slack.com

日本語の対応も可能なので是非！

Who uses CoreStore?

I'd love to hear about apps using CoreStore. Send me a message and I'll welcome any feedback!

License

CoreStore is released under an MIT license. See the LICENSE file for more information

EECS-151/fpga_labs_fa23

2023-09-11T01:42:59Z

julik/tickly

2023-09-10T02:06:49Z

Parses the subset of TCL needed for Nuke scripts

Tingenek/tcl-mqtt

2023-09-10T02:06:49Z

Small library to connect to a matt broker. Very, very basic.

muhdavi-pnl/NET17237

2023-09-10T01:45:26Z

Practice Big Data

NET17237

Practice Big Data

starzmustdie/ml-interview-questions-and-answers

2023-09-10T01:45:26Z

This repo contains my solutions to “Introduction to Machine Learning Interviews” by Chip Huyen.

Answers to Chip Huyen's Machine Learning Interview Questions

This repository contains my solutions to “Introduction to Machine Learning Interviews” by Chip Huyen. The section and question numbering has been structured to align with the original book by Chip Huyen in order to maintain consistency.

You can find the compiled PDF document in this GitHub repository or on Google Drive.

Please note: not all questions have answers, and there’s a possibility that my responses may not be entirely accurate. Therefore, it’s advisable to consider the information provided cautiously.

For any questions and contributions, you can reach me at:

email: zafir [dot] stojanovski [at] icloud [dot] com
twitter: twitter.com/starzmustdie
linkedin: linkedin.com/in/zafir-stojanovski

If you found this guide helpful, please consider supporting me by buying me a coffee. :)

alopatindev/resume

2023-09-10T01:45:26Z

My skills list document sources

LinkedInLearning/top-5-things-to-know-in-advanced-sql-4403486

2023-09-10T02:07:18Z

This is a code repository for the LinkedIn Learning course Top 5 Things to Know in Advanced SQL.

Thinkgamer/Machine-Learning-With-Python

2023-09-10T02:07:18Z

此项目是我在学习《机器学习实战》这本书时的代码记录情况，用python实现，当然也会包括一些其他的机器学习算法，使用Python实现

skilld-labs/Drupal-Umami-SDC-Dynamic-Storybook

2023-09-10T02:07:24Z

Rendering SDC in storybook with custom indexing

nodejs/corepack

2023-09-10T01:46:00Z

Zero-runtime-dependency package acting as bridge between Node projects and their package managers

corepack

Corepack is a zero-runtime-dependency Node.js script that acts as a bridge between Node.js projects and the package managers they are intended to be used with during development. In practical terms, Corepack lets you use Yarn, npm, and pnpm without having to install them.

How to Install

Default Installs

Corepack is distributed by default with all recent Node.js versions. Run corepack enable to install the required Yarn and pnpm binaries on your path.

Manual Installs

Install Corepack using npm

First uninstall your global Yarn and pnpm binaries (just leave npm). In general, you'd do this by running the following command:

npm uninstall -g yarn pnpm

# That should be enough, but if you installed Yarn without going through npm it might
# be more tedious - for example, you might need to run `brew uninstall yarn` as well.

Then install Corepack:

npm install -g corepack

We do acknowledge the irony and overhead of using npm to install Corepack, which is at least part of why the preferred option is to use the Corepack version that is distributed along with Node.js itself.

Install Corepack from source

See CONTRIBUTING.md.

Usage

When Building Packages

Just use your package managers as you usually would. Run yarn install in Yarn projects, pnpm install in pnpm projects, and npm in npm projects. Corepack will catch these calls, and depending on the situation:

If the local project is configured for the package manager you're using, Corepack will silently download and cache the latest compatible version.
If the local project is configured for a different package manager, Corepack will request you to run the command again using the right package manager - thus avoiding corruptions of your install artifacts.
If the local project isn't configured for any package manager, Corepack will assume that you know what you're doing, and will use whatever package manager version has been pinned as "known good release". Check the relevant section for more details.

When Authoring Packages

Set your package's manager with the packageManager field in package.json:

{
  "packageManager": "yarn@3.2.3+sha224.953c8233f7a92884eee2de69a1b92d1f2ec1655e66d08071ba9a02fa"
}

Here, yarn is the name of the package manager, specified at version 3.2.3, along with the SHA-224 hash of this version for validation. packageManager@x.y.z is required. The hash is optional but strongly recommended as a security practice. Permitted values for the package manager are yarn, npm, and pnpm.

Known Good Releases

When running Corepack within projects that don't list a supported package manager, it will default to a set of Known Good Releases. In a way, you can compare this to Node.js, where each version ships with a specific version of npm.

If there is no Known Good Release for the requested package manager, Corepack looks up the npm registry for the latest available version and cache it for future use.

The Known Good Releases can be updated system-wide using corepack install -g.

Offline Workflow

The utility commands detailed in the next section.

Either you can use the network while building your container image, in which case you'll simply run corepack pack to make sure that your image includes the Last Known Good release for the specified package manager.
- If you want to have all Last Known Good releases for all package managers, just use the --all flag which will do just that.
Or you're publishing your project to a system where the network is unavailable, in which case you'll preemptively generate a package manager archive from your local computer (using corepack pack -o) before storing it somewhere your container will be able to access (for example within your repository). After that it'll just be a matter of running corepack install -g --cache-only <path/to/corepack.tgz> to setup the cache.

Utility Commands

`corepack <binary name>[@<version>] [... args]`

This meta-command runs the specified package manager in the local folder. You can use it to force an install to run with a given version, which can be useful when looking for regressions.

Note that those commands still check whether the local project is configured for the given package manager (ie you won't be able to run corepack yarn install on a project where the packageManager field references pnpm).

`corepack enable [... name]`

Option	Description
`--install-directory`	Add the shims to the specified location

This command will detect where Corepack is installed and will create shims next to it for each of the specified package managers (or all of them if the command is called without parameters). Note that the npm shims will not be installed unless explicitly requested, as npm is currently distributed with Node.js through other means.

If the file system where the corepack binary is located is read-only, this command will fail. A workaround is to add the binaries as alias in your shell configuration file (e.g. in ~/.bash_aliases):

alias yarn="corepack yarn"
alias yarnpkg="corepack yarnpkg"
alias pnpm="corepack pnpm"
alias pnpx="corepack pnpx"
alias npm="corepack npm"
alias npx="corepack npx"

On Windows PowerShell, you can add functions using the $PROFILE automatic variable:

echo "function yarn { corepack yarn `$args }" >> $PROFILE
echo "function yarnpkg { corepack yarnpkg `$args }" >> $PROFILE
echo "function pnpm { corepack pnpm `$args }" >> $PROFILE
echo "function pnpx { corepack pnpx `$args }" >> $PROFILE
echo "function npm { corepack npm `$args }" >> $PROFILE
echo "function npx { corepack npx `$args }" >> $PROFILE

`corepack disable [... name]`

Option	Description
`--install-directory`	Remove the shims to the specified location

This command will detect where Node.js is installed and will remove the shims from there.

`corepack install`

Download and install the package manager configured in the local project. This command doesn't change the global version used when running the package manager from outside the project (use the `-g,--global` flag if you wish to do this).

`corepack install <-g,--global> [--all] [... name@version]`

Option	Description
`--all`	Install all Last Known Good releases

Install the selected package managers and install them on the system.

Package managers thus installed will be configured as the new default when calling their respective binaries outside of projects defining the packageManager field.

`corepack pack [--all] [... name@version]`

Option	Description
`--all`	Pack all Last Known Good releases
`--json`	Print the output folder rather than logs
`-o,--output`	Path where to generate the archive

Download the selected package managers and store them inside a tarball suitable for use with corepack install -g.

`corepack use <name@version>`

When run, this command will retrieve the latest release matching the provided descriptor, assign it to the project's package.json file, and automatically perform an install.

`corepack up`

Retrieve the latest available version for the current major release line of the package manager used in the local project, and update the project to use it.

Unlike corepack use this command doesn't take a package manager name nor a version range, as it will always select the latest available version from the same major line. Should you need to upgrade to a new major, use an explicit corepack use {name}@latest call.

Environment Variables

COREPACK_DEFAULT_TO_LATEST can be set to 0 in order to instruct Corepack not to lookup on the remote registry for the latest version of the selected package manager.
COREPACK_ENABLE_NETWORK can be set to 0 to prevent Corepack from accessing the network (in which case you'll be responsible for hydrating the package manager versions that will be required for the projects you'll run, using corepack install -g --cache-only).
COREPACK_ENABLE_STRICT can be set to 0 to prevent Corepack from throwing error if the package manager does not correspond to the one defined for the current project. This means that if a user is using the package manager specified in the current project, it will use the version specified by the project's packageManager field. But if the user is using other package manager different from the one specified for the current project, it will use the system-wide package manager version.
COREPACK_ENABLE_PROJECT_SPEC can be set to 0 to prevent Corepack from checking if the package manager corresponds to the one defined for the current project. This means that it will always use the system-wide package manager regardless of what is being specified in the project's packageManager field.
COREPACK_HOME can be set in order to define where Corepack should install the package managers. By default it is set to %LOCALAPPDATA%\node\corepack on Windows, and to $HOME/.cache/node/corepack everywhere else.
COREPACK_ROOT has no functional impact on Corepack itself; it's automatically being set in your environment by Corepack when it shells out to the underlying package managers, so that they can feature-detect its presence (useful for commands like yarn init).
COREPACK_NPM_REGISTRY sets the registry base url used when retrieving package managers from npm. Default value is https://registry.npmjs.org
COREPACK_NPM_TOKEN sets a Bearer token authorization header when connecting to a npm type registry.
COREPACK_NPM_USERNAME and COREPACK_NPM_PASSWORD to set a Basic authorization header when connecting to a npm type registry. Note that both environment variables are required and as plain text. If you want to send an empty password, explicitly set COREPACK_NPM_PASSWORD to an empty string.
HTTP_PROXY, HTTPS_PROXY, and NO_PROXY are supported through node-proxy-agent.

Troubleshooting

Networking

There are a wide variety of networking issues that can occur while running corepack commands. Things to check:

Make sure your network connection is active.
Make sure the host for your request can be resolved by your DNS; try using curl [URL] (ipv4) and curl -6 [URL] (ipv6) from your shell.
Check your proxy settings (see Environment Variables).

Contributing

See CONTRIBUTING.md.

Design

See DESIGN.md.

License (MIT)

See LICENSE.md.

langchain-ai/langchainjs

2023-09-10T01:46:00Z

Generate comic panels using a LLM + SDXL. Powered by Hugging Face 🤗

title: AI Comic Factory emoji: 👩‍🎨 colorFrom: red colorTo: yellow sdk: docker pinned: true app_port: 3000

AI Comic Factory

Running the project at home

First, I would like to highlight that everything is open-source (see here, here, here, here).

However the project isn't a monolithic Space that can be duplicated and ran immediately: it requires various components to run for the frontend, backend, LLM, SDXL etc.

If you try to duplicate the project, you will see it requires some variables:

LLM_ENGINE: can be either "INFERENCE_API" or "INFERENCE_ENDPOINT"
HF_API_TOKEN: necessary if you decide to use an inference api model or a custom inference endpoint
HF_INFERENCE_ENDPOINT_URL: necessary if you decide to use a custom inference endpoint
RENDERING_ENGINE: can only be "VIDEOCHAIN" for now, unless you code your custom solution
VIDEOCHAIN_API_URL: url to the VideoChain API server
VIDEOCHAIN_API_TOKEN: secret token to access the VideoChain API server

Please read the .env default config file for more informations. To customise a variable locally, you should create a .env.local (do not commit this file as it will contain your secrets).

-> If you intend to run it with local, cloud-hosted and/or proprietary models you are going to need to code 👨‍💻.

The LLM API (Large Language Model)

Currently the AI Comic Factory uses Llama-2 70b through an Inference Endpoint.

You have three options:

Option 1: Use an Inference API model

This is a new option added recently, where you can use one of the models from the Hugging Face Hub. By default we suggest to use CodeLlama 34b as it will provide better results than the 7b model.

To activate it, create a .env.local configuration file:

LLM_ENGINE="INFERENCE_API"

HF_API_TOKEN="Your Hugging Face token"

# codellama/CodeLlama-7b-hf" is used by default, but you can change this
# note: You should use a model able to generate JSON responses,
# so it is storngly suggested to use at least the 34b model
HF_INFERENCE_API_MODEL="codellama/CodeLlama-7b-hf"

Option 2: Use an Inference Endpoint URL

If you would like to run the AI Comic Factory on a private LLM running on the Hugging Face Inference Endpoint service, create a .env.local configuration file:

LLM_ENGINE="INFERENCE_ENDPOINT"

HF_API_TOKEN="Your Hugging Face token"

HF_INFERENCE_ENDPOINT_URL="path to your inference endpoint url"

To run this kind of LLM locally, you can use TGI (Please read this post for more information about the licensing).

Option 3: Fork and modify the code to use a different LLM system

Another option could be to disable the LLM completely and replace it with another LLM protocol and/or provider (eg. OpenAI, Replicate), or a human-generated story instead (by returning mock or static data).

Notes

It is possible that I modify the AI Comic Factory to make it easier in the future (eg. add support for OpenAI or Replicate)

The Rendering API

This API is used to generate the panel images. This is an API I created for my various projects at Hugging Face.

I haven't written documentation for it yet, but basically it is "just a wrapper ™" around other existing APIs:

The hysts/SD-XL Space by @hysts
And other APIs for making videos, adding audio etc.. but you won't need them for the AI Comic Factory

Option 1: Deploy VideoChain yourself

You will have to clone the source-code

Unfortunately, I haven't had the time to write the documentation for VideoChain yet. (When I do I will update this document to point to the VideoChain's README)

Option 2: Use another SDXL API

If you fork the project you will be able to modify the code to use the Stable Diffusion technology of your choice (local, open-source, your custom HF Space etc)

Notes

It is possible that I modify the AI Comic Factory to make it easier in the future (eg. add support for Replicate)

iwae/easyMenu

2023-09-10T01:46:00Z

AI agent and prompt chaining IDE and library

Rivet

Download | Website | Documentation

Rivet, the IDE for creating complex AI agents and prompt chaining, and embedding it in your application.

Report Bug · Request Feature · Disussions · Chat

Have a question? Need some help? Check out the Rivet Discord server!

About Rivet

Rivet Application

Rivet is a desktop application for creating complex AI agents and prompt chaining, and embedding it in your application.

Rivet currently has LLM support for:

Rivet has embedding/vector database support for:

Rivet also supports these additional integrations:

Audio Transcription from AssemblyAI

For more information on how to use the application and all of its capabilities, see the documentation!

Rivet Core

Rivet core is a TypeScript library for running graphs created in Rivet. It is used by the Rivet application, but can also be used in your own applications, so that Rivet can call into your own application's code, and your application can call into Rivet graphs.

For more information on using Rivet Core, see the Rivet Integration Getting Started page and the related API documentation.

Rivet core is available on NPM as @ironclad/rivet-core. Rivet node is available as @ironclad/rivet-node. Documentation for each is available on the Rivet website.

Getting Started

Prebuilt Binaries

Latest downloads

All Releases

Check out the releases page for all available releases.

Running from Source

See CONTRIBUTING.md for information on building and running Rivet from source.

Contributing

All types of contributions are welcome - from code to documentation, bug reports, user experience feedback, and new feature suggestions!

Take a moment to read through the CONTRIBUTING.md file for help with setting up your development environment, and how to get started contributing to Rivet.

We use the All Contributors bot to recognize all our contributors, so every contribution is acknowledged. See the Contributors section below for everyone!

Code of Conduct

The Rivet project is welcome to all contributors, and as such, we have a Code of Conduct that all contributors must follow.

Troubleshooting

If you have run into any issues while running the Rivet application, or when integrating it into your code, please check the Issues page for any existing issues, and if you can't find any, please open a new issue!

If you have any other questions on using Rivet, or have any other ideas, feel free to open a discussion!

Contributors ✨

Thanks goes to these wonderful people (emoji key):

_{Andy Brenneke} 💻 🤔 🔬 ⚠️ 🔧 👀 💬 🧑‍🏫	_{Cai GoGwilt} 💻 💼 🤔 🚧 📣 👀 🐛	_{Teddy Coleman} 💻 🐛	_{Todd Berman} 💻 🐛	_{Andrew Rothwell} ✅ 📖	_{Dominic Cooney} 🤔 🐛	_{Ankur Goyal} 📖 💻
_{Scott Bessler} 💡 💻 🐛	_{Brandon Hong} 💻 🐛	_{Niels Swimberghe} 💻	_{Terence C} 💻	_{Thomas Brodahl} 🎨	_{Justin Kwok} 🐛	_{Zhang Yi Jiang} 📖
_{Michael Kuc} 💻	_{Ikko Eltociear Ashimine} 📖

This project follows the all-contributors specification. Contributions of any kind welcome!

AntonioErdeljac/next13-discord-clone

2023-09-10T02:07:30Z

Fullstack Discord Clone: Next.js 13, React, Socket.io, Prisma, Tailwind, MySQL | Full Course 2023

This is a repository for Fullstack Discord Clone: Next.js 13, React, Socket.io, Prisma, Tailwind, MySQL | Full Course 2023

VIDEO TUTORIAL

Features:

Real-time messaging using Socket.io
Send attachments as messages using UploadThing
Delete & Edit messages in real time for all users
Create Text, Audio and Video call Channels
1:1 conversation between members
1:1 video calls between members
Member management (Kick, Role change Guest / Moderator)
Unique invite link generation & full working invite system
Infinite loading for messages in batches of 10 (tanstack/query)
Server creation and customization
Beautiful UI using TailwindCSS and ShadcnUI
Full responsivity and mobile UI
Light / Dark mode
Websocket fallback: Polling with alerts
ORM using Prisma
MySQL database using Planetscale
Authentication with Clerk

Prerequisites

Node version 18.x.x

Cloning the repository

git clone https://github.com/AntonioErdeljac/next13-discord-clone.git

Install packages

npm i

Setup .env file

NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=
CLERK_SECRET_KEY=
NEXT_PUBLIC_CLERK_SIGN_IN_URL=
NEXT_PUBLIC_CLERK_SIGN_UP_URL=
NEXT_PUBLIC_CLERK_AFTER_SIGN_IN_URL=
NEXT_PUBLIC_CLERK_AFTER_SIGN_UP_URL=


DATABASE_URL=

UPLOADTHING_SECRET=
UPLOADTHING_APP_ID=

LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
NEXT_PUBLIC_LIVEKIT_URL=

Setup Prisma

Add MySQL Database (I used PlanetScale)

npx prisma generate
npx prisma db push

Start the app

npm run dev

Available commands

Running commands with npm npm run [command]

command	description
`dev`	Starts a development instance of the app

withastro/starlight

2023-09-10T02:07:30Z

🌟 Build beautiful, accessible, high-performance documentation websites with Astro

Starlight

Starlight is a documentation website framework for Astro.

Documentation

Read the Starlight docs (they’re built with Starlight!)

Support

Get help in the Astro Discord. Post questions in our #support forum with the “starlight” tag, or visit our dedicated #starlight channel to discuss current development and more!

You can also submit bug reports and feature requests as GitHub issues.

Contributing

Join us as a Starlight contributor! These links will help you get started:

License

MIT

steven-tey/novel

2023-09-10T02:07:30Z

Notion-style WYSIWYG editor with AI-powered autocompletion.

Novel

An open-source Notion-style WYSIWYG editor with AI-powered autocompletions.

Introduction · Installation · Deploy Your Own · Setting Up Locally · Tech Stack · Contributing · License

Introduction

Novel is a Notion-style WYSIWYG editor with AI-powered autocompletions.

https://github.com/steven-tey/novel/assets/28986134/2099877f-4f2b-4b1c-8782-5d803d63be5c

Installation

To use Novel in a project, you can run the following command to install the novel NPM package:

npm i novel

Then, you can use it in your code like this:

import { Editor } from "novel";

export default function App() {
  return <Editor />;
}

The Editor is a React component that takes in the following props:

Prop	Type	Description	Default
`completionApi`	`string`	The API route to use for the OpenAI completion API.	`/api/generate`
`className`	`string`	Editor container classname.	`"relative min-h-[500px] w-full max-w-screen-lg border-stone-200 bg-white sm:mb-[calc(20vh)] sm:rounded-lg sm:border sm:shadow-lg"`
`defaultValue`	`JSONContent` or `string`	The default value to use for the editor.	`defaultEditorContent`
`extensions`	`Extension[]`	A list of extensions to use for the editor, in addition to the default Novel extensions.	`[]`
`editorProps`	`EditorProps`	Props to pass to the underlying Tiptap editor, in addition to the default Novel editor props.	`{}`
`onUpdate`	`(editor?: Editor) => void`	A callback function that is called whenever the editor is updated.	`() => {}`
`onDebouncedUpdate`	`(editor?: Editor) => void`	A callback function that is called whenever the editor is updated, but only after the defined debounce duration.	`() => {}`
`debounceDuration`	`number`	The duration (in milliseconds) to debounce the `onDebouncedUpdate` callback.	`750`
`storageKey`	`string`	The key to use for storing the editor's value in local storage.	`novel__content`

Note: Make sure to define an API endpoint that matches the completionApi prop (default is /api/generate). This is needed for the AI autocompletions to work. Here's an example: https://github.com/steven-tey/novel/blob/main/apps/web/app/api/generate/route.ts

Here's an example application: https://github.com/steven-tey/novella

Deploy Your Own

You can deploy your own version of Novel to Vercel with one click:

Setting Up Locally

To set up Novel locally, you'll need to clone the repository and set up the following environment variables:

OPENAI_API_KEY – your OpenAI API key (you can get one here)
BLOB_READ_WRITE_TOKEN – your Vercel Blob read/write token (currently still in beta, but feel free to sign up on this form for access)

If you've deployed this to Vercel, you can also use vc env pull to pull the environment variables from your Vercel project.

To run the app locally, you can run the following commands:

pnpm i
pnpm build
pnpm dev

Cross-framework support

While Novel is built for React, we also have a few community-maintained packages for non-React frameworks:

Svelte: https://novel.sh/svelte
Vue: https://novel.sh/vue

VSCode Extension

Thanks to @bennykok, Novel also has a VSCode Extension: https://novel.sh/vscode

https://github.com/steven-tey/novel/assets/28986134/58ebf7e3-cdb3-43df-878b-119e304f7373

Tech Stack

Novel is built on the following stack:

Next.js – framework
Tiptap – text editor
OpenAI - AI completions
Vercel AI SDK – AI library
Vercel – deployments
TailwindCSS – styles
Cal Sans – font

Contributing

Here's how you can contribute:

Open an issue if you believe you've encountered a bug.
Make a pull request to add new features/make quality-of-life improvements/fix bugs.

Repo Activity

License

Licensed under the Apache-2.0 license.

2TimesMeta/Javdb-Top250

2023-09-10T01:25:41Z

收集和梳理垂直领域的开源模型、数据集及评测基准。

okcar-os/android

2023-09-10T01:25:41Z

modularml/mojo

2023-09-10T01:47:40Z

The Mojo Programming Language

SimplifyJobs/New-Grad-Positions

2023-09-10T01:47:40Z

A collection of New Grad full time roles in SWE, Quant, and PM.

luban-agi/Awesome-AIGC-Tutorials

2023-09-10T01:47:40Z

Curated tutorials and resources for Large Language Models, AI Painting, and more.

nand2mario/usb_hid_host

2023-09-10T02:07:54Z

A compact USB HID host FPGA core supporting keyboards, mice and gamepads.

xhacker/PONG

2023-09-10T02:07:54Z

PONG game on Nexys3, in Verilog.

chipsalliance/systemverilog-plugin

2023-09-10T02:07:54Z

SystemVerilog support for Yosys

drodov/Computer-Architecture

2023-09-10T01:46:20Z

Computer Architecture

ethanmys/fifo-and-test-bench-

2023-09-10T02:07:56Z

Using VHDL to program a virtual fifo with memory, and test bench for verification

bhargav/dotfiles

2023-09-10T01:46:26Z

My (almost) up to date dotfiles.

kristijanhusak/vim-dadbod-ui

2023-09-10T02:08:02Z

Simple UI for https://github.com/tpope/vim-dadbod

Shougo/dpp.vim

2023-09-10T02:08:02Z

Dark powered plugin manager for Vim/neovim

python-mode/python-mode

2023-09-10T02:08:02Z

Vim python-mode. PyLint, Rope, Pydoc, breakpoints from box.

yifanjiang/lo_ooxml_wintool

2023-09-10T02:08:06Z

windows tool for libreoffice ooxml testing, mostly for generating reference documents automatically from a single sample file.

dulaiduwang003/TIME-SEA-chatgpt

2023-09-10T01:46:33Z

基于SpringBoot3开发的Ai平台含双端网页以及小程序包含chatgpt newbing claude , 双端数据同步支持自定义预设词,功能板块定义,各类绘图 web兼容手机展示

radix-vue/shadcn-vue

2023-09-10T01:46:33Z

Websites for Nekogram.

nuxt/ui

2023-09-10T02:08:09Z

Fully styled and customizable components for Nuxt.

gaozhangmin/aliyunpan

2023-09-10T02:08:09Z

小白羊网盘 - Powered by 阿里云盘。

ghostsecurity/reaper

2023-09-10T02:08:09Z

💀 Don't fear the Reaper 👻

cardonabits/haxo-hw

2023-09-11T01:44:48Z

Haxophone, an electronic musical instrument that resembles a saxophone

What is a haxophone

A haxophone is an electronic musical instrument that resembles a saxophone. It is designed to be an inexpensive and fully customizable travel saxophone. You will never get the same expressiveness or depth of sound of a real saxophone. But with a haxophone you should be able to play alone or with friends wherever you might be. The mechanical keys and unique ergonomics give it a different kind of satisfying feel.

https://user-images.githubusercontent.com/676181/133919454-f654cb48-cca9-4b6e-bcb5-27de6ea2deb8.mp4

The design is fully open, software and hardware. This means that you can make it your own, from changing the type of key switches to making your own note and instrument mappings.

Project Goals

Goals

Build a travel saxophone that is fun and satisfying to play.
Build an instrument with the same fingering as a saxophone so that muscle memory developed on the haxophone is directly transferrable to your main horn.
Keep the cost low, below $100 USD.
Easy to manufacture.
Keep the design simple, sturdy and repairable to minimize maintenance cost. Using widely available and battle-tested mechanical keys from computer keyboards is one example of this principle in use.
Fully hackable and customizable.

Non-Goals

A haxophone is not designed to be a saxophone replacement.

How does it work?

The haxophone is designed as a HAT (Hardware Attached on Top) for the Raspberry Pi family of single-board computers.

The haxophone HAT is a special type of keyboard laid out in the same way as saxophones are. It also includes a mouthpiece and a pressure sensor to detect breath intensity. The HAT also includes an audio amplifier so it does not rely on the amplifier that is built into the Raspberry Pi 3 and Raspberry Pi 4. Because of that, the HAT will also work with the least expensive of the Pi's, the Raspberry Pi Zero.

The software that runs on the Raspberry Pi is:

haxo-rs: This is the driver that detects key presses, breath and converts it into notes. It's written in Rust, and compiled on the Raspberry Pi into a native application.
fluidsynth: This is the synthesizer software that will convert notes into sounds.

Mechanical Design

In order to keep costs down, the HAT PCB (printed circuit board) is an integral part of the physical structure of the Haxophone. This means that key switches, thumb rests, mouthpiece and the Raspberry Pi are all attached directly to the PCB. The side keys (left palm and right knuckle keys) are mounted on smaller PCBs and attached to the main PCB via right angle metal brackets. This smaller PCBs are detachable pannels from the design. This brings down the manufacturing cost.

For improved ergonomics, a few parts are designed to be 3D printed. These are the mouthpiece neck, the left thumb rest, and the right thumb rest, that also incorporates a strap hook. The source 3D models for those parts are included in the repository.

Electrical Design

The haxophone hat is comprised of three subsystems: a keyboard, a pressure sensor and an audio amplifier.

Keyboard

The keyboard is organized as a 3x8 matrix. The diagram below shows the mapping from saxophone keys to keyboard column/row values.

Note that due to PCB size constraints, a design decision was made to not include a high F# key. This key is not present in all saxophones, as that note can be played using alternate fingerings with other keys. The most common mappings of the high F# are already configured on the haxophone. If you are used to a less common configuration, the software is easy to extend.

Also note that at the moment the matrix is not completely used: keys (col 0, row 7) and (col 2, row 3) are unused.

Audio Amplifier

The haxophone HAT includes an audio amplifier and a 3.5mm mini-jack. This allows connecting the haxophone to headphones or to an external amplifier.

Pinout

The diagram below shows how the HAT connects to the Raspberry Pi. The keyboard matrix uses discrete GPIOs, the pressure sensor uses I2C and the audio amplifier, I2S.

You can refer to the diagram below from Raspberry Pi Spy if you need to locate those pins on the Raspberry Pi header.

Tools/Stack

The circuit board is designed with awesome Kicad.

All 3D models are designed with Freecad, the open source 3D parametric modeler.

The code is written in Rust just for the pleasure and reliability of it.

Software

See haxo-rs for details about the software.

Manufacturing

The circuit boards were manufactured by JLPCB, who also assembled the surface mount parts.

Bill of Materials

The table below shows the main parts required to build a full Haxophone. Prices are rough estimates based on purchases made for prototypes. Shipping costs or taxes not included.

Qty	Part Description	Manufacturer	MPN	Supplier	Estimated cost USD (at quantity 5)
1	Partially-assembled Haxo HAT PCB (small SMD parts populated)	N/A	N/A	Several	$15
1	Differential pressure SMD sensor	NXP	MPXV7007DP	Arrow	~~$10~~ Post-Covid: $24
1	Set of 3D printed parts	N/A	N/A	Several	$4
1	30cm Food grade Silicon Tube ID:3mm/OD:6mm	JUNZHIDA	N/A	Amazon	$2
22	Tactile-feedback 5-pin mechanical switches	Cherry MX	MX-Brown	Banggood	$11
22	Keycaps For Mechanical Keyboards	Many options!			$10
2	4-pin 0.1in Pin Header Con Right Angle	Samtec	TSW-102-25-T-D-RA	Arrow	$2
2	4-pin 0.1in Pin Socket Con Straight	TE Connectivity	215309-2	Arrow	$2
1	40-pin 0.1in Pin Header Connector	TE Connectivity	2-535542-0	Arrow	$2
6	Mounting Bracket L-shape	Keystone	612	Mouser	$1.5
18	4-40 1/4 screws and nuts stainless	Keystone	9900, 4694	Mouser	$1
4	M2/7mm machine screws flat head stainless	Master-Carr	92010A003	Master-Carr	$0.5
3	7/16in aluminum spacers	Fascomp	FC1636-440-A	Mouser	$1
1	Sax Mouthpiece + Reed	N/A	N/A	N/A	Bring your own (but optional)
1	Neck Strap	N/A	N/A	N/A	Bring your own (but optional)
1	Raspberry Pi Zero	Broadcom	Raspberry Pi Zero	Many	$5
1	16 GB SD Card	Many	N/A	Many	$6
~~Total~~					~~$73~~ (Pre-Covid)
Total					$97 (Post-Covid)

Assembly Instructions

See Assembly Guide

TODO list

If you would like to help claim one of the open issues or create a new one for yourself. If you prefer to stay in software land, head out to the software issues.

License and Contributions

See License. We'll gladly accept contributions via Pull Requests. Contributions are accepted under the same License of the project, as captured in github's terms of service.

If you plan to contribute a major feature or improvement, consider posting your proposed design as an issue before implementing it.

Acknowledgements

Ben Gonzales has been a great source of inspiration and encouragement. His midi-Sax is a great project worth checking out.

erasmus-without-paper/ewp-specs-api-iias

2023-09-11T01:44:48Z

Specifications of EWP's Interinstitutional Agreements API.

Interinstitutional Agreements API

Summary

This document describes the Interinstitutional Agreements API. This API allows partners to compare their copies of interinstitutional Erasmus+ mobility agreements with each other, which makes it easier to spot errors. This API is complementary with the Interinstitutional Agreements Approval API where HEIs can approve agreements they exchange via the IIAs API.

Introduction

As part of the EWP project, we have thoroughly discussed many options of how to design the functionality of synchronizing IIAs between different HEIs. The IIAs API is the final result.

Features

It is distributed. Agreements (IIAs) are stored and hosted only by the institutions involved in these agreements. If the institutions use the IIA Manager (part of the Dashboard), their agreements are stored in the Dashboard repository.
All partners are equal. There is no "master" of the agreement. Since all partners of a single IIA are allowed to serve their copies of this IIA, therefore multiple conflicting copies of a single IIA may exist in the network. These conflicts are not resolved by the system itself, but our APIs allow partners to discover such conflicts early (so that they may fix them by themselves).
No history of changes. This API will serve only a single copy of the agreement (with no history of previous versions). This copy should be the copy which is currently in use by the HEI which is serving the API.

Important rules

Electronic versions of IIAs should model their former paper equivalents in a straightforward manner.
Two HEIs sign one or several IIAs with one or several cooperation conditions.
Specifications support IIAs with many cooperation conditions and each node in the network must be able to handle such IIAs to achieve this goal.
Both copies of the same IIA stored in both partners' systems must be presented to each other as exactly one IIA having the same number of corresponding cooperation conditions.
Partners should exchange identifiers of their copies of the IIA to match these copies respectively in their systems.
Regardless of whether a field is mandatory in the API, if it is present in the IIA of one HEI it is highly recommended having it in the matched IIA of the partner HEI.
Providers are free to implement their local solutions to best support the needs of their customers but under the condition that the general principle expressed in the points above is maintained.
Termination of an IIA, which has been mutually approved, should be treated as an agreement modification if any exchanges already took place. Otherwise, it should be treated as termination of the whole agreement.
To modify an IIA which has been mutually approved, HEIs SHOULD take a snapshot of the last approved version to be able to revert to it if they don't conclude a new approved version of the agreement.
An IIA that has not been mutually approved can be deleted by removing it from the EWP network. Such IIA MUST not be present in any of the IIA endpoints and an IIA CNR MUST be sent (see CNR client part section). An IIA can be removed from the EWP network only if it is permanently deleted. Identifiers of the deleted objects MUST NOT be reused for new IIAs.

Fact sheet information

If you compare our IIA XSDs to the official IIA template from the European Commission, for years 2021-20[29], you may notice that many fields seem to be missing in our XSDs. This is because we have decided to include many fields in the Institutions API and the Factsheet API instead. That is why both of the mentioned APIs MUST be implemented to use this API.

Based on the official IIA template from the European Commission for years 2021-20[29], we follow the following rules:

General information that is part of Higher Education Institutions’ profile and made publicly available to students is part of the Institutions API (and - in some cases - Organizational Units API) and the Factsheet API. The general information can be updated without formal approval of the partner.
Data on the terms of agreement that needs to be approved by both partners is part of this API. The approval is done in the IIAs Approval API.

Business requirements and processes

Business requirements and processes document clarifies the requirements for the technical solutions developed under EWP and in the local implementation that should adequately support the business processes related to the approval of IIAs at Higher Education Institutions.

IIA hash calculation

As of IIA version 7 each agreement contains an iia-hash element that replaces the conditions-hash element used in previous versions of this API.

To calculate the new hash an IIA get response XML has to be transformed using the appropriate XSLT template provided:

You can test these transformations using the provided Java class

You may need to find the right XSLT processor for these templates to work. For Java Saxon-HE-9.5.1-8.jar processor works, the previous versions fail. For more details please go to XSLT kit resources.

Security

This version of this API uses standard EWP Authentication and Security, Version 2. Server implementers choose which security methods they support by declaring them in their Manifest API entry.

Endpoints and functionality to be implemented

Server implementers MUST:

Implement the get endpoint.
Implement the index endpoint.
Implement the stats endpoint
Put the URLs of these endpoints in their manifest file, as described in manifest-entry.xsd.

The details on each of these endpoints are described on separate pages of this API specification (use the links above).

Implementers also MUST implement a Notification Sender for Interinstitutional Agreement objects. That means that an EWP host will try to deliver notifications to all Interinstitutional Agreement CNR APIs implemented throughout the EWP Network whenever related agreement objects are updated.

Data model entities involved in the response

IIA
IIA Partner
Cooperation Condition
Coordinator
Person

oreillymedia/HTMLBook

2023-09-10T02:08:46Z

Let's write books in HTML!

HTMLBook

Let’s write books in HTML! HTMLBook is an open, XHTML5-based standard for the authoring and production of both print and digital books. HTMLBook is built on the following premises:

Books are timeless. The basic "book" structure has persisted for hundreds of years and will continue to persist for our lifetimes, be it in digital or print form.
HTML is the markup language of the world for the foreseeable future.
Single-source document processing will remain valuable for the foreseeable future.

As such, HTMLBook can be characterized in the following ways:

HTMLBook is a subset of XHTML5. All HTMLBook is XHTML5, but not all XHTML5 is HTMLBook.
HTMLBook contains no additional elements or attributes outside of the XHTML5 specification.
HTMLBook is semantically tailored to the structure of a book, including more complex content used in technical and reference documents.
HTMLBook is defined with and can be validated against an XML schema.
HTMLBook stylesheets are written in CSS.

Getting Started with HTMLBook

Read the Working Draft of the HTMLBook Specification here:

http://oreillymedia.github.io/HTMLBook/

Get the HTMLBook schema here:

https://github.com/oreillymedia/HTMLBook/tree/master/schema

The main HTMLBook Schema file is htmlbook.xsd, which imports Schemas (via W3C) for MathML and SVG validation.

You can validate HTML documents against htmlbook.xsd on the command line using xmllint:

$ xmllint --noout --schema htmlbook.xsd <your_html_filename>.html

Alternatively, to validate in a GUI XML editor like oXygen, add a xsi:schemaLocation declaration to the root <html> element as follows:

<html xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.w3.org/1999/xhtml ../schema/htmlbook.xsd"
 xmlns="http://www.w3.org/1999/xhtml">

Notes on HTMLBook

The HTMLBook specification is currently in Working Draft status, and will continue to be developed and revised for the foreseeable future.
If you have any bugs or feature requests for HTMLBook, please submit them via the Issues tab in GitHub.
In the coming months, we will be adding additional tools to the HTMLBook project for authoring and producing books with HTMLBook: CSS stylesheets, scripts for generating EPUB/Mobi, etc.
If you would like to contribute to the HTMLBook project, send us a pull request!

vogellacompany/codeexamples-java

2023-09-10T02:08:46Z

Java and Eclipse code tutorial examples from vogella.com

vogella code examples

Example code for the Eclipse and Java tutorials from the vogella website.

Also used in the Eclipse training sessions and Java training sessions and the vogella book series.

For the Android examples please see the Android examples repository.

License

Eclipse Public License - v 1.0 is used.

fulcrum-so/ziggy-pydust

2023-09-11T01:45:04Z

A toolkit for building Python extensions in Zig.