GitHub 1C Enterprise Daily Trending

Pr-Mex/vanessa-automation

2024-12-01T01:51:14Z

BDD в 1С

SAP-samples/abap-platform-rap-workshops

2024-12-01T01:51:21Z

This repository contains materials for workshops about the ABAP RESTful Application Programming Model (RAP).

hengyoush/kyanos

2024-11-30T01:28:45Z

The open-source alert management and AIOps platform

Single pane of glass, alert deduplication, enrichment, filtering and correlation, bi-directional integrations, workflows, dashboards.

Docs · Try it out · Report Bug · Book a Demo · Website

🔍 Single pane of glass - Best-in-class customizable UI for all your alerts and incidents
🛠️ Swiss Army Knife for alerts - Deduplication, correlation, filtering and enrichment
🔄 Deep integrations - Bi-directional syncs with monitoring tools, customizable workflows
⚡ Automation - GitHub Actions for your monitoring tools
🤖 AIOps 2.0 - AI-powered correlation and summarization

See full platform documentation.

Supported Integrations

View the full list in our documentation

Missing a provider? Submit a new provider request and we'll add it quickly!

Observability Tools

AppDynamics	Axiom	Azure Monitoring	Centreon	Checkmk	Cilium
CloudWatch	Coralogix	Datadog	Dynatrace	Elastic	GCP Monitoring
Grafana	Graylog	Kibana	Netdata	New Relic	OpenObserve
Parseable	Pingdom	Prometheus	Rollbar	Sentry	SignalFX
Site24x7	Splunk	StatusCake	SumoLogic	UptimeKuma	VictoriaMetrics
Zabbix

Databases & Data Warehouses

BigQuery

ClickHouse

MongoDB

MySQL

PostgreSQL

Snowflake

Communication Platforms

Discord	Google Chat	Mailchimp	Mailgun	Mattermost	Ntfy.sh
Pushover	Resend	SendGrid	Slack	SMTP	Teams
Telegram	Twilio

Incident Management

Grafana Incident	Grafana OnCall	Ilert	Incident.io	AWS Incident Manager	OpsGenie
PagerDuty	Pagertree	SINGL4	Squadcast	Zenduty

Ticketing Tools

GitHub	GitLab	Jira	Linear	LinearB	Microsoft Planner
Redmine	ServiceNow	Trello

Container Orchecstration Platforms

Azure AKS

GKE

Kubernetes

OpenShift

Data Enrichment

Bash

OpenAI

Python

QuickChart

SSH

Webhook

Workflows

Keep is GitHub Actions for your monitoring tools.

A Keep Workflow is a declarative YAML file that automates your alert and incident management. Each workflow consists of:

Triggers - What starts the workflow (alerts, incidents, schedule or manual)
Steps - Read or fetch data (enrichment, context)
Actions - Execute operations (update tickets, send notifications, restart servers)

Here's a simple workflow that creates a Jira ticket for every critical alert from sentry for payments and api services.

For more workflows, see here.

workflow:
  id: sentry-alerts
  description: create ticket alerts for critical alerts from sentry
  triggers:
    - type: alert
      # customize the filter to run only on critical alert from sentry
      filters:
        - key: source
          value: sentry
        - key: severity
          value: critical
        # regex to match specific services
        - key: service
          value: r"(payments|ftp)"
  actions:
    - name: send-slack-message-team-payments
      # if the alert is on the payments service, slack the payments team
      if: "'{{ alert.service }}' == 'payments'"
      provider:
        type: slack
        # control which Slack configuration you want to use
        config: " {{ providers.team-payments-slack }} "
        # customize the alert message with context from {{ alert }} or any other {{ step }}
        with:
          message: |
            "A new alert from Sentry: Alert: {{ alert.name }} - {{ alert.description }}
            {{ alert}}"
    - name: create-jira-ticket-oncall-board
      # control the workflow flow with "if" and "foreach" statements
      if: "'{{ alert.service }}' == 'ftp' and not '{{ alert.ticket_id }}'"
      provider:
        type: jira
        config: " {{ providers.jira }} "
        with:
          board_name: "Oncall Board"
          custom_fields:
            customfield_10201: "Critical"
          issuetype: "Task"
          # customize the summary
          summary: "{{ alert.name }} - {{ alert.description }} (created by Keep)"
          description: |
            "This ticket was created by Keep.
            Please check the alert details below:
            {code:json} {{ alert }} {code}"
          # enrich the alerts with more context. from now on, the alert will be assigned with the ticket id, type and url
          enrich_alert:
            - key: ticket_type
              value: jira
            - key: ticket_id
              value: results.issue.key
            - key: ticket_url
              value: results.ticket_url

Enterprise Ready

Developer First - Modern REST APIs, native SDKs, and comprehensive documentation for seamless integration
Enterprise Security - Full authentication support (SSO, SAML, OIDC, LDAP) with granular access control (RBAC, ABAC) and team management
Flexible Deployment - Deploy on-premises or in air-gapped environments with cloud-agnostic architecture
Production Scale - High availability, performance-tested infrastructure supporting horizontal scaling for enterprise workloads

Getting Started

Need help? Can't find your environment listed? Reach out on Slack and we'll help you quickly.

Keep can run in various environments and configurations. The easiest way to start is with Keep's Docker Compose.

Running Keep locally.
Running Keep on Kubernetes.
Running Keep with Docker.
Running Keep on AWS ECS.
Running Keep on OpenShift.

🫵 Keepers

Top Contributors

A special thanks to our top contributors who help us make Keep great. You are more than awesome!

Want to become a top contributor? Join our Slack and DM Tal, Shahar, or Furkan.

Contributors

Thank you for contributing and continuously making Keep better, you're awesome 🫶

yamadashy/repomix

2024-11-30T01:28:45Z

📦 Repomix (formerly Repopack) is a powerful tool that packs your entire repository into a single, AI-friendly file. Perfect for when you need to feed your codebase to Large Language Models (LLMs) or other AI tools like Claude, ChatGPT, and Gemini.

📦 Repomix (formerly Repopack)

Repomix is a powerful tool that packs your entire repository into a single, AI-friendly file.
It is perfect for when you need to feed your codebase to Large Language Models (LLMs) or other AI tools like Claude, ChatGPT, and Gemini.

📢 Important Notice: Project Renamed to Repomix

[!NOTE] Due to legal considerations, this project has been renamed from "Repopack" to "Repomix". Only the name is changing; Repomix all functionality and maintainer (@yamadashy) remain the same. We are committed to ensuring a smooth transition for all users.

Migration Guide

To continue using the tool, simply install the new package:

# Install new package
npm install -g repomix

# Or use directly with npx
npx repomix

Optionally, you can also uninstall the old package:

npm uninstall -g repopack

Configuration Files

Your existing configuration files (repopack.config.json and .repopackignore) will continue to work during the transition period.
repomix will automatically detect these files and offer to migrate them to the new format (repomix.config.json and .repomixignore).

Timeline

Current: Transition period begins
December 1st, 2024: Ownership of the repopack npm package will be transferred to another party. The repomix package will continue to be maintained as usual

We appreciate your understanding and cooperation during this transition.

🌟 Features

AI-Optimized: Formats your codebase in a way that's easy for AI to understand and process.
Token Counting: Provides token counts for each file and the entire repository, useful for LLM context limits.
Simple to Use: You need just one command to pack your entire repository.
Customizable: Easily configure what to include or exclude.
Git-Aware: Automatically respects your .gitignore files.
Security-Focused: Incorporates Secretlint for robust security checks to detect and prevent inclusion of sensitive information.

🚀 Quick Start

You can try Repomix instantly in your project directory without installation:

npx repomix

Or install globally for repeated use:

# Install using npm
npm install -g repomix

# Alternatively using yarn
yarn global add repomix

# Alternatively using Homebrew (macOS)
brew install repomix

# Then run in any project directory
repomix

That's it! Repomix will generate a repomix-output.txt file in your current directory, containing your entire repository in an AI-friendly format.

📊 Usage

To pack your entire repository:

repomix

To pack a specific directory:

repomix path/to/directory

To pack specific files or directories using glob patterns:

repomix --include "src/**/*.ts,**/*.md"

To exclude specific files or directories:

repomix --ignore "**/*.log,tmp/"

To pack a remote repository:

repomix --remote https://github.com/yamadashy/repomix

# You can also use GitHub shorthand:
repomix --remote yamadashy/repomix

To initialize a new configuration file (repomix.config.json):

repomix --init

Once you have generated the packed file, you can use it with Generative AI tools like Claude, ChatGPT, and Gemini.

Prompt Examples

Once you have generated the packed file with Repomix, you can use it with AI tools like Claude, ChatGPT, and Gemini. Here are some example prompts to get you started:

Code Review and Refactoring

For a comprehensive code review and refactoring suggestions:

This file contains my entire codebase. Please review the overall structure and suggest any improvements or refactoring opportunities, focusing on maintainability and scalability.

Documentation Generation

To generate project documentation:

Based on the codebase in this file, please generate a detailed README.md that includes an overview of the project, its main features, setup instructions, and usage examples.

Test Case Generation

For generating test cases:

Analyze the code in this file and suggest a comprehensive set of unit tests for the main functions and classes. Include edge cases and potential error scenarios.

Code Quality Assessment

Evaluate code quality and adherence to best practices:

Review the codebase for adherence to coding best practices and industry standards. Identify areas where the code could be improved in terms of readability, maintainability, and efficiency. Suggest specific changes to align the code with best practices.

Library Overview

Get a high-level understanding of the library

This file contains the entire codebase of library. Please provide a comprehensive overview of the library, including its main purpose, key features, and overall architecture.

Feel free to modify these prompts based on your specific needs and the capabilities of the AI tool you're using.

Community Discussion

Check out our community discussion where users share:

Which AI tools they're using with Repomix
Effective prompts they've discovered
How Repomix has helped them
Tips and tricks for getting the most out of AI code analysis

Feel free to join the discussion and share your own experiences! Your insights could help others make better use of Repomix.

Output File Format

Repomix generates a single file with clear separators between different parts of your codebase.
To enhance AI comprehension, the output file begins with an AI-oriented explanation, making it easier for AI models to understand the context and structure of the packed repository.

Plain Text Format (default)

This file is a merged representation of the entire codebase, combining all repository files into a single document.

================================================================
File Summary
================================================================
(Metadata and usage AI instructions)

================================================================
Repository Structure
================================================================
src/
  cli/
    cliOutput.ts
    index.ts
  config/
    configLoader.ts

(...remaining directories)

================================================================
Repository Files
================================================================

================
File: src/index.js
================
// File contents here

================
File: src/utils.js
================
// File contents here

(...remaining files)

================================================================
Instruction
================================================================
(Custom instructions from `output.instructionFilePath`)

XML Format

To generate output in XML format, use the --style xml option:

repomix --style xml

The XML format structures the content in a hierarchical manner:

This file is a merged representation of the entire codebase, combining all repository files into a single document.

<file_summary>
(Metadata and usage AI instructions)
</file_summary>

<repository_structure>
src/
  cli/
    cliOutput.ts
    index.ts

(...remaining directories)
</repository_structure>

<repository_files>
<file path="src/index.js">
// File contents here
</file>

(...remaining files)
</repository_files>

<instruction>
(Custom instructions from `output.instructionFilePath`)
</instruction>

For those interested in the potential of XML tags in AI contexts:
https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags

When your prompts involve multiple components like context, instructions, and examples, XML tags can be a game-changer. They help Claude parse your prompts more accurately, leading to higher-quality outputs.

This means that the XML output from Repomix is not just a different format, but potentially a more effective way to feed your codebase into AI systems for analysis, code review, or other tasks.

Markdown Format

To generate output in Markdown format, use the --style markdown option:

repomix --style markdown

The Markdown format structures the content in a hierarchical manner:

This file is a merged representation of the entire codebase, combining all repository files into a single document.

# File Summary
(Metadata and usage AI instructions)

# Repository Structure
```
src/
  cli/
    cliOutput.ts
    index.ts
```
(...remaining directories)

# Repository Files

## File: src/index.js
```
// File contents here
```

(...remaining files)

# Instruction
(Custom instructions from `output.instructionFilePath`)

This format provides a clean, readable structure that is both human-friendly and easily parseable by AI systems.

Command Line Options

-v, --version: Show tool version
-o, --output <file>: Specify the output file name
--include <patterns>: List of include patterns (comma-separated)
-i, --ignore <patterns>: Additional ignore patterns (comma-separated)
-c, --config <path>: Path to a custom config file
--style <style>: Specify the output style (plain, xml, markdown)
--top-files-len <number>: Number of top files to display in the summary
--output-show-line-numbers: Show line numbers in the output
--copy: Additionally copy generated output to system clipboard
--remote <url>: Process a remote Git repository
--verbose: Enable verbose logging

Examples:

repomix -o custom-output.txt
repomix -i "*.log,tmp" -v
repomix -c ./custom-config.json
repomix --style xml
repomix --remote https://github.com/user/repo.git
npx repomix src

Updating Repomix

To update a globally installed Repomix:

# Using npm
npm update -g repomix

# Using yarn
yarn global upgrade repomix

Using npx repomix is generally more convenient as it always uses the latest version.

Remote Repository Processing

Repomix supports processing remote Git repositories without the need for manual cloning. This feature allows you to quickly analyze any public Git repository with a single command.

To process a remote repository, use the --remote option followed by the repository URL:

repomix --remote https://github.com/user/repo.git

You can also use GitHub's shorthand format:

repomix --remote user/repo

⚙️ Configuration

Create a repomix.config.json file in your project root for custom configurations.

repomix --init

Here's an explanation of the configuration options:

Option	Description	Default
`output.filePath`	The name of the output file	`"repomix-output.txt"`
`output.style`	The style of the output (`plain`, `xml`, `markdown`)	`"plain"`
`output.headerText`	Custom text to include in the file header	`null`
`output.instructionFilePath`	Path to a file containing detailed custom instructions	`null`
`output.removeComments`	Whether to remove comments from supported file types	`false`
`output.removeEmptyLines`	Whether to remove empty lines from the output	`false`
`output.showLineNumbers`	Whether to add line numbers to each line in the output	`false`
`output.copyToClipboard`	Whether to copy the output to system clipboard in addition to saving the file	`false`
`output.topFilesLength`	Number of top files to display in the summary. If set to 0, no summary will be displayed	`5`
`output.includeEmptyDirectories`	Whether to include empty directories in the repository structure	`false`
`include`	Patterns of files to include (using glob patterns)	`[]`
`ignore.useGitignore`	Whether to use patterns from the project's `.gitignore` file	`true`
`ignore.useDefaultPatterns`	Whether to use default ignore patterns	`true`
`ignore.customPatterns`	Additional patterns to ignore (using glob patterns)	`[]`
`security.enableSecurityCheck`	Whether to perform security checks on files	`true`

Example configuration:

{
  "output": {
    "filePath": "repomix-output.xml",
    "style": "xml",
    "headerText": "Custom header information for the packed file.",
    "removeComments": false,
    "removeEmptyLines": false,
    "showLineNumbers": false,
    "copyToClipboard": true,
    "topFilesLength": 5,
    "includeEmptyDirectories": false
  },
  "include": ["**/*"],
  "ignore": {
    "useGitignore": true,
    "useDefaultPatterns": true,
    "customPatterns": ["additional-folder", "**/*.log"]
  },
  "security": {
    "enableSecurityCheck": true
  }
}

Global Configuration

To create a global configuration file:

repomix --init --global

The global configuration file will be created in:

Windows: %LOCALAPPDATA%\Repomix\repomix.config.json
macOS/Linux: $XDG_CONFIG_HOME/repomix/repomix.config.json or ~/.config/repomix/repomix.config.json

Note: Local configuration (if present) takes precedence over global configuration.

Include and Ignore

Include Patterns

Repomix now supports specifying files to include using glob patterns. This allows for more flexible and powerful file selection:

Use **/*.js to include all JavaScript files in any directory
Use src/**/* to include all files within the src directory and its subdirectories
Combine multiple patterns like ["src/**/*.js", "**/*.md"] to include JavaScript files in src and all Markdown files

Ignore Patterns

Repomix offers multiple methods to set ignore patterns for excluding specific files or directories during the packing process:

.gitignore: By default, patterns listed in your project's .gitignore file are used. This behavior can be controlled with the ignore.useGitignore setting.
Default patterns: Repomix includes a default list of commonly excluded files and directories (e.g., node_modules, .git, binary files). This feature can be controlled with the ignore.useDefaultPatterns setting. Please see defaultIgnore.ts for more details.
.repomixignore: You can create a .repomixignore file in your project root to define Repomix-specific ignore patterns. This file follows the same format as .gitignore.
Custom patterns: Additional ignore patterns can be specified using the ignore.customPatterns option in the configuration file. You can overwrite this setting with the -i, --ignore command line option.

Priority Order (from highest to lowest):

Custom patterns ignore.customPatterns
.repomixignore
.gitignore (if ignore.useGitignore is true)
Default patterns (if ignore.useDefaultPatterns is true)

This approach allows for flexible file exclusion configuration based on your project's needs. It helps optimize the size of the generated pack file by ensuring the exclusion of security-sensitive files and large binary files, while preventing the leakage of confidential information.

Note: Binary files are not included in the packed output by default, but their paths are listed in the "Repository Structure" section of the output file. This provides a complete overview of the repository structure while keeping the packed file efficient and text-based.

Custom Instruction

The output.instructionFilePath option allows you to specify a separate file containing detailed instructions or context about your project. This allows AI systems to understand the specific context and requirements of your project, potentially leading to more relevant and tailored analysis or suggestions.

Here's an example of how you might use this feature:

Create a file named repomix-instruction.md in your project root:

# Coding Guidelines
- Follow the Airbnb JavaScript Style Guide
- Suggest splitting files into smaller, focused units when appropriate
- Add comments for non-obvious logic. Keep all text in English
- All new features should have corresponding unit tests

# Generate Comprehensive Output
- Include all content without abbreviation, unless specified otherwise
- Optimize for handling large codebases while maintaining output quality

In your repomix.config.json, add the instructionFilePath option:

{
  "output": {
    "instructionFilePath": "repomix-instruction.md",
    // other options...
  }
}

When Repomix generates the output, it will include the contents of repomix-instruction.md in a dedicated section.

Note: The instruction content is appended at the end of the output file. This placement can be particularly effective for AI systems. For those interested in understanding why this might be beneficial, Anthropic provides some insights in their documentation:
https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/long-context-tips

Put long-form data at the top: Place your long documents and inputs (~20K+ tokens) near the top of your prompt, above your query, instructions, and examples. This can significantly improve Claude's performance across all models. Queries at the end can improve response quality by up to 30% in tests, especially with complex, multi-document inputs.

Comment Removal

When output.removeComments is set to true, Repomix will attempt to remove comments from supported file types. This feature can help reduce the size of the output file and focus on the essential code content.

Supported languages include:
HTML, CSS, JavaScript, TypeScript, Vue, Svelte, Python, PHP, Ruby, C, C#, Java, Go, Rust, Swift, Kotlin, Dart, Shell, and YAML.

Note: The comment removal process is conservative to avoid accidentally removing code. In complex cases, some comments might be retained.

🔍 Security Check

Repomix includes a security check feature that uses Secretlint to detect potentially sensitive information in your files. This feature helps you identify possible security risks before sharing your packed repository.

The security check results will be displayed in the CLI output after the packing process is complete. If any suspicious files are detected, you'll see a list of these files along with a warning message.

Example output:

🔍 Security Check:
──────────────────
2 suspicious file(s) detected:
1. src/utils/test.txt
2. tests/utils/secretLintUtils.test.ts

Please review these files for potentially sensitive information.

By default, Repomix's security check feature is enabled. You can disable it by setting security.enableSecurityCheck to false in your configuration file:

{
  "security": {
    "enableSecurityCheck": false
  }
}

🤝 Contribution

We welcome contributions from the community! To get started, please refer to our Contributing Guide.

Contributors

📜 License

This project is licensed under the MIT License.

SimonCropp/Delta

2024-11-30T01:28:45Z

An approach to implementing a 304 Not Modified leveraging SqlServer change tracking

Delta

Delta is an approach to implementing a 304 Not Modified leveraging SqlServer change tracking

The approach uses a last updated timestamp from the database to generate an ETag. All dynamic requests then have that ETag checked/applied.

This approach works well when the frequency of updates is relatively low. In this scenario, the majority of requests will leverage the result in a 304 Not Modified being returned and the browser loading the content its cache.

Effectively consumers will always receive the most current data, while the load on the server remains low.

See Milestones for release notes.

Assumptions

Frequency of updates to data is relatively low compared to reads
Using either SQL Server Change Tracking and/or SQL Server Row Versioning

304 Not Modified Flow

graph TD
    Request
    CalculateEtag[Calculate current ETag<br/>based on timestamp<br/>from web assembly and SQL]
    IfNoneMatch{Has<br/>If-None-Match<br/>header?}
    EtagMatch{Current<br/>Etag matches<br/>If-None-Match?}
    AddETag[Add current ETag<br/>to Response headers]
    304[Respond with<br/>304 Not-Modified]
    Request --> CalculateEtag
    CalculateEtag --> IfNoneMatch
    IfNoneMatch -->|Yes| EtagMatch
    IfNoneMatch -->|No| AddETag
    EtagMatch -->|No| AddETag
    EtagMatch -->|Yes| 304

ETag calculation logic

The ETag is calculated from a combination several parts

AssemblyWriteTime

The last write time of the web entry point assembly

var webAssemblyLocation = Assembly.GetEntryAssembly()!.Location;
AssemblyWriteTime = File.GetLastWriteTime(webAssemblyLocation).Ticks.ToString();

^{snippet source | anchor}

SQL timestamp

A combination of change_tracking_current_version (if tracking is enabled) and @@DBTS (row version timestamp)

declare @changeTracking bigint = change_tracking_current_version();
declare @timeStamp bigint = convert(bigint, @@dbts);

if (@changeTracking is null)
  select cast(@timeStamp as varchar)
else
  select cast(@timeStamp as varchar) + '-' + cast(@changeTracking as varchar)

^{snippet source | anchor}

Suffix

An optional string suffix that is dynamically calculated at runtime based on the current HttpContext.

var app = builder.Build();
app.UseDelta(suffix: httpContext => "MySuffix");

^{snippet source | anchor}

Combining the above

internal static string BuildEtag(string timeStamp, string? suffix)
{
    if (suffix == null)
    {
        return $"\"{AssemblyWriteTime}-{timeStamp}\"";
    }

    return $"\"{AssemblyWriteTime}-{timeStamp}-{suffix}\"";
}

^{snippet source | anchor}

NuGet

Delta is shipped as two nugets:

Delta: Delivers functionality using SqlConnection and SqlTransaction.
Delta.EF: Delivers functionality using SQL Server EF Database Provider.

Only one of the above should be used.

Usage

DB Schema

Ensure SQL Server Change Tracking and/or SQL Server Row Versioning is enabled for all relevant tables.

Example SQL schema:

-- Tables

CREATE TABLE [dbo].[Companies](
	[Id] [uniqueidentifier] NOT NULL,
	[RowVersion] [timestamp] NOT NULL,
	[Content] [nvarchar](max) NULL,
 CONSTRAINT [PK_Companies] PRIMARY KEY CLUSTERED 
(
	[Id] ASC
) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

CREATE TABLE [dbo].[Employees](
	[Id] [uniqueidentifier] NOT NULL,
	[RowVersion] [timestamp] NOT NULL,
	[CompanyId] [uniqueidentifier] NOT NULL,
	[Content] [nvarchar](max) NULL,
	[Age] [int] NOT NULL,
 CONSTRAINT [PK_Employees] PRIMARY KEY CLUSTERED 
(
	[Id] ASC
) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

CREATE NONCLUSTERED INDEX [IX_Employees_CompanyId] ON [dbo].[Employees]
(
	[CompanyId] ASC
) ON [PRIMARY]

^{snippet source | anchor}

Add to WebApplicationBuilder

var builder = WebApplication.CreateBuilder();
builder.Services.AddScoped(_ => new SqlConnection(connectionString));
var app = builder.Build();
app.UseDelta();

^{snippet source | anchor}

Add to a Route Group

To add to a specific Route Group:

app.MapGroup("/group")
    .UseDelta()
    .MapGet("/", () => "Hello Group!");

^{snippet source | anchor}

ShouldExecute

Optionally control what requests Delta is executed on.

var app = builder.Build();
app.UseDelta(
    shouldExecute: httpContext =>
    {
        var path = httpContext.Request.Path.ToString();
        return path.Contains("match");
    });

^{snippet source | anchor}

Custom Connection discovery

By default, Delta uses HttpContext.RequestServices to discover the SqlConnection and SqlTransaction:

static Connection DiscoverConnection(HttpContext httpContext)
{
    var provider = httpContext.RequestServices;
    var connection = provider.GetRequiredService<SqlConnection>();
    var transaction = provider.GetService<SqlTransaction>();
    return new(connection, transaction);
}

^{snippet source | anchor}

To use custom connection discovery:

var application = webApplicationBuilder.Build();
application.UseDelta(
    getConnection: httpContext => httpContext.RequestServices.GetRequiredService<SqlConnection>());

^{snippet source | anchor}

To use custom connection and transaction discovery:

var webApplication = webApplicationBuilder.Build();
webApplication.UseDelta(
    getConnection: httpContext =>
    {
        var provider = httpContext.RequestServices;
        var sqlConnection = provider.GetRequiredService<SqlConnection>();
        var sqlTransaction = provider.GetService<SqlTransaction>();
        return new(sqlConnection, sqlTransaction);
    });

^{snippet source | anchor}

EF Usage

DbContext using RowVersion

Enable row versioning in Entity Framework

public class SampleDbContext(DbContextOptions options) :
    DbContext(options)
{
    public DbSet<Employee> Employees { get; set; } = null!;
    public DbSet<Company> Companies { get; set; } = null!;

    protected override void OnModelCreating(ModelBuilder builder)
    {
        var company = builder.Entity<Company>();
        company.HasKey(_ => _.Id);
        company
            .HasMany(_ => _.Employees)
            .WithOne(_ => _.Company)
            .IsRequired();
        company
            .Property(_ => _.RowVersion)
            .IsRowVersion()
            .HasConversion<byte[]>();

        var employee = builder.Entity<Employee>();
        employee.HasKey(_ => _.Id);
        employee
            .Property(_ => _.RowVersion)
            .IsRowVersion()
            .HasConversion<byte[]>();
    }
}

^{snippet source | anchor}

Add to WebApplicationBuilder

var builder = WebApplication.CreateBuilder();
builder.Services.AddSqlServer<SampleDbContext>(database.ConnectionString);
var app = builder.Build();
app.UseDelta<SampleDbContext>();

^{snippet source | anchor}

Add to a Route Group

To add to a specific Route Group:

app.MapGroup("/group")
    .UseDelta<SampleDbContext>()
    .MapGet("/", () => "Hello Group!");

^{snippet source | anchor}

ShouldExecute

Optionally control what requests Delta is executed on.

var app = builder.Build();
app.UseDelta<SampleDbContext>(
    shouldExecute: httpContext =>
    {
        var path = httpContext.Request.Path.ToString();
        return path.Contains("match");
    });

^{snippet source | anchor}

UseResponseDiagnostics

Response diagnostics is an opt-in feature that includes extra log information in the response headers.

Enable by setting UseResponseDiagnostics to true at startup:

DeltaExtensions.UseResponseDiagnostics = true;

^{snippet source | anchor}

Response diagnostics headers are prefixed with Delta-.

Example Response header when the Request has not If-None-Match header.

Helpers

Utility methods for working with databases using the Delta conventions.

GetLastTimeStamp

For a `SqlConnection`:

var timeStamp = await sqlConnection.GetLastTimeStamp();

^{snippet source | anchor}

For a `DbContext`:

var timeStamp = await dbContext.GetLastTimeStamp();

^{snippet source | anchor}

GetDatabasesWithTracking

Get a list of all databases with change tracking enabled.

var trackedDatabases = await sqlConnection.GetTrackedDatabases();
foreach (var db in trackedDatabases)
{
    Trace.WriteLine(db);
}

^{snippet source | anchor}

Uses the following SQL:

select d.name
from sys.databases as d inner join
  sys.change_tracking_databases as t on
  t.database_id = d.database_id

^{snippet source | anchor}

GetTrackedTables

Get a list of all tracked tables in database.

var trackedTables = await sqlConnection.GetTrackedTables();
foreach (var db in trackedTables)
{
    Trace.WriteLine(db);
}

^{snippet source | anchor}

Uses the following SQL:

select t.Name
from sys.tables as t left join
  sys.change_tracking_tables as c on t.[object_id] = c.[object_id]
where c.[object_id] is not null

^{snippet source | anchor}

IsTrackingEnabled

Determine if change tracking is enabled for a database.

var isTrackingEnabled = await sqlConnection.IsTrackingEnabled();

^{snippet source | anchor}

Uses the following SQL:

select count(d.name)
from sys.databases as d inner join
  sys.change_tracking_databases as t on
  t.database_id = d.database_id
where d.name = '{database}'

^{snippet source | anchor}

EnableTracking

Enable change tracking for a database.

await sqlConnection.EnableTracking();

^{snippet source | anchor}

Uses the following SQL:

alter database {database}
set change_tracking = on
(
  change_retention = {retentionDays} days,
  auto_cleanup = on
)

^{snippet source | anchor}

DisableTracking

Disable change tracking for a database and all tables within that database.

await sqlConnection.DisableTracking();

^{snippet source | anchor}

Uses the following SQL:

For disabling tracking on a database:

alter database [{database}] set change_tracking = off;

^{snippet source | anchor}

For disabling tracking on tables:

alter table [{table}] disable change_tracking;

^{snippet source | anchor}

SetTrackedTables

Enables change tracking for all tables listed, and disables change tracking for all tables not listed.

await sqlConnection.SetTrackedTables(["Companies"]);

^{snippet source | anchor}

Uses the following SQL:

For enabling tracking on a database:

alter database {database}
set change_tracking = on
(
  change_retention = {retentionDays} days,
  auto_cleanup = on
)

^{snippet source | anchor}

For enabling tracking on tables:

alter table [{table}] enable change_tracking

^{snippet source | anchor}

For disabling tracking on tables:

alter table [{table}] disable change_tracking;

^{snippet source | anchor}

Programmatic client usage

Delta is primarily designed to support web browsers as a client. All web browsers have the necessary 304 and caching functionally required.

In the scenario where web apis (that support using 304) are being consumed using .net as a client, consider using one of the below extensions to cache responses.

Icon

Estuary designed by Daan from The Noun Project.

payloadcms/payload

2024-12-01T01:51:16Z

Payload is the open-source, fullstack Next.js framework, giving you instant backend superpowers. Get a full TypeScript backend and admin panel instantly. Use Payload as a headless CMS or for building powerful applications.

Explore the Docs · Community Help · Roadmap · View G2 Reviews

[!IMPORTANT] 🎉 We've released 3.0! Star this repo or keep an eye on it to follow along.

Payload is the first-ever Next.js native CMS that can install directly in your existing /app folder. It's the start of a new era for headless CMS.

Benefits over a regular CMS

Deploy anywhere, including serverless on Vercel for free
Combine your front+backend in the same /app folder if you want
Don't sign up for yet another SaaS - Payload is open source
Query your database in React Server Components
Both admin and backend are 100% extensible
No vendor lock-in
Never touch ancient WP code again
Build faster, never hit a roadblock

Quickstart

Before beginning to work with Payload, make sure you have all of the required software.

pnpx create-payload-app@latest

If you're new to Payload, you should start with the website template (pnpx create-payload-app@latest -t website). It shows how to do everything - including custom Rich Text blocks, on-demand revalidation, live preview, and more. It comes with a frontend built with Tailwind all in one /app folder.

One-click templates

Jumpstart your next project by starting with a pre-made template. These are production-ready, end-to-end solutions designed to get you to market as fast as possible.

🌐 Website

Build any kind of website, blog, or portfolio from small to enterprise. Comes with a fully functional front-end built with RSCs and Tailwind.

We're constantly adding more templates to our Templates Directory. If you maintain your own template, consider adding the payload-template topic to your GitHub repository for others to find.

✨ Features

Completely free and open-source
Next.js native, built to run inside your /app folder
Use server components to extend Payload UI
Query your database directly in server components, no need for REST / GraphQL
Fully TypeScript with automatic types for your data
Auth out of the box
Versions and drafts
Localization
Block-based layout builder
Customizable React admin
Lexical rich text editor
Conditional field logic
Extremely granular Access Control
Document and field-level hooks for every action Payload provides
Intensely fast API
Highly secure thanks to HTTP-only cookies, CSRF protection, and more

Request Feature

🗒️ Documentation

Check out the Payload website to find in-depth documentation for everything that Payload offers.

Migrating from v2 to v3? Check out the 3.0 Migration Guide on how to do it.

🙋 Contributing

If you want to add contributions to this repository, please follow the instructions in contributing.md.

📚 Examples

The Examples Directory is a great resource for learning how to setup Payload in a variety of different ways, but you can also find great examples in our blog and throughout our social media.

If you'd like to run the examples, you can either copy them to a folder outside this repo or run them directly by (1) navigating to the example's subfolder (cd examples/your-example-folder) and (2) using the --ignore-workspace flag to bypass workspace restrictions (e.g., pnpm --ignore-workspace install or pnpm --ignore-workspace dev).

You can see more examples at:

🔌 Plugins

Payload is highly extensible and allows you to install or distribute plugins that add or remove functionality. There are both officially-supported and community-supported plugins available. If you maintain your own plugin, consider adding the payload-plugin topic to your GitHub repository for others to find.

🚨 Need help?

There are lots of good conversations and resources in our Github Discussions board and our Discord Server. If you're struggling with something, chances are, someone's already solved what you're up against. 👇

⭐ Like what we're doing? Give us a star

👏 Thanks to all our contributors

bluesky-social/social-app

2024-12-01T01:51:16Z

The Bluesky Social application for Web, iOS, and Android

Bluesky Social App

Welcome friends! This is the codebase for the Bluesky Social app.

Get the app itself:

Web: bsky.app
iOS: App Store
Android: Play Store

Development Resources

This is a React Native application, written in the TypeScript programming language. It builds on the atproto TypeScript packages (like @atproto/api), code for which is also open source, but in a different git repository.

There is a small amount of Go language source code (in ./bskyweb/), for a web service that returns the React Native Web application.

The Build Instructions are a good place to get started with the app itself.

The Authenticated Transfer Protocol ("AT Protocol" or "atproto") is a decentralized social media protocol. You don't need to understand AT Protocol to work with this application, but it can help. Learn more at:

Overview and Guides
Github Discussions 👈 Great place to ask questions
Protocol Specifications
Blogpost on self-authenticating data structures

The Bluesky Social application encompasses a set of schemas and APIs built in the overall AT Protocol framework. The namespace for these "Lexicons" is app.bsky.*.

Contributions

While we do accept contributions, we prioritize high quality issues and pull requests. Adhering to the below guidelines will ensure a more timely review.

Rules:

We may not respond to your issue or PR.
We may close an issue or PR without much feedback.
We may lock discussions or contributions if our attention is getting DDOSed.
We're not going to provide support for build issues.

Guidelines:

Check for existing issues before filing a new one please.
Open an issue and give some time for discussion before submitting a PR.
Stay away from PRs like...
- Changing "Post" to "Skeet."
- Refactoring the codebase, e.g., to replace MobX with Redux or something.
- Adding entirely new features without prior discussion.

Remember, we serve a wide community of users. Our day-to-day involves us constantly asking "which top priority is our top priority." If you submit well-written PRs that solve problems concisely, that's an awesome contribution. Otherwise, as much as we'd love to accept your ideas and contributions, we really don't have the bandwidth. That's what forking is for!

Forking guidelines

You have our blessing 🪄✨ to fork this application! However, it's very important to be clear to users when you're giving them a fork.

Please be sure to:

Change all branding in the repository and UI to clearly differentiate from Bluesky.
Change any support links (feedback, email, terms of service, etc) to your own systems.
Replace any analytics or error-collection systems with your own so we don't get super confused.

Security disclosures

If you discover any security issues, please send an email to security@bsky.app. The email is automatically CCed to the entire team and we'll respond promptly.

Are you a developer interested in building on atproto?

Bluesky is an open social network built on the AT Protocol, a flexible technology that will never lock developers out of the ecosystems that they help build. With atproto, third-party integration can be as seamless as first-party through custom feeds, federated services, clients, and more.

License (MIT)

See ./LICENSE for the full license.

P.S.

We ❤️ you and all of the ways you support us. Thank you for making Bluesky a great place!

AmruthPillai/Reactive-Resume

2024-12-01T01:51:16Z

A one-of-a-kind resume builder that keeps your privacy in mind. Completely secure, customizable, portable, open-source and free forever. Try it out today!

Reactive Resume

A free and open-source resume builder that simplifies the process of creating, updating, and sharing your resume.

Go to App | Docs

Description

Reactive Resume is a free and open-source resume builder that simplifies the process of creating, updating, and sharing your resume. With zero user tracking or advertising, your privacy is a top priority. The platform is extremely user-friendly and can be self-hosted in less than 30 seconds if you wish to own your data completely.

It's available in multiple languages and comes packed with features such as real-time editing, dozens of templates, drag-and-drop customisation, and integration with OpenAI for enhancing your writing.

You can share a personalised link of your resume to potential employers, track its views or downloads, and customise your page layout by dragging-and-dropping sections. The platform also supports various font options and provides dozens of templates to choose from. And yes, there's even a dark mode for a more comfortable viewing experience.

Start creating your standout resume with Reactive Resume today!

Templates

Azurill	Bronzor	Chikorita

Ditto	Kakuna	Nosepass

Onyx	Pikachu	Rhyhorn

Features

Free, forever and open-source
No telemetry, user tracking or advertising
You can self-host the application in less than 30 seconds
Available in multiple languages (help add/improve your language here)
Use your email address (or a throw-away address, no problem) to create an account
You can also sign in with your GitHub or Google account, and even set up two-factor authentication for extra security
Create as many resumes as you like under a single account, optimising each resume for every job application based on its description for a higher ATS score
Bring your own OpenAI API key and unlock features such as improving your writing, fixing spelling and grammar or changing the tone of your text in one-click
Translate your resume into any language using ChatGPT and import it back for easier editing
Create single page resumes or a resume that spans multiple pages easily
Customize the colours and layouts to add a personal touch to your resume.
Customise your page layout as you like just by dragging-and-dropping sections
Create custom sections that are specific to your industry if the existing ones don't fit
Jot down personal notes specific to your resume that's only visible to you
Lock a resume to prevent making any further edits (useful for master templates)
Dozens of templates to choose from, ranging from professional to modern
Design your resume using the standardised EuroPass design template
Supports printing resumes in A4 or Letter page formats
Design your resume with any font that's available on Google Fonts
Share a personalised link of your resume to companies or recruiters for them to get the latest updates
You can track the number of views or downloads your public resume has received
Built with state-of-the-art (at the moment) and dependable technologies that's battle tested and peer reviewed by the open-source community on GitHub
MIT License, so do what you like with the code as long as you credit the original author
And yes, there’s a dark mode too 🌓

Built With

React (Vite), for the frontend
NestJS, for the backend
Postgres (primary database)
Prisma ORM, which frees you to switch to any other relational database with a few minor changes in the code
Minio (for object storage: to store avatars, resume PDFs and previews)
Browserless (for headless chrome, to print PDFs and generate previews)
SMTP Server (to send password recovery emails)
GitHub/Google OAuth (for quickly authenticating users)
LinguiJS and Crowdin (for translation management and localization)

Star History

License

Reactive Resume is packaged and distributed using the MIT License which allows for commercial use, distribution, modification and private use provided that all copies of the software contain the same license and copyright.

By the community, for the community.
A passion project by Amruth Pillai

abi/screenshot-to-code

2024-12-01T01:40:21Z

Drop in a screenshot and convert it to clean code (HTML/Tailwind/React/Vue)

screenshot-to-code

A simple tool to convert screenshots, mockups and Figma designs into clean, functional code using AI. Now supporting Claude Sonnet 3.5 and GPT-4o!

https://github.com/abi/screenshot-to-code/assets/23818/6cebadae-2fe3-4986-ac6a-8fb9db030045

Supported stacks:

HTML + Tailwind
HTML + CSS
React + Tailwind
Vue + Tailwind
Bootstrap
Ionic + Tailwind
SVG

Supported AI models:

Claude Sonnet 3.5 - Best model!
GPT-4o - also recommended!
DALL-E 3 or Flux Schnell (using Replicate) for image generation

See the Examples section below for more demos.

We also just added experimental support for taking a video/screen recording of a website in action and turning that into a functional prototype.

Learn more about video here.

Follow me on Twitter for updates.

🌍 Hosted Version

Try it live on the hosted version (paid). If you're a large or medium enterprise (50+ employees), book a meeting to explore custom enterprise plans.

🛠 Getting Started

The app has a React/Vite frontend and a FastAPI backend.

Keys needed:

OpenAI API key with access to GPT-4 or Anthropic key (optional)
Both are recommended so you can compare results from both Claude and GPT4o

If you'd like to run the app with Ollama open source models (not recommended due to poor quality results), follow this comment.

Run the backend (I use Poetry for package management - pip install poetry if you don't have it):

cd backend
echo "OPENAI_API_KEY=sk-your-key" > .env
echo "ANTHROPIC_API_KEY=your-key" > .env
poetry install
poetry shell
poetry run uvicorn main:app --reload --port 7001

You can also set up the keys using the settings dialog on the front-end (click the gear icon after loading the frontend).

Run the frontend:

cd frontend
yarn
yarn dev

Open http://localhost:5173 to use the app.

If you prefer to run the backend on a different port, update VITE_WS_BACKEND_URL in frontend/.env.local

For debugging purposes, if you don't want to waste GPT4-Vision credits, you can run the backend in mock mode (which streams a pre-recorded response):

MOCK=true poetry run uvicorn main:app --reload --port 7001

Docker

If you have Docker installed on your system, in the root directory, run:

echo "OPENAI_API_KEY=sk-your-key" > .env
docker-compose up -d --build

The app will be up and running at http://localhost:5173. Note that you can't develop the application with this setup as the file changes won't trigger a rebuild.

🙋‍♂️ FAQs

I'm running into an error when setting up the backend. How can I fix it? Try this. If that still doesn't work, open an issue.
How do I get an OpenAI API key? See https://github.com/abi/screenshot-to-code/blob/main/Troubleshooting.md
How can I configure an OpenAI proxy? - If you're not able to access the OpenAI API directly (due to e.g. country restrictions), you can try a VPN or you can configure the OpenAI base URL to use a proxy: Set OPENAI_BASE_URL in the backend/.env or directly in the UI in the settings dialog. Make sure the URL has "v1" in the path so it should look like this: https://xxx.xxxxx.xxx/v1
How can I update the backend host that my front-end connects to? - Configure VITE_HTTP_BACKEND_URL and VITE_WS_BACKEND_URL in front/.env.local For example, set VITE_HTTP_BACKEND_URL=http://124.10.20.1:7001
Seeing UTF-8 errors when running the backend? - On windows, open the .env file with notepad++, then go to Encoding and select UTF-8.
How can I provide feedback? For feedback, feature requests and bug reports, open an issue or ping me on Twitter.

📚 Examples

NYTimes

Original	Replica

Instagram page (with not Taylor Swift pics)

https://github.com/abi/screenshot-to-code/assets/23818/503eb86a-356e-4dfc-926a-dabdb1ac7ba1

Hacker News but it gets the colors wrong at first so we nudge it

https://github.com/abi/screenshot-to-code/assets/23818/3fec0f77-44e8-4fb3-a769-ac7410315e5d

ed-donner/llm_engineering

2024-12-01T01:40:21Z

Repo to accompany my mastering LLM engineering course

LLM Engineering - Master AI and LLMs

Your 8 week journey to proficiency starts today

I'm so happy you're joining me on this path. We'll be building immensely satisfying projects in the coming weeks. Some will be easy, some will be challenging, many will ASTOUND you! The projects build on each other so you develop deeper and deeper expertise each week. One thing's for sure: you're going to have a lot of fun along the way.

A note before you begin

I'm here to help you be most successful with your learning! If you hit any snafus, or if you have any ideas on how I can improve the course, please do reach out in the platform or by emailing me direct (ed@edwarddonner.com). It's always great to connect with people on LinkedIn to build up the community - you'll find me here:
https://www.linkedin.com/in/eddonner/

Resources to accompany the course, including the slides and useful links, are here:
https://edwarddonner.com/2024/11/13/llm-engineering-resources/

Instant Gratification instructions for Week 1, Day 1

We will start the course by installing Ollama so you can see results immediately!

Download and install Ollama from https://ollama.com
On a PC, start a Command prompt / Powershell (Press Win + R, type cmd, and press Enter). On a Mac, start a Terminal (Applications > Utilities > Terminal).
Run ollama run llama3.2 or for smaller machines try ollama run llama3.2:1b
If this doesn't work, you may need to run ollama serve in another Powershell (Windows) or Terminal (Mac), and try step 3 again
And if that doesn't work on your box, I've set up this on the cloud. This is on Google Colab, which will need you to have a Google account to sign in, but is free: https://colab.research.google.com/drive/1-_f5XZPsChvfU1sJ0QqCePtIuc55LSdu?usp=sharing

Any problems, please contact me!

Then, Setup instructions

After we do the Ollama quick project, and after I introduce myself and the course, we get to work with the full environment setup.

Hopefully I've done a decent job of making these guides bulletproof - but please contact me right away if you hit roadblocks:

PC people please follow the instructions in SETUP-PC.md
Mac people please follow the instructions in SETUP-mac.md
Linux people, the Mac instructions should be close enough!

An important point on API costs (which are optional! No need to spend if you don't wish)

During the course, I'll suggest you try out the leading models at the forefront of progress, known as the Frontier models. I'll also suggest you run open-source models using Google Colab. These services have some charges, but I'll keep cost minimal - like, a few cents at a time. And I'll provide alternatives if you'd prefer not to use them.

Please do monitor your API usage to ensure you're comfortable with spend; I've included links below. There's no need to spend anything more than a couple of dollars for the entire course. Some AI providers such as OpenAI require a minimum credit like $5 or local equivalent; we should only spend a fraction of it, and you'll have plenty of opportunity to put it to good use in your own projects. During Week 7 you have an option to spend a bit more if you're enjoying the process - I spend about $10 myself and the results make me very happy indeed! But it's not necessary in the least; the important part is that you focus on learning.

I'll also show you an alternative if you'd rather not spend anything on APIs.

How this Repo is organized

There are folders for each of the "weeks", representing modules of the class, culminating in a powerful autonomous Agentic AI solution in Week 8 that draws on many of the prior weeks.
Follow the setup instructions above, then open the Week 1 folder and prepare for joy.

The most important part

The mantra of the course is: the best way to learn is by DOING. I don't type all the code during the course; I execute it for you to see the results. You should work along with me or after each lecture, running each cell, inspecting the objects to get a detailed understanding of what's happening. Then tweak the code and make it your own. There are juicy challenges for you throughout the course. I'd love it if you wanted to push your code so I can follow along with your progress, and I can make your solutions available to others so we share in your progress. While the projects are enjoyable, they are first and foremost designed to be educational, teaching you business skills that can be put into practice in your work.

Starting in Week 3, we'll also be using Google Colab for running with GPUs

You should be able to use the free tier or minimal spend to complete all the projects in the class. I personally signed up for Colab Pro+ and I'm loving it - but it's not required.

Learn about Google Colab and set up a Google account (if you don't already have one) here

The colab links are in the Week folders and also here:

For week 3 day 1, this Google Colab shows what colab can do
For week 3 day 2, here is a colab for the HuggingFace pipelines API
For week 3 day 3, here's the colab on Tokenizers
For week 3 day 4, we go to a colab with HuggingFace models
For week 3 day 5, we return to colab to make our Meeting Minutes product
For week 7, we will use these Colab books: Day 1 | Day 2 | Days 3 and 4 | Day 5

Monitoring API charges

You can keep your API spend very low throughout this course; you can monitor spend at the dashboards: here for OpenAI, here for Anthropic and here for Google Gemini.

The charges for the exercsies in this course should always be quite low, but if you'd prefer to keep them minimal, then be sure to always choose the cheapest versions of models:

For OpenAI: Always use model gpt-4o-mini in the code instead of gpt-4o
For Anthropic: Always use model claude-3-haiku-20240307 in the code instead of the other Claude models
During week 7, look out for my instructions for using the cheaper dataset

Please do message me or email me at ed@edwarddonner.com if this doesn't work or if I can help with anything. I can't wait to hear how you get on.

Other resources

I've put together this webpage with useful resources for the course. This includes links to all the slides.
https://edwarddonner.com/2024/11/13/llm-engineering-resources/
Please keep this bookmarked, and I'll continue to add more useful links there over time.

shader-slang/slang

2024-12-01T01:40:21Z

Making it easier to work with shaders

Slang

Slang is a shading language that makes it easier to build and maintain large shader codebases in a modular and extensible fashion, while also maintaining the highest possible performance on modern GPUs and graphics APIs. Slang is based on years of collaboration between researchers at NVIDIA, Carnegie Mellon University, Stanford, MIT, UCSD and the University of Washington.

Why Slang?

The Slang shading language is designed to enable real-time graphics developers to work with large-scale, high-performance shader code.

Write Shaders Once, Run Anywhere

The Slang compiler can generate code for a wide variety of targets: D3D12, Vulkan, Metal, D3D11, OpenGL, CUDA, and even generate code to run on a CPU. For textual targets, such as Metal Shading Language (MSL) and CUDA, Slang produces readable code that preserves original identifier names, as well as the type and call structure, making it easier to debug.

Access the Latest GPU Features

Slang code is highly portable, but can still leverage unique platform capabilities, including the latest features in Direct3D and Vulkan. For example, developers can make full use of pointers when generating SPIR-V. Slang's capability system helps applications manage feature set differences across target platforms by ensuring code only uses available features during the type-checking step, before generating final code. Additionally, Slang provides flexible interop features to enable directly embedding target code or SPIR-V into generated shaders.

Leverage Neural Graphics with Automatic Differentiation

Slang can automatically generate both forward and backward derivative propagation code for complex functions that involve arbitrary control flow and dynamic dispatch. This allows existing rendering codebases to easily become differentiable, or for Slang to serve as the kernel language in a PyTorch-driven machine learning framework via slangtorch.

Scalable Software Development with Modules

Slang provides a module system that enables logical organization of code for separate compilation. Slang modules can be independently compiled offline to a custom IR (with optional obfuscation) and then linked at runtime to generate code in formats such as DXIL or SPIR-V.

Code Specialization that Works with Modules

Slang supports generics and interfaces (a.k.a. type traits/protocols), allowing for clear expression of shader specialization without the need for preprocessor techniques or string-pasting. Unlike C++ templates, Slang's generics are pre-checked and don't produce cascading error messages that are difficult to diagnose. The same generic shader can be specialized for a variety of different types to produce specialized code ahead of time, or on the fly, entirely under application control.

Easy On-ramp for HLSL and GLSL Codebases

Slang's syntax is similar to HLSL, and most existing HLSL code can be compiled with the Slang compiler out-of-the-box, or with just minor modifications. This allows existing shader codebases to immediately benefit from Slang without requiring a complete rewrite or port.

Slang provides a compatibility module that enables the use of most GLSL intrinsic functions and GLSL's parameter binding syntax.

Comprehensive Tooling Support

Slang comes with full support of IntelliSense editing features in Visual Studio Code and Visual Studio through the Language Server Protocol. Full debugging capabilities are also available through RenderDoc and SPIR-V based tools.

Getting Started

The fastest way to get started using Slang in your own development is to use a pre-built binary package, available through GitHub releases. Slang binaries are also included in the Vulkan SDK since version 1.3.296.0.

There are packages built for 32- and 64-bit Windows, as well as 64-bit Ubuntu. Each binary release includes the command-line slangc compiler, a shared library for the compiler, and the slang.h header.

See the user-guide for info on using the slangc command-line tool: Slang Command Line Usage.

If you want to try out the Slang language without installing anything, a fast and simple way is to use the Slang Playground. The playground allows you to compile Slang code to a variety of targets, and even run some simple shaders directly within the browser. The playground loads Slang compiler to your browser and runs all compilation locally. No data will be sent to any servers.

If you would like to build Slang from source, please consult the build instructions.

Documentation

The Slang project provides a variety of different documentation, but most users would be well served starting with the User's Guide.

For developers writing Slang code, the Slang Core Module Reference provides detailed documentation on Slang's built-in types and functions.

We also provide a few examples of how to integrate Slang into a rendering application.

These examples use a graphics layer that we include with Slang called "GFX" which is an abstraction library of various graphics APIs (D3D11, D2D12, OpenGL, Vulkan, CUDA, and the CPU) to support cross-platform applications using GPU graphics and compute capabilities. If you'd like to learn more about GFX, see the GFX User Guide.

Additionally, we recommend checking out Vulkan Mini Examples for more examples of using Slang's language features available on Vulkan, such as pointers and the ray tracing intrinsics.

Contributing

If you'd like to contribute to the project, we are excited to have your input. The following guidelines should be observed by contributors:

Please follow the contributor Code of Conduct.
Bugs reports and feature requests should go through the GitHub issue tracker
Changes should ideally come in as small pull requests on top of master, coming from your own personal fork of the project
Large features that will involve multiple contributors or a long development time should be discussed in issues, and broken down into smaller pieces that can be implemented and checked in in stages

Contribution guide describes the workflow for contributors at more detail.

Limitations and Support

Platform support

The Slang compiler and libraries can be built on the following platforms:

Windows	Linux	MacOS	WebAssembly
supported	supported	supported	experimental

Both x86_64 and aarch64 architectures are supported on Windows, Linux and MacOS platforms.

Target support

Slang can compile shader code to the following targets:

Target	Status	Output Formats
Direct3D 11	supported	HLSL
Direct3D 12	supported	HLSL
Vulkan	supported	SPIRV, GLSL
Metal	experimental*	Metal Shading Language
WebGPU	experimental**	WGSL
CUDA	supported	C++ (compute only)
Optix	experimental	C++ (WIP)
CPU	experimental	C++ (kernel), C++ (host), standalone executable, dynamic library

*Slang currently supports generating vertex, fragment, compute, task and mesh shaders for Metal.

**WGSL support is still work in-progress.

For greater detail, see the Supported Compilation Targets section of the User Guide

The Slang project has been used for production applications and large shader codebases, but it is still under active development. Support is currently focused on the platforms (Windows, Linux) and target APIs (Direct3D 12, Vulkan) where Slang is used most heavily. Users who are looking for support on other platforms or APIs should coordinate with the development team via the issue tracker to make sure that their use cases can be supported.

License

The Slang code itself is under the Apache 2.0 with LLVM Exception license (see LICENSE).

Builds of the core Slang tools depend on the following projects, either automatically or optionally, which may have their own licenses:

glslang (BSD)
lz4 (BSD)
miniz (MIT)
spirv-headers (Modified MIT)
spirv-tools (Apache 2.0)
ankerl::unordered_dense::{map, set} (MIT)

Slang releases may include slang-llvm which includes LLVM under the license:

llvm (Apache 2.0 License with LLVM exceptions)

Some of the tests and example programs that build with Slang use the following projects, which may have their own licenses:

glm (MIT)
stb_image and stb_image_write from the stb collection of single-file libraries (Public Domain)
tinyobjloader (MIT)

apple/swift-crypto

2024-12-01T01:51:35Z

Open-source implementation of a substantial portion of the API of Apple CryptoKit suitable for use on Linux platforms.

apache/arrow-go

2024-12-01T01:51:35Z

Official Go implementation of Apache Arrow

apple/swift-crypto

2024-12-01T01:40:44Z

Open-source implementation of a substantial portion of the API of Apple CryptoKit suitable for use on Linux platforms.

cameronapak/freedom-stack

2024-12-01T01:29:56Z

A full-stack Astro starter kit that feels freeing and is free. Ready for use, and even better when using Cursor's IDE.

delucis/astro-embed

2024-12-01T01:40:42Z

Low-JavaScript embed components for Astro websites

Azure-Samples/contoso-chat

2024-12-01T01:51:43Z

This sample has the full End2End process of creating RAG application with Prompty and Azure AI Foundry. It includes GPT 3.5 Turbo LLM application code, evaluations, deployment automation with AZD CLI, GitHub actions for evaluation and deployment and intent mapping for multiple LLM task mapping.

Azure-Samples/msdocs-nodejs-mongodb-azure-sample-app

2024-12-01T01:51:43Z

A sample Express.js app using a MongoDB database to show how to host Node.js app in Azure App service using Azure Cosmos DB

Azure-Samples/msdocs-nodejs-mongodb-azure-sample-app

2024-12-01T01:40:53Z

A sample Express.js app using a MongoDB database to show how to host Node.js app in Azure App service using Azure Cosmos DB

meta-flutter/meta-flutter

2024-12-01T01:51:48Z

Google Flutter for Yocto

Jabka666/scpcb-ue-my

2024-11-30T01:29:15Z

SCP - Containment Breach Ultimate Edition

SimonCropp/Delta

2024-11-30T01:29:38Z

An approach to implementing a 304 Not Modified leveraging SqlServer change tracking

Delta

Delta is an approach to implementing a 304 Not Modified leveraging SqlServer change tracking

The approach uses a last updated timestamp from the database to generate an ETag. All dynamic requests then have that ETag checked/applied.

Effectively consumers will always receive the most current data, while the load on the server remains low.

See Milestones for release notes.

Assumptions

Frequency of updates to data is relatively low compared to reads
Using either SQL Server Change Tracking and/or SQL Server Row Versioning

304 Not Modified Flow

graph TD
    Request
    CalculateEtag[Calculate current ETag<br/>based on timestamp<br/>from web assembly and SQL]
    IfNoneMatch{Has<br/>If-None-Match<br/>header?}
    EtagMatch{Current<br/>Etag matches<br/>If-None-Match?}
    AddETag[Add current ETag<br/>to Response headers]
    304[Respond with<br/>304 Not-Modified]
    Request --> CalculateEtag
    CalculateEtag --> IfNoneMatch
    IfNoneMatch -->|Yes| EtagMatch
    IfNoneMatch -->|No| AddETag
    EtagMatch -->|No| AddETag
    EtagMatch -->|Yes| 304

ETag calculation logic

The ETag is calculated from a combination several parts

AssemblyWriteTime

The last write time of the web entry point assembly

var webAssemblyLocation = Assembly.GetEntryAssembly()!.Location;
AssemblyWriteTime = File.GetLastWriteTime(webAssemblyLocation).Ticks.ToString();

^{snippet source | anchor}

SQL timestamp

A combination of change_tracking_current_version (if tracking is enabled) and @@DBTS (row version timestamp)

declare @changeTracking bigint = change_tracking_current_version();
declare @timeStamp bigint = convert(bigint, @@dbts);

if (@changeTracking is null)
  select cast(@timeStamp as varchar)
else
  select cast(@timeStamp as varchar) + '-' + cast(@changeTracking as varchar)

^{snippet source | anchor}

Suffix

An optional string suffix that is dynamically calculated at runtime based on the current HttpContext.

var app = builder.Build();
app.UseDelta(suffix: httpContext => "MySuffix");

^{snippet source | anchor}

Combining the above

internal static string BuildEtag(string timeStamp, string? suffix)
{
    if (suffix == null)
    {
        return $"\"{AssemblyWriteTime}-{timeStamp}\"";
    }

    return $"\"{AssemblyWriteTime}-{timeStamp}-{suffix}\"";
}

^{snippet source | anchor}

NuGet

Delta is shipped as two nugets:

Delta: Delivers functionality using SqlConnection and SqlTransaction.
Delta.EF: Delivers functionality using SQL Server EF Database Provider.

Only one of the above should be used.

Usage

DB Schema

Ensure SQL Server Change Tracking and/or SQL Server Row Versioning is enabled for all relevant tables.

Example SQL schema:

-- Tables

CREATE TABLE [dbo].[Companies](
	[Id] [uniqueidentifier] NOT NULL,
	[RowVersion] [timestamp] NOT NULL,
	[Content] [nvarchar](max) NULL,
 CONSTRAINT [PK_Companies] PRIMARY KEY CLUSTERED 
(
	[Id] ASC
) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

CREATE TABLE [dbo].[Employees](
	[Id] [uniqueidentifier] NOT NULL,
	[RowVersion] [timestamp] NOT NULL,
	[CompanyId] [uniqueidentifier] NOT NULL,
	[Content] [nvarchar](max) NULL,
	[Age] [int] NOT NULL,
 CONSTRAINT [PK_Employees] PRIMARY KEY CLUSTERED 
(
	[Id] ASC
) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

CREATE NONCLUSTERED INDEX [IX_Employees_CompanyId] ON [dbo].[Employees]
(
	[CompanyId] ASC
) ON [PRIMARY]

^{snippet source | anchor}

Add to WebApplicationBuilder

var builder = WebApplication.CreateBuilder();
builder.Services.AddScoped(_ => new SqlConnection(connectionString));
var app = builder.Build();
app.UseDelta();

^{snippet source | anchor}

Add to a Route Group

To add to a specific Route Group:

app.MapGroup("/group")
    .UseDelta()
    .MapGet("/", () => "Hello Group!");

^{snippet source | anchor}

ShouldExecute

Optionally control what requests Delta is executed on.

var app = builder.Build();
app.UseDelta(
    shouldExecute: httpContext =>
    {
        var path = httpContext.Request.Path.ToString();
        return path.Contains("match");
    });

^{snippet source | anchor}

Custom Connection discovery

By default, Delta uses HttpContext.RequestServices to discover the SqlConnection and SqlTransaction:

static Connection DiscoverConnection(HttpContext httpContext)
{
    var provider = httpContext.RequestServices;
    var connection = provider.GetRequiredService<SqlConnection>();
    var transaction = provider.GetService<SqlTransaction>();
    return new(connection, transaction);
}

^{snippet source | anchor}

To use custom connection discovery:

var application = webApplicationBuilder.Build();
application.UseDelta(
    getConnection: httpContext => httpContext.RequestServices.GetRequiredService<SqlConnection>());

^{snippet source | anchor}

To use custom connection and transaction discovery:

var webApplication = webApplicationBuilder.Build();
webApplication.UseDelta(
    getConnection: httpContext =>
    {
        var provider = httpContext.RequestServices;
        var sqlConnection = provider.GetRequiredService<SqlConnection>();
        var sqlTransaction = provider.GetService<SqlTransaction>();
        return new(sqlConnection, sqlTransaction);
    });

^{snippet source | anchor}

EF Usage

DbContext using RowVersion

Enable row versioning in Entity Framework

public class SampleDbContext(DbContextOptions options) :
    DbContext(options)
{
    public DbSet<Employee> Employees { get; set; } = null!;
    public DbSet<Company> Companies { get; set; } = null!;

    protected override void OnModelCreating(ModelBuilder builder)
    {
        var company = builder.Entity<Company>();
        company.HasKey(_ => _.Id);
        company
            .HasMany(_ => _.Employees)
            .WithOne(_ => _.Company)
            .IsRequired();
        company
            .Property(_ => _.RowVersion)
            .IsRowVersion()
            .HasConversion<byte[]>();

        var employee = builder.Entity<Employee>();
        employee.HasKey(_ => _.Id);
        employee
            .Property(_ => _.RowVersion)
            .IsRowVersion()
            .HasConversion<byte[]>();
    }
}

^{snippet source | anchor}

Add to WebApplicationBuilder

var builder = WebApplication.CreateBuilder();
builder.Services.AddSqlServer<SampleDbContext>(database.ConnectionString);
var app = builder.Build();
app.UseDelta<SampleDbContext>();

^{snippet source | anchor}

Add to a Route Group

To add to a specific Route Group:

app.MapGroup("/group")
    .UseDelta<SampleDbContext>()
    .MapGet("/", () => "Hello Group!");

^{snippet source | anchor}

ShouldExecute

Optionally control what requests Delta is executed on.

var app = builder.Build();
app.UseDelta<SampleDbContext>(
    shouldExecute: httpContext =>
    {
        var path = httpContext.Request.Path.ToString();
        return path.Contains("match");
    });

^{snippet source | anchor}

UseResponseDiagnostics

Response diagnostics is an opt-in feature that includes extra log information in the response headers.

Enable by setting UseResponseDiagnostics to true at startup:

DeltaExtensions.UseResponseDiagnostics = true;

^{snippet source | anchor}

Response diagnostics headers are prefixed with Delta-.

Example Response header when the Request has not If-None-Match header.

Helpers

Utility methods for working with databases using the Delta conventions.

GetLastTimeStamp

For a `SqlConnection`:

var timeStamp = await sqlConnection.GetLastTimeStamp();

^{snippet source | anchor}

For a `DbContext`:

var timeStamp = await dbContext.GetLastTimeStamp();

^{snippet source | anchor}

GetDatabasesWithTracking

Get a list of all databases with change tracking enabled.

var trackedDatabases = await sqlConnection.GetTrackedDatabases();
foreach (var db in trackedDatabases)
{
    Trace.WriteLine(db);
}

^{snippet source | anchor}

Uses the following SQL:

select d.name
from sys.databases as d inner join
  sys.change_tracking_databases as t on
  t.database_id = d.database_id

^{snippet source | anchor}

GetTrackedTables

Get a list of all tracked tables in database.

var trackedTables = await sqlConnection.GetTrackedTables();
foreach (var db in trackedTables)
{
    Trace.WriteLine(db);
}

^{snippet source | anchor}

Uses the following SQL:

select t.Name
from sys.tables as t left join
  sys.change_tracking_tables as c on t.[object_id] = c.[object_id]
where c.[object_id] is not null

^{snippet source | anchor}

IsTrackingEnabled

Determine if change tracking is enabled for a database.

var isTrackingEnabled = await sqlConnection.IsTrackingEnabled();

^{snippet source | anchor}

Uses the following SQL:

select count(d.name)
from sys.databases as d inner join
  sys.change_tracking_databases as t on
  t.database_id = d.database_id
where d.name = '{database}'

^{snippet source | anchor}

EnableTracking

Enable change tracking for a database.

await sqlConnection.EnableTracking();

^{snippet source | anchor}

Uses the following SQL:

alter database {database}
set change_tracking = on
(
  change_retention = {retentionDays} days,
  auto_cleanup = on
)

^{snippet source | anchor}

DisableTracking

Disable change tracking for a database and all tables within that database.

await sqlConnection.DisableTracking();

^{snippet source | anchor}

Uses the following SQL:

For disabling tracking on a database:

alter database [{database}] set change_tracking = off;

^{snippet source | anchor}

For disabling tracking on tables:

alter table [{table}] disable change_tracking;

^{snippet source | anchor}

SetTrackedTables

Enables change tracking for all tables listed, and disables change tracking for all tables not listed.

await sqlConnection.SetTrackedTables(["Companies"]);

^{snippet source | anchor}

Uses the following SQL:

For enabling tracking on a database:

alter database {database}
set change_tracking = on
(
  change_retention = {retentionDays} days,
  auto_cleanup = on
)

^{snippet source | anchor}

For enabling tracking on tables:

alter table [{table}] enable change_tracking

^{snippet source | anchor}

For disabling tracking on tables:

alter table [{table}] disable change_tracking;

^{snippet source | anchor}

Programmatic client usage

Delta is primarily designed to support web browsers as a client. All web browsers have the necessary 304 and caching functionally required.

In the scenario where web apis (that support using 304) are being consumed using .net as a client, consider using one of the below extensions to cache responses.

Icon

Estuary designed by Daan from The Noun Project.

yaobiao131/downkyicore

2024-12-01T01:52:06Z

哔哩下载姬(跨平台版)downkyi，哔哩哔哩网站视频下载工具，支持批量下载，支持8K、HDR、杜比视界，提供工具箱（音视频提取、去水印等）。

哔哩下载姬(跨平台版)

下载

更新日志

介绍

基于哔哩下载姬Windows版和AvaloniaUI制作的跨平台版本(支持Windows、linux、macOS)。
开发这个版本目的是由于本人日常使用macOS，当我想下载up视频是偶然间发现了哔哩下载姬Windows版，发现很好用，就是不能支持macOS使用，就基于AvaloniaUI重新开发了一个跨平台版本。

使用说明

软件自带.NET6、ffmpeg、aria2运行环境、无需自行安装
默认下载路径:
- Windows: 软件运行目录下的Media文件夹
- macOS: ~/Library/Application Support/DownKyi/Media
- linux: ~/.config/DownKyi/Media

免责申明

本软件只提供视频解析，不提供任何资源上传、存储到服务器的功能。
本软件仅解析来自B站的内容，不会对解析到的音视频进行二次编码，部分视频会进行有限的格式转换、拼接等操作。
本软件解析得到的所有内容均来自B站UP主上传、分享，其版权均归原作者所有。内容提供者、上传者（UP主）应对其提供、上传的内容承担全部责任。
本软件提供的所有内容，仅可用作学习交流使用，未经原作者授权，禁止用于其他用途。请在下载24小时内删除。为尊重作者版权，请前往资源的原始发布网站观看，支持原创，谢谢。
因使用本软件产生的版权问题，软件作者概不负责。

dotnet/runtime

2024-12-01T01:52:06Z

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.

.NET Runtime

This repo contains the code to build the .NET runtime, libraries and shared host (dotnet) installers for all supported platforms, as well as the sources to .NET runtime and libraries.

What is .NET?

Official Starting Page: https://dotnet.microsoft.com

How to use .NET (with VS, VS Code, command-line CLI)
- Install official releases
- Documentation (Get Started, Tutorials, Porting from .NET Framework, API reference, ...)
  - Deploying apps
Support (Releases, OS Versions, ...)
Roadmap

How can I contribute?

We welcome contributions! Many people all over the world have helped make this project better.

Contributing explains what kinds of contributions we welcome
Workflow Instructions explains how to build and test
Dogfooding .NET explains how to get nightly builds of the runtime and its libraries to test them in your own projects.

Reporting security issues and security bugs

Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) secure@microsoft.com. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the Security TechCenter. You can also find these instructions in this repo's Security doc.

Also see info about related Microsoft .NET Bounty Program.

Filing issues

This repo should contain issues that are tied to the runtime, the class libraries and frameworks, the installation of the dotnet binary (sometimes known as the muxer) and the installation of the .NET runtime and libraries.

For other issues, please file them to their appropriate sibling repos. We have links to many of them on our new issue page.

Useful Links

.NET source index / .NET Framework source index
API Reference docs
.NET API Catalog (incl. APIs from daily builds and API usage info)
API docs writing guidelines - useful when writing /// comments
.NET Discord Server - a place to discuss the development of .NET and its ecosystem

.NET Foundation

.NET Runtime is a .NET Foundation project.

There are many .NET related projects on GitHub.

.NET home repo - links to 100s of .NET projects, from Microsoft and the community.
ASP.NET Core home - the best place to start learning about ASP.NET Core.

This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community. For more information, see the .NET Foundation Code of Conduct.

General .NET OSS discussions: .NET Foundation Discussions

License

.NET (including the runtime repo) is licensed under the MIT license.

dotnet/eShop

2024-12-01T01:52:06Z

A reference .NET application implementing an eCommerce site

eShop Reference Application - "AdventureWorks"

A reference .NET application implementing an e-commerce website using a services-based architecture using .NET Aspire.

Getting Started

This version of eShop is based on .NET 9.

Previous eShop versions:

.NET 8

Prerequisites

Clone the eShop repository: https://github.com/dotnet/eshop
Install & start Docker Desktop

Windows with Visual Studio

Install Visual Studio 2022 version 17.10 or newer.
- Select the following workloads:
  - ASP.NET and web development workload.
  - .NET Aspire SDK component in Individual components.
  - Optional: .NET Multi-platform App UI development to run client apps

Run the following commands in a Powershell & Terminal running as Administrator to automatically configure your environment with the required tools to build and run this application. (Note: A restart is required and included in the script below.)

install-Module -Name Microsoft.WinGet.Configuration -AllowPrerelease -AcceptLicense -Force
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
get-WinGetConfiguration -file .\.configurations\vside.dsc.yaml | Invoke-WinGetConfiguration -AcceptConfigurationAgreements

From Dev Home go to Machine Configuration -> Clone repositories. Enter the URL for this repository. In the confirmation screen look for the section Configuration File Detected and click Run File.

Mac, Linux, & Windows without Visual Studio

Install the latest .NET 9 SDK

Run the following commands in a Powershell & Terminal running as Administrator to automatically configuration your environment with the required tools to build and run this application. (Note: A restart is required after running the script below.)

Install Visual Studio Code and related extensions

install-Module -Name Microsoft.WinGet.Configuration -AllowPrerelease -AcceptLicense  -Force
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
get-WinGetConfiguration -file .\.configurations\vscode.dsc.yaml | Invoke-WinGetConfiguration -AcceptConfigurationAgreements

Note: These commands may require sudo

Optional: Install Visual Studio Code with C# Dev Kit
Optional: Install .NET MAUI Workload

Note: When running on Mac with Apple Silicon (M series processor), Rosetta 2 for grpc-tools.

Running the solution

[!WARNING] Remember to ensure that Docker is started

(Windows only) Run the application from Visual Studio:

Open the eShop.Web.slnf file in Visual Studio
Ensure that eShop.AppHost.csproj is your startup project
Hit Ctrl-F5 to launch Aspire

Or run the application from your terminal:

dotnet run --project src/eShop.AppHost/eShop.AppHost.csproj

then look for lines like this in the console output in order to find the URL to open the Aspire dashboard:

Login to the dashboard at: http://localhost:19888/login?t=uniquelogincodeforyou

You may need to install ASP.NET Core HTTPS development certificates first, and then close all browser tabs. Learn more at https://aka.ms/aspnet/https-trust-dev-cert

Azure Open AI

When using Azure OpenAI, inside eShop.AppHost/appsettings.json, add the following section:

  "ConnectionStrings": {
    "OpenAi": "Endpoint=xxx;Key=xxx;"
  }

Replace the values with your own. Then, in the eShop.AppHost Program.cs, set this value to true

bool useOpenAI = false;

Here's additional guidance on the .NET Aspire OpenAI component.

Use Azure Developer CLI

You can use the Azure Developer CLI to run this project on Azure with only a few commands. Follow the next instructions:

Install the latest or update to the latest Azure Developer CLI (azd).
Log in azd (if you haven't done it before) to your Azure account:

azd auth login

Initialize azd from the root of the repo.

azd init

During init:
- Select Use code in the current directory. Azd will automatically detect the .NET Aspire project.
- Confirm .NET (Aspire) and continue.
- Select which services to expose to the Internet (exposing webapp is enough to test the sample).
- Finalize the initialization by giving a name to your environment.
Create Azure resources and deploy the sample by running:

azd up

Notes:

The operation takes a few minutes the first time it is ever run for an environment.
At the end of the process, azd will display the url for the webapp. Follow that link to test the sample.
You can run azd up after saving changes to the sample to re-deploy and update the sample.
Report any issues to azure-dev repo.
FAQ and troubleshoot for azd.

Contributing

For more information on contributing to this repo, read the contribution documentation and the Code of Conduct.

Sample data

The sample catalog data is defined in catalog.json. Those product names, descriptions, and brand names are fictional and were generated using GPT-35-Turbo, and the corresponding product images were generated using DALL·E 3.

eShop on Azure

For a version of this app configured for deployment on Azure, please view the eShop on Azure repo.

dotnet/ai-samples

2024-12-01T01:41:13Z

page_type: sample languages:

azdeveloper
bicep
csharp
powershell products:
azure
ai-services
azure-openai urlFragment: ai-samples name: AI Samples for .NET description: .NET samples demonstrating how to use AI in your .NET applications.

AI Samples for .NET

Welcome to the official home for .NET samples demonstrating how to use AI in your .NET applications. If you're new to AI, start at the top and work your way down. Otherwise, jump in wherever suits your interests.

Microsoft.Extensions.AI (preview)

Microsoft.Extensions.AI is a set of core .NET libraries developed in collaboration with the .NET ecosystem, including Semantic Kernel. These libraries provide a unified layer of C# abstractions for interacting with AI services, such as small and large language models (SLMs and LLMs) and embeddings.

Core benefits:

Unified API: Offers a consistent and standard set of APIs and conventions for integrating AI services into .NET applications.
Flexibility: Allows .NET library authors to use AI services without forcing a specific AI provider, making it parameterizable with any provider.
Ease of Use: Enables .NET application developers to experiment with different packages using the same underlying abstractions and to utilize a single API throughout their application.
Componentization: Facilitates the addition of new capabilities and simplifies the componentization and testing of applications.

For more details, see the Introducing Microsoft.Extensions.AI Preview blog post.

Topic	GitHub Link
Abstraction implementations	GitHub Link
Azure OpenAI	GitHub Link
OpenAI	GitHub Link
Azure AI Inference	GitHub Link
Ollama	GitHub Link

Build 2024 - Infusing your .NET Apps with AI: Practical Tools and Techniques

Discover how to bring AI into your .NET application! This session covers the tools, libraries, and best practices for incorporating LLMs or other AI capabilities to create an "intelligent app". We'll explore practical examples, including how to leverage Azure AI services and the .NET AI ecosystem, to enhance your apps with AI.

Youtube: Infusing your .NET Apps with AI: Practical Tools and Techniques

Tutorial: Infusing your .NET Apps with AI: Practical Tools and Techniques

#	Topic	YouTube Link and Time	GitHub Link
1	Hello Semantic Kernel	3m 0s	Hello Semantic Kernel
2	Add Chat History	5m 40s	Add Chat History
3	Add Plugin - Function Call	7m 10s	Add Plugin - Function Call
4	Add Logging	9m 24s	Add Logging
5	Add Plugin - Bing Search	11m 15s	Add Plugin - Bing Search
6	Modify Kernel Behavior with Dependency Injection	12m 37s	Modifying Kernel Behavior with Dependency Injection
7	Using Semantic Kernel in a Web App	15m 57s	Using Semantic Kernel in a Web App

Quickstarts using OpenAI

#	Topic	GitHub Link
1	Text Summary	Hike Benefits Summary Project
2	Hiker AI	Hiker AI Project
3	Chat Context/Data	Chatting About my Previous Hikes Project
4	Hiker AI Pro (Tool extension)	Hiker AI Pro
5	Generating images	Hike Images Project
REF	Using Milvus	Coming Soon
REF	Using Qdrant	Coming Soon

Quickstarts using the Azure OpenAI SDK

#	Topic	GitHub Link
1	Text Summary	Hike Benefits Summary Project
2	Hiker AI	Hiker AI Project
3	Chat Context/Data	Chatting About my Previous Hikes Project
4	Hiker AI Pro (Tool extension)	Hiker AI Pro
5	Generating images	Hike Images Project

Chat samples

#	Topic	GitHub Link
1	Customer Support	Customer Support Project

LLM Core Concepts

#	Topic	GitHub Link
REF	Tokenizer	Coming Soon
REF	Embeddings	Coming Soon
REF	RAG	Coming Soon
REF	Prompts / Prompt Engineering	Coming Soon

End to End Examples

#	Topic	GitHub Link
E2E	Azure Search .	Azure Search Repository
E2E	E-Shop Sample	E-Shop Repository

Local Models

#	Topic	GitHub Link
REF	Phi	Phi
REF	Llama 2	Coming Soon

Local Embedding Models

#	Topic	GitHub Link
REF	Clip	Coming Soon

Tokenizers

#	Topic	GitHub Link
REF	TikToken	Coming Soon

Vector Databases

#	Topic	GitHub Link
REF	Azure AI Search	Coming Soon
REF	PostgreSQL + pgvector	Coming Soon
REF	Milvus	Coming Soon
REF	Qdrant	Coming Soon

Reference Examples

#	Topic	GitHub Link
REF	Vector<T>	Coming Soon

.NET Foundation

There are many .NET related projects on GitHub.

.NET home repo - links to 100s of .NET projects, from Microsoft and the community.
ASP.NET Core home - the best place to start learning about ASP.NET Core.

This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community. For more information, see the .NET Foundation Code of Conduct.

License

.NET (including the csharp-notebooks repo) is licensed under the MIT license.

HotCakeX/Harden-Windows-Security

2024-12-01T01:41:13Z

Harden Windows Safely, Securely using Official Supported Microsoft methods and proper explanation | Always up-to-date and works with the latest build of Windows | Provides tools and Guides for Personal, Enterprise, Government and Military security levels | Read The Rationale https://github.com/HotCakeX/Harden-Windows-Security/blob/main/Rationale.md

Harden Windows Security | A New Threat to Malware

Harden Windows Safely, Securely, Only With Official Microsoft Methods

Hardening Categories How To Use Features Related Trust Support Security Recommendations Resources License Wiki Basic FAQs Roadmap

[!IMPORTANT]
Click/Tap on Each of the Items Below to Access Them on This GitHub Repository

App Control for Business Resources

The WDACConfig Module for App Control for Business

Read the Rationale Behind This GitHub Repository

[!NOTE]
Windows by default is secure and safe, this repository does not imply nor claim otherwise. Just like anything, you have to use it wisely and don't compromise yourself with reckless behavior and bad user configuration; Nothing is foolproof. This repository only uses the tools and features that have already been implemented by Microsoft in Windows OS to fine-tune it towards the highest security and locked-down state, using well-documented, supported, recommended and official methods. Continue reading for comprehensive info.

How To Use

Start The Harden Windows Security Using GUI (Graphical User Interface)

(irm 'https://raw.githubusercontent.com/HotCakeX/Harden-Windows-Security/main/Harden-Windows-Security.ps1')+'P'|iex

Install the Harden Windows Security Module from PowerShell Gallery

Check the documentation and How to use

Click/Tap here for commands

Install-Module -Name 'Harden-Windows-Security-Module' -Force

Protect-WindowsSecurity -GUI

Confirm-SystemCompliance

Unprotect-WindowsSecurity

Requirements

PowerShell (latest version), Install it from 🛍️ Microsoft Store or using Winget: Winget install Microsoft.PowerShell

Any device that meets the Windows 11 hardware and Virtualization Based Security requirements.

Virtualization technology and Secure Boot enabled in your UEFI settings. Official guide - How to enable Secure Boot on: HP - Lenovo - Dell.

No 3rd party AV installed.

Latest available version of Windows installed.

[!TIP]
Restart your device after applying the hardening measures.

Features

Everything always stays up-to-date with the newest proactive security measures that are industry standards and scalable.

Everything is in clear text, nothing hidden, no 3rd party executable or pre-compiled binary is involved.

No Windows functionality is removed/disabled against Microsoft's recommendations.

All of the links and sources are from official Microsoft websites, straight from the source. No bias, No FUD, No misinformation and definitely No old obsolete methods. That's why there are no links to 3rd party news websites, forums, made up blogs/articles, and such.

With the following exceptions

Link Count	Link	Reason
1	Intel website	i7 13700k product page
1	state.gov	List of State Sponsors of Terrorism
1	orpa.princeton.edu	OFAC Sanctioned Countries
2	Wikipedia	TLS - providing additional information
1	UK Cyber Security Centre	TLS - providing additional information
1	Security.Stackexchange Q&A	TLS - providing additional information
1	browserleaks.com/tls	TLS - Browser test
1	clienttest.ssllabs.com	TLS - Browser test
1	scanigma.com/knowledge-base	TLS - providing additional information
1	cloudflare.com/ssl/reference/	TLS - providing additional information
1	github.com/ssllabs/research/	TLS - providing additional information
1	Wayback Machine	Providing additional information about Edge Browser

The module primarily uses Group policies, the Microsoft recommended way of configuring Windows. It also uses PowerShell cmdlets where Group Policies aren't available, and finally uses a few registry keys to configure security measures that can neither be configured using Group Policies nor PowerShell cmdlets. This is why the module doesn't break anything or cause unwanted behavior.

This Readme page lists all of the security measures applied by the module.

When a hardening measure is no longer necessary because it's applied by default by Microsoft on new builds of Windows, it will also be removed from the module in order to prevent any problems and because it won't be necessary anymore.

The module can be run infinite number of times, it's made in a way that it won't make any duplicate changes.

Applying these hardening measures makes your PC compliant with Microsoft Security Baselines and Secured-core PC specifications (provided that you use modern hardware that supports the latest Windows security features) - See what makes a Secured-core PC - Check Device Guard category for more info

Secured-core – recommended for the most sensitive systems and industries like financial, healthcare, and government agencies. Builds on the previous layers and leverages advanced processor capabilities to provide protection from firmware attacks.

Since I originally created this repository for myself and people I care about, I always maintain it to the highest possible standard.

If you have multiple accounts on your device, you only need to apply the hardening measures 1 time with Admin privileges, that will make system-wide changes. Then you can optionally run the module, without Admin privileges, for each standard user to apply the Non-Admin category.

[!WARNING]
For your own security, exercise caution when considering any other 3rd-party tools, programs, or scripts claiming to harden or modify Windows OS in any way. Verify their legitimacy thoroughly before use and after each release. Avoid blind trust in 3rd party Internet sources. Additionally, if they don't adhere to the rules mentioned above, they can cause system damage, unknown issues, and bugs.

Read the Trust section to see how you can 100% Trust this repository.

How are Group Policies for this module created and maintained?

💡 (back to top)

Hardening Categories

From Top to bottom in order:

Commands that require Administrator Privileges (click/tap on each of these to see in-depth info)
Commands that don't require Administrator Privileges
- Non-Admin Commands

Indicator	Description
	Security measure is applied using PowerShell cmdlets or Registry
	Security measure is applied using Group Policies
	CSP for the security measure
	Sub-category - prompts for additional confirmation

💡 (back to top)

Microsoft Security Baselines

A security baseline is a group of Microsoft-recommended configuration settings that explains their security implications. These settings are based on feedback from Microsoft security engineering teams, product groups, partners, and customers.

Continue reading in the official documentation

Optional Overrides for Microsoft Security Baselines

Highly recommended to apply these overrides, the module will ask you whether you want to apply them or not. Use Optional Overrides when applying the hardening measures on Azure VMs.

💡 (back to categories)

Microsoft 365 Apps Security Baselines

The security baseline for Microsoft 365 Apps for enterprise is published twice a year, usually in June and December.

More info in Microsoft Learn

Microsoft Security Baselines Version Matrix

💡 (back to categories)

Microsoft Defender

Extends the Cloud Security Scan time to the maximum amount of 60 seconds, by default it is 10 seconds. You need to be aware that this means actions like downloading and opening an unknown file will make Microsoft Defender send samples of it to the Cloud for more advanced analysis and it can take a maximum of 60 seconds from the time you try to open that unknown file to the time when it will be opened (if deemed safe). CSP
- Here is an example of the notification you will see in Windows 11 if that happens.

Configures the Cloud Block/Protection Level to the maximum level of Zero Tolerance and Block At First Sight. No unknown file can run on your system without first being recognized by the Microsoft's Security Graph and other globally omniscient systems. CSP

Configures the Microsoft Defender to send all samples automatically. Increasing protection by participating in the SpyNet / MAPS network. CSP

Sets the SpyNet membership to Advanced, improving Cloud Protection. CSP

Enables file hash computation; designed to allow admins to force the anti-malware solution to "compute file hashes for every executable file that is scanned if it wasn't previously computed" to "improve blocking for custom indicators in Microsoft Defender Advanced Threat Protection (Microsoft Defender ATP). CSP

Clears Quarantined items after 1 day instead of the default behavior of keeping them indefinitely. CSP

Allows Microsoft Defender to download security updates even on a metered connection. CSP

Enables Microsoft Defender to scan mapped network drives during full scan. CSP

Enables Microsoft Defender to scan emails. The engine will parse the mailbox and mail files. CSP

Enables Microsoft Defender to scan Removable Drives. CSP

Enables Microsoft Defender to scan Reparse Points. CSP

Forces Microsoft Defender to scan network files. CSP

Sets the Signature Update Interval to every 3 hours instead of automatically. CSP
- Change logs for security intelligence updates
- Configure and validate Microsoft Defender Antivirus network connections
- Security intelligence updates for Microsoft Defender Antivirus and other Microsoft antimalware
- Microsoft Safety Scanner
- Paste the following PowerShell code to retrieve the latest available online versions of the Platform, Signatures, and Engine for Microsoft Defender
- ```
$X = irm "https://www.microsoft.com/security/encyclopedia/adlpackages.aspx?action=info"
@{Engine = $X.versions.engine; Signatures = $X.versions.signatures.'#text'; Platform = $X.versions.platform} | ft -AutoSize
```

Forces Microsoft Defender to check for new virus and spyware definitions before it runs a scan. CSP

Makes Microsoft Defender run catch-up scans for scheduled quick scans. A computer can miss a scheduled scan, usually because the computer is off at the scheduled time, but now after the computer misses two scheduled quick scans, Microsoft Defender runs a catch-up scan the next time someone logs onto the computer. CSP

Enables Network Protection of Microsoft Defender CSP

Enables scanning of restore points CSP

Makes sure Async Inspection for Network protection of Microsoft Defender is turned on - Network protection now has a performance optimization that allows Block mode to start asynchronously inspecting long connections after they're validated and allowed by SmartScreen, which might provide a potential reduction in the cost that inspection has on bandwidth and can also help with app compatibility problems. CSP

Enables Smart App Control (if it's in Evaluation mode): adds significant protection from new and emerging threats by blocking apps that are malicious or untrusted. Smart App Control also helps to block potentially unwanted apps, which are apps that may cause your device to run slowly, display unexpected ads, offer extra software you didn't want, or do other things you don't expect.
- Smart App Control is User-Mode (and enforces Kernel-Mode) App Control for Business, more info in the Wiki. You can see its status in System Information and enable it manually from Microsoft Defender app's GUI. It is very important for Windows and Windows Defender intelligence updates to be always up-to-date in order for Smart App Control to work properly as it relies on live intelligence and definition data from the cloud and other sources to make a Smart decision about programs and files it encounters.
- Smart App Control uses ISG (Intelligent Security Graph). The ISG isn't a "list" of apps. Rather, it uses the same vast security intelligence and machine learning analytics that power Microsoft Defender SmartScreen and Microsoft Defender Antivirus to help classify applications as having "known good", "known bad", or "unknown" reputation. This cloud-based AI is based on trillions of signals collected from Windows endpoints and other data sources and processed every 24 hours. As a result, the decision from the cloud can change.
- Smart App Control can block a program entirely from running or only some parts of it in which case your app or program will continue working just fine most of the time. It's improved a lot since it was introduced, and it continues doing so. Consider turning it on after clean installing a new OS and fully updating it.
- Smart App Control enforces the Microsoft Recommended Driver Block rules and the Microsoft Recommended Block Rules
- Once you turn Smart App Control off, it can't be turned on without resetting or reinstalling Windows.

Enables "Send optional diagnostic data" because it is required for Smart App Control to operate when it's in evaluation mode or turned on, and for communication with Intelligent Security Graph (ISG). This setting will be automatically applied if Smart App Control is already turned on or you choose to turn it on. CSP

Enables Controlled Folder Access. It helps protect your valuable data from malicious apps and threats, such as ransomware. Controlled folder access protects your data by checking apps against a list of known, trusted apps. Due to the recent wave of global ransomware attacks, it is important to use this feature to protect your valuables files, specially OneDrive folders. CSP
- If it blocks a program from accessing one of your folders it protects, and you absolutely trust that program, then you can add it to exclusion list using Microsoft Defender GUI or PowerShell. you can also query the list of allowed apps using PowerShell (commands below). with these commands, you can backup your personalized list of allowed apps, that are relevant to your system, and restore them in case you clean install your Windows.
- The root of the OneDrive folders of all the user accounts will be added to the protected folders list of Controlled Folder Access, to provide Ransomware protection for the entire OneDrive folder. CSP
- ```
# Add multiple programs to the exclusion list of Controlled Folder Access
Add-MpPreference -ControlledFolderAccessAllowedApplications 'C:\Program Files\App\app.exe','C:\Program Files\App2\app2.exe'
```
- ```
# Get the list of all allowed apps
(Get-MpPreference).ControlledFolderAccessAllowedApplications
```

Enables Mandatory ASLR, It might cause compatibility issues only for some poorly-made 3rd party programs, specially portable ones. CSP
- Automatically detects and excludes the Git executables of GitHub Desktop and Git (Standalone version) from mandatory ASLR if they are installed on the system. More info here
- You can add Mandatory ASLR override for a trusted program using the PowerShell command below or in the Program Settings section of Exploit Protection in Microsoft Defender app.
  - Set-ProcessMitigation -Name "C:\TrustedApp.exe" -Disable ForceRelocateImages

Applies Exploit Protections/Process Mitigations from this list to the following programs: CSP
- All channels of Microsoft Edge browser
- Quick Assist app
- Some System processes
- Microsoft 365 apps
- More apps and processes will be added to the list over time once they are properly validated to be fully compatible.
- Exploit Protection configurations are also accessible in XML format within this repository. When implementing exploit protections using an XML file, the existing exploit mitigations will seamlessly integrate rather than being overwritten. Should there be pre-existing exploit protections applied to an executable on the system, and the XML file specifies different mitigations for the same executable, these protections will be merged and applied collectively.

Turns on Data Execution Prevention (DEP) for all applications, including 32-bit programs. By default, the output of BCDEdit /enum "{current}" (in PowerShell) for the NX bit is OptIn but this module sets it to AlwaysOn

Check for the latest virus and spyware security intelligence on startup. CSP

Specifies the maximum depth to scan archive files to the maximum possible value of 4,294,967,295 CSP

Defines the maximum size of downloaded files and attachments to be scanned and set it to the maximum possible value of 10,000,000 KB or 10 GB. the default is 20480 KB or ~20MB CSP

Enables the Enhanced Phishing Protection service. CSP

Enables notifying user of malicious and phishing scenarios in Microsoft Defender Enhanced Phishing Protection. CSP

Enables the feature in Enhanced Phishing Protection in Microsoft Defender SmartScreen that warns users if they reuse their work or school password. CSP

Enables warning users if they type their work or school passwords in unsafe apps. CSP

Enables automatic data collection (formerly known as Capture Threat Window) of Enhanced Phishing Protection in Microsoft Defender SmartScreen for security analysis from a suspicious website or app. CSP

Creates scheduled task for fast weekly Microsoft recommended driver block list update.. You won't see this prompt if the task already exists and is enabled or running.

Set Microsoft Defender engine and platform update channel to beta. CSP CSP

Defines the number of days before spyware security intelligence is considered out of date to 2. The default is 7. CSP

Defines the number of days before virus security intelligence is considered out of date to 2. The default is 7. CSP

Sets the default action for Severe and High threat levels to Remove, for Medium and Low threat levels to Quarantine. CSP

Configures real-time protection and Security Intelligence Updates to be enabled during OOBE. CSP

Enables the Intel TDT (Intel® Threat Detection Technology) integration with Microsoft Defender. CSP

Disables Performance Mode - Security risks in relation to Dev Drive CSP

Enables a network protection setting that blocks malicious network traffic instead of displaying a warning. CSP

Configures the Brute-Force Protection to use cloud aggregation to block IP addresses that are over 99% likely malicious CSP

Configures the Brute-Force Protection to detect and block attempts to forcibly sign in and initiate sessions CSP

Sets the internal feature logic to determine blocking time for the Brute-Force Protections CSP

Configures the Remote Encryption Protection to use cloud intel and context, and block when confidence level is above 90%. CSP

Configures the Remote Encryption Protection to detect and block attempts to replace local files with encrypted versions from another device CSP

Sets the internal feature logic to determine blocking time for the Remote Encryption Protection CSP

Extends the brute-force protection coverage in the Microsoft Defender Antivirus to block local network addresses. CSP

Enables ECS Configurations in the Microsoft Defender. They improve product health and security by automatically fixing any possible issues/bugs that may arise, in a timely manner.

Enables Network Protection to be configured into block or audit mode on Windows Server. CSP

[!TIP]
Performance analyzer for Microsoft Defender Antivirus

💡 (back to categories)

Attack surface reduction rules

Attack surface reduction rules target certain software behaviors, such as: CSP

Launching executable files and scripts that attempt to download or run files
Running obfuscated or otherwise suspicious scripts
Performing behaviors that apps don't usually initiate during normal day-to-day work

Such software behaviors are sometimes seen in legitimate applications. However, these behaviors are often considered risky because they are commonly abused by attackers through malware. Attack surface reduction rules can constrain software-based risky behaviors and help keep your organization safe.

Reducing your attack surface means protecting your devices and network, which leaves attackers with fewer ways to perform attacks. Configuring attack surface reduction rules in Windows can help!

[!TIP]
all 19 available Attack Surface Reduction rules shown in the official chart will be enabled. The Harden Windows Security application also allows you to individually configure each Attack Surface Reduction rule.

💡 (back to categories)

Bitlocker Settings

The module sets up and configures Bitlocker using official documentation, with the most secure configuration and military grade encryption algorithm, XTS-AES-256, to protect the confidentiality and integrity of all information at rest. CSP CSP
- It offers 2 security levels for OS drive encryption: Enhanced and Normal.
- In Normal security level, the OS drive is encrypted with TPM and Startup PIN. This provides very high security for your data, specially with a PIN that's long, complicated (uppercase and lowercase letters, symbols, numbers, spaces) and isn't the same as your Windows Hello PIN.
- In Enhanced security level, the OS drive is encrypted with TPM and Startup PIN and Startup key. This provides the highest level of protection by offering Multifactor Authentication. You will need to enter your PIN and also plug in a flash drive, containing a special BitLocker key, into your device in order to unlock it. Continue reading more about it here.
- Once the OS drive is encrypted, for every other non-OS drive, there will be prompts for confirmation before encrypting it. The encryption will use the same algorithm as the OS drive and uses Auto-unlock key protector. Removable flash drives are skipped.
- The recovery information of all of the drives are saved in a single well-formatted text file in the root of the OS drive C:\BitLocker-Recovery-Info-All-Drives.txt. It's very important to keep it in a safe and reachable place as soon as possible, e.g., in OneDrive's Personal Vault which requires additional authentication to access. See here and here for more info. You can use it to unlock your drives if you ever forget your PIN, lose your Startup key (USB Flash Drive) or TPM no longer has the correct authorization (E.g., after a firmware change).
- TPM has special anti-hammering logic which prevents malicious user from guessing the authorization data indefinitely. Microsoft defines that maximum number of failed attempts in Windows is 32 and every single failed attempt is forgotten after 2 hours. This means that every continuous two hours of powered on (and successfully booted) operation without an event which increases the counter will cause the counter to decrease by 1. You can view all the details using this PowerShell command: Get-TPM.
- Check out Lock Screen category for more info about the recovery password and the 2nd anti-hammering mechanism.
- BitLocker will bring you a real security against the theft of your device if you strictly abide by the following basic rules:
  - As soon as you have finished working, either Hibernate or shut Windows down and allow for every shadow of information to disappear from RAM within 2 minutes. This practice is recommended in High-Risk Environments.
  - Do not mix 3rd party encryption software and tools with Bitlocker. Bitlocker creates a secure end-to-end encrypted ecosystem for your device and its peripherals, this secure ecosystem is backed by things such as software, Virtualization Technology, TPM 2.0 and UEFI firmware, Bitlocker protects your data and entire device against real-life attacks and threats. You can encrypt your external SSDs and flash drives with Bitlocker too.

[!IMPORTANT]
AMD Zen 2 and 3 CPUs have a vulnerability in them, if you use one of them, make sure your Bitlocker Startup PIN is at least 16 characters long (max is 20).

Enables or disables DMA protection from Bitlocker Countermeasures based on the status of Kernel DMA protection. Kernel DMA Protection is not compatible with other BitLocker DMA attacks countermeasures. It is recommended to disable the BitLocker DMA attacks countermeasures if the system supports Kernel DMA Protection (this module does that exactly). Kernel DMA Protection provides higher security bar for the system over the BitLocker DMA attack countermeasures, while maintaining usability of external peripherals. you can check the status of Kernel DMA protection using this official guide. CSP
- Kernel DMA Protection (Memory Access Protection) for OEMs page shows the requirements for Kernel DMA Protection. for Intel CPUs, support for requirements such as VT-X and VT-D can be found in each CPU's respective product page. e.g. Intel i7 13700K

Disallows standard (non-Administrator) users from changing the Bitlocker Startup PIN or password CSP

Requires you to choose a PIN that contains at least 10 characters CSP

(Only on Physical machines) Enables Hibernate and adds Hibernate to Start menu's power options. CSP
- Devices that support Modern Standby have the most security because (S1-S3) power states which belong to the legacy sleep modes are not available. In Modern Standby, security components remain vigilant and the OS stays protected. Applying Microsoft Security Baselines also automatically disables the legacy (S1-S3) sleep states.

sets Hibernate to full

Enables network connectivity in standby on modern standby-capable systems. This ensures security updates for Microsoft Defender and Windows will be installed automatically. CSP

Disallows access to Bitlocker-protected removable data drives from earlier versions of Windows.

Refer to this official documentation about the countermeasures of Bitlocker

💡 (back to categories)

Device Guard

Virtualization-Based Security + UEFI Lock CSP
- Validate enabled Windows Defender Device Guard hardware-based security features

Secure boot (without requiring DMA protection) for Virtualization-Based Security CSP
- This is in accordance with Microsoft's recommendation. This option provides Secure Boot with as much protection as is supported by a given computer’s hardware. A computer with input/output memory management units (IOMMUs) will have Secure Boot with DMA protection. A computer without IOMMUs will simply have Secure Boot enabled.
- Secure boot has 2 parts, part 1 is enforced using the Group Policy by this module, but for part 2, you need to enable Secure Boot in your UEFI firmware settings if it's not enabled by default (which is the case on older hardware).
- (Kernel) DMA protection hardware requirements

Virtualization-based protection of Code Integrity + UEFI Lock CSP

Require UEFI Memory Attributes Table (MAT) CSP

Windows Defender Credential Guard + UEFI Lock CSP
- Windows Defender Device Guard and Windows Defender Credential Guard hardware readiness tool
- Windows Defender Credential Guard requirements

System Guard Secure Launch and SMM protection (Firmware Protection) CSP
- How to verify System Guard Secure Launch is configured and running

Kernel Mode Hardware Enforced Stack Protection

Local Security Authority (LSA) process Protection + UEFI Lock CSP

Enables Machine Identity Isolation Configuration in Enforcement mode.

Enables Virtualization-based Security and Memory Integrity in Mandatory mode.
- Special care should be used before enabling this mode, since, in case of any failure of the virtualization modules, the system will refuse to boot.

[!TIP]
Most of the Device Guard and Virtualization-Based Security features are Automatically enabled by default on capable and modern hardware. The rest of them will be enabled and configured to the most secure state after you apply the Microsoft Security Baselines and the Harden Windows Security policies.

Check out Secured-Core PC requirements.

Memory integrity and VBS enablement

[!IMPORTANT]
About UEFI Lock

UEFI locked security measures are rooted in Proof of Physical Presence and they can't be disabled by modifying Group Policy, registry keys or other Administrative tasks. The only way to disable UEFI locked security measures is to have physical access to the computer, reboot and access the UEFI settings, supply the credentials to access the UEFI, turn off Secure Boot, reboot the system and then you will be able to disable those security measures with Administrator privileges.

[!NOTE]
Device Protection in Windows Security Gives You One of These 4 Hardware Scores

Standard hardware security not supported

This means that your device does not meet at least one of the requirements of Standard Hardware Security.

Your device meets the requirements for Standard Hardware Security.

TPM 2.0

Secure boot

DEP

UEFI MAT

Your device meets the requirements for Enhanced Hardware Security

TPM 2.0

Secure boot

DEP

UEFI MAT

Memory Integrity

Your device has all Secured-core PC features enabled

TPM 2.0

Secure boot

DEP

UEFI MAT

Memory Integrity

System Management Mode (SMM)

💡 (back to categories)

TLS Security

Changes made by this category only affect things that use Schannel SSP: that includes IIS web server, built-in inbox Windows apps and some other programs supplied by Microsoft, including Windows network communications, but not 3rd party software that use portable stacks like Java, nodejs, python or php.

If you want to read more: Demystifying Schannel

[!NOTE]
This category checks whether Battle.net client is installed on the system and if it is then includes TLS_RSA_WITH_AES_256_CBC_SHA as an additional cipher suite in the policy due to a known issue. The way Battle.net client is detected is by checking the presence of Battle.net.exe or Battle.net Launcher.exe in C:\Program Files (x86)\Battle.net\ folder.

Disables TLS 1 and TLS 1.1 security protocols that only exist for backward compatibility. All modern software should and do use TLS 1.2 and TLS 1.3. CSP CSP

Disables MD5 Hashing Algorithm that is only available for backward compatibility

Disables the following weak ciphers that are only available for backward compatibility: "DES 56-bit","RC2 40-bit","RC2 56-bit","RC2 128-bit","RC4 40-bit","RC4 56-bit","RC4 64-bit","RC4 128-bit","3DES 168-bit (Triple DES 168)"

Configures the TLS to only use the following secure cipher suites and in this exact order: CSP

TLS_CHACHA20_POLY1305_SHA256
TLS_AES_256_GCM_SHA384
TLS_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256

Configures TLS ECC Curves to use the following prioritized Curves order: CSP

nistP521
curve25519
NistP384
NistP256

By default, in Windows, the order is this:

curve25519
NistP256
NistP384

Read more in this Wiki post

💡 (back to categories)

Lock Screen

Automatically locks device after X seconds of inactivity (just like mobile phones), which is set to 120 seconds (2 minutes) in this module, you can change that to any value you like. CSP

Requires CTRL+ALT+DEL on the lock screen, kernel protected set of key strokes. The reason and logic behind it is: CSP
- A malicious user might install malware that looks like the standard sign-in dialog box for the Windows operating system and capture a user's password. The attacker can then sign into the compromised account with whatever level of user rights that user has.

Enables a security anti-hammering feature that sets a threshold of 5 for the number of failed sign-in attempts that causes the device to be locked by using BitLocker. Sign-in attempts include Windows password or Windows Hello authentication methods. This threshold means, if the specified maximum number of failed sign-in attempts is exceeded, the device will invalidate the Trusted Platform Module (TPM) protector and any other protector except the 48-digit recovery password, and then reboot. During Device Lockout mode, the computer or device only boots into the touch-enabled Windows Recovery Environment (WinRE) until an authorized user enters the recovery password to restore full access.
- This module (in the Bitlocker category) automatically saves the 48-digit recovery password of each drive in itself, the location of it will also be visible on the PowerShell console when you run it. It is very important to keep it in a safe and reachable place, e.g. in OneDrive's Personal Vault which requires authentication to access. See Here and Here for more info about OneDrive's Personal Vault

Configures account lockout policy: Account lockout threshold, Sets the number of allowed failed sign-in attempts to 5. In combination with other policies in this category, this means every 5 failed sign-in attempts will need a full day to pass before 5 more attempts can be made, otherwise Bitlocker will engage, system will be restarted and 48-digit Bitlocker code will be asked. This policy greatly prevents brute force attempts. CSP

Configures account lockout policy: Sets Account lockout duration to 1440 minutes or 1 day. In combination with other policies in this category, this means every 5 failed sign-in attempts will need a full day to pass before 5 more attempts can be made, otherwise Bitlocker will engage, system will be restarted and 48-digit Bitlocker code will be asked. CSP

Configures account lockout policy: Sets Reset account lockout counter to 1440 minutes or 1 day. In combination with other policies in this category, this means every 5 failed sign-in attempts will need a full day to pass before 5 more attempts can be made, otherwise Bitlocker will engage, system will be restarted and 48-digit Bitlocker code will be asked. CSP

Hides email address of the Microsoft account on lock screen, if your device is in a trusted place like at home then this isn't necessary.

Don't display username at sign-in; If a user signs in as Other user, the full name of the user isn't displayed during sign-in. In the same context, if users type their email address and password at the sign-in screen and press Enter, the displayed text "Other user" remains unchanged, and is no longer replaced by the user's first and last name, as in previous versions of Windows 10. Additionally, if users enter their domain user name and password and click Submit, their full name isn't shown until the Start screen displays. CSP
- Useful If you have devices that store sensitive data, with monitors displayed in unsecured locations, or if you have devices with sensitive data that are remotely accessed, revealing logged on user's full names or domain account names

Don't display last signed-in; This security policy setting determines whether the name of the last user to sign in to the device is displayed on the Secure Desktop. If this policy is enabled, the full name of the last user to successfully sign in isn't displayed on the Secure Desktop, nor is the user's sign-in tile displayed. Additionally, if the Switch user feature is used, the full name and sign-in tile aren't displayed. The sign-in screen requests both Username + Windows Hello credentials. CSP
- This feature can be useful to enable if you live in High-Risk Environments and you don't want anyone to get any information about your accounts when you aren't logged-in.
- This policy will prevent you from using "Forgot my PIN" feature in lock screen or logon screen. If you forget your PIN, you won't be able to recover it.
- If you use Windows Hello Face or Fingerprint, you can easily login using those credential providers without the need to supply username first.

Don't Display Network Selection UI on Lock Screen (like WIFI Icon); This setting allows you to control whether anyone can interact with available networks UI on the logon screen. Once enabled, the device's network connectivity state cannot be changed without signing into Windows. Suitable for High-Risk Environments. CSP

Applies the following PIN Complexity rules to Windows Hello CSP. Please note that, by default, any character can be set as a PIN. However, the following policies ensure that certain characters are always included as a minimum requirement.
- Must include digits CSP
- Expires every 180 days (default behavior is to never expire) CSP
  - Setting an expiration date ensures that, in the event of theft, a threat actor cannot indefinitely attempt to guess the PIN. After 180 days, the PIN expires, rendering it unusable even if guessed correctly. To reset the PIN, authentication via a Microsoft account or EntraID—likely inaccessible to the attacker—will be required. Combined with anti-hammering and BitLocker policies, this expiration guarantees that a threat actor cannot endlessly persist in guessing the PIN.
- History of the 1 most recent selected PIN is preserved to prevent the user from reusing it CSP
- Must include lower-case letters CSP

💡 (back to categories)

User Account Control

Prompt for elevation of privilege on secure desktop for all binaries in Administrator accounts, which presents the sign-in UI and restricts functionality and access to the system until the sign-in requirements are satisfied. The secure desktop's primary difference from the user desktop is that only trusted processes running as SYSTEM are allowed to run here (that is, nothing is running at the user's privilege level). The path to get to the secure desktop from the user desktop must also be trusted through the entire chain. CSP
- Default Behavior: Prompt for consent for non-Windows binaries: When an operation for a non-Microsoft application requires elevation of privilege, the user is prompted on the secure desktop to select either Permit or Deny. If the user selects Permit, the operation continues with the user's highest available privilege.
- Harden Windows Security Behavior: When an operation requires elevation of privilege, the user is prompted on the secure desktop to select either Permit or Deny. If the user selects Permit, the operation continues with the user's highest available privilege.

Only elevate executables that are signed and validated by enforcing cryptographic signatures on any interactive application that requests elevation of privilege. One of the Potential impacts of it is that it can prevent certain poorly designed programs from prompting for UAC. CSP

Hides the entry points for Fast User Switching. CSP
- This policy will prevent you from using "Forgot my PIN" feature in lock screen or logon screen. If you forget your PIN, you won't be able to recover it.

Sets the behavior of the elevation prompt for Standard users to Prompt for Credentials on the Secure Desktop. CSP

Configures the type of Admin Approval Mode to be Admin Approval Mode with enhanced privilege protection.

💡 (back to categories)

Windows Firewall

Sets the Network Location of all connections to Public; Public network means less trust to other network devices.

Makes sure Windows Firewall is enabled for all profiles (which is the default) CSP CSP CSP

Sets inbound and outbound default actions for Domain Firewall Profile to Block; because this module is Not intended to be used on devices that are part of a domain or controlled by an Active Directory Domain Controller, since they will have their own policies and policy management systems in place. CSP CSP

Enables Windows Firewall logging for Domain, Private and Public profiles, sets the log file size for each of them to the max 32.767 MB. Defines separate log files for each of the firewall profiles. Logs only dropped packets for Private and Public profiles, Logs both dropped and successful packets for Domain profile. CSP CSP CSP CSP CSP CSP CSP CSP CSP

Disables Multicast DNS (mDNS) UDP-in Firewall Rules for all 3 Firewall profiles, This might interfere with Miracast screen sharing, which relies on the Public profile, and homes where the Private profile is not selected, but it does add an extra measure of security in public places, like a coffee shop.
- The domain name .local which is used in mDNS (Multicast DNS) is a special-use domain name reserved by the Internet Engineering Task Force (IETF) so that it may not be installed as a top-level domain in the Domain Name System (DNS) of the Internet.

💡 (back to categories)

Optional Windows Features

The module disables the following rarely used features in Windows optional features (Control Panel):
- PowerShell v2: because it's old and doesn't support AMSI.
- Work Folders client: not used when your computer is not part of a domain or enterprise network.
- Internet Printing Client: used in combination with IIS web server, old feature, can be disabled without causing problems further down the road.
- Windows Media Player (legacy): isn't needed anymore, Windows 11 has a modern media player app.
- Microsoft Defender Application Guard, it's deprecated. Learn more about Microsoft Edge Security Features here.

Uninstalls these optional features (Windows Settings -> Apps -> Optional Features):
- Notepad (system): legacy Notepad program. Windows 11 has multi-tabbed modern Notepad app.
- VBSCRIPT: a legacy deprecated scripting engine component, Microsoft does not recommend using this component unless and until it is really required.
- Internet Explorer mode for Edge browser: It's only used by a few possible organizations that have very old internal websites.
- WMIC: Old and deprecated, not secure and is in Microsoft recommended block rules.
- WordPad: Old and deprecated. None of the new features of Word documents are supported in it. Recommended to use Word Online, Notepad or M365 Word.
- PowerShell ISE: Old PowerShell environment that doesn't support versions above 5.1. Highly recommended to use Visual Studio Code for PowerShell usage and learning. You can even replicate the ISE experience in Visual Studio Code. You can access Visual Studio Code online in your browser without the need to install anything.
- Steps Recorder: it's deprecated.

Enables these optional features (Control Panel):
- Windows Sandbox: install, test and use programs in a disposable virtual operation system, completely separate from your main OS
- Hyper-V: a great hybrid hypervisor (Type 1 and Type 2) to run virtual machines on. check out this Hyper-V Wiki page

💡 (back to categories)

Windows Networking

Disables NetBIOS over TCP/IP on all network interfaces.

Disables Smart Multi-Homed Name Resolution because it uses NetBIOS and LLMNR, protocols that shouldn't be used anymore. CSP

Disables LMHOSTS lookup protocol on all network adapters, legacy feature that's not used anymore.

Disables Printing over HTTP because HTTP is not encrypted and it's an old feature that's not used anymore. CSP

Clears all the entries in Remotely accessible registry paths.

Clears all the entries in Remotely accessible registry paths and subpaths.

Sets the minimum required SMB version for Client to 3.1.1 which is the latest available version at the moment and was introduced years ago with Windows 10.

Sets the minimum required SMB version for Server to 3.1.1 which is the latest available version at the moment and was introduced years ago with Windows 10.

Blocks NTLM completely for SMB.

Requires encryption for SMB client/workstation.

Enables encryption for SMB Server. Its status can be checked using the following PowerShell command: (get-SmbServerConfiguration).EncryptData. If the returned value is $True then SMB Encryption is turned on.

Enables QUIC for SMB Client.

Enables QUIC for SMB Server.

Configures the Cipher Suites from the default value of AES_128_GCM,AES_128_CCM,AES_256_GCM,AES_256_CCM to AES_256_GCM,AES_256_CCM,AES_128_GCM,AES_128_CCM for the SMB Client. CSP

Configures the Cipher Suites from the default value of AES_128_GCM,AES_128_CCM,AES_256_GCM,AES_256_CCM to AES_256_GCM,AES_256_CCM,AES_128_GCM,AES_128_CCM for the SMB Server. CSP

💡 (back to categories)

Miscellaneous Configurations

Sets Early launch antimalware engine's status to 8 which is Good only. The default value is 3, which allows good, unknown and 'bad but critical'. that is the default value, because setting it to 8 can prevent your computer from booting if the driver it relies on is critical but at the same time unknown or bad. CSP
- By being launched first by the kernel, ELAM is ensured to be launched before any third-party software and is therefore able to detect malware in the boot process and prevent it from initializing. ELAM drivers must be specially signed by Microsoft to ensure they are started by the Windows kernel early in the boot process.

Disables location services (Location, Windows Location Provider, Location Scripting) system wide. Websites and apps won't be able to use your precise location, however they will still be able to detect your location using your IP address. CSP CSP CSP

Enables svchost.exe mitigations. built-in system services hosted in svchost.exe processes will have stricter security policies enabled on them. These stricter security policies include a policy requiring all binaries loaded in these processes to be signed by Microsoft, and a policy disallowing dynamically generated code. CSP
- Requires Business Windows licenses. e.g., Windows 11 pro for Workstations, Enterprise or Education.

Turns on Enhanced mode search for Windows indexer. The default is classic mode. CSP
- This causes some UI elements in the search settings in Windows settings to become unavailable for Standard user accounts to view, because it will be a managed feature by an Administrator.

Enforce the Administrator role for adding printer drivers CSP

Enables SMB/LDAP Signing CSP CSP

Enables Edge browser (stable/beta/dev channels) to download and install updates on any network, metered or not; because the updates are important and should not be suppressed.

Enables all Windows users to use Hyper-V and Windows Sandbox by adding all Windows users to the "Hyper-V Administrators" security group using its SID. By default, only Administrators can use Hyper-V or Windows Sandbox.

Creates custom views for Windows Event Viewer to help keep tabs on important security events:
- Attack Surface Reduction Rules
- Controlled Folder Access
- Exploit Protection
- Network Protection
- MSI and Scripts for WDAC Auditing
- Sudden Shut down events (due to power outage)
- Code Integrity Operational (WDAC)
- Restarts (By user or by the System/Apps)
- Workstation Locks and Unlocks
- Checks to make sure Other Logon/Logoff Events Audit is active CSP
- Failed Login attempts via PIN at lock screen
  - Error/Status code 0xC0000064 indicates wrong PIN entered at lock screen
- USB storage Connects & Disconnects (Flash drives, phones etc.)

Enables WinVerifyTrust Signature Validation, a security feature related to WinVerifyTrust function that handles Windows Authenticode signature verification for portable executable (PE) files.

Enables Command line process auditing. CSP

Enables the RPC Endpoint Mapper Client Authentication policy. CSP

Enables a policy that requests claims and compound authentication for Dynamic Access Control and Kerberos armoring. CSP

Enables Windows Protected Print. CSP

Configures the SSH client's configurations to use the following secure MACs (Message Authentication Codes): MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com.

Enables support for long paths.

Force strong key protection for user keys stored on the computer. User is prompted when the key is first used.

💡 (back to categories)

Windows Update Configurations

Windows updates are extremely important. They always should be installed as fast as possible to stay secure and if a reboot is required, it should be done immediately. Threat actors can weaponize publicly disclosed vulnerabilities the same day their POC (Proof-Of-Concept) is released..

In Windows by default, devices will scan daily, automatically download and install any applicable updates at a time optimized to reduce interference with usage, and then automatically try to restart when the end user is away.

The following policies the module configures make sure the default behavior explained above is tightly enforced.

Enables Windows Update to download and install updates on any network, metered or not; because the updates are important and should not be suppressed, that's what bad actors would want. CSP

Enables "Receive Updates for other Microsoft products" (such as PowerShell)

Enables "Notify me when a restart is required to finish updating". CSP

Specifies the number of days before quality updates are installed on devices automatically to 1 day. CSP

Specifies the number of days before feature updates are installed on devices automatically to 1 day. CSP

Sets the number of grace period days before feature updates are installed on devices automatically to 1 day. CSP

Sets the number of grace period days before quality updates are installed on devices automatically to 1 day. CSP

Configures the automatic updates to happen every day, automatically be downloaded and installed, notify users for restart. CSP

Enables features introduced via servicing that are off by default so that users will be able to get new features after having Windows Update settings managed by Group Policy as the result of running this category. CSP

💡 (back to categories)

Edge Browser configurations

Block 3rd party cookies - Recommendatory policy

Set Edge to use system's DNS over HTTPS

Automatic HTTPS upgrade of HTTP connections

Enable Encrypted Client Hello

Disable Basic HTTP authentication scheme

Allow devices using this hardening category to receive new features and experimentations like normal devices

Enforce the audio process to run sandboxed

Sets the share additional operating system region setting to never - Recommendatory policy

Disables the following weak Cipher Suites
- Site 1 to test TLS in your browser
- Site 2 to test TLS in your browser

TLS_RSA_WITH_AES_256_CBC_SHA  Reason: NO Perfect Forward Secrecy, CBC, SHA1
TLS_RSA_WITH_AES_128_CBC_SHA  Reason: NO Perfect Forward Secrecy, CBC, SHA1
TLS_RSA_WITH_AES_128_GCM_SHA256  Reason: NO Perfect Forward Secrecy
TLS_RSA_WITH_AES_256_GCM_SHA384  Reason: NO Perfect Forward Secrecy
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA  Reason: CBC, SHA1
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA  Reason: CBC, SHA1

Due to security reasons, many policies cannot be used when you are signed into Edge browser using personal Microsoft account. This module does not use any of those policies. When those policies are applied, they are ignored by the browser and edge://policy/ shows an error for them.

You can view all of the policies being applied to your Edge browser by visiting this page: edge://policy/
You can find all of the available internal Edge pages in here: edge://about/

Useful links:

💡 (back to categories)

Certificate Checking Commands

In this category, the module downloads and runs sigcheck64.exe from Sysinternals, then lists valid certificates not rooted to the Microsoft Certificate Trust List in the User and Machine certificate stores. Except for some possible Microsoft certificates, Windows insider builds certificates or certificates that have your own computer's name, which are perfectly safe and should not be deleted, All other certificates that will be listed should be treated as dangerous and removed from your system immediately.

💡 (back to categories)

Country IP Blocking

The module uses the newest range of IPv4 and IPv6 addresses of State Sponsors of Terrorism and OFAC Sanctioned Countries, directly from official IANA sources repository, then creates 2 rules (inbound and outbound) for each list in Windows firewall, completely blocking connections to and from those countries.

Once you have those Firewall rules added, you can use this method to see if any of the blocked connections were from/to those countries.

[!NOTE]
Threat actors can use VPN, VPS etc. to mask their originating IP address and location. So don't take this category as the perfect solution for network protection.

💡 (back to categories)

Downloads Defense Measures

To combat the threat of more sophisticated malware, a preemptive measure is taken by creating and deploying a WDAC policy on the system. This policy blocks the execution of executables and other potentially harmful file types in the Downloads folder, using the WDACConfig module.

This policy defends the system from malware that can launch itself automatically after being downloaded from the Internet. The user must ensure the file's safety and explicitly transfer it to a different folder before running it.

The WDAC policy employs a wildcard pattern to prevent any file from running in the Downloads folder. Additionally, it verifies that the system downloads folder in the user directory matches the downloads folder in the Edge browser's settings. If there is a discrepancy, a warning message is displayed on the console.

Creates a custom WDAC policy that blocks the execution of the following executables:

wscript.exe
mshta.exe
cscript.exe

They are insecure, unsandboxed script hosts that pose a security risk.

All of the policies can be easily removed using the Unprotect-WindowsSecurity or Remove-WDACConfig cmdlets.

💡 (back to categories)

Non-Admin Commands

You don't need Admin privileges to run this category, because no system-wide changes is made. Changes in this category only apply to the current user account that is running the PowerShell session.

Shows known file extensions in File explorer

Shows hidden files, folders and drives (toggles the control panel folder options item)

Disables websites accessing local language list - good for privacy

Turns off safe search in Windows search, will enable +18 content to appear in searches; essentially toggles the button in: Windows settings > privacy and security > search permissions > safe search

Enables Clipboard History

Turns on text suggestions when typing on the physical keyboard

Turns on "Multilingual text suggestions" for the current user, toggles the option in Windows settings

Turns off sticky key shortcut of pressing shift key 5 times fast

Disables Show reminders and incoming VoIP calls on the lock screen

💡 (back to categories)

Azure DevOps Repository (mirror)

Harden Windows Security website

Official global IANA IP block for each country

Windows Security Blog

WinSecureDNSMgr

Privacy, Anonymity and Compartmentalization

Trust

How can you 100% trust this repository and know that nothing shady is going on?

This repository uses effective methods that make it easy to verify:

Change log history is present on GitHub. (Despite some of my awkward documentation typos)
Artifact attestations are used to establish provenance for builds. It guarantees that the package(s) you download from this repository are 100% created from the source code that exist in this repository.
SBOMs (Software Bill of Materials) are generated for the entire repository to comply with data protection standards and providing transparency. Together with attestation they provide SLSA L2 security level for the build process. In the future, the workflows will be upgraded to comply with SLSA L3 level.
You can open the files in Visual Studio Code / Visual Studio Code Web / GitHub CodeSpace, and view them in a nice and easy to read environment, they are well formatted, commented and indented.
Commits are verified either with my GPG key or SSH key and Vigilant mode is turned on in my GitHub account.
You can fork this repository, verify it until that point in time, then verify any subsequent changes/updates I push to this repository, at your own pace (using Sync fork and Compare options on your fork), and if you are happy with the changes, allow it to be merged with your own copy/fork on your GitHub account.
Explanations for some of the files used by the Harden Windows Security module, ask about any other file(s) if you have questions, they are all in clear text.
- Registry.csv includes some of the security measures' registry data.
- ProcessMitigations.csv includes the process mitigations data.
- Default Security Policy.inf contains security policy data used during unprotect actions to restore defaults.
- Registry resources.csv Includes the data used for compliance checking.
- Harden-Windows-Security.ps1 is the boot-strapper for the Harden Windows Security module.
How Are Group Policies Used by the Harden Windows Security Module?
How are Group Policies for this module created and maintained?
How to verify Security-Baselines-X directory and 100% trust it?

[!TIP]
All files in this repository are zipped and automatically submitted to VirusTotal for scanning. Any available packages in the last release is also directly uploaded for scanning. It is done through a GitHub Action that is triggered every time a release is made or a PR is merged. Find the history of the uploaded files in my Virus Total profile.

💡 (back to top)

Support

If you have any questions, requests, suggestions etc. about this GitHub repository and its content, please open a new discussion or Issue.

Reporting a vulnerability on this GitHub repository.

I can also be reached privately at: spynetgirl@outlook.com

💡 (back to top)

Security Recommendations

Always download your operation system from official Microsoft websites. Right now, Windows 11 is the latest version of Windows, its ISO file can be downloaded from this official Microsoft server. One of the worst things you can do to your own security and privacy is downloading your OS, which is the root of all the active and passive security measures, from a 3rd party website claiming they have the official unmodified files. There are countless bad things that can happen as the result of it such as threat actors embedding malware or backdoors inside the customized OS, or pre-installing customized root CA certificates in your OS so that they can perform TLS termination and view all of your HTTPS and encrypted Internet data in plain clear text, even if you use VPN. Having a poisoned and compromised certificate store is the endgame for you, and that's just the tip of the iceberg.
- Refer to Wiki to see how to create Bootable USB flash drive with no 3rd party tools

Whenever you want to install a program or app, first use the Microsoft Store or Winget, if the program or app you are looking for isn't available in there, then download it from its official website. Somebody created a nice web interface for interacting with Winget CLI here. Using Winget or Microsoft store provides many benefits:
- Microsoft store UWP apps are secure in nature, digitally signed, in MSIX format. That means, installing and uninstalling them is guaranteed and there won't be any leftovers after uninstalling.
- Microsoft store has Win32 apps too, they are traditional .exe installers that we are all familiar with. The store has a library feature that makes it easy to find the apps you previously installed.
- Both Microsoft and Winget check the hash of the files by default, if a program or file is tampered, they will warn you and block the installation, whereas when you manually download a program from a website, you will have to manually verify the file hash with the hash shown on the website, if any.

Use Secure DNS; Windows 11 natively supports DNS over HTTPS and DNS over TLS.
- Use my WinSecureDNSMgr module to easily configure DNS over HTTPS in Windows

Only use Microsoft Edge for browser; It has the Highest-rated protection against phishing and malware, available by default on Windows OS, has tightly integrated valuable Security features such as Microsoft Defender Application Guard, Microsoft Defender SmartScreen, Hardware Enforced Stack Protection, Arbitrary Code Guard (ACG), Control Flow Guard (CFG), Tracking Prevention and Trusted built-in Secure Network feature from Cloudflare just to name a few.

Always enable Two-Factor/Multi-Factor Authentication on websites, apps and services that you use. Preferably, use Microsoft Authenticator app which has backup and restore feature, so you never lose access to your TOTPs (Time-Based One-Time Passwords) even if you lose your phone. Available for Android and IOS.

Make sure OneDrive backup for important folders (Desktop/Documents/Pictures) is enabled. It is fast, secure and works in any network condition and since it's x64 (64-bit), it can handle a Lot of small and large files simultaneously.

When considering the use of a VPN, it is crucial to exercise discernment and only resort to it when absolutely necessary. A VPN can be a vital tool if you reside in a totalitarian, communist, or dictatorial regime, or in a nation where democratic principles are not upheld. However, if you live in a country that does not fall into these categories, it may be wise to reconsider the necessity of using a VPN. Your local ISP (Internet Service Provider) is likely more trustworthy than the ISP associated with a remote VPN server. By using a VPN, you are merely transferring the trust you place in your local ISP to an unknown entity—the ISP utilized by the VPN provider. It is important not to be swayed by the deceptive marketing tactics employed by VPN companies. The true identities, political affiliations, backgrounds, and loyalties of those behind these services often remain shrouded in mystery. In the permissive and open societies of the Western world, it is conceivable that a VPN service could be established by entities with questionable intentions, including state sponsors of terrorism or other hostile actors. Such services could be utilized to gather intelligence, conduct data mining, and track users, posing significant risks to your privacy and security.
- There are situations where using VPN can provide security and privacy. For example, when using a public WiFi hotspot or basically any network that you don't have control over. In such cases, use Cloudflare WARP which uses WireGuard protocol, or as mentioned, use Secure Network in Edge browser that utilizes the same secure Cloudflare network. It's free, it's from an American company that has global radar and lots of insight about countries in the world in real-time, at least 19.7% of all websites use it (2022). Safe to say it's one of the backbones of the Internet.

Go passwordless with your Microsoft account and use Windows Hello authentication. In your Microsoft account which has Outlook service, you can create up to 10 Email aliases in addition to the 1 Email address you get when you made your Microsoft account, that means without creating a new account, you can have 11 Email addresses all of which will use the same inbox and account. You can specify which one of those Email aliases can be used to sign into your account, in the sign in preferences of your Microsoft account settings. So for example, when going passwordless, if you need you can give one of your Email aliases to others for communication or add it to a public profile of yours, then block sign in using that Email alias so nobody can send you authenticator notifications by entering that Email alias in the sign in page, and use the other 10 aliases that are private to sign into your Microsoft account with peace of mind. You can create a rule in your Outlook so that all of the Emails sent to your public Email alias will be stored in a different folder, apart from your other inbox emails. All of this can be done using free Microsoft account and Outlook webapp.

Set a strong password for the UEFI firmware of your device so that it will ask for password before allowing any changes to be made to firmware. You can also configure the password to be required on startup.

Use NTFS (which is the default Filesystem in Windows) or ReFS (Resilient File System, newer). In addition to all their benefits, they support Mark Of The Web (MOTW) or zone.identifier. When a file is downloaded to a device running Windows, Mark of the Web is added to the file, identifying its source as being from the internet. You can read all the information about it in here. If your USB flash drive is formatted as FAT32, change it to NTFS, because FAT32 does not keep the MOTW of the files. If the file you are downloading is compressed in .zip format, make sure you open/extract it using Windows built-in support for .zip files because it keeps the MOTW of the files. If the compressed file you downloaded is in other formats such as .7zip or .rar, make sure you use an archive program that supports keeping the mark of the Web of files after extraction. One of those programs is NanaZip which is a fork of 7zip, available in Microsoft Store and GitHub, compared to 7zip, it has better and modern GUI, and the application is digitally signed. After installation, open it, navigate to Tools at the top then select Options, set Propagate zone.id stream to Yes. You can use this PowerShell command to find all the info about the Zone Identifier of the files you downloaded from the Internet.

Get-Content <Path-To-File> -stream zone.identifier

When using Xbox, make sure you configure sign-in preference and set it to either Ask for my PIN or Lock it down. The latter is the most secure one since it will require authentication using Microsoft Authenticator app. Ask for my PIN is recommended for the most people because it will only require a PIN to be entered using controller.

A few reminders about open source programs:
- Unless you are a skilled programmer who can understand and verify every line of code in the source, and spends time to personally build the software from the source, and repeats all the aforementioned tasks for each subsequent version, then seeing the source code won't have any effect on you because you aren't able to understand nor verify it.
- Do not assume that the entire Open Source community audits and verifies every line of code just because the source code is available, as we've seen in the XZ utility's backdoor by state sponsored actors, they can have backdoors implanted in them in broad daylight and nobody might notice it for a long time.
- The majority of open source programs are unsigned, meaning they don't have a digital signature, their developers haven't bought and used a code signing certificate to sign their program. Among other problems, this might pose a danger to the end-users by making it harder to create trust for those programs in security solutions such as Application Control or App Whitelisting, and makes it hard to authenticate them. Read Microsoft's Introduction to Code Signing. Use Azure Trusted Signing which is affordable.

Use Microsoft account (MSA) or Microsoft Entra ID to sign into Windows. Never use local administrators. Real security is achieved when there is no local administrator and identities are managed using Entra ID. You will be able to enforce Multi-factor unlock, for example use PIN + Fingerprint or PIN + Facial recognition, to unlock your device.

Enable Random Hardware Addresses In Windows Settings -> Network & Internet -> WIFI. Currently, there is no Group Policy or associated registry key to automatically turn it on, that is why it is mentioned here in the security recommendations section. It has various security and privacy benefits such as your device cannot be uniquely identified by its hardware Mac address and the routers your connect to cannot uniquely identify you. You can set it to change daily in your WIFI network adapter's settings in Windows settings for even more benefits.

More Security Recommendations coming soon...

💡 (back to top)

Resources

Microsoft.com
- Force firmware code to be measured and attested by Secure Launch
Microsoft Learn - Technical Documentation
- Secure Launch—the Dynamic Root of Trust for Measurement (DRTM)
- Quick guide to Windows as a service
Germany Intelligence Agency - BND - Federal Office for Information Security
- Analysis of Device Guard
- Device Guard Differential Analysis
Microsoft Tech Community - Official blogs and documentations
Microsoft Security baselines - Security baselines from Microsoft
Microsoft Security Response Center (MSRC) YouTube channel
- BlueHat Seattle 2019 || Guarding Against Physical Attacks: The Xbox One Story
- Security Update Guide: The Microsoft Security Response Center (MSRC) investigates all reports of security vulnerabilities affecting Microsoft products and services, and provides the information here as part of the ongoing effort to help you manage security risks and help keep your systems protected.
- Microsoft Security Response Center Blog
Microsoft Security Blog
Microsoft Podcasts
Bug Bounty Program - With bounties worth up to 250,000$
Microsoft Active Protections Program
Security Update Guide FAQs
Microsoft On the Issues - Assessments, Investigations and Reports of APTs (Advanced Persistent Threats¹) and nation-sponsored cyberattack operations globally
A high level overview paper by Microsoft (in PDF), framework for cybersecurity information sharing and risk reduction.
Microsoft Threat Modeling Tool - for software architects and developers
Important events to monitor
Windows Security portal
Security auditing
Microsoft SysInternals Sysmon for Windows Event Collection or SIEM
Privileged Access Workstations
Enhanced Security Administrative Environment (ESAE)
New Zealand 2016 Demystifying the Windows Firewall – Learn how to irritate attackers without crippli
Download Windows virtual machines ready for development
UK National Cyber Security Centre Advice & guidance
Global threat activity
Microsoft Zero Trust
Understanding malware & other threats, phrases
Malware naming
Microsoft Digital Defense Report
Microsoft Defender for Individuals
Submit a file for malware analysis
Submit a driver for analysis
Service health status
Microsoft Defender Threat Intelligence
- Free community edition vs Premium edition comparison
Microsoft Virus Initiative
Digital Detectives @Microsoft
Australia's Essential Eight
NIST 800-53
DoD's CMMC (Cybersecurity Maturity Model Certification)
ISO 27001
DoD Cyber Stigs (Security Technical Implementation Guides)
NIST SP 800-171 Rev. 2 - Protecting Controlled Unclassified Information in Nonfederal Systems and Organizations
Clean source principle
Windows Message Center
Deprecated features for Windows client
Microsoft Cybersecurity Reference Architectures
BlueHat IL 2023 - David Weston - Default Security
Windows Security best practices for integrating and managing security tools
Microsoft Exploitability Index
The Microsoft Incident Response Ninja Hub

💡 (back to top)

Roadmap

💡 (back to top)

License

Using MIT License. Free information without any paywall or things of that nature. The only mission of this GitHub repository is to give all Windows users accurate, up to date and correct facts and information about how to stay secure and safe in dangerous environments, and to stay not one, but Many steps, ahead of threat actors.

Credits

Some of the icons are from icons8
Windows, Azure etc. are trademarks of Microsoft Corporation

💡 (back to top)

LANDrop/LANDrop

2024-12-01T01:30:24Z

Drop any files to any devices on your LAN.

Note: This repository does not reflect the latest LANDrop releases. We decided to temporarily close source LANDrop and we might re-open source it in the future. Thanks for your understanding!

Drop any files to any devices on your LAN. No need to use instant messaging for that anymore.

LANDrop is a cross-platform tool that you can use to conveniently transfer photos, videos, and other types of files to other devices on the same local network.

You can download prebuilts of LANDrop from the official website.

Features

Cross platform: when we say it, we mean it. iOS, Android, macOS, Windows, Linux, name yours.
Ultra fast: uses your local network for transferring. Internet speed is not a limit.
Easy to use: intuitive UI. You know how to use it when you see it.
Secure: uses state-of-the-art cryptography algorithm. No one else can see your files.
No cellular data: outside? No problem. LANDrop can work on your personal hotspot, without consuming cellular data.
No compression: doesn't compress your photos and videos when sending.

Building

The AppImage we provide as the prebuilt for Linux might not work on your machine. You can build LANDrop by yourself if the prebuilt doesn't work for you.

To build LANDrop:

Download and install the dependencies: Qt and libsodium
If you are using a Debian-based distro, such as Ubuntu, you can install libsodium via
```
sudo apt install libsodium-dev
```

Clone this repository

git clone https://github.com/LANDrop/LANDrop

Run the following commands

mkdir -p LANDrop/build
cd LANDrop/build
qmake ../LANDrop
make -j$(nproc)
sudo make install

You can now run LANDrop via
```
landrop
```

elalish/manifold

2024-12-01T01:30:24Z

Geometry library for topological robustness

About Manifold

C++ Documentation | TS Documentation | Algorithm Documentation | Blog Posts | Web Examples

Manifold is a geometry library dedicated to creating and operating on manifold triangle meshes. A manifold mesh is a mesh that represents a solid object, and so is very important in manufacturing, CAD, structural analysis, etc. Manifold also supports arbitrary vertex properties and enables mapping of materials for rendering use-cases. Our primary goal is reliability: guaranteed manifold output without caveats or edge cases. Our secondary goal is performance: efficient algorithms that make extensive use of parallelization, or pipelining when only a single thread is available.

Users

Here is an incomplete list of our users, whose integrations may be anywhere from in-progress to released. Please feel free to send a PR to update this list with your own project - it's quite difficult for us to keep track.


OpenSCAD	IFCjs	Dactyl Web Configurator
Nomad Sculpt	Grid.Space	badcad
Godot Engine	OCADml	Flitter
BRL-CAD	PolygonJS	Spherene
Babylon.js	trimesh	Gypsum
Valence 3D

Bindings & Packages

If C++ isn't your cup of tea, Manifold also has bindings to many other languages, some maintained in this repository, and others elsewhere.

Language	Packager	Name	Maintenance
C	N/A	N/A	internal
TS/JS	npm	manifold-3d	internal
Python	PyPI	manifold3d	internal
Java	N/A	manifold	external
Clojure	N/A	clj-manifold3d	external
C#	NuGet	ManifoldNET	external
Julia	Packages	ManifoldBindings.jl	external
OCaml	N/A	OManifold	external

Frontend Sandboxes

ManifoldCAD.org

If you like OpenSCAD / JSCAD, you might also like ManifoldCAD - our own solid modelling web app where you script in JS/TS. This uses our npm package, manifold-3d, built via WASM. It's not quite as fast as our raw C++, but it's hard to beat for interoperability.

Python Colab Example

If you prefer Python to JS/TS, make your own copy of the example notebook above. It demonstrates interop between our manifold3d PyPI library and the popular trimesh library, including showing the interactive model right in the notebook and saving 3D model output.

Manifold Library

This library is fast with guaranteed manifold output. As such you need manifold meshes as input, which this library can create using constructors inspired by the OpenSCAD API, as well as a level set function for evaluating signed-distance functions (SDF) that improves significantly over Marching Cubes. You can also pass in your own mesh data, but you'll get an error status if the imported mesh isn't manifold. We provide a Merge function to fix slightly non-manifold meshes, but in general you may need one of the automated repair tools that exist mostly for 3D printing.

The most significant contribution here is a guaranteed-manifold mesh Boolean algorithm, which I believe is the first of its kind. If you know of another, please open a discussion - a mesh Boolean algorithm robust to edge cases has been an open problem for many years. Likewise, if the Boolean here ever fails you, please submit an issue! This Boolean forms the basis of a CAD kernel, as it allows simple shapes to be combined into more complex ones.

Manifold has full support for arbitrary vertex properties, and also has IDs that make it easy to keep track of materials and what surfaces belong to what input objects or faces. See our web example for a simple demonstration of combining objects with unique textures.

Also included are a novel and powerful suite of refining functions for smooth mesh interpolation. They handle smoothing both triangles and quads, as well as keeping polygonal faces flat. You can easily create sharp or small-radius edges where desired, or even drive the curvature by normal vectors.

To aid in speed, this library makes extensive use of parallelization through TBB, if enabled. Not everything is so parallelizable, for instance a polygon triangulation algorithm is included which is serial. Even if compiled with parallel backend, the code will still fall back to the serial version of the algorithms if the problem size is small. The WASM build is serial-only for now, but still fast.

Look in the samples directory for examples of how to use this library to make interesting 3D models. You may notice that some of these examples bare a certain resemblance to my OpenSCAD designs on Thingiverse, which is no accident. Much as I love OpenSCAD, my library is dramatically faster and the code is more flexible.

Dependencies

Manifold no longer has any required dependencies! However, we do have several optional dependencies, of which the first two are strongly encouraged:

Name	CMake Flag	Provides
`TBB`	`MANIFOLD_PAR=ON`	Parallel acceleration
`Clipper2`	`MANIFOLD_CROSS_SECTION=ON`	2D: `CrossSection`
`Assimp`	`MANIFOLD_EXPORT=ON`	Basic Mesh I/O
`Nanobind`	`MANIFOLD_PYBIND=ON`	Python bindings
`Emscripten`	`MANIFOLD_JSBIND=ON`	JS bindings via WASM
`GTest`	`MANIFOLD_TEST=ON`	Testing framework

3D Formats

Please avoid saving to STL files! They are lossy and inefficient - when saving a manifold mesh to STL there is no guarantee that the re-imported mesh will still be manifold, as the topology is lost. Please consider using 3MF instead, as this format was designed from the beginning for manifold meshes representing solid objects.

If you use vertex properties for things like interpolated normals or texture UV coordinates, glTF is recommended, specifically using the EXT_mesh_manifold extension. This allows for the lossless and efficient transmission of manifoldness even with property boundaries. Try our make-manifold page to add this extension to your existing glTF/GLB models.

Manifold provides an optional MeshIO component based on Assimp, but it is limited in functionality and is primarily to aid in testing. If you are using our npm module, we have a much more capable gltf-io.ts you can use instead. For other languages we strongly recommend using existing packages that focus on 3D file I/O, e.g. trimesh for Python, particularly when using vertex properties or materials.

Building

Only CMake, a C++ compiler, and Python are required to be installed and set up to build this library (it has been tested with GCC, LLVM, MSVC). However, a variety of optional dependencies can bring in more functionality, see below.

Build and test (Ubuntu or similar):

git clone --recurse-submodules https://github.com/elalish/manifold.git
cd manifold
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON .. && make
test/manifold_test

CMake flags (usage e.g. -DMANIFOLD_DEBUG=ON):

MANIFOLD_JSBIND=[OFF, <ON>]: Build js binding (when using the emscripten toolchain).
MANIFOLD_CBIND=[<OFF>, ON]: Build C FFI binding.
MANIFOLD_PYBIND=[OFF, <ON>]: Build python binding, requires nanobind.
MANIFOLD_PAR=[<OFF>, ON]: Enables multi-thread parallelization, requires tbb.
MANIFOLD_CROSS_SECTION=[OFF, <ON>]: Build CrossSection for 2D support (needed by language bindings), requires Clipper2.
MANIFOLD_EXPORT=[<OFF>, ON]: Enables MeshIO and GLB export of 3D models from the tests, requires assimp.
MANIFOLD_DEBUG=[<OFF>, ON]: Enables internal assertions and exceptions. This incurs around 20% runtime overhead.
MANIFOLD_TEST=[OFF, <ON>]: Build unit tests, requires GTest.
TRACY_ENABLE=[<OFF>, ON]: Enable integration with tracy profiler. See profiling section below.

Dependency version override:

MANIFOLD_USE_BUILTIN_TBB=[<OFF>, ON]: Use builtin version of tbb.
MANIFOLD_USE_BUILTIN_CLIPPER2=[<OFF>, ON]: Use builtin version of clipper2.
MANIFOLD_USE_BUILTIN_NANOBIND=[<OFF>, ON]: Use builtin version of nanobind.

Note: These three options can force the build to avoid using the system version of the dependency. This will either use the provided source directory via FETCHCONTENT_SOURCE_DIR_* (see below), or fetch the source from GitHub. Note that the dependency will be built as static dependency to avoid dynamic library conflict. When the system package is unavailable, the option will be automatically set to true.

Offline building (with missing dependencies/dependency version override):

MANIFOLD_DOWNLOADS=[OFF, <ON>]: Automatically download missing dependencies. Need to set FETCHCONTENT_SOURCE_DIR_* if the dependency * is missing.
FETCHCONTENT_SOURCE_DIR_TBB: path to tbb source (if MANIFOLD_PAR is enabled).
FETCHCONTENT_SOURCE_DIR_CLIPPER2: path to tbb source (if MANIFOLD_CROSS_SECTION is enabled).
FETCHCONTENT_SOURCE_DIR_NANOBIND: path to nanobind source (if MANIFOLD_PYBIND is enabled).
FETCHCONTENT_SOURCE_DIR_GOOGLETEST: path to googletest source (if MANIFOLD_TEST is enabled).

Note: When FETCHCONTENT_SOURCE_DIR_* is set, CMake will use the provided source directly without downloading regardless of the value of MANIFOLD_DOWNLOADS.

The build instructions used by our CI are in manifold.yml, which is a good source to check if something goes wrong and for instructions specific to other platforms, like Windows.

WASM

To build the JS WASM library, first install NodeJS and set up emscripten:

(on Mac):

brew install nodejs
brew install emscripten

(on Linux):

sudo apt install nodejs
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install latest
./emsdk activate latest
source ./emsdk/emsdk_env.sh

Then build:

cd manifold
mkdir buildWASM
cd buildWASM
emcmake cmake -DCMAKE_BUILD_TYPE=Release .. && emmake make
node test/manifold_test.js

Python

The CMake script will build the python binding manifold3d automatically. To use the extension, please add $BUILD_DIR/bindings/python to your PYTHONPATH, where $BUILD_DIR is the build directory for CMake. Examples using the python binding can be found in bindings/python/examples. To see exported samples, run:

sudo apt install pkg-config libpython3-dev python3 python3-distutils python3-pip
pip install trimesh pytest
python3 run_all.py -e

Run the following code in the interpreter for python binding documentation:

>>> import manifold3d
>>> help(manifold3d)

For more detailed documentation, please refer to the C++ API.

Windows Shenanigans

Windows users should build with -DBUILD_SHARED_LIBS=OFF, as enabling shared libraries in general makes things very complicated.

The DLL file for manifoldc (C FFI bindings) when built with msvc is in ${CMAKE_BINARY_DIR}/bin/${BUILD_TYPE}/manifoldc.dll. For example, for the following command, the path relative to the project root directory is build/bin/Release/manifoldc.dll.

cmake . -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DMANIFOLD_DEBUG=ON -DMANIFOLD_PAR=${{matrix.parallel_backend}} -A x64 -B build

Contributing

Contributions are welcome! A lower barrier contribution is to simply make a PR that adds a test, especially if it repros an issue you've found. Simply name it prepended with DISABLED_, so that it passes the CI. That will be a very strong signal to me to fix your issue. However, if you know how to fix it yourself, then including the fix in your PR would be much appreciated!

Formatting

There is a formatting script format.sh that automatically formats everything. It requires clang-format, black formatter for python and gersemi for formatting cmake files.

Note that our script can run with clang-format older than 18, but the GitHub action check may fail due to slight differences between different versions of clang-format. In that case, either update your clang-format version or apply the patch from the GitHub action log.

Profiling

There is now basic support for the Tracy profiler for our tests. To enable tracing, compile with -DTRACY_ENABLE=on cmake option, and run the test with Tracy server running. To enable memory profiling in addition to tracing, compile with -DTRACY_MEMORY_USAGE=ON in addition to -DTRACY_ENABLE=ON.

About the author

This library was started by Emmett Lalish. I am currently a Google employee and this is my 20% project, not an official Google project. At my day job I'm the maintainer of <model-viewer>. I was the first employee at a 3D video startup, Omnivor, and before that I worked on 3D printing at Microsoft, including 3D Builder. Originally an aerospace engineer, I started at a small DARPA contractor doing seedling projects, one of which became Sea Hunter. I earned my doctorate from the University of Washington in control theory and published some papers.

shader-slang/slang

2024-12-01T01:52:10Z

Making it easier to work with shaders

Slang

Why Slang?

The Slang shading language is designed to enable real-time graphics developers to work with large-scale, high-performance shader code.

Write Shaders Once, Run Anywhere

Access the Latest GPU Features

Leverage Neural Graphics with Automatic Differentiation

Scalable Software Development with Modules

Code Specialization that Works with Modules

Easy On-ramp for HLSL and GLSL Codebases

Slang provides a compatibility module that enables the use of most GLSL intrinsic functions and GLSL's parameter binding syntax.

Comprehensive Tooling Support

Getting Started

See the user-guide for info on using the slangc command-line tool: Slang Command Line Usage.

If you would like to build Slang from source, please consult the build instructions.

Documentation

The Slang project provides a variety of different documentation, but most users would be well served starting with the User's Guide.

For developers writing Slang code, the Slang Core Module Reference provides detailed documentation on Slang's built-in types and functions.

We also provide a few examples of how to integrate Slang into a rendering application.

Additionally, we recommend checking out Vulkan Mini Examples for more examples of using Slang's language features available on Vulkan, such as pointers and the ray tracing intrinsics.

Contributing

If you'd like to contribute to the project, we are excited to have your input. The following guidelines should be observed by contributors:

Please follow the contributor Code of Conduct.
Bugs reports and feature requests should go through the GitHub issue tracker
Changes should ideally come in as small pull requests on top of master, coming from your own personal fork of the project
Large features that will involve multiple contributors or a long development time should be discussed in issues, and broken down into smaller pieces that can be implemented and checked in in stages

Contribution guide describes the workflow for contributors at more detail.

Limitations and Support

Platform support

The Slang compiler and libraries can be built on the following platforms:

Windows	Linux	MacOS	WebAssembly
supported	supported	supported	experimental

Both x86_64 and aarch64 architectures are supported on Windows, Linux and MacOS platforms.

Target support

Slang can compile shader code to the following targets:

Target	Status	Output Formats
Direct3D 11	supported	HLSL
Direct3D 12	supported	HLSL
Vulkan	supported	SPIRV, GLSL
Metal	experimental*	Metal Shading Language
WebGPU	experimental**	WGSL
CUDA	supported	C++ (compute only)
Optix	experimental	C++ (WIP)
CPU	experimental	C++ (kernel), C++ (host), standalone executable, dynamic library

*Slang currently supports generating vertex, fragment, compute, task and mesh shaders for Metal.

**WGSL support is still work in-progress.

For greater detail, see the Supported Compilation Targets section of the User Guide

License

The Slang code itself is under the Apache 2.0 with LLVM Exception license (see LICENSE).

Builds of the core Slang tools depend on the following projects, either automatically or optionally, which may have their own licenses:

glslang (BSD)
lz4 (BSD)
miniz (MIT)
spirv-headers (Modified MIT)
spirv-tools (Apache 2.0)
ankerl::unordered_dense::{map, set} (MIT)

Slang releases may include slang-llvm which includes LLVM under the license:

llvm (Apache 2.0 License with LLVM exceptions)

Some of the tests and example programs that build with Slang use the following projects, which may have their own licenses:

glm (MIT)
stb_image and stb_image_write from the stb collection of single-file libraries (Public Domain)
tinyobjloader (MIT)

bitcoin/bitcoin

2024-12-01T01:52:10Z

Bitcoin Core integration/staging tree

https://bitcoincore.org

For an immediately usable, binary version of the Bitcoin Core software, see https://bitcoincore.org/en/download/.

What is Bitcoin Core?

Bitcoin Core connects to the Bitcoin peer-to-peer network to download and fully validate blocks and transactions. It also includes a wallet and graphical user interface, which can be optionally built.

Further information about Bitcoin Core is available in the doc folder.

License

Bitcoin Core is released under the terms of the MIT license. See COPYING for more information or see https://opensource.org/licenses/MIT.

Development Process

The master branch is regularly built (see doc/build-*.md for instructions) and tested, but it is not guaranteed to be completely stable. Tags are created regularly from release branches to indicate new official, stable release versions of Bitcoin Core.

The https://github.com/bitcoin-core/gui repository is used exclusively for the development of the GUI. Its master branch is identical in all monotree repositories. Release branches and tags do not exist, so please do not fork that repository unless it is for development reasons.

The contribution workflow is described in CONTRIBUTING.md and useful hints for developers can be found in doc/developer-notes.md.

Testing

Testing and code review is the bottleneck for development; we get more pull requests than we can review and test on short notice. Please be patient and help out by testing other people's pull requests, and remember this is a security-critical project where any mistake might cost people lots of money.

Automated Testing

Developers are strongly encouraged to write unit tests for new code, and to submit new unit tests for old code. Unit tests can be compiled and run (assuming they weren't disabled during the generation of the build system) with: ctest. Further details on running and extending unit tests can be found in /src/test/README.md.

There are also regression and integration tests, written in Python. These tests can be run (if the test dependencies are installed) with: build/test/functional/test_runner.py (assuming build is your build directory).

The CI (Continuous Integration) systems make sure that every pull request is built for Windows, Linux, and macOS, and that unit/sanity tests are run automatically.

Manual Quality Assurance (QA) Testing

Changes should be tested by somebody other than the developer who wrote the code. This is especially important for large or high-risk changes. It is useful to add a test plan to the pull request description if testing the changes is not straightforward.

Translations

Changes to translations as well as new translations can be submitted to Bitcoin Core's Transifex page.

Translations are periodically pulled from Transifex and merged into the git repository. See the translation process for details on how this works.

Important: We do not accept translation changes as GitHub pull requests because the next pull from Transifex would automatically overwrite them again.

carla-simulator/carla

2024-12-01T01:52:10Z

Open-source simulator for autonomous driving research.

CARLA Simulator

CARLA is an open-source simulator for autonomous driving research. CARLA has been developed from the ground up to support development, training, and validation of autonomous driving systems. In addition to open-source code and protocols, CARLA provides open digital assets (urban layouts, buildings, vehicles) that were created for this purpose and can be used freely. The simulation platform supports flexible specification of sensor suites and environmental conditions.

[!NOTE] This is the development branch ue4-dev for the Unreal Engine 4.26 version of CARLA. This branch exists in parallel with the Unreal Engine 5.3 version of CARLA, in the ue5-dev branch. Please be sure that this version of CARLA is suitable for your needs as there are significant differences between the UE 4.26 and UE 5.3 versions of CARLA.

Download CARLA

Linux:

Windows:

[!WARNING] The CARLA package downloads are now provided using the BackBlaze CDN. The Amazon Web Service download links have been discontinued. Please ensure you update any relevant information in repositories using the CARLA simulator package versions.

Recommended system

Intel i7 gen 9th - 11th / Intel i9 gen 9th - 11th / AMD ryzen 7 / AMD ryzen 9
+32 GB RAM memory
NVIDIA RTX 3070 / NVIDIA RTX 3080 / NVIDIA RTX 4090
Ubuntu 20.04

Documentation

The CARLA documentation is hosted on ReadTheDocs. Please see the following key links:

CARLA Ecosystem

Repositories associated with the CARLA simulation platform:

CARLA Autonomous Driving leaderboard: Automatic platform to validate Autonomous Driving stacks
Scenario_Runner: Engine to execute traffic scenarios in CARLA 0.9.X
ROS-bridge: Interface to connect CARLA 0.9.X to ROS
Driving-benchmarks: Benchmark tools for Autonomous Driving tasks
Conditional Imitation-Learning: Training and testing Conditional Imitation Learning models in CARLA
AutoWare AV stack: Bridge to connect AutoWare AV stack to CARLA
Reinforcement-Learning: Code for running Conditional Reinforcement Learning models in CARLA
RoadRunner: MATLAB GUI based application to create road networks in the ASAM OpenDRIVE format
Map Editor: Standalone GUI application to enhance RoadRunner maps with traffic lights and traffic signs information

Like what you see? Star us on GitHub to support the project!

Paper

If you use CARLA, please cite our CoRL’17 paper.

CARLA: An Open Urban Driving Simulator
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, Vladlen Koltun; PMLR 78:1-16 [PDF] [talk]

@inproceedings{Dosovitskiy17,
  title = {{CARLA}: {An} Open Urban Driving Simulator},
  author = {Alexey Dosovitskiy and German Ros and Felipe Codevilla and Antonio Lopez and Vladlen Koltun},
  booktitle = {Proceedings of the 1st Annual Conference on Robot Learning},
  pages = {1--16},
  year = {2017}
}

Building CARLA

Clone this repository locally from GitHub:

git clone https://github.com/carla-simulator/carla.git .

Also, clone the CARLA fork of the Unreal Engine into an appropriate location:

git clone --depth 1 -b carla https://github.com/CarlaUnreal/UnrealEngine.git .

Once you have cloned the repositories, follow the instructions for building in Linux or building in Windows.

Contributing

Please take a look at our Contribution guidelines.

F.A.Q.

If you run into problems, check our FAQ.

Licenses

CARLA licenses

CARLA specific code is distributed under MIT License.

CARLA specific assets are distributed under CC-BY License.

CARLA Dependency and Integration licenses

The ad-rss-lib library compiled and linked by the RSS Integration build variant introduces LGPL-2.1-only License.

Unreal Engine 4 follows its own license terms.

CARLA uses three dependencies as part of the SUMO integration:

PROJ, a generic coordinate transformation software which uses the X/MIT open source license.
SQLite, part of the PROJ dependencies, which is in the public domain.
Xerces-C, a validating XML parser, which is made available under the Apache Software License, Version 2.0.

CARLA uses one dependency as part of the Chrono integration:

Eigen, a C++ template library for linear algebra which uses the MPL2 license.

CARLA uses the Autodesk FBX SDK for converting FBX to OBJ in the import process of maps. This step is optional, and the SDK is located here

This software contains Autodesk® FBX® code developed by Autodesk, Inc. Copyright 2020 Autodesk, Inc. All rights, reserved. Such code is provided "as is" and Autodesk, Inc. disclaims any and all warranties, whether express or implied, including without limitation the implied warranties of merchantability, fitness for a particular purpose or non-infringement of third party rights. In no event shall Autodesk, Inc. be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of such code."

Serial-Studio/Serial-Studio

2024-12-01T01:41:09Z

Flexible data visualization software for embedded devices and projects

Serial Studio

Serial Studio is a multi-platform, versatile data visualization tool designed for embedded engineers, students, hackers, and teachers. It allows users to visualize, analyze, and represent data from their projects and devices without needing custom, project-specific visualization software. Serial Studio offers a flexible solution that adapts to a wide range of use cases, making it ideal for both educational and professional environments.

The tool was born out of my experience in multiple CanSat-based competitions, where I often found myself developing new Ground Station Software for each project. Over time, I realized it would be more efficient and sustainable to maintain a single, flexible Ground Station Software that allows users to define how incoming data is processed and displayed.

Today, Serial Studio is a powerful and adaptable tool, suitable not only for CanSat competitions but for any data acquisition and visualization project. It supports data retrieval from a wide range of sources, including hardware and software serial ports, MQTT, Bluetooth Low Energy (BLE), and network sockets (TCP/UDP).

Read this document in other languages: Español 简体中文 Deutsch Русский Français

Features

Cross-platform: Compatible with Windows, macOS, and Linux.
CSV Export: Easily saves received data in CSV files for further analysis or processing.
Support for multiple data sources: Handles a wide variety of sources, including serial ports, MQTT, Bluetooth Low Energy (BLE), and network sockets (TCP/UDP).
Customizable visualization: Allows users to define and display data using various widgets, configurable via the project editor to meet specific needs.
Customizable frame analysis: Provides the option to modify a JavaScript function to interpret incoming data frames, enabling the preprocessing of raw sensor data and handling of complex binary formats.
MQTT publishing and receiving: Sends and receives data over the internet, enabling real-time data visualization from anywhere in the world.

Visit the Wiki for comprehensive guides, including:

Installation Instructions: Set up Serial Studio on Windows, macOS, or Linux.
Quick Start Guide: Learn how to connect your device and visualize data in minutes.
Advanced Topics: Explore data flow, frame parsing, and building custom dashboards.
Examples: Examples with code, projects, and explanations to help you learn Serial Studio.

Development

Requirements

To compile Serial Studio, the only required dependency is Qt, preferrably with all plugins & modules installed. The desktop application compiles with Qt 6.8.0.

If you're compiling on GNU/Linux, you’ll also need to install libgl1-mesa-dev:

sudo apt install libgl1-mesa-dev

Once Qt is installed, you can compile the project by opening the CMakeLists.txt file in your preferred IDE or by using the command line:

mkdir build
cd build 
cmake ../ -DPRODUCTION_OPTIMIZATION=ON -DCMAKE_BUILD_TYPE=Release
cmake --build . -j 16

Support & Tipping

Open source software thrives on collaboration, creativity, and the generosity of its users. By supporting Serial Studio, you are directly contributing to its growth, sustainability, and ability to impact countless developers, makers, educators and innovators around the world.

If this project has helped you or inspired you, consider supporting it with a donation. Even small contributions make a big difference. Your generosity will keep this tool evolving and help more people to unlock their potential with open source software.

👉 Donate/Tip Here

License

This project is licensed under the MIT License. For more details, see the LICENSE file.

OpenXRay/xray-16

2024-12-01T01:41:09Z

Improved version of the X-Ray Engine, the game engine used in the world-famous S.T.A.L.K.E.R. game series by GSC Game World. Join OpenXRay! ;)

OpenXRay

OpenXRay is an improved version of the X-Ray Engine, the game engine used in the world-famous S.T.A.L.K.E.R. game series by GSC Game World.

Goals

Make it a drop-in replacement for original engine.
1. Aim at 99% compatibility and same behaviour, where possible.
2. Compile engine into a single executable file that you can just drop into bin folder. (see #210)
Support all three games in the series: SOC/CS/COP. (see Supported games below)
Fix original S.T.A.L.K.E.R. series bugs.
Introduce a solid platform for modmakers:
1. Add frame/render graph for those who want to add new graphics features.
2. Improve performance via refactoring the code, parallelizing the engine, making it multithreaded.
3. Add new scripting, development and debugging features.
4. New game SDK with new features.
Clean up engine code, make it easily portable to new platforms, minimize platform-specific code.
Enhance player's experience with new graphics, gameplay and other features that can be enabled optionally. (by default, we stay close to vanilla)

Main differences from original X-Ray are:

Support for 64-bit.
Support for ARM, ARM64, E2K (Elbrus 2000), PPC64LE.
Works on Linux, macOS, OSL (Elbrus OS).
New OpenGL renderer. (currently, requires OpenGL 4.1 minimum, lowering to at least OpenGL 3.3 is planned)
Improved performance, better FPS.
Original bugs fixes.
New features for modmakers.
Gamepad support. (not yet finished, but you can try already, see #943)
New game SDK being currently developed. (see Game Editor)

You can see the detailed differences table here

Supported games

OpenXRay is based on X-Ray 1.6.02, used in S.T.A.L.K.E.R.: Call of Pripyat, so initially it supported only this game.
Currently, we are working on support for all three games in the series.

Call of Pripyat	Clear Sky	Shadow of Chernobyl
Yes	Release candidate (see #382). Minor bugs possible, but game is stable finishable.	Not supported yet (see #392)

Documentation:

Make sure to visit our wiki.

How to
Build and setup	On Windows	On Linux
Install and play	On Windows	-

Contributing

All contributions are more than welcomed. There are several ways how you can contribute:

Community

Play and enjoy the game, file an Issue when you encounter any bugs, or you have an enhancement request.

Join us on our Discord, subscribe to our YouTube channel, join our VK group, leave a comment, put a like and communicate there!
Also you can put a star on this repository :)

Development

Join our efforts in making our beloved game better, send pull requests, participate in discussions and code reviews!

It is a place to share ideas on what to implement, gather people that want to work on the engine, and work on the source code. However, the following things should be taken into consideration:

We want to keep the game close to the vanilla, so if you want to introduce new gameplay features, make sure it is optional, and doesn't break compatibility with original game resources (i.e. everything in gamedata folder and .db*/.xdb archives). You also may want to add non-gameplay features, fix bugs, or improve engine performance and code quality.
Major changes should be discussed before implementation.

Take a look at our Issues page:

See issues labeled as good first issue to get familiar with the engine code in practice.
You may also want to look at issues labeled as help wanted. Some of them are difficult ones, though.

The dev branch is the default and base branch for the project. It is used for development, and all pull requests should go there. But be aware that this branch sometimes may be broken, and we can only rarely do force pushes to this branch.

Be advised that this is a community project not sanctioned by GSC Game World in any way – and they remain the copyright holders of all the original source code and S.T.A.L.K.E.R. franchise. However, they know about many community projects, including this, and support the S.T.A.L.K.E.R. community efforts to make the game better.

Funding

You may provide financial support for this project by donating via different ways:

Boosty – a large part of the team is located in Russia, if you have an ability to donate through Boosty, please use it, since we don't have an ability to withdraw funds from services like Patreon, etc. to our local Russian banking cards/accounts.
GitHub Sponsors, Patreon, Open Collective – funds raised from these services will be used to support our developers outside of Russia, and also we may use them to pay for paid services on GitHub, AppVeyor, etc.
BTC: 363ZUoWcQe9fDvRPK9Kee2YuPdyhSFQpr2
ETH: 0x45a4fe8566e76946591e1eeabf190aa09b1cdb66
TRX: TGx7QAhTPsRcwnb4mwCtNDU7NF6kuoACpt
Please, contact @xottab_duty in our Discord if you would like to use another cryptocurrency.

Thank you for your support!

Thanks

GSC Game World – for creating S.T.A.L.K.E.R. and supporting the community;
Loxotron – for making the engine sources available;
All the OpenXRay contributors – for making the project better.
- The first OpenXRay team (2014-2017) – for being at the origins of the project.
  - nitrocaster – original project founder.
  - Kaffeine – initial work on the Linux port, refactoring, polishing.
  - Armada651 – creation of the OpenGL renderer, work on the build system, other project maintenance work.
  - andrew-boyarshin – work on the build system.
  - Swartz27 – work on renderer features.
  - awdavies – project maintenance work.
- The second OpenXRay team (2017-now) – for continuing work on the project.
  - Xottab_DUTY – current project leader.
  - intorr – work on the project quality. (memory leaks, refactoring, optimizations)
  - eagleivg – main part of the work on Linux port.
  - q4a – main part of the work on Linux port.
  - SkyLoader – OpenGL renderer improvements and polishing, other project work.
  - qweasdd136963 – supporting the OXR_COC project (Call of Chernobyl port to latest OpenXRay), other project work on new features, refactoring and bug fixing.
  - JohnDoe_71Rus – our regular tester.
  - Chip_exe – work on Linux port, maintaining AUR package, our regular tester.
  - a1batross – work on Linux port.
  - The Sin! – new features, refactoring, bug fixing polishing.
  - Zegeri – work on Linux port, code quality, fixes, polishing.
  - drug007 – work on Linux port.
  - vTurbine – work on renderer unification, refactoring, polishing.
  - Zigatun – work on ARM port.
  - Masterkatze – work on the build system, bug fixing.
  - Chugunov Roman – work on porting Call of Chernobyl to latest OpenXRay, extending functionality for modmakers.
  - yohjimane – work on original game bugs fixes and new features.
- Other contributors:
  - alexgdi – work on organizing project infrastructure, external dependencies.
  - NeoAnomaly – help with debug functionality on Windows.
  - RainbowZerg – work on the renderer features, bug fixing.
  - FozeSt – help with some fixes and features.
  - mrnotbadguy – work on gamepads support and bug fixing.
  - devnexen – work on FreeBSD support and portability.
  - ZeeWanderer – work on the build system.
  - GeorgeIvlev – work on the build system, bug fixing.
  - TmLev – work on code quality and Docker support.
  - Plotja – work on new gameplay features, bug fixes, portability, polishing.
  - dimhotepus – work on code quality.
  - HeapRaid – work on renderer cleanup, code quality, portability.
  - Vertver – work on macOS support.
  - Lnd-stoL – work on macOS support.
  - GermanAizek – work on code quality, finding and fixing vanilla bugs.
  - dasehak – work on FreeBSD support, finding and fixing vanilla bugs.
Particular projects:
- Oxygen – for being our friends and giving tips and help with new features, optimizations, bug fixes, etc.
- Shoker Weapon Mod and Shoker – for contributing new features, bug fixing.
- Im-Dex – for the work on the engine.
- OGSR – for amazing work on Shadow of Chernobyl.
- Call of Chernobyl and its contributors – for useful new features, bug fixes and optimizations.
- Lost Alpha – for their effort on restoring the old game concept.
- Lost Alpha DC – for continuing work on Lost Alpha.
Individuals:
- tamlin-mike – for work on the build system.
- Vincent – for work on the Linux port.
- abramcumner – for useful fixes and additions.
- Morrey – for work on Clear Sky support and his Return to Clear Sky mod.
Companies:
- CoderGears – thanks for providing a free Pro Licence for CppDepend, an amazing and powerful tool for C and C++.
- PVS-Studio LLC – thanks for proving us a free licence for PVS-Studio, a static analyzer for C, C++, C#, and Java code.

If your work is being used in our project and you are not mentioned here or in the contributors page, please, write to us and we will add you. Or send us a pull request with you added to this list ;)

alibaba/MNN

2024-12-01T01:41:09Z

MNN is a blazing fast, lightweight deep learning framework, battle-tested by business-critical use cases in Alibaba

中文版本

MNN Homepage

Intro

MNN is a highly efficient and lightweight deep learning framework. It supports inference and training of deep learning models and has industry-leading performance for inference and training on-device. At present, MNN has been integrated into more than 30 apps of Alibaba Inc, such as Taobao, Tmall, Youku, DingTalk, Xianyu, etc., covering more than 70 usage scenarios such as live broadcast, short video capture, search recommendation, product searching by image, interactive marketing, equity distribution, security risk control. In addition, MNN is also used on embedded devices, such as IoT.

MNN-LLM is a large language model runtime solution developed based on the MNN engine. The mission of this project is to deploy LLM models locally on everyone's platforms(Mobile Phone/PC/IOT). It supports popular large language models such as Qianwen, Baichuan, Zhipu, LLAMA, and others. MNN-LLM User guide

MNN-Diffusion is a stable diffusion model runtime solution developed based on the MNN engine. The mission of this project is to deploy stable diffusion models locally on everyone's platforms. MNN-Diffusion User guide

Inside Alibaba, MNN works as the basic module of the compute container in the Walle System, the first end-to-end, general-purpose, and large-scale production system for device-cloud collaborative machine learning, which has been published in the top system conference OSDI’22. The key design principles of MNN and the extensive benchmark testing results (vs. TensorFlow, TensorFlow Lite, PyTorch, PyTorch Mobile, TVM) can be found in the OSDI paper. The scripts and instructions for benchmark testing are put in the path “/benchmark”. If MNN or the design of Walle helps your research or production use, please cite our OSDI paper as follows:

@inproceedings {proc:osdi22:walle,
    author = {Chengfei Lv and Chaoyue Niu and Renjie Gu and Xiaotang Jiang and Zhaode Wang and Bin Liu and Ziqi Wu and Qiulin Yao and Congyu Huang and Panos Huang and Tao Huang and Hui Shu and Jinde Song and Bin Zou and Peng Lan and Guohuan Xu and Fei Wu and Shaojie Tang and Fan Wu and Guihai Chen},
    title = {Walle: An {End-to-End}, {General-Purpose}, and {Large-Scale} Production System for {Device-Cloud} Collaborative Machine Learning},
    booktitle = {16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)},
    year = {2022},
    isbn = {978-1-939133-28-1},
    address = {Carlsbad, CA},
    pages = {249--265},
    url = {https://www.usenix.org/conference/osdi22/presentation/lv},
    publisher = {USENIX Association},
    month = jul,
}

Documentation and Workbench

MNN's docs are in place in Read the docs.

You can also read docs/README to build docs's html.

MNN Workbench could be downloaded from MNN's homepage, which provides pretrained models, visualized training tools, and one-click deployment of models to devices.

Key Features

Lightweight

Optimized for devices, no dependencies, can be easily deployed to mobile devices and a variety of embedded devices.
iOS platform: static library size will full option for armv7+arm64 platforms is about 12MB, size increase of linked executables is about 2M.
Android platform: core so size is about 800KB (armv7a - c++_shared).
Using MNN_BUILD_MINI can reduce package size by about 25%, with a limit of fixed model input size
Support FP16 / Int8 quantize, can reduce model size 50%-70%

Versatility

Supports Tensorflow, Caffe, ONNX,Torchscripts and supports common neural networks such as CNN, RNN, GAN, Transformer.
Supports AI model with multi-inputs or multi-outputs, every kind of dimension format, dynamic inputs, controlflow.
MNN supports approximate full OPs used for the AI Model. The converter supports 178 Tensorflow OPs, 52 Caffe OPs, 163 Torchscripts OPs, 158 ONNX OPs.
Supports iOS 8.0+, Android 4.3+, and embedded devices with POSIX interface.
Supports hybrid computing on multiple devices. Currently supports CPU and GPU.

High performance

Implements core computing with lots of optimized assembly code to make full use of the ARM / x64 CPU.
Use Metal / OpenCL / Vulkan to support GPU inference on mobile.
Use CUDA and tensorcore to support NVIDIA GPU for better performance
Convolution and transposition convolution algorithms are efficient and stable. The Winograd convolution algorithm is widely used to better symmetric convolutions such as 3x3,4x4,5x5,6x6,7x7.
Twice speed increase for the new architecture ARM v8.2 with FP16 half-precision calculation support. 2.5 faster to use sdot for ARM v8.2 and VNNI.

Ease of use

Support use MNN's OP to do numerical calculating like numpy.
Support lightweight image process module like OpenCV, which is only 100k.
Support build model and train it on PC / mobile.
MNN Python API helps ML engineers to easily use MNN to infer, train, and process images, without dipping their toes in C++ code.

The Architecture / Precision MNN supported is shown below:

S ：Support and work well, deeply optimized, recommend to use
A ：Support and work well, can use
B ：Support but has bug or not optimized, no recommend to use
C ：Not Support

Architecture / Precision		Normal	FP16	BF16	Int8
CPU	Native	B	C	B	B
	x86/x64-SSE4.1	A	B	B	A
	x86/x64-AVX2	S	B	B	A
	x86/x64-AVX512	S	B	B	S
	ARMv7a	S	S (ARMv8.2)	S	S
	ARMv8	S	S (ARMv8.2)	S(ARMv8.6)	S
GPU	OpenCL	A	S	C	C
	Vulkan	A	A	C	C
	Metal	A	S	C	C
	CUDA	A	S	C	C
NPU	CoreML	B	B	C	C
	HIAI	B	C	C	B
	NNAPI	B	B	C	C

Tools

Base on MNN (Tensor compute engine), we provided a series of tools for inference, train and general computation.

MNN-Converter: Convert other models to MNN models for inference, such as Tensorflow(lite), Caffe, ONNX, Torchscripts. And do graph optimization to reduce computation.
MNN-Compress: Compress model to reduce size and increase performance / speed
MNN-Express: Support model with controlflow, use MNN's OP to do general-purpose computing.
MNN-CV: An OpenCV-like library, but based on MNN and then much more lightweight.
MNN-Train: Support train MNN model.

How to Discuss and Get Help From the MNN Community

The group discussions are predominantly Chinese. But we welcome and will help English speakers.

Dingtalk discussion groups:

Group #1 (Full): 23329087

Group #2 (Full): 23350225

Group #3: QR code:

Historical Paper

The preliminary version of MNN, as mobile inference engine and with the focus on manual optimization, has also been published in MLSys 2020. Please cite the paper, if MNN previously helped your research:

@inproceedings{alibaba2020mnn,
  author = {Jiang, Xiaotang and Wang, Huan and Chen, Yiliu and Wu, Ziqi and Wang, Lichuan and Zou, Bin and Yang, Yafeng and Cui, Zongyang and Cai, Yu and Yu, Tianhang and Lv, Chengfei and Wu, Zhihua},
  title = {MNN: A Universal and Efficient Inference Engine},
  booktitle = {MLSys},
  year = {2020}
}

License

Apache 2.0

Acknowledgement

MNN participants: Taobao Technology Department, Search Engineering Team, DAMO Team, Youku and other Alibaba Group employees.

MNN refers to the following projects:

ashvardanian/SimSIMD

2024-12-01T01:30:13Z

Up to 200x Faster Dot Products & Similarity Metrics — for Python, Rust, C, JS, and Swift, supporting f64, f32, f16 real & complex, i8, and bit vectors using SIMD for both AVX2, AVX-512, NEON, SVE, & SVE2 📐

Computing dot-products, similarity measures, and distances between low- and high-dimensional vectors is ubiquitous in Machine Learning, Scientific Computing, Geo-Spatial Analysis, and Information Retrieval. These algorithms generally have linear complexity in time, constant or linear complexity in space, and are data-parallel. In other words, it is easily parallelizable and vectorizable and often available in packages like BLAS (level 1) and LAPACK, as well as higher-level numpy and scipy Python libraries. Ironically, even with decades of evolution in compilers and numerical computing, most libraries can be 3-200x slower than hardware potential even on the most popular hardware, like 64-bit x86 and Arm CPUs. Moreover, most lack mixed-precision support, which is crucial for modern AI! The rare few that support minimal mixed precision, run only on one platform, and are vendor-locked, by companies like Intel and Nvidia. SimSIMD provides an alternative. 1️⃣ SimSIMD functions are practically as fast as memcpy. 2️⃣ Unlike BLAS, most kernels are designed for mixed-precision and bit-level operations. 3️⃣ SimSIMD often ships more binaries than NumPy and has more backends than most BLAS implementations, and more high-level interfaces than most libraries.

Features

SimSIMD (Arabic: "سيمسيم دي") is a mixed-precision math library of over 350 SIMD-optimized kernels extensively used in AI, Search, and DBMS workloads. Named after the iconic "Open Sesame" command that opened doors to treasure in Ali Baba and the Forty Thieves, SimSimd can help you 10x the cost-efficiency of your computational pipelines. Implemented distance functions include:

Euclidean (L2) and Cosine (Angular) spatial distances for Vector Search. docs
Dot-Products for real & complex vectors for DSP & Quantum computing. docs
Hamming (~ Manhattan) and Jaccard (~ Tanimoto) bit-level distances. docs
Set Intersections for Sparse Vectors and Text Analysis. docs
Mahalanobis distance and Quadratic forms for Scientific Computing. docs
Kullback-Leibler and Jensen–Shannon divergences for probability distributions. docs
Fused-Multiply-Add (FMA) and Weighted Sums to replace BLAS level 1 functions. docs
For Levenshtein, Needleman–Wunsch, and Smith-Waterman, check StringZilla.
🔜 Haversine and Vincenty's formulae for Geospatial Analysis.

Moreover, SimSIMD...

handles float64, float32, float16, and bfloat16 real & complex vectors.
handles int8 integral, int4 sub-byte, and b8 binary vectors.
handles sparse uint32 and uint16 sets, and weighted sparse vectors.
is a zero-dependency header-only C 99 library.
has Python, Rust, JS, and Swift bindings.
has Arm backends for NEON, Scalable Vector Extensions (SVE), and SVE2.
has x86 backends for Haswell, Skylake, Ice Lake, Genoa, and Sapphire Rapids.
with both compile-time and runtime CPU feature detection easily integrates anywhere!

Due to the high-level of fragmentation of SIMD support in different x86 CPUs, SimSIMD generally uses the names of select Intel CPU generations for its backends. They, however, also work on AMD CPUs. Intel Haswell is compatible with AMD Zen 1/2/3, while AMD Genoa Zen 4 covers AVX-512 instructions added to Intel Skylake and Ice Lake. You can learn more about the technical implementation details in the following blog-posts:

Benchmarks

NumPy	C 99	SimSIMD
cosine distances between 1536d vectors in `int8`
🚧 overflows	x86: 10,548,600 ops/s arm: 11,379,300 ops/s	x86: 16,151,800 ops/s arm: 13,524,000 ops/s
cosine distances between 1536d vectors in `bfloat16`
🚧 not supported	x86: 119,835 ops/s arm: 403,909 ops/s	x86: 9,738,540 ops/s arm: 4,881,900 ops/s
cosine distances between 1536d vectors in `float16`
x86: 40,481 ops/s arm: 21,451 ops/s	x86: 501,310 ops/s arm: 871,963 ops/s	x86: 7,627,600 ops/s arm: 3,316,810 ops/s
cosine distances between 1536d vectors in `float32`
x86: 253,902 ops/s arm: 46,394 ops/s	x86: 882,484 ops/s arm: 399,661 ops/s	x86: 8,202,910 ops/s arm: 3,400,620 ops/s
cosine distances between 1536d vectors in `float64`
x86: 212,421 ops/s arm: 52,904 ops/s	x86: 839,301 ops/s arm: 837,126 ops/s	x86: 1,538,530 ops/s arm: 1,678,920 ops/s
eculidean distance between 1536d vectors in `int8`
x86: 252,113 ops/s arm: 177,443 ops/s	x86: 6,690,110 ops/s arm: 4,114,160 ops/s	x86: 18,989,000 ops/s arm: 18,878,200 ops/s
eculidean distance between 1536d vectors in `bfloat16`
🚧 not supported	x86: 119,842 ops/s arm: 1,049,230 ops/s	x86: 9,727,210 ops/s arm: 4,233,420 ops/s
eculidean distance between 1536d vectors in `float16`
x86: 54,621 ops/s arm: 71,793 ops/s	x86: 196,413 ops/s arm: 911,370 ops/s	x86: 19,466,800 ops/s arm: 3,522,760 ops/s
eculidean distance between 1536d vectors in `float32`
x86: 424,944 ops/s arm: 292,629 ops/s	x86: 1,295,210 ops/s arm: 1,055,940 ops/s	x86: 8,924,100 ops/s arm: 3,602,650 ops/s
eculidean distance between 1536d vectors in `float64`
x86: 334,929 ops/s arm: 237,505 ops/s	x86: 1,215,190 ops/s arm: 905,782 ops/s	x86: 1,701,740 ops/s arm: 1,735,840 ops/s

For benchmarks we mostly use 1536-dimensional vectors, like the embeddings produced by the OpenAI Ada API. The code was compiled with GCC 12, using glibc v2.35. The benchmarks performed on Arm-based Graviton3 AWS c7g instances and r7iz Intel Sapphire Rapids. Most modern Arm-based 64-bit CPUs will have similar relative speedups. Variance withing x86 CPUs will be larger.

Similar speedups are often observed even when compared to BLAS and LAPACK libraries underlying most numerical computing libraries, including NumPy and SciPy in Python. Broader benchmarking results:

Using SimSIMD in Python

The package is intended to replace the usage of numpy.inner, numpy.dot, and scipy.spatial.distance. Aside from drastic performance improvements, SimSIMD significantly improves accuracy in mixed precision setups. NumPy and SciPy, processing int8, uint8 or float16 vectors, will use the same types for accumulators, while SimSIMD can combine int8 enumeration, int16 multiplication, and int32 accumulation to avoid overflows entirely. The same applies to processing float16 and bfloat16 values with float32 precision.

Installation

Use the following snippet to install SimSIMD and list available hardware acceleration options available on your machine:

pip install simsimd
python -c "import simsimd; print(simsimd.get_capabilities())"   # for hardware introspection
python -c "import simsimd; help(simsimd)"                       # for documentation

With precompiled binaries, SimSIMD ships .pyi interface files for type hinting and static analysis. You can check all the available functions in python/annotations/__init__.pyi.

One-to-One Distance

import simsimd
import numpy as np

vec1 = np.random.randn(1536).astype(np.float32)
vec2 = np.random.randn(1536).astype(np.float32)
dist = simsimd.cosine(vec1, vec2)

Supported functions include cosine, inner, sqeuclidean, hamming, jaccard, kulbackleibler, jensenshannon, and intersect. Dot products are supported for both real and complex numbers:

vec1 = np.random.randn(768).astype(np.float64) + 1j * np.random.randn(768).astype(np.float64)
vec2 = np.random.randn(768).astype(np.float64) + 1j * np.random.randn(768).astype(np.float64)

dist = simsimd.dot(vec1.astype(np.complex128), vec2.astype(np.complex128))
dist = simsimd.dot(vec1.astype(np.complex64), vec2.astype(np.complex64))
dist = simsimd.vdot(vec1.astype(np.complex64), vec2.astype(np.complex64)) # conjugate, same as `np.vdot`

Unlike SciPy, SimSIMD allows explicitly stating the precision of the input vectors, which is especially useful for mixed-precision setups. The dtype argument can be passed both by name and as a positional argument:

dist = simsimd.cosine(vec1, vec2, "int8")
dist = simsimd.cosine(vec1, vec2, "float16")
dist = simsimd.cosine(vec1, vec2, "float32")
dist = simsimd.cosine(vec1, vec2, "float64")
dist = simsimd.hamming(vec1, vec2, "bin8")

Binary distance functions are computed at a bit-level. Meaning a vector of 10x 8-bit integers will be treated as a sequence of 80 individual bits or dimensions. This differs from NumPy, that can't handle smaller-than-byte types, but you can still avoid the bin8 argument by reinterpreting the vector as booleans:

vec1 = np.random.randint(2, size=80).astype(np.uint8).packbits().view(np.bool_)
vec2 = np.random.randint(2, size=80).astype(np.uint8).packbits().view(np.bool_)
hamming_distance = simsimd.hamming(vec1, vec2)
jaccard_distance = simsimd.jaccard(vec1, vec2)

With other frameworks, like PyTorch, one can get a richer type-system than NumPy, but the lack of good CPython interoperability makes it hard to pass data without copies. Here is an example of using SimSIMD with PyTorch to compute the cosine similarity between two bfloat16 vectors:

import numpy as np
buf1 = np.empty(8, dtype=np.uint16)
buf2 = np.empty(8, dtype=np.uint16)

# View the same memory region with PyTorch and randomize it
import torch
vec1 = torch.asarray(memoryview(buf1), copy=False).view(torch.bfloat16)
vec2 = torch.asarray(memoryview(buf2), copy=False).view(torch.bfloat16)
torch.randn(8, out=vec1)
torch.randn(8, out=vec2)

# Both libs will look into the same memory buffers and report the same results
dist_slow = 1 - torch.nn.functional.cosine_similarity(vec1, vec2, dim=0)
dist_fast = simsimd.cosine(buf1, buf2, "bfloat16")

It also allows using SimSIMD for half-precision complex numbers, which NumPy does not support. For that, view data as continuous even-length np.float16 vectors and override type-resolution with complex32 string.

vec1 = np.random.randn(1536).astype(np.float16)
vec2 = np.random.randn(1536).astype(np.float16)
simd.dot(vec1, vec2, "complex32")
simd.vdot(vec1, vec2, "complex32")

When dealing with sparse representations and integer sets, you can apply the intersect function to two 1-dimensional arrays of uint16 or uint32 integers:

from random import randint
import numpy as np
import simsimd as simd

length1, length2 = randint(1, 100), randint(1, 100)
vec1 = np.sort(np.random.randint(0, 1000, length1).astype(np.uint16))
vec2 = np.sort(np.random.randint(0, 1000, length2).astype(np.uint16))

slow_result = len(np.intersect1d(vec1, vec2))
fast_result = simd.intersect(vec1, vec2)
assert slow_result == fast_result

One-to-Many Distances

Every distance function can be used not only for one-to-one but also one-to-many and many-to-many distance calculations. For one-to-many:

vec1 = np.random.randn(1536).astype(np.float32) # rank 1 tensor
batch1 = np.random.randn(1, 1536).astype(np.float32) # rank 2 tensor
batch2 = np.random.randn(100, 1536).astype(np.float32)

dist_rank1 = simsimd.cosine(vec1, batch2)
dist_rank2 = simsimd.cosine(batch1, batch2)

Many-to-Many Distances

All distance functions in SimSIMD can be used to compute many-to-many distances. For two batches of 100 vectors to compute 100 distances, one would call it like this:

batch1 = np.random.randn(100, 1536).astype(np.float32)
batch2 = np.random.randn(100, 1536).astype(np.float32)
dist = simsimd.cosine(batch1, batch2)

Input matrices must have identical shapes. This functionality isn't natively present in NumPy or SciPy, and generally requires creating intermediate arrays, which is inefficient and memory-consuming.

Many-to-Many All-Pairs Distances

One can use SimSIMD to compute distances between all possible pairs of rows across two matrices (akin to scipy.spatial.distance.cdist). The resulting object will have a type DistancesTensor, zero-copy compatible with NumPy and other libraries. For two arrays of 10 and 1,000 entries, the resulting tensor will have 10,000 cells:

import numpy as np
from simsimd import cdist, DistancesTensor

matrix1 = np.random.randn(1000, 1536).astype(np.float32)
matrix2 = np.random.randn(10, 1536).astype(np.float32)
distances: DistancesTensor = simsimd.cdist(matrix1, matrix2, metric="cosine")   # zero-copy, managed by SimSIMD
distances_array: np.ndarray = np.array(distances, copy=True)                    # now managed by NumPy

Element-wise Kernels

SimSIMD also provides mixed-precision element-wise kernels, where the input vectors and the output have the same numeric type, but the intermediate accumulators are of a higher precision.

import numpy as np
from simsimd import fma, wsum

# Let's take two FullHD video frames
first_frame = np.random.randn(1920 * 1024).astype(np.uint8)  
second_frame = np.random.randn(1920 * 1024).astype(np.uint8)
average_frame = np.empty_like(first_frame)
wsum(first_frame, second_frame, alpha=0.5, beta=0.5, out=average_frame)

# Slow analog with NumPy:
slow_average_frame = (0.5 * first_frame + 0.5 * second_frame).astype(np.uint8)

Similarly, the fma takes three arguments and computes the fused multiply-add operation. In applications like Machine Learning you may also benefit from using the "brain-float" format not natively supported by NumPy. In 3D Graphics, for example, we can use FMA to compute the Phong shading model:

# Assume a FullHD frame with random values for simplicity
light_intensity = np.random.rand(1920 * 1080).astype(np.float16)  # Intensity of light on each pixel
diffuse_component = np.random.rand(1920 * 1080).astype(np.float16)  # Diffuse reflectance on the surface
specular_component = np.random.rand(1920 * 1080).astype(np.float16)  # Specular reflectance for highlights
output_color = np.empty_like(light_intensity)  # Array to store the resulting color intensity

# Define the scaling factors for diffuse and specular contributions
alpha = 0.7  # Weight for the diffuse component
beta = 0.3   # Weight for the specular component

# Formula: color = alpha * light_intensity * diffuse_component + beta * specular_component
fma(light_intensity, diffuse_component, specular_component, 
    dtype="float16", # Optional, unless it can't be inferred from the input
    alpha=alpha, beta=beta, out=output_color)

# Slow analog with NumPy for comparison
slow_output_color = (alpha * light_intensity * diffuse_component + beta * specular_component).astype(np.float16)

Multithreading and Memory Usage

By default, computations use a single CPU core. To override this behavior, use the threads argument. Set it to 0 to use all available CPU cores. Here is an example of dealing with large sets of binary vectors:

ndim = 1536 # OpenAI Ada embeddings
matrix1 = np.packbits(np.random.randint(2, size=(10_000, ndim)).astype(np.uint8))
matrix2 = np.packbits(np.random.randint(2, size=(1_000, ndim)).astype(np.uint8))

distances = simsimd.cdist(matrix1, matrix2, 
    metric="hamming",   # Unlike SciPy, SimSIMD doesn't divide by the number of dimensions
    out_dtype="uint8",  # so we can use `uint8` instead of `float64` to save memory.
    threads=0,          # Use all CPU cores with OpenMP.
    dtype="bin8",       # Override input argument type to `bin8` eight-bit words.
)

By default, the output distances will be stored in double-precision float64 floating-point numbers. That behavior may not be space-efficient, especially if you are computing the hamming distance between short binary vectors, that will generally fit into 8x smaller uint8 or uint16 types. To override this behavior, use the dtype argument.

Helper Functions

You can turn specific backends on or off depending on the exact environment. A common case may be avoiding AVX-512 on older AMD CPUs and Intel Ice Lake CPUs to ensure the CPU doesn't change the frequency license and throttle performance.

$ simsimd.get_capabilities()
> {'serial': True, 'neon': False, 'sve': False, 'neon_f16': False, 'sve_f16': False, 'neon_bf16': False, 'sve_bf16': False, 'neon_i8': False, 'sve_i8': False, 'haswell': True, 'skylake': True, 'ice': True, 'genoa': True, 'sapphire': True, 'turin': True}
$ simsimd.disable_capability("sapphire")
$ simsimd.enable_capability("sapphire")

Using Python API with USearch

Want to use it in Python with USearch? You can wrap the raw C function pointers SimSIMD backends into a CompiledMetric and pass it to USearch, similar to how it handles Numba's JIT-compiled code.

from usearch.index import Index, CompiledMetric, MetricKind, MetricSignature
from simsimd import pointer_to_sqeuclidean, pointer_to_cosine, pointer_to_inner

metric = CompiledMetric(
    pointer=pointer_to_cosine("f16"),
    kind=MetricKind.Cos,
    signature=MetricSignature.ArrayArraySize,
)

index = Index(256, metric=metric)

Using SimSIMD in Rust

To install, add the following to your Cargo.toml:

[dependencies]
simsimd = "..."

Before using the SimSIMD library, ensure you have imported the necessary traits and types into your Rust source file. The library provides several traits for different distance/similarity kinds - SpatialSimilarity, BinarySimilarity, and ProbabilitySimilarity.

Spatial Similarity: Cosine and Euclidean Distances

use simsimd::SpatialSimilarity;

fn main() {
    let vector_a: Vec<f32> = vec![1.0, 2.0, 3.0];
    let vector_b: Vec<f32> = vec![4.0, 5.0, 6.0];

    // Compute the cosine similarity between vector_a and vector_b
    let cosine_similarity = f32::cosine(&vector_a, &vector_b)
        .expect("Vectors must be of the same length");

    println!("Cosine Similarity: {}", cosine_similarity);

    // Compute the squared Euclidean distance between vector_a and vector_b
    let sq_euclidean_distance = f32::sqeuclidean(&vector_a, &vector_b)
        .expect("Vectors must be of the same length");

    println!("Squared Euclidean Distance: {}", sq_euclidean_distance);
}

Spatial similarity functions are available for f64, f32, f16, and i8 types.

Dot-Products: Inner and Complex Inner Products

use simsimd::SpatialSimilarity;
use simsimd::ComplexProducts;

fn main() {
    let vector_a: Vec<f32> = vec![1.0, 2.0, 3.0, 4.0];
    let vector_b: Vec<f32> = vec![5.0, 6.0, 7.0, 8.0];

    // Compute the inner product between vector_a and vector_b
    let inner_product = SpatialSimilarity::dot(&vector_a, &vector_b)
        .expect("Vectors must be of the same length");

    println!("Inner Product: {}", inner_product);

    // Compute the complex inner product between complex_vector_a and complex_vector_b
    let complex_inner_product = ComplexProducts::dot(&vector_a, &vector_b)
        .expect("Vectors must be of the same length");

    let complex_conjugate_inner_product = ComplexProducts::vdot(&vector_a, &vector_b)
        .expect("Vectors must be of the same length");

    println!("Complex Inner Product: {:?}", complex_inner_product); // -18, 69
    println!("Complex C. Inner Product: {:?}", complex_conjugate_inner_product); // 70, -8
}

Complex inner products are available for f64, f32, and f16 types.

Probability Distributions: Jensen-Shannon and Kullback-Leibler Divergences

use simsimd::SpatialSimilarity;

fn main() {
    let vector_a: Vec<f32> = vec![1.0, 2.0, 3.0];
    let vector_b: Vec<f32> = vec![4.0, 5.0, 6.0];

    let cosine_similarity = f32::jensenshannon(&vector_a, &vector_b)
        .expect("Vectors must be of the same length");

    println!("Cosine Similarity: {}", cosine_similarity);

    let sq_euclidean_distance = f32::kullbackleibler(&vector_a, &vector_b)
        .expect("Vectors must be of the same length");

    println!("Squared Euclidean Distance: {}", sq_euclidean_distance);
}

Probability similarity functions are available for f64, f32, and f16 types.

Binary Similarity: Hamming and Jaccard Distances

Similar to spatial distances, one can compute bit-level distance functions between slices of unsigned integers:

use simsimd::BinarySimilarity;

fn main() {
    let vector_a = &[0b11110000, 0b00001111, 0b10101010];
    let vector_b = &[0b11110000, 0b00001111, 0b01010101];

    // Compute the Hamming distance between vector_a and vector_b
    let hamming_distance = u8::hamming(&vector_a, &vector_b)
        .expect("Vectors must be of the same length");

    println!("Hamming Distance: {}", hamming_distance);

    // Compute the Jaccard distance between vector_a and vector_b
    let jaccard_distance = u8::jaccard(&vector_a, &vector_b)
        .expect("Vectors must be of the same length");

    println!("Jaccard Distance: {}", jaccard_distance);
}

Binary similarity functions are available only for u8 types.

Half-Precision Floating-Point Numbers

Rust has no native support for half-precision floating-point numbers, but SimSIMD provides a f16 type. It has no functionality - it is a transparent wrapper around u16 and can be used with half or any other half-precision library.

use simsimd::SpatialSimilarity;
use simsimd::f16 as SimF16;
use half::f16 as HalfF16;

fn main() {
    let vector_a: Vec<HalfF16> = ...
    let vector_b: Vec<HalfF16> = ...

    let buffer_a: &[SimF16] = unsafe { std::slice::from_raw_parts(a_half.as_ptr() as *const SimF16, a_half.len()) };
    let buffer_b: &[SimF16] = unsafe { std::slice::from_raw_parts(b_half.as_ptr() as *const SimF16, b_half.len()) };

    // Compute the cosine similarity between vector_a and vector_b
    let cosine_similarity = SimF16::cosine(&vector_a, &vector_b)
        .expect("Vectors must be of the same length");

    println!("Cosine Similarity: {}", cosine_similarity);
}

Half-Precision Brain-Float Numbers

The "brain-float-16" is a popular machine learning format. It's broadly supported in hardware and is very machine-friendly, but software support is still lagging behind. Unlike NumPy, you can already use bf16 datatype in SimSIMD. Luckily, to downcast f32 to bf16 you only have to drop the last 16 bits:

import numpy as np
import simsimd as simd

a = np.random.randn(ndim).astype(np.float32)
b = np.random.randn(ndim).astype(np.float32)

# NumPy doesn't natively support brain-float, so we need a trick!
# Luckily, it's very easy to reduce the representation accuracy
# by simply masking the low 16-bits of our 32-bit single-precision
# numbers. We can also add `0x8000` to round the numbers.
a_f32rounded = ((a.view(np.uint32) + 0x8000) & 0xFFFF0000).view(np.float32)
b_f32rounded = ((b.view(np.uint32) + 0x8000) & 0xFFFF0000).view(np.float32)

# To represent them as brain-floats, we need to drop the second half
a_bf16 = np.right_shift(a_f32rounded.view(np.uint32), 16).astype(np.uint16)
b_bf16 = np.right_shift(b_f32rounded.view(np.uint32), 16).astype(np.uint16)

# Now we can compare the results
expected = np.inner(a_f32rounded, b_f32rounded)
result = simd.inner(a_bf16, b_bf16, "bf16")

Dynamic Dispatch in Rust

SimSIMD provides a dynamic dispatch mechanism to select the most advanced micro-kernel for the current CPU. You can query supported backends and use the SimSIMD::capabilities function to select the best one.

println!("uses neon: {}", capabilities::uses_neon());
println!("uses sve: {}", capabilities::uses_sve());
println!("uses haswell: {}", capabilities::uses_haswell());
println!("uses skylake: {}", capabilities::uses_skylake());
println!("uses ice: {}", capabilities::uses_ice());
println!("uses genoa: {}", capabilities::uses_genoa());
println!("uses sapphire: {}", capabilities::uses_sapphire());
println!("uses turin: {}", capabilities::uses_turin());

Using SimSIMD in JavaScript

To install, choose one of the following options depending on your environment:

npm install --save simsimd
yarn add simsimd
pnpm add simsimd
bun install simsimd

The package is distributed with prebuilt binaries, but if your platform is not supported, you can build the package from the source via npm run build. This will automatically happen unless you install the package with the --ignore-scripts flag or use Bun. After you install it, you will be able to call the SimSIMD functions on various TypedArray variants:

const { sqeuclidean, cosine, inner, hamming, jaccard } = require('simsimd');

const vectorA = new Float32Array([1.0, 2.0, 3.0]);
const vectorB = new Float32Array([4.0, 5.0, 6.0]);

const distance = sqeuclidean(vectorA, vectorB);
console.log('Squared Euclidean Distance:', distance);

Other numeric types and precision levels are supported as well. For double-precision floating-point numbers, use Float64Array:

const vectorA = new Float64Array([1.0, 2.0, 3.0]);
const vectorB = new Float64Array([4.0, 5.0, 6.0]);
const distance = cosine(vectorA, vectorB);

When doing machine learning and vector search with high-dimensional vectors you may want to quantize them to 8-bit integers. You may want to project values from the $[-1, 1]$ range to the $[-127, 127]$ range and then cast them to Int8Array:

const quantizedVectorA = new Int8Array(vectorA.map(v => (v * 127)));
const quantizedVectorB = new Int8Array(vectorB.map(v => (v * 127)));
const distance = cosine(quantizedVectorA, quantizedVectorB);

A more extreme quantization case would be to use binary vectors. You can map all positive values to 1 and all negative values and zero to 0, packing eight values into a single byte. After that, Hamming and Jaccard distances can be computed.

const { toBinary, hamming } = require('simsimd');

const binaryVectorA = toBinary(vectorA);
const binaryVectorB = toBinary(vectorB);
const distance = hamming(binaryVectorA, binaryVectorB);

Using SimSIMD in Sift

To install, simply add the following dependency to you Package.swift:

dependencies: [
    .package(url: "https://github.com/ashvardanian/simsimd")
]

The package provides the most common spatial metrics for Int8, Float16, Float32, and Float64 vectors.

import SimSIMD

let vectorA: [Int8] = [1, 2, 3]
let vectorB: [Int8] = [4, 5, 6]

let cosineSimilarity = vectorA.cosine(vectorB)  // Computes the cosine similarity
let dotProduct = vectorA.dot(vectorB)           // Computes the dot product
let sqEuclidean = vectorA.sqeuclidean(vectorB)  // Computes the squared Euclidean distance

Using SimSIMD in C

For integration within a CMake-based project, add the following segment to your CMakeLists.txt:

FetchContent_Declare(
    simsimd
    GIT_REPOSITORY https://github.com/ashvardanian/simsimd.git
    GIT_SHALLOW TRUE
)
FetchContent_MakeAvailable(simsimd)

After that, you can use the SimSIMD library in your C code in several ways. Simplest of all, you can include the headers, and the compiler will automatically select the most recent CPU extensions that SimSIMD will use.

#include <simsimd/simsimd.h>

int main() {
    simsimd_f32_t vector_a[1536];
    simsimd_f32_t vector_b[1536];
    simsimd_kernel_punned_t distance_function = simsimd_metric_punned(
        simsimd_metric_cos_k,   // Metric kind, like the angular cosine distance
        simsimd_datatype_f32_k, // Data type, like: f16, f32, f64, i8, b8, and complex variants
        simsimd_cap_any_k);     // Which CPU capabilities are we allowed to use
    simsimd_distance_t distance;
    distance_function(vector_a, vector_b, 1536, &distance);
    return 0;
}

Dynamic Dispatch in C

To avoid hard-coding the backend, you can rely on c/lib.c to prepackage all possible backends in one binary, and select the most recent CPU features at runtime. That feature of the C library is called dynamic dispatch and is extensively used in the Python, JavaScript, and Rust bindings. To test which CPU features are available on the machine at runtime, use the following APIs:

int uses_dynamic_dispatch = simsimd_uses_dynamic_dispatch(); // Check if dynamic dispatch was enabled
simsimd_capability_t capabilities = simsimd_capabilities();  // Returns a bitmask

int uses_neon = simsimd_uses_neon();
int uses_sve = simsimd_uses_sve();
int uses_haswell = simsimd_uses_haswell();
int uses_skylake = simsimd_uses_skylake();
int uses_ice = simsimd_uses_ice();
int uses_genoa = simsimd_uses_genoa();
int uses_sapphire = simsimd_uses_sapphire();

To override compilation settings and switch between runtime and compile-time dispatch, define the following macro:

#define SIMSIMD_DYNAMIC_DISPATCH 1 // or 0

Spatial Distances: Cosine and Euclidean Distances

#include <simsimd/simsimd.h>

int main() {
    simsimd_i8_t i8[1536];
    simsimd_i8_t u8[1536];
    simsimd_f64_t f64s[1536];
    simsimd_f32_t f32s[1536];
    simsimd_f16_t f16s[1536];
    simsimd_bf16_t bf16s[1536];
    simsimd_distance_t distance;

    // Cosine distance between two vectors
    simsimd_cos_i8(i8s, i8s, 1536, &distance);
    simsimd_cos_u8(u8s, u8s, 1536, &distance);
    simsimd_cos_f16(f16s, f16s, 1536, &distance);
    simsimd_cos_f32(f32s, f32s, 1536, &distance);
    simsimd_cos_f64(f64s, f64s, 1536, &distance);
    simsimd_cos_bf16(bf16s, bf16s, 1536, &distance);
    
    // Euclidean distance between two vectors
    simsimd_l2sq_i8(i8s, i8s, 1536, &distance);
    simsimd_l2sq_u8(u8s, u8s, 1536, &distance);
    simsimd_l2sq_f16(f16s, f16s, 1536, &distance);
    simsimd_l2sq_f32(f32s, f32s, 1536, &distance);
    simsimd_l2sq_f64(f64s, f64s, 1536, &distance);
    simsimd_l2sq_bf16(bf16s, bf16s, 1536, &distance);

    return 0;
}

Dot-Products: Inner and Complex Inner Products

#include <simsimd/simsimd.h>

int main() {
    // SimSIMD provides "sized" type-aliases without relying on `stdint.h`
    simsimd_i8_t i8[1536];
    simsimd_i8_t u8[1536];
    simsimd_f16_t f16s[1536];
    simsimd_f32_t f32s[1536];
    simsimd_f64_t f64s[1536];
    simsimd_bf16_t bf16s[1536];
    simsimd_distance_t product;

    // Inner product between two real vectors
    simsimd_dot_i8(i8s, i8s, 1536, &product);
    simsimd_dot_u8(u8s, u8s, 1536, &product);
    simsimd_dot_f16(f16s, f16s, 1536, &product);
    simsimd_dot_f32(f32s, f32s, 1536, &product);
    simsimd_dot_f64(f64s, f64s, 1536, &product);
    simsimd_dot_bf16(bf16s, bf16s, 1536, &product);

    // SimSIMD provides complex types with `real` and `imag` fields
    simsimd_f64c_t f64s[768];
    simsimd_f32c_t f32s[768];
    simsimd_f16c_t f16s[768];
    simsimd_bf16c_t bf16s[768];
    simsimd_distance_t products[2]; // real and imaginary parts

    // Complex inner product between two vectors
    simsimd_dot_f16c(f16cs, f16cs, 768, &products[0]);
    simsimd_dot_f32c(f32cs, f32cs, 768, &products[0]);
    simsimd_dot_f64c(f64cs, f64cs, 768, &products[0]);
    simsimd_dot_bf16c(bf16cs, bf16cs, 768, &products[0]);

    // Complex conjugate inner product between two vectors
    simsimd_vdot_f16c(f16cs, f16cs, 768, &products[0]);
    simsimd_vdot_f32c(f32cs, f32cs, 768, &products[0]);
    simsimd_vdot_f64c(f64cs, f64cs, 768, &products[0]);
    simsimd_vdot_bf16c(bf16cs, bf16cs, 768, &products[0]);
    return 0;
}

Binary Distances: Hamming and Jaccard Distances

#include <simsimd/simsimd.h>

int main() {
    simsimd_b8_t b8s[1536 / 8]; // 8 bits per word
    simsimd_distance_t distance;
    simsimd_hamming_b8(b8s, b8s, 1536 / 8, &distance);
    simsimd_jaccard_b8(b8s, b8s, 1536 / 8, &distance);
    return 0;
}

Probability Distributions: Jensen-Shannon and Kullback-Leibler Divergences

#include <simsimd/simsimd.h>

int main() {
    simsimd_f64_t f64s[1536];
    simsimd_f32_t f32s[1536];
    simsimd_f16_t f16s[1536];
    simsimd_distance_t divergence;

    // Jensen-Shannon divergence between two vectors
    simsimd_js_f16(f16s, f16s, 1536, &divergence);
    simsimd_js_f32(f32s, f32s, 1536, &divergence);
    simsimd_js_f64(f64s, f64s, 1536, &divergence);

    // Kullback-Leibler divergence between two vectors
    simsimd_kl_f16(f16s, f16s, 1536, &divergence);
    simsimd_kl_f32(f32s, f32s, 1536, &divergence);
    simsimd_kl_f64(f64s, f64s, 1536, &divergence);
    return 0;
}

Half-Precision Floating-Point Numbers

If you aim to utilize the _Float16 functionality with SimSIMD, ensure your development environment is compatible with C 11. For other SimSIMD functionalities, C 99 compatibility will suffice. To explicitly disable half-precision support, define the following macro before imports:

#define SIMSIMD_NATIVE_F16 0 // or 1
#define SIMSIMD_NATIVE_BF16 0 // or 1
#include <simsimd/simsimd.h>

Compilation Settings and Debugging

SIMSIMD_DYNAMIC_DISPATCH:

By default, SimSIMD is a header-only library. But if you are running on different generations of devices, it makes sense to pre-compile the library for all supported generations at once, and dispatch at runtime. This flag does just that and is used to produce the simsimd.so shared library, as well as the Python and other bindings.

For Arm: SIMSIMD_TARGET_NEON, SIMSIMD_TARGET_SVE, SIMSIMD_TARGET_SVE2, SIMSIMD_TARGET_NEON_F16, SIMSIMD_TARGET_SVE_F16, SIMSIMD_TARGET_NEON_BF16, SIMSIMD_TARGET_SVE_BF16. For x86: (SIMSIMD_TARGET_HASWELL, SIMSIMD_TARGET_SKYLAKE, SIMSIMD_TARGET_ICE, SIMSIMD_TARGET_GENOA, SIMSIMD_TARGET_SAPPHIRE, SIMSIMD_TARGET_TURIN, SIMSIMD_TARGET_SIERRA.

By default, SimSIMD automatically infers the target architecture and pre-compiles as many kernels as possible. In some cases, you may want to explicitly disable some of the kernels. Most often it's due to compiler support issues, like the lack of some recent intrinsics or low-precision numeric types. In other cases, you may want to disable some kernels to speed up the compilation process and trim the binary size.

SIMSIMD_SQRT, SIMSIMD_RSQRT, SIMSIMD_LOG:

By default, for non-SIMD backends, SimSIMD may use libc functions like sqrt and log. Those are generally very accurate, but slow, and introduce a dependency on the C standard library. To avoid that you can override those definitions with your custom implementations, like: #define SIMSIMD_RSQRT(x) (1 / sqrt(x)).

Algorithms & Design Decisions 📚

In general there are a few principles that SimSIMD follows:

Avoid loop unrolling.
Never allocate memory.
Never throw exceptions or set errno.
Keep all function arguments the size of the pointer.
Avoid returning from public interfaces, use out-arguments instead.
Don't over-optimize for old CPUs and single- and double-precision floating-point numbers.
Prioritize mixed-precision and integer operations, and new ISA extensions.
Prefer saturated arithmetic and avoid overflows.

Possibly, in the future:

Best effort computation silencing NaN components in low-precision inputs.
Detect overflows and report the distance with a "signaling" NaN.

Last, but not the least - don't build unless there is a demand for it. So if you have a specific use-case, please open an issue or a pull request, and ideally, bring in more users with similar needs.

Cosine Similarity, Reciprocal Square Root, and Newton-Raphson Iteration

The cosine similarity is the most common and straightforward metric used in machine learning and information retrieval. Interestingly, there are multiple ways to shoot yourself in the foot when computing it. The cosine similarity is the inverse of the cosine distance, which is the cosine of the angle between two vectors.

\text{CosineSimilarity}(a, b) = \frac{a \cdot b}{\|a\| \cdot \|b\|}

\text{CosineDistance}(a, b) = 1 - \frac{a \cdot b}{\|a\| \cdot \|b\|}

In NumPy terms, SimSIMD implementation is similar to:

import numpy as np

def cos_numpy(a: np.ndarray, b: np.ndarray) -> float:
    ab, a2, b2 = np.dot(a, b), np.dot(a, a), np.dot(b, b) # Fused in SimSIMD
    if a2 == 0 and b2 == 0: result = 0                    # Same in SciPy
    elif ab == 0: result = 1                              # Division by zero error in SciPy
    else: result = 1 - ab / (sqrt(a2) * sqrt(b2))         # Bigger rounding error in SciPy
    return result

In SciPy, however, the cosine distance is computed as 1 - ab / np.sqrt(a2 * b2). It handles the edge case of a zero and non-zero argument pair differently, resulting in a division by zero error. It's not only less efficient, but also less accurate, given how the reciprocal square roots are computed. The C standard library provides the sqrt function, which is generally very accurate, but slow. The rsqrt in-hardware implementations are faster, but have different accuracy characteristics.

SSE rsqrtps and AVX vrsqrtps: $1.5 \times 2^{-12}$ maximal relative error.
AVX-512 vrsqrt14pd instruction: $2^{-14}$ maximal relative error.
NEON frsqrte instruction has no documented error bounds, but can be $2^{-3}$.

To overcome the limitations of the rsqrt instruction, SimSIMD uses the Newton-Raphson iteration to refine the initial estimate for high-precision floating-point numbers. It can be defined as:

x_{n+1} = x_n \cdot (3 - x_n \cdot x_n) / 2

On 1536-dimensional inputs on Intel Sapphire Rapids CPU a single such iteration can result in a 2-3 orders of magnitude relative error reduction:

Datatype	NumPy Error	SimSIMD w/out Iteration	SimSIMD
`bfloat16`	1.89e-08 ± 1.59e-08	3.07e-07 ± 3.09e-07	3.53e-09 ± 2.70e-09
`float16`	1.67e-02 ± 1.44e-02	2.68e-05 ± 1.95e-05	2.02e-05 ± 1.39e-05
`float32`	2.21e-08 ± 1.65e-08	3.47e-07 ± 3.49e-07	3.77e-09 ± 2.84e-09
`float64`	0.00e+00 ± 0.00e+00	3.80e-07 ± 4.50e-07	1.35e-11 ± 1.85e-11

Curved Spaces, Mahalanobis Distance, and Bilinear Quadratic Forms

The Mahalanobis distance is a generalization of the Euclidean distance, which takes into account the covariance of the data. It's very similar in its form to the bilinear form, which is a generalization of the dot product.

\text{BilinearForm}(a, b, M) = a^T M b

\text{Mahalanobis}(a, b, M) = \sqrt{(a - b)^T M^{-1} (a - b)}

Bilinear Forms can be seen as one of the most important linear algebraic operations, surprisingly missing in BLAS and LAPACK. They are versatile and appear in various domains:

In Quantum Mechanics, the expectation value of an observable $A$ in a state $\psi$ is given by $\langle \psi | A | \psi \rangle$, which is a bilinear form.
In Machine Learning, in Support Vector Machines (SVMs), bilinear forms define kernel functions that measure similarity between data points.
In Differential Geometry, the metric tensor, which defines distances and angles on a manifold, is a bilinear form on the tangent space.
In Economics, payoff functions in certain Game Theoretic problems can be modeled as bilinear forms of players' strategies.
In Physics, interactions between electric and magnetic fields can be expressed using bilinear forms.

Broad applications aside, the lack of a specialized primitive for bilinear forms in BLAS and LAPACK means significant performance overhead. A $vector * matrix * vector$ product is a scalar, whereas its constituent parts ($vector * matrix$ and $matrix * vector$) are vectors:

They need memory to be stored in: $O(n)$ allocation.
The data will be written to memory and read back, wasting CPU cycles.

SimSIMD doesn't produce intermediate vector results, like a @ M @ b, but computes the bilinear form directly.

Set Intersection, Galloping, and Binary Search

The set intersection operation is generally defined as the number of elements that are common between two sets, represented as sorted arrays of integers. The most common way to compute it is a linear scan:

size_t intersection_size(int *a, int *b, size_t n, size_t m) {
    size_t i = 0, j = 0, count = 0;
    while (i < n && j < m) {
        if (a[i] < b[j]) i++;
        else if (a[i] > b[j]) j++;
        else i++, j++, count++;
    }
    return count;
}

Alternatively, one can use the binary search to find the elements in the second array that are present in the first one. On every step the checked region of the second array is halved, which is called the galloping search. It's faster, but only when large arrays of very different sizes are intersected. Third approach is to use the SIMD instructions to compare multiple elements at once:

Using string-intersection instructions on x86, like pcmpestrm.
Using integer-intersection instructions in AVX-512, like vp2intersectd.
Using vanilla equality checks present in all SIMD instruction sets.

After benchmarking, the last approach was chosen, as it's the most flexible and often the fastest.

Complex Dot Products, Conjugate Dot Products, and Complex Numbers

Complex dot products are a generalization of the dot product to complex numbers. They are supported by most BLAS packages, but almost never in mixed precision. SimSIMD defines dot and vdot kernels as:

\text{dot}(a, b) = \sum_{i=0}^{n-1} a_i \cdot b_i

\text{vdot}(a, b) = \sum_{i=0}^{n-1} a_i \cdot \bar{b_i}

Where $\bar{b_i}$ is the complex conjugate of $b_i$. Putting that into Python code for scalar arrays:

def dot(a: List[number], b: List[number]) -> number:
    a_real, a_imaginary = a[0::2], a[1::2]
    b_real, b_imaginary = b[0::2], b[1::2]
    ab_real, ab_imaginary = 0, 0
    for ar, ai, br, bi in zip(a_real, a_imaginary, b_real, b_imaginary):
        ab_real += ar * br - ai * bi
        ab_imaginary += ar * bi + ai * br
    return ab_real, ab_imaginary

def vdot(a: List[number], b: List[number]) -> number:
    a_real, a_imaginary = a[0::2], a[1::2]
    b_real, b_imaginary = b[0::2], b[1::2]
    ab_real, ab_imaginary = 0, 0
    for ar, ai, br, bi in zip(a_real, a_imaginary, b_real, b_imaginary):
        ab_real += ar * br + ai * bi
        ab_imaginary += ar * bi - ai * br
    return ab_real, ab_imaginary

Logarithms in Kullback-Leibler & Jensen–Shannon Divergences

The Kullback-Leibler divergence is a measure of how one probability distribution diverges from a second, expected probability distribution. Jensen-Shannon divergence is a symmetrized and smoothed version of the Kullback-Leibler divergence, which can be used as a distance metric between probability distributions.

\text{KL}(P || Q) = \sum_{i} P(i) \log \frac{P(i)}{Q(i)}

\text{JS}(P, Q) = \frac{1}{2} \text{KL}(P || M) + \frac{1}{2} \text{KL}(Q || M), M = \frac{P + Q}{2}

Both functions are defined for non-negative numbers, and the logarithm is a key part of their computation.

Mixed Precision in Fused-Multiply-Add and Weighted Sums

The Fused-Multiply-Add (FMA) operation is a single operation that combines element-wise multiplication and addition with different scaling factors. The Weighted Sum is it's simplified variant without element-wise multiplication.

\text{FMA}_i(A, B, C, \alpha, \beta) = \alpha \cdot A_i \cdot B_i + \beta \cdot C_i

\text{WSum}_i(A, B, \alpha, \beta) = \alpha \cdot A_i + \beta \cdot B_i

In NumPy terms, the implementation may look like:

import numpy as np
def wsum(A: np.ndarray, B: np.ndarray, /, Alpha: float, Beta: float) -> np.ndarray:
    assert A.dtype == B.dtype, "Input types must match and affect the output style"
    return (Alpha * A + Beta * B).astype(A.dtype)
def fma(A: np.ndarray, B: np.ndarray, C: np.ndarray, /, Alpha: float, Beta: float) -> np.ndarray:
    assert A.dtype == B.dtype and A.dtype == C.dtype, "Input types must match and affect the output style"
    return (Alpha * A * B + Beta * C).astype(A.dtype)

The tricky part is implementing those operations in mixed precision, where the scaling factors are of different precision than the input and output vectors. SimSIMD uses double-precision floating-point scaling factors for any input and output precision, including i8 and u8 integers and f16 and bf16 floats. Depending on the generation of the CPU, given native support for f16 addition and multiplication, the f16 temporaries are used for i8 and u8 multiplication, scaling, and addition. For bf16, native support is generally limited to dot-products with subsequent partial accumulation, which is not enough for the FMA and WSum operations, so f32 is used as a temporary.

Auto-Vectorization & Loop Unrolling

On the Intel Sapphire Rapids platform, SimSIMD was benchmarked against auto-vectorized code using GCC 12. GCC handles single-precision float but might not be the best choice for int8 and _Float16 arrays, which have been part of the C language since 2011.

Kind	GCC 12 `f32`	GCC 12 `f16`	SimSIMD `f16`	`f16` improvement
Inner Product	3,810 K/s	192 K/s	5,990 K/s	31 x
Cosine Distance	3,280 K/s	336 K/s	6,880 K/s	20 x
Euclidean Distance ²	4,620 K/s	147 K/s	5,320 K/s	36 x
Jensen-Shannon Divergence	1,180 K/s	18 K/s	2,140 K/s	118 x

Dynamic Dispatch

Most popular software is precompiled and distributed with fairly conservative CPU optimizations, to ensure compatibility with older hardware. Database Management platforms, like ClickHouse, and Web Browsers, like Google Chrome,need to run on billions of devices, and they can't afford to be picky about the CPU features. For such users SimSIMD provides a dynamic dispatch mechanism, which selects the most advanced micro-kernel for the current CPU at runtime.

Subset	F	CD	ER	PF
Knights Landing (Xeon Phi x200, 2016)	Yes	Yes	No
Knights Mill (Xeon Phi x205, 2017)	Yes	No
Skylake-SP, Skylake-X (2017)	No	No	Yes	No
Cannon Lake (2018)	Yes	No
Cascade Lake (2019)	No	Yes	No
Cooper Lake (2020)	Yes	No
Ice Lake (2019)	Yes	No	Yes	No
Tiger Lake (2020)	Yes	No
Rocket Lake (2021)	No
Alder Lake (2021)	Partial	Partial
Zen 4 (2022)	Yes	Yes	No
Sapphire Rapids (2023)	No	Yes
Zen 5 (2024)	Yes	No

You can compile SimSIMD on an old CPU, like Intel Haswell, and run it on a new one, like AMD Genoa, and it will automatically use the most advanced instructions available. Reverse is also true, you can compile on a new CPU and run on an old one, and it will automatically fall back to the most basic instructions. Moreover, the very first time you prove for CPU capabilities with simsimd_capabilities(), it initializes the dynamic dispatch mechanism, and all subsequent calls will be faster and won't face race conditions in multi-threaded environments.

Target Specific Backends

SimSIMD exposes all kernels for all backends, and you can select the most advanced one for the current CPU without relying on built-in dispatch mechanisms. That's handy for testing and benchmarking, but also in case you want to dispatch a very specific kernel for a very specific CPU, bypassing SimSIMD assignment logic. All of the function names follow the same pattern: simsimd_{function}_{type}_{backend}.

The backend can be serial, haswell, skylake, ice, genoa, sapphire, turin, neon, or sve.
The type can be f64, f32, f16, bf16, f64c, f32c, f16c, bf16c, i8, or b8.
The function can be dot, vdot, cos, l2sq, hamming, jaccard, kl, js, or intersect.

To avoid hard-coding the backend, you can use the simsimd_kernel_punned_t to pun the function pointer and the simsimd_capabilities function to get the available backends at runtime. To match all the function names, consider a RegEx:

SIMSIMD_PUBLIC void simsimd_\w+_\w+_\w+\(

On Linux, you can use the following command to list all unique functions:

$ grep -oP 'SIMSIMD_PUBLIC void simsimd_\w+_\w+_\w+\(' include/simsimd/*.h | sort | uniq
> include/simsimd/binary.h:SIMSIMD_PUBLIC void simsimd_hamming_b8_haswell(
> include/simsimd/binary.h:SIMSIMD_PUBLIC void simsimd_hamming_b8_ice(
> include/simsimd/binary.h:SIMSIMD_PUBLIC void simsimd_hamming_b8_neon(
> include/simsimd/binary.h:SIMSIMD_PUBLIC void simsimd_hamming_b8_serial(
> include/simsimd/binary.h:SIMSIMD_PUBLIC void simsimd_hamming_b8_sve(
> include/simsimd/binary.h:SIMSIMD_PUBLIC void simsimd_jaccard_b8_haswell(
> include/simsimd/binary.h:SIMSIMD_PUBLIC void simsimd_jaccard_b8_ice(
> include/simsimd/binary.h:SIMSIMD_PUBLIC void simsimd_jaccard_b8_neon(
> include/simsimd/binary.h:SIMSIMD_PUBLIC void simsimd_jaccard_b8_serial(
> include/simsimd/binary.h:SIMSIMD_PUBLIC void simsimd_jaccard_b8_sve(

betaflight/betaflight

2024-12-01T01:52:00Z

Open Source Flight Controller Firmware

Betaflight is flight controller software (firmware) used to fly multi-rotor craft and fixed wing craft.

This fork differs from Baseflight and Cleanflight in that it focuses on flight performance, leading-edge feature additions, and wide target support.

Events

Date	Event
28-04-2024	Firmware 4.5 Release

News

Requirements for the submission of new and updated targets

The following new requirements for pull requests adding new targets or modifying existing targets are put in place from now on:

Read the hardware specification
No new F3 based targets will be accepted;
For any new target that is to be added, only a Unified Target config into https://github.com/betaflight/unified-targets/tree/master/configs/default needs to be submitted. See the instructions for how to create a Unified Target configuration. If there is no Unified Target for the MCU type of the new target (see instructions above), then a 'legacy' format target definition into src/main/target/ has to be submitted as well;
For changes to existing targets, the change needs to be applied to the Unified Target config in https://github.com/betaflight/unified-targets/tree/master/configs/default. If no Unified Target configuration for the target exists, a new Unified Target configuration will have to be created and submitted. If there is no Unified Target for the MCU type of the new target (see instructions above), then an update to the 'legacy' format target definition in src/main/target/ has to be submitted alongside the update to the Unified Target configuration.

Features

Betaflight has the following features:

Multi-color RGB LED strip support (each LED can be a different color using variable length WS2811 Addressable RGB strips - use for Orientation Indicators, Low Battery Warning, Flight Mode Status, Initialization Troubleshooting, etc)
DShot (150, 300 and 600), Multishot, Oneshot (125 and 42) and Proshot1000 motor protocol support
Blackbox flight recorder logging (to onboard flash or external microSD card where equipped)
Support for targets that use the STM32 F4, G4, F7 and H7 processors
PWM, PPM, SPI, and Serial (SBus, SumH, SumD, Spektrum 1024/2048, XBus, etc) RX connection with failsafe detection
Multiple telemetry protocols (CRSF, FrSky, HoTT smart-port, MSP, etc)
RSSI via ADC - Uses ADC to read PWM RSSI signals, tested with FrSky D4R-II, X8R, X4R-SB, & XSR
OSD support & configuration without needing third-party OSD software/firmware/comm devices
OLED Displays - Display information on: Battery voltage/current/mAh, profile, rate profile, mode, version, sensors, etc
In-flight manual PID tuning and rate adjustment
PID and filter tuning using sliders
Rate profiles and in-flight selection of them
Configurable serial ports for Serial RX, Telemetry, ESC telemetry, MSP, GPS, OSD, Sonar, etc - Use most devices on any port, softserial included
VTX support for Unify Pro and IRC Tramp
and MUCH, MUCH more.

Installation & Documentation

See: https://betaflight.com/docs/wiki

Support and Developers Channel

There's a dedicated Discord server here:

https://discord.gg/n4E6ak4u3c

We also have a Facebook Group. Join us to get a place to talk about Betaflight, ask configuration questions, or just hang out with fellow pilots.

https://www.facebook.com/groups/betaflightgroup/

Etiquette: Don't ask to ask and please wait around long enough for a reply - sometimes people are out flying, asleep or at work and can't answer immediately.

Configuration Tool

To configure Betaflight you should use the Betaflight-configurator GUI tool (Windows/OSX/Linux) which can be found here:

https://github.com/betaflight/betaflight-configurator/releases/latest

Contributing

Contributions are welcome and encouraged. You can contribute in many ways:

implement a new feature in the firmware or in configurator (see below);
documentation updates and corrections;
How-To guides - received help? Help others!
bug reporting & fixes;
new feature ideas & suggestions;
provide a new translation for configurator, or help us maintain the existing ones (see below).

The best place to start is the Betaflight Discord (registration here). Next place is the github issue tracker:

https://github.com/betaflight/betaflight/issues https://github.com/betaflight/betaflight-configurator/issues

Before creating new issues please check to see if there is an existing one, search first otherwise you waste people's time when they could be coding instead!

If you want to contribute to our efforts financially, please consider making a donation to us through PayPal.

If you want to contribute financially on an ongoing basis, you should consider becoming a patron for us on Patreon.

Developers

Contribution of bugfixes and new features is encouraged. Please be aware that we have a thorough review process for pull requests, and be prepared to explain what you want to achieve with your pull request. Before starting to write code, please read our development guidelines and coding style definition.

GitHub actions are used to run automatic builds

Translators

We want to make Betaflight accessible for pilots who are not fluent in English, and for this reason we are currently maintaining translations into 21 languages for Betaflight Configurator: Català, Dansk, Deutsch, Español, Euskera, Français, Galego, Hrvatski, Bahasa Indonesia, Italiano, 日本語, 한국어, Latviešu, Português, Português Brasileiro, polski, Русский язык, Svenska, 简体中文, 繁體中文. We have got a team of volunteer translators who do this work, but additional translators are always welcome to share the workload, and we are keen to add additional languages. If you would like to help us with translations, you have got the following options:

if you help by suggesting some updates or improvements to translations in a language you are familiar with, head to crowdin and add your suggested translations there;
if you would like to start working on the translation for a new language, or take on responsibility for proof-reading the translation for a language you are very familiar with, please head to the Betaflight Discord chat (registration here), and join the 'translation' channel - the people in there can help you to get a new language added, or set you up as a proof reader.

Hardware Issues

Betaflight does not manufacture or distribute their own hardware. While we are collaborating with and supported by a number of manufacturers, we do not do any kind of hardware support. If you encounter any hardware issues with your flight controller or another component, please contact the manufacturer or supplier of your hardware, or check RCGroups https://rcgroups.com/forums/showthread.php?t=2464844 to see if others with the same problem have found a solution.

Betaflight Releases

https://github.com/betaflight/betaflight/releases

Open Source / Contributors

Betaflight is software that is open source and is available free of charge without warranty to all users.

Betaflight is forked from Cleanflight, so thanks goes to all those who have contributed to Cleanflight and its origins.

Origins for this fork (Thanks!):

Alexinparis (for MultiWii),
timecop (for Baseflight),
Dominic Clifton (for Cleanflight),
borisbstyle (for Betaflight), and
Sambas (for the original STM32F4 port).

The Betaflight Configurator is forked from Cleanflight Configurator and its origins.

Origins for Betaflight Configurator:

Dominic Clifton (for Cleanflight configurator), and
ctn (for the original Configurator).

Big thanks to current and past contributors:

Budden, Martin (martinbudden)
Bardwell, Joshua (joshuabardwell)
Blackman, Jason (blckmn)
ctzsnooze
Höglund, Anders (andershoglund)
Ledvina, Petr (ledvinap) - IO code awesomeness!
kc10kevin
Keeble, Gary (MadmanK)
Keller, Michael (mikeller) - Configurator brilliance
Kravcov, Albert (skaman82) - Configurator brilliance
MJ666
Nathan (nathantsoi)
ravnav
sambas - bringing us the F4
savaga
Stålheim, Anton (KiteAnton)

And many many others who haven't been mentioned....

arendst/Tasmota

2024-12-01T01:52:00Z

Alternative firmware for ESP8266 and ESP32 based devices with easy configuration using webUI, OTA updates, automation using timers or rules, expandability and entirely local control over MQTT, HTTP, Serial or KNX. Full documentation at

Alternative firmware for ESP8266 and ESP32 based devices with easy configuration using webUI, OTA updates, automation using timers or rules, expandability and entirely local control over MQTT, HTTP, Serial or KNX. Written for PlatformIO.

In light of current events we like to support the people behind PlatformIO Project, especially Ivan Kravets, and wish them the strength to help stop the war. See platformio-is-ukrainian-project-please-help-us-stop-the-war for what you can do.

Easy install

Easy initial installation of Tasmota can be performed using the Tasmota WebInstaller.

If you like Tasmota, give it a star, or fork it and contribute!

See RELEASENOTES.md for release information.

Firmware binaries can be downloaded from http://ota.tasmota.com/tasmota/release/ or http://ota.tasmota.com/tasmota32/release/ for ESP32 binaries.

Development

See CHANGELOG.md for detailed change information.

Unless your Tasmota powered device exhibits a problem or lacks a feature that you need, leave your device alone - it works so don’t make unnecessary changes! If the release version (i.e., the master branch) exhibits unexpected behaviour for your device and configuration, you should upgrade to the latest development version instead to see if your problem is resolved as some bugs in previous releases or development builds may already have been resolved.

Every commit made to the development branch, which is compiling successfully, will post new binary files at http://ota.tasmota.com/tasmota/ (this web address can be used for OTA updates too). It is important to note that these binaries are based on the current development codebase. These commits are tested as much as is possible and are typically quite stable. However, it is infeasible to test on the hundreds of different types of devices with all the available configuration options permitted.

Note that there is a chance, as with any upgrade, that the device may not function as expected. You must always account for the possibility that you may need to flash the device via the serial programming interface if the OTA upgrade fails. Even with the master release, you should always attempt to test the device or a similar prototype before upgrading a device which is in production or is hard to reach. And, as always, make a backup of the device configuration before beginning any firmware update.

Disclaimer

⚠ DANGER OF ELECTROCUTION ⚠

If your device connects to mains electricity (AC power) there is danger of electrocution if not installed properly. If you don't know how to install it, please call an electrician (Beware: certain countries prohibit installation without a licensed electrician present). Remember: SAFETY FIRST. It is not worth the risk to yourself, your family and your home if you don't know exactly what you are doing. Never tinker or try to flash a device using the serial programming interface while it is connected to MAINS ELECTRICITY (AC power).

We don't take any responsibility nor liability for using this software nor for the installation or any tips, advice, videos, etc. given by any member of this site or any related site.

Note

Please do not ask to add new devices unless it requires additional code for new features. If the device is not listed as a module, try using Templates first. If it is not listed in the Tasmota Device Templates Repository create your own Template.

Quick Install

Download one of the released binaries from http://ota.tasmota.com/tasmota/release/ or http://ota.tasmota.com/tasmota32/release/ and flash it to your hardware using our installation guide.

Important User Compilation Information

If you want to compile Tasmota yourself keep in mind the following:

For ESP8285 based devices only Flash Mode DOUT is supported. Do not use Flash Mode DIO / QIO / QOUT as it might seem to brick your device.
For ESP8285 based devices Tasmota uses a 1M linker script WITHOUT spiffs 1M (no SPIFFS) for optimal code space.
To make compile time changes to Tasmota use the user_config_override.h file. It assures keeping your custom settings when you download and compile a new version. You have to make a copy from the provided user_config_override_sample.h file and add your setting overrides.

Configuration Information

Please refer to the installation and configuration articles in our documentation.

Migration Information

See migration path for instructions how to migrate to a major version.

Do not upgrade from minimal to minimal version. It will most likely fail at some point and will require flashing via serial. If you do have to use minimal versions, always OTA to a full version of the same release before applying next minimal version.

Pay attention to the following version breaks due to dynamic settings updates:

Migrate to Sonoff-Tasmota 3.9.x
Migrate to Sonoff-Tasmota 4.x
Migrate to Sonoff-Tasmota 5.14
Migrate to Sonoff-Tasmota 6.7.1 (http://ota.tasmota.com/tasmota/release_6.7.1/sonoff.bin) - NOTICE underscore as a dash is not supported in older versions
Migrate to Tasmota 7.2.0 (http://ota.tasmota.com/tasmota/release-7.2.0/tasmota.bin)

--- Major change in parameter storage layout ---

Migrate to Tasmota 8.5.1 (http://ota.tasmota.com/tasmota/release-8.5.1/tasmota.bin)

--- Major change in internal GPIO function representation ---

Migrate to Tasmota 9.1 (http://ota.tasmota.com/tasmota/release-9.1.0/tasmota.bin.gz)
Upgrade to latest release (http://ota.tasmota.com/tasmota/release/tasmota.bin.gz)

While fallback or downgrading is common practice it was never supported due to Settings additions or changes in newer releases. Starting with release v9.1.0 Imogen the internal GPIO function representation has changed in such a way that fallback is only possible to the latest GPIO configuration before installing v9.1.0.

Support Information

For a database of supported devices see Tasmota Device Templates Repository

If you're looking for support on Tasmota there are some options available:

Documentation

Documentation Site: For information on how to flash Tasmota, configure, use and expand it
FAQ and Troubleshooting: For information on common problems and solutions.
Commands Information: For information on all the commands supported by Tasmota.

Support's Community

Tasmota Discussions: For Tasmota usage questions, Feature Requests and Projects.
Tasmota Users Chat: For support, troubleshooting and general questions. You have better chances to get fast answers from members of the Tasmota Community.
Search in Issues: You might find an answer to your question by searching current or closed issues.
Software Problem Report: For reporting problems of Tasmota Software.

Contribute

You can contribute to Tasmota by

Providing Pull Requests (Features, Proof of Concepts, Language files or Fixes)
Testing new released features and report issues
Donating to acquire hardware for testing and implementing or out of gratitude
Contributing missing documentation for features and devices

Credits

People helping to keep the show on the road:

Sfromis providing extensive user support
Barbudor providing user support and code fixes and additions
David Lang providing initial issue resolution and code optimizations
Heiko Krupp for his IRSend, HTU21, SI70xx and Wemo/Hue emulation drivers
Wiktor Schmidt for Travis CI implementation
Thom Dietrich for PlatformIO optimizations
Marinus van den Broek for his EspEasy groundwork
Pete Ba for more user friendly energy monitor calibration
Lobradov providing compile optimization tips
Flexiti for his initial timer implementation
reloxx13 for his TasmoAdmin management tool
Joachim Banzhaf for his TSL2561 library and driver
Andre Thomas for providing many drivers
Gijs Noorlander for his MHZ19, SenseAir and updated PubSubClient drivers
Erik Montnemery for his HomeAssistant Discovery concept and many code tuning tips
Federico Leoni for continued HomeAssistant Discovery support
Aidan Mountford for his HSB support
Daniel Ztolnai for his Serial Bridge implementation
Gerhard Mutz for multiple sensor & display drivers, Sunrise/Sunset, and scripting
Nuno Ferreira for his HC-SR04 driver
Adrian Scillato for his (security)fixes and implementing and maintaining KNX
Gennaro Tortone for implementing and maintaining Eastron drivers
Raymond Mouthaan for managing Wemos Wiki information
Norbert Richter for his decode-config.py tool
Joel Stein, digiblur and Shantur Rathore for their Tuya research and driver
Frogmore42 for providing many issue answers
Jason2866 for platformio support and providing many issue answers
Blakadder for managing the document site and providing template management
Stephan Hadinger for refactoring light driver, enhancing HueEmulation, LVGL, Zigbee and Berry support
tmo for designing the official Tasmota logo
Stefan Bode for his Shutter and Deep sleep drivers
Jacek Ziółkowski for his TDM management tool and Tasmotizer flashing tool
Christian Staars for NRF24L01 and HM-10 Bluetooth sensor support
Paul Diem for UDP Group communication support
Jörg Schüler-Maroldt for his initial ESP32 port
Javier Arigita for his thermostat driver
Simon Hailes for ESP32 Bluetooth extensions
Many more providing Tips, Wips, Pocs, PRs and Donations

License

This program is licensed under GPL-3.0-only

portapack-mayhem/mayhem-firmware

2024-12-01T01:52:00Z

Custom firmware for the HackRF+PortaPack H1/H2/H4

[!WARNING] IF YOU'VE PAID FOR MAYHEM OR ANY PREPACKAGED VERSIONS, YOU'RE BEING SCAMMED.

The only legitimate link to our repositories is the portapack-mayhem organization on GitHub.

PortaPack Mayhem

This is a fork of the Havoc firmware, which itself was a fork of the PortaPack firmware, an add-on for the HackRF. A fork is a derivate, in this case one that has extra features and fixes when compared to the older versions.

PortaPack H2+HackRF+battery (clone) with a custom 3d printed case

What is this?

If you are new to HackRF+PortaPack+Mayhem, check these:

Frequently Asked Questions

This repository expands upon the previous work by many people and aims to constantly add new features, bugfixes and generate documentation to make further development easier. Collaboration is always welcomed and appreciated.

What to buy?

✔ The fabulous PortaPack H4M Mayhem, featuring numerous improvements and accessories. Sold by our friends at OpenSourceSDRLab. Join their giveaways on discord (check the badge on top).

✔ A recommended one is this PortaPack H2, that includes everything you need with the plastic case "inspired" on this.

✔ Some individuals lean towards the H2 with a metal enclosure, but its advantages remain debated. Share your insights on our wiki.

⚠ Be cautious , ask the seller about compatibility with the latest releases. Look out for the description of the item, if they provide the firmware files for an older version or they have custom setup instructions, this means it might be NOT compatible, for example:

⚠ If it looks too different, this might mean that they are using their own recipe, check the different models in our wiki. For example all the H3 and clones of that version use their own version of the firmware. They do not contribute the changes back and eventually you will be left with a device that nobody maintains:

Where is the latest version?

The current stable release is on the page. Follow the instructions you can find in the release description. The latest (nightly) release can be found here.

How can I collaborate

You can write documentation, fix bugs and answer issues or add new functionality. Please check the following guide with details.

Consider that the hardware and firmware has been created and maintain by a lot of people, so always try collaborating your time and effort first. For coding related questions, if something does not fit as an issue, please join our Discord by clicking the chat badge on top.

To support the people behind the hardware, please buy a genuine HackRF and PortaPack.

What if I really want something specific?

If what you need can be relevant in general, you can request a feature. Alternatively, go to our Discord by clicking the chat badge on top and discuss there.

What if I need help?

First, check the documentation. If you find a bug or you think the problem is related to the current repository, please open an issue.

You can reach the official community in Facebook, and our Discord by clicking the chat badge on top.

hengyoush/kyanos

2024-12-01T01:41:03Z

Visualize the time packets spend in the kernel, watch & analyze in command line.

kyanos

简体中文 | English

English Document

What is kyanos

Kyanos is an eBPF-based network issue analysis tool that enables you to capture network requests, such as HTTP, Redis, and MySQL requests.
It also helps you analyze abnormal network issues and quickly troubleshooting without the complex steps of packet capturing, downloading, and analysis.

Powerful Traffic Filtering: Not only can filter based on traditional IP/port information, can also filter by process/container, L7 protocol information, request/response byte size, latency, and more.

# Filter by pid
./kyanos watch --pids 1234
# Filter by container id
./kyanos watch --container-id abc
# Filter by Redis key
./kyanos watch redis --keys my-key1,my-key2
# Filter by response byte size
./kyanos watch --resp-size 10000

Advanced Analysis Capabilities : Unlike tcpdump, which only provides fine-grained packet capture, Kyanos supports aggregating captured packet metrics across various dimensions, quickly providing the critical data most useful for troubleshooting.
Imagine if the bandwidth of your HTTP service is suddenly maxed out—how would you quickly analyze which IPs and which requests are causing it?
With Kyanos, you just need one command: kyanos stat http --bigresp to find the largest response byte sizes sent to remote IPs and view specific data on request and response metrics.
In-Depth Kernel-Level Latency Details: In real-world, slow queries to remote services like Redis can be challenging to diagnose precisely. Kyanos provides kernel trace points from the arrival of requests/responses at the network card to the kernel socket buffer, displaying these details in a visual format. This allows you to identify exactly which stage is causing delays.

Lightweight and Dependency-Free: Almost zero dependencies—just a single binary file and one command, with all results displayed in the command line.
Automatic SSL Traffic Decryption : All captured requests and responses are presented in plaintext.

Examples

Capture HTTP Traffic with Latency Details

Run the command:

./kyanos watch http

The result is as follows:

Capture Redis Traffic with Latency Details

Run the command:

./kyanos watch redis

The result is as follows:

Identify the Slowest Requests in the Last 5 Seconds

Run the command:

 ./kyanos stat --slow --time 5

The result is as follows:

❗ Requirements

Kyanos currently supports kernel versions 3.10(from 3.10.0-957) and 4.14 or above (with plans to support versions between 4.7 and 4.14 in the future).

You can check your kernel version using uname -r.

🎯 How to get kyanos

You can download a statically linked binary compatible with x86_64 and arm64 architectures from the release page:

tar xvf kyanos_vx.x.x_linux_x86.tar.gz

Then, run:

kyanos watch

If the following table appears: 🎉 Congratulations! Kyanos has started successfully.

📝 Documentation

English Document

⚙ Usage

The simplest usage captures all protocols currently supported by Kyanos:

./kyanos watch

Each request-response record is stored as a row in a table, with each column capturing basic information about that request. You can use the arrow keys or j/k to move up and down through the records:

Press Enter to access the details view:

In the details view, the first section shows Latency Details. Each block represents a "node" that the data packet passes through, such as the process, network card, and socket buffer.
Each block includes a time value indicating the time elapsed from the previous node to this node, showing the process flow from the process sending the request to the network card, to the response being copied to the socket buffer, and finally read by the process, with each step’s duration displayed.

The second section provides Detailed Request and Response Content, split into Request and Response parts, and truncates content over 1024 bytes.

For targeted traffic capture, such as HTTP traffic:

./kyanos watch http

You can narrow it further to capture traffic for a specific HTTP path:

./kyanos watch http --path /abc

Learn more: Kyanos Docs

🤝 Feedback and Contributions

[!IMPORTANT] If you encounter any issues or bugs while using the tool, please feel free to ask questions in the issue tracker.

🗨️ Contacts

For more detailed inquiries, you can use the following contact methods:

My Email: hengyoush1@163.com
My Blog: http://blog.deadlock.cloud

google/AFL

2024-12-01T01:41:03Z

american fuzzy lop - a security-oriented fuzzer

american fuzzy lop

Originally developed by Michal Zalewski lcamtuf@google.com.

See QuickStartGuide.txt if you don't have time to read this file.

1) Challenges of guided fuzzing

Fuzzing is one of the most powerful and proven strategies for identifying security issues in real-world software; it is responsible for the vast majority of remote code execution and privilege escalation bugs found to date in security-critical software.

Unfortunately, fuzzing is also relatively shallow; blind, random mutations make it very unlikely to reach certain code paths in the tested code, leaving some vulnerabilities firmly outside the reach of this technique.

There have been numerous attempts to solve this problem. One of the early approaches - pioneered by Tavis Ormandy - is corpus distillation. The method relies on coverage signals to select a subset of interesting seeds from a massive, high-quality corpus of candidate files, and then fuzz them by traditional means. The approach works exceptionally well, but requires such a corpus to be readily available. In addition, block coverage measurements provide only a very simplistic understanding of program state, and are less useful for guiding the fuzzing effort in the long haul.

Other, more sophisticated research has focused on techniques such as program flow analysis ("concolic execution"), symbolic execution, or static analysis. All these methods are extremely promising in experimental settings, but tend to suffer from reliability and performance problems in practical uses - and currently do not offer a viable alternative to "dumb" fuzzing techniques.

2) The afl-fuzz approach

American Fuzzy Lop is a brute-force fuzzer coupled with an exceedingly simple but rock-solid instrumentation-guided genetic algorithm. It uses a modified form of edge coverage to effortlessly pick up subtle, local-scale changes to program control flow.

Simplifying a bit, the overall algorithm can be summed up as:

Load user-supplied initial test cases into the queue,
Take next input file from the queue,
Attempt to trim the test case to the smallest size that doesn't alter the measured behavior of the program,
Repeatedly mutate the file using a balanced and well-researched variety of traditional fuzzing strategies,
If any of the generated mutations resulted in a new state transition recorded by the instrumentation, add mutated output as a new entry in the queue.
Go to 2.

The discovered test cases are also periodically culled to eliminate ones that have been obsoleted by newer, higher-coverage finds; and undergo several other instrumentation-driven effort minimization steps.

As a side result of the fuzzing process, the tool creates a small, self-contained corpus of interesting test cases. These are extremely useful for seeding other, labor- or resource-intensive testing regimes - for example, for stress-testing browsers, office applications, graphics suites, or closed-source tools.

The fuzzer is thoroughly tested to deliver out-of-the-box performance far superior to blind fuzzing or coverage-only tools.

3) Instrumenting programs for use with AFL

When source code is available, instrumentation can be injected by a companion tool that works as a drop-in replacement for gcc or clang in any standard build process for third-party code.

The instrumentation has a fairly modest performance impact; in conjunction with other optimizations implemented by afl-fuzz, most programs can be fuzzed as fast or even faster than possible with traditional tools.

The correct way to recompile the target program may vary depending on the specifics of the build process, but a nearly-universal approach would be:

$ CC=/path/to/afl/afl-gcc ./configure
$ make clean all

For C++ programs, you'd would also want to set CXX=/path/to/afl/afl-g++.

The clang wrappers (afl-clang and afl-clang++) can be used in the same way; clang users may also opt to leverage a higher-performance instrumentation mode, as described in llvm_mode/README.llvm.

When testing libraries, you need to find or write a simple program that reads data from stdin or from a file and passes it to the tested library. In such a case, it is essential to link this executable against a static version of the instrumented library, or to make sure that the correct .so file is loaded at runtime (usually by setting LD_LIBRARY_PATH). The simplest option is a static build, usually possible via:

$ CC=/path/to/afl/afl-gcc ./configure --disable-shared

Setting AFL_HARDEN=1 when calling 'make' will cause the CC wrapper to automatically enable code hardening options that make it easier to detect simple memory bugs. Libdislocator, a helper library included with AFL (see libdislocator/README.dislocator) can help uncover heap corruption issues, too.

PS. ASAN users are advised to review notes_for_asan.txt file for important caveats.

4) Instrumenting binary-only apps

When source code is NOT available, the fuzzer offers experimental support for fast, on-the-fly instrumentation of black-box binaries. This is accomplished with a version of QEMU running in the lesser-known "user space emulation" mode.

QEMU is a project separate from AFL, but you can conveniently build the feature by doing:

$ cd qemu_mode
$ ./build_qemu_support.sh

For additional instructions and caveats, see qemu_mode/README.qemu.

The mode is approximately 2-5x slower than compile-time instrumentation, is less conducive to parallelization, and may have some other quirks.

5) Choosing initial test cases

To operate correctly, the fuzzer requires one or more starting file that contains a good example of the input data normally expected by the targeted application. There are two basic rules:

Keep the files small. Under 1 kB is ideal, although not strictly necessary. For a discussion of why size matters, see perf_tips.txt.
Use multiple test cases only if they are functionally different from each other. There is no point in using fifty different vacation photos to fuzz an image library.

You can find many good examples of starting files in the testcases/ subdirectory that comes with this tool.

PS. If a large corpus of data is available for screening, you may want to use the afl-cmin utility to identify a subset of functionally distinct files that exercise different code paths in the target binary.

6) Fuzzing binaries

The fuzzing process itself is carried out by the afl-fuzz utility. This program requires a read-only directory with initial test cases, a separate place to store its findings, plus a path to the binary to test.

For target binaries that accept input directly from stdin, the usual syntax is:

$ ./afl-fuzz -i testcase_dir -o findings_dir /path/to/program [...params...]

For programs that take input from a file, use '@@' to mark the location in the target's command line where the input file name should be placed. The fuzzer will substitute this for you:

$ ./afl-fuzz -i testcase_dir -o findings_dir /path/to/program @@

You can also use the -f option to have the mutated data written to a specific file. This is useful if the program expects a particular file extension or so.

Non-instrumented binaries can be fuzzed in the QEMU mode (add -Q in the command line) or in a traditional, blind-fuzzer mode (specify -n).

You can use -t and -m to override the default timeout and memory limit for the executed process; rare examples of targets that may need these settings touched include compilers and video decoders.

Tips for optimizing fuzzing performance are discussed in perf_tips.txt.

Note that afl-fuzz starts by performing an array of deterministic fuzzing steps, which can take several days, but tend to produce neat test cases. If you want quick & dirty results right away - akin to zzuf and other traditional fuzzers - add the -d option to the command line.

7) Interpreting output

See the status_screen.txt file for information on how to interpret the displayed stats and monitor the health of the process. Be sure to consult this file especially if any UI elements are highlighted in red.

The fuzzing process will continue until you press Ctrl-C. At minimum, you want to allow the fuzzer to complete one queue cycle, which may take anywhere from a couple of hours to a week or so.

There are three subdirectories created within the output directory and updated in real time:

queue/ - test cases for every distinctive execution path, plus all the starting files given by the user. This is the synthesized corpus mentioned in section 2. Before using this corpus for any other purposes, you can shrink it to a smaller size using the afl-cmin tool. The tool will find a smaller subset of files offering equivalent edge coverage.
crashes/ - unique test cases that cause the tested program to receive a fatal signal (e.g., SIGSEGV, SIGILL, SIGABRT). The entries are grouped by the received signal.
hangs/ - unique test cases that cause the tested program to time out. The default time limit before something is classified as a hang is the larger of 1 second and the value of the -t parameter. The value can be fine-tuned by setting AFL_HANG_TMOUT, but this is rarely necessary.

Crashes and hangs are considered "unique" if the associated execution paths involve any state transitions not seen in previously-recorded faults. If a single bug can be reached in multiple ways, there will be some count inflation early in the process, but this should quickly taper off.

The file names for crashes and hangs are correlated with parent, non-faulting queue entries. This should help with debugging.

When you can't reproduce a crash found by afl-fuzz, the most likely cause is that you are not setting the same memory limit as used by the tool. Try:

$ LIMIT_MB=50
$ ( ulimit -Sv $[LIMIT_MB << 10]; /path/to/tested_binary ... )

Change LIMIT_MB to match the -m parameter passed to afl-fuzz. On OpenBSD, also change -Sv to -Sd.

Any existing output directory can be also used to resume aborted jobs; try:

$ ./afl-fuzz -i- -o existing_output_dir [...etc...]

If you have gnuplot installed, you can also generate some pretty graphs for any active fuzzing task using afl-plot. For an example of how this looks like, see http://lcamtuf.coredump.cx/afl/plot/.

8) Parallelized fuzzing

Every instance of afl-fuzz takes up roughly one core. This means that on multi-core systems, parallelization is necessary to fully utilize the hardware. For tips on how to fuzz a common target on multiple cores or multiple networked machines, please refer to parallel_fuzzing.txt.

The parallel fuzzing mode also offers a simple way for interfacing AFL to other fuzzers, to symbolic or concolic execution engines, and so forth; again, see the last section of parallel_fuzzing.txt for tips.

9) Fuzzer dictionaries

By default, afl-fuzz mutation engine is optimized for compact data formats - say, images, multimedia, compressed data, regular expression syntax, or shell scripts. It is somewhat less suited for languages with particularly verbose and redundant verbiage - notably including HTML, SQL, or JavaScript.

To avoid the hassle of building syntax-aware tools, afl-fuzz provides a way to seed the fuzzing process with an optional dictionary of language keywords, magic headers, or other special tokens associated with the targeted data type -- and use that to reconstruct the underlying grammar on the go:

http://lcamtuf.blogspot.com/2015/01/afl-fuzz-making-up-grammar-with.html

To use this feature, you first need to create a dictionary in one of the two formats discussed in dictionaries/README.dictionaries; and then point the fuzzer to it via the -x option in the command line.

(Several common dictionaries are already provided in that subdirectory, too.)

There is no way to provide more structured descriptions of the underlying syntax, but the fuzzer will likely figure out some of this based on the instrumentation feedback alone. This actually works in practice, say:

http://lcamtuf.blogspot.com/2015/04/finding-bugs-in-sqlite-easy-way.html

PS. Even when no explicit dictionary is given, afl-fuzz will try to extract existing syntax tokens in the input corpus by watching the instrumentation very closely during deterministic byte flips. This works for some types of parsers and grammars, but isn't nearly as good as the -x mode.

If a dictionary is really hard to come by, another option is to let AFL run for a while, and then use the token capture library that comes as a companion utility with AFL. For that, see libtokencap/README.tokencap.

10) Crash triage

The coverage-based grouping of crashes usually produces a small data set that can be quickly triaged manually or with a very simple GDB or Valgrind script. Every crash is also traceable to its parent non-crashing test case in the queue, making it easier to diagnose faults.

Having said that, it's important to acknowledge that some fuzzing crashes can be difficult to quickly evaluate for exploitability without a lot of debugging and code analysis work. To assist with this task, afl-fuzz supports a very unique "crash exploration" mode enabled with the -C flag.

In this mode, the fuzzer takes one or more crashing test cases as the input, and uses its feedback-driven fuzzing strategies to very quickly enumerate all code paths that can be reached in the program while keeping it in the crashing state.

Mutations that do not result in a crash are rejected; so are any changes that do not affect the execution path.

The output is a small corpus of files that can be very rapidly examined to see what degree of control the attacker has over the faulting address, or whether it is possible to get past an initial out-of-bounds read - and see what lies beneath.

Oh, one more thing: for test case minimization, give afl-tmin a try. The tool can be operated in a very simple way:

$ ./afl-tmin -i test_case -o minimized_result -- /path/to/program [...]

The tool works with crashing and non-crashing test cases alike. In the crash mode, it will happily accept instrumented and non-instrumented binaries. In the non-crashing mode, the minimizer relies on standard AFL instrumentation to make the file simpler without altering the execution path.

The minimizer accepts the -m, -t, -f and @@ syntax in a manner compatible with afl-fuzz.

Another recent addition to AFL is the afl-analyze tool. It takes an input file, attempts to sequentially flip bytes, and observes the behavior of the tested program. It then color-codes the input based on which sections appear to be critical, and which are not; while not bulletproof, it can often offer quick insights into complex file formats. More info about its operation can be found near the end of technical_details.txt.

11) Going beyond crashes

Fuzzing is a wonderful and underutilized technique for discovering non-crashing design and implementation errors, too. Quite a few interesting bugs have been found by modifying the target programs to call abort() when, say:

Two bignum libraries produce different outputs when given the same fuzzer-generated input,
An image library produces different outputs when asked to decode the same input image several times in a row,
A serialization / deserialization library fails to produce stable outputs when iteratively serializing and deserializing fuzzer-supplied data,
A compression library produces an output inconsistent with the input file when asked to compress and then decompress a particular blob.

Implementing these or similar sanity checks usually takes very little time; if you are the maintainer of a particular package, you can make this code conditional with #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION (a flag also shared with libfuzzer) or #ifdef __AFL_COMPILER (this one is just for AFL).

12) Common-sense risks

Please keep in mind that, similarly to many other computationally-intensive tasks, fuzzing may put strain on your hardware and on the OS. In particular:

Your CPU will run hot and will need adequate cooling. In most cases, if cooling is insufficient or stops working properly, CPU speeds will be automatically throttled. That said, especially when fuzzing on less suitable hardware (laptops, smartphones, etc), it's not entirely impossible for something to blow up.
Targeted programs may end up erratically grabbing gigabytes of memory or filling up disk space with junk files. AFL tries to enforce basic memory limits, but can't prevent each and every possible mishap. The bottom line is that you shouldn't be fuzzing on systems where the prospect of data loss is not an acceptable risk.
Fuzzing involves billions of reads and writes to the filesystem. On modern systems, this will be usually heavily cached, resulting in fairly modest "physical" I/O - but there are many factors that may alter this equation. It is your responsibility to monitor for potential trouble; with very heavy I/O, the lifespan of many HDDs and SSDs may be reduced.

A good way to monitor disk I/O on Linux is the 'iostat' command:

    $ iostat -d 3 -x -k [...optional disk ID...]

13) Known limitations & areas for improvement

Here are some of the most important caveats for AFL:

AFL detects faults by checking for the first spawned process dying due to a signal (SIGSEGV, SIGABRT, etc). Programs that install custom handlers for these signals may need to have the relevant code commented out. In the same vein, faults in child processed spawned by the fuzzed target may evade detection unless you manually add some code to catch that.
As with any other brute-force tool, the fuzzer offers limited coverage if encryption, checksums, cryptographic signatures, or compression are used to wholly wrap the actual data format to be tested.

To work around this, you can comment out the relevant checks (see experimental/libpng_no_checksum/ for inspiration); if this is not possible, you can also write a postprocessor, as explained in experimental/post_library/.
There are some unfortunate trade-offs with ASAN and 64-bit binaries. This isn't due to any specific fault of afl-fuzz; see notes_for_asan.txt for tips.
There is no direct support for fuzzing network services, background daemons, or interactive apps that require UI interaction to work. You may need to make simple code changes to make them behave in a more traditional way. Preeny may offer a relatively simple option, too - see: https://github.com/zardus/preeny

Some useful tips for modifying network-based services can be also found at: https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop
AFL doesn't output human-readable coverage data. If you want to monitor coverage, use afl-cov from Michael Rash: https://github.com/mrash/afl-cov
Occasionally, sentient machines rise against their creators. If this happens to you, please consult http://lcamtuf.coredump.cx/prep/.

Beyond this, see INSTALL for platform-specific tips.

14) Special thanks

Many of the improvements to afl-fuzz wouldn't be possible without feedback, bug reports, or patches from:

  Jann Horn                             Hanno Boeck
  Felix Groebert                        Jakub Wilk
  Richard W. M. Jones                   Alexander Cherepanov
  Tom Ritter                            Hovik Manucharyan
  Sebastian Roschke                     Eberhard Mattes
  Padraig Brady                         Ben Laurie
  @dronesec                             Luca Barbato
  Tobias Ospelt                         Thomas Jarosch
  Martin Carpenter                      Mudge Zatko
  Joe Zbiciak                           Ryan Govostes
  Michael Rash                          William Robinet
  Jonathan Gray                         Filipe Cabecinhas
  Nico Weber                            Jodie Cunningham
  Andrew Griffiths                      Parker Thompson
  Jonathan Neuschfer                    Tyler Nighswander
  Ben Nagy                              Samir Aguiar
  Aidan Thornton                        Aleksandar Nikolich
  Sam Hakim                             Laszlo Szekeres
  David A. Wheeler                      Turo Lamminen
  Andreas Stieger                       Richard Godbee
  Louis Dassy                           teor2345
  Alex Moneger                          Dmitry Vyukov
  Keegan McAllister                     Kostya Serebryany
  Richo Healey                          Martijn Bogaard
  rc0r                                  Jonathan Foote
  Christian Holler                      Dominique Pelle
  Jacek Wielemborek                     Leo Barnes
  Jeremy Barnes                         Jeff Trull
  Guillaume Endignoux                   ilovezfs
  Daniel Godas-Lopez                    Franjo Ivancic
  Austin Seipp                          Daniel Komaromy
  Daniel Binderman                      Jonathan Metzman
  Vegard Nossum                         Jan Kneschke
  Kurt Roeckx                           Marcel Bohme
  Van-Thuan Pham                        Abhik Roychoudhury
  Joshua J. Drake                       Toby Hutton
  Rene Freingruber                      Sergey Davidoff
  Sami Liedes                           Craig Young
  Andrzej Jackowski                     Daniel Hodson

Thank you!

15) Contact

Questions? Concerns? Bug reports? Please use GitHub.

There is also a mailing list for the project; to join, send a mail to afl-users+subscribe@googlegroups.com. Or, if you prefer to browse archives first, try: https://groups.google.com/group/afl-users.

apache/cloudberry

2024-12-01T01:41:03Z

One advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.

Apache Cloudberry (Incubating)

Introduction

Apache Cloudberry (Incubating), created by the original developers of Greenplum Database, is one advanced and mature open-source Massively Parallel Processing (MPP) database, which evolves from the open-source version of the Pivotal Greenplum Database®️ but features a newer PostgreSQL kernel and more advanced enterprise capabilities. It can serve as a data warehouse and can also be used for large-scale analytics and AI/ML workloads.

Build and try out

Build from source

You can follow these guides to build Cloudberry on Linux OS (including RHEL/Rocky Linux, and Ubuntu) and macOS.

Try out quickly

Welcome to try out Cloudberry via building one Docker-based Sandbox, which is tailored to help you gain a basic understanding of Cloudberry's capabilities and features a range of materials, including tutorials, sample code, and crash courses.

Repositories

This is the main repository for Apache Cloudberry (Incubating). Alongside this, there are several ecosystem repositories for Cloudberry, including the website, extensions, connectors, adapters, and other utilities.

apache/cloudberry-site: website and documentation sources.
apache/cloudberry-bootcamp: help you quickly try out Cloudberry via one Docker-based Sandbox.
apache/cloudberry-gpbackup: backup utility for Cloudberry.
apache/cloudberry-go-libs: go-libs for Cloudberry.
apache/cloudberry-gpbackup-s3-plugin: S3 plugin for use with Cloudberry backup utility.
apache/cloudberry-pxf: Platform Extension Framework (PXF) for Cloudberry.

Community & Support

We have many channels for community members to discuss, ask for help, feedback, and chat:

Type	Description
Slack	Click to Join the real-time chat on Slack for QA, Dev, Events, and more. Don't miss out! Check out the Slack guide to learn more.
Q&A	Ask for help when running/developing Cloudberry, visit GitHub Discussions - QA.
New ideas / Feature Requests	Share ideas for new features, visit GitHub Discussions - Ideas.
Report bugs	Problems and issues in Apache Cloudberry core. If you find bugs, welcome to submit them here.
Report a security vulnerability	View our security policy to learn how to report and contact us.
Community events	Including meetups, webinars, conferences, and more events, visit the Events page and subscribe events calendar.
Documentation	Official documentation for Cloudberry. You can explore it to discover more details about us.

Contribution

Contributions can be diverse, such as code enhancements, bug fixes, feature proposals, documents, marketing, and so on. No contribution is too small, we encourage all types of contributions. Cloudberry community welcomes contributions from anyone, new and experienced! Our contribution guide will help you get started with the contribution.

Type	Description
Code contribution	Learn how to contribute code to the Cloudberry, including coding preparation, conventions, workflow, review, and checklist following the code contribution guide.
Submit the proposal	Proposing major changes to Cloudberry through proposal guide.
Doc contribution	We need you to join us to help us improve the documentation, see the doc contribution guide.

Roadmap

You can check our Cloudberry Roadmap 2024 out to see the product plans and goals we want to achieve in 2024. Welcome to share your thoughts and ideas to join us in shaping the future of Apache Cloudberry (Incubating). (We will update the Roadmap after entering the Incubator.)

Acknowledgment

Thanks to PostgreSQL, Greenplum Database and other great open source projects to make Apache Cloudberry has a sound foundation.

License

Cloudberry is licensed under the Apache License, Version 2.0. For details, see the LICENSE.

ASF Incubator disclaimer

Apache Cloudberry is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required for all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

ecmwf/ecbuild

2024-12-01T01:52:26Z

A CMake-based build system, consisting of a collection of CMake macros and functions that ease the managing of software build systems

oozcitak/xmlbuilder-js

2024-11-30T01:30:02Z

An XML builder for node.js

SawfishWM/sawfish

2024-12-01T01:41:34Z

Sawfish Window-Manager

This is sawfish, a highly configurable window manager for X11.

It is released under the terms of the GNU GPL, copyright (c) John Harper jsh@users.sourceforge.net

Maintainer until 2007 was: John Harper Maintainer until Dec 2008 was: Janek Kozicki cosurgi@gmail.com Maintainer since Dec 2008 is: Christopher Bratusek nano@jpberlin.de

For more details, see the sawfish wiki: http://sawfish.wikia.com/

Manifesto

Sawfish is an extensible window manager using an Emacs Lisp-like scripting language--all window decorations are configurable, the basic idea is to have as much user-interface policy as possible controlled through the Lisp language.

Despite this extensibility its policy is currently very minimal compared to most window managers. Its aim is simply to manage windows in the most flexible and attractive manner possible. As such it does not implement desktop backgrounds, applications docks, or other things that may be achieved through separate applications.

All high-level wm functions are implemented in Lisp for future extensibility or redefinition. Currently this includes menus (using GTK+), interactive window moving and resizing, virtual desktop, iconification, focus/transient window policies, frame theme definitions and much more.

Also, most received events are exported to the Lisp environment through ``key-bindings'' and hooks, similar to in Emacs. These events include pointer behaviour and many internal X11 events (enter/leave, focus-in/focus-out, map/unmap, etc..)

sindresorhus/github-markdown-css

2024-12-01T01:52:38Z

The minimal amount of CSS to replicate the GitHub Markdown style

github-markdown-css

The minimal amount of CSS to replicate the GitHub Markdown style

The CSS is generated. Contributions should go to this repo.

Demo

Install

Download manually, from CDNJS, or with npm:

npm install github-markdown-css

Usage

Import the github-markdown.css file and add a markdown-body class to the container of your rendered Markdown and set a width for it. GitHub uses 980px width and 45px padding, and 15px padding for mobile.

<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="github-markdown.css">
<style>
	.markdown-body {
		box-sizing: border-box;
		min-width: 200px;
		max-width: 980px;
		margin: 0 auto;
		padding: 45px;
	}

	@media (max-width: 767px) {
		.markdown-body {
			padding: 15px;
		}
	}
</style>
<article class="markdown-body">
	<h1>Unicorns</h1>
	<p>All the things</p>
</article>

You can use GitHub's /markdown API to turn Markdown into the HTML that GitHub generates, which works well with the CSS in this repo. Other Markdown parsers will mostly work with these styles too. To mimic how GitHub highlights code, you can use starry-night with your Markdown parser of choice.

There are 3 themes provided in this package:

github-markdown.css: (default) Automatically switches between light and dark through @media (prefers-color-scheme).
github-markdown-light.css: Light-only.
github-markdown-dark.css: Dark-only.

You may know that now GitHub supports more than 2 themes including dark_dimmed, dark_high_contrast and colorblind variants. If you want to try these themes, you can generate them on your own! See next section.

How

See generate-github-markdown-css for how it's generated and ability to generate your own.

Dev

Run npm run make to update the CSS.

IBM/plex

2024-12-01T01:52:38Z

The package of IBM’s typeface, IBM Plex.

IBM Plex® typeface

Meet the IBM Plex® typeface, our corporate typeface family. It’s global, it’s versatile and it’s distinctly IBM.

We designed the IBM Plex typeface carefully to meet our needs as a global tech company and express who we are as IBMers. It took two years and a lot of work to get here, but today, we have a signature typeface we’re proud of and excited to share with the world. Discover more about our development of the IBM Plex typeface.

The IBM Plex typeface is an open-source project available for download and various uses following the Open Font License (OFL). The IBM Plex family comes in Sans, Serif, Mono, and Sans Condensed, all with Roman and true italics. Plex has been designed to work well in user interface (UI) environments and other mediums. This project provides all source files and multiple formats to support most typographical situations. Currently, IBM Plex Sans supports extended Latin, Arabic, Chinese (Traditional), Cyrillic, Devanagari, Greek, Hebrew, Japanese, Korean, and Thai. The Chinese SC (Simplified) version will be available in Q4 2024.

Additionally, we have introduced IBM Plex® Math. This highly anticipated release includes over 5,000 new glyphs, covering a wide range of mathematical symbols such as alphanumeric, double-struck, Fraktur, operators, script, icons, arrows, Greek letters, phonetics, technical and geometric shapes — making it one of the most complete math fonts available today. IBM Plex Math provides a fresh and comprehensive alternative to STIX and Microsoft’s Cambria. Its compatibility with IBM Plex Serif Regular makes it an ideal choice for IBM researchers and mathematicians.

Thanks for trying the IBM Plex typeface! We hope you like it.

Add the IBM Plex typeface to your device

Please download the latest zip files from our releases page for installation.

IBM Plex typeface packages

Package	NPM
IBM Plex Math	@ibm/plex-math
IBM Plex Mono	@ibm/plex-mono
IBM Plex Sans	@ibm/plex-sans
IBM Plex Sans Arabic	@ibm/plex-sans-arabic
IBM Plex Sans Condensed	@ibm/plex-sans-condensed
IBM Plex Sans Devanagari	@ibm/plex-sans-devanagari
IBM Plex Sans Hebrew	@ibm/plex-sans-hebrew
IBM Plex Sans Japanese	@ibm/plex-sans-jp
IBM Plex Sans Korean	@ibm/plex-sans-kr
IBM Plex Sans Chinese TC	@ibm/plex-sans-tc
IBM Plex Sans Thai	@ibm/plex-sans-thai
IBM Plex Sans Thai Looped	@ibm/plex-sans-thai-looped
IBM Plex Serif	@ibm/plex-serif

primefaces/sakai-react

2024-12-01T01:52:38Z

Free React Admin Template

This is a Next.js project bootstrapped with create-next-app.

Getting Started

First, run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev

Open http://localhost:3000 with your browser to see the result.

Learn More

To learn more about Next.js, take a look at the following resources:

Next.js Documentation - learn about Next.js features and API.
Learn Next.js - an interactive Next.js tutorial.

MeiK2333/github-style

2024-12-01T01:41:42Z

github-style

Init hugo site

hugo new site mysite
cd mysite

Install the theme

git submodule add git@github.com:MeiK2333/github-style.git themes/github-style

Update the theme

If you just installed the theme, it is already in the latest version. If not, you can update using the below commands

cd themes/github-style
git pull

Then, you need to rename the previous posts folder to post

cd <you-project-folder>
mv content/posts content/post

Setup readme

hugo new readme.md
echo '`Hello World!`' > content/readme.md

Pin post

---
pin: true
---

Add new post

Hugo will create a post with draft: true, change it to false in order for it to show in the website.

hugo new post/title_of_the_post.md

Limit display content

Approach 1: use summary

---
title: "title"
date: 2019-10-22T18:46:47+08:00
draft: false
summary: "The summary content"
---

Approach 2: use ``

Use  to separate content that will display in the posts page as abstraction and the rest of the content. This is different from summary, as summary will not appear in the post.

---
title: "title"
date: 2019-10-22T18:46:47+08:00
draft: false
---
abstraction show in the post page
<!--more-->
other content

Add last modified date

add to config.toml

lastmod = true

[frontmatter]
  lastmod = ["lastmod", ":fileModTime", ":default"]

Use gitalk to support comments

add to config.toml

enableGitalk = true

  [params.gitalk]
    clientID = "Your client ID"
    clientSecret = "Your client secret"
    repo = "repo"
    owner = "Your Github username"
    admin = "Your Github username"
    id = "location.pathname"
    labels = "gitalk"
    perPage = 30
    pagerDirection = "last"
    createIssueManually = true
    distractionFreeMode = false

Support LaTeX

In you post add math: true to front matter

---
katex: math
---

Then the katex script will auto render the string enclosed by delimiters.

# replace ... with latex formula
display inline \\( ... \\)
display block $$ ... $$

Support MathJax

you can add MathJax:true to frontmatter

mathJax: true

config.toml example

baseURL = "https://meik2333.com/"
languageCode = "zh-cn"
title = "MeiK's blog"
theme = "github-style"
pygmentsCodeFences = true
pygmentsUseClasses = true

[params]
  author = "MeiK"
  description = "In solitude, where we are least alone."
  github = "MeiK2333"
  facebook = "MeiK2333"
  twitter = "MeiK2333"
  linkedin = "MeiK2333"
  instagram = "MeiK2333"
  tumblr = "MeiK2333"
  email = "meik2333@gmail.com"
  url = "https://meik2333.com"
  keywords = "blog, google analytics"
  rss = true
  lastmod = true
  userStatusEmoji = "😀"
  favicon = "/images/github.png"
  avatar = "/images/avatar.png"
  headerIcon = "/images/GitHub-Mark-Light-32px.png"
  location = "China"
  enableGitalk = true

  [params.gitalk]
    clientID = "Your client ID"
    clientSecret = "Your client secret"
    repo = "repo"
    owner = "MeiK2333"
    admin = "MeiK2333"
    id = "location.pathname"
    labels = "gitalk"
    perPage = 15
    pagerDirection = "last"
    createIssueManually = true
    distractionFreeMode = false

  [[params.links]]
    title = "Link"
    href = "https://github.com/meik2333"
  [[params.links]]
    title = "Link2"
    href = "https://meik2333.com"
    icon = "https://meik2333.com/images/avatar.png"

[frontmatter]
  lastmod = ["lastmod", ":fileModTime", ":default"]

[services]
  [services.googleAnalytics]
    ID = "UA-123456-789"

Support collapsible block

You can create a collapsible block like this:

{{<details "summary title">}}

block content

{{</details>}}

And it will show like this:

summary title

block content

Support local search

add to config.toml

[params]
  enableSearch = true

[outputs]
  home = ["html", "json"]

[outputFormats.json]
  mediaType = "application/json"
  baseName = "index"
  isPlainText = false

We can do local search now, it is implemented by fuse.js.

deploy.sh example

There are various way to deploy to github, here is a link to official document.

Here is an sample. Note line 22 have env HUGO_ENV="production", makes sure googleAnalysis is loaded during production, but is not loaded when we are testing it in localhost.

#!/bin/sh

if [ "`git status -s`" ]
then
    echo "The working directory is dirty. Please commit any pending changes."
    exit 1;
fi

echo "Deleting old publication"
rm -rf public
mkdir public
git worktree prune
rm -rf .git/worktrees/public/

echo "Checking out gh-pages branch into public"
git worktree add -B gh-pages public origin/gh-pages

echo "Removing existing files"
rm -rf public/*

echo "Generating site"
env HUGO_ENV="production" hugo -t github-style

echo "Updating gh-pages branch"
cd public && git add --all && git commit -m "Publishing to gh-pages (publish.sh)"

#echo "Pushing to github"
#git push --all

Then you can verify the site is working and use git push --all to push the change to github. If you don't want to check again every time, you can uncomment the #git push --all in the script.

TODO

重写标题导航，那玩意儿引入的 JS 在控制台报错。

efeslab/Nanoflow

2024-12-01T01:41:41Z

A throughput-oriented high-performance serving framework for LLMs

yunweneric/flutter-widgethub

2024-11-30T01:30:22Z

From ready-to-use components and flexible layout blocks to smooth animations and striking effects, Flutter Widget Hub is a collection of some pretty UIS aimed at helping developers create beautiful, functional flutter apps quickly and efficiently.

Welcome to Flutter Widget Hub Build Stunning Flutter Apps with Ease

Features

Templates

A wide range of pre-built UI templates from app clones to demo apps all in one place
Blocs

A collection of reusable layout blocks to help you create consistent and responsive layouts
Animations

Smooth and captivating animations that can be easily applied to any element.
Effects

Visual effects ranging from shadows and gradients to more complex transformations

Setup

Version support

FVM is compiled in flutter version 3.22.1. You can use FVM to manage your flutter version
Running app

You can find the main_dev.dart and main_prod.dart files in the lib/app/core folder. In the .vscode folder, you can find the launch config for dev and prod environment

NB: Ensure you have an env.json or dev.env.json file out of your project in a folder called envs/fwh
Building app with wasm support

flutter build web --wasm --target lib/app/core/main_dev.dart --dart-define-from-file=../envs/fwh/dev.env.json

flutter build web --wasm --target lib/app/core/main_prod.dart --dart-define-from-file=../envs/fwh/env.json

Request New Component

If you will like the community to build a custom component tailored for a specific use case, you can Request a component that and label it accordingly.

Contribution Guide

We welcome contributions from the community to make this SDK better! Find the complete Contribution guide here

Code of Conduct

Please note that this project is governed by a Code of Conduct. We expect all contributors and users to adhere to it. The Code of Conduct can be found in the CODE_OF_CONDUCT.md file in this repository.

License

This Library is released under the MIT License. You can find the license details in the LICENSE file.

Support

For any questions, issues, or feature requests, feel free to create an issue in this repository or contact us at yunweneric@gmail.com.

Acknowledgements

We would like to thank all contributors who have helped in making flutter widget hub project possible.

HemantKArya/BloomeeTunes

2024-12-01T01:52:47Z

🌸Bloomee is a cross-platform music app designed to bring you ad-free tunes from various sources. 🌼🎵

Bloomee🌸

Bloomee is experimental cross-platform open Source Music player designed to bring you Ad-free tunes from various sources. Dive into a world of limitless music from platforms like YouTube and Jio Saavn, with more sources blooming soon! 🌼🎵

Why Bloomee?

🌟 Ad-Free Experience: Say goodbye to interruptions and enjoy uninterrupted musical bliss.

🌍 Multi-Source Player: Access your favorite tracks from diverse platforms, with more sources continually joining our melody garden.

🚀 Flutter-Powered Learning: Bloomee is not just about music; it's about learning and growing with Flutter and BLoC architecture. Explore the intersection of beautiful design and smooth functionality while mastering the art of app development.

Features

🚫 Ad-Free Music
📝 Lyrics Support (Time Synced)
📊 Scrobble music with Last.FM
🎵 Offline Music Experience
🔽 Import playlists from various source.
⏲️ Sleep Timer
🎧 Personalized Playlist Creation and Sharing
🌐 Daily Updated Global Charts. (Billboard, Last.fm and more.)
🖥️ Support for Android, Windows and Linux.
📉 Minimal Data Usage
💾 Space Efficient
🏃 Lightweight App
🔓 Open Source
💿 Play Automatic Related Songs
🎸 Share your playlists with others
🎼 AI-Generated Playlist
💡 AI-Based Recommendations
🆎 Multi-Language support

Download for Android, Windows & Linux(dev) 😍

Now you can support me via

Contribute to BloomeeTunes! 🎶

🌱 Every Note Counts: Your contribution, no matter how small, adds to the richness of Bloomee. Whether you're fixing a bug, enhancing features, or suggesting improvements, your notes in our melody matter. Also I'm new to flutter, So every contribution will help me and this project.

🚀 Learn and Grow: Contribute to Bloomee and enhance your Flutter and BLoC skills. Every pull request is an opportunity to learn, and we encourage contributors of all levels to join the journey.

How to Contribute:

Fork the Repository: Start your Bloomee journey by forking main(Branch).
Clone Locally: Clone the forked repository to your local machine.
Create a Branch: Create a new branch for your contribution.
Make Changes: Make your contribution - fix a bug, add a feature, or improve the documentation.
Push Changes: Push your changes to your forked repository.
Create a Pull Request: Submit a pull request, and let your notes join the Bloomee Symphony!

Get in touch with me at :-

DJDoubleD/refreezer

2024-12-01T01:52:47Z

An alternative Deezer music streaming & downloading client, based on Freezer.

An alternative Deezer music streaming & downloading client, based on Freezer. The entire codebase has been updated/rewritten to be compatible with the latest version of flutter, the dart SDK & android (current build target is API level 34).

Screenshots

More Android Phone

Android Auto

Features & changes

Restored all features of the old Freezer app, most notably:
- Restored all login options
- Restored Highest quality streaming and download options (premium account required, free accounts limited to MP3 128kbps)
Support downloading to external storage (sdcard) for android 11 and up
Restored homescreen and added new Flow & Mood smart playlist options
Fixed Log-out (no need for restart anymore)
Improved/fixed queue screen and queue handling (shuffle & rearranging)
Updated lyrics screen to also support unsynced lyrics
Some minor UI changes to better accomadate horizontal/tablet view
Updated entire codebase to fully support latest flutter & dart SDK versions
Updated to gradle version 8.5.1
Removed included c libraries (openssl & opencrypto) and replaced them with custom native java implementation
Replaced the included decryptor-jni c library with a custom native java implementation
Implemented null-safety
Removed the need of custom just_audio & audio_service plugin versions & refactored source code to use the latest version of the official plugins
Multiple other fixes

Compile from source

Install the latest flutter SDK: https://flutter.dev/docs/get-started/install
(Optional) Generate keys for release build: https://flutter.dev/docs/deployment/android

Download source:

git clone https://github.com/DJDoubleD/ReFreezer
git submodule init
git submodule update

Create a .env file in the lib folder of the project and add the following content:

# Deezer API credentials
deezerClientId = '<Your_Deezer_Client_Id>';
deezerClientSecret = '<Your_Deezer_Client_Secret>';

# LastFM API credentials
lastFmApiKey = '<Your_LastFM_API_Key>'
lastFmApiSecret = 'Your_LastFM_API_Secret'

Build generated files:

Use following script to (re)build generated classes in submodules and main project:

.\run_build_runner.ps1

or run these commands manually in the relevant submodules to (re)build the generated files:

flutter pub get
dart run build_runner clean
dart run build_runner build --delete-conflicting-outputs

Compile:

 flutter build apk --split-per-abi --release

NOTE: You have to use own keys, or build debug using flutter build apk --debug

Generate your own keys

Step 1: create a JKS file

Make sure your Java JDK folder is in your PATH variable and execute the following command in the ./android folder of the project:

keytool -genkey -v -keystore ./keys.jks -keyalg RSA -keysize 2048 -validity 10000 -alias <YourKeyAlias>

Follow the instructions show in the commandline to fill out the desired fields of your keystore.

Step 2: create a key.properties file

Inside the ./android (next to the keys.jks file from step 1), create a file key.properties with the following contents:

storePassword=<storePassword>
keyPassword=<keyPassword>
keyAlias=<YourKeyAlias>
storeFile=../keys.jks

Disclaimer & Legal

ReFreezer was not developed for piracy, but educational and private usage. It may be illegal to use this in your country! I will not be responsible for how you use ReFreezer.

ReFreezer uses both Deezer's public and internal API's, but is not endorsed, certified or otherwise approved in any way by Deezer.

The Deezer brand and name is the registered trademark of its respective owner.

ReFreezer has no partnership, sponsorship or endorsement with Deezer.

By using ReFreezer you agree to the following: https://www.deezer.com/legal/cgu

deckerst/aves

2024-12-01T01:52:47Z

Aves is a gallery and metadata explorer app, built for Android with Flutter.

Aves

Aves is a gallery and metadata explorer app. It is built for Android, with Flutter.

Compare versions

Features

Aves can handle all sorts of images and videos, including your typical JPEGs and MP4s, but also more exotic things like multi-page TIFFs, SVGs, old AVIs and more!

It scans your media collection to identify motion photos, panoramas (aka photo spheres), 360° videos, as well as GeoTIFF files.

Navigation and search is an important part of Aves. The goal is for users to easily flow from albums to photos to tags to maps, etc.

Aves integrates with Android (from KitKat to Android 14, including Android TV) with features such as widgets, app shortcuts, screen saver and global search handling. It also works as a media viewer and picker.

Screenshots

Changelog

The list of changes for past and future releases is available here.

Permissions

Aves requires a few permissions to do its job:

read contents of shared storage: the app only accesses media files, and modifying them requires explicit access grants from the user,
read locations from media collection: necessary to display the media coordinates, and to group them by country (via reverse geocoding),
have network access: necessary for the map view, and most likely for precise reverse geocoding too,
view network connections: checking for connection states allows Aves to gracefully degrade features that depend on internet.

Contributing

Issues

Bug reports and feature requests are welcome, but read the guidelines first. If you have questions, check out the discussions.

Code

At this stage this project does not accept PRs.

Translations

Translations are powered by Weblate and the effort of wonderfully generous volunteers.

If you want to translate this app in your language and share the result, there is a guide.

Donations

Some users have expressed the wish to financially support the project. Thanks! ❤️

Project Setup

Before running or building the app, update the dependencies for the desired flavor:

# scripts/apply_flavor_play.sh

To build the project, create a file named <app dir>/android/key.properties. It should contain a reference to a keystore for app signing, and other necessary credentials. See key_template.properties for the expected keys.

To run the app:

# ./flutterw run -t lib/main_play.dart --flavor play

cake-tech/cake_wallet

2024-12-01T01:41:52Z

The open source repository for Cake Wallet, a noncustodial multi-currency wallet, and Monero.com, a noncustodial Monero-only wallet. Need help? Check out https://guides.cakewallet.com

Cake Wallet

Cake Wallet is an open-source, non-custodial, and private multi-currency crypto wallet for Android, iOS, macOS, and Linux.

Cake Wallet includes support for several cryptocurrencies, including:

Monero (XMR)
Bitcoin (BTC)
Ethereum (ETH)
Litecoin (LTC)
Bitcoin Cash (BCH)
Polygon (Pol)
Solana (SOL)
Nano (XNO)
Haven (XHV)

Features

App-Wide Features

Completely noncustodial. Your keys, your coins.
Built-in exchange for dozens of pairs
Easily pay cryptocurrency invoices with fixed rate exchanges
Buy cryptocurrency (BTC/LTC/XMR/ETH) with credit/debit/bank
Sell cryptocurrency by bank transfer
Scan QR codes for easy cryptocurrency transfers
Create several wallets
Select your own custom nodes/servers
Address book
Backup to an external location or iCloud
Send to OpenAlias, Unstoppable Domains, Yats, and FIO Crypto Handles
Set desired network fee level
Store local transaction notes
Extremely simple user experience
Convenient exchange and sending templates for recurring payments
Create donation links and invoices in the receive screen
Robust privacy settings (eg: Tor-only connections)
Robust security settings (eg: Cake 2FA)

Monero Specific Features

The Monero view key is retained on the device for maximum privacy
Full support for Monero subaddresses and accounts
Specify restore height for faster syncing
Specify multiple recipients for batch sending
Optionally set Monero nodes as trusted for faster syncing
Specify a proxy for Monero nodes, compatible with Tor and i2p

Bitcoin Specific Features

Bitcoin coin control (specify specific outputs to spend)
Automatically generate new addresses
Specify multiple recipients for batch sending

Ethereum Specific Features

Store ETH and all ERc-20 tokens
Add custom tokens by contract address
Enable or disable Etherscan for transaction history

Litecoin Specific Features

Litecoin coin control (specify specific outputs to spend)
Automatically generate new addresses
Specify multiple recipients for batch sending

Haven Specific Features

Send, receive, and store XHV and all xAssets like xUSD, xEUR, xAG, etc.

Monero.com by Cake Wallet for Android and iOS

Open Source Monero-Only Wallet

Exchanging to/from other assets is also supported.

Support

We have 24/7 free support. Please contact support@cakewallet.com

We have excellent user guides, which are also open-source and open for contributions: https://guides.cakewallet.com

Build Instructions

More instructions to follow

For instructions on how to build for Android: please view file howto-build-android.md

Contributing

Improving translations

Edit the applicable strings_XX.arb file in res/values/ and open a pull request with the changes.

Current list of language files:

English
Spanish
French
German
Italian
Portuguese
Dutch
Polish
Croatian
Russian
Ukrainian
Hindi
Japanese
Chinese
Korean
Thai
Arabic
Turkish
Burmese
Urdu
Bulgarian
Czech
Indonesian
Hausa
Yoruba

Add a new language

Create a new strings_XX.arb file in res/values/, replacing XX with the language's ISO 639-1 code.
Edit the strings in this file, replacing XXX below with the translation for each string.

"welcome": "Welcome to", -> "welcome": "XXX",

For strings where there is a variable, denoted by a $ symbol and braces, such as ${status}, the string in braces should not be translated. For example, when editing line 106:

"time" : "${minutes}m ${seconds}s"

The only parts to be translated, if needed, are the values m and s after the variables.

Add the language to lib/entities/language_service.dart under both supportedLocales and localeCountryCode. Use the name of the language in the local language and in English in parentheses after for supportedLocales. Use the ISO 3166-1 alpha-3 code for localeCountryCode. You must choose one country, so choose the country with the most native speakers of this language or is otherwise best associated with this language.
Add a relevant flag to assets/images/flags/XXXX.png, replacing XXXX with the 3 letters localeCountryCode. The image must be 42x26 pixels with 3 pixels of transparent margin on all 4 sides. You can resize the flag with paint.net to 36x20 pixels, expand the canvas to 42x26 pixels with the flag anchored in the middle, and then manually delete the 3 pixels on each side to make it transparent. Or you can use another program like Photoshop.
Add the new language code to tool/utils/translation/translation_constants.dart

Add a new fiat currency

Check with Cake Wallet support to see if the desired new fiat currency is available through our fiat API. Not all fiat currencies are.
If the currency is associated strongly with a specific issuing country, map the ISO 4217 currency code with the applicable ISO 3166-1 alpha-3 code in lib/entities/fiat_currency.dart. If the currency is used in a whole region or organization, then map with a reasonable interpretation of this (eg: eur countryCode for EUR symbol).
Add the raw mapping underneath in lib/entities/fiat_currency.dart following the same format as the others.
Add a flag of the issuing country or organization to assets/images/flags/XXXX.png, replacing XXXX with the ISO 3166-1 alpha-3 code used above (eg: usa.png, eur.png). Do not add this if the flag with the same name already exists. The image must be 42x26 pixels with a 3 pixels of transparent margin on all 4 sides.

astral-sh/uv-docker-example

2024-11-30T01:30:33Z

An example of using uv in Docker images

uv-docker-example

An example project for using uv in Docker images, with a focus on best practices for developing with the project mounted in the local image.

See the uv Docker integration guide for more background.

Trying it out

A run.sh utility is provided for quickly building the image and starting a container. This script demonstrates best practices for developing using the container, using bind mounts for the project and virtual environment directories.

To build and run the web application in the container using docker run:

$ ./run.sh

Then, check out http://localhost:8000 to see the website.

A Docker compose configuration is also provided to demonstrate best practices for developing using the container with Docker compose. Docker compose is more complex than using docker run, but has more robust support for various workflows.

To build and run the web application using Docker compose:

docker compose up --watch

By default, the image is set up to start the web application. However, a command-line interface is provided for demonstration purposes as well.

To run the command-line entrypoint in the container:

$ ./run.sh hello

Project overview

Dockerfile

The Dockerfile defines the image and includes:

Installation of uv
Installing the project dependencies and the project separately for optimal image build caching
Placing environment executables on the PATH
Running the web application

The multistage.Dockerfile example extends the Dockerfile example to use multistage builds to reduce the final size of the image.

Dockerignore file

The .dockerignore file includes an entry for the .venv directory to ensure the .venv is not included in image builds. Note that the .dockerignore file is not applied to volume mounts during container runs.

Run script

The run.sh script includes an example of invoking docker run for local development, mounting the source code for the project into the container so that edits are reflected immediately.

Docker compose file

The compose.yml file includes a Docker compose definition for the web application. It includes a watch directive for Docker compose, which is a best-practice method for updating the container on local changes.

Application code

The Python application code for the project is at src/uv_docker_example/__init__.py — there's a command line entrypoint and a basic FastAPI application — both of which just display "hello world" output.

Project definition

The project at pyproject.toml includes Ruff as an example development dependency, includes FastAPI as a dependency, and defines a hello entrypoint for the application.

Useful commands

To check that the environment is up-to-date after image builds:

$ ./run.sh uv sync --frozen
Audited 2 packages ...

To enter a bash shell in the container:

$ ./run.sh /bin/bash

To build the image without running anything:

$ docker build .

To build the multistage image:

$ docker build . --file multistage.Dockerfile

medusajs/docker-medusa

2024-12-01T01:52:59Z

This repo is managed by the Medusa Community. Medusa does not provide official support for Docker, but we will accept fixes and documentation. Use at your own risk.

Medusa

Medusa Admin | Website | Blog | LinkedIn | Twitter | Documentation | Notion

Medusa is an open-source headless commerce engine that enables developers to create amazing digital commerce experiences.

Please note

This repo is managed by the Medusa Community. Medusa does not provide official support for Docker, but we will accept fixes and documentation. Use at your own risk.

This project is inteded for development only at this time.

The files for both the Medusa server and the Storefront are loaded in Bind Mounts allowing you to change the server functionality and have the change be hot-reloaded onto your running containers.

Requirements

To use Docker with Medusa, you should have created a Medusa project. Check out our Quickstart to get started.

Additionally, you should have docker and docker-compose installed on your system.

Getting Started

To set up Medusa in a development environment with Docker, you should copy files docker-compose.yml, docker-compose.override.yml, backend/develop.sh, and backend/Dockerfile` to your Medusa project.

Then build the images since they are not published on dockerhub. This is accomplished by adding the --build flag as shown below:

docker compose up --build

Having already built the Docker images you can run docker compose without the --build flag.

docker compose up

Your local Medusa setup is now running with each of the services occupying the following ports:

Medusa Server: 9000
Medusa Admin: 7000
Storefront: 8000
postgres: 5432
redis: 6379

Note: If you change the dependencies of your projects by adding new packages you can simply rebuild that package with the same tag test and run docker compose up once again to update your environment.

Seeding your Medusa store

To add seed data to your medusa store run this command in a seperate

docker exec medusa-server medusa seed -f ./data/seed.json

Running Medusa with docker in production

This repository and each of the services contain dockerfiles for both development and production, named Dockerfile and Dockerfile.prod respectively. The Dockerfile.prod copies the local files from disk and builds a production ready image based on your local development progress. Your specific needs for a production like container might differ from the Dockerfile.prod but it should provide a template and an idea of the requirements for each of the basic services.

To run the services in a production state docker compose is simply run with the docker-compose.production.yml file as well as the basic docker-compose.yml file as seen below. If you wish to build the production ready images and then start them run docker compose up with the --build flag as described above.

docker compose up -f docker-compose.yml -f docker-compose.production.yml up

docker-compose.production.yml contains production relevant overrides to the services described in the docker-compose.yml development file.

Try it out

curl -X GET localhost:9000/store/products | python -m json.tool

After the seed script has run you will have the following things in you database:

a User with the email: admin@medusa-test.com and password: supersecret
a Region called Default Region with the countries GB, DE, DK, SE, FR, ES, IT
a Shipping Option called Standard Shipping which costs 10 EUR
a Product called Cool Test Product with 4 Product Variants that all cost 19.50 EUR

Visit docs.medusa-commerce.com for further guides.

Website | Notion Home | Twitter | Docs

LondheShubham153/node-todo-cicd

2024-12-01T01:53:14Z

oban-bg/oban

2024-12-01T01:53:15Z

💎 Robust job processing in Elixir, backed by modern PostgreSQL and SQLite3

bluzky/salad_ui

2024-12-01T01:53:15Z

Phoenix Liveview component library inspired by shadcn UI

anoma/anoma

2024-12-01T01:53:15Z

Reference implementation of Anoma

dandavison/xenops

2024-11-30T01:30:51Z

An editing environment for LaTeX mathematical documents

[6-minute video introduction]

1. Overview

Xenops is a LaTeX editing environment for mathematical documents in Emacs.

Please see the 6-minute video introduction.

Installation instructions are below. To use Xenops, open a .tex file in Emacs as usual, and do M-x xenops-mode. The key features are:

LaTeX math, tables, and TikZ diagrams are automatically displayed as SVG images in the Emacs buffer.

You don't need to do anything: rendering is triggered automatically. The result is that you can work on the document in Emacs without needing to frequently check the appearance of the PDF. Rendering is asynchronous, so it doesn't interrupt your writing.
A LaTeX file edited with Xenops is just a normal .tex file.

Other people collaborating on the same document do not have to use Xenops (or Emacs), and using Xenops does not involve adding non-LaTeX content to your .tex files.
Source code in minted blocks can be executed, and is syntax-highlighted.

Mathematica and SymPy are both capable of returning symbolic expressions as LaTeX code. This means that you can perform a symbolic calculation in a Mathematica or SymPy minted block, and Xenops will render the symbolic expression that results as an image of LaTeX-typeset traditional math notation.
Xenops provides an optional decluttered view of your document.

This view can be toggled on/off with M-x xenops-xen-mode. It works by hiding common LaTeX markup, and restyling certain document elements.
Images on disk are displayed in the Emacs buffer, and screenshots can be captured from the system clipboard.

So, for example, \includegraphics{myfile.png} will be displayed as an image, and if you capture a screenshot to the system clipboard, then paste (e.g. C-y) will prompt for a file name, save the image to disk, insert the \includegraphics element, and display the image.

When using Xenops, you can continue to use auctex: Xenops replaces the preview-latex functionality in auctex.

Xenops can also be used with org-mode documents that contain LaTeX fragments.

2. Getting started

Ensure that you have LaTeX installed on your machine.

The commands which latex and which dvisvgm must both return paths to the executables. dvisvgm should be present as part of your LaTeX installation, but it's also available here.
Ensure that your Emacs version is at least Emacs 26.

This is necessary because Xenops uses emacs-aio.

If you are using MacOS, install emacs from homebrew using the emacs-mac package (aka Mitsuharu's Emacs build), since it uses WebKit to render SVG, resulting in beautifully crisp images.
Install Xenops from MELPA.
```
M-x package-refresh-contents RET
M-x package-install RET xenops RET
```
To activate Xenops, use M-x xenops-mode after opening a LaTeX file.

Optional: activate xenops-mode automatically for LaTeX files.

(add-hook 'latex-mode-hook #'xenops-mode)
(add-hook 'LaTeX-mode-hook #'xenops-mode)

Run M-x xenops-doctor

You must run Emacs as a GUI application, not as a terminal application. This is because Xenops displays images in Emacs buffers.

M-x xenops-doctor will check all the requirements listed above and some others. You should see this:
If so, move on to the next section, "How to use Xenops". If not, the xenops-doctor output will explain what is wrong.

3. How to use Xenops

You don't need to learn any commands for Xenops to render your math: you just type, and Xenops will render it. The animation at the top of this README gives an idea of what this looks like (refresh the page to make it play again).

More explicitly, here are the basic steps for working with math in Xenops:

Enter some LaTeX math code, using any of the standard delimiters:
$...$ , $...$, \[...\], \tikz ...;, \begin{equation|align|table|tabular|tikzpicture}.
Move the cursor out of the block: an image of the math will appear.
Move the cursor over the image and hit Enter: the image will disappear and you can edit the LaTeX code.

If you want Xenops to reveal the LaTeX code for editing automatically when the cursor moves into the block, set the variable xenops-reveal-on-entry.

You may also want to create images for all math/table/TikZ elements in the document. To do that, position the cursor outside any element and issue xenops-dwim.

The other command to know from the start is xenops-doctor: run this if something is not working.

The following sections give more detailed instructions and examples.

3.1 Rendering math, tables and TikZ content

This section gives step-by-step instructions and shows what to do if it is not working.

Open a file containing LaTeX math content:

\documentclass{article}
\usepackage{amsmath}
\begin{document}
An example equation is
\begin{equation*}
\frac{\partial \mathcal{L}}{\partial q} - \frac{d}{dt}\frac{\partial \mathcal{L}}{\partial \dot{q}} = 0.
\end{equation*}
The document continues.
\end{document}

Xenops recognizes the usual delimiters: $...$ , $...$, \[...\], \tikz ...;, \begin{equation|align|table|tabular|tikzpicture}.

Move the cursor into a math element:
Move the cursor out of that element. The content in the block will shortly be replaced by an image:
To remove the image and continue editing, click on it or move the cursor over the image and hit Enter.
What if something went wrong?

There are a few things that could go wrong. For example, Emacs might not be able to find the latex executable on your machine, or latex might fail due to errors in your code.

If something went wrong, a yellow warning triangle will appear at the beginning of the math block:

To investigate what the problem is, you can hover the mouse over the triangle, or right-click on the triangle to bring up a menu. Here's what you might see if you hover the mouse over the warning triangle:

So that's fairly clear: either latex is not installed, or Emacs is not finding the executable. Xenops has another troubleshooting tool which it would make sense to use at this point: M-x xenops-doctor. Here's what is displayed when we run that:

Alternatively, right-click on the warning triangle brings up a menu:
Select "View failing command output" to see the errors from the latex process, or select "Copy failing command" in order to debug the problem from the command-line.

3.2 Rendering all the content in the document

Place the cursor anywhere in the document, but not in a math/table/TikZ element, and issue M-x xenops-dwim.

This will kick off one asynchronous processing task for every renderable element in the buffer whose image is not already cached. You can carry on editing the document while these are being processed. They will gradually complete, and the images will be inserted in the buffer.

Alternatively, you can select a region, and xenops-dwim will act on just the elements in that region.

3.3 Executing code

Emacs provides sophisticated facilities for executing blocks of code written in many different languages: see org-babel.

Xenops makes use of this to allow minted blocks in LaTeX buffers to be executed. Place point in a minted code block and issue M-x xenops-dwim:

(Use (require 'ob-python) to enable python execution.)

There are many possibilities with documents that are a hybrid of code, code results, and traditional content: please see the org-babel documentation.

With Xenops, we can use this to check calculations done by hand against the output of a symbolic algebra package.

Both SymPy and Mathematica can return their results to the Emacs buffer as LaTeX code, and Xenops will render this immediately as an image. The result is that it feels as if Sympy/Mathematica are returning their results as an image, typeset in traditional mathematical notation, which can be helpful for a quick check of manually-obtained results:

3.3.1 SymPy example

\begin{minted}{python3} :sympy t :results latex
from sympy import integrate, symbols
t, y, tau = symbols('t y tau')

def picard(f, y_prev, a, b):
    return b + integrate(f.subs([(t, tau), (y, y_prev)]), (tau, a, t))

a, b = 0, 1

picard_iterates = []
y = b
for i in [1, 2, 3]:
    f = (1 - 2*t) * y
    y = picard(f, y, a, b)
    picard_iterates.append(y)

return picard_iterates
\end{minted}

After xenops-dwim on the code block, the Xenops buffer looks like this:

3.3.2 Mathematica example

The derivative is
\begin{align*}
  \frac{\dif}{\dif x} \frac{y'}{\sqrt{1 + y'^2}}
  &= \frac{y''\sqrt{1 + y'^2} - y'\frac{1}{2}\frac{1}{\sqrt{1 + y'^2}}2y'y''}{1 + y'^2} \\
  &= \frac{y''}{\sqrt{1 + y'^2}} - \frac{y'^2y''}{(1 + y'^2)^{3/2}}.
\end{align*}

(Check this in Mathematica):
\begin{minted}{wolfram} :results latex
D[y'[x]/Sqrt[1 + y'[x]^2], x]
\end{minted}

After xenops-dwim on the math and code blocks, the Xenops buffer looks like this:

Note that for SymPy, the header arguments :sympy t :results latex are necessary to tell Xenops that this is not a normal python code block and that it should insert SymPy code to automatically format the result as LaTeX. For Mathematica, :results latex suffices for this.

3.4 Working with images

The size of images displayed in the buffer can be changed with xenops-increase-size and xenops-decrease-size.

Xenops recognizes the \includegraphics command, and these images will be displayed by xenops-render. If you capture a screenshot to your system clipboard and then paste (e.g. C-y) into the LaTeX buffer, Xenops will notice that you're pasting image data, prompt for a file to save it in, and insert the \includegraphics link (implemented in MacOS only currently, see issue #1):

Some facts:
\begin{mdframed}
  \includegraphics[width=400pt]{xenops.png}
\end{mdframed}
The document continues.

3.5 Xen mode

The command xenops-xen-mode toggles an alternative cleaner view that hides common LaTeX markup and applies some visual styling. These are changes in appearance only -- the actual text content of the document is never changed and hitting save after turning on Xen mode will not cause any new changes to be written to disk.

In the example below, Xen-mode has made the following changes to the visual appearance of the document:

\begin{align}, \end{align}, \begin{minted}, \end{minted} have been replaced with a unicode character (⚡).
\section markup has been hidden and the section heading has been styled and indented according to its level.
Some LaTeX mathematical syntax has been replaced by unicode characters.

	Math block in edit mode	Math block rendered
Default view
Xen mode

4. Command and variable reference

For commands that operate on one or more elements the rule is:

If there is an element at the current cursor position, operate on that element.
If there is an active selection, operate on elements in the selected region.
Otherwise, operate on all elements in the buffer.

Command		Default keybinding
`xenops-dwim`	Render, Reveal, Regenerate, or Execute element(s). (do-what-i-mean)	`C-c !`

`xenops-render`	Render element(s) as an image.

`xenops-reveal`	Remove image and reveal element(s) for editing.	`RET` on an element, click on an element

`xenops-regenerate`	Regenerate image for element(s), refusing to use a cached version.

`xenops-execute`	Execute code block.

`xenops-doctor`	Check your Emacs environment for any problems that will prevent Xenops from working correctly.

`xenops-goto-element`	Use avy to jump to an element.	`C-c , g`

`xenops-copy-and-paste-element`	Use avy to copy an element from elsewhere and paste it at point.	`C-c , c`

`<copy>`	If the cursor is on a rendered element, copy the entire element.	`C-w` etc

`<paste>`	If the clipboard contains an image, write the image to disk and insert an `\includegraphics` link.	`C-y` etc

`xenops-show-waiting-tasks`	Display a count of latex background processing tasks that are waiting in the queue

`xenops-cancel-waiting-tasks`	Cancel all latex background processing tasks that are waiting in the queue

`xenops-increase-size`	Increase size of image(s).

`xenops-decrease-size`	Decrease size of image(s).

`xenops-xen-mode`	Toggle Xen mode on/off.	`C-c /`

`xenops-select-font-family`	Select a font for the Xenops buffer.

`xenops-clear-latex-preamble-cache`	Force Xenops to recreate the latex preamble for the current document. This is necessary if you have added e.g. a `\usepackage` line.

`xenops-math-concatenate`	Concatenate contiguous math blocks in the active region.

In addition, the keybinding C-c , is bound to a keymap behind which all Xenops commands are available.

Variable		Default value
`xenops-cache-directory`	The location of the directory in which Xenops saves SVG images.	~/.emacs.d/xenops/cache

`xenops-font-family`	The font family to use in Xenops buffers. (You can choose this interactively using `M-x xenops-select-font-family`.)	nil

`xenops-font-height`	The font height for the main font.	160

`xenops-font-height-code`	The font height for the font used for LaTeX code, and code in code blocks.	140

`xenops-math-image-scale-factor`	Scaling factor for SVG math images. Determines image size in the image file that is cached on disk.	1.0

`xenops-math-latex-max-tasks-in-flight`	The maximum number of latex processing tasks that are permitted to be simultaneously active.	32

`xenops-reveal-on-entry`	Should a math block be revealed automatically when point moves inside it?	nil

`xenops-tooltip-delay`	The time delay in seconds before displaying a tooltip on mouseover.	0.2

When generating image previews, there are a few things one may want to configure. For example, the image type (SVG is the default, but PNG is an alternative), and details of the LaTeX document that is constructed when creating each image preview. Xenops shares these configuration mechanisms with org-mode, which also offers image preview functionality (and supports SVG). Here are the relevant org-mode variables:

Variable		Default value
`org-format-latex-header`	The document header used when processing LaTeX fragments. You will need to change this if you need to use a different `documentclass`.	(a long string)

5. Tips

Xenops cannot currently parse $...$ when it is broken across lines. Please try to keep them on one line for now! fill-paragraph (M-q) helps with this.
The command xenops-copy-and-paste-element is very useful: when writing a mathematical document one very often wants to write an expression that is similar to something that already exists elsewhere in the document. The default binding is C-c , c but I recommend binding it to something more convenient: personally I am using s-[.
Use fill-paragraph (M-q) while writing. In xenops-mode, in addition to its normal behavior, it ensures that all content in the paragraph is rendered.
Xenops caches the SVG images. If it notices that it already has the image for some LaTeX math/table/TikZ content, then it will not re-run latex. The cache location on disk is determined by the variable xenops-cache-directory.
Xenops understands the auctex variable TeX-master. This means that you can work on multi-file LaTeX projects with Xenops: as long as TeX-master is set correctly, then Xenops will include the necessary LaTeX packages and preamble when it is generating images for math/table/TikZ fragments. Please see the auctex documentation.
If you are encountering any problems, the first thing to try is M-x xenops-doctor. Beyond that, please don't hesitate to open Github issues!

6. Contributing

Xenops can already be used profitably for serious work. But there are also many areas for further work. Please don't hesitate to get in touch (dandavison7@gmail.com), submit a PR, or open a Github issue.

7. Credit

_{Streaked Xenops (Xenops rutilans) image by Dubi Shapiro.}

purcell/emacs-reformatter

2024-12-01T01:53:19Z

Define commands which run reformatters on the current Emacs buffer

This library lets elisp authors easily define an idiomatic command to reformat the current buffer using a command-line program, together with an optional minor mode which can apply this command automatically on save.

By default, reformatter.el expects programs to read from stdin and write to stdout, and you should prefer this mode of operation where possible. If this isn't possible with your particular formatting program, refer to the options for reformatter-define, and see the examples in the package's tests.

In its initial release it supports only reformatters which can read from stdin and write to stdout, but a more versatile interface will be provided as development continues.

As an example, let's define a reformat command that applies the "dhall format" command. We'll assume here that we've already defined a variable dhall-command which holds the string name or path of the dhall executable:

(reformatter-define dhall-format
  :program dhall-command
  :args '("format")
  :lighter " DF")

The reformatter-define macro expands to code which generates dhall-format-buffer and dhall-format-region interactive commands, and a local minor mode called dhall-format-on-save-mode. The :args and :program expressions will be evaluated at runtime, so they can refer to variables that may (later) have a buffer-local value. A custom variable will be generated for the mode lighter, with the supplied value becoming the default.

The generated minor mode allows idiomatic per-directory or per-file customisation, via the "modes" support baked into Emacs' file-local and directory-local variables mechanisms. For example, users of the above example might add the following to a project-specific .dir-locals.el file:

((dhall-mode
   (mode . dhall-format-on-save)))

See the documentation for reformatter-define, which provides a number of options for customising the generated code.

Library authors might like to provide autoloads for the generated code, e.g.:

;;;###autoload (autoload 'dhall-format-buffer "current-file" nil t)
;;;###autoload (autoload 'dhall-format-region "current-file" nil t)
;;;###autoload (autoload 'dhall-format-on-save-mode "current-file" nil t)

Examples of usage in the wild

To find reverse dependencies, look for "Needed by" on the MELPA page for reformatter. Here are some specific examples:

dhall-mode.el
elm-format.el, in elm-mode
sqlformat.el
Here is the author defining a reformatter in his own configuration

Rationale

I contribute to a number of Emacs programming language modes and tools, and increasingly use code reformatters in my daily work. It's surprisingly difficult to write robust, correct code to apply these reformatters, given that it must consider such issues as:

Missing programs
Buffers not yet saved to a file
Displaying error output
Colorising ANSI escape sequences in any error output
Handling file encodings correctly

With this library, I hope to help the community standardise on best practices, and make things easier for tool authors and end users alike.

FAQ

How is this different from format-all.el?

format-all is a very different approach: it aims to provide a single minor mode which you then enable and configure to do the right thing (including nothing) for all the languages you use. It even tries to tell you how to install missing programs. It's an interesting project, but IMO it's hard to design the configuration for such a grand unified approach, and it can get complex. For example, you'd have to be able to configure which of two possible reformatters you want to use for a specific language, and to be able to do that on a per-project basis.

In contrast reformatter produces small, self-contained and separate formatters and minor modes which all work consistently and are individually configured. It makes it possible to replace existing formatter code, and it's also very convenient for users to define their own ad-hoc reformatter wrappers

Installation

Manual

Ensure reformatter.el is in a directory on your load-path, and add the following to your ~/.emacs or ~/.emacs.d/init.el:

(require 'reformatter)

MELPA

If you're an Emacs 24 user or you have a recent version of package.el you can install reformatter from the MELPA repository. The version of reformatter there will always be up-to-date.

About

Author: Steve Purcell

Homepage: https://github.com/purcell/emacs-reformatter

💝 Support this project and my other Open Source work

💼 LinkedIn profile

✍ sanityinc.com

clojure-emacs/clojure-mode

2024-12-01T01:53:19Z

Emacs support for the Clojure(Script) programming language

Clojure Mode

clojure-mode is an Emacs major mode that provides font-lock (syntax highlighting), indentation, navigation and refactoring support for the Clojure(Script) programming language.

This documentation tracks the master branch of clojure-mode. Some of the features and settings discussed here might not be available in older releases (including the current stable release). Please, consult the relevant git tag (e.g. 5.19.0) if you need documentation for a specific clojure-mode release.

Installation

Available on the major package.el community maintained repos - MELPA Stable and MELPA repos.

MELPA Stable is the recommended repo as it has the latest stable version. MELPA has a development snapshot for users who don't mind (infrequent) breakage but don't want to run from a git checkout.

You can install clojure-mode using the following command:

M-x package-install [RET] clojure-mode [RET]

or if you'd rather keep it in your dotfiles:

(unless (package-installed-p 'clojure-mode)
  (package-install 'clojure-mode))

If the installation doesn't work try refreshing the package list:

M-x package-refresh-contents

Bundled major modes

The clojure-mode package actually bundles together several major modes:

clojure-mode is a major mode for editing Clojure code
clojurescript-mode is a major mode for editing ClojureScript code
clojurec-mode is a major mode for editing .cljc source files

All the major modes derive from clojure-mode and provide more or less the same functionality. Differences can be found mostly in the font-locking - e.g. ClojureScript has some built-in constructs that are not present in Clojure.

The proper major mode is selected automatically based on the extension of the file you're editing.

Having separate major modes gives you the flexibility to attach different hooks to them and to alter their behavior individually (e.g. add extra font-locking just to clojurescript-mode) .

Note that all modes derive from clojure-mode, so things you add to clojure-mode-hook and clojure-mode-map will affect all the derived modes as well.

Configuration

In the spirit of Emacs, pretty much everything you can think of in clojure-mode is configurable.

To see a list of available configuration options do M-x customize-group RET clojure.

Indentation options

The default indentation rules in clojure-mode are derived from the community Clojure Style Guide. Please, refer to the guide for the general Clojure indentation rules.

If you'd like to use the alternative "fixed/tonsky" indentation style you should update your configuration accordingly:

(setq clojure-indent-style 'always-indent
      clojure-indent-keyword-style 'always-indent
      clojure-enable-indent-specs nil)

Read on for more details on the available indentation-related configuration options.

Indentation of docstrings

By default multi-line docstrings are indented with 2 spaces, as this is a somewhat common standard in the Clojure community. You can however adjust this by modifying clojure-docstring-fill-prefix-width. Set it to 0 if you don't want multi-line docstrings to be indented at all (which is pretty common in most lisps).

Indentation of function forms

The indentation of function forms is configured by the variable clojure-indent-style. It takes three possible values:

always-align (the default)

(some-function
 10
 1
 2)
(some-function 10
               1
               2)

always-indent

(some-function
  10
  1
  2)
(some-function 10
  1
  2)

align-arguments

(some-function
  10
  1
  2)
(some-function 10
               1
               2)

Note: Prior to clojure-mode 5.10, the configuration options for clojure-indent-style used to be keywords, but now they are symbols. Keywords will still be supported at least until clojure-mode 6.

Indentation of keywords

Similarly we have the clojure-indent-keyword-style, which works in the following way:

always-align (default) - All args are vertically aligned with the first arg in case (A), and vertically aligned with the function name in case (B).

(:require [foo.bar]
          [bar.baz])
(:require
 [foo.bar]
 [bar.baz])

always-indent - All args are indented like a macro body.

(:require [foo.bar]
   [bar.baz])
(:x
   location
   0)

align-arguments - Case (A) is indented like always-align, and case (B) is indented like a macro body.

(:require [foo.bar]
          [bar.baz])
(:x
   location
   0)

Indentation of macro forms

The indentation of special forms and macros with bodies is controlled via put-clojure-indent, define-clojure-indent and clojure-backtracking-indent. Nearly all special forms and built-in macros with bodies have special indentation settings in clojure-mode. You can add/alter the indentation settings in your personal config. Let's assume you want to indent ->> and -> like this:

(->> something
  ala
  bala
  portokala)

You can do so by putting the following in your config:

(put-clojure-indent '-> 1)
(put-clojure-indent '->> 1)

This means that the body of the ->/->> is after the first argument.

A more compact way to do the same thing is:

(define-clojure-indent
  (-> 1)
  (->> 1))

To indent something like a definition (defn) you can do something like:

(put-clojure-indent '>defn :defn)

You can also specify different indentation settings for symbols prefixed with some ns (or ns alias):

(put-clojure-indent 'do 0)
(put-clojure-indent 'my-ns/do 1)

The bodies of certain more complicated macros and special forms (e.g. letfn, deftype, extend-protocol, etc) are indented using a contextual backtracking indentation method, require more sophisticated indent specifications. Here are a few examples:

(define-clojure-indent
  (implement '(1 (1)))
  (letfn     '(1 ((:defn)) nil))
  (proxy     '(2 nil nil (1)))
  (reify     '(:defn (1)))
  (deftype   '(2 nil nil (1)))
  (defrecord '(2 nil nil (1)))
  (specify   '(1 (1)))
  (specify   '(1 (1))))

These follow the same rules as the :style/indent metadata specified by cider-nrepl. For instructions on how to write these specifications, see this document. The only difference is that you're allowed to use lists instead of vectors.

The indentation of special arguments is controlled by clojure-special-arg-indent-factor, which by default indents special arguments a further lisp-body-indent when compared to ordinary arguments.

An example of the default formatting is:

(defrecord MyRecord
    [my-field])

Note that defrecord has two special arguments, followed by the form's body - namely the record's name and its fields vector.

Setting clojure-special-arg-indent-factor to 1, results in:

(defrecord MyRecord
  [my-field])

You can completely disable the effect of indentation specs like this:

(setq clojure-enable-indent-specs nil)

Indentation of Comments

clojure-mode differentiates between comments like ;, ;;, etc. By default clojure-mode treats ; as inline comments and always indents those. You can change this behaviour like this:

(add-hook 'clojure-mode-hook (lambda () (setq-local comment-column 0)))

You might also want to change comment-add to 0 in that way, so that Emacs comment functions (e.g. comment-region) would use ; by default instead of ;;.

Note: Check out this section of the Clojure style guide to understand better the semantics of the different comment levels and why clojure-mode treats them differently by default.

Vertical alignment

You can vertically align sexps with C-c SPC. For instance, typing this combo on the following form:

(def my-map
  {:a-key 1
   :other-key 2})

Leads to the following:

(def my-map
  {:a-key     1
   :other-key 2})

This can also be done automatically (as part of indentation) by turning on clojure-align-forms-automatically. This way it will happen whenever you select some code and hit TAB.

Font-locking

clojure-mode features static font-locking (syntax highlighting) that you can extend yourself if needed. As typical for Emacs, it's based on regular expressions. You can find the default font-locking rules in clojure-font-lock-keywords. Here's how you can add font-locking for built-in Clojure functions and vars:

(defvar clojure-built-in-vars
  '(;; clojure.core
    "accessor" "aclone"
    "agent" "agent-errors" "aget" "alength" "alias"
    "all-ns" "alter" "alter-meta!" "alter-var-root" "amap"
    ;; omitted for brevity
    ))

(defvar clojure-built-in-dynamic-vars
  '(;; clojure.test
    "*initial-report-counters*" "*load-tests*" "*report-counters*"
    "*stack-trace-depth*" "*test-out*" "*testing-contexts*" "*testing-vars*"
    ;; clojure.xml
    "*current*" "*sb*" "*stack*" "*state*"
    ))

(font-lock-add-keywords 'clojure-mode
                        `((,(concat "(\\(?:\.*/\\)?"
                                    (regexp-opt clojure-built-in-vars t)
                                    "\\>")
                           1 font-lock-builtin-face)))

(font-lock-add-keywords 'clojure-mode
                        `((,(concat "\\<"
                                    (regexp-opt clojure-built-in-dynamic-vars t)
                                    "\\>")
                           0 font-lock-builtin-face)))

Note: The package clojure-mode-extra-font-locking provides such additional font-locking for Clojure built-ins.

As you might imagine one problem with this font-locking approach is that because it's based on regular expressions you'll get some false positives here and there (there's no namespace information, and no way for clojure-mode to know what var a symbol resolves to). That's why clojure-mode's font-locking defaults are conservative and minimalistic.

Precise font-locking requires additional data that can obtained from a running REPL (that's how CIDER's dynamic font-locking works) or from static code analysis.

When it comes to non built-in definitions, clojure-mode needs to be manually instructed how to handle the docstrings and highlighting. Here's an example:

(put '>defn 'clojure-doc-string-elt 2)

(font-lock-add-keywords 'clojure-mode
                        `((,(concat "(\\(?:" clojure--sym-regexp "/\\)?"
                                    "\\(>defn\\)\\>")
                           1 font-lock-keyword-face)))

Note: The clojure-doc-string-elt attribute is processed by the function clojure-font-lock-syntactic-face-function.

Refactoring support

The available refactorings were originally created and maintained by the clj-refactor.el team. The ones implemented in Elisp only are gradually migrated to clojure-mode.

Threading macros related features

clojure-thread: Thread another form into the surrounding thread. Both ->> and -> variants are supported.

clojure-unwind: Unwind a threaded expression. Supports both ->> and ->.

clojure-thread-first-all: Introduce the thread first macro (->) and rewrite the entire form. With a prefix argument do not thread the last form.

clojure-thread-last-all: Introduce the thread last macro and rewrite the entire form. With a prefix argument do not thread the last form.

clojure-unwind-all: Fully unwind a threaded expression removing the threading macro.

Cycling things

clojure-cycle-privacy: Cycle privacy of defs or defns. Use metadata explicitly with setting clojure-use-metadata-for-privacy to t for defns too.

clojure-cycle-not: Add or remove a not form around the current form.

clojure-cycle-when: Find the closest when or when-not up the syntax tree and toggle it.

clojure-cycle-if: Find the closest if or if-not up the syntax tree and toggle it. Also transpose the else and then branches, keeping the semantics the same as before.

Convert collection

Convert any given collection at point to list, quoted list, map, vector or set.

Let expression

clojure-introduce-let: Introduce a new let form. Put the current form into its binding form with a name provided by the user as a bound name. If called with a numeric prefix put the let form Nth level up in the form hierarchy.

clojure-move-to-let: Move the current form to the closest let's binding form. Replace all occurrences of the form in the body of the let.

clojure-let-forward-slurp-sexp: Slurp the next form after the let into the let. Replace all occurrences of the bound forms in the form added to the let form. If called with a prefix argument slurp the next n forms.

clojure-let-backward-slurp-sexp: Slurp the form before the let into the let. Replace all occurrences of the bound forms in the form added to the let form. If called with a prefix argument slurp the previous n forms.

paredit-convolute-sexp is advised to replace occurrences of bound forms with their bound names when convolute is used on a let form.

Rename ns alias

clojure-rename-ns-alias: Rename an alias inside a namespace declaration, and all of its usages in the buffer

If there is an active selected region, only rename usages of aliases within the region, without affecting the namespace declaration.

Add arity to a function

clojure-add-arity: Add a new arity to an existing single-arity or multi-arity function.

Related packages

clojure-mode-extra-font-locking provides additional font-locking for built-in methods and macros. The font-locking is pretty imprecise, because it doesn't take namespaces into account and it won't font-lock a function at all possible positions in a sexp, but if you don't mind its imperfections you can easily enable it:

(require 'clojure-mode-extra-font-locking)

The code in clojure-mode-font-locking used to be bundled with clojure-mode before version 3.0.

You can also use the code in this package as a basis for extending the font-locking further (e.g. functions/macros from more namespaces). Generally you should avoid adding special font-locking for things that don't have fairly unique names, as this will result in plenty of incorrect font-locking. CIDER users should avoid this package, as CIDER does its own dynamic font-locking, which is namespace-aware and doesn't produce almost any false positives.

clj-refactor provides refactoring support.
Enabling CamelCase support for editing commands(like forward-word, backward-word, etc) in clojure-mode is quite useful since we often have to deal with Java class and method names. The built-in Emacs minor mode subword-mode provides such functionality:

(add-hook 'clojure-mode-hook #'subword-mode)

The use of paredit when editing Clojure (or any other Lisp) code is highly recommended. It helps ensure the structure of your forms is not compromised and offers a number of operations that work on code structure at a higher level than just characters and words. To enable it for Clojure buffers:

(add-hook 'clojure-mode-hook #'paredit-mode)

smartparens is an excellent (newer) alternative to paredit. Many Clojure hackers have adopted it recently and you might want to give it a try as well. To enable smartparens use the following code:

(add-hook 'clojure-mode-hook #'smartparens-strict-mode)

RainbowDelimiters is a minor mode which highlights parentheses, brackets, and braces according to their depth. Each successive level is highlighted in a different color. This makes it easy to spot matching delimiters, orient yourself in the code, and tell which statements are at a given depth. Assuming you've already installed RainbowDelimiters you can enable it like this:

(add-hook 'clojure-mode-hook #'rainbow-delimiters-mode)

aggressive-indent-mode automatically adjust the indentation of your code, while you're writing it. Using it together with clojure-mode is highly recommended. Provided you've already installed aggressive-indent-mode you can enable it like this:

(add-hook 'clojure-mode-hook #'aggressive-indent-mode)

REPL Interaction

One of the fundamental aspects of Lisps in general, and Clojure in particular, is the notion of interactive programming - building your programs by continuously changing the state of the running Lisp program (as opposed to doing something more traditional like making a change and re-running the program afterwards to see the changes in action). To get the most of clojure-mode you'll have to combine it with some tool which will allow you to interact with your Clojure program (a.k.a. process/REPL).

A number of options exist for connecting to a running Clojure process and evaluating code interactively.

Basic REPL

inf-clojure provides basic interaction with a Clojure REPL process. It's very similar in nature and supported functionality to inferior-lisp-mode for Common Lisp.

CIDER

CIDER is a powerful Clojure interactive development environment, similar to SLIME for Common Lisp.

If you're into Clojure and Emacs you should definitely check it out.

Tutorials

Tutorials, targeting Emacs beginners, are available at clojure-doc.org and Clojure for the Brave and the True. Keep in mind, however, that they might be out-of-date.

Caveats

clojure-mode is a capable tool, but it's certainly not perfect. This section lists a couple of general design problems/limitations that might affect your experience negatively.

General Issues

clojure-mode derives a lot of functionality directly from lisp-mode (an Emacs major mode for Common Lisp), which simplified the initial implementation, but also made it harder to implement certain functionality. Down the road it'd be nice to fully decouple clojure-mode from lisp-mode.

See this ticket for a bit more details.

Indentation Performance

clojure-mode's indentation engine is a bit slow. You can speed things up significantly by disabling clojure-use-backtracking-indent, but this will break the indentation of complex forms like deftype, defprotocol, reify, letfn, etc.

We should look into ways to optimize the performance of the backtracking indentation logic. See this ticket for more details.

Font-locking Implementation

As mentioned above, the font-locking is implemented in terms of regular expressions which makes it both slow and inaccurate.

Changelog

An extensive changelog is available here.

License

Distributed under the GNU General Public License; type C-h C-c to view it.

tumashu/posframe

2024-12-01T01:53:19Z

Pop a posframe (just a child-frame) at point, posframe is a **GNU ELPA** package!

Created 2021-06-01 Tue 10:41

#+TITLE: Pop a posframe (just a frame) at point #+AUTHOR: Feng Shu

#+html: #+html: #+html:

What is posframe? Posframe can pop up a frame at point, this posframe is a child-frame connected to its root window's buffer.

The main advantages are:

It is fast enough for daily usage :-)
It works well with CJK languages.

NOTE:

For MacOS users, posframe needs Emacs version >= 26.0.91
GNOME users with GTK3 builds need Emacs 27 or later. See variable `posframe-gtk-resize-child-frames' which auto-detects this configuration.

More details:
1. [[https://git.savannah.gnu.org/cgit/emacs.git/commit/?h=emacs-27&id=c49d379f17bcb0ce82604def2eaa04bda00bd5ec][Fix some problems with moving and resizing child frames]]
2. [[https://lists.gnu.org/archive/html/emacs-devel/2020-01/msg00343.html][Emacs's set-frame-size can not work well with gnome-shell?]]

[[file:./snapshots/posframe-1.png]]

Installation

#+begin_example (require 'posframe) #+end_example

Usage

** Create a posframe

*** Simple way #+begin_example (when (posframe-workable-p) (posframe-show " my-posframe-buffer" :string "This is a test" :position (point))) #+end_example

*** Advanced way #+begin_example (defvar my-posframe-buffer " my-posframe-buffer")

(with-current-buffer (get-buffer-create my-posframe-buffer) (erase-buffer) (insert "Hello world"))

(when (posframe-workable-p) (posframe-show my-posframe-buffer :position (point))) #+end_example

*** Arguments

#+begin_example C-h f posframe-show #+end_example

** Hide a posframe #+begin_example (posframe-hide " my-posframe-buffer") #+end_example

** Hide all posframes #+begin_example M-x posframe-hide-all #+end_example

** Delete a posframe

Delete posframe and its buffer #+begin_example (posframe-delete " my-posframe-buffer") #+end_example
Only delete the frame #+begin_example (posframe-delete-frame " my-posframe-buffer") #+end_example ** Delete all posframes #+begin_example M-x posframe-delete-all #+end_example

Note: this command will delete all posframe buffers. You probably shouldn't use it if you are sharing a buffer between posframe and other packages.

** posframe-arghandler

posframe-arghandler feature has been removed from posframe-1.1, user can use advice feature instead.

** Mouse banish Default setting will work well in most case, but for EXWM user, suggest use the below config.

#+begin_src emacs-lisp (setq posframe-mouse-banish-function #'posframe-mouse-banish-simple) #+end_src

millejoh/emacs-ipython-notebook

2024-12-01T01:42:23Z

Jupyter notebook client in Emacs

========================================================== EIN -- Emacs IPython Notebook |build-status| |melpa-dev|

.. image:: https://github.com/dickmao/emacs-ipython-notebook/blob/master/thumbnail.png :target: https://youtu.be/8VzWc9QeOxE :alt: Kaggle Notebooks in AWS

Emacs IPython Notebook (EIN), despite its name, is a jupyter client for all languages. It does not work under non-WSL Windows environments.

As of 2023, EIN has been sunset for a number of years having been unable to keep up with jupyter's web-first ecosystem. Even during its heyday EIN never fully reconciled emac's monolithic buffer architecture to the notebook's by-cell discretization, leaving gaping functional holes like crippled undo.

Certainly in 2012 when jupyter was much smaller, an emacs client made perfect sense. With many years of hindsight, it's now clear the json-driven, git-averse notebook format is anathema to emacs's plain text ethos.

.. |build-status| image:: https://github.com/millejoh/emacs-ipython-notebook/workflows/CI/badge.svg :target: https://github.com/millejoh/emacs-ipython-notebook/actions :alt: Build Status .. |melpa-dev| image:: https://melpa.org/packages/ein-badge.svg :target: http://melpa.org/#/ein :alt: MELPA current version .. _Jupyter: http://jupyter.org .. _Babel: https://orgmode.org/worg/org-contrib/babel/intro.html .. _Org: https://orgmode.org .. _[tkf]: http://tkf.github.io .. _[gregsexton]: https://github.com/gregsexton/ob-ipython

Install

As described in Getting started_, ensure melpa's whereabouts in init.el or .emacs::

(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/"))

Then

M-x package-install RET ein RET

Alternatively, directly clone this repo and make install.

For jupyterlab 3.0+, reconfigure the subcommand from "notebook" to "server".

M-x customize-option RET ein:jupyter-server-use-subcommand RET

Usage

Start EIN using ONE of the following:

Open an .ipynb file, press C-c C-o, or,
M-x ein:run launches a jupyter process from emacs, or,
M-x ein:login to a running jupyter server, or,

M-x ein:stop prompts to halt local and remote jupyter services.

Alternatively, ob-ein_.

.. _Cask: https://cask.readthedocs.io/en/latest/guide/installation.html .. _Getting started: http://melpa.org/#/getting-started

FAQ

How do I...

... report a bug? Note EIN is tested only for released GNU Emacs versions 26.3 and later. Pre-release versions will not work.

First try emacs -Q -f package-initialize -f ein:dev-start-debug and reproduce the bug. The -Q skips any user configuration that might interfere with EIN.

Then file an issue using M-x ein:dev-bug-report-template.

... display images inline? We find inserting images into emacs disruptive, and so default to spawning an external viewer. To override this, ::

  M-x customize-group RET ein
  Ein:Output Area Inlined Images

... configure the external image viewer? ::

  M-x customize-group RET mailcap
  Mailcap User Mime Data

On a typical Linux system, one might configure a viewer for MIME Type image/png as a shell command convert %s -background white -alpha remove -alpha off - | display -immutable.

... get IDE-like behavior? You can't. EIN's architecture is fundamentally incompatible with LSP.

.. _Issues: https://github.com/millejoh/emacs-ipython-notebook/issues .. _prevailing documentation: http://millejoh.github.io/emacs-ipython-notebook .. _spacemacs layer: https://github.com/syl20bnr/spacemacs/tree/master/layers/%2Blang/ipython-notebook .. _company-mode: https://github.com/company-mode/company-mode .. _jupyterhub: https://github.com/jupyterhub/jupyterhub .. _elpy: https://melpa.org/#/elpy .. _math-preview: https://gitlab.com/matsievskiysv/math-preview .. _program modes: https://www.gnu.org/software/emacs/manual/html_node/emacs/Program-Modes.html .. _undo boundaries: https://www.gnu.org/software/emacs/manual/html_node/elisp/Undo.html

ob-ein

Configuration:

M-x customize-group RET org-babel Org Babel Load Languages: Insert (ein . t) For example, '((emacs-lisp . t) (ein . t))

Snippet:

#+BEGIN_SRC ein-python :session localhost import numpy, math, matplotlib.pyplot as plt %matplotlib inline x = numpy.linspace(0, 2*math.pi) plt.plot(x, numpy.sin(x)) #+END_SRC

The :session is the notebook url, e.g., http://localhost:8888/my.ipynb, or simply localhost, in which case org evaluates anonymously. A port may also be specified, e.g., localhost:8889.

Language can be ein-python, ein-r, or ein-julia. The relevant jupyter kernel_ must be installed before use. Additional languages can be configured via::

M-x customize-group RET ein Ob Ein Languages

.. _polymode: https://github.com/polymode/polymode .. _ob-ipython: https://github.com/gregsexton/ob-ipython .. _scimax: https://github.com/jkitchin/scimax .. _jupyter kernel: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

.. _gat utility: https://dickmaogat.readthedocs.io/en/latest/install.html .. _gat usage: https://dickmaogat.readthedocs.io/en/latest/usage.html .. _batch mode: https://nbconvert.readthedocs.io/en/latest/execute_api.html .. _dickmao/Kaggler: https://github.com/dickmao/Kaggler/tree/gcspath#importing-datasets

Keymap (C-h m)

Key Binding

C- ein:worksheet-goto-next-input-km C- ein:worksheet-goto-prev-input-km M-S- ein:worksheet-execute-cell-and-insert-below-km M- ein:worksheet-not-move-cell-down-km M- ein:worksheet-not-move-cell-up-km

C-x C-s ein:notebook-save-notebook-command-km C-x C-w ein:notebook-rename-command-km

M-RET ein:worksheet-execute-cell-and-goto-next-km M-, ein:pytools-jump-back-command M-. ein:pytools-jump-to-source-command

C-c C-a ein:worksheet-insert-cell-above-km C-c C-b ein:worksheet-insert-cell-below-km C-c C-c ein:worksheet-execute-cell-km C-u C-c C-c ein:worksheet-execute-all-cells C-c C-e ein:worksheet-toggle-output-km C-c C-f ein:file-open-km C-c C-k ein:worksheet-kill-cell-km C-c C-l ein:worksheet-clear-output-km C-c RET ein:worksheet-merge-cell-km C-c C-n ein:worksheet-goto-next-input-km C-c C-o ein:notebook-open-km C-c C-p ein:worksheet-goto-prev-input-km C-c C-q ein:notebook-kill-kernel-then-close-command-km C-c C-r ein:notebook-reconnect-session-command-km C-c C-s ein:worksheet-split-cell-at-point-km C-c C-t ein:worksheet-toggle-cell-type-km C-c C-u ein:worksheet-change-cell-type-km C-c C-v ein:worksheet-set-output-visibility-all-km C-c C-w ein:worksheet-copy-cell-km C-c C-y ein:worksheet-yank-cell-km C-c C-z ein:notebook-kernel-interrupt-command-km C-c C-S-l ein:worksheet-clear-all-output-km C-c C-# ein:notebook-close-km C-c C-$ ein:tb-show-km C-c C-/ ein:notebook-scratchsheet-open-km C-c C-; ein:shared-output-show-code-cell-at-point-km C-c ein:worksheet-move-cell-down-km C-c ein:worksheet-move-cell-up-km

C-c C-x C-r ein:notebook-restart-session-command-km

C-c M-w ein:worksheet-copy-cell-km

This is a minor mode. If called interactively, toggle the ‘Ein:Notebook mode’ mode. If the prefix argument is positive, enable the mode, and if it is zero or negative, disable the mode.

If called from Lisp, toggle the mode if ARG is ‘toggle’. Enable the mode if ARG is nil, omitted, or is a positive number. Disable the mode if ARG is a negative number.

To check whether the minor mode is enabled in the current buffer, evaluate ‘ein:notebook-mode’.

The mode’s hook is called both when the mode is enabled and when it is disabled.

epgsql/epgsql

2024-12-01T01:53:25Z

Erlang PostgreSQL client library.

esmf-org/esmf

2024-12-01T01:53:32Z

The Earth System Modeling Framework (ESMF) is a suite of software tools for developing high-performance, multi-component Earth science modeling applications.

foxssake/netfox

2024-12-01T01:53:37Z

Addons for building multiplayer games with Godot

don-tnowe/godot-resources-as-sheets-plugin

2024-12-01T01:53:37Z

I needed to edit many Resource files, so figured someone would too.

nezhahq/nezha

2024-12-01T01:32:15Z

Self-hosted, lightweight server and website monitoring and O&M tool

LOGO designed by 熊大 .

) Nezha Monitoring: Self-hostable, lightweight, servers and websites monitoring and O&M tool.

Supports monitoring system status, HTTP (SSL certificate change, upcoming expiration, expired), TCP, Ping and supports push alerts, run scheduled tasks and web terminal.

>> Telegram Channel: 哪吒监控（中文通知频道）

>> Telegram Group: Nezha Monitoring Global (English Only), 哪吒监控（中文群组）

>> Use Cases | 我们的用户 (Google)

User Guide

Screenshots

用户前台 @hamster1963	管理后台 @uubulb

hamster1963/nezha-dash-react	nezhahq/admin-frontend

Supported Languages

English
中文
Español

You can change the dashboard language in the settings page (/setting) after the dashboard is installed.

Contributors

Special Thanks

IPInfo for providing an accurate GeoIP Database.

Star History

AlexxIT/go2rtc

2024-12-01T01:53:48Z

Ultimate camera streaming application with support RTSP, RTMP, HTTP-FLV, WebRTC, MSE, HLS, MP4, MJPEG, HomeKit, FFmpeg, etc.

Ultimate camera streaming application with support RTSP, WebRTC, HomeKit, FFmpeg, RTMP, etc.

zero-dependency and zero-config small app for all OS (Windows, macOS, Linux, ARM)
zero-delay for many supported protocols (lowest possible streaming latency)
streaming from RTSP, RTMP, DVRIP, HTTP (FLV/MJPEG/JPEG/TS), USB Cameras and other sources
streaming from any sources, supported by FFmpeg
streaming to RTSP, WebRTC, MSE/MP4, HomeKit HLS or MJPEG
publish any source to popular streaming services (YouTube, Telegram, etc.)
first project in the World with support streaming from HomeKit Cameras
support H265 for WebRTC in browser (Safari only, read more)
on the fly transcoding for unsupported codecs via FFmpeg
play audio files and live streams on some cameras with speaker
multi-source 2-way codecs negotiation
- mixing tracks from different sources to single stream
- auto match client supported codecs
- 2-way audio for some cameras
streaming from private networks via ngrok
can be integrated to any smart home platform or be used as standalone app

Inspired by:

series of streaming projects from @deepch
webrtc go library and whole @pion team
rtsp-simple-server idea from @aler9
GStreamer framework pipeline idea
MediaSoup framework routing idea
HomeKit Accessory Protocol from @brutella
creator of the project's logo @v_novoseltsev

Fast start

Download binary or use Docker or Home Assistant Add-on or Integration
Open web interface: http://localhost:1984/

Optionally:

add your streams to config file
setup external access to webrtc

Developers:

write your own web interface
integrate web api into your smart home platform

go2rtc: Binary

Download binary for your OS from latest release:

go2rtc_win64.zip - Windows 10+ 64-bit
go2rtc_win32.zip - Windows 7+ 32-bit
go2rtc_win_arm64.zip - Windows ARM 64-bit
go2rtc_linux_amd64 - Linux 64-bit
go2rtc_linux_i386 - Linux 32-bit
go2rtc_linux_arm64 - Linux ARM 64-bit (ex. Raspberry 64-bit OS)
go2rtc_linux_arm - Linux ARM 32-bit (ex. Raspberry 32-bit OS)
go2rtc_linux_armv6 - Linux ARMv6 (for old Raspberry 1 and Zero)
go2rtc_linux_mipsel - Linux MIPS (ex. Xiaomi Gateway 3, Wyze cameras)
go2rtc_mac_amd64.zip - macOS 10.13+ Intel 64-bit
go2rtc_mac_arm64.zip - macOS ARM 64-bit

Don't forget to fix the rights chmod +x go2rtc_xxx_xxx on Linux and Mac.

go2rtc: Docker

The Docker container alexxit/go2rtc supports multiple architectures including amd64, 386, arm64, and arm. This container offers the same functionality as the Home Assistant Add-on but is designed to operate independently of Home Assistant. It comes preinstalled with FFmpeg, ngrok, and Python.

go2rtc: Home Assistant Add-on

Install Add-On:
- Settings > Add-ons > Plus > Repositories > Add https://github.com/AlexxIT/hassio-addons
- go2rtc > Install > Start
Setup Integration

go2rtc: Home Assistant Integration

WebRTC Camera custom component can be used on any Home Assistant installation, including HassWP on Windows. It can automatically download and use the latest version of go2rtc. Or it can connect to an existing version of go2rtc. Addon installation in this case is optional.

go2rtc: Dev version

Latest, but maybe unstable version:

Binary: latest nightly release
Docker: alexxit/go2rtc:master or alexxit/go2rtc:master-hardware versions
Hass Add-on: go2rtc master or go2rtc master hardware versions

Configuration

by default go2rtc will search go2rtc.yaml in the current work directory
api server will start on default 1984 port (TCP)
rtsp server will start on default 8554 port (TCP)
webrtc will use port 8555 (TCP/UDP) for connections
ffmpeg will use default transcoding options

Configuration options and a complete list of settings can be found in the wiki.

Available modules:

streams
api - HTTP API (important for WebRTC support)
rtsp - RTSP Server (important for FFmpeg support)
webrtc - WebRTC Server
mp4 - MSE, MP4 stream and MP4 snapshot Server
hls - HLS TS or fMP4 stream Server
mjpeg - MJPEG Server
ffmpeg - FFmpeg integration
ngrok - ngrok integration (external access for private network)
hass - Home Assistant integration
log - logs config

Module: Streams

go2rtc support different stream source types. You can config one or multiple links of any type as stream source.

Available source types:

rtsp - RTSP and RTSPS cameras with two way audio support
rtmp - RTMP streams
http - HTTP-FLV, MPEG-TS, JPEG (snapshots), MJPEG streams
onvif - get camera RTSP link and snapshot link using ONVIF protocol
ffmpeg - FFmpeg integration (HLS, files and many others)
ffmpeg:device - local USB Camera or Webcam
exec - get media from external app output
echo - get stream link from bash or python
expr - get stream link via built-in expression language
homekit - streaming from HomeKit Camera
bubble - streaming from ESeeCloud/dvr163 NVR
dvrip - streaming from DVR-IP NVR
tapo - TP-Link Tapo cameras with two way audio support
kasa - TP-Link Kasa cameras
gopro - GoPro cameras
ivideon - public cameras from Ivideon service
hass - Home Assistant integration
isapi - two way audio for Hikvision (ISAPI) cameras
roborock - Roborock vacuums with cameras
webrtc - WebRTC/WHEP sources
webtorrent - WebTorrent source from another go2rtc

Two way audio

Supported for sources:

RTSP cameras with ONVIF Profile T (back channel connection)
DVRIP cameras
TP-Link Tapo cameras
Hikvision ISAPI cameras
Roborock vacuums models with cameras
Exec audio on server
Any Browser as IP-camera

Two way audio can be used in browser with WebRTC technology. The browser will give access to the microphone only for HTTPS sites (read more).

go2rtc also support play audio files and live streams on this cameras.

Source: RTSP

streams:
  sonoff_camera: rtsp://rtsp:12345678@192.168.1.123/av_stream/ch0
  dahua_camera:
    - rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
    - rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=1
  amcrest_doorbell:
    - rtsp://username:password@192.168.1.123:554/cam/realmonitor?channel=1&subtype=0#backchannel=0
  unifi_camera: rtspx://192.168.1.123:7441/fD6ouM72bWoFijxK
  glichy_camera: ffmpeg:rtsp://username:password@192.168.1.123/live/ch00_1

Recommendations

Amcrest Doorbell users may want to disable two way audio, because with an active stream you won't have a call button working. You need to add #backchannel=0 to the end of your RTSP link in YAML config file
Dahua Doorbell users may want to change backchannel audio codec
Reolink users may want NOT to use RTSP protocol at all, some camera models have a very awful unusable stream implementation
Ubiquiti UniFi users may want to disable HTTPS verification. Use rtspx:// prefix instead of rtsps://. And don't use ?enableSrtp suffix
TP-Link Tapo users may skip login and password, because go2rtc support login without them
If your camera has two RTSP links - you can add both of them as sources. This is useful when streams has different codecs, as example AAC audio with main stream and PCMU/PCMA audio with second stream
If the stream from your camera is glitchy, try using ffmpeg source. It will not add CPU load if you won't use transcoding
If the stream from your camera is very glitchy, try to use transcoding with ffmpeg source

Other options

Format: rtsp...#{param1}#{param2}#{param3}

Add custom timeout #timeout=30 (in seconds)
Ignore audio - #media=video or ignore video - #media=audio
Ignore two way audio API #backchannel=0 - important for some glitchy cameras
Use WebSocket transport #transport=ws...

RTSP over WebSocket

streams:
  # WebSocket with authorization, RTSP - without
  axis-rtsp-ws:  rtsp://192.168.1.123:4567/axis-media/media.amp?overview=0&camera=1&resolution=1280x720&videoframeskipmode=empty&Axis-Orig-Sw=true#transport=ws://user:pass@192.168.1.123:4567/rtsp-over-websocket
  # WebSocket without authorization, RTSP - with
  dahua-rtsp-ws: rtsp://user:pass@192.168.1.123/cam/realmonitor?channel=1&subtype=1&proto=Private3#transport=ws://192.168.1.123/rtspoverwebsocket

Source: RTMP

You can get stream from RTMP server, for example Nginx with nginx-rtmp-module.

streams:
  rtmp_stream: rtmp://192.168.1.123/live/camera1

Source: HTTP

Support Content-Type:

HTTP-FLV (video/x-flv) - same as RTMP, but over HTTP
HTTP-JPEG (image/jpeg) - camera snapshot link, can be converted by go2rtc to MJPEG stream
HTTP-MJPEG (multipart/x) - simple MJPEG stream over HTTP
MPEG-TS (video/mpeg) - legacy streaming format

Source also support HTTP and TCP streams with autodetection for different formats: MJPEG, H.264/H.265 bitstream, MPEG-TS.

streams:
  # [HTTP-FLV] stream in video/x-flv format
  http_flv: http://192.168.1.123:20880/api/camera/stream/780900131155/657617
  
  # [JPEG] snapshots from Dahua camera, will be converted to MJPEG stream
  dahua_snap: http://admin:password@192.168.1.123/cgi-bin/snapshot.cgi?channel=1

  # [MJPEG] stream will be proxied without modification
  http_mjpeg: https://mjpeg.sanford.io/count.mjpeg

  # [MJPEG or H.264/H.265 bitstream or MPEG-TS]
  tcp_magic: tcp://192.168.1.123:12345

  # Add custom header
  custom_header: "https://mjpeg.sanford.io/count.mjpeg#header=Authorization: Bearer XXX"

PS. Dahua camera has bug: if you select MJPEG codec for RTSP second stream - snapshot won't work.

Source: ONVIF

New in v1.5.0

The source is not very useful if you already know RTSP and snapshot links for your camera. But it can be useful if you don't.

WebUI > Add webpage support ONVIF autodiscovery. Your server must be on the same subnet as the camera. If you use docker, you must use "network host".

streams:
  dahua1: onvif://admin:password@192.168.1.123
  reolink1: onvif://admin:password@192.168.1.123:8000
  tapo1: onvif://admin:password@192.168.1.123:2020

Source: FFmpeg

You can get any stream or file or device via FFmpeg and push it to go2rtc. The app will automatically start FFmpeg with the proper arguments when someone starts watching the stream.

FFmpeg preistalled for Docker and Hass Add-on users
Hass Add-on users can target files from /media folder

Format: ffmpeg:{input}#{param1}#{param2}#{param3}. Examples:

streams:
  # [FILE] all tracks will be copied without transcoding codecs
  file1: ffmpeg:/media/BigBuckBunny.mp4

  # [FILE] video will be transcoded to H264, audio will be skipped
  file2: ffmpeg:/media/BigBuckBunny.mp4#video=h264

  # [FILE] video will be copied, audio will be transcoded to pcmu
  file3: ffmpeg:/media/BigBuckBunny.mp4#video=copy#audio=pcmu

  # [HLS] video will be copied, audio will be skipped
  hls: ffmpeg:https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/gear5/prog_index.m3u8#video=copy

  # [MJPEG] video will be transcoded to H264
  mjpeg: ffmpeg:http://185.97.122.128/cgi-bin/faststream.jpg#video=h264

  # [RTSP] video with rotation, should be transcoded, so select H264
  rotate: ffmpeg:rtsp://rtsp:12345678@192.168.1.123/av_stream/ch0#video=h264#rotate=90

All trascoding formats has built-in templates: h264, h265, opus, pcmu, pcmu/16000, pcmu/48000, pcma, pcma/16000, pcma/48000, aac, aac/16000.

But you can override them via YAML config. You can also add your own formats to config and use them with source params.

ffmpeg:
  bin: ffmpeg  # path to ffmpeg binary
  h264: "-codec:v libx264 -g:v 30 -preset:v superfast -tune:v zerolatency -profile:v main -level:v 4.1"
  mycodec: "-any args that supported by ffmpeg..."
  myinput: "-fflags nobuffer -flags low_delay -timeout 5000000 -i {input}"
  myraw: "-ss 00:00:20"

You can use go2rtc stream name as ffmpeg input (ex. ffmpeg:camera1#video=h264)
You can use video and audio params multiple times (ex. #video=copy#audio=copy#audio=pcmu)
You can use rotate param with 90, 180, 270 or -90 values, important with transcoding (ex. #video=h264#rotate=90)
You can use width and/or height params, important with transcoding (ex. #video=h264#width=1280)
You can use drawtext to add a timestamp (ex. drawtext=x=2:y=2:fontsize=12:fontcolor=white:box=1:boxcolor=black)
- This will greatly increase the CPU of the server, even with hardware acceleration
You can use raw param for any additional FFmpeg arguments (ex. #raw=-vf transpose=1)
You can use input param to override default input template (ex. #input=rtsp/udp will change RTSP transport from TCP to UDP+TCP)
- You can use raw input value (ex. #input=-timeout 5000000 -i {input})
- You can add your own input templates

Source: FFmpeg Device

You can get video from any USB-camera or Webcam as RTSP or WebRTC stream. This is part of FFmpeg integration.

check available devices in Web interface
video_size and framerate must be supported by your camera!
for Linux supported only video for now
for macOS you can stream Facetime camera or whole Desktop!
for macOS important to set right framerate

Format: ffmpeg:device?{input-params}#{param1}#{param2}#{param3}

streams:
  linux_usbcam:   ffmpeg:device?video=0&video_size=1280x720#video=h264
  windows_webcam: ffmpeg:device?video=0#video=h264
  macos_facetime: ffmpeg:device?video=0&audio=1&video_size=1280x720&framerate=30#video=h264#audio=pcma

PS. It is recommended to check the available devices in the WebUI add page.

Source: Exec

Exec source can run any external application and expect data from it. Two transports are supported - pipe (from v1.5.0) and RTSP.

If you want to use RTSP transport - the command must contain the {output} argument in any place. On launch, it will be replaced by the local address of the RTSP server.

pipe reads data from app stdout in different formats: MJPEG, H.264/H.265 bitstream, MPEG-TS. Also pipe can write data to app stdin in two formats: PCMA and PCM/48000.

The source can be used with:

FFmpeg - go2rtc ffmpeg source just a shortcut to exec source
FFplay - play audio on your server
GStreamer
Raspberry Pi Cameras
any your own software

Pipe commands support parameters (format: exec:{command}#{param1}#{param2}):

killsignal - signal which will be send to stop the process (numeric form)
killtimeout - time in seconds for forced termination with sigkill
backchannel - enable backchannel for two-way audio

streams:
  stream: exec:ffmpeg -re -i /media/BigBuckBunny.mp4 -c copy -rtsp_transport tcp -f rtsp {output}
  picam_h264: exec:libcamera-vid -t 0 --inline -o -
  picam_mjpeg: exec:libcamera-vid -t 0 --codec mjpeg -o -
  pi5cam_h264: exec:libcamera-vid -t 0 --libav-format h264 -o -
  canon: exec:gphoto2 --capture-movie --stdout#killsignal=2#killtimeout=5
  play_pcma: exec:ffplay -fflags nobuffer -f alaw -ar 8000 -i -#backchannel=1
  play_pcm48k: exec:ffplay -fflags nobuffer -f s16be -ar 48000 -i -#backchannel=1

Source: Echo

Some sources may have a dynamic link. And you will need to get it using a bash or python script. Your script should echo a link to the source. RTSP, FFmpeg or any of the supported sources.

Docker and Hass Add-on users has preinstalled python3, curl, jq.

Check examples in wiki.

streams:
  apple_hls: echo:python3 hls.py https://developer.apple.com/streaming/examples/basic-stream-osx-ios5.html

Source: Expr

New in v1.8.2

Like echo source, but uses the built-in expr expression language (read more).

Source: HomeKit

Important:

You can use HomeKit Cameras without Apple devices (iPhone, iPad, etc.), it's just a yet another protocol
HomeKit device can be paired with only one ecosystem. So, if you have paired it to an iPhone (Apple Home) - you can't pair it with Home Assistant or go2rtc. Or if you have paired it to go2rtc - you can't pair it with iPhone
HomeKit device should be in same network with working mDNS between device and go2rtc

go2rtc support import paired HomeKit devices from Home Assistant. So you can use HomeKit camera with Hass and go2rtc simultaneously. If you using Hass, I recommend pairing devices with it, it will give you more options.

You can pair device with go2rtc on the HomeKit page. If you can't see your devices - reload the page. Also try reboot your HomeKit device (power off). If you still can't see it - you have a problems with mDNS.

If you see a device but it does not have a pair button - it is paired to some ecosystem (Apple Home, Home Assistant, HomeBridge etc). You need to delete device from that ecosystem, and it will be available for pairing. If you cannot unpair device, you will have to reset it.

Important:

HomeKit audio uses very non-standard AAC-ELD codec with very non-standard params and specification violation
Audio can't be played in VLC and probably any other player
Audio should be transcoded for using with MSE, WebRTC, etc.

Recommended settings for using HomeKit Camera with WebRTC, MSE, MP4, RTSP:

streams:
  aqara_g3:
    - hass:Camera-Hub-G3-AB12
    - ffmpeg:aqara_g3#audio=aac#audio=opus

RTSP link with "normal" audio for any player: rtsp://192.168.1.123:8554/aqara_g3?video&audio=aac

This source is in active development! Tested only with Aqara Camera Hub G3 (both EU and CN versions).

Source: Bubble

New in v1.6.1

Other names: ESeeCloud, dvr163.

you can skip username, password, port, ch and stream if they are default
setup separate streams for different channels and streams

streams:
  camera1: bubble://username:password@192.168.1.123:34567/bubble/live?ch=0&stream=0

Source: DVRIP

New in v1.2.0

Other names: DVR-IP, NetSurveillance, Sofia protocol (NETsurveillance ActiveX plugin XMeye SDK).

you can skip username, password, port, channel and subtype if they are default
setup separate streams for different channels
use subtype=0 for Main stream, and subtype=1 for Extra1 stream
only the TCP protocol is supported

streams:
  only_stream: dvrip://username:password@192.168.1.123:34567?channel=0&subtype=0
  only_tts: dvrip://username:password@192.168.1.123:34567?backchannel=1
  two_way_audio:
    - dvrip://username:password@192.168.1.123:34567?channel=0&subtype=0
    - dvrip://username:password@192.168.1.123:34567?backchannel=1

Source: Tapo

New in v1.2.0

TP-Link Tapo proprietary camera protocol with two way audio support.

stream quality is the same as RTSP protocol
use the cloud password, this is not the RTSP password! you do not need to add a login!
you can also use UPPERCASE MD5 hash from your cloud password with admin username
some new camera firmwares requires SHA256 instead of MD5

streams:
  # cloud password without username
  camera1: tapo://cloud-password@192.168.1.123
  # admin username and UPPERCASE MD5 cloud-password hash
  camera2: tapo://admin:UPPERCASE-MD5@192.168.1.123
  # admin username and UPPERCASE SHA256 cloud-password hash
  camera3: tapo://admin:UPPERCASE-SHA256@192.168.1.123

echo -n "cloud password" | md5 | awk '{print toupper($0)}'
echo -n "cloud password" | shasum -a 256 | awk '{print toupper($0)}'

Source: Kasa

New in v1.7.0

TP-Link Kasa non-standard protocol more info.

username - urlsafe email, alex@gmail.com -> alex%40gmail.com
password - base64password, secret1 -> c2VjcmV0MQ==

streams:
  kc401: kasa://username:password@192.168.1.123:19443/https/stream/mixed

Tested: KD110, KC200, KC401, KC420WS, EC71.

Source: GoPro

New in v1.8.3

Support streaming from GoPro cameras, connected via USB or Wi-Fi to Linux, Mac, Windows. Read more.

Source: Ivideon

Support public cameras from service Ivideon.

streams:
  quailcam: ivideon:100-tu5dkUPct39cTp9oNEN2B6/0

Source: Hass

Support import camera links from Home Assistant config files:

Generic Camera, setup via GUI
HomeKit Camera
ONVIF
Roborock vacuums with camera

hass:
  config: "/config"  # skip this setting if you Hass Add-on user

streams:
  generic_camera: hass:Camera1  # Settings > Integrations > Integration Name
  aqara_g3: hass:Camera-Hub-G3-AB12

WebRTC Cameras (from v1.6.0)

Any cameras in WebRTC format are supported. But at the moment Home Assistant only supports some Nest cameras in this fomat.

Important. The Nest API only allows you to get a link to a stream for 5 minutes. Do not use this with Frigate! If the stream expires, Frigate will consume all available ram on your machine within seconds. It's recommended to use Nest source - it supports extending the stream.

streams:
  # link to Home Assistant Supervised
  hass-webrtc1: hass://supervisor?entity_id=camera.nest_doorbell
  # link to external Hass with Long-Lived Access Tokens
  hass-webrtc2: hass://192.168.1.123:8123?entity_id=camera.nest_doorbell&token=eyXYZ...

RTSP Cameras

By default, the Home Assistant API does not allow you to get dynamic RTSP link to a camera stream. So more cameras, like Tuya, and possibly others can also be imported by using this method.

Source: ISAPI

New in v1.3.0

This source type support only backchannel audio for Hikvision ISAPI protocol. So it should be used as second source in addition to the RTSP protocol.

streams:
  hikvision1:
    - rtsp://admin:password@192.168.1.123:554/Streaming/Channels/101
    - isapi://admin:password@192.168.1.123:80/

Source: Nest

New in v1.6.0

Currently only WebRTC cameras are supported.

For simplicity, it is recommended to connect the Nest/WebRTC camera to the Home Assistant. But if you can somehow get the below parameters - Nest/WebRTC source will work without Hass.

streams:
  nest-doorbell: nest:?client_id=***&client_secret=***&refresh_token=***&project_id=***&device_id=***

Source: Roborock

New in v1.3.0

This source type support Roborock vacuums with cameras. Known working models:

Roborock S6 MaxV - only video (the vacuum has no microphone)
Roborock S7 MaxV - video and two way audio
Roborock Qrevo MaxV - video and two way audio

Source support load Roborock credentials from Home Assistant custom integration or the core integration. Otherwise, you need to log in to your Roborock account (MiHome account is not supported). Go to: go2rtc WebUI > Add webpage. Copy roborock://... source for your vacuum and paste it to go2rtc.yaml config.

If you have graphic pin for your vacuum - add it as numeric pin (lines: 123, 456, 789) to the end of the roborock-link.

Source: WebRTC

New in v1.3.0

This source type support four connection formats.

whep

WebRTC/WHEP - is replaced by WebRTC/WISH standard for WebRTC video/audio viewers. But it may already be supported in some third-party software. It is supported in go2rtc.

go2rtc

This format is only supported in go2rtc. Unlike WHEP it supports asynchronous WebRTC connection and two way audio.

openipc (from v1.7.0)

Support connection to OpenIPC cameras.

wyze (from v1.6.1)

Supports connection to Wyze cameras, using WebRTC protocol. You can use docker-wyze-bridge project to get connection credentials.

kinesis (from v1.6.1)

Supports Amazon Kinesis Video Streams, using WebRTC protocol. You need to specify signalling WebSocket URL with all credentials in query params, client_id and ice_servers list in JSON format.

streams:
  webrtc-whep:    webrtc:http://192.168.1.123:1984/api/webrtc?src=camera1
  webrtc-go2rtc:  webrtc:ws://192.168.1.123:1984/api/ws?src=camera1
  webrtc-openipc: webrtc:ws://192.168.1.123/webrtc_ws#format=openipc#ice_servers=[{"urls":"stun:stun.kinesisvideo.eu-north-1.amazonaws.com:443"}]
  webrtc-wyze:    webrtc:http://192.168.1.123:5000/signaling/camera1?kvs#format=wyze
  webrtc-kinesis: webrtc:wss://...amazonaws.com/?...#format=kinesis#client_id=...#ice_servers=[{...},{...}]

PS. For kinesis sources you can use echo to get connection params using bash/python or any other script language.

Source: WebTorrent

New in v1.3.0

This source can get a stream from another go2rtc via WebTorrent protocol.

streams:
  webtorrent1: webtorrent:?share=huofssuxaty00izc&pwd=k3l2j9djeg8v8r7e

Incoming sources

By default, go2rtc establishes a connection to the source when any client requests it. Go2rtc drops the connection to the source when it has no clients left.

Go2rtc also can accepts incoming sources in RTSP, RTMP, HTTP and WebRTC/WHIP formats
Go2rtc won't stop such a source if it has no clients
You can push data only to existing stream (create stream with empty source in config)
You can push multiple incoming sources to same stream
You can push data to non empty stream, so it will have additional codecs inside

Examples

RTSP with any codec

ffmpeg -re -i BigBuckBunny.mp4 -c copy -rtsp_transport tcp -f rtsp rtsp://localhost:8554/camera1

HTTP-MJPEG with MJPEG codec

ffmpeg -re -i BigBuckBunny.mp4 -c mjpeg -f mpjpeg http://localhost:1984/api/stream.mjpeg?dst=camera1

HTTP-FLV with H264, AAC codecs

ffmpeg -re -i BigBuckBunny.mp4 -c copy -f flv http://localhost:1984/api/stream.flv?dst=camera1

MPEG-TS with H264 codec

ffmpeg -re -i BigBuckBunny.mp4 -c copy -f mpegts http://localhost:1984/api/stream.ts?dst=camera1

Incoming: Browser

New in v1.3.0

You can turn the browser of any PC or mobile into an IP-camera with support video and two way audio. Or even broadcast your PC screen:

Create empty stream in the go2rtc.yaml
Go to go2rtc WebUI
Open links page for you stream
Select camera+microphone or display+speaker option
Open webrtc local page (your go2rtc should work over HTTPS!) or share link via WebTorrent technology (work over HTTPS by default)

Incoming: WebRTC/WHIP

New in v1.3.0

You can use OBS Studio or any other broadcast software with WHIP protocol support. This standard has not yet been approved. But you can download OBS Studio dev version:

Settings > Stream > Service: WHIP > http://192.168.1.123:1984/api/webrtc?dst=camera1

Stream to camera

New in v1.3.0

go2rtc support play audio files (ex. music or TTS) and live streams (ex. radio) on cameras with two way audio support (RTSP/ONVIF cameras, TP-Link Tapo, Hikvision ISAPI, Roborock vacuums, any Browser).

API example:

POST http://localhost:1984/api/streams?dst=camera1&src=ffmpeg:http://example.com/song.mp3#audio=pcma#input=file

you can stream: local files, web files, live streams or any format, supported by FFmpeg
you should use ffmpeg source for transcoding audio to codec, that your camera supports
you can check camera codecs on the go2rtc WebUI info page when the stream is active
some cameras support only low quality PCMA/8000 codec (ex. Tapo)
it is recommended to choose higher quality formats if your camera supports them (ex. PCMA/48000 for some Dahua cameras)
if you play files over http-link, you need to add #input=file params for transcoding, so file will be transcoded and played in real time
if you play live streams, you should skip #input param, because it is already in real time
you can stop active playback by calling the API with the empty src parameter
you will see one active producer and one active consumer in go2rtc WebUI info page during streaming

Publish stream

New in v1.8.0

You can publish any stream to streaming services (YouTube, Telegram, etc.) via RTMP/RTMPS. Important:

Supported codecs: H264 for video and AAC for audio
AAC audio is required for YouTube, videos without audio will not work
You don't need to enable RTMP module listening for this task

You can use API:

POST http://localhost:1984/api/streams?src=camera1&dst=rtmps://...

Or config file:

publish:
  # publish stream "video_audio_transcode" to Telegram
  video_audio_transcode:
    - rtmps://xxx-x.rtmp.t.me/s/xxxxxxxxxx:xxxxxxxxxxxxxxxxxxxxxx
  # publish stream "audio_transcode" to Telegram and YouTube
  audio_transcode:
    - rtmps://xxx-x.rtmp.t.me/s/xxxxxxxxxx:xxxxxxxxxxxxxxxxxxxxxx
    - rtmp://xxx.rtmp.youtube.com/live2/xxxx-xxxx-xxxx-xxxx-xxxx

streams:
  video_audio_transcode:
    - ffmpeg:rtsp://user:pass@192.168.1.123/stream1#video=h264#hardware#audio=aac
  audio_transcode:
    - ffmpeg:rtsp://user:pass@192.168.1.123/stream1#video=copy#audio=aac

Telegram Desktop App > Any public or private channel or group (where you admin) > Live stream > Start with... > Start streaming.
YouTube > Create > Go live > Stream latency: Ultra low-latency > Copy: Stream URL + Stream key.

Module: API

The HTTP API is the main part for interacting with the application. Default address: http://localhost:1984/.

Important! go2rtc passes requests from localhost and from unix socket without HTTP authorisation, even if you have it configured! It is your responsibility to set up secure external access to API. If not properly configured, an attacker can gain access to your cameras and even your server.

API description.

Module config

you can disable HTTP API with listen: "" and use, for example, only RTSP client/server protocol
you can enable HTTP API only on localhost with listen: "127.0.0.1:1984" setting
you can change API base_path and host go2rtc on your main app webserver suburl
all files from static_dir hosted on root path: /
you can use raw TLS cert/key content or path to files

api:
  listen: ":1984"    # default ":1984", HTTP API port ("" - disabled)
  username: "admin"  # default "", Basic auth for WebUI
  password: "pass"   # default "", Basic auth for WebUI
  base_path: "/rtc"  # default "", API prefix for serve on suburl (/api => /rtc/api)
  static_dir: "www"  # default "", folder for static files (custom web interface)
  origin: "*"        # default "", allow CORS requests (only * supported)
  tls_listen: ":443" # default "", enable HTTPS server
  tls_cert: |        # default "", PEM-encoded fullchain certificate for HTTPS
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls_key: |         # default "", PEM-encoded private key for HTTPS
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----
  unix_listen: "/tmp/go2rtc.sock"  # default "", unix socket listener for API

PS:

MJPEG over WebSocket plays better than native MJPEG because Chrome bug
MP4 over WebSocket was created only for Apple iOS because it doesn't support MSE and native MP4

Module: RTSP

You can get any stream as RTSP-stream: rtsp://192.168.1.123:8554/{stream_name}

You can enable external password protection for your RTSP streams. Password protection always disabled for localhost calls (ex. FFmpeg or Hass on same server).

rtsp:
  listen: ":8554"    # RTSP Server TCP port, default - 8554
  username: "admin"  # optional, default - disabled
  password: "pass"   # optional, default - disabled
  default_query: "video&audio"  # optional, default codecs filters

By default go2rtc provide RTSP-stream with only one first video and only one first audio. You can change it with the default_query setting:

default_query: "mp4" - MP4 compatible codecs (H264, H265, AAC)
default_query: "video=all&audio=all" - all tracks from all source (not all players can handle this)
default_query: "video=h264,h265" - only one video track (H264 or H265)
default_query: "video&audio=all" - only one first any video and all audio as separate tracks

Module: RTMP

New in v1.8.0

You can get any stream as RTMP-stream: rtmp://192.168.1.123/{stream_name}. Only H264/AAC codecs supported right now.

Incoming stream in RTMP-format tested only with OBS Studio and Dahua camera. Different FFmpeg versions has differnt problems with this format.

rtmp:
  listen: ":1935"  # by default - disabled!

Module: WebRTC

In most cases WebRTC uses direct peer-to-peer connection from your browser to go2rtc and sends media data via UDP. It can't pass media data through your Nginx or Cloudflare or Nabu Casa HTTP TCP connection! It can automatically detects your external IP via public STUN server. It can establish a external direct connection via UDP hole punching technology even if you not open your server to the World.

But about 10-20% of users may need to configure additional settings for external access if mobile phone or go2rtc server behing Symmetric NAT.

by default, WebRTC uses both TCP and UDP on port 8555 for connections
you can use this port for external access
you can change the port in YAML config:

webrtc:
  listen: ":8555"  # address of your local server and port (TCP/UDP)

Static public IP

forward the port 8555 on your router (you can use same 8555 port or any other as external port)
add your external IP-address and external port to YAML config

webrtc:
  candidates:
    - 216.58.210.174:8555  # if you have static public IP-address

Dynamic public IP

forward the port 8555 on your router (you can use same 8555 port or any other as the external port)
add stun word and external port to YAML config
- go2rtc automatically detects your external address with STUN-server

webrtc:
  candidates:
    - stun:8555  # if you have dynamic public IP-address

Private IP

setup integration with ngrok service

ngrok:
  command: ...

Hard tech way 1. Own TCP-tunnel

If you have personal VPS, you can create TCP-tunnel and setup in the same way as "Static public IP". But use your VPS IP-address in YAML config.

Hard tech way 2. Using TURN-server

If you have personal VPS, you can install TURN server (e.g. coturn, config example).

webrtc:
  ice_servers:
    - urls: [stun:stun.l.google.com:19302]
    - urls: [turn:123.123.123.123:3478]
      username: your_user
      credential: your_pass

Module: HomeKit

New in v1.7.0

HomeKit module can work in two modes:

export any H264 camera to Apple HomeKit
transparent proxy any Apple HomeKit camera (Aqara, Eve, Eufy, etc.) back to Apple HomeKit, so you will have all camera features in Apple Home and also will have RTSP/WebRTC/MP4/etc. from your HomeKit camera

Important

HomeKit cameras supports only H264 video and OPUS audio

Minimal config

streams:
  dahua1: rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0
homekit:
  dahua1:  # same stream ID from streams list, default PIN - 19550224

Full config

streams:
  dahua1:
    - rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0
    - ffmpeg:dahua1#video=h264#hardware  # if your camera doesn't support H264, important for HomeKit
    - ffmpeg:dahua1#audio=opus           # only OPUS audio supported by HomeKit

homekit:
  dahua1:                   # same stream ID from streams list
    pin: 12345678           # custom PIN, default: 19550224
    name: Dahua camera      # custom camera name, default: generated from stream ID
    device_id: dahua1       # custom ID, default: generated from stream ID
    device_private: dahua1  # custom key, default: generated from stream ID

Proxy HomeKit camera

Video stream from HomeKit camera to Apple device (iPhone, AppleTV) will be transmitted directly
Video stream from HomeKit camera to RTSP/WebRTC/MP4/etc. will be transmitted via go2rtc

streams:
  aqara1:
    - homekit://...
    - ffmpeg:aqara1#audio=aac#audio=opus  # optional audio transcoding

homekit:
  aqara1:  # same stream ID from streams list

Module: WebTorrent

New in v1.3.0

This module support:

Share any local stream via WebTorrent technology
Get any incoming stream from PC or mobile via WebTorrent technology
Get any remote go2rtc source via WebTorrent technology

Securely and free. You do not need to open a public access to the go2rtc server. But in some cases (Symmetric NAT) you may need to set up external access to WebRTC module.

To generate sharing link or incoming link - goto go2rtc WebUI (stream links page). This link is temporary and will stop working after go2rtc is restarted!

You can create permanent external links in go2rtc config:

webtorrent:
  shares:
    super-secret-share:  # share name, should be unique among all go2rtc users!
      pwd: super-secret-password
      src: rtsp-dahua1   # stream name from streams section

Link example: https://alexxit.github.io/go2rtc/#share=02SNtgjKXY&pwd=wznEQqznxW&media=video+audio

TODO: article how it works...

Module: ngrok

With ngrok integration you can get external access to your streams in situations when you have Internet with private IP-address.

ngrok is pre-installed for Docker and Hass Add-on users
you may need external access for two different things:
- WebRTC stream, so you need tunnel WebRTC TCP port (ex. 8555)
- go2rtc web interface, so you need tunnel API HTTP port (ex. 1984)
ngrok support authorization for your web interface
ngrok automatically adds HTTPS to your web interface

The ngrok free subscription has the following limitations:

You can reserve a free domain for serving the web interface, but the TCP address you get will always be random and change with each restart of the ngrok agent (not a problem for webrtc stream)
You can forward multiple ports from a single agent, but you can only run one ngrok agent on the free plan

go2rtc will automatically get your external TCP address (if you enable it in ngrok config) and use it with WebRTC connection (if you enable it in webrtc config).

You need to manually download the ngrok agent app for your OS and register with the ngrok service.

Tunnel for only WebRTC Stream

You need to add your ngrok authtoken and WebRTC TCP port to YAML:

ngrok:
  command: ngrok tcp 8555 --authtoken eW91IHNoYWxsIG5vdCBwYXNzCnlvdSBzaGFsbCBub3QgcGFzcw

Tunnel for WebRTC and Web interface

You need to create ngrok.yaml config file and add it to go2rtc config:

ngrok:
  command: ngrok start --all --config ngrok.yaml

ngrok config example:

version: "2"
authtoken: eW91IHNoYWxsIG5vdCBwYXNzCnlvdSBzaGFsbCBub3QgcGFzcw
tunnels:
  api:
    addr: 1984  # use the same port as in go2rtc config
    proto: http
    basic_auth:
      - admin:password  # you can set login/pass for your web interface
  webrtc:
    addr: 8555  # use the same port as in go2rtc config
    proto: tcp

See the ngrok agent documentation for more details on the ngrok configuration file.

Module: Hass

The best and easiest way to use go2rtc inside the Home Assistant is to install the custom integration WebRTC Camera and custom lovelace card.

But go2rtc is also compatible and can be used with RTSPtoWebRTC built-in integration.

You have several options on how to add a camera to Home Assistant:

Camera RTSP source => Generic Camera
Camera any source => go2rtc config => Generic Camera
- Install any go2rtc
- Add your stream to go2rtc config
- Hass > Settings > Integrations > Add Integration > ONVIF > Host: 127.0.0.1, Port: 1984
- Hass > Settings > Integrations > Add Integration > Generic Camera > Stream Source URL: rtsp://127.0.0.1:8554/camera1 (change to your stream name, leave everything else as is)

You have several options on how to watch the stream from the cameras in Home Assistant:

Camera Entity => Picture Entity Card => Technology HLS, codecs: H264/H265/AAC, poor latency.
Camera Entity => RTSPtoWebRTC => Picture Entity Card => Technology WebRTC, codecs: H264/PCMU/PCMA/OPUS, best latency.
- Install any go2rtc
- Hass > Settings > Integrations > Add Integration > RTSPtoWebRTC > http://127.0.0.1:1984/
- RTSPtoWebRTC > Configure > STUN server: stun.l.google.com:19302
- Use Picture Entity or Picture Glance lovelace card
Camera Entity or Camera URL => WebRTC Camera => Technology: WebRTC/MSE/MP4/MJPEG, codecs: H264/H265/AAC/PCMU/PCMA/OPUS, best latency, best compatibility.
- Install and add WebRTC Camera custom integration
- Use WebRTC Camera custom lovelace card

You can add camera entity_id to go2rtc config if you need transcoding:

streams:
  "camera.hall": ffmpeg:{input}#video=copy#audio=opus

PS. Default Home Assistant lovelace cards don't support 2-way audio. You can use 2-way audio from Add-on Web UI. But you need use HTTPS to access the microphone. This is a browser restriction and cannot be avoided.

PS. There is also another nice card with go2rtc support - Frigate Lovelace Card.

Module: MP4

Provides several features:

MSE stream (fMP4 over WebSocket)
Camera snapshots in MP4 format (single frame), can be sent to Telegram
HTTP progressive streaming (MP4 file stream) - bad format for streaming because of high start delay. This format doesn't work in all Safari browsers, but go2rtc will automatically redirect it to HLS/fMP4 it this case.

API examples:

MP4 snapshot: http://192.168.1.123:1984/api/frame.mp4?src=camera1 (H264, H265)
MP4 stream: http://192.168.1.123:1984/api/stream.mp4?src=camera1 (H264, H265, AAC)
MP4 file: http://192.168.1.123:1984/api/stream.mp4?src=camera1 (H264, H265*, AAC, OPUS, MP3, PCMA, PCMU, PCM)
- You can use mp4, mp4=flac and mp4=all param for codec filters
- You can use duration param in seconds (ex. duration=15)
- You can use filename param (ex. filename=record.mp4)
- You can use rotate param with 90, 180 or 270 values
- You can use scale param with positive integer values (ex. scale=4:3)

Module: HLS

New in v1.1.0

HLS is the worst technology for real-time streaming. It can only be useful on devices that do not support more modern technology, like WebRTC, MSE/MP4.

The go2rtc implementation differs from the standards and may not work with all players.

API examples:

HLS/TS stream: http://192.168.1.123:1984/api/stream.m3u8?src=camera1 (H264)
HLS/fMP4 stream: http://192.168.1.123:1984/api/stream.m3u8?src=camera1&mp4 (H264, H265, AAC)

Module: MJPEG

Important. For stream as MJPEG format, your source MUST contain the MJPEG codec. If your stream has a MJPEG codec - you can receive MJPEG stream or JPEG snapshots via API.

You can receive an MJPEG stream in several ways:

some cameras support MJPEG codec inside RTSP stream (ex. second stream for Dahua cameras)
some cameras has HTTP link with MJPEG stream
some cameras has HTTP link with snapshots - go2rtc can convert them to MJPEG stream
you can convert H264/H265 stream from your camera via FFmpeg integraion

With this example, your stream will have both H264 and MJPEG codecs:

streams:
  camera1:
    - rtsp://rtsp:12345678@192.168.1.123/av_stream/ch0
    - ffmpeg:camera1#video=mjpeg

API examples:

MJPEG stream: http://192.168.1.123:1984/api/stream.mjpeg?src=camera1
JPEG snapshots: http://192.168.1.123:1984/api/frame.jpeg?src=camera1
- You can use width/w and/or height/h params
- You can use rotate param with 90, 180, 270 or -90 values
- You can use hardware/hw param read more

PS. This module also supports streaming to the server console (terminal) in the animated ASCII art format (read more):

Module: Log

You can set different log levels for different modules.

log:
  level: info  # default level
  api: trace
  exec: debug
  ngrok: info
  rtsp: warn
  streams: error
  webrtc: fatal

Security

By default go2rtc starts the Web interface on port 1984 and RTSP on port 8554, as well as use port 8555 for WebRTC connections. The three ports are accessible from your local network. So anyone on your local network can watch video from your cameras without authorization. The same rule applies to the Home Assistant Add-on.

This is not a problem if you trust your local network as much as I do. But you can change this behaviour with a go2rtc.yaml config:

api:
  listen: "127.0.0.1:1984" # localhost

rtsp:
  listen: "127.0.0.1:8554" # localhost

webrtc:
  listen: ":8555" # external TCP/UDP port

local access to RTSP is not a problem for FFmpeg integration, because it runs locally on your server
local access to API is not a problem for Home Assistant Add-on, because Hass runs locally on same server and Add-on Web UI protected with Hass authorization (Ingress feature)
external access to WebRTC TCP port is not a problem, because it used only for transmit encrypted media data
- anyway you need to open this port to your local network and to the Internet in order for WebRTC to work

If you need Web interface protection without Home Assistant Add-on - you need to use reverse proxy, like Nginx, Caddy, ngrok, etc.

PS. Additionally WebRTC will try to use the 8555 UDP port for transmit encrypted media. It works without problems on the local network. And sometimes also works for external access, even if you haven't opened this port on your router (read more). But for stable external WebRTC access, you need to open the 8555 port on your router for both TCP and UDP.

Codecs filters

go2rtc can automatically detect which codecs your device supports for WebRTC and MSE technologies.

But it cannot be done for RTSP, HTTP progressive streaming, HLS technologies. You can manually add a codec filter when you create a link to a stream. The filters work the same for all three technologies. Filters do not create a new codec. They only select the suitable codec from existing sources. You can add new codecs to the stream using the FFmpeg transcoding.

Without filters:

RTSP will provide only the first video and only the first audio (any codec)
MP4 will include only compatible codecs (H264, H265, AAC)
HLS will output in the legacy TS format (H264 without audio)

Some examples:

rtsp://192.168.1.123:8554/camera1?mp4 - useful for recording as MP4 files (e.g. Hass or Frigate)
rtsp://192.168.1.123:8554/camera1?video=h264,h265&audio=aac - full version of the filter above
rtsp://192.168.1.123:8554/camera1?video=h264&audio=aac&audio=opus - H264 video codec and two separate audio tracks
rtsp://192.168.1.123:8554/camera1?video&audio=all - any video codec and all audio codecs as separate tracks
http://192.168.1.123:1984/api/stream.m3u8?src=camera1&mp4 - HLS stream with MP4 compatible codecs (HLS/fMP4)
http://192.168.1.123:1984/api/stream.m3u8?src=camera1&mp4=flac - HLS stream with PCMA/PCMU/PCM audio support (HLS/fMP4), won't work on old devices
http://192.168.1.123:1984/api/stream.mp4?src=camera1&mp4=flac - MP4 file with PCMA/PCMU/PCM audio support, won't work on old devices (ex. iOS 12)
http://192.168.1.123:1984/api/stream.mp4?src=camera1&mp4=all - MP4 file with non standard audio codecs, won't work on some players

Codecs madness

AVC/H.264 video can be played almost anywhere. But HEVC/H.265 has a lot of limitations in supporting with different devices and browsers. It's all about patents and money, you can't do anything about it.

Device	WebRTC	MSE	HTTP*	HLS
latency	best	medium	bad	bad
- Desktop Chrome 107+ - Desktop Edge - Android Chrome 107+	H264 PCMU, PCMA OPUS	H264, H265* AAC, FLAC* OPUS	H264, H265* AAC, FLAC* OPUS, MP3	no
Desktop Firefox	H264 PCMU, PCMA OPUS	H264 AAC, FLAC* OPUS	H264 AAC, FLAC* OPUS	no
- Desktop Safari 14+ - iPad Safari 14+ - iPhone Safari 17.1+	H264, H265* PCMU, PCMA OPUS	H264, H265 AAC, FLAC*	no!	H264, H265 AAC, FLAC*
iPhone Safari 14+	H264, H265* PCMU, PCMA OPUS	no!	no!	H264, H265 AAC, FLAC*
macOS Hass App	no	no	no	H264, H265 AAC, FLAC*

HTTP* - HTTP Progressive Streaming, not related with Progressive download, because the file has no size and no end

Chrome H265: read this and read this
Edge H265: read this
Desktop Safari H265: Menu > Develop > Experimental > WebRTC H265
iOS Safari H265: Settings > Safari > Advanced > Experimental > WebRTC H265

Audio

Go2rtc support automatic repack PCMA/PCMU/PCM codecs to FLAC for MSE/MP4/HLS so they will work almost anywhere
WebRTC audio codecs: PCMU/8000, PCMA/8000, OPUS/48000/2
OPUS and MP3 inside MP4 is part of the standard, but some players do not support them anyway (especially Apple)

Apple devices

all Apple devices don't support HTTP progressive streaming
iPhones don't support MSE technology because it competes with the HTTP Live Streaming (HLS) technology, invented by Apple
HLS is the worst technology for live streaming, it still exists only because of iPhones

Codec names

H264 = H.264 = AVC (Advanced Video Coding)
H265 = H.265 = HEVC (High Efficiency Video Coding)
PCMA = G.711 PCM (A-law) = PCM A-law (alaw)
PCMU = G.711 PCM (µ-law) = PCM mu-law (mulaw)
PCM = L16 = PCM signed 16-bit big-endian (s16be)
AAC = MPEG4-GENERIC
MP3 = MPEG-1 Audio Layer III or MPEG-2 Audio Layer III

Built-in transcoding

There are no plans to embed complex transcoding algorithms inside go2rtc. FFmpeg source does a great job with this. Including hardware acceleration support.

But go2rtc has some simple algorithms. They are turned on automatically, you do not need to set them up additionally.

PCM for MSE/MP4/HLS

Go2rtc can pack PCMA, PCMU and PCM codecs into an MP4 container so that they work in all browsers and all built-in players on modern devices. Including Apple QuickTime:

PCMA/PCMU => PCM => FLAC => MSE/MP4/HLS

Resample PCMA/PCMU for WebRTC

By default WebRTC support only PCMA/8000 and PCMU/8000. But go2rtc can automatically resample PCMA and PCMU codec with with a different sample rate. Also go2rtc can transcode PCM codec to PCMA/8000, so WebRTC can play it:

PCM/xxx => PCMA/8000 => WebRTC
PCMA/xxx => PCMA/8000 => WebRTC
PCMU/xxx => PCMU/8000 => WebRTC

Important

FLAC codec not supported in a RTSP stream. If you using Frigate or Hass for recording MP4 files with PCMA/PCMU/PCM audio - you should setup transcoding to AAC codec.
PCMA and PCMU are VERY low quality codecs. Them support only 256! different sounds. Use them only when you have no other options.

Codecs negotiation

For example, you want to watch RTSP-stream from Dahua IPC-K42 camera in your Chrome browser.

this camera support 2-way audio standard ONVIF Profile T
this camera support codecs H264, H265 for send video, and you select H264 in camera settings
this camera support codecs AAC, PCMU, PCMA for send audio (from mic), and you select AAC/16000 in camera settings
this camera support codecs AAC, PCMU, PCMA for receive audio (to speaker), you don't need to select them
your browser support codecs H264, VP8, VP9, AV1 for receive video, you don't need to select them
your browser support codecs OPUS, PCMU, PCMA for send and receive audio, you don't need to select them
you can't get camera audio directly, because its audio codecs doesn't match with your browser codecs
- so you decide to use transcoding via FFmpeg and add this setting to config YAML file
- you have chosen OPUS/48000/2 codec, because it is higher quality than the PCMU/8000 or PCMA/8000

Now you have stream with two sources - RTSP and FFmpeg:

streams:
  dahua:
    - rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
    - ffmpeg:rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0#audio=opus

go2rtc automatically match codecs for you browser and all your stream sources. This called multi-source 2-way codecs negotiation. And this is one of the main features of this app.

PS. You can select PCMU or PCMA codec in camera setting and don't use transcoding at all. Or you can select AAC codec for main stream and PCMU codec for second stream and add both RTSP to YAML config, this also will work fine.

Projects using go2rtc

Frigate 12+ - open source NVR built around real-time AI object detection
Frigate Lovelace Card - custom card for Home Assistant
OpenIPC - Alternative IP Camera firmware from an open community
wz_mini_hacks - Custom firmware for Wyze cameras
EufyP2PStream - A small project that provides a Video/Audio Stream from Eufy cameras that don't directly support RTSP
ioBroker.euSec - ioBroker adapter for control Eufy security devices
MMM-go2rtc - MagicMirror² Module
ring-mqtt - Ring devices to MQTT Bridge

Distributions

Cameras experience

Dahua - reference implementation streaming protocols, a lot of settings, high stream quality, multiple streaming clients
EZVIZ - awful RTSP protocol realisation, many bugs in SDP
Hikvision - a lot of proprietary streaming technologies
Reolink - some models has awful unusable RTSP realisation and not best RTMP alternative (I recommend that you contact Reolink support for new firmware), few settings
Sonoff - very low stream quality, no settings, not best protocol implementation
TP-Link - few streaming clients, packet loss?
Chinese cheap noname cameras, Wyze Cams, Xiaomi cameras with hacks (usual has /live/ch00_1 in RTSP URL) - awful but usable RTSP protocol realisation, low stream quality, few settings, packet loss?

TIPS

Using apps for low RTSP delay

ffplay -fflags nobuffer -flags low_delay "rtsp://192.168.1.123:8554/camera1"
VLC > Preferences > Input / Codecs > Default Caching Level: Lowest Latency

Snapshots to Telegram

FAQ

Q. What's the difference between go2rtc, WebRTC Camera and RTSPtoWebRTC?

go2rtc is a new version of the server-side WebRTC Camera integration, completely rewritten from scratch, with a number of fixes and a huge number of new features. It is compatible with native Home Assistant RTSPtoWebRTC integration. So you can use default lovelace Picture Entity or Picture Glance.

Q. Should I use go2rtc addon or WebRTC Camera integration?

go2rtc is more than just viewing your stream online with WebRTC/MSE/HLS/etc. You can use it all the time for your various tasks. But every time the Hass is rebooted - all integrations are also rebooted. So your streams may be interrupted if you use them in additional tasks.

Basic users can use WebRTC Camera integration. Advanced users can use go2rtc addon or Frigate 12+ addon.

Q. Which RTSP link should I use inside Hass?

You can use direct link to your cameras there (as you always do). go2rtc support zero-config feature. You may leave streams config section empty. And your streams will be created on the fly on first start from Hass. And your cameras will have multiple connections. Some from Hass directly and one from go2rtc.

Also you can specify your streams in go2rtc config file and use RTSP links to this addon. With additional features: multi-source codecs negotiation or FFmpeg transcoding for unsupported codecs. Or use them as source for Frigate. And your cameras will have one connection from go2rtc. And go2rtc will have multiple connection - some from Hass via RTSP protocol, some from your browser via WebRTC/MSE/HLS protocols.

Use any config what you like.

Q. What about lovelace card with support 2-way audio?

At this moment I am focused on improving stability and adding new features to go2rtc. Maybe someone could write such a card themselves. It's not difficult, I have some sketches.

TwiN/gatus

2024-12-01T01:53:48Z

⛑ Automated developer-oriented status page

Gatus is a developer-oriented health dashboard that gives you the ability to monitor your services using HTTP, ICMP, TCP, and even DNS queries as well as evaluate the result of said queries by using a list of conditions on values like the status code, the response time, the certificate expiration, the body and many others. The icing on top is that each of these health checks can be paired with alerting via Slack, Teams, PagerDuty, Discord, Twilio and many more.

I personally deploy it in my Kubernetes cluster and let it monitor the status of my core applications: https://status.twin.sh/

Looking for a managed solution? Check out Gatus.io.

Quick start

docker run -p 8080:8080 --name gatus twinproduction/gatus

You can also use GitHub Container Registry if you prefer:

docker run -p 8080:8080 --name gatus ghcr.io/twin/gatus

For more details, see Usage

❤ Like this project? Please consider sponsoring me.

Have any feedback or questions? Create a discussion.

Why Gatus?

Before getting into the specifics, I want to address the most common question:

Why would I use Gatus when I can just use Prometheus’ Alertmanager, Cloudwatch or even Splunk?

Neither of these can tell you that there’s a problem if there are no clients actively calling the endpoint. In other words, it's because monitoring metrics mostly rely on existing traffic, which effectively means that unless your clients are already experiencing a problem, you won't be notified.

Gatus, on the other hand, allows you to configure health checks for each of your features, which in turn allows it to monitor these features and potentially alert you before any clients are impacted.

A sign you may want to look into Gatus is by simply asking yourself whether you'd receive an alert if your load balancer was to go down right now. Will any of your existing alerts be triggered? Your metrics won’t report an increase in errors if no traffic makes it to your applications. This puts you in a situation where your clients are the ones that will notify you about the degradation of your services rather than you reassuring them that you're working on fixing the issue before they even know about it.

Features

The main features of Gatus are:

Highly flexible health check conditions: While checking the response status may be enough for some use cases, Gatus goes much further and allows you to add conditions on the response time, the response body and even the IP address.
Ability to use Gatus for user acceptance tests: Thanks to the point above, you can leverage this application to create automated user acceptance tests.
Very easy to configure: Not only is the configuration designed to be as readable as possible, it's also extremely easy to add a new service or a new endpoint to monitor.
Alerting: While having a pretty visual dashboard is useful to keep track of the state of your application(s), you probably don't want to stare at it all day. Thus, notifications via Slack, Mattermost, Messagebird, PagerDuty, Twilio, Google chat and Teams are supported out of the box with the ability to configure a custom alerting provider for any needs you might have, whether it be a different provider or a custom application that manages automated rollbacks.
Metrics
Low resource consumption: As with most Go applications, the resource footprint that this application requires is negligibly small.
Badges:
Dark mode

Usage

Quick start

docker run -p 8080:8080 --name gatus twinproduction/gatus

You can also use GitHub Container Registry if you prefer:

docker run -p 8080:8080 --name gatus ghcr.io/twin/gatus

If you want to create your own configuration, see Docker for information on how to mount a configuration file.

Here's a simple example:

endpoints:
  - name: website                 # Name of your endpoint, can be anything
    url: "https://twin.sh/health"
    interval: 5m                  # Duration to wait between every status check (default: 60s)
    conditions:
      - "[STATUS] == 200"         # Status must be 200
      - "[BODY].status == UP"     # The json path "$.status" must be equal to UP
      - "[RESPONSE_TIME] < 300"   # Response time must be under 300ms

  - name: make-sure-header-is-rendered
    url: "https://example.org/"
    interval: 60s
    conditions:
      - "[STATUS] == 200"                          # Status must be 200
      - "[BODY] == pat(*<h1>Example Domain</h1>*)" # Body must contain the specified header

This example would look similar to this:

By default, the configuration file is expected to be at config/config.yaml.

You can specify a custom path by setting the GATUS_CONFIG_PATH environment variable.

If GATUS_CONFIG_PATH points to a directory, all *.yaml and *.yml files inside said directory and its subdirectories are merged like so:

All maps/objects are deep merged (i.e. you could define alerting.slack in one file and alerting.pagerduty in another file)
All slices/arrays are appended (i.e. you can define endpoints in multiple files and each endpoint will be added to the final list of endpoints)
Parameters with a primitive value (e.g. metrics, alerting.slack.webhook-url, etc.) may only be defined once to forcefully avoid any ambiguity
- To clarify, this also means that you could not define alerting.slack.webhook-url in two files with different values. All files are merged into one before they are processed. This is by design.

💡 You can also use environment variables in the configuration file (e.g. $DOMAIN, ${DOMAIN})

See examples/docker-compose-postgres-storage/config/config.yaml for an example.

If you want to test it locally, see Docker.

Configuration

Parameter	Description	Default
`metrics`	Whether to expose metrics at `/metrics`.	`false`
`storage`	Storage configuration.	`{}`
`alerting`	Alerting configuration.	`{}`
`endpoints`	Endpoints configuration.	Required `[]`
`external-endpoints`	External Endpoints configuration.	`[]`
`security`	Security configuration.	`{}`
`disable-monitoring-lock`	Whether to disable the monitoring lock.	`false`
`skip-invalid-config-update`	Whether to ignore invalid configuration update. See Reloading configuration on the fly.	`false`
`web`	Web configuration.	`{}`
`web.address`	Address to listen on.	`0.0.0.0`
`web.port`	Port to listen on.	`8080`
`web.read-buffer-size`	Buffer size for reading requests from a connection. Also limit for the maximum header size.	`8192`
`web.tls.certificate-file`	Optional public certificate file for TLS in PEM format.	``
`web.tls.private-key-file`	Optional private key file for TLS in PEM format.	``
`ui`	UI configuration.	`{}`
`ui.title`	Title of the document.	`Health Dashboard ǀ Gatus`
`ui.description`	Meta description for the page.	`Gatus is an advanced...`.
`ui.header`	Header at the top of the dashboard.	`Health Status`
`ui.logo`	URL to the logo to display.	`""`
`ui.link`	Link to open when the logo is clicked.	`""`
`ui.buttons`	List of buttons to display below the header.	`[]`
`ui.buttons[].name`	Text to display on the button.	Required `""`
`ui.buttons[].link`	Link to open when the button is clicked.	Required `""`
`maintenance`	Maintenance configuration.	`{}`

If you want more verbose logging, you may set the GATUS_LOG_LEVEL environment variable to DEBUG. Conversely, if you want less verbose logging, you can set the aforementioned environment variable to WARN, ERROR or FATAL. The default value for GATUS_LOG_LEVEL is INFO.

Endpoints

Endpoints are URLs, applications, or services that you want to monitor. Each endpoint has a list of conditions that are evaluated on an interval that you define. If any condition fails, the endpoint is considered as unhealthy. You can then configure alerts to be triggered when an endpoint is unhealthy once a certain threshold is reached.

Parameter	Description	Default
`endpoints`	List of endpoints to monitor.	Required `[]`
`endpoints[].enabled`	Whether to monitor the endpoint.	`true`
`endpoints[].name`	Name of the endpoint. Can be anything.	Required `""`
`endpoints[].group`	Group name. Used to group multiple endpoints together on the dashboard. See Endpoint groups.	`""`
`endpoints[].url`	URL to send the request to.	Required `""`
`endpoints[].method`	Request method.	`GET`
`endpoints[].conditions`	Conditions used to determine the health of the endpoint. See Conditions.	`[]`
`endpoints[].interval`	Duration to wait between every status check.	`60s`
`endpoints[].graphql`	Whether to wrap the body in a query param (`{"query":"$body"}`).	`false`
`endpoints[].body`	Request body.	`""`
`endpoints[].headers`	Request headers.	`{}`
`endpoints[].dns`	Configuration for an endpoint of type DNS. See Monitoring an endpoint using DNS queries.	`""`
`endpoints[].dns.query-type`	Query type (e.g. MX).	`""`
`endpoints[].dns.query-name`	Query name (e.g. example.com).	`""`
`endpoints[].ssh`	Configuration for an endpoint of type SSH. See Monitoring an endpoint using SSH.	`""`
`endpoints[].ssh.username`	SSH username (e.g. example).	Required `""`
`endpoints[].ssh.password`	SSH password (e.g. password).	Required `""`
`endpoints[].alerts`	List of all alerts for a given endpoint. See Alerting.	`[]`
`endpoints[].client`	Client configuration.	`{}`
`endpoints[].ui`	UI configuration at the endpoint level.	`{}`
`endpoints[].ui.hide-conditions`	Whether to hide conditions from the results. Note that this only hides conditions from results evaluated from the moment this was enabled.	`false`
`endpoints[].ui.hide-hostname`	Whether to hide the hostname in the result.	`false`
`endpoints[].ui.hide-url`	Whether to ensure the URL is not displayed in the results. Useful if the URL contains a token.	`false`
`endpoints[].ui.dont-resolve-failed-conditions`	Whether to resolve failed conditions for the UI.	`false`
`endpoints[].ui.badge.reponse-time`	List of response time thresholds. Each time a threshold is reached, the badge has a different color.	`[50, 200, 300, 500, 750]`

External Endpoints

Unlike regular endpoints, external endpoints are not monitored by Gatus, but they are instead pushed programmatically. This allows you to monitor anything you want, even when what you want to check lives in an environment that would not normally be accessible by Gatus.

For instance:

You can create your own agent that lives in a private network and pushes the status of your services to a publicly-exposed Gatus instance
You can monitor services that are not supported by Gatus
You can implement your own monitoring system while using Gatus as the dashboard

Parameter	Description	Default
`external-endpoints`	List of endpoints to monitor.	`[]`
`external-endpoints[].enabled`	Whether to monitor the endpoint.	`true`
`external-endpoints[].name`	Name of the endpoint. Can be anything.	Required `""`
`external-endpoints[].group`	Group name. Used to group multiple endpoints together on the dashboard. See Endpoint groups.	`""`
`external-endpoints[].token`	Bearer token required to push status to.	Required `""`
`external-endpoints[].alerts`	List of all alerts for a given endpoint. See Alerting.	`[]`

Example:

external-endpoints:
  - name: ext-ep-test
    group: core
    token: "potato"
    alerts:
      - type: discord
        description: "healthcheck failed"
        send-on-resolved: true

To push the status of an external endpoint, the request would have to look like this:

POST /api/v1/endpoints/{key}/external?success={success}&error={error}

Where:

{key} has the pattern <GROUP_NAME>_<ENDPOINT_NAME> in which both variables have , /, _, , and . replaced by -.
- Using the example configuration above, the key would be core_ext-ep-test.
{success} is a boolean (true or false) value indicating whether the health check was successful or not.
{error}: a string describing the reason for a failed health check. If {success} is false, this should contain the error message; if the check is successful, it can be omitted or left empty.

You must also pass the token as a Bearer token in the Authorization header.

Conditions

Here are some examples of conditions you can use:

Condition	Description	Passing values	Failing values
`[STATUS] == 200`	Status must be equal to 200	200	201, 404, ...
`[STATUS] < 300`	Status must lower than 300	200, 201, 299	301, 302, ...
`[STATUS] <= 299`	Status must be less than or equal to 299	200, 201, 299	301, 302, ...
`[STATUS] > 400`	Status must be greater than 400	401, 402, 403, 404	400, 200, ...
`[STATUS] == any(200, 429)`	Status must be either 200 or 429	200, 429	201, 400, ...
`[CONNECTED] == true`	Connection to host must've been successful	true	false
`[RESPONSE_TIME] < 500`	Response time must be below 500ms	100ms, 200ms, 300ms	500ms, 501ms
`[IP] == 127.0.0.1`	Target IP must be 127.0.0.1	127.0.0.1	0.0.0.0
`[BODY] == 1`	The body must be equal to 1	1	`{}`, `2`, ...
`[BODY].user.name == john`	JSONPath value of `$.user.name` is equal to `john`	`{"user":{"name":"john"}}`
`[BODY].data[0].id == 1`	JSONPath value of `$.data[0].id` is equal to 1	`{"data":[{"id":1}]}`
`[BODY].age == [BODY].id`	JSONPath value of `$.age` is equal JSONPath `$.id`	`{"age":1,"id":1}`
`len([BODY].data) < 5`	Array at JSONPath `$.data` has less than 5 elements	`{"data":[{"id":1}]}`
`len([BODY].name) == 8`	String at JSONPath `$.name` has a length of 8	`{"name":"john.doe"}`	`{"name":"bob"}`
`has([BODY].errors) == false`	JSONPath `$.errors` does not exist	`{"name":"john.doe"}`	`{"errors":[]}`
`has([BODY].users) == true`	JSONPath `$.users` exists	`{"users":[]}`	`{}`
`[BODY].name == pat(john*)`	String at JSONPath `$.name` matches pattern `john*`	`{"name":"john.doe"}`	`{"name":"bob"}`
`[BODY].id == any(1, 2)`	Value at JSONPath `$.id` is equal to `1` or `2`	1, 2	3, 4, 5
`[CERTIFICATE_EXPIRATION] > 48h`	Certificate expiration is more than 48h away	49h, 50h, 123h	1h, 24h, ...
`[DOMAIN_EXPIRATION] > 720h`	The domain must expire in more than 720h	4000h	1h, 24h, ...

Placeholders

Placeholder	Description	Example of resolved value
`[STATUS]`	Resolves into the HTTP status of the request	`404`
`[RESPONSE_TIME]`	Resolves into the response time the request took, in ms	`10`
`[IP]`	Resolves into the IP of the target host	`192.168.0.232`
`[BODY]`	Resolves into the response body. Supports JSONPath.	`{"name":"john.doe"}`
`[CONNECTED]`	Resolves into whether a connection could be established	`true`
`[CERTIFICATE_EXPIRATION]`	Resolves into the duration before certificate expiration (valid units are "s", "m", "h".)	`24h`, `48h`, 0 (if not protocol with certs)
`[DOMAIN_EXPIRATION]`	Resolves into the duration before the domain expires (valid units are "s", "m", "h".)	`24h`, `48h`, `1234h56m78s`
`[DNS_RCODE]`	Resolves into the DNS status of the response	`NOERROR`

Functions

Function	Description	Example
`len`	If the given path leads to an array, returns its length. Otherwise, the JSON at the given path is minified and converted to a string, and the resulting number of characters is returned. Works only with the `[BODY]` placeholder.	`len([BODY].username) > 8`
`has`	Returns `true` or `false` based on whether a given path is valid. Works only with the `[BODY]` placeholder.	`has([BODY].errors) == false`
`pat`	Specifies that the string passed as parameter should be evaluated as a pattern. Works only with `==` and `!=`.	`[IP] == pat(192.168.*)`
`any`	Specifies that any one of the values passed as parameters is a valid value. Works only with `==` and `!=`.	`[BODY].ip == any(127.0.0.1, ::1)`

💡 Use pat only when you need to. [STATUS] == pat(2*) is a lot more expensive than [STATUS] < 300.

Storage

Parameter	Description	Default
`storage`	Storage configuration	`{}`
`storage.path`	Path to persist the data in. Only supported for types `sqlite` and `postgres`.	`""`
`storage.type`	Type of storage. Valid types: `memory`, `sqlite`, `postgres`.	`"memory"`
`storage.caching`	Whether to use write-through caching. Improves loading time for large dashboards. Only supported if `storage.type` is `sqlite` or `postgres`	`false`

The results for each endpoint health check as well as the data for uptime and the past events must be persisted so that they can be displayed on the dashboard. These parameters allow you to configure the storage in question.

If storage.type is memory (default):

# Note that this is the default value, and you can omit the storage configuration altogether to achieve the same result.
# Because the data is stored in memory, the data will not survive a restart.
storage:
  type: memory

If storage.type is sqlite, storage.path must not be blank:

storage:
  type: sqlite
  path: data.db

See examples/docker-compose-sqlite-storage for an example.

If storage.type is postgres, storage.path must be the connection URL:

storage:
  type: postgres
  path: "postgres://user:password@127.0.0.1:5432/gatus?sslmode=disable"

See examples/docker-compose-postgres-storage for an example.

Client configuration

In order to support a wide range of environments, each monitored endpoint has a unique configuration for the client used to send the request.

Parameter	Description	Default
`client.insecure`	Whether to skip verifying the server's certificate chain and host name.	`false`
`client.ignore-redirect`	Whether to ignore redirects (true) or follow them (false, default).	`false`
`client.timeout`	Duration before timing out.	`10s`
`client.dns-resolver`	Override the DNS resolver using the format `{proto}://{host}:{port}`.	`""`
`client.oauth2`	OAuth2 client configuration.	`{}`
`client.oauth2.token-url`	The token endpoint URL	required `""`
`client.oauth2.client-id`	The client id which should be used for the `Client credentials flow`	required `""`
`client.oauth2.client-secret`	The client secret which should be used for the `Client credentials flow`	required `""`
`client.oauth2.scopes[]`	A list of `scopes` which should be used for the `Client credentials flow`.	required `[""]`
`client.proxy-url`	The URL of the proxy to use for the client	`""`
`client.identity-aware-proxy`	Google Identity-Aware-Proxy client configuration.	`{}`
`client.identity-aware-proxy.audience`	The Identity-Aware-Proxy audience. (client-id of the IAP oauth2 credential)	required `""`
`client.tls.certificate-file`	Path to a client certificate (in PEM format) for mTLS configurations.	`""`
`client.tls.private-key-file`	Path to a client private key (in PEM format) for mTLS configurations.	`""`
`client.tls.renegotiation`	Type of renegotiation support to provide. (`never`, `freely`, `once`).	`"never"`
`client.network`	The network to use for ICMP endpoint client (`ip`, `ip4` or `ip6`).	`"ip"`

📝 Some of these parameters are ignored based on the type of endpoint. For instance, there's no certificate involved in ICMP requests (ping), therefore, setting client.insecure to true for an endpoint of that type will not do anything.

This default configuration is as follows:

client:
  insecure: false
  ignore-redirect: false
  timeout: 10s

Note that this configuration is only available under endpoints[], alerting.mattermost and alerting.custom.

Here's an example with the client configuration under endpoints[]:

endpoints:
  - name: website
    url: "https://twin.sh/health"
    client:
      insecure: false
      ignore-redirect: false
      timeout: 10s
    conditions:
      - "[STATUS] == 200"

This example shows how you can specify a custom DNS resolver:

endpoints:
  - name: with-custom-dns-resolver
    url: "https://your.health.api/health"
    client:
      dns-resolver: "tcp://8.8.8.8:53"
    conditions:
      - "[STATUS] == 200"

This example shows how you can use the client.oauth2 configuration to query a backend API with Bearer token:

endpoints:
  - name: with-custom-oauth2
    url: "https://your.health.api/health"
    client:
      oauth2:
        token-url: https://your-token-server/token
        client-id: 00000000-0000-0000-0000-000000000000
        client-secret: your-client-secret
        scopes: ['https://your.health.api/.default']
    conditions:
      - "[STATUS] == 200"

This example shows how you can use the client.identity-aware-proxy configuration to query a backend API with Bearer token using Google Identity-Aware-Proxy:

endpoints:
  - name: with-custom-iap
    url: "https://my.iap.protected.app/health"
    client:
      identity-aware-proxy:
        audience: "XXXXXXXX-XXXXXXXXXXXX.apps.googleusercontent.com"
    conditions:
      - "[STATUS] == 200"

📝 Note that Gatus will use the gcloud default credentials within its environment to generate the token.

This example shows you how you can use the client.tls configuration to perform an mTLS query to a backend API:

endpoints:
  - name: website
    url: "https://your.mtls.protected.app/health"
    client:
      tls:
        certificate-file: /path/to/user_cert.pem
        private-key-file: /path/to/user_key.pem
        renegotiation: once
    conditions:
      - "[STATUS] == 200"

📝 Note that if running in a container, you must volume mount the certificate and key into the container.

Alerting

Gatus supports multiple alerting providers, such as Slack and PagerDuty, and supports different alerts for each individual endpoints with configurable descriptions and thresholds.

Alerts are configured at the endpoint level like so:

Parameter	Description	Default
`alerts`	List of all alerts for a given endpoint.	`[]`
`alerts[].type`	Type of alert. See table below for all valid types.	Required `""`
`alerts[].enabled`	Whether to enable the alert.	`true`
`alerts[].failure-threshold`	Number of failures in a row needed before triggering the alert.	`3`
`alerts[].success-threshold`	Number of successes in a row before an ongoing incident is marked as resolved.	`2`
`alerts[].send-on-resolved`	Whether to send a notification once a triggered alert is marked as resolved.	`false`
`alerts[].description`	Description of the alert. Will be included in the alert sent.	`""`

Here's an example of what an alert configuration might look like at the endpoint level:

endpoints:
  - name: example
    url: "https://example.org"
    conditions:
      - "[STATUS] == 200"
    alerts:
      - type: slack
        description: "healthcheck failed"
        send-on-resolved: true

📝 If an alerting provider is not properly configured, all alerts configured with the provider's type will be ignored.

Parameter	Description	Default
`alerting.custom`	Configuration for custom actions on failure or alerts. See Configuring Custom alerts.	`{}`
`alerting.discord`	Configuration for alerts of type `discord`. See Configuring Discord alerts.	`{}`
`alerting.email`	Configuration for alerts of type `email`. See Configuring Email alerts.	`{}`
`alerting.github`	Configuration for alerts of type `github`. See Configuring GitHub alerts.	`{}`
`alerting.gitlab`	Configuration for alerts of type `gitlab`. See Configuring GitLab alerts.	`{}`
`alerting.googlechat`	Configuration for alerts of type `googlechat`. See Configuring Google Chat alerts.	`{}`
`alerting.gotify`	Configuration for alerts of type `gotify`. See Configuring Gotify alerts.	`{}`
`alerting.jetbrainsspace`	Configuration for alerts of type `jetbrainsspace`. See Configuring JetBrains Space alerts.	`{}`
`alerting.matrix`	Configuration for alerts of type `matrix`. See Configuring Matrix alerts.	`{}`
`alerting.mattermost`	Configuration for alerts of type `mattermost`. See Configuring Mattermost alerts.	`{}`
`alerting.messagebird`	Configuration for alerts of type `messagebird`. See Configuring Messagebird alerts.	`{}`
`alerting.ntfy`	Configuration for alerts of type `ntfy`. See Configuring Ntfy alerts.	`{}`
`alerting.opsgenie`	Configuration for alerts of type `opsgenie`. See Configuring Opsgenie alerts.	`{}`
`alerting.pagerduty`	Configuration for alerts of type `pagerduty`. See Configuring PagerDuty alerts.	`{}`
`alerting.pushover`	Configuration for alerts of type `pushover`. See Configuring Pushover alerts.	`{}`
`alerting.slack`	Configuration for alerts of type `slack`. See Configuring Slack alerts.	`{}`
`alerting.teams`	Configuration for alerts of type `teams`. (Deprecated) See Configuring Teams alerts.	`{}`
`alerting.teams-workflows`	Configuration for alerts of type `teams-workflows`. See Configuring Teams Workflow alerts.	`{}`
`alerting.telegram`	Configuration for alerts of type `telegram`. See Configuring Telegram alerts.	`{}`
`alerting.twilio`	Settings for alerts of type `twilio`. See Configuring Twilio alerts.	`{}`

Configuring Discord alerts

Parameter	Description	Default
`alerting.discord`	Configuration for alerts of type `discord`	`{}`
`alerting.discord.webhook-url`	Discord Webhook URL	Required `""`
`alerting.discord.title`	Title of the notification	`":helmet_with_white_cross: Gatus"`
`alerting.discord.default-alert`	Default alert configuration. See Setting a default alert	N/A
`alerting.discord.overrides`	List of overrides that may be prioritized over the default configuration	`[]`
`alerting.discord.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`
`alerting.discord.overrides[].webhook-url`	Discord Webhook URL	`""`

alerting:
  discord:
    webhook-url: "https://discord.com/api/webhooks/**********/**********"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: discord
        description: "healthcheck failed"
        send-on-resolved: true

Configuring Email alerts

Parameter	Description	Default
`alerting.email`	Configuration for alerts of type `email`	`{}`
`alerting.email.from`	Email used to send the alert	Required `""`
`alerting.email.username`	Username of the SMTP server used to send the alert. If empty, uses `alerting.email.from`.	`""`
`alerting.email.password`	Password of the SMTP server used to send the alert. If empty, no authentication is performed.	`""`
`alerting.email.host`	Host of the mail server (e.g. `smtp.gmail.com`)	Required `""`
`alerting.email.port`	Port the mail server is listening to (e.g. `587`)	Required `0`
`alerting.email.to`	Email(s) to send the alerts to	Required `""`
`alerting.email.default-alert`	Default alert configuration. See Setting a default alert	N/A
`alerting.email.client.insecure`	Whether to skip TLS verification	`false`
`alerting.email.overrides`	List of overrides that may be prioritized over the default configuration	`[]`
`alerting.email.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`
`alerting.email.overrides[].to`	Email(s) to send the alerts to	`""`

alerting:
  email:
    from: "from@example.com"
    username: "from@example.com"
    password: "hunter2"
    host: "mail.example.com"
    port: 587
    to: "recipient1@example.com,recipient2@example.com"
    client:
      insecure: false
    # You can also add group-specific to keys, which will
    # override the to key above for the specified groups
    overrides:
      - group: "core"
        to: "recipient3@example.com,recipient4@example.com"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: email
        description: "healthcheck failed"
        send-on-resolved: true

  - name: back-end
    group: core
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[CERTIFICATE_EXPIRATION] > 48h"
    alerts:
      - type: email
        description: "healthcheck failed"
        send-on-resolved: true

⚠ Some mail servers are painfully slow.

Configuring Gitea alerts

Parameter	Description	Default
`alerting.gitea`	Configuration for alerts of type `gitea`	`{}`
`alerting.gitea.repository-url`	Gitea repository URL (e.g. `https://gitea.com/TwiN/example`)	Required `""`
`alerting.gitea.token`	Personal access token to use for authentication. Must have at least RW on issues and RO on metadata.	Required `""`
`alerting.github.default-alert`	Default alert configuration. See Setting a default alert.	N/A

The Gitea alerting provider creates an issue prefixed with alert(gatus): and suffixed with the endpoint's display name for each alert. If send-on-resolved is set to true on the endpoint alert, the issue will be automatically closed when the alert is resolved.

alerting:
  gitea:
    repository-url: "https://gitea.com/TwiN/test"
    token: "349d63f16......"

endpoints:
  - name: example
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 75"
    alerts:
      - type: gitea
        failure-threshold: 2
        success-threshold: 3
        send-on-resolved: true
        description: "Everything's burning AAAAAHHHHHHHHHHHHHHH"

Configuring GitHub alerts

Parameter	Description	Default
`alerting.github`	Configuration for alerts of type `github`	`{}`
`alerting.github.repository-url`	GitHub repository URL (e.g. `https://github.com/TwiN/example`)	Required `""`
`alerting.github.token`	Personal access token to use for authentication. Must have at least RW on issues and RO on metadata.	Required `""`
`alerting.github.default-alert`	Default alert configuration. See Setting a default alert.	N/A

The GitHub alerting provider creates an issue prefixed with alert(gatus): and suffixed with the endpoint's display name for each alert. If send-on-resolved is set to true on the endpoint alert, the issue will be automatically closed when the alert is resolved.

alerting:
  github:
    repository-url: "https://github.com/TwiN/test"
    token: "github_pat_12345..."

endpoints:
  - name: example
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 75"
    alerts:
      - type: github
        failure-threshold: 2
        success-threshold: 3
        send-on-resolved: true
        description: "Everything's burning AAAAAHHHHHHHHHHHHHHH"

Configuring GitLab alerts

Parameter	Description	Default
`alerting.gitlab`	Configuration for alerts of type `gitlab`	`{}`
`alerting.gitlab.webhook-url`	GitLab alert webhook URL (e.g. `https://gitlab.com/yourusername/example/alerts/notify/gatus/xxxxxxxxxxxxxxxx.json`)	Required `""`
`alerting.gitlab.authorization-key`	GitLab alert authorization key.	Required `""`
`alerting.gitlab.severity`	Override default severity (critical), can be one of `critical, high, medium, low, info, unknown`	`""`
`alerting.gitlab.monitoring-tool`	Override the monitoring tool name (gatus)	`"gatus"`
`alerting.gitlab.environment-name`	Set gitlab environment's name. Required to display alerts on a dashboard.	`""`
`alerting.gitlab.service`	Override endpoint display name	`""`
`alerting.gitlab.default-alert`	Default alert configuration. See Setting a default alert.	N/A

The GitLab alerting provider creates an alert prefixed with alert(gatus): and suffixed with the endpoint's display name for each alert. If send-on-resolved is set to true on the endpoint alert, the alert will be automatically closed when the alert is resolved. See https://docs.gitlab.com/ee/operations/incident_management/integrations.html#configuration to configure the endpoint.

alerting:
  gitlab:
    webhook-url: "https://gitlab.com/hlidotbe/example/alerts/notify/gatus/xxxxxxxxxxxxxxxx.json"
    authorization-key: "12345"

endpoints:
  - name: example
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 75"
    alerts:
      - type: gitlab
        failure-threshold: 2
        success-threshold: 3
        send-on-resolved: true
        description: "Everything's burning AAAAAHHHHHHHHHHHHHHH"

Configuring Google Chat alerts

Parameter	Description	Default
`alerting.googlechat`	Configuration for alerts of type `googlechat`	`{}`
`alerting.googlechat.webhook-url`	Google Chat Webhook URL	Required `""`
`alerting.googlechat.client`	Client configuration. See Client configuration.	`{}`
`alerting.googlechat.default-alert`	Default alert configuration. See Setting a default alert.	N/A
`alerting.googlechat.overrides`	List of overrides that may be prioritized over the default configuration	`[]`
`alerting.googlechat.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`
`alerting.googlechat.overrides[].webhook-url`	Google Chat Webhook URL	`""`

alerting:
  googlechat:
    webhook-url: "https://chat.googleapis.com/v1/spaces/*******/messages?key=**********&token=********"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: googlechat
        description: "healthcheck failed"
        send-on-resolved: true

Configuring Gotify alerts

Parameter	Description	Default
`alerting.gotify`	Configuration for alerts of type `gotify`	`{}`
`alerting.gotify.server-url`	Gotify server URL	Required `""`
`alerting.gotify.token`	Token that is used for authentication.	Required `""`
`alerting.gotify.priority`	Priority of the alert according to Gotify standards.	`5`
`alerting.gotify.title`	Title of the notification	`"Gatus: <endpoint>"`
`alerting.gotify.default-alert`	Default alert configuration. See Setting a default alert.	N/A

alerting:
  gotify:
    server-url: "https://gotify.example"
    token: "**************"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: gotify
        description: "healthcheck failed"
        send-on-resolved: true

Here's an example of what the notifications look like:

Configuring JetBrains Space alerts

Parameter	Description	Default
`alerting.jetbrainsspace`	Configuration for alerts of type `jetbrainsspace`	`{}`
`alerting.jetbrainsspace.project`	JetBrains Space project name	Required `""`
`alerting.jetbrainsspace.channel-id`	JetBrains Space Chat Channel ID	Required `""`
`alerting.jetbrainsspace.token`	Token that is used for authentication.	Required `""`
`alerting.jetbrainsspace.default-alert`	Default alert configuration. See Setting a default alert	N/A
`alerting.jetbrainsspace.overrides`	List of overrides that may be prioritized over the default configuration	`[]`
`alerting.jetbrainsspace.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`

alerting:
  jetbrainsspace:
    project: myproject
    channel-id: ABCDE12345
    token: "**************"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
    alerts:
      - type: jetbrainsspace
        description: "healthcheck failed"
        send-on-resolved: true

Here's an example of what the notifications look like:

Configuring Matrix alerts

Parameter	Description	Default
`alerting.matrix`	Configuration for alerts of type `matrix`	`{}`
`alerting.matrix.server-url`	Homeserver URL	`https://matrix-client.matrix.org`
`alerting.matrix.access-token`	Bot user access token (see https://webapps.stackexchange.com/q/131056)	Required `""`
`alerting.matrix.internal-room-id`	Internal room ID of room to send alerts to (can be found in Room Settings > Advanced)	Required `""`
`alerting.matrix.default-alert`	Default alert configuration. See Setting a default alert	N/A

alerting:
  matrix:
    server-url: "https://matrix-client.matrix.org"
    access-token: "123456"
    internal-room-id: "!example:matrix.org"

endpoints:
  - name: website
    interval: 5m
    url: "https://twin.sh/health"
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: matrix
        send-on-resolved: true
        description: "healthcheck failed"

Configuring Mattermost alerts

Parameter	Description	Default
`alerting.mattermost`	Configuration for alerts of type `mattermost`	`{}`
`alerting.mattermost.webhook-url`	Mattermost Webhook URL	Required `""`
`alerting.mattermost.channel`	Mattermost channel name override (optional)	`""`
`alerting.mattermost.client`	Client configuration. See Client configuration.	`{}`
`alerting.mattermost.default-alert`	Default alert configuration. See Setting a default alert.	N/A
`alerting.mattermost.overrides`	List of overrides that may be prioritized over the default configuration	`[]`
`alerting.mattermost.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`
`alerting.mattermist.overrides[].webhook-url`	Mattermost Webhook URL	`""`

alerting:
  mattermost:
    webhook-url: "http://**********/hooks/**********"
    client:
      insecure: true

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: mattermost
        description: "healthcheck failed"
        send-on-resolved: true

Here's an example of what the notifications look like:

Configuring Messagebird alerts

Parameter	Description	Default
`alerting.messagebird`	Configuration for alerts of type `messagebird`	`{}`
`alerting.messagebird.access-key`	Messagebird access key	Required `""`
`alerting.messagebird.originator`	The sender of the message	Required `""`
`alerting.messagebird.recipients`	The recipients of the message	Required `""`
`alerting.messagebird.default-alert`	Default alert configuration. See Setting a default alert	N/A

Example of sending SMS text message alert using Messagebird:

alerting:
  messagebird:
    access-key: "..."
    originator: "31619191918"
    recipients: "31619191919,31619191920"

endpoints:
  - name: website
    interval: 5m
    url: "https://twin.sh/health"
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: messagebird
        failure-threshold: 3
        send-on-resolved: true
        description: "healthcheck failed"

Configuring Ntfy alerts

Parameter	Description	Default
`alerting.ntfy`	Configuration for alerts of type `ntfy`	`{}`
`alerting.ntfy.topic`	Topic at which the alert will be sent	Required `""`
`alerting.ntfy.url`	The URL of the target server	`https://ntfy.sh`
`alerting.ntfy.token`	Access token for restricted topics	`""`
`alerting.ntfy.email`	E-mail address for additional e-mail notifications	`""`
`alerting.ntfy.click`	Website opened when notification is clicked	`""`
`alerting.ntfy.priority`	The priority of the alert	`3`
`alerting.ntfy.disable-firebase`	Whether message push delivery via firebase should be disabled. ntfy.sh defaults to enabled	`false`
`alerting.ntfy.disable-cache`	Whether server side message caching should be disabled. ntfy.sh defaults to enabled	`false`
`alerting.ntfy.default-alert`	Default alert configuration. See Setting a default alert	N/A

ntfy is an amazing project that allows you to subscribe to desktop and mobile notifications, making it an awesome addition to Gatus.

Example:

alerting:
  ntfy:
    topic: "gatus-test-topic"
    priority: 2
    token: faketoken
    default-alert:
      failure-threshold: 3
      send-on-resolved: true

endpoints:
  - name: website
    interval: 5m
    url: "https://twin.sh/health"
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: ntfy

Configuring Opsgenie alerts

Parameter	Description	Default
`alerting.opsgenie`	Configuration for alerts of type `opsgenie`	`{}`
`alerting.opsgenie.api-key`	Opsgenie API Key	Required `""`
`alerting.opsgenie.priority`	Priority level of the alert.	`P1`
`alerting.opsgenie.source`	Source field of the alert.	`gatus`
`alerting.opsgenie.entity-prefix`	Entity field prefix.	`gatus-`
`alerting.opsgenie.alias-prefix`	Alias field prefix.	`gatus-healthcheck-`
`alerting.opsgenie.tags`	Tags of alert.	`[]`
`alerting.opsgenie.default-alert`	Default alert configuration. See Setting a default alert	N/A

Opsgenie provider will automatically open and close alerts.

alerting:
  opsgenie:
    api-key: "00000000-0000-0000-0000-000000000000"

Configuring PagerDuty alerts

Parameter	Description	Default
`alerting.pagerduty`	Configuration for alerts of type `pagerduty`	`{}`
`alerting.pagerduty.integration-key`	PagerDuty Events API v2 integration key	`""`
`alerting.pagerduty.overrides`	List of overrides that may be prioritized over the default configuration	`[]`
`alerting.pagerduty.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`
`alerting.pagerduty.overrides[].integration-key`	PagerDuty Events API v2 integration key	`""`
`alerting.pagerduty.default-alert`	Default alert configuration. See Setting a default alert	N/A

It is highly recommended to set endpoints[].alerts[].send-on-resolved to true for alerts of type pagerduty, because unlike other alerts, the operation resulting from setting said parameter to true will not create another incident but mark the incident as resolved on PagerDuty instead.

Behavior:

By default, alerting.pagerduty.integration-key is used as the integration key
If the endpoint being evaluated belongs to a group (endpoints[].group) matching the value of alerting.pagerduty.overrides[].group, the provider will use that override's integration key instead of alerting.pagerduty.integration-key's

alerting:
  pagerduty:
    integration-key: "********************************"
    # You can also add group-specific integration keys, which will
    # override the integration key above for the specified groups
    overrides:
      - group: "core"
        integration-key: "********************************"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 30s
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: pagerduty
        failure-threshold: 3
        success-threshold: 5
        send-on-resolved: true
        description: "healthcheck failed"

  - name: back-end
    group: core
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[CERTIFICATE_EXPIRATION] > 48h"
    alerts:
      - type: pagerduty
        failure-threshold: 3
        success-threshold: 5
        send-on-resolved: true
        description: "healthcheck failed"

Configuring Pushover alerts

Parameter	Description	Default
`alerting.pushover`	Configuration for alerts of type `pushover`	`{}`
`alerting.pushover.application-token`	Pushover application token	`""`
`alerting.pushover.user-key`	User or group key	`""`
`alerting.pushover.title`	Fixed title for all messages sent via Pushover	Name of your App in Pushover
`alerting.pushover.priority`	Priority of all messages, ranging from -2 (very low) to 2 (emergency)	`0`
`alerting.pushover.resolved-priority`	Override the priority of messages on resolved, ranging from -2 (very low) to 2 (emergency)	`0`
`alerting.pushover.sound`	Sound of all messages See sounds for all valid choices.	`""`
`alerting.pushover.default-alert`	Default alert configuration. See Setting a default alert	N/A

alerting:
  pushover:
    application-token: "******************************"
    user-key: "******************************"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 30s
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: pushover
        failure-threshold: 3
        success-threshold: 5
        send-on-resolved: true
        description: "healthcheck failed"

Configuring Slack alerts

Parameter	Description	Default
`alerting.slack`	Configuration for alerts of type `slack`	`{}`
`alerting.slack.webhook-url`	Slack Webhook URL	Required `""`
`alerting.slack.default-alert`	Default alert configuration. See Setting a default alert	N/A
`alerting.slack.overrides`	List of overrides that may be prioritized over the default configuration	`[]`
`alerting.slack.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`
`alerting.slack.overrides[].webhook-url`	Slack Webhook URL	`""`

alerting:
  slack:
    webhook-url: "https://hooks.slack.com/services/**********/**********/**********"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 30s
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: slack
        description: "healthcheck failed 3 times in a row"
        send-on-resolved: true
      - type: slack
        failure-threshold: 5
        description: "healthcheck failed 5 times in a row"
        send-on-resolved: true

Here's an example of what the notifications look like:

Configuring Teams alerts (Deprecated)

[!CAUTION] Deprecated: Office 365 Connectors within Microsoft Teams are being retired (Source: Microsoft DevBlog). Existing connectors will continue to work until December 2025. The new Teams Workflow Alerts should be used with Microsoft Workflows instead of this legacy configuration.

Parameter	Description	Default
`alerting.teams`	Configuration for alerts of type `teams`	`{}`
`alerting.teams.webhook-url`	Teams Webhook URL	Required `""`
`alerting.teams.default-alert`	Default alert configuration. See Setting a default alert	N/A
`alerting.teams.title`	Title of the notification	`"🚨 Gatus"`
`alerting.teams.overrides`	List of overrides that may be prioritized over the default configuration	`[]`
`alerting.teams.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`
`alerting.teams.overrides[].webhook-url`	Teams Webhook URL	`""`
`alerting.teams.client.insecure`	Whether to skip TLS verification	`false`

alerting:
  teams:
    webhook-url: "https://********.webhook.office.com/webhookb2/************"
    client:
      insecure: false
    # You can also add group-specific to keys, which will
    # override the to key above for the specified groups
    overrides:
      - group: "core"
        webhook-url: "https://********.webhook.office.com/webhookb3/************"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 30s
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: teams
        description: "healthcheck failed"
        send-on-resolved: true

  - name: back-end
    group: core
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[CERTIFICATE_EXPIRATION] > 48h"
    alerts:
      - type: teams
        description: "healthcheck failed"
        send-on-resolved: true

Here's an example of what the notifications look like:

Configuring Teams Workflow alerts

[!NOTE] This alert is compatible with Workflows for Microsoft Teams. To setup the workflow and get the webhook URL, follow the Microsoft Documentation.

Parameter	Description	Default
`alerting.teams-workflows`	Configuration for alerts of type `teams`	`{}`
`alerting.teams-workflows.webhook-url`	Teams Webhook URL	Required `""`
`alerting.teams-workflows.title`	Title of the notification	`"⛑ Gatus"`
`alerting.teams-workflows.default-alert`	Default alert configuration. See Setting a default alert	N/A
`alerting.teams-workflows.overrides`	List of overrides that may be prioritized over the default configuration	`[]`
`alerting.teams-workflows.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`
`alerting.teams-workflows.overrides[].webhook-url`	Teams WorkFlow Webhook URL	`""`

alerting:
  teams-workflows:
    webhook-url: "https://********.webhook.office.com/webhookb2/************"
    # You can also add group-specific to keys, which will
    # override the to key above for the specified groups
    overrides:
      - group: "core"
        webhook-url: "https://********.webhook.office.com/webhookb3/************"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 30s
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: teams-workflows
        description: "healthcheck failed"
        send-on-resolved: true

  - name: back-end
    group: core
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[CERTIFICATE_EXPIRATION] > 48h"
    alerts:
      - type: teams-workflows
        description: "healthcheck failed"
        send-on-resolved: true

Here's an example of what the notifications look like:

Configuring Telegram alerts

Parameter	Description	Default
`alerting.telegram`	Configuration for alerts of type `telegram`	`{}`
`alerting.telegram.token`	Telegram Bot Token	Required `""`
`alerting.telegram.id`	Telegram User ID	Required `""`
`alerting.telegram.api-url`	Telegram API URL	`https://api.telegram.org`
`alerting.telegram.client`	Client configuration. See Client configuration.	`{}`
`alerting.telegram.default-alert`	Default alert configuration. See Setting a default alert	N/A
`alerting.telegram.overrides`	List of overrides that may be prioritized over the default configuration	`[]`
`alerting.telegram.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`
`alerting.telegram.overrides[].token`	Telegram Bot Token for override default value	`""`
`alerting.telegram.overrides[].id`	Telegram User ID for override default value	`""`

alerting:
  telegram:
    token: "123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11"
    id: "0123456789"

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 30s
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
    alerts:
      - type: telegram
        send-on-resolved: true

Here's an example of what the notifications look like:

Configuring Twilio alerts

Parameter	Description	Default
`alerting.twilio`	Settings for alerts of type `twilio`	`{}`
`alerting.twilio.sid`	Twilio account SID	Required `""`
`alerting.twilio.token`	Twilio auth token	Required `""`
`alerting.twilio.from`	Number to send Twilio alerts from	Required `""`
`alerting.twilio.to`	Number to send twilio alerts to	Required `""`
`alerting.twilio.default-alert`	Default alert configuration. See Setting a default alert	N/A

alerting:
  twilio:
    sid: "..."
    token: "..."
    from: "+1-234-567-8901"
    to: "+1-234-567-8901"

endpoints:
  - name: website
    interval: 30s
    url: "https://twin.sh/health"
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: twilio
        failure-threshold: 5
        send-on-resolved: true
        description: "healthcheck failed"

Configuring AWS SES alerts

Parameter	Description	Default
`alerting.aws-ses`	Settings for alerts of type `aws-ses`	`{}`
`alerting.aws-ses.access-key-id`	AWS Access Key ID	Optional `""`
`alerting.aws-ses.secret-access-key`	AWS Secret Access Key	Optional `""`
`alerting.aws-ses.region`	AWS Region	Required `""`
`alerting.aws-ses.from`	The Email address to send the emails from (should be registered in SES)	Required `""`
`alerting.aws-ses.to`	Comma separated list of email address to notify	Required `""`
`alerting.aws-ses.default-alert`	Default alert configuration. See Setting a default alert	N/A

alerting:
  aws-ses:
    access-key-id: "..."
    secret-access-key: "..."
    region: "us-east-1"
    from: "status@example.com"
    to: "user@example.com"

endpoints:
  - name: website
    interval: 30s
    url: "https://twin.sh/health"
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: aws-ses
        failure-threshold: 5
        send-on-resolved: true
        description: "healthcheck failed"

If the access-key-id and secret-access-key are not defined Gatus will fall back to IAM authentication.

Make sure you have the ability to use ses:SendEmail.

Configuring custom alerts

Parameter	Description	Default
`alerting.custom`	Configuration for custom actions on failure or alerts	`{}`
`alerting.custom.url`	Custom alerting request url	Required `""`
`alerting.custom.method`	Request method	`GET`
`alerting.custom.body`	Custom alerting request body.	`""`
`alerting.custom.headers`	Custom alerting request headers	`{}`
`alerting.custom.client`	Client configuration. See Client configuration.	`{}`
`alerting.custom.default-alert`	Default alert configuration. See Setting a default alert	N/A

While they're called alerts, you can use this feature to call anything.

For instance, you could automate rollbacks by having an application that keeps tracks of new deployments, and by leveraging Gatus, you could have Gatus call that application endpoint when an endpoint starts failing. Your application would then check if the endpoint that started failing was part of the recently deployed application, and if it was, then automatically roll it back.

Furthermore, you may use the following placeholders in the body (alerting.custom.body) and in the url (alerting.custom.url):

[ALERT_DESCRIPTION] (resolved from endpoints[].alerts[].description)
[ENDPOINT_NAME] (resolved from endpoints[].name)
[ENDPOINT_GROUP] (resolved from endpoints[].group)
[ENDPOINT_URL] (resolved from endpoints[].url)
[RESULT_ERRORS] (resolved from the health evaluation of a given health check)

If you have an alert using the custom provider with send-on-resolved set to true, you can use the [ALERT_TRIGGERED_OR_RESOLVED] placeholder to differentiate the notifications. The aforementioned placeholder will be replaced by TRIGGERED or RESOLVED accordingly, though it can be modified (details at the end of this section).

For all intents and purposes, we'll configure the custom alert with a Slack webhook, but you can call anything you want.

alerting:
  custom:
    url: "https://hooks.slack.com/services/**********/**********/**********"
    method: "POST"
    body: |
      {
        "text": "[ALERT_TRIGGERED_OR_RESOLVED]: [ENDPOINT_GROUP] - [ENDPOINT_NAME] - [ALERT_DESCRIPTION] - [RESULT_ERRORS]"
      }
endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 30s
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: custom
        failure-threshold: 10
        success-threshold: 3
        send-on-resolved: true
        description: "health check failed"

Note that you can customize the resolved values for the [ALERT_TRIGGERED_OR_RESOLVED] placeholder like so:

alerting:
  custom:
    placeholders:
      ALERT_TRIGGERED_OR_RESOLVED:
        TRIGGERED: "partial_outage"
        RESOLVED: "operational"

As a result, the [ALERT_TRIGGERED_OR_RESOLVED] in the body of first example of this section would be replaced by partial_outage when an alert is triggered and operational when an alert is resolved.

Setting a default alert

Parameter	Description	Default
`alerting.*.default-alert.enabled`	Whether to enable the alert	N/A
`alerting.*.default-alert.failure-threshold`	Number of failures in a row needed before triggering the alert	N/A
`alerting.*.default-alert.success-threshold`	Number of successes in a row before an ongoing incident is marked as resolved	N/A
`alerting.*.default-alert.send-on-resolved`	Whether to send a notification once a triggered alert is marked as resolved	N/A
`alerting.*.default-alert.description`	Description of the alert. Will be included in the alert sent	N/A

⚠ You must still specify the type of the alert in the endpoint configuration even if you set the default alert of a provider.

While you can specify the alert configuration directly in the endpoint definition, it's tedious and may lead to a very long configuration file.

To avoid such problem, you can use the default-alert parameter present in each provider configuration:

alerting:
  slack:
    webhook-url: "https://hooks.slack.com/services/**********/**********/**********"
    default-alert:
      description: "health check failed"
      send-on-resolved: true
      failure-threshold: 5
      success-threshold: 5

As a result, your Gatus configuration looks a lot tidier:

endpoints:
  - name: example
    url: "https://example.org"
    conditions:
      - "[STATUS] == 200"
    alerts:
      - type: slack

  - name: other-example
    url: "https://example.com"
    conditions:
      - "[STATUS] == 200"
    alerts:
      - type: slack

It also allows you to do things like this:

endpoints:
  - name: example
    url: "https://example.org"
    conditions:
      - "[STATUS] == 200"
    alerts:
      - type: slack
        failure-threshold: 5
      - type: slack
        failure-threshold: 10
      - type: slack
        failure-threshold: 15

Of course, you can also mix alert types:

alerting:
  slack:
    webhook-url: "https://hooks.slack.com/services/**********/**********/**********"
    default-alert:
      failure-threshold: 3
  pagerduty:
    integration-key: "********************************"
    default-alert:
      failure-threshold: 5

endpoints:
  - name: endpoint-1
    url: "https://example.org"
    conditions:
      - "[STATUS] == 200"
    alerts:
      - type: slack
      - type: pagerduty

  - name: endpoint-2
    url: "https://example.org"
    conditions:
      - "[STATUS] == 200"
    alerts:
      - type: slack
      - type: pagerduty

Configuring Zulip alerts

Parameter	Description	Default
`alerting.zulip`	Configuration for alerts of type `discord`	`{}`
`alerting.zulip.bot-email`	Bot Email	Required `""`
`alerting.zulip.bot-api-key`	Bot API key	Required `""`
`alerting.zulip.domain`	Full organization domain (e.g.: yourZulipDomain.zulipchat.com)	Required `""`
`alerting.zulip.channel-id`	The channel ID where Gatus will send the alerts	Required `""`
`alerting.zulip.overrides[].group`	Endpoint group for which the configuration will be overridden by this configuration	`""`
`alerting.zulip.overrides[].bot-email`	.	`""`
`alerting.zulip.overrides[].bot-api-key`	.	`""`
`alerting.zulip.overrides[].domain`	.	`""`
`alerting.zulip.overrides[].channel-id`	.	`""`

alerting:
  zulip:
    bot-email: gatus-bot@some.zulip.org
    bot-api-key: "********************************"
    domain: some.zulip.org
    channel-id: 123456

endpoints:
  - name: website
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
    alerts:
      - type: zulip
        description: "healthcheck failed"
        send-on-resolved: true

Maintenance

If you have maintenance windows, you may not want to be annoyed by alerts. To do that, you'll have to use the maintenance configuration:

Parameter	Description	Default
`maintenance.enabled`	Whether the maintenance period is enabled	`true`
`maintenance.start`	Time at which the maintenance window starts in `hh:mm` format (e.g. `23:00`)	Required `""`
`maintenance.duration`	Duration of the maintenance window (e.g. `1h`, `30m`)	Required `""`
`maintenance.timezone`	Timezone of the maintenance window format (e.g. `Europe/Amsterdam`). See List of tz database time zones for more info	`UTC`
`maintenance.every`	Days on which the maintenance period applies (e.g. `[Monday, Thursday]`). If left empty, the maintenance window applies every day	`[]`

Here's an example:

maintenance:
  start: 23:00
  duration: 1h
  timezone: "Europe/Amsterdam"
  every: [Monday, Thursday]

Note that you can also specify each day on separate lines:

maintenance:
  start: 23:00
  duration: 1h
  timezone: "Europe/Amsterdam"
  every:
    - Monday
    - Thursday

Security

Parameter	Description	Default
`security`	Security configuration	`{}`
`security.basic`	HTTP Basic configuration	`{}`
`security.oidc`	OpenID Connect configuration	`{}`

Basic Authentication

Parameter	Description	Default
`security.basic`	HTTP Basic configuration	`{}`
`security.basic.username`	Username for Basic authentication.	Required `""`
`security.basic.password-bcrypt-base64`	Password hashed with Bcrypt and then encoded with base64 for Basic authentication.	Required `""`

The example below will require that you authenticate with the username john.doe and the password hunter2:

security:
  basic:
    username: "john.doe"
    password-bcrypt-base64: "JDJhJDEwJHRiMnRFakxWazZLdXBzRERQazB1TE8vckRLY05Yb1hSdnoxWU0yQ1FaYXZRSW1McmladDYu"

⚠ Make sure to carefully select to cost of the bcrypt hash. The higher the cost, the longer it takes to compute the hash, and basic auth verifies the password against the hash on every request. As of 2023-01-06, I suggest a cost of 9.

OIDC

Parameter	Description	Default
`security.oidc`	OpenID Connect configuration	`{}`
`security.oidc.issuer-url`	Issuer URL	Required `""`
`security.oidc.redirect-url`	Redirect URL. Must end with `/authorization-code/callback`	Required `""`
`security.oidc.client-id`	Client id	Required `""`
`security.oidc.client-secret`	Client secret	Required `""`
`security.oidc.scopes`	Scopes to request. The only scope you need is `openid`.	Required `[]`
`security.oidc.allowed-subjects`	List of subjects to allow. If empty, all subjects are allowed.	`[]`

security:
  oidc:
    issuer-url: "https://example.okta.com"
    redirect-url: "https://status.example.com/authorization-code/callback"
    client-id: "123456789"
    client-secret: "abcdefghijk"
    scopes: ["openid"]
    # You may optionally specify a list of allowed subjects. If this is not specified, all subjects will be allowed.
    #allowed-subjects: ["johndoe@example.com"]

Confused? Read Securing Gatus with OIDC using Auth0.

TLS Encryption

Gatus supports basic encryption with TLS. To enable this, certificate files in PEM format have to be provided.

The example below shows an example configuration which makes gatus respond on port 4443 to HTTPS requests:

web:
  port: 4443
  tls:
    certificate-file: "certificate.crt"
    private-key-file: "private.key"

Metrics

To enable metrics, you must set metrics to true. Doing so will expose Prometheus-friendly metrics at the /metrics endpoint on the same port your application is configured to run on (web.port).

Metric name	Type	Description	Labels	Relevant endpoint types
gatus_results_total	counter	Number of results per endpoint	key, group, name, type, success	All
gatus_results_code_total	counter	Total number of results by code	key, group, name, type, code	DNS, HTTP
gatus_results_connected_total	counter	Total number of results in which a connection was successfully established	key, group, name, type	All
gatus_results_duration_seconds	gauge	Duration of the request in seconds	key, group, name, type	All
gatus_results_certificate_expiration_seconds	gauge	Number of seconds until the certificate expires	key, group, name, type	HTTP, STARTTLS

See examples/docker-compose-grafana-prometheus for further documentation as well as an example.

Connectivity

Parameter	Description	Default
`connectivity`	Connectivity configuration	`{}`
`connectivity.checker`	Connectivity checker configuration	Required `{}`
`connectivity.checker.target`	Host to use for validating connectivity	Required `""`
`connectivity.checker.interval`	Interval at which to validate connectivity	`1m`

While Gatus is used to monitor other services, it is possible for Gatus itself to lose connectivity to the internet. In order to prevent Gatus from reporting endpoints as unhealthy when Gatus itself is unhealthy, you may configure Gatus to periodically check for internet connectivity.

All endpoint executions are skipped while the connectivity checker deems connectivity to be down.

connectivity:
  checker:
    target: 1.1.1.1:53
    interval: 60s

Remote instances (EXPERIMENTAL)

This feature allows you to retrieve endpoint statuses from a remote Gatus instance.

There are two main use cases for this:

You have multiple Gatus instances running on different machines, and you wish to visually expose the statuses through a single dashboard
You have one or more Gatus instances that are not publicly accessible (e.g. behind a firewall), and you wish to retrieve

This is an experimental feature. It may be removed or updated in a breaking manner at any time. Furthermore, there are known issues with this feature. If you'd like to provide some feedback, please write a comment in #64. Use at your own risk.

Parameter	Description	Default
`remote`	Remote configuration	`{}`
`remote.instances`	List of remote instances	Required `[]`
`remote.instances.endpoint-prefix`	String to prefix all endpoint names with	`""`
`remote.instances.url`	URL from which to retrieve endpoint statuses	Required `""`

remote:
  instances:
    - endpoint-prefix: "status.example.org-"
      url: "https://status.example.org/api/v1/endpoints/statuses"

Deployment

Many examples can be found in the .examples folder, but this section will focus on the most popular ways of deploying Gatus.

Docker

To run Gatus locally with Docker:

docker run -p 8080:8080 --name gatus twinproduction/gatus

Other than using one of the examples provided in the .examples folder, you can also try it out locally by creating a configuration file, we'll call it config.yaml for this example, and running the following command:

docker run -p 8080:8080 --mount type=bind,source="$(pwd)"/config.yaml,target=/config/config.yaml --name gatus twinproduction/gatus

If you're on Windows, replace "$(pwd)" by the absolute path to your current directory, e.g.:

docker run -p 8080:8080 --mount type=bind,source=C:/Users/Chris/Desktop/config.yaml,target=/config/config.yaml --name gatus twinproduction/gatus

To build the image locally:

docker build . -t twinproduction/gatus

Helm Chart

Helm must be installed to use the chart. Please refer to Helm's documentation to get started.

Once Helm is set up properly, add the repository as follows:

helm repo add twin https://twin.github.io/helm-charts
helm repo update
helm install gatus twin/gatus

To get more details, please check chart's configuration.

Terraform

Gatus can be deployed on Terraform by using the following module: terraform-kubernetes-gatus.

Running the tests

go test -v ./...

Using in Production

See the Deployment section.

FAQ

Sending a GraphQL request

By setting endpoints[].graphql to true, the body will automatically be wrapped by the standard GraphQL query parameter.

For instance, the following configuration:

endpoints:
  - name: filter-users-by-gender
    url: http://localhost:8080/playground
    method: POST
    graphql: true
    body: |
      {
        users(gender: "female") {
          id
          name
          gender
          avatar
        }
      }
    conditions:
      - "[STATUS] == 200"
      - "[BODY].data.users[0].gender == female"

will send a POST request to http://localhost:8080/playground with the following body:

{"query":"      {\n        users(gender: \"female\") {\n          id\n          name\n          gender\n          avatar\n        }\n      }"}

Recommended interval

📝 This does not apply if disable-monitoring-lock is set to true, as the monitoring lock is what tells Gatus to only evaluate one endpoint at a time.

To ensure that Gatus provides reliable and accurate results (i.e. response time), Gatus only evaluates one endpoint at a time In other words, even if you have multiple endpoints with the same interval, they will not execute at the same time.

You can test this yourself by running Gatus with several endpoints configured with a very short, unrealistic interval, such as 1ms. You'll notice that the response time does not fluctuate - that is because while endpoints are evaluated on different goroutines, there's a global lock that prevents multiple endpoints from running at the same time.

Unfortunately, there is a drawback. If you have a lot of endpoints, including some that are very slow or prone to timing out (the default timeout is 10s), then it means that for the entire duration of the request, no other endpoint can be evaluated.

The interval does not include the duration of the request itself, which means that if an endpoint has an interval of 30s and the request takes 2s to complete, the timestamp between two evaluations will be 32s, not 30s.

While this does not prevent Gatus' from performing health checks on all other endpoints, it may cause Gatus to be unable to respect the configured interval, for instance:

Endpoint A has an interval of 5s, and times out after 10s to complete
Endpoint B has an interval of 5s, and takes 1ms to complete
Endpoint B will be unable to run every 5s, because endpoint A's health evaluation takes longer than its interval

To sum it up, while Gatus can handle any interval you throw at it, you're better off having slow requests with higher interval.

As a rule of thumb, I personally set the interval for more complex health checks to 5m (5 minutes) and simple health checks used for alerting (PagerDuty/Twilio) to 30s.

Default timeouts

Endpoint type	Timeout
HTTP	10s
TCP	10s
ICMP	10s

To modify the timeout, see Client configuration.

Monitoring a TCP endpoint

By prefixing endpoints[].url with tcp:\\, you can monitor TCP endpoints at a very basic level:

endpoints:
  - name: redis
    url: "tcp://127.0.0.1:6379"
    interval: 30s
    conditions:
      - "[CONNECTED] == true"

Placeholders [STATUS] and [BODY] as well as the fields endpoints[].body, endpoints[].headers, endpoints[].method and endpoints[].graphql are not supported for TCP endpoints.

This works for applications such as databases (Postgres, MySQL, etc.) and caches (Redis, Memcached, etc.).

📝 [CONNECTED] == true does not guarantee that the endpoint itself is healthy - it only guarantees that there's something at the given address listening to the given port, and that a connection to that address was successfully established.

Monitoring a UDP endpoint

By prefixing endpoints[].url with udp:\\, you can monitor UDP endpoints at a very basic level:

endpoints:
  - name: example
    url: "udp://example.org:80"
    conditions:
      - "[CONNECTED] == true"

Placeholders [STATUS] and [BODY] as well as the fields endpoints[].body, endpoints[].headers, endpoints[].method and endpoints[].graphql are not supported for UDP endpoints.

This works for UDP based application.

Monitoring a SCTP endpoint

By prefixing endpoints[].url with sctp:\\, you can monitor Stream Control Transmission Protocol (SCTP) endpoints at a very basic level:

endpoints:
  - name: example
    url: "sctp://127.0.0.1:38412"
    conditions:
      - "[CONNECTED] == true"

Placeholders [STATUS] and [BODY] as well as the fields endpoints[].body, endpoints[].headers, endpoints[].method and endpoints[].graphql are not supported for SCTP endpoints.

This works for SCTP based application.

Monitoring a WebSocket endpoint

By prefixing endpoints[].url with ws:// or wss://, you can monitor WebSocket endpoints at a very basic level:

endpoints:
  - name: example
    url: "wss://example.com/"
    body: "status"
    conditions:
      - "[CONNECTED] == true"
      - "[BODY].result >= 0"

The [BODY] placeholder contains the output of the query, and [CONNECTED] shows whether the connection was successfully established.

Monitoring an endpoint using ICMP

By prefixing endpoints[].url with icmp:\\, you can monitor endpoints at a very basic level using ICMP, or more commonly known as "ping" or "echo":

endpoints:
  - name: ping-example
    url: "icmp://example.com"
    conditions:
      - "[CONNECTED] == true"

Only the placeholders [CONNECTED], [IP] and [RESPONSE_TIME] are supported for endpoints of type ICMP. You can specify a domain prefixed by icmp://, or an IP address prefixed by icmp://.

If you run Gatus on Linux, please read the Linux section on https://github.com/prometheus-community/pro-bing#linux if you encounter any problems.

Monitoring an endpoint using DNS queries

Defining a dns configuration in an endpoint will automatically mark said endpoint as an endpoint of type DNS:

endpoints:
  - name: example-dns-query
    url: "8.8.8.8" # Address of the DNS server to use
    dns:
      query-name: "example.com"
      query-type: "A"
    conditions:
      - "[BODY] == 93.184.215.14"
      - "[DNS_RCODE] == NOERROR"

There are two placeholders that can be used in the conditions for endpoints of type DNS:

The placeholder [BODY] resolves to the output of the query. For instance, a query of type A would return an IPv4.
The placeholder [DNS_RCODE] resolves to the name associated to the response code returned by the query, such as NOERROR, FORMERR, SERVFAIL, NXDOMAIN, etc.

Monitoring an endpoint using SSH

You can monitor endpoints using SSH by prefixing endpoints[].url with ssh:\\:

endpoints:
  - name: ssh-example
    url: "ssh://example.com:22" # port is optional. Default is 22.
    ssh:
      username: "username"
      password: "password"
    body: |
      {
        "command": "uptime"
      }
    interval: 1m
    conditions:
      - "[CONNECTED] == true"
      - "[STATUS] == 0"

The following placeholders are supported for endpoints of type SSH:

[CONNECTED] resolves to true if the SSH connection was successful, false otherwise
[STATUS] resolves the exit code of the command executed on the remote server (e.g. 0 for success)

Monitoring an endpoint using STARTTLS

If you have an email server that you want to ensure there are no problems with, monitoring it through STARTTLS will serve as a good initial indicator:

endpoints:
  - name: starttls-smtp-example
    url: "starttls://smtp.gmail.com:587"
    interval: 30m
    client:
      timeout: 5s
    conditions:
      - "[CONNECTED] == true"
      - "[CERTIFICATE_EXPIRATION] > 48h"

Monitoring an endpoint using TLS

Monitoring endpoints using SSL/TLS encryption, such as LDAP over TLS, can help detect certificate expiration:

endpoints:
  - name: tls-ldaps-example
    url: "tls://ldap.example.com:636"
    interval: 30m
    client:
      timeout: 5s
    conditions:
      - "[CONNECTED] == true"
      - "[CERTIFICATE_EXPIRATION] > 48h"

Monitoring domain expiration

You can monitor the expiration of a domain with all endpoint types except for DNS by using the [DOMAIN_EXPIRATION] placeholder:

endpoints:
  - name: check-domain-and-certificate-expiration
    url: "https://example.org"
    interval: 1h
    conditions:
      - "[DOMAIN_EXPIRATION] > 720h"
      - "[CERTIFICATE_EXPIRATION] > 240h"

⚠ The usage of the [DOMAIN_EXPIRATION] placeholder requires Gatus to send a request to the official IANA WHOIS service through a library and in some cases, a secondary request to a TLD-specific WHOIS server (e.g. whois.nic.sh). To prevent the WHOIS service from throttling your IP address if you send too many requests, Gatus will prevent you from using the [DOMAIN_EXPIRATION] placeholder on an endpoint with an interval of less than 5m.

disable-monitoring-lock

Setting disable-monitoring-lock to true means that multiple endpoints could be monitored at the same time.

While this behavior wouldn't generally be harmful, conditions using the [RESPONSE_TIME] placeholder could be impacted by the evaluation of multiple endpoints at the same time, therefore, the default value for this parameter is false.

There are three main reasons why you might want to disable the monitoring lock:

You're using Gatus for load testing (each endpoint are periodically evaluated on a different goroutine, so technically, if you create 100 endpoints with a 1 seconds interval, Gatus will send 100 requests per second)
You have a lot of endpoints to monitor
You want to test multiple endpoints at very short intervals (< 5s)

Reloading configuration on the fly

For the sake of convenience, Gatus automatically reloads the configuration on the fly if the loaded configuration file is updated while Gatus is running.

By default, the application will exit if the updating configuration is invalid, but you can configure Gatus to continue running if the configuration file is updated with an invalid configuration by setting skip-invalid-config-update to true.

Keep in mind that it is in your best interest to ensure the validity of the configuration file after each update you apply to the configuration file while Gatus is running by looking at the log and making sure that you do not see the following message:

The configuration file was updated, but it is not valid. The old configuration will continue being used.

Failure to do so may result in Gatus being unable to start if the application is restarted for whatever reason.

I recommend not setting skip-invalid-config-update to true to avoid a situation like this, but the choice is yours to make.

If you are not using a file storage, updating the configuration while Gatus is running is effectively the same as restarting the application.

📝 Updates may not be detected if the config file is bound instead of the config folder. See #151.

Endpoint groups

Endpoint groups are used for grouping multiple endpoints together on the dashboard.

endpoints:
  - name: frontend
    group: core
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

  - name: backend
    group: core
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

  - name: monitoring
    group: internal
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

  - name: nas
    group: internal
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

  - name: random endpoint that is not part of a group
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

The configuration above will result in a dashboard that looks like this:

Exposing Gatus on a custom path

Currently, you can expose the Gatus UI using a fully qualified domain name (FQDN) such as status.example.org. However, it does not support path-based routing, which means you cannot expose it through a URL like example.org/status/.

For more information, see https://github.com/TwiN/gatus/issues/88.

Exposing Gatus on a custom port

By default, Gatus is exposed on port 8080, but you may specify a different port by setting the web.port parameter:

web:
  port: 8081

If you're using a PaaS like Heroku that doesn't let you set a custom port and exposes it through an environment variable instead, you can use that environment variable directly in the configuration file:

web:
  port: ${PORT}

Configuring a startup delay

If, for any reason, you need Gatus to wait for a given amount of time before monitoring the endpoints on application start, you can use the GATUS_DELAY_START_SECONDS environment variable to make Gatus sleep on startup.

Keeping your configuration small

While not specific to Gatus, you can leverage YAML anchors to create a default configuration. If you have a large configuration file, this should help you keep things clean.

Example

default-endpoint: &defaults
  group: core
  interval: 5m
  client:
    insecure: true
    timeout: 30s
  conditions:
    - "[STATUS] == 200"

endpoints:
  - name: anchor-example-1
    <<: *defaults               # This will merge the configuration under &defaults with this endpoint
    url: "https://example.org"

  - name: anchor-example-2
    <<: *defaults
    group: example              # This will override the group defined in &defaults
    url: "https://example.com"

  - name: anchor-example-3
    <<: *defaults
    url: "https://twin.sh/health"
    conditions:                # This will override the conditions defined in &defaults
      - "[STATUS] == 200"
      - "[BODY].status == UP"

Proxy client configuration

You can configure a proxy for the client to use by setting the proxy-url parameter in the client configuration.

endpoints:
  - name: website
    url: "https://twin.sh/health"
    client:
      proxy-url: http://proxy.example.com:8080
    conditions:
      - "[STATUS] == 200"

How to fix 431 Request Header Fields Too Large error

Depending on where your environment is deployed and what kind of middleware or reverse proxy sits in front of Gatus, you may run into this issue. This could be because the request headers are too large, e.g. big cookies.

By default, web.read-buffer-size is set to 8192, but increasing this value like so will increase the read buffer size:

web:
  read-buffer-size: 32768

Badges

Uptime

Gatus can automatically generate an SVG badge for one of your monitored endpoints. This allows you to put badges in your individual applications' README or even create your own status page if you desire.

The path to generate a badge is the following:

/api/v1/endpoints/{key}/uptimes/{duration}/badge.svg

Where:

{duration} is 30d (alpha), 7d, 24h or 1h
{key} has the pattern <GROUP_NAME>_<ENDPOINT_NAME> in which both variables have , /, _, , and . replaced by -.

For instance, if you want the uptime during the last 24 hours from the endpoint frontend in the group core, the URL would look like this:

https://example.com/api/v1/endpoints/core_frontend/uptimes/7d/badge.svg

If you want to display an endpoint that is not part of a group, you must leave the group value empty:

https://example.com/api/v1/endpoints/_frontend/uptimes/7d/badge.svg

Example:

![Uptime 24h](https://status.twin.sh/api/v1/endpoints/core_blog-external/uptimes/24h/badge.svg)

If you'd like to see a visual example of each badge available, you can simply navigate to the endpoint's detail page.

Health

The path to generate a badge is the following:

/api/v1/endpoints/{key}/health/badge.svg

Where:

{key} has the pattern <GROUP_NAME>_<ENDPOINT_NAME> in which both variables have , /, _, , and . replaced by -.

For instance, if you want the current status of the endpoint frontend in the group core, the URL would look like this:

https://example.com/api/v1/endpoints/core_frontend/health/badge.svg

Health (Shields.io)

The path to generate a badge is the following:

/api/v1/endpoints/{key}/health/badge.shields

Where:

{key} has the pattern <GROUP_NAME>_<ENDPOINT_NAME> in which both variables have , /, _, , and . replaced by -.

For instance, if you want the current status of the endpoint frontend in the group core, the URL would look like this:

https://example.com/api/v1/endpoints/core_frontend/health/badge.shields

See more information about the Shields.io badge endpoint here.

Response time

The endpoint to generate a badge is the following:

/api/v1/endpoints/{key}/response-times/{duration}/badge.svg

Where:

{duration} is 30d (alpha), 7d, 24h or 1h
{key} has the pattern <GROUP_NAME>_<ENDPOINT_NAME> in which both variables have , /, _, , and . replaced by -.

How to change the color thresholds of the response time badge

To change the response time badges' threshold, a corresponding configuration can be added to an endpoint. The values in the array correspond to the levels [Awesome, Great, Good, Passable, Bad] All five values must be given in milliseconds (ms).

endpoints:
- name: nas
  group: internal
  url: "https://example.org/"
  interval: 5m
  conditions:
    - "[STATUS] == 200"
  ui:
    badge:
      response-time:
        thresholds: [550, 850, 1350, 1650, 1750]

API

Gatus provides a simple read-only API that can be queried in order to programmatically determine endpoint status and history.

All endpoints are available via a GET request to the following endpoint:

/api/v1/endpoints/statuses

Example: https://status.twin.sh/api/v1/endpoints/statuses

Specific endpoints can also be queried by using the following pattern:

/api/v1/endpoints/{group}_{endpoint}/statuses

Example: https://status.twin.sh/api/v1/endpoints/core_blog-home/statuses

Gzip compression will be used if the Accept-Encoding HTTP header contains gzip.

The API will return a JSON payload with the Content-Type response header set to application/json. No such header is required to query the API.

Installing as binary

You can download Gatus as a binary using the following command:

go install github.com/TwiN/gatus/v5@latest

High level design overview

aceberg/WatchYourLAN

2024-12-01T01:53:48Z

Lightweight network IP scanner. Can be used to notify about new hosts and monitor host online/offline history

WatchYourLAN

Lightweight network IP scanner with web GUI. Features:

Send notification when new host is found
Monitor hosts online/offline history
Keep a list of all hosts in the network
Send data to InfluxDB2 to make a Grafana dashboard

More screenshots

Expand

Quick start

Expand

Replace $YOURTIMEZONE with correct time zone and $YOURIFACE with network interface you want to scan. Network mode must be host. Set $DOCKERDATAPATH for container to save data:

docker run --name wyl \
	-e "IFACES=$YOURIFACE" \
	-e "TZ=$YOURTIMEZONE" \
	--network="host" \
	-v $DOCKERDATAPATH/wyl:/data/WatchYourLAN \
    aceberg/watchyourlan

Web GUI should be at http://localhost:8840

Auth

Expand

WatchYourLAN does not have built-in auth option. But you can use it with SSO tools like Authelia, or my simple auth app ForAuth.
Here is an example docker-compose-auth.yml.

⚠ WARNING!
Please, don't forget that WYL needs host network mode to work. So, WYL port will be exposed in this setup. You need to limit access to it with firewall or other measures.

Install on Linux

Expand

All binary packages can be found in latest release. There are .deb, .rpm, .apk (Alpine Linux) and .tar.gz files.

Supported architectures: amd64, i386, arm_v5, arm_v6, arm_v7, arm64.
Dependencies: arp-scan, tzdata.

For amd64 there is a deb repo available

Config

Expand

Configuration can be done through config file, GUI or environment variables. Variable names is config_v2.yaml file are the same, but in lowcase.

Basic config

Variable	Description	Default
TZ	Set your timezone for correct time
HOST	Listen address	0.0.0.0
PORT	Port for web GUI	8840
THEME	Any theme name from https://bootswatch.com in lowcase or additional	sand
COLOR	Background color: light or dark	dark
NODEPATH	Path to local node modules
SHOUTRRR_URL	WatchYourLAN uses Shoutrrr to send notifications. It is already integrated, just needs a correct URL. Examples for Discord, Email, Gotify, Matrix, Ntfy, Pushover, Slack, Telegram, Generic Webhook and etc are here

Scan settings

Variable	Description	Default
IFACES	Interfaces to scan. Could be one or more, separated by space. See docs/VLAN_ARP_SCAN.md.
TIMEOUT	Time between scans (seconds)	120
ARP_ARGS	Arguments for `arp-scan`. Enable `debug` log level to see resulting command. (Example: `-r 1`). See docs/VLAN_ARP_SCAN.md.
ARP_STRS ARP_STRS_JOINED	See docs/VLAN_ARP_SCAN.md.
LOG_LEVEL	Log level: `debug`, `info`, `warn` or `error`	info
TRIM_HIST	Remove history after (hours)	48
HIST_IN_DB	Store History in DB - if `false`, the History will be stored only in memory and will be lost on app restart. Though, it will keep the app DB smaller (and InfluxDB is recommended for long term History storage)	false
USE_DB	Either `sqlite` or `postgres`	sqlite
PG_CONNECT	Address to connect to PostgreSQL. (Example: `postgres://username:password@192.168.0.1:5432/dbname?sslmode=disable`). Full list of URL parameters here

InfluxDB2 config

This config matches Grafana's config for InfluxDB data source

Variable	Description	Default	Example
INFLUX_ENABLE	Enable export to InfluxDB2	false	true
INFLUX_SKIP_TLS	Skip TLS Verify	false	true
INFLUX_ADDR	Address:port of InfluxDB2 server		https://192.168.2.3:8086/
INFLUX_BUCKET	InfluxDB2 bucket		test
INFLUX_ORG	InfluxDB2 org		home
INFLUX_TOKEN	Secret token, generated by InfluxDB2

Config file

Expand

Config file name is config_v2.yaml. Example:

arp_args: ""
color: dark
hist_in_db: false
host: 0.0.0.0
ifaces: enp4s0
influx_addr: ""
influx_bucket: ""
influx_enable: false
influx_org: ""
influx_skip_tls: false
influx_token: ""
log_level: info
nodepath: ""
pg_connect: ""
port: "8840"
shoutrrr_url: "gotify://192.168.0.1:8083/AwQqpAae.rrl5Ob/?title=Unknown host detected&DisableTLS=yes"
theme: sand
timeout: 60
trim_hist: 48
use_db: sqlite

Options

Expand

Key	Description	Default
-d	Path to config dir	/data/WatchYourLAN
-n	Path to node modules (see below)

Local network only

Expand

By default, this app pulls themes, icons and fonts from the internet. But, in some cases, it may be useful to have an independent from global network setup. I created a separate image with all necessary modules and fonts. Run with Docker:

docker run --name node-bootstrap          \
    -p 8850:8850                          \
    aceberg/node-bootstrap

docker run --name wyl \
	-e "IFACES=$YOURIFACE" \
	-e "TZ=$YOURTIMEZONE" \
	--network="host" \
	-v $DOCKERDATAPATH/wyl:/data/WatchYourLAN \
    aceberg/watchyourlan -n "http://$YOUR_IP:8850"

Or use docker-compose

API & Integrations

Expand

API

Moved to docs/API.md

Integrations

Python API client by drwahl

Thanks

Expand

All go packages listed in dependencies
Favicon and logo: Access point icons created by Freepik - Flaticon
Bootstrap
Themes: Free themes for Bootstrap

sirupsen/logrus

2024-12-01T01:42:51Z

Structured, pluggable logging for Go.

Logrus

Logrus is a structured logger for Go (golang), completely API compatible with the standard library logger.

Logrus is in maintenance-mode. We will not be introducing new features. It's simply too hard to do in a way that won't break many people's projects, which is the last thing you want from your Logging library (again...).

This does not mean Logrus is dead. Logrus will continue to be maintained for security, (backwards compatible) bug fixes, and performance (where we are limited by the interface).

I believe Logrus' biggest contribution is to have played a part in today's widespread use of structured logging in Golang. There doesn't seem to be a reason to do a major, breaking iteration into Logrus V2, since the fantastic Go community has built those independently. Many fantastic alternatives have sprung up. Logrus would look like those, had it been re-designed with what we know about structured logging in Go today. Check out, for example, Zerolog, Zap, and Apex.

Seeing weird case-sensitive problems? It's in the past been possible to import Logrus as both upper- and lower-case. Due to the Go package environment, this caused issues in the community and we needed a standard. Some environments experienced problems with the upper-case variant, so the lower-case was decided. Everything using logrus will need to use the lower-case: github.com/sirupsen/logrus. Any package that isn't, should be changed.

To fix Glide, see these comments. For an in-depth explanation of the casing issue, see this comment.

Nicely color-coded in development (when a TTY is attached, otherwise just plain text):

With log.SetFormatter(&log.JSONFormatter{}), for easy parsing by logstash or Splunk:

{"animal":"walrus","level":"info","msg":"A group of walrus emerges from the
ocean","size":10,"time":"2014-03-10 19:57:38.562264131 -0400 EDT"}

{"level":"warning","msg":"The group's number increased tremendously!",
"number":122,"omg":true,"time":"2014-03-10 19:57:38.562471297 -0400 EDT"}

{"animal":"walrus","level":"info","msg":"A giant walrus appears!",
"size":10,"time":"2014-03-10 19:57:38.562500591 -0400 EDT"}

{"animal":"walrus","level":"info","msg":"Tremendously sized cow enters the ocean.",
"size":9,"time":"2014-03-10 19:57:38.562527896 -0400 EDT"}

{"level":"fatal","msg":"The ice breaks!","number":100,"omg":true,
"time":"2014-03-10 19:57:38.562543128 -0400 EDT"}

With the default log.SetFormatter(&log.TextFormatter{}) when a TTY is not attached, the output is compatible with the logfmt format:

time="2015-03-26T01:27:38-04:00" level=debug msg="Started observing beach" animal=walrus number=8
time="2015-03-26T01:27:38-04:00" level=info msg="A group of walrus emerges from the ocean" animal=walrus size=10
time="2015-03-26T01:27:38-04:00" level=warning msg="The group's number increased tremendously!" number=122 omg=true
time="2015-03-26T01:27:38-04:00" level=debug msg="Temperature changes" temperature=-4
time="2015-03-26T01:27:38-04:00" level=panic msg="It's over 9000!" animal=orca size=9009
time="2015-03-26T01:27:38-04:00" level=fatal msg="The ice breaks!" err=&{0x2082280c0 map[animal:orca size:9009] 2015-03-26 01:27:38.441574009 -0400 EDT panic It's over 9000!} number=100 omg=true

To ensure this behaviour even if a TTY is attached, set your formatter as follows:

	log.SetFormatter(&log.TextFormatter{
		DisableColors: true,
		FullTimestamp: true,
	})

Logging Method Name

If you wish to add the calling method as a field, instruct the logger via:

log.SetReportCaller(true)

This adds the caller as 'method' like so:

{"animal":"penguin","level":"fatal","method":"github.com/sirupsen/arcticcreatures.migrate","msg":"a penguin swims by",
"time":"2014-03-10 19:57:38.562543129 -0400 EDT"}

time="2015-03-26T01:27:38-04:00" level=fatal method=github.com/sirupsen/arcticcreatures.migrate msg="a penguin swims by" animal=penguin

Note that this does add measurable overhead - the cost will depend on the version of Go, but is between 20 and 40% in recent tests with 1.6 and 1.7. You can validate this in your environment via benchmarks:

go test -bench=.*CallerTracing

Case-sensitivity

The organization's name was changed to lower-case--and this will not be changed back. If you are getting import conflicts due to case sensitivity, please use the lower-case import: github.com/sirupsen/logrus.

Example

The simplest way to use Logrus is simply the package-level exported logger:

package main

import (
  log "github.com/sirupsen/logrus"
)

func main() {
  log.WithFields(log.Fields{
    "animal": "walrus",
  }).Info("A walrus appears")
}

Note that it's completely api-compatible with the stdlib logger, so you can replace your log imports everywhere with log "github.com/sirupsen/logrus" and you'll now have the flexibility of Logrus. You can customize it all you want:

package main

import (
  "os"
  log "github.com/sirupsen/logrus"
)

func init() {
  // Log as JSON instead of the default ASCII formatter.
  log.SetFormatter(&log.JSONFormatter{})

  // Output to stdout instead of the default stderr
  // Can be any io.Writer, see below for File example
  log.SetOutput(os.Stdout)

  // Only log the warning severity or above.
  log.SetLevel(log.WarnLevel)
}

func main() {
  log.WithFields(log.Fields{
    "animal": "walrus",
    "size":   10,
  }).Info("A group of walrus emerges from the ocean")

  log.WithFields(log.Fields{
    "omg":    true,
    "number": 122,
  }).Warn("The group's number increased tremendously!")

  log.WithFields(log.Fields{
    "omg":    true,
    "number": 100,
  }).Fatal("The ice breaks!")

  // A common pattern is to re-use fields between logging statements by re-using
  // the logrus.Entry returned from WithFields()
  contextLogger := log.WithFields(log.Fields{
    "common": "this is a common field",
    "other": "I also should be logged always",
  })

  contextLogger.Info("I'll be logged with common and other field")
  contextLogger.Info("Me too")
}

For more advanced usage such as logging to multiple locations from the same application, you can also create an instance of the logrus Logger:

package main

import (
  "os"
  "github.com/sirupsen/logrus"
)

// Create a new instance of the logger. You can have any number of instances.
var log = logrus.New()

func main() {
  // The API for setting attributes is a little different than the package level
  // exported logger. See Godoc.
  log.Out = os.Stdout

  // You could set this to any `io.Writer` such as a file
  // file, err := os.OpenFile("logrus.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
  // if err == nil {
  //  log.Out = file
  // } else {
  //  log.Info("Failed to log to file, using default stderr")
  // }

  log.WithFields(logrus.Fields{
    "animal": "walrus",
    "size":   10,
  }).Info("A group of walrus emerges from the ocean")
}

Fields

Logrus encourages careful, structured logging through logging fields instead of long, unparseable error messages. For example, instead of: log.Fatalf("Failed to send event %s to topic %s with key %d"), you should log the much more discoverable:

log.WithFields(log.Fields{
  "event": event,
  "topic": topic,
  "key": key,
}).Fatal("Failed to send event")

We've found this API forces you to think about logging in a way that produces much more useful logging messages. We've been in countless situations where just a single added field to a log statement that was already there would've saved us hours. The WithFields call is optional.

In general, with Logrus using any of the printf-family functions should be seen as a hint you should add a field, however, you can still use the printf-family functions with Logrus.

Default Fields

Often it's helpful to have fields always attached to log statements in an application or parts of one. For example, you may want to always log the request_id and user_ip in the context of a request. Instead of writing log.WithFields(log.Fields{"request_id": request_id, "user_ip": user_ip}) on every line, you can create a logrus.Entry to pass around instead:

requestLogger := log.WithFields(log.Fields{"request_id": request_id, "user_ip": user_ip})
requestLogger.Info("something happened on that request") # will log request_id and user_ip
requestLogger.Warn("something not great happened")

Hooks

You can add hooks for logging levels. For example to send errors to an exception tracking service on Error, Fatal and Panic, info to StatsD or log to multiple places simultaneously, e.g. syslog.

Logrus comes with built-in hooks. Add those, or your custom hook, in init:

import (
  log "github.com/sirupsen/logrus"
  "gopkg.in/gemnasium/logrus-airbrake-hook.v2" // the package is named "airbrake"
  logrus_syslog "github.com/sirupsen/logrus/hooks/syslog"
  "log/syslog"
)

func init() {

  // Use the Airbrake hook to report errors that have Error severity or above to
  // an exception tracker. You can create custom hooks, see the Hooks section.
  log.AddHook(airbrake.NewHook(123, "xyz", "production"))

  hook, err := logrus_syslog.NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, "")
  if err != nil {
    log.Error("Unable to connect to local syslog daemon")
  } else {
    log.AddHook(hook)
  }
}

Note: Syslog hook also support connecting to local syslog (Ex. "/dev/log" or "/var/run/syslog" or "/var/run/log"). For the detail, please check the syslog hook README.

A list of currently known service hooks can be found in this wiki page

Level logging

Logrus has seven logging levels: Trace, Debug, Info, Warning, Error, Fatal and Panic.

log.Trace("Something very low level.")
log.Debug("Useful debugging information.")
log.Info("Something noteworthy happened!")
log.Warn("You should probably take a look at this.")
log.Error("Something failed but I'm not quitting.")
// Calls os.Exit(1) after logging
log.Fatal("Bye.")
// Calls panic() after logging
log.Panic("I'm bailing.")

You can set the logging level on a Logger, then it will only log entries with that severity or anything above it:

// Will log anything that is info or above (warn, error, fatal, panic). Default.
log.SetLevel(log.InfoLevel)

It may be useful to set log.Level = logrus.DebugLevel in a debug or verbose environment if your application has that.

Note: If you want different log levels for global (log.SetLevel(...)) and syslog logging, please check the syslog hook README.

Entries

Besides the fields added with WithField or WithFields some fields are automatically added to all logging events:

time. The timestamp when the entry was created.
msg. The logging message passed to {Info,Warn,Error,Fatal,Panic} after the AddFields call. E.g. Failed to send event.
level. The logging level. E.g. info.

Environments

Logrus has no notion of environment.

If you wish for hooks and formatters to only be used in specific environments, you should handle that yourself. For example, if your application has a global variable Environment, which is a string representation of the environment you could do:

import (
  log "github.com/sirupsen/logrus"
)

func init() {
  // do something here to set environment depending on an environment variable
  // or command-line flag
  if Environment == "production" {
    log.SetFormatter(&log.JSONFormatter{})
  } else {
    // The TextFormatter is default, you don't actually have to do this.
    log.SetFormatter(&log.TextFormatter{})
  }
}

This configuration is how logrus was intended to be used, but JSON in production is mostly only useful if you do log aggregation with tools like Splunk or Logstash.

Formatters

The built-in logging formatters are:

logrus.TextFormatter. Logs the event in colors if stdout is a tty, otherwise without colors.
- Note: to force colored output when there is no TTY, set the ForceColors field to true. To force no colored output even if there is a TTY set the DisableColors field to true. For Windows, see github.com/mattn/go-colorable.
- When colors are enabled, levels are truncated to 4 characters by default. To disable truncation set the DisableLevelTruncation field to true.
- When outputting to a TTY, it's often helpful to visually scan down a column where all the levels are the same width. Setting the PadLevelText field to true enables this behavior, by adding padding to the level text.
- All options are listed in the generated docs.
logrus.JSONFormatter. Logs fields as JSON.
- All options are listed in the generated docs.

Third party logging formatters:

FluentdFormatter. Formats entries that can be parsed by Kubernetes and Google Container Engine.
GELF. Formats entries so they comply to Graylog's GELF 1.1 specification.
logstash. Logs fields as Logstash Events.
prefixed. Displays log entry source along with alternative layout.
zalgo. Invoking the Power of Zalgo.
nested-logrus-formatter. Converts logrus fields to a nested structure.
powerful-logrus-formatter. get fileName, log's line number and the latest function's name when print log; Save log to files.
caption-json-formatter. logrus's message json formatter with human-readable caption added.

You can define your formatter by implementing the Formatter interface, requiring a Format method. Format takes an *Entry. entry.Data is a Fields type (map[string]interface{}) with all your fields as well as the default ones (see Entries section above):

type MyJSONFormatter struct {
}

log.SetFormatter(new(MyJSONFormatter))

func (f *MyJSONFormatter) Format(entry *Entry) ([]byte, error) {
  // Note this doesn't include Time, Level and Message which are available on
  // the Entry. Consult `godoc` on information about those fields or read the
  // source of the official loggers.
  serialized, err := json.Marshal(entry.Data)
    if err != nil {
      return nil, fmt.Errorf("Failed to marshal fields to JSON, %w", err)
    }
  return append(serialized, '\n'), nil
}

Logger as an `io.Writer`

Logrus can be transformed into an io.Writer. That writer is the end of an io.Pipe and it is your responsibility to close it.

w := logger.Writer()
defer w.Close()

srv := http.Server{
    // create a stdlib log.Logger that writes to
    // logrus.Logger.
    ErrorLog: log.New(w, "", 0),
}

Each line written to that writer will be printed the usual way, using formatters and hooks. The level for those entries is info.

This means that we can override the standard library logger easily:

logger := logrus.New()
logger.Formatter = &logrus.JSONFormatter{}

// Use logrus for standard log output
// Note that `log` here references stdlib's log
// Not logrus imported under the name `log`.
log.SetOutput(logger.Writer())

Rotation

Log rotation is not provided with Logrus. Log rotation should be done by an external program (like logrotate(8)) that can compress and delete old log entries. It should not be a feature of the application-level logger.

Tools

Tool	Description
Logrus Mate	Logrus mate is a tool for Logrus to manage loggers, you can initial logger's level, hook and formatter by config file, the logger will be generated with different configs in different environments.
Logrus Viper Helper	An Helper around Logrus to wrap with spf13/Viper to load configuration with fangs! And to simplify Logrus configuration use some behavior of Logrus Mate. sample

Testing

Logrus has a built in facility for asserting the presence of log messages. This is implemented through the test hook and provides:

decorators for existing logger (test.NewLocal and test.NewGlobal) which basically just adds the test hook
a test logger (test.NewNullLogger) that just records log messages (and does not output any):

import(
  "github.com/sirupsen/logrus"
  "github.com/sirupsen/logrus/hooks/test"
  "github.com/stretchr/testify/assert"
  "testing"
)

func TestSomething(t*testing.T){
  logger, hook := test.NewNullLogger()
  logger.Error("Helloerror")

  assert.Equal(t, 1, len(hook.Entries))
  assert.Equal(t, logrus.ErrorLevel, hook.LastEntry().Level)
  assert.Equal(t, "Helloerror", hook.LastEntry().Message)

  hook.Reset()
  assert.Nil(t, hook.LastEntry())
}

Fatal handlers

Logrus can register one or more functions that will be called when any fatal level message is logged. The registered handlers will be executed before logrus performs an os.Exit(1). This behavior may be helpful if callers need to gracefully shutdown. Unlike a panic("Something went wrong...") call which can be intercepted with a deferred recover a call to os.Exit(1) can not be intercepted.

...
handler := func() {
  // gracefully shutdown something...
}
logrus.RegisterExitHandler(handler)
...

Thread safety

By default, Logger is protected by a mutex for concurrent writes. The mutex is held when calling hooks and writing logs. If you are sure such locking is not needed, you can call logger.SetNoLock() to disable the locking.

Situation when locking is not needed includes:

You have no hooks registered, or hooks calling is already thread-safe.
Writing to logger.Out is already thread-safe, for example:
1. logger.Out is protected by locks.
2. logger.Out is an os.File handler opened with O_APPEND flag, and every write is smaller than 4k. (This allows multi-thread/multi-process writing)
  
  (Refer to http://www.notthewizard.com/2014/06/17/are-files-appends-really-atomic/)

VictoriaMetrics/VictoriaMetrics

2024-12-01T01:42:51Z

VictoriaMetrics: fast, cost-effective monitoring solution and time series database

VictoriaMetrics

VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and managing time series data. It delivers high performance and reliability, making it an ideal choice for businesses of all sizes.

Here are some resources and information about VictoriaMetrics:

Documentation: docs.victoriametrics.com
Case studies: Grammarly, Roblox, Wix,....
Available: Binary releases, Docker images, Source code
Deployment types: Single-node version, Cluster version, and Enterprise version
Changelog: CHANGELOG, and How to upgrade
Community: Slack, X (Twitter), LinkedIn, YouTube

Yes, we open-source both the single-node VictoriaMetrics and the cluster version.

Prominent features

VictoriaMetrics is optimized for timeseries data, even when old time series are constantly replaced by new ones at a high rate, it offers a lot of features:

Long-term storage for Prometheus or as a drop-in replacement for Prometheus and Graphite in Grafana.
Powerful stream aggregation: Can be used as a StatsD alternative.
Ideal for big data: Works well with large amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various Enterprise workloads.
Query language: Supports both PromQL and the more performant MetricsQL.
Easy to setup: No dependencies, single small binary, configuration through command-line flags, but the default is also fine-tuned; backup and restore with instant snapshots.
Global query view: Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics and queried via a single query.
Various Protocols: Support metric scraping, ingestion and backfilling in various protocol.
- Prometheus exporters, Prometheus remote write API, Prometheus exposition format.
- InfluxDB line protocol over HTTP, TCP and UDP.
- Graphite plaintext protocol with tags.
- OpenTSDB put message.
- HTTP OpenTSDB /api/put requests.
- JSON line format.
- Arbitrary CSV data.
- Native binary format.
- DataDog agent or DogStatsD.
- NewRelic infrastructure agent.
- OpenTelemetry metrics format.
NFS-based storages: Supports storing data on NFS-based storages such as Amazon EFS, Google Filestore.
And many other features such as metrics relabeling, cardinality limiter, etc.

Enterprise version

In addition, the Enterprise version includes extra features:

Anomaly detection: Automation and simplification of your alerting rules, covering complex anomalies found in metrics data.
Backup automation: Automates regular backup procedures.
Multiple retentions: Reducing storage costs by specifying different retentions for different datasets.
Downsampling: Reducing storage costs and increasing performance for queries over historical data.
Stable releases with long-term support lines (LTS).
Comprehensive support: First-class consulting, feature requests and technical support provided by the core VictoriaMetrics dev team.
Many other features, which you can read about on the Enterprise page.

Contact us if you need enterprise support for VictoriaMetrics. Or you can request a free trial license here, downloaded Enterprise binaries are available at Github Releases.

We strictly apply security measures in everything we do. VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. See Security page for more details.

Benchmarks

Some good benchmarks VictoriaMetrics achieved:

Minimal memory footprint: handling millions of unique timeseries with 10x less RAM than InfluxDB, up to 7x less RAM than Prometheus, Thanos or Cortex.
Highly scalable and performance for data ingestion and querying, 20x outperforms InfluxDB and TimescaleDB.
High data compression: 70x more data points may be stored into limited storage than TimescaleDB, 7x less storage space is required than Prometheus, Thanos or Cortex.
Reducing storage costs: 10x more effective than Graphite according to the Grammarly case study.
A single-node VictoriaMetrics can replace medium-sized clusters built with competing solutions such as Thanos, M3DB, Cortex, InfluxDB or TimescaleDB. See VictoriaMetrics vs Thanos, Measuring vertical scalability, Remote write storage wars - PromCon 2019.
Optimized for storage: Works well with high-latency IO and low IOPS (HDD and network storage in AWS, Google Cloud, Microsoft Azure, etc.).

Community and contributions

Feel free asking any questions regarding VictoriaMetrics:

If you like VictoriaMetrics and want to contribute, then please read these docs.

VictoriaMetrics Logo

The provided ZIP file contains three folders with different logo orientations. Each folder includes the following file types:

JPEG: Preview files
PNG: Preview files with transparent background
AI: Adobe Illustrator files

VictoriaMetrics Logo Usage Guidelines

Font

Font Used: Lato Black
Download here: Lato Font

Color Palette

Black #000000
Purple #4d0e82
Orange #ff2e00
White #ffffff

Logo Usage Rules

Only use the Lato Black font as specified.
Maintain sufficient clear space around the logo for visibility.
Do not modify the spacing, alignment, or positioning of design elements.
You may resize the logo as needed, but ensure all proportions remain intact.

Thank you for your cooperation!

semaphoreui/semaphore

2024-12-01T01:42:51Z

Modern UI and powerful API for Ansible, Terraform, OpenTofu, PowerShell and other DevOps tools.

Semaphore UI

Modern UI for Ansible, Terraform, OpenTofu, PowerShell and other DevOps tools.

If your project has grown and deploying from the terminal is no longer feasible, then Semaphore UI is the tool you need.

Live Demo

Try the latest version of Semaphore at https://cloud.semaphoreui.com.

What is Semaphore UI?

Semaphore UI is a modern web interface for managing popular DevOps tools.

Semaphore UI allows you to:

Easily run Ansible playbooks, Terraform and OpenTofu code, as well as Bash and PowerShell scripts.
Receive notifications about failed tasks.
Control access to your deployment system.

Key Concepts

Projects is a collection of related resources, configurations, and tasks. Each project allows you to organize and manage your automation efforts in one place, defining the scope of tasks such as deploying applications, running scripts, or orchestrating cloud resources. Projects help group resources, inventories, task templates, and environments for streamlined automation workflows.
Task Templates are reusable definitions of tasks that can be executed on demand or scheduled. A template specifies what actions should be performed, such as running Ansible playbooks, Terraform configurations, or other automation tasks. By using templates, you can standardize tasks and easily re-execute them with minimal effort, ensuring consistent results across different environments.
Task is a specific instance of a job or operation executed by Semaphore. It refers to running a predefined action (like an Ansible playbook or a script) using a task template. Tasks can be initiated manually or automatically through schedules and are tracked to give you detailed feedback on the execution, including success, failure, and logs.
Schedules allow you to automate task execution at specified times or intervals. This feature is useful for running periodic maintenance tasks, backups, or deployments without manual intervention. You can configure recurring schedules to ensure important automation tasks are performed regularly and on time.
Inventory is a collection of target hosts (servers, virtual machines, containers, etc.) on which tasks will be executed. The inventory includes details about the managed nodes such as IP addresses, SSH credentials, and grouping information. It allows for dynamic control over which environments and hosts your automation will interact with.
Environment refers to a configuration context that holds sensitive information such as environment variables and secrets used by tasks during execution. It separates sensitive data from task templates and allows you to switch between different setups while running the same task template across different environments securely.

Getting Started

You can install Semaphore using the following methods:

Docker
SaaS (Semaphore Cloud)
Deploy a VM from a marketplace (AWS, DigitalOcean, etc.)
Snap
Binary file
Debian or RPM package

Docker

The most popular way to install Semaphore is via Docker.

docker run -p 3000:3000 --name semaphore \
	-e SEMAPHORE_DB_DIALECT=bolt \
	-e SEMAPHORE_ADMIN=admin \
	-e SEMAPHORE_ADMIN_PASSWORD=changeme \
	-e SEMAPHORE_ADMIN_NAME=Admin \
	-e SEMAPHORE_ADMIN_EMAIL=admin@localhost \
	-d semaphoreui/semaphore:latest

We recommend using the Container Configurator to get the ideal Docker configuration for Semaphore.

SaaS

We offer a SaaS solution for using Semaphore UI without installation. Check it out at Semaphore Cloud.

Deploy VM from Marketplace

Supported cloud providers:

Semaphore Run
AWS
Yandex Cloud
DigitalOcean (coming soon)

Other Installation Methods

For more installation options, visit our Installation page.

Documentation

Contribution

Contribution Guide
Dev Container (default user admin / changeme)

License

MIT © Denis Gukov

palantir/gradle-git-version

2024-12-01T01:54:02Z

a Gradle plugin that uses `git describe` to produce a version string.

webcrumbs-community/webcrumbs

2024-12-01T01:32:35Z

Build, re(use) and share your own JavaScript plugins that effortlessly match your website's style. 🌟 Star to support our work!

Looking for Hacktoberfest? Check out the rules here! Hacktoberfest at Webcrumbs
🌟 Star the repository and sign up at webcrumbs.org to know about our launches!

WebCrumbs aspires to be an industry-standard solution for modern web development, creating the first open ecosystem of plugins for the JavaScript community and related frameworks (like React, Nextjs, Vue, Svelte, and others). Whether you're a developer or not, you'll find it easy to create, manage, and extend your own websites with our intuitive admin panel and a rich ecosystem of plugins developed by the community.

If you love what WebCrumbs is doing, consider starring us on GitHub. Your support is key to refining our product and growing our community. Star WebCrumbs on GitHub.

Star the repository: If you haven't yet (yes, you!), give us a star. It's free and means a lot to us. Your support helps increase our visibility and keeps us motivated.

Write about us: If you believe in what WebCrumbs is building, share it with your network and help us grow the community. Write about us on your favorite blog, like DEV.to, Medium, Substack, and more.

Join our discord community: It's where the real-time chats happen. Join us through this link.

Share your opinion: We want to hear your feedback and ideas! Every opinion counts, and we're excited to hear from you. Share your opinion.

Share with friends: We're looking for passionate, creative minds to join our mission. So, share with the kind of brilliant developers we know you are.

"Check out Webcrumbs!
It's open source and lets you build, reuse, and share JavaScript plugins.
Perfect for starting your own collection."

This project operates under the AGPL Open Source License and is a community-driven endeavor.

Credits
Spearheaded and maintained by Webcrumbs Team.

tpn/pdfs

2024-12-01T01:54:15Z

Technically-oriented PDF Collection (Papers, Specs, Decks, Manuals, etc)

EbookFoundation/free-programming-books

2024-12-01T01:54:15Z

📚 Freely available programming books

List of Free Learning Resources In Many Languages

Search the list at https://ebookfoundation.github.io/free-programming-books-search/ .

This page is available as an easy-to-read website. Access it by clicking on .

Intro

This list was originally a clone of StackOverflow - List of Freely Available Programming Books with contributions from Karan Bhangui and George Stocker.

The list was moved to GitHub by Victor Felder for collaborative updating and maintenance. It has grown to become one of GitHub's most popular repositories.

The Free Ebook Foundation now administers the repo, a not-for-profit organization devoted to promoting the creation, distribution, archiving, and sustainability of free ebooks. Donations to the Free Ebook Foundation are tax-deductible in the US.

How To Contribute

Please read CONTRIBUTING. If you're new to GitHub, welcome! Remember to abide by our adapted from Code of Conduct too (translations also available).

Click on these badges to see how you might be able to help:

How To Share

Share on Facebook
Share on LinkedIn
Share on Mastodon/Fediverse
Share on Telegram
Share on 𝕏 (Twitter)

Resources

This project lists books and other resources grouped by genres:

Books

English, By Programming Language

English, By Subject

Other Languages

Cheat Sheets

All Languages

Free Online Courses

Interactive Programming Resources

Problem Sets and Competitive Programming

Problem Sets

Podcast - Screencast

Free Podcasts and Screencasts:

Programming Playgrounds

Write, compile, and run your code within a browser. Try it out!

Translations

Volunteers have translated many of our Contributing, How-to, and Code of Conduct documents into languages covered by our lists.

English
... More languages ...

You might notice that there are some missing translations here - perhaps you would like to help out by contributing a translation?

License

Each file included in this repository is licensed under the CC BY License.

techchipnet/CamPhish

2024-12-01T01:54:15Z

Grab cam shots from target's phone front camera or PC webcam just sending a link.

CamPhish

Grab cam shots from target's phone front camera or PC webcam just sending a link.

What is CamPhish?

CamPhish is techniques to take cam shots of target's phone front camera or PC webcam. CamPhish Hosts a fake website on in built PHP server and uses ngrok & serveo to generate a link which we will forward to the target, which can be used on over internet. website asks for camera permission and if the target allows it, this tool grab camshots of target's device

Features

In this tool I added two automatic webpage templates for engaged target on webpage to get more picture of cam

Festival Wishing
Live YouTube TV
Online Meeting [Beta]

simply enter festival name or youtube's video ID

This Tool Tested On :

Kali Linux
Termux
MacOS
Ubuntu
Parrot Sec OS

Installing and requirements

This tool require PHP for webserver, SSH or serveo link. First run following command on your terminal

apt-get -y install php openssh git wget

Installing (Kali Linux/Termux):

git clone https://github.com/techchipnet/CamPhish
cd CamPhish
bash camphish.sh

Change Log:

Version: 1.7: Fix and add support

fixed: termux failed to get home directory
Add support for Apple sillicon (M1/M2/M3 ARM64)
Add support for arm64 like Raspberry Pi

Version: 1.6: Fix ngrok direct link generate

Version: 1.5: Add new online meeting template

Version: 1.4: Ngrok authtoken update

Version: 1.3: Fix ngrok direct link

For More Video subcribe TechChip YouTube Channel

CamPhish is created to help in penetration testing and it's not responsible for any misuse or illegal purposes.

CamPhish is inspired by https://github.com/thelinuxchoice/ Big thanks to @thelinuxchoice

tpn/pdfs

2024-12-01T01:43:15Z

Technically-oriented PDF Collection (Papers, Specs, Decks, Manuals, etc)

brave/brave-core

2024-12-01T01:43:15Z

Core engine for the Brave browser for mobile and desktop. For issues https://github.com/brave/brave-browser/issues

Brave Core

Brave Core is a set of changes, APIs, and scripts used for customizing Chromium to make the Brave browser. Please also check https://github.com/brave/brave-browser

Follow @brave on Twitter for important announcements.

Resources

Community

You can ask questions and interact with the community in the following locations:

Brave Community
community channel on Brave Software's Slack

react-native-share/react-native-share

2024-11-30T01:31:58Z

Social share, sending simple data to other apps.

react-native-share

React Native Share, is a simple tool for sharing messages and files with other apps.

Getting started 🚀

Expo Managed Workflow

For Expo projects, you will need to be on a managed workflow and use the following command:

npx expo install react-native-share

Configure you app.config.ts or app.json to use the permissions needed by the library:

{
  "plugins": [
    [
      "react-native-share",
      {
        "ios": [
          "fb",
          "instagram",
          "twitter",
          "tiktoksharesdk",
        ],
        "android": [
          "com.facebook.katana",
          "com.instagram.android",
          "com.twitter.android",
          "com.zhiliaoapp.musically",
        ]
      }
    ]
  ]
}

ios parameter will take care of adding queries (LSApplicationQueriesSchemes) to the Info.plist.

<key>LSApplicationQueriesSchemes</key>
<array>
  <string>fb</string>
  <string>instagram</string>
  <string>twitter</string>
  <string>tiktoksharesdk</string>
</array>

android parameter will take care of adding queries to the AndroidManifest.xml.

<queries>
  <package android:name="com.facebook.katana" />
  <package android:name="com.instagram.android" />
  <package android:name="com.twitter.android" />
  <package android:name="com.zhiliaoapp.musically" />
</queries>

And prebuild the project with expo prebuild.

Bare React Native

If you are using react-native >= 0.7X and/or the new arch you just need to do a simple:

yarn add react-native-share

Or if are using npm:

npm i react-native-share --save

After that, we need to install the dependencies to use the project on iOS(you can skip this part if you are using this on Android).

Now run a simple: npx pod-install or cd ios && pod install. After that, you should be able to use the library on both Platforms, iOS and Android.

Usage

Then simply import:

import Share from 'react-native-share';

Share.open(options)
  .then((res) => {
    console.log(res);
  })
  .catch((err) => {
    err && console.log(err);
  });

Which you do something similar to this:

Documentation

If you are using a older version of react-native or react-native-share, having any problem or want to know how use Share.open and other functions, please refer to our new docs and help us improve that.🚀

freeplane/freeplane

2024-11-30T01:31:58Z

Application for Mind Mapping, Knowledge Management, Project Management. Develop, organize and communicate your ideas and knowledge in the most effective way.

Freeplane

Freeplane is a free and open source software application that supports thinking, sharing information, getting things done at work, in school and at home. It provides you a set of tools for mind mapping (also known as concept mapping or information mapping) and navigating the mapped information. Freeplane is also a more robust and superior alternative to Xmind, Mindmeister, and similar mind mapping software.

Freeplane is written in Java using OSGi and Java Swing. It runs on any operating system that has a current version of Java installed. It can be installed or can run from removable storage like a USB drive.

Download and install the latest version over at Sourceforge. If you would like to report a bug, you can go report it over at Issues.

The documentation can be found at . There, you will find How-To Guides, FAQs, Examples and explanations about the functions of Freeplane.

Hop on to our Discussions if you have any questions, ideas, or thoughts you'd like to share. Contributors are very much welcome, of course!

Features Rundown

How to Start Contributing

We're currently looking for contributors for developing the documentation. If you can write simple step-by-step guides, translate existing text into English, transfer text from our old documentation into the new one, then we could use your help. You can start a discussion post saying you want to contribute to the documentation and the Freeplane team will respond and assist you.

If you have other ways of contributing: developing an add-on, sharing your pre-configured mindmap, or suggestions about future development, please feel free to join us in the Discussions

Every contributor or team member freely decides what task they are going to work on. However, for making the best decision regarding development, it's advised that we first propose and suggest the idea to the community through a discussion post as to enable early discussion and community feedback.

Where to Download

Download and install the latest version over at Sourceforge.

valkey-io/valkey-glide

2024-11-30T01:31:58Z

An open source Valkey client library that supports Valkey and Redis open source 6.2, 7.0 and 7.2. Valkey GLIDE is designed for reliability, optimized performance, and high-availability, for Valkey and Redis OSS based applications. GLIDE is a multi language client library, written in Rust with programming language bindings, such as Java and Python

Valkey GLIDE

Valkey General Language Independent Driver for the Enterprise (GLIDE), is an open-source Valkey client library. Valkey GLIDE is one of the official client libraries for Valkey, and it supports all Valkey commands. Valkey GLIDE supports Valkey 7.2 and above, and Redis open-source 6.2, 7.0 and 7.2. Application programmers use Valkey GLIDE to safely and reliably connect their applications to Valkey- and Redis OSS- compatible services. Valkey GLIDE is designed for reliability, optimized performance, and high-availability, for Valkey and Redis OSS based applications. It is sponsored and supported by AWS, and is pre-configured with best practices learned from over a decade of operating Redis OSS-compatible services used by hundreds of thousands of customers. To help ensure consistency in application development and operations, Valkey GLIDE is implemented using a core driver framework, written in Rust, with language specific extensions. This design ensures consistency in features across languages and reduces overall complexity.

Supported Engine Versions

Valkey GLIDE is API-compatible with the following engine versions:

Engine Type	6.2	7.0	7.2	8.0
Valkey	-	-	V	V
Redis	V	V	V	-

Current Status and upcoming releases

In the current release, Valkey GLIDE is available for Python, Java and Node.js. Support for GO is actively under development, with plans to include more programming languages in the future.

v1.2 (Dec. 2024)

Vector Similarity Search and JSON modules support
Availability zone routing for Read from Replica

v1.3 (TBD)

GO support
Observability with Open Telemetry
Client side metrics

Getting Started

Getting Help

If you have any questions, feature requests, encounter issues, or need assistance with this project, please don't hesitate to open a GitHub issue. Our community and contributors are here to help you. Before creating an issue, we recommend checking the existing issues to see if your question or problem has already been addressed. If not, feel free to create a new issue, and we'll do our best to assist you. Please provide as much detail as possible in your issue description, including:

A clear and concise title
Detailed description of the problem or question
Reproducible test case or step-by-step instructions
Valkey GLIDE version in use
Operating system details
Server version
Cluster or standalone setup information, including topology, number of shards, number of replicas, and data types used
Relevant modifications you've made
Any unusual aspects of your environment or deployment
Log files

Contributing

GitHub is a platform for collaborative coding. If you're interested in writing code, we encourage you to contribute by submitting pull requests from forked copies of this repository. Additionally, please consider creating GitHub issues for reporting bugs and suggesting new features. Feel free to comment on issues that interest. For more info see Contributing.

License

Apache License 2.0

StarRocks/starrocks

2024-12-01T01:54:33Z

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

Download | Docs | Benchmarks | Demo

StarRocks is the world's fastest open query engine for sub-second, ad-hoc analytics both on and off the data lakehouse. With average query performance 3x faster than other popular alternatives, StarRocks is a query engine that eliminates the need for denormalization and adapts to your use cases, without having to move your data or rewrite SQL. A Linux Foundation project.

Learn more 👉🏻 What Is StarRocks: Features and Use Cases

Features

🚀 Native vectorized SQL engine: StarRocks adopts vectorization technology to make full use of the parallel computing power of CPU, achieving sub-second query returns in multi-dimensional analyses, which is 5 to 10 times faster than previous systems.
📊 Standard SQL: StarRocks supports ANSI SQL syntax (fully supported TPC-H and TPC-DS). It is also compatible with the MySQL protocol. Various clients and BI software can be used to access StarRocks.
💡 Smart query optimization: StarRocks can optimize complex queries through CBO (Cost Based Optimizer). With a better execution plan, the data analysis efficiency will be greatly improved.
⚡ Real-time update: The updated model of StarRocks can perform upsert/delete operations according to the primary key, and achieve efficient query while concurrent updates.
🪟 Intelligent materialized view: The materialized view of StarRocks can be automatically updated during the data import and automatically selected when the query is executed.
✨ Querying data in data lakes directly: StarRocks allows direct access to data from Apache Hive™, Apache Iceberg™, Delta Lake™ and Apache Hudi™ without importing.
🎛️ Resource management: This feature allows StarRocks to limit resource consumption for queries and implement isolation and efficient use of resources among tenants in the same cluster.
💠 Easy to maintain: Simple architecture makes StarRocks easy to deploy, maintain and scale out. StarRocks tunes its query plan agilely, balances the resources when the cluster is scaled in or out, and recovers the data replica under node failure automatically.

Architecture Overview

StarRocks’s streamlined architecture is mainly composed of two modules: Frontend (FE) and Backend (BE). The entire system eliminates single points of failure through seamless and horizontal scaling of FE and BE, as well as replication of metadata and data.

Starting from version 3.0, StarRocks supports a new shared-data architecture, which can provide better scalability and lower costs.

Resources

📚 Read the docs

Section	Description
Quick Starts	How-tos and Tutorials.
Deploy	Learn how to run and configure StarRocks.
Docs	Full documentation.
Blogs	StarRocks deep dive and user stories.

❓ Get support

Slack community: join technical discussions, ask questions, and meet other users!
YouTube channel: subscribe to the latest video tutorials and webcasts.
GitHub issues: report an issue with StarRocks.

Contributing to StarRocks

We welcome all kinds of contributions from the community, individuals and partners. We owe our success to your active involvement.

See Contributing.md to get started.
Set up StarRocks development environment:

Understand our GitHub workflow for opening a pull request; use this PR Template when submitting a pull request.
Pick a good first issue and start contributing.

📝 License: StarRocks is licensed under Apache License 2.0.

👥 Community Membership: Learn more about different contributor roles in StarRocks community.

💬 Developer Group： Please join our Google Groups to discuss StarRocks features, project directions, issues, pull requests, or share suggestions.

Used By

This project is used by the following companies. Learn more about their use cases:

dromara/Sa-Token

2024-12-01T01:54:33Z

一个轻量级 Java 权限认证框架，让鉴权变得简单、优雅！—— 登录认证、权限认证、分布式Session会话、微服务网关鉴权、单点登录、OAuth2.0

Sa-Token v1.39.0

一个轻量级 Java 权限认证框架，让鉴权变得简单、优雅！

在线文档：https://sa-token.cc

Sa-Token 介绍

Sa-Token 是一个轻量级 Java 权限认证框架，目前拥有五大核心模块：登录认证、权限认证、单点登录、OAuth2.0、微服务鉴权。

简单示例展示：（点击展开 / 折叠）

Sa-Token 旨在以简单、优雅的方式完成系统的权限认证部分，以登录认证为例，你只需要：

// 会话登录，参数填登录人的账号id 
StpUtil.login(10001);

无需实现任何接口，无需创建任何配置文件，只需要这一句静态代码的调用，便可以完成会话登录认证。

如果一个接口需要登录后才能访问，我们只需调用以下代码：

// 校验当前客户端是否已经登录，如果未登录则抛出 `NotLoginException` 异常
StpUtil.checkLogin();

在 Sa-Token 中，大多数功能都可以一行代码解决：

踢人下线：

// 将账号id为 10077 的会话踢下线 
StpUtil.kickout(10077);

权限认证：

// 注解鉴权：只有具备 `user:add` 权限的会话才可以进入方法
@SaCheckPermission("user:add")
public String insert(SysUser user) {
    // ... 
    return "用户增加";
}

路由拦截鉴权：

// 根据路由划分模块，不同模块不同鉴权 
registry.addInterceptor(new SaInterceptor(handler -> {
	SaRouter.match("/user/**", r -> StpUtil.checkPermission("user"));
	SaRouter.match("/admin/**", r -> StpUtil.checkPermission("admin"));
	SaRouter.match("/goods/**", r -> StpUtil.checkPermission("goods"));
	SaRouter.match("/orders/**", r -> StpUtil.checkPermission("orders"));
	SaRouter.match("/notice/**", r -> StpUtil.checkPermission("notice"));
	// 更多模块... 
})).addPathPatterns("/**");

当你受够 Shiro、SpringSecurity 等框架的三拜九叩之后，你就会明白，相对于这些传统老牌框架，Sa-Token 的 API 设计是多么的简单、优雅！

核心模块一览：（点击展开 / 折叠）

登录认证 —— 单端登录、多端登录、同端互斥登录、七天内免登录。
权限认证 —— 权限认证、角色认证、会话二级认证。
踢人下线 —— 根据账号id踢人下线、根据Token值踢人下线。
注解式鉴权 —— 优雅的将鉴权与业务代码分离。
路由拦截式鉴权 —— 根据路由拦截鉴权，可适配 restful 模式。
Session会话 —— 全端共享Session,单端独享Session,自定义Session,方便的存取值。
持久层扩展 —— 可集成 Redis，重启数据不丢失。
前后台分离 —— APP、小程序等不支持 Cookie 的终端也可以轻松鉴权。
Token风格定制 —— 内置六种 Token 风格，还可：自定义 Token 生成策略。
记住我模式 —— 适配 [记住我] 模式，重启浏览器免验证。
二级认证 —— 在已登录的基础上再次认证，保证安全性。
模拟他人账号 —— 实时操作任意用户状态数据。
临时身份切换 —— 将会话身份临时切换为其它账号。
同端互斥登录 —— 像QQ一样手机电脑同时在线，但是两个手机上互斥登录。
账号封禁 —— 登录封禁、按照业务分类封禁、按照处罚阶梯封禁。
密码加密 —— 提供基础加密算法，可快速 MD5、SHA1、SHA256、AES 加密。
会话查询 —— 提供方便灵活的会话查询接口。
Http Basic认证 —— 一行代码接入 Http Basic、Digest 认证。
全局侦听器 —— 在用户登陆、注销、被踢下线等关键性操作时进行一些AOP操作。
全局过滤器 —— 方便的处理跨域，全局设置安全响应头等操作。
多账号体系认证 —— 一个系统多套账号分开鉴权（比如商城的 User 表和 Admin 表）
单点登录 —— 内置三种单点登录模式：同域、跨域、同Redis、跨Redis、前后端分离等架构都可以搞定。
单点注销 —— 任意子系统内发起注销，即可全端下线。
OAuth2.0认证 —— 轻松搭建 OAuth2.0 服务，支持openid模式。
分布式会话 —— 提供共享数据中心分布式会话方案。
微服务网关鉴权 —— 适配Gateway、ShenYu、Zuul等常见网关的路由拦截认证。
RPC调用鉴权 —— 网关转发鉴权，RPC调用鉴权，让服务调用不再裸奔
临时Token认证 —— 解决短时间的 Token 授权问题。
独立Redis —— 将权限缓存与业务缓存分离。
Quick快速登录认证 —— 为项目零代码注入一个登录页面。
标签方言 —— 提供 Thymeleaf 标签方言集成包，提供 beetl 集成示例。
jwt集成 —— 提供三种模式的 jwt 集成方案，提供 token 扩展参数能力。
RPC调用状态传递 —— 提供 dubbo、grpc 等集成包，在RPC调用时登录状态不丢失。
参数签名 —— 提供跨系统API调用签名校验模块，防参数篡改，防请求重放。
自动续签 —— 提供两种Token过期策略，灵活搭配使用，还可自动续签。
开箱即用 —— 提供SpringMVC、WebFlux、Solon 等常见框架集成包，开箱即用。
最新技术栈 —— 适配最新技术栈：支持 SpringBoot 3.x，jdk 17。

SSO 单点登录

Sa-Token SSO 分为三种模式，解决同域、跨域、共享Redis、跨Redis、前后端一体、前后端分离……等不同架构下的 SSO 接入问题：

系统架构	采用模式	简介	文档链接
前端同域 + 后端同 Redis	模式一	共享Cookie同步会话	文档、示例
前端不同域 + 后端同 Redis	模式二	URL重定向传播会话	文档、示例
前端不同域 + 后端不同Redis	模式三	Http请求获取会话	文档、示例

前端同域：就是指多个系统可以部署在同一个主域名之下，比如：c1.domain.com、c2.domain.com、c3.domain.com
后端同Redis：就是指多个系统可以连接同一个Redis。（此处并非要所有项目数据都放在一个Redis中，Sa-Token提供 [权限缓存与业务缓存分离] 的解决方案）
如果既无法做到前端同域，也无法做到后端同Redis，可以走模式三，Http请求校验 ticket 获取会话。
提供 NoSdk 模式示例，不使用 Sa-Token 的系统也可以对接。
提供 sso-server 接口文档，不使用 java 语言的系统也可以对接。
提供前后端分离整合方案：无论是 sso-server 还是 sso-client 的前后端分离都可以整合。
提供安全校验：域名校验、ticket校验、参数签名校验，有效防 ticket 劫持，防请求重放等攻击。
参数防丢：笔者曾试验多个SSO框架，均有参数丢失情况，比如登录前是：http://a.com?id=1&name=2，登录成功后就变成了：http://a.com?id=1，Sa-Token-SSO 内有专门算法保证了参数不丢失，登录成功后精准原路返回。
提供用户数据同步/迁移方案的建议：开发前统一迁移、运行时实时数据同步、根据关联字段匹配、根据 center_id 字段匹配等。
提供直接可运行的 demo 示例，帮助你快速熟悉 SSO 大致登录流程。

OAuth2 授权认证

Sa-Token-OAuth2 模块分为四种授权模式，解决不同场景下的授权需求

授权模式	简介
授权码（Authorization Code）	OAuth2.0 标准授权步骤，Server 端向 Client 端下放 Code 码，Client 端再用 Code 码换取授权 Token
隐藏式（Implicit）	无法使用授权码模式时的备用选择，Server 端使用 URL 重定向方式直接将 Token 下放到 Client 端页面
密码式（Password）	Client直接拿着用户的账号密码换取授权 Token
客户端凭证（Client Credentials）	Server 端针对 Client 级别的 Token，代表应用自身的资源授权

详细参考文档：https://sa-token.cc/doc.html#/oauth2/readme

开源集成案例

[ Snowy ]：国内首个国密前后分离快速开发平台，采用 Vue3 + AntDesignVue3 + Vite + SpringBoot + Mp + HuTool + SaToken。
[ RuoYi-Vue-Plus ]：重写RuoYi-Vue所有功能集成 Sa-Token+Mybatis-Plus+Jackson+Xxl-Job+knife4j+Hutool+OSS 定期同步
[Smart-Admin]：SmartAdmin国内首个以「高质量代码」为核心，「简洁、高效、安全」中后台快速开发平台；
[ 灯灯 ]：专注于多租户解决方案的微服务中后台快速开发平台。租户模式支持独立数据库(DATASOURCE模式)、共享数据架构(COLUMN模式) 和非租户模式(NONE模式)✨
[ EasyAdmin ]：一个基于SpringBoot2 + Sa-Token + Mybatis-Plus + Snakerflow + Layui 的后台管理系统，灵活多变可前后端分离，也可单体，内置代码生成器、权限管理、工作流引擎等
[ sa-admin-server ]：基于 sa-admin-ui 的后台管理开发脚手架。

还有更多优秀开源案例无法逐一展示，请参考：Awesome-Sa-Token

友情链接

[ OkHttps ]：轻量级 http 通信框架，API无比优雅，支持 WebSocket、Stomp 协议
[ Bean Searcher ]：专注高级查询的只读 ORM，使一行代码实现复杂列表检索！
[ Jpom ]：简而轻的低侵入式在线构建、自动部署、日常运维、项目监控软件。
[ TLog ]：一个轻量级的分布式日志标记追踪神器。
[ hippo4j ]：强大的动态线程池框架，附带监控报警功能。
[ hertzbeat ]：易用友好的开源实时监控告警系统，无需Agent，高性能集群，强大自定义监控能力。
[ Solon ]：一个更现代感的应用开发框架：更快、更小、更自由。
[ Chat2DB ]：一个AI驱动的数据库管理和BI工具，支持Mysql、pg、Oracle、Redis等22种数据库的管理。

代码托管

Gitee：https://gitee.com/dromara/sa-token
GitHub：https://github.com/dromara/sa-token
GitCode：https://gitcode.com/dromara/sa-token

交流群

QQ交流群：823181187 点击加入

微信交流群：

(扫码添加微信，备注：sa-token，邀您加入群聊)

加入群聊的好处：

第一时间收到框架更新通知。
第一时间收到框架 bug 通知。
第一时间收到新增开源案例通知。
和众多大佬一起互相 (huá shuǐ) 交流 (mō yú)。

gyoogle/tech-interview-for-developer

2024-12-01T01:54:33Z

👶🏻 신입 개발자 전공 지식 & 기술 면접 백과사전 📖

tech-interview-for-developer

👶🏻 신입 개발자 전공 지식 & 기술 면접 백과사전 📖

Collaborator

Commit convention rule : 날짜-[주제]-내용-상태

ex) 2019-10-14 [Algorithm] Sort Add/Update/Delete

잘못된 내용은 이슈와 PR로 알려주세요 💡

🙏도움을 주신 분들🙏

💖후원하기💝

⏩ ⏩ ⏩ 웹 사이트에서 편하게 공부하세요! Click! ⏪ ⏪ ⏪

👨🏻‍🏫 기술 면접 감 잡기 👩🏻‍🏫

📌 Computer Science

📌 Algorithm

✏️ TEST
- 삼성 소프트웨어 역량테스트 PRO 등급 준비

📌 Design Pattern

📌 Interview

📌 Language

📌 Web

📌 Linux

📌 New Technology

📌 Seminar

📌 ETC

License

apache/camel

2024-12-01T01:43:37Z

Apache Camel is an open source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data.

Apache Camel

Apache Camel is an Open Source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data.

Introduction

Camel empowers you to define routing and mediation rules in a variety of domain-specific languages (DSL, such as Java, XML, Groovy and YAML). This means you get smart completion of routing rules in your IDE, whether in a Java or XML editor.

Apache Camel uses URIs to enable easier integration with all kinds of transport or messaging model including HTTP, ActiveMQ, JMS, JBI, SCA, MINA or CXF together with working with pluggable Data Format options. Apache Camel is a small library that has minimal dependencies for easy embedding in any Java application. Apache Camel lets you work with the same API regardless of the transport type, making it possible to interact with all the components provided out-of-the-box, with a good understanding of the API.

Apache Camel has powerful Bean Binding and integrated seamlessly with popular frameworks such as Spring, Quarkus, and CDI.

Apache Camel has extensive testing support allowing you to easily unit test your routes.

Components

Apache Camel comes alongside several artifacts with components, data formats, languages, and kinds. The up-to-date list is available online at the Camel website:

Components: https://camel.apache.org/components/latest/
Data Formats: https://camel.apache.org/components/latest/dataformats/
Languages: https://camel.apache.org/components/latest/languages/
Miscellaneous: https://camel.apache.org/components/latest/#_miscellaneous_components

Examples

Apache Camel comes with many examples. The up to date list is available online at GitHub:

Examples: https://github.com/apache/camel-examples/tree/main#welcome-to-the-apache-camel-examples

Getting Started

To help you get started, try the following links:

Getting Started

https://camel.apache.org/getting-started.html

The beginner examples are another powerful alternative pathway for getting started with Apache Camel.

Examples: https://github.com/apache/camel-examples/tree/main#welcome-to-the-apache-camel-examples

Building

https://camel.apache.org/camel-core/contributing/

Contributions

We welcome all kinds of contributions, the details of which are specified here:

https://github.com/apache/camel/blob/main/CONTRIBUTING.md

Please refer to the website for details of finding the issue tracker, email lists, GitHub, chat

Website: https://camel.apache.org/

GitHub (source): https://github.com/apache/camel

Issue tracker: https://issues.apache.org/jira/projects/CAMEL

Mailing-list: https://camel.apache.org/community/mailing-list/

Chat: https://camel.zulipchat.com/

StackOverflow: https://stackoverflow.com/questions/tagged/apache-camel

Twitter: https://twitter.com/ApacheCamel

Support

For additional help, support, we recommend referencing this page first:

https://camel.apache.org/community/support/

Getting Help

If you get stuck somewhere, please feel free to reach out to us on either StackOverflow, Chat, or the email mailing list.

Please help us make Apache Camel better — we appreciate any feedback you may have.

Enjoy!

The Camel riders!

Licensing

The terms for software licensing are detailed in the LICENSE.txt file,
located in the working directory.

comet-ml/opik

2024-12-01T01:43:37Z

Open-source end-to-end LLM Development Platform

Opik
Open-source end-to-end LLM Development Platform

Confidently evaluate, test and monitor LLM applications.

Website • Slack community • Twitter • Documentation

🚀 What is Opik?

Opik is an open-source platform for evaluating, testing and monitoring LLM applications. Built by Comet.

You can use Opik for:

Development:
- Tracing: Track all LLM calls and traces during development and production (Quickstart, Integrations
- Annotations: Annotate your LLM calls by logging feedback scores using the Python SDK or the UI.
Evaluation: Automate the evaluation process of your LLM application:
- Datasets and Experiments: Store test cases and run experiments (Datasets, Evaluate your LLM Application)
- LLM as a judge metrics: Use Opik's LLM as a judge metric for complex issues like hallucination detection, moderation and RAG evaluation (Answer Relevance, Context Precision
- CI/CD integration: Run evaluations as part of your CI/CD pipeline using our PyTest integration
Production Monitoring: Monitor your LLM application in production and easily close the feedback loop by adding error traces to your evaluation datasets.

[!TIP]
If you are looking for features that Opik doesn't have today, please raise a new Github discussion topic 🚀

🛠️ Installation

Opik is available as a fully open source local installation or using Comet.com as a hosted solution. The easiest way to get started with Opik is by creating a free Comet account at comet.com.

If you'd like to self-host Opik, you can do so by cloning the repository and starting the platform using Docker Compose:

# Clone the Opik repository
git clone https://github.com/comet-ml/opik.git

# Navigate to the opik/deployment/docker-compose directory
cd opik/deployment/docker-compose

# Start the Opik platform
docker compose up --detach

# You can now visit http://localhost:5173 on your browser!

For more information about the different deployment options, please see our deployment guides:

Installation methods	Docs link
Local instance
Kubernetes

🏁 Get Started

To get started, you will need to first install the Python SDK:

pip install opik

Once the SDK is installed, you can configure it by running the opik configure command:

opik configure

This will allow you to configure Opik locally by setting the correct local server address or if you're using the Cloud platform by setting the API Key

[!TIP]
You can also call the opik.configure(use_local=True) method from your Python code to configure the SDK to run on the local installation.

You are now ready to start logging traces using the Python SDK.

📝 Logging Traces

The easiest way to get started is to use one of our integrations. Opik supports:

Integration	Description	Documentation
OpenAI	Log traces for all OpenAI LLM calls	Documentation
LiteLLM	Call any LLM model using the OpenAI format	Documentation
LangChain	Log traces for all LangChain LLM calls	Documentation
Bedrock	Log traces for all Bedrock LLM calls	Documentation
Anthropic	Log traces for all Anthropic LLM calls	Documentation
Gemini	Log traces for all Gemini LLM calls	Documentation
Groq	Log traces for all Groq LLM calls	Documentation
LangGraph	Log traces for all LangGraph executions	Documentation
LlamaIndex	Log traces for all LlamaIndex LLM calls	Documentation
Ollama	Log traces for all Ollama LLM calls	Documentation
Predibase	Fine-tune and serve open-source Large Language Models	Documentation
Ragas	Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines	Documentation
watsonx	Log traces for all watsonx LLM calls	Documentation

[!TIP]
If the framework you are using is not listed above, feel free to open an issue or submit a PR with the integration.

If you are not using any of the frameworks above, you can also use the track function decorator to log traces:

import opik

opik.configure(use_local=True) # Run locally

@opik.track
def my_llm_function(user_question: str) -> str:
    # Your LLM code here

    return "Hello"

[!TIP]
The track decorator can be used in conjunction with any of our integrations and can also be used to track nested function calls.

🧑‍⚖️ LLM as a Judge metrics

The Python Opik SDK includes a number of LLM as a judge metrics to help you evaluate your LLM application. Learn more about it in the metrics documentation.

To use them, simply import the relevant metric and use the score function:

from opik.evaluation.metrics import Hallucination

metric = Hallucination()
score = metric.score(
    input="What is the capital of France?",
    output="Paris",
    context=["France is a country in Europe."]
)
print(score)

Opik also includes a number of pre-built heuristic metrics as well as the ability to create your own. Learn more about it in the metrics documentation.

🔍 Evaluating your LLM Application

Opik allows you to evaluate your LLM application during development through Datasets and Experiments.

You can also run evaluations as part of your CI/CD pipeline using our PyTest integration.

🤝 Contributing

There are many ways to contribute to Opik:

Submit bug reports and feature requests
Review the documentation and submit Pull Requests to improve it
Speaking or writing about Opik and letting us know
Upvoting popular feature requests to show your support

To learn more about how to contribute to Opik, please see our contributing guidelines.

elastic/logstash

2024-12-01T01:43:37Z

Logstash - transport and process your logs, events, or other data

Logstash

Logstash is part of the Elastic Stack along with Beats, Elasticsearch and Kibana. Logstash is a server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash." (Ours is Elasticsearch, naturally.). Logstash has over 200 plugins, and you can write your own very easily as well.

For more info, see https://www.elastic.co/products/logstash

Documentation and Getting Started

You can find the documentation and getting started guides for Logstash on the elastic.co site

For information about building the documentation, see the README in https://github.com/elastic/docs

Downloads

You can download officially released Logstash binaries, as well as debian/rpm packages for the supported platforms, from downloads page.

Need Help?

Logstash Plugins

Logstash plugins are hosted in separate repositories under the logstash-plugins github organization. Each plugin is a self-contained Ruby gem which gets published to RubyGems.org.

Writing your own Plugin

Logstash is known for its extensibility. There are hundreds of plugins for Logstash and you can write your own very easily! For more info on developing and testing these plugins, please see the working with plugins section

Plugin Issues and Pull Requests

Please open new issues and pull requests for plugins under its own repository

For example, if you have to report an issue/enhancement for the Elasticsearch output, please do so here.

Logstash core will continue to exist under this repository and all related issues and pull requests can be submitted here.

Developing Logstash Core

Prerequisites

Install JDK version 11 or 17. Make sure to set the JAVA_HOME environment variable to the path to your JDK installation directory. For example set JAVA_HOME=<JDK_PATH>
Install JRuby 9.2.x It is recommended to use a Ruby version manager such as RVM or rbenv.
Install rake and bundler tool using gem install rake and gem install bundler respectively.

RVM install (optional)

If you prefer to use rvm (ruby version manager) to manage Ruby versions on your machine, follow these directions. In the Logstash folder:

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -sSL https://get.rvm.io | bash -s stable --ruby=$(cat .ruby-version)

Check Ruby version

Before you proceed, please check your ruby version by:

$ ruby -v

The printed version should be the same as in the .ruby-version file.

Building Logstash

The Logstash project includes the source code for all of Logstash, including the Elastic-Licensed X-Pack features and functions; to run Logstash from source using only the OSS-licensed code, export the OSS environment variable with a value of true:

export OSS=true

Set up the location of the source code to build

export LOGSTASH_SOURCE=1
export LOGSTASH_PATH=/YOUR/LOGSTASH/DIRECTORY

Install dependencies with `gradle` (recommended)[^1]

Install development dependencies

./gradlew installDevelopmentGems

Install default plugins and other dependencies

./gradlew installDefaultGems

Verify the installation

To verify your environment, run the following to start Logstash and send your first event:

bin/logstash -e 'input { stdin { } } output { stdout {} }'

This should start Logstash with stdin input waiting for you to enter an event

hello world
2016-11-11T01:22:14.405+0000 0.0.0.0 hello world

Advanced: Drip Launcher

Drip is a tool that solves the slow JVM startup problem while developing Logstash. The drip script is intended to be a drop-in replacement for the java command. We recommend using drip during development, in particular for running tests. Using drip, the first invocation of a command will not be faster but the subsequent commands will be swift.

To tell logstash to use drip, set the environment variable JAVACMD=`which drip`.

Example (but see the Testing section below before running rspec for the first time):

JAVACMD=`which drip` bin/rspec

Caveats

Drip does not work with STDIN. You cannot use drip for running configs which use the stdin plugin.

Building Logstash Documentation

To build the Logstash Reference (open source content only) on your local machine, clone the following repos:

logstash - contains main docs about core features

logstash-docs - contains generated plugin docs

docs - contains doc build files

Make sure you have the same branch checked out in logstash and logstash-docs. Check out master in the docs repo.

Run the doc build script from within the docs repo. For example:

./build_docs.pl --doc ../logstash/docs/index.asciidoc --chunk=1 -open

Testing

Most of the unit tests in Logstash are written using rspec for the Ruby parts. For the Java parts, we use junit. For testing you can use the test rake tasks and the bin/rspec command, see instructions below:

Core tests

1- To run the core tests you can use the Gradle task:

./gradlew test

or use the rspec tool to run all tests or run a specific test:

bin/rspec
bin/rspec spec/foo/bar_spec.rb

Note that before running the rspec command for the first time you need to set up the RSpec test dependencies by running:

./gradlew bootstrap

2- To run the subset of tests covering the Java codebase only run:

./gradlew javaTests

3- To execute the complete test-suite including the integration tests run:

./gradlew check

4- To execute a single Ruby test run:

SPEC_OPTS="-fd -P logstash-core/spec/logstash/api/commands/default_metadata_spec.rb" ./gradlew :logstash-core:rubyTests --tests org.logstash.RSpecTests

5- To execute single spec for integration test, run:

./gradlew integrationTests -PrubyIntegrationSpecs=specs/slowlog_spec.rb

Sometimes you might find a change to a piece of Logstash code causes a test to hang. These can be hard to debug.

If you set LS_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" you can connect to a running Logstash with your IDEs debugger which can be a great way of finding the issue.

Plugins tests

To run the tests of all currently installed plugins:

rake test:plugins

You can install the default set of plugins included in the logstash package:

rake test:install-default

Note that if a plugin is installed using the plugin manager bin/logstash-plugin install ... do not forget to also install the plugins development dependencies using the following command after the plugin installation:

bin/logstash-plugin install --development

Building Artifacts

Built artifacts will be placed in the LS_HOME/build directory, and will create the directory if it is not already present.

You can build a Logstash snapshot package as tarball or zip file

./gradlew assembleTarDistribution
./gradlew assembleZipDistribution

OSS-only artifacts can similarly be built with their own gradle tasks:

./gradlew assembleOssTarDistribution
./gradlew assembleOssZipDistribution

You can also build .rpm and .deb, but the fpm tool is required.

rake artifact:rpm
rake artifact:deb

and:

rake artifact:rpm_oss
rake artifact:deb_oss

Using a Custom JRuby Distribution

If you want the build to use a custom JRuby you can do so by setting a path to a custom JRuby distribution's source root via the custom.jruby.path Gradle property.

E.g.

./gradlew clean test -Pcustom.jruby.path="/path/to/jruby"

Project Principles

Community: If a newbie has a bad time, it's a bug.
Software: Make it work, then make it right, then make it fast.
Technology: If it doesn't do a thing today, we can make it do it tomorrow.

Contributing

All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.

Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.

It is more important that you are able to contribute.

For more information about contributing, see the CONTRIBUTING file.

Footnotes

[^1]:

Use bundle instead of gradle to install dependencies

#### Alternatively, instead of using `gradle` you can also use `bundle`:

* Install development dependencies

    ```sh
    bundle config set --local path vendor/bundle
    bundle install
    ```

* Bootstrap the environment:

    ```sh
    rake bootstrap
    ```

* You can then use `bin/logstash` to start Logstash, but there are no plugins installed. To install default plugins, you can run:

    ```sh
    rake plugin:install-default
    ```

This will install the 80+ default plugins which makes Logstash ready to connect to multiple data sources, perform transformations and send the results to Elasticsearch and other destinations.
</details>

besuper/TwitchNoSub

2024-11-30T01:32:04Z

Easily manage the brightness of your monitors in Windows from the system tray

Twinkle Tray

Twinkle Tray enables brightness control on external displays in Windows 10 & 11. Even though Windows is capable of adjusting the backlight on most monitors, it doesn't support external monitors natively. Windows also lacks any options to manage the brightness of multiple displays. This app inserts a new icon into your system tray, where you can click to have instant access to the brightness levels of all compatible displays.

Features

Adds brightness sliders to the system tray, similar to the built-in Windows volume flyout.
Seamlessly blends in with Windows 10 and Windows 11. Uses your Personalization settings to match your taskbar.
Can automatically change monitor brightness depending on the time of day or when idle.
Bind hotkeys to adjust the brightness of specific or all displays.
Normalize backlight across different monitors.
Control DDC/CI features such as contrast.
Starts up with Windows.

Design & Personalization

Twinkle Tray will automatically adjust the look and feel to match your Windows version and preferences. Additional options are available to select the Windows version and theme of your choice.

Download

Download the lastest version from twinkletray.com or the Releases page.

Install via Package Manager

Windows Package Manager

winget install xanderfrangos.twinkletray

Chocolatey (unofficial)

Chocolatey users can download and install Twinkle Tray from Chocolatey's Community Repository by installing the twinkle-tray package:

choco install twinkle-tray

To upgrade to the latest approved package version, run the following command:

choco upgrade twinkle-tray

This package is not maintained at this repository. Please do not create issues relating to the package here. Instead, go to the package page and follow the Package Triage Process.

Scoop (unofficial)

Scoop users can download and install Twinkle Tray from Scoop's Extras bucket by installing the twinkle-tray package:

scoop bucket add extras
scoop install extras/twinkle-tray

To upgrade to the latest approved package version, run the following command:

scoop update twinkle-tray

This package is not maintained at this repository. Please do not create issues relating to the package here. Instead, go to ScoopInstallers/Extras and search for an existing issue or discussion and create a new issue or discussion if one does not already exist.

Usage

Download from the Releases page and run the installer EXE.
Once installation has finished, you should see the Twinkle Tray icon in your system tray.
Click the icon to bring up the Adjust Brightness flyout.
Click away to hide the flyout.
Right-click the system tray icon to quit.

Compatibility

Twinkle Tray uses DDC/CI and WMI to communicate with your monitors. Most monitors offer DDC/CI compatibility, but it may be off by default. Make sure you have the appropriate option(s) enabled on your monitor so that it can work with Twinkle Tray. Refer to your monitor's user manual for more information.

Known issues:

The AMD Radeon Control Panel can interfere with Twinkle Tray. Ensure "Custom Colors" is not enabled.
VGA/DVI may not be compatible.
USB/Thunderbolt/Surface docks with HDMI or DisplayPort may not be compatible.
DDC/CI features such as brightness control and power state may cause certain models of monitors to behave poorly. This applies to any DDC/CI software, not just Twinkle Tray.

If some of your monitors are not being detected, please see this page for troubleshooting steps.

Command Line Arguments

Twinkle Tray (v1.13.0+) supports requesting brightness changes from the command line. Twinkle Tray must already be running. One monitor argument and one brightness argument are required. Multiple arguments will override each other.

For example: "%LocalAppData%\Programs\twinkle-tray\Twinkle Tray.exe" --MonitorNum=1 --Offset=-30 will adjust monitor number 1 by -30 brightness.

Supported args:

--List List all displays. (available in v1.14.0+)
--MonitorNum Select monitor by number. Starts at 1. Example: --MonitorNum=2
--MonitorID Select monitor by internal ID. Partial or whole matches accepted. Example: --MonitorID="UID2353"
--All Flag to select all monitors.
--Set Set brightness percentage. Example: --Set=95
--Offset Adjust brightness percentage. Example: --Offset=-20
--VCP Send a specific DDC/CI VCP code and value instead of brightness. The first part is the VCP code (decimal or hexadecimal), and the second is the value. Example: --VCP="0xD6:5" (available in v1.14.4+)
--Overlay Flag to show new brightness levels in the overlay Example: --Overlay
--Panel Flag to show new brightness levels in the panel Example: --Panel

This feature is not available on the Windows Store version of Twinkle Tray.

Localization

Thanks to several contributors, Twinkle Tray is localized for multiple languages. If you'd like to create or update a localization, see this page for details. Special thanks to Weblate for allowing free use of their service.

Localization progress

Build Instructions

If you wish to run a development build of Twinkly Tray:

Download or clone.
Install the build tools for node-gyp, if not already installed. You may already have these from installing NodeJS.
Run npm install.
Run npm run build to build an executable or npm start to run a development build.

Note: Twinkle Tray must be built on Windows.

Special Thanks

Twinkle Tray was built using frameworks & libraries such as Electron, Node.js, node-ddcci, and win32-displayconfig. Thanks to Weblate for allowing free use of their service, along with the many contributors to the localizations of Twinkle Tray. The app would not be nearly as popular without all of your help. And thank you for the many donations, small and large, over the years.

License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

stdlib-js/stdlib

2024-12-01T01:54:45Z

✨ Standard library for JavaScript and Node.js. ✨

We believe in a future in which the web is a preferred environment for numerical computation. To help realize this future, we've built stdlib.

stdlib (/ˈstændərd lɪb/ "standard lib") is a standard library with an emphasis on numerical and scientific computation, written in JavaScript (and C) for execution in browsers and in Node.js. The library provides a collection of robust, high performance libraries for mathematics, statistics, data processing, streams, and more and includes many of the utilities you would expect from a standard library.

What sets stdlib apart is its fully decomposable architecture, which allows you to swap out and mix and match APIs and functionality to cater to your exact preferences and use cases.

When you use stdlib, you can be confident that you are using the most thorough, rigorous, well-written, studied, documented, tested, measured, and high-quality code available.

Want to join us in bringing numerical computing to the web? Start by starring the project. 🌟

Explore this GitHub repository for stdlib's source code and documentation. For guidance on developing stdlib, refer to the development guide.

Thank you for being a part of our community! Your support is invaluable to us!

Resources

External Resources

Features

150+ special math functions.
35+ probability distributions, with support for evaluating probability density functions (PDFs), cumulative distribution functions (CDFs), quantiles, moments, and more.
40+ seedable pseudorandom number generators (PRNGs).
200+ general utilities for data transformation, functional programming, and asynchronous control flow.
200+ assertion utilities for data validation and feature detection.
50+ sample datasets for testing and development.
A plot API for data visualization and exploratory data analysis.
Native add-ons for interfacing with BLAS libraries, with pure JavaScript fallbacks.
A benchmark framework supporting TAP.
REPL environment with integrated help and examples.
Can be bundled using Browserify, Webpack, and other bundlers for use in web browsers.
Every function is accompanied by TypeScript declaration files, ensuring type safety and facilitating intelligent code completion in IDEs.

Installation

To accommodate various use cases, stdlib can be used in multiple ways. The preferred method of use depends on your individual use case. We've provided some user stories to help you identify the best approach. 😃

While this project's installation instructions defaults to using npm for package management, installation via other package managers, such as yarn, should be a matter of simply swapping out npm commands with those of the relevant package manager.

User Stories

I want to perform data analysis and data science tasks in JavaScript and Node.js, similar to how I might use Python, Julia, R, and MATLAB.
- Install the entire project as a command-line utility.
I am building a web application.
- I plan on using Browserify, Webpack, and other bundlers for use in web browsers.
  - Install individual packages. Installing the entire project is likely unnecessary and will lead to slower installation times.
- I would like to vendor a custom bundle containing various stdlib functionality.
  - Follow the steps for creating custom bundles.
- I would like to include stdlib functionality by just using a script tag.
  - I would like to use ES Modules.
    - Use an individual package's ES Module build.
  - I would like to use a pre-built bundle (possibly via a CDN, such as unpkg or jsDelivr).
    - Install (or consume via a CDN) an individual package's pre-built UMD browser bundle.
- I am interested in using a substantial amount of functionality found in a top-level stdlib namespace and don't want to separately install hundreds of individual packages (e.g., if building an on-line calculator application and wanting all of stdlib's math functionality).
  - Install one or more top-level namespaces. Installing the entire project is likely unnecessary and will lead to slower installation times. Installing a top-level namespace is likely to mean installing functionality which will never be used; however, installing a top-level namespace is likely to be easier and less time-consuming than installing many individual packages separately.
    
    When bundling, installing a top-level namespace should not be a concern, as individual functionality can still be independently required/imported. Project installation times may, however, be somewhat slower.
I am building a Node.js server application.
- I am interested in using various functionality found in stdlib.
  - Install individual packages. Installing the entire project is likely unnecessary and will lead to slower installation times.
- I would like to vendor stdlib functionality and avoid dependency trees.
  - Install individual package UMD bundles.
- I am interested in using a substantial amount of functionality found in a top-level stdlib namespace and don't want to separately install hundreds of individual packages.
  - Install one or more top-level namespaces. Installing the entire project is likely unnecessary and will lead to slower installation times. Installing a top-level namespace is likely to mean installing functionality which will never be used; however, installing a top-level namespace is likely to be easier and less time-consuming than installing many individual packages separately.
I am using Deno.
- Import individual packages using pre-built Deno builds.
I would like to use stdlib functionality in an Observable notebook.
- Consume a pre-built browser bundles via a CDN, such as unpkg or jsDelivr.
I want to hack at stdlib, possibly even creating customized builds to link to platform-specific native libraries (such as Intel's MKL or some other numerical library).
- Install the project as a system library by cloning this repository and following the installation instructions as described in the development guide.

Complete Library

To install the entire project as a library or application dependency,

$ npm install @stdlib/stdlib

Once installed, stdlib packages can be individually required/imported to minimize load times and decrease bundle sizes. For example, to use require

var ndarray = require( '@stdlib/ndarray/array' );

var arr = ndarray( [ [ 1, 2 ], [ 3, 4 ] ] );
// returns <ndarray>

and to use import

import ndarray from '@stdlib/ndarray/array';

var arr = ndarray( [ [ 1, 2 ], [ 3, 4 ] ] );
// returns <ndarray>

Individual Packages

stdlib is designed to allow decomposition of the main project into individual packages which can be independently consumed. Accordingly, users of the project can avoid installing all project functionality and only install the exact functionality they need.

To install individual packages, replace forward slashes / after @stdlib/ with hyphens -. For example,

$ npm install @stdlib/ndarray-array

Once installed, individual packages can be required/imported. For example, to use require

var ndarray = require( '@stdlib/ndarray-array' );

var arr = ndarray( [ [ 1, 2 ], [ 3, 4 ] ] );
// returns <ndarray>

and to use import

import ndarray from '@stdlib/ndarray-array';

var arr = ndarray( [ [ 1, 2 ], [ 3, 4 ] ] );
// returns <ndarray>

Namespaces

stdlib is comprised of various top-level namespaces (i.e., collections of related functionality united by common themes). For example, to install all math functionality found in the top-level math namespace,

$ npm install @stdlib/math

Once installed, packages within a top-level namespace can be individually required/imported to minimize load times and decrease bundle sizes. For example, to use require

var sin = require( '@stdlib/math/base/special/sin' );

var v = sin( 3.14 );
// returns <number>

and to use import

import sin from '@stdlib/math/base/special/sin';

var v = sin( 3.14 );
// returns <number>

Note: installing nested namespaces found within top-level namespaces (e.g., math/base) is not supported. Consider installing individual packages or the relevant top-level namespace.

Command-line Utility

To install globally for use as a command-line utility and/or use the REPL,

$ npm install -g @stdlib/stdlib

which will expose the stdlib command. For example, to see available sub-commands

$ stdlib help

and to run the REPL

$ stdlib repl

Environment Builds

ES Modules

To use ES Modules via a <script> tag, use ES Module builds available in each package's repository via a dedicated esm branch (e.g., see the esm branch for @stdlib/math-base-special-erf). For example,

<script type="module">
import linspace from 'https://cdn.jsdelivr.net/gh/stdlib-js/array-base-linspace@esm/index.mjs';
import erf from 'https://cdn.jsdelivr.net/gh/stdlib-js/math-base-special-erf@esm/index.mjs';

const x = linspace( -10.0, 10.0, 100 );

for ( let i = 0; i < x.length; i++ ) {
    console.log( 'x: %d, erf(x): %d', x[ i ], erf( x[ i ] ) );
}
</script>

Deno

To use individual packages in Deno, use Deno builds available in each package's repository via a dedicated deno branch (e.g., see the deno branch for @stdlib/ndarray-array). For example,

import ndarray from 'https://cdn.jsdelivr.net/gh/stdlib-js/ndarray-array@deno/mod.js';

var arr = ndarray( [ [ 1, 2 ], [ 3, 4 ] ] );
// returns <ndarray>

jQuery-like Bundle

For those wanting a jQuery-like bundle, one can use pre-built distributable UMD bundles for use in browser environments or as shared ("vendored") libraries in server environments available in each package's repository via a dedicated umd branch. See sections UMD and Node.js for more details.

UMD

To use UMD bundles either via a <script> tag or in Observable, use UMD browser builds available in each package's repository via a dedicated umd branch (e.g., see the umd branch for @stdlib/math-base-special-erf). For example,

<script type="text/javascript" src="https://cdn.jsdelivr.net/gh/stdlib-js/array-base-linspace@umd/browser.js"></script>
<script type="text/javascript" src="https://cdn.jsdelivr.net/gh/stdlib-js/math-base-special-erf@umd/browser.js"></script>
<script type="text/javascript">
(function () {

var x = linspace( -10.0, 10.0, 100 );

for ( var i = 0; i < x.length; i++ ) {
    console.log( 'x: %d, erf(x): %d', x[ i ], erf( x[ i ] ) );
}

})();
</script>

Node.js

To vendor stdlib functionality and avoid installing dependency trees, use UMD server builds available in each package's repository via a dedicated umd branch (e.g., see the umd branch for @stdlib/math-base-special-erf). For example,

var linspace = require( '/path/to/vendor/umd/@stdlib/array-base-linspace' );
var erf = require( '/path/to/vendor/umd/@stdlib/math-base-special-erf' );

var x = linspace( -10.0, 10.0, 100 );

for ( var i = 0; i < x.length; i++ ) {
    console.log( 'x: %d, erf(x): %d', x[ i ], erf( x[ i ] ) );
}

Custom Bundles

To create a custom bundle based on project needs,

follow the download, configuration, and installation instructions as described in the development guide.
navigate to the local installation directory.
run the following command to print help documentation for providing a list of stdlib package names to bundle
```
$ NODE_PATH=./lib/node_modules node ./bin/cli bundle-pkg-list -- -h
```

modify and run the above command with the list of packages to bundle

$ NODE_PATH=./lib/node_modules node ./bin/cli bundle-pkg-list -- <pkg> <pkg> <pkg> ...

Upon generating a bundle, the bundle can be loaded via a <script> tag as described above for pre-built distributable UMD bundles.

System Library

To install as a system library (e.g., for the purposes of creating custom builds), follow the download, configuration, and installation instructions as described in the development guide.

Prerequisites

Installing and running stdlib for use in Node.js requires the following prerequisites:

Node.js: JavaScript runtime (version >= 0.10)
npm: package manager (version > 2.7.0; if Node < 1.0.0, version > 2.7.0 and < 4.0.0; if Node <= 10.x.x, version > 2.7.0 and < 6.0.0)

Most functionality in stdlib is implemented in JavaScript and no further prerequisites are required to use stdlib (i.e., you can safely avoid installing any additional prerequisites); however, some implementations try to capture performance benefits by using native bindings and/or WebAssembly. While not required to run stdlib, as every stdlib implementation has a JavaScript fallback, the following dependencies are required for building native add-ons, including linking to BLAS and LAPACK libraries:

GNU make: development utility and task runner
GNU bash: an sh-compatible shell
gcc & g++ or Clang: C/C++ compilation and linking (g++ version >= 4.8; clang version >= 3.5, Xcode version >=8.3.1 on OS X)
gfortran: Fortran compilation and linking (version >= 4.8)

While not required to run stdlib, the following dependencies are required for automatically downloading external libraries:

curl, wget, or fetch (FreeBSD): utilities for downloading remote resources

The following external libraries can be automatically downloaded and compiled from source using make:

OpenBLAS: optimized BLAS library
Electron: framework for cross-platform desktop applications

Contributing

First time contributor?

See the contributing guidelines.

Already an expert?

Fork the repository.
Clone the forked repository
```
$ git clone --depth=1 https://github.com/<username>/stdlib.git
```
where <username> is your GitHub username.
Navigate to the stdlib directory
```
$ cd stdlib
```
Install dependencies
```
$ make install-node-modules
```
Initialize your stdlib development environment
```
$ make init
```

Users

The following organizations and key stakeholders trust and rely on stdlib:

Does your organization use stdlib? If so, we'd love to hear from you!

Governance

For information about the governance of the stdlib project, see GOVERNANCE.md.

License

See LICENSE.

Copyright

Status

Version

Community

GitSquared/edex-ui

2024-12-01T01:54:45Z

A cross-platform, customizable science fiction terminal emulator with advanced monitoring & touchscreen support.

(Project archived oct. 18th 2021)

eDEX-UI is a fullscreen, cross-platform terminal emulator and system monitor that looks and feels like a sci-fi computer interface.

Heavily inspired from the TRON Legacy movie effects (especially the Board Room sequence), the eDEX-UI project was originally meant to be "DEX-UI with less « art » and more « distributable software »".

While keeping a futuristic look and feel, it strives to maintain a certain level of functionality and to be usable in real-life scenarios, with the larger goal of bringing science-fiction UXs to the mainstream.

It might or might not be a joke taken too seriously.

Jump to:
Features — Screenshots — Questions & Answers — Download — Featured In — Contributor Instructions — Credits

Sponsor

Want to help support my open-source experiments and learn some cool JavaScript tricks at the same time?

Click the banner below and sign up to Bytes, the only newsletter cool enough to be recommended by eDEX-UI.

Features

Fully featured terminal emulator with tabs, colors, mouse events, and support for curses and curses-like applications.
Real-time system (CPU, RAM, swap, processes) and network (GeoIP, active connections, transfer rates) monitoring.
Full support for touch-enabled displays, including an on-screen keyboard.
Directory viewer that follows the CWD (current working directory) of the terminal.
Advanced customization using themes, on-screen keyboard layouts, CSS injections. See the wiki for more info.
Optional sound effects made by a talented sound designer for maximum hollywood hacking vibe.

Screenshots

neofetch on eDEX-UI 2.2 with the default "tron" theme & QWERTY keyboard

Checking out available themes in eDEX's config dir with ranger on eDEX-UI 2.2 with the "blade" theme

cmatrix on eDEX-UI 2.2 with the experimental "tron-disrupted" theme, and the user-contributed DVORAK keyboard

Editing eDEX-UI source code with nvim on eDEX-UI 2.2 with the custom horizon-full theme

Q&A

How do I get it?

Click on the little badges under the eDEX logo at the top of this page, or go to the Releases tab, or download it through one of the available repositories (Homebrew, AUR...).

Public release binaries are unsigned (why). On Linux, you will need to chmod +x the AppImage file in order to run it.

I have a problem!

Search through the Issues to see if yours has already been reported. If you're confident it hasn't been reported yet, feel free to open up a new one. If you see your issue and it's been closed, it probably means that the fix for it will ship in the next version, and you'll have to wait a bit.

Can you disable the keyboard/the filesystem display?

You can't disable them (yet) but you can hide them. See the tron-notype theme.

Why is the file browser saying that "Tracking Failed"? (Windows only)

On Linux and macOS, eDEX tracks where you're going in your terminal tab to display the content of the current folder on-screen. Sadly, this is technically impossible to do on Windows right now, so the file browser reverts back to a "detached" mode. You can still use it to browse files & directories and click on files to input their path in the terminal.

Can this run on a Raspberry Pi / ARM device?

We provide prebuilt arm64 builds. For other platforms, see this issue comment, and the thread on issue #818.

Is this repo actively maintained?

No, after a 3 years run, this project has been archived. See the announcement.

How did you make this?

Glad you're interested! See #272.

This is so cool.

Thanks! If you feel like it, you can follow me on Twitter to hear about new stuff I'm making.

Featured in...

Linux Uprising Blog
My post on r/unixporn
Korben article (in french)
Hacker News
This tweet that made me smile
BoingBoing article - Apparently i'm a "French hacker"
OReilly 4 short links
Hackaday
Developpez.com (another french link)
GitHub Blog's Release Radar November 2018
opensource.com Productive Tools for 2019
O'Reilly 4 short links (again)
LinuxLinks
Linux For Everyone (Youtube)
BestOfJS Rising Stars 2020
The Geek Freaks (Youtube/German)
JSNation Open Source Awards 2021 (Nominee - Fun Side Project of the Year)

Useful commands for the nerds

IMPORTANT NOTE: the following instructions are meant for running eDEX from the latest unoptimized, unreleased, development version. If you'd like to get stable software instead, refer to these instructions.

Starting from source:

on *nix systems (You'll need the Xcode command line tools on macOS):

clone the repository
npm run install-linux
npm run start

on Windows:

start cmd or powershell as administrator
clone the repository
npm run install-windows
npm run start

Building

Note: Due to native modules, you can only build targets for the host OS you are using.

npm install (NOT install-linux or install-windows)
npm run build-linux or build-windows or build-darwin

The script will minify the source code, recompile native dependencies and create distributable assets in the dist folder.

Getting the bleeding edge

If you're interested in running the latest in-development version but don't want to compile source code yourself, you can can get pre-built nightly binaries on GitHub Actions: click the latest commits, and download the artifacts bundle for your OS.

Credits

eDEX-UI's source code was primarily written by me, Squared. If you want to get in touch with me or find other projects I'm involved in, check out my website.

PixelyIon helped me get started with Windows compatibility and offered some precious advice when I started to work on this project seriously.

IceWolf composed the sound effects on v2.1.x and above. He makes really cool stuff, check out his music!

Thanks

Of course, eDEX would never have existed if I hadn't stumbled upon the amazing work of Seena on r/unixporn.

This project uses a bunch of open-source libraries, frameworks and tools, see the full dependency graph.

I want to namely thank the developers behind xterm.js, systeminformation and SmoothieCharts.

Huge thanks to Rob "Arscan" Scanlon for making the fantastic ENCOM Globe, also inspired by the TRON: Legacy movie, and distributing it freely. His work really puts the icing on the cake.

Licensing

Licensed under the GPLv3.0.

alexlenail/NN-SVG

2024-12-01T01:54:45Z

Publication-ready NN-architecture schematics.

NN-SVG

| Docs | Contributing

Illustrations of Neural Network architectures are often time-consuming to produce, and machine learning researchers all too often find themselves constructing these diagrams from scratch by hand.

NN-SVG is a tool for creating Neural Network (NN) architecture drawings parametrically rather than manually. It also provides the ability to export those drawings to Scalable Vector Graphics (SVG) files, suitable for inclusion in academic papers or web pages.

The tool provides the ability to generate figures of three kinds: classic Fully-Connected Neural Network (FCNN) figures, Convolutional Neural Network (CNN) figures of the sort introduced in the LeNet paper, and Deep Neural Network figures following the style introduced in the AlexNet paper. The former two are accomplished using the D3 javascript library and the latter with the javascript library Three.js. NN-SVG provides the ability to style the figure to the user's liking via many size, color, and layout parameters.

I hope this tool will save machine learning researchers time, and I hope this software might also serve as a pedagogical tool in some contexts.

Citation

LeNail, (2019). NN-SVG: Publication-Ready Neural Network Architecture Schematics.
Journal of Open Source Software, 4(33), 747, https://doi.org/10.21105/joss.00747

XengShi/materialYouNewTab

2024-12-01T01:43:45Z

A Simple New Tab ( browsers's home page ) inspired with Google's 'Material You' design

Material You NewTab

Material You NewTab is a versatile browser extension that personalizes your New Tab page with customizable themes and a welcoming message, all while integrating seamlessly with your preferred search engine.

Features • Additional Features • Currently Supported Languages • Installation Guide
Usage • Contributing • Issues and Support • Privacy Policy • License

🔴 🟡 🟢 Test live : https://xengshi.github.io/materialYouNewTab/

✨ Features

Customizable Themes: Choose from a selection of themes to match your style.
Personalized Greeting: Add a custom message or your name, so you're greeted each time you open a new tab.
Integrated Search: Search directly from the New Tab using your preferred search engine—Google, DuckDuckGo, Bing, or YouTube.
Browser Compatibility: Supports major browsers like Chrome, Edge, Firefox, Brave, and Safari.

🛠️ Additional Features

Responsive and Modern Design:
- The extension is built with a clean, modern interface inspired by Material You, offering a consistent aesthetic that’s pleasing to the eye.
- Elements are intuitively placed, ensuring ease of use and accessibility for users of all ages.
Seamless Integration with Browsers:
- The extension integrates smoothly with all major browsers, including Chrome, Edge, Brave, Firefox, and Safari, providing a cohesive experience across platforms.

🌐 Currently Supported Languages (16)

Language	Code	Translator(s)
English	en	XengShi, Prem
Bangla - বাংলা	bn	It'z RJ, Prem
Chinese (Simplified) - 中文	zh-CN	Guy Berryman, Tianli
Czech - Čeština	cz	Michal Šmahel
Hindi - हिंदी	hi	XengShi, Prem
Indonesian - Bahasa Indonesia	idn	Ayyas-RF
Italian - Italiano	it	Claudio Di Maio
Japanese - 日本語	ja	XengShi
Korean - 한국어	ko	XengShi
Marathi - मराठी	mr	Prem
Portuguese - Português	pt-BR	Maycon Vitor Correa
Russian - Русский	ru	giwi
Spanish - Español	es-ES	XengShi
Turkish - Türkçe	tr	Nobody
Uzbek - O'zbek	uz	Firdavs
Vietnamese - Tiếng Việt	vn	Diep

📥 Installation Guide

Download

Clone the Repository:
```
git clone https://github.com/XengShi/materialYouNewTab.git
```
- Alternatively, download the latest version as a ZIP file by clicking the Download ZIP button below:
Direct Download:
- On the repository page, click the Code button, then select Download ZIP from the dropdown.

Installation

Chromium-Based Browsers

Install from Chrome Web Store:
- Chrome Web Store link
Or follow the manual steps below:
Manual Installation:
- Prepare the Extension Folder:
  - Ensure you’ve cloned the repository or downloaded the ZIP file and extracted it.
- Open Extensions Page: Open your preferred browser and enter the following text in the address bar
  - Chrome: chrome://extensions
  - Edge: edge://extensions
  - Brave: brave://extensions
  - Opera: opera://extensions
- Enable Developer Mode:
  - Turn on Developer Mode in the extensions page.
- Load Unpacked Extension:
  - Click on Load unpacked and select your extracted extension folder.

Below are screenshots of the process to guide you through each step visually.

Firefox

Install from Mozilla Extensions Store:
- Mozilla Add-ons link

Or follow the manual steps below:

Manual Installation:
- Clone the repository or download and extract the ZIP file.
- Prepare the Manifest:
  - Delete the existing manifest.json file.
  - Rename manifest(firefox).json to manifest.json.
- Load Temporary Add-on:
  - Go to about:debugging#/runtime/this-firefox, select "This Firefox", then click Load Temporary Add-on.
  - Browse and select the updated manifest.json.

Safari

Clone the repository or download and extract the ZIP file.
Enable Developer Mode:
- Open Safari, go to Preferences > Advanced and enable Show Develop menu in the menu bar.
Install Extension:
- Go to Develop > Show Extension Builder, click the "+" button, and select your extension folder.
- Click Install.

Installation Video

For a step-by-step walkthrough, watch this installation guide video.

📖 Usage

Welcome Greeting:
- The extension greets you with a personalized message, like "Hello! It’z RJ," and displays the current date. This feature can be customized in the settings to show your name or preferred text, making your browsing experience feel more personal.
Clock Widget:
- A modern, animated clock is displayed prominently on the left side, keeping you aware of the current time without needing to check elsewhere.
Weather Information:
- Displays real-time weather information for your location, including temperature, weather condition (like "Sunny"), humidity, and "Feels like" temperature.
- Users can set their location (like "Dhaka" in the screenshot) to get accurate weather data. A visual icon represents the weather, and temperature is displayed in degrees Celsius.
Search Bar:
- The search bar allows you to enter queries directly from the new tab page. This makes it quick and easy to start searching without switching to a different page or window.
- Below the search bar, there are search engine options that allow you to choose your preferred search engine: Google, DuckDuckGo, Bing, Brave, or YouTube.
Quick Access Social Media Icons:
- At the bottom, there are quick-access icons for popular platforms, including YouTube, Email, Telegram, WhatsApp, Twitter, and more.
- Clicking on any of these icons will redirect you to the respective platform, allowing for easy access to your frequently used services.
AI Tools Button:
- On the bottom left, an "AI Tools" button is available, which could provide access to AI-powered tools directly from the New Tab page. This feature can be helpful for quick, AI-driven assistance or tasks.

🤝 Contributing

Contributions are welcome! If you’d like to contribute:

Fork the repository.
Create a feature branch: git checkout -b feature/YourFeature.
Commit your changes: git commit -m 'Add Your Feature'.
Push to the branch: git push origin feature/YourFeature.
Open a pull request.

❓ Issues and Support

If you encounter any bugs or issues, please submit an issue on the GitHub repository.

🔒 Privacy Policy

For information on how we handle privacy, please read our Privacy Policy.

📜 License

This project is licensed under the GNU General Public License v3.0 (GPL-3.0). See the LICENSE file for details.

GitHub @XengShi · Telegram @xengshi · Email xengshi@duck.com

metowolf/vCards

2024-12-01T01:43:45Z

📡️ vCards 中国黄页 - 优化 iOS/Android 来电、信息界面体验

vCards CN

导入常用联系人头像，优化 iOS 来电、信息界面体验。

使用指南

订阅：CardDAV 服务/ 或参考教程自建

采用订阅方式导入，优势是会自动更新，也更方便区分和管理个人通讯录和黄页，不会混合两种列表。

服务器：vcards.metowolf.com
用户名：cn
密码：cn 或任意填写

步骤：

iOS：「设置」--「通讯录」--「账户」--「添加账户」-- 「其他」--「添加 CardDAV 账户」
Mac：「通讯录」--「设置」--「账户」--「其他通讯录账户」

下载导入

到 https://github.com/metowolf/vCards/releases 下载最新的打包文件 archive.zip；
解压后，根据不同平台的指南导入 vcf 文件至 iCloud 中，推荐单独创建「黄页」分组方便管理和隐藏。

macOS

iOS/web

请求收录

打开 https://github.com/metowolf/vCards/issues/new/choose 页面，选择「vCard 新增请求」
完整填写相关信息
提交 issue，等待处理

参与维护

在 /data/类别/ 里添加 yaml 与 png 文件
在根目录下执行 yarn test 检查格式规范
提交 pull requests，等待合并

号码收录

由于不同地区不同运营商的 106 短信推送号段存在差异，项目不作收录，建议将本项目作为一个基础模板，导入联系人后可以按以下方式自行补充其余号码

图标设计

采用 PNG 编码
画布大小 width：200px；height：200px
logo 居中放置
- 圆形尺寸 140w140h
- 正矩形尺寸 120w120h
- 长矩形尺寸 160w80h
- 无 svg 需要使用 Inkscape 改绘转换
- 特殊情况特殊处理
图像大小压缩在 20 kB 内

致谢

114 百事通提供查询接口
百度手机卫士提供查询接口
中国可信号码数据中心提供查询接口

denysdovhan/wtfjs

2024-12-01T01:43:45Z

🤪 A list of funny and tricky JavaScript examples

What the f*ck JavaScript?

A list of funny and tricky JavaScript examples

JavaScript is a great language. It has a simple syntax, large ecosystem and, what is most important, a great community.

At the same time, we all know that JavaScript is quite a funny language with tricky parts. Some of them can quickly turn our everyday job into hell, and some of them can make us laugh out loud.

The original idea for WTFJS belongs to Brian Leroux. This list is highly inspired by his talk “WTFJS” at dotJS 2012:

Node Packaged Manuscript

You can install this handbook using npm. Just run:

$ npm install -g wtfjs

You should be able to run wtfjs at the command line now. This will open the manual in your selected $PAGER. Otherwise, you may continue reading on here.

The source is available here: https://github.com/denysdovhan/wtfjs

Translations

Currently, there are these translations of wtfjs:

Help translating to your language

Note: Translations are maintained by their translators. They may not contain every example, and existing examples may be outdated.

💪🏻 Motivation

Just for fun

— “Just for Fun: The Story of an Accidental Revolutionary”, Linus Torvalds

The primary goal of this list is to collect some crazy examples and explain how they work, if possible. Just because it's fun to learn something that we didn't know before.

If you are a beginner, you can use these notes to get a deeper dive into JavaScript. I hope these notes will motivate you to spend more time reading the specification.

If you are a professional developer, you can consider these examples as a great reference for all of the quirks and unexpected edges of our beloved JavaScript.

In any case, just read this. You're probably going to find something new.

⚠️ Note: If you enjoy reading this document, please, consider supporting the author of this collection.

✍🏻 Notation

// -> is used to show the result of an expression. For example:

1 + 1; // -> 2

// > means the result of console.log or another output. For example:

console.log("hello, world!"); // > hello, world!

// is just a comment used for explanations. Example:

// Assigning a function to foo constant
const foo = function() {};

👀 Examples

`[]` is equal `![]`

Array is equal not array:

[] == ![]; // -> true

💡 Explanation:

The abstract equality operator converts both sides to numbers to compare them, and both sides become the number 0 for different reasons. Arrays are truthy, so on the right, the opposite of a truthy value is false, which is then coerced to 0. On the left, however, an empty array is coerced to a number without becoming a boolean first, and empty arrays are coerced to 0, despite being truthy.

Here is how this expression simplifies:

+[] == +![];
0 == +false;
0 == 0;
true;

`true` is not equal `![]`, but not equal `[]` too

Array is not equal true, but not Array is not equal true too; Array is equal false, not Array is equal false too:

true == []; // -> false
true == ![]; // -> false

false == []; // -> true
false == ![]; // -> true

💡 Explanation:

true == []; // -> false
true == ![]; // -> false

// According to the specification

true == []; // -> false

toNumber(true); // -> 1
toNumber([]); // -> 0

1 == 0; // -> false

true == ![]; // -> false

![]; // -> false

true == false; // -> false

false == []; // -> true
false == ![]; // -> true

// According to the specification

false == []; // -> true

toNumber(false); // -> 0
toNumber([]); // -> 0

0 == 0; // -> true

false == ![]; // -> true

![]; // -> false

false == false; // -> true

7.2.15 Abstract Equality Comparison

true is false

!!"false" == !!"true"; // -> true
!!"false" === !!"true"; // -> true

💡 Explanation:

Consider this step-by-step:

// true is 'truthy' and represented by value 1 (number), 'true' in string form is NaN.
true == "true"; // -> false
false == "false"; // -> false

// 'false' is not the empty string, so it's a truthy value
!!"false"; // -> true
!!"true"; // -> true

7.2.15 Abstract Equality Comparison

baNaNa

"b" + "a" + +"a" + "a"; // -> 'baNaNa'

This is an old-school joke in JavaScript, but remastered. Here's the original one:

"foo" + +"bar"; // -> 'fooNaN'

💡 Explanation:

The expression is evaluated as 'foo' + (+'bar'), which converts 'bar' to not a number.

`NaN` is not a `NaN`

NaN === NaN; // -> false

💡 Explanation:

The specification strictly defines the logic behind this behavior:

If Type(x) is different from Type(y), return false.

If Type(x) is Number, then

If x is NaN, return false.

If y is NaN, return false.

… … …

— 7.2.14 Strict Equality Comparison

Following the definition of NaN from the IEEE:

Four mutually exclusive relations are possible: less than, equal, greater than, and unordered. The last case arises when at least one operand is NaN. Every NaN shall compare unordered with everything, including itself.

— “What is the rationale for all comparisons returning false for IEEE754 NaN values?” at StackOverflow

`Object.is()` and `===` weird cases

Object.is() determines if two values have the same value or not. It works similar to the === operator but there are a few weird cases:

Object.is(NaN, NaN); // -> true
NaN === NaN; // -> false

Object.is(-0, 0); // -> false
-0 === 0; // -> true

Object.is(NaN, 0 / 0); // -> true
NaN === 0 / 0; // -> false

💡 Explanation:

In JavaScript lingo, NaN and NaN are the same value but they're not strictly equal. NaN === NaN being false is apparently due to historical reasons so it would probably be better to accept it as it is.

Similarly, -0 and 0 are strictly equal, but they're not the same value.

For more details about NaN === NaN, see the above case.

It's a fail

You would not believe, but …

(![] + [])[+[]] +
  (![] + [])[+!+[]] +
  ([![]] + [][[]])[+!+[] + [+[]]] +
  (![] + [])[!+[] + !+[]];
// -> 'fail'

💡 Explanation:

By breaking that mass of symbols into pieces, we notice that the following pattern occurs often:

![] + []; // -> 'false'
![]; // -> false

So we try adding [] to false. But due to a number of internal function calls (binary + Operator -> ToPrimitive -> [[DefaultValue]]) we end up converting the right operand to a string:

![] + [].toString(); // 'false'

Thinking of a string as an array we can access its first character via [0]:

"false"[0]; // -> 'f'

The rest is obvious, but the i is tricky. The i in fail is grabbed by generating the string 'falseundefined' and grabbing the element on index ['10'].

More examples:

+![]          // -> 0
+!![]         // -> 1
!![]          // -> true
![]           // -> false
[][[]]        // -> undefined
+!![] / +![]  // -> Infinity
[] + {}       // -> "[object Object]"
+{}           // -> NaN

Brainfuck beware: JavaScript is after you!
Writing a sentence without using the Alphabet — generate any phrase using JavaScript

`[]` is truthy, but not `true`

An array is a truthy value, however, it's not equal to true.

!![]       // -> true
[] == true // -> false

💡 Explanation:

Here are links to the corresponding sections in the ECMA-262 specification:

`null` is falsy, but not `false`

Despite the fact that null is a falsy value, it's not equal to false.

!!null; // -> false
null == false; // -> false

At the same time, other falsy values, like 0 or '' are equal to false.

0 == false; // -> true
"" == false; // -> true

💡 Explanation:

The explanation is the same as for previous example. Here's the corresponding link:

7.2.15 Abstract Equality Comparison

`document.all` is an object, but it is undefined

⚠️ This is part of the Browser API and won't work in a Node.js environment ⚠️

Despite the fact that document.all is an array-like object and it gives access to the DOM nodes in the page, it responds to the typeof function as undefined.

document.all instanceof Object; // -> true
typeof document.all; // -> 'undefined'

At the same time, document.all is not equal to undefined.

document.all === undefined; // -> false
document.all === null; // -> false

But at the same time:

document.all == null; // -> true

💡 Explanation:

document.all used to be a way to access DOM elements, in particular with old versions of IE. While it has never been a standard it was broadly used in the old age JS code. When the standard progressed with new APIs (such as document.getElementById) this API call became obsolete and the standard committee had to decide what to do with it. Because of its broad use they decided to keep the API but introduce a willful violation of the JavaScript specification. The reason why it responds to false when using the Strict Equality Comparison with undefined while true when using the Abstract Equality Comparison is due to the willful violation of the specification that explicitly allows that.

— “Obsolete features - document.all” at WhatWG - HTML spec — “Chapter 4 - ToBoolean - Falsy values” at YDKJS - Types & Grammar

Minimal value is greater than zero

Number.MIN_VALUE is the smallest number, which is greater than zero:

Number.MIN_VALUE > 0; // -> true

💡 Explanation:

Number.MIN_VALUE is 5e-324, i.e. the smallest positive number that can be represented within float precision, i.e. that's as close as you can get to zero. It defines the best resolution that floats can give you.

Now the overall smallest value is Number.NEGATIVE_INFINITY although it's not really numeric in a strict sense.

— “Why is 0 less than Number.MIN_VALUE in JavaScript?” at StackOverflow

20.1.2.9 Number.MIN_VALUE

function is not a function

⚠️ A bug present in V8 v5.5 or lower (Node.js <=7) ⚠️

All of you know about the annoying undefined is not a function, but what about this?

// Declare a class which extends null
class Foo extends null {}
// -> [Function: Foo]

new Foo() instanceof null;
// > TypeError: function is not a function
// >     at … … …

💡 Explanation:

This is not a part of the specification. It's just a bug that has now been fixed, so there shouldn't be a problem with it in the future.

Super constructor null of Foo is not a constructor

It's continuation of story with previous bug in modern environment (tested with Chrome 71 and Node.js v11.8.0).

class Foo extends null {}
new Foo() instanceof null;
// > TypeError: Super constructor null of Foo is not a constructor

💡 Explanation:

This is not a bug because:

Object.getPrototypeOf(Foo.prototype); // -> null

If the class has no constructor the call from prototype chain. But in the parent has no constructor. Just in case, I’ll clarify that null is an object:

typeof null === "object";

Therefore, you can inherit from it (although in the world of the OOP for such terms would have beaten me). So you can't call the null constructor. If you change this code:

class Foo extends null {
  constructor() {
    console.log("something");
  }
}

You see the error:

ReferenceError: Must call super constructor in derived class before accessing 'this' or returning from derived constructor

And if you add super:

class Foo extends null {
  constructor() {
    console.log(111);
    super();
  }
}

JS throws an error:

TypeError: Super constructor null of Foo is not a constructor

An explanation of this issue by @geekjob

Adding arrays

What if you try to add two arrays?

[1, 2, 3] + [4, 5, 6]; // -> '1,2,34,5,6'

💡 Explanation:

The concatenation happens. Step-by-step, it looks like this:

[1, 2, 3] +
  [4, 5, 6][
    // call toString()
    (1, 2, 3)
  ].toString() +
  [4, 5, 6].toString();
// concatenation
"1,2,3" + "4,5,6";
// ->
("1,2,34,5,6");

Trailing commas in array

You've created an array with 4 empty elements. Despite all, you'll get an array with three elements, because of trailing commas:

let a = [, , ,];
a.length; // -> 3
a.toString(); // -> ',,'

💡 Explanation:

Trailing commas (sometimes called "final commas") can be useful when adding new elements, parameters, or properties to JavaScript code. If you want to add a new property, you can simply add a new line without modifying the previously last line if that line already uses a trailing comma. This makes version-control diffs cleaner and editing code might be less troublesome.

— Trailing commas at MDN

Array equality is a monster

Array equality is a monster in JS, as you can see below:

[] == ''   // -> true
[] == 0    // -> true
[''] == '' // -> true
[0] == 0   // -> true
[0] == ''  // -> false
[''] == 0  // -> true

[null] == ''      // true
[null] == 0       // true
[undefined] == '' // true
[undefined] == 0  // true

[[]] == 0  // true
[[]] == '' // true

[[[[[[]]]]]] == '' // true
[[[[[[]]]]]] == 0  // true

[[[[[[ null ]]]]]] == 0  // true
[[[[[[ null ]]]]]] == '' // true

[[[[[[ undefined ]]]]]] == 0  // true
[[[[[[ undefined ]]]]]] == '' // true

💡 Explanation:

You should watch very carefully for the above examples! The behaviour is described in section 7.2.15 Abstract Equality Comparison of the specification.

`undefined` and `Number`

If we don't pass any arguments into the Number constructor, we'll get 0. The value undefined is assigned to formal arguments when there are no actual arguments, so you might expect that Number without arguments takes undefined as a value of its parameter. However, when we pass undefined, we will get NaN.

Number(); // -> 0
Number(undefined); // -> NaN

💡 Explanation:

According to the specification:

If no arguments were passed to this function's invocation, let n be +0.
Else, let n be ? ToNumber(value).
In case of undefined, ToNumber(undefined) should return NaN.

Here's the corresponding section:

`parseInt` is a bad guy

parseInt is famous by its quirks:

parseInt("f*ck"); // -> NaN
parseInt("f*ck", 16); // -> 15

💡 Explanation: This happens because parseInt will continue parsing character-by-character until it hits a character it doesn't know. The f in 'f*ck' is the hexadecimal digit 15.

Parsing Infinity to integer is something…

//
parseInt("Infinity", 10); // -> NaN
// ...
parseInt("Infinity", 18); // -> NaN...
parseInt("Infinity", 19); // -> 18
// ...
parseInt("Infinity", 23); // -> 18...
parseInt("Infinity", 24); // -> 151176378
// ...
parseInt("Infinity", 29); // -> 385849803
parseInt("Infinity", 30); // -> 13693557269
// ...
parseInt("Infinity", 34); // -> 28872273981
parseInt("Infinity", 35); // -> 1201203301724
parseInt("Infinity", 36); // -> 1461559270678...
parseInt("Infinity", 37); // -> NaN

Be careful with parsing null too:

parseInt(null, 24); // -> 23

💡 Explanation:

It's converting null to the string "null" and trying to convert it. For radixes 0 through 23, there are no numerals it can convert, so it returns NaN. At 24, "n", the 14th letter, is added to the numeral system. At 31, "u", the 21st letter, is added and the entire string can be decoded. At 37 on there is no longer any valid numeral set that can be generated and NaN is returned.

— “parseInt(null, 24) === 23… wait, what?” at StackOverflow

Don't forget about octals:

parseInt("06"); // 6
parseInt("08"); // 8 if support ECMAScript 5
parseInt("08"); // 0 if not support ECMAScript 5

💡 Explanation: If the input string begins with "0", radix is eight (octal) or 10 (decimal). Exactly which radix is chosen is implementation-dependent. ECMAScript 5 specifies that 10 (decimal) is used, but not all browsers support this yet. For this reason always specify a radix when using parseInt.

parseInt always convert input to string:

parseInt({ toString: () => 2, valueOf: () => 1 }); // -> 2
Number({ toString: () => 2, valueOf: () => 1 }); // -> 1

Be careful while parsing floating point values

parseInt(0.000001); // -> 0
parseInt(0.0000001); // -> 1
parseInt(1 / 1999999); // -> 5

💡 Explanation: ParseInt takes a string argument and returns an integer of the specified radix. ParseInt also strips anything after and including the first non-digit in the string parameter. 0.000001 is converted to a string "0.000001" and the parseInt returns 0. When 0.0000001 is converted to a string it is treated as "1e-7" and hence parseInt returns 1. 1/1999999 is interpreted as 5.00000250000125e-7 and parseInt returns 5.

Math with `true` and `false`

Let's do some math:

true + true; // -> 2
(true + true) * (true + true) - true; // -> 3

Hmmm… 🤔

💡 Explanation:

We can coerce values to numbers with the Number constructor. It's quite obvious that true will be coerced to 1:

Number(true); // -> 1

The unary plus operator attempts to convert its value into a number. It can convert string representations of integers and floats, as well as the non-string values true, false, and null. If it cannot parse a particular value, it will evaluate to NaN. That means we can coerce true to 1 easier:

+true; // -> 1

When you're performing addition or multiplication, the ToNumber method is invoked. According to the specification, this method returns:

If argument is true, return 1. If argument is false, return +0.

That's why we can add boolean values as regular numbers and get correct results.

Corresponding sections:

HTML comments are valid in JavaScript

You will be impressed, but <!-- (which is known as HTML comment) is a valid comment in JavaScript.

// valid comment
<!-- valid comment too

💡 Explanation:

Impressed? HTML-like comments were intended to allow browsers that didn't understand the <script> tag to degrade gracefully. These browsers, e.g. Netscape 1.x are no longer popular. So there is really no point in putting HTML comments in your script tags anymore.

Since Node.js is based on the V8 engine, HTML-like comments are supported by the Node.js runtime too. Moreover, they're a part of the specification:

B.1.3 HTML-like Comments

`NaN` is not a number

Type of NaN is a 'number':

typeof NaN; // -> 'number'

💡 Explanation:

Explanations of how typeof and instanceof operators work:

`[]` and `null` are objects

typeof []; // -> 'object'
typeof null; // -> 'object'

// however
null instanceof Object; // false

💡 Explanation:

The behavior of typeof operator is defined in this section of the specification:

13.5.3 The typeof Operator

According to the specification, the typeof operator returns a string according to Table 37: typeof Operator Results. For null, ordinary, standard exotic and non-standard exotic objects, which do not implement [[Call]], it returns the string "object".

However, you can check the type of an object by using the toString method.

Object.prototype.toString.call([]);
// -> '[object Array]'

Object.prototype.toString.call(new Date());
// -> '[object Date]'

Object.prototype.toString.call(null);
// -> '[object Null]'

Magically increasing numbers

999999999999999; // -> 999999999999999
9999999999999999; // -> 10000000000000000

10000000000000000; // -> 10000000000000000
10000000000000000 + 1; // -> 10000000000000000
10000000000000000 + 1.1; // -> 10000000000000002

💡 Explanation:

This is caused by IEEE 754-2008 standard for Binary Floating-Point Arithmetic. At this scale, it rounds to the nearest even number. Read more:

6.1.6 The Number Type
IEEE 754 on Wikipedia

Precision of `0.1 + 0.2`

A well-known joke. An addition of 0.1 and 0.2 is deadly precise:

0.1 + 0.2; // -> 0.30000000000000004
0.1 + 0.2 === 0.3; // -> false

💡 Explanation:

The answer for the ”Is floating point math broken?” question on StackOverflow:

The constants 0.2 and 0.3 in your program will also be approximations to their true values. It happens that the closest double to 0.2 is larger than the rational number 0.2 but that the closest double to 0.3 is smaller than the rational number 0.3. The sum of 0.1 and 0.2 winds up being larger than the rational number 0.3 and hence disagreeing with the constant in your code.

This problem is so known that there is even a website called 0.30000000000000004.com. It occurs in every language that uses floating-point math, not just JavaScript.

Patching numbers

You can add your own methods to wrapper objects like Number or String.

Number.prototype.isOne = function() {
  return Number(this) === 1;
};

(1.0).isOne(); // -> true
(1).isOne(); // -> true
(2.0).isOne(); // -> false
(7).isOne(); // -> false

💡 Explanation:

Obviously, you can extend the Number object like any other object in JavaScript. However, it's not recommended if the behavior of the defined method is not a part of the specification. Here is the list of Number's properties:

20.1 Number Objects

Comparison of three numbers

1 < 2 < 3; // -> true
3 > 2 > 1; // -> false

💡 Explanation:

Why does this work that way? Well, the problem is in the first part of an expression. Here's how it works:

1 < 2 < 3; // 1 < 2 -> true
true < 3; // true -> 1
1 < 3; // -> true

3 > 2 > 1; // 3 > 2 -> true
true > 1; // true -> 1
1 > 1; // -> false

We can fix this with Greater than or equal operator (>=):

3 > 2 >= 1; // true

Read more about Relational operators in the specification:

12.10 Relational Operators

Funny math

Often the results of arithmetic operations in JavaScript might be quite unexpected. Consider these examples:

 3  - 1  // -> 2
 3  + 1  // -> 4
'3' - 1  // -> 2
'3' + 1  // -> '31'

'' + '' // -> ''
[] + [] // -> ''
{} + [] // -> 0
[] + {} // -> '[object Object]'
{} + {} // -> '[object Object][object Object]'

'222' - -'111' // -> 333

[4] * [4]       // -> 16
[] * []         // -> 0
[4, 4] * [4, 4] // NaN

💡 Explanation:

What's happening in the first four examples? Here's a small table to understand addition in JavaScript:

Number  + Number  -> addition
Boolean + Number  -> addition
Boolean + Boolean -> addition
Number  + String  -> concatenation
String  + Boolean -> concatenation
String  + String  -> concatenation

What about other examples? A ToPrimitive and ToString methods are being implicitly called for [] and {} before addition. Read more about evaluation process in the specification:

Notably, {} + [] here is the exception. The reason why it differs from [] + {} is that, without parenthesis, it is interpreted as a code block and then a unary +, converting [] into a number. It sees the following:

{
  // a code block here
}
+[]; // -> 0

To get the same output as [] + {} we can wrap it in parenthesis.

({} + []); // -> [object Object]

Addition of RegExps

Did you know you can add numbers like this?

// Patch a toString method
RegExp.prototype.toString =
  function() {
    return this.source;
  } /
  7 /
  -/5/; // -> 2

💡 Explanation:

21.2.5.10 get RegExp.prototype.source

Strings aren't instances of `String`

"str"; // -> 'str'
typeof "str"; // -> 'string'
"str" instanceof String; // -> false

💡 Explanation:

The String constructor returns a string:

typeof String("str"); // -> 'string'
String("str"); // -> 'str'
String("str") == "str"; // -> true

Let's try with a new:

new String("str") == "str"; // -> true
typeof new String("str"); // -> 'object'

Object? What's that?

new String("str"); // -> [String: 'str']

More information about the String constructor in the specification:

21.1.1 The String Constructor

Calling functions with backticks

Let's declare a function which logs all params into the console:

function f(...args) {
  return args;
}

No doubt, you know you can call this function like this:

f(1, 2, 3); // -> [ 1, 2, 3 ]

But did you know you can call any function with backticks?

f`true is ${true}, false is ${false}, array is ${[1, 2, 3]}`;
// -> [ [ 'true is ', ', false is ', ', array is ', '' ],
// ->   true,
// ->   false,
// ->   [ 1, 2, 3 ] ]

💡 Explanation:

Well, this is not magic at all if you're familiar with Tagged template literals. In the example above, f function is a tag for template literal. Tags before template literal allow you to parse template literals with a function. The first argument of a tag function contains an array of string values. The remaining arguments are related to the expressions. Example:

function template(strings, ...keys) {
  // do something with strings and keys…
}

This is the magic behind famous library called 💅 styled-components, which is popular in the React community.

Link to the specification:

12.3.7 Tagged Templates

Call call call

Found by @cramforce

console.log.call.call.call.call.call.apply(a => a, [1, 2]);

💡 Explanation:

Attention, it could break your mind! Try to reproduce this code in your head: we're applying the call method using the apply method. Read more:

A `constructor` property

const c = "constructor";
c[c][c]('console.log("WTF?")')(); // > WTF?

💡 Explanation:

Let's consider this example step-by-step:

// Declare a new constant which is a string 'constructor'
const c = "constructor";

// c is a string
c; // -> 'constructor'

// Getting a constructor of string
c[c]; // -> [Function: String]

// Getting a constructor of constructor
c[c][c]; // -> [Function: Function]

// Call the Function constructor and pass
// the body of new function as an argument
c[c][c]('console.log("WTF?")'); // -> [Function: anonymous]

// And then call this anonymous function
// The result is console-logging a string 'WTF?'
c[c][c]('console.log("WTF?")')(); // > WTF?

An Object.prototype.constructor returns a reference to the Object constructor function that created the instance object. In case with strings it is String, in case with numbers it is Number and so on.

Object as a key of object's property

{ [{}]: {} } // -> { '[object Object]': {} }

💡 Explanation:

Why does this work so? Here we're using a Computed property name. When you pass an object between those brackets, it coerces object to a string, so we get the property key '[object Object]' and the value {}.

We can make "brackets hell" like this:

({ [{}]: { [{}]: {} } }[{}][{}]); // -> {}

// structure:
// {
//   '[object Object]': {
//     '[object Object]': {}
//   }
// }

Accessing prototypes with `proto`

As we know, primitives don't have prototypes. However, if we try to get a value of __proto__ for primitives, we would get this:

(1).__proto__.__proto__.__proto__; // -> null

💡 Explanation:

This happens because when something doesn't have a prototype, it will be wrapped into a wrapper object using the ToObject method. So, step-by-step:

(1).__proto__; // -> [Number: 0]
(1).__proto__.__proto__; // -> {}
(1).__proto__.__proto__.__proto__; // -> null

Here is more information about __proto__:

`${{Object}}`

What is the result of the expression below?

`${{ Object }}`;

The answer is:

// -> '[object Object]'

💡 Explanation:

We defined an object with a property Object using Shorthand property notation:

{
  Object: Object;
}

Then we've passed this object to the template literal, so the toString method calls for that object. That's why we get the string '[object Object]'.

12.2.9 Template Literals
Object initializer at MDN

Destructuring with default values

Consider this example:

let x,
  { x: y = 1 } = { x };
y;

The example above is a great task for an interview. What the value of y? The answer is:

// -> 1

💡 Explanation:

let x,
  { x: y = 1 } = { x };
y;
//  ↑       ↑           ↑    ↑
//  1       3           2    4

With the example above:

We declare x with no value, so it's undefined.
Then we pack the value of x into the object property x.
Then we extract the value of x using destructuring and want to assign it to y. If the value is not defined, then we're going to use 1 as the default value.
Return the value of y.

Object initializer at MDN

Dots and spreading

Interesting examples could be composed with spreading of arrays. Consider this:

[...[..."..."]].length; // -> 3

💡 Explanation:

Why 3? When we use the spread operator, the @@iterator method is called, and the returned iterator is used to obtain the values to be iterated. The default iterator for string spreads a string into characters. After spreading, we pack these characters into an array. Then we spread this array again and pack it back to an array.

A '...' string consists with three . characters, so the length of resulting array is 3.

Now, step-by-step:

[...'...']             // -> [ '.', '.', '.' ]
[...[...'...']]        // -> [ '.', '.', '.' ]
[...[...'...']].length // -> 3

Obviously, we can spread and wrap the elements of an array as many times as we want:

[...'...']                 // -> [ '.', '.', '.' ]
[...[...'...']]            // -> [ '.', '.', '.' ]
[...[...[...'...']]]       // -> [ '.', '.', '.' ]
[...[...[...[...'...']]]]  // -> [ '.', '.', '.' ]
// and so on …

Labels

Not many programmers know about labels in JavaScript. They are kind of interesting:

foo: {
  console.log("first");
  break foo;
  console.log("second");
}

// > first
// -> undefined

💡 Explanation:

The labeled statement is used with break or continue statements. You can use a label to identify a loop, and then use the break or continue statements to indicate whether a program should interrupt the loop or continue its execution.

In the example above, we identify a label foo. After that console.log('first'); executes and then we interrupt the execution.

Nested labels

a: b: c: d: e: f: g: 1, 2, 3, 4, 5; // -> 5

💡 Explanation:

Similar to previous examples, follow these links:

Insidious `try..catch`

What will this expression return? 2 or 3?

(() => {
  try {
    return 2;
  } finally {
    return 3;
  }
})();

The answer is 3. Surprised?

💡 Explanation:

13.15 The try Statement

Is this multiple inheritance?

Take a look at the example below:

new class F extends (String, Array) {}(); // -> F []

Is this a multiple inheritance? Nope.

💡 Explanation:

The interesting part is the value of the extends clause ((String, Array)). The grouping operator always returns its last argument, so (String, Array) is actually just Array. That means we've just created a class which extends Array.

A generator which yields itself

Consider this example of a generator which yields itself:

(function* f() {
  yield f;
})().next();
// -> { value: [GeneratorFunction: f], done: false }

As you can see, the returned value is an object with its value equal to f. In that case, we can do something like this:

(function* f() {
  yield f;
})()
  .next()
  .value()
  .next()(
    // -> { value: [GeneratorFunction: f], done: false }

    // and again
    function* f() {
      yield f;
    }
  )()
  .next()
  .value()
  .next()
  .value()
  .next()(
    // -> { value: [GeneratorFunction: f], done: false }

    // and again
    function* f() {
      yield f;
    }
  )()
  .next()
  .value()
  .next()
  .value()
  .next()
  .value()
  .next();
// -> { value: [GeneratorFunction: f], done: false }

// and so on
// …

💡 Explanation:

To understand why this works that way, read these sections of the specification:

A class of class

Consider this obfuscated syntax playing:

typeof new class {
  class() {}
}(); // -> 'object'

It seems like we're declaring a class inside of class. Should be an error, however, we get the string 'object'.

💡 Explanation:

Since ECMAScript 5 era, keywords are allowed as property names. So think about it as this simple object example:

const foo = {
  class: function() {}
};

And ES6 standardized shorthand method definitions. Also, classes can be anonymous. So if we drop : function part, we're going to get:

class {
  class() {}
}

The result of a default class is always a simple object. And its typeof should return 'object'.

Non-coercible objects

With well-known symbols, there's a way to get rid of type coercion. Take a look:

function nonCoercible(val) {
  if (val == null) {
    throw TypeError("nonCoercible should not be called with null or undefined");
  }

  const res = Object(val);

  res[Symbol.toPrimitive] = () => {
    throw TypeError("Trying to coerce non-coercible object");
  };

  return res;
}

Now we can use this like this:

// objects
const foo = nonCoercible({ foo: "foo" });

foo * 10; // -> TypeError: Trying to coerce non-coercible object
foo + "evil"; // -> TypeError: Trying to coerce non-coercible object

// strings
const bar = nonCoercible("bar");

bar + "1"; // -> TypeError: Trying to coerce non-coercible object
bar.toString() + 1; // -> bar1
bar === "bar"; // -> false
bar.toString() === "bar"; // -> true
bar == "bar"; // -> TypeError: Trying to coerce non-coercible object

// numbers
const baz = nonCoercible(1);

baz == 1; // -> TypeError: Trying to coerce non-coercible object
baz === 1; // -> false
baz.valueOf() === 1; // -> true

💡 Explanation:

Tricky arrow functions

Consider the example below:

let f = () => 10;
f(); // -> 10

Okay, fine, but what about this:

let f = () => {};
f(); // -> undefined

💡 Explanation:

You might expect {} instead of undefined. This is because the curly braces are part of the syntax of the arrow functions, so f will return undefined. It is however possible to return the {} object directly from an arrow function, by enclosing the return value with brackets.

let f = () => ({});
f(); // -> {}

Arrow functions can not be a constructor

Consider the example below:

let f = function() {
  this.a = 1;
};
new f(); // -> f { 'a': 1 }

Now, try do to the same with an arrow function:

let f = () => {
  this.a = 1;
};
new f(); // -> TypeError: f is not a constructor

💡 Explanation:

Arrow functions cannot be used as constructors and will throw an error when used with new. Because they have a lexical this, and do not have a prototype property, so it would not make much sense.

`arguments` and arrow functions

Consider the example below:

let f = function() {
  return arguments;
};
f("a"); // -> { '0': 'a' }

Now, try do to the same with an arrow function:

let f = () => arguments;
f("a"); // -> Uncaught ReferenceError: arguments is not defined

💡 Explanation:

Arrow functions are a lightweight version of regular functions with a focus on being short and lexical this. At the same time arrow functions do not provide a binding for the arguments object. As a valid alternative use the rest parameters to achieve the same result:

let f = (...args) => args;
f("a");

Arrow functions at MDN.

Tricky return

return statement is also tricky. Consider this:

(function() {
  return
  {
    b: 10;
  }
})(); // -> undefined

💡 Explanation:

return and the returned expression must be in the same line:

(function() {
  return {
    b: 10
  };
})(); // -> { b: 10 }

This is because of a concept called Automatic Semicolon Insertion, which automagically inserts semicolons after most newlines. In the first example, there is a semicolon inserted between the return statement and the object literal, so the function returns undefined and the object literal is never evaluated.

Chaining assignments on object

var foo = { n: 1 };
var bar = foo;

foo.x = foo = { n: 2 };

foo.x; // -> undefined
foo; // -> {n: 2}
bar; // -> {n: 1, x: {n: 2}}

From right to left, {n: 2} is assigned to foo, and the result of this assignment {n: 2} is assigned to foo.x, that's why bar is {n: 1, x: {n: 2}} as bar is a reference to foo. But why foo.x is undefined while bar.x is not ?

💡 Explanation:

Foo and bar references the same object {n: 1}, and lvalues are resolved before assignations. foo = {n: 2} is creating a new object, and so foo is updated to reference that new object. The trick here is foo in foo.x = ... as a lvalue was resolved beforehand and still reference the old foo = {n: 1} object and update it by adding the x value. After that chain assignments, bar still reference the old foo object, but foo reference the new {n: 2} object, where x is not existing.

It's equivalent to:

var foo = { n: 1 };
var bar = foo;

foo = { n: 2 }; // -> {n: 2}
bar.x = foo; // -> {n: 1, x: {n: 2}}
// bar.x point to the address of the new foo object
// it's not equivalent to: bar.x = {n: 2}

Accessing object properties with arrays

var obj = { property: 1 };
var array = ["property"];

obj[array]; // -> 1

// this also works with nested arrays
var nestedArray = [[[[[[[[[["property"]]]]]]]]]];
obj[nestedArray]; // -> 1

What about pseudo-multidimensional arrays?

var map = {};
var x = 1;
var y = 2;
var z = 3;

map[[x, y, z]] = true;
map[[x + 10, y, z]] = true;

map["1,2,3"]; // -> true
map["11,2,3"]; // -> true

💡 Explanation:

The brackets [] operator converts the passed expression using toString. Converting a one-element array to a string is akin to converting the contained element to the string:

["property"].toString(); // -> 'property'

`Number.toFixed()` display different numbers

Number.toFixed() can behave a bit strange in different browsers. Check out this example:

(0.7875).toFixed(3);
// Firefox: -> 0.787
// Chrome: -> 0.787
// IE11: -> 0.788
(0.7876).toFixed(3);
// Firefox: -> 0.788
// Chrome: -> 0.788
// IE11: -> 0.788

💡 Explanation:

While your first instinct may be that IE11 is correct and Firefox/Chrome are wrong, the reality is that Firefox/Chrome are more directly obeying standards for numbers (IEEE-754 Floating Point), while IE11 is minutely disobeying them in (what is probably) an effort to give clearer results.

You can see why this occurs with a few quick tests:

// Confirm the odd result of rounding a 5 down
(0.7875).toFixed(3); // -> 0.787
// It looks like it's just a 5 when you expand to the
// limits of 64-bit (double-precision) float accuracy
(0.7875).toFixed(14); // -> 0.78750000000000
// But what if you go beyond the limit?
(0.7875).toFixed(20); // -> 0.78749999999999997780

Floating point numbers are not stored as a list of decimal digits internally, but through a more complicated methodology that produces tiny inaccuracies that are usually rounded away by toString and similar calls, but are actually present internally.

In this case, that "5" on the end was actually an extremely tiny fraction below a true 5. Rounding it at any reasonable length will render it as a 5... but it is actually not a 5 internally.

IE11, however, will report the value input with only zeros appended to the end even in the toFixed(20) case, as it seems to be forcibly rounding the value to reduce the troubles from hardware limits.

See for reference NOTE 2 on the ECMA-262 definition for toFixed.

20.1.3.3 Number.prototype.toFixed (fractionDigits)

`Math.max()` less than `Math.min()`

I find this example hilarious:

Math.min() > Math.max(); // -> true
Math.min() < Math.max(); // -> false

💡 Explanation:

This is a simple one. Let's consider each part of this expression separately:

Math.min(); // -> Infinity
Math.max(); // -> -Infinity
Infinity > -Infinity; // -> true

Why so? Well, Math.max() is not the same thing as Number.MAX_VALUE. It does not return the largest possible number.

Math.max takes arguments, tries to convert the to numbers, compares each one and then returns the largest remaining. If no arguments are given, the result is −∞. If any value is NaN, the result is NaN.

The opposite is happening for Math.min. Math.min returns ∞, if no arguments are given.

15.8.2.11 Math.max
15.8.2.11 Math.min
Why is Math.max() less than Math.min()? by Charlie Harvey

Comparing `null` to `0`

The following expressions seem to introduce a contradiction:

null == 0; // -> false
null > 0; // -> false
null >= 0; // -> true

How can null be neither equal to nor greater than 0, if null >= 0 is actually true? (This also works with less than in the same way.)

💡 Explanation:

The way these three expressions are evaluated are all different and are responsible for producing this unexpected behavior.

First, the abstract equality comparison null == 0. Normally, if this operator can't compare the values on either side properly, it converts both to numbers and compares the numbers. Then, you might expect the following behavior:

// This is not what happens
(null == 0 + null) == +0;
0 == 0;
true;

However, according to a close reading of the spec, the number conversion doesn't actually happen on a side that is null or undefined. Therefore, if you have null on one side of the equal sign, the other side must be null or undefined for the expression to return true. Since this is not the case, false is returned.

Next, the relational comparison null > 0. The algorithm here, unlike that of the abstract equality operator, will convert null to a number. Therefore, we get this behavior:

null > 0
+null = +0
0 > 0
false

Finally, the relational comparison null >= 0. You could argue that this expression should be the result of null > 0 || null == 0; if this were the case, then the above results would mean that this would also be false. However, the >= operator in fact works in a very different way, which is basically to take the opposite of the < operator. Because our example with the greater than operator above also holds for the less than operator, that means this expression is actually evaluated like so:

null >= 0;
!(null < 0);
!(+null < +0);
!(0 < 0);
!false;
true;

Same variable redeclaration

JS allows to redeclare variables:

a;
a;
// This is also valid
a, a;

Works also in strict mode:

var a, a, a;
var a;
var a;

💡 Explanation:

All definitions are merged into one definition.

13.3.2 Variable Statement

Default behavior Array.prototype.sort()

Imagine that you need to sort an array of numbers.

[10, 1, 3].sort(); // -> [ 1, 10, 3 ]

💡 Explanation:

The default sort order is built upon converting the elements into strings, then comparing their sequences of UTF-16 code units values.

22.1.3.25 Array.prototype.sort ( comparefn )

Hint

Pass compareFn if you try to sort anything but string.

[10, 1, 3].sort((a, b) => a - b); // -> [ 1, 3, 10 ]

resolve() won't return Promise instance

const theObject = {
  a: 7
};
const thePromise = new Promise((resolve, reject) => {
  resolve(theObject);
}); // Promise instance object

thePromise.then(value => {
  console.log(value === theObject); // > true
  console.log(value); // > { a: 7 }
});

The value which is resolved from thePromise is exactly theObject.

How about input another Promise into the resolve function?

const theObject = new Promise((resolve, reject) => {
  resolve(7);
}); // Promise instance object
const thePromise = new Promise((resolve, reject) => {
  resolve(theObject);
}); // Promise instance object

thePromise.then(value => {
  console.log(value === theObject); // > false
  console.log(value); // > 7
});

💡 Explanation:

This function flattens nested layers of promise-like objects (e.g. a promise that resolves to a promise that resolves to something) into a single layer.

Promise.resolve() on MDN

The specification is ECMAScript 25.6.1.3.2 Promise Resolve Functions. But it is not quite human-friendly.

`{}{}` is undefined

Write them in the console. They will return the value defined in the last object.

{}{}; // -> undefined
{}{}{}; // -> undefined
{}{}{}{}; // -> undefined
{foo: 'bar'}{}; // -> 'bar'
{}{foo: 'bar'}; // -> 'bar'
{}{foo: 'bar'}{}; // -> 'bar'
{a: 'b'}{c:' d'}{}; // -> 'd'
{a: 'b', c: 'd'}{}; // > SyntaxError: Unexpected token ':'
({}{}); // > SyntaxError: Unexpected token '{'

💡 Explanation:

When inspecting each {}, they returns undefined. If you inspect {foo: 'bar'}{}, you will find {foo: 'bar'} is 'bar'.

There are two meanings for {}: an object or a block. For example, the {} in () => {} means block. So we need to use () => ({}) to return an object.

Let's use {foo: 'bar'} as a block. Write this snippet in your console:

if (true) {
  foo: "bar";
} // -> 'bar'

Surprisingly, it behaviors the same! You can guess here that {foo: 'bar'}{} is a block.

`arguments` binding

Consider this function:

function a(x) {
  arguments[0] = "hello";
  console.log(x);
}

a(); // > undefined
a(1); // > "hello"

💡 Explanation:

arguments is an Array-like object that contains the values of the arguments passed to that function. When no arguments are passed, then there's no x to override.

The arguments object on MDN

An `alert` from hell

This on is literally from hell:

[666]["\155\141\160"]["\143\157\156\163\164\162\165\143\164\157\162"](
  "\141\154\145\162\164(666)"
)(666); // alert(666)

💡 Explanation:

This one is based on octal escape sequences and multiple strings.

Any character with a character code lower than 256 (i.e. any character in the extended ASCII range) can be escaped using its octal-encoded character code, prefixed with \. An example above is basically and alert ecoded by octal escape sequances.

An infinite timeout

Guess what would happen if we set an infinite timeout?

setTimeout(() => console.log("called"), Infinity); // -> <timeoutId>
// > 'called'

It will executed immediately instead of infinity delay.

💡 Explanation:

Usually, runtime stores the delay as a 32-bit signed integer internally. This causes an integer overflow, resulting in the timeout being executed immediately.

For example, in Node.js we will get this warning:

(node:1731) TimeoutOverflowWarning: Infinity does not fit into a 32-bit signed integer.
Timeout duration was set to 1.
(Use `node --trace-warnings ...` to show where the warning was created)

WindowOrWorkerGlobalScope.setTimeout() on MDN
Node.js Documentation on Timers
Timers on W3C

A `setTimeout` object

Guess what would happen if we set an callback that's not a function to setTimeout?

setTimeout(123, 100); // -> <timeoutId>
// > 'called'

This is fine.

setTimeout('{a: 1}', 100); // -> <timeoutId>
// > 'called'

This is also fine.

setTimeout({a: 1}, 100); // -> <timeoutId>
// > 'Uncaught SyntaxError: Unexpected identifier               setTimeout (async) (anonymous) @ VM__:1'

This throws an SyntaxError.

Note that this can easily happen if your function returns an object and you call it here instead of passing it! What if the content - policy is set to self?

setTimeout(123, 100); // -> <timeoutId>
// > console.error("[Report Only] Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'report-sample' 'self' ")

The console refuses to run it at all!

💡 Explanation:

WindowOrWorkerGlobalScope.setTimeout() can be called with code as first argument, which will be passed on to eval, which is bad. Eval will coerce her input to String, and evaluate what is produced, so Objects becomes '[object Object]' which has hmmm ... an 'Unexpected identifier'!

eval() on MDN (don't use this)
WindowOrWorkerGlobalScope.setTimeout() on MDN
Content Security Policy
Timers on W3C

Double dot

Let's try to coerce a number to a string:

27.toString() // > Uncaught SyntaxError: Invalid or unexpected token

Maybe we should try with two dots?

27..toString(); // -> '27'

But why doesn't first example work?

💡 Explanation:

It's just a language grammar limitation.

The . character presents an ambiguity. It can be understood to be the member operator, or a decimal, depending on its placement.

The specification's interpretation of the . character in that particular position is that it will be a decimal. This is defined by the numeric literal syntax of ECMAScript.

You must always use parenthesis or an addition dot to make such expression valid.

(27).toString(); // -> '27'
// or
27..toString(); // -> '27'

Usage of toString in JavaScript on StackOverflow
Why does 10..toString() work, but 10.toString() does not?

Extra Newness

I present this as an oddity for your amusement.

class Foo extends Function {
  constructor(val) {
    super();
    this.prototype.val = val;
  }
}

new new Foo(":D")().val; // -> ':D'

💡 Explanation:

Constructors in JavaScript are just functions with some special treatment. By extending Function using the class syntax you create a class that, when instantiated, is now a function, which you can then additionally instantiate.

While not exhaustively tested, I believe the last statement can be analyzed thus:

new new Foo(":D")().val(new newFooInstance()).val;
veryNewFooInstance.val;
// -> ':D'

As a tiny addendum, doing new Function('return "bar";') of course creates a function with the body return "bar";. Since super() in the constructor of our Foo class is calling Function's constructor, it should come as no surprise now to see that we can additionally manipulate things in there.

class Foo extends Function {
  constructor(val) {
    super(`
      this.val = arguments[0];
    `);
    this.prototype.val = val;
  }
}

var foo = new new Foo(":D")("D:");
foo.val; // -> 'D:'
delete foo.val; // remove the instance prop 'val', deferring back to the prototype's 'val'.
foo.val; // -> ':D'

Class Extends Function: Extra Newness

Why you should use semicolons

Writing some standard JavaScript… and then BOOM!

class SomeClass {
  ["array"] = []
  ["string"] = "str"
}

new SomeClass().array; // -> 'str'

What the …?

💡 Explanation:

Once again, this is all thanks to the Automatic Semicolon Insertion.

An example above is basically the same as:

class SomeClass {
  ["array"] = ([]["string"] = "str");
}

You basically assign a string str into an array property.

An original tweet with an example by Ryan Cavanaugh
TC39 meeting when they debated about it

Split a string by a space

Have you ever tried to split a string by a space?

"".split(""); // -> []
// but…
"".split(" "); // -> [""]

💡 Explanation:

This is expected behaviour. Its responsibility is to divide the input string every time a separator occurs in that input string. When you pass in an empty string it'll never find a separator and thus return that string.

Let's quote the specification:

The substrings are determined by searching from left to right for occurrences of separator; these occurrences are not part of any String in the returned array, but serve to divide up the String value.

22.1.3.21 String.prototype.split
An original tween with an example by Ryan Cavanaugh
A tween with an explanation by Nabil Tharwat

A stringified string

This caused a bug that I've been solving for a few days:

JSON.stringify("production") === "production"; // -> false

💡 Explanation:

Let's see what JSON.stringify is returning:

JSON.stringify("production"); // -> '"production"'

It is actually a stringified string, so it's true:

'"production"' === "production"; // -> false

ECMA-404 The JSON Data Interchange Standard.

Non-strict comparison of a number to `true`

1 == true; // -> true
// but…
Boolean(1.1); // -> true
1.1 == true; // -> false

💡 Explanation:

According to the specification:

The comparison x == y, where x and y are values, produces true or false. Such a comparison is performed as follows:

If Type(x) is Number and Type(y) is String, return the result of the comparison x == ! ToNumber(y).

So this comparison is performed like this:

1 == true;
1 == Number(true);
1 == 1; // -> true
// but…
1.1 == true;
1.1 == Number(true);
1.1 == 1; // -> false

7.2.15 Abstract Equality Comparison

📚 Other resources

wtfjs.com — a collection of those very special irregularities, inconsistencies and just plain painfully unintuitive moments for the language of the web.
Wat — A lightning talk by Gary Bernhardt from CodeMash 2012
What the... JavaScript? — Kyle Simpsons talk for Forward 2 attempts to “pull out the crazy” from JavaScript. He wants to help you produce cleaner, more elegant, more readable code, then inspire people to contribute to the open source community.
Zeros in JavaScript — a comparison table of ==, ===, + and * in JavaScript.

🤝 Supporting

Hi! I work on this project in my spare time, in addition to my primary job. I hope you enjoy reading it. If you do, please, consider supporting me 🙏.

Every single donation is important. Your donation is gonna make a clear statement: My work is valued.

🙏 Thank you for your support! 🙏

Service	Link	Action
Patreon	Become a patron
BuyMeACoffee	Buy me a cup of ☕️ or 🥤
Bitcoin	`1EJsKs6rPsqa7QLoVLpe3wgcdL9Q8WmDxE`
Ethereum	`0x6aF39C917359897ae6969Ad682C14110afe1a0a1`

⚠️ Note: I live in Ukraine and services like PayPal and Stripe don't work with Ukrainian bank accounts. This means there's no way for me to set up GitHub Sponsors, OpenCollective, or services relied on them. Sorry, those are the only ways you can support me for now.

🎓 License

Zefan-Cai/KVCache-Factory

2024-12-01T01:33:17Z

Unified KV Cache Compression Methods for Auto-Regressive Models

News

[2024-11-28] Çhange the name to KVCache-Factory! The target of our project is now a unified framework of KV Cache compression of diverse models.
[2024-06-25] Support multi-GPUs inference with big LLMs now! Try out PyramidKV on LlaMa-3-70B-Instruct!
[2024-06-10] Support PyramidKV, SnapKV, H2O and StreamingLLM at Flash Attention v2, Sdpa Attention now! If your devices (i.e., V100, 3090) does not support Flash Attention v2, you can set attn_implementation=sdpa to try PyramidKV at Sdpa Attention!

TODO:

Support implementation of Streaming LLM, H2O and SnapKV
Support Mistral model
Support implementation of Needle
Support KV cache compression without Flash Attention v2 (i.e. Sdpa Attention) for V100
Support multi-GPU inference for 70B LlaMa-3
Introduce new functions to support kv cache budget allocation (i.e., supports for percentage.)
Support Mixtral
Support Batch Inference
Support KV cache compression at decoding stage

Performence

Visualization: Inefficient Attention

The Llama model attention map with 3 documents is represented as follows:

we provide a notebook visualization.ipynb to reproduce the visualization result of each Llama-2-7b-hf model layer for a given 3 document.

Model attention maps for different layers would be stored at ./attention

Requirements

transformers >= 4.41
flash-attn >= 2.4.0.post1

Installation


git clone https://github.com/Zefan-Cai/PyramidKV.git
cd PyramidKV
pip install -r requirements.txt .

Inference

We support inference code on LongBench to repuduce our result.

Please refer to scripts/scripts_longBench/eval.sh to modify the parameters according to your requirements.

Our codebase support Flash Attention v2, Sdpa Attention, etc. The results presented in our paper in based on Flash Attention v2.

export CUDA_VISIBLE_DEVICES=$1

method=$2 # Support PyramidKV, SnapKV, H2O, StreamingLLM
max_capacity_prompts=64 # 128,2048 in paper
attn_implementation=$3 # Support "flash_attention_2", "sdpa", "eager".
source_path=$4
model_path=$5
save_dir=${source_path}"results_long_bench" # path to result save_dir

python3 run_longbench.py \
    --method ${method} \
    --model_path ${model_path} \
    --max_capacity_prompts ${max_capacity_prompts} \
    --attn_implementation ${attn_implementation} \
    --save_dir ${save_dir} \
    --use_cache True

CUDA_VISIBLE_DEVICES: For multi-GPU inference for big LLMs, just need to specify CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7. For single GPU inference, just need to specify CUDA_VISIBLE_DEVICES=0.
model_path: Path to your model. Support "Llama-3-8B-Instruct" for now.
method: Support PyramidKV, SnapKV, StreamingLLM, H2O.
max_capacity_prompts: Selected KV Size in each layer. （e.g. 128, 2048 in paper）. When method is "PyramidKV", given that the total number of KV remains unchanged, the specific KV length for each layer will be modified accordingly
save_dir: Path to your dir to save LongBench result.

After modifying parameters, run:


sh scripts/scripts_longBench/eval.sh

Needle in haystack

We support inference code on Needle in haystack to repuduce our result.

Please refer to scripts/scripts_needle/eval.sh to modify the parameters according to your requirements.

Our codebase support Flash Attention v2, Sdpa Attention, etc. The results presented in our paper in based on Flash Attention v2.


METHOD='pyramidkv'       # ['full', 'pyramidkv', 'snapkv', 'streamingllm', 'h2o']
MAX_CAPACITY_PROMPT=96  # [64, 96, 128, 256, 512, 1024, 2048, ...]
attn_implementation="flash_attention_2" # Support "flash_attention_2", "sdpa", "".
TAG=test


# For Llama3-8b

(
python -u run_needle_in_haystack.py --s_len 1000 --e_len 8001\
    --model_provider LLaMA3 \
    --model_name /mnt/workspace/zhiyuanhu/yuliang/models/llama3-8b_raw \
    --attn_implementation ${attn_implementation} \
    --step 100 \
    --method $METHOD \
    --max_capacity_prompt $MAX_CAPACITY_PROMPT \
    --model_version LlaMA3_${METHOD}_${MAX_CAPACITY_PROMPT}_${TAG}
) 2>&1  | tee results_needle/logs/LlaMA3_${METHOD}_${MAX_CAPACITY_PROMPT}_${TAG}.log

Both LLaMA3 and Mistral2 inference support on single GPU.
model_provider: LLaMA3 or Mistral2
model_name: Path to your model. Support "Llama-3-8B-Instruct" "Mistral-7B-Instruct-v0.2" and for now.
step: The increase of context length.
method: Support PyramidKV, SnapKV, StreamingLLM, H2O.
max_capacity_prompt: Selected KV Size in each layer. （e.g. 128, 2048 in paper）. When method is "PyramidKV", given that the total number of KV remains unchanged, the specific KV length for each layer will be modified accordingly

To reproduce our results, run

bash scripts/scripts_needle/eval.sh

After inference, run

python scripts/scripts_needle/visualize.py

to draw the img, you should change FOLDER_PATH in visualize.py to your output path (the argument of --model_version in eval.sh).

Citation

If you find PyramidKV useful for your research and applications, please kindly cite using this BibTeX:

@article{cai2024pyramidkv,
  title={Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling},
  author={Cai, Zefan and Zhang, Yichi and Gao, Bofei and Liu, Yuliang and Liu, Tianyu and Lu, Keming and Xiong, Wayne and Dong, Yue and Chang, Baobao and Hu, Junjie and Xiao Wen},
  journal={arXiv preprint arXiv:2406.02069},
  year={2024}
}

@article{fu2024not,
  title={Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning},
  author={Fu, Yu and Cai, Zefan and Asi, Abedelkadir and Xiong, Wayne and Dong, Yue and Xiao, Wen},
  journal={arXiv preprint arXiv:2410.19258},
  year={2024}
}

Acknowledgement

Thanks [SnapKV] SnapKV: LLM Knows What You are Looking for Before Generation for providing open-source code to support the expansion of this project.

langchain-ai/langchain

2024-12-01T01:54:58Z

🦜🔗 Build context-aware reasoning applications

🦜️🔗 LangChain

⚡ Build context-aware reasoning applications ⚡

Looking for the JS/TS library? Check out LangChain.js.

To help you ship LangChain apps to production faster, check out LangSmith. LangSmith is a unified developer platform for building, testing, and monitoring LLM applications. Fill out this form to speak with our sales team.

Quick Install

With pip:

pip install langchain

With conda:

conda install langchain -c conda-forge

🤔 What is LangChain?

LangChain is a framework for developing applications powered by large language models (LLMs).

For these applications, LangChain simplifies the entire application lifecycle:

Open-source libraries: Build your applications using LangChain's open-source building blocks, components, and third-party integrations. Use LangGraph to build stateful agents with first-class streaming and human-in-the-loop support.
Productionization: Inspect, monitor, and evaluate your apps with LangSmith so that you can constantly optimize and deploy with confidence.
Deployment: Turn your LangGraph applications into production-ready APIs and Assistants with LangGraph Cloud.

Open-source libraries

langchain-core: Base abstractions and LangChain Expression Language.
langchain-community: Third party integrations.
- Some integrations have been further split into partner packages that only rely on langchain-core. Examples include langchain_openai and langchain_anthropic.
langchain: Chains, agents, and retrieval strategies that make up an application's cognitive architecture.
LangGraph: A library for building robust and stateful multi-actor applications with LLMs by modeling steps as edges and nodes in a graph. Integrates smoothly with LangChain, but can be used without it. To learn more about LangGraph, check out our first LangChain Academy course, Introduction to LangGraph, available here.

Productionization:

LangSmith: A developer platform that lets you debug, test, evaluate, and monitor chains built on any LLM framework and seamlessly integrates with LangChain.

Deployment:

LangGraph Cloud: Turn your LangGraph applications into production-ready APIs and Assistants.

🧱 What can you build with LangChain?

❓ Question answering with RAG

Documentation
End-to-end Example: Chat LangChain and repo

🧱 Extracting structured output

Documentation
End-to-end Example: SQL Llama2 Template

🤖 Chatbots

Documentation
End-to-end Example: Web LangChain (web researcher chatbot) and repo

And much more! Head to the Tutorials section of the docs for more.

🚀 How does LangChain help?

The main value props of the LangChain libraries are:

Components: composable building blocks, tools and integrations for working with language models. Components are modular and easy-to-use, whether you are using the rest of the LangChain framework or not
Off-the-shelf chains: built-in assemblages of components for accomplishing higher-level tasks

Off-the-shelf chains make it easy to get started. Components make it easy to customize existing chains and build new ones.

LangChain Expression Language (LCEL)

LCEL is a key part of LangChain, allowing you to build and organize chains of processes in a straightforward, declarative manner. It was designed to support taking prototypes directly into production without needing to alter any code. This means you can use LCEL to set up everything from basic "prompt + LLM" setups to intricate, multi-step workflows.

Overview: LCEL and its benefits
Interface: The standard Runnable interface for LCEL objects
Primitives: More on the primitives LCEL includes
Cheatsheet: Quick overview of the most common usage patterns

Components

Components fall into the following modules:

📃 Model I/O

This includes prompt management, prompt optimization, a generic interface for chat models and LLMs, and common utilities for working with model outputs.

📚 Retrieval

Retrieval Augmented Generation involves loading data from a variety of sources, preparing it, then searching over (a.k.a. retrieving from) it for use in the generation step.

🤖 Agents

Agents allow an LLM autonomy over how a task is accomplished. Agents make decisions about which Actions to take, then take that Action, observe the result, and repeat until the task is complete. LangChain provides a standard interface for agents, along with LangGraph for building custom agents.

📖 Documentation

Please see here for full documentation, which includes:

Introduction: Overview of the framework and the structure of the docs.
Tutorials: If you're looking to build something specific or are more of a hands-on learner, check out our tutorials. This is the best place to get started.
How-to guides: Answers to “How do I….?” type questions. These guides are goal-oriented and concrete; they're meant to help you complete a specific task.
Conceptual guide: Conceptual explanations of the key parts of the framework.
API Reference: Thorough documentation of every class and method.

🌐 Ecosystem

🦜🛠️ LangSmith: Trace and evaluate your language model applications and intelligent agents to help you move from prototype to production.
🦜🕸️ LangGraph: Create stateful, multi-actor applications with LLMs. Integrates smoothly with LangChain, but can be used without it.
🦜🕸️ LangGraph Platform: Deploy LLM applications built with LangGraph into production.

💁 Contributing

As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.

For detailed information on how to contribute, see here.

🌟 Contributors

facebookresearch/segment-anything

2024-12-01T01:54:58Z

The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

Latest updates -- SAM 2: Segment Anything in Images and Videos

Please check out our new release on Segment Anything Model 2 (SAM 2).

SAM 2 code: https://github.com/facebookresearch/segment-anything-2
SAM 2 demo: https://sam2.metademolab.com/
SAM 2 paper: https://arxiv.org/abs/2408.00714

Segment Anything Model 2 (SAM 2) is a foundation model towards solving promptable visual segmentation in images and videos. We extend SAM to video by considering images as a video with a single frame. The model design is a simple transformer architecture with streaming memory for real-time video processing. We build a model-in-the-loop data engine, which improves model and data via user interaction, to collect our SA-V dataset, the largest video segmentation dataset to date. SAM 2 trained on our data provides strong performance across a wide range of tasks and visual domains.

Segment Anything

Meta AI Research, FAIR

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alex Berg, Wan-Yen Lo, Piotr Dollar, Ross Girshick

[Paper] [Project] [Demo] [Dataset] [Blog] [BibTeX]

The Segment Anything Model (SAM) produces high quality object masks from input prompts such as points or boxes, and it can be used to generate masks for all objects in an image. It has been trained on a dataset of 11 million images and 1.1 billion masks, and has strong zero-shot performance on a variety of segmentation tasks.

Installation

The code requires python>=3.8, as well as pytorch>=1.7 and torchvision>=0.8. Please follow the instructions here to install both PyTorch and TorchVision dependencies. Installing both PyTorch and TorchVision with CUDA support is strongly recommended.

Install Segment Anything:

pip install git+https://github.com/facebookresearch/segment-anything.git

or clone the repository locally and install with

git clone git@github.com:facebookresearch/segment-anything.git
cd segment-anything; pip install -e .

The following optional dependencies are necessary for mask post-processing, saving masks in COCO format, the example notebooks, and exporting the model in ONNX format. jupyter is also required to run the example notebooks.

pip install opencv-python pycocotools matplotlib onnxruntime onnx

Getting Started

First download a model checkpoint. Then the model can be used in just a few lines to get masks from a given prompt:

from segment_anything import SamPredictor, sam_model_registry
sam = sam_model_registry["<model_type>"](checkpoint="<path/to/checkpoint>")
predictor = SamPredictor(sam)
predictor.set_image(<your_image>)
masks, _, _ = predictor.predict(<input_prompts>)

or generate masks for an entire image:

from segment_anything import SamAutomaticMaskGenerator, sam_model_registry
sam = sam_model_registry["<model_type>"](checkpoint="<path/to/checkpoint>")
mask_generator = SamAutomaticMaskGenerator(sam)
masks = mask_generator.generate(<your_image>)

Additionally, masks can be generated for images from the command line:

python scripts/amg.py --checkpoint <path/to/checkpoint> --model-type <model_type> --input <image_or_folder> --output <path/to/output>

See the examples notebooks on using SAM with prompts and automatically generating masks for more details.

ONNX Export

SAM's lightweight mask decoder can be exported to ONNX format so that it can be run in any environment that supports ONNX runtime, such as in-browser as showcased in the demo. Export the model with

python scripts/export_onnx_model.py --checkpoint <path/to/checkpoint> --model-type <model_type> --output <path/to/output>

See the example notebook for details on how to combine image preprocessing via SAM's backbone with mask prediction using the ONNX model. It is recommended to use the latest stable version of PyTorch for ONNX export.

Web demo

The demo/ folder has a simple one page React app which shows how to run mask prediction with the exported ONNX model in a web browser with multithreading. Please see demo/README.md for more details.

Model Checkpoints

Three model versions of the model are available with different backbone sizes. These models can be instantiated by running

from segment_anything import sam_model_registry
sam = sam_model_registry["<model_type>"](checkpoint="<path/to/checkpoint>")

Click the links below to download the checkpoint for the corresponding model type.

default or vit_h: ViT-H SAM model.
vit_l: ViT-L SAM model.
vit_b: ViT-B SAM model.

Dataset

See here for an overview of the datastet. The dataset can be downloaded here. By downloading the datasets you agree that you have read and accepted the terms of the SA-1B Dataset Research License.

We save masks per image as a json file. It can be loaded as a dictionary in python in the below format.

{
    "image"                 : image_info,
    "annotations"           : [annotation],
}

image_info {
    "image_id"              : int,              # Image id
    "width"                 : int,              # Image width
    "height"                : int,              # Image height
    "file_name"             : str,              # Image filename
}

annotation {
    "id"                    : int,              # Annotation id
    "segmentation"          : dict,             # Mask saved in COCO RLE format.
    "bbox"                  : [x, y, w, h],     # The box around the mask, in XYWH format
    "area"                  : int,              # The area in pixels of the mask
    "predicted_iou"         : float,            # The model's own prediction of the mask's quality
    "stability_score"       : float,            # A measure of the mask's quality
    "crop_box"              : [x, y, w, h],     # The crop of the image used to generate the mask, in XYWH format
    "point_coords"          : [[x, y]],         # The point coordinates input to the model to generate the mask
}

Image ids can be found in sa_images_ids.txt which can be downloaded using the above link as well.

To decode a mask in COCO RLE format into binary:

from pycocotools import mask as mask_utils
mask = mask_utils.decode(annotation["segmentation"])

See here for more instructions to manipulate masks stored in RLE format.

License

The model is licensed under the Apache 2.0 license.

Contributing

See contributing and the code of conduct.

Contributors

The Segment Anything project was made possible with the help of many contributors (alphabetical):

Aaron Adcock, Vaibhav Aggarwal, Morteza Behrooz, Cheng-Yang Fu, Ashley Gabriel, Ahuva Goldstand, Allen Goodman, Sumanth Gurram, Jiabo Hu, Somya Jain, Devansh Kukreja, Robert Kuo, Joshua Lane, Yanghao Li, Lilian Luong, Jitendra Malik, Mallika Malhotra, William Ngan, Omkar Parkhi, Nikhil Raina, Dirk Rowe, Neil Sejoor, Vanessa Stark, Bala Varadarajan, Bram Wasti, Zachary Winstrom

Citing Segment Anything

If you use SAM or SA-1B in your research, please use the following BibTeX entry.

@article{kirillov2023segany,
  title={Segment Anything},
  author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{\'a}r, Piotr and Girshick, Ross},
  journal={arXiv:2304.02643},
  year={2023}
}

datawhalechina/self-llm

2024-12-01T01:54:58Z

《开源大模型食用指南》基于Linux环境快速部署开源大模型，更适合中国宝宝的部署教程

开源大模型食用指南

本项目是一个围绕开源大模型、针对国内初学者、基于 Linux 平台的中国宝宝专属大模型教程，针对各类开源大模型提供包括环境配置、本地部署、高效微调等技能在内的全流程指导，简化开源大模型的部署、使用和应用流程，让更多的普通学生、研究者更好地使用开源大模型，帮助开源、自由的大模型更快融入到普通学习者的生活中。

本项目的主要内容包括：

基于 Linux 平台的开源 LLM 环境配置指南，针对不同模型要求提供不同的详细环境配置步骤；
针对国内外主流开源 LLM 的部署使用教程，包括 LLaMA、ChatGLM、InternLM 等；
开源 LLM 的部署应用指导，包括命令行调用、在线 Demo 部署、LangChain 框架集成等；
开源 LLM 的全量微调、高效微调方法，包括分布式全量微调、LoRA、ptuning 等。

项目的主要内容就是教程，让更多的学生和未来的从业者了解和熟悉开源大模型的食用方法！任何人都可以提出issue或是提交PR，共同构建维护这个项目。

想要深度参与的同学可以联系我们，我们会将你加入到项目的维护者中。

学习建议：本项目的学习建议是，先学习环境配置，然后再学习模型的部署使用，最后再学习微调。因为环境配置是基础，模型的部署使用是基础，微调是进阶。初学者可以选择Qwen1.5，InternLM2，MiniCPM等模型优先学习。

注：如果有同学希望了解大模型的模型构成，以及从零手写RAG、Agent和Eval等任务，可以学习Datawhale的另一个项目Tiny-Universe，大模型是当下深度学习领域的热点，但现有的大部分大模型教程只在于教给大家如何调用api完成大模型的应用，而很少有人能够从原理层面讲清楚模型结构、RAG、Agent 以及 Eval。所以该仓库会提供全部手写，不采用调用api的形式，完成大模型的 RAG 、 Agent 、Eval 任务。

注：考虑到有同学希望在学习本项目之前，希望学习大模型的理论部分，如果想要进一步深入学习 LLM 的理论基础，并在理论的基础上进一步认识、应用 LLM，可以参考 Datawhale 的 so-large-llm课程。

注：如果有同学在学习本课程之后，想要自己动手开发大模型应用。同学们可以参考 Datawhale 的动手学大模型应用开发课程，该项目是一个面向小白开发者的大模型应用开发教程，旨在基于阿里云服务器，结合个人知识库助手项目，向同学们完整的呈现大模型应用开发流程。

项目意义

什么是大模型？

大模型（LLM）狭义上指基于深度学习算法进行训练的自然语言处理（NLP）模型，主要应用于自然语言理解和生成等领域，广义上还包括机器视觉（CV）大模型、多模态大模型和科学计算大模型等。

百模大战正值火热，开源 LLM 层出不穷。如今国内外已经涌现了众多优秀开源 LLM，国外如 LLaMA、Alpaca，国内如 ChatGLM、BaiChuan、InternLM（书生·浦语）等。开源 LLM 支持用户本地部署、私域微调，每一个人都可以在开源 LLM 的基础上打造专属于自己的独特大模型。

然而，当前普通学生和用户想要使用这些大模型，需要具备一定的技术能力，才能完成模型的部署和使用。对于层出不穷又各有特色的开源 LLM，想要快速掌握一个开源 LLM 的应用方法，是一项比较有挑战的任务。

本项目旨在首先基于核心贡献者的经验，实现国内外主流开源 LLM 的部署、使用与微调教程；在实现主流 LLM 的相关部分之后，我们希望充分聚集共创者，一起丰富这个开源 LLM 的世界，打造更多、更全面特色 LLM 的教程。星火点点，汇聚成海。

我们希望成为 LLM 与普罗大众的阶梯，以自由、平等的开源精神，拥抱更恢弘而辽阔的 LLM 世界。

项目受众

本项目适合以下学习者：

想要使用或体验 LLM，但无条件获得或使用相关 API；
希望长期、低成本、大量应用 LLM；
对开源 LLM 感兴趣，想要亲自上手开源 LLM；
NLP 在学，希望进一步学习 LLM；
希望结合开源 LLM，打造领域特色的私域 LLM；
以及最广大、最普通的学生群体。

项目规划及进展

本项目拟围绕开源 LLM 应用全流程组织，包括环境配置及使用、部署应用、微调等，每个部分覆盖主流及特点开源 LLM：

Example 系列

Chat-嬛嬛： Chat-甄嬛是利用《甄嬛传》剧本中所有关于甄嬛的台词和语句，基于LLM进行LoRA微调得到的模仿甄嬛语气的聊天语言模型。
Tianji-天机：天机是一款基于人情世故社交场景，涵盖提示词工程、智能体制作、数据获取与模型微调、RAG 数据清洗与使用等全流程的大语言模型系统应用教程。

已支持模型

Qwen2.5-Coder
- Qwen2.5-Coder-7B-Instruct FastApi部署调用 @赵文恺
- Qwen2.5-Coder-7B-Instruct Langchian接入 @杨晨旭
- Qwen2.5-Coder-7B-Instruct WebDemo 部署 @王泽宇
- Qwen2.5-Coder-7B-Instruct vLLM 部署 @王泽宇
- Qwen2.5-Coder-7B-Instruct Lora 微调 @荞麦
- Qwen2.5-Coder-7B-Instruct Lora 微调 SwanLab 可视化记录版 @杨卓
Qwen2-vl
- Qwen2-vl-2B FastApi 部署调用 @姜舒凡
- Qwen2-vl-2B WebDemo 部署 @赵伟
- Qwen2-vl-2B vLLM 部署 @荞麦
- Qwen2-vl-2B Lora 微调 @李柯辰
- Qwen2-vl-2B Lora 微调 SwanLab 可视化记录版 @林泽毅
Qwen2.5
- Qwen2.5-7B-Instruct FastApi 部署调用 @娄天奥
- Qwen2.5-7B-Instruct langchain 接入 @娄天奥
- Qwen2.5-7B-Instruct vLLM 部署调用 @姜舒凡
- Qwen2.5-7B-Instruct WebDemo 部署 @高立业
- Qwen2.5-7B-Instruct Lora 微调 @左春生
- Qwen2.5-7B-Instruct o1-like 推理链实现 @姜舒凡
- Qwen2.5-7B-Instruct Lora 微调 SwanLab 可视化记录版 @林泽毅
Apple OpenELM
- OpenELM-3B-Instruct FastApi 部署调用 @王泽宇
- OpenELM-3B-Instruct Lora 微调 @王泽宇
Llama3_1-8B-Instruct
- Llama3_1-8B-Instruct FastApi 部署调用 @不要葱姜蒜
- Llama3_1-8B-Instruct langchain 接入 @张晋
- Llama3_1-8B-Instruct WebDemo 部署 @张晋
- Llama3_1-8B-Instruct Lora 微调 @不要葱姜蒜
Gemma-2-9b-it
- Gemma-2-9b-it FastApi 部署调用 @不要葱姜蒜
- Gemma-2-9b-it langchain 接入 @不要葱姜蒜
- Gemma-2-9b-it WebDemo 部署 @不要葱姜蒜
- Gemma-2-9b-it Peft Lora 微调 @不要葱姜蒜
Yuan2.0
- Yuan2.0-2B FastApi 部署调用 @张帆
- Yuan2.0-2B Langchain 接入 @张帆
- Yuan2.0-2B WebDemo部署 @张帆
- Yuan2.0-2B vLLM部署调用 @张帆
- Yuan2.0-2B Lora微调 @张帆
Yuan2.0-M32
- Yuan2.0-M32 FastApi 部署调用 @张帆
- Yuan2.0-M32 Langchain 接入 @张帆
- Yuan2.0-M32 WebDemo部署 @张帆
DeepSeek-Coder-V2
- DeepSeek-Coder-V2-Lite-Instruct FastApi 部署调用 @姜舒凡
- DeepSeek-Coder-V2-Lite-Instruct langchain 接入 @姜舒凡
- DeepSeek-Coder-V2-Lite-Instruct WebDemo 部署 @Kailigithub
- DeepSeek-Coder-V2-Lite-Instruct Lora 微调 @余洋
哔哩哔哩 Index-1.9B
- Index-1.9B-Chat FastApi 部署调用 @邓恺俊
- Index-1.9B-Chat langchain 接入 @张友东
- Index-1.9B-Chat WebDemo 部署 @九月
- Index-1.9B-Chat Lora 微调 @姜舒凡
Qwen2
- Qwen2-7B-Instruct FastApi 部署调用 @康婧淇
- Qwen2-7B-Instruct langchain 接入 @不要葱姜蒜
- Qwen2-7B-Instruct WebDemo 部署 @三水
- Qwen2-7B-Instruct vLLM 部署调用 @姜舒凡
- Qwen2-7B-Instruct Lora 微调 @散步
GLM-4
- GLM-4-9B-chat FastApi 部署调用 @张友东
- GLM-4-9B-chat langchain 接入 @谭逸珂
- GLM-4-9B-chat WebDemo 部署 @何至轩
- GLM-4-9B-chat vLLM 部署 @王熠明
- GLM-4-9B-chat Lora 微调 @肖鸿儒
Qwen 1.5
- Qwen1.5-7B-chat FastApi 部署调用 @颜鑫
- Qwen1.5-7B-chat langchain 接入 @颜鑫
- Qwen1.5-7B-chat WebDemo 部署 @颜鑫
- Qwen1.5-7B-chat Lora 微调 @不要葱姜蒜
- Qwen1.5-72B-chat-GPTQ-Int4 部署环境 @byx020119
- Qwen1.5-MoE-chat Transformers 部署调用 @丁悦
- Qwen1.5-7B-chat vLLM推理部署 @高立业
- Qwen1.5-7B-chat Lora 微调接入SwanLab实验管理平台 @黄柏特
谷歌-Gemma
- gemma-2b-it FastApi 部署调用 @东东
- gemma-2b-it langchain 接入 @东东
- gemma-2b-it WebDemo 部署 @东东
- gemma-2b-it Peft Lora 微调 @东东
phi-3
- Phi-3-mini-4k-instruct FastApi 部署调用 @郑皓桦
- Phi-3-mini-4k-instruct langchain 接入 @郑皓桦
- Phi-3-mini-4k-instruct WebDemo 部署 @丁悦
- Phi-3-mini-4k-instruct Lora 微调 @丁悦
CharacterGLM-6B
- CharacterGLM-6B Transformers 部署调用 @孙健壮
- CharacterGLM-6B FastApi 部署调用 @孙健壮
- CharacterGLM-6B webdemo 部署 @孙健壮
- CharacterGLM-6B Lora 微调 @孙健壮
LLaMA3-8B-Instruct
- LLaMA3-8B-Instruct FastApi 部署调用 @高立业
- LLaMA3-8B-Instruct langchain 接入 @不要葱姜蒜
- LLaMA3-8B-Instruct WebDemo 部署 @不要葱姜蒜
- LLaMA3-8B-Instruct Lora 微调 @高立业
XVERSE-7B-Chat
- XVERSE-7B-Chat transformers 部署调用 @郭志航
- XVERSE-7B-Chat FastApi 部署调用 @郭志航
- XVERSE-7B-Chat langchain 接入 @郭志航
- XVERSE-7B-Chat WebDemo 部署 @郭志航
- XVERSE-7B-Chat Lora 微调 @郭志航
TransNormerLLM
- TransNormerLLM-7B-Chat FastApi 部署调用 @王茂霖
- TransNormerLLM-7B-Chat langchain 接入 @王茂霖
- TransNormerLLM-7B-Chat WebDemo 部署 @王茂霖
- TransNormerLLM-7B-Chat Lora 微调 @王茂霖
BlueLM Vivo 蓝心大模型
- BlueLM-7B-Chat FatApi 部署调用 @郭志航
- BlueLM-7B-Chat langchain 接入 @郭志航
- BlueLM-7B-Chat WebDemo 部署 @郭志航
- BlueLM-7B-Chat Lora 微调 @郭志航
InternLM2
- InternLM2-7B-chat FastApi 部署调用 @不要葱姜蒜
- InternLM2-7B-chat langchain 接入 @不要葱姜蒜
- InternLM2-7B-chat WebDemo 部署 @郑皓桦
- InternLM2-7B-chat Xtuner Qlora 微调 @郑皓桦
DeepSeek 深度求索
- DeepSeek-7B-chat FastApi 部署调用 @不要葱姜蒜
- DeepSeek-7B-chat langchain 接入 @不要葱姜蒜
- DeepSeek-7B-chat WebDemo @不要葱姜蒜
- DeepSeek-7B-chat Lora 微调 @不要葱姜蒜
- DeepSeek-7B-chat 4bits量化 Qlora 微调 @不要葱姜蒜
- DeepSeek-MoE-16b-chat Transformers 部署调用 @Kailigithub
- DeepSeek-MoE-16b-chat FastApi 部署调用 @Kailigithub
- DeepSeek-coder-6.7b finetune colab @Swiftie
- Deepseek-coder-6.7b webdemo colab @Swiftie
MiniCPM
- MiniCPM-2B-chat transformers 部署调用 @Kailigithub
- MiniCPM-2B-chat FastApi 部署调用 @Kailigithub
- MiniCPM-2B-chat langchain 接入 @不要葱姜蒜
- MiniCPM-2B-chat webdemo 部署 @Kailigithub
- MiniCPM-2B-chat Lora && Full 微调 @不要葱姜蒜
- 官方友情链接：面壁小钢炮MiniCPM教程 @OpenBMB
- 官方友情链接：MiniCPM-Cookbook @OpenBMB
Qwen-Audio
- Qwen-Audio FastApi 部署调用 @陈思州
- Qwen-Audio WebDemo @陈思州
Qwen
- Qwen-7B-chat Transformers 部署调用 @李娇娇
- Qwen-7B-chat FastApi 部署调用 @李娇娇
- Qwen-7B-chat WebDemo @李娇娇
- Qwen-7B-chat Lora 微调 @不要葱姜蒜
- Qwen-7B-chat ptuning 微调 @肖鸿儒
- Qwen-7B-chat 全量微调 @不要葱姜蒜
- Qwen-7B-Chat 接入langchain搭建知识库助手 @李娇娇
- Qwen-7B-chat 低精度训练 @肖鸿儒
- Qwen-1_8B-chat CPU 部署 @散步
Yi 零一万物
- Yi-6B-chat FastApi 部署调用 @李柯辰
- Yi-6B-chat langchain接入 @李柯辰
- Yi-6B-chat WebDemo @肖鸿儒
- Yi-6B-chat Lora 微调 @李娇娇
Baichuan 百川智能
- Baichuan2-7B-chat FastApi 部署调用 @惠佳豪
- Baichuan2-7B-chat WebDemo @惠佳豪
- Baichuan2-7B-chat 接入 LangChain 框架 @惠佳豪
- Baichuan2-7B-chat Lora 微调 @惠佳豪
InternLM
- InternLM-Chat-7B Transformers 部署调用 @小罗
- InternLM-Chat-7B FastApi 部署调用 @不要葱姜蒜
- InternLM-Chat-7B WebDemo @不要葱姜蒜
- Lagent+InternLM-Chat-7B-V1.1 WebDemo @不要葱姜蒜
- 浦语灵笔图文理解&创作 WebDemo @不要葱姜蒜
- InternLM-Chat-7B 接入 LangChain 框架 @Logan Zou
Atom (llama2)
- Atom-7B-chat WebDemo @Kailigithub
- Atom-7B-chat Lora 微调 @Logan Zou
- Atom-7B-Chat 接入langchain搭建知识库助手 @陈思州
- Atom-7B-chat 全量微调 @Logan Zou
ChatGLM3
- ChatGLM3-6B Transformers 部署调用 @丁悦
- ChatGLM3-6B FastApi 部署调用 @丁悦
- ChatGLM3-6B chat WebDemo @不要葱姜蒜
- ChatGLM3-6B Code Interpreter WebDemo @不要葱姜蒜
- ChatGLM3-6B 接入 LangChain 框架 @Logan Zou
- ChatGLM3-6B Lora 微调 @肖鸿儒

通用环境配置

pip、conda 换源 @不要葱姜蒜
AutoDL 开放端口 @不要葱姜蒜
模型下载
- hugging face @不要葱姜蒜
- hugging face 镜像下载 @不要葱姜蒜
- modelscope @不要葱姜蒜
- git-lfs @不要葱姜蒜
- Openxlab
Issue && PR
- Issue 提交 @肖鸿儒
- PR 提交 @肖鸿儒
- fork更新 @肖鸿儒

致谢

核心贡献者

宋志学(不要葱姜蒜)-项目负责人（Datawhale成员-中国矿业大学(北京)）
肖鸿儒（Datawhale成员-同济大学）
郭志航（内容创作者）
张帆（内容创作者-Datawhale成员）
姜舒凡（内容创作者-鲸英助教）
李娇娇（Datawhale成员）
丁悦（Datawhale-鲸英助教）
邹雨衡-项目负责人（Datawhale成员-对外经济贸易大学）
惠佳豪（Datawhale-宣传大使）
王茂霖（内容创作者-Datawhale成员）
孙健壮（内容创作者-对外经济贸易大学）
东东（内容创作者-谷歌开发者机器学习技术专家）
高立业（内容创作者-DataWhale成员）
王泽宇（内容创作者-太原理工大学-鲸英助教）
Kailigithub （Datawhale成员）
郑皓桦（内容创作者）
李柯辰（Datawhale成员）
林泽毅（内容创作者-SwanLab产品负责人）
陈思州（Datawhale成员）
散步（Datawhale成员）
颜鑫（Datawhale成员）
荞麦（内容创作者-Datawhale成员）
Swiftie （小米NLP算法工程师）
黄柏特（内容创作者-西安电子科技大学）
张友东（内容创作者-Datawhale成员）
余洋（内容创作者-Datawhale成员）
张晋（内容创作者-Datawhale成员）
娄天奥（内容创作者-中国科学院大学-鲸英助教）
左春生（内容创作者-Datawhale成员）
杨卓（内容创作者-西安电子科技大学-鲸英助教）
小罗（内容创作者-Datawhale成员）
谭逸珂（内容创作者-对外经济贸易大学）
王熠明（内容创作者-Datawhale成员）
何至轩（内容创作者-鲸英助教）
康婧淇（内容创作者-Datawhale成员）
三水（内容创作者-鲸英助教）
九月（内容创作者-Datawhale意向成员）
邓恺俊（内容创作者-Datawhale成员）
杨晨旭（内容创作者-太原理工大学-鲸英助教）
赵文恺（内容创作者-太原理工大学-鲸英助教）
赵伟（内容创作者-鲸英助教）

注：排名根据贡献程度排序

其他

特别感谢@Sm1les对本项目的帮助与支持
部分lora代码和讲解参考仓库：https://github.com/zyds/transformers-code.git
如果有任何想法可以联系我们 DataWhale 也欢迎大家多多提出 issue
特别感谢以下为教程做出贡献的同学！

Star History

facebookresearch/sam2

2024-12-01T01:43:58Z

The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

SAM 2: Segment Anything in Images and Videos

AI at Meta, FAIR

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer

[Paper] [Project] [Demo] [Dataset] [Blog] [BibTeX]

Latest updates

09/30/2024 -- SAM 2.1 Developer Suite (new checkpoints, training code, web demo) is released

A new suite of improved model checkpoints (denoted as SAM 2.1) are released. See Model Description for details.
- To use the new SAM 2.1 checkpoints, you need the latest model code from this repo. If you have installed an earlier version of this repo, please first uninstall the previous version via pip uninstall SAM-2, pull the latest code from this repo (with git pull), and then reinstall the repo following Installation below.
The training (and fine-tuning) code has been released. See training/README.md on how to get started.
The frontend + backend code for the SAM 2 web demo has been released. See demo/README.md for details.

Installation

SAM 2 needs to be installed first before use. The code requires python>=3.10, as well as torch>=2.3.1 and torchvision>=0.18.1. Please follow the instructions here to install both PyTorch and TorchVision dependencies. You can install SAM 2 on a GPU machine using:

git clone https://github.com/facebookresearch/sam2.git && cd sam2

pip install -e .

If you are installing on Windows, it's strongly recommended to use Windows Subsystem for Linux (WSL) with Ubuntu.

To use the SAM 2 predictor and run the example notebooks, jupyter and matplotlib are required and can be installed by:

pip install -e ".[notebooks]"

Note:

It's recommended to create a new Python environment via Anaconda for this installation and install PyTorch 2.3.1 (or higher) via pip following https://pytorch.org/. If you have a PyTorch version lower than 2.3.1 in your current environment, the installation command above will try to upgrade it to the latest PyTorch version using pip.
The step above requires compiling a custom CUDA kernel with the nvcc compiler. If it isn't already available on your machine, please install the CUDA toolkits with a version that matches your PyTorch CUDA version.
If you see a message like Failed to build the SAM 2 CUDA extension during installation, you can ignore it and still use SAM 2 (some post-processing functionality may be limited, but it doesn't affect the results in most cases).

Please see INSTALL.md for FAQs on potential issues and solutions.

Getting Started

Download Checkpoints

First, we need to download a model checkpoint. All the model checkpoints can be downloaded by running:

cd checkpoints && \
./download_ckpts.sh && \
cd ..

or individually from:

(note that these are the improved checkpoints denoted as SAM 2.1; see Model Description for details.)

Then SAM 2 can be used in a few lines as follows for image and video prediction.

Image prediction

SAM 2 has all the capabilities of SAM on static images, and we provide image prediction APIs that closely resemble SAM for image use cases. The SAM2ImagePredictor class has an easy interface for image prompting.

import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor

checkpoint = "./checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"
predictor = SAM2ImagePredictor(build_sam2(model_cfg, checkpoint))

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(<your_image>)
    masks, _, _ = predictor.predict(<input_prompts>)

Please refer to the examples in image_predictor_example.ipynb (also in Colab here) for static image use cases.

SAM 2 also supports automatic mask generation on images just like SAM. Please see automatic_mask_generator_example.ipynb (also in Colab here) for automatic mask generation in images.

Video prediction

For promptable segmentation and tracking in videos, we provide a video predictor with APIs for example to add prompts and propagate masklets throughout a video. SAM 2 supports video inference on multiple objects and uses an inference state to keep track of the interactions in each video.

import torch
from sam2.build_sam import build_sam2_video_predictor

checkpoint = "./checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"
predictor = build_sam2_video_predictor(model_cfg, checkpoint)

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<your_video>)

    # add new prompts and instantly get the output on the same frame
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>):

    # propagate the prompts to get masklets throughout the video
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...

Please refer to the examples in video_predictor_example.ipynb (also in Colab here) for details on how to add click or box prompts, make refinements, and track multiple objects in videos.

Load from 🤗 Hugging Face

Alternatively, models can also be loaded from Hugging Face (requires pip install huggingface_hub).

For image prediction:

import torch
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained("facebook/sam2-hiera-large")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(<your_image>)
    masks, _, _ = predictor.predict(<input_prompts>)

For video prediction:

import torch
from sam2.sam2_video_predictor import SAM2VideoPredictor

predictor = SAM2VideoPredictor.from_pretrained("facebook/sam2-hiera-large")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<your_video>)

    # add new prompts and instantly get the output on the same frame
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>):

    # propagate the prompts to get masklets throughout the video
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...

Model Description

SAM 2.1 checkpoints

The table below shows the improved SAM 2.1 checkpoints released on September 29, 2024.

Model	Size (M)	Speed (FPS)	SA-V test (J&F)	MOSE val (J&F)	LVOS v2 (J&F)
sam2.1_hiera_tiny (config, checkpoint)	38.9	47.2	76.5	71.8	77.3
sam2.1_hiera_small (config, checkpoint)	46	43.3 (53.0 compiled*)	76.6	73.5	78.3
sam2.1_hiera_base_plus (config, checkpoint)	80.8	34.8 (43.8 compiled*)	78.2	73.7	78.2
sam2.1_hiera_large (config, checkpoint)	224.4	24.2 (30.2 compiled*)	79.5	74.6	80.6

SAM 2 checkpoints

The previous SAM 2 checkpoints released on July 29, 2024 can be found as follows:

Model	Size (M)	Speed (FPS)	SA-V test (J&F)	MOSE val (J&F)	LVOS v2 (J&F)
sam2_hiera_tiny (config, checkpoint)	38.9	47.2	75.0	70.9	75.3
sam2_hiera_small (config, checkpoint)	46	43.3 (53.0 compiled*)	74.9	71.5	76.4
sam2_hiera_base_plus (config, checkpoint)	80.8	34.8 (43.8 compiled*)	74.7	72.8	75.8
sam2_hiera_large (config, checkpoint)	224.4	24.2 (30.2 compiled*)	76.0	74.6	79.8

* Compile the model by setting compile_image_encoder: True in the config.

Segment Anything Video Dataset

See sav_dataset/README.md for details.

Training SAM 2

You can train or fine-tune SAM 2 on custom datasets of images, videos, or both. Please check the training README on how to get started.

Web demo for SAM 2

We have released the frontend + backend code for the SAM 2 web demo (a locally deployable version similar to https://sam2.metademolab.com/demo). Please see the web demo README for details.

License

The SAM 2 model checkpoints, SAM 2 demo code (front-end and back-end), and SAM 2 training code are licensed under Apache 2.0, however the Inter Font and Noto Color Emoji used in the SAM 2 demo code are made available under the SIL Open Font License, version 1.1.

Contributing

See contributing and the code of conduct.

Contributors

The SAM 2 project was made possible with the help of many contributors (alphabetical):

Karen Bergan, Daniel Bolya, Alex Bosenberg, Kai Brown, Vispi Cassod, Christopher Chedeau, Ida Cheng, Luc Dahlin, Shoubhik Debnath, Rene Martinez Doehner, Grant Gardner, Sahir Gomez, Rishi Godugu, Baishan Guo, Caleb Ho, Andrew Huang, Somya Jain, Bob Kamma, Amanda Kallet, Jake Kinney, Alexander Kirillov, Shiva Koduvayur, Devansh Kukreja, Robert Kuo, Aohan Lin, Parth Malani, Jitendra Malik, Mallika Malhotra, Miguel Martin, Alexander Miller, Sasha Mitts, William Ngan, George Orlin, Joelle Pineau, Kate Saenko, Rodrick Shepard, Azita Shokrpour, David Soofian, Jonathan Torres, Jenny Truong, Sagar Vaze, Meng Wang, Claudette Ward, Pengchuan Zhang.

Third-party code: we use a GPU-based connected component algorithm adapted from cc_torch (with its license in LICENSE_cctorch) as an optional post-processing step for the mask predictions.

Citing SAM 2

If you use SAM 2 or the SA-V dataset in your research, please use the following BibTeX entry.

@article{ravi2024sam2,
  title={SAM 2: Segment Anything in Images and Videos},
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2408.00714},
  url={https://arxiv.org/abs/2408.00714},
  year={2024}
}

IBM/data-prep-kit

2024-12-01T01:43:58Z

Open source project for data preparation of LLM application builders

Data Prep Kit

Data Prep Kit is a community project to democratize and accelerate unstructured data preparation for LLM app developers. With the explosive growth of LLM-enabled use cases, developers are faced with the enormous challenge of preparing use case-specific unstructured data to fine-tune, instruct-tune the LLMs or to build RAG applications for LLMs. As the variety of use cases grow, so does the need to support:

New ways of transforming the data to enhance the performance of the resulting LLMs for each specific use case.
A large variety in the scale of data to be processed, from laptop-scale to datacenter-scale
Support for different data modalities including language, code, vision, multimodal etc

Data Prep Kit offers implementations of commonly needed data preparation steps, called modules or transforms, for both Code and Language modalities, with vision to extend to images, speech and multimodal data. The goal is to offer high-level APIs for developers to quickly get started in working with their data, without needing expertise in the underlying runtimes and frameworks.

📝 Table of Contents

📖 About

Data Prep Kit is a toolkit for streamlining data preparation for developers looking to build LLM-enabled applications via fine-tuning, RAG or instruction-tuning. Data Prep Kit contributes a set of modules that the developer can get started with to easily build data pipelines suitable for their use case. These modules have been tested while producing pre-training datasets for the Granite open source LLM models.

The modules are built on common frameworks (for Spark and Ray), called the data processing library that allows the developers to build new custom modules that readily scale across a variety of runtimes.

Features of the toolkit:

It aims to accelerate unstructured data prep for the "long tail" of LLM use cases.
It offers a growing set of module implementations across multiple runtimes, targeting laptop-scale to datacenter-scale processing.
It provides a growing set of sample data processing pipelines that can be used for real enterprise use cases.
It provides the Data processing library to enable contribution of new custom modules targeting new use cases.
It uses Kubeflow Pipelines-based workflow automation.

Data modalities supported today: Code and Natural Language.

🚀 Getting Started

Fastest way to experience Data Prep Kit

With no setup necessary, let's use a Google Colab friendly notebook to try Data Prep Kit. This is a simple transform to extract content from PDF files: examples/notebooks/Run_your_first_transform_colab.ipynb | . (Here are some tips for running Data Prep Kit transforms on Google Colab. For this simple example, these tips are either already taken care of, or are not needed.)

Create a Virtual Environment

To run on a local machine, follow these steps to quickly set up and deploy the Data Prep Kit in your virtual Python environment.

conda create -n data-prep-kit -y python=3.11
conda activate data-prep-kit
python --version

Check if the python version is 3.11.

If you are using a linux system, install gcc using the below commands:

conda install gcc_linux-64
conda install gxx_linux-64

Next, install the data prep toolkit library. This library installs both the python and ray versions of the transforms. For better management of dependencies, it is recommended to install the same tagged version of both the library and the transform.

pip3 install  'data-prep-toolkit[ray]==0.2.2.dev1'
pip3 install  'data-prep-toolkit-transforms[ray,all]==0.2.2.dev1'
pip3 install jupyterlab   ipykernel  ipywidgets

## install custom kernel
python -m ipykernel install --user --name=data-prep-kit --display-name "dataprepkit"

Test, your installation. If you are able to import these data-prep-kit libraries successfully in python, your installation has succeeded.

## start python interpreter
$   python

# import DPK libraries
>>> from data_processing_ray.runtime.ray import RayTransformLauncher
>>> from data_processing.runtime.pure_python import PythonTransformLauncher

If there are no errors, you are good to go!

Run your first transform locally

Let's try the same simple transform to extract content from PDF files on a local machine.

Local Notebook versions

You can try either one or both of the following two versions:

Option 1: Pure python notebook: examples/notebooks/Run_your_first_transform_python.ipynb - easiest to get started
Option 2: Ray version: This one uses Ray framework for parallel execution while still allowing local processing - examples/notebooks/Run_your_first_transform_ray.ipynb

To run the notebooks, launch jupyter from the same virtual environment you created using the command below.

jupyter lab

After opening the jupyter notebook, change the kernel to dataprepkit, so all libraries will be properly loaded.

Explore more examples here.

Run your first data prep pipeline

Now that you have run a single transform, the next step is to explore how to put these transforms together to run a data prep pipeline for an end to end use case like fine tuning a model or building a RAG application. This notebook gives an example of how to build an end to end data prep pipeline for fine tuning for code LLMs. Similarly, this notebook is a fine tuning example of an end-to-end sample data pipeline designed for processing language datasets. You can also explore how to build a RAG pipeline here.

Current list of transforms

The matrix below shows the the combination of modules and supported runtimes. All the modules can be accessed here and can be combined to form data processing pipelines, as shown in the examples folder.

Modules	Python-only	Ray	Spark	KFP on Ray
Data Ingestion
Code (from zip) to Parquet	✅	✅		✅
PDF to Parquet	✅	✅		✅
HTML to Parquet	✅	✅		✅
Web to Parquet	✅
Universal (Code & Language)
Exact dedup filter	✅	✅		✅
Fuzzy dedup filter		✅		✅
Unique ID annotation	✅	✅	✅	✅
Filter on annotations	✅	✅	✅	✅
Profiler	✅	✅	✅	✅
Resize	✅	✅	✅	✅
HAP	✅	✅		✅
Tokenizer	✅	✅		✅
Language-only
Language identification	✅	✅		✅
Document quality	✅	✅		✅
Document chunking for RAG	✅	✅		✅
Text encoder	✅	✅		✅
PII Annotator/Redactor	✅	✅		✅
Code-only
Programming language annotation	✅	✅		✅
Code quality annotation	✅	✅		✅
Malware annotation	✅	✅		✅
Header cleanser	✅	✅		✅
Semantic file ordering		✅
License Select Annotation	✅	✅		✅

Contributors are welcome to add new modules to expand to other data modalities as well as add runtime support for existing modules!

Add your own transform

At the core of the framework, is a data processing library, that provides a systematic way to implement the data processing modules. The library is python-based and enables the application of "transforms" to a one or more input data files to produce one or more output data files. We use the popular parquet format to store the data (code or language). Every parquet file follows a set schema. A user can use one or more transforms (or modules) as discussed above to process their data. A transform can follow one of the two patterns: annotator or filter.

Annotator An annotator transform adds information during the processing by adding one more columns to the parquet files. The annotator design also allows a user to verify the results of the processing before the actual filtering of the data.
Filter A filter transform processes the data and outputs the transformed data, e.g., exact deduplication. A general purpose SQL-based filter transform enables a powerful mechanism for identifying columns and rows of interest for downstream processing.

For a new module to be added, a user can pick the right design based on the processing to be applied. More details here.

One can leverage Python-based processing logic and the Data Processing Library to easily build and contribute new transforms. We have provided an example transform that can serve as a template to add new simple transforms. Follow the step by step tutorial to help you add your own new transform.

For a deeper understanding of the library's architecture, its transforms, and available runtimes, we encourage the reader to consult the comprehensive overview document alongside dedicated sections on transforms and runtimes.

Additionally, check out our video tutorial for a visual, example-driven guide on adding custom modules.

💻 -> 🖥️☁️ From laptop to cluster

Data-prep-kit provides the flexibility to transition your projects from proof-of-concept (PoC) stage to full-scale production mode, offering all the necessary tools to run your data transformations at high volume. In this section, we enable you how to run your transforms at scale and how to automate them.

Scaling of Transforms

To enable processing of large data volumes leveraging multi-mode clusters, Ray or Spark wrappers are provided, to readily scale out the Python implementations.

A generalized workflow is shown here.

Automation

The toolkit also supports transform execution automation based on Kubeflow pipelines (KFP), tested on a locally deployed Kind cluster and external OpenShift clusters. There is an automation to create a Kind cluster and deploy all required components on it. The KFP implementation is based on the KubeRay Operator for creating and managing the Ray cluster and KubeRay API server to interact with the KubeRay operator. An additional framework along with several kfp components is used to simplify the pipeline implementation.

A simple transform pipeline tutorial explains the pipeline creation and execution. In addition, if you want to combine several transformers in a single pipeline, you can look at multi-steps pipeline

When you finish working with the cluster, and want to clean up or destroy it. See the clean up the cluster

Run your first transform using command line options

You can run transforms via docker image or using virtual environments. This document shows how to run a transform using virtual environment. You can follow this document to run using docker image.

Citations

If you use Data Prep Kit in your research, please cite our paper:

@misc{wood2024dataprepkitgettingdataready,
      title={Data-Prep-Kit: getting your data ready for LLM application development}, 
      author={David Wood and Boris Lublinsky and Alexy Roytman and Shivdeep Singh 
      and Constantin Adam and Abdulhamid Adebayo and Sungeun An and Yuan Chi Chang 
      and Xuan-Hong Dang and Nirmit Desai and Michele Dolfi and Hajar Emami-Gohari 
      and Revital Eres and Takuya Goto and Dhiraj Joshi and Yan Koyfman 
      and Mohammad Nassar and Hima Patel and Paramesvaran Selvam and Yousaf Shah  
      and Saptha Surendran and Daiki Tsuzuku and Petros Zerfos and Shahrokh Daijavad},
      year={2024},
      eprint={2409.18164},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2409.18164}, 
}

DataExpert-io/data-engineer-handbook

2024-12-01T01:43:58Z

This is a repo with links to everything you'd ever want to learn about data engineering

The Data Engineering Handbook

This repo has all the resources you need to become an amazing data engineer!

Getting started

If you are new to data engineering, start by following this 2024 breaking into data engineering roadmap

If you are here for the 6-week free YouTube boot camp you can check out

For more applied learning:

Check out the projects section for more hands-on examples!
Check out the interviews section for more advice on how to pass data engineering interviews!
Check out the books section for a list of high quality data engineering books
Check out the communities section for a list of high quality data engineering communities to join
Check out the newsletter section to learn via email

Resources

Great list of over 25 books

Great list of over 10 communities to join:

Top must-join communities for DE:

Top must-join communities for ML:

Companies:

Orchestration
- Mage
- Astronomer
- Prefect
- Dagster
- Airflow
- Kestra
- Shipyard
- Hamilton
Data Lake / Cloud
- Tabular
- Microsoft
- Databricks
- Onehouse
- Delta Lake
Data Warehouse
- Snowflake
- Firebolt
Data Quality
- dbt
- Gable
- Great Expectations
- Streamdal
- Coalesce
- Soda
- DQOps
- HEDDA.IO
Education Companies
Analytics / Visualization
- Preset
- Starburst
- Metabase
- Looker Studio
- Tableau
- Power BI
- Hex
- Apache Superset
- Evidence
Data Integration
- Cube
- Fivetran
- Airbyte
- dlt
- Sling
- Meltano
Modern OLAP
- Apache Druid
- ClickHouse
- Apache Pinot
- Apache Kylin
- DuckDB
- QuestDB
- StarRocks
LLM application library
Real-Time Data

Data Engineering blogs of companies:

Data Engineering Whitepapers:

Social Media Accounts

Here's the mostly comprehensive list of data engineering creators: (You have to have at least 5k followers somewhere to be added!)

Name	YouTube	LinkedIn	X/Twitter	Instagram	TikTok
Zach Wilson	Data with Zach (70k+)	Zach Wilson (400k+)	EcZachly (30k+)	eczachly (150k+)	@eczachly (70k+)
Shashank Mishra	E-learning Bridge (100k+)	Shashank Mishra (100k+)
Seattle Data Guy	Seattle Data Guy (100k+)	Ben Rogojan (100k+)	SeattleDataGuy (10k+)
TrendyTech	TrendyTech (100k+)	Sumit Mittal (100k+)
Darshil Parmar	Darshil Parmar (100k+)	Darshil Parmar (100k+)
Andreas Kretz	Andreas Kretz (100k+)	Andreas Kretz (100k+)		learndataengineering (5k+)
ByteByteGo	ByteByteGo (1m+)	Alex Xu (100k+)	alexxubyte (100k+)
The Ravit Show	The Ravit Show (100k+)
Guy in a Cube	Guy in a Cube (100k+)
Adam Marczak	Adam Marczak (100k+)
nullQueries	nullQueries (100k+)
TECHTFQ by Thoufiq	TECHTFQ by Thoufiq (100k+)
SQLBI	SQLBI (100k+)	Marco Russo (50k+)	marcorus (10k+)
Azure Lib	Azure Lib (10k+)	Deepak Goyal (100k+)
Advancing Analytics	Advancing Analytics (10k+)	Simon Whiteley (10k+)
Kahan Data Solutions	Kahan Data Solutions (10k+)
Ankit Bansal	Ankit Bansal (10k+)	Ankit Bansal (50k+)
Mr. K Talks Tech	Mr. K Talks Tech (10k+)
Li Yin		Li Yin (10k+)
Jaco van Gelder		Jaco van Gelder (10k+)
Joseph Machado		Joseph Machado (10k+)	startdataeng (5k+)
Eric Roby		Eric Roby (10k+)
Simon Späti		Simon Späti (10k+)
Dipankar Mazumdar		Dipankar Mazumdar (5k+)
Daniel Ciocirlan		Daniel Ciocirlan (5k+)
Hugo Lu		Hugo Lu (5k+)
Tobias Macey		Tobias Macey (5k+)
Marcos Ortiz		Marcos Ortiz (5k+)
Julien Hurault		Julien Hurault (5k+)
Alex Freberg	Alex The Analyst (100k+)	Alex Freberg (100k+)			@alex_the_analyst (10k+)
Marc Lamberti		Marc Lamberti (50k+)
Chip Huyen		Chip Huyen (250k+)
Alex Merced	Alex Merced Data	Alex Merced (30k+)	@amdatalakehouse	@alexmercedcoder
John Kutay	John Kutay	John Kutay (5k+)	@JohnKutay
Lakshmi Sontenam		Lakshmi Sontenam (9.5k+)
Hassaan Akbar		Hassaan Akbar (5k+)
Samuel Focht	Python Basics (10k+)
Constantin Lungu		Constantin Lungu (10k+)
Ijaz Ali		Ijaz Ali (24K+)
Subhankar		Subhankar (5k+)
Ankur Ranjan	Big Data Show (100k+)	Ankur Ranjan (48k+)
Lenny		Lenny A (6k+)
Mehdi Ouazza	Mehdio DataTV (3k+)	Mehdi Ouazza (20k+)	mehd_io		@mehdio_datatv
ITVersity	ITVersity (67k+)	Durga Gadiraju (48k+)
Arnaud Milleker		Arnaud Milleker (7k+)
Soumil Shah	[Soumil Shah] (https://www.youtube.com/@SoumilShah) (50k)	Soumil Shah (8k+)
Ananth Packkildurai		Ananth Packkildurai (18k+)

Great Podcasts

Great list of 20+ newsletters

Top must follow newsletters for data engineering:

Glossaries:

Design Patterns

Courses / Academies

DataExpert.io course use code HANDBOOK10 for a discount!
LearnDataEngineering.com
Technical Freelancer Academy Use code zwtech for a discount!
IBM Data Engineering for Everyone
Qwiklabs
DataCamp
Udemy Courses from Shruti Mantri
Rock the JVM teaches Spark (in Scala), Flink and others
Data Engineering Zoomcamp by DataTalksClub
Efficient Data Processing in Spark
Scaler
DataTeams - Data Engingeer hiring platform
Udemy Courses from Daniel Blanco

Certifications Courses

deniscerri/ytdlnis

2024-12-01T01:55:08Z

Android Video/Audio Downloader app using yt-dlp

YTDLnis

YTDLnis is a free and open source video/audio downloader using yt-dlp for Android 7.0 and above.

Created by Denis Çerri

The links above are the only trusted sources of YTDLnis. Everything else is not related to me.

💡 Features:

download audio/video files from more than 1000 websites
process playlists
- edit every playlist item separately just like in a normal download item.
- select a common format for all items and/or select multiple audio formats in case you are downloading them as a video
- select a download path for all items
- select a filename template for all items
- batch update download type to audio/video/custom command in one click
queue downloads and schedule them by date and time
- you can also schedule multiple items at the same time
download multiple items at the same time
use custom commands and templates or go full yt-dlp mode with a built in terminal
- you can backup and restore templates, so you can share with your buddies
COOKIES support. Log in with your accounts and download private/unavailable videos, unlock premium formats etc.
cut videos based on timestamps and video chapters (This yt-dlp feature is experimental on the original project)
- you can make unlimited cuts
remove sponsorblock elements from the item
- embed them as a chapters in your video
embed subtitles/metadata/chapters etc
modify metadata such as title and author
split item into separate files depending on its chapters
select different download formats
bottom card right from the share menu, no need to open the app
- you can create a txt file and fill it with links/playlists/search queries separate by a new line and the app will process them
search or insert a link from the app
- you can stack searches so you can process them at the same time
log downloads in case of problems
re-download cancelled or failed downloads
- you can use gestures to swipe left to redownload and right to delete
- you can long click the redownload button in the details sheet to show the download card for more functionality
incognito mode when you don't want to save a download history or logs
quick download mode
- download immediately without having to wait for data to process. Turn off the bottom card and it will instantly start
open / share downloaded files right from the finished notification
most yt-dlp features are implemented, suggestions are welcome
Material You interface
Theming options
Backup and restore features. (Almost everything can be backed up)
MVVM Architecture w/ WorkManager

📲 Screenshots

💬 Contact

Join our Telegram Channel or Discord for discussion, announcements, and releases!

🔑 Connect with ReVanced and LibreTube

The App's package name is "com.deniscerri.ytdl"

📝 Help Translate on Weblate

✔️🤖 Connect with Tasker / Macrodroid etc

You can use intents on apps like tasker or macrodroid to push commands to the app to run a download without user interaction Accepted variables:

TYPE -> it can be: audio,video,command
BACKGROUND -> it can be: true,false. If its true the app won't show the download card no matter what and run the download in the background
COMMAND -> if your preferred type is audio/video or you set the TYPE variable as one of them, this will be appended to your Extra Commands string. If the type is command, the whole command you wrote here will be used for the download

Tasker Example of downloading an audio in the background

Create Send Intent task
Action: android.intent.action.SEND
Cat: Default
Mime Type: text/*
Extra: android.intent.extra.TEXT:url (instead of url write the url of the video you want to download)
Extra: TYPE:audio
Extra: BACKGROUND:true

😇 Contributing

If you would like to contribute. Please read the Contributing section.

📄 License

GNU GPL v3.0

⚠️ Warning

Except for the source code licensed under the GPLv3 license, all other parties are prohibited from using the 'YTDLnis' name as a downloader app, and the same is true for its derivatives. Derivatives include but are not limited to forks and unofficial builds.

🙏 Thanks

yt-dlp and its contributors for making this tool possible. Without it this app wouldn't exist.
youtubedl-android for porting yt-dlp to android
dvd for showing how to implement youtubedl-android
seal for certain design elements and features i wanted to use aswell in this app when i started developing
decipher3114 for the app icon

and alot of other internet forum strangers.

firebase/quickstart-android

2024-12-01T01:55:08Z

Firebase Quickstart Samples for Android

Firebase Quickstarts for Android

A collection of quickstart samples demonstrating the Firebase APIs on Android. For more information, see https://firebase.google.com.

Samples

You can open each of the following samples as an Android Studio project, and run them on a mobile device or a virtual device (AVD). When doing so you need to add each sample app you wish to try to a Firebase project on the Firebase console. You can add multiple sample apps to the same Firebase project. There's no need to create separate projects for each app.

To add a sample app to a Firebase project, use the applicationId value specified in the app/build.gradle file of the app as the Android package name. Download the generated google-services.json file, and copy it to the app/ directory of the sample you wish to run.

How to make contributions?

Please read and follow the steps in the CONTRIBUTING.md

bitwarden/android

2024-12-01T01:55:08Z

Bitwarden mobile app for Android.

Bitwarden Android

Compatibility

Minimum SDK: 29
Target SDK: 35
Device Types Supported: Phone and Tablet
Orientations Supported: Portrait and Landscape

Setup

Clone the repository:

$ git clone https://github.com/bitwarden/android

Create a user.properties file in the root directory of the project and add the following properties:
- gitHubToken: A "classic" Github Personal Access Token (PAT) with the read:packages scope (ex: gitHubToken=gph_xx...xx). These can be generated by going to the Github tokens page. See the Github Packages user documentation concerning authentication for more details.
- localSdk: A boolean value to determine if the SDK should be loaded from the local maven artifactory (ex: localSdk=true). This is particularly useful when developing new SDK capabilities. Review Linking SDK to clients for more details.
Setup the code style formatter:

All code must follow the guidelines described in the Code Style Guidelines document. To aid in adhering to these rules, all contributors should apply docs/bitwarden-style.xml as their code style scheme. In IntelliJ / Android Studio:
- Navigate to Preferences > Editor > Code Style.
- Hit the Manage button next to Scheme.
- Select Import.
- Find the bitwarden-style.xml file in the project's docs/ directory.
- Import "from" BitwardenStyle "to" BitwardenStyle.
- Hit Apply and OK to save the changes and exit Preferences.
Note that in some cases you may need to restart Android Studio for the changes to take effect.

All code should be formatted before submitting a pull request. This can be done manually but it can also be helpful to create a macro with a custom keyboard binding to auto-format when saving. In Android Studio on OS X:
- Select Edit > Macros > Start Macro Recording
- Select Code > Optimize Imports
- Select Code > Reformat Code
- Select File > Save All
- Select Edit > Macros > Stop Macro Recording
This can then be mapped to a set of keys by navigating to Android Studio > Preferences and editing the macro under Keymap (ex : shift + command + s).

Please avoid mixing formatting and logical changes in the same commit/PR. When possible, fix any large formatting issues in a separate PR before opening one to make logical changes to the same code. This helps others focus on the meaningful code changes when reviewing the code.

Dependencies

Application Dependencies

The following is a list of all third-party dependencies included as part of the application beyond the standard Android SDK.

AndroidX Appcompat
- https://developer.android.com/jetpack/androidx/releases/appcompat
- Purpose: Allows access to new APIs on older API versions.
- License: Apache 2.0
AndroidX Autofill
- https://developer.android.com/jetpack/androidx/releases/autofill
- Purpose: Allows access to tools for building inline autofill UI.
- License: Apache 2.0
AndroidX Biometrics
- https://developer.android.com/jetpack/androidx/releases/biometric
- Purpose: Authenticate with biometrics or device credentials.
- License: Apache 2.0
AndroidX Browser
- https://developer.android.com/jetpack/androidx/releases/browser
- Purpose: Displays webpages with the user's default browser.
- License: Apache 2.0
AndroidX CameraX Camera2
- https://developer.android.com/jetpack/androidx/releases/camera
- Purpose: Display and capture images for barcode scanning.
- License: Apache 2.0
AndroidX Compose
- https://developer.android.com/jetpack/androidx/releases/compose
- Purpose: A Kotlin-based declarative UI framework.
- License: Apache 2.0
AndroidX Core SplashScreen
- https://developer.android.com/jetpack/androidx/releases/core
- Purpose: Backwards compatible SplashScreen API implementation.
- License: Apache 2.0
AndroidX Credentials
- https://developer.android.com/jetpack/androidx/releases/credentials
- Purpose: Unified access to user's credentials.
- License: Apache 2.0
AndroidX Lifecycle
- https://developer.android.com/jetpack/androidx/releases/lifecycle
- Purpose: Lifecycle aware components and tooling.
- License: Apache 2.0
AndroidX Room
- https://developer.android.com/jetpack/androidx/releases/room
- Purpose: A convenient SQLite-based persistence layer for Android.
- License: Apache 2.0
AndroidX Security
- https://developer.android.com/jetpack/androidx/releases/security
- Purpose: Safely manage keys and encrypt files and sharedpreferences.
- License: Apache 2.0
AndroidX WorkManager
- https://developer.android.com/jetpack/androidx/releases/work
- Purpose: The WorkManager is used to schedule deferrable, asynchronous tasks that must be run reliably.
- License: Apache 2.0
Dagger Hilt
- https://github.com/google/dagger
- Purpose: Dependency injection framework.
- License: Apache 2.0
Firebase Cloud Messaging
- https://github.com/firebase/firebase-android-sdk
- Purpose: Allows for push notification support. (NOTE: This dependency is not included in builds distributed via F-Droid.)
- License: Apache 2.0
Firebase Crashlytics
- https://github.com/firebase/firebase-android-sdk
- Purpose: SDK for crash and non-fatal error reporting. (NOTE: This dependency is not included in builds distributed via F-Droid.)
- License: Apache 2.0
Glide
- https://github.com/bumptech/glide
- Purpose: Image loading and caching.
- License: BSD, part MIT and Apache 2.0
kotlinx.collections.immutable
- https://github.com/Kotlin/kotlinx.collections.immutable
- Purpose: Immutable collection interfaces and implementation prototypes for Kotlin.
- License: Apache 2.0
kotlinx.coroutines
- https://github.com/Kotlin/kotlinx.coroutines
- Purpose: Kotlin coroutines library for asynchronous and reactive code.
- License: Apache 2.0
kotlinx.serialization
- https://github.com/Kotlin/kotlinx.serialization/
- Purpose: JSON serialization library for Kotlin.
- License: Apache 2.0
kotlinx.serialization converter
- https://github.com/square/retrofit/tree/trunk/retrofit-converters/kotlinx-serialization
- Purpose: Converter for Retrofit 2 and kotlinx.serialization.
- License: Apache 2.0
OkHttp 3
- https://github.com/square/okhttp
- Purpose: An HTTP client used by the library to intercept and log traffic.
- License: Apache 2.0
Retrofit 2
- https://github.com/square/retrofit
- Purpose: A networking layer interface.
- License: Apache 2.0
Timber
- https://github.com/JakeWharton/timber
- Purpose: Extensible logging library for Android.
- License: Apache 2.0
zxcvbn4j
- https://github.com/nulab/zxcvbn4j
- Purpose: Password strength estimation.
- License: MIT
ZXing
- https://github.com/zxing/zxing
- Purpose: Barcode scanning and generation.
- License: Apache 2.0

Development Environment Dependencies

The following is a list of additional third-party dependencies used as part of the local development environment. This includes test-related artifacts as well as tools related to code quality and linting. These are not present in the final packaged application.

detekt
- https://github.com/detekt/detekt
- Purpose: A static code analysis tool for the Kotlin programming language.
- License: Apache 2.0
JUnit 5
- https://github.com/junit-team/junit5
- Purpose: Unit Testing framework for testing application code.
- License: Eclipse Public License 2.0
MockK
- https://github.com/mockk/mockk
- Purpose: Kotlin-friendly mocking library.
- License: Apache 2.0
Robolectric
- https://github.com/robolectric/robolectric
- Purpose: A unit testing framework for code directly depending on the Android framework.
- License: MIT
Turbine
- https://github.com/cashapp/turbine
- Purpose: A small testing library for kotlinx.coroutine's Flow.
- License: Apache 2.0

CI/CD Dependencies

The following is a list of additional third-party dependencies used as part of the CI/CD workflows. These are not present in the final packaged application.

Fastlane
- https://fastlane.tools/
- Purpose: Automates building, signing, and distributing applications.
- License: MIT
Kover
- https://github.com/Kotlin/kotlinx-kover
- Purpose: Kotlin code coverage toolset.
- License: Apache 2.0

oxyroid/M3UAndroid

2024-12-01T01:44:08Z

FOSS Player, which made of jetpack compose. Android 8.0 and above supported.

M3UAndroid

M3U is a FREE stream media player on android devices, which made of jetpack compose. Android 8.0 and above supported.

Device support

Most Android devices, including smartphones, TVs, and tablets.

Screenshots

📢 Translations Wanted 📢

Please submit a pull request if you want to help with translation.

Official:

English
Simplified Chinese
(You can also provide better translations for the above languages via pull requests)

From PRs:

Spanish, thanks @sguinetti
Romanian, thanks @iboboc
Brazilian Portuguese, thanks @Suburbanno

Introduce

Basic

Features

Playlist Management.
Streaming media analysis capabilities.
Xtream support.
DLNA screencast.
i18n (Internationalization).

Android Development

M3U is an app that attempts to use the latest libraries and tools. As a summary:

Entirely written in Kotlin.
UI completely written in Jetpack Compose.
Material3 design system.
Uses Kotlin Coroutines throughout.
Uses many of the Architecture Components, including: Room, Lifecycle, Navigation.
Uses Hilt for dependency injection.
Uses Lint Checks for code scanning.
Uses KSP & KotlinPoet for Code Generating.
FFmepg-kit & ExoPlayer.

Installation

Get the SNAPSHOT package here

Community

You can join the Telegram Channel for update information and alpha & beta packages.

You can also join the Telegram Group for discussing.

Contributing

View this file to learn about how to contribute this repository.

Refer to the file to learn about the Code Specification of this repository.

Star History

License

M3UAndroid is distributed under the terms of the Apache License (Version 2.0). See the license for more information.

karb94/neoscroll.nvim

2024-12-01T01:44:26Z

Smooth scrolling neovim plugin written in lua

Neoscroll: a smooth scrolling neovim plugin written in lua

Demo

https://user-images.githubusercontent.com/41967813/121818668-7b36c800-cc80-11eb-8c3a-45a4767b8f05.mp4

Features

Smooth scrolling for window movement commands (mappings optional): <C-u>, <C-d>, <C-b>, <C-f>, <C-y>, <C-e>, zt, zz, zb.
Takes into account folds.
A single scrolling function that accepts either the number of lines or the percentage of the window to scroll.
Scroll any window.
Cursor is hidden while scrolling (optional) for a more pleasing scrolling experience.
Customizable scrolling behaviour.
You can use predefined easing functions for the scrolling animation.
Performance mode that turns off syntax highlighting while scrolling for slower machines or files with heavy regex syntax highlighting.
Cancel scroll by scrolling in the opposite direction.
Simulated "stop on key release" when holding down a key to scroll.
Scroll any window by window-ID

Installation

You will need neovim 0.5 for this plugin to work. Install it using your favorite plugin manager:

With Packer: use 'karb94/neoscroll.nvim'
With vim-plug: Plug 'karb94/neoscroll.nvim'

With lazy.nvim, create the file ~/.config/nvim/lua/plugins/neoscroll.lua:

return {
  "karb94/neoscroll.nvim",
  config = function ()
    require('neoscroll').setup({})
  end
}

Options

Read :help neoscroll-options for a detailed description of all the options.

setup() with all the options and their default values:

require('neoscroll').setup({
  mappings = {                 -- Keys to be mapped to their corresponding default scrolling animation
    '<C-u>', '<C-d>',
    '<C-b>', '<C-f>',
    '<C-y>', '<C-e>',
    'zt', 'zz', 'zb',
  },
  hide_cursor = true,          -- Hide cursor while scrolling
  stop_eof = true,             -- Stop at <EOF> when scrolling downwards
  respect_scrolloff = false,   -- Stop scrolling when the cursor reaches the scrolloff margin of the file
  cursor_scrolls_alone = true, -- The cursor will keep on scrolling even if the window cannot scroll further
  duration_multiplier = 1.0    -- Global duration multiplier
  easing = 'linear',           -- Default easing function
  pre_hook = nil,              -- Function to run before the scrolling animation starts
  post_hook = nil,             -- Function to run after the scrolling animation ends
  performance_mode = false,    -- Disable "Performance Mode" on all buffers.
  ignored_events = {           -- Events ignored while scrolling
      'WinScrolled', 'CursorMoved'
  },
})

You can map a smaller set of default mappings:

require('neoscroll').setup({ mappings = {'<C-u>', '<C-d>', '<C-b>', '<C-f>'} })

Or you can disable all default mappings by passing an empty list:

require('neoscroll').setup({ mappings = {} })

The section below explains how to create your own custom mappings.

Custom mappings

You can create your own scrolling mappings using the following lua functions:

scroll(lines, opts)
ctrl_u
ctrl_d
ctrl_b
ctrl_f
zt(opts)
zz(opts)
zb(opts)

Read :help neoscroll.scroll() and :help neoscroll-helper-functions for more details.

You can use the following syntactic sugar in your init.lua to define lua function mappings in normal, visual and select modes:

neoscroll = require('neoscroll')
local keymap = {
  ["<C-u>"] = function() neoscroll.ctrl_u({ duration = 250 }) end;
  ["<C-d>"] = function() neoscroll.ctrl_d({ duration = 250 }) end;
  ["<C-b>"] = function() neoscroll.ctrl_b({ duration = 450 }) end;
  ["<C-f>"] = function() neoscroll.ctrl_f({ duration = 450 }) end;
  ["<C-y>"] = function() neoscroll.scroll(-0.1, { move_cursor=false; duration = 100 }) end;
  ["<C-e>"] = function() neoscroll.scroll(0.1, { move_cursor=false; duration = 100 }) end;
  ["zt"]    = function() neoscroll.zt({ half_win_duration = 250 }) end;
  ["zz"]    = function() neoscroll.zz({ half_win_duration = 250 }) end;
  ["zb"]    = function() neoscroll.zb({ half_win_duration = 250 }) end;
}
local modes = { 'n', 'v', 'x' }
for key, func in pairs(keymap) do
  vim.keymap.set(modes, key, func)
end

Easing functions

By default the scrolling animation has a constant speed (linear), i.e. the time between each line scroll is constant. If you want to smooth the start and end of the scrolling animation you can pass the name of one of the easing functions that Neoscroll provides to the scroll() function. You can use any of the following easing functions: linear, quadratic, cubic, quartic, quintic, circular, sine. Neoscroll will then adjust the time between each line scroll using the selected easing function. This dynamic time adjustment can make animations more pleasing to the eye.

To learn more about easing functions here are some useful links:

Examples

Using the same syntactic sugar introduced in Custom mappings we can write the following config:

neoscroll = require('neoscroll')
neoscroll.setup({
  -- Default easing function used in any animation where
  -- the `easing` argument has not been explicitly supplied
  easing = "quadratic"
})
local keymap = {
  -- Use the "sine" easing function
  ["<C-u>"] = function() neoscroll.ctrl_u({ duration = 250; easing = 'sine' }) end;
  ["<C-d>"] = function() neoscroll.ctrl_d({ duration = 250; easing = 'sine' }) end;
  -- Use the "circular" easing function
  ["<C-b>"] = function() neoscroll.ctrl_b({ duration = 450; easing = 'circular' }) end;
  ["<C-f>"] = function() neoscroll.ctrl_f({ duration = 450; easing = 'circular' }) end;
  -- When no value is passed the `easing` option supplied in `setup()` is used
  ["<C-y>"] = function() neoscroll.scroll(-0.1, { move_cursor=false; duration = 100 }) end;
  ["<C-e>"] = function() neoscroll.scroll(0.1, { move_cursor=false; duration = 100 }) end;
}
local modes = { 'n', 'v', 'x' }
for key, func in pairs(keymap) do
    vim.keymap.set(modes, key, func)
end

`pre_hook` and `post_hook` functions

Set pre_hook and post_hook functions to run custom code before and/or after the scrolling animation. The function will be called with the info parameter which can be optionally passed to scroll() (or any of the provided wrappers). This can be used to conditionally run different hooks for different types of scrolling animations.

For example, if you want to hide the cursorline only for <C-d>/<C-u> scrolling animations you can do something like this:

require('neoscroll').setup({
  pre_hook = function(info) if info == "cursorline" then vim.wo.cursorline = false end end,
  post_hook = function(info) if info == "cursorline" then vim.wo.cursorline = true end end
})
local keymap = {
  ["<C-u>"] = function() neoscroll.ctrl_u({ duration = 250; info = 'cursorline' }) end;
  ["<C-d>"] = function() neoscroll.ctrl_d({ duration = 250; info = 'cursorline' }) end;
}
local modes = { 'n', 'v', 'x' }
for key, func in pairs(keymap) do
  vim.keymap.set(modes, key, func)
end

Keep in mind that the info variable is not restricted to a string. It can also be a table with multiple key-pair values.

Known issues

<C-u>, <C-d>, <C-b>, <C-f> mess up macros (issue).

Acknowledgements

This plugin was inspired by vim-smoothie and neo-smooth-scroll.nvim. Big thank you to their authors!

yetone/avante.nvim

2024-12-01T01:44:26Z

Use your Neovim like using Cursor AI IDE!

avante.nvim

avante.nvim is a Neovim plugin designed to emulate the behaviour of the Cursor AI IDE. It provides users with AI-driven code suggestions and the ability to apply these recommendations directly to their source files with minimal effort.

[!NOTE]

🥰 This project is undergoing rapid iterations, and many exciting features will be added successively. Stay tuned!

https://github.com/user-attachments/assets/510e6270-b6cf-459d-9a2f-15b397d1fe53

https://github.com/user-attachments/assets/86140bfd-08b4-483d-a887-1b701d9e37dd

Features

AI-Powered Code Assistance: Interact with AI to ask questions about your current code file and receive intelligent suggestions for improvement or modification.
One-Click Application: Quickly apply the AI's suggested changes to your source code with a single command, streamlining the editing process and saving time.

Installation

For building binary if you wish to build from source, then cargo is required. Otherwise curl and tar will be used to get prebuilt binary from GitHub.

lazy.nvim (recommended)

{
  "yetone/avante.nvim",
  event = "VeryLazy",
  lazy = false,
  version = false, -- set this if you want to always pull the latest change
  opts = {
    -- add any opts here
  },
  -- if you want to build from source then do `make BUILD_FROM_SOURCE=true`
  build = "make",
  -- build = "powershell -ExecutionPolicy Bypass -File Build.ps1 -BuildFromSource false" -- for windows
  dependencies = {
    "stevearc/dressing.nvim",
    "nvim-lua/plenary.nvim",
    "MunifTanjim/nui.nvim",
    --- The below dependencies are optional,
    "hrsh7th/nvim-cmp", -- autocompletion for avante commands and mentions
    "nvim-tree/nvim-web-devicons", -- or echasnovski/mini.icons
    "zbirenbaum/copilot.lua", -- for providers='copilot'
    {
      -- support for image pasting
      "HakonHarnes/img-clip.nvim",
      event = "VeryLazy",
      opts = {
        -- recommended settings
        default = {
          embed_image_as_base64 = false,
          prompt_for_file_name = false,
          drag_and_drop = {
            insert_mode = true,
          },
          -- required for Windows users
          use_absolute_path = true,
        },
      },
    },
    {
      -- Make sure to set this up properly if you have lazy=true
      'MeanderingProgrammer/render-markdown.nvim',
      opts = {
        file_types = { "markdown", "Avante" },
      },
      ft = { "markdown", "Avante" },
    },
  },
}

vim-plug


" Deps
Plug 'stevearc/dressing.nvim'
Plug 'nvim-lua/plenary.nvim'
Plug 'MunifTanjim/nui.nvim'

" Optional deps
Plug "hrsh7th/nvim-cmp"
Plug 'nvim-tree/nvim-web-devicons' "or Plug 'echasnovski/mini.icons'
Plug 'HakonHarnes/img-clip.nvim'
Plug 'zbirenbaum/copilot.lua'

" Yay, pass source=true if you want to build from source
Plug 'yetone/avante.nvim', { 'branch': 'main', 'do': 'make' }
autocmd! User avante.nvim lua << EOF
require('avante_lib').load()
require('avante').setup()
EOF

mini.deps

local add, later, now = MiniDeps.add, MiniDeps.later, MiniDeps.now

add({
  source = 'yetone/avante.nvim',
  monitor = 'main',
  depends = {
    'stevearc/dressing.nvim',
    'nvim-lua/plenary.nvim',
    'MunifTanjim/nui.nvim',
    'echasnovski/mini.icons'
  },
  hooks = { post_checkout = function() vim.cmd('make') end }
})
--- optional
add({ source = 'hrsh7th/nvim-cmp' })
add({ source = 'zbirenbaum/copilot.lua' })
add({ source = 'HakonHarnes/img-clip.nvim' })
add({ source = 'MeanderingProgrammer/render-markdown.nvim' })

now(function() require('avante_lib').load() end)
later(function() require('render-markdown').setup({...}) end)
later(function()
  require('img-clip').setup({...}) -- config img-clip
  require("copilot").setup({...}) -- setup copilot to your liking
  require("avante").setup({...}) -- config for avante.nvim
end)

Lua

-- deps:
require('cmp').setup ({
  -- use recommended settings from above
})
require('img-clip').setup ({
  -- use recommended settings from above
})
require('copilot').setup ({
  -- use recommended settings from above
})
require('render-markdown').setup ({
  -- use recommended settings from above
})
require('avante_lib').load()
require('avante').setup ({
  -- Your config here!
})

NOTE: For avante.tokenizers and templates to work, make sure to call require('avante_lib').load() somewhere when entering the editor. We will leave the users to decide where it fits to do this, as this varies among configurations. (But we do recommend running this after where you set your colorscheme)

[!IMPORTANT]

avante.nvim is currently only compatible with Neovim 0.10.1 or later. Please ensure that your Neovim version meets these requirements before proceeding.

[!NOTE]

Recommended Neovim options:

-- views can only be fully collapsed with the global statusline
vim.opt.laststatus = 3

[!TIP]

Any rendering plugins that support markdown should work with Avante as long as you add the supported filetype Avante. See https://github.com/yetone/avante.nvim/issues/175 and this comment for more information.

Default setup configuration

See config.lua#L9 for the full config

{
  ---@alias Provider "claude" | "openai" | "azure" | "gemini" | "cohere" | "copilot" | string
  provider = "claude", -- Recommend using Claude
  auto_suggestions_provider = "claude", -- Since auto-suggestions are a high-frequency operation and therefore expensive, it is recommended to specify an inexpensive provider or even a free provider: copilot
  claude = {
    endpoint = "https://api.anthropic.com",
    model = "claude-3-5-sonnet-20241022",
    temperature = 0,
    max_tokens = 4096,
  },
  ---Specify the special dual_boost mode
  ---1. enabled: Whether to enable dual_boost mode. Default to false.
  ---2. first_provider: The first provider to generate response. Default to "openai".
  ---3. second_provider: The second provider to generate response. Default to "claude".
  ---4. prompt: The prompt to generate response based on the two reference outputs.
  ---5. timeout: Timeout in milliseconds. Default to 60000.
  ---How it works:
  --- When dual_boost is enabled, avante will generate two responses from the first_provider and second_provider respectively. Then use the response from the first_provider as provider1_output and the response from the second_provider as provider2_output. Finally, avante will generate a response based on the prompt and the two reference outputs, with the default Provider as normal.
  ---Note: This is an experimental feature and may not work as expected.
  dual_boost = {
    enabled = false,
    first_provider = "openai",
    second_provider = "claude",
    prompt = "Based on the two reference outputs below, generate a response that incorporates elements from both but reflects your own judgment and unique perspective. Do not provide any explanation, just give the response directly. Reference Output 1: [{{provider1_output}}], Reference Output 2: [{{provider2_output}}]",
    timeout = 60000, -- Timeout in milliseconds
  },
  behaviour = {
    auto_suggestions = false, -- Experimental stage
    auto_set_highlight_group = true,
    auto_set_keymaps = true,
    auto_apply_diff_after_generation = false,
    support_paste_from_clipboard = false,
    minimize_diff = true, -- Whether to remove unchanged lines when applying a code block
  },
  mappings = {
    --- @class AvanteConflictMappings
    diff = {
      ours = "co",
      theirs = "ct",
      all_theirs = "ca",
      both = "cb",
      cursor = "cc",
      next = "]x",
      prev = "[x",
    },
    suggestion = {
      accept = "<M-l>",
      next = "<M-]>",
      prev = "<M-[>",
      dismiss = "<C-]>",
    },
    jump = {
      next = "]]",
      prev = "[[",
    },
    submit = {
      normal = "<CR>",
      insert = "<C-s>",
    },
    sidebar = {
      apply_all = "A",
      apply_cursor = "a",
      switch_windows = "<Tab>",
      reverse_switch_windows = "<S-Tab>",
    },
  },
  hints = { enabled = true },
  windows = {
    ---@type "right" | "left" | "top" | "bottom"
    position = "right", -- the position of the sidebar
    wrap = true, -- similar to vim.o.wrap
    width = 30, -- default % based on available width
    sidebar_header = {
      enabled = true, -- true, false to enable/disable the header
      align = "center", -- left, center, right for title
      rounded = true,
    },
    input = {
      prefix = "> ",
      height = 8, -- Height of the input window in vertical layout
    },
    edit = {
      border = "rounded",
      start_insert = true, -- Start insert mode when opening the edit window
    },
    ask = {
      floating = false, -- Open the 'AvanteAsk' prompt in a floating window
      start_insert = true, -- Start insert mode when opening the ask window
      border = "rounded",
      ---@type "ours" | "theirs"
      focus_on_apply = "ours", -- which diff to focus after applying
    },
  },
  highlights = {
    ---@type AvanteConflictHighlights
    diff = {
      current = "DiffText",
      incoming = "DiffAdd",
    },
  },
  --- @class AvanteConflictUserConfig
  diff = {
    autojump = true,
    ---@type string | fun(): any
    list_opener = "copen",
    --- Override the 'timeoutlen' setting while hovering over a diff (see :help timeoutlen).
    --- Helps to avoid entering operator-pending mode with diff mappings starting with `c`.
    --- Disable by setting to -1.
    override_timeoutlen = 500,
  },
}

Usage

Given its early stage, avante.nvim currently supports the following basic functionalities:

[!IMPORTANT]

Avante will only support Claude, and OpenAI (and its variants including azure)out-of-the-box due to its high code quality generation. For all OpenAI-compatible providers, see wiki for more details.

[!IMPORTANT]

Due to the poor performance of other models, avante.nvim only recommends using the claude-3.5-sonnet model. All features can only be guaranteed to work properly on the claude-3.5-sonnet model. We do not accept changes to the code or prompts to accommodate other models. Otherwise, it will greatly increase our maintenance costs. We hope everyone can understand. Thank you!

[!IMPORTANT]

For most consistency between neovim session, it is recommended to set the environment variables in your shell file. By default, Avante will prompt you at startup to input the API key for the provider you have selected.

For Claude:
export ANTHROPIC_API_KEY=your-api-key
For OpenAI:
export OPENAI_API_KEY=your-api-key
For Azure OpenAI:
export AZURE_OPENAI_API_KEY=your-api-key

Open a code file in Neovim.
Use the :AvanteAsk command to query the AI about the code.
Review the AI's suggestions.
Apply the recommended changes directly to your code with a simple command or key binding.

Note: The plugin is still under active development, and both its functionality and interface are subject to significant changes. Expect some rough edges and instability as the project evolves.

Key Bindings

The following key bindings are available for use with avante.nvim:

Key Binding	Description
`Leaderaa`	show sidebar
`Leaderar`	refresh sidebar
`Leaderaf`	switch sidebar focus
`Leaderae`	edit selected blocks
`co`	choose ours
`ct`	choose theirs
`ca`	choose all theirs
`c0`	choose none
`cb`	choose both
`cc`	choose cursor
`]x`	move to previous conflict
`[x`	move to next conflict
`[[`	jump to previous codeblocks (results window)
`]]`	jump to next codeblocks (results windows)

[!NOTE]

If you are using lazy.nvim, then all keymap here will be safely set, meaning if <leader>aa is already binded, then avante.nvim won't bind this mapping. In this case, user will be responsible for setting up their own. See notes on keymaps for more details.

Highlight Groups

Highlight Group	Description	Notes
AvanteTitle	Title
AvanteReversedTitle	Used for rounded border
AvanteSubtitle	Selected code title
AvanteReversedSubtitle	Used for rounded border
AvanteThirdTitle	Prompt title
AvanteReversedThirdTitle	Used for rounded border
AvanteConflictCurrent	Current conflict highlight	Default to `Config.highlights.diff.current`
AvanteConflictIncoming	Incoming conflict highlight	Default to `Config.highlights.diff.incoming`
AvanteConflictCurrentLabel	Current conflict label highlight	Default to shade of `AvanteConflictCurrent`
AvanteConflictIncomingLabel	Incoming conflict label highlight	Default to shade of `AvanteConflictIncoming`
AvantePopupHint	Usage hints in popup menus
AvanteInlineHint	The end-of-line hint displayed in visual mode

See highlights.lua for more information

Custom prompts

By default, avante.nvim provides three different modes to interact with: planning, editing, and suggesting, followed with three different prompts per mode.

planning: Used with require("avante").toggle() on sidebar
editing: Used with require("avante").edit() on selection codeblock
suggesting: Used with require("avante").get_suggestion():suggest() on Tab flow.

Users can customize the system prompts via Config.system_prompt. We recommend calling this in a custom Autocmds depending on your need:

vim.api.nvim_create_autocmd("User", {
  pattern = "ToggleMyPrompt",
  callback = function() require("avante.config").override({system_prompt = "MY CUSTOM SYSTEM PROMPT"}) end,
})

vim.keymap.set("n", "<leader>am", function() vim.api.nvim_exec_autocmds("User", { pattern = "ToggleMyPrompt" }) end, { desc = "avante: toggle my prompt" })

If one wish to custom prompts for each mode, avante.nvim will check for project root based on the given buffer whether it contains the following patterns: *.{mode}.avanterules.

The rules for root hierarchy:

lsp workspace folders
lsp root_dir
root pattern of filename of the current buffer
root pattern of cwd

Example folder structure for custom prompt

If you have the following structure:

.
├── .git/
├── typescript.planning.avanterules
├── snippets.editing.avanterules
└── src/

typescript.planning.avanterules will be used for planning mode
snippets.editing.avanterules`` will be used for editing` mode
the default suggesting prompt from avante.nvim will be used for suggesting mode.

[!important]

*.avanterules is a jinja template file, in which will be rendered using minijinja. See templates for example on how to extend current templates.

TODOs

Chat with current file
Apply diff patch
Chat with the selected block
Slash commands
Edit the selected block
Smart Tab (Cursor Flow)
Chat with project (You can use @codebase to chat with the whole project)
CoT
Chat with selected files

Roadmap

Enhanced AI Interactions: Improve the depth of AI analysis and recommendations for more complex coding scenarios.
LSP + Tree-sitter + LLM Integration: Integrate with LSP and Tree-sitter and LLM to provide more accurate and powerful code suggestions and analysis.

Contributing

Contributions to avante.nvim are welcome! If you're interested in helping out, please feel free to submit pull requests or open issues. Before contributing, ensure that your code has been thoroughly tested.

See wiki for more recipes and tricks.

Acknowledgments

We would like to express our heartfelt gratitude to the contributors of the following open-source projects, whose code has provided invaluable inspiration and reference for the development of avante.nvim:

Nvim Plugin	License	Functionality	Location
git-conflict.nvim	No License	Diff comparison functionality	lua/avante/diff.lua
ChatGPT.nvim	Apache 2.0 License	Calculation of tokens count	lua/avante/utils/tokens.lua
img-clip.nvim	MIT License	Clipboard image support	lua/avante/clipboard.lua
copilot.lua	MIT License	Copilot support	lua/avante/providers/copilot.lua
jinja.vim	MIT License	Template filetype support	syntax/jinja.vim
codecompanion.nvim	MIT License	Secrets logic support	lua/avante/providers/init.lua
aider	Apache 2.0 License	Planning mode user prompt	lua/avante/templates/planning.avanterules

The high quality and ingenuity of these projects' source code have been immensely beneficial throughout our development process. We extend our sincere thanks and respect to the authors and contributors of these projects. It is the selfless dedication of the open-source community that drives projects like avante.nvim forward.

License

avante.nvim is licensed under the Apache 2.0 License. For more details, please refer to the LICENSE file.

Star History

sanjay-kv/Open-source-Practice

2024-12-01T01:55:38Z

GSSoC is a 3-month long #OpenSource program by GirlScript Foundation, India.

container-storage-interface/spec

2024-12-01T01:55:38Z

Container Storage Interface (CSI) Specification.

somepaulo/MoreWaita

2024-12-01T01:55:57Z

An expanded Adwaita-styled companion icon theme with extra icons for popular apps to complement Gnome Shell's original icons.

recodehive/Opensource-practice

2024-12-01T01:56:02Z

nf-core/rnavar

2024-12-01T01:47:35Z

gatk4 RNA variant calling pipeline

nf-core/clipseq

2024-12-01T01:47:35Z

CLIP sequencing analysis pipeline for QC, pre-mapping, genome mapping, UMI deduplication, and multiple peak-calling options.

nf-core/spatialvi

2024-12-01T01:47:35Z

Pipeline for processing spatially-resolved gene counts with spatial coordinates and image data. Designed for 10x Genomics Visium transcriptomics.

MarceColl/zen-browser-flake

2024-12-01T01:56:21Z

Nix Flake for the Zen Browser

oddlama/nix-topology

2024-12-01T01:56:21Z

🍁 Generate infrastructure and network diagrams directly from your NixOS configurations

lilyinstarlight/nixos-cosmic

2024-12-01T01:56:21Z

Flake for using COSMIC on NixOS

dmarcotte/easy-move-resize

2024-11-30T01:33:42Z

Easy SSL pinning validation and reporting for iOS, macOS, tvOS and watchOS.

TrustKit

TrustKit is an open source framework that makes it easy to deploy SSL public key pinning and reporting in any iOS 12+, macOS 10.13+, tvOS 12+ or watchOS 4+ App; it supports both Swift and Objective-C Apps.

If you need SSL pinning/reporting in your Android App. we have also released TrustKit for Android at https://github.com/datatheorem/TrustKit-Android.

Overview

TrustKit provides the following features:

Simple API to configure an SSL pinning policy and enforce it within an App. The policy settings are heavily based on the HTTP Public Key Pinning specification.
Sane implementation by pinning the certificate's Subject Public Key Info, as opposed to the certificate itself or the public key bits.
Reporting mechanism to notify a server about pinning validation failures happening within the App, when an unexpected certificate chain is detected. This is similar to the report-uri directive described in the HPKP specification. The reporting mechanism can also be customized within the App by leveraging pin validation notifications sent by TrustKit.
Auto-pinning functionality by swizzling the App's NSURLConnection and NSURLSession delegates in order to automatically add pinning validation to the App's HTTPS connections; this allows deploying TrustKit without even modifying the App's source code.

Getting Started

Read the Getting Started guide.
Check out the API documentation.
TrustKit was initially released at Black Hat USA 2015 and was also featured on PayPal's engineering blog.

Sample Usage

Deploying SSL pinning in the App requires initializing TrustKit with a pinning policy (domains, Subject Public Key Info hashes, and additional settings).

The policy can be configured within the App's Info.plist:

Alternatively, the pinning policy can be set programmatically:

    NSDictionary *trustKitConfig =
  @{
    kTSKSwizzleNetworkDelegates: @NO,
    kTSKPinnedDomains : @{
            @"www.datatheorem.com" : @{
                    kTSKExpirationDate: @"2017-12-01",
                    kTSKPublicKeyHashes : @[
                            @"HXXQgxueCIU5TTLHob/bPbwcKOKw6DkfsTWYHbxbqTY=",
                            @"0SDf3cRToyZJaMsoS17oF72VMavLxj/N7WBNasNuiR8="
                            ],
                    kTSKEnforcePinning : @NO,
                    },
            @"yahoo.com" : @{
                    kTSKPublicKeyHashes : @[
                            @"TQEtdMbmwFgYUifM4LDF+xgEtd0z69mPGmkp014d6ZY=",
                            @"rFjc3wG7lTZe43zeYTvPq8k4xdDEutCmIhI5dn4oCeE=",
                            ],
                    kTSKIncludeSubdomains : @YES
                    }
            }};
    
    [TrustKit initSharedInstanceWithConfiguration:trustKitConfig];

The policy can also be set programmatically in Swift Apps:

        let trustKitConfig = [
            kTSKSwizzleNetworkDelegates: false,
            kTSKPinnedDomains: [
                "yahoo.com": [
                    kTSKExpirationDate: "2017-12-01",
                    kTSKPublicKeyHashes: [
                        "JbQbUG5JMJUoI6brnx0x3vZF6jilxsapbXGVfjhN8Fg=",
                        "WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="
                    ],]]] as [String : Any]
        
        TrustKit.initSharedInstance(withConfiguration:trustKitConfig)

After TrustKit has been initialized, a TSKPinningValidator instance can be retrieved from the TrustKit singleton, and can be used to perform SSL pinning validation in the App's network delegates. For example in an NSURLSessionDelegate:

- (void)URLSession:(NSURLSession *)session 
              task:(NSURLSessionTask *)task 
didReceiveChallenge:(NSURLAuthenticationChallenge *)challenge 
 completionHandler:(void (^)(NSURLSessionAuthChallengeDisposition disposition, NSURLCredential *credential))completionHandler {
{
    TSKPinningValidator *pinningValidator = [[TrustKit sharedInstance] pinningValidator];
    // Pass the authentication challenge to the validator; if the validation fails, the connection will be blocked
    if (![pinningValidator handleChallenge:challenge completionHandler:completionHandler])
    {
        // TrustKit did not handle this challenge: perhaps it was not for server trust
        // or the domain was not pinned. Fall back to the default behavior
        completionHandler(NSURLSessionAuthChallengePerformDefaultHandling, nil);
    }
}

For more information, see the Getting Started guide.

Credits

TrustKit is a joint-effort between the mobile teams at Data Theorem and Yahoo. See AUTHORS for details.

License

TrustKit is released under the MIT license. See LICENSE for details.

SAP/macOS-enterprise-privileges

2024-12-01T01:56:29Z

For Mac users in enterprise environments, this application gives users control over the administration of their machine by elevating their level of access to administrator privileges on macOS. Users can set a timeframe in the application's settings to perform specific tasks, such as installing or removing an application.

Privileges

Privileges is an application for macOS which allows users to work as a standard user for day-to-day tasks, by providing a quick and easy way to request administrator rights. Users can set a timeframe in the application’s settings to perform specific tasks, such as installing or removing an application. To request admin rights, simply click on the Privileges icon in your Dock.

Working as standard user instead of an administrator adds another layer of security to your Mac and is considered a security best practice. We believe all users, including all developers, can benefit from using Privileges.

Privileges supports the following macOS versions:

macOS 15.x
macOS 14.x
macOS 13.x
macOS 12.x
macOS 11.x

Features

🛠️ Easy install

🚀 Perfect for day-to-day use

⏰ Turn on admin rights anytime

🔐 Enjoy standard user security

🔥 Command line use supported

New Privileges 2 features 🔥

📦 Installer package

⛔️ Revoke admin rights at login

⏳ Unified expiration interval for administrator privileges

▶️ Run actions on privilege change

👆 Command line tool now also supports Touch ID

⚙️ AppleScript support

🪝 Webhooks

🔠 Localized in 40 languages

Demo

⚡️ Have a look at how quick and easy you can request admin rights

Application Management

As of Privileges 1.5.0, it is possible to manage settings for Privileges or the PrivilegesCLI command line tool using a macOS configuration profile. For more details, please click here.

Articles

We are happy to share with you the article about privileges from 9to5Mac:

Apple @ Work: Privileges for macOS is the open source tool that all Apple IT departments need

Documentation

📚 If you want to learn more about Privileges features, make sure to take a look at our wiki or the links below:

License

Copyright (c) 2018-2024 SAP SE or an SAP affiliate company and macOS-enterprise-privileges contributors. Please see our LICENSE for copyright and license information. Detailed information including third-party components and their licensing/copyright information is available via the REUSE tool.

Security

Found a security-related issue or vulnerability and want to notify us? Please see here for how to report it.

Support

This project is 'as-is' with no support, no changes being made. You are welcome to make changes to improve it but we are not available for questions or support of any kind.

romaonthego/RESideMenu

2024-12-01T01:56:29Z

iOS 7/8 style side menu with parallax effect.

RESideMenu

iOS 7/8 style side menu with parallax effect inspired by Dribbble shots (first and second).

Since version 4.0 you can add menu view controllers on both left and right sides of your content view controller.

Requirements

Xcode 6 or higher
Apple LLVM compiler
iOS 6.0 or higher
ARC

Demo

Build and run the RESideMenuExample project in Xcode to see RESideMenu in action. For storyboards integration demo, build and run RESideMenuStoryboardsExample.

Installation

CocoaPods

The recommended approach for installating RESideMenu is via the CocoaPods package manager, as it provides flexible dependency management and dead simple installation. For best results, it is recommended that you install via CocoaPods >= 0.28.0 using Git >= 1.8.0 installed via Homebrew.

Install CocoaPods if not already available:

$ [sudo] gem install cocoapods
$ pod setup

Change to the directory of your Xcode project:

$ cd /path/to/MyProject
$ touch Podfile
$ edit Podfile

Edit your Podfile and add RESideMenu:

platform :ios, '6.0'
pod 'RESideMenu', '~> 4.0.7'

Install into your Xcode project:

$ pod install

Open your project in Xcode from the .xcworkspace file (not the usual project file)

$ open MyProject.xcworkspace

Please note that if your installation fails, it may be because you are installing with a version of Git lower than CocoaPods is expecting. Please ensure that you are running Git >= 1.8.0 by executing git --version. You can get a full picture of the installation details by executing pod install --verbose.

Manual Install

All you need to do is drop RESideMenu files into your project, and add #include "RESideMenu.h" to the top of classes that will use it.

Example Usage

In your AppDelegate's - (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions create the view controller and assign content and menu view controllers.

// Create content and menu controllers
//
DEMONavigationController *navigationController = [[DEMONavigationController alloc] initWithRootViewController:[[DEMOHomeViewController alloc] init]];
DEMOLeftMenuViewController *leftMenuViewController = [[DEMOLeftMenuViewController alloc] init];
DEMORightMenuViewController *rightMenuViewController = [[DEMORightMenuViewController alloc] init];

// Create side menu controller
//
RESideMenu *sideMenuViewController = [[RESideMenu alloc] initWithContentViewController:navigationController
                                                                leftMenuViewController:leftMenuViewController
                                                               rightMenuViewController:rightMenuViewController];
sideMenuViewController.backgroundImage = [UIImage imageNamed:@"Stars"];

// Make it a root controller
//
self.window.rootViewController = sideMenuViewController;

Present the menu view controller:

[self.sideMenuViewController presentLeftMenuViewController];

[self.sideMenuViewController presentRightMenuViewController];

Switch content view controllers:

#import <RESideMenu/RESideMenu.h>

....

[self.sideMenuViewController setContentViewController:viewController animated:YES];
[self.sideMenuViewController hideMenuViewController];

Storyboards Example

Create a subclass of RESideMenu. In this example we call it DEMORootViewController.
In the Storyboard designate the root view's owner as DEMORootViewController.
Make sure to #import "RESideMenu.h" in DEMORootViewController.h.
Add more view controllers to your Storyboard, and give them identifiers "leftMenuViewController", "rightMenuViewController" and "contentViewController". Note that in the new XCode the identifier is called "Storyboard ID" and can be found in the Identity inspector.
Add a method awakeFromNib to DEMORootViewController.m with the following code:

- (void)awakeFromNib
{
    self.contentViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"contentViewController"];
    self.leftMenuViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"leftMenuViewController"];
    self.rightMenuViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"rightMenuViewController"];
}

Customization

You can customize the following properties of RESideMenu:

@property (assign, readwrite, nonatomic) NSTimeInterval animationDuration;
@property (strong, readwrite, nonatomic) UIImage *backgroundImage;
@property (assign, readwrite, nonatomic) BOOL panGestureEnabled;
@property (assign, readwrite, nonatomic) BOOL panFromEdge;
@property (assign, readwrite, nonatomic) NSUInteger panMinimumOpenThreshold;
@property (assign, readwrite, nonatomic) BOOL interactivePopGestureRecognizerEnabled;
@property (assign, readwrite, nonatomic) BOOL scaleContentView;
@property (assign, readwrite, nonatomic) BOOL scaleBackgroundImageView;
@property (assign, readwrite, nonatomic) BOOL scaleMenuView;
@property (assign, readwrite, nonatomic) BOOL contentViewShadowEnabled;
@property (assign, readwrite, nonatomic) UIColor *contentViewShadowColor;
@property (assign, readwrite, nonatomic) CGSize contentViewShadowOffset;
@property (assign, readwrite, nonatomic) CGFloat contentViewShadowOpacity;
@property (assign, readwrite, nonatomic) CGFloat contentViewShadowRadius;
@property (assign, readwrite, nonatomic) CGFloat contentViewScaleValue;
@property (assign, readwrite, nonatomic) CGFloat contentViewInLandscapeOffsetCenterX;
@property (assign, readwrite, nonatomic) CGFloat contentViewInPortraitOffsetCenterX;
@property (assign, readwrite, nonatomic) CGFloat parallaxMenuMinimumRelativeValue;
@property (assign, readwrite, nonatomic) CGFloat parallaxMenuMaximumRelativeValue;
@property (assign, readwrite, nonatomic) CGFloat parallaxContentMinimumRelativeValue;
@property (assign, readwrite, nonatomic) CGFloat parallaxContentMaximumRelativeValue;
@property (assign, readwrite, nonatomic) CGAffineTransform menuViewControllerTransformation;
@property (assign, readwrite, nonatomic) BOOL parallaxEnabled;
@property (assign, readwrite, nonatomic) BOOL bouncesHorizontally;
@property (assign, readwrite, nonatomic) UIStatusBarStyle menuPreferredStatusBarStyle;
@property (assign, readwrite, nonatomic) BOOL menuPrefersStatusBarHidden;

If you set a backgroundImage, don't forget to set the Menu View Controller's background color to clear color.

You can implement RESideMenuDelegate protocol to receive the following messages:

- (void)sideMenu:(RESideMenu *)sideMenu didRecognizePanGesture:(UIPanGestureRecognizer *)recognizer;
- (void)sideMenu:(RESideMenu *)sideMenu willShowMenuViewController:(UIViewController *)menuViewController;
- (void)sideMenu:(RESideMenu *)sideMenu didShowMenuViewController:(UIViewController *)menuViewController;
- (void)sideMenu:(RESideMenu *)sideMenu willHideMenuViewController:(UIViewController *)menuViewController;
- (void)sideMenu:(RESideMenu *)sideMenu didHideMenuViewController:(UIViewController *)menuViewController;

Contact

Roman Efimov

License

RESideMenu is available under the MIT license.

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

SAP/macOS-enterprise-privileges

2024-12-01T01:47:49Z

Privileges

Privileges supports the following macOS versions:

macOS 15.x
macOS 14.x
macOS 13.x
macOS 12.x
macOS 11.x

Features

🛠️ Easy install

🚀 Perfect for day-to-day use

⏰ Turn on admin rights anytime

🔐 Enjoy standard user security

🔥 Command line use supported

New Privileges 2 features 🔥

📦 Installer package

⛔️ Revoke admin rights at login

⏳ Unified expiration interval for administrator privileges

▶️ Run actions on privilege change

👆 Command line tool now also supports Touch ID

⚙️ AppleScript support

🪝 Webhooks

🔠 Localized in 40 languages

Demo

⚡️ Have a look at how quick and easy you can request admin rights

Application Management

As of Privileges 1.5.0, it is possible to manage settings for Privileges or the PrivilegesCLI command line tool using a macOS configuration profile. For more details, please click here.

Articles

We are happy to share with you the article about privileges from 9to5Mac:

Apple @ Work: Privileges for macOS is the open source tool that all Apple IT departments need

Documentation

📚 If you want to learn more about Privileges features, make sure to take a look at our wiki or the links below:

License

Security

Found a security-related issue or vulnerability and want to notify us? Please see here for how to report it.

Support

This project is 'as-is' with no support, no changes being made. You are welcome to make changes to improve it but we are not available for questions or support of any kind.

button/DeepLinkKit

2024-12-01T01:47:49Z

A splendid route-matching, block-based way to handle your deep links.

DeepLink Kit

Overview

DeepLink Kit is a splendid route-matching, block-based way to handle your deep links. Rather than decide how to format your URLs, parse them, pass data, and navigate to specific content or perform actions, this library and a few lines of code will get you on your way.

Full Documentation

Guide to add Universal Links to your app

Check it out

Try the DeepLinkKit sample project by running the following command:

pod try "DeepLinkKit"

Installation

CocoaPods

DeepLinkKit is available through CocoaPods. To install the library, simply add the following line to your Podfile:

pod "DeepLinkKit"

Carthage

To install via Carthage, add the following line to your Cartfile:

github "button/DeepLinkKit"

Other

If you don't use CocoaPods or Carthage, you can include all of the source files from the DeepLinkKit directory in your project.

Usage

Add deep link support to your app in 5 minutes or less following these simple steps.

Note: As of 1.0.0, all imports should be updated to import <DeepLinkKit/DeepLinkKit.h>.

1. Make sure you have a URL scheme registered for your app in your Info.plist

2. Import DeepLinkKit

#import <DeepLinkKit/DeepLinkKit.h>

3. Create an instance of DPLDeepLinkRouter in your app delegate

- (BOOL)application:(UIApplication *)application
        didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {

  self.router = [[DPLDeepLinkRouter alloc] init];

  return YES;
}

4. Register a route handler

Objective-C

self.router[@"/log/:message"] = ^(DPLDeepLink *link) {
  NSLog(@"%@", link.routeParameters[@"message"]);
};

Swift

self.router.register("/log/:message") { link in
    if let link = link {
        print("\(link.routeParameters["message"])")
    }
}

5. Pass incoming URLs to the router

- (BOOL)application:(UIApplication *)application
            openURL:(NSURL *)url
  sourceApplication:(NSString *)sourceApplication
         annotation:(id)annotation {

  return [self.router handleURL:url withCompletion:NULL];
}

6. Passing NSUserActivity objects to the router (optional)

Note: If your application supports Apple's new universal links, implement the following in your app delegate:

- (BOOL)application:(UIApplication *)application
        continueUserActivity:(NSUserActivity *)userActivity
          restorationHandler:(void (^)(NSArray *))restorationHandler {

    return [self.router handleUserActivity:userActivity withCompletion:NULL];
}

Learn more about the DeepLinkKit by reading our Integration Guide.

Route Registration Examples

URLs coming into your app will be in a similar format to the following: <scheme>://<host>/<path-component>/<path-component>

When registering routes, it's important to note that the first forward slash in your registered route determines the start of the path to be matched. A route component before the first forward slash will be considered to be the host.

Say you have an incoming URL of twitter://timeline


// Matches the URL.
router[@"timeline"] = ^{ … }

// Does not match the URL.
router[@"/timeline"] = ^{ … }

In another example, a URL of twitter://dpl.com/timeline

// Matches the URL.
router[@"/timeline"] = ^{ … }

// Does not match the URL.
router[@"timeline"] = ^{ … }

You can also be scheme specific. If you support multiple URL schemes in your app, you can register routes specific to those schemes as follows:

An incoming URL of scheme-one://timeline

// Matches the URL.
router[@"scheme-one://timeline"] = ^{ … }

// Does not match the URL.
router[@"scheme-two://timeline"] = ^{ … }

Regex Route Matching

You can use regex in your route patterns as well to give you maximum flexibility.

Match any url

The following will match all incoming urls

router[@".*"] ^(DPLDeepLink *link){
  // This will match all incoming links
}

Note: Routes are matched in the order they're registered so registering this route first will prevent all other more specific routes from matching.

Match any url with a given scheme

The following will match all incoming links with the scheme, myscheme://

router[@"myscheme://.*"] ^(DPLDeepLink *link){
  // matches all urls with a scheme of `myscheme://`
}

You can name your regex groups too

The following will match any url with a host of trycaviar.com and hand you :path in the route params.

// Given the url ‘https://trycaviar.com/manhattan/nicoletta-297`
router[@"trycaviar.com/:path(.*)"] ^(DPLDeepLink *link){
  // `link[@"path"]` => @"manhattan/nicoletta-297"
}

Match multiple path components

In this example, you'll get :city and :restaurant in the route params.

// Given the url ‘https://trycaviar.com/manhattan/nicoletta-297`
router[@"trycaviar.com/:city([a-zA-Z]+)/:restaurant(.*)"] ^(DPLDeepLink *link){
  // `link[@"city"]` => @"manhattan"
  // `link[@"restaurant"]` => @"nicoletta-297"
}

If the restaurant ids are numbers, you could limit your matches as follows.

// Given the url ‘https://trycaviar.com/manhattan/297`
router[@"trycaviar.com/:city([a-zA-Z]+)/:restaurant([0-9])"] ^(DPLDeepLink *link){
  // `link[@"city"]` => @"manhattan"
  // `link[@"restaurant"]` => @"297"
}

Name some groups and not others

// Lets say the url is ‘https://trycaviar.com/manhattan/pizza/nicoletta-297`
router[@"trycaviar.com/:city([a-zA-Z]+)/[a-z]+/:restaurant(.*)"] ^(DPLDeepLink *link){
  // `link[@"city"]` => @"manhattan"
  // `link[@"restaurant"]` => @"nicoletta-297"
}

The above would match ‘https://trycaviar.com/manhattan/pizza/nicoletta-297’ but not ‘https://trycaviar.com/manhattan/PIZZA/nicoletta-297’ or ‘https://trycaviar.com/manhattan/pizza-places/nicoletta-297’, etc

AppLinks Support

Does your app support AppLinks? You can easily handle incoming AppLinks by importing the AppLinks category DPLDeepLink+AppLinks. The AppLinks category provides convenience accessors to all AppLinks 1.0 properties.

router[@"/timeline"] = ^(DPLDeepLink *link) {
  NSURL *referrerURL  = link.referralURL;
  NSString *someValue = link.extras[@"some-key"];
}

Running the Demo

To run the example project, run pod try DeepLinkKit in your terminal. You can also clone the repo, and run pod install from the project root. If you don't have CocoaPods, begin by follow this guide.

There are two demo apps, SenderDemo, and ReceiverDemo. ReceiverDemo has some registered routes that will handle specific deep links. SenderDemo has a couple actions that will deep link out to ReceiverDemo for fulfillment.

Run theSenderDemo build scheme first, then stop the simulator and switch the build scheme to ReceiverDemo and run again. Now you can switch back to the SenderDemo app in the simulator and tap on one of the actions.

Creating Deep Links

You can also create deep links with DPLMutableDeepLink. Between two DeepLinkKit integrated apps, you can pass complex objects via deep link from one app to another app and easily get that object back on the other end.

In the first app:


DPLMutableDeepLink *link = [[DPLMutableDeepLink alloc] initWithString:@"app-two://categories"];
link[@"brew-types"] = @[@"Ale", @"Lager", @"Stout", @"Wheat"]
link[@"beers"] = @{
  @"ales": @[
    @{
        @"name": @"Southern Tier Pumking Ale",
        @"price": @799
    },
    @{
        @"name": @"Sierra Nevada Celebration Ale",
        @"price": @799
    }
  ],
  @"lagers": @[
     ...
  ],
  ...
}

[[UIApplication sharedApplication] openURL:link.URL];

In the second app:

router[@"categories"] = ^(DPLDeepLink *link) {
  NSArray *brewTypes  = link[@"brew-types"];
  NSDictionary *beers = link[@"beers"];
}

Authors

Wes Smith
Chris Maddern

License

DeepLinkKit is available under the MIT license. See the LICENSE file for more info.

Contributing

We'd love to see your ideas for improving this library. The best way to contribute is by submitting a pull request. We'll do our best to respond to you as soon as possible. You can also submit a new Github issue if you find bugs or have questions. )

Please make sure to follow our general coding style and add test coverage for new features!

aquasecurity/trivy-checks

2024-12-01T01:37:07Z

aquasecurity/trivy-checks

2024-12-01T01:56:38Z

aquasecurity/trivy-checks

2024-12-01T01:47:58Z

ostat/gridfinity_extended_openscad

2024-12-01T01:56:41Z

Gridfinity Extended OpenSCAD Model

znuny/Znuny

2024-12-01T01:56:55Z

Znuny/Znuny LTS is a fork of the ((OTRS)) Community Edition, one of the most flexible web-based ticketing systems used for Customer Service, Help Desk, IT Service Management.

Znuny

Znuny is a continuation of the ((OTRS)) Community Edition (version 6.0.30) which was declared end of life (EOL) at the end of December 2020.

The primary goal for this project is to provide a maintained and stable version of the well known ticket system and improve it with new features.

The second goal is to reestablish a connection to the community.

License

The project is distributed under the GNU General Public License (GPL v3) - see the accompanying COPYING file for general license information. If you need more details you can have a look here.

Documentation

You can find documentation here. The source code of Znuny is publicly available on GitHub.

You want to get in touch?

Software requirements

Operating system

Linux (Debian or Red Hat preferred)
Perl 5.16.0 or higher

Web server

Apache 2 + mod_perl2 or higher (recommended)
Web server with CGI support (CGI is not recommended)

Databases

MySQL 8.0 or higher
MariaDB 10.3 or higher
PostgreSQL 12.0 or higher
Oracle 19c or higher

Browsers

These browsers are NOT supported:
- Internet Explorer before version 11
- Firefox before version 31
- Safari before version 6

Vendor

This project is mainly funded by Znuny GmbH, Berlin. If you need professional support or consulting, feel free to contact us.

Znuny Website

cplusplus/papers

2024-12-01T01:56:55Z

ISO/IEC JTC1 SC22 WG21 paper scheduling and management

papers

ISO/IEC JTC1 SC22 WG21 paper scheduling and management.

This repo is for tracking the status of papers and recording their progress, it is not for discussion of the content of papers.

Each proposal being actively considered by WG21 has an open issue. The labels on the issue indicate its current status.

liske/needrestart

2024-12-01T01:48:10Z

Restart daemons after library updates.

needrestart

About

needrestart checks which daemons need to be restarted after library upgrades. It is inspired by checkrestart from the debian-goodies package.

There are some hook scripts in the ex/ directory (to be used with apt and dpkg. The scripts will call needrestart after any package installation/upgrades.

needrestart should work on GNU/Linux. It has limited functionality on GNU/kFreeBSD since /proc/<pid>/maps does not show removed file links.

Restarting Services

needrestart supports but does not require systemd (available since v0.6). If systemd is used you should use libpam-systemd, too. If needrestart detects systemd it will assume that libpam-systemd is used and relies on cgroup names to detect if a process belongs to a user session or a daemon. If you do not use libpam-systemd you should set $nrconf{has_pam_systemd} to 0 within needrestart.conf.

If systemd is not available or does not return a service name needrestart uses hooks to identify the corresponding System V init script. The shipped hooks support the following package managers:

dpkg
rpm
pacman

The service command is used to run the traditional System V init script.

Frontends

needrestart uses a modular approach based on perl packages providing the user interface. The following frontends are shipped:

NeedRestart::UI::Debconf using debconf
NeedRestart::UI::stdio fallback using stdio interaction

Kernel & Microcode

needrestart 0.8 brings a obsolete kernel detection feature. Since needrestart 3.5 it is possible to filter kernel image filenames (required on Raspberry Pi).

In needrestart 3.0 a processor microcode update detection feature for Intel CPUs has been added. Since needrestart 3.5 the AMD CPU support has been added.

Interpreters

needrestart 0.8 brings an interpreter scanning feature. Interpreters not only map binary (shared) objects but also use plaintext source files. The interpreter detection tries to check for outdated source files since they may contain security issues, too. This is only a heuristic and might fail to detect all relevant source files. The following interpreter scanners are shipped:

NeedRestart::Interp::Java
NeedRestart::Interp::Perl
NeedRestart::Interp::Python
NeedRestart::Interp::Ruby

Containers

needrestart 2.1 detects some container technologies. If a process is part of a container it might not be possible to restart it using Sys-V/systemd.

There are special perl packages (NeedRestart::CONT::*) implementing the container detection and restarting. The following container detectors are shipped:

NeedRestart::CONT::docker
NeedRestart::CONT::LXC
NeedRestart::CONT::machined

Batch Mode

needrestart can be run in batch mode to use the results within other programs or scripts.

There is also a nagios plugin mode available.

PHPOffice/PhpSpreadsheet

2024-12-01T01:56:51Z

A pure PHP library for reading and writing spreadsheet files

PhpSpreadsheet

PhpSpreadsheet is a library written in pure PHP and offers a set of classes that allow you to read and write various spreadsheet file formats such as Excel and LibreOffice Calc.

Installation

See the install instructions.

Documentation

Read more about it, including install instructions, in the official documentation. Or check out the API documentation.

Please ask your support questions on StackOverflow, or have a quick chat on Gitter.

Patreon

I am now running a Patreon to support the work that I do on PhpSpreadsheet.

Supporters will receive access to articles about working with PhpSpreadsheet, and how to use some of its more advanced features.

Posts already available to Patreon supporters:

The Dating Game
- A look at how MS Excel (and PhpSpreadsheet) handle date and time values.
Looping the Loop
- Advice on Iterating through the rows and cells in a worksheet.

And for Patrons at levels actively using PhpSpreadsheet:

Behind the Mask
- A look at Number Format Masks.

The Next Article (currently Work in Progress):

Formula for Success
- How to debug formulae that don't produce the expected result.

My aim is to post at least one article each month, taking a detailed look at some feature of MS Excel and how to use that feature in PhpSpreadsheet, or on how to perform different activities in PhpSpreadsheet.

Planned posts for the future include topics like:

Tables
Structured References
AutoFiltering
Array Formulae
Conditional Formatting
Data Validation
Value Binders
Images
Charts

After a period of six months exclusive to Patreon supporters, articles will be incorporated into the public documentation for the library.

PHPExcel vs PhpSpreadsheet ?

PhpSpreadsheet is the next version of PHPExcel. It breaks compatibility to dramatically improve the code base quality (namespaces, PSR compliance, use of latest PHP language features, etc.).

Because all efforts have shifted to PhpSpreadsheet, PHPExcel will no longer be maintained. All contributions for PHPExcel, patches and new features, should target PhpSpreadsheet master branch.

Do you need to migrate? There is an automated tool for that.

License

PhpSpreadsheet is licensed under MIT.

larastan/larastan

2024-12-01T01:56:51Z

⚗️ Adds code analysis to Laravel improving developer productivity and code quality.

⚗️ About Larastan

Larastan was created by Can Vural and Nuno Maduro, got artwork designed by @Caneco, is maintained by Can Vural and Viktor Szépe, and is a PHPStan extension for Laravel. Larastan focuses on finding errors in your code. It catches whole classes of bugs even before you write tests for the code.

Adds static typing to Laravel to improve developer productivity and code quality
Supports most of Laravel's beautiful magic
Discovers bugs in your code

While by definition, "static analysis" doesn't load any of your application's code. Larastan boots your application's container, so it can resolve types that are only possible to compute at runtime. That's why we use the term "code analysis" instead of "static analysis".

Supported Laravel versions

Laravel Version	Larastan Version
< 9	1.x
> 9.0 && <11.15	2.x
11.15+	3.0+

✨ Getting Started In 3 Steps

Requires:

PHP 8.2+
Laravel 11.15+

1: First, you may use Composer to install Larastan as a development dependency into your Laravel project:

composer require --dev "larastan/larastan:^3.0"

Using Larastan for analysing Laravel packages? You may need to install orchestra/testbench.

2: Then, create a phpstan.neon or phpstan.neon.dist file in the root of your application. It might look like this:

includes:
    - vendor/larastan/larastan/extension.neon
    - vendor/nesbot/carbon/extension.neon

parameters:

    paths:
        - app/

    # Level 10 is the highest level
    level: 5

#    ignoreErrors:
#        - '#PHPDoc tag @var#'
#
#    excludePaths:
#        - ./*/*/FileToBeExcluded.php

For all available options, please take a look at the PHPStan documentation: https://phpstan.org/config-reference

3: Finally, you may start analyzing your code using the phpstan console command:

./vendor/bin/phpstan analyse

If you are getting the error Allowed memory size exhausted, then you can use the --memory-limit option fix the problem:

./vendor/bin/phpstan analyse --memory-limit=2G

Ignoring errors

Ignoring a specific error can be done either with a php comment or in the configuration file:

// @phpstan-ignore-next-line
$test->badMethod();

$test->badMethod(); // @phpstan-ignore-line

When ignoring errors in PHPStan's configuration file, they are ignored by writing a regex based on error messages:

parameters:
    ignoreErrors:
        - '#Call to an undefined method .*badMethod\(\)#'

Baseline file

In older codebases it might be hard to spend the time fixing all the code to pass a high PHPStan Level.

To get around this a baseline file can be generated. The baseline file will create a configuration file with all of the current errors, so new code can be written following a higher standard than the old code. (PHPStan Docs)

./vendor/bin/phpstan analyse --generate-baseline

Rules

A list of configurable rules specific to Laravel can be found here.

Features

A list of Larastan features can be found here.

Custom PHPDoc types

A list of PHPDoc types specific to Larastan can be found here.

Custom PHPStan config parameters

A list of custom config parameters that you can use in your PHPStan config file can be found here.

Errors To Ignore

Some parts of Laravel are currently too magical for Larastan/PHPStan to understand. We listed common errors to ignore, add them as needed

👊🏻 Contributing

Thank you for considering contributing to Larastan. All the contribution guidelines are mentioned here.

📖 License

Larastan is an open-sourced software licensed under the MIT license.

matomo-org/device-detector

2024-12-01T01:56:51Z

The Universal Device Detection library will parse any User Agent and detect the browser, operating system, device used (desktop, tablet, mobile, tv, cars, console, etc.), brand and model.

DeviceDetector

Code Status

Description

The Universal Device Detection library that parses User Agents and Browser Client Hints to detect devices (desktop, tablet, mobile, tv, cars, console, etc.), clients (browsers, feed readers, media players, PIMs, ...), operating systems, brands and models.

Usage

Using DeviceDetector with composer is quite easy. Just add matomo/device-detector to your projects requirements.

composer require matomo/device-detector

And use some code like this one:

require_once 'vendor/autoload.php';

use DeviceDetector\ClientHints;
use DeviceDetector\DeviceDetector;
use DeviceDetector\Parser\Device\AbstractDeviceParser;

// OPTIONAL: Set version truncation to none, so full versions will be returned
// By default only minor versions will be returned (e.g. X.Y)
// for other options see VERSION_TRUNCATION_* constants in DeviceParserAbstract class
AbstractDeviceParser::setVersionTruncation(AbstractDeviceParser::VERSION_TRUNCATION_NONE);

$userAgent = $_SERVER['HTTP_USER_AGENT']; // change this to the useragent you want to parse

// Client Hints are optional
// If you want to use them your server must announce that it supports client hints, using the Accept-CH header to specify the hints that it is interested in receiving.
// See e.g. https://developer.mozilla.org/en-US/docs/Web/HTTP/Client_hints
$clientHints = ClientHints::factory($_SERVER);

$dd = new DeviceDetector($userAgent, $clientHints);

// OPTIONAL: Set caching method
// By default static cache is used, which works best within one php process (memory array caching)
// To cache across requests use caching in files or memcache
// $dd->setCache(new Doctrine\Common\Cache\PhpFileCache('./tmp/'));

// OPTIONAL: Set custom yaml parser
// By default Spyc will be used for parsing yaml files. You can also use another yaml parser.
// You may need to implement the Yaml Parser facade if you want to use another parser than Spyc or [Symfony](https://github.com/symfony/yaml)
// $dd->setYamlParser(new DeviceDetector\Yaml\Symfony());

// OPTIONAL: If called, getBot() will only return true if a bot was detected  (speeds up detection a bit)
// $dd->discardBotInformation();

// OPTIONAL: If called, bot detection will completely be skipped (bots will be detected as regular devices then)
// $dd->skipBotDetection();

$dd->parse();

if ($dd->isBot()) {
  // handle bots,spiders,crawlers,...
  $botInfo = $dd->getBot();
} else {
  $clientInfo = $dd->getClient(); // holds information about browser, feed reader, media player, ...
  $osInfo = $dd->getOs();
  $device = $dd->getDeviceName();
  $brand = $dd->getBrandName();
  $model = $dd->getModel();
}

Methods check device type:

$dd->isSmartphone();
$dd->isFeaturePhone();
$dd->isTablet();
$dd->isPhablet();
$dd->isConsole();
$dd->isPortableMediaPlayer();
$dd->isCarBrowser();
$dd->isTV();
$dd->isSmartDisplay();
$dd->isSmartSpeaker();
$dd->isCamera();
$dd->isWearable();
$dd->isPeripheral();

Methods check client type:

$dd->isBrowser();
$dd->isFeedReader();
$dd->isMobileApp();
$dd->isPIM();
$dd->isLibrary();
$dd->isMediaPlayer();

Get OS family:

use DeviceDetector\Parser\OperatingSystem;

$osFamily = OperatingSystem::getOsFamily($dd->getOs('name'));

Get browser family:

use DeviceDetector\Parser\Client\Browser;

$browserFamily = Browser::getBrowserFamily($dd->getClient('name'));

Instead of using the full power of DeviceDetector it might in some cases be better to use only specific parsers. If you aim to check if a given useragent is a bot and don't require any of the other information, you can directly use the bot parser.

require_once 'vendor/autoload.php';

use DeviceDetector\Parser\Bot AS BotParser;

$botParser = new BotParser();
$botParser->setUserAgent($userAgent);

// OPTIONAL: discard bot information. parse() will then return true instead of information
$botParser->discardDetails();

$result = $botParser->parse();

if (!is_null($result)) {
    // do not do anything if a bot is detected
    return;
}

// handle non-bot requests

Using without composer

Alternatively to using composer you can also use the included autoload.php. This script will register an autoloader to dynamically load all classes in DeviceDetector namespace.

Device Detector requires a YAML parser. By default Spyc parser is used. As this library is not included you need to include it manually or use another YAML parser.

<?php

include_once 'path/to/spyc/Spyc.php';
include_once 'path/to/device-detector/autoload.php';

use DeviceDetector\ClientHints;
use DeviceDetector\DeviceDetector;
use DeviceDetector\Parser\Device\AbstractDeviceParser;

// OPTIONAL: Set version truncation to none, so full versions will be returned
// By default only minor versions will be returned (e.g. X.Y)
// for other options see VERSION_TRUNCATION_* constants in DeviceParserAbstract class
AbstractDeviceParser::setVersionTruncation(AbstractDeviceParser::VERSION_TRUNCATION_NONE);

$userAgent = $_SERVER['HTTP_USER_AGENT']; // change this to the useragent you want to parse
$clientHints = ClientHints::factory($_SERVER); // client hints are optional

$dd = new DeviceDetector($userAgent, $clientHints);

// ...

Caching

By default, DeviceDetector uses a built-in array cache. To get better performance, you can use your own caching solution:

You can create a class that implement DeviceDetector\Cache\CacheInterface
Or if your project uses a PSR-6 or PSR-16 compliant caching system (like symfony/cache or matthiasmullie/scrapbook), you can inject them the following way:

// Example with PSR-6 and Symfony
$cache = new \Symfony\Component\Cache\Adapter\ApcuAdapter();
$dd->setCache(
    new \DeviceDetector\Cache\PSR6Bridge($cache)
);

// Example with PSR-16 and ScrapBook
$cache = new \MatthiasMullie\Scrapbook\Psr16\SimpleCache(
    new \MatthiasMullie\Scrapbook\Adapters\Apc()
);
$dd->setCache(
    new \DeviceDetector\Cache\PSR16Bridge($cache)
);

// Example with Doctrine
$cache = new \Doctrine\Common\Cache\ApcuCache();
$dd->setCache(
    new \DeviceDetector\Cache\DoctrineBridge($cache)
);

// Example with Laravel
$dd->setCache(
    new \DeviceDetector\Cache\LaravelCache()
);

Contributing

Hacking the library

This is a free/libre library under license LGPL v3 or later.

Your pull requests and/or feedback is very welcome!

Listing all user agents from your logs

Sometimes it may be useful to generate the list of most used user agents on your website, extracting this list from your access logs using the following command:

zcat ~/path/to/access/logs* | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -n20000 > /home/matomo/top-user-agents.txt

Contributors

Created by the Matomo team, Stefan Giehl, Matthieu Aubry, Michał Gaździk, Tomasz Majczak, Grzegorz Kaszuba, Piotr Banaszczyk and contributors.

Together we can build the best Device Detection library.

We are looking forward to your contributions and pull requests!

Tests

Running tests

cd /path/to/device-detector
curl -sS https://getcomposer.org/installer | php
php composer.phar install
./vendor/bin/phpunit

Device Detector for other languages

There are already a few ports of this tool to other languages:

.NET https://github.com/totpero/DeviceDetector.NET
Ruby https://github.com/podigee/device_detector
JavaScript/TypeScript/NodeJS https://github.com/etienne-martin/device-detector-js
NodeJS https://github.com/sanchezzzhak/node-device-detector
Python 3 https://github.com/thinkwelltwd/device_detector
Crystal https://github.com/creadone/device_detector
Elixir https://github.com/elixir-inspector/ua_inspector
Java https://github.com/deevvicom/device-detector
Java https://github.com/PaniniGelato/java-device-detector
Rust https://github.com/simplecastapps/rust-device-detector
Rust https://github.com/stry-rs/device-detector
Go https://github.com/gamebtc/devicedetector
Go https://github.com/umutbasal/device-detector-go
Go https://github.com/robicode/device-detector

Icon packs

If you are looking for icons to use alongside Device Detector, these repositories can be of use:

Official Matomo pack
Unofficial Simbiat pack

What Device Detector is able to detect

The lists below are auto generated and updated from time to time. Some of them might not be complete.

Last update: 2024/11/24

List of detected operating systems:

AIX, Android, Android TV, Alpine Linux, Amazon Linux, AmigaOS, Armadillo OS, AROS, tvOS, Arch Linux, AOSC OS, ASPLinux, Azure Linux, BackTrack, Bada, Baidu Yi, BeOS, BlackBerry OS, BlackBerry Tablet OS, blackPanther OS, Bliss OS, Brew, BrightSignOS, Caixa Mágica, CentOS, CentOS Stream, Clear Linux OS, ClearOS Mobile, Chrome OS, Chromium OS, China OS, CyanogenMod, Debian, Deepin, DragonFly, DVKBuntu, ElectroBSD, EulerOS, Fedora, Fenix, Firefox OS, Fire OS, Foresight Linux, Freebox, FreeBSD, FRITZ!OS, FydeOS, Fuchsia, Gentoo, GENIX, GEOS, gNewSense, GridOS, Google TV, HP-UX, Haiku OS, iPadOS, HarmonyOS, HasCodingOS, HELIX OS, IRIX, Inferno, Java ME, Joli OS, KaiOS, Kali, Kanotix, KIN OS, Knoppix, KreaTV, Kubuntu, GNU/Linux, LeafOS, LindowsOS, Linspire, Lineage OS, Liri OS, Loongnix, Lubuntu, Lumin OS, LuneOS, VectorLinux, Mac, Maemo, Mageia, Mandriva, MeeGo, MocorDroid, moonOS, Motorola EZX, Mint, MildWild, MorphOS, NetBSD, MTK / Nucleus, MRE, NeXTSTEP, NEWS-OS, Nintendo, Nintendo Mobile, Nova, OS/2, OSF1, OpenBSD, OpenVMS, OpenVZ, OpenWrt, Opera TV, Oracle Linux, Ordissimo, Pardus, PCLinuxOS, PICO OS, Plasma Mobile, PlayStation Portable, PlayStation, Proxmox VE, Puffin OS, PureOS, Qtopia, Raspberry Pi OS, Raspbian, Red Hat, Red Star, RedOS, Revenge OS, risingOS, RISC OS, Rocky Linux, Roku OS, Rosa, RouterOS, Remix OS, Resurrection Remix OS, REX, RazoDroiD, Sabayon, SUSE, Sailfish OS, Scientific Linux, SeewoOS, SerenityOS, Sirin OS, Slackware, Solaris, Star-Blade OS, Syllable, Symbian, Symbian OS, Symbian OS Series 40, Symbian OS Series 60, Symbian^3, TencentOS, ThreadX, Tizen, TiVo OS, TmaxOS, Turbolinux, Ubuntu, ULTRIX, UOS, VIDAA, ViziOS, watchOS, Wear OS, WebTV, Whale OS, Windows, Windows CE, Windows IoT, Windows Mobile, Windows Phone, Windows RT, WoPhone, Xbox, Xubuntu, YunOS, Zenwalk, ZorinOS, iOS, palmOS, Webian, webOS

List of detected browsers:

Via, Pure Mini Browser, Pure Lite Browser, Raise Fast Browser, Rabbit Private Browser, Fast Browser UC Lite, Fast Explorer, Lightning Browser, Cake Browser, IE Browser Fast, Vegas Browser, OH Browser, OH Private Browser, XBrowser Mini, Sharkee Browser, Lark Browser, Pluma, Anka Browser, Azka Browser, Dragon Browser, Easy Browser, Dark Web Browser, Dark Browser, 18+ Privacy Browser, 115 Browser, 1DM Browser, 1DM+ Browser, 2345 Browser, 360 Secure Browser, 360 Phone Browser, 7654 Browser, Avant Browser, ABrowse, Acoo Browser, AdBlock Browser, Adult Browser, Ai Browser, Airfind Secure Browser, ANT Fresco, ANTGalio, Aloha Browser, Aloha Browser Lite, ALVA, Amaya, Amaze Browser, Amerigo, Amigo, Android Browser, AOL Explorer, AOL Desktop, AOL Shield, AOL Shield Pro, Aplix, AppBrowzer, AppTec Secure Browser, APUS Browser, Arora, Arctic Fox, Amiga Voyager, Amiga Aweb, APN Browser, Arachne, Arc Search, Armorfly Browser, Arvin, Ask.com, Asus Browser, Atom, Atomic Web Browser, Atlas, Avast Secure Browser, AVG Secure Browser, Avira Secure Browser, AwoX, Awesomium, Basic Web Browser, Beaker Browser, Beamrise, BF Browser, BlackBerry Browser, Bluefy, BrowseHere, Browser Hup Pro, Baidu Browser, Baidu Spark, Bang, Bangla Browser, Basilisk, Belva Browser, Beyond Private Browser, Beonex, Berry Browser, Bitchute Browser, BizBrowser, BlackHawk, Bloket, Bunjalloo, B-Line, Black Lion Browser, Blue Browser, Bonsai, Borealis Navigator, Brave, BriskBard, BroKeep Browser, Browspeed Browser, BrowseX, Browzar, Browlser, Browser Mini, BrowsBit, Biyubi, Byffox, BXE Browser, Camino, Catalyst, Catsxp, Cave Browser, CCleaner, CG Browser, ChanjetCloud, Chedot, Cherry Browser, Centaury, Cliqz, Coc Coc, CoolBrowser, Colibri, Columbus Browser, Comodo Dragon, Coast, Charon, CM Browser, CM Mini, Chrome Frame, Headless Chrome, Chrome, Chrome Mobile iOS, Conkeror, Chrome Mobile, Chowbo, Classilla, CoolNovo, Colom Browser, CometBird, Comfort Browser, COS Browser, Cornowser, Chim Lac, ChromePlus, Chromium, Chromium GOST, Cyberfox, Cheshire, Cromite, Crow Browser, Crusta, Craving Explorer, Crazy Browser, Cunaguaro, Chrome Webview, CyBrowser, dbrowser, Peeps dBrowser, Dark Web, Dark Web Private, Debuggable Browser, Decentr, Deepnet Explorer, deg-degan, Deledao, Delta Browser, Desi Browser, DeskBrowse, Dezor, Diigo Browser, DoCoMo, Dolphin, Dolphin Zero, Dorado, Dot Browser, Dooble, Dillo, DUC Browser, DuckDuckGo Privacy Browser, East Browser, Ecosia, Edge WebView, Every Browser, Epic, Elinks, EinkBro, Element Browser, Elements Browser, Eolie, Explore Browser, eZ Browser, EudoraWeb, EUI Browser, GNOME Web, G Browser, Espial TV Browser, fGet, Falkon, Faux Browser, Fire Browser, Fiery Browser, Firefox Mobile iOS, Firebird, Fluid, Fennec, Firefox, Firefox Focus, Firefox Reality, Firefox Rocket, Firefox Klar, Float Browser, Flock, Floorp, Flow, Flow Browser, Firefox Mobile, Fireweb, Fireweb Navigator, Flash Browser, Flast, Flyperlink, FOSS Browser, FreeU, Freedom Browser, Frost, Frost+, Fulldive, Galeon, Gener8, Ghostery Privacy Browser, GinxDroid Browser, Glass Browser, Godzilla Browser, Good Browser, Google Earth, Google Earth Pro, GOG Galaxy, GoBrowser, GoKu, GO Browser, GreenBrowser, Habit Browser, Halo Browser, Harman Browser, HasBrowser, Hawk Turbo Browser, Hawk Quick Browser, Helio, Hexa Web Browser, Hi Browser, hola! Browser, Holla Web Browser, HotBrowser, HotJava, HONOR Browser, HTC Browser, Huawei Browser Mobile, Huawei Browser, HUB Browser, iBrowser, iBrowser Mini, IBrowse, iDesktop PC Browser, iCab, iCab Mobile, iNet Browser, Iridium, Iron Mobile, IceCat, IceDragon, Isivioo, IVVI Browser, Iceweasel, Impervious Browser, Incognito Browser, Inspect Browser, Insta Browser, Internet Explorer, Internet Browser Secure, Internet Webbrowser, Intune Managed Browser, Indian UC Mini Browser, InBrowser, Involta Go, IE Mobile, Iron, Japan Browser, Jasmine, JavaFX, Jelly, Jig Browser, Jig Browser Plus, JioSphere, JUZI Browser, K.Browser, Keepsafe Browser, KeepSolid Browser, Kids Safe Browser, Kindle Browser, K-meleon, K-Ninja, Konqueror, Kapiko, Keyboard Browser, Kinza, Kitt, Kiwi, Kode Browser, KUN, KUTO Mini Browser, Kylo, Kazehakase, Cheetah Browser, Ladybird, Lagatos Browser, Legan Browser, Lexi Browser, Lenovo Browser, LieBaoFast, LG Browser, Light, Lightning Browser Plus, Lilo, Links, Liri Browser, LogicUI TV Browser, Lolifox, Lotus, Lovense Browser, LT Browser, LuaKit, LUJO TV Browser, Lulumi, Lunascape, Lunascape Lite, Lynx, Lynket Browser, Mandarin, MarsLab Web Browser, MaxBrowser, mCent, MicroB, NCSA Mosaic, Meizu Browser, Mercury, Me Browser, Mobile Safari, Midori, Midori Lite, MixerBox AI, Mobicip, Mi Browser, Mobile Silk, Mogok Browser, Motorola Internet Browser, Minimo, Mint Browser, Maxthon, MaxTube Browser, Maelstrom, Mises, Mmx Browser, MxNitro, Mypal, Monument Browser, MAUI WAP Browser, Naenara Browser, Navigateur Web, Naked Browser, Naked Browser Pro, NFS Browser, Ninetails, Nokia Browser, Nokia OSS Browser, Nokia Ovi Browser, Norton Private Browser, Nox Browser, NOMone VR Browser, NOOK Browser, NetSurf, NetFront, NetFront Life, NetPositive, Netscape, NextWord Browser, Ninesky, NTENT Browser, Nuanti Meta, Nuviu, Ocean Browser, Oculus Browser, Odd Browser, Opera Mini iOS, Obigo, Odin, Odin Browser, OceanHero, Odyssey Web Browser, Off By One, Office Browser, OhHai Browser, OnBrowser Lite, ONE Browser, Onion Browser, ONIONBrowser, Opera Crypto, Opera GX, Opera Neon, Opera Devices, Opera Mini, Opera Mobile, Opera, Opera Next, Opera Touch, Orbitum, Orca, Ordissimo, Oregano, Origin In-Game Overlay, Origyn Web Browser, OrNET Browser, Openwave Mobile Browser, OpenFin, Open Browser, Open Browser 4U, Open Browser fast 5G, Open Browser Lite, Open TV Browser, OmniWeb, Otter Browser, Owl Browser, OJR Browser, Palm Blazer, Pale Moon, Polypane, Prism, Oppo Browser, Opus Browser, Palm Pre, Puffin Cloud Browser, Puffin Incognito Browser, Puffin Secure Browser, Puffin Web Browser, Palm WebPro, Palmscape, Pawxy, Peach Browser, Perfect Browser, Perk, Phantom.me, Phantom Browser, Phoenix, Phoenix Browser, Photon, Pintar Browser, PirateBrowser, PICO Browser, PlayFree Browser, PocketBook Browser, Polaris, Polarity, PolyBrowser, Presearch, Privacy Browser, PrivacyWall, Privacy Explorer Fast Safe, Privacy Pioneer Browser, Private Internet Browser, Proxy Browser, Proxyium, Proxynet, ProxyFox, ProxyMax, Pi Browser, PronHub Browser, PSI Secure Browser, Reqwireless WebViewer, Roccat, Microsoft Edge, Qazweb, Qiyu, QJY TV Browser, Qmamu, Quick Search TV, QQ Browser Lite, QQ Browser Mini, QQ Browser, Quick Browser, Qutebrowser, Quark, QupZilla, Qwant Mobile, QtWeb, QtWebEngine, Rakuten Browser, Rakuten Web Search, Raspbian Chromium, RCA Tor Explorer, Realme Browser, Rekonq, RockMelt, Roku Browser, Samsung Browser, Samsung Browser Lite, Sailfish Browser, SberBrowser, Seewo Browser, SEMC-Browser, Sogou Explorer, Sogou Mobile Browser, SOTI Surf, Soul Browser, Soundy Browser, Safari, Safari Technology Preview, Safe Exam Browser, SalamWeb, Savannah Browser, SavySoda, Secure Browser, SFive, Shiira, Sidekick, SimpleBrowser, SilverMob US, Singlebox, Sizzy, Skye, Skyfire, SkyLeap, Seraphic Sraf, SiteKiosk, Sleipnir, SlimBoat, Slimjet, SP Browser, Sony Small Browser, Secure Private Browser, SecureX, Stampy Browser, 7Star, Smart Browser, Smart Search & Web Browser, Smart Lenovo Browser, Smooz, Snowshoe, Spark, Spectre Browser, Splash, Sputnik Browser, Sunrise, Sunflower Browser, SuperBird, Super Fast Browser, SuperFast Browser, Sushi Browser, surf, Surf Browser, Surfy Browser, Stargon, START Internet Browser, Stealth Browser, Steam In-Game Overlay, Streamy, Swiftfox, Swiftweasel, Seznam Browser, Sweet Browser, SX Browser, T+Browser, T-Browser, t-online.de Browser, TalkTo, Tao Browser, tararia, Thor, Tor Browser, TenFourFox, Tenta Browser, Tesla Browser, Tizen Browser, Tint Browser, TrueLocation Browser, TUC Mini Browser, TUSK, Tungsten, ToGate, Total Browser, TQ Browser, TweakStyle, TV Bro, TV-Browser Internet, U Browser, UBrowser, UC Browser, UC Browser HD, UC Browser Mini, UC Browser Turbo, Ui Browser Mini, UPhone Browser, UR Browser, Uzbl, Ume Browser, vBrowser, Vast Browser, VD Browser, Veera, Venus Browser, Vewd Browser, VibeMate, Nova Video Downloader Pro, Viasat Browser, Vivaldi, vivo Browser, Vivid Browser Mini, Vision Mobile Browser, Vertex Surf, VMware AirWatch, VMS Mosaic, Vonkeror, Vuhuv, Wear Internet Browser, Web Explorer, Web Browser & Explorer, Webian Shell, WebDiscover, WebPositive, Weltweitimnetz Browser, Wexond, Waterfox, Wave Browser, Wavebox, Whale Browser, Whale TV Browser, wOSBrowser, w3m, WeTab Browser, World Browser, Wolvic, Wukong Browser, Wyzo, YAGI, Yahoo! Japan Browser, Yandex Browser, Yandex Browser Corp, Yandex Browser Lite, Yaani Browser, Yo Browser, Yolo Browser, YouCare, YouBrowser, Yuzu Browser, xBrowser, MMBOX XBrowser, X Browser Lite, X-VPN, xBrowser Pro Super Fast, XNX Browser, XtremeCast, xStand, Xiino, XnBrowse, Xooloo Internet, Xvast, Zetakey, Zvu, Zirco Browser, Zordo Browser, ZTE Browser

List of detected browser engines:

WebKit, Blink, Trident, Text-based, Dillo, iCab, Elektra, Presto, Clecko, Gecko, KHTML, NetFront, Edge, NetSurf, Servo, Goanna, EkiohFlow, Arachne, LibWeb

List of detected libraries:

aiohttp, Akka HTTP, Android License Verification Library, AnyEvent HTTP, Apache HTTP Client, Aria2, Artifactory, Axios, Azure Blob Storage, Azure Data Factory, Babashka HTTP Client, Boto3, Buildah, BuildKit, C++ REST SDK, CakePHP, CarrierWave, Containerd, containers, cPanel HTTP Client, cpp-httplib, cri-o, curl, Cygwin, Cypress, Dart, Deno, docker, Down, Electron Fetch, Emacs, Embarcadero URI Client, ESP32 HTTP Client, Faraday, fasthttp, ffmpeg, FFUF, FileDownloader, Free Download Manager, GeoIP Update, git-annex, go-container registry, Go-http-client, Google HTTP Java Client, got, GRequests, gRPC-Java, GStreamer, Guzzle (PHP HTTP Client), gvfs, hackney, Harbor registry client, Helm, HTML Parser, http.rb, HTTP:Tiny, HTTPie, httplib2, httprs, HTTPX, HTTP_Request2, ICAP Client, Insomnia REST Client, iOS Application, IPinfo, Jakarta Commons HttpClient, Jaunt, Java, Java HTTP Client, jsdom, KaiOS Downloader, Kiwi TCMS, Kiwi TCMS API, libdnf, libpod, libsoup, Libsyn, LUA OpenResty NGINX, Mandrill PHP, Mechanize, Mikrotik Fetch, Msray-Plus, Node Fetch, OKDownload Library, OkHttp, Open Build Service, Pa11y, Perl, Perl REST::Client, PhantomJS, PHP cURL Class, Podgrab, Postman Desktop, PRDownloader, Python Requests, Python urllib, QbHttp, quic-go, r-curl, Radio Downloader, ReactorNetty, req, request, Requests, reqwest, REST Client for Ruby, RestSharp, Resty, resty-requests, ruby, Safari View Service, ScalaJ HTTP, Skopeo, SlimerJS, Slim Framework, Stealer, superagent, Symfony, Typhoeus, uclient-fetch, Ultimate Sitemap Parser, undici, Unirest for Java, urlgrabber (yum), uTorrent, Wget, Windows HTTP, WinHttp WinHttpRequest, WWW-Mechanize, XML-RPC

List of detected media players:

Alexa, Amarok, Audacious, Banshee, Boxee, Clementine, Deezer, DIGA, Downcast, FlyCast, Foobar2000, Google Podcasts, HTC Streaming Player, Hubhopper, iTunes, JHelioviewer, JRiver Media Center, Juice, Just Audio, Kasts, Kodi, MediaMonkey, Miro, MixerBox, MPlayer, mpv, MusicBee, Music Player Daemon, NexPlayer, Nightingale, QuickTime, Songbird, SONOS, Sony Media Go, Stagefright, StudioDisplay, SubStream, VLC, Winamp, Windows Media Player, XBMC, Xtream Player, YouView

List of detected mobile apps:

'sodes, +Simple, 1Password, 2tch, ActionExtension, Adobe Acrobat Reader, Adobe Creative Cloud, Adobe IPM, Adobe NGL, Adobe Synchronizer, Adori, Agora, Aha Radio 2, AIDA64, Airr, Airsonic, AliExpress, Alipay, AllHitMusicRadio, All You Can Books, Amazon Fire, Amazon Music, Amazon Shopping, Ameba, Anchor, AnchorFM, AndroidDownloadManager, Anghami, AntennaPod, AntiBrowserSpy, AN WhatsApp, Anybox, Anytime Podcast Player, APK Downloader, Apollo, Apple iMessage, Apple News, Apple Podcasts, Apple Reminders, Apple TV, Arvocast, ASUS Updater, Audacy, Audials, Audible, Audio, Audiobooks, Audio Now, Autoplius.lt, Avid Link, Awasu, Background Intelligent Transfer Service, Baidu Box App, Baidu Input, Ballz, Bank Millenium, Battle.net, BB2C, BBC News, Bear, Be Focused, BetBull, BeyondPod, Bible, Bible KJV, Binance, Bitcoin Core, Bitsboard, Bitwarden, Blackboard, Blitz, Blue Proxy, BlueStacks, Bolt, BonPrix, Bookmobile, Bookshelf, Boom, Boom360, Boomplay, Bose Music, Bose SoundTouch, bPod, Breez, Bridge, Broadcast, Broadway Podcast Network, Browser-Anonymizer, Browser app, BrowserPlus, Bullhorn, BuzzVideo, CamScanner, Capital, capsule.fm, Castamatic, Castaway, CastBox, Castro, Castro 2, CCleaner, CGN, ChMate, Chrome Update, Ciisaa, Citrix Workspace, Classic FM, Client, Clipbox+, Clovia, COAF SMART Citizen, Coinbase, Cooler, Copied, Cortana, Cosmicast, Covenant Eyes, CPod, CPU-Z, CrosswalkApp, Crypto.com DeFi Wallet, CSDN, Damus, Daum, De Standaard, De Telegraaf, DevCasts, DeviantArt, DingTalk, DIRECTV, Discord, DManager, Dogecoin Core, DoggCatcher, Don't Waste My Time!, douban App, DoubleTwist CloudPlayer, Doughnut, Douyin, Downcast, Downie, Downloader, Dr. Watson, DStream Air, Edge Update, EMAudioPlayer, Emby Theater, Epic Games Launcher, ESET Remote Administrator, eToro, Evolve Podcast, Expedia, Expo, F-Secure SAFE, Facebook, Facebook Audience Network, Facebook Groups, Facebook Lite, Facebook Messenger, Facebook Messenger Lite, faidr, Fathom, FeedR, FeedStation, Fiddler Classic, Files, Flipboard App, Flipp, FM WhatsApp, Focus Keeper, Focus Matrix, Fountain, Freespoke, Gaana, Garmin fenix 5X, Garmin Forerunner, GBWhatsApp, GetPodcast, Git, GitHub Desktop, GlobalProtect, GoEuro, Gold, GoldenPod, GoLoud, GoNative, Goodpods, GoodReader, Google Assistant, Google Drive, Google Fiber TV, Google Go, Google Lens, Google Nest Hub, Google Photos, Google Play Newsstand, Google Plus, Google Podcasts, Google Search App, Google Tag Manager, GroupMe, Guacamole, Hago, Hammel, HandBrake, HardCast, Hark Audio, Heart, HermesPod, HeyTapBrowser, HiCast, HideX, Hik-Connect, Himalaya, HiSearch, HisThumbnail, HistoryHound, Hotels.com, HP Smart, HTTP request maker, Huawei Mobile Services, HyperCatcher, iCatcher, iHeartRadio, IMO HD Video Calls & Chat, IMO International Calls & Chat, Instabridge, Instacast, Instagram, Instapaper, iPlayTV, IPTV, IPTV Pro, iVoox, Jam, JaneStyle, JioSaavn, Jitsi Meet, JJ2GO, Jungle Disk, Just Listen, Kajabi, KakaoTalk, Keeper Password Manager, Kids Listen, KidsPod, Kik, KKBOX, Klarna, KPN Veilig, Kwai, Kwai Pro, L.A. Times, Landis+Gyr AIM Browser, Lark, Laughable, Lazada, LBC, LG Player, Line, LinkedIn, Listen, LiSTNR, Liulo, Logi Options+, LoseIt!, Luminary, Macrium Reflect, MBolsa, Megaphone, MEmpresas, Menucast, Mercantile Bank of Michigan, MessengerX, Meta Business Suite, Metacast, MetaMask, MetaTrader, Microsoft Bing, Microsoft Copilot, Microsoft Lync, Microsoft Office, Microsoft Office Access, Microsoft Office Excel, Microsoft Office Mobile, Microsoft Office OneDrive for Business, Microsoft Office OneNote, Microsoft Office PowerPoint, Microsoft Office Project, Microsoft Office Publisher, Microsoft Office Visio, Microsoft Office Word, Microsoft OneDrive, Microsoft Power Query, Microsoft Start, Microsoft Store, Mimir, mobile.de, MobileSMS, Mojeek, MOMO, MoonFM, mowPod, Moya, MX Player, My Bentley, MyTuner, My Watch Party, My World, nate, Naver, NAVER Dictionary, NET.mede, Netflix, NewsArticle App, Newsly, Nextcloud, NPR, NRC, NRC Audio, NTV Mobil, NuMuKi Browser, Obsidian, OBS Studio, Odnoklassniki, OfferUp, Opal Travel, Opera News, Opera Updater, Orange Radio, Outcast, Overcast, Overhaul FM, Paint by Number, Palco MP3, Pandora, Papers, PeaCast, Perplexity, Petal Search, Pic Collage, Pinterest, Player FM, PLAYit, Plex Media Server, Pocket Casts, Podbay, Podbean, Podcast & Radio Addict, Podcast App, Podcast Guru, Podcastly, Podcast Player, Podcast Republic, Podcat, Podcatcher Deluxe, Podchaser, Podclipper, PodCruncher, Podeo, Podfriend, Podhero, Podimo, PodKast, Podkicker, Podkicker Classic, Podkicker Pro, PodLP, PodMe, PodMN, PodNL, Podopolo, Podplay, Pods, PodTrapper, podU, Podurama, Podverse, Podvine, Podyssey, PowerShell, Procast, PugPig Bolt, Q-municate, qBittorrent, QQ, QQMusic, QuickCast, Quicksilver, Quora, R, radio.at, radio.de, radio.dk, radio.es, radio.fr, radio.it, radio.net, radio.pl, radio.pt, radio.se, RadioApp, Radio Italiane, Radioline, Radio Next, RadioPublic, Rave Social, Razer Synapse, RDDocuments, Reddit, Redditor, rekordbox, Repod, Reuters News, Rhythmbox, RNPS Action Cards, Roblox, RoboForm, Rocket Chat, RSSDemon, RSSRadio, Rutube, SachNoi, Safari Search Helper, SafeIP, Samsung Magician, Samsung Podcasts, SearchCraft, ServeStream, Shadow, Shadowrocket, Shopee, ShowMe, Signal, Sina Weibo, Siri, SiriusXM, Skyeng, Skyeng Teachers, Skype, Skype for Business, Slack, Snapchat, Snipd, SoFi, SogouSearch App, SohuNews, Soldier, Sonnet, Sony PlayStation 5, SOOP, SoundOn, SoundWaves, SPORT1, Spotify, Spreaker, Startsiden, Stitcher, StoryShots, Streamlabs OBS, Stream Master, Strimio, Surfshark, Swinsian, Swoot, Taobao, Teams, Tencent Docs, The Crossword, The Epoch Times, The New York Times, The Wall Street Journal, Theyub, Threads, Thunder, tieba, TikTok, TikTok Lite, TIM, TiviMate, TopBuzz, TopSecret Chat, TownNews Now, TracePal, Trade Me, TradingView, Treble.fm, TRP Retail Locator, TuneIn Radio, TuneIn Radio Pro, Turtlecast, Tuya Smart Life, TVirl, twinkle, Twitch Studio, Twitter, Twitterrific, U-Cursos, Ubook Player, UCast, Uconnect LIVE, Uforia, Unibox, UnityPlayer, Viber, Victor Reader Stream 3, Victor Reader Stream New Generation, Victor Reader Stream Trek, Virgin Radio, Visha, Visual Studio Code, Vodacast, Vuhuv, Vuze, waipu.tv, Walla News, WatchFree+, Wattpad, Wayback Machine, WebDAV, WeChat, WeChat Share Extension, Whatplay, WhatsApp, WhatsApp+2, WhatsApp Business, Whisper, WH Questions, Windows Antivirus, Windows CryptoAPI, Windows Delivery Optimization, Windows Push Notification Services, Windows Update Agent, Wireshark, Wirtschafts Woche, WNYC, Word Cookies!, WPS Office, Wynk Music, Xiao Yu Zhou, XING, XShare, XSplit Broadcaster, Y8 Browser, Yahoo! Japan, Yahoo OneSearch, YakYak, Yandex, Yandex Music, Yapa, Yelp Mobile, YouTube, Youtube Music, Yo WhatsApp, Zalo, ZEIT ONLINE, Zen, ZEPETO, Zoho Chat, Zune and mobile apps using AFNetworking or Electron

List of detected PIMs (personal information manager):

Airmail, Apple Mail, Barca, Basecamp, BathyScaphe, BlueMail, DAVdroid, eM Client, Evernote, Foxmail, Franz, Gmail, JaneView, Live5ch, Lotus Notes, mailapp, MailBar, Mailbird, Mail Master, Mailspring, Microsoft Outlook, NAVER Mail, Notion, Outlook Express, Postbox, Raindrop.io, Rambox Pro, SeaMonkey, Spicebird, The Bat!, Thunderbird, Windows Mail, Yahoo! Mail, Yahoo Mail

List of detected feed readers:

Akregator, Apple PubSub, BashPodder, Breaker, castero, castget, FeedDemon, Feeddler RSS Reader, gPodder, JetBrains Omea Reader, Liferea, NetNewsWire, Newsbeuter, NewsBlur, NewsBlur Mobile App, Newsboat, Playapod, PodPuppy, PritTorrent, Pulp, QuiteRSS, ReadKit, Reeder, RSS Bandit, RSS Junkie, RSSOwl, Stringer

List of brands with detected devices:

2E, 3GNET, 3GO, 3Q, 4Good, 4ife, 5IVE, 7 Mobile, 10moons, 360, 8848, A&K, A1, A95X, AAUW, Accent, Accesstyle, Ace, Aceline, Acer, Acteck, actiMirror, Adreamer, Adronix, Advan, Advance, Advantage Air, AEEZO, AFFIX, AfriOne, AGM, AG Mobile, AIDATA, Ainol, Airis, Airness, AIRON, Airpha, Airtel, Airties, AirTouch, AIS, Aiuto, Aiwa, Ajib, Akai, AKIRA, Alba, Alcatel, Alcor, ALDI NORD, ALDI SÜD, Alfawise, Alienware, Aligator, AllCall, AllDocube, ALLINmobile, Allview, Allwinner, Alps, Altech UEC, Altice, Altimo, altron, Altus, AMA, Amazon, Amazon Basics, AMCV, AMGOO, Amigoo, Amino, Amoi, ANBERNIC, ANCEL, Andowl, Angelcare, AngelTech, Anker, Anry, ANS, ANXONIT, AOC, Aocos, Aocwei, AOpen, Aoro, Aoson, AOYODKG, Apple, Aquarius, Archos, Arian Space, Arival, Ark, ArmPhone, Arnova, ARRIS, Artel, Artizlee, ArtLine, Arçelik, Asano, Asanzo, Ask, Aspera, ASSE, Assistant, Astro, Asus, AT&T, Athesi, Atlantic Electrics, Atmaca Elektronik, ATMAN, ATMPC, ATOL, Atom, Atouch, Atozee, Attila, Atvio, Audiovox, AUPO, AURIS, Autan, AUX, Avaya, Avenzo, AVH, Avvio, Awow, AWOX, AXEN, Axioo, AXXA, Axxion, AYA, AYYA, Azeyou, AZOM, Azumi Mobile, Azupik, b2m, Backcell, BAFF, BangOlufsen, Barnes & Noble, BARTEC, BASE, BAUHN, BBK, BB Mobile, BDF, BDQ, BDsharing, Beafon, Becker, Beeline, Beelink, Beetel, Beista, Beko, Bell, Bellphone, Benco, Benesse, BenQ, BenQ-Siemens, BenWee, Benzo, Beyond, Bezkam, BGH, Biegedy, Bigben, BIHEE, BilimLand, Billion, Billow, BioRugged, Bird, Bitel, Bitmore, Bittium, Bkav, Black Bear, Black Box, Black Fox, Blackpcs, Blackton, Blackview, Blaupunkt, Bleck, BLISS, Blloc, Blow, Blu, Bluboo, Bluebird, Bluedot, Bluegood, BlueSky, Bluewave, BluSlate, BMAX, Bmobile, BMW, BMXC, Bobarry, bogo, Bolva, Bookeen, Boost, Botech, Boway, bq, BrandCode, Brandt, BRAVE, Bravis, BrightSign, Brigmton, Brondi, BROR, BS Mobile, Bubblegum, Bundy, Bush, BuzzTV, BYD, BYJU'S, BYYBUO, C5 Mobile, CADENA, CAGI, Camfone, Canaima, Canal+, Canal Digital, Canguro, Capitel, Captiva, Carbon Mobile, Carrefour, Casio, Casper, Cat, Cavion, CCIT, Cecotec, Ceibal, Celcus, Celkon, Cell-C, Cellacom, CellAllure, Cellution, CENTEK, Centric, CEPTER, CG Mobile, CGV, Chainway, Changhong, CHCNAV, Cherry Mobile, Chico Mobile, ChiliGreen, China Mobile, China Telecom, Chuwi, C Idea, CipherLab, Citycall, CKK Mobile, Claresta, Clarmin, CLAYTON, ClearPHONE, Clementoni, Cloud, Cloudfone, Cloudpad, Clout, Clovertek, CMF, CnM, Cobalt, Coby Kyros, COLORROOM, Colors, Comio, Compal, Compaq, COMPUMAX, ComTrade Tesla, Conceptum, Concord, ConCorde, Condor, Connectce, Connex, Conquest, Continental Edison, Contixo, COOD-E, Coolpad, Coopers, CORN, Cosmote, Covia, Cowon, COYOTE, CPDEVICE, CreNova, Crescent, Cricket, Crius Mea, Crony, Crosscall, Crown, Ctroniq, Cube, CUBOT, Cuiud, Cultraview, CVTE, Cwowdefu, CX, Cyrus, D-Link, D-Tech, Daewoo, Danew, DangcapHD, Dany, Daria, DASS, Datalogic, Datamini, Datang, Datawind, Datsun, Dazen, DbPhone, Dbtel, Dcode, DEALDIG, Dell, Denali, Denver, Desay, DeWalt, DEXP, DEYI, DF, DGTEC, DIALN, Dialog, Dicam, Digi, Digicel, DIGICOM, Digidragon, DIGIFORS, Digihome, Digiland, Digit4G, Digma, DIJITSU, DIKOM, DIMO, Dinalink, Dinax, DING DING, DIORA, DISH, Disney, Ditecma, Diva, DiverMax, Divisat, DIXON, DL, DMM, DMOAO, DNS, DoCoMo, Doffler, Dolamee, Dom.ru, Doogee, Doopro, Doov, Dopod, Doppio, DORLAND, Doro, DPA, DRAGON, Dragon Touch, Dreamgate, DreamStar, DreamTab, Droidlogic, Droxio, DSDevices, DSIC, Dtac, DUDU AUTO, Dune HD, DUNNS Mobile, Durabook, Duubee, Dykemann, Dyon, E-Boda, E-Ceros, E-TACHI, E-tel, Eagle, EagleSoar, EAS Electric, Easypix, EBEN, EBEST, Echo Mobiles, ecom, ECON, ECOO, ECS, Edenwood, EE, EFT, EGL, EGOTEK, Ehlel, Einstein, EKINOX, EKO, Eks Mobility, EKT, ELARI, ELE-GATE, Elecson, Electroneum, ELECTRONIA, Elekta, Elektroland, Element, Elenberg, Elephone, Elevate, Elong Mobile, Eltex, Ematic, Emporia, ENACOM, Energizer, Energy Sistem, Engel, ENIE, Enot, eNOVA, Entity, Envizen, Ephone, Epic, Epik One, Epson, Equator, Ergo, Ericsson, Ericy, Erisson, Essential, Essentielb, eSTAR, ETOE, Eton, eTouch, Etuline, Eurocase, Eurostar, Evercoss, Everest, Everex, Everis, Evertek, Evolio, Evolveo, Evoo, EVPAD, EvroMedia, evvoli, EWIS, EXCEED, Exmart, ExMobile, EXO, Explay, Express LUCK, ExtraLink, Extrem, Eyemoo, EYU, Ezio, Ezze, F&U, F+, F2 Mobile, F150, Facebook, Facetel, Facime, Fairphone, Famoco, Famous, Fantec, FaRao Pro, Farassoo, FarEasTone, Fengxiang, Fenoti, FEONAL, Fero, FFF SmartLife, Figgers, FiGi, FiGO, FiiO, Filimo, FILIX, FinePower, Finlux, FireFly Mobile, FISE, FITCO, Fluo, Fly, FLYCAT, FLYCOAY, FMT, FNB, FNF, Fobem, Fondi, Fonos, FOODO, FORME, Formuler, Forstar, Fortis, FortuneShip, FOSSiBOT, Fourel, Four Mobile, Foxconn, FoxxD, FPT, free, Freetel, FreeYond, Frunsi, Fuego, Fujitsu, Funai, Fusion5, Future Mobile Technology, Fxtec, G-PLUS, G-TiDE, G-Touch, Galactic, Galaxy Innovations, Gamma, Garmin-Asus, Gateway, Gazer, GDL, Geanee, Geant, Gear Mobile, Gemini, General Mobile, Genesis, GEOFOX, Geotel, Geotex, GEOZON, Getnord, GFive, Gfone, Ghia, Ghong, Ghost, Gigabyte, Gigaset, Gini, Ginzzu, Gionee, GIRASOLE, Globex, Globmall, GlocalMe, Glofiish, GLONYX, Glory Star, GLX, GOCLEVER, Gocomma, GoGEN, GOLDBERG, GoldMaster, GoldStar, Gol Mobile, Goly, Gome, GoMobile, GOODTEL, Google, Goophone, Gooweel, GOtv, Gplus, Gradiente, Graetz, Grape, Great Asia, Gree, Green Lion, Green Orange, Greentel, Gresso, Gretel, GroBerwert, Grundig, Gtel, GTMEDIA, GTX, Guophone, GVC Pro, H96, H133, Hafury, Haier, Haipai, Haixu, Hamlet, Hammer, Handheld, HannSpree, Hanseatic, Hanson, HAOQIN, HAOVM, Hardkernel, Harper, Hartens, Hasee, Hathway, HDC, HeadWolf, HEC, Heimat, Helio, Hemilton, HERO, HexaByte, Hezire, Hi, Hi-Level, Hiberg, HiBy, High Q, Highscreen, HiGrace, HiHi, HiKing, HiMax, Hi Nova, HIPER, Hipstreet, Hiremco, Hisense, Hitachi, Hitech, HKC, HKPro, HLLO, HMD, HOFER, Hoffmann, HOLLEBERG, Homatics, Hometech, Homtom, Honeywell, HongTop, HONKUAHG, Hoozo, Hopeland, Horizon, Horizont, Hosin, Hotel, Hot Pepper, HOTREALS, Hotwav, How, HP, HTC, Huadoo, Huagan, Huavi, Huawei, Hugerock, Humanware, Humax, Hurricane, Huskee, Hyatta, Hykker, Hyrican, Hytera, Hyundai, Hyve, i-Cherry, I-INN, i-Joy, i-mate, i-mobile, I-Plus, iBall, iBerry, ibowin, iBrit, IconBIT, iData, iDino, iDroid, iFIT, iGet, iHome Life, iHunt, I KALL, Ikea, IKI Mobile, iKoMo, iKon, iKonia, IKU Mobile, iLA, iLepo, iLife, iMan, Imaq, iMars, iMI, IMO Mobile, Imose, Impression, iMuz, iNavi, INCAR, Inch, Inco, iNew, Infiniton, Infinix, InFocus, InfoKit, Infomir, InFone, Inhon, Inka, Inkti, InnJoo, Innos, Innostream, Inoi, iNo Mobile, iNOVA, inovo, INQ, Insignia, INSYS, Intek, Intel, Intex, Invens, Inverto, Invin, iOcean, IOTWE, iOutdoor, iPEGTOP, iPro, iQ&T, IQM, IRA, Irbis, iReplace, Iris, iRobot, iRola, iRulu, iSafe Mobile, iStar, iSWAG, IT, iTel, iTruck, IUNI, iVA, iView, iVooMi, ivvi, iWaylink, iXTech, iYou, iZotron, Jambo, JAY-Tech, Jedi, Jeep, Jeka, Jesy, JFone, Jiake, Jiayu, Jinga, Jin Tu, Jio, Jivi, JKL, Jolla, Joy, JoySurf, JPay, JREN, Jumper, Juniper Systems, Just5, JVC, JXD, K-Lite, K-Touch, Kaan, Kaiomy, Kalley, Kanji, Kapsys, Karbonn, Kata, KATV1, Kazam, Kazuna, KDDI, Kempler & Strauss, Kenbo, Kendo, Keneksi, KENSHI, KENWOOD, Kenxinda, Khadas, Kiano, kidiby, Kingbox, Kingstar, Kingsun, KINGZONE, Kinstone, Kiowa, Kivi, Klipad, KN Mobile, Kocaso, Kodak, Kogan, Komu, Konka, Konrow, Koobee, Koolnee, Kooper, KOPO, Korax, Koridy, Koslam, Kraft, KREZ, KRIP, KRONO, Krüger&Matz, KT-Tech, KUBO, KuGou, Kuliao, Kult, Kumai, Kurio, KVADRA, Kvant, Kydos, Kyocera, Kyowon, Kzen, KZG, L-Max, LAIQ, Land Rover, Landvo, Lanin, Lanix, Lark, Laser, Laurus, Lava, LCT, Leader Phone, Leagoo, Leben, LeBest, Lectrus, Ledstar, LeEco, Leelbox, Leff, Legend, Leke, Lemco, LEMFO, Lemhoov, Lenco, Lenovo, Leotec, Le Pan, Lephone, Lesia, Lexand, Lexibook, LG, Liberton, Lifemaxx, Lime, Lingbo, Lingwin, Linnex, Linsar, Linsay, Listo, LNMBBS, Loewe, Logic, Logic Instrument, Logicom, Logik, Logitech, LOKMAT, LongTV, Loview, Lovme, LPX-G, LT Mobile, Lumigon, Lumitel, Lumus, Luna, Luxor, Lville, LW, LYF, LYOTECH LABS, M-Horse, M-KOPA, M-Tech, M.T.T., M3 Mobile, M4tel, MAC AUDIO, Macoox, Mafe, MAG, MAGCH, Magicsee, Magnus, Majestic, Malata, Mango, Manhattan, Mann, Manta Multimedia, Mantra, Mara, Marshal, Mascom, Massgo, Masstel, Master-G, Mastertech, Matco Tools, Matrix, Maunfeld, Maxcom, Maxfone, Maximus, Maxtron, MAXVI, Maxwell, Maxwest, MAXX, Maze, Maze Speed, MBI, MBK, MBOX, MDC Store, MDTV, meanIT, Mecer, Mecool, Mediacom, MediaTek, Medion, MEEG, Megacable, MegaFon, MEGA VISION, Meitu, Meizu, Melrose, MeMobile, Memup, MEO, MESWAO, Meta, Metz, MEU, MicroMax, Microsoft, Microtech, Mightier, Minix, Mint, Mintt, Mio, Mione, mipo, Miray, Mitchell & Brown, Mito, Mitsubishi, Mitsui, MIVO, MIWANG, MIXC, MiXzo, MLAB, MLLED, MLS, MMI, Mobell, Mobicel, MobiIoT, Mobiistar, Mobile Kingdom, Mobiola, Mobistel, MobiWire, Mobo, Mobvoi, Modecom, Mode Mobile, Mofut, Moondrop, Mosimosi, Motiv, Motorola, Motorola Solutions, Movic, MOVISUN, Movitel, Moxee, mPhone, Mpman, MSI, MStar, MTC, MTN, Multilaser, MultiPOS, MwalimuPlus, MYFON, MyGica, MygPad, Mymaga, MyMobile, MyPhone (PH), myPhone (PL), Myria, Myros, Mystery, MyTab, MyWigo, N-one, Nabi, NABO, Nanho, Naomi Phone, NASCO, National, Navcity, Navitech, Navitel, Navon, NavRoad, NEC, Necnot, Nedaphone, Neffos, NEKO, Neo, neoCore, Neolix, Neomi, Neon IQ, Neoregent, NetBox, Netgear, Netmak, NETWIT, NeuImage, NeuTab, NEVIR, Newal, New Balance, New Bridge, Newgen, Newland, Newman, Newsday, NewsMy, Nexa, NEXBOX, Nexian, NEXON, NEXT, Next & NextStar, Nextbit, NextBook, NextTab, NGM, NG Optics, NGpon, Nikon, NINETEC, NINETOLOGY, Nintendo, nJoy, NOA, Noain, Nobby, Noblex, NOBUX, noDROPOUT, NOGA, Nokia, Nomi, Nomu, Noontec, Nordfrost, Nordmende, NORMANDE, NorthTech, Nos, Nothing, Nous, Novacom, Novex, Novey, NoviSea, NOVO, NTT West, NuAns, Nubia, NUU Mobile, NuVision, Nuvo, Nvidia, NYX Mobile, O+, O2, Oale, Oangcc, OASYS, Obabox, Ober, Obi, OCEANIC, Odotpad, Odys, Oilsky, OINOM, Ok, Okapi, Okapia, Oking, OKSI, OKWU, Olax, Olkya, Ollee, OLTO, Olympia, OMIX, Onda, OneClick, OneLern, OnePlus, Onida, Onix, Onkyo, ONN, ONVO, ONYX BOOX, Ookee, Ooredoo, OpelMobile, Openbox, Ophone, OPPO, Opsson, Optoma, Orange, Orange Pi, Orava, Orbic, Orbita, Orbsmart, Ordissimo, Orion, OSCAL, OTTO, OUJIA, Ouki, Oukitel, OUYA, Overmax, Ovvi, Owwo, OX TAB, OYSIN, Oysters, Oyyu, OzoneHD, P-UP, Pacific Research Alliance, Packard Bell, Padpro, PAGRAER, Paladin, Palm, Panacom, Panasonic, Panavox, Pano, Panodic, Panoramic, Pantech, PAPYRE, Parrot Mobile, Partner Mobile, PCBOX, PCD, PCD Argentina, PC Smart, PEAQ, Pelitt, Pendoo, Penta, Pentagram, Perfeo, Phicomm, Philco, Philips, Phonemax, phoneOne, Pico, PINE, Pioneer, Pioneer Computers, PiPO, PIRANHA, Pixela, Pixelphone, PIXPRO, Pixus, Planet Computers, Platoon, Play Now, Ployer, Plum, PlusStyle, Pluzz, PocketBook, POCO, Point Mobile, Point of View, Polar, PolarLine, Polaroid, Polestar, PolyPad, Polytron, Pomp, Poppox, POPTEL, Porsche, Portfolio, Positivo, Positivo BGH, PPTV, Premier, Premio, Prestigio, PRIME, Primepad, Primux, Pritom, Prixton, PROFiLO, Proline, Prology, ProScan, PROSONIC, Protruly, ProVision, PULID, Punos, Purism, PVBox, Q-Box, Q-Touch, Q.Bell, QFX, Qilive, QIN, QLink, QMobile, Qnet Mobile, QTECH, Qtek, Quantum, Quatro, Qubo, Quechua, Quest, Quipus, Qumo, Qware, QWATT, R-TV, Rakuten, Ramos, Raspberry, Ravoz, Raylandz, Razer, RCA Tablets, RCT, Reach, Readboy, Realix, Realme, RED, Redbean, Redfox, RedLine, Redway, Reeder, REGAL, RelNAT, Relndoo, Remdun, Renova, RENSO, rephone, Retroid Pocket, Revo, Revomovil, Rhino, Ricoh, Rikomagic, RIM, Ringing Bells, Rinno, Ritmix, Ritzviva, Riviera, Rivo, Rizzen, ROADMAX, Roadrover, Roam Cat, ROCH, Rocket, ROiK, Rokit, Roku, Rombica, Ross&Moor, Rover, RoverPad, Royole, RoyQueen, RTK, RT Project, RugGear, RuggeTech, Ruggex, Ruio, Runbo, Rupa, Ryte, S-Color, S-TELL, S2Tel, Saba, Safaricom, Sagem, Sagemcom, Saiet, SAILF, Salora, Samsung, Samtech, Samtron, Sanei, Sankey, Sansui, Santin, SANY, Sanyo, Savio, Sber, Schneider, Schok, Scoole, Scosmos, Seatel, SEBBE, Seeken, SEEWO, SEG, Sega, SEHMAX, Selecline, Selenga, Selevision, Selfix, SEMP TCL, Sencor, Sendo, Senkatel, SENNA, Senseit, Senwa, SERVO, Seuic, Sewoo, SFR, SGIN, Shanling, Sharp, Shift Phones, Shivaki, Shtrikh-M, Shuttle, Sico, Siemens, Sigma, Silelis, Silent Circle, Silva Schneider, Simbans, simfer, Simply, Singtech, Siragon, Sirin Labs, Siswoo, SK Broadband, SKG, SKK Mobile, Sky, Skyline, SkyStream, Skytech, Skyworth, Smadl, Smailo, Smart, Smartab, SmartBook, SMARTEC, Smart Electronic, Smartex, Smartfren, Smartisan, Smart Kassel, Smarty, Smooth Mobile, Smotreshka, SMT Telecom, SMUX, SNAMI, SobieTech, Soda, Softbank, Soho Style, Solas, SOLE, SOLO, Solone, Sonim, SONOS, Sony, Sony Ericsson, SOSH, SoulLink, Soundmax, Soyes, Spark, Sparx, SPC, Spectralink, Spectrum, Spice, Sprint, SPURT, SQOOL, SSKY, Star, Starlight, Starmobile, Starway, Starwind, STF Mobile, STG Telecom, STK, Stonex, Storex, StrawBerry, Stream, STRONG, Stylo, Subor, Sugar, Sumvision, Sunmax, Sunmi, Sunny, Sunstech, SunVan, Sunvell, SUNWIND, SuperBOX, Super General, SuperSonic, SuperTab, Supra, Supraim, Surfans, Surge, Suzuki, Sveon, Swipe, SWISSMOBILITY, Swisstone, Switel, SWOFY, Syco, SYH, Sylvania, Symphony, Syrox, System76, T-Mobile, T96, TADAAM, TAG Tech, Taiga System, Takara, Talius, Tambo, Tanix, TAUBE, TB Touch, TCL, TCL SCBC, TD Systems, TD Tech, TeachTouch, Technicolor, Technika, TechniSat, Technopc, TECHNOSAT, TechnoTrend, TechPad, TechSmart, Techstorm, Techwood, Teclast, Tecno Mobile, TecToy, TEENO, Teknosa, Tele2, Telefunken, Telego, Telenor, Telia, Telit, Telkom, Telly, Telma, TeloSystems, Telpo, Temigereev, TENPLUS, Teracube, Tesco, Tesla, TETC, Tetratab, teXet, ThL, Thomson, Thuraya, TIANYU, Tibuta, Tigers, Time2, Timovi, TIMvision, Tinai, Tinmo, TiPhone, Tivax, TiVo, TJC, TJD, TOKYO, Tolino, Tone, TOOGO, Tooky, Top-Tech, TopDevice, TOPDON, Topelotek, Top House, Toplux, TOPSHOWS, Topsion, Topway, Torex, Torque, TOSCIDO, Toshiba, Touchmate, Touch Plus, TOX, TPS, Transpeed, TrekStor, Trevi, TriaPlay, Trident, Trifone, Trimble, Trio, Tronsmart, True, True Slim, Tsinghua Tongfang, TTEC, TTfone, TTK-TV, TuCEL, Tunisie Telecom, Turbo, Turbo-X, TurboKids, TurboPad, Turkcell, Tuvio, TVC, TwinMOS, TWM, Twoe, TWZ, TYD, Tymes, Türk Telekom, U-Magic, U.S. Cellular, UD, UE, UGINE, Ugoos, Uhans, Uhappy, Ulefone, Umax, UMIDIGI, Umiio, Unblock Tech, Uniden, Unihertz, Unimax, Uniqcell, Uniscope, Unistrong, Unitech, United Group, UNIWA, Unknown, Unnecto, Unnion Technologies, UNNO, Unonu, Unowhy, UOOGOU, Urovo, UTime, UTOK, UTStarcom, UZ Mobile, V-Gen, V-HOME, V-HOPE, v-mobile, VAIO, VALE, VALEM, VALTECH, VANGUARD, Vankyo, VANWIN, Vargo, Vastking, VAVA, VC, VDVD, Vega, Veidoo, Vekta, Venso, Venstar, Venturer, VEON, Verico, Verizon, Vernee, Verssed, Versus, Vertex, Vertu, Verykool, Vesta, Vestel, VETAS, Vexia, VGO TEL, ViBox, Victurio, VIDA, Videocon, Videoweb, ViewSonic, VIIPOO, VIKUSHA, VILLAON, VIMOQ, Vinabox, Vinga, Vinsoc, Vios, Viper, Vipro, Virzo, Vision Technology, Vision Touch, Visual Land, Vitelcom, Vityaz, Viumee, Vivax, VIVIBright, VIVIMAGE, Vivo, VIWA, Vizio, Vizmo, VK Mobile, VKworld, VNPT Technology, VOCAL, Vodacom, Vodafone, VOGA, VOLIA, VOLKANO, Volla, Volt, Vonino, Vontar, Vorago, Vorcom, Vorke, Vormor, Vortex, Voto, VOX, Voxtel, Voyo, Vsmart, Vsun, VUCATIMES, Vue Micro, Vulcan, VVETIME, Völfen, W&O, WAF, Wainyok, Walker, Walton, Waltter, Wanmukang, WANSA, WE, We. by Loewe., Webfleet, Web TV, WeChip, Wecool, Weelikeit, Weiimi, Weimei, WellcoM, WELLINGTON, Western Digital, Westpoint, Wexler, White Mobile, Whoop, Wieppo, Wigor, Wiko, Wileyfox, Winds, Wink, Winmax, Winnovo, Winstar, Wintouch, Wiseasy, WIWA, WizarPos, Wizz, Wolder, Wolfgang, Wolki, WONDER, Wonu, Woo, Wortmann, Woxter, WOZIFAN, WS, X-AGE, X-BO, X-Mobile, X-TIGI, X-View, X.Vision, X88, X96, X96Q, Xcell, XCOM, Xcruiser, XElectron, XGEM, XGIMI, Xgody, Xiaodu, Xiaolajiao, Xiaomi, Xion, Xolo, Xoro, XPPen, XREAL, Xshitou, Xsmart, Xtouch, Xtratech, Xwave, XY Auto, Yandex, Yarvik, YASIN, YELLYOUTH, YEPEN, Yes, Yestel, Yezz, Yoka TV, Yooz, Yota, YOTOPT, Youin, Youwei, Ytone, Yu, Yuandao, YU Fly, YUHO, YUMKEM, YUNDOO, Yuno, YunSong, Yusun, Yxtel, Z-Kai, Zaith, Zamolxe, Zatec, Zealot, Zeblaze, Zebra, Zeeker, Zeemi, Zen, Zenek, Zentality, Zfiner, ZH&K, Zidoo, ZIFRO, Zigo, ZIK, Zinox, Ziox, Zonda, Zonko, Zoom, ZoomSmart, Zopo, ZTE, Zuum, Zync, ZYQ, Zyrex, ZZB, öwn

List of detected bots:

2GDPR, 2ip, 360 Monitoring, 360JK, 360Spider, 1001FirmsBot, Abonti, Aboundexbot, AccompanyBot, Acoon, AdAuth, Adbeat, AddThis.com, ADMantX, ADmantX Service Fetcher, Adsbot, Adscanner, AdsTxtCrawler, AdsTxtCrawlerTP, adstxtlab.com, Aegis, aHrefs Bot, AhrefsSiteAudit, aiHitBot, Alexa Crawler, Alexa Site Audit, Allloadin Favicon Bot, AlltheWeb, Amazon AdBot, Amazon Bot, Amazon ELB, Amazon Route53 Health Check, Amorank Spider, Analytics SEO Crawler, Ant, Anthropic AI, ApacheBench, Applebot, AppSignalBot, Arachni, archive.org bot, ArchiveBot, ArchiveBox, Arquivo.pt, ARSNova Filter System, Asana, Ask Jeeves, AspiegelBot, Automattic Analytics, Awario, Backlink-Check.de, BacklinkCrawler, BackupLand, Baidu Spider, Barkrowler, Barracuda Sentinel, BazQux Reader, BBC Forge URL Monitor, BBC Page Monitor, BDCbot, BDFetch, Better Uptime Bot, BingBot, Birdcrawlerbot, BitlyBot, BitSight, Blackbox Exporter, Blekkobot, BLEXBot Crawler, Bloglines, Bloglovin, Blogtrottr, BoardReader, BoardReader Blog Indexer, Botify, Bountii Bot, BrandVerity, Bravebot, BrightBot, BrightEdge, Browsershots, BUbiNG, Buck, BuiltWith, Butterfly Robot, Bytespider, CareerBot, Castopod, Castro 2, Catchpoint, CATExplorador, ccBot crawler, CensysInspect, Charlotte, Chartable, ChatGPT-User, CheckHost, CheckMark Network, Choosito, Chrome Privacy Preserving Prefetch Proxy, Cincraw, CISPA Web Analyzer, CLASSLA-web, ClaudeBot, Clickagy, Cliqzbot, CloudFlare Always Online, CloudFlare AMP Fetcher, Cloudflare Custom Hostname Verification, Cloudflare Diagnostics, Cloudflare Health Checks, Cloudflare Observatory, Cloudflare Security Insights, Cloudflare Smart Transit, Cloudflare SSL Detector, Cloudflare Traffic Manager, CloudServerMarketSpider, Cocolyzebot, Cohere AI, Collectd, colly, CommaFeed, COMODO DCV, Comscore, ContentKing, Convertify, Cookiebot, Cotoyogi, Crawldad, Crawlson, Crawly Project, CriteoBot, CrowdTangle, CSS Certificate Spider, CSSCheck, CyberFind Crawler, Cyberscan, Cốc Cốc Bot, DaspeedBot, Datadog Agent, DataForSeoBot, datagnionbot, Datanyze, Dataprovider, DataXu, Daum, Dazoobot, Deepfield Genome, deepnoc, Deep SEARCH 9, DepSpid, Detectify, Diffbot, Discobot, Discord Bot, Disqus, DNSResearchBot, DomainAppender, Domain Codex, DomainCrawler, Domain Re-Animator Bot, Domain Research Project, Domains Project, DomainStatsBot, DomCop Bot, DotBot, Dotcom Monitor, Dubbotbot, DuckAssistBot, DuckDuckBot, ducks.party, DuplexWeb-Google, DynatraceSynthetic, Easou Spider, eCairn-Grabber, EFF Do Not Track Verifier, Elastic Synthetics, EMail Exractor, EmailWolf, Embedly, Entfer, evc-batch, Everyfeed, ExaBot, ExactSeek Crawler, Example3, Exchange check, Expanse, EyeMonit, Ezgif, Ezooms, eZ Publish Link Validator, FacebookBot, Facebook Crawler, Faveeo, Feedbin, FeedBurner, Feedly, Feedspot, Feed Wrangler, Femtosearch, Fever, Findxbot, Flipboard, FontRadar, fragFINN, FreeWebMonitoring, FreshRSS, Functionize, Gaisbot, GDNP, GeedoBot, GeedoProductSearch, Generic Bot, Genieo Web filter, Ghost Inspector, Gigablast, Gigabot, GitCrawlerBot, GitHubCopilotChat, Gluten Free Crawler, Gmail Image Proxy, Gobuster, Golfe, Goo, Google-CloudVertexBot, Google-Document-Conversion, Google-Safety, Google Apps Script, Google Area 120 Privacy Policy Fetcher, Googlebot, Googlebot News, Google Cloud Scheduler, Google Docs, Google Favicon, Google PageSpeed Insights, Google Partner Monitoring, Google Search Console, Google Sheets, Google Slides, Google Stackdriver Monitoring, Google StoreBot, Google Structured Data Testing Tool, Google Transparency Report, Gowikibot, Gozle, GPTBot, Grammarly, Grapeshot, Gregarius, GTmetrix, GumGum Verity, hackermention, Hatena Bookmark, Hatena Favicon, Headline, Heart Rails Capture, Heritrix, Heureka Feed, htmlyse, HTTPMon, httpx, HuaweiWebCatBot, HubPages, HubSpot, ICC-Crawler, ichiro, IDG/IT, Iframely, IIS Site Analysis, ImageSift, Inetdex Bot, Infegy, InfoTigerBot, Inktomi Slurp, inoreader, Inspici, InsytfulBot, Intelligence X, Interactsh, InternetMeasurement, IONOS Crawler, IP-Guide Crawler, IPIP, IPS Agent, IsItWP, iTMS, JobboerseBot, JungleKeyThumbnail, K6, KadoBot, Kaspersky, KeyCDN Tools, Keys.so, Kiwi TCMS GitOps, KlarnaBot, KomodiaBot, Konturbot, Kouio, Kozmonavt, KStandBot, l9explore, l9tcpid, LAC IA Harvester, Larbin web crawler, LastMod Bot, LCC, leak.info, LeakIX, Let's Encrypt Validation, LetSearch, Lighthouse, Linespider, Linkdex Bot, LinkedIn Bot, LinkpadBot, LinkPreview, LinkWalker, LiveJournal, LTX71, Lumar, LumtelBot, Lycos, MaCoCu, MADBbot, Magpie-Crawler, MagpieRSS, Mail.Ru Bot, MakeMerryBot, Marginalia, MariaDB/MySQL Knowledge Base, masscan, masscan-ng, Mastodon Bot, Matomo, Meanpath Bot, Mediatoolkit Bot, MegaIndex, MeltwaterNews, Meta-ExternalAgent, Meta-ExternalFetcher, MetaInspector, MetaJobBot, MicroAdBot, Microsoft Preview, Miniature.io, Mixnode, Mixrank Bot, MJ12 Bot, Mnogosearch, MojeekBot, Monitor.Us, Monitor Backlinks, Monsidobot, Montastic Monitor, MoodleBot Linkchecker, Morningscore Bot, MTRobot, MuckRack, Munin, MuscatFerret, Nagios check_http, NalezenCzBot, NameProtectBot, nbertaupete95, Neevabot, Netcraft Survey Bot, netEstate, NetLyzer FastProbe, Netpeak Checker, NetResearchServer, NetSystemsResearch, NetTrack, Netvibes, NETZZAPPEN, NewsBlur, NewsGator, Newslitbot, NiceCrawler, Nimbostratus Bot, NLCrawler, Nmap, NodePing, Notify Ninja, Nutch-based Bot, Nuzzel, OAI-SearchBot, oBot, Octopus, Odin, Odnoklassniki Bot, Oh Dear, Omgili bot, OmtrBot, Onalytica, OnlineOrNot Bot, Openindex Spider, OpenLinkProfiler, OpenWebSpider, Orange Bot, OSZKbot, Outbrain, Overcast Podcast Sync, OWLer, Pageburst, Page Modified Pinger, PagePeeker, PageThing, Pandalytics, Panscient, PaperLiBot, Paqlebot, parse.ly, PATHspider, PayPal IPN, PDR Labs, Peer39, PerplexityBot, Petal Bot, Phantomas, phpMyAdmin, PHP Server Monitor, Picsearch bot, Pigafetta, PingAdmin.Ru, Pingdom Bot, Pinterest, PiplBot, Plesk Screenshot Service, Plukkie, Pocket, Podroll Analyzer, PodUptime, Pompos, Prerender, PritTorrent, Probely, Project Patchwatch, Project Resonance, Prometheus, PRTG Network Monitor, Punk Map, Quantcast, QuerySeekerSpider, Quora Bot, Quora Link Preview, Qwantbot, Rainmeter, RamblerMail Image Proxy, RavenCrawler, Reddit Bot, RedekenBot, RenovateBot, Repo Lookout, ReqBin, Research Scan, Riddler, Robozilla, RocketMonitorBot, Rogerbot, ROI Hunter, RSSRadio Bot, RuxitSynthetic, Ryowl, SabsimBot, SafeDNSBot, Sandoba//Crawler, Scamadviser External Hit, Scooter, ScoutJet, Scraping Robot, Scrapy, Screaming Frog SEO Spider, ScreenerBot, Sectigo DCV, security.txt scanserver, Seekport, Sellers.Guide, semaltbot, Semantic Scholar Bot, SemrushBot, Semrush Reputation Management, Sensika Bot, Sentry Bot, Senuto, Seobility, SEOENGBot, SEOkicks, SEOkicks-Robot, seolyt, Seolyt Bot, Seoscanners.net, Serendeputy Bot, Serenety, serpstatbot, Server Density, Seznam Bot, Seznam Email Proxy, Seznam Zbozi.cz, sfFeedReader, ShopAlike, Shopify Partner, ShopWiki, SilverReader, SimplePie, Sirdata, SISTRIX Crawler, SISTRIX Optimizer, Site24x7 Defacement Monitor, Site24x7 Website Monitoring, SiteAuditBot, Sitebulb, Siteimprove, SitemapParser-VIPnytt, SiteOne Crawler, SiteScore, SiteSucker, Sixy.ch, Skype URI Preview, Slackbot, SMTBot, Snapchat Ads, Snapchat Proxy, Snap URL Preview Service, Sogou Spider, Soso Spider, Sparkler, Spawning AI, SpazioDati, Speedy, SpiderLing, Spinn3r, SplitSignalBot, Spotify, Sprinklr, Sputnik Bot, Sputnik Favicon Bot, Sputnik Image Bot, sqlmap, SSL Labs, start.me, Startpagina Linkchecker, Statista, StatOnline.ru, StatusCake, Steam Chat URL Lookup, Steve Bot, Stract, Sublinq, Substack Content Fetch, SuggestBot, Superfeedr Bot, SurdotlyBot, Survey Bot, Swiftbot, Swisscows Favicons, Synapse, t3versions, Taboolabot, TactiScout, Tag Inspector, Tarmot Gezgin, tchelebi, TelegramBot, Tenable.asm, TestCrawler, The British Library Legal Deposit Bot, The Knowledge AI, theoldreader, The Trade Desk Content, ThinkChaos, ThousandEyes, TigerBot, Timpibot, TinEye, TinEye Crawler, Tiny Tiny RSS, TLSProbe, TraceMyFile, Trendiction Bot, Trendsmap, Turnitin, TurnitinBot, TweetedTimes Bot, Tweetmeme Bot, Twingly Recon, Twitterbot, Twurly, UCSB Network Measurement, UkrNet Mail Proxy, uMBot, UniversalFeedParser, Uptime-Kuma, Uptimebot, UptimeRobot, Uptimia, URLAppendBot, URLinspector, URLSuMaBot, Vagabondo, ValidBot, Velen Public Web Crawler, Vercel Bot, VeryHip, Viber Url Downloader, VirusTotal Cloud, Visual Site Mapper Crawler, VK Robot, VK Share Button, vuhuvBot, VU Server Health Scanner, W3C CSS Validator, W3C I18N Checker, W3C Link Checker, W3C Markup Validation Service, W3C MobileOK Checker, W3C P3P Validator, W3C Unified Validator, Wappalyzer, WDG HTML Validator, WebbCrawler, WebCEO, WebDataStats, WebMon, Weborama, WebPageTest, WebPros, Website-info, WebSitePulse, WebThumbnail, webtru, Webwiki, WellKnownBot, WeSEE:Search, WeViKaBot, WhatCMS, WhatsMyIP.org, WhereGoes, Who.is Bot, Wibybot, WikiDo, Willow Internet Crawler, WireReaderBot, WooRank, WordPress, WordPress.com mShots, Workona, Wotbox, wp.com feedbot, XenForo, XoviBot, YaCy, Yahoo! Cache System, Yahoo! Japan, Yahoo! Japan ASR, Yahoo! Japan BRW, Yahoo! Japan WSC, Yahoo! Link Preview, Yahoo! Mail Proxy, Yahoo! Slurp, Yahoo Gemini, YaK, Yandex Bot, Yeti/Naverbot, Yottaa Site Monitor, YouBot, Youdao Bot, Yourls, Yunyun Bot, Zaldamo, Zao, Ze List, Zeno, zgrab, Zookabot, ZoomBot, ZoominfoBot, Zotero Translation Server, ZumBot

php/pie

2024-12-01T01:48:18Z

The PHP Installer for Extensions

🥧 PIE (PHP Installer for Extensions)

You will need PHP 8.1 or newer to run PIE, but PIE can install an extension to any installed PHP version.

If you are an extension maintainer wanting to add PIE support to your extension, please read extension-maintainers.

Installing PIE

Manual installation

Download pie.phar either:
- latest stable release
- latest unstable nightly
Verify the PHAR's source with gh attestation verify pie.phar --repo php/pie
You may then invoke PIE with php pie.phar <command>

Further installation details can be found in the usage docs. This documentation assumes you have moved pie.phar into your $PATH, e.g. /usr/local/bin/pie on non-Windows systems.

Extensions that support PIE

A list of extensions that support PIE can be found on https://packagist.org/extensions.

Installing an extension using PIE

You can install an extension using the install command. For example, to install the example_pie_extension extension, you would run:

$ pie install example/example-pie-extension
This command may need elevated privileges, and may prompt you for your password.
You are running PHP 8.3.10
Target PHP installation: 8.3.10 nts, on Linux/OSX/etc x86_64 (from /usr/bin/php8.3)
Found package: example/example-pie-extension:1.0.1 which provides ext-example_pie_extension
phpize complete.
Configure complete.
Build complete: /tmp/pie_downloader_66e0b1de73cdb6.04069773/example-example-pie-extension-769f906/modules/example_pie_extension.so
Install complete: /usr/lib/php/20230831/example_pie_extension.so
You must now add "extension=example_pie_extension" to your php.ini
$

memstechtips/UnattendedWinstall

2024-12-01T01:37:50Z

Personalized Unattended Answer Files that helps automatically debloat and customize Windows 10 & 11 during the installation process.

UnattendedWinstall

Introduction

UnattendedWinstall leverages Microsoft's Answer Files (or Unattend files) to automate and customize Windows installations.
It enables modifications to Windows Settings and Packages directly in the Windows ISO during setup.

Why Use an Answer File?

Security

Provides transparency by allowing inspection of all changes in the answer file.
Runs directly on official Windows ISOs from Microsoft, eliminating the need for unofficial sources.
Utilizes a Microsoft-supported feature designed for streamlined mass deployment of Windows installations.

Automation

Enables automated configuration across multiple devices, saving time and effort by eliminating repetitive manual setups.

[!NOTE] UnattendedWinstall has been tested and optimized for personal use. For those interested in customizing further, create your own answer file following this video guide.

Versions

Support the Project

If UnattendedWinstall has been useful to you, consider supporting the project, it really does help!

Feedback and Community

If you have feedback, suggestions, or need help with UnattendedWinstall, please feel free to join the discussion on GitHub or our Discord community:

Requirements

Windows 10 or Windows 11
- (Tested on Windows 10 22H2 & Windows 11 24H2)
- (32-bit, 64-bit and arm64 is supported)

What Does UnattendedWinstall Do?

The UnattendedWinstall answer file comes with detailed descriptions for nearly all configurations and registry tweaks, which are available for inspection here on GitHub. For customization, download the answer file and open it in editors like Cursor or VSCode.

Sources and Contributions

Click to Show

Base Answer File Generation:
- Schneegans Unattend Generator
Tweaks & Optimizations:
- ChrisTitusTech WinUtil
- FR33THY's Ultimate Windows Optimization Guide
Additional Tweaks:

Key Features

Ability to choose Windows Edition (Pro is not enforced anymore as in v2.0.0)
Bypasses Windows 11 system requirements
Disables Windows Defender services by default
- prompted to enable after Windows installation
Disables User Account Control by default
- prompted to enable after Windows installation
Allows execution of PowerShell scripts by default
Skips forced Microsoft account creation during Windows setup
Removes preinstalled bloatware apps except Microsoft Edge, Notepad and Calculator
- Copilot and Recall is Disabled.
Sets privacy-related registry keys to disable telemetry
Limits Windows Update to install only security updates for one year
Optimizes registry with various optimization and customization-related keys
- See the "Set-RecommendedHKLMRegistry" and "Set-RecommendedHKCURegistry" functions for more information
Disables unnecessary scheduled tasks
Configures Windows services for optimal performance
Enables the Ultimate Performance power plan

[!NOTE] Use the UWScript.ps1 file once Windows is installed to reapply or revert settings in case Windows Update resets some of the settings or if you encounter any issues.
It can also be used to achieve a similar experience to UnattendedWinstall on an existing Windows installation without reinstalling Windows.

Before Running the Script

Ensure you open PowerShell as an administrator. Additionally, set the execution policy to allow script execution by running the following command:
Set-ExecutionPolicy Unrestricted
Running PowerShell with elevated permissions and enabling script execution will ensure that UWScript.ps1 can apply the necessary system changes.

Usage Instructions

To use an answer file, include autounattend.xml at the root of your Windows Installation Media to be executed during Windows setup.

[!IMPORTANT]
Ensure the answer file is named autounattend.xml; otherwise, it won’t be recognized by the installer.

Using Memory's WIMUtil (Highly Preferred)

To use WIMUtil, follow these steps to launch PowerShell as an Administrator and run the installation script:

Open PowerShell as Administrator:
- Windows 10/11: Right-click on the Start button and select Windows PowerShell (Admin) or Windows Terminal (Admin).
  PowerShell will open in a new window.
Confirm Administrator Privileges:
- If prompted by the User Account Control (UAC), click Yes to allow PowerShell to run as an administrator.
Paste and Run the Command:
- Copy the following command:
```
irm "https://github.com/memstechtips/WIMUtil/raw/main/src/WIMUtil.ps1" | iex
```
- To paste into PowerShell, Right-Click or press Ctrl + V in the PowerShell or Terminal window.
  This should automatically paste your copied command.
- Press Enter to execute the command.

Once launched, WIMUtil guides you through a wizard:

Select or Download Windows ISO
Add Latest UnattendedWinstall Answer File Automatically
Extract and Add Current Device Drivers to Installation Media
Create New ISO with Customizations Included
Create a Bootable USB Flash Drive with Ventoy
Copy the New ISO File to the Ventoy Flash Drive
Boot from the USB flash drive, choose your ISO & Install Windows

For more info, check out the official WIMUtil GitHub Repo.

Old Methods

Method 1: Create a Bootable Windows Installation USB

Video Tutorial

Click to Show Instructions

Download the autounattend.xml file and save it on your computer.
Create a Windows 10 or Windows 11 Bootable Installation USB drive with Rufus or the Media Creation Tool.
Important
- Some users have reported issues with the Media Creation Tool when creating the Windows Installation USB. Use it at your own discretion.
- When using Rufus, don’t select any of the checkboxes in “Customize Your Windows Experience,” as it creates another autounattend.xml file that might overwrite settings in the UnattendedWinstall file.
Copy the autounattend.xml file you downloaded in Step 1 to the root of the Bootable Windows Installation USB you created in Step 2.
Boot from the Windows Installation USB, do a clean install of Windows as normal, and the scripts will run automatically.

Method 2: Create a Custom ISO File

Video Tutorial

Click to Show Instructions

Download the autounattend.xml file and save it on your computer.
Download the Windows 10 or Windows 11 ISO file depending on the version you want.
Download and install AnyBurn
- In AnyBurn, select the “Edit Image File” option.
- Navigate to and select the Official Windows ISO file you downloaded in Step 2.
- Click on “Add” and select the autounattend.xml file you downloaded in Step 1, or just click and drag the autounattend.xml into the AnyBurn window.
- Click on “Next,” then on “Create Now.” You should be prompted to overwrite the ISO file; click on “Yes.”
- Once the process is complete, close AnyBurn.
Use the ISO file to install Windows on a Virtual Machine OR use a program like Rufus or Ventoy to create a bootable USB flash drive with the edited Windows ISO file.

Important

When using Rufus, don’t select any of the checkboxes in “Customize Your Windows Experience,” as it creates another autounattend.xml file that might overwrite settings in the UnattendedWinstall file.

Boot from the Windows Installation USB, do a clean install of Windows as normal, and the scripts will run automatically.

Method 3: Use Ventoy Auto Install Plugin

Video Tutorial

Click to Show Instructions

Download the autounattend.xml file and save it on your computer.
Download the Windows 10 or Windows 11 ISO file, depending on the version you want.
Download and install Ventoy to your desired USB flash drive.
Prepare the folder structure:
- In your newly created Ventoy USB disk, create the following folders: ISO and Templates.
  They should be at the root of the drive.
- Inside of the ISO folder, create a new folder called Windows.
- Copy your Windows ISO files in the ISO\Windows folder.
- Copy your autounattend.xml into the Templates folder.
Start VentoyPlugson. Depending on your OS, the steps might differ.
- On Windows, run the VentoyPlugson.exe file.
- A new browser window should open up with a Ventoy web interface ready to go.
- Select the Auto Install Plugin menu from the list.
- Click on the Add button.
- Select [parent] to make the whole Windows ISO folder benefit from the plugin.
- In the Directory Path, paste in the absolute path to your Windows folder.
  example: F:\ISO\Windows (Replace F with your drive letter.)
- In the Template Path, paste in the absolute path to your autounattend.xml file.
  example: F:\Templates\autounattend.xml (Replace F with your drive letter.)
  (PSA: If you have more autounattend.xml files, you can add them later on!)
- Click on OK and you should see a message saying that the configuration has been saved successfully.
- Close the VentoyPlugson browser window and stop the VentoyPlugson application.
Boot from the Ventoy USB drive in the computer where you want to install windows.
- After selecting a Windows ISO to boot from, you will be prompted to boot with the /Templates/autounattend.xml file.
- Select that option and the autounattend.xml will be automatically executed during installation.

FAQ

How can I apply these settings to an existing Windows installation?

Run the UWScript.ps1 file or use the Chris Titus Tech Windows Utility (Video).

Can this answer file be used for an in-place upgrade?

No, in-place upgrades do not support answer files.

Why is Windows still updating automatically?

Feature updates are delayed for a year; however, security and driver updates continue as usual.

Why don't I have internet after installing Windows?

Click to Show Instructions

If you’re unable to connect to the internet after installation, it’s likely because your Wi-Fi or LAN (Ethernet) drivers are missing. Windows sometimes doesn’t include all necessary drivers for network adapters, especially if they’re specific to your device.

To resolve this, follow these steps:

Download your network driver from the manufacturer’s website on another computer with internet access. Look for Wi-Fi or LAN drivers specific to your device model.
Transfer the driver to your Windows installation via USB drive.
Install the driver on your Windows system and restart if necessary.

After installation, you should be able to connect to the internet.

How can I access the previous "IoT-LTSC-Like," "Standard," and "Core" versions of the file(s)?

You still have access to the previous files here: Version 1.0.0 Release.

[!NOTE]
You need to download the Source Code.zip file. Once extracted, you’ll have access to all the previous v1.0.0 files.

Why isn't Microsoft Edge Uninstalled?

Click to Show Explanation

I spent a lot of time trying to find a way to uninstall Microsoft Edge during Windows installation. However, it was challenging because of differences between Windows 10 22H2 and Windows 11 24H2. My goal is to use Microsoft’s supported uninstall methods, and I plan to add an easy Edge removal option in future releases.

In the meantime, if you wish to remove Edge after Windows installation, consider using this script by FR33THY. FR33THY’s Ultimate Windows Optimization Guide was a major inspiration for version 2.0.0 of this project, and I highly recommend exploring it for additional Windows optimization tips.

How can I add my own Registry Tweaks to v2.0.0 of the `autounattend.xml` file?

Click to Show Instructions

You can also still add your own registry entries to the v2.0.0 file, and it is actually easier if you understand where to add it. I'll give a brief explanation.

For registry entries that apply to the local machine, i.e., HKEY_LOCAL_MACHINE registry keys, you can find the function SetRecommendedHKLMRegistry in the autounattend.xml file, see here: https://github.com/memstechtips/UnattendedWinstall/blob/93305192ed6d64e0f5b98a89f447927480285354/autounattend.xml#L1981

and then add whatever registry entries you want to add in .reg format, like the rest of the entries are set, and just make sure you add it before the "@ to make it part of the .reg file that will be generated, see here: https://github.com/memstechtips/UnattendedWinstall/blob/93305192ed6d64e0f5b98a89f447927480285354/autounattend.xml#L3412

and it will then be applied to the registry.

Similarly, if you have HKEY_CURRENT_USER registry keys, you can add those to the User Customization.ps1 file in the same way as explained above, starting here: https://github.com/memstechtips/UnattendedWinstall/blob/93305192ed6d64e0f5b98a89f447927480285354/autounattend.xml#L3912 so below the Windows Registry Editor Version 5.00 and then ending before the "@ here: https://github.com/memstechtips/UnattendedWinstall/blob/93305192ed6d64e0f5b98a89f447927480285354/autounattend.xml#L4423

Note
The above links might not take you to the correct lines of code once new versions of the file are released, but it does take you to the correct lines on v2.0.0.

memstechtips/UnattendedWinstall

2024-12-01T01:57:08Z

Personalized Unattended Answer Files that helps automatically debloat and customize Windows 10 & 11 during the installation process.

UnattendedWinstall

Introduction

Why Use an Answer File?

Security

Provides transparency by allowing inspection of all changes in the answer file.
Runs directly on official Windows ISOs from Microsoft, eliminating the need for unofficial sources.
Utilizes a Microsoft-supported feature designed for streamlined mass deployment of Windows installations.

Automation

Enables automated configuration across multiple devices, saving time and effort by eliminating repetitive manual setups.

[!NOTE] UnattendedWinstall has been tested and optimized for personal use. For those interested in customizing further, create your own answer file following this video guide.

Versions

Support the Project

If UnattendedWinstall has been useful to you, consider supporting the project, it really does help!

Feedback and Community

If you have feedback, suggestions, or need help with UnattendedWinstall, please feel free to join the discussion on GitHub or our Discord community:

Requirements

Windows 10 or Windows 11
- (Tested on Windows 10 22H2 & Windows 11 24H2)
- (32-bit, 64-bit and arm64 is supported)

What Does UnattendedWinstall Do?

Sources and Contributions

Click to Show

Base Answer File Generation:
- Schneegans Unattend Generator
Tweaks & Optimizations:
- ChrisTitusTech WinUtil
- FR33THY's Ultimate Windows Optimization Guide
Additional Tweaks:

Key Features

Ability to choose Windows Edition (Pro is not enforced anymore as in v2.0.0)
Bypasses Windows 11 system requirements
Disables Windows Defender services by default
- prompted to enable after Windows installation
Disables User Account Control by default
- prompted to enable after Windows installation
Allows execution of PowerShell scripts by default
Skips forced Microsoft account creation during Windows setup
Removes preinstalled bloatware apps except Microsoft Edge, Notepad and Calculator
- Copilot and Recall is Disabled.
Sets privacy-related registry keys to disable telemetry
Limits Windows Update to install only security updates for one year
Optimizes registry with various optimization and customization-related keys
- See the "Set-RecommendedHKLMRegistry" and "Set-RecommendedHKCURegistry" functions for more information
Disables unnecessary scheduled tasks
Configures Windows services for optimal performance
Enables the Ultimate Performance power plan

[!NOTE] Use the UWScript.ps1 file once Windows is installed to reapply or revert settings in case Windows Update resets some of the settings or if you encounter any issues.
It can also be used to achieve a similar experience to UnattendedWinstall on an existing Windows installation without reinstalling Windows.

Before Running the Script

Ensure you open PowerShell as an administrator. Additionally, set the execution policy to allow script execution by running the following command:
Set-ExecutionPolicy Unrestricted
Running PowerShell with elevated permissions and enabling script execution will ensure that UWScript.ps1 can apply the necessary system changes.

Usage Instructions

To use an answer file, include autounattend.xml at the root of your Windows Installation Media to be executed during Windows setup.

[!IMPORTANT]
Ensure the answer file is named autounattend.xml; otherwise, it won’t be recognized by the installer.

Using Memory's WIMUtil (Highly Preferred)

To use WIMUtil, follow these steps to launch PowerShell as an Administrator and run the installation script:

Open PowerShell as Administrator:
- Windows 10/11: Right-click on the Start button and select Windows PowerShell (Admin) or Windows Terminal (Admin).
  PowerShell will open in a new window.
Confirm Administrator Privileges:
- If prompted by the User Account Control (UAC), click Yes to allow PowerShell to run as an administrator.
Paste and Run the Command:
- Copy the following command:
```
irm "https://github.com/memstechtips/WIMUtil/raw/main/src/WIMUtil.ps1" | iex
```
- To paste into PowerShell, Right-Click or press Ctrl + V in the PowerShell or Terminal window.
  This should automatically paste your copied command.
- Press Enter to execute the command.

Once launched, WIMUtil guides you through a wizard:

Select or Download Windows ISO
Add Latest UnattendedWinstall Answer File Automatically
Extract and Add Current Device Drivers to Installation Media
Create New ISO with Customizations Included
Create a Bootable USB Flash Drive with Ventoy
Copy the New ISO File to the Ventoy Flash Drive
Boot from the USB flash drive, choose your ISO & Install Windows

For more info, check out the official WIMUtil GitHub Repo.

Old Methods

Method 1: Create a Bootable Windows Installation USB

Video Tutorial

Click to Show Instructions

Download the autounattend.xml file and save it on your computer.
Create a Windows 10 or Windows 11 Bootable Installation USB drive with Rufus or the Media Creation Tool.
Important
- Some users have reported issues with the Media Creation Tool when creating the Windows Installation USB. Use it at your own discretion.
- When using Rufus, don’t select any of the checkboxes in “Customize Your Windows Experience,” as it creates another autounattend.xml file that might overwrite settings in the UnattendedWinstall file.
Copy the autounattend.xml file you downloaded in Step 1 to the root of the Bootable Windows Installation USB you created in Step 2.
Boot from the Windows Installation USB, do a clean install of Windows as normal, and the scripts will run automatically.

Method 2: Create a Custom ISO File

Video Tutorial

Click to Show Instructions

Download the autounattend.xml file and save it on your computer.
Download the Windows 10 or Windows 11 ISO file depending on the version you want.
Download and install AnyBurn
- In AnyBurn, select the “Edit Image File” option.
- Navigate to and select the Official Windows ISO file you downloaded in Step 2.
- Click on “Add” and select the autounattend.xml file you downloaded in Step 1, or just click and drag the autounattend.xml into the AnyBurn window.
- Click on “Next,” then on “Create Now.” You should be prompted to overwrite the ISO file; click on “Yes.”
- Once the process is complete, close AnyBurn.
Use the ISO file to install Windows on a Virtual Machine OR use a program like Rufus or Ventoy to create a bootable USB flash drive with the edited Windows ISO file.

Important

When using Rufus, don’t select any of the checkboxes in “Customize Your Windows Experience,” as it creates another autounattend.xml file that might overwrite settings in the UnattendedWinstall file.

Boot from the Windows Installation USB, do a clean install of Windows as normal, and the scripts will run automatically.

Method 3: Use Ventoy Auto Install Plugin

Video Tutorial

Click to Show Instructions

Download the autounattend.xml file and save it on your computer.
Download the Windows 10 or Windows 11 ISO file, depending on the version you want.
Download and install Ventoy to your desired USB flash drive.
Prepare the folder structure:
- In your newly created Ventoy USB disk, create the following folders: ISO and Templates.
  They should be at the root of the drive.
- Inside of the ISO folder, create a new folder called Windows.
- Copy your Windows ISO files in the ISO\Windows folder.
- Copy your autounattend.xml into the Templates folder.
Start VentoyPlugson. Depending on your OS, the steps might differ.
- On Windows, run the VentoyPlugson.exe file.
- A new browser window should open up with a Ventoy web interface ready to go.
- Select the Auto Install Plugin menu from the list.
- Click on the Add button.
- Select [parent] to make the whole Windows ISO folder benefit from the plugin.
- In the Directory Path, paste in the absolute path to your Windows folder.
  example: F:\ISO\Windows (Replace F with your drive letter.)
- In the Template Path, paste in the absolute path to your autounattend.xml file.
  example: F:\Templates\autounattend.xml (Replace F with your drive letter.)
  (PSA: If you have more autounattend.xml files, you can add them later on!)
- Click on OK and you should see a message saying that the configuration has been saved successfully.
- Close the VentoyPlugson browser window and stop the VentoyPlugson application.
Boot from the Ventoy USB drive in the computer where you want to install windows.
- After selecting a Windows ISO to boot from, you will be prompted to boot with the /Templates/autounattend.xml file.
- Select that option and the autounattend.xml will be automatically executed during installation.

FAQ

How can I apply these settings to an existing Windows installation?

Run the UWScript.ps1 file or use the Chris Titus Tech Windows Utility (Video).

Can this answer file be used for an in-place upgrade?

No, in-place upgrades do not support answer files.

Why is Windows still updating automatically?

Feature updates are delayed for a year; however, security and driver updates continue as usual.

Why don't I have internet after installing Windows?

Click to Show Instructions

To resolve this, follow these steps:

Download your network driver from the manufacturer’s website on another computer with internet access. Look for Wi-Fi or LAN drivers specific to your device model.
Transfer the driver to your Windows installation via USB drive.
Install the driver on your Windows system and restart if necessary.

After installation, you should be able to connect to the internet.

How can I access the previous "IoT-LTSC-Like," "Standard," and "Core" versions of the file(s)?

You still have access to the previous files here: Version 1.0.0 Release.

[!NOTE]
You need to download the Source Code.zip file. Once extracted, you’ll have access to all the previous v1.0.0 files.

Why isn't Microsoft Edge Uninstalled?

Click to Show Explanation

How can I add my own Registry Tweaks to v2.0.0 of the `autounattend.xml` file?

Click to Show Instructions

You can also still add your own registry entries to the v2.0.0 file, and it is actually easier if you understand where to add it. I'll give a brief explanation.

and it will then be applied to the registry.

Note
The above links might not take you to the correct lines of code once new versions of the file are released, but it does take you to the correct lines on v2.0.0.

cisagov/ScubaGear

2024-12-01T01:57:08Z

Automation to assess the state of your M365 tenant against CISA's baselines

ScubaGear is an assessment tool that verifies that a Microsoft 365 (M365) tenant’s configuration conforms to the policies described in the Secure Cloud Business Applications (SCuBA) Security Configuration Baseline documents.

Note: This documentation can be read using GitHub Pages.

Target Audience

ScubaGear is for M365 administrators who want to assess their tenant environments against CISA Secure Configuration Baselines.

Overview

ScubaGear uses a three-step process:

Step One - PowerShell code queries M365 APIs for various configuration settings.
Step Two - It then calls Open Policy Agent (OPA) to compare these settings against Rego security policies written per the baseline documents.
Step Three - Finally, it reports the results of the comparison as HTML, JSON, and CSV.

Getting Started

To install ScubaGear from PSGallery, open a PowerShell 5 terminal on a Windows computer and install the module:

# Install ScubaGear
Install-Module -Name ScubaGear

To install its dependencies:

# Install the minimum required dependencies
Initialize-SCuBA

To verify that it is installed:

# Check the version
Invoke-SCuBA -Version

To run ScubaGear:

# Assess all products
Invoke-SCuBA -ProductNames *

Note: Successfully running ScubaGear requires certain prerequisites and configuration settings. To learn more, read through the sections below.

The following sections should be read in order.

Installation

Prerequisites

Execution

Configuration

Troubleshooting

Misc

Assumptions

Project License

Unless otherwise noted, this project is distributed under the Creative Commons Zero license. With developer approval, contributions may be submitted with an alternate compatible license. If accepted, those contributions will be listed herein with the appropriate license.

automatic-ripping-machine/automatic-ripping-machine

2024-11-30T01:34:41Z

Automatic Ripping Machine (ARM) Scripts

Automatic Ripping Machine (ARM)

Overview

Insert an optical disc (Blu-ray, DVD, CD) and checks to see if it's audio, video (Movie or TV), or data, then rips it.

See: https://b3n.org/automatic-ripping-machine

Features

Detects insertion of disc using udev
Determines disc type...
- If video (Blu-ray or DVD)
  - Retrieve title from disc or OMDb API to name the folder "Movie Title (Year)" so that Plex or Emby can pick it up
  - Determine if video is Movie or TV using OMDb API
  - Rip using MakeMKV or HandBrake (can rip all features or main feature)
  - Eject disc and queue up Handbrake transcoding when done
  - Transcoding jobs are asynchronously batched from ripping
  - Send notifications via IFTTT, Pushbullet, Slack, Discord, and many more!
- If audio (CD) - rip using abcde (get disc-data and album art from musicbrainz)
- If data (Blu-ray, DVD, or CD) - make an ISO backup
Headless, designed to be run from a server
Can rip from multiple-optical drives in parallel
Python Flask UI to interact with ripping jobs, view logs, update jobs, etc

Usage

Insert disc
Wait for disc to eject
Repeat

Requirements

A system capable of running Docker containers
One or more optical drives to rip Blu-rays, DVDs, and CDs
Lots of drive space (I suggest using a NAS) to store your movies

Install

For normal installation please see the wiki.

For docker installation please see here.

Troubleshooting

Please see the wiki for troubleshooting.

Contributing

Pull requests are welcome. Please see the Contributing Guide

If you set ARM up in a different environment (hardware/OS/virtual/etc.), please consider submitting a howto to the wiki.

License

MIT License

icloud-photos-downloader/icloud_photos_downloader

2024-11-30T01:34:41Z

A command-line tool to download photos from iCloud

iCloud Photos Downloader

A command-line tool to download all your iCloud photos.
Works on Linux, Windows, and macOS; laptop, desktop, and NAS
Available as an executable for direct downloading and through package managers/ecosystems (Docker, PyPI, AUR, npm)
Developed and maintained by volunteers (we are always looking for help).

See Documentation for more details. Also, check Issues

We aim to release new versions once a week (Friday), if there is something worth delivering.

Install and Run

There are three ways to run icloudpd:

Download executable for your platform from the GitHub Release and run it
Use package manager to install, update, and, in some cases, run (Docker, PyPI, AUR, npm)
Build and run from the source

See Documentation for more details

Features

Three modes of operation:
- Copy - download new photos from iCloud (default mode)
- Sync - download new photos from iCloud and delete local files that were removed in iCloud (--auto-delete option)
- Move - download new photos from iCloud and delete photos in iCloud (--delete-after-download option)
Support for Live Photos (image and video as separate files) and RAW images (including RAW+JPEG)
Automatic de-duplication of photos with the same name
One time download and an option to monitor for iCloud changes continuously (--watch-with-interval option)
Optimizations for incremental runs (--until-found and --recent options)
Photo metadata (EXIF) updates (--set-exif-datetime option)
... and many more (use --help option to get full list)

Experimental Mode

Some changes are added to the experimental mode before they graduate into the main package. Details

Usage

To keep your iCloud photo collection synchronized to your local system:

icloudpd --directory /data --username my@email.address --watch-with-interval 3600

[!IMPORTANT] It is icloudpd, not icloud executable

[!TIP] Synchronization logic can be adjusted with command-line parameters. Run icloudpd --help to get full list.

To independently create and authorize a session (and complete 2SA/2FA validation if needed) on your local system:

icloudpd --username my@email.address --password my_password --auth-only

[!TIP] This feature can also be used to check and verify that the session is still authenticated.

Contributing

Want to contribute to iCloud Photos Downloader? Awesome! Check out the contributing guidelines to get involved.

opendatalab/MinerU

2024-12-01T01:57:19Z

A high-quality tool for convert PDF to Markdown and JSON.一站式开源高质量数据提取工具，将PDF转换成Markdown和JSON格式。

English | 简体中文

PDF-Extract-Kit: High-Quality PDF Extraction Toolkit🔥🔥🔥

👋 join us on Discord and WeChat

Changelog

2024/11/22 0.10.0 released. Introducing hybrid OCR text extraction capabilities,
- Significantly improved parsing performance in complex text distribution scenarios such as dense formulas, irregular span regions, and text represented by images.
- Combines the dual advantages of accurate content extraction and faster speed in text mode, and more precise span/line region recognition in OCR mode.
2024/11/15 0.9.3 released. Integrated RapidTable for table recognition, improving single-table parsing speed by more than 10 times, with higher accuracy and lower GPU memory usage.
2024/11/06 0.9.2 released. Integrated the StructTable-InternVL2-1B model for table recognition functionality.
2024/10/31 0.9.0 released. This is a major new version with extensive code refactoring, addressing numerous issues, improving performance, reducing hardware requirements, and enhancing usability:
- Refactored the sorting module code to use layoutreader for reading order sorting, ensuring high accuracy in various layouts.
- Refactored the paragraph concatenation module to achieve good results in cross-column, cross-page, cross-figure, and cross-table scenarios.
- Refactored the list and table of contents recognition functions, significantly improving the accuracy of list blocks and table of contents blocks, as well as the parsing of corresponding text paragraphs.
- Refactored the matching logic for figures, tables, and descriptive text, greatly enhancing the accuracy of matching captions and footnotes to figures and tables, and reducing the loss rate of descriptive text to near zero.
- Added multi-language support for OCR, supporting detection and recognition of 84 languages.For the list of supported languages, see OCR Language Support List.
- Added memory recycling logic and other memory optimization measures, significantly reducing memory usage. The memory requirement for enabling all acceleration features except table acceleration (layout/formula/OCR) has been reduced from 16GB to 8GB, and the memory requirement for enabling all acceleration features has been reduced from 24GB to 10GB.
- Optimized configuration file feature switches, adding an independent formula detection switch to significantly improve speed and parsing results when formula detection is not needed.
- Integrated PDF-Extract-Kit 1.0:
  - Added the self-developed doclayout_yolo model, which speeds up processing by more than 10 times compared to the original solution while maintaining similar parsing effects, and can be freely switched with layoutlmv3 via the configuration file.
  - Upgraded formula parsing to unimernet 0.2.1, improving formula parsing accuracy while significantly reducing memory usage.
  - Due to the repository change for PDF-Extract-Kit 1.0, you need to re-download the model. Please refer to How to Download Models for detailed steps.
2024/09/27 Version 0.8.1 released, Fixed some bugs, and providing a localized deployment version of the online demo and the front-end interface.
2024/09/09: Version 0.8.0 released, supporting fast deployment with Dockerfile, and launching demos on Huggingface and Modelscope.
2024/08/30: Version 0.7.1 released, add paddle tablemaster table recognition option
2024/08/09: Version 0.7.0b1 released, simplified installation process, added table recognition functionality
2024/08/01: Version 0.6.2b1 released, optimized dependency conflict issues and installation documentation
2024/07/05: Initial open-source release

MinerU

Project Introduction

MinerU is a tool that converts PDFs into machine-readable formats (e.g., markdown, JSON), allowing for easy extraction into any format. MinerU was born during the pre-training process of InternLM. We focus on solving symbol conversion issues in scientific literature and hope to contribute to technological development in the era of large models. Compared to well-known commercial products, MinerU is still young. If you encounter any issues or if the results are not as expected, please submit an issue on issue and attach the relevant PDF.

https://github.com/user-attachments/assets/4bea02c9-6d54-4cd6-97ed-dff14340982c

Key Features

Remove headers, footers, footnotes, page numbers, etc., to ensure semantic coherence.
Output text in human-readable order, suitable for single-column, multi-column, and complex layouts.
Preserve the structure of the original document, including headings, paragraphs, lists, etc.
Extract images, image descriptions, tables, table titles, and footnotes.
Automatically recognize and convert formulas in the document to LaTeX format.
Automatically recognize and convert tables in the document to HTML format.
Automatically detect scanned PDFs and garbled PDFs and enable OCR functionality.
OCR supports detection and recognition of 84 languages.
Supports multiple output formats, such as multimodal and NLP Markdown, JSON sorted by reading order, and rich intermediate formats.
Supports various visualization results, including layout visualization and span visualization, for efficient confirmation of output quality.
Supports both CPU and GPU environments.
Compatible with Windows, Linux, and Mac platforms.

Quick Start

If you encounter any installation issues, please first consult the FAQ.
If the parsing results are not as expected, refer to the Known Issues.
There are three different ways to experience MinerU:

[!WARNING] Pre-installation Notice—Hardware and Software Environment Support

To ensure the stability and reliability of the project, we only optimize and test for specific hardware and software environments during development. This ensures that users deploying and running the project on recommended system configurations will get the best performance with the fewest compatibility issues.

By focusing resources on the mainline environment, our team can more efficiently resolve potential bugs and develop new features.

In non-mainline environments, due to the diversity of hardware and software configurations, as well as third-party dependency compatibility issues, we cannot guarantee 100% project availability. Therefore, for users who wish to use this project in non-recommended environments, we suggest carefully reading the documentation and FAQ first. Most issues already have corresponding solutions in the FAQ. We also encourage community feedback to help us gradually expand support.

Operating System
Ubuntu 22.04 LTS	Windows 10 / 11	macOS 11+
CPU	x86_64(unsupported ARM Linux)	x86_64(unsupported ARM Windows)	x86_64 / arm64
Memory	16GB or more, recommended 32GB+
Python Version	3.10(Please make sure to create a Python 3.10 virtual environment using conda)
Nvidia Driver Version	latest (Proprietary Driver)	latest	None
CUDA Environment	Automatic installation [12.1 (pytorch) + 11.8 (paddle)]	11.8 (manual installation) + cuDNN v8.7.0 (manual installation)	None
GPU Hardware Support List	GPU VRAM 8GB or more	2080~2080Ti / 3060Ti~3090Ti / 4060~4090 8G VRAM can enable all acceleration features	None

Online Demo

Stable Version (Stable version verified by QA):

Test Version (Synced with dev branch updates, testing new features):

Quick CPU Demo

1. Install magic-pdf

conda create -n MinerU python=3.10
conda activate MinerU
pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com

2. Download model weight files

Refer to How to Download Model Files for detailed instructions.

3. Modify the Configuration File for Additional Configuration

After completing the 2. Download model weight files step, the script will automatically generate a magic-pdf.json file in the user directory and configure the default model path. You can find the magic-pdf.json file in your 【user directory】.

[!TIP] The user directory for Windows is "C:\Users\username", for Linux it is "/home/username", and for macOS it is "/Users/username".

You can modify certain configurations in this file to enable or disable features, such as table recognition:

[!NOTE] If the following items are not present in the JSON, please manually add the required items and remove the comment content (standard JSON does not support comments).

{
    // other config
    "layout-config": {
        "model": "layoutlmv3" // Please change to "doclayout_yolo" when using doclayout_yolo.
    },
    "formula-config": {
        "mfd_model": "yolo_v8_mfd",
        "mfr_model": "unimernet_small",
        "enable": true  // The formula recognition feature is enabled by default. If you need to disable it, please change the value here to "false".
    },
    "table-config": {
        "model": "rapid_table",  // Default to using "rapid_table", can be switched to "tablemaster" or "struct_eqtable".
        "enable": false, // The table recognition feature is disabled by default. If you need to enable it, please change the value here to "true".
        "max_time": 400
    }
}

Using GPU

If your device supports CUDA and meets the GPU requirements of the mainline environment, you can use GPU acceleration. Please select the appropriate guide based on your system:

Ubuntu 22.04 LTS + GPU
Windows 10/11 + GPU
Quick Deployment with Docker

[!IMPORTANT] Docker requires a GPU with at least 8GB of VRAM, and all acceleration features are enabled by default.

Before running this Docker, you can use the following command to check if your device supports CUDA acceleration on Docker.
docker run --rm --gpus=all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi

wget https://github.com/opendatalab/MinerU/raw/master/Dockerfile
docker build -t mineru:latest .
docker run --rm -it --gpus=all mineru:latest /bin/bash
magic-pdf --help

Usage

Command Line

Using MinerU via Command Line

[!TIP] For more information about the output files, please refer to the Output File Description.

API

Using MinerU via Python API

For detailed implementation, refer to:

Deploy Derived Projects

Derived projects include secondary development projects based on MinerU by project developers and community developers,
such as application interfaces based on Gradio, RAG based on llama, web demos similar to the official website, lightweight multi-GPU load balancing client/server ends, etc. These projects may offer more features and a better user experience.
For specific deployment methods, please refer to the Derived Project README

Development Guide

TODO

Reading order based on the model
Recognition of index and list in the main text
Table recognition
Code block recognition in the main text
Chemical formula recognition
Geometric shape recognition

Known Issues

Reading order is determined by the model based on the spatial distribution of readable content, and may be out of order in some areas under extremely complex layouts.
Vertical text is not supported.
Tables of contents and lists are recognized through rules, and some uncommon list formats may not be recognized.
Only one level of headings is supported; hierarchical headings are not currently supported.
Code blocks are not yet supported in the layout model.
Comic books, art albums, primary school textbooks, and exercises cannot be parsed well.
Table recognition may result in row/column recognition errors in complex tables.
OCR recognition may produce inaccurate characters in PDFs of lesser-known languages (e.g., diacritical marks in Latin script, easily confused characters in Arabic script).
Some formulas may not render correctly in Markdown.

FAQ

FAQ in Chinese

FAQ in English

All Thanks To Our Contributors

License Information

LICENSE.md

This project currently uses PyMuPDF to achieve advanced functionality. However, since it adheres to the AGPL license, it may impose restrictions on certain usage scenarios. In future iterations, we plan to explore and replace it with a more permissive PDF processing library to enhance user-friendliness and flexibility.

Acknowledgments

Citation

@misc{wang2024mineruopensourcesolutionprecise,
      title={MinerU: An Open-Source Solution for Precise Document Content Extraction}, 
      author={Bin Wang and Chao Xu and Xiaomeng Zhao and Linke Ouyang and Fan Wu and Zhiyuan Zhao and Rui Xu and Kaiwen Liu and Yuan Qu and Fukai Shang and Bo Zhang and Liqun Wei and Zhihao Sui and Wei Li and Botian Shi and Yu Qiao and Dahua Lin and Conghui He},
      year={2024},
      eprint={2409.18839},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2409.18839}, 
}

@article{he2024opendatalab,
  title={Opendatalab: Empowering general artificial intelligence with open datasets},
  author={He, Conghui and Li, Wei and Jin, Zhenjiang and Xu, Chao and Wang, Bin and Lin, Dahua},
  journal={arXiv preprint arXiv:2407.13773},
  year={2024}
}

Star History

Magic-doc

Magic-Doc Fast speed ppt/pptx/doc/docx/pdf extraction tool

Magic-html

Magic-HTML Mixed web page extraction tool

LibraHp/GetQzonehistory

2024-12-01T01:57:19Z

获取QQ空间发布的历史说说

GetQzonehistory(获取qq发布的历史说说)

已经打包好的单文件可以直接执行，在release页面。

免责声明【必读】

本工具仅供学习和技术研究使用，不得用于任何商业或非法行为，否则后果自负。

本工具的作者不对本工具的安全性、完整性、可靠性、有效性、正确性或适用性做任何明示或暗示的保证，也不对本工具的使用或滥用造成的任何直接或间接的损失、责任、索赔、要求或诉讼承担任何责任。

本工具的作者保留随时修改、更新、删除或终止本工具的权利，无需事先通知或承担任何义务。

本工具的使用者应遵守相关法律法规，尊重QQ的版权和隐私，不得侵犯QQ或其他第三方的合法权益，不得从事任何违法或不道德的行为。

本工具的使用者在下载、安装、运行或使用本工具时，即表示已阅读并同意本免责声明。如有异议，请立即停止使用本工具，并删除所有相关文件。

该项目通过获取QQ空间的历史消息列表来获取该账号下发布的所有说说（当然消息列表中没有的就获取不到，例如一些仅自己可见的说说）B站有详细食用教程

主要实现还是通过模拟登录QQ空间来获取历史消息列表，然后进行数据分析，最后将爬取的说说存放到/resource/result目录下

由于对python编程还不是很熟悉，所以代码有很多疏漏，可以通过自己的想法来完善代码

目录结构

project/
├── resource/                # 资源目录
│   ├── config/              # 配置目录，文件保存位置配置
│   │   └── config.ini
│   ├── result/              # 导出结果的目录，格式为“你的qq.xlsx”
│   │   ├── ...
│   │   └── ...
│   ├── temp/                # 缓存目录
│   │   ├── ...
│   │   └── ...
│   ├── user/                # 用户信息
│   │   ├── ...
│   │   └── ...
├── util/                    # 单元工具目录
│   ├── ConfigUtil.py        # 读取配置
│   ├── GetAllMomentsUtil.py # 获取未删除的所有说说
│   ├── LoginUtil.py         # 登录相关
│   ├── RequestUtil.py       # 请求数据相关
│   └── ToolsUtil.py         # 工具
├── main.py                  # 主程序入口
├── fetch_all_message.py     # 主程序入口
├── README.md                # 项目说明文件
├── requirements.txt         # 依赖项列表
└── LICENSE                  # 许可证文件

安装

使用虚拟环境（推荐）

# 克隆储存库
git clone https://github.com/LibraHp/GetQzonehistory.git
# 打开目录
cd GetQzonehistory
# 创建名为 myenv 的虚拟环境
python -m venv myenv
# 激活虚拟环境。在终端或命令提示符中运行以下命令：
# 对于 Windows：
myenv\Scripts\activate
# 对于 macOS/Linux：
source myenv/bin/activate
# 安装依赖
pip install -i https://mirrors.aliyun.com/pypi/simple/ -r requirements.txt
# 运行脚本
python main.py

使用本机环境（不推荐）

# 克隆储存库
git clone https://github.com/LibraHp/GetQzonehistory.git
# 打开目录
cd GetQzonehistory
# 安装依赖
pip install -i https://mirrors.aliyun.com/pypi/simple/ -r requirements.txt
# 运行脚本
python main.py

参考

登录方法参考自 python-QQ空间扫码登录

donnemartin/system-design-primer

2024-12-01T01:57:19Z

Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.

English ∙ 日本語 ∙ 简体中文 ∙ 繁體中文 | العَرَبِيَّة‎ ∙ বাংলা ∙ Português do Brasil ∙ Deutsch ∙ ελληνικά ∙ עברית ∙ Italiano ∙ 한국어 ∙ فارسی ∙ Polski ∙ русский язык ∙ Español ∙ ภาษาไทย ∙ Türkçe ∙ tiếng Việt ∙ Français | Add Translation

Help translate this guide!

The System Design Primer

Motivation

Learn how to design large-scale systems.

Prep for the system design interview.

Learn how to design large-scale systems

Learning how to design scalable systems will help you become a better engineer.

System design is a broad topic. There is a vast amount of resources scattered throughout the web on system design principles.

This repo is an organized collection of resources to help you learn how to build systems at scale.

Learn from the open source community

This is a continually updated, open source project.

Contributions are welcome!

Prep for the system design interview

In addition to coding interviews, system design is a required component of the technical interview process at many tech companies.

Practice common system design interview questions and compare your results with sample solutions: discussions, code, and diagrams.

Additional topics for interview prep:

Anki flashcards

The provided Anki flashcard decks use spaced repetition to help you retain key system design concepts.

Great for use while on-the-go.

Coding Resource: Interactive Coding Challenges

Looking for resources to help you prep for the Coding Interview?

Check out the sister repo Interactive Coding Challenges, which contains an additional Anki deck:

Coding deck

Contributing

Learn from the community.

Feel free to submit pull requests to help:

Fix errors
Improve sections
Add new sections
Translate

Content that needs some polishing is placed under development.

Review the Contributing Guidelines.

Index of system design topics

Summaries of various system design topics, including pros and cons. Everything is a trade-off.

Each section contains links to more in-depth resources.

Study guide

Suggested topics to review based on your interview timeline (short, medium, long).

Q: For interviews, do I need to know everything here?

A: No, you don't need to know everything here to prepare for the interview.

What you are asked in an interview depends on variables such as:

How much experience you have
What your technical background is
What positions you are interviewing for
Which companies you are interviewing with
Luck

More experienced candidates are generally expected to know more about system design. Architects or team leads might be expected to know more than individual contributors. Top tech companies are likely to have one or more design interview rounds.

Start broad and go deeper in a few areas. It helps to know a little about various key system design topics. Adjust the following guide based on your timeline, experience, what positions you are interviewing for, and which companies you are interviewing with.

Short timeline - Aim for breadth with system design topics. Practice by solving some interview questions.
Medium timeline - Aim for breadth and some depth with system design topics. Practice by solving many interview questions.
Long timeline - Aim for breadth and more depth with system design topics. Practice by solving most interview questions.

	Short	Medium	Long
Read through the System design topics to get a broad understanding of how systems work	👍	👍	👍
Read through a few articles in the Company engineering blogs for the companies you are interviewing with	👍	👍	👍
Read through a few Real world architectures	👍	👍	👍
Review How to approach a system design interview question	👍	👍	👍
Work through System design interview questions with solutions	Some	Many	Most
Work through Object-oriented design interview questions with solutions	Some	Many	Most
Review Additional system design interview questions	Some	Many	Most

How to approach a system design interview question

How to tackle a system design interview question.

The system design interview is an open-ended conversation. You are expected to lead it.

You can use the following steps to guide the discussion. To help solidify this process, work through the System design interview questions with solutions section using the following steps.

Step 1: Outline use cases, constraints, and assumptions

Gather requirements and scope the problem. Ask questions to clarify use cases and constraints. Discuss assumptions.

Who is going to use it?
How are they going to use it?
How many users are there?
What does the system do?
What are the inputs and outputs of the system?
How much data do we expect to handle?
How many requests per second do we expect?
What is the expected read to write ratio?

Step 2: Create a high level design

Outline a high level design with all important components.

Sketch the main components and connections
Justify your ideas

Step 3: Design core components

Dive into details for each core component. For example, if you were asked to design a url shortening service, discuss:

Generating and storing a hash of the full url
- MD5 and Base62
- Hash collisions
- SQL or NoSQL
- Database schema
Translating a hashed url to the full url
- Database lookup
API and object-oriented design

Step 4: Scale the design

Identify and address bottlenecks, given the constraints. For example, do you need the following to address scalability issues?

Load balancer
Horizontal scaling
Caching
Database sharding

Discuss potential solutions and trade-offs. Everything is a trade-off. Address bottlenecks using principles of scalable system design.

Back-of-the-envelope calculations

You might be asked to do some estimates by hand. Refer to the Appendix for the following resources:

Source(s) and further reading

Check out the following links to get a better idea of what to expect:

System design interview questions with solutions

Common system design interview questions with sample discussions, code, and diagrams.

Solutions linked to content in the solutions/ folder.

Question
Design Pastebin.com (or Bit.ly)	Solution
Design the Twitter timeline and search (or Facebook feed and search)	Solution
Design a web crawler	Solution
Design Mint.com	Solution
Design the data structures for a social network	Solution
Design a key-value store for a search engine	Solution
Design Amazon's sales ranking by category feature	Solution
Design a system that scales to millions of users on AWS	Solution
Add a system design question	Contribute

Object-oriented design interview questions with solutions

Common object-oriented design interview questions with sample discussions, code, and diagrams.

Solutions linked to content in the solutions/ folder.

Note: This section is under development

Question
Design a hash map	Solution
Design a least recently used cache	Solution
Design a call center	Solution
Design a deck of cards	Solution
Design a parking lot	Solution
Design a chat server	Solution
Design a circular array	Contribute
Add an object-oriented design question	Contribute

System design topics: start here

New to system design?

First, you'll need a basic understanding of common principles, learning about what they are, how they are used, and their pros and cons.

Step 1: Review the scalability video lecture

Scalability Lecture at Harvard

Topics covered:
- Vertical scaling
- Horizontal scaling
- Caching
- Load balancing
- Database replication
- Database partitioning

Step 2: Review the scalability article

Scalability

Topics covered:
- Clones
- Databases
- Caches
- Asynchronism

Next steps

Next, we'll look at high-level trade-offs:

Performance vs scalability
Latency vs throughput
Availability vs consistency

Keep in mind that everything is a trade-off.

Then we'll dive into more specific topics such as DNS, CDNs, and load balancers.

Performance vs scalability

A service is scalable if it results in increased performance in a manner proportional to resources added. Generally, increasing performance means serving more units of work, but it can also be to handle larger units of work, such as when datasets grow.¹

Another way to look at performance vs scalability:

If you have a performance problem, your system is slow for a single user.
If you have a scalability problem, your system is fast for a single user but slow under heavy load.

Source(s) and further reading

Latency vs throughput

Latency is the time to perform some action or to produce some result.

Throughput is the number of such actions or results per unit of time.

Generally, you should aim for maximal throughput with acceptable latency.

Source(s) and further reading

Understanding latency vs throughput

Availability vs consistency

CAP theorem

Source: CAP theorem revisited

In a distributed computer system, you can only support two of the following guarantees:

Consistency - Every read receives the most recent write or an error
Availability - Every request receives a response, without guarantee that it contains the most recent version of the information
Partition Tolerance - The system continues to operate despite arbitrary partitioning due to network failures

Networks aren't reliable, so you'll need to support partition tolerance. You'll need to make a software tradeoff between consistency and availability.

CP - consistency and partition tolerance

Waiting for a response from the partitioned node might result in a timeout error. CP is a good choice if your business needs require atomic reads and writes.

AP - availability and partition tolerance

Responses return the most readily available version of the data available on any node, which might not be the latest. Writes might take some time to propagate when the partition is resolved.

AP is a good choice if the business needs to allow for eventual consistency or when the system needs to continue working despite external errors.

Source(s) and further reading

Consistency patterns

With multiple copies of the same data, we are faced with options on how to synchronize them so clients have a consistent view of the data. Recall the definition of consistency from the CAP theorem - Every read receives the most recent write or an error.

Weak consistency

After a write, reads may or may not see it. A best effort approach is taken.

This approach is seen in systems such as memcached. Weak consistency works well in real time use cases such as VoIP, video chat, and realtime multiplayer games. For example, if you are on a phone call and lose reception for a few seconds, when you regain connection you do not hear what was spoken during connection loss.

Eventual consistency

After a write, reads will eventually see it (typically within milliseconds). Data is replicated asynchronously.

This approach is seen in systems such as DNS and email. Eventual consistency works well in highly available systems.

Strong consistency

After a write, reads will see it. Data is replicated synchronously.

This approach is seen in file systems and RDBMSes. Strong consistency works well in systems that need transactions.

Source(s) and further reading

Transactions across data centers

Availability patterns

There are two complementary patterns to support high availability: fail-over and replication.

Fail-over

Active-passive

With active-passive fail-over, heartbeats are sent between the active and the passive server on standby. If the heartbeat is interrupted, the passive server takes over the active's IP address and resumes service.

The length of downtime is determined by whether the passive server is already running in 'hot' standby or whether it needs to start up from 'cold' standby. Only the active server handles traffic.

Active-passive failover can also be referred to as master-slave failover.

Active-active

In active-active, both servers are managing traffic, spreading the load between them.

If the servers are public-facing, the DNS would need to know about the public IPs of both servers. If the servers are internal-facing, application logic would need to know about both servers.

Active-active failover can also be referred to as master-master failover.

Disadvantage(s): failover

Fail-over adds more hardware and additional complexity.
There is a potential for loss of data if the active system fails before any newly written data can be replicated to the passive.

Replication

Master-slave and master-master

This topic is further discussed in the Database section:

Availability in numbers

Availability is often quantified by uptime (or downtime) as a percentage of time the service is available. Availability is generally measured in number of 9s--a service with 99.99% availability is described as having four 9s.

99.9% availability - three 9s

Duration	Acceptable downtime
Downtime per year	8h 45min 57s
Downtime per month	43m 49.7s
Downtime per week	10m 4.8s
Downtime per day	1m 26.4s

99.99% availability - four 9s

Duration	Acceptable downtime
Downtime per year	52min 35.7s
Downtime per month	4m 23s
Downtime per week	1m 5s
Downtime per day	8.6s

Availability in parallel vs in sequence

If a service consists of multiple components prone to failure, the service's overall availability depends on whether the components are in sequence or in parallel.

In sequence

Overall availability decreases when two components with availability < 100% are in sequence:

Availability (Total) = Availability (Foo) * Availability (Bar)

If both Foo and Bar each had 99.9% availability, their total availability in sequence would be 99.8%.

In parallel

Overall availability increases when two components with availability < 100% are in parallel:

Availability (Total) = 1 - (1 - Availability (Foo)) * (1 - Availability (Bar))

If both Foo and Bar each had 99.9% availability, their total availability in parallel would be 99.9999%.

Domain name system

Source: DNS security presentation

A Domain Name System (DNS) translates a domain name such as www.example.com to an IP address.

DNS is hierarchical, with a few authoritative servers at the top level. Your router or ISP provides information about which DNS server(s) to contact when doing a lookup. Lower level DNS servers cache mappings, which could become stale due to DNS propagation delays. DNS results can also be cached by your browser or OS for a certain period of time, determined by the time to live (TTL).

NS record (name server) - Specifies the DNS servers for your domain/subdomain.
MX record (mail exchange) - Specifies the mail servers for accepting messages.
A record (address) - Points a name to an IP address.
CNAME (canonical) - Points a name to another name or CNAME (example.com to www.example.com) or to an A record.

Services such as CloudFlare and Route 53 provide managed DNS services. Some DNS services can route traffic through various methods:

Weighted round robin
- Prevent traffic from going to servers under maintenance
- Balance between varying cluster sizes
- A/B testing
Latency-based
Geolocation-based

Disadvantage(s): DNS

Accessing a DNS server introduces a slight delay, although mitigated by caching described above.
DNS server management could be complex and is generally managed by governments, ISPs, and large companies.
DNS services have recently come under DDoS attack, preventing users from accessing websites such as Twitter without knowing Twitter's IP address(es).

Source(s) and further reading

Content delivery network

Source: Why use a CDN

A content delivery network (CDN) is a globally distributed network of proxy servers, serving content from locations closer to the user. Generally, static files such as HTML/CSS/JS, photos, and videos are served from CDN, although some CDNs such as Amazon's CloudFront support dynamic content. The site's DNS resolution will tell clients which server to contact.

Serving content from CDNs can significantly improve performance in two ways:

Users receive content from data centers close to them
Your servers do not have to serve requests that the CDN fulfills

Push CDNs

Push CDNs receive new content whenever changes occur on your server. You take full responsibility for providing content, uploading directly to the CDN and rewriting URLs to point to the CDN. You can configure when content expires and when it is updated. Content is uploaded only when it is new or changed, minimizing traffic, but maximizing storage.

Sites with a small amount of traffic or sites with content that isn't often updated work well with push CDNs. Content is placed on the CDNs once, instead of being re-pulled at regular intervals.

Pull CDNs

Pull CDNs grab new content from your server when the first user requests the content. You leave the content on your server and rewrite URLs to point to the CDN. This results in a slower request until the content is cached on the CDN.

A time-to-live (TTL) determines how long content is cached. Pull CDNs minimize storage space on the CDN, but can create redundant traffic if files expire and are pulled before they have actually changed.

Sites with heavy traffic work well with pull CDNs, as traffic is spread out more evenly with only recently-requested content remaining on the CDN.

Disadvantage(s): CDN

CDN costs could be significant depending on traffic, although this should be weighed with additional costs you would incur not using a CDN.
Content might be stale if it is updated before the TTL expires it.
CDNs require changing URLs for static content to point to the CDN.

Source(s) and further reading

Load balancer

Source: Scalable system design patterns

Load balancers distribute incoming client requests to computing resources such as application servers and databases. In each case, the load balancer returns the response from the computing resource to the appropriate client. Load balancers are effective at:

Preventing requests from going to unhealthy servers
Preventing overloading resources
Helping to eliminate a single point of failure

Load balancers can be implemented with hardware (expensive) or with software such as HAProxy.

Additional benefits include:

SSL termination - Decrypt incoming requests and encrypt server responses so backend servers do not have to perform these potentially expensive operations
- Removes the need to install X.509 certificates on each server
Session persistence - Issue cookies and route a specific client's requests to same instance if the web apps do not keep track of sessions

To protect against failures, it's common to set up multiple load balancers, either in active-passive or active-active mode.

Load balancers can route traffic based on various metrics, including:

Random
Least loaded
Session/cookies
Round robin or weighted round robin
Layer 4
Layer 7

Layer 4 load balancing

Layer 4 load balancers look at info at the transport layer to decide how to distribute requests. Generally, this involves the source, destination IP addresses, and ports in the header, but not the contents of the packet. Layer 4 load balancers forward network packets to and from the upstream server, performing Network Address Translation (NAT).

Layer 7 load balancing

Layer 7 load balancers look at the application layer to decide how to distribute requests. This can involve contents of the header, message, and cookies. Layer 7 load balancers terminate network traffic, reads the message, makes a load-balancing decision, then opens a connection to the selected server. For example, a layer 7 load balancer can direct video traffic to servers that host videos while directing more sensitive user billing traffic to security-hardened servers.

At the cost of flexibility, layer 4 load balancing requires less time and computing resources than Layer 7, although the performance impact can be minimal on modern commodity hardware.

Horizontal scaling

Load balancers can also help with horizontal scaling, improving performance and availability. Scaling out using commodity machines is more cost efficient and results in higher availability than scaling up a single server on more expensive hardware, called Vertical Scaling. It is also easier to hire for talent working on commodity hardware than it is for specialized enterprise systems.

Disadvantage(s): horizontal scaling

Scaling horizontally introduces complexity and involves cloning servers
- Servers should be stateless: they should not contain any user-related data like sessions or profile pictures
- Sessions can be stored in a centralized data store such as a database (SQL, NoSQL) or a persistent cache (Redis, Memcached)
Downstream servers such as caches and databases need to handle more simultaneous connections as upstream servers scale out

Disadvantage(s): load balancer

The load balancer can become a performance bottleneck if it does not have enough resources or if it is not configured properly.
Introducing a load balancer to help eliminate a single point of failure results in increased complexity.
A single load balancer is a single point of failure, configuring multiple load balancers further increases complexity.

Source(s) and further reading

Reverse proxy (web server)

Source: Wikipedia

A reverse proxy is a web server that centralizes internal services and provides unified interfaces to the public. Requests from clients are forwarded to a server that can fulfill it before the reverse proxy returns the server's response to the client.

Additional benefits include:

Increased security - Hide information about backend servers, blacklist IPs, limit number of connections per client
Increased scalability and flexibility - Clients only see the reverse proxy's IP, allowing you to scale servers or change their configuration
SSL termination - Decrypt incoming requests and encrypt server responses so backend servers do not have to perform these potentially expensive operations
- Removes the need to install X.509 certificates on each server
Compression - Compress server responses
Caching - Return the response for cached requests
Static content - Serve static content directly
- HTML/CSS/JS
- Photos
- Videos
- Etc

Load balancer vs reverse proxy

Deploying a load balancer is useful when you have multiple servers. Often, load balancers route traffic to a set of servers serving the same function.
Reverse proxies can be useful even with just one web server or application server, opening up the benefits described in the previous section.
Solutions such as NGINX and HAProxy can support both layer 7 reverse proxying and load balancing.

Disadvantage(s): reverse proxy

Introducing a reverse proxy results in increased complexity.
A single reverse proxy is a single point of failure, configuring multiple reverse proxies (ie a failover) further increases complexity.

Source(s) and further reading

Application layer

Source: Intro to architecting systems for scale

Separating out the web layer from the application layer (also known as platform layer) allows you to scale and configure both layers independently. Adding a new API results in adding application servers without necessarily adding additional web servers. The single responsibility principle advocates for small and autonomous services that work together. Small teams with small services can plan more aggressively for rapid growth.

Workers in the application layer also help enable asynchronism.

Microservices

Related to this discussion are microservices, which can be described as a suite of independently deployable, small, modular services. Each service runs a unique process and communicates through a well-defined, lightweight mechanism to serve a business goal. ¹

Pinterest, for example, could have the following microservices: user profile, follower, feed, search, photo upload, etc.

Service Discovery

Systems such as Consul, Etcd, and Zookeeper can help services find each other by keeping track of registered names, addresses, and ports. Health checks help verify service integrity and are often done using an HTTP endpoint. Both Consul and Etcd have a built in key-value store that can be useful for storing config values and other shared data.

Disadvantage(s): application layer

Adding an application layer with loosely coupled services requires a different approach from an architectural, operations, and process viewpoint (vs a monolithic system).
Microservices can add complexity in terms of deployments and operations.

Source(s) and further reading

Database

Source: Scaling up to your first 10 million users

Relational database management system (RDBMS)

A relational database like SQL is a collection of data items organized in tables.

ACID is a set of properties of relational database transactions.

Atomicity - Each transaction is all or nothing
Consistency - Any transaction will bring the database from one valid state to another
Isolation - Executing transactions concurrently has the same results as if the transactions were executed serially
Durability - Once a transaction has been committed, it will remain so

There are many techniques to scale a relational database: master-slave replication, master-master replication, federation, sharding, denormalization, and SQL tuning.

Master-slave replication

The master serves reads and writes, replicating writes to one or more slaves, which serve only reads. Slaves can also replicate to additional slaves in a tree-like fashion. If the master goes offline, the system can continue to operate in read-only mode until a slave is promoted to a master or a new master is provisioned.

Source: Scalability, availability, stability, patterns

Disadvantage(s): master-slave replication

Additional logic is needed to promote a slave to a master.
See Disadvantage(s): replication for points related to both master-slave and master-master.

Master-master replication

Both masters serve reads and writes and coordinate with each other on writes. If either master goes down, the system can continue to operate with both reads and writes.

Source: Scalability, availability, stability, patterns

Disadvantage(s): master-master replication

You'll need a load balancer or you'll need to make changes to your application logic to determine where to write.
Most master-master systems are either loosely consistent (violating ACID) or have increased write latency due to synchronization.
Conflict resolution comes more into play as more write nodes are added and as latency increases.
See Disadvantage(s): replication for points related to both master-slave and master-master.

Disadvantage(s): replication

There is a potential for loss of data if the master fails before any newly written data can be replicated to other nodes.
Writes are replayed to the read replicas. If there are a lot of writes, the read replicas can get bogged down with replaying writes and can't do as many reads.
The more read slaves, the more you have to replicate, which leads to greater replication lag.
On some systems, writing to the master can spawn multiple threads to write in parallel, whereas read replicas only support writing sequentially with a single thread.
Replication adds more hardware and additional complexity.

Source(s) and further reading: replication

Federation

Source: Scaling up to your first 10 million users

Federation (or functional partitioning) splits up databases by function. For example, instead of a single, monolithic database, you could have three databases: forums, users, and products, resulting in less read and write traffic to each database and therefore less replication lag. Smaller databases result in more data that can fit in memory, which in turn results in more cache hits due to improved cache locality. With no single central master serializing writes you can write in parallel, increasing throughput.

Disadvantage(s): federation

Federation is not effective if your schema requires huge functions or tables.
You'll need to update your application logic to determine which database to read and write.
Joining data from two databases is more complex with a server link.
Federation adds more hardware and additional complexity.

Source(s) and further reading: federation

Scaling up to your first 10 million users

Sharding

Source: Scalability, availability, stability, patterns

Sharding distributes data across different databases such that each database can only manage a subset of the data. Taking a users database as an example, as the number of users increases, more shards are added to the cluster.

Similar to the advantages of federation, sharding results in less read and write traffic, less replication, and more cache hits. Index size is also reduced, which generally improves performance with faster queries. If one shard goes down, the other shards are still operational, although you'll want to add some form of replication to avoid data loss. Like federation, there is no single central master serializing writes, allowing you to write in parallel with increased throughput.

Common ways to shard a table of users is either through the user's last name initial or the user's geographic location.

Disadvantage(s): sharding

You'll need to update your application logic to work with shards, which could result in complex SQL queries.
Data distribution can become lopsided in a shard. For example, a set of power users on a shard could result in increased load to that shard compared to others.
- Rebalancing adds additional complexity. A sharding function based on consistent hashing can reduce the amount of transferred data.
Joining data from multiple shards is more complex.
Sharding adds more hardware and additional complexity.

Source(s) and further reading: sharding

Denormalization

Denormalization attempts to improve read performance at the expense of some write performance. Redundant copies of the data are written in multiple tables to avoid expensive joins. Some RDBMS such as PostgreSQL and Oracle support materialized views which handle the work of storing redundant information and keeping redundant copies consistent.

Once data becomes distributed with techniques such as federation and sharding, managing joins across data centers further increases complexity. Denormalization might circumvent the need for such complex joins.

In most systems, reads can heavily outnumber writes 100:1 or even 1000:1. A read resulting in a complex database join can be very expensive, spending a significant amount of time on disk operations.

Disadvantage(s): denormalization

Data is duplicated.
Constraints can help redundant copies of information stay in sync, which increases complexity of the database design.
A denormalized database under heavy write load might perform worse than its normalized counterpart.

Source(s) and further reading: denormalization

Denormalization

SQL tuning

SQL tuning is a broad topic and many books have been written as reference.

It's important to benchmark and profile to simulate and uncover bottlenecks.

Benchmark - Simulate high-load situations with tools such as ab.
Profile - Enable tools such as the slow query log to help track performance issues.

Benchmarking and profiling might point you to the following optimizations.

Tighten up the schema

MySQL dumps to disk in contiguous blocks for fast access.
Use CHAR instead of VARCHAR for fixed-length fields.
- CHAR effectively allows for fast, random access, whereas with VARCHAR, you must find the end of a string before moving onto the next one.
Use TEXT for large blocks of text such as blog posts. TEXT also allows for boolean searches. Using a TEXT field results in storing a pointer on disk that is used to locate the text block.
Use INT for larger numbers up to 2^32 or 4 billion.
Use DECIMAL for currency to avoid floating point representation errors.
Avoid storing large BLOBS, store the location of where to get the object instead.
VARCHAR(255) is the largest number of characters that can be counted in an 8 bit number, often maximizing the use of a byte in some RDBMS.
Set the NOT NULL constraint where applicable to improve search performance.

Use good indices

Columns that you are querying (SELECT, GROUP BY, ORDER BY, JOIN) could be faster with indices.
Indices are usually represented as self-balancing B-tree that keeps data sorted and allows searches, sequential access, insertions, and deletions in logarithmic time.
Placing an index can keep the data in memory, requiring more space.
Writes could also be slower since the index also needs to be updated.
When loading large amounts of data, it might be faster to disable indices, load the data, then rebuild the indices.

Avoid expensive joins

Denormalize where performance demands it.

Partition tables

Break up a table by putting hot spots in a separate table to help keep it in memory.

Tune the query cache

In some cases, the query cache could lead to performance issues.

Source(s) and further reading: SQL tuning

NoSQL

NoSQL is a collection of data items represented in a key-value store, document store, wide column store, or a graph database. Data is denormalized, and joins are generally done in the application code. Most NoSQL stores lack true ACID transactions and favor eventual consistency.

BASE is often used to describe the properties of NoSQL databases. In comparison with the CAP Theorem, BASE chooses availability over consistency.

Basically available - the system guarantees availability.
Soft state - the state of the system may change over time, even without input.
Eventual consistency - the system will become consistent over a period of time, given that the system doesn't receive input during that period.

In addition to choosing between SQL or NoSQL, it is helpful to understand which type of NoSQL database best fits your use case(s). We'll review key-value stores, document stores, wide column stores, and graph databases in the next section.

Key-value store

Abstraction: hash table

A key-value store generally allows for O(1) reads and writes and is often backed by memory or SSD. Data stores can maintain keys in lexicographic order, allowing efficient retrieval of key ranges. Key-value stores can allow for storing of metadata with a value.

Key-value stores provide high performance and are often used for simple data models or for rapidly-changing data, such as an in-memory cache layer. Since they offer only a limited set of operations, complexity is shifted to the application layer if additional operations are needed.

A key-value store is the basis for more complex systems such as a document store, and in some cases, a graph database.

Source(s) and further reading: key-value store

Document store

Abstraction: key-value store with documents stored as values

A document store is centered around documents (XML, JSON, binary, etc), where a document stores all information for a given object. Document stores provide APIs or a query language to query based on the internal structure of the document itself. Note, many key-value stores include features for working with a value's metadata, blurring the lines between these two storage types.

Based on the underlying implementation, documents are organized by collections, tags, metadata, or directories. Although documents can be organized or grouped together, documents may have fields that are completely different from each other.

Some document stores like MongoDB and CouchDB also provide a SQL-like language to perform complex queries. DynamoDB supports both key-values and documents.

Document stores provide high flexibility and are often used for working with occasionally changing data.

Source(s) and further reading: document store

Wide column store

Source: SQL & NoSQL, a brief history

Abstraction: nested map ColumnFamily<RowKey, Columns<ColKey, Value, Timestamp>>

A wide column store's basic unit of data is a column (name/value pair). A column can be grouped in column families (analogous to a SQL table). Super column families further group column families. You can access each column independently with a row key, and columns with the same row key form a row. Each value contains a timestamp for versioning and for conflict resolution.

Google introduced Bigtable as the first wide column store, which influenced the open-source HBase often-used in the Hadoop ecosystem, and Cassandra from Facebook. Stores such as BigTable, HBase, and Cassandra maintain keys in lexicographic order, allowing efficient retrieval of selective key ranges.

Wide column stores offer high availability and high scalability. They are often used for very large data sets.

Source(s) and further reading: wide column store

Graph database

Source: Graph database

Abstraction: graph

In a graph database, each node is a record and each arc is a relationship between two nodes. Graph databases are optimized to represent complex relationships with many foreign keys or many-to-many relationships.

Graphs databases offer high performance for data models with complex relationships, such as a social network. They are relatively new and are not yet widely-used; it might be more difficult to find development tools and resources. Many graphs can only be accessed with REST APIs.

Source(s) and further reading: graph

Source(s) and further reading: NoSQL

SQL or NoSQL

Source: Transitioning from RDBMS to NoSQL

Reasons for SQL:

Structured data
Strict schema
Relational data
Need for complex joins
Transactions
Clear patterns for scaling
More established: developers, community, code, tools, etc
Lookups by index are very fast

Reasons for NoSQL:

Semi-structured data
Dynamic or flexible schema
Non-relational data
No need for complex joins
Store many TB (or PB) of data
Very data intensive workload
Very high throughput for IOPS

Sample data well-suited for NoSQL:

Rapid ingest of clickstream and log data
Leaderboard or scoring data
Temporary data, such as a shopping cart
Frequently accessed ('hot') tables
Metadata/lookup tables

Source(s) and further reading: SQL or NoSQL

Cache

Source: Scalable system design patterns

Caching improves page load times and can reduce the load on your servers and databases. In this model, the dispatcher will first lookup if the request has been made before and try to find the previous result to return, in order to save the actual execution.

Databases often benefit from a uniform distribution of reads and writes across its partitions. Popular items can skew the distribution, causing bottlenecks. Putting a cache in front of a database can help absorb uneven loads and spikes in traffic.

Client caching

Caches can be located on the client side (OS or browser), server side, or in a distinct cache layer.

CDN caching

CDNs are considered a type of cache.

Web server caching

Reverse proxies and caches such as Varnish can serve static and dynamic content directly. Web servers can also cache requests, returning responses without having to contact application servers.

Database caching

Your database usually includes some level of caching in a default configuration, optimized for a generic use case. Tweaking these settings for specific usage patterns can further boost performance.

Application caching

In-memory caches such as Memcached and Redis are key-value stores between your application and your data storage. Since the data is held in RAM, it is much faster than typical databases where data is stored on disk. RAM is more limited than disk, so cache invalidation algorithms such as least recently used (LRU) can help invalidate 'cold' entries and keep 'hot' data in RAM.

Redis has the following additional features:

Persistence option
Built-in data structures such as sorted sets and lists

There are multiple levels you can cache that fall into two general categories: database queries and objects:

Row level
Query-level
Fully-formed serializable objects
Fully-rendered HTML

Generally, you should try to avoid file-based caching, as it makes cloning and auto-scaling more difficult.

Caching at the database query level

Whenever you query the database, hash the query as a key and store the result to the cache. This approach suffers from expiration issues:

Hard to delete a cached result with complex queries
If one piece of data changes such as a table cell, you need to delete all cached queries that might include the changed cell

Caching at the object level

See your data as an object, similar to what you do with your application code. Have your application assemble the dataset from the database into a class instance or a data structure(s):

Remove the object from cache if its underlying data has changed
Allows for asynchronous processing: workers assemble objects by consuming the latest cached object

Suggestions of what to cache:

User sessions
Fully rendered web pages
Activity streams
User graph data

When to update the cache

Since you can only store a limited amount of data in cache, you'll need to determine which cache update strategy works best for your use case.

Cache-aside

Source: From cache to in-memory data grid

The application is responsible for reading and writing from storage. The cache does not interact with storage directly. The application does the following:

Look for entry in cache, resulting in a cache miss
Load entry from the database
Add entry to cache
Return entry

def get_user(self, user_id):
    user = cache.get("user.{0}", user_id)
    if user is None:
        user = db.query("SELECT * FROM users WHERE user_id = {0}", user_id)
        if user is not None:
            key = "user.{0}".format(user_id)
            cache.set(key, json.dumps(user))
    return user

Memcached is generally used in this manner.

Subsequent reads of data added to cache are fast. Cache-aside is also referred to as lazy loading. Only requested data is cached, which avoids filling up the cache with data that isn't requested.

Disadvantage(s): cache-aside

Each cache miss results in three trips, which can cause a noticeable delay.
Data can become stale if it is updated in the database. This issue is mitigated by setting a time-to-live (TTL) which forces an update of the cache entry, or by using write-through.
When a node fails, it is replaced by a new, empty node, increasing latency.

Write-through

Source: Scalability, availability, stability, patterns

The application uses the cache as the main data store, reading and writing data to it, while the cache is responsible for reading and writing to the database:

Application adds/updates entry in cache
Cache synchronously writes entry to data store
Return

Application code:

set_user(12345, {"foo":"bar"})

Cache code:

def set_user(user_id, values):
    user = db.query("UPDATE Users WHERE id = {0}", user_id, values)
    cache.set(user_id, user)

Write-through is a slow overall operation due to the write operation, but subsequent reads of just written data are fast. Users are generally more tolerant of latency when updating data than reading data. Data in the cache is not stale.

Disadvantage(s): write through

When a new node is created due to failure or scaling, the new node will not cache entries until the entry is updated in the database. Cache-aside in conjunction with write through can mitigate this issue.
Most data written might never be read, which can be minimized with a TTL.

Write-behind (write-back)

Source: Scalability, availability, stability, patterns

In write-behind, the application does the following:

Add/update entry in cache
Asynchronously write entry to the data store, improving write performance

Disadvantage(s): write-behind

There could be data loss if the cache goes down prior to its contents hitting the data store.
It is more complex to implement write-behind than it is to implement cache-aside or write-through.

Refresh-ahead

Source: From cache to in-memory data grid

You can configure the cache to automatically refresh any recently accessed cache entry prior to its expiration.

Refresh-ahead can result in reduced latency vs read-through if the cache can accurately predict which items are likely to be needed in the future.

Disadvantage(s): refresh-ahead

Not accurately predicting which items are likely to be needed in the future can result in reduced performance than without refresh-ahead.

Disadvantage(s): cache

Need to maintain consistency between caches and the source of truth such as the database through cache invalidation.
Cache invalidation is a difficult problem, there is additional complexity associated with when to update the cache.
Need to make application changes such as adding Redis or memcached.

Source(s) and further reading

Asynchronism

Source: Intro to architecting systems for scale

Asynchronous workflows help reduce request times for expensive operations that would otherwise be performed in-line. They can also help by doing time-consuming work in advance, such as periodic aggregation of data.

Message queues

Message queues receive, hold, and deliver messages. If an operation is too slow to perform inline, you can use a message queue with the following workflow:

An application publishes a job to the queue, then notifies the user of job status
A worker picks up the job from the queue, processes it, then signals the job is complete

The user is not blocked and the job is processed in the background. During this time, the client might optionally do a small amount of processing to make it seem like the task has completed. For example, if posting a tweet, the tweet could be instantly posted to your timeline, but it could take some time before your tweet is actually delivered to all of your followers.

Redis is useful as a simple message broker but messages can be lost.

RabbitMQ is popular but requires you to adapt to the 'AMQP' protocol and manage your own nodes.

Amazon SQS is hosted but can have high latency and has the possibility of messages being delivered twice.

Task queues

Tasks queues receive tasks and their related data, runs them, then delivers their results. They can support scheduling and can be used to run computationally-intensive jobs in the background.

Celery has support for scheduling and primarily has python support.

Back pressure

If queues start to grow significantly, the queue size can become larger than memory, resulting in cache misses, disk reads, and even slower performance. Back pressure can help by limiting the queue size, thereby maintaining a high throughput rate and good response times for jobs already in the queue. Once the queue fills up, clients get a server busy or HTTP 503 status code to try again later. Clients can retry the request at a later time, perhaps with exponential backoff.

Disadvantage(s): asynchronism

Use cases such as inexpensive calculations and realtime workflows might be better suited for synchronous operations, as introducing queues can add delays and complexity.

Source(s) and further reading

Communication

Source: OSI 7 layer model

Hypertext transfer protocol (HTTP)

HTTP is a method for encoding and transporting data between a client and a server. It is a request/response protocol: clients issue requests and servers issue responses with relevant content and completion status info about the request. HTTP is self-contained, allowing requests and responses to flow through many intermediate routers and servers that perform load balancing, caching, encryption, and compression.

A basic HTTP request consists of a verb (method) and a resource (endpoint). Below are common HTTP verbs:

Verb	Description	Idempotent*	Safe	Cacheable
GET	Reads a resource	Yes	Yes	Yes
POST	Creates a resource or trigger a process that handles data	No	No	Yes if response contains freshness info
PUT	Creates or replace a resource	Yes	No	No
PATCH	Partially updates a resource	No	No	Yes if response contains freshness info
DELETE	Deletes a resource	Yes	No	No

*Can be called many times without different outcomes.

HTTP is an application layer protocol relying on lower-level protocols such as TCP and UDP.

Source(s) and further reading: HTTP

Transmission control protocol (TCP)

Source: How to make a multiplayer game

TCP is a connection-oriented protocol over an IP network. Connection is established and terminated using a handshake. All packets sent are guaranteed to reach the destination in the original order and without corruption through:

Sequence numbers and checksum fields for each packet
Acknowledgement packets and automatic retransmission

If the sender does not receive a correct response, it will resend the packets. If there are multiple timeouts, the connection is dropped. TCP also implements flow control and congestion control. These guarantees cause delays and generally result in less efficient transmission than UDP.

To ensure high throughput, web servers can keep a large number of TCP connections open, resulting in high memory usage. It can be expensive to have a large number of open connections between web server threads and say, a memcached server. Connection pooling can help in addition to switching to UDP where applicable.

TCP is useful for applications that require high reliability but are less time critical. Some examples include web servers, database info, SMTP, FTP, and SSH.

Use TCP over UDP when:

You need all of the data to arrive intact
You want to automatically make a best estimate use of the network throughput

User datagram protocol (UDP)

Source: How to make a multiplayer game

UDP is connectionless. Datagrams (analogous to packets) are guaranteed only at the datagram level. Datagrams might reach their destination out of order or not at all. UDP does not support congestion control. Without the guarantees that TCP support, UDP is generally more efficient.

UDP can broadcast, sending datagrams to all devices on the subnet. This is useful with DHCP because the client has not yet received an IP address, thus preventing a way for TCP to stream without the IP address.

UDP is less reliable but works well in real time use cases such as VoIP, video chat, streaming, and realtime multiplayer games.

Use UDP over TCP when:

You need the lowest latency
Late data is worse than loss of data
You want to implement your own error correction

Source(s) and further reading: TCP and UDP

Remote procedure call (RPC)

Source: Crack the system design interview

In an RPC, a client causes a procedure to execute on a different address space, usually a remote server. The procedure is coded as if it were a local procedure call, abstracting away the details of how to communicate with the server from the client program. Remote calls are usually slower and less reliable than local calls so it is helpful to distinguish RPC calls from local calls. Popular RPC frameworks include Protobuf, Thrift, and Avro.

RPC is a request-response protocol:

Client program - Calls the client stub procedure. The parameters are pushed onto the stack like a local procedure call.
Client stub procedure - Marshals (packs) procedure id and arguments into a request message.
Client communication module - OS sends the message from the client to the server.
Server communication module - OS passes the incoming packets to the server stub procedure.
Server stub procedure - Unmarshalls the results, calls the server procedure matching the procedure id and passes the given arguments.
The server response repeats the steps above in reverse order.

Sample RPC calls:

GET /someoperation?data=anId

POST /anotheroperation
{
  "data":"anId";
  "anotherdata": "another value"
}

RPC is focused on exposing behaviors. RPCs are often used for performance reasons with internal communications, as you can hand-craft native calls to better fit your use cases.

Choose a native library (aka SDK) when:

You know your target platform.
You want to control how your "logic" is accessed.
You want to control how error control happens off your library.
Performance and end user experience is your primary concern.

HTTP APIs following REST tend to be used more often for public APIs.

Disadvantage(s): RPC

RPC clients become tightly coupled to the service implementation.
A new API must be defined for every new operation or use case.
It can be difficult to debug RPC.
You might not be able to leverage existing technologies out of the box. For example, it might require additional effort to ensure RPC calls are properly cached on caching servers such as Squid.

Representational state transfer (REST)

REST is an architectural style enforcing a client/server model where the client acts on a set of resources managed by the server. The server provides a representation of resources and actions that can either manipulate or get a new representation of resources. All communication must be stateless and cacheable.

There are four qualities of a RESTful interface:

Identify resources (URI in HTTP) - use the same URI regardless of any operation.
Change with representations (Verbs in HTTP) - use verbs, headers, and body.
Self-descriptive error message (status response in HTTP) - Use status codes, don't reinvent the wheel.
HATEOAS (HTML interface for HTTP) - your web service should be fully accessible in a browser.

Sample REST calls:

GET /someresources/anId

PUT /someresources/anId
{"anotherdata": "another value"}

REST is focused on exposing data. It minimizes the coupling between client/server and is often used for public HTTP APIs. REST uses a more generic and uniform method of exposing resources through URIs, representation through headers, and actions through verbs such as GET, POST, PUT, DELETE, and PATCH. Being stateless, REST is great for horizontal scaling and partitioning.

Disadvantage(s): REST

With REST being focused on exposing data, it might not be a good fit if resources are not naturally organized or accessed in a simple hierarchy. For example, returning all updated records from the past hour matching a particular set of events is not easily expressed as a path. With REST, it is likely to be implemented with a combination of URI path, query parameters, and possibly the request body.
REST typically relies on a few verbs (GET, POST, PUT, DELETE, and PATCH) which sometimes doesn't fit your use case. For example, moving expired documents to the archive folder might not cleanly fit within these verbs.
Fetching complicated resources with nested hierarchies requires multiple round trips between the client and server to render single views, e.g. fetching content of a blog entry and the comments on that entry. For mobile applications operating in variable network conditions, these multiple roundtrips are highly undesirable.
Over time, more fields might be added to an API response and older clients will receive all new data fields, even those that they do not need, as a result, it bloats the payload size and leads to larger latencies.

RPC and REST calls comparison

Operation	RPC	REST
Signup	POST /signup	POST /persons
Resign	POST /resign { "personid": "1234" }	DELETE /persons/1234
Read a person	GET /readPerson?personid=1234	GET /persons/1234
Read a person’s items list	GET /readUsersItemsList?personid=1234	GET /persons/1234/items
Add an item to a person’s items	POST /addItemToUsersItemsList { "personid": "1234"; "itemid": "456" }	POST /persons/1234/items { "itemid": "456" }
Update an item	POST /modifyItem { "itemid": "456"; "key": "value" }	PUT /items/456 { "key": "value" }
Delete an item	POST /removeItem { "itemid": "456" }	DELETE /items/456

Source: Do you really know why you prefer REST over RPC

Source(s) and further reading: REST and RPC

Security

This section could use some updates. Consider contributing!

Security is a broad topic. Unless you have considerable experience, a security background, or are applying for a position that requires knowledge of security, you probably won't need to know more than the basics:

Encrypt in transit and at rest.
Sanitize all user inputs or any input parameters exposed to user to prevent XSS and SQL injection.
Use parameterized queries to prevent SQL injection.
Use the principle of least privilege.

Source(s) and further reading

Appendix

You'll sometimes be asked to do 'back-of-the-envelope' estimates. For example, you might need to determine how long it will take to generate 100 image thumbnails from disk or how much memory a data structure will take. The Powers of two table and Latency numbers every programmer should know are handy references.

Powers of two table

Power           Exact Value         Approx Value        Bytes
---------------------------------------------------------------
7                             128
8                             256
10                           1024   1 thousand           1 KB
16                         65,536                       64 KB
20                      1,048,576   1 million            1 MB
30                  1,073,741,824   1 billion            1 GB
32                  4,294,967,296                        4 GB
40              1,099,511,627,776   1 trillion           1 TB

Source(s) and further reading

Powers of two

Latency numbers every programmer should know

Latency Comparison Numbers
--------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy            10,000   ns       10 us
Send 1 KB bytes over 1 Gbps network     10,000   ns       10 us
Read 4 KB randomly from SSD*           150,000   ns      150 us          ~1GB/sec SSD
Read 1 MB sequentially from memory     250,000   ns      250 us
Round trip within same datacenter      500,000   ns      500 us
Read 1 MB sequentially from SSD*     1,000,000   ns    1,000 us    1 ms  ~1GB/sec SSD, 4X memory
HDD seek                            10,000,000   ns   10,000 us   10 ms  20x datacenter roundtrip
Read 1 MB sequentially from 1 Gbps  10,000,000   ns   10,000 us   10 ms  40x memory, 10X SSD
Read 1 MB sequentially from HDD     30,000,000   ns   30,000 us   30 ms 120x memory, 30X SSD
Send packet CA->Netherlands->CA    150,000,000   ns  150,000 us  150 ms

Notes
-----
1 ns = 10^-9 seconds
1 us = 10^-6 seconds = 1,000 ns
1 ms = 10^-3 seconds = 1,000 us = 1,000,000 ns

Handy metrics based on numbers above:

Read sequentially from HDD at 30 MB/s
Read sequentially from 1 Gbps Ethernet at 100 MB/s
Read sequentially from SSD at 1 GB/s
Read sequentially from main memory at 4 GB/s
6-7 world-wide round trips per second
2,000 round trips per second within a data center

Latency numbers visualized

Source(s) and further reading

Additional system design interview questions

Common system design interview questions, with links to resources on how to solve each.

Question	Reference(s)
Design a file sync service like Dropbox	youtube.com
Design a search engine like Google	queue.acm.org stackexchange.com ardendertat.com stanford.edu
Design a scalable web crawler like Google	quora.com
Design Google docs	code.google.com neil.fraser.name
Design a key-value store like Redis	slideshare.net
Design a cache system like Memcached	slideshare.net
Design a recommendation system like Amazon's	hulu.com ijcai13.org
Design a tinyurl system like Bitly	n00tc0d3r.blogspot.com
Design a chat app like WhatsApp	highscalability.com
Design a picture sharing system like Instagram	highscalability.com highscalability.com
Design the Facebook news feed function	quora.com quora.com slideshare.net
Design the Facebook timeline function	facebook.com highscalability.com
Design the Facebook chat function	erlang-factory.com facebook.com
Design a graph search function like Facebook's	facebook.com facebook.com facebook.com
Design a content delivery network like CloudFlare	figshare.com
Design a trending topic system like Twitter's	michael-noll.com snikolov .wordpress.com
Design a random ID generation system	blog.twitter.com github.com
Return the top k requests during a time interval	cs.ucsb.edu wpi.edu
Design a system that serves data from multiple data centers	highscalability.com
Design an online multiplayer card game	indieflashblog.com buildnewgames.com
Design a garbage collection system	stuffwithstuff.com washington.edu
Design an API rate limiter	https://stripe.com/blog/
Design a Stock Exchange (like NASDAQ or Binance)	Jane Street Golang Implementation Go Implementation
Add a system design question	Contribute

Real world architectures

Articles on how real world systems are designed.

Source: Twitter timelines at scale

Don't focus on nitty gritty details for the following articles, instead:

Identify shared principles, common technologies, and patterns within these articles
Study what problems are solved by each component, where it works, where it doesn't
Review the lessons learned

Type	System	Reference(s)
Data processing	MapReduce - Distributed data processing from Google	research.google.com
Data processing	Spark - Distributed data processing from Databricks	slideshare.net
Data processing	Storm - Distributed data processing from Twitter	slideshare.net

Data store	Bigtable - Distributed column-oriented database from Google	harvard.edu
Data store	HBase - Open source implementation of Bigtable	slideshare.net
Data store	Cassandra - Distributed column-oriented database from Facebook	slideshare.net
Data store	DynamoDB - Document-oriented database from Amazon	harvard.edu
Data store	MongoDB - Document-oriented database	slideshare.net
Data store	Spanner - Globally-distributed database from Google	research.google.com
Data store	Memcached - Distributed memory caching system	slideshare.net
Data store	Redis - Distributed memory caching system with persistence and value types	slideshare.net

File system	Google File System (GFS) - Distributed file system	research.google.com
File system	Hadoop File System (HDFS) - Open source implementation of GFS	apache.org

Misc	Chubby - Lock service for loosely-coupled distributed systems from Google	research.google.com
Misc	Dapper - Distributed systems tracing infrastructure	research.google.com
Misc	Kafka - Pub/sub message queue from LinkedIn	slideshare.net
Misc	Zookeeper - Centralized infrastructure and services enabling synchronization	slideshare.net
	Add an architecture	Contribute

Company architectures

Company	Reference(s)
Amazon	Amazon architecture
Cinchcast	Producing 1,500 hours of audio every day
DataSift	Realtime datamining At 120,000 tweets per second
Dropbox	How we've scaled Dropbox
ESPN	Operating At 100,000 duh nuh nuhs per second
Google	Google architecture
Instagram	14 million users, terabytes of photos What powers Instagram
Justin.tv	Justin.Tv's live video broadcasting architecture
Facebook	Scaling memcached at Facebook TAO: Facebook’s distributed data store for the social graph Facebook’s photo storage How Facebook Live Streams To 800,000 Simultaneous Viewers
Flickr	Flickr architecture
Mailbox	From 0 to one million users in 6 weeks
Netflix	A 360 Degree View Of The Entire Netflix Stack Netflix: What Happens When You Press Play?
Pinterest	From 0 To 10s of billions of page views a month 18 million visitors, 10x growth, 12 employees
Playfish	50 million monthly users and growing
PlentyOfFish	PlentyOfFish architecture
Salesforce	How they handle 1.3 billion transactions a day
Stack Overflow	Stack Overflow architecture
TripAdvisor	40M visitors, 200M dynamic page views, 30TB data
Tumblr	15 billion page views a month
Twitter	Making Twitter 10000 percent faster Storing 250 million tweets a day using MySQL 150M active users, 300K QPS, a 22 MB/S firehose Timelines at scale Big and small data at Twitter Operations at Twitter: scaling beyond 100 million users How Twitter Handles 3,000 Images Per Second
Uber	How Uber scales their real-time market platform Lessons Learned From Scaling Uber To 2000 Engineers, 1000 Services, And 8000 Git Repositories
WhatsApp	The WhatsApp architecture Facebook bought for $19 billion
YouTube	YouTube scalability YouTube architecture

Company engineering blogs

Architectures for companies you are interviewing with.

Questions you encounter might be from the same domain.

Source(s) and further reading

Looking to add a blog? To avoid duplicating work, consider adding your company blog to the following repo:

kilimchoi/engineering-blogs

Under development

Interested in adding a section or helping complete one in-progress? Contribute!

Distributed computing with MapReduce
Consistent hashing
Scatter gather
Contribute

Credits

Credits and sources are provided throughout this repo.

Special thanks to:

Contact info

Feel free to contact me to discuss any issues, questions, or comments.

My contact info can be found on my GitHub page.

License

I am providing code and resources in this repository to you under an open source license. Because this is my personal repository, the license you receive to my code and resources is from me and not my employer (Facebook).

Copyright 2017 Donne Martin

Creative Commons Attribution 4.0 International License (CC BY 4.0)

http://creativecommons.org/licenses/by/4.0/

GitHub 1C Enterprise Daily Trending

Pr-Mex/vanessa-automation

SAP-samples/abap-platform-rap-workshops

hengyoush/kyanos

The open-source alert management and AIOps platform

Supported Integrations

Observability Tools

Databases & Data Warehouses

Communication Platforms

Incident Management

Ticketing Tools

Container Orchecstration Platforms

Data Enrichment

Workflows

Enterprise Ready

Getting Started

🫵 Keepers

Top Contributors

Contributors

yamadashy/repomix

📦 Repomix (formerly Repopack)

📢 Important Notice: Project Renamed to Repomix

Migration Guide

Configuration Files

Timeline

🌟 Features

🚀 Quick Start

📊 Usage

Prompt Examples

Code Review and Refactoring

Documentation Generation

Test Case Generation

Code Quality Assessment

Library Overview

Community Discussion

Output File Format

Plain Text Format (default)

XML Format

Markdown Format

Command Line Options

Updating Repomix

Remote Repository Processing

⚙️ Configuration

Global Configuration

Include and Ignore

Include Patterns

Ignore Patterns

Custom Instruction

Comment Removal

🔍 Security Check

🤝 Contribution

Contributors

📜 License

SimonCropp/Delta

Delta

Assumptions

304 Not Modified Flow

ETag calculation logic

AssemblyWriteTime

SQL timestamp

Suffix

Combining the above

NuGet

Usage

DB Schema

Add to WebApplicationBuilder

Add to a Route Group

ShouldExecute

Custom Connection discovery

EF Usage

DbContext using RowVersion

Add to WebApplicationBuilder

Add to a Route Group

ShouldExecute

UseResponseDiagnostics

Helpers

GetLastTimeStamp

For a SqlConnection:

For a DbContext:

GetDatabasesWithTracking

For a `SqlConnection`:

For a `DbContext`: