GitHub 1C Enterprise Daily Trending

phil-opp/blog_os

2024-09-16T01:28:33Z

Writing an OS in Rust

Blog OS

This repository contains the source code for the Writing an OS in Rust series at os.phil-opp.com.

If you have questions, open an issue or chat with us on Gitter.

Where is the code?

The code for each post lives in a separate git branch. This makes it possible to see the intermediate state after each post.

The code for the latest post is available here.

You can find the branch for each post by following the (source code) link in the post list below. The branches are named post-XX where XX is the post number, for example post-03 for the VGA Text Mode post or post-07 for the Hardware Interrupts post. For build instructions, see the Readme of the respective branch.

You can check out a branch in a subdirectory using git worktree:

git worktree add code post-10

The above command creates a subdirectory named code that contains the code for the 10th post ("Heap Allocation").

Posts

The goal of this project is to provide step-by-step tutorials in individual blog posts. We currently have the following set of posts:

Bare Bones:

Interrupts:

Memory Management:

Multitasking:

Async/Await (source code)

First Edition Posts

The current version of the blog is already the second edition. The first edition is outdated and no longer maintained, but might still be useful. The posts of the first edition are:

Click to expand

Bare Bones:

Memory Management:

Exceptions:

Additional Resources:

License

This project, with exception of the blog/content folder, is licensed under either of

Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.

For licensing of the blog/content folder, see the blog/content/README.md.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

frdel/agent-zero

2024-09-16T01:28:33Z

Agent Zero AI framework

Agent Zero

Personal and organic AI framework

Agent Zero is not a predefined agentic framework. It is designed to be dynamic, organically growing, and learning as you use it.
Agent Zero is fully transparent, readable, comprehensible, customizable and interactive.
Agent Zero uses the computer as a tool to accomplish its (your) tasks.

Now with UI:

Key concepts

General-purpose assistant

Agent Zero is not pre-programmed for specific tasks (but can be). It is meant to be a general-purpose personal assistant. Give it a task, and it will gather information, execute commands and code, cooperate with other agent instances, and do its best to accomplish it.
It has a persistent memory, allowing it to memorize previous solutions, code, facts, instructions, etc., to solve tasks faster and more reliably in the future.

Computer as a tool

Agent Zero uses the operating system as a tool to accomplish its tasks. It has no single-purpose tools pre-programmed. Instead, it can write its own code and use the terminal to create and use its own tools as needed.
The only default tools in its arsenal are online search, memory features, communication (with the user and other agents), and code/terminal execution. Everything else is created by the agent itself or can be extended by the user.
Tool usage functionality has been developed from scratch to be the most compatible and reliable, even with very small models.

Multi-agent cooperation

Every agent has a superior agent giving it tasks and instructions. Every agent then reports back to its superior.
In the case of the first agent in the chain (Agent 0), the superior is the human user; the agent sees no difference.
Every agent can create its subordinate agent to help break down and solve subtasks. This helps all agents keep their context clean and focused.

Completely customizable and extensible

Almost nothing in this framework is hard-coded. Nothing is hidden. Everything can be extended or changed by the user.
The whole behavior is defined by a system prompt in the prompts/default/agent.system.md file. Change this prompt and change the framework dramatically.
The framework does not guide or limit the agent in any way. There are no hard-coded rails that agents have to follow.
Every prompt, every small message template sent to the agent in its communication loop, can be found in the prompts/ folder and changed.
Every default tool can be found in the python/tools/ folder and changed or copied to create new predefined tools.
Of course, it is open-source (except for some tools like Perplexity, but that will be replaced with an open-source alternative as well in the future).

Communication is key

Give your agent a proper system prompt and instructions, and it can do miracles.
Agents can communicate with their superiors and subordinates, asking questions, giving instructions, and providing guidance. Instruct your agents in the system prompt on how to communicate effectively.
The terminal interface is real-time streamed and interactive. You can stop and intervene at any point. If you see your agent heading in the wrong direction, just stop and tell it right away.
There is a lot of freedom in this framework. You can instruct your agents to regularly report back to superiors asking for permission to continue. You can instruct them to use point-scoring systems when deciding when to delegate subtasks. Superiors can double-check subordinates' results and dispute. The possibilities are endless.

Nice features to have

Output is very clean, colorful, readable and interactive; nothing is hidden.
The same colorful output you see in the terminal is automatically saved to HTML file in logs/ folder for every session.
Agent output is streamed in real-time, allowing the user to read along and intervene at any time.
No coding is required, only prompting and communication skills.
With a solid system prompt, the framework is reliable even with small models, including precise tool usage.

Keep in mind

Agent Zero can be dangerous! With proper instruction, Agent Zero is capable of many things, even potentially dangerous to your computer, data, or accounts. Always run Agent Zero in an isolated environment (like the built in docker container) and be careful what you wish for.
Agent Zero is not pre-programmed; it is prompt-based. The whole framework contains only a minimal amount of code and does not guide the agent in any way. Everything lies in the system prompt in the prompts/ folder. Here you can rewrite the whole framework behavior to your needs. If your agent fails to communicate properly, use tools, reason, use memory, find answers - just instruct it better.
If you cannot provide the ideal environment, let your agent know. Agent Zero is made to be used in an isolated virtual environment (for safety) with some tools preinstalled and configured. If you cannot provide all the necessary conditions or API keys, just change the system prompt and tell your agent what operating system and tools are at its disposal. Nothing is hard-coded; if you do not tell your agent about a certain tool, it will not know about it and will not try to use it.

Known problems

The system prompt sucks. You can do better. If you do, help me please :)
The communication between agent and terminal in docker container via SSH can sometimes break and stop producing outputs. Sometimes it is because the agent runs something like "server.serve_forever()" which causes the terminal to hang, sometimes a random error can occur. Restarting the agent and/or the docker container helps.
The agent can break his operating system. Sometimes the agent can deactivate virtual environment, uninstall packages, change config etc. Again, removing the docker container and cleaning up the work_dir/ is enough to fix that.

Ideal environment

Docker container: The perfect environment to run Agent Zero is the built-in docker container. The agent can download the image frdel/agent-zero-exe on its own and start the container, you only need to have docker running (like the Docker Desktop application).
Python: Python has to be installed on the system to run the framework.
Internet access: The agent will need internet access to use its online knowledge tool and execute commands and scripts requiring a connection. If you do not need your agent to be online, you can alter its prompts in the prompts/ folder and make it fully local.

Setup

A detailed setup guide with a video can be found here: /docs/installation. Scroll down to see the readme file.

⚠️ Changes to launch files since v0.6:
main.py file has been replaced with run_ui.py (webui) and run_cli.py (terminal) launch files. configuration has been moved to initialize.py for both webui and terminal launch files.

apache/iotdb

2024-09-16T01:28:33Z

Apache IoTDB

English | 中文

IoTDB

Overview

IoTDB (Internet of Things Database) is a data management system for time series data, which provides users with specific services, including data collection, storage and analysis. Due to its lightweight structure, high performance and usable features, together with its seamless integration with the Hadoop and Spark ecosystem, IoTDB meets the requirements of massive dataset storage, high throughput data input, and complex data analysis in the industrial IoT field.

Click for More Information

IoTDB depends on TsFile which is a columnar storage file format designed for time series data. The branch iotdb of TsFile project is used to deploy SNAPSHOT version for IoTDB project.

Main Features

The main features of IoTDB are as follows:

Flexible deployment strategy. IoTDB provides users with a one-click installation tool on either the cloud platform or the terminal devices, and a data synchronization tool bridging the data on cloud platform and terminals.
Low cost on hardware. IoTDB can reach a high compression ratio of disk storage.
Efficient directory structure. IoTDB supports efficient organization for complex time series data structures from intelligent networking devices, organization for time series data from devices of the same type, and fuzzy searching strategy for massive and complex directory of time series data.
High-throughput read and write. IoTDB supports millions of low-power devices' strong connection data access, high-speed data read and write for intelligent networking devices and mixed devices mentioned above.
Rich query semantics. IoTDB supports time alignment for time series data across devices and measurements, computation in time series field (frequency domain transformation) and rich aggregation function support in time dimension.
Easy to get started. IoTDB supports SQL-like language, JDBC standard API and import/export tools which are easy to use.
Seamless integration with state-of-the-practice Open Source Ecosystem. IoTDB supports analysis ecosystems, such as Hadoop and Spark, as well as visualization tools, such as Grafana.

For the latest information about IoTDB, please visit IoTDB official website. If you encounter any problems or identify any bugs while using IoTDB, please report an issue in Jira.

Outline

Quick Start

This short guide will walk you through the basic process of using IoTDB. For a more detailed introduction, please visit our website's User Guide.

Prerequisites

To use IoTDB, you need to have:

Java >= 1.8 (1.8, 11 to 17 are verified. Please make sure the environment path has been set accordingly).
Maven >= 3.6 (If you want to compile and install IoTDB from source code).
Set the max open files num as 65535 to avoid the "too many open files" error.

(Optional) Set the somaxconn as 65535 to avoid "connection reset" error when the system is under high load.

# Linux
> sudo sysctl -w net.core.somaxconn=65535

# FreeBSD or Darwin
> sudo sysctl -w kern.ipc.somaxconn=65535

Linux

(This guide is based on an installation of Ubuntu 22.04.)

Git

Make sure Git is installed, if it's missing, simply install it via:

sudo apt install git

Java

Make sure Java is installed, if it's missing, simply install it via:

sudo apt install default-jdk

Flex

sudo apt install flex

Bison

sudo apt install bison

Boost

sudo apt install libboost-all-dev

OpenSSL header files

Usually OpenSSL is already installed, however it's missing the header files we need to compile. So ensure these are installed:

sudo apt install libssl-dev

Mac OS

Git

First ensure git works.

Usually on a new Mac, as soon as you simply type git in a Terminal window, a popup will come up and ask if you want to finish installing the Mac developer tools. Just say yes. As soon as this is finished, you are free to use git.

Homebrew

Then install Homebrew - If this hasn't been installed yet, as we are going to be installing everything using Homebrew.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Java

As soon as that's done install Java, if this hasn't been installed yet:

brew install java

Depending on your version of Homebrew, it will tell you to do one of the following (depending on the type of processor in your device).

Mainly on the Intel-based models:

sudo ln -sfn /usr/local/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk

Mainly on the ARM-based models:

sudo ln -sfn /opt/homebrew/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk

CPP Prerequisites

Building Thrift requires us to add two more dependencies to the picture.

This however is only needed when enabling the with-cpp profile:

brew install boost
brew install bison
brew install openssl

Windows

Chocolatey

Then install Chocolatey - If this hasn't been installed yet, as we are going to be installing everything using Chocolatey.

https://chocolatey.org/install

Git

choco install git.install

Java

choco install openjdk

Visual Studio 19 2022

choco install visualstudio2022community
choco install visualstudio2022buildtools
choco install visualstudio2022-workload-nativedesktop

Flex / Bison

choco install winflexbison

Boost

choco install boost-msvc-14.2

OpenSSL

choco install openssl

Installation

IoTDB provides three installation methods, you can refer to the following suggestions, choose the one fits you best:

Installation from source code. If you need to modify the code yourself, you can use this method.
Installation from binary files. Download the binary files from the official website. This is the recommended method, in which you will get a binary released package which is out-of-the-box.
Using Docker：The path to the dockerfile is here.

Here in the Quick Start, we give a brief introduction of using source code to install IoTDB. For further information, please refer to User Guide.

Build from source

Prepare Thrift compiler

Skip this chapter if you are using Windows.

As we use Thrift for our RPC module (communication and protocol definition), we involve Thrift during the compilation, so Thrift compiler 0.13.0 (or higher) is required to generate Thrift Java code. Thrift officially provides binary compiler for Windows, but unfortunately, they do not provide that for Unix OSs.

If you have permission to install new software, use apt install or yum install or brew install to install the Thrift compiler. (If you already have installed the thrift compiler, skip this step.) Then, you may add the following parameter when running Maven: -Dthrift.download-url=http://apache.org/licenses/LICENSE-2.0.txt -Dthrift.exec.absolute.path=<YOUR LOCAL THRIFT BINARY FILE>.

If not, then you have to compile the thrift compiler, and it requires you install a boost library first. Therefore, we compiled a Unix compiler ourselves and put it onto GitHub, and with the help of a maven plugin, it will be downloaded automatically during compilation. This compiler works fine with gcc8 or later, Ubuntu MacOS, and CentOS, but previous versions and other OSs are not guaranteed.

If you can not download the thrift compiler automatically because of a network problem, you can download it by yourself, and then either: rename your thrift file to {project_root}\thrift\target\tools\thrift_0.12.0_0.13.0_linux.exe; or, add Maven commands: -Dthrift.download-url=http://apache.org/licenses/LICENSE-2.0.txt -Dthrift.exec.absolute.path=<YOUR LOCAL THRIFT BINARY FILE>.

Compile IoTDB

You can download the source code from:

git clone https://github.com/apache/iotdb.git

The default dev branch is the master branch, if you want to use a released version x.x.x:

git checkout vx.x.x

Or checkout to the branch of a big version, e.g., the branch of 1.0 is rel/1.0.

git checkout rel/x.x

Build IoTDB from source

Under the root path of iotdb:

> mvn clean package -pl distribution -am -DskipTests

After being built, the IoTDB distribution is located at the folder: "distribution/target".

Only build cli

Under the iotdb/iotdb-client path:

> mvn clean package -pl cli -am -DskipTests

After being built, the IoTDB cli is located at the folder "cli/target".

Build Others

Use -P with-cpp for compiling the cpp client. (For more details, read client-cpp's Readme file.)

NOTE: Directories "thrift/target/generated-sources/thrift", "thrift-sync/target/generated-sources/thrift", "thrift-cluster/target/generated-sources/thrift", "thrift-influxdb/target/generated-sources/thrift" and "antlr/target/generated-sources/antlr4" need to be added to sources roots to avoid compilation errors in the IDE.

In IDEA, you just need to right click on the root project name and choose "Maven->Reload Project" after you run mvn package successfully.

Configurations

Configuration files are under the "conf" folder.

environment config module (datanode-env.bat, datanode-env.sh),
system config module (iotdb-datanode.properties)
log config module (logback.xml).

For more information, please see Config Manual.

Start

You can go through the following steps to test the installation. If there is no error returned after execution, the installation is completed.

Start IoTDB

Users can start 1C1D IoTDB by the start-standalone script under the sbin folder.

# Unix/OS X
> sbin/start-standalone.sh

# Windows
> sbin\start-standalone.bat

Use IoTDB

Use Cli

IoTDB offers different ways to interact with server, here we introduce the basic steps of using Cli tool to insert and query data.

After installing IoTDB, there is a default user 'root', its default password is also 'root'. Users can use this default user to login Cli to use IoTDB. The start-up script of Cli is the start-cli script in the folder sbin. When executing the script, user should assign IP, PORT, USER_NAME and PASSWORD. The default parameters are "-h 127.0.0.1 -p 6667 -u root -pw -root".

Here is the command for starting the Cli:

# Unix/OS X
> sbin/start-cli.sh -h 127.0.0.1 -p 6667 -u root -pw root

# Windows
> sbin\start-cli.bat -h 127.0.0.1 -p 6667 -u root -pw root

The command line cli is interactive, so you should see the welcome logo and statements if everything is ready:

 _____       _________  ______   ______
|_   _|     |  _   _  ||_   _ `.|_   _ \
  | |   .--.|_/ | | \_|  | | `. \ | |_) |
  | | / .'`\ \  | |      | |  | | |  __'.
 _| |_| \__. | _| |_    _| |_.' /_| |__) |
|_____|'.__.' |_____|  |______.'|_______/  version x.x.x


IoTDB> login successfully
IoTDB>

Basic commands for IoTDB

Now, let us introduce the way of creating timeseries, inserting data and querying data.

The data in IoTDB is organized as timeseries. Each timeseries includes multiple data–time pairs, and is owned by a database. Before defining a timeseries, we should define a database using CREATE DATABASE first, and here is an example:

IoTDB> CREATE DATABASE root.ln

We can also use SHOW DATABASES to check the database being created:

IoTDB> SHOW DATABASES
+-------------+
|     Database|
+-------------+
|      root.ln|
+-------------+
Total line number = 1

After the database is set, we can use CREATE TIMESERIES to create a new timeseries. When creating a timeseries, we should define its data type and the encoding scheme. Here we create two timeseries:

IoTDB> CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN, ENCODING=PLAIN
IoTDB> CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=FLOAT, ENCODING=RLE

In order to query the specific timeseries, we can use SHOW TIMESERIES . represent the location of the timeseries. The default value is "null", which queries all the timeseries in the system (the same as using "SHOW TIMESERIES root"). Here are some examples:

Querying all timeseries in the system:

IoTDB> SHOW TIMESERIES
+-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
|                   Timeseries|Alias|Database|DataType|Encoding|Compression|Tags|Attributes|
+-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
|root.ln.wf01.wt01.temperature| null|      root.ln|   FLOAT|     RLE|     SNAPPY|null|      null|
|     root.ln.wf01.wt01.status| null|      root.ln| BOOLEAN|   PLAIN|     SNAPPY|null|      null|
+-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
Total line number = 2

Querying a specific timeseries (root.ln.wf01.wt01.status):

IoTDB> SHOW TIMESERIES root.ln.wf01.wt01.status
+------------------------+-----+-------------+--------+--------+-----------+----+----------+
|              timeseries|alias|database|dataType|encoding|compression|tags|attributes|
+------------------------+-----+-------------+--------+--------+-----------+----+----------+
|root.ln.wf01.wt01.status| null|      root.ln| BOOLEAN|   PLAIN|     SNAPPY|null|      null|
+------------------------+-----+-------------+--------+--------+-----------+----+----------+
Total line number = 1

Inserting timeseries data is a basic operation of IoTDB, you can use the ‘INSERT’ command to finish this. Before insertion, you should assign the timestamp and the suffix path name:

IoTDB> INSERT INTO root.ln.wf01.wt01(timestamp,status) values(100,true);
IoTDB> INSERT INTO root.ln.wf01.wt01(timestamp,status,temperature) values(200,false,20.71)

The data that you have just inserted will be displayed as follows:

IoTDB> SELECT status FROM root.ln.wf01.wt01
+------------------------+------------------------+
|                    Time|root.ln.wf01.wt01.status|
+------------------------+------------------------+
|1970-01-01T00:00:00.100Z|                    true|
|1970-01-01T00:00:00.200Z|                   false|
+------------------------+------------------------+
Total line number = 2

You can also query several timeseries data using one SQL statement:

IoTDB> SELECT * FROM root.ln.wf01.wt01
+------------------------+-----------------------------+------------------------+
|                    Time|root.ln.wf01.wt01.temperature|root.ln.wf01.wt01.status|
+------------------------+-----------------------------+------------------------+
|1970-01-01T00:00:00.100Z|                         null|                    true|
|1970-01-01T00:00:00.200Z|                        20.71|                   false|
+------------------------+-----------------------------+------------------------+
Total line number = 2

To change the time zone in Cli, you can use the following SQL:

IoTDB> SET time_zone=+08:00
Time zone has set to +08:00
IoTDB> SHOW time_zone
Current time zone: Asia/Shanghai

Add then the query result will show using the new time zone.

IoTDB> SELECT * FROM root.ln.wf01.wt01
+-----------------------------+-----------------------------+------------------------+
|                         Time|root.ln.wf01.wt01.temperature|root.ln.wf01.wt01.status|
+-----------------------------+-----------------------------+------------------------+
|1970-01-01T08:00:00.100+08:00|                         null|                    true|
|1970-01-01T08:00:00.200+08:00|                        20.71|                   false|
+-----------------------------+-----------------------------+------------------------+
Total line number = 2

The commands to exit the Cli are:

IoTDB> quit
or
IoTDB> exit

For more information about the commands supported by IoTDB SQL, please see User Guide.

Stop IoTDB

The server can be stopped with "ctrl-C" or the following script:

# Unix/OS X
> sbin/stop-standalone.sh

# Windows
> sbin\stop-standalone.bat

The use of CSV Import and Export Tool

see The use of CSV Import and Export Tool

Frequent Questions for Compiling

see Frequent Questions when Compiling the Source Code

Contact Us

QQ Group

Apache IoTDB User Group: 659990460

Wechat Group

Add friend: tietouqiao or liutaohua001, and then we'll invite you to the group.

Slack

Slack channel

see Join the community for more!

mrousavy/nitro

2024-09-17T01:25:30Z

🔥 Insanely fast native C++, Swift or Kotlin modules with a statically compiled binding layer to JSI

Nitro Modules are highly efficient native modules with a statically compiled binding layer to JSI, and typesafe JS bindings. It consists of two parts:

react-native-nitro-modules: The core C++ library powering all nitro modules
nitrogen: A code-generator for nitro module library authors

Installation

Install react-native-nitro-modules from npm:

npm i react-native-nitro-modules
cd ios && pod install

Documentation

Supported Platforms

Cross-platform

Cross-platform native modules can be built with C++. Any custom C++ types can be used and bridged to JS with minimal overhead.

JS <-> C++ type converters are statically generated ahead of time - no more dynamic lookups or runtime parser errors! 🥳

iOS

iOS native modules and view components can be written either in pure C++, or pure Swift. Thanks to Swift 5.9, Swift Nitro Modules bridge directly to C++ instead of going through Objective-C message sends. Woohoo, no more Objective-C, and zero overhead C++ -> Swift calls! 🥳

Android

Android native modules and view components can be written either in pure C++, or pure Kotlin/Java. Thanks to fbjni, even complex types can be effortlessly bridged to Kotlin/Java with minimal overhead! 🔥

Contributing

See the contributing guide to learn how to contribute to the repository and the development workflow.

License

MIT

clj-commons/kibit

2024-09-17T01:25:51Z

There's a function for that!

truenas/py-libzfs

2024-09-16T01:29:58Z

Python libzfs bindings

zhad3/zrenderer

2024-09-16T01:30:00Z

A renderer for Ragnarok Online sprites

tomsik68/docker-xampp

2024-09-16T01:30:10Z

Dockerfile to build an image containing XAMPP(MySQL + PHP + PHPMyAdmin) running on Debian system with SSH server

PHP version	Corresponding tag
8.2.4	`tomsik68/xampp:8`
7.4.33	`tomsik68/xampp:7`
5.6.40	`tomsik68/xampp:5`

For PHP 8, start your container like this:

docker run --name myXampp -p 41061:22 -p 41062:80 -d -v ~/my_web_pages:/www tomsik68/xampp:8

docker-xampp

This image is intended for PHP+MySQL development. For convenience, it also runs SSH server to connect to. Both MySQL and phpmyadmin use default XAMPP password.

Any questions?

Please read this readme before asking anything.

If you didn't find answer to your question, create an issue or just ask on Gitter. I'll try to help if I can!

Running the image

This image uses /www directory for your page files, so you need to mount it.

docker run --name myXampp -p 41061:22 -p 41062:80 -d -v ~/my_web_pages:/www tomsik68/xampp

The command above will expose the SSH server on port 41061 and HTTP server on port 41062. Feel free to use your own name for the container.

To browse to your web page, visit this URL: http://localhost:41062/www And to open up the XAMPP interface: http://localhost:41062/

Default credentials

service	username	password
ssh	root	root

Additional How-tos

My app can't function in `/www` folder

No problem, just mount your app in the /opt/lampp/htdocs folder, for example:

docker run --name myXampp -p 41061:22 -p 41062:80 -d -v ~/my_web_pages:/opt/lampp/htdocs tomsik68/xampp

ssh connection

default SSH password is 'root'.

ssh root@localhost -p 41061

get a shell terminal inside your container

docker exec -ti myXampp bash

use binaries provided by XAMPP

Inside docker container:

export PATH=/opt/lampp/bin:$PATH

You can then use mysql and friends installed in /opt/lampp/bin in your current bash session. If you want this to persist, you need to add it to your user or system-wide .bashrc (inside container).

Use your own configuration

In your home directory, create a my_apache_conf directory in which you place any number of apache configuration files. As soon as they end with the .conf extension, they will be used by xampp. Make sure to use the following flag in your command: -v ~/my_apache_conf:/opt/lampp/apache2/conf.d, for example:

docker run --name myXampp -p 41061:22 -p 41062:80 -d -v ~/my_web_pages:/www  -v ~/my_apache_conf:/opt/lampp/apache2/conf.d tomsik68/xampp

Restart the server

Once you have modified configuration for example

docker exec myXampp /opt/lampp/lampp restart

phpmyadmin

If you used the flag -p 41062:80 with docker run, just browse to http://localhost:41062/phpmyadmin/ .

Docker compose example

version: '3.8'

services:
  xampp:
    image: tomsik68/xampp
    ports:
      - "44162:22"
      - "41063:80"
    volumes:
      # Mount your web pages
      - ./xampp/my_pages:/www
      # Mount your apache configuration
      - ./xampp/my_apache_conf/:/opt/lampp/apache2/conf.d
      #Save MySQL data to be persistent
      # add the dbs as needed
      - ./xampp/mysql/mydb:/opt/lampp/var/mysql/mydb
    restart: always

Use a Different XAMPP or PHP Version

Currently, the Docker image is built only for PHP 5, 7 and 8. If you need another version, you can easily build a Docker image yourself, here's how:

Clone this repo (local clone is sufficient, no need to fork)
Find the URL to a URL to your desired version. List of versions can be found here: https://sourceforge.net/projects/xampp/files/XAMPP%20Linux/
Build the image using e.g. docker build --build-arg XAMPP_URL="https://www.apachefriends.org/xampp-files/5.6.40/xampp-linux-x64-5.6.40-1-installer.run?from_af=true" .

OPM/opm-tests

2024-09-16T01:30:18Z

Data sets intended for integration and regression testing reservoir simulations

edance/openpace

2024-09-17T01:26:44Z

🏃 Track Runs and Workouts

OpenRCT2/objects

2024-09-16T01:30:30Z

Official objects for OpenRCT2

Contains objects from RollerCoaster Tycoon 1, RollerCoaster Tycoon 2 and their expansion packs. For the main development and codebase of OpenRCT2, visit OpenRCT2/OpenRCT2.

Build Status

gdquest-demos/godot-shaders

2024-09-17T01:27:06Z

A large library of free and open-source shaders for the Godot game engine. Here, you'll get 2D and 3D shaders with playable demos.

golang/example

2024-09-16T01:30:52Z

Go example projects

This repository contains a collection of Go programs and libraries that demonstrate the language, standard libraries, and tools.

Clone the project

$ git clone https://go.googlesource.com/example
$ cd example

https://go.googlesource.com/example is the canonical Git repository. It is mirrored at https://github.com/golang/example.

hello and hello/reverse

$ cd hello
$ go build
$ ./hello -help

A trivial "Hello, world" program that uses a library package.

The hello command covers:

The basic form of an executable command
Importing packages (from the standard library and the local repository)
Printing strings (fmt)
Command-line flags (flag)
Logging (log)

The reverse reverse covers:

The basic form of a library
Conversion between string and []rune
Table-driven unit tests (testing)

helloserver

$ cd helloserver
$ go run .

A trivial "Hello, world" web server.

Topics covered:

Command-line flags (flag)
Logging (log)
Web servers (net/http)

outyet

$ cd outyet
$ go run .

A web server that answers the question: "Is Go 1.x out yet?"

Topics covered:

Command-line flags (flag)
Web servers (net/http)
HTML Templates (html/template)
Logging (log)
Long-running background processes
Synchronizing data access between goroutines (sync)
Exporting server state for monitoring (expvar)
Unit and integration tests (testing)
Dependency injection
Time (time)

appengine-hello

A trivial "Hello, world" App Engine application intended to be used as the starting point for your own code. Please see Google App Engine SDK for Go and Quickstart for Go in the App Engine Standard Environment.

gotypes

The go/types package is a type-checker for Go programs. It is one of the most complex packages in Go's standard library, so we have provided this tutorial to help you find your bearings. It comes with several example programs that you can obtain using go get and play with as you learn to build tools that analyze or manipulate Go programs.

template

A trivial web server that demonstrates the use of the template package's block feature.

slog-handler-guide

The log/slog package supports structured logging. It features a flexible backend in the form of a Handler interface. This guide can help you write your own handler.

drupal-composer/drupal-scaffold

2024-09-16T01:33:38Z

🚧 Composer Plugin for updating the Drupal scaffold files when using drupal/core

drupal-scaffold

DEPRECATED - DO NOT USE

This project is deprecated. Please use drupal/core-composer-scaffold.

If you are still using this plugin because you are on Drupal 8.7.x or earlier, please note that it is possible to use core-composer-scaffold with older versions of Drupal 8. Start with the template project drupal/recommended-project, and then pin to an earlier version of drupal/core-recommended after the initial create-project is done. Note that it's generally easier to remove composer.lock and vendor first when moving back to older versions of Drupal core.

OLD README FOLLOWS

Composer plugin for automatically downloading Drupal scaffold files (like index.php, update.php, …) when using drupal/core via Composer.

It is recommended that the vendor directory be placed in its standard location at the project root, outside of the Drupal root; however, the location of the vendor directory and the name of the Drupal root may be placed in whatever location suits the project. Drupal-scaffold will generate the autoload.php file at the Drupal root to require the Composer-generated autoload file in the vendor directory.

Usage

Run composer require drupal-composer/drupal-scaffold:dev-master in your composer project before installing or updating drupal/core.

Once drupal-scaffold is required by your project, it will automatically update your scaffold files whenever composer update changes the version of drupal/core installed.

Configuration

You can configure the plugin with providing some settings in the extra section of your root composer.json.

{
  "extra": {
    "drupal-scaffold": {
      "source": "https://git.drupalcode.org/project/drupal/raw/{version}/{path}",
      "excludes": [
        "google123.html",
        "robots.txt"
      ],
      "includes": [
        "sites/default/example.settings.my.php"
      ],
      "initial": {
        "sites/default/default.services.yml": "sites/default/services.yml",
        "sites/default/default.settings.php": "sites/default/settings.php"
      },
      "omit-defaults": false
    }
  }
}

The source option may be used to specify the URL to download the scaffold files from; the default source is drupal.org. The literal string {version} in the source option is replaced with the current version of Drupal core being updated prior to download. You can also define source as an array to have fallbacks in case of any HTTP issues.

{
  "extra": {
    "drupal-scaffold": {
      "source": [
        "https://git.drupalcode.org/project/drupal/raw/{version}/{path}",
        "https://raw.githubusercontent.com/drupal/drupal/{version}/{path}"
      ]
    }
  }
}

With the drupal-scaffold option excludes, you can provide additional paths that should not be copied or overwritten. The plugin provides no excludes by default.

Default includes are provided by the plugin:

.csslintrc
.editorconfig
.eslintignore
.eslintrc (Drupal <= 8.2.x)
.eslintrc.json (Drupal >= 8.3.x)
.gitattributes
.ht.router.php (Drupal >= 8.5.x)
.htaccess
index.php
robots.txt
sites/default/default.settings.php
sites/default/default.services.yml
sites/development.services.yml
sites/example.settings.local.php
sites/example.sites.php
update.php
web.config

When setting omit-defaults to true, neither the default excludes nor the default includes will be provided; in this instance, only those files explicitly listed in the excludes and includes options will be considered. If omit-defaults is false (the default), then any items listed in excludes or includes will be in addition to the usual defaults.

The initial hash lists files that should be copied over only if they do not exist in the destination. The key specifies the path to the source file, and the value indicates the path to the destination file.

Limitation

When using Composer to install or update the Drupal development branch, the scaffold files are always taken from the HEAD of the branch (or, more specifically, from the most recent development .tar.gz archive). This might not be what you want when using an old development version (e.g. when the version is fixed via composer.lock). To avoid problems, always commit your scaffold files to the repository any time that composer.lock is committed. Note that the correct scaffold files are retrieved when using a tagged release of drupal/core (recommended).

Custom command

The plugin by default is only downloading the scaffold files when installing or updating drupal/core. If you want to call it manually, you have to add the command callback to the scripts-section of your root composer.json, like this:

{
  "scripts": {
    "drupal-scaffold": "DrupalComposer\\DrupalScaffold\\Plugin::scaffold"
  }
}

After that you can manually download the scaffold files according to your configuration by using composer drupal:scaffold.

It is assumed that the scaffold files will be committed to the repository, to ensure that the correct files are used on the CI server (see Limitation, above). After running composer install for the first time commit the scaffold files to your repository.

hechoendrupal/drupal-console-extend-plugin

2024-09-16T01:33:38Z

Drupal Console Extend Composer Plugin

Drupal Console Extend Plugin

Composer plugin to discover Drupal Console commands using a standard package/library.

Install this project:

composer require drupal/console-extend-plugin

timescale/pgai

2024-09-16T01:33:45Z

Bring AI models closer to your PostgreSQL data

frdel/agent-zero

2024-09-16T01:34:02Z

A python parametric CAD scripting framework based on OCCT

CadQuery

Quick Links

Documentation | Cheatsheet | Discord | Google Group | GUI Editor

What is CadQuery

CadQuery is an intuitive, easy-to-use Python module for building parametric 3D CAD models. Using CadQuery, you can write short, simple scripts that produce high quality CAD models. It is easy to make many different objects using a single script that can be customized.

CadQuery is often compared to OpenSCAD. Like OpenSCAD, CadQuery is an open-source, script based, parametric model generator. However, CadQuery stands out in many ways and has several key advantages:

The scripts use a standard programming language, Python, and thus can benefit from the associated infrastructure. This includes many standard libraries and IDEs.
CadQuery's CAD kernel Open CASCADE Technology (OCCT) is much more powerful than the CGAL used by OpenSCAD. Features supported natively by OCCT include NURBS, splines, surface sewing, STL repair, STEP import/export, and other complex operations, in addition to the standard CSG operations supported by CGAL
Ability to import/export STEP and the ability to begin with a STEP model, created in a CAD package, and then add parametric features. This is possible in OpenSCAD using STL, but STL is a lossy format.
CadQuery scripts require less code to create most objects, because it is possible to locate features based on the position of other features, workplanes, vertices, etc.
CadQuery scripts can build STL, STEP, AMF and 3MF faster than OpenSCAD.

CadQuery was built to be used as a Python library without any GUI. This makes it great for use cases such as integration into servers, or creating scientific and engineering scripts. Options for visualization are also available including CQ-Editor and JupyterLab.

For those who are interested, the OCP repository contains the current OCCT wrapper used by CQ.

Key features

Build 3D models with scripts that are as close as possible to how you would describe the object to a human.
Create parametric models that can be very easily customized by end users.
Output high quality (loss-less) CAD formats like STEP and DXF in addition to STL, VRML, AMF and 3MF.
Provide a non-proprietary, plain text model format that can be edited and executed with only a web browser.
Offer advanced modeling capabilities such as fillets, curvilinear extrudes, parametric curves and lofts.
Build nested assemblies out of individual parts and other assemblies.

Why this fork

The original version of CadQuery was built on the FreeCAD API. This was great because it allowed for fast development and easy cross-platform capability. However, we eventually started reaching the limits of the API for some advanced operations and selectors. This 2.0 version of CadQuery is based directly on a Python wrapper of the OCCT kernel. This gives us a great deal more control and flexibility, at the expense of some simplicity and having to handle the cross-platform aspects of deployment ourselves. We believe this is a worthwhile trade-off to allow CadQuery to continue to grow and expand in the future.

Getting started

To learn more about designing with CadQuery, visit the documentation, examples, and cheatsheet.

To get started playing around with CadQuery and see its capabilities, take a look at the CQ-editor GUI. This easy-to-use IDE is a great way to get started desiging with CadQuery. The CQ-editor installer bundles both CQ-editor and CadQuery (recommended). See the CQ-editor installation instructions.

The CadQuery library (with or without CQ-editor) and its dependencies may be installed using conda, or pip. Note that conda (or the CQ-editor installer) is the better supported option.

See the documentation for detailed CadQuery installation instructions.

There are also videos covering installation:

Linux installation video
Windows installation video

CadQuery Installation Via Conda

To first install the Conda package manager see Install the Conda Package Manager, and Mambaforge for a minimal installer.

mamba install is recommended over conda install for faster and less memory intensive cadquery installation.

# Set up a new environment
conda create -n cadquery

# Activate the new environment
conda activate cadquery

# Install the latest released version
mamba install -c conda-forge cadquery

# Or install the dev version to get the latest changes
mamba install -c conda-forge -c cadquery cadquery=master

CadQuery Installation Via Pip

CadQuery has a complex set of dependencies including OCP, which is our set of bindings to the OpenCASCADE CAD kernel. OCP is distributed as binary wheels for Linux, MacOS and Windows. However, there are some limitations. Only Python 3.9 through 3.12 are currently supported, and some older Linux distributions such as Ubuntu 18.04 are not supported. If the pip installation method does not work for your system, you can try the conda installation method.

It is highly recommended that a virtual environment is used when installing CadQuery, although it is not strictly required. Installing CadQuery via pip requires a up-to-date version of pip, which can be obtained with the following command line (or a slight variation thereof).

python3 -m pip install --upgrade pip

Once a current version of pip is installed, CadQuery can be installed using the following command line.

pip install cadquery

It is also possible to install the very latest changes directly from CadQuery's GitHub repository, with the understanding that sometimes breaking changes can occur. To install from the git repository, run the following command line.

pip install git+https://github.com/CadQuery/cadquery.git

CQ-editor GUI

CQ-editor is an IDE that allows users to edit CadQuery model scripts in a GUI environment. It includes features such as:

A graphical debugger that allows you to step through your scripts.
A CadQuery stack inspector.
Export to various formats, including STEP and STL, directly from the menu.

Jupyter

CadQuery supports Jupyter out-of-the-box. Run your CadQuery code in the notebook and visualize the model by calling display(<CadQuery object>).

JupyterLab installation.

Getting help

You can find the full CadQuery documentation at cadquery.readthedocs.io.

We also have a Google Group to make it easy to get help from other CadQuery users. We want you to feel welcome and encourage you to join the group and introduce yourself. We would also love to hear what you are doing with CadQuery.

There is a Discord server as well. You can ask for help in the general channel.

Projects using CadQuery

Here are just a few examples of how CadQuery is being used.

FxBricks Lego Train System

FxBricks uses CadQuery in the product development pipeline for their Lego train system. FxBricks has also given back to the community by creating documentation for their CAD pipeline. They have also assembled cq-kit, a library containing utility classes and functions to extend the capabilities of CadQuery. Thanks to @michaelgale and @fx-bricks for this example.

Hexidor Board Game Development

Hexidor is an expanded take on the Quoridor board game, and the development process has been chronicled here. CadQuery was used to generate the game board. Thanks to Bruce for this example.

Spindle assembly

Thanks to @marcus7070 for this example from here.

3D Printed Resin Mold

Thanks to @eddieliberato for sharing this example of an anti-kink resin mold for a cable.

License

CadQuery is licensed under the terms of the Apache Public License, version 2.0.

Contributing

Contributions from the community are welcome and appreciated.

You do not need to be a software developer to have a big impact on this project. Contributions can take many forms including, but not limited to, the following:

Writing and improving documentation
Triaging bugs
Submitting bugs and feature requests
Creating tutorial videos and blog posts
Helping other users get started and solve problems
Telling others about this project
Helping with translations and internationalization
Helping with accessibility
Contributing bug fixes and new features

It is asked that all contributions to this project be made in a respectful and considerate way. Please use the Python Community Code of Conduct's guidelines as a reference.

Contributing code

If you are going to contribute code, make sure to follow this steps:

Consider opening an issue first to discuss what you have in mind
Try to keep it as short and simple as possible (if you want to change several things, start with just one!)
Fork the CadQuery repository, clone your fork and create a new branch to start working on your changes
Create a conda development environment with something like:
- mamba env create -n cq-dev -f environment.yml
Activate the new conda environment:
- conda activate cq-dev
If desired, install the master branch of cq-editor (Note; a release version may not be compatible with the master branch of cadquery):
- mamba install -c cadquery -c conda-forge cq-editor=master Installing cq-editor adds another instance of cadquery which overrides the clone just added. Fix this by reinstalling cadquery using pip:
- pip install -e .
Before making any changes verify that the current tests pass. Run pytest from the root of your cadquery clone, there should be no failures and the output will look similar to this:
- ======= 215 passed, 57 warnings in 13.95s =======
Start with the tests! How should CadQuery behave after your changes? Make sure to add some tests to the test suite to ensure proper behavior
Make sure your tests have assertions checking all the expected results
Add a nice docstring to the test indicating what the test is doing; if there is too much to explain, consider splitting the test in two!
Go ahead and implement the changes
Add a nice docstring to the functions/methods/classes you implement describing what they do, what the expected parameters are and what it returns (if anything)
Update the documentation if there is any change to the public API
Consider adding an example to the documentation showing your cool new feature!
Make sure nothing is broken (run the complete test suite with pytest)
Run black from our fork to autoformat your code and make sure your code style complies with CadQuery's
Push the changes to your fork and open a pull-request upstream
Keep an eye on the automated feedback you will receive from the CI pipelines; if there is a test failing or some code is not properly formatted, you will be notified without human intervention
Be prepared for constructive feedback and criticism!
Be patient and respectful, remember that those reviewing your code are also working hard (sometimes reviewing changes is harder than implementing them!)

How to Report a Bug

When filing a bug report issue, please be sure to answer these questions:

What version of the software are you running?
What operating system are you running the software on?
What are the steps to reproduce the bug?

How to Suggest a Feature or Enhancement

If you find yourself wishing for a feature that does not exist, you are probably not alone. There are bound to be others out there with similar needs. Open an issue which describes the feature you would like to see, why you need it, and how it should work.

ai-to-ai/Auto-Gmail-Creator

2024-09-17T01:35:23Z

Open Source Bulk Auto Gmail Creator Bot with Selenium & Seleniumwire ( Python ). Feel free to contact me with Django/Flask, ML, AI, GPT, Automation, Scraping.

Auto-Gmail-Creator

Summary

Latest Open Source Bulk Auto Google Account ( Gmail ) Regiteration Bot Script 2023

Important Notice for already cloned/ forked users

I developed stealth webdriver that avoid detection during the account creation. You can contact me to get it by giving me a coffee. btw, The only thing remain is to find cheap, robust sms providers as well as proxies.

Introduction

This is the automation script for Python lovers to learn basics about automation as well as commercial for marketers. Fork me on GitHub

According to Jonathan's desription ,only about five gmail addresses can be verified on a single phone number.

This script uses sms-activate.org api for phone verification and more services will be added.

To run the script, you don't need to download Chromedriver or Geckodriver manually. The script does it automatically with webdriver manager.

You can customize this script to avoid getting blocked. I am also developing another script with Requests but going to keep it private since it will be blocked if I let it public.

Also, Feel free to contact me if you have any project regarding Automation, Scraping, Machine Learning.

Usage

Install Python 3.x.
pip install -r requirements.txt
python app.py
'Created.txt' will be generated for successful creation.
if you already installed python packages for my previous script, I recommend to upgrade the webdriver-manager package by pip install webdriver-manager -U

Customize

Browser [ Chrome, Firefox ]

Switch from Chrome to Firefox by commenting 2 lines.

#options = ChromeOptions()
options = FirefoxOptions()

#driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options = options, seleniumwire_options=seleniumwire_options)

driver = webdriver.Firefox(service=Service(GeckoDriverManager().install()), options = options, seleniumwire_options=seleniumwire_options)

Manual or Automatic User info generation

You edit the 'User.csv' with given type such as First name, Last name, Password, Birthday, Username(optional) from the second line. If the 5th parameter on user.csv is not passed by userBot generates username automatically adding FN + dot + LN + random 5 digits.(john.doe12345@gmail.com) Thanks to BourneXu, Script generates random popular usernames. You can set this variant as "True" to use this functionality to automate generation.

AUTO_GENERATE_UERINFO = True

Proxy

If you want to use socks proxy, please remove comment theses lines. Free Proxy list is here

    SOCKS_PROXY = "socks5://user:pass@ip:port"

Headless or With UI (Optional)

    options.add_argument('--headless')

Profile (Optional)

You can add your own profile if you want by specifying the path.

    options.add_argument("--incognito")
    options.add_argument(r"--user-data-dir=C:\\Users\\Username\\AppData\\Local\\Google\\Chrome\\User Data")
    options.add_argument(r'--profile-directory=ProfileName')

To-Do

Try to simulate user's mouse action wity pyautogui and adding cookie, recovery email, more sms services such as Durian, 5sims. To bypass bot-detection, I am all ears to hear from you.

Images

Running
Edit user.csv

With Notepad

With Excel
Bot will create chrome browser repeatedly for each gmail.
You can visit sms-activate.org to see it's apis.
To see the country code, you can hit here.

Email

tr.soft.engineer@gmail.com

https://t.me/leoshabit

Discord

leoshabit

Skype

https://join.skype.com/invite/H6S0RFA69GNK

Github

https://github.com/ai-to-ai

Phone

+13035365033

ropensci/rix

2024-09-16T01:34:13Z

Reproducible development environments for R with Nix

rix: Reproducible Environments with Nix

Reproducible Environments with Nix

Introduction

{rix} is an R package that leverages Nix, a powerful package manager focusing on reproducible builds. With Nix, it is possible to create project-specific environments that contain a project-specific version of R and R packages (as well as other tools or languages, if needed). This project-specific environment will also include all the required system-level dependencies that can be difficult to install, such as GDAL for packages for geospatial analysis for example. Nix installs software as a complete “bundle” that include all of the software’s dependencies, and all of the dependencies’ dependencies and so on. Nix is an incredibly useful piece of software for ensuring reproducibility of projects, in research or otherwise.

Some other use cases include, for example, running web applications like Shiny apps or {plumber} APIs in a controlled environment, or executing {targets} pipelines with the right version of R and dependencies, or use environments managed by Nix to work interactively using an IDE.

In essence, this means that you can use {rix} and Nix to replace {renv} and Docker with one single tool, but the approach is quite different: {renv} records specific versions of individual packages, while {rix} provides a complete snapshot of the R ecosystem at a specific point in time, but also snapshots all the required dependencies to make your project-specific R environment work. In contrast, to ensure complete reproducibility with {renv}, it must be combined with Docker, in order to include system-level dependencies (like GDAL, as per the example above).

Nix has a fairly steep learning curve though. Nix is a complex piece of software that comes with its own programming language, which is also called Nix. Its purpose is to solve a complex problem: defining instructions on how to build software packages and manage configurations in a declarative way, using functional programming principles. This makes sure that software gets installed in a fully reproducible manner, on any operating system or hardware, but with the caveat that users must learn the Nix programming language and get into the “functional programming approach to software management” mindset, which is unusual.

{rix} provides functions to help you write Nix expressions (written in the Nix language). These expressions will be the inputs for the Nix package manager, to build sets of software packages and provide them in a reproducible development environment. These environments can be used for interactive data analysis, or reproduced when running pipelines in CI/CD systems. The Nixpkgs collection includes currently more than 100.000 pieces of software available through the Nix package manager.

With {rix}, you can define development environments, or shells, that contain the required tools needed to analyze data using R. These environments are isolated from each other and project-specific: this means that a project can use one version of R and R packages, and another environment another version of R and R packages. However, extra care is required if you already have R installed through the usual method for your operating system, as these development environments are not totally isolated from the rest of your system. Unlike Docker, where a running container cannot acces anything from the host system, unless explicitely configured to do so, Nix development shells are nothing but environments that add more software to the list of already available software (the so-called PATH). As such, it is possible to access anything (files and software) already present on the system from a running Nix shell. Thus, {rix} also provides a function called rix_init() that helps isolate R sessions running inside Nix environments from the rest of your system. This avoids clashes between the Nix-specific library of R packages and the user library of R packages should you already have R installed and managed by the usual method for your operating system.

It is also possible to add any other software package available on Nixpkgs to a Nix environment, for example IDEs such as RStudio or VS Code. The Nix R ecosystem currently includes almost the entirety of CRAN and Bioconductor packages (there is around a hundred CRAN or Biocondcuctor packages that are unavailable through Nix). Like with any other programming language or software, it is also possible to install older releases of R packages, or install packages from GitHub at defined states, as well as local packages in the .tar.gz format.

The Nix package manager is extremely powerful. Not only does it handle all the dependencies of any package extremely well in a deterministic manner, it is also possible with it to reproduce environments containing old releases of software. It is thus possible to build environments containing R version 4.0.0 (for example) to run an old project that was originally developed on that version of R.

If you need other tools or languages like Python or Julia, this can also be done easily. Nix is available for Linux, macOS and Windows (via WSL2) and {rix} comes with the following features:

define complete development environments as code and use them anywhere;
install project-specific complete R environments, which can be different from each other;
run single R functions (and objects in the call stack) in a different environment (potentially with a different R version and R packages) for an interactive R session, and get back the output of that function using with_nix();

{rix} does not require Nix to be installed on your system to generate expressions. This means that you can generate expressions on a system on which you cannot easily install software, and then use these expressions on the cloud or on a CI/CD environment to build the project there.

If you have R installed, you can start straight away from your R session by first installing {rix}:

install.packages("rix", repos = c(
  "https://ropensci.r-universe.dev",
  "https://cloud.r-project.org"
))

library("rix")

Now try to build an expression using rix():

library(rix)

path_default_nix <- "."

rix(
  r_ver = "4.3.3",
  r_pkgs = c("dplyr", "ggplot2"),
  system_pkgs = NULL,
  git_pkgs = NULL,
  ide = "code",
  project_path = path_default_nix,
  overwrite = TRUE,
  print = TRUE
)

This generates a file called default.nix in the path path_default_nix with the correct expression to build this environment. To build the environment, the Nix package manager must be installed. If you have Nix installed, you can build the expression above using the nix-build terminal command and then enter the environment using nix-shell. The vignettes included in the package walk you through the whole workflow.

Quick start for returning users

If you are not familiar with Nix or {rix} skip to the next section.

Click to expand

If you are already familiar with Nix and R, and simply want to get started as quickly as possible, you can start by installing Nix using the installer from Determinate Systems a company that provides services and tools built on Nix:

curl --proto '=https' --tlsv1.2 -sSf \
    -L https://install.determinate.systems/nix | \
     sh -s -- install

You can check that everything works well by trying to build the Nix expression that ships with {rix}. Nix expressions are typically saved into files with the name default.nix or shell.nix. This expression installs the latest version of R and {rix} in a separate, reproducible environment:

file.copy(
  # default.nix is the file containing the Nix expression
  from = system.file("extdata", "default.nix", package = "rix"),
  to = ".", overwrite = TRUE
)

# nix_build() is a wrapper around the command line tool `nix-build`
nix_build(project_path = ".")

If everything worked well, you should see a file called result next to default.nix. You can now enter this newly built development environment by opening a terminal in that folder and typing nix-shell. You should be immediately dropped into an interactive R session.

If you don’t have R installed, but have the Nix package manager installed, you can run a temporary R session with R using this command (it will build the same environment as the one above):

nix-shell --expr "$(curl -sl https://raw.githubusercontent.com/ropensci/rix/main/inst/extdata/default.nix)"

You can then create new development environment definitions, build them, and start using them.

Getting started for new users

To get started with {rix} and Nix, you should read the following vignette vignette("a-getting-started") (online documentation). The vignettes are numbered to get you to learn how to use {rix} and Nix smoothly. There’s a lot of info, so take your time reading the vignettes. Don’t hesitate to open an issue if something is not clear.

Docker

You can also try out Nix inside Docker. To know more, read vignette("z-advanced-topic-using-nix-inside-docker") link.

How is Nix different from Docker+renv/{groundhog}/{rang}/(Ana/Mini)Conda/Guix? or Why Nix?

Docker and renv

Let’s start with arguably the most popular combo for reproducibility in the R ecosystem, Docker+{renv}.

{renv} snapshots the state of the library of R packages for a project, nothing more, nothing less, unless you also use {rspm} or {bspm} in combination to {renv}: this will install the required system-level dependencies automatically. {renv} can then be used to restore the library of packages on another machine, but it is the user’s responsibility to ensure that the right version of R and system-level dependencies are available on that other machine. This is why {renv} is often coupled with a versioned Docker image, such as the images from the Rocker project. Combining both provides a very robust way to serve applications such as Shiny apps, but it can be awkward to develop interactively with this setup, which is why most of the time, people work on their current setup, and dockerize the setup once when they’re done. However, you need to make sure to keep updating the image, as the underlying operating system will eventually reach end of life. Eventually, you might even have to update the whole stack as it could become impossible to install the version of R and R packages you used on a recent Docker image. This can be a good thing actually; it could be the opportunity to update your app and make sure that it benefits from the latest security patches. However for reproducibility in research, this is not something that you should be doing because it could have an impact on historical results.

What we suggest instead, is to keep using Docker if you are already invested in the ecosystem, and continue to use it to deploy and serve applications and archive research. But instead of using {renv} to get the right packages, you combine Docker and Nix. This way, you have a nice separation of concerns: Docker will only be used as a platter to serve code, while the environment will be handled by Nix. You could even use an image that gets continuously updated such as ubuntu:latest as a base: it doesn’t matter that the image is always changing, since the environment that will be doing the heavy lifting inside the container is completely reproducible thanks to Nix.

Exactly the same reasoning can be applied to {groundhog}, {rang} or the CRAN snapshots of Posit in combination to Docker instead of {renv}.

Ana/Mini-conda and Mamba

Anaconda, Miniconda, Mamba, Micromamba… (henceforth we’ll refer to these as Conda) and Nix have much in common: they are multiplatform package managers and both can be used to setup reproducible development environments for many languages, such as R or Python. Using conda-lock one can generate fully reproducible lock files that can then be used by Conda to build the environment as defined in the lock file. The main difference between Conda and Nix is conceptual and might not seem that important for end-users: Conda is a procedural package manager, while Nix is a functional package manager. In practice this means that environments managed by Conda are mutable and users are not prevented from changing their environment interactively, and then re-generate the lock file. This is quite comfortable when working interactively, but can lead to issues where dependency management might get borked.

In the case of Nix however, environments are immutable: you cannot add software into a running Nix environment. You will need to stop working, re-define the environment, rebuild it and then use it. While this might sound more tedious (it is) it forces users to work more “cleanly” and avoids many issues from dynamically changing an environment. If it is not possible to build that environment, it fails as early as possible and forces you to deal with the issue. A mutating environment could lead you into a false sense of safeness.

Another major difference is that Conda does not include the entirety of CRAN nor Bioconductor, which is the case for Nix. According to Anaconda’s Documentation 6000 CRAN packages are available through Conda (as of writing in July 2024, CRAN has 21’000+ packages). Nix also includes almost all of Bioconductor packages, and Conda includes them trough the Bioconda project, however, we were not able to find if Bioconda contains all of Bioconductor. According to Bioconda’s FAQ, Bioconductor data packages are not included.

How is Nix different from Guix?

Just like Nix, Guix is a functional package manager with a focus on reproducible builds. We won’t go into technical differences/similarities, but only to pratical ones for end-users of the R programming language. If you want to know about technical aspects, read this https://news.ycombinator.com/item?id=18910683. The main shortcoming of Guix for R users is that not all CRAN or Bioconductor packages are included, nor is Guix available on Windows or macOS.

Is {rix} all there is?

No, there are other tools that you might want to check out, especially if you want to set up polyglot environments (even though it is possible to use {rix} to set up an environment with R and Python packages for example).

Take a look at https://devenv.sh/ and https://prefix.dev/ if you want to explore other tools that make using Nix easier!

Contributing

Refer to Contributing.md to learn how to contribute to the package.

Please note that this package is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Thanks

Thanks to the Nix community for making Nix possible, and thanks to the community of R users on Nix for their work packaging R and CRAN/Bioconductor packages for Nix (in particular Justin Bedő, Rémi Nicole, nviets, Chris Hammill, László Kupcsik, Simon Lackerbauer, MrTarantoga and every other person from the Matrix Nixpkgs R channel).

Finally, thanks to David Solito for creating {rix}’s logo!

jamesgolick/resource_controller

2024-09-16T01:34:32Z

Rails RESTful controller abstraction plugin.

= Resource Controller

resource_controller makes RESTful controllers easier, more maintainable, and super readable. With the RESTful controller pattern hidden away, you can focus on what makes your controller special.

== Get It

Install it as a plugin:

script/plugin install git://github.com/giraffesoft/resource_controller.git

Or grab the source

git clone git://github.com/giraffesoft/resource_controller.git

= Usage

Creating a basic RESTful controller is as easy as...

class PostsController < ResourceController::Base end

...or if you prefer, you can use the method-call syntax. If you need to inherit from some other class, this syntax is definitely for you:

class PostsController < ApplicationController resource_controller end

Both syntaxes are identical in their behavior. Just make sure you call resource_controller before you use any other r_c functionality in your controller.

Nobody just uses the default RESTful controller, though. resource_controller provides a simple API for customizations.

== Action Lifecycle

It's really easy to make changes to the lifecycle of your actions.

Note: We had to call the new accessor "new_action", since new is somewhat reserved in ruby.

=== Before and After

class ProjectsController < ResourceController::Base

new_action.before do
  3.times { object.tasks.build }
end

create.after do
  object.creator = current_user
end

end

=== Flash

class ProjectsController < ResourceController::Base create.flash "Can you believe how easy it is to use resource_controller? Neither could I!" end

=== respond_to

You can add to what's already there...

class ProjectsController < ResourceController::Base
create.wants.js { render :template => "show.rjs" } end

Or you can create a whole new block. This syntax destroys everything that's there, and starts again...

class ProjectsController < ResourceController::Base
create.response do |wants| wants.html wants.js { render :template => "show.rjs" } end end

If you have a nested resource and want to redirect to the parent after create/update and destroy you can do this in the object controller

class CommentsController < ResourceController::Base belongs_to :post

create.wants.html { redirect_to smart_url(parent_url_options) } 
update.wants.html { redirect_to smart_url(parent_url_options) } 
destroy.wants.html { redirect_to smart_url(parent_url_options) }

end

=== Scoping

Because sometimes you want to make a bunch of customizations at once, most of the helpers accept blocks that make grouping calls really easy. Is it a DSL? Maybe; maybe not. But, it's definitely awesome.

With actions that can fail, the scoping defaults to success. That means that create.flash == create.success.flash.

class ProjectsController < ResourceController::Base

create do
  flash "Object successfully created!"
  wants.js { render :template => "show.rjs" }
  
  failure.wants.js { render :template => "display_errors.rjs" }
end

destroy do
  flash "You destroyed your project.  Good work."
  
  failure do
    flash "You cannot destroy that project.  Stop trying!"
    wants.js { render :template => "display_errors.rjs" }
  end
end

end

== Singleton Resource

If you want to create a singleton RESTful controller inherit from ResourceController::Singleton.

class AccountsController < ResourceController::Singleton end

...or if need to inherit from some other class:

class AccountsController < ApplicationController resource_controller :singleton end

Note: This type of controllers handle a single resource only so the index action and all the collection helpers (collection_url, collection_path...) are not available for them.

Loading objects in singletons is similar to plural controllers with one exception. For non-nested singleton controllers you should override the object method as it defaults to nil for them.

class AccountsController < ResourceController::Singleton private def object @object ||= Account.find(session[:account_id]) end end

In other cases you can use the default logic and override it only if you use permalinks or anything special.

Singleton nesting with both :has_many and :has_one associations is provided...

map.resource :account, :has_many => :options # /account/options, account is a singleton parent map.resources :users, :has_one => :image # /users/1/image, image is a singleton child

If you have the :has_many association with a singleton parent remember to override parent_object for your :has_many controller as it returns nil by default in this case.

class OptionsController < ResourceController::Base belongs_to :account

protected
def parent_object
  Account.find(session[:account_id])
end

end

== Helpers (ResourceController::Helpers)

=== Loading objects

You want to add something like pagination to your controller...

class PostsController < ResourceController::Base private def collection @collection ||= end_of_association_chain.find(:all, :page => {:size => 10, :current => params[:page]}) end end

Or maybe you used a permalink...

class PostsController < ResourceController::Base private def object @object ||= end_of_association_chain.find_by_permalink(param) end end

=== Building objects

Maybe you have some alternative way of building objects...

class PostsController < ResourceController::Base private def build_object @object ||= end_of_association_chain.build_my_object_some_funky_way object_params end end

...and there are tons more helpers in the ResourceController::Helpers

== Nested Resources

Nested controllers can be a pain, especially if routing is such that you may or may not have a parent. Not so with Resource Controller.

class CommentsController < ResourceController::Base belongs_to :post end

All of the finding, and creation, and everything will be done at the scope of the post automatically.

== Namespaced Resources

...are handled automatically, and any namespaces are always available, symbolized, in array form @ ResourceController::Helpers#namespaces

== Polymorphic Resources

Everything, including url generation is handled completely automatically. Take this example...

comment.rb

class Comment belongs_to :commentable, :polymorphic => true end

comments_controller.rb

class CommentsController < ResourceController::Base belongs_to :post, :product, :user end Note: Your model doesn't have to be polymorphic in the ActiveRecord sense. It can be associated in whichever way you want.

routes.rb

map.resources :posts, :has_many => :comments map.resources :products, :has_many => :comments map.resources :users, :has_many => :comments

All you have to do is that, and r_c will infer whichever relationship is present, and perform all the actions at the scope of the parent object.

=== Parent Helpers

You also get some helpers for reflecting on your parent.

parent? # => true/false is there a parent present? parent_type # => :post parent_model # => Post parent_object # => @post

=== Non-standard resource names

resource_controller supports overrides for every non-standard configuration of resources.

The most common example is where the resource has a different name than the associated model. Simply overriding the model_name helper will get resource_controller working with your model.

map.resources :tags ... class PhotoTag < ActiveRecord::Base ... class TagsController < ResourceController::Base private def model_name 'photo_tag' end end

In the above example, the variable, and params will be set to @tag, @tags, and params[:tag]. If you'd like to change that, override object_name.

def object_name 'photo_tag' end

If you're using a non-standard controller name, but everything else is standard, overriding resource_name will propagate through all of the other helpers.

map.resources :tags, :controller => "somethings" ... class Tag < ActiveRecord::Base ... class SomethingsController < ResourceController::Base private def resource_name 'tag' end end

Finally, the route_name helper is used by Urligence to determine which url helper to call, so if you have non-standard route names, override it.

map.resources :tags, :controller => "taggings" ... class Taggings < ActiveRecord::Base ... class TaggingsController < ResourceController::Base private def route_name 'tag' end end

== Url Helpers

Thanks to Urligence, you also get some free url helpers.

No matter what your controller looks like...

[edit_|new_]object_url # is the equivalent of saying [edit_|new_]post_url(@post) [edit_|new_]object_url(some_other_object) # allows you to specify an object, but still maintain any paths or namespaces that are present

collection_url # is like saying posts_url

Url helpers are especially useful when working with polymorphic controllers.

/posts/1/comments

object_url # => /posts/1/comments/#{@comment.to_param} object_url(comment) # => /posts/1/comments/#{comment.to_param} edit_object_url # => /posts/1/comments/#{@comment.to_param}/edit collection_url # => /posts/1/comments

/products/1/comments

object_url # => /products/1/comments/#{@comment.to_param} object_url(comment) # => /products/1/comments/#{comment.to_param} edit_object_url # => /products/1/comments/#{@comment.to_param}/edit collection_url # => /products/1/comments

/comments

object_url # => /comments/#{@comment.to_param} object_url(comment) # => /comments/#{comment.to_param} edit_object_url # => /comments/#{@comment.to_param}/edit collection_url # => /comments

Or with namespaced, nested controllers...

/admin/products/1/options

object_url # => /admin/products/1/options/#{@option.to_param} object_url(option) # => /admin/products/1/options/#{option.to_param} edit_object_url # => /admin/products/1/options/#{@option.to_param}/edit collection_url # => /admin/products/1/options

You get the idea. Everything is automagical! All parameters are inferred.

== Credits

resource_controller was created, and is maintained by {James Golick}[http://jamesgolick.com].

== License

resource_controller is available under the {MIT License}[http://en.wikipedia.org/wiki/MIT_License]

technoweenie/attachment_fu

2024-09-16T01:34:32Z

Treat an ActiveRecord model as a file attachment, storing its patch, size, content type, etc.

attachment-fu

attachment_fu is a plugin by Rick Olson (aka technoweenie http://techno-weenie.net) and is the successor to acts_as_attachment. To get a basic run-through of its capabilities, check out Mike Clark's tutorial http://clarkware.com/cgi/blosxom/2007/02/24#FileUploadFu.

attachment_fu functionality

attachment_fu facilitates file uploads in Ruby on Rails. There are a few storage options for the actual file data, but the plugin always at a minimum stores metadata for each file in the database.

There are four storage options for files uploaded through attachment_fu: File system Database file Amazon S3 Rackspace (Mosso) Cloud Files

Each method of storage many options associated with it that will be covered in the following section. Something to note, however, is that the Amazon S3 storage requires you to modify config/amazon_s3.yml, the Rackspace Cloud Files storage requires you to modify config/rackspace_cloudfiles.yml, and the Database file storage requires an extra table.

attachment_fu models

For all three of these storage options a table of metadata is required. This table will contain information about the file (hence the 'meta') and its location. This table has no restrictions on naming, unlike the extra table required for database storage, which must have a table name of db_files (and by convention a model of DbFile).

In the model there are two methods made available by this plugins: has_attachment and validates_as_attachment.

has_attachment(options = {}) This method accepts the options in a hash: :content_type # Allowed content types. # Allows all by default. Use :image to allow all standard image types. :min_size # Minimum size allowed. # 1 byte is the default. :max_size # Maximum size allowed. # 1.megabyte is the default. :size # Range of sizes allowed. # (1..1.megabyte) is the default. This overrides the :min_size and :max_size options. :resize_to # Used by RMagick to resize images. # Pass either an array of width/height, or a geometry string. :thumbnails # Specifies a set of thumbnails to generate. # This accepts a hash of filename suffixes and RMagick resizing options. # This option need only be included if you want thumbnailing. :thumbnail_class # Set which model class to use for thumbnails. # This current attachment class is used by default. :path_prefix # Path to store the uploaded files in. # Uses public/#{table_name} by default for the filesystem, and just #{table_name} for the S3 and Cloud Files backend.
# Setting this sets the :storage to :file_system. :partition # Whether to partiton files in directories like /0000/0001/image.jpg. Default is true. Only applicable to the :file_system backend. :storage # Specifies the storage system to use.. # Defaults to :db_file. Options are :file_system, :db_file, :s3, and :cloud_files. :cloudfront # If using S3 for storage, this option allows for serving the files via Amazon CloudFront. # Defaults to false. :processor # Sets the image processor to use for resizing of the attached image. # Options include ImageScience, Rmagick, and MiniMagick. Default is whatever is installed. :uuid_primary_key # If your model's primary key is a 128-bit UUID in hexadecimal format, then set this to true. :association_options # attachment_fu automatically defines associations with thumbnails with has_many and belongs_to. If there are any additional options that you want to pass to these methods, then specify them here.

Examples: has_attachment :max_size => 1.kilobyte has_attachment :size => 1.megabyte..2.megabytes has_attachment :content_type => 'application/pdf' has_attachment :content_type => ['application/pdf', 'application/msword', 'text/plain'] has_attachment :content_type => :image, :resize_to => [50,50] has_attachment :content_type => ['application/pdf', :image], :resize_to => 'x50' has_attachment :thumbnails => { :thumb => [50, 50], :geometry => 'x50' } has_attachment :storage => :file_system, :path_prefix => 'public/files' has_attachment :storage => :file_system, :path_prefix => 'public/files', :content_type => :image, :resize_to => [50,50], :partition => false has_attachment :storage => :file_system, :path_prefix => 'public/files', :thumbnails => { :thumb => [50, 50], :geometry => 'x50' } has_attachment :storage => :s3 has_attachment :store => :s3, :cloudfront => true has_attachment :storage => :cloud_files

validates_as_attachment This method prevents files outside of the valid range (:min_size to :max_size, or the :size range) from being saved. It does not however, halt the upload of such files. They will be uploaded into memory regardless of size before validation.

Example: validates_as_attachment

attachment_fu migrations

Fields for attachment_fu metadata tables... in general: size, :integer # file size in bytes content_type, :string # mime type, ex: application/mp3 filename, :string # sanitized filename that reference images: height, :integer # in pixels width, :integer # in pixels that reference images that will be thumbnailed: parent_id, :integer # id of parent image (on the same table, a self-referencing foreign-key). # Only populated if the current object is a thumbnail. thumbnail, :string # the 'type' of thumbnail this attachment record describes.
# Only populated if the current object is a thumbnail. # Usage: # [ In Model 'Avatar' ] # has_attachment :content_type => :image, # :storage => :file_system, # :max_size => 500.kilobytes, # :resize_to => '320x200>', # :thumbnails => { :small => '10x10>', # :thumb => '100x100>' } # [ Elsewhere ] # @user.avatar.thumbnails.first.thumbnail #=> 'small' that reference files stored in the database (:db_file): db_file_id, :integer # id of the file in the database (foreign key)

Field for attachment_fu db_files table: data, :binary # binary file data, for use in database file storage

attachment_fu views

There are two main views tasks that will be directly affected by attachment_fu: upload forms and displaying uploaded images.

There are two parts of the upload form that differ from typical usage.

Include ':multipart => true' in the html options of the form_for tag. Example: <% form_for(:attachment_metadata, :url => { :action => "create" }, :html => { :multipart => true }) do |form| %>
Use the file_field helper with :uploaded_data as the field name. Example: <%= form.file_field :uploaded_data %>

Displaying uploaded images is made easy by the public_filename method of the ActiveRecord attachment objects using file system, s3, and Cloud Files storage.

public_filename(thumbnail = nil) Returns the public path to the file. If a thumbnail prefix is specified it will return the public file path to the corresponding thumbnail. Examples: attachment_obj.public_filename #=> /attachments/2/file.jpg attachment_obj.public_filename(:thumb) #=> /attachments/2/file_thumb.jpg attachment_obj.public_filename(:small) #=> /attachments/2/file_small.jpg

When serving files from database storage, doing more than simply downloading the file is beyond the scope of this document.

attachment_fu controllers

There are two considerations to take into account when using attachment_fu in controllers.

The first is when the files have no publicly accessible path and need to be downloaded through an action.

Example: def readme send_file '/path/to/readme.txt', :type => 'plain/text', :disposition => 'inline' end

See the possible values for send_file for reference.

The second is when saving the file when submitted from a form. Example in view: <%= form.file_field :attachable, :uploaded_data %>

Example in controller: def create @attachable_file = AttachmentMetadataModel.new(params[:attachable]) if @attachable_file.save flash[:notice] = 'Attachment was successfully created.' redirect_to attachable_url(@attachable_file)
else render :action => :new end end

attachement_fu scripting

You may wish to import a large number of images or attachments. The following example shows how to upload a file from a script.

#!/usr/bin/env ./script/runner

required to use ActionController::TestUploadedFile

require 'action_controller' require 'action_controller/test_process.rb'

path = "./public/images/x.jpg"

mimetype is a string like "image/jpeg". One way to get the mimetype for a given file on a UNIX system

mimetype = `file -ib #{path}`.gsub(/\n/,"")

mimetype = "image/jpeg"

This will "upload" the file at path and create the new model.

@attachable = AttachmentMetadataModel.new(:uploaded_data => ActionController::TestUploadedFile.new(path, mimetype)) @attachable.save

kwai/blaze

2024-09-16T01:34:35Z

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.

BLAZE

The Blaze accelerator for Apache Spark leverages native vectorized execution to accelerate query processing. It combines the power of the Apache Arrow-DataFusion library and the scale of the Spark distributed computing framework.

Blaze takes a fully optimized physical plan from Spark, mapping it into DataFusion's execution plan, and performs native plan computation in Spark executors.

Blaze is composed of the following high-level components:

Spark Extension: hooks the whole accelerator into Spark execution lifetime.
Spark Shims: specialized codes for different versions of spark.
Native Engine: implements the native engine in rust, including:
- ExecutionPlan protobuf specification
- JNI gateway
- Customized operators, expressions, functions

Based on the inherent well-defined extensibility of DataFusion, Blaze can be easily extended to support:

Various object stores.
Operators.
Simple and Aggregate functions.
File formats.

We encourage you to extend DataFusion capability directly and add the supports in Blaze with simple modifications in plan-serde and extension translation.

Build from source

To build Blaze, please follow the steps below:

Install Rust

The native execution lib is written in Rust. So you're required to install Rust (nightly) first for compilation. We recommend you to use rustup.

Install JDK+Maven

Blaze has been well tested on jdk8 and maven3.5, should work fine with higher versions.

Check out the source code.

git clone git@github.com:blaze-init/blaze.git
cd blaze

Build the project.

Specify shims package of which spark version that you would like to run on. _Currently we have supported these shims:

spark303 - for spark3.0.x
spark313 - for spark3.1.x
spark324 - for spark3.2.x
spark333 - for spark3.3.x
spark351 - for spark3.5.x.

You could either build Blaze in dev mode for debugging or in release mode to unlock the full potential of Blaze.

SHIM=spark333 # or spark303/spark313/spark320/spark324/spark333/spark351
MODE=release # or pre
mvn package -P"${SHIM}" -P"${MODE}"

After the build is finished, a fat Jar package that contains all the dependencies will be generated in the target directory.

Build with docker

You can use the following command to build a centos-7 compatible release:

SHIM=spark333 MODE=release ./release-docker.sh

Run Spark Job with Blaze Accelerator

This section describes how to submit and configure a Spark Job with Blaze support.

move blaze jar package to spark client classpath (normally spark-xx.xx.xx/jars/).
add the follow confs to spark configuration in spark-xx.xx.xx/conf/spark-default.conf:

spark.blaze.enable true
spark.sql.extensions org.apache.spark.sql.blaze.BlazeSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.blaze.shuffle.BlazeShuffleManager
spark.memory.offHeap.enabled false

# suggested executor memory configuration
spark.executor.memory 4g
spark.executor.memoryOverhead 4096

submit a query with spark-sql, or other tools like spark-thriftserver:

spark-sql -f tpcds/q01.sql

Performance

Check Benchmark Results with the latest date for the performance comparison with vanilla Spark 3.3.3. The benchmark result shows that Blaze save about 50% time on TPC-DS/TPC-H 1TB datasets. Stay tuned and join us for more upcoming thrilling numbers.

TPC-DS Query time: (How can I run TPC-DS benchmark?)

TPC-H Query time:

We also encourage you to benchmark Blaze and share the results with us. 🤗

Community

We're using Discussions to connect with other members of our community. We hope that you:

Ask questions you're wondering about.
Share ideas.
Engage with other community members.
Welcome others who are open-minded. Remember that this is a community we build together 💪 .

License

Blaze is licensed under the Apache 2.0 License. A copy of the license can be found here.

mspnp/azure-databricks-streaming-analytics

2024-09-17T01:36:04Z

Stream processing with Azure Databricks

This reference architecture shows an end-to-end stream processing pipeline. This type of pipeline has four stages: ingest, process, store, and analysis and reporting. For this reference architecture, the pipeline ingests data from two sources, performs a join on related records from each stream, enriches the result, and calculates an average in real time. The results are stored for further analysis.

Scenario: A taxi company collects data about each taxi trip. For this scenario, we assume there are two separate devices sending data. The taxi has a meter that sends information about each ride — the duration, distance, and pickup and dropoff locations. A separate device accepts payments from customers and sends data about fares. To spot ridership trends, the taxi company wants to calculate the average tip per mile driven, in real time, for each neighborhood.

Deploy the solution

A deployment for this reference architecture is available on GitHub.

Prerequisites

Clone, fork, or download this GitHub repository.
Install Docker to run the data generator.
Install Azure CLI 2.0.
Install Databricks CLI.
From a command prompt, bash prompt, or PowerShell prompt, sign into your Azure account as follows:
```
az login
```
Optional - Install a Java IDE, with the following resources:
- JDK 1.8
- Scala SDK 2.12
- Maven 3.6.3
Note: Instructions are included for building via a docker container if you do not want to install a Java IDE.

Download the New York City taxi and neighborhood data files

Create a directory named DataFile in the root of the cloned Github repository in your local file system.
Open a web browser and navigate to https://uofi.app.box.com/v/NYCtaxidata/folder/2332219935.
Click the Download button on this page to download a zip file of all the taxi data for that year.
Extract the zip file to the DataFile directory.

Note: This zip file contains other zip files. Don't extract the child zip files.

The directory structure should look like the following:
```
/DataFile
    /FOIL2013
        trip_data_1.zip
        trip_data_2.zip
        trip_data_3.zip
        ...
```
Open a web browser and navigate to https://www.census.gov/geographies/mapping-files/time-series/geo/cartographic-boundary.html#ti1400387013.
Under the section County Subdivisions click the dropdown an select New York.
Copy the cb_2019_36_cousub_500k.zip file from your browser's downloads directory to the DataFile directory.

Deploy the Azure resources

From a shell or Windows Command Prompt, run the following command and follow the sign-in prompt:
```
az login
```
Navigate to the folder named azure in the GitHub repository directory:
```
cd azure
```

Run the following commands to deploy the Azure resources:

export resourceGroup='[Resource group name]'
export resourceLocation='[Region]'
export eventHubNamespace='[Event Hubs namespace name]'
export databricksWorkspaceName='[Azure Databricks workspace name]'
export cosmosDatabaseAccount='[Cosmos DB database name]'
export logAnalyticsWorkspaceName='[Log Analytics workspace name]'
export logAnalyticsWorkspaceRegion='[Log Analytics region]'

# Create a resource group
az group create --name $resourceGroup --location $resourceLocation

# Deploy resources
az deployment group create --resource-group $resourceGroup \
 --template-file ./deployresources.json --parameters \
 eventHubNamespace=$eventHubNamespace \
    databricksWorkspaceName=$databricksWorkspaceName \
 cosmosDatabaseAccount=$cosmosDatabaseAccount \
 logAnalyticsWorkspaceName=$logAnalyticsWorkspaceName \
 logAnalyticsWorkspaceRegion=$logAnalyticsWorkspaceRegion

The output of the deployment is written to the console once complete. Search the output for the following JSON:

"outputs": {
        "cosmosDb": {
          "type": "Object",
          "value": {
            "hostName": <value>,
            "secret": <value>,
            "username": <value>
          }
        },
        "eventHubs": {
          "type": "Object",
          "value": {
            "taxi-fare-eh": <value>,
            "taxi-ride-eh": <value>
          }
        },
        "logAnalytics": {
          "type": "Object",
          "value": {
            "secret": <value>,
            "workspaceId": <value>
          }
        }
},

These values are the secrets that will be added to Databricks secrets in upcoming sections. Keep them secure until you add them in those sections.

Add a Cassandra table to the Cosmos DB Account

In the Azure portal, navigate to the resource group created in the deploy the Azure resources section above. Click on Azure Cosmos DB Account. Create a table with the Cassandra API.
In the overview blade, click add table.
When the add table blade opens, enter newyorktaxi in the Keyspace name text box.
In the enter CQL command to create the table section, enter neighborhoodstats in the text box beside newyorktaxi.

In the text box below, enter the following:

(neighborhood text, window_end timestamp, number_of_rides bigint, total_fare_amount double, total_tip_amount double, average_fare_amount double, average_tip_amount double, primary key(neighborhood, window_end))

In the Table throughput section confirm that Autoscale is selected and that value 4000 is in the Table Max RU/s text box.
Click OK.

Add the Databricks secrets using the Databricks CLI

Tip: Make sure you have authenticated your Databricks CLI configuration. The simplest method in bash is to run:
export DATABRICKS_AAD_TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d | jq .accessToken --raw-output)
databricks configure --aad-token --host <enter Databricks Workspace URL from Portal>
The resource GUID (2ff814a6-3304-4ab8-85cb-cd0e6f879c1d) is a fixed value. For other options see Set up authentication in the Azure Databricks documentation. If you see a JSONDecodeError error when running a command, your token has exired and you can refresh by running the commands above again.

First, enter the secrets for EventHub:

Using the Azure Databricks CLI installed in step 4 of the prerequisites, create the Azure Databricks secret scope:
```
databricks secrets create-scope --scope "azure-databricks-job"
```
Add the secret for the taxi ride EventHub:
```
databricks secrets put --scope "azure-databricks-job" --key "taxi-ride"
```
Once executed, this command opens the vi editor. Enter the taxi-ride-eh value from the eventHubs output section in step 4 of the deploy the Azure resources section. Save and exit vi (if in edit mode hit ESC, then type ":wq").
Add the secret for the taxi fare EventHub:
```
databricks secrets put --scope "azure-databricks-job" --key "taxi-fare"
```
Once executed, this command opens the vi editor. Enter the taxi-fare-eh value from the eventHubs output section in step 4 of the deploy the Azure resources section. Save and exit vi (if in edit mode hit ESC, then type ":wq").

Next, enter the secrets for Cosmos DB:

Using the Azure Databricks CLI, add the secret for the Cosmos DB user name:
```
databricks secrets put --scope azure-databricks-job --key "cassandra-username"
```
Once executed, this command opens the vi editor. Enter the username value from the CosmosDb output section in step 4 of the deploy the Azure resources section. Save and exit vi (if in edit mode hit ESC, then type ":wq").
Next, add the secret for the Cosmos DB password:
```
databricks secrets put --scope azure-databricks-job --key "cassandra-password"
```
Once executed, this command opens the vi editor. Enter the secret value from the CosmosDb output section in step 4 of the deploy the Azure resources section. Save and exit vi (if in edit mode hit ESC, then type ":wq").

Note: If using an Azure Key Vault-backed secret scope, the scope must be named azure-databricks-job and the secrets must have the exact same names as those above.

Add the Census Neighborhoods data file to the Databricks file system

Create a directory in the Databricks file system:
```
dbfs mkdirs dbfs:/azure-databricks-job
```
Navigate to the DataFile folder and enter the following:
```
dbfs cp cb_2020_36_cousub_500k.zip dbfs:/azure-databricks-job/
```
Note: The filename may change if you obtain a shapefile for a different year.

Build the .jar files for the Databricks job

To build the jars using a docker container from a bash prompt change to the azure directory and run:
```
docker run -it --rm -v `pwd`:/streaming_azuredatabricks_azure -v ~/.m2:/root/.m2 maven:3.6.3-jdk-8 mvn -f /streaming_azuredatabricks_azure/pom.xml package
```
Note: Alternately, use your Java IDE to import the Maven project file named pom.xml located in the azure directory. Perform a clean build.
The outputs of the build is a file named azure-databricks-job-1.0-SNAPSHOT.jar in the ./AzureDataBricksJob/target directory.

Create a Databricks cluster

In the Databricks workspace, click Compute, then click Create cluster. Enter the cluster name you created in step 3 of the configure custom logging for the Databricks job section above.
Select Standard for Cluster Mode.
Set Databricks runtime version to 7.3 Extended Support (Scala 2.12, Apache Spark 3.0.1)
Deselect Enable autoscaling.
Set Worker Type to Standard_DS3_v2.
Set Workers to 2.
Set Driver Type to Same as worker

Optional - Configure Azure Log Analytics
1. Follow the instructions in Monitoring Azure Databricks to build the monitoring library and upload the resulting library files to your workspace.
2. Click on Advanced Options then Init Scripts.
3. Enter dbfs:/databricks/spark-monitoring/spark-monitoring.sh.
4. Click the Add button.
Click the Create Cluster button.

Install dependent libraries on cluster

In the Databricks user interface, click on the home button.
Click on Compute in the navigtation menu on the left then click on the cluster you created in the Create a Databricks cluster step.
Click on Libraries, then click Install New.
In the Library Source control, select Maven.
Under the Maven Coordinates text box, enter com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.21.
Select Install.
Repeat steps 3 - 6 for the com.datastax.spark:spark-cassandra-connector-assembly_2.12:3.0.1 Maven coordinate.
Repeat steps 3 - 5 for the org.geotools:gt-shapefile:23.0 Maven coordinate.
Enter https://repo.osgeo.org/repository/release/ in the Repository text box.
Click Install.

Create a Databricks job

Copy the azure-databricks-job-1.0-SNAPSHOT.jar file to the Databricks file system by entering the following command in the Databricks CLI:
```
databricks fs cp --overwrite AzureDataBricksJob/target/azure-databricks-job-1.0-SNAPSHOT.jar dbfs:/azure-databricks-job/
```
In the Databricks workspace, click "Jobs", "create job".
Enter a job name.
In the Task area, change Type to JAR and Enter com.microsoft.pnp.TaxiCabReader in the Main Class field.
Under Dependent Libraries click Add, this opens the Add dependent library dialog box.
Change Library Source to DBFS/ADLS, confirm that Library Type is Jar and enter dbfs:/azure-databricks-job/azure-databricks-job-1.0-SNAPSHOT.jar in the File Path text box and select Add.

In the Parameters field, enter the following (replace <Cosmos DB Cassandra host name> with a value from above):

["-n","jar:file:/dbfs/azure-databricks-job/cb_2020_36_cousub_500k.zip!/cb_2020_36_cousub_500k.shp","--taxi-ride-consumer-group","taxi-ride-eh-cg","--taxi-fare-consumer-group","taxi-fare-eh-cg","--window-interval","1 hour","--cassandra-host","<Cosmos DB Cassandra host name>"]

Under Cluster, click the drop down arrow and select the cluster created the Create a Databricks cluster section.
Click Create
Select the Runs tab and click Run Now.

Run the data generator

Navigate to the directory onprem in the GitHub repository.
```
cd ../onprem
```
Update the values in the file main.env as follows:
```
RIDE_EVENT_HUB=[Connection string for the taxi-ride event hub]
FARE_EVENT_HUB=[Connection string for the taxi-fare event hub]
RIDE_DATA_FILE_PATH=/DataFile/FOIL2013
MINUTES_TO_LEAD=0
PUSH_RIDE_DATA_FIRST=false
```
The connection string for the taxi-ride event hub is the taxi-ride-eh value from the eventHubs output section in step 4 of the deploy the Azure resources section. The connection string for the taxi-fare event hub the taxi-fare-eh value from the eventHubs output section in step 4 of the deploy the Azure resources section.
Run the following command to build the Docker image.
```
docker build --no-cache -t dataloader .
```
Navigate back to the repository root directory.
```
cd ..
```

Run the following command to run the Docker image.

docker run -v `pwd`/DataFile:/DataFile --env-file=onprem/main.env dataloader:latest

The output should look like the following:

Created 10000 records for TaxiFare
Created 10000 records for TaxiRide
Created 20000 records for TaxiFare
Created 20000 records for TaxiRide
Created 30000 records for TaxiFare
...

Hit CTRL+C to cancel the generation of data.

Verify the solution is running

To verify the Databricks job is running correctly, open the Azure portal and navigate to the Cosmos DB database. Open the Data Explorer blade and examine the data in the neighborhoodstats table, you should see results similar to:

average_fare _amount	average_tip _amount	neighborhood	number_of_rides	total_fare _amount	total_tip _amount	window_end
10.5	1.0	Bronx	1	10.5	1.0	1/1/2013 8:02:00 AM +00:00
12.67	2.6	Brooklyn	3	38	7.8	1/1/2013 8:02:00 AM +00:00
14.98	0.73	Manhattan	52	779	37.83	1/1/2013 8:02:00 AM +00:00
...	...	...	...	...	...	...

[1] Donovan, Brian; Work, Dan (2016): New York City Taxi Trip Data (2010-2013). University of Illinois at Urbana-Champaign. https://doi.org/10.13012/J8PN93H8

ava-labs/avalanche-starter-kit

2024-09-17T01:36:32Z

SikandarJODD/svelte-animations

2024-09-17T01:36:44Z

Svelte Components using Tailwind CSS & Framer Motion

pavel-demin/red-pitaya-notes

2024-09-16T01:35:17Z

Notes on the Red Pitaya Open Source Instrument

aws-amplify/amplify-ui

2024-09-16T01:35:43Z

Amplify UI is a collection of accessible, themeable, performant React (and more!) components that can connect directly to the cloud.

AWS Amplify

Amplify UI

Amplify UI is an open-source UI library with cloud-connected components that are endlessly customizable, accessible, and can integrate into any application. Amplify UI consists of:

Connected components that simplify complex cloud-connected workflows, like Authenticator.
Primitive components that create consistency across Amplify UI and allow you to build complete applications that fit your brand, like Buttons and Badges.
Data-bound components that make it easy to display dynamic data, like DataStoreCollections.
Theming capabilities that allow you to customize the appearance of Amplify UI to match your brand.

Package Name	Downloads	Version
@aws-amplify/ui-react
@aws-amplify/ui-vue
@aws-amplify/ui-angular

Documentation

https://ui.docs.amplify.aws/

Getting started

https://ui.docs.amplify.aws/getting-started/installation

Component Matrix

Connected Components	React	React Native	Angular	Vue
Authenticator	✅	✅	✅	✅
In-App Messaging	✅	✅
Geo (MapView)	✅
Account Settings	✅
Storage (FileUploader)	✅
Storage (StorageImage)	✅
Liveness (FaceLivenessDetector)	✅

Primitives	React	Angular	Vue
Alert	✅
Autocomplete	✅
Badge	✅
Button	✅
Card	✅
CheckboxField	✅
Collection	✅
Divider	✅
Expander	✅
Flex	✅
Grid	✅
Heading	✅
HighlightMatch	✅
Icon	✅
Image	✅
Link	✅
Loader	✅
Menu	✅
Pagination	✅
PasswordField	✅
PhoneNumberField	✅
Placeholder	✅
RadioGroupField	✅
Rating	✅
ScrollView	✅
SearchField	✅
SelectField	✅
SliderField	✅
StepperField	✅
SwitchField	✅
Table	✅
Tabs	✅
Text	✅
TextAreaField	✅
TextField	✅
ToggleButton	✅
View	✅
VisuallyHidden	✅

Frequently asked questions

What are the major benefits of the new version of Amplify UI?

Better developer experience Connected-components like Authenticator are being written with framework-specific implementations so that they follow framework conventions and are easier to integrate into your application.
Endlessly customizable Every detail of Amplify UI is customizable to match your brand. Style all of Amplify UI with themes, override components with your own, or build your own UI and use Amplify for complex state management.
Accessible Amplify UI components follow WCAG and WAI-ARIA best practices and guidelines such as color contrast, keyboard navigation, accessible labels, and focus management.
Primitive components (React only right now) Primitive components are used in the connected components, like Authenticator, you can also customize them and use them to build the rest of your UI.

Why are you building primitives?

We are building more cloud-connected components and want to have consistency between them, while also allowing you to build your UI with the same primitives so you can have a consistent UI development experience.

How does this compare to other UI libraries like Tailwind, Chakra, Supabase, or Material-UI?

Amplify UI consists of both primitive components like Buttons, Badges, and Cards, as well as cloud-connected and data-bound components like the Authenticator. We are taking heavy inspiration from open-source frameworks like Tailwind, Chakra, Supabase, Radix, Adobe Spectrum, Material-UI, and others. In fact, one of the core ideas with the new Amplify UI is the ability to integrate seamlessly into any application, including ones using those UI frameworks. For example, you can use Tailwind classes to style Amplify UI components or Chakra components like buttons inside Amplify connected-components like the Authenticator.

Where should I file bugs and requests?

Bugs and feature requests for Amplify UI

You can also use the above link to report a bug or a feature request for previous version of Amplify UI Components.

As we continue to work on the new Amplify UI we will move UI-related issues in the amplify-js repository over here to work on them. We will continue to maintain major bug and security fixes for all existing UI packages and versions. New development for UI components will happen in this repository and eventually be published under the @react npm tag.

Contributing

CONTRIBUTING.md

jeetsukumaran/vim-pythonsense

2024-09-17T01:37:24Z

A Vim plugin that provides text objects and motions for Python classes, methods, functions, and doc strings.