First of all, everyone of you is welcome to contribute to the project. Whether small changes like typo fixes, simple bug fixes or large feature implementations, every contribution is a step further to make this project way nicer. 😄
Depending on the scale of the contribution, you might need some general understanding of the languages and frameworks used and of some simple development patterns which are applied. But also beginners are absolutely welcome.
Let me give you a quick overview over all modules, structures and used languages/frameworks of the project, so you know what you are working with.
shinpuru is configured using either a YAML or JSON config file which can be passed by -c command line parameter.
Take a look at the example configuration file which holds rich documentation about each configuration key.
For development, you can take the provided my.private.config.yml and enter your credentials. Then, rename it to private.config.yaml, so your secrets will not be commited to the repository by accident.
The backend of shinpuru is completely written in Go (golang). To communicate with Discord, the API wrapper discordgo is used. DiscordGo provides very low level bindings to the Discord API with very little utilities around, therefore a lot of utility packages were created. These can be found in the pkg/ directory. These are the main utility packages used in shinpuru:
- acceptmsg creates an embed message with a Buttons (message components). Then, you can execute code depending on the button pressed.
- discordutil provides general utility functions like getting message links, retrieving objects first from the discordgo cache and, when not available there, from the Discord API or checking if a user has admin privileges.
- embedbuilder helps building embeds using the builder pattern.
- fetch is a widely used package in shinpuru used to get objects like users, members or roles by either ID, name or mention. This is designed to be as fuzzy as possible matching objects to provide a better experience to the user.
Take a look at the packages in the pkg yourself. All of them are as well documented as I was possible to and some also have unit tests where you can see some examples how to use them. 😉
Also, a lot of shared functionalities which require shinpuru specific dependencies are located in the internal/util directory. There you can find some utilities which can be used to access the imagestore, karma system, metrics or votes.
For command handling, shinpuru uses ken. Take a look there and in the examples. As represented there, commands are handled and defined in shinpuru the same way. All command definitions can be found in the internal/slashcommands directory. If you want to add a command, just implement ken's Command interface and take a look how the other commands are implemented to match the conventions applied in the other commands. After that, register the command in the commandhandler InitCommandHandler() function..
Since version 1.17.0, shinpuru switched to dgrs for state and sharding management. Because a Discord Bot needs to fetch a lot of information from the Discord API (like Users, Guilds, Channels, and so on), it would be kind of stupid to do this every time the data is needed. So, every Discord Bot uses a state manager which caches these information after fetching it from the API once. DiscordGo uses a simple internal map structure for that, but because I wanted to have more control over the state management and also, because I wanted to take some load from the Garbage Collector, I've implemented this Redis-based state manager. Here you can find the documentation and more information of it, because it is widely used across shinpuru's code, of course.
Discord event handlers and listeners can be found in the listeners package. A listener is a struct which exposes one or more event handler methods. Listeners must be registered botsession InitDiscordBotSession() function using the session.AddHandler(listeners.NewYourListener(container).Handler) method.
First of all, you can find a Database interface at internal/services/database. This is mainly used to interact with the database. There, you can also find the specific database drivers available, which are currently mysql, sqlite and redis.
shinpuru mainly uses MySQL/MariaDB as database. You can also use SQLite3 for development, but this is not tested anymore and may not be reliable anymore. It is recommended to set up a MariaDB instance on your server or dev system for development. Here you can find some resources how to set up MariaDB on mainly used systems:
- Windows: https://mid.as/kb/00197/install-configure-mariadb-on-windows
- Linux: https://opensource.com/article/20/10/mariadb-mysql-linux
- Docker: https://hub.docker.com/_/mariadb/
Redis is used as database cache. The RedisMiddleware generally inherits functionalities from the specified database middleware instance and only overwrites using the specified functions. The database cache always keeps the cache as well as the database hot and always first tries to get objects from cache and, if not available there, from database.
If you want to add functionalities to the database in your contributions, add the functions to the database interface as well as to the MySQL database driver and, if you need caching, the middleware functions to the redis caching middleware.
If you want to add a column to an existing table, take a look in the migrations implementation. There, you can add a migration function with the SQL statements which will be executed in order to migrate the database structure to the new state. If you add an entirely new table, you don't need to add a migration function. Just add the table definition in the setup() method in the mysql driver.
The
MysqlMiddlewareis very "low level" and directly works with SQL statements instead of using an ORM or something like this. Don't be overwhelmed by the size of the middleware file. Its just because same functionalities are re-used over and over again, which is not very nice, but to be honest, the middleware is very old and I don't find the time to rewrite it and migrate the current database after that.
shinpuru utilizes a simple object storage for storing images and backup files, described by the Storage interface in internal/services/storage. Currently, shinpuru implements two storage drivers: A firect file storage driver and a minio object storage driver, which can also connect to other object storages like Amazon S3 or Google cloud storage.
The web interface communicates with the shinpuru backend over a RESTful HTTP API. Therefore, fiber is used as HTTP framework. Most of the code of the web server is in the internal/services/webserver directory. The web server is split up in Router's and Controller's. Routers are for versioning the API (e.g. /api/v1, /api/v2, ...) and controllers split up the endpoints in different logical sections (e.g. /guilds, /backups, /guilds/:id/members, ...). Also, there are models, which define the object structure of request and response objects as well as some transformation functions, for example to transform a discordgo.Guild object to a models.Guild object.
If you want to add API endpoints, just add the endpoints to one of the controllers (don't forget to register the endpoint in the controller's Setup method!), or create a new entire controller, which then needs to be registered in the API Route. If you need service dependencies in your controller, just add it to the controllers struct and get it from the passed di.Container (more explained below) in the Setup method.
Also, fiber works a lot with middlewares, which can be chained anywhere into the fiber route chain. In shinpuru's implementation, there are three main types of middlewares.
- The high level middlewares like the rate limiter, CORS or file system middleware, which are set before all incomming requests.
- Controller specific middlewares which are defined in the router. Mainly, this is used for the authorization middleware, which checks for auth tokens in the requests. This middleware is required by some controllers and not required for others.
- Endpoint specific middlewares which are defined for specific endpoints only. Mainly, this is used for the permission middleware which checks for required user permissions to execute specific endpoints.
Here you can see a simple overview over the routing structure of the shinpuru webserver.
If you are unfamiliar with the concepts of dependency injection, please read this blog post I have recently written about DI, also with examples in Go. 😉
shinpuru widely uses DI (dependency injection) to share service instances using the package di from sarulabs. It's a really straight forward implementation of a DI container which does not take use of reflection, which makes it quite simple and fast. Also, the di cares about constructing service instances when they are needed and tearing them down when they are no more needed.
The whole service specification happens in the main function of shinpuru in the cmd/shinpuru/main file. For example, the database service initialization looks like following:
diBuilder.Add(di.Def{
Name: static.DiDatabase,
Build: func(ctn di.Container) (interface{}, error) {
return inits.InitDatabase(ctn), nil
},
Close: func(obj interface{}) error {
database := obj.(database.Database)
util.Log.Info("Shutting down database connection...")
database.Close()
return nil
},
})As you can see, all service identifiers are registered in the internal/util/static/di file.
After building the diBuilder, you will have a di.Container to work with where you can get any service registered. Because all services are registered in the App scope, once they are initialized, all requests are getting the same instance of the service. This makes service development very easy, because every service is getting passed the same service container and every service can grab the instance of any other registered service instance.
When you want to use a service, just take it from the passed service container by the specified identifier. Let's take a look at the starboard listener, for example:
func NewListenerStarboard(container di.Container) *ListenerStarboard {
cfg := container.Get(static.DiConfig).(config.Provider)
var publicAddr string
if cfg.WebServer != nil {
publicAddr = cfg.WebServer.PublicAddr
}
return &ListenerStarboard{
db: container.Get(static.DiDatabase).(database.Database),
st: container.Get(static.DiObjectStorage).(storage.Storage),
publicAddr: publicAddr,
}
}As you can see, the NewListenerStarboard function is getting passed the di.Container from somewhere above. Then, the config is taken from the container to resolve the public web server address, if specified. Also, the database as well as the storage service instance is retrieved.
The only thing important to keep in mind is that you should always build your service dependency structure like a tree, and not like a circle. That means, when service A needs service B to be built, service B can not depend on service A on construction.
shinpuru also has an internal LifeCycleTimer which works as job scheduler which is responsible for checking expired votes, cleaning up expired access tokens in the database and creating guild backups.
The package robfig/cron is used to schedule tasks. An instance of cron.Cron is wrapped into CronLifeCycleWrapper so it can be provided via dependency injection using the LifeCycleTimer interface.
This package is using a crontab styled syntax to schedule jobs. Take a look in the InitLCTimer() initializer function to see an example on how the jobs are scheduled.
The shinpuru web frontend is a React SPA, which is directly hosted form the shinpuru web server. The source files are located at /web. The web app uses Yarn for package management and Vite as module bundler. The code is written in TypeScript and uses React Functional Components. For styling, styled-components is used to provide simply, hirachic and highly dynamic component styling. For global state management, zustand is used which is a very simple alternative to state managers like redux based on useState style hooks. The app is localized using i18next. All locales are in the public/locales directory as JSON files for each route. If you need to add or modify text anywhere, there is your starting point. Of course, if you want to translate for more locales, feel free to contribute your translation. 😄
For communication with the API, a custom package has been written (which will soon also be available as NPM package). With the use of the custom useApi hook, you can access the API from anywhere in the app. There are also a lot of more custom hooks available, which you can use. Also, feel free to add your owns if you want.
Translation files for the web app can be found in web/public/locales/. If you want to ad missing translations or add a translation for a new language, you can use the provided merge-langs script. It takes a --base language pack and applies it to the --target language pack by inserting missing language keys from the base pack or adding missing translation files from the base pack to the target pack.
Here is an example how to execute the script.
python3 scripts/merge-langs.py --base web/public/locales/en-US --target web/public/locales/de
If you want to add translations for a new language, just use a new folder as target path.
python3 scripts/merge-langs.py --base web/public/locales/en-US --target web/public/locales/it
There are two main ways to set up a development environment for shinpuru.
First of all, create a fork of this repository.
Then, clone the repository to your PC either using HTTPS, SSH or the Git CLI.
Of course, you need to download and install the Go compiler toolchain. Please follow these instructions to do so.
Also, to compile the web frontend, you need to install NodeJS. Please follow these instructions to do so. Also, you need to install Yarn which is used as package manager for the web app. Therefore, just follow these instructions.
This repository also provides a Makefile with a lot of useful recipies for development. Just enter make help to get a quick overview over all make recipes.
Read this to install GNU Make on Linux and this to install it on Windows.
Now - if not already done - install Docker and Docker Compose on your system. Here you can find a detailed explanation on how to do so on your type of system.
After that, you can simply start up all services required via Docker Compose.
$ docker-compose -f docker-compose.dev.yml up -d
Now, copy the development config template from config/my.private.config.yml to config/private.config.yml and enter your Discord Credentials for your development bot application.
Finally, start the development instance with Make.
$ make run
Alternatively, a Terraform Template for Coder is also provided. If you have a Coder instance, just upload the template and create a workspace based on it.
$ coder template create -d terraform shinpuru && \
coder create --template="shinpuru" shinpuru
All required services are automatically set up in your workspace. You just need to copy the config template from config/coder.private.config.yml to config/private.config.yml, enter your Discord bot application credentials and then, run the dev instance with Make.
$ make run
Now, install the Remote - SSH (ms-vscode-remote.remote-ssh) extension in VSCode and connect to your workspace instance. Here you can read how to do so.
You might need to add some port worwardings to access PhpMyAdmin or the Minio Console.
So, you want to contribute to shinpuru but you don't know what exactly you want to do? Just take a look into the Issue page, there you can find some open bug reports, feature requests or simply idea proposals which are currently open. Just grab one of them you are interested in. You can also leave a comment under the issue that you want to contribute to it. Of course, that is not mandatory in any way, so you don't need to supply a PR after that if you don't have the time or whatever else. 😉
If you have any questions, please hit me on my Dev Discord (zekro#0001) or on Twitter. You can also simply send me an e-mail. 😉