eightball/PluralKit
1
0
Fork 0
mirror of https://github.com/PluralKit/PluralKit.git synced 2026-02-04 13:06:50 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

chore: add config options and update dev-docs

Browse source
This commit is contained in:
asleepyskye 2025-05-02 01:12:33 -04:00
parent 94a3276979
commit a9c3875a95
5 changed files with 133 additions and 78 deletions
Download patch file Download diff file Expand all files Collapse all files

48
dev-docs/README.md
View file

@ -2,11 +2,19 @@
Most of PluralKit's code is written in C#, and is split into 3 projects: `PluralKit.Core` (supporting libraries), `PluralKit.Bot` (the Discord bot), and `PluralKit.API` (ASP.NET webserver with controllers for most API endpoints).
There is an ongoing effort to port this code to Rust, and we have a handful of crates already. Currently, the main Rust services are `gateway` (connects to Discord to receive events), `api` (handles authentication and rate-limiting for the public API, as well as a couple of private endpoints), and `scheduled_tasks` (background cron job runner, for statistics and miscellaneous cleanup).
There is an ongoing effort to port this code to Rust, and we have a handful of crates already.
Currently, the main Rust services are:
- `gateway` - connects to Discord to receive events
- `api` - handles authentication and rate-limiting for the public API, as well as a couple of private endpoints
- `scheduled_tasks` - background cron job runner, for statistics and miscellaneous cleanup.
- `avatars` - handles avatar storage and cleanup
- `dispatch` - dispatches webhook events
Additionally, `libpk` handles runtime configuration and database functions.
At the very least, `PluralKit.Bot` and `gateway` are required for the bot to run. While code still exists to connect to the Discord gateway directly from the C# bot, this is no longer a supported configuration and may break in the future.
Service-specific documentation can be found for the C# services in [dotnet.md](./dotnet.md), and for the Rust services in [rust.md](./rust.md) (todo; there is a temporary mostly-accurate document at [RUNNING.md](./RUNNING.md)).
Service-specific documentation can be found for the C# services in [dotnet.md](./dotnet.md), and for the Rust services in [rust.md](./rust.md).
## Building/running
@ -16,6 +24,42 @@ The production instance of PluralKit uses Docker images built in CI. These take
PluralKit services are configured with environment variables; see service-specific documentation for details. Generally, the configuration from the self-host `docker-compose.yml` should get you started.
### process-compose basic steps
Your .env should contain at least the following for the bot to run (see the C#/Rust service specific docs for more on configuration):
```
RUST_LOG="info"
pluralkit__discord__bot_token="<BOT_TOKEN_HERE>"
pluralkit__discord__client_id="<CLIENT_ID_HERE>"
pluralkit__db__db_password="postgres"
pluralkit__db__data_db_uri="postgresql://postgres@localhost:5432/pluralkit"
pluralkit__db__data_redis_addr="redis://localhost:6379"
pluralkit__discord__client_secret=1
pluralkit__discord__max_concurrency=1
pluralkit__runtime_config_key="gateway"
PluralKit__Bot__Token="<BOT_TOKEN_HERE>"
PluralKit__Bot__Client="<CLIENT_ID_HERE>"
PluralKit__Database="Host=localhost;Port=5432;Username=postgres;Password=postgres;Database=pluralkit"
PluralKit__RedisAddr="127.0.0.1:6379"
PluralKit__Bot__DisableGateway="true"
PluralKit__Bot__EventAwaiterTarget="http://127.0.0.1:5002/events"
PluralKit__Bot__HttpListenerAddr="127.0.0.1"
PluralKit__Bot__HttpCacheUrl="127.0.0.1:5000"
```
1. Clone the repository: `git clone https://github.com/PluralKit/PluralKit`
2. Create a `.env` configuration file in the `PluralKit` directory *(see above)*
3. Build and run: `nix run .#dev`
- This will download the dependencies, build, and run PluralKit
- If Nix is not setup to allow flakes, you may need to add `--extra-experimental-features nix-command --extra-experimental-features flakes` to the command
- If the `pluralkit-bot` process fails to run, you can restart it by selecting it and pressing `Ctrl-R`
```
[nix-shell:~]$ git clone https://github.com/PluralKit/PluralKit
[nix-shell:~]$ cd PluralKit
[nix-shell:~/PluralKit]$ nano .env
[nix-shell:~/PluralKit]$ nix run .#dev
```
## Upgrading database from legacy version
If you have an instance of the Python version of the bot (from the `legacy` branch), you may need to take extra database migration steps.
For more information, see [LEGACYMIGRATE.md](./LEGACYMIGRATE.md).

71
dev-docs/RUNNING.md
View file

@ -1,71 +0,0 @@
# Technical Overview:
PluralKit is composed of several different parts, some of which optional and not needed for the bot to function in a testing environment.
##### Required:
- Bot (*PluralKit.Bot*)
- Gateway
- PostgreSQL Database
- Redis Database
##### Optional:
- API (*PluralKit.API*)
- Scheduled Tasks (*scheduled_tasks*)
*Optionally, it can also integrate with Sentry for error reporting, and InfluxDB for aggregate statistics. In production, we use [VictoriaMetrics](https://victoriametrics.com/) InfluxDB competible endpoint, to query metrics in a Prometheus-compatible format.
The bot and API are built using .NET 8, with other optional components used for scaling (ex. scheduled_tasks) built using Rust.
# Building + Running
**The below procedures are intended for development and testing use only!
Newer versions of the bot have moved to using [Nix](https://nixos.org/) for simplified builds. A docker compose file is also provided, but not recommended as it is not actively maintained.
See [Configuration](./CONFIGURATION.md) for full details on configuration as only the basics will be covered here.
Configuration is done through a JSON configuration file `pluralkit.conf` placed in the bot's working directory. An example of the configuration format can be seen in [pluralkit.conf.example](pluralkit.conf.example).
The minimum configuration needed for the bot to function must include the following:
- **`PluralKit.Bot.Token`**: the Discord bot token to connect with
- **`PluralKit.Bot.ClientId`**: the ID of the bot's user account, used for calculating the bot's own permissions and for the link in `pk;invite`
- **`PluralKit.Database`**: the URI of the PostgreSQL database to connect to (in [ADO.NET Npgsql format](https://www.connectionstrings.com/npgsql/))
- **`PluralKit.RedisAddr`**: the `host:port` of the Redis database to connect to
**When running using Docker, you do not need to specify the Postgres or Redis URLs as these will be overwritten by environment variables in the compose file.**
**When using Nix, the Database URI Username, Password, and Database fields must match what the database was setup with in the `flake.nix` file!**
The bot can also take configuration from environment variables, which will override the values read from the file. Here, use `:` (colon) or `__` (double underscore) as a level separator (eg. `export PluralKit__Bot__Token=foobar123`) as per [ASP.NET config](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-3.1#environment-variables).
## Nix (recommended)
The bot, databases, and services are available to run all in one as a Nix flake (using process-compose-flake).
As of the writing of this document, there are a few caveats with the current flake file.
- The database URI in the config must match the username, password, and database specified during database creation (currently `postgres`, `postgres`, and `pluralkit` respectively).
- Not all services are added to the flake file yet.
**Additionally, as of the writing of this document, the `pluralkit-gateway` service reads environment variables to get the bot token and database URLs, so you also need to create a `.env` file in the `PluralKit` directory with the following variables:**
```
pluralkit__db__db_password="postgres"
pluralkit__db__data_db_uri="postgresql://postgres@localhost:5432/pluralkit"
pluralkit__db__data_redis_addr="redis://localhost:6379"
pluralkit__discord__bot_token="BOT_TOKEN_GOES_HERE"
pluralkit__discord__client_id="BOT_CLIENT_ID_GOES_HERE"
pluralkit__discord__client_secret=1
pluralkit__discord__max_concurrency=1
pluralkit__runtime_config_key="gateway"
```
**(This should match the username/password/database specified in the flake file and the configuration file AND the redis address in the configuration file should be changed to match this)**
*(assuming you already have Git installed, if not, you can start a shell with git by running `nix-shell -p git`)*
1. Clone the repository: `git clone https://github.com/PluralKit/PluralKit`
2. Create a `pluralkit.conf` configuration file in the `PluralKit` directory
- Again, the DB URI parameters must match what's in the `flake.nix` file
- Also again, the Redis address must match what is in the .env file created next
3. Create a `.env` configuration file in the `PluralKit` directory *(see above)*
4. Build and run: `nix run .#dev`
- This will download the dependencies, build, and run PluralKit
- If Nix is not setup to allow flakes, you may need to add `--extra-experimental-features nix-command --extra-experimental-features flakes` to the command
- If the `pluralkit-bot` process fails to run, you can restart it by selecting it and pressing `Ctrl-R`
```
[nix-shell:~]$ git clone https://github.com/PluralKit/PluralKit
[nix-shell:~]$ cd PluralKit
[nix-shell:~/PluralKit]$ cp pluralkit.conf.example pluralkit.conf
[nix-shell:~/PluralKit]$ nano pluralkit.conf
[nix-shell:~/PluralKit]$ nano .env
[nix-shell:~/PluralKit]$ nix run .#dev
```

40
dev-docs/dotnet.md Normal file
View file

@ -0,0 +1,40 @@
## Configuration
Configuration was previously done through a JSON configuration file `pluralkit.conf` placed in the bot's working directory. To simplify things however and maintain consistency with the Rust services, it is now recommended to use environment variables.
The minimum configuration needed for the dotnet part of the bot to function include the following:
- **`PluralKit__Bot__Token`**: the Discord bot token to connect with
- **`PluralKit__Bot__ClientId`**: the ID of the bot's user account, used for calculating the bot's own permissions and for the link in `pk;invite`
- **`PluralKit__Database`**: the URI of the PostgreSQL database to connect to (in [ADO.NET Npgsql format](https://www.connectionstrings.com/npgsql/))
- **`PluralKit__RedisAddr`**: the `host:port` of the Redis database to connect to
**When using Nix, the Database URI Username, Password, and Database fields must match what the database was setup with in the `flake.nix` file!**
### Available Configuration Values:
| Name | Description | Rust Equivalent (if applicable) |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------- |
| **`PluralKit__Api__ClientId`** | the ID of the bot's user account, used for calculating the bot's own permissions and for the link in `pk;invite`. | *`pluralkit__discord__client_id`* |
| **`PluralKit__Api__ClientSecret`** | the client secret of the application, used for OAuth with Discord | *`pluralkit__discord__client_secret`* |
| **`PluralKit__Api__TrustAuth`** | | |
| **`PluralKit__Bot__AdminRole`** | Discord role ID used to determine if a user can use `pk;admin` commands | |
| **`PluralKit__Bot__AvatarServiceUrl`** | the URL of the avatar service | |
| **`PluralKit__Bot__ClientId`** | the ID of the bot's user account, used for calculating the bot's own permissions and for the link in `pk;invite`. | *`pluralkit__discord__client_id`* |
| **`PluralKit__Bot__Cluster__TotalShards`** | the total number of shards | *`pluralkit__discord__cluster__total_shards`* |
| **`PluralKit__Bot__DisableGateway`** | (deprecated) boolean used to enable or disable the inbuilt gateway functions, should be true if using Rust `gateway` | |
| **`PluralKit__Bot__DiscordBaseUrl`** | the base Discord API url used for HTTP API requests | *`pluralkit__discord__api_base_url`* |
| **`PluralKit__Bot__EventAwaiterTarget`** | the target bind address used to receive events from `gateway` over http | |
| **`PluralKit__Bot__HttpCacheUrl`** | the URL of the http cache to use, as of now, the `gateway` service | |
| **`PluralKit__Bot__HttpListenerAddr`** | the base bind address | |
| **`PluralKit__Bot__Token`** | the Discord bot token to connect with | *`pluralkit__discord__bot_token`* |
| **`PluralKit__Database`** | the URI of the database to connect to (in [ADO.NET Npgsql format](https://www.connectionstrings.com/npgsql/)) | *`pluralkit__db__data_db_uri`* (diff formatting) |
| **`PluralKit__MessagesDatabase`** | the URI of the database for message storage to connect to (in [ADO.NET Npgsql format](https://www.connectionstrings.com/npgsql/)) | *`pluralkit__db__messages_db_uri`* (diff formatting) |
| **`PluralKit__RedisAddr`** | the `host:port` of a Redis database to connect to | *`pluralkit__db__data_redis_addr`* (diff formatting) |
| **`PluralKit__DispatchProxyToken`** | the token used to authenticate with the dispatch proxy service | |
| **`PluralKit__DispatchProxyUrl`** | the URL of the dispatch proxy service | |
| **`PluralKit__ConsoleLogLevel`** | the minimum [log level](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.logging.loglevel?view=net-9.0-pp) used for logging | |
| **`PluralKit__InfluxUrl`** | the URL to an [InfluxDB](https://www.influxdata.com/products/influxdb-overview/) server to report aggregate statistics to. An example of these stats can be seen on [the public stats page](https://stats.pluralkit.me). | |
| **`PluralKit__InfluxDb`** | the name of an InfluxDB database to report statistics to. If either this field or `PluralKit.InfluxUrl` are absent, InfluxDB reporting will be disabled. | |
| **`PluralKit__SentryUrl`** | the [Sentry](https://sentry.io/welcome/) client key/DSN to report runtime errors to. If absent, disables Sentry integration. | |

47
dev-docs/rust.md Normal file
View file

@ -0,0 +1,47 @@
## Services Overview
TODO: write more about what each service does (and quirks)
## Configuration
Configuration is done through environment variables. A potentially uncompleted and/or outdated list is as follows:
#### Key:
- G - gateway
- A - api
- ST - scheduled_tasks
- AV - avatars
- D - dispatch
| Used by: | Name | Description |
| --------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| G, ST | **`pluralkit__discord__bot_token`** | the Discord bot token to connect with |
| G, A | **`pluralkit__discord__client_id`** | the ID of the bot's user account, used for calculating the bot's own permissions and for the link in `pk;invite`. |
| A | **`pluralkit__discord__client_secret`** | the client secret of the application, used for OAuth with Discord |
| G | **`pluralkit__discord__cluster__total_shards`** | the total number of shards |
| G | **`pluralkit__discord__cluster__total_nodes`** | the total number of clusters |
| G | **`pluralkit__discord__cluster__node_id`** | the ID of the cluster -- overwritten in libpk on run for k8s configuration |
| G | **`pluralkit__discord__max_concurrency`** | number of identify requests per 5 seconds -- see Discord docs |
| G, ST | **`pluralkit__discord__api_base_url`** | the base Discord API url used for HTTP API requests |
| G, A, ST, AV | **`pluralkit__db__data_db_uri`** | the URI of the PostgreSQL data database in [libpq format](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) |
| G, A, ST, AV | **`pluralkit__db__data_redis_addr`** | the address of the Redis instance, in [standard Redis format](https://redis.io/docs/latest/develop/clients/nodejs/connect/) |
| G, A, ST, AV | **`pluralkit__db__db_password`** | the password to use for PostgreSQL database connections |
| G, A, ST, AV | **`pluralkit__db__messages_db_uri`**<br> | the URI of the PostgreSQL messages database in [libpq format](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) |
| G, A, ST, AV | **`pluralkit__db__stats_db_uri`** | the URI of the PostgreSQL statistics database in [libpq format](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) |
| ST | **`pluralkit__scheduled_tasks__expected_gateway_count`** | the total number of expected running gateway instances |
| ST | **`pluralkit__scheduled_tasks__gateway_url`** | the base URL used for querying statistics from gateway instances |
| ST | **`pluralkit__scheduled_tasks__set_guild_count`** | boolean used to determine if the guild count should be updated in Redis for the bot status |
| A | **`pluralkit__api__addr`** | the bind address used for the Rust API |
| A | **`pluralkit__api__ratelimit_redis_addr`** | the address of a Redis instance to use for request ratelimiting |
| A | **`pluralkit__api__remote_url`** | the remote url of the dotnet API instance |
| A | **`pluralkit__api__temp_token2`** | the token used in the API for fetching app authorization |
| AV | **`pluralkit__avatars__cdn_url`** | the CDN address used for avatar storage |
| AV | **`pluralkit__avatars__cloudflare_token`** | the Cloudflare token to use for avatar cache cleanup |
| AV | **`pluralkit__avatars__cloudflare_zone_id`** | the Cloudflare zone id to use for avatar cache cleanup |
| AV | **`pluralkit__avatars__s3__application_id`** | the application id of the s3 instance to use for avatar storage |
| AV | **`pluralkit__avatars__s3__application_key`** | the application key of the s3 instance to use for avatar storage |
| AV | **`pluralkit__avatars__s3__bucket`** | the bucket to use for avatar storage |
| AV | **`pluralkit__avatars__s3__endpoint`** | the endpoint URL of the s3 instance to use for avatar storage |
| G, A, ST, AV, D | **`pluralkit__json_log`** | boolean used to enable or disable JSON log formatting |
| G | **`pluralkit__runtime_config_key`** | key used to determine runtime platform (used in prod, if dev/self-hosting, use 'gateway') |
| G, A, ST, AV, D | **`pluralkit__run_metrics_server`** | boolean used to enable or disable the inbuilt Prometheus format metrics server |
| G, A, ST, AV, D | **`pluralkit__sentry_url`** | the URL of a sentry instance to publish errors to |

5
flake.nix
View file

@ -103,9 +103,6 @@
process-compose."dev" = let
dataDir = ".nix-process-compose";
pluralkitConfCheck = ''
[[ -f "pluralkit.conf" ]] || (echo "pluralkit config not found, please copy pluralkit.conf.example to pluralkit.conf and edit it" && exit 1)
'';
sourceDotenv = ''
[[ -f ".env" ]] && echo "sourcing .env file..." && export "$(xargs < .env)"
'';
@ -159,7 +156,6 @@
text = ''
${sourceDotenv}
set -x
${pluralkitConfCheck}
exec cargo build --bin ${name}
'';
};
@ -177,7 +173,6 @@
text = ''
${sourceDotenv}
set -x
${pluralkitConfCheck}
exec ${mkBotEnv "dotnet build -c Release -o obj/"}/bin/env
'';
};